Batching with missing data

@tom-andersson et al. I wonder if you can help clear up some difficulty that @MartinSJRogers and I are having with batched training for gridded data with missing values using deepsensor.
As yet I'm unable to work out whether we're doing something incorrectly or whether there are bugs in deepsensor's implementation.
We're working with gridded data with missing values represented as NaNs as specified in the Data Requirements section of the docs.
When setting a batch_size during training, concat_tasks is called, in the below snippet, the remove_target_nans() method is called:
https://github.com/alan-turing-institute/deepsensor/blob/aeccc097699c5963fa81b4601cfa11fad5daa41b/deepsensor/data/task.py#L477-L489
This results in a ValueError raised later in concat_tasks since there are different numbers of targets in each batch:
ValueError: All tasks must have the same number of targets to concatenate: got [9460279, 10432117, 8255541, 10345501]. To train with Task batches containing differing numbers of targets, run the model individually over each task and average the losses.

and as a result we don't get to the calls to mask_nans_numpy and mask_nans_nps towards the end of concat_tasks.
I'm confused by this, since from the message above "Cannot concatenate tasks that have had NaNs masked. " "Masking will be applied automatically after concatenation." and the later call to mask_nans_{numpy,nps} it seems like this should be handled by those methods.
When I remove the call to remove_target_nans here for testing, of course the batch sizes are the same and the ValueError above isn't raised, the rest of concat_tasks runs and mask_nans_{numpy,nps} are called successfully.
This, however, results in an error further down the line in which the Masked object from neuralprocesses is found not to have the dtype attribute:

Full stack trace
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[12], line 15
     13 trainer = Trainer(model, lr=5e-5)
     14 for epoch in tqdm(range(1)):
---> 15     batch_losses = trainer(train_tasks, tqdm_notebook=True, batch_size=None) # error here due to filesize. I have attempted using batch_size = n,
     16                                         # but get seperate error asserting that the number of targets in each batch must be the same.
     17                                         # Todo- work out how to calculate number of targets in each batch, and ensure the batch size allows me to honour this assertion.
     18     losses.append(np.mean(batch_losses))

File _/deepsensor/train/train.py:177, in Trainer.__call__(self, tasks, batch_size, progress_bar, tqdm_notebook)
    170 def __call__(
    171     self,
    172     tasks: List[Task],
   (...)
    175     tqdm_notebook=False,
    176 ) -> List[float]:
--> 177     return train_epoch(
    178         model=self.model,
    179         tasks=tasks,
    180         batch_size=batch_size,
    181         opt=self.opt,
    182         progress_bar=progress_bar,
    183         tqdm_notebook=tqdm_notebook,
    184     )

File _/deepsensor/train/train.py:145, in train_epoch(model, tasks, lr, batch_size, opt, progress_bar, tqdm_notebook)
    143     else:
    144         task = tasks[batch_i]
--> 145     batch_loss = train_step(task)
    146     batch_losses.append(batch_loss)
    148 return batch_losses

File _/deepsensor/train/train.py:116, in train_epoch.<locals>.train_step(tasks)
    114 task_losses = []
    115 for task in tasks:
--> 116     task_losses.append(model.loss_fn(task, normalise=True))
    117 mean_batch_loss = B.mean(B.stack(*task_losses))
    118 mean_batch_loss.backward()

File _/deepsensor/model/convnp.py:869, in ConvNP.loss_fn(self, task, fix_noise, num_lv_samples, normalise)
    865 task = ConvNP.modify_task(task)
    867 context_data, xt, yt, model_kwargs = convert_task_to_nps_args(task)
--> 869 logpdfs = backend.nps.loglik(
    870     self.model,
    871     context_data,
    872     xt,
    873     yt,
    874     **model_kwargs,
    875     fix_noise=fix_noise,
    876     num_samples=num_lv_samples,
    877     normalise=normalise,
    878 )
    880 loss = -B.mean(logpdfs)
    882 return loss

File _conda-envs/deepsensor/lib/python3.11/site-packages/plum/function.py:399, in Function.__call__(self, *args, **kw_args)
    397 def __call__(self, *args, **kw_args):
    398     method, return_type = self._resolve_method_with_cache(args=args)
--> 399     return _convert(method(*args, **kw_args), return_type)

File _conda-envs/deepsensor/lib/python3.11/site-packages/neuralprocesses/model/loglik.py:113, in loglik(model, *args, **kw_args)
    110 @_dispatch
    111 def loglik(model: Model, *args, **kw_args):
    112     state = B.global_random_state(B.dtype(args[-2]))
--> 113     state, logpdfs = loglik(state, model, *args, **kw_args)
    114     B.set_global_random_state(state)
    115     return logpdfs

File _conda-envs/deepsensor/lib/python3.11/site-packages/plum/function.py:399, in Function.__call__(self, *args, **kw_args)
    397 def __call__(self, *args, **kw_args):
    398     method, return_type = self._resolve_method_with_cache(args=args)
--> 399     return _convert(method(*args, **kw_args), return_type)

File _conda-envs/deepsensor/lib/python3.11/site-packages/neuralprocesses/model/loglik.py:48, in loglik(state, model, contexts, xt, yt, num_samples, batch_size, normalise, fix_noise, dtype_lik, **kw_args)
     12 @_dispatch
     13 def loglik(
     14     state: B.RandomState,
   (...)
     25     **kw_args,
     26 ):
     27     \"\"\"Log-likelihood objective.
     28
     29     Args:
   (...)
     46         tensor: Log-likelihoods.
     47     \"\"\"
---> 48     float = B.dtype_float(yt)
     49     float64 = B.promote_dtypes(float, np.float64)
     51     # For the likelihood computation, default to using a 64-bit version of the data
     52     # type of `yt`.

File _conda-envs/deepsensor/lib/python3.11/site-packages/plum/function.py:399, in Function.__call__(self, *args, **kw_args)
    397 def __call__(self, *args, **kw_args):
    398     method, return_type = self._resolve_method_with_cache(args=args)
--> 399     return _convert(method(*args, **kw_args), return_type)

File _conda-envs/deepsensor/lib/python3.11/site-packages/lab/types.py:342, in dtype_float(x)
    340 @dispatch
    341 def dtype_float(x):
--> 342     return dtype_float(dtype(x))

File _conda-envs/deepsensor/lib/python3.11/site-packages/plum/function.py:399, in Function.__call__(self, *args, **kw_args)
    397 def __call__(self, *args, **kw_args):
    398     method, return_type = self._resolve_method_with_cache(args=args)
--> 399     return _convert(method(*args, **kw_args), return_type)

File _conda-envs/deepsensor/lib/python3.11/site-packages/lab/types.py:236, in dtype(a)
    226 @dispatch
    227 def dtype(a):
    228     \"\"\"Determine the data type of an object.
    229
    230     Args:
   (...)
    234         dtype: Data type of `a`.
    235     \"\"\"
--> 236     return a.dtype

AttributeError: 'Masked' object has no attribute 'dtype'"

Are we doing something incorrectly here? Or are there bugs in the implementation. Happy to add more docs when we've worked out what we're doing or contribute bug fixes if required!
Lastly, for non-batched training, are mask_nans_{numpy,nps} run somewhere else? I notice that they're called in modify_task but I'm not yet sure when this is called.

@davidwilby Not sure if this helps, I ran into something similar a while ago where the target sets have different number of targets across tasks while batching, and I adapted the concat_tasks function to randomly subsample the targets to a common batch size:
for target_set_i in range(n_target_sets):
    # Raise error if target sets have different numbers of targets across tasks
    n_target_obs = [task["Y_t"][target_set_i].size for task in tasks]
    if not all([n == n_target_obs[0] for n in n_target_obs]):
        # for this target set adapt the number of observations across tasks to min_n_target_obs
        shapes = [task["Y_t"][target_set_i].shape[-1] for task in tasks]
        min_n = min(shapes)

        for task in tasks:
            rand_indices = np.random.choice(
                np.arange(task["Y_t"][target_set_i].shape[-1]),
                size=min_n,
                replace=False,
            )
            task["Y_t"][target_set_i] = task["Y_t"][target_set_i][..., rand_indices]
            task["X_t"][target_set_i] = task["X_t"][target_set_i][..., rand_indices]

Not sure if this helps, but I would agree that it would be nice to include some functionality that handles this for batched training as this can happen quiet frequently.

Hi @davidwilby + @MartinSJRogers, thank you for raising this :) this boils down to a few things:

We can have missing data/NaNs on the context side because ConvNPs represent missing data as zeros in the density channels of the context set encodings. This is handled by neuralprocesses.Masked objects which are constructed by the Task.mask_nans_{numpy,nps} methods you mentioned.
However, we can't have NaNs in target values because backpropagation would fail.
We can remove NaNs from target sets, and NP models can happily train on varying-length target arrays when this happens.
However, we can't concatenate varying-length arrays into a single array (and there is no support in deepsensor/neuralprocesses for padding the target arrays and then masking the padded values from the loss).
Therefore we can't run in batch mode if there are missing values in the targets.

The workaround, as suggested in the error message, is to manually run the model multiple times in a for loop over your 'batch', and then average the losses within your model update. This gives you the smoother loss surface of batch training, but unfortunately it doesn't give you the computational efficiency of running on multiple examples in parallel on a GPU.
@nilsleh's workaround of subsampling to the smallest number of targets is a nice idea, although the model will see fewer target points per batch than it would otherwise, so this is a trade-off between computational efficiency and learning efficiency. If the number of non-missing target points are similar between all Tasks, which looks to be the case from [9460279, 10432117, 8255541, 10345501], then it might not be a bad shout.

When I remove the call to remove_target_nans here for testing, of course the batch sizes are the same and the ValueError above isn't raised, the rest of concat_tasks runs and mask_nans_{numpy,nps} are called successfully. This, however, results in an error further down the line in which the Masked object from neuralprocesses is found not to have the dtype attribute:

neuralprocesses stack traces can be confusing and the dtype error isn't clear, but you can't have neuralprocesses.Masked objects in the targets. The targets need to be vanilla tensors. Only context data can have neuralprocesses.Masked objects, and the missing data will be dealt with under the hood, as mentioned above.
Hope this clears things up, and please close if so :)

I have a related question to NaNs in target sets, which is the case for the data that I am working with. If I don't modify anything and use the provided Trainer code as such:
trainer = Trainer(model, lr=5e-5)
batch_losses = trainer(train_tasks, batch_size=None)

A single task looks like this in the loss_fn computation after the modify_task call here
time: Timestamp/2013-06-15 12:00:00
ops: ['str/batch_dim', 'str/float32', 'str/numpy_mask', 'str/nps_mask', 'str/tensor']
X_c: ['Tensor/torch.float32/torch.Size([1, 2, 256000])', 'Tensor/torch.float32/torch.Size([1, 2, 256000])', 'Tensor/torch.float32/torch.Size([1, 2, 256000])']
Y_c: ['Masked/(y=torch.float32/torch.Size([1, 1, 256000]))/(mask=torch.float32/torch.Size([1, 1, 256000]))', 'Masked/(y=torch.float32/torch.Size([1, 1, 256000]))/(mask=torch.float32/torch.Size([1, 1, 256000]))', 'Tensor/torch.float32/torch.Size([1, 1, 256000])']
X_t: ['Tensor/torch.float32/torch.Size([1, 2, 224000])']
Y_t: ['Masked/(y=torch.float32/torch.Size([1, 1, 224000]))/(mask=torch.float32/torch.Size([1, 1, 224000]))']

And I get the error: AttributeError: 'Masked' object has no attribute 'dtype'
If I change the loss function to remove the targets before modifying the task by adding task.remove_target_nans(), while keeping batch_size=None or just changing the trainer batch_size>2 because then remove_target_nans() is called in concat_tasks a single task looks like this:
time: Timestamp/2013-06-15 12:00:00
ops: ['str/target_nans_removed', 'str/batch_dim', 'str/float32', 'str/numpy_mask', 'str/nps_mask', 'str/tensor']
X_c: ['Tensor/torch.float32/torch.Size([1, 2, 256000])', 'Tensor/torch.float32/torch.Size([1, 2, 256000])', 'Tensor/torch.float32/torch.Size([1, 2, 256000])']
Y_c: ['Masked/(y=torch.float32/torch.Size([1, 1, 256000]))/(mask=torch.float32/torch.Size([1, 1, 256000]))', 'Masked/(y=torch.float32/torch.Size([1, 1, 256000]))/(mask=torch.float32/torch.Size([1, 1, 256000]))', 'Tensor/torch.float32/torch.Size([1, 1, 256000])']
X_t: ['Tensor/torch.float32/torch.Size([1, 2, 215843])']
Y_t: ['Tensor/torch.float32/torch.Size([1, 1, 215843])']

but then I get the neuralprocess library error: AssertionError: Expected not a parallel of elements, but got inputs and outputs in parallel.

Full Stacktrace

File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/deepsensor/train/train.py", line 116, in train_step
    task_losses.append(model.loss_fn(task, normalise=True))
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/deepsensor/model/convnp.py", line 870, in loss_fn
    logpdfs = backend.nps.loglik(
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/neuralprocesses/model/loglik.py", line 113, in loglik
    state, logpdfs = loglik(state, model, *args, **kw_args)
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/neuralprocesses/model/loglik.py", line 64, in loglik
    state, pred = model(
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/plum/function.py", line 489, in __call__
    return self._f(self._instance, *args, **kw_args)
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/neuralprocesses/model/model.py", line 101, in __call__
    return self(
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/plum/function.py", line 489, in __call__
    return self._f(self._instance, *args, **kw_args)
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/neuralprocesses/model/model.py", line 72, in __call__
    xz, pz = code(self.encoder, xc, yc, xt, root=True, **enc_kw_args)
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/neuralprocesses/__init__.py", line 21, in f_wrapped
    return f(*args, **kw_args)
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/neuralprocesses/coders/functional.py", line 39, in code
    return code(coder.coder, xz, z, x, **kw_args)
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/neuralprocesses/__init__.py", line 21, in f_wrapped
    return f(*args, **kw_args)
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/neuralprocesses/chain.py", line 56, in code
    xz, z = code(link, xz, z, x, **kw_args)
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/neuralprocesses/__init__.py", line 21, in f_wrapped
    return f(*args, **kw_args)
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/neuralprocesses/chain.py", line 56, in code
    xz, z = code(link, xz, z, x, **kw_args)
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/plum/function.py", line 399, in __call__
    return _convert(method(*args, **kw_args), return_type)
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/neuralprocesses/__init__.py", line 21, in f_wrapped
    return f(*args, **kw_args)
  File "/opt/anaconda3/envs/oceanEnv/lib/python3.10/site-packages/neuralprocesses/coders/shaping.py", line 179, in code
    raise AssertionError(
AssertionError: Expected not a parallel of elements, but got inputs and outputs in parallel.

</details>

And I am not sure what I have done wrong.

Hi @nilsleh, the AttributeError: 'Masked' object has no attribute 'dtype' is exactly what is described above - essentially you can't train a model with NaNs in targets. It is unfortunate that when target NaNs are present the error message is confusing. As you say, using batch_size > 1 means target NaNs are automatically removed within the concat_tasks method.
Regarding the AssertionError: Expected not a parallel of elements, but got inputs and outputs in parallel, I have never seen that neuralprocesses error before. The shape of the Task looks fine, but I am missing context for what exact code you call prior to this. Would you be able to produce an MWE in a Colab by generating random data?

Hi @tom-andersson thanks for the reply, I have created a gist with the accompanying data I am using. The data tar file also contains the normalization parameters for the data processor.

Glad you could solve this @nilsleh  - to copy over your solution from the neuralprocesses GitHub for future reference:

I had forgotten to pass in the task_loader as an argument to the ConvNP model as I was using multiple context sets. And that just initializes a model with default parameters than then result in a mismatch, when you try to pass in your "actual" data.

There were no complaints about closing this issue, so closing now.


Implement the Cuckoo Cycle PoW

Figure out how to plug in the Cuckoo Cycle PoW into Erlang.
Implement using ports or NIF.

Let's make sure we can run it in parallel with everything else.

I am extracting code from the the main miner in the Cuckoo Cycle code of John Tomp (https://github.com/tromp/cuckoo) and build an Erlang interface using NIFs.

Cuckoo Cycle takes a string (which can be the hash of the block) and a nonce and creates a pair of siphash keys. These, together with the nonce, are used to create paths. The algorithm looks for 42-long cycles in these paths.
An attempt with a specific nonce value takes roughly 6 seconds on my laptop and it succeeds with a 2.2% chance. It results in an array of 42 integers representing the cycle found. Tromp suggested adding a difficulty by calculating the SHA265 hash of the solution and discarding the solution unless the hash is below the difficulty, just like with SHA256 PoW. (N. B. the success rate can be tuned by changing the node/edge ratio (1/2 in the suggested implementation) but this is nonlinear).
There are two methods verifying a solution:

We can redo the calculation from the block hash and nonce. This takes about 6 seconds as it includes re-generating the original solution.
Using the actual solution (array of 42 numbers) we can verify the solution by simple checking if it forms a cycle, this takes only 50-100 microseconds on my laptop. Though this is much faster, it requires storing the solution for aiding later verification.

Do you mean storing the solution in the block header, together with the hash and nonce? @sennui

I think, that storing the solution was one of the ideas behind democratizing miners and giving a slack to cheap mobile phones. (Jumping 42 pointers is way cheaper even than computing sha256)
@wagerlabs Yes, I would add it to the block and respectively include it in a header.
It would be an equivalent of current solution for, say, bitcoin: hash is a solution and its public.
If we want to make it fixed 42 numbers, we can drive the difficulty in a way mentioned by @akorosmezey

+1
On Wed, Sep 6, 2017 at 3:30 PM sennui<EMAIL_ADDRESS>wrote:

I think, that storing the solution was one of the ideas behind
democratizing miners and giving a slack to cheap mobile phones. (Jumping 42
pointers is way cheaper even than computing sha256)
@wagerlabs https://github.com/wagerlabs Yes, I would add it to the
block and respectively include it in a header.
It would be an equivalent of current solution for, say, bitcoin: hash is a
solution and its public.
If we want to make it fixed 42 numbers, we can drive the difficulty in a
way mentioned by @akorosmezey https://github.com/akorosmezey
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aeternity/epoch/issues/5#issuecomment-327468179, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AABJZ0INOl17kU3L4yEZGb08MZ2P2uHRks5sfpBYgaJpZM4PC1Dh
.

--
__
CTO, æternity
http://linkedin.com/in/joelreymont

@sennui raised the question where the output of printf calls in the NIF goes. Unfortunately it goes to the console. I can remove most of them (the ones from my code) but there are still others from the original code. In order to remove them I need to fork the repo or just include the sources.
(A simple solution would be to redirect stdout to /dev/null for the duration of the call but this may affect other processes running parallel and in case of a crash we do not have the chance of undo the redirection.)

PR #84 created that adds a call with default values to the API. @wagerlabs supported this idea.


Unable to locate credentials

Facing an issue while accessing the S3 bucket of ncei-wcsd-archive even with anon=True.

Install AWS CLI From
https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html, and configure using the aws configure command. Set the appropriate API key and Secret Key, which will generate the credentials file under the .aws folder.


History of translation changes

Is your feature request related to a problem? Please describe.
History of translation changes
Describe the solution you'd like
To see the history of translation changes.
Describe alternatives you've considered
No alternatives at the moment.

History feature added


Docs for personal BinderHub CORS setup

Proposed change
Consolidated documentation for setting up CORS headers on personal BinderHub deployments with helm.
Alternative options
Leave the documentation as it is - and I will just be grateful for an already awesome tool!
Who would use this feature?
Anyone wishing to connect to a personal BinderHub deployment via clients that rely on CORS requests, like Juniper.
(Optional): Suggest a solution
My knowledge of CORS headers, etc. is quite limited. But as far as I have pieced together from the MyBinder deployment config/ other Github issues, the following minimal additions must be added to config.yaml:
binderhub:
  cors: &cors
    allowOrigin: "*"

  jupyterhub:
    custom:
      cors: *cors

And I remember reading a comment somewhere from @minrk that only authenticated requests are supposed to work with cors, so maybe the authentication options should also be added?
If I ever get my own cors setup to work, I would be happy to contribute this section to the documentation myself. Unfortunately, my naive attempts with the above options have failed so far. For reference, I am trying to use juniper on this page to connect to this binder. For debugging, the page console is logging connection events as experienced by juniper, which is where I'm seeing the CORS error. I'm not giving up, but maybe someone has some helpful tips in the meantime.

Just adding my 2 cents here. I am a complete newbie to this as well but I recently added these headers to my config.yaml for a binder hosted on AWS and they worked a charm! I wrote all my steps out in a guide, here is the section with my config.yaml in case it's helpful.

Thanks @TomasBeuzen, that did it! I'm pretty sure I just had the yaml nesting wrong. Had I seen your config.yaml this morning it would have saved me quite a few headaches 😅
So my problem is solved. If you're up for it, I would recommend that you (or I if you don't want to!) make a pull request for the documentation with a small section about CORS and includes your full working config.yaml (probably under Customization, next to the https stuff?)

Glad I could help! (I also had many headaches trying to sort this out so I know what it's like!). I recently opened #1151 to add some more documentation supporting HTTPS for AWS users. This CORS stuff probably fits pretty naturally into/around that PR so I can include that too. Will link this issue once I open the PR :)

Do you want to make a separate PR for the CORS documentation? In general smaller PRs or PRs that touch only one topic are easier and faster to review than bigger ones.

Sure sounds good


prevent variable aliasing in any way

the following code
func OpenLogFile(lo LogOptions) (io.WriteCloser, error) {
    var logw io.WriteCloser
    switch lo.LogPath {
    case "stderr":
        logw = os.Stderr
    default:
        logw, err := openLogFileInternal(lo)
        if err != nil {
            return logw, err
        }
    }
    log.SetOutput(logw)
    return logw, nil
}

will alias the logw variable: since logw is declared outside, and in the default switch case another logw is created with :=, every default case will return a nil. Fun ensues.
Go should either prevent this behaviour with a panic/error/warning or should note that the variable is initialized in an external block and properly initialize it rather than creating a new one.

Can't change this now. https://golang.org/doc/go1compat
We will certainly revisit variable declarations and scoping rules in Go 2.

Declare err at the outer scope and use = instead of :=.
This proposal is an incompatible language change, so
it's not going to happen, at least not for Go 1.

Declare err at the outer scope and use = instead of :=.

The problem is that it's very easy to typo = and :=.

This proposal is an incompatible language change, so it's not going to happen, at least not for Go 1.

Yep!

It's fine if this waits until Go 2, just don't forget about it when it is time :)

Even for Go 2, I think a better way is to have golint or govet detect such
cases rather than make a language change.

sure, as long as it automated and the developer is notified of this in some way


Biting Cold & Into the Breach Hats

Adds Hats for the Biting Cold and Into the Breach packs
Also a few other commits that I guess got duplicated since their changes are already in main lol

Don't worry about the commit duplication, it happened because we squash merged a few PRs. Git is still intelligent enough to recognize that they represent changes that were already incorporated into the main repo.


Change java record type DB meta data for all date related fields from String to Date

Most database i/o is performed against reading and writing JRT (Java Record Type) for specific table within database.   All current JRTs have date fields listed as String.   Need to convert all JRTs with 'date' field from String to java.util.Date to align with actual Database/Field types in MySQL and SQLite.   All date related fields within the database schema are defined as 'Date' or 'DateTime'.   JRTs will be updated to reflect this.   Dates passed to the database are nullable and should be either a legitimate YYYY-MM-DD date or Null.   All references to String dates are to be removed in the JRTs ...as has been done at the database schema level.

The JRT type for all date related fields will remain as String.   The initial design decision for blank dates as null in MySQL adds difficulty to using actual java.uti.Dates at the JRT level.  In addition, a conversion from java.util.date to java.sql.date would be required.   Better left as-is...and continue usage of setDBDate methods across diff types of DBs.


cors issue when calling an API

@prasmussen I am trying to call the glot run API ( https://run.glot.io/languages/javascript/latest ) to run the code with API key created in glot.io website from my reactJS app hosted in heroku, but the API is throwing cors issues with statusCode 404.
When I tried to run the same API through postman it's running fine but it's causing cors issue from the browser
Error:
Access to XMLHttpRequest at 'https://run.glot.io/languages/javascript/latest' from origin 'https://mydomain.io' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
I understand if I call this API from localhost it will throw the cors issues but I cannot able to understand why it's throwing this error even from a hosted server

The api does not support CORS at this time.
Using the api from the browser would mean that anyone could see your api key and could for example create and delete snippets using your account.
I want to improve the way api keys are handled before supporting CORS.

The api does not support CORS at this time.
Using the api from the browser would mean that anyone could see your api key and could for example create and delete snippets using your account.
I want to improve the way api keys are handled before supporting CORS.


Edit staged proposal bug

Edit staged proposals doesn`t work
[Nik - just wanted to report the same]
Steps to reproduce:
1 - Create proposal
2 - Publish on staging
3 - Click on "Edit Proposal" in the overview
Expected result:
I am editing the proposal
Actual result:
A screen appears with "Complete your draft proposal".
The button "Continue Proposal" does not work - nothing happens clicking on it
I can't edit the staged proposal.

Hi @evgenibir
Expected result:
Continue editing the proposal and publishing.
Atual result:
Can't continue the create the proposal or publishing it
Video: https://adilo.bigcommand.com/watch/c8Lx9hLv
Staging Demoxdaox
Google Chrome V 109.0.5414.87

@evgenibir check @pmmm last comment i updated video


Github action for creating Google Calendar Reminders upon issue assignment

This is a pull request for issue #624 , creating a Google Calendar Reminder when the user assigns themselves to an issue

Upon issue assignment, the action will append a comment with a link to create a Google Calendar Event for the assigned issue's tentative due date.  The issue's title will be encoded in the URL link and set as the event title by Default as shown in the following example:

The user is free to adjust their desired dates, times, reminder types, etc.  The due date is tentative and can be adjusted by the user.


Create number column type

New column-type "number" created, which can have suffix and prefix, too.

Hey @m5taha5m given columns are read only, would it work to simply add suffixes and prefixes to the text column rather than having to create a new column/documentation etc?
Based on this implementation all decimals will be stripped, which may not work for all use cases.

@m5taha5m , thanks a lot for your PR. Merged it, it will be available publicly in the next patch update, today or in a few days.
@b8ne and @lloy0076 - thank you for reviewing this. I've just implemented the changes you requested and merged the PR. To be more accurate:

added the number column type, with prefix, suffix and decimals;
added prefix, suffix and a custom limit parameters to the text column;

Also changed the documentation to show these features, here.
Thank you. Cheers!


Add appengine-web.xml and fix artifact ID

Making these changes allowed me to run the project locally.

Why does google cloud build fail? What does that even mean

Why does google cloud build fail? What does that even mean

I enabled that when I linked the GCP project but I just disabled it. If you push an empty commit git commit --amend, it should no longer run.

That fixed it, thanks!


Backport 3795

Backport and version gate the original fix for use statement sorting with raw identifiers (#3795)
Let me know if any additional tests need to be added to this.
Closes #5362

It seems the integration tests are failing because of issues related to https://github.com/clap-rs/clap/issues/3822

So I think the issue is two fold. I believe we're running into an issue with cargo install --path . --force installing a newer version of clap than we have listed in our Cargo.toml and then we're hitting the deprecation warning. We're using clap==3.1. Here's a snippet from the help docs for cargo help install:

By default, the Cargo.lock file that is included with the package will be ignored. This means that Cargo will recompute
which versions of dependencies to use, possibly using newer versions that have been released since the package was
published. The --locked flag can be used to force Cargo to use the packaged Cargo.lock file if it is available. This may be useful for ensuring reproducible builds, to use the exact same set of dependencies that were available when the package was published. It may also be useful if a newer version of a dependency is published that no longer builds on your system, or has other problems. The downside to using --locked is that you will not receive any fixes or updates to any dependency. Note that Cargo did not start publishing Cargo.lock files until version 1.37, which means packages published with prior versions will not have a Cargo.lock file available.

And here's a snippet from the bitflags integration test when running cargo install --path . --force:

Downloaded clap v3.2.1
Downloaded serde_derive v1.0.137
Downloaded itertools v0.10.3
Downloaded heck v0.4.0
Downloaded unicode-segmentation v1.9.0
Downloaded memchr v2.5.0
Downloaded thiserror v1.0.31
Downloaded serde_json v1.0.81
Downloaded bstr v0.2.17
Downloaded itoa v1.0.2
Downloaded cargo-platform v0.1.2
Downloaded serde v1.0.137
Downloaded anyhow v1.0.57
Downloaded rustc-workspace-hack v1.0.0
Downloaded log v0.4.17
Downloaded proc-macro2 v1.0.39
Downloaded quote v1.0.18
Downloaded os_str_bytes v6.1.0
Downloaded globset v0.4.9
Downloaded env_logger v0.9.0
Downloaded atty v0.2.14
Downloaded dirs-next v2.0.0
Downloaded clap_derive v3.2.1

Based on the help text it seems like the remedy is to add the --locked flag

On a side note if you take a look at one of the integration tests that passed (clippy) for example
(export logs for future viewing logs_6801.zip You'll notice that it was reported as a success, but the test actually failed. Not sure if this is a problem with the ci/integration.sh script of with GitHub Acitions.

)

On a side note if you take a look at one of the integration tests that passed (clippy) for example
(exported logs for future viewing -> logs_6801.zip), you'll notice that it was reported as a success, but the test actually failed.

This is intentional, but in my opinion is more reflective of the limited utility these tests really provide. See below for more details
https://github.com/rust-lang/rustfmt/blob/7b73b60faca71d01d900e49831fcb84553e93019/.github/workflows/integration.yml#L33-L38
https://github.com/rust-lang/rustfmt/blob/7b73b60faca71d01d900e49831fcb84553e93019/.github/workflows/integration.yml#L55-L56

Ahh I see. Thanks for pointing that out!


Update 'go get' for Go 1.4

According to the Go issue, that is the correct fix, so looks good.  Nice find.
Also, we will have to address dependency management at some point, but since we distribute the binaries, the only people really at risk are those that build from source, which isn't necessary if we distribute binaries.

OK, thanks @corylanou -- merging on a green build.

:heart:


TensorFlow Tutorials not compatible with latest release.

Until the docs are updated and the Colab backend is upgraded to use the latest TensorFlow version, please flag the tutorials that are not compatible with the latest stable release so it's clear to users with the latest version that they must either rollback or modify their scripts to user Keras 2 to follow the tutorials as they are.

@Ayman250,
Thank you for reporting the issue. Could you please provide tensorflow tutorial examples where you are facing the issue in the execution. Thank you!

Here is one example:
https://www.tensorflow.org/tutorials/load_data/pandas_dataframe
Running tensorflow v 2.17.0 there is an error on the line:
normalizer.adapt(numeric_features)
local variable 'input_shape' referenced before assignment
You need to convert this to a tensor first it cannot be done implicitly.
numeric_features_tensor = tf.convert_to_tensor(numeric_features)
There are likely more issues downstream in this tutorial.

In the past, documentation was tested before the final release. It seems now that step has been skipped.


Process newsletter msgs

Fixes https://github.com/WhiskeySockets/Baileys/issues/628

The PR worked for me!!

#919


Zeek Broker error (Broker::PEER_UNAVAILABLE): (invalid-node, *localhost:47760, "unable to connect to remote peer

Hello sir,
While solving the issue #82 I have encountered new issue. I have tried installing into fresh system. And reinstalled all the components again freshly. So here are the following conclusion I have reached till this point:
1- when I am compiling the threat bus and other tools like inmem, misp, zeek from the cloning the github, its not installing properly for some reasons. threatbus was stucking at :
2020-12-16 17:57:31 INFO     [threatbus] Starting plugins...
2- What is working for me in that in "venv" envoirnment running the following command for installing threatbus and related utilities using:
pip install threatbus
pip install threatbus-inmem
pip install threatbus-misp
pip install threatbus-zeek

3- Seems like since the day zeek has been updated there is some issue started occurring between zeek and threatbus communication, because zeek version i am running is: 3.2.3
4- Threatbus misp communication is working fine for me.
So now the persistent issue is as shown in screenshot: (I will replace the API key later so didnt hide it here, once all fixed, atleast I hope so)
//Also tried with http://localhost
error in /opt/zeek/share/zeek/base/frameworks/broker/./log.zeek, line 80: Broker error (Broker::PEER_UNAVAILABLE): (invalid-node, *localhost:47760, "unable to connect to remote peer")

Sir, if my issue can be resolved, I will be very grateful to you.Thank you very much.
Kind Regards
Nillay

Hi @nillay

when I am compiling the threat bus and other tools like inmem, misp, zeek from the cloning the github, its not installing properly for some reasons. threatbus was stucking at : 020-12-16 17:57:31 INFO [threatbus] Starting plugins...

That is the case when no plugins are installed. You can install Threat Bus and plugins directly from source as follows:

clone the tenzir/threatbus repository via git clone git@github.com:tenzir/threatbus.git
navigate to the repository root folder cd threatbus
install the Threat Bus and the plugins you need directly from source:

pip3 install .
pip3 install plugins/apps/threatbus_misp
pip3 install plugins/apps&& && /threatbus_zeek
pip3 install plugins/backbones/threatbus_inmem

Note that it is not enough to just install Threat Bus, as it does not do anything without plugins.

What is working for me in that in "venv" envoirnment running the following command for installing threatbus and related utilities [...]

That is because you install Threat Bus and the plugins from PyPI. You can achieve the same by installing from source as described above.

Seems like since the day zeek has been updated there is some issue started occurring between zeek and threatbus communication, because zeek version i am running is: 3.2.3

I started both Threat Bus and Zeek locally, using Zeek version v3.2.3. I cannot reproduce any errors, the communication works fine on my test machine.

So now the persistent issue is as shown in screenshot

The original Zeek script we provide in threatbus/apps/zeek uses port 47761. From your screenshot I can tell that you modified the Zeek script and changed the port to 47760. To make sure you did not accidentally change anything else, please revert the Zeek script to the original state as we provide it in the repository. To start Zeek with a different Threat Bus port, don't change the script, but instead pass the port as command line option as follows:
zeek -i enp0s3 -C threatbus.zeek -- "Tenzir::broker_port=47760/tcp"

Does the error persist?
In case the error persists:

please run netstat -tulpen and paste the output
please clone the Threat Bus repository and run this Zeek test script via python3 zeek_sender.py: The script will send a message to Threat Bus on localhost:47761  via broker. This should lead to a Debug log message in Threat Bus.

@0ortmann sir,
Thank you for detailed steps to reproduce the issue, I have tried all the steps mentioned above, and attaching the screenshot of the current updates, Also when I am running the zeek_sender.py using python3, threatbus is responding properly but for some reasons zeek in not doing the same. May be something I doing seems to be very foolish, since It was working before. Also want to drag your attention to the fact that when I tried installing from the threatbus folder, after all installation is done, threatbus is not able to connect to broker and throwing error about no broker module found. But anyway once all done and threatbus started finding the broker component, I am hereby attaching the screenshot for the same. Thanks a lot sir for taking our precious time for helping me troubleshooting the issue. I thank you from bottom of my heart.

Zeek/Broker can unfortunately not be installed via pip / is not available on PyPI. Instead, one has to install it manually. When working in a virtualenv, you need to make your system's site-packages available to the virtualenv, otherwise it cannot find broker. I.e. create it with  --system-site-packages, like this:

python3 -m venv --system-site-packages venv

Please note that we released a new version of Threat Bus yesterday. Please update to the latest version, if possible.
Thanks for the output of netstat. I'm at a loss understanding why Zeek won't find the connection to Threat Bus. What versions are you using exactly for

Zeek zeek --version
Broker python3 -c "import broker; print(f'{broker.Version.MAJOR}.{broker.Version.MINOR}.{broker.Version.PATCH}')" 1.4.0
Ubuntu uname -a
Lastly, how did you install Zeek and Broker - did you build from source or do you use apt packages, and if so, from where did you download it?

@0ortmann sir, thanks a lot for taking your precious time in understanding the issue.... and thank you very much for detailed pointers on the queries I am having. Sir I have updated the the threatbus just now by pulling the fresh copy from git here, so now I am sharing the current situation as follows:

python3 -m venv --system-site-packages venv
Sir I just updated my venv using above command into my environment. Then compiled the broker, threatbus after that by the latest gits.

Sir for me also Its kinda nightmware that why zeek is not talking to threatbus, when it was doing perfectly once. for some reason Zeek is not able to find the running threatbus pub/sub agent. I installed the zeek and all tools freshly on another machine to check if issue persist. And surprisingly it is occurring even in fresh machine.

One question I want to ask:
I found that you also prepared zeek+broker docker, also the threatbus. I was trying to use that but I am unable to understand how can I supply threatbus.zeek file to zeek since "-v" is reserved for version info in zeek..... I want to try that too, I hope that it may work, but not confident enough that how to supply custom zeek scripts like threatbus.zeek and how to tweak the local.zeek. Although git of that docker explains a lot of commands, I tried them but may be I am failing to understand the usage properly.
Anyways here are following information I am trying to provide related to the issue:
Zeek Version:

Firewall status:

Cheking Threatbus + Zeek communication again:

python3 -c "import broker; print(f'{broker.Version.MAJOR}.{broker.Version.MINOR}.{broker.Version.PATCH}')"

uname -a

System update status for update and dist-upgrade:

How did I Install Zeek:
Dependencies:
$ sudo su
$ apt-get update
$ apt-get upgrade
$ apt install curl
$ apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev
$ echo 'deb https://download.opensuse.org/repositories/security:/zeek/xUbuntu_20.04/ /' | sudo tee /etc/apt/sources.list.d/security:zeek.list
$ curl -fsSL https://download.opensuse.org/repositories/security:/zeek/xUbuntu_20.04/Release.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/security:zeek.gpg > /dev/null
$ sudo apt-get update

Zeek:
$ sudo apt-get install zeek
Verifying if Zeek installed correctly:
$ /opt/zeek/bin/zeek –h

As root/sudo, create the /opt/zeek directory and set ownership to the zeek user.
sudo chown -R user:user /opt/zeek <my user name in place of "user">
sudo chmod 750 /opt/zeek
How did I Install Broker:
I tried both python2 virtualenv and python3 venv for creating the venv envoirnment:
Installing venv (Python3):
$ cd /Users/user/sandbox/broker/
$ python3 -m venv ./venv

Activated venv:
. /Users/user/sandbox/broker/venv/bin/activate
Brief:
As per zeek broker documentation,zeek uses C++ actor frameworks. although zeek broker git provides caf bundled with the package. I tried two ways, first was the Installation of CAF separately and then installation of Zeek Broker with python dependencies. Second way the using of CAF comes with Zeek broker itself.
How did I Install CAF as first alternative way as Broker dependency:
$ git clone --recursive https://github.com/actor-framework/actor-framework
$ cd actor-framework
$ ./configure
$ cd build
$ make
$ make test
$ sudo make install

Installation of Zeek Broker with Python Dependencies: (While keep activating the "venv")
$ git clone --recursive https://github.com/zeek/broker
$ ./configure --prefix=/Users/user/sandbox/broker --python-prefix=$(python -c 'import sys; print(sys.exec_prefix)')
$ make install
$ python -c 'import broker; print(broker.__file__)' /Users/user/sandbox/broker/venv/lib/python3.7/site-packages/broker/__init__.py

(last command reflects the path if broker installed properly otherwise gives error.)
How did I Installed Threatbus
Although this command gives error:
pip3 install plugins/apps&& && /threatbus_zeek
bash: syntax error near unexpected token `&&'

But here is the way threatbus and additional plugins was installed:
$ git clone --recursive https://github.com/tenzir/threatbus
$ cd threatbus
$ pip3 install .
$ pip3 install threatbus
$ pip3 install plugins/apps/*
$ pip3 install plugins/apps/threatbus_misp
$ pip3 install plugins/backbones/threatbus_inmem

Sir, so i tried here to provide all the process I followed to install the abovesaid tools. Please let me know if any other details are required too.
Kind Regards
Nillay

Hi @nillay
You are using Broker 2.1.0 and Zeek 3.2.3. These versions are not compatible. See the Broker 2.0.0-rc1 release notes:

Broker 2.0.0 now depends on CAF 0.18.0 with a wire format targeting compatibility with Zeek 4.0.x.

Please use Broker 1.4.0.
When you build Broker, simply check out the Broker release version via
cd broker
git checkout v1.4.0
git submodule update --init --recursive

Then install Broker like usual.

Sigh! Sir what a relief! I was completely clueless about the cause. Sir, Can't thank you enough for saving me. I am and will always be grateful for the kind of pain you took to help me out.
Attaching screenshot for the same:

Sir, although MISP is not reflecting the incident when its been seen by threatbus and zeek. But atleast now zeek is getting intel from MISP.
Sir, thank you again for all the efforts and precious time you put in helping me out.
Kind Regards
Nillay

Hi @nillay
You are using Broker 2.1.0 and Zeek 3.2.3. These versions are not compatible. See the Broker 2.0.0-rc1 release notes:

Broker 2.0.0 now depends on CAF 0.18.0 with a wire format targeting compatibility with Zeek 4.0.x.

Please use Broker 1.4.0.
When you build Broker, simply check out the Broker release version via
cd broker
git checkout v1.4.0
git submodule update --init --recursive

Then install Broker like usual.

I'm glad we could resolve this issue :)

@0ortmann sir all credits goes to you.


Allow $var nad @macro to be treated as one word.

This is the configuration as I currently have it. It makes selection variables much easier.
However, I'm not sure that sublime-settings is the right place to put it, and it will override any custom word_separators settings.

You introduced a trailing comma, which is incompatible with ST2.

Ok, made that change to the pull request.

@MattDiesel Please fix the conflicts


BN#7 feat(buyer should receive an order confirmation)

PR Description
As a buyer, I want to receive a confirmation email with a tracking number after completing checkout, so that I can track the status of my order and receive updates on its delivery.
Description of tasks that were expected to be completed
List all the tasks and sub-tasks that were assigned to you Make sure you check every completed task. list them in this style

[X] Upon successful payment and order completion, the system should automatically generate a unique tracking number for the order.
[X] The system should use the buyer's email address provided during checkout to send a confirmation email containing the order details and tracking number.
[X] The confirmation email should also include a link to the order status page on the website, where the buyer can view the latest updates on their order.

Description of tasks that were expected to be completed and that are not yet completed

[X] The system should generate a unique tracking number for each order upon successful payment.
[X] The confirmation email should be sent to the email address provided by the buyer during checkout.
[X] The confirmation email should contain the order details and tracking number.
[X] The confirmation email should include a link to the order status page on the website.
[X] The buyer should be able to view the expected delivery date and any updates on the order status on the website.

How this may be tested?
Please describe the tests that you ran to verify your functionalities. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration.

run npm i to install all required dependencies
run npm run dev
Swagger/post

Screenshots (If appropriate/applicable)
Provide screenshots related to your PR in case it's appropriate and applicable
Please check this Checklist before you submit your PR:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my code
[x] I have commented on my code, particularly in hard-to-understand areas
[x] My code generates no warnings
[x] There are no vulnerabilities

replaced it


Use Formatter over printf for string interpolation

@ryandub Is this still relevant to your interests?

@Gifflen - I believe it should still be done as format is the preferred method going forward.


Center map when creating new location with "+" button

Ensures the "+" (add new location) button centre on GPS location rather than the centre of the map so that offline users don't have to remember to centre map first.

 Have the "+" (add new location) button center on GPS location rather than center of map


DLP - instructions for next steps

On the dataset landing page (and only for collaborators on the dataset) we should create a collapsable sidebar that describes the instructions for the next steps:

filling the rest of the metadata
installing the python CLI (@yarikoptic - we should publish this to conda as well)
using the CLI to organize and upload.

It should also contain instructions for how to contact the team to be added as a collaborator to the dataset.
For download (this would be available for all), there should be a separate page with explicit instructions for how to download the data using the CLI.

@satra - I think most or all of the items are already implemented. I'm not completely sure about the "instructions for how to contact the team to be added", contact person is not enough?
Please close or comment/update.

This kind of thing is covered by the "empty states" designs that @jtomeck has included in the DLP redesign mockups. We may need more than is seen there, but it would be good to update this issue with new thoughts about how to proceed based on those designs.

let's close this issue for now. between adding intro.js to code, updating the handbook, and the empty states instructions, this should be covered.


Update Rx subspec to Swift 3 syntax

can you rebase against swift-3?

I didn't actually test to see if the test worked because Tests aren't yet building. (it should, i hope)

update to the latest rxswift and :shipit:

I would vote to not remove the disposable but this shouldn't block the swift 3 module.

Talked with @vinnyt about this.  We're going to favor dropping the DisposeKey return here.  FYI @brightredchilli

@Benuuu @vinnyt rebased onto swift-3, updated to beta2, ran tests, renamed OperatorTests to Rx+ExtensionsTest.


build: rename package

The current name is somewhat too generic and not really well worded.
I propose to rename it to IEEH PowerFactory Tools (package name: ieeh-powerfactory-tools; module name: powerfactory_tools) in order to be in line with future packages.
As long as the published package is relatively young, a package name change wouldnt be too problematic.

@masc622 @FiedLa @SebastianDD  what do you think?


Astyle

Adding astyle config files with formatting rules mimicing project coding style and using them to auto-format code.

Thank you!


Is there new release?

Is there new release?
I see the latest release info is
version 0.1.0
date  2022/8/10
So I want to know about it...

I'm going to ship an update tomorrow morning 👍


Make project useable by downstream CMake projects

This PR makes three independent changes to the build set-up that make it possible (in my setup) to use this project through add_subdirectory and modern CMake targets. I am trying to implement best practices as advocated here and here, among others.
More precisely, I want to include the project like this:
add_subdirectory("${Skyr_ROOT_DIR}" skyr-url)
target_link_libraries(downstream-library
    PUBLIC Skyr::skyr-url)

The three commits solve the following three problems:

Only the target skyr-url exist, but is not aliased. It is common practice to alias the public targets of a project in a namespace.
The include directories are set directory wide, so they work to compile the examples, but downstream projects need to set them manually. By making them target properties, downstream projects do not need to do anything -- CMake sets the include flags automatically.
It is currently not possible to link the generated libskyr-url.a into a shared library because it is not position independent (GCC's -fPIC). CMake has a platform/compiler-independent mechanism to achieve that.

This looks great, thanks very much!


Scaling to create a proccess per cpu core overwhelms grobid service

When I  try and index PDFs on my development machine, I get frequent failures because the grobid service is overwhelmed by the number of processes allocated for ingesting the PDFs:
ERROR [2023-12-03 11:57:31,564] org.grobid.service.process.GrobidRestProcessFiles: Could not get an engine from the pool within configured time. Sending service unavailable.

The default concurrency setting in grobid is 10.  On my machine os.cpu_count() returns 16, so we are creating more processes that available engines in the grobid pool.
Whilst this is not an issue in paperetl itself, I think anyone for whom os.cpu_count() returns > 10 will hit this issue.  The impact could be mitigated by adding a note to the documentation to suggest users increase the default concurrency limit in grobid if they this error. https://grobid.readthedocs.io/en/latest/Configuration/#service-configuration
I am happy to create a PR for this if you agree.

Happily would take a PR on this.
A brief note in this section would be great. https://github.com/neuml/paperetl#additional-dependencies


case should use [] instead of {}

What's up?
Currently, this is syntax for case:
from employees
derive distance = case {
  city == "Calgary" => 0,
  city == "Edmonton" => 300,
}

In the last dev call, @snth made a point that case syntax should be more similar to arrays than tuples, since the elements of the case block:

must have the same type,
are not named.

This is the proposed syntax:
from employees
derive distance = case [
  city == "Calgary" => 0,
  city == "Edmonton" => 300,
]

Thanks @aljazerzen . Very much agree with this.

Someone might want to match Calgary OR Edmonton.
Someone might want to range match income between 1000 and 2000.

(if it's no extra work — and I don't think it's more than a couple of lines which already exist, in the parser — we could support {} for a while too. Though we don't really have a notion of deprecation warnings, which means we wouldn't have a great story for removing it)

I am generally supportive of a deprecation strategy but given our current manpower I think we might want to keep "moving fast and breaking things" while we still can.

Yes, agree on reflection


Add Chrome as dependency

Because the https://github.com/GoogleChrome/chrome-launcher lib is used to start Chrome per default
May further enhancements could change this: #10

Thanks!


Added clock timer

A popular time management technique uses a timer to break down work into intervals (usually
25 mins) followed by a small break.
created a BASH script to aid this process. Taken a command line argument for the
number of iterations (work + break). Print notifications about the breaks (5 mins) or time to work
(25 mins) in the terminal. Every 4 iterations, include a long break (15 mins). Print notification
when all the cycles are complete.
Example usage:
bash timer.sh 2
Output structure:
#1 work
#1 break time
#2 work
#2 break time
Finished

Working perfectly


Add context to requiring Service Network Configuration

https://www.pivotaltracker.com/story/show/159276003

These changes need to be in 1.3 and 1.4, as well.
Our PMs need to review our docs changes: @AmberAlston or @kbakas-piv

@AmberAlston @kbakas-piv As Rachel mentioned, we'll hold off on merging this until we get your approval.
I brought over some edits to match the formatting of Miranda and I's commits to the 1.1 version of this PR. You can see how it will look in the image below. Again, happy to make changes if you have any feedback or if we didn't maintain the meaning in our edits.
cc @rheaton

This looks good to me. Content makes sense and the formatting is a good upgrade.
@ljarzynski


Update ethers dependencies to fix "invalid fragment object" error

See https://github.com/ethers-io/ethers.js/issues/1493.

See ethers-io/ethers.js#1493.

@paulrberg has this issue been resolved? I'm having the same kind of error on version 3.4.0.

I'm not sure, I haven't used this lately.

I'm not sure, I haven't used this lately.

Thanks for your feedback. Looks like this is just a warning, my tests and compilation run perfectly despite this error message.


[Visual]: embedded issues

... why does it think theyre all checked? tried resetting tracker state, no effect. happens on any random start setting.

Yeah, sorry, that's from some other changes that I didn't test super well. The main site should be identical aside from these graphical issues and the item tooltips. I'll be pushing a fix for this specifically soon, but there are likely other related graphical problems.

in that case, i'll keep this open, and report any issues i see in use.

Ok, this should be fixed on the dev site now. I'm not seeing any other graphical issues, but there probably still are some - I appreciate you keeping an eye out!

@kbranch found another one: landfill single entrance locations are missing the "checked" graphic.

Got it, should be fixed - thanks!


Pixart JS Homework QT

Finally Finished!
comfort level: 4 now because of help from Andre. My comfort level was 1 this afternoon.
complete: 5

:+1:  Great job on this.


onclose is not called in JS client on server timeout

Versions of Server-Side NuGet Packages: 1.0.2
Versions of Client-Side NuGet/NPM Packages: 1.0.2
Are you using the C# client or the JavaScript client: JavaScript
The Server you are using (Kestrel/HttpSysServer/IIS/IIS Express/Azure Web App/etc.): Kestrel
The Operating System on the Server (Windows/Linux/macOS): Win
The Operating System on the Client (Windows/Linux/macOS): Win
The Browser on the client, if using the JavaScript client (IE/Chrome/Edge/Firefox/etc.): Chrome

I'm initializing my connection with this code:
connection = new signalR.HubConnectionBuilder()
  .withUrl("/signalR", {
    transport: signalR.HttpTransportType.WebSockets,
    serverTimeoutInMilliseconds: 30 * 1000
  })
  .configureLogging(/* my custom logger */)
  .build();

connection.onclose(_handleDisconnected);

If I simply pause my server, _handleDisconnected is not called after 30 secs but after ~ 1.5 min.
I've debugged a little and found that HubConnection.serverTimeout is actually called after ~ 30 sec which in turn calls this.connection.stop but doesn't trigger the onclose event.
Also this.webSocket.close(); within WebSocketTransport.stop is called...

Looks like you might be hitting https://github.com/aspnet/SignalR/issues/2557#issuecomment-402798865
This was fixed in 2.2.0-preview1. You can try out our previews by using the "next" tag when installing the client via npm, or grab one of the preview versions explicitly<EMAIL_ADDRESS>

Perfect, working as expected in the mentioned preview version!
Until this makes it into stable I can intercept WebSocketTransport.stop and call the close handler myself.
Thanks for the fast & helpful response!

Out of curiosity: Is there any timeline for when those changes are released as stable?

Our Roadmap https://github.com/aspnet/Home/wiki/Roadmap


Make transports optional

Put extra transports behind feature flags to make some dependencies optional.
While at it, also made sure no warnings are emitted on the recent rust version.
Resolves https://github.com/paritytech/litep2p/issues/161.

Thank you, very appreciated!


Incorrect namespacing when enums are used

When enums are used in the schema, the resulting API.swift makes use of JSONEncodable and JSONDecodable. However, they're incorrectly name spaced to Apollo, as shown below:

Simply removing the Apollo. fixes this problem.

It looks like this is fixed in #7. Closing.


Remove dockerExampleRunner

The docker example runner is no longer required due to the configure example code being pulled out of the runner

we should also move the RestExample into the example model module, so we can then use that on all the examples just having different restConfig.json files to point to different config services

merged into develop


Changed name scope within bidirectional, fixing #5820

I don't feel like I understand what exactly name scopes do very well, so I might have done this incorrectly, but in my tests, this fixes #5820 (you can't save models with bidirectional layers, because there are overlapping names; I only ran into this with theano, incidentally; tensorflow worked fine).

Can you try to print out the name of the ops and variables being created in each sublayer?

I don't know about any ops that are created, but the names of the weight variables are as follows; I assume the ops have similar naming differences:
We have three names here: (1) the bidirectional wrapper name [here bidirectional], (2) the forward/backward layer names [here gru for both, which Bidirectional changes to forward_gru and backward_gru], and (3) the weight names in the forward/backward layers [here kernel, recurrent_kernel, bias].
Before this PR, the weight names being saved in the h5 file were these:

bidirectional/kernel [for forward layer]
bidirectional/recurrent_kernel [for forward layer]
bidirectional/bias [for forward layer]
bidirectional/kernel [for backward layer; crash happens here]

After this PR, the weight names being saved are:

bidirectional/forward_gru/kernel
bidirectional/forward_gru/recurrent_kernel
bidirectional/forward_gru/bias
bidirectional/backward_gru/kernel
bidirectional/backward_gru/recurrent_kernel
bidirectional/backward_gru/bias

(With a possible _1 or some other index appended to the layer names, depending on the context of running this.)

LGTM, thanks


Workload tests adapted to centralized result logging

Description
Workload tests adapted to cenralized result logging
Issues:
Refs: #1861
How has this been tested:

[ ] Covered by existing integration testing
[ ] Added integration testing to cover
[ ] Verified all A/C passes

[ ] develop
[ ] master
[ ] tag/other branch

[ ] Test environment

[ ] Shared Packet K8s cluster
[ ] New Packet K8s cluster
[ ] Kind cluster

[ ] Have not tested

Types of changes:

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation update

Checklist:
Documentation

[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] No updates required.

Code Review

[ ] Does the test handle fatal exceptions, ie. rescue block

Issue

[ ] Tasks in issue are checked off

spec test result the same as for main branch (still some failures but those don't not seem to be caused by this change).
Ready for review.

spec test result the same as for main branch (still some failures but those don't not seem to be caused by this change). Ready for review.

We'll try to pull down and push up to verify github actions with no issues and manual testing prior to merging.

rebased

Just tried running the build for this. There are some tests that are failing on repeated attempts.
https://github.com/cnti-testcatalog/testsuite/actions/runs/8232211049

fixed.
@HashNuke in the link provided I saw following failures of the spec tests:
curl, helm, operator_test, points , validate_config
I fixed reasons for: helm, operator_test, points , validate_config.
Failing "curl" specs: I don't know, I think it is not related to this change. For me it is failing even on main branch.
Can you please re-run github actions? Thanks.

LGTM. Merged this.
Build is passing - https://github.com/cnti-testcatalog/testsuite/actions/runs/8342423721


Cleanup: ZC Bcast Post API code in CkArray methods

BTW, what happens to ZC code path if there are no array elements on a PE? Can the ZC messages be received by CkArray in that case, and will it attempt to do localElemVec[0] even though it is empty or does it take that into account somewhere?

BTW, what happens to ZC code path if there are no array elements on a PE? Can the ZC messages be received by CkArray in that case, and will it attempt to do localElemVec[0] even though it is empty or does it take that into account somewhere?

This was a case I didn't handle in the previous code. I always had more than 1 array element/PE in my tests, so I overlooked this corner case. I'll add len > 0 to the if condition to ensure that localElemVec[0] is only used when there is a 0th element.

Thanks @juanjgalvez, I had not considered the case of len being 0 on each PE. There is an issue with ZC Bcast Post API whenever the first PE of the logical node has 0 elements. I should either mark that as a known issue or post a fix before 6.10.
From slack chat on #core:
nbhat4  How likely is it for the number of chare array elements of a chare array to be less than the number of total PEs? (since ideally, the number of chare array elements ~ 10 * CkNumPes())
I am asking this because I realized that ZC Bcast Post API for chare arrays doesn’t handle some corner cases. (Currently, it assumes that there’s an array element on the first PE of a logical node).
For 6.10, I am inclined to mark this as a known issue, since supporting it will change the implementation quite a bit.

swhite  OpenAtom uses numChareArrayElements < Cores for specific chare arrays, though they have multiple different chare arrays so in that way make use of all cores
subcommunicators in AMPI are similar

nbhat4  Ah okay, makes sense. Yeah, that assumption that there’ll be a chare array element on every logical node was fundamentally incorrect.
So, for ZC Bcast Post API, the mapping of chare array elements to PEs makes all the difference. (for eg, there might be chare array elements on the root node and leaves, but none on intermediate nodes. In such a case, the leaves cannot rget from their immediate parents, they should rget from the last parent that has an element).

swhite  This only effects the post api version of ZC Bcast?

nbhat4  Yes, that’s right. (Since the ZC Bcast Send API allocates a buffer on every process (logical node) irrespective of the number of elements present.) (edited)


 Non-functional changes, just working toward factoring out the SDK interface and some cleanup.
Best reviewed by individual commits.
Does this introduce a breaking change?

[ ] Yes
[x] No

Pull Request Type
What kind of change does this Pull Request introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[x] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

Pull Request Checklist

[ ] Public API changes
[ ] Verified for cross-platform
[ ] Synced with iOS, Web team
[ ] Internal review completed
[ ] UI Changes

[ ] Screen captures included
[ ] Tested for Light/Dark mode
[ ] Tested for screen rotation
[ ] Tested on Tablet and small screen device (5")
[ ] Include localization changes

[ ] Tests

[ ] Memory leak analysis performed
[ ] Tested on API 21, 26 and latest
[ ] Tested for background mode
[ ] Unit Tests Included
[ ] UI Automated Tests Included

How to Test

Open...
Click on...

What to Check
Verify that the following are valid

...

Other Information

Did basic smoke testing running the demo app on a device, connecting ACS-ACS and ACS-Teams, it all appears to work.

tested - looks good


Can't pull image from Docker registry (gitlab) using second grid

Inception
When I'am trying to install app to the my production grid (on another server, agent without master), I caught the following error:
GridServiceInstanceDeployer::ServiceError: Docker::Error::ServerError: ... latest: denied: access forbidden (on node node1)
It's seems because kontena cannot to connect with gitlab private registry. Inside a kontena logs, they contain following errors:
error parsing HTTP 403 response body: invalid character 'F' looking for beginning of value: "Forbidden\n" or just denied: access forbidden
List of external registries show all needed registries to pull containers using kontena. I also tried login from server (docker login) and run them docker pull - it works properly without kontena deploying.
My gitlab with registry generate following logs (only when kontena start deploying):
2017-08-09_16:35:45.88954 time="2017-08-09T19:35:45.889492198+03:00" level=warning
 msg="error authorizing context: authorization token required" environment=production
go.version=go1.8.1 http.request.host=registry.domain.com http.request.id=879f2af3-c98b-4047-
8a35-2dff66c4d0ae http.request.method=GET http.request.remoteaddr=IP http.request.uri="/v2/"
http.request.useragent="docker/17.03.2-ce go/go1.7.5 git-commit/f5ec1e2 kernel/4.4.0-89-generic
os/linux arch/amd64 UpstreamClient(Swipely/Docker-API 1.32.1)" instance.id=5d7d7dc0-c2ad-4252-
b9c9-3f72151a9d35 service=registry version=v2.6.1-1-gdd544a8
2017-08-09_16:35:45.88956 <IP_ADDRESS> - - [09/Aug/2017:19:35:45 +0300] "GET /v2/ HTTP/1.1" 401 87
"" "docker/17.03.2-ce go/go1.7.5 git-commit/f5ec1e2 kernel/4.4.0-89-generic os/linux arch/amd64
UpstreamClient(Swipely/Docker-API 1.32.1)"

I think that it kontena bug.
P.S. If I use official private docker registry - it works and container to be pull & running on the server.
P.P.S. kontena works fine with development (first) grid, when master and agent installed on the same host.
This problem same discussed in Slack:
https://kontenacommunity.slack.com/archives/C1U90NQN8/p1502296926034575
How to reproduce?
Following this steps:

install gitlab & configure gitlab registry -> server1
1.1. create repo, with container, build & push
install master server & agent on server2
2.1. add external registry to kontena cli client
try deploy, it just works... (on first grid)
create another grid named production (or different name), switch to here and copy env vars
install agent on server 3, paste env vars
after 'kontena node ls ' showing it, try deploy this app to second grid.
kontena fails on pull step.

And see the screenshots, how to I'm trying deploy to production grid:
http://telegra.ph/Test-story-08-09
Links
Some links which used for solve problem (but isn't successfully).

https://gitlab.com/gitlab-com/support-forum/issues/1830
https://gitlab.com/gitlab-org/gitlab-ce/issues/29863
https://github.com/moby/moby/issues/33434

More information about attempts can be found in Slack thread.
__
So how I don't use Slack every day, please connect with me using this issue thread.

Kontena 1.4 will have #2695 to fix https://github.com/swipely/docker-api/pull/485 for https://github.com/moby/moby/issues/33434, but I'm not sure if that's what's happening here or not.

Could you try to make the pull from the node directly? I.e. log into the node with ssh and do:
docker login ...
docker pull ...

Just to eliminate any connectivity issue from the grid node to the registry.

Can you show the complete kontena-agent logs from the deploy that fails? In particular, does the Kontena::Workers::ImagePullWorker log pulling image: ... or pulling image with credentials: ...?


NO-JIRA Adding a test

This test starts 2 servers and send messages to
a queue until it enters into paging state. Then
it changes the max-size to -1, restarts the 2 servers
and consumes all the messages. If verifies that
even if the max-size has changed all the paged
messages will be depaged and consumed. No stuck
messages after restarting.

@howardgao Could you please add a more descriptive commit message.

@mtaylor I've added a few more lines to the commit message.


[Feature-10629][Improvement] Support multi cluster environments - namespace use cluster

Purpose of the pull request
namespace bind to cluster, when namespace operator get config from cluster config

Brief change log

Modify cluster update，delete
Modify namespace create，update，delete

Verify this pull request

This change added tests and can be verified as follows:

Update K8SNamespaceServiceTest
Update ClusterServiceTest.

close #10629

Codecov Report

Merging #10650 (d5a733f) into dev (cd62de9) will decrease coverage by 0.20%.
The diff coverage is 52.38%.

@@             Coverage Diff              @@
##                dev   #10650      +/-   ##
============================================
- Coverage     41.11%   40.90%   -0.21%
+ Complexity     4902     4837      -65
============================================
  Files           897      897
  Lines         36245    36211      -34
  Branches       3993     4013      +20
============================================
- Hits          14902    14812      -90
- Misses        19878    19920      +42
- Partials       1465     1479      +14

Impacted Files
Coverage Δ

...che/dolphinscheduler/api/k8s/K8sClientService.java
4.25% <0.00%> (-2.13%)
:arrow_down:

...uler/api/service/impl/K8SNamespaceServiceImpl.java
68.33% <50.00%> (-6.06%)
:arrow_down:

...rg/apache/dolphinscheduler/api/k8s/K8sManager.java
58.53% <56.66%> (+11.16%)
:arrow_up:

...scheduler/api/service/impl/ClusterServiceImpl.java
85.71% <58.33%> (-2.57%)
:arrow_down:

...ache/dolphinscheduler/dao/entity/K8sNamespace.java
75.40% <66.66%> (-1.26%)
:arrow_down:

...heduler/api/controller/K8sNamespaceController.java
80.00% <100.00%> (ø)

.../org/apache/dolphinscheduler/api/enums/Status.java
100.00% <100.00%> (ø)

.../org/apache/dolphinscheduler/common/Constants.java
82.60% <100.00%> (+0.79%)
:arrow_up:

...inscheduler/server/master/config/MasterConfig.java
35.00% <0.00%> (-41.75%)
:arrow_down:

...inscheduler/server/worker/config/WorkerConfig.java
52.38% <0.00%> (-24.09%)
:arrow_down:

... and 31 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cd62de9...d5a733f. Read the comment docs.


docs: zh_cn index page

Codecov Report

Merging #68 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #68   +/-   ##
=======================================
  Coverage   78.29%   78.29%
=======================================
  Files         181      181
  Lines        5141     5141
  Branches      628      628
=======================================
  Hits         4025     4025
  Misses        867      867
  Partials      249      249

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2bd85fe...dfc425a. Read the comment docs.


How to add to the homekit using the type of television?

Could you prepare a ready-made flow for adding a TV to the HomeKit via node Homekit “node-red-contrib-homekit-bridged” using your plugin?
It would be amazing!

I'm not using node-red-contrib-homekit-bridged, so i fear i can't help you with that. Maybe the maintainers of the homekit nodes can?


HI3518EV100 / majestic runs into timeout even when idle

After less than a minute running (serving images or being idle) majestic runs into timeout and can no longer serve images:
root@openipc-hi3518ev100:~# killall majestic; SENSOR=ov9712 majestic
00:03:00 [    main] main@135                      Majestic Lite for HiSilicon, version master+e5e75e4, built on 2022-05-12
00:03:00 [app_conf] load_config@345               Using /etc/majestic.yaml as main configuration
00:03:00 [watchdog] watchdog_start@101            Detected Hisilicon Watchdog
00:03:00 [watchdog] watchdog_start@121            Watchdog timeout set to 10 seconds
00:03:01 [     sdk] try_sensor_config@3103        SENSOR=ov9712
00:03:01 [     sdk] find_sensor_config@3065       matched sensor config: ov9712_i2c_dc_720p.ini
00:03:01 [     sdk] start_sdk@246                 Using /etc/sensors/ov9712_i2c_dc_720p.ini sensor configuration file
00:03:01 [     sdk] start_sdk@255                 App was built with MPP version: Hi3518_MPP_V1.0.B.0.
00:03:01 [     sdk] start_sdk@258                 Current MPP version: HI_VERSION=Hi3518_MPP_V1.0.B.0
00:03:01 [     grp] GROUP_UnregisterChn@1813      [   grp] Group: 1 has been start recive picture, not surpport unregister
00:03:01 [      vi] VIU_DRV_DisableChn@1470       [    vi] !! Disable ViChn:0 timeout 120ms!!!
00:03:01 [     sdk] start_sdk@281                 sensor ov9712
00:03:01 [     sdk] start_sdk@293                   dev [1280x720]@0x0 25fps, BGGR
00:03:01 [  sensor] try_to_load@19                trying to load /usr/lib/sensors/libsns_ov9712_i2c_dc.so
00:03:01 [     hal] hisi_free_mem_ex@704          Free MMZ mem before allocation: 32640KB
00:03:01 [     sdk] dump_vb_configuration@2027    VB configuration:
00:03:01 [     sdk] dump_vb_configuration@2037      [0]: 1382400 x 5
00:03:01 [     sdk] dump_vb_configuration@2037      [1]: 1843200 x 1
00:03:01 [     hal] hisi_free_mem_ex@704          Free MMZ mem after allocation: 24076KB
00:03:01 [     sdk] init_sensor@2231              Sensor driver loaded
00:03:01 [     sdk] log_venc_chn@1141             H.264 1280x720 25fps 4096Kbit  25 GOP
00:03:01 [     sdk] create_vpss_chn@888             new venc: 0   vpss_grp: 0,   vpss_chn: 0
00:03:01 [     sdk] start_video@1712                venc took 3400KB of MMZ [675]
00:03:01 [     sdk] init_chn@1176                 JPEG snapshot snapshot venc_chn 1  1280x720
00:03:01 [     sdk] start_jpeg@1257                 jpeg_enc took 916KB of MMZ [900]
00:03:01 [     sdk] start_sdk@392                 HiSilicon SDK started
00:03:01 [     hal] hisi_free_mem_ex@704          Free MMZ mem finally: 15604KB
00:03:01 [   httpd] new_http_server@346           HTTP server started on :::80
00:03:01 [    rtsp] rtsp_init@30                  RTSP server started on port 554
00:03:01 [   netip] netip_start@2013              NETIP server started on port 34567
00:03:01 [    gpio] set_gpio@25                   set_gpio(39, 0)
00:03:01 [    gpio] set_gpio@25                   set_gpio(38, 0)
00:03:01 [   night] dn_start@220                  Starting monitor for light sensor
00:03:01 [    gpio] set_gpio@25                   set_gpio(39, 1)
00:03:01 [    gpio] set_gpio@25                   set_gpio(38, 0)
00:03:01 [    gpio] set_gpio@25                   set_gpio(39, 0)
00:03:01 [    gpio] set_gpio@25                   set_gpio(38, 1)
00:03:02 [    gpio] set_gpio@25                   set_gpio(39, 0)
00:03:02 [    gpio] set_gpio@25                   set_gpio(38, 0)
00:03:02 [   night] set_night_mode@117            Change mode to DAY
00:03:02 [    gpio] set_gpio@25                   set_gpio(39, 0)
00:03:02 [    gpio] set_gpio@25                   set_gpio(38, 1)
00:03:02 [    gpio] set_gpio@25                   set_gpio(39, 0)
00:03:02 [    gpio] set_gpio@25                   set_gpio(38, 0)
00:03:49 [     sdk] venc_read@813                 Timeout get any data from venc channel 0
00:03:49 [     sdk] venc_read@833                 NOTE: Current frame is NULL!
00:03:51 [     sdk] venc_read@813                 Timeout get any data from venc channel 0
00:03:51 [     sdk] venc_read@833                 NOTE: Current frame is NULL!
00:03:53 [     sdk] venc_read@813                 Timeout get any data from venc channel 0
00:03:53 [     sdk] venc_read@833                 NOTE: Current frame is NULL!
00:03:55 [     sdk] venc_read@813                 Timeout get any data from venc channel 0
00:03:55 [     sdk] venc_read@833                 NOTE: Current frame is NULL!
00:03:57 [     sdk] venc_read@813                 Timeout get any data from venc channel 0
00:03:57 [     sdk] venc_read@833                 NOTE: Current frame is NULL!
^C00:03:58 [     log] printf@261                    Interrupt signal received
00:03:58 [    gpio] set_gpio@25                   set_gpio(39, 0)
00:03:58 [    gpio] set_gpio@25                   set_gpio(38, 0)
00:03:58 [     sdk] stop_chn@1746                 Stopped 1 channel
00:03:58 [     sdk] stop_chn@1746                 Stopped 0 channel
00:03:58 [     sdk] ISP_thread_proc@160           Shutdown isp_run thread
00:03:58 [      vi] VIU_DRV_DisableChn@1470       [    vi] !! Disable ViChn:0 timeout 120ms!!!
00:03:58 [     sdk] stop_sdk@2983                 Stop sdk Ok!
00:03:58 [    main] main@223                      Shutdown main thread

root@openipc-hi3518ev100:~# killall majestic; SENSOR=ov9712 majestic
killall: majestic: no process killed
00:07:17 [    main] main@135                      Majestic Lite for HiSilicon, version master+e5e75e4, built on 2022-05-12
00:07:17 [app_conf] load_config@345               Using /etc/majestic.yaml as main configuration
00:07:17 [watchdog] watchdog_start@101            Detected Hisilicon Watchdog
00:07:17 [watchdog] watchdog_start@121            Watchdog timeout set to 10 seconds
00:07:17 [     sdk] try_sensor_config@3103        SENSOR=ov9712
00:07:17 [     sdk] find_sensor_config@3065       matched sensor config: ov9712_i2c_dc_720p.ini
00:07:17 [     sdk] start_sdk@246                 Using /etc/sensors/ov9712_i2c_dc_720p.ini sensor configuration file
00:07:17 [     sdk] start_sdk@255                 App was built with MPP version: Hi3518_MPP_V1.0.B.0.
00:07:17 [     sdk] start_sdk@258                 Current MPP version: HI_VERSION=Hi3518_MPP_V1.0.B.0
00:07:17 [     sdk] start_sdk@281                 sensor ov9712
00:07:17 [     sdk] start_sdk@293                   dev [1280x720]@0x0 25fps, BGGR
00:07:17 [  sensor] try_to_load@19                trying to load /usr/lib/sensors/libsns_ov9712_i2c_dc.so
00:07:18 [     hal] hisi_free_mem_ex@704          Free MMZ mem before allocation: 32640KB
00:07:18 [     sdk] dump_vb_configuration@2027    VB configuration:
00:07:18 [     sdk] dump_vb_configuration@2037      [0]: 1382400 x 5
00:07:18 [     sdk] dump_vb_configuration@2037      [1]: 1843200 x 1
00:07:18 [     hal] hisi_free_mem_ex@704          Free MMZ mem after allocation: 24076KB
00:07:18 [     sdk] init_sensor@2231              Sensor driver loaded
00:07:18 [     sdk] log_venc_chn@1141             H.264 1280x720 25fps 4096Kbit  25 GOP
00:07:18 [     sdk] create_vpss_chn@888             new venc: 0   vpss_grp: 0,   vpss_chn: 0
00:07:18 [     sdk] start_video@1712                venc took 3400KB of MMZ [675]
00:07:18 [     sdk] init_chn@1176                 JPEG snapshot snapshot venc_chn 1  1280x720
00:07:18 [     sdk] start_jpeg@1257                 jpeg_enc took 916KB of MMZ [900]
00:07:18 [     sdk] start_sdk@392                 HiSilicon SDK started
00:07:18 [     hal] hisi_free_mem_ex@704          Free MMZ mem finally: 15604KB
00:07:18 [   httpd] new_http_server@346           HTTP server started on :::80
00:07:18 [    rtsp] rtsp_init@30                  RTSP server started on port 554
00:07:18 [   netip] netip_start@2013              NETIP server started on port 34567
00:07:18 [    gpio] set_gpio@25                   set_gpio(39, 0)
00:07:18 [    gpio] set_gpio@25                   set_gpio(38, 0)
00:07:18 [   night] dn_start@220                  Starting monitor for light sensor
00:07:18 [    gpio] set_gpio@25                   set_gpio(39, 1)
00:07:18 [    gpio] set_gpio@25                   set_gpio(38, 0)
00:07:18 [    gpio] set_gpio@25                   set_gpio(39, 0)
00:07:18 [    gpio] set_gpio@25                   set_gpio(38, 1)
00:07:18 [    gpio] set_gpio@25                   set_gpio(39, 0)
00:07:18 [    gpio] set_gpio@25                   set_gpio(38, 0)
00:07:18 [   night] set_night_mode@117            Change mode to DAY
00:07:18 [    gpio] set_gpio@25                   set_gpio(39, 0)
00:07:18 [    gpio] set_gpio@25                   set_gpio(38, 1)
00:07:18 [    gpio] set_gpio@25                   set_gpio(39, 0)
00:07:18 [    gpio] set_gpio@25                   set_gpio(38, 0)
00:07:26 [     sdk] venc_read@813                 Timeout get any data from venc channel 0
00:07:26 [     sdk] venc_read@833                 NOTE: Current frame is NULL!
00:07:28 [     sdk] venc_read@813                 Timeout get any data from venc channel 0
00:07:28 [     sdk] venc_read@833                 NOTE: Current frame is NULL!
00:07:29 [     sdk] venc_read@813                 Timeout get any data from venc channel 0
00:07:30 [     sdk] venc_read@833                 NOTE: Current frame is NULL!
^C00:07:31 [     log] printf@261                    Interrupt signal received
00:07:31 [    gpio] set_gpio@25                   set_gpio(39, 0)
00:07:31 [    gpio] set_gpio@25                   set_gpio(38, 0)
00:07:31 [     sdk] stop_chn@1746                 Stopped 1 channel
00:07:31 [     sdk] stop_chn@1746                 Stopped 0 channel
00:07:31 [     sdk] ISP_thread_proc@160           Shutdown isp_run thread
00:07:31 [      vi] VIU_DRV_DisableChn@1470       [    vi] !! Disable ViChn:0 timeout 120ms!!!
00:07:31 [     sdk] stop_sdk@2983                 Stop sdk Ok!
00:07:31 [    main] main@223                      Shutdown main thread

root@openipc-hi3518ev100:~# killall majestic; SENSOR=ov9712 majestic
killall: majestic: no process killed
00:07:53 [    main] main@135                      Majestic Lite for HiSilicon, version master+e5e75e4, built on 2022-05-12
00:07:53 [app_conf] load_config@345               Using /etc/majestic.yaml as main configuration
00:07:53 [watchdog] watchdog_start@101            Detected Hisilicon Watchdog
00:07:53 [watchdog] watchdog_start@121            Watchdog timeout set to 10 seconds
00:07:53 [     sdk] try_sensor_config@3103        SENSOR=ov9712
00:07:53 [     sdk] find_sensor_config@3065       matched sensor config: ov9712_i2c_dc_720p.ini
00:07:53 [     sdk] start_sdk@246                 Using /etc/sensors/ov9712_i2c_dc_720p.ini sensor configuration file
00:07:53 [     sdk] start_sdk@255                 App was built with MPP version: Hi3518_MPP_V1.0.B.0.
00:07:53 [     sdk] start_sdk@258                 Current MPP version: HI_VERSION=Hi3518_MPP_V1.0.B.0
00:07:53 [     sdk] start_sdk@281                 sensor ov9712
00:07:53 [     sdk] start_sdk@293                   dev [1280x720]@0x0 25fps, BGGR
00:07:53 [  sensor] try_to_load@19                trying to load /usr/lib/sensors/libsns_ov9712_i2c_dc.so
00:07:53 [     hal] hisi_free_mem_ex@704          Free MMZ mem before allocation: 32640KB
00:07:53 [     sdk] dump_vb_configuration@2027    VB configuration:
00:07:53 [     sdk] dump_vb_configuration@2037      [0]: 1382400 x 5
00:07:53 [     sdk] dump_vb_configuration@2037      [1]: 1843200 x 1
00:07:53 [     hal] hisi_free_mem_ex@704          Free MMZ mem after allocation: 24076KB
00:07:53 [     sdk] init_sensor@2231              Sensor driver loaded
00:07:54 [     sdk] log_venc_chn@1141             H.264 1280x720 25fps 4096Kbit  25 GOP
00:07:54 [     sdk] create_vpss_chn@888             new venc: 0   vpss_grp: 0,   vpss_chn: 0
00:07:54 [     sdk] start_video@1712                venc took 3400KB of MMZ [675]
00:07:54 [     sdk] init_chn@1176                 JPEG snapshot snapshot venc_chn 1  1280x720
00:07:54 [     sdk] start_jpeg@1257                 jpeg_enc took 916KB of MMZ [900]
00:07:54 [     sdk] start_sdk@392                 HiSilicon SDK started
00:07:54 [     hal] hisi_free_mem_ex@704          Free MMZ mem finally: 15604KB
00:07:54 [   httpd] new_http_server@346           HTTP server started on :::80
00:07:54 [    rtsp] rtsp_init@30                  RTSP server started on port 554
00:07:54 [   netip] netip_start@2013              NETIP server started on port 34567
00:07:54 [    gpio] set_gpio@25                   set_gpio(39, 0)
00:07:54 [    gpio] set_gpio@25                   set_gpio(38, 0)
00:07:54 [   night] dn_start@220                  Starting monitor for light sensor
00:07:54 [    gpio] set_gpio@25                   set_gpio(39, 1)
00:07:54 [    gpio] set_gpio@25                   set_gpio(38, 0)
00:07:54 [    gpio] set_gpio@25                   set_gpio(39, 0)
00:07:54 [    gpio] set_gpio@25                   set_gpio(38, 1)
00:07:54 [    gpio] set_gpio@25                   set_gpio(39, 0)
00:07:54 [    gpio] set_gpio@25                   set_gpio(38, 0)
00:07:54 [   night] set_night_mode@117            Change mode to DAY
00:07:54 [    gpio] set_gpio@25                   set_gpio(39, 0)
00:07:54 [    gpio] set_gpio@25                   set_gpio(38, 1)
00:07:54 [    gpio] set_gpio@25                   set_gpio(39, 0)
00:07:54 [    gpio] set_gpio@25                   set_gpio(38, 0)
00:07:57 [     sdk] sdk_take_jpeg@1777            take jpeg venc_chn(1)   1280x720  q:50
00:07:57 [     sdk] VENC_SaveJpeg@701             JPEG snapshot was received from hardware encoder
00:07:57 [  on_jpg] send_regular_jpeg@76          JPEG was sent to client
00:07:58 [     sdk] sdk_take_jpeg@1777            take jpeg venc_chn(1)   1280x720  q:50
00:07:58 [     sdk] VENC_SaveJpeg@701             JPEG snapshot was received from hardware encoder
00:07:58 [  on_jpg] send_regular_jpeg@76          JPEG was sent to client
00:07:59 [     sdk] sdk_take_jpeg@1777            take jpeg venc_chn(1)   1280x720  q:50
00:07:59 [     sdk] VENC_SaveJpeg@701             JPEG snapshot was received from hardware encoder
00:07:59 [  on_jpg] send_regular_jpeg@76          JPEG was sent to client
00:08:00 [     sdk] sdk_take_jpeg@1777            take jpeg venc_chn(1)   1280x720  q:50
00:08:00 [     sdk] VENC_SaveJpeg@701             JPEG snapshot was received from hardware encoder
00:08:00 [  on_jpg] send_regular_jpeg@76          JPEG was sent to client
00:08:07 [     sdk] sdk_take_jpeg@1777            take jpeg venc_chn(1)   1280x720  q:50
00:08:07 [     sdk] VENC_SaveJpeg@701             JPEG snapshot was received from hardware encoder
00:08:07 [  on_jpg] send_regular_jpeg@76          JPEG was sent to client
00:08:49 [     sdk] venc_read@813                 Timeout get any data from venc channel 0
00:08:49 [     sdk] venc_read@833                 NOTE: Current frame is NULL!
00:08:51 [     sdk] venc_read@813                 Timeout get any data from venc channel 0
00:08:51 [     sdk] venc_read@833                 NOTE: Current frame is NULL!
00:08:53 [     sdk] venc_read@813                 Timeout get any data from venc channel 0
00:08:53 [     sdk] venc_read@833                 NOTE: Current frame is NULL!
00:08:55 [     sdk] venc_read@813                 Timeout get any data from venc channel 0
00:08:55 [     sdk] venc_read@833                 NOTE: Current frame is NULL!
00:08:57 [     sdk] venc_read@813                 Timeout get any data from venc channel 0
00:08:57 [     sdk] venc_read@833                 NOTE: Current frame is NULL!
00:08:57 [     sdk] sdk_take_jpeg@1777            take jpeg venc_chn(1)   1280x720  q:50
00:08:59 [     sdk] venc_read@813                 Timeout get any data from venc channel 0
00:08:59 [     sdk] venc_read@833                 NOTE: Current frame is NULL!
00:09:01 [     sdk] venc_read@813                 Timeout get any data from venc channel 0
00:09:01 [     sdk] venc_read@833                 NOTE: Current frame is NULL!
^C00:09:02 [     log] printf@261                    Interrupt signal received
00:09:02 [    gpio] set_gpio@25                   set_gpio(39, 0)
00:09:02 [    gpio] set_gpio@25                   set_gpio(38, 0)
00:09:02 [     grp] GROUP_UnregisterChn@1813      [   grp] Group: 1 has been start recive picture, not surpport unregister
00:09:02 [     sdk] stop_chn@1735                 Cannot unregister VENC chn 1
00:09:02 [     sdk] stop_chn@1735                 ERR_VPSS_NOT_PERM: The operation is forbidden
00:09:02 [     sdk] stop_chn@1746                 Stopped 0 channel
00:09:02 [     sdk] ISP_thread_proc@160           Shutdown isp_run thread
00:09:02 [      vi] VIU_DRV_DisableChn@1470       [    vi] !! Disable ViChn:0 timeout 120ms!!!
00:09:02 [     sdk] stop_sdk@2983                 Stop sdk Ok!
00:09:02 [    main] main@223                      Shutdown main thread

(At least when running from nfsroot) clock is configured to 27MHz (devmem 0x20030030 32 --> 0x5). Bringing it down to 24MHz (devmem 0x20030030 32 0x1) solves the problem.


1.18.2 JSON5 file still resetting

json5 config file resetting every launch
I know we commented before how the issue was resolved, but if that change could happen with the 1.18.2 version that would be great. I saved a copy as a JSON file to see if that would work and hope Minecraft would prioritize it but this does not seem to be the case.

Yes this is a known issue for older versions cause of json5 usage instead of json.
Out of time reasons I only work on the latest version tho

Ok I guess I won't be using this mod then. Thanks for getting back to me.

Then this can get closed I guess


chore(deps): upgrade dependencies

Upgrades project dependencies. See details in workflow run.

Automatically created by projen via the "upgrade-k8s-25-main" workflow

⚪ Backport skipped
The pull request was not backported as there were no branches to backport to. If this is a mistake, please apply the desired version labels or run the backport tool manually.
Manual backport
To create the backport manually run:
backport --pr 1735

Questions ?
Please refer to the Backport tool documentation