Testing MIB model export with bioimage.io

Hi,
I have prepared a small test network (attached) that has been trained in DeepMIB and exported in onnx and tensorflow/keras formats.
The network is a simple Unet with two down-sampling steps:

The network was trained to detect white spots over a black backfground:

The expected results of the network application (the test image in the attached archive):

Network settings;
Input patch size: 224x224x1
Padding: same
Encoder depth: 2
Number of classes: 2
Number of first filters: 2
Filter size: 3
There is a readme file for the tensorflow model that describes how it can be converted into TensorFlow SavedModel format.
It would be great to know whether these models can be run on bioimage.io
Thanks!
Best regards,
Ilya
DeepMIB_test.zip

Hi Ilya, thanks for your interest in contributing your models and connect your tool to the model zoo. Could you please take a look at our docs here:  https://bioimage.io/docs/#/contribute_models/README
cc @esgomezm is there other resources or help we can provide to Ilya? Please feel free to comment here.

Hi Wei,
we had discussion yesterday on the meeting and the idea was that I provide via issues a small test network exported from Matlab as a quick test to check compatibility with bioimage. Depending on how it goes, I can continue and write a proper output function to generate required metadata.

@IvanHCenalmor can you please take a look at this one? I would suggest downloading the model and seeing if you can consume it with the bioimageio.core library or make the needed changes to make it work. I would then keep the track on this issue so @Ajaxels can take a look at it. Does this sound good?

I had a quick look into this, and conversion of the model into the bioimage.io format works (see snippet for creating a  bioimage.io model from the ONNX weights below). The model seems to run, however it cannot be properly tested because the zip only contains the model input and not the expected model output. (I just used the png for the model output from above, but that does not work).
@Ajaxels: can you please provide the expected output for this model in a suitable format (e.g. tif) as well?
import imageio
import numpy as np
from bioimageio.core.build_spec import build_model

# create the test input and output data as numpy arrays:

# the input -> the model seems to expect inputs of size 224 x 224 pixels, so we crop to this size
# (if not the model test throws an error)
input_ = imageio.imread("../DeepMIB_test/Spots2D_01.tif")[:224, :224]
output = imageio.imread("../DeepMIB_test/test-out.png")
print(input_.shape)
print(output.shape)
np.save("../DeepMIB_test/test-in.npy", input_[None, None])
np.save("../DeepMIB_test/test-out.npy", output[None])

# create doc file
with open("../DeepMIB_test/doc.md", "w") as f:
    f.write("# My Doc")

build_model(
    weight_uri="../DeepMIB_test/onnx/Unet_2layers_224px.onnx",
    test_inputs=["../DeepMIB_test/test-in.npy"],
    test_outputs=["../DeepMIB_test/test-out.npy"],
    input_axes=["bcyx"],
    output_axes=["byxc"],
    output_path="./model-onnx.zip",
    name="test",
    description="test",
    authors=[{"name": "test"}],
    tags=["test"],
    documentation="../DeepMIB_test/doc.md",
    cite=[{"text": "Test et al.", "doi": "test"}],
    opset_version=10,  # IMPORTANT: SET THE CORRECT OPSET VERSION (10 is just a guess)
)

Thanks @Ajaxels!
The labels are still not quite in the formar that we would need; but we don't need to get stuck on this for now.
I managed to produce a valid model export from your weights (using predictions from the model as the expected outputs).
I will bring this up in our meeting today and we will discuss how to proceed and then get back to you.

@constantinpape ,
what format you would like to have? should it be 0 and 255?

What we would need is the exact data that comes out of the model.
Given this pseudocode:
input_ = load_input_patch()  #  the input for the model, which has shape 1x1x224x224 (batch x channel x width x height)
prediction = model(input_)
# prediction has shape 1 x 2 x 224 x 224 (batch x channel x width x height)

we would need to have the prediction data. (Note that your model only accepts inputs of shape 224 x 224, at least when using the onnx weights that you have provided). In particular this means that the data should have 2 channels and should be floating point (presumably in the range 0 to 1).
But as I said, for now I can just generate the predictions myself and everything seems to work. (This should not be done for actual model exports though, because the goal here is to provide test data for the model to check that the export was done correctly, but for our initial tests this is fine.)

you mean probability map (scrores), right? i.e. not the binary segmentation?

I can make that one as well. The problem is to find a suitable format. Can you read mat-files?
I can probably also convert it to numpy matrix in a similar way as you have in the script above.

I scale predictions from floats 0-1 to uint8: 0-255 for the export.
I re-scaled the predictions back to 0-1, but it has probably caused some rounding of values...
Score_Spots2D_01.zip
I can add an option to generate score files in range of 0-1 to make the procedure more straight forward.

Thanks @Ajaxels.
The format is correct and now this almost works. There is still some mismatched between the scores you provide in the .mat file and the predictions from the onnx model.
There are two potential reasons for this:

differences at the boundary because of prediction with tiling
differences in the ONNX version

I will follow up on this later.

I now created a gist to collect all the steps for the model conversion: https://gist.github.com/constantinpape/de4ad27fad437129867716a790c70740.
@Ajaxels: as I mentioned earlier there are two potential issues that cause the test in there to fail:

tiling artifacts because prediction is run differently for your large output image.

Do you think you could provide an input image and the corresponding score / probability output that come from a single pass of the model, without tiling or cropping the results? Note that the ONNX models expects inputs of the exact size 224x224 pixels.

a difference in the ONNX Opset Version. I have just randomly set this to 10, and the model at least runs with this version, but a version mismatch may cause some numeric issue.

Could you find out the Opset Version of your ONNX export?

@constantinpape
Sorry I only today realized that you added more comments!
Here is the updated version, where I modified the input images and the network to detect larger spots on noisy images:
Input image:

Input image+ground truth:

The tested images have the dimensions matching the input patch size of the network.
Folder in the attached archive:

onnx folder -> 2 onnx models exported with opset 10

DeepMIB_test2\onnx\Unet_2layers_noise_224px_allLayers_OS10.onnx  -> all layers of MATLAB network with the final ""Segmentation-Layer", which is a generalized dice loss.
DeepMIB_test2\onnx\Unet_2layers_noise_224px_softmax_OS10.onnx -> the same, but ommiting the segmentation layer, i.e. the final layer is softmax

prediction_images, images for prediction using the trained network

prediction_labels, corresponding labels containing the ground truth

results_labels, resulting bitmap labels (0/1) generated by the network

results_scores, probabilities for predictions (scale from 0 to 1):
The resulting MAT-file has 3 variables:
.imgOut: [2075×2125×2 single] -> predictions
.imgVariable: 'imgOut' -> variable that encodes name of the variable with predictions
.options: [1×1 struct] -> structure with additional options, at the moment has a sequence of axes:
options.dimOrder -> 'yxczt'

Please let me know if this is enough!
DeepMIB_test2.zip

Hi @Ajaxels,
thanks for providing the new file and it works now!
I have updated the gist that explains how to export the model in python: https://gist.github.com/constantinpape/de4ad27fad437129867716a790c70740
And the exported model is available here: https://owncloud.gwdg.de/index.php/s/VTcGgfUDSydzbSZ
So we can conclude that this works in principle!

@constantinpape
Fantastic! Thanks!
I will try to reproduce this on my side using your instructions!

@constantinpape ,
I tested the function and it generates outputs in two places:

there are files in "exported_model" subfolder:

The files are slightly different from the model that you generated.
in the same directory with the script there is another export:

The files look the same as the ones that you've generated, but yaml is missing.

Is it intentional that two models are generated?

Hi @Ajaxels,
you can ignore all the files generated by the script except for exported_model/model-onnx.zip. This contains the zipped model with all relevant files. (The function has to create some temporary files, which is what you see when you run the model. This can be avoided by wrapping this in a temp folder, but I am not sure if this is relevant for you right now.)


Inline suggestion disabled causes no suggestion anymore (VS Code, last v)

this is related to:
https://github.com/codota/TabNine/issues/552
My first experience with tabnine was great because it was only popup suggestions. Now, after some updates, I get only inline suggestions which I find glitchy, confusing and hard to apply. I want to disable inline suggestions but to put them in popup suggestions. I changed inline_suggestion_mode from null to false,  but now tabnine seems to do nothing... What would be the propper settings for this?
gz#20401

@florin-botea
Please see https://github.com/codota/TabNine/issues/581, to see if it fixes the issue.

You can close this.
I'done the following:
-uninstall tabnine
-deleted %appdata%/tabnine
-searched TabNine in %appdata% (cache) and deleted anything I have found
-reinstalled TabNine, seted to Hybrid model
-changed inline_suggestions in tabnine_config.json to false
now its work like a charm!
Please take in consideration to not remove popup suggestions in the future, else we will be very sad :( . Thanks!


IPTC and Exif data discarded

ImageMagick used by exposed discards all the Exif and IPTC data that may be embedded in the original photos by default. But as IPTC data holds information about copyright- and contact information of the photographer (although easily deletable), these information should be kept in the photos wherever possible.

unfortunately imagemagick does have some issues with exif. The proper way to do this is with exiftool but I don't really want to add another dependency.
I think this might be an acceptable loss? If copyright info etc is important you could use the zip function which stores a copy of the original, exif and all.


Re-land "Fix unable to find bundled Java version"

Reverts flutter/flutter#119981
Re-land of https://github.com/flutter/flutter/pull/119244
Fixes https://github.com/flutter/flutter/issues/106674
This was reverted because of a flake in Mac web_tool_tests, however, even after the revert this flake persists: https://ci.chromium.org/p/flutter/builders/prod/Mac web_tool_tests/10455

Firebase pre-submit failures are un-related to this change, but upstream https://github.com/flutter/flutter/issues/120111


Add unwrap to interfaces

It would be useful to have a <T extends Span> unwrap()  method on the Span interface (and similar methods on similar interfaces), similar to JPA's. It would make it easier to call implementation-specific methods.
Before:
    Span span = tracer.buildSpan("test-operation").start();
    long traceId = ((JaegerSpanContext) span.context()).getTraceId();

After:
    Span span = tracer.buildSpan("test-operation").start();
    long traceId = span.context().<JaegerSpanContext>unwrap().getTraceId();

Before:
    JaegerSpanContext expectedContext = (JaegerSpanContext) jaegerSpan.context();

After:
    JaegerSpanContext expectedContext = jaegerSpan.context().unwrap();

Please note there is a bit of discussion in PR 211 about the desirability of adding this to the API

We may want to move this to a cross-language discussion as part of Trace Identifiers: https://github.com/opentracing/specification/issues/123


Make Jumbotron CTA dynamic

Need a dynamic CTA for jumbotron. It's optional.
Also add dynamic {{url}} to the link

Added {{# cta }} wrapper around the CTA button. Also added {{ url }} to make the link dynamic. Should I add separate url for cta link so it isn't set to the default?

This has been addressed. Close #5


Update skylab-dev and common packages.yaml (jedi-ufs-env updates)

Description
Update skylab-dev template and common packages.yaml to reflect changes in the associated PR https://github.com/NOAA-EMC/spack/pull/203.
This PR contains all the changes in #409, #413. If testing is successful, then merging this PR will automatically mark the other two as merged.
Issues
Fixes https://github.com/NOAA-EMC/spack-stack/issues/366
Fixes https://github.com/NOAA-EMC/spack-stack/issues/378
Fixes https://github.com/NOAA-EMC/spack-stack/issues/133
Testing

Testing on Dom's macOS: MISSING
CI testing: MISSING
Testing on one HPC with Intel and GNU: MISSING

I think we are slowly getting there with the CI tests passing. But let's merge https://github.com/NOAA-EMC/spack-stack/pull/413 and https://github.com/NOAA-EMC/spack/pull/201 first, assuming that those tests pass, then update this PR and the associated spack PR, update the description in both, and hopefully get a last round of CI tests to pass quickly.


Review defaults in Helper and Bicep

Review the defaults in the helper and bicep to cut down on number of parameters for the majority of use-cases, and maybe remove some configurations that we should not be advocating for.
For example, AAD and RBAC and no-admin should be true by default,

Can this be combined with the work @nellyk is doing around the default for outbound?


Price cross in orderbook

In my test application, the orderbook is maintained by subscribing to the L2Update channel. When receiving a snapshot message, it creates a orderbook struct and it handles each l2update message after that. Sometimes I see the price crosses, i.e. best bid is higher than the best ask.
It might be a bug in my code, but I also want to check with users of this library and see if you are having the same issue.

you're meant to use also the sequence no to determine if you need to reconnect or not.
it's hard to see what the problem is without looking at the code in it's entirety.

@zhengyangfeng00 any update?


Lots of improvements

Added tests
Added "stalled branch" detection
Added "only submodules" filter
Speedup
Covered "Repo has no commits yet" state

Support for git submodules would be awesome! Great work, hope it can be incorporated into mgitstatus.

Git submodules is a must-have! Thanks for adding this support.
However, I have 2 issues:

It doesn't work on ZSH: git config --file=.gitmodules --get-regexp ^^submodule.*\.path$ | cut -d " " -f 2
It doesn't work on macOS: find -L "$DIR" $FIND_OPTS -depth -name ".git" -printf "%h\0" (no -printf flag on BSD's find)

It worked great on Bash/Linux; I'm hoping this can be resolved and merged soon.
Cheers!

@braydentraas Does gfind solve the issue on Mac: https://superuser.com/a/293200/187576?

@ceremcem Yes it does, but now readarray isn't available on mac/BSD either (or zsh even on linux)

As I don't have Mac on hand, I'm not sure how to proceed here. Are you able to implement the necessary changes and test on Mac? (I even didn't know this repo has been supporting Mac)

@ceremcem I've made a few changes and submitted a PR to your repo. Works for me now on MacOS/ZSH and Linux/Bash
Thanks again!

Any idea why this is not yet merged?

Don't know.

@ceremcem thank you for your updates!


Conditional imports generate modulepreloads for everything

The vite_entry_script_tags twig function adds modulepreload links for conditional imports (if (condition) { import('some-module.ts'); }).
This should probably not happen, since the whole point of conditional imports is to avoid loading the resource at all when it's not needed.

Hi @thor-juhasz, you're totally right. I fix this and add preloadDynamicImports option to vite_entry_link_tags if someone still wanted this behavior https://symfony-vite.pentatrion.com/guide/twig-functions.html#vite-entry-link-tags

May I suggest avoiding breaking changes in the default behvaior though.
Have the default value for preloadDynamicImports as true, until you release the next major version?

Ah I missed that you were making a major anyway.
I did have a look at how this bundle generates the module preloads.. I was under the impression that this block generates the preloads for "all" dependencies.. Not just the dynamically imported ones.
Won't this stop all module preloads completely?

There'is no breaking changes,
preloadDynamicImports is only available in the v4.x release (11-07-2023)

Yes you are right, it must be fixed... I reopen the issue

May I suggest a way to resolve this:
vite-plugin-symfony generates the entrypoints.json. In each entrypoint, you could introduce a new key, dynamicImports, which naturally, will list all the dynamic imports.
Then, in this bundle, if preloadDynamicImports is true, add the dynamic imports to the list of modules to be preloaded.
But as you are already in a new major release, this is considered a breaking change.
So, having true as the default value for this option avoids unexpected behavior from those that upgrade from v4.0.0 to v4.1.0 (assuming that would be the version that implements this).
And finally, I suggest releasing v4.0.2 which reverts v4.0.1 since:

it's a patch version but implements a feature (does this project follow semver?)
the feature does not behave as the option name suggests (removing all module preloads as opposed to only dynamic ones)
the default behavior changes the behavior from v4.0.0 (breaking change)

thank you for your feedback, I have just developed the patch which follows your advice, I am thinking of putting it in a 4.0.2 which fixes a bug since the release date of the 4.0.0 is from this morning.

I try to follow semver as much as possible but until recently imports and dynamic imports were not preloaded at all due to a bug.  I guess it won't be so disturbing ?

Possible.. 😅 but most devs never end up interacting with a repository even if they find issues, just hoping others take care of it (I'm guilty of that myself as well). 1500 weekly downloads (according to npm) is still rather low (webpack-encore with 160k) so..... 🤷🏼♂

yes we are not yet at the level of webpack-encore :smile:


Allow serializing non-HTMLForm elements

My use case here is that I have a form inside a table row, unfortunately html does not allow that so we need to have something like: form > table > tr, tr, tr where every tr is a virtual form of sorts. If I serialize the form obviously I get all trs together which isn't what I need, I just want the inputs of a given tr tag. The patch allows this.

Why can't you have form elements inside a table? I am pretty certain you can and just need an outer closing form tag.
I am hesitant to accept this PR because I don't see why or how html is preventing you from having a form enclosing your input.

You can have a form element inside a td, but not around nor within a tr (enclosing multiple td's). That just breaks the markup.
See https://jsfiddle.net/yrma5gmv/ for a demo.. As you can see in the console interestingly the form.one works fine and the inputs are accessible as being in the form, however the styling breaks because the elements aren't really part of the form anymore in the DOM, as the DOM gets rewritten to something like:

And I presume that you want to have two form in the same table for a reason and can't split up the table into two tables any other way?
Is using divs an option here instead?
Alternatively, could you just use one form for the whole thing and prefix the input names with sections like <input name="section1.name">...<input name="section2.name">. Tho I suppose that would not let you know which form section was submitted.
I guess what I am trying to get at is a better understanding of your particular UX flow that you are creating because I am almost certain there is a way to do it without changing the behavior of this module. The reason I am hesitant to change the behavior like that is because the stated purpose of the module is serialize a form.
To make this possible tho I could see an API being expose like "serialize single element". Then you could pass it in whatever you want maybe? One issue I see with the PR here is whether it captures all form elements or we would constantly be updating with new tags, etc and I want to avoid that too.

I don't really need a form per se at all, as it's all handled by JS, I just need the form element to take care of a submit event in case user presses enter, then I need a form element to use with this library..
Using divs would technically be an option except I have a table that works fine and it's an actual table in which using divs would be a bit absurd.
I just see this patch as a nice added value in case someone passes something that's not a form, it'll find form elements in it. I don't see any reason to be bound to a form element per se in this day and age of one page apps and full JS frontends. I see your reluctance with regard to the list of tags included, but tbh new form inputs are not exactly a common occurrence and it's only a fallback method so as long as users provide a form tag the lib is future proof.

Still needed in 2019 😄


Update the way of getting length of dataloader

Previously in dist_inference_one_layer, node_mini_batch_gnn_predict, dist_minibatch_inference, edge_mini_batch_gnn_predict, we use len(list(dataloader)) to get the length of a dataload which is costy. Change the implementation to use len(datalaoder)
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Modification on GSgnnNodeSemiSupDataLoader looks good to me


added forward invocation

in case when you don't want to make the user to move logic from UIViewControllers to another BehaviorDefiner.

you must assign forwardDelegate before you will replace delegate with Behavior

Since we determined in #19 that BLKDelegateSplitter provides the same functionality, I'm going to close this.


linear policy?

Hi I got here after reading your paper "Towards Generalization and Simplicity in Continuous Control" and I was wondering if the file https://github.com/aravindr93/mjrl/blob/master/mjrl/policies/gaussian_mlp.py was the one used in your paper? The paper claims to use linear policies but it seems this network in an MLP?

Hi @zafarali thanks for the interest. The paper indeed uses linear policy. For the paper, we used a bare numpy implementation without any autograd (I am planning to make this public as well, but will likely not do any further development in that repo). To get linear policies with this repo, you can simply strip off the hidden layers in the MLP policy. Please let me know if some further clarification is needed.

Thanks for your reply!
So is it safe to say that no gpu's were needed? How long did your experiments take?
I found this surrogate loss in the REINFORCE code, can you elaborate a bit on this? What does "CPI" stand for and will using this surrogate re-create the vanilla REINFORCE algorithm?
https://github.com/aravindr93/mjrl/blob/7705640fed18f9740e2d4bc9ee79bfee4133b044/mjrl/algos/batch_reinforce.py#L37-L44

Yes, no GPUs are needed for the OpenAI gym continuous control tasks. This is standard practice in other code bases as well (e.g. rallab).
The experiments can take anywhere between 5 minutes to 2 hours depending on complexity of task (from simple swimmer to complex humanoid) on a single good workstation.
CPI stands for Conservative Policy Iteration (paper: http://www.cs.cmu.edu/~./jcl/papers/aoarl/Final.pdf), which was the paper that originally proposed that surrogate. This gradient of the surrogate will be identical to REINFORCE as proved in that paper. The LR will indeed equal 1 when using on-policy samples. This general code structure is useful for other algorithms that use off-policy samples (e.g. PPO, Q-prop).

Thanks, this is super useful!

Looking at linear_nn_comparison.py, together with a linear policy an mlp is used as the value function approximator. Is this what was used for the experiments in the paper as well?


Not Found in Apt after following Steps.

Operating system: Ubuntu 16.04.1 LTS
Messenger for Desktop version: Current

I followed the steps of adding the GPG key, then adding the apt sources, then apt update, and still, I get:
user@hostname:/etc/apt/sources.list.d$ sudo apt-get install messengerfordesktop
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package messengerfordesktop

Ideas?

Duplicate https://github.com/aluxian/Messenger-for-Desktop/issues/1133


Support imports in __init__.py instead of full paths to objects

In my package all functions are accessible from the main namespace by importing them in the __init__.py file (see https://github.com/frazane/scoringrules/blob/main/scoringrules/init.py). However, when I use e.g. ::: scoringrules.brier_score in the markdown files I get an error ('scoringrules.brier_score' not found). On the other hand, ::: scoringrules._brier.brier_score works.
It would be great if one could use the same namespaces as defined by the public API instead of having to write the full paths.

It's possible. The feature will be implemented in the next version.


Current master (1d5c07c4) panics when built with golang 1.17.1

Summary
Cloned a fresh copy of the current master (1d5c07c4) and built it with go version go1.17.1 linux/amd64 successfully.  But the resulting executable panics when simply asking for the version.
ldeffenb@osm:~$ mkdir master2
ldeffenb@osm:~$ cd master2
ldeffenb@osm:~/master2$ git clone https://github.com/ethersphere/bee.git
Cloning into 'bee'...
remote: Enumerating objects: 20253, done.
remote: Counting objects: 100% (1604/1604), done.
remote: Compressing objects: 100% (837/837), done.
remote: Total 20253 (delta 975), reused 1205 (delta 729), pack-reused 18649
Receiving objects: 100% (20253/20253), 6.34 MiB | 6.34 MiB/s, done.
Resolving deltas: 100% (13455/13455), done.
ldeffenb@osm:~/master2$ cd bee
ldeffenb@osm:~/master2/bee$ make binary
mkdir dist
go version
go version go1.17.1 linux/amd64
go build -trimpath -ldflags "-s -w -X github.com/ethersphere/bee.commitHash=""1d5c07c4"" -X github.com/ethersphere/bee.commitTime=""1631792277""" -o dist/bee ./cmd/bee
ldeffenb@osm:~/master2/bee$ dist/bee version
panic: qtls.ClientHelloInfo doesn't match

goroutine 1 [running]:
github.com/marten-seemann/qtls-go1-15.init.0()
        github.com/marten-seemann/qtls-go1-15@v0.1.1/unsafe.go:20 +0x132

Steps to reproduce
Clone a new copy, build it with golang 1.17.1 and try to run dist/bee version
Expected behavior
If it built, I expected it to run.
Actual behavior
Panics when simply asking for version.

Note:  I rolled back my golang to go version go1.16.1 linux/amd64 and it builds and runs successfully.  As I mentioned in #2515, it would be good if the makefile could validate the required golang version to avoid very confusing error messages.  Especially when 1.15 and 1.17 both fail in different ways, but 1.16 works!

there's an open PR to upgrade to go 1.17. it doesn't work out of the box. it will be handled soon.


Handle invalid Addresses

https://github.com/AztecProtocol/aztec-packages/issues/8969 will require computing the Point that backs an AztecAddress from its x coordinate, but the x coordinate might not be in the curve in the first place. If this happens, then we won't be able to do elliptic curve math on it, so we need an escape hatch.
Since we define addresses as values that represent the x coordinate of a point in the curve, these address values are truly invalid, and cannot be used for anything. No decryption can be done with them, nor can they do things such as nullify notes since they won't be able to compute this invalid x coord. Therefore, it is fine to simply skip whatever action needed to be done to avoid King of the Hill attack vectors. For example, in the case of sending a note we should not send it at all.
This means that we should likely change AztecAddress::to_point so that it results something like an Option, to handle the invalid derivation scenario.

Some hopefully helpful theory:
For grumpkin, $y^2 = x^3 - 17$. There exist values $x \in \mathbb{F}$ for which no $y$ satisfies this equation. What does that mean? If we take some $x$, and we compute $t = x^3 - 17$, then $\sqrt{t}$ does not exist in $\mathbb{F}$.
Why don't square roots always exist in prime finite fields? They just don't.
Take $\mathbb{F}_7 = {0, 1, 2, 3, 4, 5, 6}$. $0^2 = 0$, $1^2 = 1$, $2^2 = 4$, $3^2 = 9 = 2$, $4^2 = 16 = 2$, $5^2 = 25 = 4$, $6^2 = 36 = 1$. None of the answers were 3 or 5, so 3 and 5 have no square root in $\mathbb{F}_7$.
So in our context, if we shrunk our field to $\mathbb{F}_7$, and someone said "My address is 3", then for $x = 3$, $x^3 - 17 = 27 - 17 = 10 = 3$, and we just showed that $3$ has no square root in $\mathbb{F}_7$, so address=3 is an invalid address in that finite field.
In a circuit, how can we efficiently demonstrate that an address $x$ is invalid? We compute $t = x^3 - 17$ and then we pass $t$ into the sqrt function below to check whether it has a square root, and return it if it does.
This code (from another project I was working on) should do it:
global KNOWN_NON_RESIDUE = 5;

unconstrained fn __sqrt(x: Field) -> (bool, Field) {
    let is_sq = std::ec::is_square(x);
    let mut maybe_sqrt = 0;
    if is_sq {
        maybe_sqrt = std::ec::sqrt(x);
        assert(maybe_sqrt * maybe_sqrt == x); // to catch bugs
    } else {
        // If x a non-residue, x * KNOWN_NON_RESIDUE will be a residue, since -1 * -1 = 1 when considering their legendre symbols.
        let demo_not_square = x * KNOWN_NON_RESIDUE;
        maybe_sqrt = std::ec::sqrt(demo_not_square);
        assert(maybe_sqrt * maybe_sqrt == demo_not_square); // to catch bugs
    }
    (is_sq, maybe_sqrt)
}

fn sqrt(x: Field) -> (bool, Field) {
    let (is_sq, maybe_sqrt) = __sqrt(x);
    let mut out: (bool, Field) = (false, 0);
    if is_sq {
        let sqrt = maybe_sqrt;
        assert(sqrt * sqrt == x);
        out = (true, sqrt);
    } else {
        let not_sqrt = maybe_sqrt;
        let demo_x_not_square = x * KNOWN_NON_RESIDUE;
        // Given that this is square, it implies x is not square.
        assert(not_sqrt * not_sqrt == demo_x_not_square);
    }
    out
}

Saying a value $t \in \mathbb{F}$ is a "quadratic non residue" is basically a mathsy way of saying "$t$ does not have a square root". Conversely, saying a value "is a quadratic residue" effectively means it does have a square root.
If you take a known non-residue (5 is a non-residue in the Noir Field), then:
"Modulo a prime, the product of two nonresidues is a residue and the product of a nonresidue and a (nonzero) residue is a nonresidue."
In other words:
If we learn that $t$ is square, from the unconstrained function, then we can provide its square root to the constrained function.
If we learn that $t$ is not a square, from the unconstrained function, then we cannot provide its square root. But we can compute t * KNOWN_NON_RESIDUE, the result of which, according to the rule above, is a residue. So we can provide the square root of t * KNOWN_NON_RESIDUE as a way of demonstrating that $t$ has no square root in this case!


 is moved around in project files

This happens in all fsproj and csproj files in a .NET Framework 4.6.1 only solution.
The tag is :
When doing an "install", the tag is moved before all the Choose tags. When  doing "auto-restore on", or possibly some other command, the tag is moved to the very bottom of the Project tag.
In other words, the tag moves around needlessly, and cause unneeded changes in commits unless taken care of.
Known workarounds
Alternative 1 : Go through files to be committed, and remove unneeded changes.
Alternative 2 : Run some command that doesn't effectively do anything except restore the placement of the tag. I do an "auto-restore on" to achieve this.
At some point in my development I had to decide which placement I wanted to be the right one, and I decided to go with the placement at the bottom. So I do "install" and then follow up by some other "dummy" command that fixes the placement.
Other info
Version 5.172.2

Also see #3209. Could be the same issue.

Can you please upload a zip with a repro?
Bent Tranberg<EMAIL_ADDRESS>schrieb am Mi., 20. Juni 2018,
19:41:

Also see #3209 https://github.com/fsprojects/Paket/issues/3209. Could
be the same issue.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/fsprojects/Paket/issues/3263#issuecomment-398836296,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNKXcgXSfejAh0iPlAk1E9s93Bcmoks5t-ok1gaJpZM4UvsvZ
.

@forki there is a repro in the other issue. Problem is that paket moves it to the end which potentially changes semantics


Add mailer example

@razonyang looks great overall but still needs some polishing. If you prefer, I can do it myself.
Also, seems we're lacking HTML helper that's useful for views. Going to extract it soon.

@samdark I've fixed some minor problems, as to the rest of problems(lack of enough knowledge and good idea), I need your help with them, you can do it yourself, thank you :smile:

Btw, I met some problems during the time of writing this example, it seems hard to retrieve some services, such as logger, mailer since yii\helpers\Yii was removed.
For example, I prefer to separate form logic from controller like this:
public function contact()
{
    $form = new ContactForm();
    $response = $form->handle();
    // ...
}

class ContactForm
{
    // What is the proper way to inject dependencies here?
}

Another missing feature of this example is that it doesn't perform any validations(such as email). :(

Logger and Mailer are supposed to be injected via constructors. Either you need to register ContactForm in DI container (then dependencies will be injected automatically) or pass instances from controller:
public function contact(Mailer $mailer, Logger $logger)
{
    $form = new ContactForm($mailer, $logger);
    $response = $form->handle();
    // ...
}

class ContactForm
{
    public function __construct(Mailer $mailer, Logger $logger)
    {
          // ...
    }
}

Merged! Moved contact form into its own controller afterwards.

Thank you very much.


Ability to work in Next.js

I've tried to make it more customisable and improve implementation in Next.js environment.
You can pass the same configurable properties as props to the <SpatialNavigation> component:
https://github.com/luke-chang/js-spatial-navigation#configuration
See example:
https://codesandbox.io/s/6j0p7jv963
You can also change the focusable and active class names and customise initialisation, again by passing the following props to the <SpatialNavigation> component

focusableClassName: String e.g 'js-focusable'
activeClassName: String e.g. 'is-active'
customInit: Function e.g. customInit(config) { ... }

Keen to get your thoughts...

Thanks 😄


Add minimum chromosome length parameter to (pairwise) macro-synteny microservices

This is very similar to issue #41, except chromosomes will be filtered by their physical length instead of how many genes they have. We'll call this parameter optionalChromosomeLength in the protocol buffer files and min_chromosome_length in the Python code.
Make it so!

Done in commit 1ade8e144d26994c462e2961d11ff3350194535e. Closing.


Players falling off the screen

In meetings, players are falling off the screen instead of being put on other pages. (Same goes for the shapeshifter menu.)

In meetings, players are falling off the screen instead of being put on other pages. (Same goes for the shapeshifter menu.)

I am using the latest artifact btw.

Do you use any other mods?

Do you use any other mods?

Yes.

BetterPolus
ColorsPlus
Reactor (of course)
Submerged

Giving a quick look I don't see any incompatibilities. Could you try disabling them one by one? You can use Reactor.Debugger plugin to spawn dummies to overflow first page

Testing is a bit hard, because I do it alone right now. Last time I tested, I had to open 19 instances of Among Us. :/

As I just said you can use freeplay and Reactor.Debugger mod to spawn enough dummies

Oh, I didn't know that. I'm not sure if I can test everything today though.

It still happens with just CrowdedMod and Reactor installed.

Could you try with https://github.com/CrowdedMods/CrowdedMod/actions/runs/2521993373 ?

In this patch, the ability to turn the page on the vote is gone

https://github.com/CrowdedMods/CrowdedMod/actions/runs/2523720996

Well I've been doing this through Github editor but it needs more research
Will fix soon

Hello!
https://github.com/CrowdedMods/CrowdedMod/actions/runs/2529236406

Voting has been fixed in this patch, but the shapeshifter menu remains dropdown

Closing as completed and merging #41
Please make a new issue for shapeshifter UI


Add timezone to chrome driver for ModelDiffTest

ModelDiffTest uses chrome browser which fails in timezones other than UTC. This PR explicitly sets the timezone for the chrome driver

I think the sonar failure should go away if you raise your PRs by checking out this project and making branches from there, rather than from your fork.

I've come across the readme mentioning to raise the PR via a fork. I'll create the branch from main directly next time.


Re creating the issue - Unexpected behaviour when pasting TOC in the editor

I am creating the issue again because I haven't received any help on the issue.
The issue is https://github.com/nhn/tui.editor/issues/2004
Sorry for the inconvenience.

@ats1999
Sorry for late replying for your prev issue.
It seems like a complicated issue, so I'll check it within this week and comment here!

Sure, thanks


pub.dev error: No valid SDK

We are getting this error at pub.dev:

Details also at https://pub.dev/api/packages/effective_dart/metrics?pretty&reports
~The reason is probably because of https://github.com/dart-lang/pana/issues/591 which has been fixed by https://github.com/dart-lang/pana/pull/592 (but will be published later).~

Temporary fixed in https://github.com/tenhobi/effective_dart/commit/e9620c1b5cd2815a184dcefde37246160b2d26ec


Updated FAQ for R UMAP implementations

See here: #227
Also fixed some typos and added a link to the reticulate documentation.

Coverage remained the same at 94.536% when pulling 40ffab4f22a13a86e8556ffe87700bbd0169ca43 on vicramr:master into ec860e9280dcd4a80c820c73c4cd5fb77cd59278 on lmcinnes:master.

Thanks!


Cannot validate domain (possibly because it is pure IPv6)

Log entry (with domain and IP changed)
2016-12-20T15:11:35.539614335Z time="2016-12-20T15:11:35Z" level=info msg="process certificates requests for ingresses" context=kubelego
2016-12-20T15:11:35.540744315Z time="2016-12-20T15:11:35Z" level=info msg="creating new secret" context=secret name=echomap-tls namespace=default
2016-12-20T15:11:35.540765345Z time="2016-12-20T15:11:35Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=echomap namespace=default
2016-12-20T15:11:35.540782762Z time="2016-12-20T15:11:35Z" level=info msg="requesting certificate for foo.example.com,bar.example.com" context="ingress_tls" name=echomap namespace=default
2016-12-20T15:12:49.697426907Z time="2016-12-20T15:12:49Z" level=warning msg="authorization failed after 1m0s: reachabily test failed: Get http://bar.example.com/.well-known/acme-challenge/_selftest: dial tcp [here:goes:my:correct:ipv6:address]:80: connect: network is unreachable" context=acme domain=bar.example.com
2016-12-20T15:12:56.120965529Z time="2016-12-20T15:12:56Z" level=warning msg="authorization failed after 1m0s: reachabily test failed: Get http://foo.example.com/.well-known/acme-challenge/_selftest: dial tcp [here:goes:my:correct:ipv6:address]:80: connect: network is unreachable" context=acme domain=foo.example.com
2016-12-20T15:12:56.121006599Z time="2016-12-20T15:12:56Z" level=error msg="Error while process certificate requests: no domain could be authorized successfully" context=kubelego

The trick is that I can easily go to http://foo.example.com/.well-known/acme-challenge/_selftest from absolutely remote IPv6 machine. No autorization, nothing is required, just works.
According to
https://community.letsencrypt.org/t/support-for-ipv6-only-hosts/354/60
and
https://letsencrypt.org/2016/07/26/full-ipv6-support.html
Pure IPv6 is supported. Unfortunately in this case I cannot plain IPv4, so I can only rely on IPv6.

Also I've looked through the code, it seems like the query is made not by Let's Encrypt, but by the code of Kube Lego.
I cannot kubectl into kube-lego container, as the containes is pretty stripped of shell tools, but I've logged in into neighboring container in the same machine as kube-lego and it can "curl" the url just fine

Maybe it is related to
https://github.com/golang/go/issues/8453?

Or like https://github.com/golang/go/issues/9801

Btw, if this turns out hard to fix, can we make self-check optional?

"fixed" it just for myself by throwing away self-check and building custom image. This is just FYI to show that this was the issue. Lets Encrypt issues the certificate without a problem

Hey @igor-petruk, how did you tackle building the custom image? I am running into a situation where the self-check is failing due to issues with NAT, but the service is perfectly reachable from the outside. Disabling the self check would help a lot.

Hi,
I've just commented out the check, built a custom image and I've put it
into my internal docker registry. That check is probably formally needed by
Let's Encrypt, but it is terribly unreliable.
2017-05-29 9:11 GMT+01:00 WillemdeKok<EMAIL_ADDRESS>
Hey @igor-petruk https://github.com/igor-petruk, how did you tackle
building the custom image? I am running into a situation where the
self-check is failing due to issues with NAT, but the service is perfectly
reachable from the outside. Disabling the self check would help a lot.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jetstack/kube-lego/issues/66#issuecomment-304600456,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABWTIMlyp4_-uWBObnw4wucKSGo-ymlQks5r-n2WgaJpZM4LR9As
.


notation sign error messages are not helpful to understand what parameter is missing

What is the areas you experience the issue in?
Notation CLI
What is not working as expected?
The following command notation sign --signature-format cose --key $REMOTE_KEY_NAME $DEST_IMAGE returns the following error message: Error: missing reference
From that error message is not clear what reference is missing. Actually the error is even more confusing because reference assumes reference to the artifact and the actual error was missing key.
What did you expect to happen?
Meaningful error like: "Missing key parameter required for signing" or missing "Artifact reference required for signing".
How can we reproduce it?
Issue the command listed above.
Describe your environment
N/A
What is the version of your Notation CLI or Notation Library?
Notation: Notary v2, A tool to sign, store, and verify artifacts.
Version:     1.0.0-rc.3
Go version:  go1.20.1
Git commit:  233c0ea71edfb68b951eb54a739a101e2a05cd26

Move to post v1 release, as it is not critical for v1 release. cc @vaninrao10

Fixed by https://github.com/notaryproject/notation/pull/810


Contributing Guide link on README.md is not working

Visit README.md and click on Contributing Guide link : https://github.com/GoogleCloudPlatform/generative-ai/CONTRIBUTING.md

Please review PR#8


Grouped Update Messages Incorrect

Is there an existing issue for this?

[X] I have searched the existing issues

Package ecosystem
Nuget
Package manager version
No response
Language version
C# 12
Manifest location and content before the Dependabot update
No response
dependabot.yml content
No response
Updated dependency
No response
What you expected to see, versus what you actually saw
Nuget grouped updates are updating the packages based on grouping parameters, but the update message is not matching the grouping parameters.
For example, I would only see Microsoft based updates in my microsoft group, but I'm also seeing other packages referenced in the update message.
For the actual file updates, I'm only seeing the packages matched by the grouping filers. Here is what my grouping looks like:
groups:
      microsoft:
        patterns:
          - 'Microsoft*'
      default:
        patterns:
          - '*'
        exclude-patterns:
          - 'Microsoft*'

Native package manager behavior
No response
Images of the diff or a link to the PR, issue, or logs
No response
Smallest manifest that reproduces the issue
No response

@watercable76 , @honeyankit is about to deploy #9240 which should resolve this.

I am unsure if this is related, but I've noticed that even though the grouped updates work, the message is incorrect because there are duplicate entries.


When will ESP32C2's WiFi and BLE library be supported (IDFGH-7451)

When will ESP32C2's WiFi and BLE library be supported?
I'm going to test it with esp8684H4.

Hi @dreamcmi
We have already support now!
Thanks.

Thanks for reporting, feel free to reopen.


Ever-increasing memory with a custom runner and a single spec file

Subject of the issue
In a custom RSpec runner I'm using, I've noticed an ever-increasing memory usage with each spec file (the same file, actually) that the runner executes. I don't know if this expected or not, so I'm opening an issue here to get your insights as well.
Your environment

Ruby version: 2.5.5 (also tested with  2.7.1)
rspec-core version: 3.9.2

Steps to reproduce
Clone the reproduction repository:
$ git clone git@github.com:agis/rspec-memory-issue.git
$ cd rspec-memory-issue
$ bundle install
$ bundle exec ruby test.rb

Expected behavior
I'd expect memory usage to reach a plateau.
Actual behavior
Memory usage grows indefinitely (reaches some gigabytes after 2-3 minutes). This is probably due to the fact that RSpec creates a class for each describe/it block if the spec file. However, these classes never get garbage collected (which is expected I suppose), but they also get redefined every time the same file gets loaded.
For instance, ObjectSpace.each_object(RSpec::Core::ExampleGroup).each.count shows ever increasing values with each loop:
iteration #1: 28 (a few kilobytes)
...
iteration #1096: 3232 (100MB)
...
iteration #4100: 8552 (325MB)
.....

Likewise, the defined classes look like this:
ObjectSpace.each_object(Class) { |x| puts x }
#...
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::HdisajfidajidjsIajfdklsajFkldsajLkfdsa
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::Hmmmmmmm::HeyHeyHey
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::HdisajfidajidjsIajfdklsajFkldsajLkfdsa::FkofkofkoekoFkeoAkofeKaokfEo
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::HdisajfidajidjsIajfdklsajFkldsajLkfdsa
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::Hmmmmmmm::HeyHeyHey
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::Hmmmmmmm
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::HdisajfidajidjsIajfdklsajFkldsajLkfdsa::FkofkofkoekoFkeoAkofeKaokfEo
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::HdisajfidajidjsIajfdklsajFkldsajLkfdsa
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::HdisajfidajidjsIajfdklsajFkldsajLkfdsa::FkofkofkoekoFkeoAkofeKaokfEo
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::Hmmmmmmm::HeyHeyHey
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::Hmmmmmmm
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::Hmmmmmmm
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::HdisajfidajidjsIajfdklsajFkldsajLkfdsa::FkofkofkoekoFkeoAkofeKaokfEo
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::HdisajfidajidjsIajfdklsajFkldsajLkfdsa::FkofkofkoekoFkeoAkofeKaokfEo
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::HdisajfidajidjsIajfdklsajFkldsajLkfdsa
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::Hmmmmmmm::HeyHeyHey
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::Hmmmmmmm
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::Hmmmmmmm::HeyHeyHey
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::Hmmmmmmm
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::HdisajfidajidjsIajfdklsajFkldsajLkfdsa::FkofkofkoekoFkeoAkofeKaokfEo
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::HdisajfidajidjsIajfdklsajFkldsajLkfdsa
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::Hmmmmmmm::HeyHeyHey
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::Hmmmmmmm
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::HdisajfidajidjsIajfdklsajFkldsajLkfdsa::FkofkofkoekoFkeoAkofeKaokfEo
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::HdisajfidajidjsIajfdklsajFkldsajLkfdsa
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::Hmmmmmmm::HeyHeyHey
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo::Hmmmmmmm
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo
RSpec::ExampleGroups::SomethingWsomewhereFoFoFo
=> 45350

So they're repeatedly defined as different class objects. Is this expected behavior?
Thanks in advance.

So they're repeatedly defined as different class objects. Is this expected behavior?

Yes, you've called reset which clears the old constants and allows the specs to be reloaded, you can see the same behaviour from plain Ruby.
# Repeated class re-opening does not creating new classes.
2.7.1 :001 > class AC; class DC; end; end
 => nil
2.7.1 :002 > class AC; class DC; end; end
 => nil
2.7.1 :003 > x = []; ObjectSpace.each_object(Class) { |y| x << y if y.to_s =~ /^AC/ }; x
 => [AC::DC, AC]

# However removing the constant does allow new classes to be created with the same name.
2.7.1 :004 > AC.send :remove_const, "DC"
 => AC::DC
2.7.1 :005 > class AC; class DC; end; end
 => nil
2.7.1 :006 > AC.send :remove_const, "DC"
 => AC::DC
2.7.1 :007 > class AC; class DC; end; end
 => nil
2.7.1 :008 >  x = []; ObjectSpace.each_object(Class) { |y| x << y if y.to_s =~ /^AC/ }; x
 => [AC::DC, AC::DC, AC::DC, AC]

Why do we do this? Because otherwise reloaded files could conflict, its a lot easier to remove the whole thing than it is to try to work out whats changed. Theres also no guarantee you'd be running the same files again, so it makes sense to clean house.
In fact if you skip the reset you should find they get different names (there's conflict detection code), to the "world" that the file is reloaded is irrelevant, it's a new spec file to load and parse.
Would we expect memory usage to go up? Probably not, running this with a manual GC.compact does seem to keep the memory usage down, but not plateau, so there is possible a leak somewhere, interestingly your memory profile output focuses a lot on the option parser.

Would we expect memory usage to go up? Probably not, running this with a manual GC.compact does seem to keep the memory usage down, but not plateau, so there is possible a leak somewhere, interestingly your memory profile output focuses a lot on the option parser.

I've been running GC.compact every 1000 iterations, yet after 10k iterations memory is at ~800MB (from a few kilobytes in the first iterations), so yes, it doesn't make a significant difference.
The memory profile points to optparse as far as "allocated memory/objects" are concerned. But this memory is actually reclaimed by MRI, so I believe it's not relevant in our case. We actually care about "retained memory/objects", and the profile points at rspec-core and specifically rspec/core/metadata.rb.
For reference, I'm attaching the memory profile I took: memory_profile_3453_1601972084.txt Interestingly, the profile reports only a few megabytes (150MB) worth of retained data whereas the dump was taken when the process' RSS was at 2GB. So I suspect that the profile is not accurate. I'll try to dig into this.

That said, if the class objects are never cleared as you demonstrated and are never garbage collected (which they aren't), isn't it expected for the memory to be ever-increasing?

That said, if the class objects are never cleared as you demonstrated and are never garbage collected (which they aren't), isn't it expected for the memory to be ever-increasing?

They are cleared by us, the output you are seeing is their name, not the classes themselves. Unless remove_const does actually keep the old class around somewhere, in which case it sound like a Ruby bug, we are keeping no reference to the classes ourselves.

Actually they're class objects (i.e the actual classes themselves). For example:
[7] pry(main)> whereami

From: /home/hyu/dev/rspec-memleak/test.rb:38 :

    33: sleep 3
    34:
    35: loop do
    36:   binding.pry if $debug
    37:
 => 38:   puts "run #{i}"
    39:   i += 1
    40:
    41:   RSpec.reset
    42:   RSpec.clear_examples
    43:

# demonstrating that it's actually classes that are kept around
[12] pry(main)> RSpec.reset
=> nil
[13] pry(main)> RSpec.clear_examples
=> {}
[14] pry(main)> x = ObjectSpace.each_object(Class).to_a.last
=> RSpec::ExampleGroups::SomethingWsomewhereFoFoFo
[15] pry(main)> x.object_id
=><PHONE_NUMBER>0120
[16] pry(main)> x.new
=> #<RSpec::ExampleGroups::SomethingWsomewhereFoFoFo (no description provided)>
[20] pry(main)> x.class
=> Class
[21] pry(main)> x.superclass
=> RSpec::Core::ExampleGroup

[18] pry(main)> ObjectSpace.each_object(Class).to_a.count
=> 8251
[19] pry(main)> puts i
1313
=> nil

Open to suggested improvements, but this isn't a memory leak, it's Ruby being Ruby...

I'll investigate more, cause something looks off here.  I'll have to get familiar with RSpec's inner workings, but in the meantime, if you have any pointers or ideas, let me know please.

No as far as I know we don't hold any reference after a rest, if you find one feel free to open a PR.

Closing due to inactivity during the monorepo migration, I'm pretty certain theres not much we can do here anyway.


[Cloud SQL Proxy] Encrypt/decrypt passwords with KMS

Currently the MySQL root password and hive user password are hardcoded in the script, which isn't very secure.
I suggest we update the script to fetch those secrets from a GCS bucket and decrypt them using keys stored in KMS.
If you agree, then I'm happy to write a patch for it. I should be able to keep it backwards-compatible to avoid breaking existing code.

GCS + KMS sounds good to me.

Ok great, I'll work on a patch.

Here's my (uninformed) train of thought. Disclaimer: we should talk to @AngusDavis , @bsidhom , @dennishuo who know a lot more about security.
These init actions are intended to be examples (starting points). Most of them are good enough for "Hello, world", but you will likely need to customize them to fit your needs. In general, you should copy these scripts to your own GCS bucket. This also ensures that breaking changes to these examples do not break your pipelines.
So the current way to specify a password is to copy the script into your own GCS bucket and hardcode the password.

Security

GCS ACLs (IAM?)

Pros

No changes required (for dataproc)

Cons

Users need to fork the script.

Alright, so I'll go down the thought experiment of this proposal. If we don't want to fork the script, there needs to be a way to specify the root and hive user passwords. My first thought is metadata keys.

Security

Need to secure whatever script is running gcloud dataproc clusters create, since it will include the password
Dataproc IAM (metadata shows up in gcloud dataproc clusters describe
GCE IAM (metadata shows up in all instances in gcloud compute instances describe

Pros

No extra dependencies
In line with how the other init actions work (see Jupyter's AUTH_TOKEN).

Cons

For security-conscious users, this could be more work than forking the script.

This issue suggests KMS + encrypted password in GCS.

Security

KMS

Pros

KMS has nice security features like revoking keys

Cons

Adds complexity -- you need a GCS file for each user and at least one KMS key
You have to specify the password in plaintext to Cloud SQL when creating the instance or creating users. You can't give it this KMS key + GCS file

In addition, we need to remember that any user that can ssh to a Dataproc cluster has root access, meaning they can read any file on the VM and view the arguments passed to any process. So in other words, I think any user that can ssh can see the passwords. Also, any user with permission to change user passwords in Cloud SQL can do so.
I'm not trying to discourage you from writing this patch, but I'm not yet convinced that this complexity is significantly better than just hardcoding the password in your version of the script. For example, isn't changing the password in Cloud SQL just as good as revoking a key in KMS?

@karth295 Thanks for the thorough feedback. You raise some great questions and I understand your reservations.
I agree that the init actions presented in this repository should be considered as starting points, as it's unlikely that they would cover every possible use case that one might need in a large production setup. That said, they're also an opportunity to show some recommended patterns and best practices wherever possible. So yes it's true, it's just a matter of finding the right balance between over-simplification and over-complication :)
To answer your question ("isn't changing the password in Cloud SQL just as good as revoking a key in KMS?"). These would indeed have essentially the same effect. However, my original motivation here was to remove any passwords from the code, so that you could then check that code into a source repository.
When you say that "any user that can ssh to a Dataproc cluster has root access", could you clarify exactly what that means and how this happens? Also, could that perhaps be prevented by using OS Login?
Regarding the option of passing passwords directly as metadata values, I would be wary of that as the passwords would then be logged in the command history when creating the cluster (and possibly also in HTTP logs?).
Maybe if I take a step back, I think that ideally the goals that would be great to achieve are:

Make it acceptable to check an init action's code into a source repository, i.e. free of any passwords.
Prevent any passwords from being logged anywhere (command history, HTTP logs, etc.).
Prevent non-sudo users to access to the passwords stored on the VM's disk.
Make it as easy as possible to reset passwords (or revoke keys) with minimal disruption.

I'll think a bit more about it. Any further feedback welcome. Thanks!

Ah, thanks for the detailed response. I didn't even consider checking in init actions into a repository -- that's an excellent reason for this proposal. It's great to have an outsider perspective :)
You only need to worry about security on the cluster if there are users that should not have access as root to the particular cloud sql instance(s). If that's the case, could we write a script that does not need to authenticate as root to Cloud SQL, but instead just as a user with appropriate access? (I assume if you don't need to create a user you don't need to auth as root).
That's generally the principle Dataproc follows -- the cluster has an identity, and any user that can ssh or submit jobs to the cluster can act as that identity. This is how we recommend users securely run jobs against data in GCS or BQ -- create a cluster per job, or at least a cluster per identity.
But to directly answer your question: OS Login sounds possible. In addition, I believe most Dataproc job types run drivers as root. So you would need to disable Dataproc job submission for unprivileged users: https://cloud.google.com/dataproc/docs/concepts/iam/granular-iam. But this is pretty difficult.

So I've got a working patch that should be backwards compatible: https://github.com/jphalip/dataproc-initialization-actions/compare/c402333a7837bc12c6a2617b0f2effb084287332...jphalip:cloud-sql-proxy-kms
The patch also makes it work without the MySQL root user, in which case you'd just have to create the database and hive user separately upstream.
I might be missing something, but I'm not really concerned about SSH, as you can control SSH access via SSH metadata keys or OS Login on your compute instances.
However, I'm concerned about the fact that the initialization action's log is stored in the staging bucket, and that the log displays the passwords in clear. For example, here's a snippet of what appears in the log:
...
++ gsutil cat gs://[PROJECT]/hive-password.encrypted
++ gcloud kms decrypt --ciphertext-file - --plaintext-file - --key projects/[PROJECT]/locations/global/keyRings/my-key-ring/cryptoKeys/my-key
+ readonly db_hive_password=hive123
+ db_hive_password=hive123
...

The problem is that to allow a user to run jobs via the Dataproc jobs API, you need to give them the storage.objectViewer role on the staging bucket (as explained here), which means that the user can also view the initialization action log with the password displayed in clear.
This issue would still occur with the status quo (i.e. hardcoding the password in the init action), so the KMS approach would not be a regression. But really I'm not sure that this problem can be addressed unless the logging can be changed to not display the value of variables. Do you know if that can be done?
So anyways, this is an issue that might be OK to live with. You'd just have to clearly spell out the caveat that whoever is allowed to run jobs on the cluster would also have to be trusted at least with the hive MySQL user's password.
Any thoughts?

Hiding the password is easy -- you can just do something like
set +x # Disable printing each command
... commands that contain password ...
set -x # Enable printing each command

https://stackoverflow.com/questions/36273665/what-does-set-x-do.
And that makes sense anyway because logs shouldn't contain sensitive information.
But again, since users with access to the jobs API can run arbitrary code on the cluster as root, they'll be able to discover the hive password by looking at command line arguments to cloud-sql-proxy by running ps. So yeah, that caveat should still be in the README.

@karth295 I've submitted a patch in #260. Let me know what you think!


Write unit tests

The current project doesn't have any tests to make sure the code is secure. Here are the guidelines when it comes to writing tests:

All the unit tests must be included in the root directory under the tests folder
Even though there is no defined structure for the tests folder structure, the tests folder structure may follow the same structure as the main project structure except that each folder has to be appropriately labeled.
You must use Jest as your testing framework.
Your PR should only address testing of either one or a maximum of two components. If you want to write tests for more components, send the appropriate PRs for each component.

@khairalanam Hi, I'd like to work on a few component tests. I can start with one for the Nav.tsx.

HI @tchestnut85, that's great! I will assign this to you.


Remove version: Jackett.Jackett version 0.21.687

404 - release deleted
Microsoft Reviewers: Open in CodeFlow

URL: https://github.com/Jackett/Jackett/releases/download/v0.21.687/Jackett.Installer.Windows.exe
Response Error: The remote server returned an error: (404) Not Found.
(Automated message - build 918)


Virtual list

A simple virtual list widget requiring a known width, known equal height of each item and known number of items.

I did a pretty major refactor and ended up doing away with the trait. See my comments on 4bc3f5a for a bit more explanation as to they "why" of what I did. Hopefully you agree with most of the changes, but happy to talk through my reasoning!
I think the two remaining things are horizontal scrolling and out-of-band content refreshing. That commit was getting long enough I decided to table those and get this chunk committed.


operators [R] [CI] camel-k (0.2.0 0.3.3 0.3.4 1.0.0 1.0.0-m1 1.0.0-m2 1.0.0-m3 1.0.0-m4 1.0.0-rc1 1.0.0-rc2 1.0.1 1.1.0 1.1.1 1.2.0 1.2.1 1.3.0 1.3.1 1.3.2 1.4.0 1.4.1 1.5.0 1.5.1 1.6.0 1.6.1 1.7.0 1.8.0 1.8.1 1.8.2)

Signed-off-by: phantomjinx<EMAIL_ADDRESS>Thanks submitting your Operator. Please check below list before you create your Pull Request.
New Submissions

[ ] Are you familiar with our contribution guidelines?
[ ] Have you packaged and deployed your Operator for Operator Framework?

[ ] Have you tested your Operator in all supported installation modes?
[ ] Have you considered whether you want use semantic versioning order?
[ ] Is your submission signed?
[ ] Is operator icon set?

Updates to existing Operators

[x] Did you create a ci.yaml file according to the update instructions?
[x] Is your new CSV pointing to the previous version with the replaces property if you chose replaces-mode via the updateGraph property in ci.yaml?
[x] Is your new CSV referenced in the appropriate channel defined in the package.yaml or annotations.yaml ?
[x] Have you tested an update to your Operator when deployed via OLM?
[x] Is your submission signed?

Your submission should not

[ ] Modify more than one operator
[ ] Modify an Operator you don't own
[ ] Rename an operator - please remove and add with a different name instead
[ ] Modify any files outside the above mentioned folders
[ ] Contain more than one commit. Please squash your commits.

Operator Description must contain (in order)

[ ] Description about the managed Application and where to find more information
[ ] Features and capabilities of your Operator and how to use it
[ ] Any manual steps about potential pre-requisites for using your Operator

Operator Metadata should contain

[ ] Human readable name and 1-liner description about your Operator
[ ] Valid category name1
[ ] One of the pre-defined capability levels2
[ ] Links to the maintainer, source code and documentation
[ ] Example templates for all Custom Resource Definitions intended to be used
[ ] A quadratic logo

Remember that you can preview your CSV here.
--
1 If you feel your Operator does not fit any of the pre-defined categories, file an issue against this repo and explain your need
2 For more information see here

/unhold

/merge possible

/merge possible

@phantomjinx, please chack labels in annotations against https://github.com/redhat-openshift-ecosystem/community-operators-prod/runs/5392272910?check_suite_focus=true#step:5:19158

/test all

/merge possible


Getting errors while running in simple hurl file with :HurlVerbose

Details
Have tried with multiple files, each resulting in the following error:
Error executing Lua callback: ...lgrim/.local/share/nvim/lazy/hurl.nvim/lua/hurl/main.lua:419: attempt to concatenate field 'start_line' (a nil value)
stack traceback:
	...lgrim/.local/share/nvim/lazy/hurl.nvim/lua/hurl/main.lua:419: in function <...lgrim/.local/share/nvim/lazy/hurl.nvim/lua/hurl/main.lua:390>

I installed with the basic config using LazyVim, specificly this lua file:
return {
  "jellydn/hurl.nvim",
  dependencies = { "MunifTanjim/nui.nvim" },
  ft = "hurl",
  opts = {
    -- Show debugging info
    debug = false,
    -- Show notification on run
    show_notification = false,
    -- Show response in popup or split
    mode = "split",
    -- Default formatter
    formatters = {
      json = { "jq" }, -- Make sure you have install jq in your system, e.g: brew install jq
      html = {
        "prettier", -- Make sure you have install prettier in your system, e.g: npm install -g prettier
        "--parser",
        "html",
      },
    },
  },
  keys = {
    -- Run API request
    { "<leader>A", "<cmd>HurlRunner<CR>", desc = "Run All requests" },
    { "<leader>a", "<cmd>HurlRunnerAt<CR>", desc = "Run Api request" },
    { "<leader>te", "<cmd>HurlRunnerToEntry<CR>", desc = "Run Api request to entry" },
    { "<leader>tm", "<cmd>HurlToggleMode<CR>", desc = "Hurl Toggle Mode" },
    { "<leader>tv", "<cmd>HurlVerbose<CR>", desc = "Run Api in verbose mode" },
    -- Run Hurl request in visual mode
    { "<leader>h", ":HurlRunner<CR>", desc = "Hurl Runner", mode = "v" },
  },
}```

<details open>
<summary>Checklist</summary>

- [X] Modify `lua/hurl/main.lua` ✓ https://github.com/jellydn/hurl.nvim/commit/fed85aa9c948dbff083e30212f8b568f03daa58f [Edit](https://github.com/jellydn/hurl.nvim/edit/sweep/getting_errors_while_running_in_simple_h/lua/hurl/main.lua#L419-L420)
- [ ] Running GitHub Actions for `lua/hurl/main.lua` ⋯  [Edit](https://github.com/jellydn/hurl.nvim/edit/sweep/getting_errors_while_running_in_simple_h/lua/hurl/main.lua#L419-L420)
</details>

Hi @TimPelgrim Thank you for your report. Could you check again with latest version? It should work now.


Reduce loading timeout noise

I'm not 100% sure if this will work, but we'll soon find out.

nope didn't work, worth a try.


Ctrl + A is interpreted as Ctrl + Q

Issue Type: Bug
I have an azerty keyboard.
When i press a on Visual Studio,  a is written on the editor. OK, this was expected
However, when i press Ctrl + a, Visual Studio exits.
Visual Studio seems to see my keyboard as qwerty when Ctrl is pressed.
VS Code version: Code 1.33.1 (51b0b28134d51361cf996d2f0a1c698247aeabd8, 2019-04-11T08:20:22.771Z)
OS version: Linux x64 3.16.0-4-amd64

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz (8 x 3556)

GPU Status
2d_canvas: enabledchecker_imaging: disabled_offflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onnative_gpu_memory_buffers: disabled_softwarerasterization: disabled_softwaresurface_synchronization: enabled_onvideo_decode: unavailable_offwebgl: enabledwebgl2: enabled

Load (avg)
17, 16, 16

Memory (System)
7.67GB (0.90GB free)

Process Argv
--unity-launch

Screen Reader
no

VM
67%

Extensions (5)

Extension
Author (truncated)
Version

c-cpp-flylint
jbe
1.2.4

cpptools
ms-
0.23.0-insiders2

cmake
twx
0.0.17

cmake-tools
vec
1.1.3

clang-format
xav
1.9.0

@CKortylak Did you try configuring keyboard.dispatch: "keyCode"

@alexandrudima: Your advice was right:
Ctrl+,->Application->Keyboard->Dispatch->KeyCode
fixes the issue.
Thank you !


Correct graphic in deps/rems article.

Whoops!
There were 1 critical errors that broke the build and prevented it from being automatically deployed.
ERRORS
src/content/en/updates/2020/12/chrome-88-deps-rems.md#L9 - WF Tag 'wf_featured_image' found, but couldn't find image - /web/updates/images/2020/12/dep-rem_88.jpg

Whoops!
There were 1 critical errors that broke the build and prevented it from being automatically deployed.
ERRORS
src/content/en/updates/2020/12/chrome-88-deps-rems.md#L9 - WF Tag 'wf_featured_image' found, but couldn't find image - /web/updates/images/2020/12/dep-rem_88.jpg

:+1:

:+1:


Current_date snowflake function cause to lag in data

Hi
I have a question about lag of data in our Jira dashboards.
Until the hour 10 AM (Israel time), we can't see the data of current date, although that in Fivetran there is data of current date.
I investigated this issue and find that the int table :int_jira__issue_calendar_spine use with filter: "where date_day <= current_date".
The function current_date bring me the real date just starting from 10 AM (Israel time), and this is the reason of lag in our data.
First I will happy to understand what is the meaning of current_date, which time is it?(I'm sorry, I know that it is snowflake function, but I don't understand according by this function brings the date) .
And also I want to ask if there is an option to change it for UTC time (for example sysdate() function) or to give an option to users, to choose the desired time.
Thank a lot for your feedback, and have a nice week.
Rivki

Hi @RivkiHofman thanks so much for raising this issue.
That is very interesting that the current_date() Snowflake function only brings you time starting at 10am. I believe there should be a way to convert the current_date() to UTC to then allow for this error to no longer persist (I would imagine).
Let me look a bit more into this today and I will respond here with a working branch you should be able to test out.

Hi @RivkiHofman 👋
I have been able to do some digging into this question. While I do not have a good explanation for how the current_date() function works within Snowflake, I did see that there is a useful macro within dbt_utils that allows for timestamp conversion to UTC that will work for all warehouses!
I have just pushed a hotfix to the below working branch and opened the above PR to integrate this into the next release of the Jira package. Would you be able to test the below package dep and let me know if this 10am issue is still persisting for you?
packages:
    - git: https://github.com/fivetran/dbt_jira.git
      revision: hotfix/current-date-utc
      warn-unpinned: false

Let me know how it goes, thanks!

Hi @fivetran-joemarkiewicz
Thank a lot for your quick support and response.
We checked and it works great.
I will happy to be updated when it will go live.
Thanks and have a nice day.

Thanks so much for the quick reply! I will plan to merge this into our next release/v0.6.0 branch along with a few other changes addressed by other users of the package.

Hi @fivetran-joemarkiewicz
I jוst want to ask about the temporary package:
packages:
- git: https://github.com/fivetran/dbt_jira.git
revision: hotfix/current-date-utc
warn-unpinned: false
Can I use it in our production environment? Is it like as current version(0.5.1) + the change of current_date?
I'm afraid to declare the new version: v0.6.0 without comprehensive tests on the new features.
Is there a documentation for the new version?
Thanks and have a nice weekend.

Hey @RivkiHofman I will actually be cutting v0.6.0 later today. If you would like to test out the other features before I go live that would be great! Otherwise, I will post back here later today once the version is live.
packages:
    - git: https://github.com/fivetran/dbt_jira.git
      revision: release/v0.6.0
      warn-unpinned: false

Hi @fivetran-joemarkiewicz
I understand that the v0.6.0 will go live today, Good luck 👍 .
My question is if I can stay in the meantime with the temporary branch(revision: hotfix/current-date-utc), or maybe this branch will be deleted or changed?
Thanks.

@RivkiHofman you can keep the temporary branch in the meantime! I will be sure not to delete it 😄

Hi @RivkiHofman I wanted to let you know that the latest version of the dbt_jira package v0.6.0 is now live on the dbt hub!
Be sure to read the Release Notes to be aware of the changes that are included in the latest release along with the change from this issue, and to run a dbt run --full-refresh after upgrading in order to capture changes in the incremental models within this package.
Thanks again for helping raise this issue and test out the working branch before release! We are extremely grateful for your contributions to this package and for helping make this package better for all it's users. 🎉 ❤  Please feel free to open a new issue if you experience any other bugs or a feature request if you think of a new enhancement to that the package would benefit from integrating.


Only the first five gpio-keys of wio-terminal seem to work

Describe the bug
Using two different wio_terminal I could reproduce the weird behaviour, that only the first 5 entries in gpio-keys or buttons seem to work.
They are ordererd as:
	/* Buttons */
	gpio_keys {
		compatible = "gpio-keys";
		user_button_0: button_0 {...};
		user_button_1: button_1 {...};
		user_button_2: button_2 {...};
		joy_sel: joystick_selection {...};
		joy_down: joystick_down {...};
		joy_up: joystick_up {...};
		joy_left: joystick_left {...};
		joy_right: joystick_right {...};
	};

and just the buttons _0, _1, _2 and _selection and _down are working. This can be observed in the lvgl sample or with the following code:
static void input_cb(struct input_event *evt)
{

  char buf[1000];
  snprintf(buf, sizeof(buf), "type: %d, code: %d, value: %d", evt->type, evt->code, evt->value);
  ... // print buf to display
}
INPUT_CALLBACK_DEFINE(NULL, input_cb);

Using this code only the first five entries show any sign of reaction.
The weird behaviour is, that reordering the entries to
	/* Buttons */
	gpio_keys {
		compatible = "gpio-keys";
		user_button_0: button_0 {...};
		joy_sel: joystick_selection {...};
		joy_left: joystick_left {...};
		joy_right: joystick_right {...};
		user_button_1: button_1 {...};
		user_button_2: button_2 {...};
		joy_down: joystick_down {...};
		joy_up: joystick_up {...};
	};

makes _0, _selection, _left, _right and _1 available and allows to actually use the button matrix of the lvgl example.
To Reproduce
$ west build -b wio_terminal samples/subsys/display/lvgl
$ west flash

observe that button grid is displayed but only the select (press on joystick) has a visual reaction.
Reorder the entries as above, rebuild and flash and observe that right and left joystick movement is now working as expected.
Expected behavior
I would expect that all gpio_keys are working
Impact
Some buttons of the wio_terminal are not usable for no clear reason.
Environment (please complete the following information):

OS: Linux
Toolchain: Zephyr SDK
Commit SHA: bf2bdcf6f79cf9711b1fdee3f2d91a5e61603f29

Ah, to get the lvgl sample working one needs:
$ cat samples/subsys/display/lvgl/boards/wio_terminal.conf
CONFIG_GPIO=y
CONFIG_INPUT=y

and
$ cat samples/subsys/display/lvgl/boards/wio_terminal.overlay
/*
 * SPDX-License-Identifier: Apache-2.0
 */
#include <zephyr/dt-bindings/input/input-event-codes.h>
#include <zephyr/dt-bindings/lvgl/lvgl.h>
 / {

        lvgl_button_input {
                compatible = "zephyr,lvgl-button-input";
                //input = <&gpio_keys>;
                input-codes = <INPUT_KEY_B>;
                coordinates = <160 120>;
        };

        lvgl_keypad_input {
                compatible = "zephyr,lvgl-keypad-input";
                //input = <&gpio_keys>;
                input-codes = <INPUT_KEY_LEFT INPUT_KEY_RIGHT INPUT_KEY_ENTER>;
                lvgl-codes = <LV_KEY_LEFT LV_KEY_RIGHT LV_KEY_ENTER>;
        };
};

Ah, to get the lvgl sample working one needs

So this issue is resolved?

Oh, sorry for the unclear communication. No, this does not solve the issue but just adds the necessary overlay to map the keypad.
The issue is still existent and depending on order just the first 5 entries of gpio_keys seem to have any effect.

@joelguittet

Hello,
Not seen this during the initial integration of the wio terminal, but it was tested only with one gpio one by so to ensure the assignment of the pin is correct.
Here the issue you describe is different, not sure how it is related to the wio terminal itself, or the gpio keys?
I will have a look when I will be available after the new year celebration, if not solved in the meantime :-)
Joel

I also checked that all pin assignments are correct.

@maxhbr can you maybe check if the input events are raised in the first place? You can enable some logging for them with the CONFIG_INPUT_EVENT_DUMP Kconfig option.

@maxhbr
I'm trying to reproduce your setup, I have added the two files mentioned above samples/subsys/display/lvgl/boards/wio_terminal.conf and  samples/subsys/display/lvgl/boards/wio_terminal.overlay. Note CONFIG_GPIO is already defined in the defconfig of the Wio terminal board so it's not required.
Now I get the following error while building samples/subsys/display/lvgl :
/home/ubuntu/zephyrproject/zephyr/samples/subsys/display/lvgl/boards/wio_terminal.overlay:5:10: fatal error: zephyr/dt-bindings/lvgl/lvgl.h: No such file or directory
    5 | #include <zephyr/dt-bindings/lvgl/lvgl.h>

Seems really strange to me and I didn't found any samples where you may have copy/paste this include.
Which Zephyr version are you using ? I'm on 3.5.0.

I am on https://github.com/zephyrproject-rtos/zephyr/commit/bf2bdcf6f79cf9711b1fdee3f2d91a5e61603f29.

Ok, seems more recent than release 3.5.0, I need to update on my side. I will let you known. No other addition/modification you have made except the two files above ?

let me recreate this example as a branch based on 3.5.0

Hey, I created https://github.com/maxhbr/zephyr/tree/maxhbr/exampleFor%2367049 and it

is based on latest main (there were lvgl related breaking changes since 3.5)
add support for the wio_terminal to the lvgl example (as described above)
adds another label to the UI which shows the last recorded event (as described above, with INPUT_CALLBACK_DEFINE)

@maxhbr can you give us also the full part of the dts with gpio_keys just to make sure that they all have the correct zephyr,code property?

FWIW the issue is not with LVGL IMO, and can easily be reproduced with the input_dump sample:
west build -p auto -b wio_terminal samples/subsys/input/input_dump

after a west build -b wio_terminal samples/subsys/display/lvgl I see the following in build/zephyr/zephyr.dts:
 557   │     gpio_keys {
 558   │         compatible = "gpio-keys";
 559   │         user_button_0: button_0 {
 560   │             label = "User Button 0";
 561   │             gpios = < &portc 0x1a 0x1 >;
 562   │             zephyr,code = < 0xb >;
 563   │         };
 564   │         user_button_1: button_1 {
 565   │             label = "User Button 1";
 566   │             gpios = < &portc 0x1b 0x1 >;
 567   │             zephyr,code = < 0x2 >;
 568   │         };
 569   │         user_button_2: button_2 {
 570   │             label = "User Button 2";
 571   │             gpios = < &portc 0x1c 0x1 >;
 572   │             zephyr,code = < 0x3 >;
 573   │         };
 574   │         joy_sel: joystick_selection {
 575   │             label = "joystick selection";
 576   │             gpios = < &portd 0xa 0x1 >;
 577   │             zephyr,code = < 0x1c >;
 578   │         };
 579   │         joy_down: joystick_down {
 580   │             label = "joystick down";
 581   │             gpios = < &portd 0x8 0x1 >;
 582   │             zephyr,code = < 0x6c >;
 583   │         };
 584   │         joy_up: joystick_up {
 585   │             label = "joystick up";
 586   │             gpios = < &portd 0x14 0x1 >;
 587   │             zephyr,code = < 0x67 >;
 588   │         };
 589   │         joy_left: joystick_left {
 590   │             label = "joystick left";
 591   │             gpios = < &portd 0xc 0x1 >;
 592   │             zephyr,code = < 0x69 >;
 593   │         };
 594   │         joy_right: joystick_right {
 595   │             label = "joystick right";
 596   │             gpios = < &portd 0x9 0x1 >;
 597   │             zephyr,code = < 0x6a >;
 598   │         };
 599   │     };

I can not reproduce the same behaviour on other boards. I tested it on a stm32h747i_disco which also has a similar joystick and with the input_dump example I saw all 6 gpio_keys working.

I needed to apply the following patch:
diff --git a/boards/arm/stm32h747i_disco/stm32h747i_disco_m7.dts b/boards/arm/stm32h747i_disco/stm32h747i_
disco_m7.dts
index 514047c8a5..94de4db849 100644
--- a/boards/arm/stm32h747i_disco/stm32h747i_disco_m7.dts
+++ b/boards/arm/stm32h747i_disco/stm32h747i_disco_m7.dts
@@ -44,6 +44,21 @@
                wake_up: button {
                        status = "okay";
                };
+               joy_center: joystick_center {
+                       status = "okay";
+               };
+               joy_down: joystick_down {
+                       status = "okay";
+               };
+               joy_up: joystick_up {
+                       status = "okay";
+               };
+               joy_left: joystick_left {
+                       status = "okay";
+               };
+               joy_right: joystick_right {
+                       status = "okay";
+               };
        };

looks like gpio-keys driver is throwing an error:
D: gpio_keys_interrupt_configure [0x0xb7c4, 26]
D: gpio_keys_interrupt_configure [0x0xb7c4, 27]
D: gpio_keys_interrupt_configure [0x0xb7c4, 28]
D: gpio_keys_interrupt_configure [0x0xb7b0, 10]
D: gpio_keys_interrupt_configure [0x0xb7b0, 8]
D: gpio_keys_interrupt_configure [0x0xb7b0, 20]
E: interrupt configuration failed: -16
E: Pin 5 interrupt configuration failed: -16

Agree with @kartben I see the same issue.
The SAMD51P19A has only 16 external interrupt. Maybe they are all used ? Trying to verify....

Ok issue found.
The EIC has 16 external interrupts available, and they are affected to GPIOs. However, there are much more GPIO of course, so a single EIC input is on multiple GPIOs at the same time.
Of course to get it working, it's not possible to activate multiple source to the same EIC input.
Currently the pins used for buttons on the Wio Terminal have the following configuration:

Button 0, PC26 => EXTINT[10] => KO! This is duplicated with Joystick UP
Button 1, PC27 => EXTINT[11] => OK
Button 2, PC28 => EXTINT[12] => OK
Joystick SEL, PD10 => EXTINT[5] => OK
Joystick Down, PD8 => EXTINT[3] => OK
Joystick Up, PD20 => EXTINT[10] => KO! This is duplicated with Button 0
Joystick Left, PD12 => EXTINT[7] => OK
Joystick Right, PD9 => EXTINT[4] => OK

Disable either Button 0 (button 1 on Seeed schematic) or Joystick UP and this is now working as expected.
This is an hardware limitation of the MCU.

PS: you will likely disable Button 0, depending of your application maybe it's possible to use polling on this GPIO instead of interrupt, if you really need this.

Given the current gpio-key configuration can't work as-is, I wonder if it should be reworked in two different nodes (one for button 0, 1, 2, "okay"'d by default, and another disabled one for the 5-way switch buttons?).

[...] depending of your application maybe it's possible to use polling on this GPIO instead of interrupt, if you really need this

@fabiobaltieri would it make sense for gpio-keys driver to have a polling mode similar to (shared with?) the one available for kbd-matrix?

@fabiobaltieri would it make sense for gpio-keys driver to have a polling mode similar to (shared with?) the one available for kbd-matrix?

Yeah it does, stm32 have a similar limitations, plus some mcu don't support triggering on both edges, so it's useful to have a fallback mode. Linux has one as well (https://elixir.bootlin.com/linux/latest/source/drivers/input/keyboard/gpio_keys_polled.c) we could either have a separate driver or a "polling" mode.

Had a look at the code, should be fairly easy to add polling mode, same schema as with the keyboard matrix one. I'll give it a shot, stay tuned.

reopening as the linked PR makes a fix for this issue possible, but does not fix it per se.
@joelguittet over to you for an update of the devicetree? :)

Yes will do !


chore: Add OpenTelemetry incubating semantic conventions to shared dependencies

Follow up to https://github.com/googleapis/sdk-platform-java/pull/3020 but adding the opentelemetry-semconv-incubating dependency which lives in the same repository as the previously added opentelemetry-semconv dependency. Approval for that dependency is at b/351882514. This allows us to use the MessagingIncubatingAttributes class which provides some GCP specific semantic conventions for OpenTelemetry.

Could you please create a new buganizer issue for approval?


Unsupported URL (https://motherless.com/D63BB1F)

Checklist

[ ] I'm reporting a new site support request
[X] I've verified that I'm running youtube-dl version 2021.12.17
[X] I've checked that all provided URLs are alive and playable in a browser
[X] I've checked that none of provided URLs violate any copyrights
[X] I've searched the bugtracker for similar site support requests including closed ones

Example URLs

Single video: https://motherless.com/D63BB1F

Description

youtube-dl -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best' 'https://motherless.com/D63BB1F'

[Motherless] D63BB1F: Downloading webpage
ERROR: Unable to extract uploader_id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-f', 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best', 'https://motherless.com/0001DE4', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.10.5 (CPython) - macOS-11.6.8-x86_64-i386-64bit
[debug] exe versions: ffmpeg 5.1, ffprobe 5.1, rtmpdump 2.4
[debug] Proxy map: {}
[Motherless] 0001DE4: Downloading webpage
ERROR: Unable to extract uploader_id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor/motherless.py", line 129, in _real_extract
    uploader_id = self._html_search_regex(
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 1021, in _html_search_regex
    res = self._search_regex(pattern, string, name, default, fatal, flags, group)
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 1012, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract uploader_id; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

There are two factors to this problem:

the site has a new version and the field that was being targeted for uploader_id is no longer there;
the extraction of uploader_id crashes the extractor because of a coding error.

Use --add-header 'Cookie: _force_desktop_v1=true' to get the old site until someone updates the extractor.

The uploader_id  can be found in the new site with the same basic pattern, but changing thumb-member-username in the pattern to (?:thumb-member-username|media-meta-member).
Anyone updating the extractor should run the tests to ensure that other metadata is still being correctly extracted.

Fixed in https://github.com/ytdl-org/youtube-dl/commit/82e4eca711a128138ed0b84ddb4321e403d56340

This is, in all probability, a dupe of #29626 (which should also be closed).


Create a AWS integration

I guess it's the equivalent of what we have done on DO: https://aws.amazon.com/marketplace

I assigned you @eskombro because I think you are already working on it, am I right?

Done by @eskombro! ❤


TransferDeadLetterMessageCount	Issue

Hi,
I have configured a Forward to on a subscription(With True Filter) to an another topic and made the topic(to which messages are forwarded) inactive. When I sent a message on to the topic, since the Forward Topic is not active it has to increase the Transfer Dead Letter Message count ( as per Microsoft Doc https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-counters) but instead it is increasing the Dead Letter Message count. Is there any configurations that are to be done to get my expected behavior or there is any misunderstanding from my end.
Pricing Tier : Standard

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: f3fb5e3e-205d-8e04-a5c9-63cc2a29d39d
Version Independent ID: d5720861-9ba6-bdc1-0dc2-c640c3fc9298
Content: Azure Service Bus - message count - Azure Service Bus
Content Source: articles/service-bus-messaging/message-counters.md
Service: service-bus-messaging
GitHub Login: @spelluru
Microsoft Alias: spelluru

#reassign:EldertGrootenboer

Thank you for your feedback. We have opened an investigation task for this in our backlog, and will update this issue when we have more information.

#please-close
We will update the issue when we have more information. Thans.


Fix the training and validating S3 data path

Description of changes:
You mistook the path of the training and validating.
You forgot to write the bucket-name and prefix.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Hi, the path to the local file is used during local mode training. You should not experience issues when executing training on the local notebook instance with local files. Could you provide additional details on the issues you are experiencing? Thanks.


Added backup endpoint to s3 and restore on startIndex for Primary and Standalone Mode

Added backup endpoint to s3 and restore on startIndex for Primary and Standalone Mode.
Fixed tests.
Removed older workaround for remote storage locally

Still need to figure out snapshot/restore of taxoDir for Primary mode.
Probably need to add some more replication tests now that we have a more concrete backup/restore implementattion

closes #22


Path too long to file Microsoft.Dynamics.Commerce.Essentials.Telemetry.cs

When cloning this repository to a Windows machine, you get an error that the filepath to the file Commerce/Essentials/Telemetry/Metadata/VSProjects/Commerce Essentials - Telemetry/Microsoft.Dynamics.Commerce.Essentials.Telemetry/Microsoft.Dynamics.Commerce.Essentials.Telemetry.cs is too long.
In VS it is also marked as deleted when loading any solution.

Remote: Total 4038 (delta 267), reused 205 (delta 184), pack-reused 3623
Error: unable to create file Commerce/Essentials/Telemetry/Metadata/VSProjects/Commerce Essentials - Telemetry/Microsoft.Dynamics.Commerce.Essentials.Telemetry/Microsoft.Dynamics.Commerce.Essentials.Telemetry.cs: Filename too long

facing same issue


cannot find package /extra25519

this is the issue : https://github.com/agl/ed25519/issues/27
mb the solution : https://godoc.org/golang.org/x/crypto/ed25519 this dont work.
you can swap agl with decred : extra "github.com/decred/ed25519/extra25519"

after the fix

now i have an other issue:

Hey @chenak-a , thanks a lot for raising this issue.
Taking a more detailed look, we use agl/ed25519 for a public/private point conversion between elliptic curves Curve25519/Ed25519, mostly for using the same private key for signature and encryption using NaCl.
This issue is tracking including this conversion, which would be the nicest way to solve this. In the meantime, I seems we can consider the age implementation of this conversions. age is done by Fillipo Valsorda which is most probably the best person to rely on for Go crypto.
Apart from doing the change, we should add some tests between the old and new implementations to really check things.

Good call on the age implementation, I would agree with your assessment here @jsign. Not that it matters too much, but our JS libs for crypto also mongomery/edwards key conversion. This issue https://github.com/golang/go/issues/20504 also appears to suggest age (among other solutions).