custom ca not working on istio 1.7

(NOTE: This is used to report product bugs:
To report a security vulnerability, please visit https://istio.io/about/security-vulnerabilities/
To ask questions about how to use Istio, please visit https://discuss.istio.io
)
Bug description
After generate cacerts secrets, install istio with IstioOperator.
istiod service running well,but all istio proxy sidecar or ingressgateway cannot work well.
this is logs:
2020-09-08T11:59:14.178620Z	info	sds	SDS gRPC server for workload UDS starts, listening on "./etc/istio/proxy/SDS"

2020-09-08T11:59:14.178680Z	info	Starting proxy agent
2020-09-08T11:59:14.178720Z	info	sds	Start SDS grpc server
2020-09-08T11:59:14.178712Z	info	Opening status port 15020

2020-09-08T11:59:14.178781Z	info	Received new config, creating new Envoy epoch 0
2020-09-08T11:59:14.178823Z	info	Epoch 0 starting
2020-09-08T11:59:14.233702Z	info	Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster grpc-verify-grpc.grpc-verify --service-node sidecar~<IP_ADDRESS>~grpc-verify-grpc-6f947867b7-cvk9q.grpc-verify~grpc-verify.svc.cluster.local --local-address-ip-version v4 --log-format-prefix-with-location 0 --log-format %Y-%m-%dT%T.%fZ	%l	envoy %n	%v -l warning --component-log-level misc:error --concurrency 2]
2020-09-08T11:59:14.263787Z	warning	envoy runtime	Unable to use runtime singleton for feature envoy.reloadable_features.activate_fds_next_event_loop
2020-09-08T11:59:14.289783Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-09-08T11:59:14.289801Z	warning	envoy config	Unable to establish new stream
2020-09-08T11:59:14.297212Z	info	sds	resource:default new connection
2020-09-08T11:59:14.297279Z	info	sds	Skipping waiting for gateway secret
2020-09-08T11:59:14.551726Z	info	cache	Root cert has changed, start rotating root cert for SDS clients
2020-09-08T11:59:14.551778Z	info	cache	GenerateSecret default
2020-09-08T11:59:14.552226Z	info	sds	resource:default pushed key/cert pair to proxy
2020-09-08T11:59:14.554289Z	warning	envoy main	there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
2020-09-08T11:59:15.123803Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2020-09-08T11:59:15.909285Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2020-09-08T11:59:18.161375Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2020-09-08T11:59:25.069427Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2020-09-08T11:59:35.341020Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2020-09-08T11:59:45.448392Z	warn	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-09-08T11:59:47.447950Z	warn	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-09-08T11:59:49.448099Z	warn	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected

Affected product area (please put an X in all that apply)
[ ] Docs
[ ] Installation
[* ] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Affected features (please put an X in all that apply)
[ ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane
Expected behavior
Steps to reproduce the bug
Version (include the output of istioctl version --remote and kubectl version --short and helm version if you used Helm)
How was Istio installed?
Environment where bug was observed (cloud vendor, OS, etc)
Additionally, please consider attaching a cluster state archive by attaching
the dump file to this issue.

Finnally I found that truth:
If you use a  a root cert generated by openssl , you should  set "extendedKeyUsage = serverAuth,clientAuth" !
Because connections between envoy <=> istio use mTLS !!!

At the start of the log there should be some output about the root cert that will be used. Can you compare that to the cacerts you provisioned and to ./var/run/secrets/istio/root-cert.pem on the proxy?


Changing access to $SiteUrl array

I ran into an issue where $SiteUrl[0] wouldn't return the first element. I changed it to use Select -First 1 which is more safe than simply acessing [0]. Subsequently Connect-PnPOnline would fail which would lead to more issues. I haven't created an issue for this yet.
Category

[x] Bug Fix
[ ] New Feature

Thank you Andreas!


Update a whole bunch of links to datasette.io instead of datasette.readthedocs.io

https://ripgrep.datasette.io/-/ripgrep?pattern=(datasette|sqlite-utils)\.readthedocs\.io

https://github.com/Homebrew/homebrew-core/pull/67983


Analyzers should infer pods from deployments

Many analysis tasks are easier to model in terms of pods instead of deployments, but particularly in the files-only case, there may not be any files defining pods. Generally users will have deployment resources that will result in pods, and it would be useful for our analysis to be able to smoothly handle that case.
Right now, an analyzer (e.g. the gateway ports analyzer) that pays attention to pods can only practically run against a live cluster, unless users are hand-defining their pods. If a user only has yaml for Deployments, we won't be able to detect gateway port issues related to them until after they've pushed the changes to their cluster.
One approach would be to add a transformer (like we do for Ingress) that infers pod resources from deployments.

https://github.com/istio/istio/pull/19583 is another example of an analyzer that would benefit from this.

Some things that make this tricky to do right:

Some analyzers really only care about live pods (e.g. injection.VersionAnalyzer) and it would be awkward if those analyzers started to trigger false positives on inferred pods.
For a given pod, it's a bit awkward to tell what deployment created it. The most robust looking way is to look at the ownerReferences in the pod metadata, but there's a layer of indirection that goes through replicaSets to get there. If we do go this route, we'd have to add replicaSets to our metadata.yaml just for this use case.


Guidance requested on using doc.retokenize

I'm having a great time with this library, but running into the following issue:
As mentioned in https://github.com/explosion/holmes-extractor/issues/2#issuecomment-1195536077, I am manually setting entities using doc.set_ents. I found that some obvious / verbatim search phrases (e.g. something like ENTITYCUSTOM_ENT1 something ENTITYCUSTOM_ENT2) would not match properly until I made sure to retokenize the doc before registering it with the manager:
docs = []
spans = filter_spans(spans=spans)
doc.set_ents(spans, default='unmodified')
with doc.retokenize() as retokenizer:
    for span in spans:
        retokenizer.merge(span, {'POS': 'NOUN'})
docs.append(doc)
# ...
for doc in docs:
    manager.register_serialized_document(doc.to_bytes(), label=label)

However, this also appears to have created a mis-alignment in doc length and I am not sure how. Some search phrases end up throwing this error when a match is found:
IndexError: [E026] Error accessing token at position 145: out of bounds in Doc of length 88.

I was able to debug that particular doc, and found that my retokenized doc is 88 (word) tokens. If I ran the doc.text through manager.nlp, however, the resulting doc is 145 word tokens. This suggests to me that somewhere the merged span information is lost, perhaps after some internal call to manager.nlp?

Yes, the problem here is that Holmes (and Coreferee on which it relies) are both applied to the document when it is first parsed and before retokenization. This means that the Coreferee and Holmes information stored on the document will no longer be valid after you retokenize it.
The problem should be solved if you run the document back through both extensions after retokenization:
coreferee_ext = manager.nlp.get_pipe("coreferee")
holmes_ext = manager.nlp.get_pipe("holmes")
coreferee_ext(doc)
holmes_ext(doc)


Python endpoint name detector

spp.jetbrains.marker.jvm.psi.EndpointDetector is able to extract the endpoint name from Spring MVC annotations. For example, the following:
import org.springframework.web.bind.annotation.RequestMapping;
public class TestController {
    @RequestMapping(value = "/doGet", method = RequestMethod.GET)
    public void doGet() {}
}

EndpointDetector will extract GET:/doGet as the endpoint name.

This task is to create the equivalent for Python Flask annotations such that:
@app.route('/todos', methods=['GET'])
def get_todos():
    if len(todos) == 0:
        return jsonify([])
    return jsonify(list(todos.values()))

will extract GET:/todos as the endpoint name.

You can use https://github.com/sourceplusplus/tutorial-python/blob/master/app.py as a testing file.
Just determine the endpoint name for now. EndpointDetector also finds the endpoint id but that will be done after this task.

Make sure to use the latest code in dev branch. Specifically this commit: https://github.com/sourceplusplus/interface-jetbrains/pull/717/commits/92e0f2f7bae0822159909650c41c96a662bc386d
It adds better Python artifact naming.

I tried importing the python application to Idea editor but I am not able to place marker. I tried to configure runIde to open pycharm but failing with below error. Please suggest how I can test the plugin for Python.
Cannot find builtin plugin 'java' for IDE: .gradle\caches\modules-2\files-2.1\com.jetbrains.intellij.pycharm\pycharmPC\2021.3.3\ed07827b1346cfa285580cda980365a9cc00e0de\pycharmPC-2021.3.3

Forgot to mention you need some edits to make Python work. PyCharm doesn't look like it's working at the moment.
Revert your changes to gradle.properties and edit SourceMarkerPlugin from:
if (PYCHARM_PRODUCT_CODES.contains(productCode)) {

to:
if (true) {


Docker build failing with current instructions

I tried using the Dockerfile.build instructions docker build -f build/Dockerfile.build -t bwbuild:v1 . but it's failing when compiling the Bottled Water source code (the last one).
It seems this may have to do with the current gcc version used in Debian and the PostgreSQL 9.4 dev source code, respectively the pqexpbuffer.h header file.
Are you able to reproduce and confirm this? Should I use a specific compiler version?
GCC and G++ packages installed with apt-get install:
cpp-4.9 amd64 4.9.2-10
cpp amd64 4:4.9.2-2
libgcc-4.9-dev amd64 4.9.2-10
gcc-4.9 amd64 4.9.2-10
gcc amd64 4:4.9.2-2
libstdc++-4.9-dev amd64 4.9.2-10
g++-4.9 amd64 4.9.2-10
g++ amd64 4:4.9.2-2

Error below:
Step 6 : RUN cd /root/bottledwater &&     make clean && make && make install && cd / &&     tar czf bottledwater-ext.tar.gz usr/lib/postgresql/9.4/lib/bottledwater.so usr/share/postgresql/9.4/extension/bottledwater* &&     cp /root/bottledwater/kafka/bottledwater /root/bottledwater/client/bwtest /usr/local/bin &&     tar czf bottledwater-bin.tar.gz usr/local/bin/bottledwater usr/local/bin/bwtest
 ---> Running in 5525be9d5f5a
make -C ext clean
make[1]: Entering directory '/root/bottledwater/ext'
rm -f bottledwater.so   libbottledwater.a  libbottledwater.pc
rm -f io_util.o logdecoder.o oid2avro.o schema_cache.o protocol.o protocol_server.o snapshot.o
make[1]: Leaving directory '/root/bottledwater/ext'
make -C client clean
make[1]: Entering directory '/root/bottledwater/client'
rm -f replication.o protocol.o protocol_client.o connect.o bwtest.o bwtest libbottledwater.a
make[1]: Leaving directory '/root/bottledwater/client'
make -C kafka clean
make[1]: Entering directory '/root/bottledwater/kafka'
rm -f bottledwater.o registry.o bottledwater
make[1]: Leaving directory '/root/bottledwater/kafka'
make -C ext all
make[1]: Entering directory '/root/bottledwater/ext'
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -fpic -I/usr/local/include  -std=c99 -I. -I./ -I/usr/include/postgresql/9.4/server -I/usr/include/postgresql/internal -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/tcl8.6  -c -o io_util.o io_util.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -fpic -I/usr/local/include  -std=c99 -I. -I./ -I/usr/include/postgresql/9.4/server -I/usr/include/postgresql/internal -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/tcl8.6  -c -o logdecoder.o logdecoder.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -fpic -I/usr/local/include  -std=c99 -I. -I./ -I/usr/include/postgresql/9.4/server -I/usr/include/postgresql/internal -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/tcl8.6  -c -o oid2avro.o oid2avro.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -fpic -I/usr/local/include  -std=c99 -I. -I./ -I/usr/include/postgresql/9.4/server -I/usr/include/postgresql/internal -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/tcl8.6  -c -o schema_cache.o schema_cache.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -fpic -I/usr/local/include  -std=c99 -I. -I./ -I/usr/include/postgresql/9.4/server -I/usr/include/postgresql/internal -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/tcl8.6  -c -o protocol.o protocol.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -fpic -I/usr/local/include  -std=c99 -I. -I./ -I/usr/include/postgresql/9.4/server -I/usr/include/postgresql/internal -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/tcl8.6  -c -o protocol_server.o protocol_server.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -I/usr/include/mit-krb5 -fPIC -pie -DLINUX_OOM_SCORE_ADJ=0 -fno-omit-frame-pointer -fpic -I/usr/local/include  -std=c99 -I.   The problem is an incompatible change in libpq-dev 9.5 (changes some macros for function attributes), which breaks the Bottled Water build.  It seems at some point the postgres:9.4 Docker image updated to include libpq-dev 9.5.  I can reproduce this after clearing my Docker build cache.
I've pushed a pin-libpq branch to my fork that fixes the issue for me - @scordeiro can you confirm if it works for you?

@samstokes Yes, it works, many thanks for the quick fix!
May I suggest this build file and future BW's Docker build files always explicitly keep/lock the necessary versions of the required packages? Is it possible?
I believe it may help to eliminate issues like this so that the default file always builds without breaking.

@scordeiro I've merged my fix into master, sorry for the delay in doing so.
Re:

May I suggest this build file and future BW's Docker build files always explicitly keep/lock the necessary versions of the required packages? Is it possible?

I agree in principle with pinning dependencies to known-good versions, but the situation is a little complicated here:

The only semantic requirement Bottled Water makes of Postgres is logical replication, present in 9.4+, so we don't really require a specific version; we should really be able to build against any version 9.4+, give or take this build error which is a bug (#49).
It's actually fairly tricky to pin all your dependencies in a Debian system, unless you explicitly list out all your transitive dependencies, which could be tens or hundreds of packages.  (I don't know of an equivalent to Ruby's Gemfile.lock, which does explicitly track the version of every transitive dependency, but is also managed by the Bundler tool rather than by hand.)  Also, if you do that, you lose access to security updates.
At least until we have a continuous integration setup (#51), the only way we have of discovering this sort of incompatibility with a new package version is by someone running into it :(

It makes sense to pin libpq-dev until #49 is fixed, since that's the only package with which we have a known compile-time incompatibility.  For the reasons above though I'm tempted to propose actually reverting my fix once we do fix the build against libpq-dev 9.5.

@samstokes Thanks for the insight.
I found a (new) problem with your fix, have you tried to build an image with a few days afterward?
I'm now having issues using $PG_VERSION because the latest debian jessie image appends some "garbage" to the version variable when you query it, it looks like 9.4.*+deb8u1 now instead of only 9.4.*, this causes the build to fail:
E: Version '9.4.5-2.pgdg80+1' for 'libpq5' was not found
E: Version '9.4.5-2.pgdg80+1' for 'libpq-dev' was not found
E: Version '9.4.5-2.pgdg80+1' for 'postgresql-server-dev-9.4' was not found

I had to replace the PG_VERSION with 9.4\* so the correct libs are fetch, this is how the first part of the build file looks like:
FROM postgres:9.4

RUN apt-get update && \
    # --force-yes is needed because we are downgrading libpq5 to $PG_VERSION
    # (set by the postgres:9.4 Docker image).  Confusingly the postgres:9.4
    # Docker image includes libpq5 version 9.5.x, which we are not yet
    # compatible with, even though the Postgres version (and $PG_VERSION) are
    # 9.4.x.
    apt-get install -y --force-yes \
        build-essential \
        cmake \
        curl \
        libcurl4-openssl-dev \
        libjansson-dev \
        libpq5=9.4\* \
        libpq-dev=9.4\* \
        pkg-config \
        postgresql-server-dev-9.4=9.4\*

And the PostgreSQL installed libs:
Get:2 http://apt.postgresql.org/pub/repos/apt/ jessie-pgdg/9.4 libpq5 amd64 9.4.6-1.pgdg80+1 [123 kB]
Get:11 http://apt.postgresql.org/pub/repos/apt/ jessie-pgdg/9.4 libpq-dev amd64 9.4.6-1.pgdg80+1 [163 kB]
Get:15 http://apt.postgresql.org/pub/repos/apt/ jessie-pgdg/main postgresql-server-dev-9.4 amd64 9.4.6-1.pgdg80+1 [636 kB]

I hope this fix helps overcoming any compilation issue that may arise against PostgreSQL 9.4 compilations.

@scordeiro thanks for the report - it's a pain that even tagged docker images can change and break like that!
I just tried a build with the latest postgres:9.4 image (via docker pull postgres:9.4 followed by a rebuild) and I can't reproduce the problem you described.  Could you check whether you still get the error with the latest image?

@samstokes Now it's working for 9.4. I noticed the image got updated again. By the time I did the test that failed, I did update the postgres:9.4 image with a pull, so maybe something changed in the meanwhile.
I also built it with postgres:9.5, I changed the postgresql-server-9.4 to postgresql-server-dev-${PG_MAJOR} so it can pick-up the new package version only by switching the major version in FROM postgres:9.*.
FYI, to ensure the kind of error I got doesn't happen again, I replaced the ${PG_VERSION} variable with ${PG_MAJOR}*, mind the asterisk * so it can choose the latest minor version without worrying with any suffixes that the OS maintainers may add that can break it.
Also great to know you were able to build it against 9.5.  Did you need to make any changes to the source to get around the original compile issue with 9.5 headers, or did that also get magically fixed by a newer version? :)

@samstokes Please check PR #55 :)
I've added a simple script in the build directory to help build and rebuild the Docker infrastructure that is suggested in the main page, for now it supports only 9.4 and 9.5, you can test the build against either of those.
Mind only that if you already have the containers created with Bottled Water's standard names and you want to keep using them, you'll probably want to stop and rename them before using the new script.
Feel free to comment and suggest any improvements.


BUG-cannot create scenario with label containing accented char like é à

Description
cannot create scenario with label containing accented char like é à
How to reproduce

A code fragmentimport taipy as tp;
scenario = tp.create_scenario(...name = "Frédéric")
...

Expected behavior
Either the scenario is created or an explicit error is shown
Runtime environment
Please specify relevant indications.

Taipy version: 3.0.0.dev3

@FredLL-Avaiga  hmmm I was able to create a scenario using this code fragment:
import taipy as tp
from taipy import Config

scenario_config = Config.configure_scenario("scenario_1")
scenario = tp.create_scenario(scenario_config, name = "Frédéric")

print(scenario)   # Frédéric
print(scenario.name)   # Frédéric

So not sure what the problem is, can you help explain it in more details?

I tried through the UI ...

Do we know the input value going into core? might be due to different encoding mechanism as well?

When I create a Scenario with name="a", it is listed by the get_cycles_scenarios()
When I create a Scenario with name="é", it is not listed by the get_cycles_scenarios()


OpenApi-format filtering

When passing the following as filter options:
{
  "operationIds": ["proxyCall"],
  "tags": ["Proxy"]
}

The operations is removed, but the tag is not resulting in an empty folder in the Postman collection.
portman: 0.1.0
openapi-format being using within portman: 1.2.3

@nicklloyd what is the actual result? And what is expected?

Fixed in the 1.2.5 release of openapi-format


bug: check depth for create (and call?) < 1024

Bug Report

Current behavior:
Currently, we entering a new sub context, we don't check depth < 1024
We know that it's unlucky there'll ever be enough gas to have such depth, but still, we need to check those invariants for specs and ef-tests.

Expected behavior:
In create and call, check for depth < 1024
Open to proposal for this:

an execution context field
something else

See #845


You'd better change this to match the changes

You'd better change this to match the changes so as to better user's experience

@harrish0225 : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off


Issue #521: Fix UserData members to be Blobs for base64 encoding

I'm not 100% sure whether I should change the UserData structure as well:
https://github.com/boto/botocore/blob/develop/botocore/data/aws/ec2/2014-10-01.normal.json#L11136

Coverage decreased (-0.58%) to 93.87% when pulling 1fa5a47f4ea354189bb70b858619dda8ec65b8c6 on charleso:issue/521-userdata-blob into 875dfe5c495689015c0fda3eadf2a7d8ccf820d0 on boto:develop.

Coverage decreased (-0.58%) to 93.87% when pulling 1fa5a47f4ea354189bb70b858619dda8ec65b8c6 on charleso:issue/521-userdata-blob into 875dfe5c495689015c0fda3eadf2a7d8ccf820d0 on boto:develop.

@charleso
It is my bad that I did not mention this in my comment on the issue you opened, but as per our contributing guidelines, we do not accept pull request that update the .normal.json models because we pull these from an internal upstream source, and the change would need to happen at the source.
We will look into updating the shape for UserData upstream though and removing the custom handler we have to account for the wrong type. Thanks again though for pointing this out. Closing PR.


CPO-combined

Combined API's to be used in demo
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[ ] No

Closing as the demo has been completed and this branch is no longer needed


handle BOOTArray

This handles https://github.com/lizmat/eigenstates/issues/3 .

Thank you for the PR.  The solution is however very CPU intensive, I've devised a different solution for it, and used the test from this PR.
Version 0.10 now available on zef.  Closing this now


Castlevania 64?

Is there any limits to the current recompiler, technically?
As in, are there any games that use hardware features that can't be recompiled into modern code?

This is not the place to request specific games. As far as limitations go, any ones I know of are already listed in the project's readme.


pangene lookup can break by 414: Request-URI Too Long

A gene list that is under the max 100 items can nevertheless break the tool internally ("Failed to load data") when the number of characters of the gene list exceeds a certain size (<2000). Since the current implementation requires full yuck in each input gene, it means the effective limit is smaller than 100, which is probably fine for now but the mysterious errors received for gene lists in the gray zone are fairly unfriendly. Long term we are going to need to do something that allows us to have longer lists.

Perhaps the solution here is to simply disable the querystring parameter functionality for this component. What do you think of this @adf-ncgr?

Yeah, I think we had talked about that once; I'm in favor.

after the code update to disable adding the long gene list to the query string this is the error I see in the console, which seems to indicate that there's a long URI in a GET to the intermine server:
{message: '414: Request-URI Too Long', locations: Array(1), path: Array(1), extensions: {…}}extensions: code: "INTERNAL_SERVER_ERROR"response: {url: 'https://mines.legumeinfo.org/legumemine/service/qu…alue%3D%27gnm4%27%2F%3E%3C%2Fquery%3E&format=json', status: 414, statusText: 'Request-URI Too Long', body: '<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">…limit for this server.
\n
\n\n'}body: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n\n<title>414 Request-URI Too Long</title>\n\n
Request-URI Too Long
\n
The requested URL's length exceeds the capacity\nlimit for this server.
\n
\n\n"status: 414statusText: "Request-URI Too Long"url: "https://mines.legumeinfo.org/legumemine/service/q[[Prototype]]: Object[[Prototype]]: Objectlocations: [{…}]message: "414: Request-URI Too Long"path: ['panGenePairs'][[Prototype]]: Object 0 [{…}]
The query string update may have solved an earlier error, but apparently we're not out of the woods. @alancleary let me know if you want me to send you the gene list that triggers this

@adf-ncgr Sorry for the delay. Yes, please send me the gene list. This result seems to indicate that the original error wasn't even related to the web component querystring parameters. If this is the case, then do we actually want to remove the querystring functionality from the component (via PR #404)?

OK, just emailed it to you. I'm not %100 sure whether or not the original error was the same as this, but I think it's probably still a good idea to keep the gene list out of the query string. It looked like in the new version all query parameters were being kept out not just the long gene list, but that's probably fine. At this point, I'd say whatever is easiest for now and we can revisit later if needed.

I was able to replicate the error using the gene list you sent. The stacktrace in the console clearly states that the error is coming from Intermine. This can be confirmed by searching the Intermine repo on GitHub for the error message which reveals a comment in the code explaining that Intermine has a length limit on the URLs it serves. So the only way I see around this is to further reduce the number of genes the lookup component accepts or sending requests to Intermine via POSTs. @adf-ncgr do you know if the latter is possible? I naively tried to convert our existing GET requests to POSTs but Intermine didn't like it.

I would have thought POSTing was an option, but I don't know for sure. That seems like the way to go if possible. I will look into it, but if we can't make that work in short order I guess we'll have to just reduce the number of genes as you suggest.

Eavesdropping on the Intermine web interface reveals that imjs is in fact using POST requests. For some reason, though, I can't get the request structured correctly using the Apollo RESTDataSource API. This may be due to the fact that the body of the request sent by imjs is encoded as a URL querystring parameters string, rather than serialized JSON...

Thanks for looking into it; if I understand what you're describing, it sounds like it should be solvable one way or another though perhaps not by using official RESTDataSource API means? Alternatively, we could try to tweak it on the intermine end, though that may be more involved. I've pretty much been in meetings all day but should have some time soon to look at it more closely, then maybe tomorrow we can discuss further.

So apparently there's some headers that have to be included with the POST request otherwise Intermine will try and parse it like a GET. Also, the way the body of the post needs to be encoded as a string is very cryptic. I can successfully make a request using the RESTDataSource API if I hard-code the body from a post request sent via the Intermine web interface but if I attempt to programmatically encode the same string via the GraphQL server then it breaks. Egad, this is annoying!

It's working! I can successfully submit the complete gene list you sent me. I'll follow up via email regarding next steps.

AWESOME! glad you got it to work despite the need to do some olde-time cussing (are the kids really saying "egad" these days?)
look forward to your next steps email

This sent us down quite the rabbit hole over in the graphql-server repo; see graql-server PR #171. In the end, the error was coming from InterMine and the solution was for the GraphQL server to send POST requests to InterMine instead of GET requests. So for now, the web component repo will continue to send requests to the GraphQL server via GET requests, but I have already confirmed that Apollo accepts POST requests, so we can switch our web component client to make POST requests if/when necessary.


Try to lookup in the shared dict when the lock times out

This attempts to recover from fairly rare race condition that might cause some threads to exit in a timeout state while the value is actually available. I've seen it happening only under very high load and it was still pretty rare.

Due to race conditions with other blocked requests in other worker
processes, the lock might time out because other processes took the
shared dict causing the resty.lock spin lock to wait again and again.
This just works around the problem by querying the shared dict for lock
times out. The real problem is that a dumb lock is not the right
synchronization tool here: a condition-style lock that releases all
stalled requests without requiring further locking would be more
efficient as it would prevent the unnecessary timeout at the first
place.

The real problem is that a dumb lock is not the right
synchronization tool here: a condition-style lock that releases all
stalled requests without requiring further locking would be more
efficient as it would prevent the unnecessary timeout at the first
place.

Indeed. We can probably write our own little helper for this purpose. It seems to me like a "dumber" lock (one that does not subsequently lock after the first unlock) would be fine here.

Patch committed to master. Thanks for the report!


Trying to convert lmsys/fastchat-t5-3b-v1.0

Not really an bug with Transformers.js, but with the conversion script.
Got an error when trying to convert lmsys/fastchat-t5-3b-v1.0 with text2text-generation-with-past task.
Using task text2text-generation works fine, though.
Am I missing something?
And is there a way to run the model without it being created with -with-past?
Currently when I run const pipe = await pipeline("text2text-generation", "lmsys/fastchat-t5-3b-v1.0"); it triggers
Error: File not found. Could not locate "models/lmsys/fastchat-t5-3b-v1.0/seq2seq-lm-with-past/encoder_model.onnx".

Files inside models/lmsys/fastchat-t5-3b-v1.0/seq2seq-lm-with-past/ are the following:
added_tokens.json
config.json
generation_config.json
model.onnx
model.onnx_data
special_tokens_map.json
spiece.model
tokenizer.json
tokenizer_config.json

How to reproduce
Run:
python -m scripts.convert --model_id lmsys/fastchat-t5-3b-v1.0 --from_hub --quantize --task text2text-generation-with-past

Expect an output like this:
Using framework PyTorch: 2.0.0
Overriding 1 configuration item(s)
        - use_cache -> True
Asked a sequence length of 16, but a sequence length of 1 will be used with use_past == True for `input_ids`.
Asked a sequence length of 16, but a sequence length of 1 will be used with use_past == True for `decoder_input_ids`.
/opt/homebrew/lib/python3.10/site-packages/transformers/modeling_utils.py:828: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if causal_mask.shape[1] < attention_mask.shape[1]:
/opt/homebrew/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:507: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  elif past_key_value.shape[2] != key_value_states.shape[1]:
In-place op on output of tensor.shape. See https://pytorch.org/docs/master/onnx.html#avoid-inplace-operations-when-using-tensor-shape-in-tracing-mode
In-place op on output of tensor.shape. See https://pytorch.org/docs/master/onnx.html#avoid-inplace-operations-when-using-tensor-shape-in-tracing-mode
================ Diagnostic Run torch.onnx.export version 2.0.0 ================
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

Saving external data to one file...
Asked a sequence length of 16, but a sequence length of 1 will be used with use_past == True for `input_ids`.
Asked a sequence length of 16, but a sequence length of 1 will be used with use_past == True for `decoder_input_ids`.
Traceback (most recent call last):
  File<EMAIL_ADDRESS>line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File<EMAIL_ADDRESS>line 86, in _run_code
    exec(code, run_globals)
  File "/Users/user/Repositories/transformers.js/scripts/convert.py", line 310, in <module>
    main()
  File "/Users/user/Repositories/transformers.js/scripts/convert.py", line 282, in main
    _, onnx_outputs = export_models(
  File "/opt/homebrew/lib/python3.10/site-packages/optimum/exporters/onnx/convert.py", line 609, in export_models
    export(
  File "/opt/homebrew/lib/python3.10/site-packages/optimum/exporters/onnx/convert.py", line 722, in export
    config.fix_dynamic_axes(output, device=device, input_shapes=input_shapes, dtype=dtype)
  File "/opt/homebrew/lib/python3.10/site-packages/optimum/exporters/onnx/base.py", line 285, in fix_dynamic_axes
    outputs = session.run(None, onnx_inputs)
  File "/opt/homebrew/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 200, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Invalid Feed Input Name:past_key_values.9.encoder.value

Expected behavior
Was expecting it to work, the same way python -m scripts.convert --model_id lmsys/fastchat-t5-3b-v1.0 --from_hub --quantize --task text2text-generation worked.
Environment

Transformers.js version: N/A
Browser (if applicable): N/A
Operating system (if applicable): MacOS

I think this is due to a recent update of optimum - which will be fixed in the next release. One problem I am aware of, however, is that trying to convert large models (e.g., 3b param models) will produce an external data file, which is currently not supported by Transformers.js.
Are you able to export with optimum directly? Your command should look something like:
optimum-cli export onnx -m lmsys/fastchat-t5-3b-v1.0 output

Ah, nice! Using that command made it generate some more files. But the command exited with the following exception:
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/bin/optimum-cli", line 8, in <module>
    sys.exit(main())
  File "/opt/homebrew/lib/python3.10/site-packages/optimum/commands/optimum_cli.py", line 163, in main
    service.run()
  File "/opt/homebrew/lib/python3.10/site-packages/optimum/commands/export/onnx.py", line 219, in run
    main_export(
  File "/opt/homebrew/lib/python3.10/site-packages/optimum/exporters/onnx/__main__.py", line 366, in main_export
    raise Exception(
Exception: An error occured during validation, but the model was saved nonetheless at output. Detailed error: [ONNXRuntimeError] : 1 : FAIL : Load model from output/decoder_model_merged.onnx failed:/Users/runner/work/1/s/onnxruntime/core/graph/model.cc:146 onnxruntime::Model::Model(onnx::ModelProto &&, const onnxruntime::PathString &, const onnxruntime::IOnnxRuntimeOpSchemaRegistryList *, const logging::Logger &, const onnxruntime::ModelOptions &) Unsupported model IR version: 9, max supported IR version: 8

List of files generated (including the encoder_model.onnx, which was not being generated by the conversion script):
added_tokens.json
config.json
decoder_model.onnx
decoder_model.onnx_data
decoder_model_merged.onnx
decoder_model_merged.onnx_data
decoder_with_past_model.onnx
decoder_with_past_model.onnx_data
encoder_model.onnx
encoder_model.onnx_data
generation_config.json
special_tokens_map.json
spiece.model
tokenizer.json
tokenizer_config.json

Thanks 👍 The first error message you received looks like a bug with optimum. If you'd like, you can raise an issue on their repo.
For now (even if you did get conversion working), the model is currently just slightly too large to run with the current version of Transformers.js (which doesn't support the external format: .onnx_data). I will hopefully get around to adding support for it in the coming week or so, but I am just prioritizing other things first.

Thanks for your work on this lib!
The pipeline didn't work for lmsys/fastchat-t5-3b-v1.0, although all the files were there. When running transformers.js in Node, and it's exiting with the following message:
Error: no available backend found. ERR:
Error: Failed to load model with error: /Users/runner/work/1/s/onnxruntime/core/graph/model.cc:146 onnxruntime::Model::Model(onnx::ModelProto &&, const onnxruntime::PathString &, const onnxruntime::IOnnxRuntimeOpSchemaRegistryList *, const logging::Logger &, const onnxruntime::ModelOptions &) Unsupported model IR version: 9, max supported IR version: 8

But I'm happy it's close to getting it working! The other T5 model (LaMini-Flan-T5-783M) is working great, and it's already pretty good for its purpose.

Right, it's failing because the .onnx file does not contain all the model parameters. Those parameters are stored in .onnx_data.
This is due to a limitation with protobuf, which has a 2GB limit. See here for more information.
This also means optimum's validation code doesn't load the external data format (which also appears to be a bug).
However, I do intend to add support for the external data format :) Just got a lot on my plate right now haha.

Updated title to be a feature request for the external data file format (which is used for models larger than 2GB).

@xenova Does transformers.js support models larger than 2 GB yet?

Any news on this one? I am trying to load bge-m3 (full, not quantized, dtype='fp32') using the v3 branch (onnxruntime-1.17.x) in nodejs and I am getting the following error:
Error: Deserialize tensor embeddings.word_embeddings.weight failed.GetFileLength for ./model.onnx_data failed:Invalid fd was supplied: -1
(which is weird, since the model.onnx_data file is there).

Pinging for any updates on this too! I am trying to load Xenova/TinyLlama-1.1B-Chat-v1.0 using the v3 branch (onnxruntime-1.17.x) and there is a decoder_model_merged.onnx_data file but I still get this.
Error: ERROR_CODE: 1, ERROR_MESSAGE: Deserialize tensor onnx::MatMul_7210 failed.Failed to load external data file "./model.onnx_data", error: Module.MountedFiles is not available.


Migrate strorage-plus to v0.12.0

This seems to have been done https://github.com/CosmWasm/cosmwasm-plus/blob/master/packages/storage-plus/Cargo.toml#L16


Frame Freeze using HumanPose

Ubuntu 18.02 Python 3.7 OpenCV 4 with i5 cpu and GTX 1660 Super GPU.   Mediapipe has been compiled with OpenGL ES.
When streaming from a webcam (1920 x 1080 @ 60 fps), the video is freezing after a few frames.   Any recommendations?

Hi @mcdonasd1212 ,can you please recorded video and code changes  to investigate further.

The issue was related to memory usage from the camera. This can be considered closed.


Add indicator to avatar

Details

Fixed Issues

$ https://github.com/Expensify/Expensify/issues/218719
Tests

This needs manually changing data/code to test (since the API is not done yet):

Expose Onyx variable to window ie: window.Onyx = Onyx on a file that imports Onyx
Set some errors in the employee list of a policy you have, ie: Onyx.set('policyMemberList_2D0BDD1B7A684C67'<EMAIL_ADDRESS>{pending: 'add', errors: {1: 'This is an error'}}})
Check avatar has a red dot

Check the policy you set the error in has a red dot

Check manage members of that policy (but not others) has the red dot

Here add an error and pendingAction: ie: .map(person => ({...person, pendingAction: 'delete', errors: {1: 'caca'}}))
Check the member has the error displayed

Note that dismissing the error is not working now, since the error is hardcoded in the code above

PR Review Checklist

Contributor (PR Author) Checklist

[ ] I linked the correct issue in the ### Fixed Issues section above
[ ] I wrote clear testing steps that cover the changes made in this PR

[ ] I added steps for local testing in the Tests section
[ ] I added steps for Staging and/or Production testing in the QA steps section
[ ] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[ ] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[ ] I included screenshots or videos for tests on all platforms
[ ] I ran the tests on all platforms & verified they passed on:

[ ] iOS / native
[ ] Android / native
[ ] iOS / Safari
[ ] Android / Chrome
[ ] MacOS / Chrome
[ ] MacOS / Desktop

[ ] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[ ] I followed proper code patterns (see Reviewing the code)

[ ] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[ ] I verified that comments were added to code that is not self explanatory
[ ] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[ ] I verified any copy / text shown in the product was added in all src/languages/* files
[ ] I verified any copy / text that was added to the app is correct English and approved by marketing by tagging the marketing team on the original GH to get the correct copy.
[ ] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[ ] I verified the JSDocs style guidelines (in STYLE.md) were followed

[ ] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[ ] I followed the guidelines as stated in the Review Guidelines
[ ] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[ ] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[ ] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[ ] If a new component is created I verified that:

[ ] A similar component doesn't exist in the codebase
[ ] All props are defined accurately and each prop has a /** comment above it */
[ ] Any functional components have the displayName property
[ ] The file is named correctly
[ ] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[ ] The only data being stored in the state is data necessary for rendering and nothing else
[ ] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[ ] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[ ] All JSX used for rendering exists in the render method
[ ] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[ ] If a new CSS style is added I verified that:

[ ] A similar style doesn't already exist
[ ] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[ ] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[ ] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[ ] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

PR Reviewer Checklist
The Contributor+ will copy/paste it into a new comment and complete it after the author checklist is completed

[ ] I have verified the author checklist is complete (all boxes are checked off).
[ ] I verified the correct issue is linked in the ### Fixed Issues section above
[ ] I verified testing steps are clear and they cover the changes made in this PR

[ ] I verified the steps for local testing are in the Tests section
[ ] I verified the steps for Staging and/or Production testing are in the QA steps section
[ ] I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[ ] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[ ] I checked that screenshots or videos are included for tests on all platforms
[ ] I verified tests pass on all platforms & I tested again on:

[ ] iOS / native
[ ] Android / native
[ ] iOS / Safari
[ ] Android / Chrome
[ ] MacOS / Chrome
[ ] MacOS / Desktop

[ ] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[ ] I verified proper code patterns were followed (see Reviewing the code)

[ ] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
[ ] I verified that comments were added to code that is not self explanatory
[ ] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[ ] I verified any copy / text shown in the product was added in all src/languages/* files
[ ] I verified any copy / text that was added to the app is correct English and approved by marketing by tagging the marketing team on the original GH to get the correct copy.
[ ] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[ ] I verified the JSDocs style guidelines (in STYLE.md) were followed

[ ] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[ ] I verified that this PR follows the guidelines as stated in the Review Guidelines
[ ] I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
[ ] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[ ] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[ ] If a new component is created I verified that:

[ ] A similar component doesn't exist in the codebase
[ ] All props are defined accurately and each prop has a /** comment above it */
[ ] Any functional components have the displayName property
[ ] The file is named correctly
[ ] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[ ] The only data being stored in the state is data necessary for rendering and nothing else
[ ] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[ ] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[ ] All JSX used for rendering exists in the render method
[ ] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[ ] If a new CSS style is added I verified that:

[ ] A similar style doesn't already exist
[ ] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[ ] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[ ] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[ ] I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

QA Steps

[ ] Verify that no errors appear in the JS console

Screenshots

Web

Mobile Web

Desktop

iOS

Android

@iwiznia can you add some testing steps? please

Yeah sorry, sent this in late yesterday and wanted an early review (expecting comments and changes anyway). Doing that now. This is again a hard to test PR since the backend is not ready and so manual things are needed.

did an initial review, looks good so far - a few questions, will item.login change?

Probably...

Seems @aldo-expensify is OOO, so merging as is. If there are additional comments, I can address those in another PR

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

🚀 Deployed to staging by @iwiznia in version: 1.1.87-0 🚀

platform
result

🤖 android 🤖
failure ❌

🖥 desktop 🖥
failure ❌

🍎 iOS 🍎
failure ❌

🕸 web 🕸
failure ❌

🚀 Deployed to production by @yuwenmemon in version: 1.1.87-9 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


refactor(instructions): remove unnecessary exists checks on instructions

Exists checks are not required, as it is not possible to get a non-existant Instruction instance

:tada: This PR is included in version 7.0.0-alpha.5 :tada:
The release is available on:

npm package (@alpha dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 7.0.0-beta.1 :tada:
The release is available on:

npm package (@beta dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 7.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


WIP: Log with slog to stdout in JSON form

Before:
go run ./ agent --one-shot --output-path /foo/bar --api-token should-not-be-required  --agent-config-file examples/cert-manager-agent.yaml 2>stderr >stdout

stdout

stderr
2024/10/15 11:56:20 Preflight agent version: development ()
2024/10/15 11:56:20 Using the Jetstack Secure API Token auth mode since --api-token was specified.
2024/10/15 11:56:20 Using deprecated Endpoint configuration. User Server instead.
2024/10/15 11:56:20 starting "k8s/secrets.v1" datagatherer
2024/10/15 11:56:20 starting "k8s/certificates.v1alpha2.cert-manager.io" datagatherer
2024/10/15 11:56:20 starting "k8s/ingresses.v1beta1.networking.k8s.io" datagatherer
2024/10/15 11:56:20 starting "k8s/certificaterequests.v1alpha2.cert-manager.io" datagatherer
W1015 11:56:20.490541  347732 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list networking.k8s.io/v1beta1, Resource=ingresses: the server could not find the requested resource
2024/10/15 11:56:20 server missing resource for datagatherer of "networking.k8s.io/v1beta1, Resource=ingresses"
W1015 11:56:20.495094  347732 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list cert-manager.io/v1alpha2, Resource=certificaterequests: the server could not find the requested resource
2024/10/15 11:56:20 server missing resource for datagatherer of "cert-manager.io/v1alpha2, Resource=certificaterequests"
W1015 11:56:20.497450  347732 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list cert-manager.io/v1alpha2, Resource=certificates: the server could not find the requested resource
2024/10/15 11:56:20 server missing resource for datagatherer of "cert-manager.io/v1alpha2, Resource=certificates"
W1015 11:56:21.904811  347732 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list cert-manager.io/v1alpha2, Resource=certificaterequests: the server could not find the requested resource
2024/10/15 11:56:21 server missing resource for datagatherer of "cert-manager.io/v1alpha2, Resource=certificaterequests"
W1015 11:56:22.043669  347732 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list cert-manager.io/v1alpha2, Resource=certificates: the server could not find the requested resource
2024/10/15 11:56:22 server missing resource for datagatherer of "cert-manager.io/v1alpha2, Resource=certificates"
W1015 11:56:22.080127  347732 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list networking.k8s.io/v1beta1, Resource=ingresses: the server could not find the requested resource
2024/10/15 11:56:22 server missing resource for datagatherer of "networking.k8s.io/v1beta1, Resource=ingresses"
W1015 11:56:23.938818  347732 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list cert-manager.io/v1alpha2, Resource=certificates: the server could not find the requested resource
2024/10/15 11:56:23 server missing resource for datagatherer of "cert-manager.io/v1alpha2, Resource=certificates"
W1015 11:56:24.441234  347732 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list networking.k8s.io/v1beta1, Resource=ingresses: the server could not find the requested resource
2024/10/15 11:56:24 server missing resource for datagatherer of "networking.k8s.io/v1beta1, Resource=ingresses"
W1015 11:56:24.564356  347732 reflector.go:561] pkg/mod/k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: failed to list cert-manager.io/v1alpha2, Resource=certificaterequests: the server could not find the requested resource
2024/10/15 11:56:24 server missing resource for datagatherer of "cert-manager.io/v1alpha2, Resource=certificaterequests"
2024/10/15 11:56:25 failed to complete initial sync of "k8s-dynamic" data gatherer "k8s/certificates.v1alpha2.cert-manager.io": timed out waiting for Kubernetes caches to sync
2024/10/15 11:56:25 failed to complete initial sync of "k8s-dynamic" data gatherer "k8s/ingresses.v1beta1.networking.k8s.io": timed out waiting for Kubernetes caches to sync
2024/10/15 11:56:25 failed to complete initial sync of "k8s-dynamic" data gatherer "k8s/certificaterequests.v1alpha2.cert-manager.io": timed out waiting for Kubernetes caches to sync
2024/10/15 11:56:25 successfully gathered 22 items from "k8s/secrets.v1" datagatherer
2024/10/15 11:56:25 successfully gathered 0 items from "k8s/certificates.v1alpha2.cert-manager.io" datagatherer
2024/10/15 11:56:25 successfully gathered 0 items from "k8s/ingresses.v1beta1.networking.k8s.io" datagatherer
2024/10/15 11:56:25 successfully gathered 0 items from "k8s/certificaterequests.v1alpha2.cert-manager.io" datagatherer
2024/10/15 11:56:25 failed to output to local file: open /foo/bar: no such file or directory
exit status 1

One-shot output with deliberate config error
$ go run ./ agent --one-shot --output-path /foo/bar --api-token should-not-be-required
{"time":"2024-10-15T11:28:22.360581783+01:00","level":"INFO","msg":"Preflight agent version: development ()"}
{"time":"2024-10-15T11:28:22.361846945+01:00","level":"INFO","msg":"Using the Jetstack Secure API Token auth mode since --api-token was specified."}
{"time":"2024-10-15T11:28:22.361875051+01:00","level":"INFO","msg":"ignoring the venafi-cloud.uploader_id field in the config file. This field is not needed in Jetstack Secure API Token mode."}
{"time":"2024-10-15T11:28:22.361899488+01:00","level":"INFO","msg":"starting \"dummy\" datagatherer"}
{"time":"2024-10-15T11:28:22.361919085+01:00","level":"INFO","msg":"starting \"dummy-fail\" datagatherer"}
{"time":"2024-10-15T11:28:22.361970856+01:00","level":"INFO","msg":"failed to output to local file: open /foo/bar: no such file or directory"}
exit status 1

Command line help
$ go run ./ agent --one-shot --output-path /foo/bar --api-token should-not-be-required  -h
The agent will periodically gather data for the configured data
        gatherers and send it to a remote backend for evaluation

Usage:
  preflight agent [flags]
  preflight agent [command]

Available Commands:
  info        print several internal parameters of the agent
  rbac        print the agent's minimal RBAC manifest

Flags:
  -c, --agent-config-file agent.yaml                      Config file location, default is agent.yaml in the current working directory. (default "./agent.yaml")
      --api-token string                                  Turns on the Jetstack Secure API Token mode. Defaults to the value of the env var API_TOKEN.
      --backoff-max-time duration                         Max time for retrying failed data gatherers (given as XhYmZs). (default 10m0s)
      --client-id string                                  Turns on the Venafi Cloud Key Pair Service Account mode. If you use this flag you don't need to use --venafi-cloud as it will assume you are authenticating with Venafi Cloud. Using this removes the need to use a credentials file.
  -k, --credentials-file string                           Location of the credentials file. For the Jetstack Secure OAuth and Venafi Cloud Key Pair Service Account modes.
      --enable-metrics                                    Enables Prometheus metrics server on the agent (port: 8081).
      --enable-pprof                                      Enables the pprof profiling server on the agent (port: 6060).
  -h, --help                                              help for agent
      --input-path string                                 For testing purposes. In conjunction with --one-shot, it allows you to push manually crafted data readings (in JSON format) to the Venafi Cloud API without the need to connect to a Kubernetes cluster.
      --install-namespace string                          For testing purposes. Namespace in which the agent is running. Only needed with the Venafi Cloud VenafiConnection modewhen running the agent outside of Kubernetes.
      --one-shot                                          For testing purposes. The agent will run once and exit. It is often used in conjunction with --output-path and/or --input-path.
      --output-path string                                For testing purposes. In conjunction with --one-shot, it allows you to write the data readings to a file instead of uploading to the server.
  -p, --period duration                                   Override time between scans in the configuration file (given as XhYmZs).
      --private-key-path string                           To be used in conjunction with --client-id. The path to the private key file for the service account.
      --strict                                            Runs agent in strict mode. No retry attempts will be made for a missing data gatherer's data.
      --venafi-connection string                          Turns on the Venafi Cloud VenafiConnection mode. This flag configures the name of the VenafiConnection to be used.
      --venafi-connection-namespace allowReferencesFrom   Namespace of the VenafiConnection to be used. It is only useful when the VenafiConnection isn't in the same namespace as the agent. The field allowReferencesFrom must be present on the cross-namespace VenafiConnection for the agent to use it.

Global Flags:
  -v, --log-level int   Set the logging verbosity. 8: >= ERROR, 4: >= WARN, 0: >=INFO, -4: >= DEBUG

Use "preflight agent [command] --help" for more information about a command.

Unit Test output
$ go test ./pkg/logs/... -v
=== RUN   TestLogs
vCert: 2024/10/15 11:07:15 log before initialize
vCert: 2024/10/15 11:07:15 INFO slog before initialize
--- PASS: TestLogs (0.00s)
PASS
ok      github.com/jetstack/preflight/pkg/logs  (cached)

Example logs from the E2E test with a deliberate config error:
{"time":"2024-10-15T10:09:19.118854375Z","level":"INFO","msg":"Preflight agent version: v1.1.0-3-gd97130c33e1830-dirty (d97130c33e183044ac01116511b11779d6f38408)"}
{"time":"2024-10-15T10:09:19.124213625Z","level":"INFO","msg":"Using the Venafi Cloud VenafiConnection auth mode since --venafi-connection was specified."}
{"time":"2024-10-15T10:09:19.124446355Z","level":"INFO","msg":"ignoring the server field specified in the config file. In Venafi Cloud VenafiConnection mode, this field is not needed."}
{"time":"2024-10-15T10:09:19.124459725Z","level":"INFO","msg":"ignoring the venafi-cloud.upload_path field in the config file. In Venafi Cloud VenafiConnection mode, this field is not needed."}
{"time":"2024-10-15T10:09:19.124464355Z","level":"INFO","msg":"ignoring the venafi-cloud.uploader_id field in the config file. This field is not needed in Venafi Cloud VenafiConnection mode."}
{"time":"2024-10-15T10:09:19.124486005Z","level":"INFO","msg":"Using period from config 1m0s"}
{"time":"2024-10-15T10:09:19.168849156Z","level":"INFO","msg":"Prometheus was enabled.\nRunning prometheus on port :8081"}
{"time":"2024-10-15T10:09:24.191872114Z","level":"INFO","msg":"Posting data to: "}
{"time":"2024-10-15T10:09:24.355308739Z","level":"INFO","msg":"retrying in 43.757356959s after error: post to server failed: while loading the VenafiConnection venafi/venafi-components: connection is not ready yet (building connection failed): chain element 1 (VCPOAuth) error: Post \"https://api.venafi.eu.example/v1/oauth2/v2.0/9a0cab61-2b00-11ee-ba09-0733b0fe5adc/token\": dial tcp: lookup api.venafi.eu.example on <IP_ADDRESS>:53: no such host"}
{"time":"2024-10-15T10:10:08.115125429Z","level":"INFO","msg":"Posting data to: "}
{"time":"2024-10-15T10:10:08.191519822Z","level":"INFO","msg":"retrying in 42.657399976s after error: post to server failed: while loading the VenafiConnection venafi/venafi-components: connection is not ready yet (building connection failed): chain element 1 (VCPOAuth) error: Post \"https://api.venafi.eu.example/v1/oauth2/v2.0/9a0cab61-2b00-11ee-ba09-0733b0fe5adc/token\": dial tcp: lookup api.venafi.eu.example on <IP_ADDRESS>:53: no such host"}
{"time":"2024-10-15T10:10:50.850078426Z","level":"INFO","msg":"Posting data to: "}
{"time":"2024-10-15T10:10:50.910425557Z","level":"INFO","msg":"retrying in 43.526318066s after error: post to server failed: while loading the VenafiConnection venafi/venafi-components: connection is not ready yet (building connection failed): chain element 1 (VCPOAuth) error: Post \"https://api.venafi.eu.example/v1/oauth2/v2.0/9a0cab61-2b00-11ee-ba09-0733b0fe5adc/token\": dial tcp: lookup api.venafi.eu.example on <IP_ADDRESS>:53: no such host"}
{"time":"2024-10-15T10:11:34.437054765Z","level":"INFO","msg":"Posting data to: "}
{"time":"2024-10-15T10:11:34.507746957Z","level":"INFO","msg":"retrying in 2m27.200451291s after error: post to server failed: while loading the VenafiConnection venafi/venafi-components: connection is not ready yet (building connection failed): chain element 1 (VCPOAuth) error: Post \"https://api.venafi.eu.example/v1/oauth2/v2.0/9a0cab61-2b00-11ee-ba09-0733b0fe5adc/token\": dial tcp: lookup api.venafi.eu.example on <IP_ADDRESS>:53: no such host"}
{"time":"2024-10-15T10:14:01.70912232Z","level":"INFO","msg":"Posting data to: "}
{"time":"2024-10-15T10:14:01.765769857Z","level":"INFO","msg":"retrying in 1m44.386748573s after error: post to server failed: while loading the VenafiConnection venafi/venafi-components: connection is not ready yet (building connection failed): chain element 1 (VCPOAuth) error: Post \"https://api.venafi.eu.example/v1/oauth2/v2.0/9a0cab61-2b00-11ee-ba09-0733b0fe5adc/token\": dial tcp: lookup api.venafi.eu.example on <IP_ADDRESS>:53: no such host"}
{"time":"2024-10-15T10:15:46.152698136Z","level":"INFO","msg":"Posting data to: "}
{"time":"2024-10-15T10:15:46.230646545Z","level":"INFO","msg":"retrying in 1m59.783143995s after error: post to server failed: while loading the VenafiConnection venafi/venafi-components: connection is not ready yet (building connection failed): chain element 1 (VCPOAuth) error: Post \"https://api.venafi.eu.example/v1/oauth2/v2.0/9a0cab61-2b00-11ee-ba09-0733b0fe5adc/token\": dial tcp: lookup api.venafi.eu.example on <IP_ADDRESS>:53: no such host"}
{"time":"2024-10-15T10:17:46.014726631Z","level":"INFO","msg":"Posting data to: "}
{"time":"2024-10-15T10:17:46.077177178Z","level":"INFO","msg":"retrying in 2m51.121170931s after error: post to server failed: while loading the VenafiConnection venafi/venafi-components: connection is not ready yet (building connection failed): chain element 1 (VCPOAuth) error: Post \"https://api.venafi.eu.example/v1/oauth2/v2.0/9a0cab61-2b00-11ee-ba09-0733b0fe5adc/token\": dial tcp: lookup api.venafi.eu.example on <IP_ADDRESS>:53: no such host"}
{"time":"2024-10-15T10:20:37.199232157Z","level":"INFO","msg":"Posting data to: "}
{"time":"2024-10-15T10:20:37.260874746Z","level":"INFO","msg":"Exiting due to fatal error uploading: post to server failed: while loading the VenafiConnection venafi/venafi-components: connection is not ready yet (building connection failed): chain element 1 (VCPOAuth) error: Post \"https://api.venafi.eu.example/v1/oauth2/v2.0/9a0cab61-2b00-11ee-ba09-0733b0fe5adc/token\": dial tcp: lookup api.venafi.eu.example on <IP_ADDRESS>:53: no such host"}

I chose a different approach. Using klog in #596


ci: Disables quality control on main

Pull Request Test Coverage Report for Build<PHONE_NUMBER>6
Details

0 of 0   changed or added relevant lines in 0 files are covered.
237 unchanged lines in 110 files lost coverage.
Overall coverage increased (+0.8%) to 78.894%

Files with Coverage Reduction
New Missed Lines
%

src/Tempest/Console/src/Components/Interactive/ConfirmComponent.php
1
94.29%

src/Tempest/Console/src/Components/Interactive/MultipleChoiceComponent.php
1
94.87%

src/Tempest/Console/src/Components/Interactive/PasswordComponent.php
1
90.48%

src/Tempest/Console/src/Components/Interactive/ProgressBarComponent.php
1
94.87%

src/Tempest/Console/src/Components/Interactive/SingleChoiceComponent.php
1
94.29%

src/Tempest/Console/src/Components/Interactive/TextBoxComponent.php
1
94.74%

src/Tempest/Console/src/Exceptions/InterruptException.php
1
0.0%

src/Tempest/Console/src/Exceptions/InvalidCommandException.php
1
18.18%

src/Tempest/Console/src/HandlesKey.php
1
0.0%

src/Tempest/Console/src/Highlight/TempestConsoleLanguage/Injections/CommentInjection.php
1
50.0%

Totals

Change from base Build<PHONE_NUMBER>2:
0.8%

Covered Lines:
5476

Relevant Lines:
6941

💛  - Coveralls


fix: fix declaration file output with out_dir set

The default declaration_dir was already calculated up above taking out_dir into account.
I'm not sure if examples is really the best place to test this? 2/3 of those failed to build without this fix...

can we get a patch version release for this please :) ?


Nest. Quick fixes

[x] Add Testnest message on the top right + configuration

](https://www.figma.com/file/ClniQyKFqN1hSpWizV5O4A/Paloma?node-id=3281%3A1986)

The user should be able to open the drop down menu but mainnet should be disabled.

[x] When the address has not activity yet, when the user presses the Activity menu should be a message "You have no activity"

https://www.figma.com/file/ClniQyKFqN1hSpWizV5O4A/Paloma?node-id=3559%3A99

[x] When I press "Switch Wallet" we should show the full background grey.

https://www.figma.com/file/ClniQyKFqN1hSpWizV5O4A/Paloma?node-id=3590%3A123

[x] Add Go back arrow in Switch Wallet view. For go back to previous view.

https://www.figma.com/file/ClniQyKFqN1hSpWizV5O4A/Paloma?node-id=3590%3A123

[x] Add Go back arrow in Send view. For go back to previous view.

https://www.figma.com/file/ClniQyKFqN1hSpWizV5O4A/Paloma?node-id=3400%3A404

[x] When the user presses the logo "NEST" should be redirected to the Balance view.

[x] Replace Extention fav icon with file: "favicon_shadow" https://drive.google.com/drive/u/1/folders/1mPLJ5V6h0TVTLvNvjxZ_gCzuicBgUCT0

@wdev733 @joycehermosilla these all look good to me except the first one. I have two comments

we should use the actual testnet name for testnet, so curreently that's "paloma-testnet-13". Also @wdev733 can you confirm that this can change if we upgrade to a new testnet?
mainnet is clickable (but nothing happens when I click it). Will this work when we actually have a mainnet?


Cst enabled by default

Coverage remained the same at 100.0% when pulling a9f3bcb2805d86c6d9a86f269567c744e870cf50 on cst_on into a3b7a0b42af407f8e7272165a0b8c940afdb989f on master.

Fixes #819


⚠ Empornium has degraded performance

In 4b7431b, Empornium ($EMP) experienced degraded performance:

HTTP code: 200
Response time: 6916 ms

Resolved: Empornium performance has improved in d374d01 after 6 minutes.


Accessibility Issue found on "Datagrid - Select All" using Voice Over on Mac

Describe the bug
Accessibility Issue found on "Datagrid - Select All" using Voice Over on Mac
Refer to video for detail:
https://www.dropbox.com/s/sbwm7izewxk9rw0/Datagrid-SelectAll-AccessibilityIssue.mp4?dl=0
How to reproduce
https://clarity.design/documentation/datagrid/full
Steps to reproduce the behavior:

Go to 'https://clarity.design/documentation/datagrid/full'
On Mac, turn on "Voice Over"
Press "Tab" key till "Apply" button
Press "Tab" key again which should focus on "Select All" of Datagrid

Expected behavior
Voice over should read out "Select All".
Versions
App

Angular: 9.0.0
Node: v12.16.2
Clarity: 3.1.0

Device:

Type: MacBook
OS: OS X
Browser: chrome
Version: 81.0.4044.122 (Official Build) (64-bit)

Additional notes
This issue is not reproducible on Windows Narrator

Clarity cares deeply about accessibility, and we work with our accessibility team to identify and triage issues. We have an internal tracking system to identify and verify accessibility issues, which has captured this already. Some issues with Clarity Angular are difficult to solve without causing side effects or breaking changes, and we have to balance concerns with fixing issues and breaking changes. As we build out Clarity Core, we are able to address these root cause accessibility issues and recommend adopting the Clarity Core implementations as they become available. We are closing this issue as we have the issues tracked internally. Please follow our development and releases to see when we release relevant Clarity Core components to make this possible.


duplicate key value violates unique constraint "IX_ApiScopes_Name"

[ ] I read and understood how to enable logging

Issue / Steps to reproduce the problem
I have implemented an API to manage API Resources of IdentityServer (IdentityServer4).
If I try to update an API Resource, I get this database error. "duplicate key value violates unique constraint "IX_ApiScopes_Name".
Code to update database:
        public void Update(IdentityServer4.Models.ApiResource apiResource)
        {
            _context.ApiResources.Update(apiResource.ToEntity());
            _context.SaveChanges();
        }

To test the update method, I get the ApiResource using get method and just change the description/displayName of the received model, and send it back to update endpoint.
{
  "enabled": true,
  "name": "api1",
  "displayName": "Updated display name",
  "description": null,
  "apiSecrets": [],
  "userClaims": [],
  "scopes": [
    {
      "name": "api1",
      "displayName": "My API",
      "description": null,
      "required": false,
      "emphasize": false,
      "showInDiscoveryDocument": true,
      "userClaims": []
    }
  ]
}

If I remove the scopes part and just change the displayName, then I get following error:
Database operation expected to affect 1 row(s) but actually affected 0 row(s). Data may have been modified or deleted since entities were loaded. See http://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.

Relevant parts of the log file
<log goes here>

Wrong repo -- this should be in the EF repo.


firestore-bigquery-export fs-bq-import-collection import script not found

Step 2: Describe your configuration

Extension name: firestore-bigquery-export

Step 3: Describe the problem
In the IMPORT_EXISTING_DOCUMENTS.md guide there is guide to run Run npx @firebaseextensions/fs-bq-import-collection script. This gives an error:
404 @firebaseextensions/fs-bq-import-collection@latest' is not in the npm registry.
Steps to reproduce:
run
npx @firebaseextensions/fs-bq-import-collection@latest
Expected result
runs the script
Actual result

npm ERR! 404 Not Found - GET https://registry.npmjs.org/@firebaseextensions%2Ffs-bq-import-collection - Not found

@diginikkari I believe the correct script is npx @firebaseextensions/firestore-bigquery-change-tracker. I sent a PR fixing the guide (see #15). Good catch!

@rosariopfernandes I tried npm install @ firebaseextensions/firestore-bigquery-change-tracker and the installation was successful, but npx @ firebaseextensions/firestore-bigquery-change-tracker cannot find the command.

@masashi-sutou You probably don't have npx installed. As described in this post, npx is "A tool for executing Node packages.", which means it will run npm install behind the scenes before executing the package (in case you don't have that package installed yet).
npx comes bundled with npm version 5.2+, so I'd recommend updating npm if you wish to use npx.

@rosariopfernandes npx is working properly with other packages. Seems like there is an issue with the command npx @firebaseextensions/firestore-bigquery-change-tracker

I get the following output when running the commpand $npx @firebaseextensions/firestore-bigquery-change-tracker
command not found: firestore-bigquery-change-tracker

@PVSPHANINDRA @xXCalleManXx You're right! The firestore-bigquery-change-tracker package should expose a fs-bq-import-collection command, but after installing the package, that command  is not found. I've also pointed this out here. Let's wait for a Firebaser to comment on it.

The problem was: the script @firebaseextensions/fs-bq-import-collection was still marked as private (for Googlers only).
But it has now been made public (see full discussion in #15) :)
Running npx @firebaseextensions/fs-bq-import-collection should work just fine now.

Yes, confirmed it as fixed.


Add a page dedicated to examples in the Foundation SDK docs

To make these examples discoverable from the Foundation SDK website as well.

@K-Phoen is there a public preview available anywhere?

Unfortunately not.
The build process is a bit convoluted, building and publishing previews isn't exactly trivial :/


merge package template from Kronos-Tools/npm-package-template

package.json

chore(devDependencies): update<EMAIL_ADDRESS>from template
chore(devDependencies): add<EMAIL_ADDRESS>from template
chore(devDependencies): add<EMAIL_ADDRESS>from template
chore(npm): delete config

Codecov Report

Merging #178 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #178   +/-   ##
=======================================
  Coverage   83.92%   83.92%
=======================================
  Files           1        1
  Lines         168      168
=======================================
  Hits          141      141
  Misses         27       27

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 183ad6a...6c5e1da. Read the comment docs.


[PUPIL-811] Hide inline svg spritesheet

Description
Music year: 1953
This wraps the inline svg sprite with a div and hides it. Applying the style to the sprite itself results in all the svgs within the spritesheet getting that style (seems odd I know), so the only option was to add the extra div.
This was introduced with the recent fix for the latest version of Chrome where svgs on the site stopped displaying.
How to test

Go to https://deploy-preview-2515--oak-web-application.netlify.thenational.academy
Click on any page, there should be no svg visible at the bottom of the page.
Test other areas of the site to check svgs are working
Check if this has resolved drag and drop issue in pupil quizzes

Checklist

[x] Added or updated tests where appropriate
[x] Manually tested across browsers / devices
[x] Considered impact on accessibility
[x] Design sign-off (N/A)
[x] Approved by product owner (N/A)
[x] Does this PR update a package with a breaking change

:tada: This PR is included in version 1.462.3 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


improve consistency of histogram presentation

Now that the team elo calculation has been fixed... i think it'd be better to align a few presentational details of the 1v1 and team plots.
in short:

share an x axis across all 3 plots

the team RM xaxis starts in the negatives (thanks to two players... shrug)
easier to compare the two elos

show size of population for each histogram
use same histogram binning for the solo and team plots

screenshots of before/after:

and with the marker:

Also.... if you're curious.. (as i was)… Here's an
animation of the team elo over the past 30 days
That's 15 days on either side of the change.
Turns out it's… not that interesting.  🙃


Add grafana dashboards for initial build/provision upon component cre…

…ation, simple builds, PaC provisions and unprovisions and triggers
STONEBLD-1773
STONEBLD-1774

/retest

How should we review this? Which parts of the generated config are relevant? Or how could we deploy it and view it ourselves?

How should we review this? Which parts of the generated config are relevant? Or how could we deploy it and view it ourselves?

in diff you can see that it shows 1 file as moved and changed, so I've used "template" from existing dashboards and changed just metric name & those descriptions

How should we review this? Which parts of the generated config are relevant? Or how could we deploy it and view it ourselves?

in diff you can see that it shows 1 file as moved and changed (so there you can see actual changes I've made basically for all of them), so I've used "template" from existing dashboards and changed just metric name & those descriptions

That helps :+1: So all the dashboards are pretty much copy-pastes of each other, just for different metrics?
Still though, if I wanted to deploy it and check it on my cluster, how would I do it? I probably know how to deploy it at least (https://redhat-appstudio.github.io/infra-deployments/docs/development/deployment.html#deploying-your-versions-of-crds) but not sure where to see the dashboard

in the cluster find namespace : appstudio-grafana
there check route, and there is url to grafana and there you can see dashboards

/test build-service-e2e

Deployed on my cluster. The dashboards are shown, however after submitting a few builds, both simple and PaC builds, number is still 0 in the dashboard by waiting for a while after the submission. "No data" is shown inside initial build dashboard. Any additional configuration should be done after bootstrapping the cluster?

Deployed on my cluster. The dashboards are shown, however after submitting a few builds, both simple and PaC builds, number is still 0 in the dashboard by waiting for a while after the submission. "No data" is shown inside initial build dashboard. Any additional configuration should be done after bootstrapping the cluster?

I have no idea
I was testing those metrics which we are sending to prometheus in dev cluster and those extended names worked
not sure what has to be done for dashboards


wxGlade event_handlers block not found, when adding a class

This is just a minor issue:
When setting wxGlade to keep user code and adding a simple
class Test():
    pass

after the # end wxGlade, where user code should go (see example below), this warning will be produced and the generated code contains a line like
<15890495231133762465125wxGlade event_handlers ClassName># end of class AboutDialog
This can easily be fixed: when moving the class to a separate file and importing it, wxGlade works fine again.
[...]
# begin wxGlade: extracode
# end wxGlade

class MyFrame(wx.Frame):
    def __init__(self, *args, **kwds):
        # begin wxGlade: MyFrame.__init__
        kwds["style"] = kwds.get("style", 0) | wx.DEFAULT_FRAME_STYLE
        wx.Frame.__init__(self, *args, **kwds)
        self.SetSize((400, 300))
        self.SetTitle("frame")

        self.panel_1 = wx.Panel(self, wx.ID_ANY)

        sizer_1 = wx.BoxSizer(wx.VERTICAL)

        sizer_1.Add((0, 0), 0, 0, 0)

        self.panel_1.SetSizer(sizer_1)

        self.Layout()

        self.Bind(wx.EVT_CLOSE, self.on_close, self)
        # end wxGlade
        class Test():
            pass

    def on_close(self, event):  # wxGlade: MyFrame.<event_handler>
        print("Event handler 'on_close' not implemented!")
        event.Skip()

# end of class MyFrame

Thanks for reporting. This should be fixed now. The regex did match leading spaces, which was not necessary.

Thanks, i can conform this fixed the issue.

Release 0.9.6 tagged / draft available


Formatting lost for specific object

I've had this issue since always in one of my projects. The newlines and indentations get lost when using spew on a certain object type. All other object both before and after dumping this one, are printed just fine.
This is the type definition and all the subtypes. It can be found in the wild here: https://github.com/ethereum/go-ethereum/blob/master/core/types/receipt.go#L46
type Receipt struct {
	PostState         []byte
	Status            uint
	CumulativeGasUsed *big.Int
	Bloom             Bloom
	Logs              []*Log
	TxHash            common.Hash
	ContractAddress   common.Address
	GasUsed           *big.Int
}

type Hash [HashLength]byte
type Address [AddressLength]byte
type Bloom [BloomByteLength]byte

type Log struct {
	Address     common.Address
	Topics      []common.Hash
	Data        []byte
	BlockNumber uint64
	TxHash      common.Hash
	TxIndex     uint
	BlockHash   common.Hash
	Index       uint
	Removed     bool
}

It looks like some of the values are breaking for formatting.
Here's an example:
(spew.ConfigState) {
 Indent: (string) (len=1) " ",
 MaxDepth: (int) 0,
 DisableMethods: (bool) false,
 DisablePointerMethods: (bool) false,
 DisablePointerAddresses: (bool) false,
 DisableCapacities: (bool) false,
 ContinueOnMethod: (bool) false,
 SortKeys: (bool) false,
 SpewKeys: (bool) false
}
(*types.Receipt)(0xc420001c80)(receipt{status=0 cgas=3500000 bloom=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 logs=[]})
(spew.ConfigState) {
 Indent: (string) (len=1) " ",
 MaxDepth: (int) 0,
 DisableMethods: (bool) false,
 DisablePointerMethods: (bool) false,
 DisablePointerAddresses: (bool) false,
 DisableCapacities: (bool) false,
 ContinueOnMethod: (bool) false,
 SortKeys: (bool) false,
 SpewKeys: (bool) false
}

Whoops, just remember that I had this issue a year ago and that it was the String() method that looks a lot like spew output.


Also using svd method?

From the code, VGG16 acceleration factor can be 3X-4X, not only by using channel-pruning method but also used svd ?  Is it true?
And What about the effect of ssd mobilenet ?

It use SVD for spatial decomposition and channel decomposition, please refer to paper for details.
haven't studied.

Oh,the paper hasn't described the svd steps, thanks a lot.

details are in:

spatial decomposition (Speeding up Convolutional Neural Networks with Low Rank Expansions)
channel decomposition (Accelerating Very Deep Convolutional Networks for Classification and Detection)


pre-upgrade and post-upgrade jobs run every time

helm version: version.BuildInfo{Version:"v3.12.1", GitCommit:"f32a527a060157990e2aa86bf45010dfb3cc8b8d", GitTreeState:"clean", GoVersion:"go1.20.4"}
I have an umbrella chart that includes some subcharts.
One subchart contains a job that has 'pre-install,pre-upgrade' hook.
The other two contain 'post-install,post-upgrade' hooks.
I found that these jobs are triggered every time I run helm upgrade.
I also tried just run helm upgrade with the exact same version of umbrella chart and those jobs still get triggered, while none of my deployments change.
Similar to this one https://github.com/helm/helm/issues/8361

You can use --no-hooks flag to disable hooks when install or upgrade.


docker pull error, "You have reached your pull rate limit"

Is there any possibility to share the Dockerfile?
I was trying to use the docker since it's difficult to build the apex library. Whereas when I used docker pull like the below,
"docker pull cogview/cuda111_torch181_deepspeed040"
I got the following error message:
"Using default tag: latest
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"
Therefore, an original Dockerfile may be better.

Hi, the dockerfile is already under env/, (although I am not 100% sure if it is the correct version).

Hi, the dockerfile is already under env/, (although I am not 100% sure if it is the correct version).

Thanks for the dockerfile! I change some components to make it suitable for my V100 GPU, and finally it works. The main changes are as follows:
cuda 10.1
cupy-cuda101
TORCH_CUDA_ARCH_LIST="6.0 6.1 7.0+PTX 7.5+PTX"
torch-1.7.1+cu101
torchvision-0.8.2+cu101


Restrict dragging outside of parent element

Is it somehow possible to
prevent grid items to go outside of container div?

@rin4ik look at this sample React-Grid-Layout Demo 16 - Bounded

@serkononenko thanks

Updated Link: https://react-grid-layout.github.io/react-grid-layout/examples/16-bounded.html


civil Engineering field

add civil engineering field

all done closing the setup this civil and remove conflict

resolve this issue


Problem running cbf_pcasl_robust_batch.R: aslPerfusion in ANTsR

I have been trying to run NeuroBattery (https://github.com/jeffduda/NeuroBattery), and I kept getting an error when it tried to run cbf_pcasl_robust_batch.R. Specifically, when it got to line 12:
pcasl.processing <- aslPerfusion(pasl, moreaccurate=FALSE, interpolation="linear" )
it complains that "interpolation="linear" " is an unused argument. When checking the help for aslPerfusion in R, there is indeed no "interpolation" argument. Removing the argument makes it so the script will run, but I'm not sure if that's appropriate or not.
The same is true for cbf_pasl_robust_batch.R (it has interpolation="sinc").

ok - thanks for the information.
note:  antsASLProcessing.R
is the one to use ...

@stnava Could you explain how to solve this problem, because I still get the error.
This is my call stack:

./run_test.sh
./process_modalities.sh
antsbin/bin/antsNeuroimagingBattery -input-directory ../data/input/PEDS012/20131101/ --output-directory ../data/output/PEDS012/20131101/ --output-name PEDS012_20131101_ --anatomical ../data/output/PEDS012/20131101/PEDS012_20131101_BrainExtractionBrain.nii.gz --anatomical-mask ../data/output/PEDS012/20131101/PEDS012_20131101_BrainExtractionMask.nii.gz --template ../data/template/PTBP/PTBP_T1_Defaced.nii.gz --rsbold-flag BOLD/bold_fc_1.nii.gz/BOLD_ --pcasl-flag PCASL/pcasl_1.nii.gz/PCASL_ --dti-flag DTI/dt.nii.gz/DTI_ --template-transform-name PEDS012_20131101_SubjectToTemplate
antsbin/bin/cbf_pcasl_robust_batch.R ../data/input/PEDS012/20131101/PCASL/PEDS012_20131101_pcasl_1.nii.gz ../data/output/PEDS012/20131101/PCASL/PEDS012_20131101_PCASL_meancbf.nii.gz

My version of ANTsR is 0.3.2

antsASLPProcessing.R is the most up to date version of ants-based asl
processing.
i dont think @jeffduda maintains the neurobattery.
it is open source, so you could debug it yourself and then send him a pull
request with the fix.
brian
On Wed, Mar 22, 2017 at 11:58 AM, Nicolas Traut<EMAIL_ADDRESS>wrote:

@stnava https://github.com/stnava Could you explain how to solve this
problem, because I still get the error.
This is my call stack:

./run_test.sh
./process_modalities.sh
antsbin/bin/antsNeuroimagingBattery -input-directory
../data/input/PEDS012/20131101/ --output-directory
../data/output/PEDS012/20131101/ --output-name PEDS012_20131101_
--anatomical ../data/output/PEDS012/20131101/PEDS012_20131101_BrainExtractionBrain.nii.gz
--anatomical-mask ../data/output/PEDS012/20131101/PEDS012_20131101_BrainExtractionMask.nii.gz
--template ../data/template/PTBP/PTBP_T1_Defaced.nii.gz --rsbold-flag
BOLD/bold_fc_1.nii.gz/BOLD_ --pcasl-flag PCASL/pcasl_1.nii.gz/PCASL_
--dti-flag DTI/dt.nii.gz/DTI_ --template-transform-name PEDS012_20131101_
SubjectToTemplate
antsbin/bin/cbf_pcasl_robust_batch.R ../data/input/PEDS012/
20131101/PCASL/PEDS012_20131101_pcasl_1.nii.gz ../data/output/PEDS012/
20131101/PCASL/PEDS012_20131101_PCASL_meancbf.nii.gz

My version of ANTsR is 0.3.2
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stnava/ANTs/issues/224#issuecomment-288446611, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AATyfr4rLhtClCnD8j3Yu3OPCeFRn9cXks5roUUXgaJpZM4F22np
.

here is what you can do:

clone neurobattery

 git clone https://github.com/jeffduda/NeuroBattery.git

get data as described in the README.txt within input and template directories

 wget http://files.figshare.com/1701182/PEDS012_20131101.zip
wget http://files.figshare.com/1510696/PTBP.zip

process the t1.  in the scripts directory, call

 bash process_t1.sh

call antsASLProcessing.R

antsASLProcessing.R \
    -s  ../data/input/PEDS012/20131101/PCASL/PEDS012_20131101_pcasl_1.nii.gz \
    -o ./XXXX \
	  -a ../data/output/PEDS012/20131101/PEDS012_20131101_ \
	  -l  ../data/template/PTBP/Labels/PTBP_T1_AAL.nii.gz \
    -t   ../data/template/PTBP/PTBP_T1_BrainCerebellum.nii.gz \
		-x 6.0 \
	  -m "regression" \
	  -d "CompCor" \
	  -r 0.95 \
	  -n 20 \
	  -e  0.8

the script is rather out of date and not all options will actually work.

i will probably edit the above response or add more as needed

my advice is for you to contribute any relevant fix by making a pull
request.
On Thu, Mar 23, 2017 at 10:54 AM Nicolas Traut<EMAIL_ADDRESS>wrote:

So your advice is to give up with ./run_test.sh?
The solution of @jashubbard https://github.com/jashubbard seemed to
solve this issue (I previously missed to install the R package robust),
but it seems effectively that there are other issues that prevent
NeuroBattery to run until the end.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/stnava/ANTs/issues/224#issuecomment-288744926, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AATyfqnt74lVLHH0lUnlsE5Bs9tzeddeks5rooeygaJpZM4F22np
.

--
brian

I'll take a stab at updating NB but it may take some time as it hasn't
really been touched in 3 years and obviously the reference code has changed
significantly in that time.
-jeff
On Thu, Mar 23, 2017 at 10:56 AM stnava<EMAIL_ADDRESS>wrote:

my advice is for you to contribute any relevant fix by making a pull
request.
On Thu, Mar 23, 2017 at 10:54 AM Nicolas Traut<EMAIL_ADDRESS>wrote:

So your advice is to give up with ./run_test.sh?
The solution of @jashubbard https://github.com/jashubbard seemed to
solve this issue (I previously missed to install the R package robust),
but it seems effectively that there are other issues that prevent
NeuroBattery to run until the end.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/stnava/ANTs/issues/224#issuecomment-288744926, or
mute
the thread
<
https://github.com/notifications/unsubscribe-auth/AATyfqnt74lVLHH0lUnlsE5Bs9tzeddeks5rooeygaJpZM4F22np
.

--
brian
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stnava/ANTs/issues/224#issuecomment-288745700, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAfWiFJQbRIXE7UBB5oum4Htknhh9DMmks5roog5gaJpZM4F22np
.

OK, I will make a pull request for the issues I managed to fix and I will submit a new issue that I identified.

Updating to the latest ANTs and ANTsR should fix this issue. Give it a try
and let me know.
On Thu, Mar 23, 2017 at 11:28 AM Nicolas Traut<EMAIL_ADDRESS>wrote:

OK, I will make a pull request for the issues I managed to fix and I will
submit a new issue that I identified.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stnava/ANTs/issues/224#issuecomment-288756425, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAfWiNuIW9IGQ5ZRhQYxIe461wYa55qqks5roo-igaJpZM4F22np
.


BlackBars On Load

I turn on TCC, and get black-bars where my UI settings are, this stays in game and out

You probably disabled transparency for some windows.


add a mode to present participants from list

Would you please specify this issue? What exactly do you mean with "present"?

A single view to each participants where the user is able to step through the list of participants.