Search performance issues

(Issue description edited by @loiclec )
Some search queries are problematic because they use too much RAM or CPU time. Furthermore, we currently don't have any way to limit the number of simultaneous search queries being executed or to limit the time they are allowed to take. This can affect the stability of a Meilisearch instance. For v1, we'd like to resolve such issues in order to:

most importantly: prevent Meilisearch from exceeding its allocated memory budget (and therefore getting shut down unexpectedly)
ensure that search queries take a reasonable amount of time to execute

This is a meta-issue listing the potential areas of improvements. Note that there already exists an internal-only document about these issues here.

[ ] https://github.com/meilisearch/meilisearch/issues/3126
[x] https://github.com/meilisearch/meilisearch/issues/3115
[x] https://github.com/meilisearch/meilisearch/issues/3116
[x] https://github.com/meilisearch/meilisearch/issues/3118
[ ] https://github.com/meilisearch/meilisearch/issues/3124
[ ] https://github.com/meilisearch/meilisearch/issues/3123
[x] https://github.com/meilisearch/meilisearch/issues/3125
[ ] https://github.com/meilisearch/meilisearch/issues/3222

Current related issues

https://github.com/meilisearch/meilisearch/issues/2966 (expected to be partly solved through a combination of the proposed changes)
https://github.com/meilisearch/product/discussions/553 (expected to be only partly solved through a combination of the proposed changes)
Crash of the "landing page" search bar observed on meilisearch.com a while ago (expected to be solved through the MatchingWords issue)
https://github.com/meilisearch/meilisearch/issues/2202 (not expected to be solved unless the query tree is refactored)

An update on the progress so far:
First, there is the issue of rate-limiting + search request timeout. This is probably the most important issue if we want to make meilisearch reliable under pressure and avoid being shut down due to excessive memory consumption:  https://github.com/meilisearch/meilisearch/issues/3126 . Note the last comment by kero, which gives a way forward to tackle the issue. However, this is not something that I am going to work on. Maybe @dureuill or @irevoire will have time to think about it/implement something?
Then:

We’ve removed a huge source of RAM consumption with a change in the data structure used by the search result highlighting logic: https://github.com/meilisearch/milli/pull/708 This change also protects meilisearch against certain malicious search queries that could consume gigabytes of memory.

Then, a few other PRs have been opened to address some worst-case scenarios leading to excessively long search times (measured in seconds or even minutes if unlucky):

https://github.com/meilisearch/milli/pull/709 reduces the time of some queries with many (8-10) common words from multiple seconds to milliseconds
https://github.com/meilisearch/milli/pull/732 changes the way we interpret multi-word synonyms (they are now treated as phrases: nyc -> "New York City" instead of loose words nyc -> New York City ) which protects meilisearch against queries containing many of these synonyms.
https://github.com/meilisearch/milli/pull/733 avoids iterating over all the words starting with a certain prefix (which could be hundreds/thousands of words) inside the implementation of the proximity ranking rule. This change slightly degrades search results in some rare cases.

I have also started improving the search benchmarks such that (1) we can more reliably detect improvements/regressions in some search algorithms and (2) we benchmark more complex search queries that take a long time to execute. This is the draft PR: https://github.com/meilisearch/milli/pull/742 It will allow us to describe the impact of the search performance improvements more accurately.

There are a few other perf optimisations to be written, which I’ll try to implement this week and the next:

https://github.com/meilisearch/meilisearch/issues/3222  affects the behaviour of the attribute  ranking rule and will also have a positive impact on the relevancy and the disk size of the index
https://github.com/meilisearch/meilisearch/issues/3124 should be an easy win but the impact on performance is a bit of a mystery to me
https://github.com/meilisearch/meilisearch/issues/3123 avoids precisely sorting documents when a search offset is given
That’s it for now :)

Issue move to Milestones v1.1 since not everything has been done (it was expected, some issues require huge implementation and refactoring)
This issue (https://github.com/meilisearch/meilisearch/issues/2754) needs to be completed to ensure the good making of this current one. See this comment

Closing this issue in favour of https://github.com/meilisearch/meilisearch/issues/3547 , which supersedes it.


Use kustomize for pachctl deploy

Now that kustomize is a part of kubectl standardize on one set of kubernetes manifests and then use overlays for each cloud provider/local. This should simplify our deployment code and ensure a consistency across all environments

Some further thoughts to this / Potential re-architecture of pachctl deploy
Modelled after kubeflow deployment process
Release manifests/overlays as deployment artifacts separately
pachctl deploy creates a .manifests folder and downloads matching pach version of manifest/overlay
pachctl deploy google runs kustomize behind the scenes, applying downloaded google overlay to downloaded manifest
Could also allow for manual control / passing of urls to manifests
Thoughts / considerations:

Operator downloads in an airgapped environment (maybe operator has cache of last x versions)
Use of pachctl in an airgapped environment

By having templates live as bare yaml in the project, we unlock the use of things like skaffold, which significantly shorten the developer working loop

I am 100% behind this, as long as we leave room for local customizations of the customizations, which I believe kustomize supports.
Common ones are the customizations applied for OpenShift deployments, including environrment variables and service definitions for pachd.

WIP Comment
Dug into this a little more, and working through the issues with converting pachctl deploy
Safe to deprecate?

[ ] dashboard-only

Handled by kustomize natively

[ ] namespace

Global Flags:

[ ] Create pach context
--create-context --dry-run        Create a context, even with --dry-run.
[ ] Custom Variables

[ ] block-cache-size
[ ] cluster-deployment-id
[ ] context string
--dash-image string
--dynamic-etcd-nodes int
--etcd-cpu-request string
--etcd-memory-request string
--etcd-storage-class string       If set, the name of an existing StorageClass to use for etcd storage. Ignored if --static-etcd-volume is set.
--expose-object-api               If set, instruct pachd to serve its object/block API on its public port (not safe with auth enabled, do not set in production).
--image-pull-secret string
--local-roles                     Use namespace-local roles instead of cluster roles. Ignored if --no-rbac is set.
--log-level string
--new-storage-layer               (feature flag) Do not set, used for testing.
--no-color                        Turn off colors.
--no-dashboard                    Don't deploy the Pachyderm UI alongside Pachyderm (experimental).
--no-expose-docker-socket         Don't expose the Docker socket to worker containers. This limits the privileges of workers which prevents them from automatically setting the container's working dir and user.
--no-guaranteed                   Don't use guaranteed QoS for etcd and pachd deployments. Turning this on (turning guaranteed QoS off) can lead to more stable local clusters (such as on Minikube), it should normally be used for production clusters.
--no-rbac                         Don't deploy RBAC roles for Pachyderm. (for k8s versions prior to 1.8)
-o, --output string                   Output format. One of: json|yaml (default "json")
--pachd-cpu-request string        (rarely set) The size of Pachd's CPU request, which we give to Kubernetes. Size is in cores (with partial cores allowed and encouraged).
--pachd-memory-request string     (rarely set) The size of PachD's memory request in addition to its block cache (set via --block-cache-size). Size is in bytes, with SI suffixes (M, K, G, Mi, Ki, Gi, etc).
--registry string                 The registry to pull images from.
--require-critical-servers-only   Only require the critical Pachd servers to startup and run without errors.
--shards int                      (rarely set) The maximum number of pachd nodes allowed in the cluster; increasing this number blindly can result in degraded performance. (default 16)
--static-etcd-volume string       Deploy etcd as a ReplicationController with one pod.  The pod uses the given persistent volume.
--tls string                      string of the form "," of the signed TLS certificate and private key that Pachd should use for TLS authentication (enables TLS-encrypted communication with Pachd)
--upload-concurrency-limit int    The maximum number of concurrent object storage uploads per Pachd instance. (default 100)

Installation environment

GCP
AWS
Microsoft

Examples

Storage Class
Annotations (iam annotations)

Storage Backend

S3
GCP Buckets
Microsoft blob storage
Minio
Local / Hostpath

Local PVs instead of hostpaths for kubernetes clusters that support it would be really helpful in preventing weirdness with dev deployments, where some stuff sticks around even when undeploying everything 👍

It seems like kustomize wants to only cover the overlay stuff and leave variables to pre-processing/ other tools kubernetes-sigs/kustomize#388
We might want to look to kfctl / kfdef for inspiration here


DocsArchive-<IP_ADDRESS>Auto archive content

Auto archive content from https://github.com/OPS-E2E-PPE/docs-archive-test

Docs Build status updates of commit fa9b399:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

archvieTargetFolder/content/someContent.md
:bulb:Suggestion

Details

.openpublishing.redirection.json
:white_check_mark:Succeeded

archvieTargetFolder/TOC.yml
:white_check_mark:Succeeded

archvieTargetFolder/content/sql-database-engine.png
:white_check_mark:Succeeded

archvieTargetFolder/docfx.json
:white_check_mark:Succeeded

archvieTargetFolder/content/someContent.md

[Suggestion] Missing attribute: author. Add the current author's GitHub ID. NOTE: This Suggestion will be elevated to a Warning on 9/1/2019.

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Add tests for byteArrayFromAscii

See comment https://github.com/Plutonomicon/cardano-browser-tx/pull/186#discussion_r828617085

The function is trivial enough to not be tested


Added a helpful message to the list command so that people know what …

…list is showing

Fixes #101


Version 1.3.2

Background
Add new icon HeavyPlus and update dependency.
💡 Feature 1

NG-2007 Add HeavyPlus icon

🛠 Fixes

Bump @types/node from 16.18.11 to 16.18.12

Let's take #308 into this release still✌🏻


Package URl
pkg:maven/org.apache.jackrabbit:jackrabbit-webdav:1.5.0
CPE
cpe:2.3:a:apache:jackrabbit:1.5.0:::::::*
CVE
CVE-2009-0026
ODC Integration
No response
ODC Version
6.5.3
Description
CVE-2009-0026 is wrongly reported for jackrabbit-webdav:1.5.0. It should only be reported only for Jackrabbit server component, not webdav component.

While it is a false-positive for this component of the Apache JackRabbit framework we do not micro-manage the vulnerabilities by framework component if the libraries are released as a whole and as such receive a single CPE in NIST NVD database.


Run test suite on CI

We could create a PR to simply run the tests on the CI machinery, thus testing all target platforms.
For reference, we might need to generate a fake config.h as suggested in https://github.com/Amber-MD/cmake-buildscripts/issues/85#issuecomment-338851577. We can benefit from CB env variables to parametrize the file:
INSTALLTYPE=serial
AMBER_SOURCE=/Users/haichit/amber_git/amber
AMBER_PREFIX=/Users/haichit/amber_git/cmake/tmp
BINDIR=/Users/haichit/amber_git/cmake/tmp/bin
DATDIR=/Users/haichit/amber_git/cmake/tmp/dat
LIBDIR=/Users/haichit/amber_git/cmake/tmp/lib
INCDIR=/Users/haichit/amber_git/cmake/tmp/include
PYTHON=python
SKIP_PYTHON=no
SHARED_SUFFIX=.dylib

I'll leave this here as a reminder. First we need osx to build :muscle:

PR was merged after applying all patches needed (and possible).
The test machinery is there if we ever need it. Instructions to readd them are provided at /recipe/.run_tests/README.md.


There is a new tag of ansible-conjur-host-identity to announce deprecation of standalone project

This project will be migrated to the ansible-conjur-collection, and will subsequently be deprecated.
AC:

[ ] Add a tag to the Ansible role project to note that it has been deprecated and has moved to the new Conjur Collection

From the conversation that started in #30:
@BradleyBoutcher proposed:

We should take the following steps to deprecate this project:

Complete migration of Role to the Collection (see #30)
Tag a release of Collection (ansible-conjur-collection#35)
Migrate docs https://github.com/cyberark/ansible-conjur-collection/issues/36
Verify functionality of role in collection and document migration steps
Merge PR #43 in conjur-host-identity
Tag a release of the newly-deprecated conjur-host-identity (1.0.0, as this is a "breaking" change) (ansible-conjur-host-identity#42)

Add deprecated tag to the conjur-host-identity page in Ansible Galaxy

I asked for PO clarification of the deprecation process, and @alexkalish replied:

Users should be clearly informed of the following:

What exactly is being deprecated?  Be detailed.
How long is the deprecation period?  Said another way, when will the feature/component be removed?
What is the alternative path to meet the same outcome?  What do we advise?

Readme is probably fine, assuming that is primary place where the component is consumed.

@BradleyBoutcher here is what I would suggest we use as an approach, and completing these tasks would be the acceptance criteria for this card:
In this repository:

Add a changelog entry that announces deprecating this role for use with Ansible 2.9+ in favor of using the role that's part of the Ansible collection (note: this means we can't tag in this repo until the role is available in a collection release). TBD whether we can also announce deprecating support for EOL versions of Ansible < 2.8.
Move the Requirements section higher up in the README, and clearly indicate that it works with Ansible >= <IP_ADDRESS>, < 2.9 (TBD whether this should just be Ansible 2.8)
Add a link to the README in the Requirements section that points to the collection if you're using Ansible 2.9+
Update any metadata to be clear about the Ansible versions this role supports, and to ensure the link to the collection will be clear in Galaxy if using Ansible 2.9+.
Once these changes are in, tag the project with a patch release to announce the deprecation, and ensure the Galaxy listing for this role is clear about the compatible Ansible versions for this iteration of the role (e.g. Ansible >= <IP_ADDRESS>, < 2.9 - TBD whether it would just be Ansible 2.8)

Open question:

Can we also deprecate support for Ansible < 2.8 and update this role to indicate it's only supported for Ansible 2.8, and support will be maintained until Ansible 2.8 is unmaintained / EOL here.

We decided to maintain support for this project while Ansible 2.8 is still supported; closing this issue for now.


[SPARK-47200][SS] Make Foreach batch sink user function error handling backward compatible

What changes were proposed in this pull request?
I checked in a previous PR (https://github.com/apache/spark/pull/45299), that handles and classifies exceptions thrown in user provided functions for foreach batch sink. This change is to make it backward compatible in order not to break current users, since users may be depending on getting the user code error from the StreamingQueryException.cause instead of StreamingQueryException.cause.cause
Why are the changes needed?
To prevent breaking existing usage pattern.
Does this PR introduce any user-facing change?
Yes, better error message with error class for ForeachBatchSink user function failures.
How was this patch tested?
updated existing tests
Was this patch authored or co-authored using generative AI tooling?
No

cc @HeartSaVioR

GA failure only happened in Spark connect, which is not related.

Thanks! Merging to master.

(Btw, I missed the JIRA ticket is reused - best practice for this case is adding [FOLLOWUP] explicitly in the title. Just FYI.)


verify.vs.igrap.R crashes on edgelist read

awesomer:release-test disa [disa-dev] $ Rscript verify.vs.igraph.R
Loading required package: igraph
Loading required package: methods
Attaching package: ‘igraph’
The following objects are masked from ‘package:stats’:
decompose, spectrum

The following object is masked from ‘package:base’:
union

Loading required package: Rcpp
conf file FlashGraphR.conf doesn't exist.
init FlashGraphR: RAID config file doesn't exist
Run FlashGraphR in standalone mode
Loading required package: MASS
Loading required package: mclust
Package 'mclust' version 5.0.2
Type 'citation("mclust")' for citing this R package in publications.
[1] "The ARI of R kms is 0.00111621632102036"
[1] "The ARI of fg is 0.184083662227963"
[1] "The ARI of R.kms =  0.73023827228347"
[1] "The ARI of fg.kms = 0.716342112683848"
Error in .Call("R_igraph_read_graph_edgelist", file, as.numeric(n), as.logical(directed),  :
At rinterface.c:4793 : Cannot read edgelist, File operation error
Calls: read.graph -> read.graph.edgelist -> .Call ->
Execution halted

It seems the error is in igraph.
I guess it's not FlashGraph error.

disa should not run verify.vs.igraph directly. He should use run_realease_test.sh


opencv-python-headless depends on windows media feature pack after 4.7 update

Expected behaviour
On previous version (<IP_ADDRESS>) of opencv-python-headless, the package did not depend on windows media feature pack.
>> pyi-bindepend.exe "C:\Users\user\anaconda3\envs\cv246\Lib\site-packages\cv2\cv2.pyd"
C:\Users\user\anaconda3\envs\cv246\Lib\site-packages\cv2\cv2.pyd {'WSOCK32.dll', 'ole32.dll', 'KERNEL32.dll', 'OLEAUT32.dll', 'python3.dll'}

Actual behaviour
More recently (<IP_ADDRESS>) of opencv-python-headless, the package depends on windows media feature pack that is not available in windows docker image (windows/servercore:ltsc2019) .
>> pyi-bindepend.exe "C:\Users\user\anaconda3\envs\cv247\Lib\site-packages\cv2\cv2.pyd"
C:\Users\user\anaconda3\envs\cv247\Lib\site-packages\cv2\cv2.pyd {'MFReadWrite.dll', 'MF.dll', 'MFPlat.DLL', 'OLEAUT32.dll', 'KERNEL32.dll', 'SHLWAPI.dll', 'ole32.dll', 'WSOCK32.dll', 'python3.dll'}

Steps to reproduce
I have installed opencv-python-headless in a conda python 3.9 environment with

pip install opencv-python-headless

@matteoaletti Thanks for the issue report. I have not found explicit changes in setup.py or build scripts that manage Media Foundation dependency. Looks like it's some side effect of Python 3.6 -> 3.7 or Github actions environment update. Could you list other container specific issues here. I want to discuss the next steps with colleagues.

Thank you for your answer.
I am not sure that there are any other container specific issues. We are using a conda environment with python 3.9 inside a windows docker container (windows/servercore:ltsc2019) and when we use the opencv-python-headless <IP_ADDRESS> we can import cv2. Same environment with version <IP_ADDRESS> and our tests produces an error during the import of cv2:

ImportError: DLL load failed while importing cv2: The specified module could not be found

We tried to debug it ourselves and found out the new DLL dependencies mentioned above.
Everything works fine outside the container in a normal windows OS where those libraries are available.
For now, we simply pinned the version to the latest 4.6.

@asmorkalov Need to check G-API module and/or obsensor feature.

Any news to resolve this issue after 4.8 version? I encountered this issue when import cv2 in aws codebuild docker container image (windows server 2019) . I tried 4.6 4.7 4.8 but only 4.6 works.

Decided to switch to plugins for videoio in headless package on Windows. In case of good luck, it'll be part of 4.9.0.x.

@asmorkalov I see that a new release of opencv is out, was this switch included in 4.9.80 ?

Likely duplicated by https://github.com/opencv/opencv-python/issues/856 (although that issue is more popular)


Bump typescript to >=4.2.0

What kind of PR is this?:

/kind chore
What this PR does / why we need it:
Updated typescript to >=4.2.0 to use latest features causing compile IntelliSense errors.
Also fixed wrong whale-api-client imports due to 0.7.0 changes.

I think it's not yet supported yet by Expo.

I think it's not yet supported yet by Expo. Let's check if the expo upgrade will work

It's mainly due to this new typescript template feature, I will revert the change in https://github.com/DeFiCh/whale/pull/285

I think it's not yet supported yet by Expo. Let's check if the expo upgrade will work

To keep it on the safe side I will just revert typescript and fix import errors.


How to construct several neural networks for different node pairs in a model?

❓ Questions & Help
Hi, I have a question analogous to heterogeneous GNN.
For example, I have two kinds of nodes, which are denoted by A and B respectively. I want to construct three same neural network for A-A pairs, A-B pairs and B-B pairs but with different weights, or construct three different neural networks for these pairs.
Since the model will be trained with a batch which will consist of several graphs, I don't know how could I implement?

Great question! This is currently not that well-implemented, and I am working on a cleaner interface. PyG provides all tools to implement A-A, A-B and B-B message passing, but batch construction is currently quite cumbersome:
I suggest to inherit from the Data class and re-implement the __inc__ method, which describes how respective edge indices should be added together:
class MyHetoData(Data):
    def __inc__(self, key, value):
        if key == 'edge_index_A_A':
            return num_nodes_A
        elif key == 'edge_index_B_B':
            return num_nodes_B
        elif key == 'edge_index_A_B':
            return torch.tensor([num_nodesA, num_nodes_B])
        else:
            return 0

Thanks for replying.


CSRF on App not working

headers() only provides read-only request data and not from response data.
CSRF is attached to response headers.
Are you able to successfully get any csrf data from any server components in the app directory
just by using headers()?

Using headers() to retrieve the response header set by edge-csrf is working in the example here: https://github.com/amorey/edge-csrf/tree/main/example-ts-appdir
I just tried it with the latest version of Next.js (13.4.16) and it also worked.
Do you have an example of it not working with the app router?

Ok, thanks for the update. Happy to hear the issue wasn't with edge-csrf.


const String -> u32 ?

Heya, I came across your article at https://casualhacks.net/blog/2020-05-31/compiletime-processing/
Was wondering if you've looked into doing this for creating compile-time hashes from strings?
For example, I'd like to be able to do const U_POSITION:u32 = hash!("u_color"); and know that it's the same value as some other usage elsewhere. Speed and avoiding collisions are important, but it doesn't need to be cryptographically secure.

That article is a bit outdated by now, there's already a hash! macro. Note that if you're using it in a const context you can simply use the const fn hash instead of the macro. The macro simply forces compiletime evaluation where the compiler would default to invoking it at runtime.
However avoiding collisions is not possible with compiletime string hashing like this as it requires global understanding of all the other string hashes.
If avoiding collisions is 100% important you're probably looking at string interning, which is trivial in this case as the compiler will already do it for you if you use static objects instead of string literals, eg. pub static U_COLOR: &str = "u_color";. You can use U_COLOR.as_ptr() as usize as a globally unique id for the "u_color" string.


vendor bump to gogo/protobuf 1.3.2

CVE-2021-3121 ( https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3121 ) covers an issue in GoGo Protobuf. It is fixed with version 1.3.2.
Update the vendoring of gogo/protobuf appropriately.
go get -u github.com/gogo/protobuf ; go mod tidy ; go mod vendor and commit the changes.

@jqmichael would you be able to (find some one to) review this? Thanks!


Guid & Date Time Data type

Hi,
In Swagger Ui parameters show what are all parameter available in controller action.

Like this. here guid data type shows like string at same date time data type also shows undefined. If give (Guid?) and (DataTime?) it will show right data type. can you pls guid me what I need to do.
In function I need to give (Guid) this data type only not (Guid?). Kindly help me.

Help me regarding this issue?

I'm afraid I don't understand what your asking. Maybe you could provide some examples?

Closing due to inactivity - as far as I can see the generated docs for Guid's and DataTime (and the nullable variants) are correct. So, if you want to re-open, you'll have to provide more detail.

Maybe what's meant here is that if I define an API operation (web API method) that takes a Guid as parameter, the "parameters" documentation section shows it being "Data Type: string" (and not Guid) like illustrated in the picture. I'm seeing this too with Swashbuckle 5.2.1. Maybe this is intentional as Guid is pretty much a .NET specific concept and would not mean much for callers using other languages?

Having better support for the format parameters would be great in my opinion. In order to generate useful proxies, we need a way to distinguish Guids from regular strings. The swagger specification states (under "Data Types") that you can use custom strings for formats. They even list "uuid" in one of there examples. Although i prefer the term "guid" it seems like the more platform agnostic term is "uuid" so I'd be fine with that as well.


Snaplayout wont hide on windows 11

Describe the bug
My client user noticed that snaplayout wont hide after he released mouse

If you don't pull it up in the center, then the menu hangs there
, it's around the corner on a minimal scale

To Reproduce

If you don't pull it up in the center, then the menu hangs there
, it's around the corner on a minimal scale

Cant reproduce on my os
Edition	Windows 11 Home
Version	22H2
Installed on	28/09/2022
OS build	22621.382
Experience	Windows Feature Experience Pack 1000.22632.1000.0
Expected behavior
Snaplayout must hide after user released window
Screenshots

OS version
Not send yet
.NET version
6.0 Core
WPF-UI NuGet version
2.0.3
Additional context
No response

Probably OS problem?


File picker not working in iPad

I'm using Xamarin.Plugin.FilePicker in MvvmCross."https://github.com/Studyxnet/FilePicker-Plugin-for-Xamarin-and-Windows" for the browser "var pickedFile = await CrossFilePicker.Current.PickFile();" getting null value from pickerfile.
it's working fine in iPhone but not working in iPad. can you please anyone suggest, what's the problem.
please share any other dll.

Please have a look and try https://github.com/jfversluis/FilePicker-Plugin-for-Xamarin-and-Windows as this repo is no longer maintained


where is save file and how to output it

http://[2408:8220:a11:5580:35b8:16b2:7304:56f1]:10921/index.html

save file is located in the game www directory/saves
l.
what do you mean "output"?


document.activeElement becomes undefined after removing focused node

I discovered this while testing a React component: essentially if you have a list of children, focus one and then clear out the list (which uses some combination of DOM APIs to remove them), document.activeElement becomes undefined instead of pointing to document.body as expected in the W3 spec.

:+1: thanks @domenic


Avoid expensive fields() method call that creates temporary map.

@bratseth PR

Omg I missed that one!


Some problem about Data processing during training with Cityscapes

I encountered the following issues during training：

Traceback (most recent call last):
File "/data2/zixuan/DSRL-subpixel/train.py", line 313, in
main()
File "/data2/zixuan/DSRL-subpixel/train.py", line 306, in main
trainer.training(epoch)
File "/data2/zixuan/DSRL-subpixel/train.py", line 110, in training
output,output_sr,fea_seg,fea_sr = self.model(input_img)
File "/home/zixuan/anaconda3/envs/DSRL/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/zixuan/anaconda3/envs/DSRL/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 159, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/zixuan/anaconda3/envs/DSRL/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/data2/zixuan/DSRL-subpixel/modeling/deeplab.py", line 46, in forward
x = self.aspp(x)
File "/home/zixuan/anaconda3/envs/DSRL/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/data2/zixuan/DSRL-subpixel/modeling/aspp.py", line 70, in forward
x5 = self.global_avg_pool(x)
File "/home/zixuan/anaconda3/envs/DSRL/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/zixuan/anaconda3/envs/DSRL/lib/python3.6/site-packages/torch/nn/modules/container.py", line 117, in forward
input = module(input)
File "/home/zixuan/anaconda3/envs/DSRL/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/zixuan/anaconda3/envs/DSRL/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 136, in forward
self.weight, self.bias, bn_training, exponential_average_factor, self.eps)
File "/home/zixuan/anaconda3/envs/DSRL/lib/python3.6/site-packages/torch/nn/functional.py", line 2054, in batch_norm
_verify_batch_size(input.size())
File "/home/zixuan/anaconda3/envs/DSRL/lib/python3.6/site-packages/torch/nn/functional.py", line 2037, in _verify_batch_size
raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))
ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 256, 1, 1])

May I ask if there is an error downloading my data or if there is an issue with my model parameter settings? Can you please refer to your format for storing Cityscapes data?

Hello, I just found out in the experiment that this is due to my batchsize setting. The above error occurred because I set the batchsize to 1. When I set the batchsize to 2 or others, this error no longer appears. May I ask if the model can set the batchsize to 1?

Hello, I also want to conduct experiments on Cityscapes data recently, but I have encountered some issues with the dataset. Could you please share with me the dataset you used through email? Thank you very much.

Hello, I also want to conduct experiments on Cityscapes data recently, but I have encountered some issues with the dataset. Could you please share with me the dataset you used through email? Thank you very much.

You can find a dataset with rain and fog on Cityscapes' official website. Here is its official download page and specific download content.
https://www.cityscapes-dataset.com/downloads/


apinto config.yml support listen range ip port

Description
Now:
listen:   # node listen port
  - 8099

admin:    # openAPI request info
  scheme: http # listen scheme
  listen: 9400 # listen port
  ip: <IP_ADDRESS> # listen ip

just a case:
listen:   # node listen port
  - 8099
  - 2000-3000,3001,3003,4000-50000

more idea:
only allow  bind ports you list, if you set nothing, there won't be any limit
and the port will used when config in the dashboard

Thank for you repay. It will make the release process lighter in the future


Add eager_update option to dependencies

Current functionality remains as is by default
eager_update=True forces dependency to always update on all installs
eager_update=False disables dependency updates even when version spec is non-exact

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

13 of 13 (100.0%)  changed or added relevant lines in 3 files are covered.
66 unchanged lines in 11 files lost coverage.
Overall coverage increased (+1.1%) to 84.273%

Files with Coverage Reduction
New Missed Lines
%

src/main/python/pybuilder/plugins/python/_coverage_util.py
1
18.68%

src/main/python/pybuilder/plugins/python/flake8_plugin.py
1
85.94%

src/main/python/pybuilder/plugins/python/integrationtest_plugin.py
1
73.18%

src/main/python/pybuilder/plugins/python/remote_tools/init.py
1
27.4%

src/main/python/pybuilder/extern/init.py
2
86.42%

src/main/python/pybuilder/plugins/python/coverage_plugin.py
2
86.81%

src/main/python/pybuilder/remote/init.py
2
76.76%

src/main/python/pybuilder/reactor.py
4
93.49%

src/main/python/pybuilder/utils.py
6
83.88%

src/main/python/pybuilder/python_env.py
13
79.94%

Totals

Change from base Build<PHONE_NUMBER>:
1.1%

Covered Lines:
5403

Relevant Lines:
6353

💛  - Coveralls


Support for pytest >= 7.0.0

This package is not compatible with pytest >= 7.0.0. Would be great to have support for pytest >= 7.0.0. Sample error message below:
The conflict is caused by:
The user requested pytest==7.0.1
pytest-parametrize-cases 0.1.1 depends on pytest<7.0.0 and >=6.1.2

I'll update it soon; for the time being you can just copy the contents of case.py into your project and import it and it should work.

Excellent, thanks for the quick response! I'll try your advice tomorrow and come back to report if it worked.
I'm not sure exactly how I came across the package -- just stumbled upon it while searching for a better/clearer alternative for test parameterization. I think I found a post on stack overflow that led me here.
Anyway, I really like the approach you have implemented, so thanks for the package!

Okay I've put out a new release, can you see if it works properly when you reinstall it?

Thanks for the new release, it does appear to be working. Much appreciated!


Ensure that client connects on each metric collection

This ensures that even if router connection is broken, it will reconnect on next attempt. It is assumed that the cost of creating a new connection for each query is not significant since the collection does not happen frequently.
Fixes #1488

One further note here:
When debugging, I found that the output of python "print" statements didn't appear in the container log. Adding the "-u" param in the Dockerfile to disable output buffering helped:
CMD ["python", "-u", "/router-metrics.py"]

@calohmn Could you have another look?


Allow space application supporter to access specific isolation segments endpoints.

Issue
Allow space application supporter to access specific isolation segments endpoints.
Context
We are introducing a new role and we want to make sure it has the right access.
Expected result
A space application support should be able access the following endpoints:
GET /v3/isolation_segments/:guid
GET /v3/isolation_segments
GET /v3/isolation_segments/:guid/relationships/organizations
GET /v3/isolation_segments/:guid/relationships/spaces
Acceptance
A space application supporter would see the same info as a space developer assigned to the same space for these and only these isolation segments endpoints.
Documentation
When I browse to any of these endpoints on v3 docs I can see the Space Application Supporter role in the list of permitted roles with an indication that this role is not fully implemented and the permissions will be changing.

While working on GET /v3/isolation_segments/:guid, it turns out the docs are incorrect in terms of who is allowed access.  The docs say that an Org Billing Manager and Org Auditor cannot use that endpoint, but it turns out they can.  For now, we are not changing the code to match the docs, will update the docs instead.


CI: package_build: provide more memory to iot-agent builders

What does this PR do?
Provide more memory to iot-agent builders

Motivation
XZ compression in parallel can be quite greedy, and sometimes fails to allocate enough memory in the current configuration
For instance: https://gitlab.ddbuild.io/DataDog/datadog-agent/-/jobs/408995911

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

Reviewer's Checklist

[ ] If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
[ ] Use the major_change label if your change either has a major impact on the code base, is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.
[ ] A release note has been added or the changelog/no-changelog label has been applied.
[ ] Changed code has automated tests for its functionality.
[ ] Adequate QA/testing plan information is provided. Except if the qa/skip-qa label, with required either qa/done or qa/no-code-change labels, are applied.
[ ] At least one team/.. label has been applied, indicating the team(s) that should QA this change.
[ ] If applicable, docs team has been notified or an issue has been opened on the documentation repo.
[ ] If applicable, the need-change/operator and need-change/helm labels have been applied.
[ ] If applicable, the k8s/<min-version> label, indicating the lowest Kubernetes version compatible with this feature.
[ ] If applicable, the config template has been updated.

/merge


Send message to whatsapp server

urrently I do not know the structure of the payload that should be sent to the server, in the documentation they speak of 3 paramatros [description, attributes, content] but the model whatsapp_binary_writter that triggers an error and closes the connection

Duplicate of #69


Hack around bug in gcc-4.8.1 template processing.

This keeps the previous behavior for other compilers that will
throw warnings on a negative shift operation, but works around
the internal compiler error in GCC.
Fixes #855

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Rebased and pushed into master as d7f199b5d


Fix: Remove trailing comma

This pull request

[x] removes a trailing comma from composer.json

Follows #2.

Closing in favor of #6.


Rosters not working

I've imported a roster, but when players that are on that roster attempt to join via keyword whisper they're given the response that they're not on the nominated roster.

Thanks for reporting this. Does this happen with multiple people? Did you check that the spelling is exactly correct in your roster list? You can try adding the name of the player manually with the add button and typing it in, can you check whether the roster stays the same after adding (which it should because it detects that this player is already on the list)?
And I assume this is with the roster invite only setting activated?


Fix departed id counting

The departing vehicle IDs were being fed by the arrived vehicles values.

Pull Request Test Coverage Report for Build 4340

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 91.964%

Totals

Change from base Build 4295:
0.0%

Covered Lines:
9178

Relevant Lines:
9980

💛  - Coveralls


the pdt_locales/en_AU.py file uses en_A for the localID instead of en_AU

A very old mistake that should be fixed - will require a major version bump
found during work on PR #118

Why does this require a major version bump? In other words: In what situations should I expect things to break?
I'm trying to figure out if I should modify code in my project that depends on this one.

@MatthiasKauer In hindsight I would agree with you - it didn't need a major version bump but I had already rolled it into the v2.0 changes
to be honest the amount of bug fixes that have been made since then is more of a reason to upgrade

Thanks for clarifying.
I'll update for sure. I'm mainly trying to figure out where I should look for things to possibly break in my own code with that upgrade.

The CHANGES.txt log is very complete, but I realize may not have all of the info you need -- the vast majority of the changes are bug fixes and I don't think we made any changes to how methods are called or their return values.
thanks!

I'm coming from that changes file. It is indeed very good. The reason I'm asking is that it lists this issue as sole reason for the major version increase. When I come here, however, it didn't feel so major at first glance. Anyhow, sorry to bother you and thanks for the explanation!
On 25 March 2016 18:46:30 CET, Mike Taylor<EMAIL_ADDRESS>wrote:

The CHANGES.txt log is very complete, but I realize may not have all of
the info you need -- the vast majority of the changes are bug fixes and
I don't think we made any changes to how methods are called or their
return values.
thanks!

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/bear/parsedatetime/issues/120#issuecomment-201383321

glad to help and thanks for the suggestion - I will definitely be more aware of how the changes file reads to others going forward!


Add Sauce Labs

Does free-for-open-source also qualify?

Yes, it qualifies.
I already added Travis CI for example, or Zeit Now which are both free for open source and/or expose source of the app, which may be ok for side projects especially if you are already publicly hosting your code on GitHub

Which category would Sauce Labs fit into?

Maybe a new "App testing" category?


Truncate the source with ellipsis

For very long client names use ellipsis so the page doesn't go crazy.

Hi @crainte
Sorry for the delay. The CSS files are generated with Sass so you should edit the file css/uchiwa-default/_table.scss instead and generate the CSS files with the command grunt sass.
Thanks!

Closing for https://github.com/sensu/uchiwa-web/pull/37


[jaeger-v2] Storage backend integration tests

Requirement
With the Jaeger storage extension for Jaeger-v2 is going to have full support of Jaeger-v1's storage backends, some unit tests on every storage backends are not enough. We need to conduct end-to-end tests of OpenTelemetry Collector pipeline to the targeted database.
Problem
There are still no integration tests to test the actual stored traces to the database from V2 Jaeger storage extension.
Proposal
Fortunately, OpenTelemetry Collector already has a testbed framework to help us conducting the end-to-end tests.
Testbed is a controlled environment and tools for conducting end-to-end tests for the OpenTelemetry Collector, including reproducible short-term benchmarks, correctness tests, long-running stability tests and maximum load stress tests. However, we will only utilize the correctness tests from testbed, it generates and sends every combinatorial trace attributes and matches every single of them with the received traces from another end.
Architecture of the integration test
Here's the architecture we will use to test the OpenTelemetry Collector pipeline from end-to-end with the designated storage backends.

Testbed components:

LoadGenerator - encapsulates DataProvider and DataSender in order to generate and send data.

Golden DataProvider - generates traces from the "Golden" dataset generated using pairwise combinatorial testing techniques. Testbed example uses PICT to generate the test data, e.g. testdata.
OTLP Trace DataSender - with the generated traces from DataProvider, the DataSender sends traces to OTLP receiver in the collector instance.

Mockbackend - encapsulates DataReceiver and provides consume functionality.

DataReceiver - we will create a custom DataReceiver that will host a Jaeger storage extension to retrieve traces from the database by pulling them using our artificial Jaeger storage receiver https://github.com/jaegertracing/jaeger/pull/5242.
Consumer - consumer does not actually a thing in MockBackend but only to make the diagram intuitive, the traces received from our artificial receiver will be stored inside MockBackend.

Correctness Test Validator - checks if the traces received from MockBackend are all matches with the generated traces from DataProvider.

Plan
The execution of integration tests will be done incrementally one by one on every supported storage backends:

[x] GRPC with memory
[ ] Badger
[ ] Cassandra
[ ] Elasticsearch
[ ] Opensearch

Open questions
No response

@james-ryans as I was reviewing the PRs that follow from this issue, I am starting to have some concerns with this approach. Here is the set of requirements that I think we need to meet:

we need to exercise the full pipeline to write data externally and verify that it makes it to the storage
* (1b) we need to write data in different formats, not just OTLP
we then also need to exercise the querying API
we need to exercise archiving capability
we need to validate that the config files we're providing in cmd/jaeger are valid by doing an e2e smoke test
* (4b) in v1 we also had some docker-compose files that need to be tested
we need to generate code coverage for some parts of the code that do not get exercised in unit tests (usually related to initializing the storage drivers)
we need to provide a capability for external plugin providers (implementing gRPC Storage API, such as Quickwit or Postgress plugin) to also run e2e test for writing and querying, as a way of certifying compatibility with Jaeger

In the current state:

OTEL testbed solves only (1)
crossdock tests solve (1b) and partially (4b)
Our storage integration tests solve (2), (3) and (5), and I think (6)
Nothing solves (4)

I think we can solve all 6 requirements by building upon our existing integration tests rather than with OTEL testbed. Perhaps we can also find a way to utilize the testbed's data generation ability and incorporate it as a step in the overall integration, but on itself I don't see how it can solve all requirements.

we can keep integration tests operating as unit tests to address (2), (3), (5)
we can abstract how integration tests write and read span data, such that in the unit test mode they would call storage API as a library, but in e2e mode they will do the same via RPC requests. This can solve (1) and (4)

note 1: right now testbed-based config for gRPC is different (#5281) from the one in cmd/jaeger, but probably artificially, if we run the main config and test writes and reads via RPCs we don't need to separate Badger storage into another process, it can run in the all-in-one mode
note 2: when bootstrapping e2e tests, perhaps we can rely on docker-compose files instead of starting docker containers manually, this will help with (4b)

we can run unit test mode and e2e mode in the same workflow so that we don't have to start-up storage backends multiple times (expensive for ES/OS/Cassandra)
not sure about (6), my guess it should be able to reuse the grpc-storage tests
the crossdock tests have two parts: testing interoperability between SDKs and exercising the receipt of different formats of data produced by various legacy SDKs. We can retire the former part, but the latter is still useful since we cannot reproduce it locally without depending on deprecating SDKs. If the whole e2e test is running in docker-compose we can find a way to reuse this.

Achieving this will streamline our integration tests by converging onto a single framework, instead of using 3 different ones for bits and pieces. This is probably a large task, so I would like to find a path of incremental improvements that lead us to the overall goal. Let's give it some thought.

Yes, that is all correct. For instance, with ES, in the unit test model the test will instantiate es.SpanWriter and when it calls writer.WriteSpan() it's an in-process call to the es storage implementation that writes directly to ES. But in e2e mode, a different SpanWriter will be instantiated that executes an OTLP-RPC request to the running collector, where it will be accepted by the receiver and written to storage by the exporter.

unit test mode
flowchart LR
    Test -->|writeSpan| SpanWriter
    SpanWriter --> B(StorageBackend)
    Test -->|readSpan| SpanReader
    SpanReader --> B

    subgraph Integration Test Executable
        Test
        SpanWriter
        SpanReader
    end

e2e test mode
flowchart LR
    Test -->|writeSpan| SpanWriter
    SpanWriter --> RPC_client
    RPC_client --> Receiver
    Receiver --> Exporter
    Exporter --> B(StorageBackend)
    Test -->|readSpan| SpanReader
    SpanReader --> RPC_client2
    RPC_client2 --> jaeger_query
    jaeger_query --> B

    subgraph Integration Test Executable
        Test
        SpanWriter
        SpanReader
        RPC_client
        RPC_client2
    end

    subgraph jaeger-v2
        Receiver
        Exporter
        jaeger_query
    end

I have created an action plan to provide us with a clear, structured pathway so we can execute this in parallel. Some thoughts are welcome if my idea doesn't match with our vision.

Prototyping the new integration tests

Implement the unit test that exercise the querying API (2) and archives (3). Initializing storage driver codes (5) should automatically covered with this tests.
My thoughts of how this will be implemented is we need only to pass the config to the setup function that will starts the storage extension, within the setup we can retrieve the SpanWriter and SpanReader. Not sure, but probably we can reuse the StorageIntegration module.
Also I found that the archiving capability only tested on Elasticsearch storage.

Extend the unit test for the e2e test, but instead of starts only the storage extension, we use config file in cmd/jaeger to spawn the whole collector pipeline (4), then implement the SpanWriter and SpanReader to send span data through gRPC requests to receiver and from jaeger_query (1).

List of the storage backends that need to be tested:

memory
gRPC
badger
cassandra
elasticsearch
opensearch

Refactoring and an example for external plugin provider.

Refactor the unit test and e2e test to run in the same workflow so they use the same storage backend. Need to extra careful with the previous written data.
Refactor bootstrapping tests to rely on docker-compose files.
Add an example on how to test external plugin providers with our gRPC storage tests.

Add the crossdock tests.

With this, we can prototype the unit test and e2e test modes in parallel. But, after the unit test is merged, we need to refactor the e2e test to have a similar structure. Once the unit test and e2e test for one of the storage backends are merged, we can continue working on the other backends. After that, we can do refactoring and an example from plan 2 in parallel. The last one is to give some thoughts and find out how to test the interoperability between SDKs and exercise the receipt of different formats of data in crossdock fashion.

And I'll try to prototype the e2e test for the gRPC storage backend since @Pushkarm029 is working on the gRPC unit test.

@james-ryans a couple thoughts

there is a PR in progress to migrate archive test from ES to the main test suite #5207
my diagrams only show the extension of the existing /integration/ tests to work in e2e mode. Do you see the benefits of also using OTEL testbed in this setup?

there is a PR in progress to migrate archive test from ES to the main test suite Extracted Archive Test #5207

Ohh wow, nice.. I overlooked that this task exists. I'll take a look at it.

my diagrams only show the extension of the existing /integration/ tests to work in e2e mode. Do you see the benefits of also using OTEL testbed in this setup?

Some components of it might be useful but we can implement it on our own with ease if we want to, probably modifying it for our specific use case. I'm thinking that we should be able to use OTEL testbed collector (testbed/testbed/in_process_collector.go) to start the jaeger-v2.
Probably, also the OTEL testbed sender component to write the span data through RPC request. However, I still need to examine it to get a concrete picture. One concern is that the sender lacks the functionality to close the RPC connection.

One main difference to me is that our integration tests generate very specific traces and then query for them in very specific ways, to actually exercise the querying capabilities & permutations. But OTEL testbed just generates a random flood of data and only checks that it all gets there (not even that, as I believe it only checks the IDs). That was really my question - what is the value of such data source? It's not really fuzz-testing since the data is still hardcoded (just permutated for the load). I could see it potentially being useful for stress testing, but we don't do that today (would need dedicated HW, not GH runners).

The sender is just a wrapper for OTLP exporter that we are able to call the ConsumeTraces func with our specific provided traces and the remaining will be handled by the sender to do the RPC requests. Your understanding might be confused between the OTEL testbed data provider with the sender. The data provider component is the one that generates random traces and pushes them through the sender. With the sender alone we should be able to utilize it for our integration tests.

@yurishkuro with the new integration requirements, we don't need to test the collector pipeline with testbed as I proposed before anymore, is that right? If it is, we can just delete it.

I think so, but that was really my question to you - if we used the testbed, what additional aspects or behavior would it be testing?

Okay. It doesn't give any benefit anymore at this point, since all the test cases are already covered by the existing StorageIntegration. But we can use some parts of the components for ourselves to provide an easier setup for the new integration tests.

Copying from https://github.com/jaegertracing/jaeger/pull/5355#discussion_r1566018600 - let's add this to the README.
flowchart LR
    Receiver --> Processor
    Processor --> Exporter
    JaegerStorageExension -->|"(1) get storage"| Exporter
    Exporter -->|"(2) write trace"| Badger

    Badger_e2e_test -->|"(1) POST /purge"| HTTP_endpoint
    JaegerStorageExension -->|"(2) getStorage()"| HTTP_endpoint
    HTTP_endpoint -.->|"(3) storage.(*Badger).Purge()"| Badger

    subgraph Jaeger Collector
        Receiver
        Processor
        Exporter

        Badger
        BadgerCleanerExtension
        HTTP_endpoint
        subgraph JaegerStorageExension
            Badger
        end
        subgraph BadgerCleanerExtension
            HTTP_endpoint
        end
    end


Bump go-connections version.  Looks like I had previously vendored a …

…merged branch,
so this just makes the version number correct.
Signed-off-by: Ying Li<EMAIL_ADDRESS>

LGTM

LGTM


MSID: 92252 Version: 1 DOI: 2<IP_ADDRESS>1235

"msas": "Neuroscience"
"msid": "92252"
"preprintDoi": "10.1101/2<IP_ADDRESS>1235"
"Reviewed Preprint posted": "2023-12-12"
"Sent for peer review": "2023-09-26"
"Posted to bioRxiv": "2023-10-09" (link: "Go to bioRxiv": "https://www.biorxiv.org/content/10.1101/2<IP_ADDRESS>1235v1/")
[PLACE PDF URL HERE WHEN AVAILABLE] See step 7
Step 1. Inform bioRxiv
Who can help: @QueenKraken, @nlisgo, @scottaubrey

[x] Manuscript is in data hub index (https://data-hub-api.elifesciences.org/enhanced-preprints/docmaps/v2/by-publisher/elife/get-by-manuscript-id?manuscript_id=92252)

Step 2. Create preview of manuscript
Who can help: @fred-atherden, @nlisgo, @scottaubrey

[ ] Manuscript is available for preview (https://prod--epp.elifesciences.org/preview/92252/v1)

[ ] Trigger import for staging: https://prod--epp.elifesciences.org/import
[ ] Sync data folder with s3://prod-elife-epp-data/data
[ ] Trigger import for staging: https://staging--epp.elifesciences.org/import
[ ] Sync data folder with s3://staging-elife-epp-data/data
[ ] Pull request created to enhanced-preprint-data
[ ] Post link to pull request below

Pull request: [PLACE LINK TO PULL REQUEST HERE]
Detailed instructions: https://github.com/elifesciences/enhanced-preprints-data#add-a-manuscript
Step 3: Awaiting public reviews and QC
Who can help: Production team

[ ] QC passed or QC not needed

Who can help: Editorial team

[x] Public reviews available on sciety (https://sciety.org/articles/activity/10.1101/2<IP_ADDRESS>1235)
[x] Manuscript and editor details at the top of this issue (Supplied by Editorial team)
[x] Reviews returned by EPP - https://prod--epp.elifesciences.org/api/reviewed-preprints/92252/v1/reviews

Example
"msas": "Genetics and Genomics", "Neuroscience"
"msid": "84628"
"version": "1"
"preprintDoi": "10.1101/2<IP_ADDRESS>4241"
"articleType": "Reviewed Preprint"
"status": "Published from the original preprint after peer review and assessment by eLife."

"Reviewed Preprint posted": "2023-01-02"
"Sent for peer review": "2022-10-28"
"Posted to bioRxiv": "2022-11-21" (link: "Go to bioRxiv": "https://www.biorxiv.org/content/10.1101/2<IP_ADDRESS>4241v1")

Editors:
Reviewing Editor
Michael B Eisen
University of California, Berkeley, United States
Senior Editor
Michael B Eisen
University of California, Berkeley, United States

Step 4: Modify manuscripts.json (no PDF)

[ ] Reviewed preprint is published to EPP (https://elifesciences.org/reviewed-preprints/92252)

[ ] Pull request created to enhanced-preprint-client
[ ] Pull request switched to "Ready for review"
[ ] Post link to pull request below

[ ] Request that a doi be registered

[ ] Post link to #enhanced-preprint comment below

Pull request: [PLACE LINK TO PULL REQUEST HERE]
#enhanced-preprint comment thread: [PLACE LINK TO COMMENT HERE]

Instructions to modify manuscripts.json

Visit: https://github.com/elifesciences/enhanced-preprints-client/actions/workflows/publish-manuscript.yaml
Click: Run workflow
Complete the form and click "Run workflow"
A successful run should result in a new pull request at https://github.com/elifesciences/enhanced-preprints-client/pulls
Open the pull request and click the "Ready for review" button to trigger tests
Once the tests pass and you are happy with the changes the PR can be merged

Example pull request: https://github.com/elifesciences/enhanced-preprints-client/pull/334/files
Once the pull request is merged in it should be available a few minutes later.

Request that a doi
Post the following in #enhanced-preprint:
@Fred can you register a doi for https://elifesciences.org/reviewed-preprints/92252

Step 5: Awaiting search reindex

[ ] Reviewed preprint is avaliable on journal homepage (https://elifesciences.org)

The search reindex is triggered once an hour. We need the reviewed preprint to be indexed as the search application serves the journal homepage.

Additional info
If needed, the jenkins pipeline to reindex search can be triggered sooner.
https://alfred.elifesciences.org/job/process/job/process-reindex-reviewed-preprints/

Step 6: Published! Request PDF generation

[ ] Post the link to the PDF url at the top of the issue

[ ] Post link to #enhanced-preprint comment below

#enhanced-preprint comment thread: [PLACE LINK TO COMMENT HERE]
Post the following to the #enhanced-preprint on slack:
@Ryan Dix-Peek please can you generate a PDF for https://elifesciences.org/reviewed-preprints/92252

Step 7: Introduce PDF to data folder and git repo

[ ] PDF is avaliable at https://github.com/elifesciences/enhanced-preprints-data/raw/master/data/92252/v1/92252-v1.pdf

Detailed instructions: https://github.com/elifesciences/enhanced-preprints-data#add-a-pdf
Step 8: Add PDF url to manuscripts.json

[ ] Reviewed preprint PDF is available for download (https://prod--epp.elifesciences.org/reviewed-preprints/92252)

[ ] Pull request created to enhanced-preprint-client
[ ] Pull request switched to "Ready for review"
[ ] Post link to pull request below

[PLACE LINK TO PULL REQUEST HERE]

Instructions to add PDF url to manuscripts.json

Visit: https://github.com/elifesciences/enhanced-preprints-client/actions/workflows/add-pdf-url-to-manuscript.yaml
Click: Run workflow
Complete the form and click "Run workflow"
A successful run should result in a new pull request at https://github.com/elifesciences/enhanced-preprints-client/pulls
Open the pull request and click the "Ready for review" button to trigger tests
Once the tests pass and you are happy with the changes the PR can be merged

Example pull request: https://github.com/elifesciences/enhanced-preprints-client/pull/397/files
Once the pull request is merged in it should be available a few minutes later.

Step 9: Done!

[ ] Kettle is on!

Just to note that the authors are extremely keen to have their VOR out this year, and I've got that ready to send through. Is it okay to send the VOR along now, @fred-atherden, even if the RP is blocked?

Hi @acollings good to know.
Yes since the reviews have been posted and are available in the docmap, we're fine for the VOR to be exported even though the RP hasn't yet been published.


Dry up post_register method in UserController

In this PR:
Add separate classes for creating users(RegisterUserJob) and apps(AddAppJob). RegisterUserJob class can be used anywhere we want to create a user with an app likewise AddAppJob class.
By moving other components to their own classes ensures that they can be tested independently
and the post_register() method stays as clean as possible.
Validating request has been moved to its own class and type hinted in post_register() method.
All the changes have test coverage.

Nice one, UserControlller is getting leaner each time with your  PR 👍


DOI links

DOI links must be entered as https://doi.org/x in order to work on both individual publication pages and on the overall publication list. Some links need editing!

Fixed by Chris' edits to the Publications page


Convert Initial dataset into csv file

Our current dataset is in a md file. We plan on converting it into a csv file in order to have it in the correct file format.

Dataset is now in a csv format. Initial md file has been renamed to still provide notes.


Algorithm updates

While writing the docs, I noticed that we did our algorithm wrong.
This should be our reference: https://eigen-books.gitbook.io/eigen-trust/~/revisions/EzbPDwFJ1NWlnID22Uvc/4_algorithm
@duguorong009 Can you fix the failing test cases?

While writing the docs, I noticed that we did our algorithm wrong. This should be our reference: https://eigen-books.gitbook.io/eigen-trust/~/revisions/EzbPDwFJ1NWlnID22Uvc/4_algorithm
@duguorong009 Can you fix the failing test cases?

@lazovicff
I checked the doc and algorithm updates, and I understand the intention.
While checking the failing test cases, I found that the normalize computation does not match the original intention.
For example, [0, 10, 20, 10] becomes [0, 0.25, 0.5. 0.5] when we follow the doc.
But, real computation is done in finite field. So, [0, 10, 20, 10] becomes [0, 10 * 40^-1, 20 * 40 ^-1 , 10 * 40^-1].
Clearly, this result is not equal to decimal, and score computation is not equal.
Hence, after every iteration of computation, the sum would be different to previous sum.
As result, I think it is not correct to add the constraint sum_initial == sum_final here

For now we can comment out sum_initial == sum_final assert, and review & merge this


YAML validation

[x] Ensure relevant VS Code configuration files are available
[x] Enable YAML lint and casing validation

AC at:
https://app.zenhub.com/workspaces/the-outlaws-6193fe815d79fc0011e741b1/issues/gh/energinet-datahub/the-outlaws/966

Codecov Report

Merging #146 (40b0118) into main (a36ea9d) will decrease coverage by 22.67%.
The diff coverage is n/a.

:exclamation: Current head 40b0118 differs from pull request most recent head a13f7bd. Consider uploading reports for the commit a13f7bd to get more accurate results

@@             Coverage Diff             @@
##             main     #146       +/-   ##
===========================================
- Coverage   81.66%   59.00%   -22.67%
===========================================
  Files         326      357       +31
  Lines        5984     7425     +1441
  Branches      598        0      -598
===========================================
- Hits         4887     4381      -506
- Misses        886     2833     +1947
  Partials      211      211

see 241 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Update MD file on new branch 2017-37-15_09-12-51

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

split/updateTestCon.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Clicked glyph underlined

Clicked glyphs are underlined. Fix this so that it doesn't get the underline.

May be browser specific.


order=random not working

@Disservin is it just me or is order=random not working for opening book? it just plays it sequentially,
here is my prompt: fast-chess.exe -openings file=openingss.epd format=epd order=random -engine cmd=Stockfish-dev-20240413-c55ae376.exe name=Engine1 -engine cmd=Stockfish-dev-20240413-c55ae376.exe name=Engine2 -each tc=10+0.1 -rounds 4 -concurrency 4 -resign movecount=1 score=600 -pgnout file=fastchess.pgn

https://github.com/disservin/fast-chess/blob/master/src/matchmaking/book/opening_book.cpp seems you forgot to add shuffle()?

oh yeah indeed

while youre on it i might want to add another bug that i just found, seems if you set the number of rounds to be greater than the number of openings in the book, the test seems to end prematurely, maybe something to do with how the book openings are recycled idk


No callback triggered on ECONNREFUSED

Hi, I'm having troubles that the promise rejection callback is not executed when the connection fails, if I run with DEBUG=amqp* I get these repeated logs:
  amqp10:client connecting to: amqp://localhost +0ms
  amqp10:transport:net Connecting to localhost:5672 via straight-up sockets +5ms
  amqp10:connection Transitioning from DISCONNECTED to START due to connect +3ms
  amqp10:connection Error from socket: Error: connect ECONNREFUSED <IP_ADDRESS>:5672 +2ms
  amqp10:client connection error:  +1ms { Error: connect ECONNREFUSED <IP_ADDRESS>:5672
    at Object.exports._errnoException (util.js:1036:11)
    at exports._exceptionWithHostPort (util.js:1059:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1080:14)
  code: 'ECONNREFUSED',
  errno: 'ECONNREFUSED',
  syscall: 'connect',
  address: '<IP_ADDRESS>',
  port: 5672 }
  amqp10:client disconnected +9ms
  amqp10:connection Transitioning from START to DISCONNECTED due to error +1ms

I need to have a working error callback somewhere but it never seem to trigger. In desperation I added client.on('error', () => {}) without knowing if that even exists.
The code that I use are:
const AMQP = require('amqp10');

const AMQP_HOST = process.env.AMQP_HOST || 'localhost';

this.client = new AMQP.Client(AMQP.ActiveMQ);
this.client.connect(`amqp://${AMQP_HOST}`).then(() => {
  console.log('connected', `amqp://${AMQP_HOST}`);
  this.client.createReciever('log').then((reciever) => {
    console.log('reciever created');
    reciever.on('message', (message) => {
      const level = message.level === 'warning' ? 'warn' : message.level;

      if (console[level] instanceof Function) {
        if (message.package) {
          console[level](
            (new Date(message.date).toISOString()),
            message.package.name,
            message.package.version,
            message.level,
            message.data
          );
        } else {
          console[level](
            (new Date(message.date).toISOString()),
            message.level,
            message.data
          );
        }
      }
    });
  }, (error) => {
    console.error(error);
    throw new Error('Unable to create a reciever for the amqp service, exiting');
  });
}, (error) => {
  console.error(error);
  throw new Error('Unable to connect to amqp service, exiting');
});

this.client.on('error', () => {
  console.error(error);
  throw new Error('Connection to amqp service whent down, exiting');
});

console.log('Test that we get here', `amqp://${AMQP_HOST}`);

I found something that might be the answer, if I set the event listener to
this.client.on(AMQP.Client.ErrorReceived, error => console.error(error));

@tirithen this is most likely because you have reconnect set to retry forever (the default). You could for instance try: this.client = new AMQP.Client({ reconnect: { retries: 1, forever: false } });

Also thanks for fast response :)


Add converting to BSONDocument

It would be great if there would be a way to convert json to BSONDocument (reactivemongo) and there would be a way to create custom handlers for some fields.

Hi, @andemi !
Two questions:

You need exactly BSONDocument, or ByteBuffer with BSON will be OK too?
What do you mean when talking about "custom handlers for some fields"? This - case class User(@key("user_name") userName: String)?

I think I hurried.
After JSON we need validate data, so direct converting JSON to BSONDocument is redundant.

BSONDocument http://reactivemongo.org/releases/0.11/api/index.html#reactivemongo.bson.BSONDocument
No, validation or changing some data

What means "validate"? It's needed for be sure thats email is email and age not 999? If it is, I suppose we can do it before converting to BSON. Anyway pushka.Ast to BSONDocument conversion will be lose data type precision: DateTime and Timestam became just Longs.
I have old plans about supporting various AST #3. It's next big thing in Pushka, so I planing to take this issue to work in few next months. I can take reactive-mongo as first external AST backend if you help me to realize requirements of reactive-mongo.

It's needed for be sure thats email is email and age not 999?
Yes, something like this.

Here some data types classes
http://reactivemongo.org/releases/0.11/documentation/bson/typeclasses.html
and
http://reactivemongo.org/releases/0.11/documentation/json/overview.html

So what "validation" means?

For example, all data that comes from frontend must be checked

Ok. "Yes, something like this" is in quote :) Such kind of validation is not Pushka's job. You can validate fields after case class was read.

Waiting for #3.


Remove unnecessary pipes from the codes

Pipes spawn subshells. Use herestring to remove unnecessary time cost on spawning the shells.

I'm wondering about code readability after this change 🤔

Agree, it becomes less obvious to understand.

@laggardkernel could you give us some numbers/statistics on how big is perf improvement? Is it reasonable to lose readability to gain minimal perf improvement?

This is similar to the UUOC, some buy it, while others don't. And you're right, the performance improvement is marginal in this situation. I pointed it out just because i write bash codes in this style.
It's more about the coding style. So just choose what you guys prefer. Maybe no need to pay much attention to it here.

Let's keep it as it is. Right now the code is more familiar to newbies and IMHO simpler to read.
Huge thanks for your PR though, @laggardkernel!


ImportError: No module named dota_kit.poly_nms_gpu.poly_overlaps

After running the following command. there are no poly_overlaps to import
sudo apt-get install swig
cd ${RoI_ROOT}/dota_kit
swig -c++ -python polyiou.i
python setup.py build_ext --inplace
cd ${RoI_ROOT}/dota_kit/poly_nms_gpu
make -j16

You should use the same python interpret for complile and run.

I also encountered the same problem.I used the same python interpret(python 2.7) for complile and run.

I also met this question.did you solved it? @

I also encountered the same problem.I used the same python interpret(python 2.7) for complile and run.

I also met this question.did you solved it?

I also encountered the same problem.I used the same python interpret(python 2.7) for complile and run.

I also met this question.did you solved it?

I have not solved it yet.

in the file add
"import sys
sys.path.append("${RoI_ROOT}")
"
but it seem that another error happened
"AttributeError: 'module' object has no attribute 'PSROIALIGNAVEPooling'"

I solved this problem by modifying   "..../core/rcnn.py"   file，add a line ---- import sys sys.path.append("${RoI_ROOT}")   at the beginning, it can work!

I solved this problem by modifying "..../core/rcnn.py" file，add a line ---- import sys sys.path.append("${RoI_ROOT}")  at the beginning, it can work!

I also encountered the same problem.I used the same python interpret(python 2.7) for complile and run.

I solved this problem by modifying "..../core/rcnn.py" file，add a line ---- import sys sys.path.append("${RoI_ROOT}") at the beginning, it can work!

I also encountered the same problem.I used the same python interpret(python 2.7) for complile and run.

I also met this question.did you solved it?

I have not solved it yet.

I solved this problem by modifying "..../core/rcnn.py" file，add a line ---- import sys sys.path.append("${RoI_ROOT}") at the beginning, it can work!

I also encountered the same problem.I used the same python interpret(python 2.7) for complile and run.

I also met this question.did you solved it?

I have not solved it yet.

I solved this problem by modifying "..../core/rcnn.py" file，add a line ---- import sys sys.path.append("${RoI_ROOT}") at the beginning, it can work!

I have add "import sys
sys.path.append("${RoI_ROOT}")
"
but it still has the same problem.(ImportError: No module named dota_kit.poly_nms_gpu.poly_overlaps)
Please help me thank you!

I have also been having this issue. "import sys" doesn't help at all, nor does directly editing the import path.

Also encounter this issue, compile all in the same anaconda envi，“import sys” does not help，hope author can help. @dingjiansw101
PS，Is there anything to do with MXnet？


Implement commands to mute ooc messages and messages sent by other characters

Issue
There should be commands that allows a player to mute OOC communication and messages sent using the message command.

Resolved by @ItsAKelmi in PR #235


When to fetch <variant, chain, realization> triplet

Currently, we fetch this information in the init view at the beginning of the experiment.
If somebody takes a long time reading the instructions or does not even presses the next button, another participant might has to wait a long time, because they are already matched by their triplets.
Only fetch this information, when a participant is ready for the experiment and presses the to the lobby button.

I think this might also be helped by the timeout implementation we discussed before. i.e. if the participant didn't move for 10 minutes, the connection will be dropped and they will be prompted to try again. Might also be a solution.


Bug: Image paste doesn't respect caret position

When there are multiple Image Nodes consecutively, pasting an cutting and pasting an image pastes it at the wrong location. I suspect this issue is happening for nodes which implement decorate.
Lexical version: 0.9.0
Steps To Reproduce

Insert three distinct images consecutively (with no spaces in between)
Move your cursor after the last image
Use Shift + Left Arrow to select the last image
Use Cmd / Ctrl + X to cut and Cmd / Ctrl + V to paste at same location

Link to code example: This issue occurs in playground.
The current behavior
Image is pasted between first and second images
The expected behavior
Image is pasted after the second image
https://user-images.githubusercontent.com/13694315/226834392-97cb6df0-60b4-49e9-9d2a-bf45fcef6351.mp4

Had a quick look into it. Turns out the insertNodes does not respect the position when the following are true:

Selection is inside an ElementNode
InsertNodes is given a block that later insertNodes will have to unwrap

We've plans to rewrite insertNodes. Ideally insertNodes is lighter and less magical. For the sake of a quick solution to this one we may want to handle the block case specifically in the clipboard code (which may eventually by the good way if we reduce the responsibilities of insertNodes).


Adding support for kpi to sixpack.Session.convert.

This is backwards compatible.

This is already implemented. Close?


Issue with the Github site

The linked github site isnt up to date, I believe you touched on that in the readme, but I don't see a link to the updated site?
Also, are there instructions/tutorial to get this up and running ourselves?
Thank you!

Hi there. I had been using travis-CI to keep my site updated, but ran out of credits with them and am waiting on more. I hadn't updated the site for a little while, but ran the python script at home and updated the site with the corresponding output.
If all you want to do is get the code to produce new data and photo files, all you would need to do is download the complete repository, go into the .travis folder (remember the . means it's a hidden folder), and move the make.py file to the parent directory. Get an API key from Alpha Vantage and use it to replace the variable called "ALPHA_KEY". Then, in an environment with all python dependencies installed, run make.py to from the parent directory to create updated files with new data. I'll try to make time to post updated models in the upcoming week, but if you'd like to do it yourself, the code is in the load_model_ex.ipynb notebook file. Keep in mind that the models can take quite a lot of time and computing power to make, and I recommend running them on an external GPU and not on your system. Please let me know if anything comes up or if any of those instructions are unclear.
Remember that once you get this up and running, you can easily change the companies and length of prediction or use the data for different purposes or visualizations. Let me know if you do!

Thanks for the help! I was able to update the models, images, and analyze/predict stocks outside of what you had included. I’d like to see the chart with actual prices vs prediction, but I cannot figure out how to get the multiline Chart to appear on the index.HTML for individual comps.
am I missing something? Thank you

Hmm, if you have all the files downloaded and it's not working then check your developer console on your browser when you pull those pages up. I expect that'll lead you in the right direction. I'll keep an eye on this issue today, so I should be able to get back to you pretty quickly if you have any more questions.


Tensorflow lite conversion error when converting the bidirectional LSTM

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 2.3.0
Python version: 3.8.10

You can collect some of this information using our environment capture script:
https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh
You can obtain the TensorFlow version with:
python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the problem
I am trying to convert a tensorflow model into the tensorflow lite format. It prompts that it failed to duplicate values for the stateful op when the conversion goes to the bidirectional lstm layer. Here below is the network structure.
Model: "sequential"

Layer (type)                 Output Shape              Param #
conv1d (Conv1D)              (None, 1558, 32)          5024
max_pooling1d (MaxPooling1D) (None, 389, 32)           0
batch_normalization (BatchNo (None, 389, 32)           128
conv1d_1 (Conv1D)            (None, 386, 64)           8256
max_pooling1d_1 (MaxPooling1 (None, 96, 64)            0
batch_normalization_1 (Batch (None, 96, 64)            256
bidirectional (Bidirectional (None, 50)                18000
dense (Dense)                (None, 32)                1632
dense_1 (Dense)              (None, 25)                825
Total params: 34,121
Trainable params: 33,929
Non-trainable params: 192

Source code / logs
Here is the code to raised this error.
converter = tf.lite.TFLiteConverter.from_saved_model(save_model_dir)
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS,                                            tf.lite.OpsSet.SELECT_TF_OPS]
 tflite_model = converter.convert()
with open('converted_model.tflite', 'wb') as f:
        f.write(tflite_model)
Here below are the logs:
2021-09-28 16:44:25.817579: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:313] Ignored output_format.
2021-09-28 16:44:25.817782: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:316] Ignored drop_control_dependency.
2021-09-28 16:44:25.819144: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: D:\VoiceRecognition\trained_model
2021-09-28 16:44:25.884916: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2021-09-28 16:44:25.885104: I tensorflow/cc/saved_model/loader.cc:234] Reading SavedModel debug info (if present) from: D:\VoiceRecognition\trained_model
2021-09-28 16:44:26.135034: I tensorflow/cc/saved_model/loader.cc:199] Restoring SavedModel bundle.
2021-09-28 16:44:26.476369: I tensorflow/cc/saved_model/loader.cc:183] Running initialization op on SavedModel bundle at path: D:\VoiceRecognition\trained_model
2021-09-28 16:44:26.761966: I tensorflow/cc/saved_model/loader.cc:303] SavedModel load for tags { serve }; Status: success: OK. Took 942808 microseconds.
loc(callsite(callsite(callsite(unknown at "sequential/bidirectional/backward_lstm/PartitionedCall@__inference__wrapped_model_44427") at "StatefulPartitionedCall@__inference_signature_wrapper_52710") at "StatefulPartitionedCall")): error: We cannot duplicate the value since it's not constant.

error: Failed to duplicate values for the stateful op

Traceback (most recent call last):
  File "C:\Users\lizai\anaconda3\envs\SpectrumAnalysis\lib\site-packages\tensorflow\lite\python\convert.py", line 196, in toco_convert_protos
    model_str = wrap_toco.wrapped_toco_convert(model_flags_str,
  File "C:\Users\lizai\anaconda3\envs\SpectrumAnalysis\lib\site-packages\tensorflow\lite\python\wrap_toco.py", line 32, in wrapped_toco_convert
    return _pywrap_toco_api.TocoConvert(
Exception: <unknown>:0: error: loc(callsite(callsite(callsite(unknown at "sequential/bidirectional/backward_lstm/PartitionedCall@__inference__wrapped_model_44427") at "StatefulPartitionedCall@__inference_signature_wrapper_52710") at "StatefulPartitionedCall")): We cannot duplicate the value since it's not constant.

<unknown>:0: note: loc("StatefulPartitionedCall"): called from
<unknown>:0: note: loc(callsite(callsite(callsite(unknown at "sequential/bidirectional/backward_lstm/PartitionedCall@__inference__wrapped_model_44427") at "StatefulPartitionedCall@__inference_signature_wrapper_52710") at "StatefulPartitionedCall")): see current operation: %23 = "tfl.unidirectional_sequence_lstm"(%22, %cst_20, %cst_21, %cst_22, %cst_23, %cst_12, %cst_13, %cst_14, %cst_15, %cst_7, %cst_7, %cst_7, %cst_16, %cst_17, %cst_18, %cst_19, %cst_7, %cst_7, %21, %21, %cst_7, %cst_7, %cst_7, %cst_7) {cell_clip = 1.000000e+01 : f32, fused_activation_function = "TANH", proj_clip = 0.000000e+00 : f32, time_major = false} : (tensor<?x96x64xf32>, tensor<25x64xf32>, tensor<25x64xf32>, tensor<25x64xf32>, tensor<25x64xf32>, tensor<25x25xf32>, tensor<25x25xf32>, tensor<25x25xf32>, tensor<25x25xf32>, none, none, none, tensor<25xf32>, tensor<25xf32>, tensor<25xf32>, tensor<25xf32>, none, none, tensor<?x25xf32>, tensor<?x25xf32>, none, none, none, none) -> tensor<?x96x25xf32>
<unknown>:0: error: Failed to duplicate values for the stateful op

I have solved this problem by converting the bidirectional lstm into to unidirectional lstm. The solution is referenced by https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/examples/experimental_new_converter/Keras_LSTM_fusion_Codelab.ipynb?pli=1#scrollTo=0-b0IKK2FGuO


GenericModelBinder integration tests part 1

Covers simple scenario for each model binder.
Covers scenarios mixing a generic model binder with a greedy model
binder.

/cc @harshgMSFT

:shipit: after addressing comment.

:shipit: after addressing comment.

Please be a gentleman and merge your changes so I can rebase on top :+1:

Merged.


Sc-7718: implement need attention endpoint to user ui

Scope of changes
Sc-7718: implement need attention endpoint to user ui
Type of change

[ ] bug fix
[x] new feature
[ ] documentation
[ ] other (describe)

Acceptance criteria
Sc-7718: implement need attention endpoint to user ui
Author checklist

[ ] I have manually tested the change and/or added automation in the form of unit tests or integration tests
[ ]  I have updated the dependencies list
[ ]  I have recompiled and included new protocol buffers to reflect changes I made
[ ]  I have added new test fixtures as needed to support added tests
[x]   Check this box if a reviewer can merge this pull request after approval (leave it unchecked if you want to do it yourself)
[x]  I have moved the associated Shortcut story to "Ready for Review"

Reviewer(s) checklist

[ ] Any new user-facing content that has been added for this PR has been QA'ed to ensure correct grammar, spelling, and understandability.

Codecov Report

Merging #695 (1d17dc0) into main (d8293fb) will decrease coverage by 5.87%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #695      +/-   ##
==========================================
- Coverage   43.95%   38.08%   -5.88%
==========================================
  Files         575      155     -420
  Lines       17156     2434   -14722
  Branches     1246      618     -628
==========================================
- Hits         7541      927    -6614
+ Misses       8427     1330    -7097
+ Partials     1188      177    -1011

Impacted Files
Coverage Δ

web/gds-user-ui/src/application/routes/routes.ts

web/gds-user-ui/src/application/store/index.ts

...user-ui/src/application/store/selectors/stepper.ts

...gds-user-ui/src/application/store/stepper.slice.ts

...b/gds-user-ui/src/components/BasicDetail/index.tsx

.../CertificateReview/BasicDetailsReviewDataTable.tsx

...components/CertificateReview/LegalPersonReview.tsx

...s/CertificateReview/LegalPersonReviewDataTable.tsx

...ponents/CertificateReview/TrixoReviewDataTable.tsx

...user-ui/src/components/CertificateReview/index.tsx

... and 410 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


the spinach coloured extension tips hurt my eyes...

:+1:

ping @jrieken

Push the code, or don't close the issue.

@joaomoreno Just pushed all the code to master from my fork.

:+1:


Fixed SSO client docs

Resolves #802.
There are currently only the two listed clients.

merging...

merged: 62a87f2a4253003650639f5c6a680ee9c92ac8f1


Add S3 Storage type and S3StorageClient

Summary
Add S3 Storage type and S3StorageClient
Changes

[ ] Client-facing API Changes
[ ] Internal API Changes
[ ] Bug Fixes
[ ] New Features
[ ] Performance Improvements
[ ] Code Style
[ ] Refactoring
[ ] Documentation
[ ] Tests

For all the boxes checked, please include additional details of the changes made in this pull request.
Testing Done
Added unit tests. This is one of the first few PRs to complete plugging in S3 storage. More test recipes will be added when S3 storage is completely plugged in.

[x] Manually Tested on local docker setup. Please include commands ran, and their output.
[ ] Added new tests for the changes made.
[ ] Updated existing tests to reflect the changes made.
[ ] No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
[ ] Some other form of testing like staging or soak time in production. Please explain.

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.
Additional Information

[ ] Breaking Changes
[ ] Deprecations
[ ] Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

Overall code looks great on first pass.

Manually Tested on local docker setup. Please include commands ran, and their output.

Was this tested on local docker setup with s3 container, or current HDFS container.

Added unit tests. This is one of the first few PRs to complete plugging in S3 storage. More test recipes will be added when S3 storage is completely plugged in.

Can you update the description with a high level PR plan on what to expect after this lands?
Also super nit, PR title [PR 1/n] Add S3 Storage type and S3StorageClient


[BUG]: address wildcard list performance

Bug Description / What Happened
Each entry in wildcard lists are currently loaded as a regular expression. With large lists, this is resulting in essentially a self-imposed Denial of Service (DoS) situation where every request is being checked and pegging CPU usage, causing clients to perceive DNS services to be unavailable.
The current RegEx for each wildcard entry is as follows:
https://github.com/wranders/coredns-filter/blob/0f142c8cd0709d65244366c527f1f1f69575e445/action.go#L166
This could be can be slightly optimized to:
(^|\\.)%s

but would still probably result in the same problem.
Another solution must be explored.
Perhaps unifying the disparate slices into a trie where wildcards could be identified as a dedicated field. Reversing domains such as com.example.* and returning a block result if a node contains a block field. This would incur a drastically higher memory footprint, but that seems preferable to a situation where CPU usage renders the entire DNS server unresponsive.
Another potential solution would be to just run HasSuffix for wildcard entries, though I don't know how this would compare to current CPU usage.
Example:
if strings.HasSuffix(request, ".example.com" {
    # Block request
}

Expected Behavior
Responsive DNS services with large block lists
Steps to reproduce

Use a large wildcard blocklist
???
lose DNS services

Additional Info
The v0.2.4 release has functional RegEx/wildcard support, but should only be used with very small lists.
OS
Raspberry Pi OS 11, Linux 5.10.103-v7l, armv7l
Version
v0.2.4
Logs
Sep 24 23:12:56 dns1 coredns[18610]: [ERROR] plugin/errors: 2 settings-win.data.microsoft.com. A: EOF
Sep 24 23:12:56 dns1 coredns[18610]: [ERROR] plugin/errors: 2 connectivitycheck.gstatic.com. A: EOF
Sep 24 23:12:56 dns1 coredns[18610]: [ERROR] plugin/errors: 2 www.gstatic.com. A: EOF
Sep 24 23:12:57 dns1 coredns[18610]: [ERROR] plugin/errors: 2 updates.coreos.fedoraproject.org. AAAA: EOF
Sep 24 23:12:57 dns1 coredns[18610]: [ERROR] plugin/errors: 2 www.gstatic.com. A: EOF
Sep 24 23:12:57 dns1 coredns[18610]: [ERROR] plugin/errors: 2 unagi-na.amazon.com. A: EOF
Sep 24 23:12:58 dns1 coredns[18610]: [ERROR] plugin/errors: 2 connectivitycheck.gstatic.com. A: EOF
Sep 24 23:12:59 dns1 coredns[18610]: [ERROR] plugin/errors: 2 unagi-na.amazon.com. AAAA: EOF
Sep 24 23:13:01 dns1 coredns[18610]: [ERROR] plugin/errors: 2 unagi-na.amazon.com. AAAA: EOF
Sep 24 23:13:02 dns1 coredns[18610]: [ERROR] plugin/errors: 2 updates.coreos.fedoraproject.org. A: EOF
Sep 24 23:13:03 dns1 coredns[18610]: [ERROR] plugin/errors: 2 connectivitycheck.gstatic.com. A: EOF
Sep 24 23:13:03 dns1 coredns[18610]: [ERROR] plugin/errors: 2 connectivitycheck.gstatic.com. A: EOF
Sep 24 23:13:03 dns1 coredns[18610]: [ERROR] plugin/errors: 2 connectivitycheck.gstatic.com. A: EOF
Sep 24 23:13:03 dns1 coredns[18610]: [ERROR] plugin/errors: 2 ab992t7ewg8z.na.api.amazonvideo.com. A: EOF
Sep 24 23:13:03 dns1 coredns[18610]: [ERROR] plugin/errors: 2 unagi-na.amazon.com. AAAA: EOF
Sep 24 23:13:03 dns1 coredns[18610]: [ERROR] plugin/errors: 2 clients4.google.com. HTTPS: EOF
Sep 24 23:13:03 dns1 coredns[18610]: [ERROR] plugin/errors: 2 unagi-na.amazon.com. AAAA: EOF
Sep 24 23:13:04 dns1 coredns[18610]: [ERROR] plugin/errors: 2 connectivitycheck.gstatic.com. A: EOF

and so on, due to inability to respond to requests.

Just found this..
Looks great.. "I love what you have done with the place.."
I have almost 10k chromebooks that I do recursion for.. and have an 'adblock syntax' list of almost 5k domains
Have you seen this project?
0xERR0R/blocky
The logging in coredns is better for what I am doing..
I presently use the forward plugin and an import of:
except lan lan1 home myfiosgateway mynetworksettings.com discord.gg etc..
you get a million of these:
[ERROR] plugin/errors: 2 wpad.lan. AAAA: plugin/forward: no next plugin found
[ERROR] plugin/errors: 2 wpad.lan. A: plugin/forward: no next plugin found
[ERROR] plugin/errors: 2 wpad.lan. HTTPS: plugin/forward: no next plugin found

Definitely would help test..