Implement trunc_sat ops

Shouldn't we merge the missing instructions? @NikVolf @pepyakin

Shouldn't we merge the missing instructions? @NikVolf @pepyakin

I don't believe those are from V1 spec are they?

I don't believe those are from V1 spec are they?

I don't know which revision added these, but they're standard now. You can see them at the bottom of instruction index: https://webassembly.github.io/spec/core/appendix/index-instructions.html#index-instr

I don't know which revision added these, but they're standard now. You can see them at the bottom of instruction index: https://webassembly.github.io/spec/core/appendix/index-instructions.html#index-instr

I think they should be implemented in similar way to atomics:
https://github.com/paritytech/parity-wasm/blob/bb191e850da6ccfbb6b098dddd5a882a59afc212/src/elements/ops.rs#L308
(dedicated entry in enum and under feature-flag)

I think they should be implemented in similar way to atomics:
https://github.com/paritytech/parity-wasm/blob/bb191e850da6ccfbb6b098dddd5a882a59afc212/src/elements/ops.rs#L308
(dedicated entry in enum and under feature-flag)

Why? Atomics are not standard, but these are.

Why? Atomics are not standard, but these are.

I think because while being part of the standard they are not part of the MVP. I found no easy way to look that up. That said, I am OK with adding instructions that are part of the standard without a feature flag no matter when they where added.
Users of parity-wasm that can't handle these new instructions must make sure to reject then. As a matter of fact this is exactly what pallet-contracts does.

I think because while being part of the standard they are not part of the MVP. I found no easy way to look that up. That said, I am OK with adding instructions that are part of the standard without a feature flag no matter when they where added.
Users of parity-wasm that can't handle these new instructions must make sure to reject then. As a matter of fact this is exactly what pallet-contracts does.

Why? Atomics are not standard, but these are.

parity-wasm cannot afford to produce no error on something that produced error in previous versions
Those ops were later addition

Why? Atomics are not standard, but these are.

parity-wasm cannot afford to produce no error on something that produced error in previous versions
Those ops were later addition

Are there any major drawbacks to your use case when guarding the new instructions behind a new feature flag? It will certainly make our life easier.

Are there any major drawbacks to your use case when guarding the new instructions behind a new feature flag? It will certainly make our life easier.

Closing for lack of activity. @osa1 open a new PR if you want to see this merged (behind a feature flag).


:bn and :bp crash omnivim

I've seen some inconsistent behavior here, so I'm not quite sure what causes this, but right now I can reproduce it by opening onivim2 and running :bn or :bp. On v0.5.0

Thanks for reporting, @iamcodemaker !
Unfortunately I'm not able to reproduce this... maybe some extra info will help:

What platform are you using? Mac/Windows/Linux?
Do you have any other files open, or are you able to reproduce immediately after opening and running :bn or :bp?
Does it reproduce on a newer build? Either the stable (0.5.5) or nightly? It's possible it's fixed by a PR that has come in since 0.5.0, like https://github.com/onivim/oni2/pull/2351

I'm on Mac. I was able to reproduce this by running onivim2 and then immediately running :bn. I'll test with 0.5.5.

Unable to reproduce on 0.5.2120

Thanks @iamcodemaker for checking in latest 👍


LLVM error on High Sierra

Building cortex-m-rt on nightly, beta and stable in Mac OS High Sierra (tested on two machines) results in the following error:
LLVM ERROR: Global variable '__RESET_VECTOR' has an invalid section specifier '.vector_table.reset_vector': mach-o section specifier requires a segment whose length is between 1 and 16 characters.

It build correctly on my Ubuntu 18.04 machine.

mach-o section specifier requires a segment whose length is between 1 and 16 characters

Are you compiling the cortex-m-rt crate for the host (for mac)?
If that's the case we can drop the link_section when compiling for mac; trying to link a cortex-m program compiled for the host (mac / Linux) is going to fail anyway.

 I forgot to set the default target in .cargo/config.

The cortex-m crate goes out of its way to build at least on Linux, would be nice if all the core crates consistently built on Linux, macOS, and Windows

we ran into a similar issue in the defmt project.
we found out that on macOS the longest linker section name can be .section,0123456789ABCDEF (note: comma after .section not period and a max of 16 characters after the comma)
this issue can probably be fixed by adding a cfg_attr(is_macos, link_section = "..") to the relevant static variables

we ran into a similar issue in the defmt project.
we found out that on macOS the longest linker section name can be .section,0123456789ABCDEF (note: comma after .section not period and a max of 16 characters after the comma)
this issue can probably be fixed by adding a cfg_attr(is_macos, link_section = "..") to the relevant static variables

Indeed, I've encountered it frequently.

Indeed, I've encountered it frequently.

What does this fix do? It replaces section name with exactly the same but taking half a kilobyte more space in source code; am I missing something there?

What does this fix do? It replaces section name with exactly the same but taking half a kilobyte more space in source code; am I missing something there?

@berkus Not sure what you mean. What are you seeing and on which system? This is changing the section names for macOS only because it has a weird length restriction. It should not affect anything except to prevent linker errors on macOS.

@berkus Not sure what you mean. What are you seeing and on which system? This is changing the section names for macOS only because it has a weird length restriction. It should not affect anything except to prevent linker errors on macOS.

@therealprof ok,  lets go with the way of example, shall we.
#[cfg_attr(target_os = "macos", link_section = ".vector_table,interrupts")]
#[cfg_attr(not(target_os = "macos"), link_section = ".vector_table.interrupts")]

On  macOS the  section  name would be .vector_table.interrupts,  however on any other system it would be .vector_table.interrupts which is EXACTLY THE SAME.
So now you've reduced readability, understandability and maintainability of the  code to provide exactly the same section names for every system out there.
What am I missing? I could have missed something, to be precise am looking  at this commit.
Here it  is, condensed:
#[cfg_attr(target_os = "macos", link_section = ".HardFault,user")]
#[cfg_attr(not(target_os = "macos"), link_section = ".HardFault.user")] // NB: same name

#[cfg_attr(target_os = "macos", link_section = ".vector_table,reset_vector")]
#[cfg_attr(not(target_os = "macos"), link_section = ".vector_table.reset_vector")] // NB: same name

#[cfg_attr(target_os = "macos", link_section = ".HardFault,default")]
#[cfg_attr(not(target_os = "macos"), link_section = ".HardFault.default")] // NB: same name

#[cfg_attr(target_os = "macos", link_section = ".vector_table,exceptions")]
#[cfg_attr(not(target_os = "macos"), link_section = ".vector_table.exceptions")] // NB: same name

#[cfg_attr(target_os = "macos", link_section = ".vector_table,interrupts")]
#[cfg_attr(not(target_os = "macos"), link_section = ".vector_table.interrupts")] // NB: same name

@therealprof ok,  lets go with the way of example, shall we.
#[cfg_attr(target_os = "macos", link_section = ".vector_table,interrupts")]
#[cfg_attr(not(target_os = "macos"), link_section = ".vector_table.interrupts")]

On  macOS the  section  name would be .vector_table.interrupts,  however on any other system it would be .vector_table.interrupts which is EXACTLY THE SAME.
So now you've reduced readability, understandability and maintainability of the  code to provide exactly the same section names for every system out there.
What am I missing? I could have missed something, to be precise am looking  at this commit.
Here it  is, condensed:
#[cfg_attr(target_os = "macos", link_section = ".HardFault,user")]
#[cfg_attr(not(target_os = "macos"), link_section = ".HardFault.user")] // NB: same name

#[cfg_attr(target_os = "macos", link_section = ".vector_table,reset_vector")]
#[cfg_attr(not(target_os = "macos"), link_section = ".vector_table.reset_vector")] // NB: same name

#[cfg_attr(target_os = "macos", link_section = ".HardFault,default")]
#[cfg_attr(not(target_os = "macos"), link_section = ".HardFault.default")] // NB: same name

#[cfg_attr(target_os = "macos", link_section = ".vector_table,exceptions")]
#[cfg_attr(not(target_os = "macos"), link_section = ".vector_table.exceptions")] // NB: same name

#[cfg_attr(target_os = "macos", link_section = ".vector_table,interrupts")]
#[cfg_attr(not(target_os = "macos"), link_section = ".vector_table.interrupts")] // NB: same name

Ahha, now I see the  comma vs period, my bad!

Ahha, now I see the  comma vs period, my bad!

@therealprof btw, could it be using the comma on  all systems, not only macos?

@therealprof btw, could it be using the comma on  all systems, not only macos?

Not sure sorry, I'm actually curious why this is supposed to work on macOS (NB: I haven't actually tested it myself). Maybe some internal linker magic to substitute the "." back?

Not sure sorry, I'm actually curious why this is supposed to work on macOS (NB: I haven't actually tested it myself). Maybe some internal linker magic to substitute the "." back?

@therealprof on mac this is  two names, ".vector_table" for the segment and "reset_vector" for the section name.
I will test on linux if a section name with comma works, the ELF file format spec says why not so  the only limiting factor would be linker's parser.

@therealprof on mac this is  two names, ".vector_table" for the segment and "reset_vector" for the section name.
I will test on linux if a section name with comma works, the ELF file format spec says why not so  the only limiting factor would be linker's parser.

We could probably just remove the link_section attribute entirely on MacOS, right? It's not going to actually do anything as I understand it, since we can't build final libraries or executables targeting MacOS with cortex-m-rt and it doesn't use our linker script (where the names matter) on MacOS either; this change just lets people on MacOS run cargo check/test without a target specified. I wouldn't want to swap the ARM builds or linker script to using commas for section names just so that Macs can check the code when compiling natively, though.

We could probably just remove the link_section attribute entirely on MacOS, right? It's not going to actually do anything as I understand it, since we can't build final libraries or executables targeting MacOS with cortex-m-rt and it doesn't use our linker script (where the names matter) on MacOS either; this change just lets people on MacOS run cargo check/test without a target specified. I wouldn't want to swap the ARM builds or linker script to using commas for section names just so that Macs can check the code when compiling natively, though.


Need to compile manually on MacOS

Using the Makefile I'll get an error trying to run Terraform apply or other Terraform CRUD actions:
*terraform.EvalInitProvider, err: fork/exec ...  exec format error
There is no support in the Makefile for other Architectures / Operating Systems (My Mac included)
So after compiling manually with Mac compatible parameters, I was able to run terraform apply with paperspace provider.
Figured I'd note it here, if anyone else was running into this.
This could probably be fixed by removing hardcoded environment variables for GOOS=linux GOARCH=amd64 and assume users running go build have values exported to their shell, or Golang resolves them.

Hi @TimDurward, this has been resolved as of<PHONE_NUMBER>aa6c15974428bb472809339aac6596 :) Better late than never! We've picked back up development on this recently. If you happen to still be using this provider or do, and find any other issues, we'd love to hear from ya~


fix: trim only replaces last occurence of trim

:heavy_check_mark: Includes additions to the docs, found in the docs/readme file, as required - (Not required)
:heavy_check_mark: Includes a new line in the changelog titling the change
:heavy_check_mark: The new and changed code you have written is unit tested and passes the eslint
:heavy_check_mark: The body of the text for this PR includes enough text for a reviewer to understand the context of the change
I'm working on a project which has entities called 'ModelCode', 'ModelRange', 'ModelCodeLabel' etc.
We found that the current implementation of trimming using string.replace(trim, '') was replacing the first instance of 'Model' that was found in the string.
This was causing odd errors and turning 'ModelCode' into 'CodeModel'.
This fix means that only the last instance of the trim string will be replaced from singular or plural schema models.

@acr-tom-bee  looks good! Thanks for the contribution :)
I've edited the PR to dev not main.

@j-d-carmichael  Thanks for the quick merge :) Sorry about the branch mix up. I wasn't sure whether fixes went straight into main or dev first. I'll pop any future ones onto dev.


Transpile cloud-api/ through babel?

Hello,
Is this possible?
I took a quick look at the source, and couldn't find any apparent module to hook into the build process.
-d

you are right, the cli does not need to be transpiled by babel.

No, I mean running my code through babel in the awsmobile push command.
I traced the building process to here.
Am I correct in thinking that function is what I need to modify to hook into the build process?

correct.
before the lambda function is zipped, you need to make sure it has already been correctly built.


Added auto-complete for Python IDF objects

Added:
PythonPlugin:Instance
PythonPlugin:SearchPaths
PythonPlugin:Variables .. which may not actually be an idf obj..

@aaron-boranian   Added the couple pyems IDF objs that I've been using recently


ND321 Is not available to download

Describe the bug
I am following all the steps mentioned in the wiki page but I am not able to download the Hybrid Cloud Nanodegree i.e. ND321 which comes under Udacity Scholarship Program.
Kindly do check if anything can be done regarding the same if not then kindly suggest any other path/way for the same.

https://www.youtube.com/watch?v=Bilsm50HO1s


CIV-4981 Refactor notification deadline calc.

Before creating a pull request make sure that:

[x] commit messages are meaningful and follow good commit message guidelines
[x] README and other documentation has been updated / added (if needed)
[x] tests have been updated / new tests has been added (if needed)
[x] Screenshot attached for the Evidence of manual testing
[x] Functional, Smoke and API tests have been run locally
[x] Full gradle build and run tests with coverage have been completed (if needed)

Please remove this line and everything above and fill the following sections:
JIRA link (if applicable)
https://tools.hmcts.net/jira/browse/CIV-4981
Change description
Refactor notification deadline calc. to add extra day for each holiday
Testing branch and instruction
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[ ] No

Screenshot are attached in the ticket

Fucntional

smoke, api, functional


add RAG: cpu/cuda/mps as a config variable rather than hardcode

File: backend/config.py has the following hardcoded variable:
RAG_EMBEDDING_MODEL_DEVICE_TYPE = os.environ.get(
    "RAG_EMBEDDING_MODEL_DEVICE_TYPE", "cpu"
)

I use open-webui locally without Docker, and whenever I do an update I have config.py replaced, however the hardcoded variable I have to change everytime to "mps" and I forget a lot of the time.

Related #1165


Fix issues that make test -race fail

Tests with goroutines instead of regular t.Parallel()
Calling dictionaryHelper without a mutex from multiple tests

Codecov Report

Merging #72 into master will decrease coverage by 0.9%.
The diff coverage is 7.07%.

@@            Coverage Diff             @@
##           master      #72      +/-   ##
==========================================
- Coverage   56.52%   55.61%   -0.91%
==========================================
  Files          65       65
  Lines        4692     4772      +80
==========================================
+ Hits         2652     2654       +2
- Misses       1881     1959      +78
  Partials      159      159

Impacted Files
Coverage Δ

template/generated_options.go
0% <0%> (ø)
:arrow_up:

collections/generic_list.go
0% <0%> (ø)
:arrow_up:

collections/dictionary.go
0% <0%> (ø)
:arrow_up:

utils/generated_colors.go
29.03% <0%> (-70.97%)
:arrow_down:

context.go
76.66% <100%> (+0.53%)
:arrow_up:

collections/implementation/list_helper.go
94.54% <100%> (ø)
:arrow_up:

collections/implementation/dict_helper.go
94.81% <100%> (ø)
:arrow_up:

main.go
57.28% <25%> (ø)
:arrow_up:

collections/implementation/base_helper.go
79.5% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 85a3fef...cdc0da6. Read the comment docs.


M3-1253 Add SSH key event messages

change: Added event messages for SSH key addition and deletion.
change: Added user event menu click handler for SSH key addition and deletion.

Coverage decreased (-0.008%) to 32.107% when pulling e918113b09a939810688ca476c4ab9a7a8409ec6 on rjstires:M3-1253 into 76aae46aa959b44fb73fdc68dc94ff6ce7bee6f4 on linode:develop.


Option to handle incomplete NVD version information

I'm using Dependency Track "for real", monitoring several products and hundreds of components.
Sometimes DTrack does not report a vulnerability I know is applicable. The reason is that the "affected version" information in NVD XML sources is incomplete. A frequent example is that only the version in which the vulnerability is found is reported, nothing being said about earlier versions. When the set of the affected versions is nontrivial (happens regularly with e.g. OpenSSL), it is seldom fully correct in the NVD XML.
I see this as a showstopper for many users: the purpose of DTrack is to not miss the vulnerabilities.
But obviously the report cannot be better than the source data it is based upon.
One idea might be to give an option to "report vulnerabilities for all versions" for a given component when the user selects a component version in the application. This way the user will get also non-applicable vulnerabilities, but will never miss the relevant ones (if only the product name is correct).
I do not know how such "feature request" fits in the present vision of DTrack. But otherwise I found that I'm reading the XML diff to be sure that I have not missed something critical, which kind of defeats the purpose of using DTrack.

I also use dtrack 1.x in production and have the same experience. One major difference between v1 and v3 is that v1 relies solely on dependency-check to provide it CVEs and other relevant vulnerability information. v3 on the other hand contains a full mirror of the NVD regardless of whether or not tracked components have any of the vulnerabilities or not.
One of the primary use cases for this is to be able to look up a CVE and see (on the same page) a list of projects that may be affected by it. This partially works today.
I was leaning towards the idea you had, but hadn't thought it all the way through yet. This would require the ability to perform "scans" not using dependency-check, but rather some simple internal logic that leverages the full mirror of the NVD that v3 has. I think this would be possible in v3. Not sure if it will make it into 3.0.0 or not, or a minor release afterwards. But I think this feature would be really valuable.


lchown permission denied (non-root user)

Hi,
I'm trying to get the sshfs volume working for my backup stack, but everytime I try to mount the endpoint I get prompted with permission denied.
With the user that's running docker-compose, I can access the folder so I'm not sure why this is going wrong.
Error when running docker-compose:
Error response from daemon: failed to copy file info for /var/lib/docker/plugins/a1318307eab6ed3e2bcfffa7a5226d557e7a2a6f12b4d48113d2e79063da848d/propagated-mount/e6c0b9b0593381f295d4a5c925f8ad8b: failed to chown /var/lib/docker/plugins/a1318307eab6ed3e2bcfffa7a5226d557e7a2a6f12b4d48113d2e79063da848d/propagated-mount/e6c0b9b0593381f295d4a5c925f8ad8b: lchown /var/lib/docker/plugins/a1318307eab6ed3e2bcfffa7a5226d557e7a2a6f12b4d48113d2e79063da848d/propagated-mount/e6c0b9b0593381f295d4a5c925f8ad8b: permission denied.

You can workaround this issue for now by leaving a dump file on the remote location.

I didnt try it yet with docker compose, but I also couldn't make any connections. I finally solved it by:

installing the plugin as: docker plugin install --grant-all-permissions vieux/sshfs sshkey.source=/root/.ssh/
setting up piblic/private key for root (sudo -i , then ssh-keygen -b 4096 -t rsa )
setting up the volume: docker volume create -d vieux/sshfs -o sshcmd=sven@<IP_ADDRESS>:/home/user/ sshvolume

Hope this helps either you or the development team.

I'm getting this error message as well:
docker run -ti -v nextcloud-html:/var/www/html nextcloud /bin/bash
docker: Error response from daemon: failed to copy file info for /var/lib/docker/plugins/2bb483b4a3521a55bff3d7284d4f1646b74092430c307ba7fea29306ea92fa89/propagated-mount/21558aea1829ef2d4bd4c0b95f4e5f60: failed to chown /var/lib/docker/plugins/2bb483b4a3521a55bff3d7284d4f1646b74092430c307ba7fea29306ea92fa89/propagated-mount/21558aea1829ef2d4bd4c0b95f4e5f60: lchown /var/lib/docker/plugins/2bb483b4a3521a55bff3d7284d4f1646b74092430c307ba7fea29306ea92fa89/propagated-mount/21558aea1829ef2d4bd4c0b95f4e5f60: permission denied.
See 'docker run --help'.
In my case this seems to happen because I have use two sshfs volumes (nextcloud-db and nextcloud-html) on the same remote ssh server like in this example:
docker volume inspect nextcloud-db
[
{
"CreatedAt": "0001-01-01T00:00:00Z",
"Driver": "vieux/sshfs:latest",
"Labels": null,
"Mountpoint": "/mnt/volumes/2e01dcda104ec5521369b7d308810632",
"Name": "nextcloud-db",
"Options": {
"allow_other": "",
"gid": "999",
"password": "xxx",
"sshcmd": "uxxx-sub1@uxxx.your-storagebox.de:nextcloud_db",
"uid": "999"
},
"Scope": "local"
}
]
docker volume inspect nextcloud-html
[
{
"CreatedAt": "0001-01-01T00:00:00Z",
"Driver": "vieux/sshfs:latest",
"Labels": null,
"Mountpoint": "/mnt/volumes/21558aea1829ef2d4bd4c0b95f4e5f60",
"Name": "nextcloud-html",
"Options": {
"allow_other": "",
"password": "xxx",
"sshcmd": "uxxx-sub2@uxxx.your-storagebox.de:",
"uid": "33"
},
"Scope": "local"
}
]
I figured out that it seems I cannot use both volumes at the same time.
It is possible to mount volume nextcloud-db to a mariadb container, but if I startup a second container (see docker run in my 3rd Line) this happens. If I stop/rm the mariadb container first, my 3rd line executes successfully.
Please note that both volumes are hosted on the same host (but accessed via different ssh-users). Could it be related to the different uid-Values I use in the volumes?
best regards,
Christoph

You can workaround this issue by leaving a dump file on the remote location.

What exactly do you mean by that? Do you have got an example? How can I produce such a dump file?

I am having the same issue and can't find a solution. Trying to use docker-compose to attach sshfs volumes.
docker version: 20.10.7
host: ubuntu 20.04 (kernel 5.4.0-80-generic)
Tried with public key authentication as well as password. Volume is created, user has full r/w/x over ssh with either method, can manually edit. Run docker-compse and watch auth logs on storage server, no problems there. Removed ACLs, set remote dirs to 777 and the parent folders.
Every time after the volume driver authenticates, docker-compose fails with the following error:
failed to copy file info for /var/lib/docker/plugins/33378cc08f72a307a93f8f14391ef77eb78de6ed6f97fc427a3c096d0f162956/propagated-mount/364cea55c25d60e3da22a307ac46c260: failed to chown /var/lib/docker/plugins/33378cc08f72a307a93f8f14391ef77eb78de6ed6f97fc427a3c096d0f162956/propagated-mount/364cea55c25d60e3da22a307ac46c260: lchown /var/lib/docker/plugins/33378cc08f72a307a93f8f14391ef77eb78de6ed6f97fc427a3c096d0f162956/propagated-mount/364cea55c25d60e3da22a307ac46c260: permission denied
Storage target is TrueNAS 12. Also have tried with Ubuntu 20.04 and a separate docker host (Mint 19), identical issues. What is going on here? None of the above suggestions worked, including creating the volume in advance, nor the fixes in issues 17, 58. 65.
Is this plugin still maintained?


Add git-leaks config

add .gitleaks.toml file to this repo please.
Currently I'm unsure where I should place it.
$REPO_ROOT/.gitleaks.toml and $REPO_ROOT/gitleaks.toml both seem to be not recognized by trunk for me.

Hi 👋  - Sorry about that, it looks like we were only respecting .gitleaks.config. Did gitleaks change it's config naming recently? Either way, it's a simple config change and I'll update it for the next release (you can also override this in trunk.yaml to get the behavior in the meantime).
For adding a .gitleaks.toml to this repo, what would you suggest? It seems like the default is pretty good (generally no configuration is needed?).

@dapirian yeah the default is pretty good. I'm still familiarizing myself with their config language but it'd be nice to have some inline-ignore functionality. Currently I have to set an allow regex to ignore AWS keys with the text EXAMPLE in them, and to ignore any lines ending in #nosec or //nosec based on https://github.com/zricethezav/gitleaks/issues/579
Any advice on how to set this in trunk.yaml?
Mine looks like ..
version: 0.1
cli:
  version: 0.8.1-beta
lint:
  enabled:
    -<EMAIL_ADDRESS>    -<EMAIL_ADDRESS>    -<EMAIL_ADDRESS>    -<EMAIL_ADDRESS>    -<EMAIL_ADDRESS>    -<EMAIL_ADDRESS>    -<EMAIL_ADDRESS>environments:
  - name: gitleaks
    environment:
      - name: GITLEAKS_CONFIG
        value: .gitleaks.toml

but it doesn't appear to be picked up. My gitleaks.toml is at the repo root.


Initial release

Hi, the note "at the initial release of integrated systems. we will support etc etc" makes it sound like the product is not G.A. yet.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e033d4a6-3152-6b73-aa78-bc17a57ee761
Version Independent ID: 38371ea2-87a8-a8a6-5ca4-577ef9fe1509
Content: General datacenter integration considerations for Azure Stack integrated systems
Content Source: articles/azure-stack/azure-stack-datacenter-integration.md
Service: azure-stack
GitHub Login: @jeffgilb
Microsoft Alias: jeffgilb

Thanks for the feedback! We are currently investigating and will update you shortly.

Good catch @cenkersoy . I'll get this updated ASAP.

I've made the necessary doc updates for this issue.

#please-close

@cenkersoy We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.


SW-6306 Added event listener to sync deal name and application internal name

#2677  👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.


Add Code of Conduct

We recommend that every repo has a code of conduct. If you don’t feel comfortable creating your own Code of Conduct from scratch we highly recommend using one of the templates provided by GitHub. If you do use a template, please read through the template and ensure that you can and will abide by the Code of Conduct.
Please follow these instructions on how to add a Code of Conduct.
Sample: https://github.com/Girlscript-WOB/Guidelines/blob/main/Project Admins/Code of conduct.md

Can you assign this issue to me as a part of GSWOB24?

Also please let me know the support email ID to be included to the code of conduct markdown file for reporting purposes.

I have raised a PR referencing this issue post assignment. Please review and share the feedback. Thank you!

No response from Project Owner over a long period of time. Hence, closing the PR for good!


Request to add wrapper of api/v1/registry/names

Is your feature request related to a problem?
No
Describe the solution you'd like
Add the following snippet to prismacloud/api/compute/_registry.py:
def registry_list_names(self, body_params=None):         result = self.execute_compute('GET', 'api/v1/registry/names', body_params=body_params)         return result
Describe alternatives you've considered
Have to implement this locally, but I use it in lambdas which have an automated deployment. It would make it easier to deploy and reduce complexity of my deployment to have it in the package installed via requirements.txt

Thanks! That endpoint is a GET, so should use query params rather than body params:
https://prisma.pan.dev/api/cloud/cwpp/registry#operation/get-registry-names
So I'll add this:
    def registry_list_image_names(self, query_params=None):
        result = self.execute_compute('GET', 'api/v1/registry/names?', query_params=query_params)
        return result

Resolved via https://github.com/PaloAltoNetworks/prismacloud-api-python/pull/126


Draft: fix for not copying empty guest Info into vmi status

This is a protective fix for the kubevirt issue for https://github.com/kubevirt/kubevirt/issues/12698
The following code changes prevents copying empty guestInfo into vmi status.
1.Send updates of guestInfo only when there is a change in guestInfo.
validate if the key matches for the guest info and then send updates.
2.validate if guestos and interface info not empty before updating the vmi status
What this PR does
Before this PR:
IP address in vmi status disappear and re-appear frequently
After this PR:
IP address stays constant in the vmi status once assigned to an interface.
Why we need it and why it was done in this way
The following tradeoffs were made:
The following alternatives were considered:
Links to places where the discussion took place:
https://github.com/kubevirt/kubevirt/issues/12698
Special notes for your reviewer

Checklist
This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

[ ] Design: A design document was considered and is present (link) or not required
[ ] PR: The PR description is expressive enough and will help future contributors
[ ] Code: Write code that humans can understand and Keep it simple
[ ] Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
[ ] Upgrade: Impact of this change on upgrade flows was considered and addressed if required
[ ] Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
[ ] Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
[ ] Community: Announcement to kubevirt-dev was considered

Release note

/cc @Acedus


ES storage improvement/refactoring

Use the same ID generation for ES in event data
Reuse RestClient instance in ESLEvents
Replace 0 with false in mapping
Set Content-Type as application/json

LGTM. I'd like to suggest us to get into the habit of filing JIRA tickets for all changes. It will make the release notes writer easier to figure out what's been changed.


Match posts that only include multiple hashtags (#x AND #y)

Hi,
Background: I've found online a list of popular hashtags to use with #starwars, such as #disney or #toys but then I would not want to download just #disney or just to download #toys... and just #starwars by itself is like 100,000 images a day, which is almost impossible to go through unless I look at posts through a different eye. I do not want to look at images of Mickey Mouse all day, unless it is a stormtrooper with Mickey Mouse ears :)
I don't know if the Instagram service used in the script would allow the implementation of a multiple hashtags "AND" function such as
"Download images from posts that have both #y AND #x", but exclude posts that have only #y or only #x?
The default folder name could be #y-#x to match the parameters passed to the function.
A work around could be to query the posts one by one that have the first hashtag in the list (#y), but BEFORE downloading the image, look into the post to see if all the other #hashtags are there (#x), and only download/save those that match all the hashtags (#y AND #x).
Just a thought, maybe there is a better way to get to it.
Thanks

Hi Matthieu,
as far as I know, Instagram does not have the option to directly show posts that match multiple hashtags. Also, since hashtags might occur in comments rather than captions, simply iterating through one hashtag's posts and analyzing their metadata won't suffice. However, the following code snippet queries multiple hashtags and then downloads the posts that were returned in all of them.
It requires the Instaloader.get_hashtag_posts() method from commit 58882f5, which is included in the yet-experimental branch v3-dev.
from instaloader import Instaloader

def download_posts_with_multiple_hashtags(instaloader, hashtags, max_per_hashtag):
    posts = []
    print("Getting posts with hashtag {}".format(hashtags[0]))
    for count, post in enumerate(instaloader.get_hashtag_posts(hashtags[0])):
        posts.append(post)
        print(count + 1, sep='', end='\r', flush=True)
        if count >= max_per_hashtag - 1:
            break

    for idx, hashtag in enumerate(hashtags[1:]):
        prev_posts = posts
        posts = []
        print("\nGetting posts with hashtag {} and {}".format(hashtag,
                                                              ','.join(hashtags[:(idx + 1)])))
        for count, post in enumerate(instaloader.get_hashtag_posts(hashtag)):
            if any(p['shortcode'] == post['shortcode'] for p in prev_posts):
                posts.append(post)
            print("{}, {} matching".format(count + 1, len(posts)), sep='', end='\r',
                  flush=True)
            if count >= max_per_hashtag - 1:
                break

    if posts:
        print("\nDownloading posts with hashtags {}".format(','.join(hashtags)))
        for count, post in enumerate(posts):
            print("[{:03d}/{:03d}] ".format(count + 1, len(posts)), end='', flush=True)
            instaloader.download_post(post, target=','.join(hashtags), profile=None)

loader = Instaloader(sleep=True, filename_pattern='{date}')
try:
    download_posts_with_multiple_hashtags(loader,
                                          hashtags=['starwars', 'disney', 'toys'],
                                          max_per_hashtag=2500)
except KeyboardInterrupt:
    pass

Since there are very many posts associated with one hashtag, the function has a max_per_hashtag parameter, and only that many posts are considered for each hashtag. Also, sleep=True seems to be absolutely required since this code issues many requests to the Instagram database server within a short time period. I got banned for a short period while testing this code, and I fear that the current sleep intervals are too short for this request pattern.
As mentioned before, you need to install Instaloader from our v3-dev branch.
$ pip3 install --upgrade git+https://github.com/Thammus/instaloader@v3-dev

( Paste above snippet into a file )

$ python3 (code-snippet.py)

The parameters you might want to change (hashtags, max_per_hashtag and the parameters in the Instaloader() constructor) are within the last few lines of the code.
Happy instaloading!
Alex

Thanks for taking a shot at it. I just tried it and I do not feel I'm getting everything. For #starwars' #disney and #toys it only saved 2 images with the 2500 max value. It felt short so I tried with another example from the 9th image I saw when searching #starwars.
I manually found that post from 5 hours ago that has #starwars and #civilwars in it
https://www.instagram.com/p/BXqVxxXBVZC
So I ran the script for the two hashtags 'starwars' and 'civilwars' but it only saved ONE image from 10 hours ago which is different from the one above
https://www.instagram.com/p/BXqVxxXBVZC
Tried again with this post that has #C3PO and #LegoC3PO posted 34 seconds ago... the script saved nothing.
While I feel very confident using the basic instaloader with only one hashtag that it will grab everything I missed since my last run and would stop when it hits the last saved post (looooovvveee it!), theoretically, it feels restricting that there is a max_per_hashtag value associated with the first hashtag. I tried putting 10,000 but then nothing happens and no images are saved, but even 10,000 is not enough.
Not sure if anything else is possible around the max_per_hashtag.
Side question: I assume the match function is not case sensitive right? and it would not matter if the hashtags are written with a space or not? "#C3PO#LegoC3PO" vs "#c3p0 #LEGOc3po"
Thanks!

Hi Matthieu,
the new --only-if option might help here. It allows to define a condition that each post has to meet to be downloaded. We can configure Instaloader to load all posts of one of the hashtags, that contain the other hashtags in their caption like this:
instaloader --fast-update --only-if='caption and "#cute" in caption.lower()' '#kitten'

Happy instaloading!
Alex

I was able to make that one work
instaloader --only-if="not is_video" #starwars
but what was mentioned above does not work
instaloader --fast-update --only-if='caption and "#cute" in caption.lower()' '#kitten'
I get this error
Traceback (most recent call last):   File "c:\python\scripts\instaloader-script.py", line 11, in <module>     load_entry_point('instaloader==3.0rc0', 'console_scripts', 'instaloader')()   File "c:\python\lib\site-packages\instaloader.py", line 1429, in main     args.profile_pic_only, args.fast_update, args.stories, args.stories_only, args.only_if)   File "c:\python\lib\site-packages\instaloader.py", line 1222, in main     filter_func = filterstr_to_filterfunc(filter_str, username is not None)   File "c:\python\lib\site-packages\instaloader.py", line 152, in filterstr_to_filterfunc     compiled_filter = compile(TransformFilterAst().visit(ast.parse(filter_str, filename=input_filename, mode='eval')),   File "c:\python\lib\ast.py", line 35, in parse     return compile(source, filename, mode, PyCF_ONLY_AST)   File "<--only-if parameter>", line 1     'caption            ^ SyntaxError: EOL while scanning string literal
please advise

This one works... the double quotes had to be outside and the single quotes inside
instaloader --fast-update --filename-pattern={profile}_{date} --only-if="not is_video and '#cute' in caption.lower()" #kitten

it is not a perfect hashtag match though it matched #cuteness instead of just #cute... that's enough for my need anyways. Thanks

I get this error
instaloader --fast-update --filename-pattern={profile}_{date} --only-if="not is_video and '#georgelucas' in caption.lower()" #starwars
[  2] #starwars
[  2] #starwars
[  2] #starwars
[  2] #starwars
[  2] #starwars
[  2] #starwars
[  2] #starwars
[  2] #starwars
[  2] #starwars Traceback (most recent call last):
File "c:\python\scripts\instaloader-script.py", line 11, in
load_entry_point('instaloader==3.0rc0', 'console_scripts', 'instaloader')()
File "c:\python\lib\site-packages\instaloader.py", line 1431, in main
args.profile_pic_only, args.fast_update, args.stories, args.stories_only, args.only_if)
File "c:\python\lib\site-packages\instaloader.py", line 1249, in main
filter_func=filter_func)
File "c:\python\lib\site-packages\instaloader.py", line 1051, in download_hashtag
if filter_func is not None and not filter_func(post):
File "c:\python\lib\site-packages\instaloader.py", line 157, in filterfunc
return bool(eval(compiled_filter, {'post': post}))
File "<--only-if parameter>", line 1, in
AttributeError: 'NoneType' object has no attribute 'lower'

And this error... that one went on for a minute before crashing
c:\python\scripts\instaloader --fast-update --filename-pattern={profile}_{date} --only-if="not is_video and '#disney' in caption.lower()" #starwars
[ 48] #starwars
[ 48] #starwars
[ 48] #starwars
[ 48] #starwars
[ 48] #starwars
[ 48] #starwars
[ 48] #starwars
[ 48] #starwars
[ 48] #starwars Traceback (most recent call last):
File "c:\python\scripts\instaloader-script.py", line 11, in
load_entry_point('instaloader==3.0rc0', 'console_scripts', 'instaloader')()
File "c:\python\lib\site-packages\instaloader.py", line 1431, in main
args.profile_pic_only, args.fast_update, args.stories, args.stories_only, args.only_if)
File "c:\python\lib\site-packages\instaloader.py", line 1249, in main
filter_func=filter_func)
File "c:\python\lib\site-packages\instaloader.py", line 1051, in download_hashtag
if filter_func is not None and not filter_func(post):
File "c:\python\lib\site-packages\instaloader.py", line 157, in filterfunc
return bool(eval(compiled_filter, {'post': post}))
File "<--only-if parameter>", line 1, in
AttributeError: 'NoneType' object has no attribute 'lower'

Add a clause to check that caption is not None before (i.e. on the left side) caption.lower() is evaluated, i.e.
instaloader --fast-update --filename-pattern={profile}_{date} --only-if="not is_video and caption and '#disney' in caption.lower()" #starwars

(and caption before '#disney' in caption.lower())
Otherwise it fails whenever a post does not have a caption.

Achievement unlocked! Works like a charm!
Thanks a bunch!

I am glad we could find a well-working solution for this problem :) Thanks for your patience and cooperation. Sharing how Instaloader is used and how its limits are reached is a significant contribution, since issues like these help making Instaloader great :)
Now I added the caption_hashtags property and closed this issue.
Happy instaloading!
Alex

Hello Alex,
Firstly, the contribution you put to get data from IG is such an outstanding performance, I appreciate it. Thank you so much!
I actually run your code to get posts from Instagram through multiple hashtags. However, once finished, I couldn't find the posts saved on my PC. Where were those posts saved on the computer, how can I find them?
Thank you!


Add future time limit iter API

Closes #14
This also refactors future time matching.
Example:
use saffron::Cron;
use chrono::prelude::*;

let cron = "*/10 * * * *".parse::<Cron>().expect("Couldn't parse expression!");
let start = Utc.ymd(1970, 1, 1).and_hms(0, 0, 0);

// effectively the same as iter_from
let _ = cron.clone().iter(start..);

// all matching times in the next 30 minutes
let _ = cron.clone().iter(start..(start + chrono::Duration::from_secs(60 * 30)));

WIP: Needs testing (originally API had eyeball testing, but new API will allow for much easier testing)

Hi! Quick question on the overall API design before I look into the implementation. Is there a reason you can't just use take_white? IMO it'd be nice to reuse the standard iterator methods, and you could achieve the goal of getting a finite number of next trigger timestamps.
use saffron::Cron;
use chrono::prelude::*;

let cron = "*/10 * * * *".parse::<Cron>().expect("Couldn't parse expression!");
let start = Utc.ymd(1970, 1, 1).and_hms(0, 0, 0);

// all matching times in the next 30 minutes
let thirty_minutes = chrono::Duration::from_secs(60 * 30);
let _ = cron.clone().iter(start).take_while(|t| t < start + thirty_minutes);

We coouuullldd but to me it's nicer to read and understand iter(start..(start + Duration::minutes(30)). Plus:

With some crons, we may have to calculate a far future just to realize it's way out of bounds like 0 0 29 2 *. We have to calculate up to the next leap year, when we could've checked a while ago and saw "yeah it's out of bounds".
I want to support reversing the iterator at some point (simply for personal reasons) and this design requires no other changes to implement that besides implementing DoubleEndedIterator.


[New game]: Want to add the game catch the objects

🎮 Game Request
Game logic and basic description
In this game you have to catch the object falling from the top
Point down the features
Game points
It consist of a circle object
you just need to catch that
Select program in which you are contributing
GSSoC24
Code of Conduct

[X] I follow CONTRIBUTING GUIDELINE of this project.

/assign

Hey @Ayushmaanagarwal1211 !
Thank you for raising an issue 💗
You can self assign the issue by commenting /assign in comment 😀
Make sure you follow CODE OF CONDUCT and CONTRIBUTING GUIDELINES 🚀
Don’t Forget to ⭐ our GameZone🎮
Make sure you join our Discord🕹

Hey @Ayushmaanagarwal1211 ! Thank you so much for your raising the issue💗
It’s all yours, you can come anytime again and make some contributions! 🚀
Alone, we can do little, but together we can do so much! 😇


[wip] pdf generation

This is the current state of the pdf work. It's pretty raw.
I am trying ghost.py instead of phantom: it's a more robust, as it doesn't bundle all of qt, but more about that below, and gives us better access to the underlying components of qt. it's error handling is not as good as phantomjs, though.
It stands up a temporary tornado server, and uses a long task outside the event loop which then kills the event loop when done.
I am getting some fascinating js errors that are heisenbergingly absent when run "full-local", i.e. no network access, but i am slowly but surely finding my way there through a mountain of undefined(0): undefiend is not a function-style errors.
Features

[X] multipage PDF generation
[X] attach raw ipynb to PDF
[ ] per-page orientation, size
[ ] easily install (work covered by Anaconda-Server/Anaconda-Notebook#43)

[X] pyqt (seems pretty good)
[ ] ghost.py
[ ] pypdf2

[x] in-notebook app download
[x] verify third-party integration

[X] bokeh
bqplot works fine in app, but is made of widgets...
[x] qgrid (master?)

[ ] "true" offline generation (can't help what third-parties do) - may be deferred

Roadblocks
I am not trying to hit the widgets yet! That will probably have to wait until pyqt5 is usable with ghost

Here's the kind of thing I'm getting out right now :)
output.pdf

The loading problem is due to baobab... maybe something isn't shimmed properly...

Yep. The issue was creaky old PyQt's webview didn't have the Function.prototype.bind defined, which was exploding in ghost. Would have happened over in phantom, too, though I might have found it much more quickly.
Output is getting a lot better:
output.pdf
I am going to do some other tests to ensure it's not cheating, but I'm confident it will be demoable now.

The examples are looking better!

Ok, really pretty ready to be done with this. nbbrowserpdf is in -c nbcio. I'm going to merge this, and we can pick up the shiny pieces if the nbpresent build is not good.


running through Falcon RTR does not provide the expected results

Hi,
We have ran the script though Falcon RTR on few vulnerable servers and did not detect anything.
The reason in every case was that the library is not present in the same folder as the executable, (e.g. .exe in \bin folder, .jar in \lib folder)
Is there a possibility to run the script through RTR with our own arguments, to scan a specific folder? It will be very useful.
According the manual there is an option to fill -CommandLine="", but I could not find its usage in the PS script...
Thanks,
Borut

Identifying vulnerable libraries is not dependent on the placement of the binary, nor even on the script wrapper.
If you're using the PS1 wrapper, it attempts to serve as a one-size solution by enumerating running processes and known configuration locations to select paths to scan. As such, it can miss latent JAR files, particularly those just placed in a semi-arbitrary location for testing.
You may invoke the cast.exe executable directly via RTR's run command and give it one or more paths to scan, or you may drop your own PS1 that invokes it against, say, C:\lib and execute that via runscript.

Already tried running cast.exe directly, am I missing something?
C:\temp> run cast.exe scan c:\temp
Max args is 1. 3 were provided
Error code: 40012

I wasn't aware of that limitation with the RTR run verb; it looks like you'll have to drop a dedicated script for your testing (or place your known-vulnerable JAR in the installation path of a running Java application).

I've asked that the RTR documentation be updated, but there is actually a correct way to solve this with run:
run C:\cast.exe -CommandLine='scan C:\path\to\malicious.zip'

Thanks for the effort, successfully ran, but unfortunately the command does not give any output in the RTR console :(
Will have to go with a Powershell script after all...


[BUG]: Code versioning, cache invalidation, and git sha commit

System Information
ZenML version: 0.20.5
Install path: /opt/anaconda3/lib/python3.9/site-packages/zenml
Python version: 3.9.7
Platform information: {'os': 'linux', 'linux_distro': 'ubuntu', 'linux_distro_like': 'debian', 'linux_distro_version': '22.10'}
Environment: native
Integrations: ['github', 'huggingface', 'pillow', 'plotly', 'pytorch', 'pytorch_lightning', 'scipy', 'sklearn']
What happened?
I have a (simplified) step function like this :
@step
def json_file_loader() -> Output(corpus=dict):
    return toto

def toto():
    with open(params.filename,"r", encoding="utf-8") as f:
        return {"corpus":json.load(f)[0,10]}

If I change in the toto function 10 to 20, the pipeline cache is not invalidated.
It works if I the complete code is in the step function :
@step
def json_file_loader() -> Output(corpus=dict):
    with open(params.filename,"r", encoding="utf-8") as f:
        return {"corpus":json.load(f)[0,10]}

Moreover, If I then commit the code to git, the pipeline cache is still not invalidated.
Code of Conduct

[X] I agree to follow this project's Code of Conduct

As a work-around, I could extract git commit, and push it into a BaseParameter class.

Correct, ZenML caching currently works by inspecting the parameters of the step as well as the source code of the step function. Detecting and inspecting all the functions that a step calls as shown in your example is not implemented as that would be quite complicated and require parsing of the AST.
We've considered using the git sha as the caching indicator as well, but it comes with its own limitations:

If we simply use the HEAD commit hash, then modifying a single unrelated file will disable caching for all your steps
If we use the commit hash that last modified the file which contains the step, we will have a similar issue that you described above (I just separated the step definition and helper function into separate files):

# my_step.py
from .utils import helper_function

@step
def my_step() -> None:
    helper_function()

# utils.py
def helper_function() -> None:
    # if we modify this, how to disable caching?
    ...

Do you have any concrete suggestions how we could improve the cache detection?

You are right that AST parsing is probably not the solution, especially with sadists like me who fancy having dynamic calls (through getattr ), and dynamically tracing the code is no solution either.
I woud like the git commit hash approach, if I could define which folders to watch (maybe pipelines and steps by default).

An alternative could be to declaratively specify dependency functions to checksum and track in a new step parameter)

I agree, no matter if we go with git sha or the source code to check if code has changed, we could definitely make it customizable as you described. I'll discuss this internally if the rest of the team agrees and when we can get to it.

@ghpu Perhaps we can shift these ideas to this issue on the roadmap made by @jsuchome https://zenml.hellonext.co/p/configure-caching-behavior-based-on-step-input-output
Maybe  we can combine and consolidate it there and close this issue?


Update tutorial on REST API Proxy

Resolves https://github.com/wso2-enterprise/choreo/issues/14212

Closing this because we are rolling out the new documentation structure and there might be conflicts. I will resend the PR and provide a link to the new one here.


Flange to be added to the ConnectorKindType enumeration

For describing the interface between ODU and antenna, the "FLANGE" shall be added to the ConnectorKindType identity.

Proposal to the 5G-xhaul call on 13th of December 2023 :
The value "FLANGE" shall be added to the ConnectorKindType identity.

UML changed as decided.

Has been updated and published in EquipmentAugment_1.0.0-tsi.240104.1330+yang.1.zip


Normal and Anomalous Labels

Hello Houssam,
First of all I'd like to thank you for sharing the code and the amazing paper you and your fellow researchers produced. I have a few questions regarding the implementations, it would be great if you could take some time and look at these.

In kdd.py, it was assumed 1 for anomalous label and 0 for normal label throughout the code, but in the function _get_dataset the anomalous data was assigned with label 0 and the normal 1 (lines 46-47), which confused me, these two lines also contradict with the comment in line 128. I did reverse the label in these two lines (keeping other settings intact, cross entropy loss, L1 norm in reconstruction loss used) and the F1 score dropped drastically. Please correct me if I was wrong.

In BiGAN_run_kdd.py when you are evaluating the anomalous data in the test set, you used the fact that 20% of the test data are anomalous and thus assigned anomalous label to the data with top 20% scores in test set (lines 319-328). Is that allowed in anomaly detection? I think an anomaly detection algorithm should be able to detect unknown anomalies.

Thank you for your contribution, it's an exciting work!
Zheyu

Hi Zheyu,
Thank you for your interest in our research! It's nice to have your feedback!

As referred to the original DAGMM paper (Zong et al, 2018) and the DSEBM paper (Zhai et al, 2016) the labels are switched for this anomaly detection task.  This is why you may have been confused...

Yes indeed. This is an interesting question as in general, we found that unsupervised anomaly detection may be very challenging. How can one define a threshold for the anomaly score without any knowledge? This still remains a question very few recent paper tackled. So what we did was reproduce the same experimental setup as the DAGMM paper (Zong et al, 2018) and also assumed to have this knowledge on the test set.
You can think of it as if you have the knowledge of what is the proportion of people having a certain disease in a population, should you do anomaly detection on medical images or any health application...

If anyone has ideas for this challenge in unsupervised anomaly detection, please suggest, this could be a nice contribution in any future research actually!

Hi Houssam,
Thank you for replying to my questions.
Yes indeed the normal data are assumed anomalous class in this task as suggested in the two papers to make the ratio of anomalies roughly 20% of the whole data, however no further justification was found in both papers of why flipping the labels other than making the anomalies a minority compared to normal labels. And the DSEBM paper shows experiments with varying ratio of anomalous data will have poorer performance when that ratio is high. But I guess the algorithm trained on swapped label can still be used to predict intrusions in KDD99 since it classifies attack/non-attack at the end.
As for the second question, the training data are actually attacks which are supposed (only my guess) to be more different from each other than non-attacks are. So I don't know whether this will impose problems if we just pick say the 95 percentile of the training scores as the threshold as the scores can be very different?

Thanks for your questions!
Regarding your questions about the experimental setup, we again followed the same setup as the one in the "DAGMM" original paper. I suggest to also contact them and have their opinion. They replied to some of our questions.
Apart from that, to set a threshold from a practical point of view, I think the experimental setup which is fully unsupervised looks a bit unrealistic. What would be more likely in an industry application would be probably to have a lot of normal training data and very few labeled samples containing anomalous data. With these data you could actually train your network without the labels but create a validation (containing both normal and anomalous data) set with few labels to choose a certain threshold (for example, maximizing the F1 score? maximizing any other metric which can be more relevant to your context?). However, do note this setup would not be fully "unsupervised", yet still more "unsupervised" than some other "semi-supervised" anomaly detection techniques.
As those questions are no more related to the code itself, I'll be glad to answer to them in separate email threads! Please do not hesitate to include the other authors and myself in the thread so that we can all have a better idea of the questions and give our opinions.
Houssam


Sasha/improvements

add image name to dataset get_item function, for easy tracking of train/test split

Closing following @datumbox comment. Additionally, this also breaks the scripts for when training on COCO, so we won't be able to merge this


Allow logo to be cache busted

Our new logo's URL is hardcoded, which doesn't allow us to leverage our beatiful asset hashing for cache busting, such as /assets/images/logos/hc2017-logo-horizontal-color.c53d2942.svg.
Simple fix!
@varkor could you review?

Great! 👍 Looks good.


Add KeyActions enum

Pull Request
Related issue
Fixes #418
What does this PR do?

add KeyAction enum with all supported cases and .unknown(String)
extends KeyAction with an initializer with String
extends KeyAction with a String representation of the enum value
adds initializer with [KeyAction] instead of String for actions to KeyParams
extends Key&KeyParams with value “enumActions” containing the actions as KeyAction representation

PR checklist
Please check if your PR fulfills the following requirements:

[x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
[X] Have you read the contributing guidelines?
[X] Have you made sure that the title is accurate and descriptive of the changes?

This is awesome, thanks! Will give it a full review shortly and look to get both merged in (in preparation for a release once a few more breaking changes are compiled).


Feature request: no-cli

It would be great if we could've a CLI for nocode, which would do nothing.

But, what would --help say?


[ArPow] Remove xliff-tasks CheckEolTargetFramework option

Once xliff-tasks removes its 2.1 dependency (https://github.com/dotnet/xliff-tasks/issues/508), the /p:CheckEolTargetFramework=false build arg should be removed - https://github.com/dotnet/installer/blob/release/6.0.1xx/src/SourceBuild/tarball/content/repos/xliff-tasks.proj#L6

This was fixed with https://github.com/dotnet/source-build/issues/3081


HLS Subtitles

Newer versions of the HLS specification define how a webvtt subtitle can be embedded inside a hls stream.
What I'm looking for is a solution to do this with the current technology available.
AFAIK this is not quite possible to do with what we've got so far so my question is weather this feature will be available and if so when.

:+1:

+1


Can we compute reference_chosen_logps, reference_rejected_logps offline?

Is there a reason why we're not computing reference_chosen_logps, reference_rejected_logps using the reference_model offline? Wouldn't be faster than dematerializing/remat the model in case of FSDP?

You're right, this would save some compute & memory, though it would slightly complicate the DPO pipeline (adding a pre-caching step). I'm not sure how significant the speedup would be in practice (since the forward/backward of the policy we have to do in either case, and the backward pass is the bottleneck anyway).
Only way to find out for sure is to try it! Wouldn't be too terribly difficult to do, if you want to open a PR :) happy to give feedback if you'd like to give it a go- not sure I'll get to this in the near future myself.

I'll close this for now, but feel free to continue the discussion/re-open if you'd like.

Thanks, I'll give it a go when I've got time 🙌


Can nodes in a swarm have different versions of docker-ce ?

I am getting "remote CA does not match fingerprint" when trying to join a new node with latest docker-ce version. Tried installing old version but not all components matched (i.e. containerd.io) and it still didn't work.

This repo is not for Swarm, please use https://github.com/docker/swarmkit


Refine Multi-cluster unit tests to remove unnecessary dependency

Refine unit test of multicluster antctl helper to remove unnecessary dependency
Add more unit tests for antctl/raw/multicluster/deploy package

Resolves #4689

/skip-all


cleaner: rewrite node shebangs

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.

[x] Have you successfully run brew typecheck with your changes locally?
[x] Have you successfully run brew tests with your changes locally?

Fixes https://github.com/Homebrew/homebrew-core/issues/176257
Closes #17539
This rewrites node shebangs during cleanup, as we do for perl
$ brew install -s typescript && head -1 $(which tsc)
#!/opt/homebrew/opt/node/bin/node

Not sure if we need tests here? rewrite_shebangs doesn't have any yet, happy to try to add some

Cautiously approve but this will affect a ton of formulae so maybe worth holding until after the next Monday release (already got a lot of other big changes on master that's taken a while to stabilise).

Agreed 👍🏻

how do we feel about merging this now? I'm rebuilding node formulae for https://github.com/Homebrew/brew/pull/17867 and having this would help us catch any potential issues quickly

There will likely be a follow up release tonight with https://github.com/Homebrew/brew/pull/17902 (if approved, else tomorrow).

Thanks @branchvincent, good to go now.


tree_implementation parameter gets overwritten with onnx backend

Hello,
I have been trying to use the gemm with onnx backend but the onnx graph did not really look like we were using gemm tree implementation:
model = convert(skl_model, backend="onnx", test_input=X, extra_config={"tree_implementation":"gemm"})

It seems that the tree_implementation parameter gets overwritten deeper in the code:
        if backend == onnx.__name__:
            # vers = LooseVersion(torch.__version__)
            # allowed_min = LooseVersion("1.6.0")
            # Pytorch <= 1.6.0 has a bug with exporting GEMM into ONNX.
            # For the moment only tree_trav is enabled for pytorch <= 1.6.0
            # if vers < allowed_min:
            extra_config[constants.TREE_IMPLEMENTATION] = "tree_trav"

I am not sure why this is done tho? Removing this part gives me back the expected behavior.
Could you give a bit more info on what is the point of this? Can I do anything to fix this?

Yes it could be that with the newest version they fixed the problem. @jfrery we had to pin the tree implementation for onnx because of a bug on onnx runtime. Can you please share your onnx + onnxruntime versions? We can try to remove the check and see if it works in our tests.

Thanks for the info. Here are the versions I am using:
onnx version: 1.10.2
onnxruntime version: 1.10.0

If I can be of any help here I would be glad to help.

Yes, we love contributions! So if you want to submit a PR which removes the pin of the tree model to tree_trav it would be great!
One thing is, can you try to do an investigation on which version of pytorch breaks the conversions? You can leave your onnx dependencies as is, and try to downgrade pytorch until the conversion using gemm returns an exception. In your PR you can then uncomment the line checking the version of pytorch, and set it to the minimum version that works, while for the earlier version we can continue to force it to use tree_trav. Does it make sense?

Yes of course I will prepare that. One question: the check is on the backend==onnx and not pytorch. I will replace that by a check on pytorch version and set it to the minimum version that works. However, I think it  is better to raise an exception rather than overwrite the parameter given by the user if the pytorch version is too low.
Does this sound right? I will have a PR soon.

Is there a reason why this line take in the onnx_ml_model rather than model:
https://github.com/microsoft/hummingbird/blob/2705c42585ac1d3ee69e65a93e5f59d7d42f44d5/tests/test_onnxml_decision_tree_converter.py#L37
I am not sure what we are trying to test by sending the onnx_ml_model here.
It seems that this test is breaking with tree_implementation="gemm" with the following error:

Test error
./test_onnxml_decision_tree_converter.py::TestONNXDecisionTreeConverter::test_decision_tree_regressor Failed with Error: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Non-zero status code returned while running Transpose node. Name:'Transpose_17' Status Message: perm: [ 1 0 ] does not align with rank of input data: 1
  File "/usr/lib/python3.8/unittest/case.py", line 60, in testPartExecutor
    yield
  File "/usr/lib/python3.8/unittest/case.py", line 676, in run
    self._callTestMethod(testMethod)
  File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod
    method()
  File "/home/dev_user/hummingbird/tests/test_onnxml_decision_tree_converter.py", line 92, in test_decision_tree_regressor
    self._test_regressor(X, model, extra_config={"tree_implementation":"tree_trav"})
  File "/home/dev_user/hummingbird/tests/test_onnxml_decision_tree_converter.py", line 66, in _test_regressor
    onnx_ml_pred, onnx_pred, output_names = self._test_decision_tree(X, model, extra_config)
  File "/home/dev_user/hummingbird/tests/test_onnxml_decision_tree_converter.py", line 54, in _test_decision_tree
    onnx_pred = onnx_model.predict(X)
  File "/home/dev_user/hummingbird/hummingbird/ml/containers/_sklearn_api_containers.py", line 119, in predict
    return self._run(self._predict, *inputs)
  File "/home/dev_user/hummingbird/hummingbird/ml/containers/_sklearn_api_containers.py", line 67, in _run
    return function(*inputs)
  File "/home/dev_user/hummingbird/hummingbird/ml/containers/sklearn/onnx_containers.py", line 216, in _predict
    return np.array(self._session.run(self._output_names, named_inputs))[0].ravel()
  File "/home/dev_user/hummingbird/env_hum/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 192, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Non-zero status code returned while running Transpose node. Name:'Transpose_17' Status Message: perm: [ 1 0 ] does not align with rank of input data: 1

But there the convert function is actually working fine so I am a bit lost here.

These tests are a little bit confusing due to sooo many different types of conversions.  (We should add more comments to our tests!)
There are two different types of conversions we can do:

converting scikit-learn into onnx (vectorized with HB)
So in this test, we would just convert model as you mentioned, which works as expected.  This was your original use case!

converting  onnx-ml (meaning, traditional onnx) into onnx (vectorized with HB)
To derive this onnx-ml model, we first create model from scikit-learn as with the first test, but then we convert_sklearn (a function from onnx tools, not HB) into onnx_ml_model on line 32 to get an onnx-ml model to test with.  We found this to be the easiest way to create these traditional onnx-ml models in a consistent way.  It seems that this is where we hit the bug with gemm.

But thinking about this a bit more, the code at the very beginning:
model = convert(SOME_MODEL_HERE, backend="onnx", test_input=X, extra_config={"tree_implementation":"gemm"})

should work fine if we are converting skl -> onnx, and only break with  onnx-ml -> onnx.  Maybe we can add an additional check if SOME_MODEL_HERE is coming from onnx-ml or not.
@interesaaat thoughts?

It looks that this error is coming from the onnxruntime. To bypass this, we can force onnxml models to not use gemm, and open an issue regarding this.
You can extra_config[constants.TREE_IMPLEMENTATION] = "tree_trav" somewhere in this method. In this way sklearn models will use gemm (eventually), while onnxml models will stil use tree_trav until we understand where this new error is coming from.


[Question] Is it possible to run llava-v1.6-mistral-7b with transformers?

Question
Coming from https://huggingface.co/liuhaotian/llava-v1.6-mistral-7b I tried to make it work in Google Colab, but without any success. Could you please tell me if it is possible to run it with transformers and show me the code snippet for it please? Or should I use llava-hf/llava-v1.6-mistral-7b-hf ?

Second this

https://github.com/haotian-liu/LLaVA/pull/1115


[UI][RulePad] the position of the star icon in RulePad causes errors.

When the star icon is not displayed, only the checkbox and the eraser icons are displayed.

However, when star icon is displayed, the position of the checkbox and the eraser icons are changed. When there is a delay in clicking the checkbox, this update in location causes the error for users when they want to click on checkbox, but they click on the eraser icon.

Suggested fix: move the star icon before the other icons to keep the position of the icons.

Yeah, I can look into the issue and move the star to the other side rather than the last.


add protobuf to the list

example: https://github.com/protocolbuffers/protobuf/issues/8460

Found it: https://github.com/protocolbuffers/protobuf/blob/bc80135f10ca2ad6f307d4a2165cdaa112d68555/.github/workflows/janitor.yml#L32


DNS Link 404's

#5597  Expected content
Link to more content when clicking on "The SendGrid Email Infrastructure Guide"
Actual content
"Quoth the Raven - 404"
https://github.com/sendgrid/docs/blob/develop/content/docs/glossary/dns.md

Hi @bobzilladev ,
We're sorry you struggled with this issue, and we appreciate the feedback! We've added this to our internal docs backlog and will resolve the issue as soon as possible.
Twilio SendGrid Docs Team


feat: support to run test via YAML editor

feat: add syntax support
feat: support to run test case directly

see also https://github.com/LinuxSuRen/api-testing/pull/28


Blank page will somehow show up when webview tag loads a url at first time.

Electron Version: 1.6.1-beta
As titled, and the screenshot attached below:

This problem happened to lulumi-browser when I upgraded electron to version 1.6.1-beta.

Hi @qazbnm456, could you please upload screenshot one more time? It doesn't load now for some reason.

@tonyganch Sure!

I think the problem has something to do with Vue.js, especially Vuex. The problem could happen if I assigned that initial page object in an early time (maybe even before render processing of Electron).
From the fix, I moved the assignment statement to beforeMount event (inside this.$store.dispatch('createTab')), and the problem has vanished away.
But, still be eager to create a repro to see what actually happens!


User might not have language configured

When using a database from the very early days of XCMailr, it might happen that some users don't have any language configured.
Obviously, any call to user.getLanguage() simply returns null and needs to be handled appropriately. Hence, we need to search our codebase for such call sites and make them null-safe.

Fixed in develop branch. Will be part of next release.


Error: Unable to Exec

I'm receiving an error "Error: Unable to Exec" when trying to upload via SFTP.
remote-sync seems to connect and authenticate okay, then displays the Error: Unable to Exec message in red. The SFTP connection works fine in other clients like Transmit.
Strangely, I also have remote-sync working fine on the same host (WP-Engine) in another account. The settings are very similar.
Is there a way of getting a more detailed error message to try to understand what is going on? Or is this a known issue? DIdn't see reference to it. Thanks!

Just out of curiosity, what is your Port #?

I have the same issue. Also using WP-Engine for hosting...  using "port": 2222
If I try to use port 22 I get:
Error: Authentication failure. Available authentication methods: publickey

If I use port 2222 I get (note replace <IP_ADDRESS> with my WPEngine IP) :
[10:12:36 AM] Connecting: username@123.123.123:2222
[10:12:39 AM] Upload: /Users/chris/Sites/test.md to /test.md ...
Error: Unable to exec

To start the upload I right click on the file and choose "Upload file"
Using Atom v.1.0.0 on OSX Yosemite 10.10.3.
{
  "transport": "scp",
  "hostname": "<IP_ADDRESS>",
  "port": 2222,
  "username": "username",
  "password": "mypassword",
  "target": "/",
  "ignore": [
    ".git/**"
  ]
}

@chrisjacob it seems that port 22 is the right port. And you should config keyfile path in .remote-sync.json according to your error msg.
So Error: Unable to exec means port wrong?

Was a fix implemented for this @yongkangchen?  Or is it not a bug in the plugin?
I get this error as well although there are no extra error messages.  The port is fine as other SFTP clients can connect and I've never used a key file on any of the servers we use.
Thanks

Same issue; WP-Engine, Using Port 2222; SFTP. Download works; upload leads to the error: "unable to exec"

+1 for the same exact problem on WPEngine, port 2222.

Same exact issue with me on 2 WP-Engine accounts that I own. When I plug these same settings into my FileZilla client, it works perfectly.

Same issues for me to
this board is closed, but it seems that people are still facing this issue?
I use Prepros.io to watch for project folder changes folder, which normally works nicely.. but with WP-engine as host I get with the following error when uploading :
: Failed to create Remote Directory /path/foo/....  Unable to exec
However in Filezilla with the same credentials and port 2222, everything works fine.

Same issue with WPEngine on port 2222

I'm also having this problem but on Rackspace Cloud Sites.

Same problem to connect squarespace developer port. 2030

Possible fix for this issue mscdex/ssh2#427

Also have this problem on Adobe's Business Catalyst

Have the same issue on Windows platform using if I try to upload file.
"transport": "scp",

How can I resolve

I have the same issue when trying to upload a file to Squarespace. Port 2030.
It authenticates correctly, but when it begins to upload I receive this error: "Error: Unable to exec"
On the flip side, I'm able to download files with no issues.

I just got off phone with WP Engine and they were not able to determine why this is failing on their end.
Does anyone have any other insight here? @patrickgilmour you mentioned that you had the remote-sync working on another WP Engine account - would you be able to provide the example settings (with username/password details removed) here so we could further diagnose?
I am setting up a support ticket with WP Engine about this and any further information would be helpful.

@derekcavaliero - Sorry! I haven 't used remote-sync for quite some time and don't have the settings file any more.

+1 for this exact issue. Would love to see a fix.

I am having the same issue, but not seeing much by way of support by the authors. No problem: free and all of that.
Does anyone know of an Atom package like this that works?

Same issue here, port 2222, sftp.

Use Webstorm, luke!

Nothing here still? I am having the exact same issue...

Do not use Atom! Use WS!

what's ws?

Webstorm!

I also have this issue with WP Engine.. port 2222 SFTP

@hellboy81 I actually switched from Atom to PHPStorm because of this very issue!

I'm having the same issue with port 2222 + Wordpress...

I think it's fairly evident that this won't be fixed by the plugin author any time soon. If you want to keep using Atom, but want remote FTP/SFTP syncing with WP Engine or port 2222, you might want to look into the new Prepros.

@v0wels I wish I knew how to close this.
It's been open since May '15. If it hasn't been fixed by then, you're right, it won't be now.
Please stop commenting on this.

Need fix as well


Requests time out when fetching pages of a search request in parallel ("HAPI-1163: Request timed out")

NOTE: Before filing a ticket, please see the following URL:
https://github.com/hapifhir/hapi-fhir/wiki/Getting-Help
Describe the bug
TL;DR: With recent versions of HAPI JPA server (not sure exactly since when) we cannot fetch pages of a search result in parallel, if the number of resources to be fetched is too large.
Details:
In fhir-data-pipes we have different methods for fetching resources from a FHIR server in parallel. One of these is through FHIR Search API; one search is done and then different pages are downloaded in parallel, i.e., many queries like this:
http://localhost:8094/fhir?_getpages=ID&_getpagesoffset=INDEX&_count=10&_summary=data

This used to work fine with HAPI FHIR JPA servers too, at least until two years ago as described in this issue. But we have recently discovered that if the number of resources to be fetched is large, then accessing the pages in parallel will fail with HAPI-1163: Request timed out after 60492ms (the time is around 60 seconds and I think comes from this line). Doing some investigations, it seems after ~2K resources are fetched, one page request, all of sudden, takes tens of seconds (other requests are instant). I have investigated this on our side and have reproduced this problem with a simple script (outside our pipeline code), as described in the same issue. If I run the pipeline with a single worker, it eventually succeeds. But having 3 or more workers, usually fails (in my experiments/setup, it failed when number of resources was more than ~5 million). I am guessing that very slow page request, blocks other parallel requests (but I am not sure).
Another input: Although I have set the number of DB connections of HAPI JPA server to 40, there is usually only one or maybe two postgres processes doing some work. When we did extensive performance experiments two years ago, we could easily flood 40+ CPU cores with postgres processes.
To Reproduce
Steps to reproduce the behavior:

Bring up a HAPI JPA server.
Upload a large number of resources (millions) of one type.
Do a single search for fetching all those resources and then fetch pages in parallel.

Expected behavior
The parallel fetch can flood all resources on the machine and succeed.
Screenshots
See this issue for when we profiled HAPI JPA server two years with similar setup.
Environment (please complete the following information):

HAPI FHIR Version: I have reproduced this with HAPI JPA docker versions 7.2.0 and 6.1.0 (not sure which HAPI version they use).
OS: Linux
Browser: It is not through a browser but a parallel pipeline.

Additional context
Here is a sample stack-trace from the HAPI JPA docker image logs:
2024-07-14 01:22:48.242 [http-nio-8080-exec-568] ERROR c.u.f.j.s.SearchCoordinatorSvcImpl [SearchCoordinatorSvcImpl.java:282] Search 109 of type SEARCH for Observation?_count=10&_summary=data&_total=accurate timed out after 60492ms
2024-07-14 01:22:48.242 [http-nio-8080-exec-568] ERROR c.u.f.r.s.i.ExceptionHandlingInterceptor [ExceptionHandlingInterceptor.java:198] Failure during REST processing
ca.uhn.fhir.rest.server.exceptions.InternalErrorException: HAPI-1163: Request timed out after 60492ms
        at ca.uhn.fhir.jpa.search.SearchCoordinatorSvcImpl.getResources(SearchCoordinatorSvcImpl.java:289)
        at ca.uhn.fhir.jpa.search.PersistedJpaBundleProvider.doSearchOrEverything(PersistedJpaBundleProvider.java:265)
        at ca.uhn.fhir.jpa.search.PersistedJpaBundleProvider.getResources(PersistedJpaBundleProvider.java:393)
        at ca.uhn.fhir.rest.server.method.ResponseBundleBuilder.pagingBuildResourceList(ResponseBundleBuilder.java:163)
        at ca.uhn.fhir.rest.server.method.ResponseBundleBuilder.buildResponsePage(ResponseBundleBuilder.java:115)
        at ca.uhn.fhir.rest.server.method.ResponseBundleBuilder.buildResponseBundle(ResponseBundleBuilder.java:57)
        at ca.uhn.fhir.rest.server.method.PageMethodBinding.handlePagingRequest(PageMethodBinding.java:99)
        at ca.uhn.fhir.rest.server.method.PageMethodBinding.doInvokeServer(PageMethodBinding.java:82)
        at ca.uhn.fhir.rest.server.method.BaseResourceReturningMethodBinding.invokeServer(BaseResourceReturningMethodBinding.java:275)
        at ca.uhn.fhir.rest.server.RestfulServer.handleRequest(RestfulServer.java:1198)
        at ca.uhn.fhir.rest.server.RestfulServer.doGet(RestfulServer.java:424)
        at ca.uhn.fhir.rest.server.RestfulServer.service(RestfulServer.java:1938)
        at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:109)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:340)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1744)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:840)

@tadgh if you know how to route this ticket please let us know. Many thanks!

Huh interesting one. I'll read over the other thread and get back to you. Thanks Jing!

Thanks @tadgh for looking into this; I was wondering if there is any update. Please let me know if you need more inputs for reproducing this problem.

Sorry, we have been slammed with the release last couple weeks. Once that settles down I can get some focus on this

Hey @bashir2 and @jingtang10 is there any chance this can be replicated in a HAPI-FHIR test? IF you can submit a PR with a failing test that would go a long way for us in terms of our ability to address this.

@tadgh can you please point me to an example/similar PR that you want me to create for reproducing this issue? The reproducing steps require a large number of resources to be uploaded to the HAPI server (as mentioned in my OP of this issue). Can I reproduce such an environment in the HAPI-FHIR test that you are suggesting?

Most certainly. Have a look at StressTesR4Test which generates ~1k resources.
Alternatively, and maybe better for your use case, is FhirResourceDaoR4SearchOptimizedTest which seems to be doing roughly what you are. You'll note that that second test creates 200 patients, and queries them in parallel via a thread pool. However, that test interacts directly against the DAOs however, so that may hide the source of failure if its upstream in the client. Wouldn't be much of a lift to use the igenericclient in a threadpool though.
Let me know if you need further guidance.

Thanks @tadgh for the pointer; it took me a while to get to this and then to set up my environment but I think I have demonstrated the issue in this commit. Looking a little bit into the details, I think I understand the root cause of this now as well.
TL;DR;
I think this is due to SearchPreFetchThresholds which is by default set to (13, 503, 2003, -1) here. Once we hit around 2000 resources, we have a slow fetch which is consistent with my initial findings.
Details:
The test I have added, sets SearchPreFetchThresholds to (70, 300, -1). It creates 10K Observation resources and then fetch them in pages of size 30. When we are fetching the 300 to 329 resources (and to a lesser extent at 60) we have a slow request. On my machine it takes over 10 seconds while other page fetches are ~10-25 ms (except 60).
I have demonstrated this in the WARN messages I have added to the end of the test. Here are some relevant log messages for 3 consecutive page fetches; note the WARN messages for _getpagesoffset=300:
2024-09-25 05:36:06.501 [main] INFO  c.u.f.j.stresstest.StressTestR4Test [StressTestR4Test.java:259] Loading page 9: http://localhost:41149/fhir/context?_getpages=89d1e925-5774-49e3-8124-9fe456b71fd6&_getpagesoffset=270&_count=30&_bundletype=searchset
2024-09-25 05:36:06.501 [main] INFO  c.u.f.r.c.i.LoggingInterceptor [LoggingInterceptor.java:82] Client request: GET http://localhost:41149/fhir/context?_getpages=89d1e925-5774-49e3-8124-9fe456b71fd6&_getpagesoffset=270&_count=30&_bundletype=searchset HTTP/1.1
2024-09-25 05:36:06.522 [main] INFO  c.u.f.r.c.i.LoggingInterceptor [LoggingInterceptor.java:127] Client response: HTTP 200 OK (Bundle/89d1e925-5774-49e3-8124-9fe456b71fd6) in 21ms
2024-09-25 05:36:06.524 [main] INFO  c.u.f.j.stresstest.StressTestR4Test [StressTestR4Test.java:259] Loading page 10: http://localhost:41149/fhir/context?_getpages=89d1e925-5774-49e3-8124-9fe456b71fd6&_getpagesoffset=300&_count=30&_bundletype=searchset
2024-09-25 05:36:06.525 [main] INFO  c.u.f.r.c.i.LoggingInterceptor [LoggingInterceptor.java:82] Client request: GET http://localhost:41149/fhir/context?_getpages=89d1e925-5774-49e3-8124-9fe456b71fd6&_getpagesoffset=300&_count=30&_bundletype=searchset HTTP/1.1
2024-09-25 05:36:17.765 [main] INFO  c.u.f.r.c.i.LoggingInterceptor [LoggingInterceptor.java:127] Client response: HTTP 200 OK (Bundle/89d1e925-5774-49e3-8124-9fe456b71fd6) in 00:00:11.239
2024-09-25 05:36:17.767 [main] WARN  c.u.f.j.stresstest.StressTestR4Test [StressTestR4Test.java:263] Loading page 10 at index 300 took too long: 00:00:11.243
2024-09-25 05:36:17.768 [main] INFO  c.u.f.j.stresstest.StressTestR4Test [StressTestR4Test.java:259] Loading page 11: http://localhost:41149/fhir/context?_getpages=89d1e925-5774-49e3-8124-9fe456b71fd6&_getpagesoffset=330&_count=30&_bundletype=searchset
2024-09-25 05:36:17.768 [main] INFO  c.u.f.r.c.i.LoggingInterceptor [LoggingInterceptor.java:82] Client request: GET http://localhost:41149/fhir/context?_getpages=89d1e925-5774-49e3-8124-9fe456b71fd6&_getpagesoffset=330&_count=30&_bundletype=searchset HTTP/1.1
2024-09-25 05:36:17.788 [main] INFO  c.u.f.r.c.i.LoggingInterceptor [LoggingInterceptor.java:127] Client response: HTTP 200 OK (Bundle/89d1e925-5774-49e3-8124-9fe456b71fd6) in 20ms

So I am not sure if there is actually anything to be fixed here, is there? I was looking at the hapi-fhir-jpaserver-starter code to see if there is a config parameter to change the default  SearchPreFetchThresholds such that at least we document this behavior on our side; but from here it seems to me that this parameter is not exposed among the configs; is my understanding correct?

There's a setter for the storage here, but it appears unused in the jpaserver-starter. It would be straightforward enough to add it to the application.properties file, and then use that value here in the JpaServerStarter.
Would that help if i whipped up a PR for that?

Yes adding that config to the jpaserver-starter helps; I am also trying to find other ways we can account for this (maybe from our pipeline's end).
What does the pre-fetching exactly mean? I remember that when doing a search, HAPI was storing the list of IDs for that particular search in DB. Does pre-fetching means creating that full-list? Is it possible to make creation of that list more gradual? I mean something similar to the idea of SearchPreFetchThresholds list but instead of having one last very large chunk, make it gradual, i.e., pre-fetch in batches of a configurable number. So for example, the last number of that list, instead of -1, would be that batch size (e.g., 10000).
If this is not easy to do, I think from our side, we should initially fetch a page that is beyond the last index (e.g., 3000) to make sure everything is pre-fetched, then flood the server with parallel page queries.

I'd rather defer to @michaelabuckley 's thoughts on this, but I can certainly help in exposing the setting.

Could you not just use -1 as the first and only element in the list? That would force a full result list immediately, but may blow the memory on the server, depending on your use case.

Could you not just use -1 as the first and only element in the list? That would force a full result list immediately, but may blow the memory on the server, depending on your use case.

The problem is that we do not control the source FHIR-server. Our pipeline is supposed to read from any FHIR-server. We have been facing this issue when our pipeline uses the FHIR Search API and the FHIR-server is HAPI. We can recommend that setting to our partners once your PR is released, but we should also be clear about other performance implications of that.
BTW, what are the memory implications of pre-fetching everything, i.e., just use -1 as you suggested?

 Not good! I didn't write this code and am not intimately familiar with it but my read on it is that it would prefetch all possible results, which could exceed memory limitations of the DB or the application server. For your use case, it may be better to fetch a known static amount per fetch, then have your pipeline adjust so that it runs like this:

first query (loads N resources in pre-fetch).
flood other queries up to known pre-fetched count (e.g. N/page-size, if prefetch 2000 and requests are paged by 50 at a time, that's 19 additional request for future pages.
Another individual "prefetch" call.
Another flood.
etc.

Just spitballing here on the options, no clue if this would be suitable for your particular use case.


I am trying to run this tutorial, I could successfully issue credential using azure B2C tenant but when I am trying to issue using my own tenant i am facing some problem. The app is not able to access the vault i think. Do let me know if i need to make any more changes in the tenant other than the steps mentioned in the tutoria.

Could not retrieve {"keyReference":"#sig_143443f3","type":"key","remoteKeyReference":"issuerSigningKeyIon-143443f3-4641-63e6-f9b4-6044bdd9345a/e7e56d07324b4effab090a2c5e76d7ca"}. Error: RestError: The policy requires the caller 'appid=27b0f364-66a5-428a-928f-e2763cbd78e6;oid=913fd730-e081-4ecb-aa5d-5a6c6ef8a5d8;iss=https://sts.windows.net/956e584f-0fea-4539-9768-31e2a0137343/' to use on-behalf-of (OBO) flow. For more information on OBO, please see https://go.microsoft.com/fwlink/?linkid=2152310
(node:5923) UnhandledPromiseRejectionWarning: RestError: The policy requires the caller 'appid=27b0f364-66a5-428a-928f-e2763cbd78e6;oid=913fd730-e081-4ecb-aa5d-5a6c6ef8a5d8;iss=https://sts.windows.net/956e584f-0fea-4539-9768-31e2a0137343/' to use on-behalf-of (OBO) flow.
[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ace782b7-663f-c6b1-9fef-38b1a4bc13f3
Version Independent ID: ce32432d-42fe-c2c6-9dc8-fa89b5857065
Content: Tutorial - Issue and verify verifiable credentials using your Azure tenant (preview)
Content Source: articles/active-directory/verifiable-credentials/issue-verify-verifiable-credentials-your-tenant.md
Service: active-directory
Sub-service: verifiable-credentials
GitHub Login: @barclayn
Microsoft Alias: barclayn

Hello @Vama97, thanks for the question! We are taking a look into this and will get back to you soon.

@Vama97 indeed it looks like your key vault was not configured properly. A couple of things to check:

double-check that your keyreference doesn't have the hash sign (#) in the configuration.
make sure you select only the Principal role when setting up the access policy for the app in the key vault. I deleted it and re-create it again.

@Vama97, does suggestion from @jcruiz help you?

Just checking in to see if you have had a chance to see the previous response. Kindly let us know if you need further assistance on this issue or is good to be closed.

Just checking in to see if you have had a chance to see the previous response. Kindly let us know if you need further assistance on this issue or is good to be closed.

Yeah, thank you. I made the changes and I amable to run it now. Thanks alot for your help 👍

We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.

I faced the similar error and I found that the cause was a space at the end of my KeyVault URI that I added in the didconfig.json file.
Removing the space, fixed the issue.
Extract from the error pointing to the issue:
 Description: HTTP 404. The resource you are looking for (or one of its dependencies) could have been removed, had its name changed, or is temporarily unavailable.  Please review the following URL and make sure that it is spelled correctly.


Add page index argument to list-built-aritfacts

Currently doesn't provide this option but its in the API

Thank you for your contribution! :)


Drawer Component

Hello,
First, great set of components...used them on a couple of projects and works very well.
I have a project I have in mind, but one of the things I need is a Drawer component.
I'm debating between material-ui and reactstrap for the project. One of the nice features of
material-ui is that it has a built-in drawer component that is easy to use.
Has anybody started/considered adding a general purpose drawer component to this library?
Thanks!

This project is a set of components matching the original bootstrap. Bootstrap itself depends on jquery and manipulates the DOM directly and doesn't mesh well with react. This library fixes that, but doesn't really provide anything that bootstrap itself doesn't provide.
There are third-party libraries which extend reactstrap to add more functionality similar to how other project extends bootstrap to add more functionality (such as any "bootstrap calendar" or "enhanced select" (select-2)).
If someone wants to make a reactstap-drawer lib, go for it!
A drawer can be created with a little bit of custom CSS and the Collapse component. Bootstrap itself does have an example and you can borrow the custom CSS from that example.