Complete PD1 Mapping Inventory

User Story
As ReportStream, I must implement the HL7v2 -> FHIR Mapping inventory for the PD1 segment in order to support onboarding future Senders to the Universal Pipeline.
Note: If a mapping does not exist in the inventory, only fields specified by NIST 2.5.1 should be considered to help limit the scope to a manageable level
Description/Use Case
Presently, RS implements mappings for the PD1 segment but there are fields and rules specified in the mapping inventory that are not implemented as of yet. In order to speed up future onboarding and prevent the issues that arise when implementing these mappings piecemeal, we should go ahead and attempt to create these mappings before we start onboarding many more senders.
Risks/Impacts/Considerations
Dev Notes
Test Plan
Test Plan: https://github.com/CDCgov/prime-reportstream/blob/master/prime-router/docs/design/proposals/mapping-inventory-test-plan.md
PR That implements test plan, use as example: https://github.com/CDCgov/prime-reportstream/pull/11147
Acceptance Criteria

[ ] PD1 segment mappings updated to match HL7v2 -> FHIR Mapping inventory ORU_R01
[ ] Tests created and passing, per the linked test plan

duplicate of #11546


pin sphinx<3.5.0 in docs build

[x] Closes #7216

Latest sphinx release seems to break things

sounds good, thanks, will close this


Web - Connect Bank account - User is hitting Onfido flow when connecting BA through the Plaid option

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:
Pre-Condition: User has a Workspace created

Access staging.new.expensify.com
Sign into a valid account
Tap Profile > Workspace > Connect Bank account
Select "Connect with Plaid" and Proceed with the "Fidelity" flow until the end

Expected Result:
User expects to be able to connect to the bank account through VBA
Actual Result:
User is hitting Onfido flow when connecting BA through the Plaid option
Workaround:
Unknown
Platform:

Where is this issue occurring?

Web
Desktop App
Mobile Web

Version Number: <IP_ADDRESS>
Reproducible in staging?: Yes
Reproducible in production?: n/a
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://user-images.githubusercontent.com/93399543/193172920-7f0ebf99-e72f-44b8-885c-2bcfd276dcb7.mp4
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
View all open jobs on GitHub

:wave: Friendly reminder that deploy blockers are time-sensitive ⏱ issues! Check out the open StagingDeployCash deploy checklist to see the list of PRs included in this release, then work quickly to do one of the following:

Identify the pull request that introduced this issue and revert it.
Find someone who can quickly fix the issue.
Fix the issue yourself.

@nkuoch could this possibly be related to https://github.com/Expensify/App/pull/11232 ?


Unbinds hotkeys when logout detected

It's not the most beautiful fix, basically checking if $Node.GetMy is allowed once per minute. And if not say logged out. Also if a vote gets 401 it sets logged out. And when setting logged out it removes the event listener.
Resolves #1700 #1699 #1592

I realized I could do this without adding any calls, just checking the status on the attempted vote and unbind key capture event if voting gets a non 200. So should be safe to merge/won't add any new load on the site.


Update for Julia 0.7/1.0?

Do you have any plans to update the codebase for 0.7? I so I'd be interested in helping.

Hi Patrick, that would be amazing!
To be honest, i haven't thought about it, but it shouldn't be too hard I think! :) I would love your help! How should we proceed?
I just enabled FemtoCleaner for this repo, which feels like a good start!

Hey, just watched your JuliaCon talk, nice work! Maybe running through the notebook using 0.7 and fixing deprecations and errors would be a good technique, followed by switching to 1.0. Probably most of the work will be discovering which dependencies need attention of course!
So I guess maybe starting a branch in your slides repo for this would be step 1.

Hey! :)
This is fixed, finally, in https://github.com/NHDaly/PaddleBattleJL/pull/4.


JSX spread child generator
JSX spread generator
ts(2609)
2609 generator
🕗 Version & Regression Information

This is the behavior in every version I tried, and I reviewed the FAQ for entries about <Search Terms above>

⏯ Playground Link
Playground code
💻 Code
function* generate() {
    for(var i=0; i <10; i++)
        yield document.createElement("span")
}
var result =
{...generate()}

🙁 Actual behavior
An error is generated if a JSX.Element generator is used.

🙂 Expected behavior
The result of the compilation conforms to javascript and the code works. so the compiler doesn't need to fail in this case.
function* generate() {
    for (var i = 0; i < 10; i++)
        yield document.createElement("span");
}
var result = React.createElement("div", null, ...generate());

Related: #51328

This is explicitly not allowed by React, see https://github.com/facebook/react/pull/13312

@RyanCavanaugh That patch is about rendering generators as components; this one is about spreading it as children. Each individual children is still a component or a tag.

That patch is about rendering generators as components; this one is about spreading it as children. Each individual child is still a plain-function component or a tag. After spreading, it would not be observable to React.

This is a child spread, which is a JSX-specific feature, so we can't say for certain that it transforms to a runtime spread (as far as I know, React has not provided a spec for what any given JSX feature is truly a sugar for) Other transpilers, even as lately as this year, have not necessarily produced an ES iterator spread in their output when consuming this (favoring instead to present the argument as-is): https://github.com/evanw/esbuild/issues/2245, so it's not a settled question.
The unquestionably safe thing to write is this:
var result =
{[...generate()]}

React is not the only library available.

Sure, but it's what 98%+ of TS users are using. We can't possibly support any JSX framework anyone might come up with


satetest/ng + Invalid URL

In my file.safetest.ts, we have this import :

import { makeSafetestBed } from 'safetest/ng';
Throws this error : TS2307: Cannot find module safetest/ng or its corresponding type declarations
Where does this module come from and how to fix this issue ?

Also, after running the command npm run safetest, I get :

I'm running it on localhost obviously, Angular 17.
My script :
 "safetest": "cross-env OPT_URL=${OPT_URL:-http://localhost:4200}  jest --runInBand --bail=5 --testMatch '**/*.safetest.ts' --setupFilesAfterEnv ./setup-safetest.js --testTimeout=30000"

I just pushed a change to the fix-windows branch. Can you try again. I was able to get this to work on my windows machine using the vanilla command prompt. You'll need to

npm install --legacy-peer-deps in root of project
npm install in examples/angular-app
Run npm start in one command prompt terminal, keep this terminal running
Run npx jest in another command prompt terminal

Oh looking at over the issue again, on windows you need to change
OPT_URL=${OPT_URL:-http://localhost:4200}

to
OPT_URL=http://localhost:4200

The ${OPT_URL:-http://localhost:4200} only works on nix based systems

Thanks a lot for the explanations, it works regarding the target URL like a charm, setting the url in the steup file is perfect.
Nonetheless, I still have the issue where I can't import the right lib, which is strange cause as you mentionned below, we see that is available, here is what I have when I want to import :

By choosing safetest/lib/ng, by running my tests it gives the following error :

Hmm, this is an issue with your tooling not following the exports field, that's why there's all these top level files like /react.js just to point to the lib one. I'll need to push a release with an ng file as well. For now you should be able to // @ts-ignore it

I just published 0.1.12 release which has this file, can you give that a try?

That's solved it, thanks a lot for your help and guiding @kolodny


fix: use proper helm param name for specifying pod cidr

Introduced in d3f2bcb2364d ("Doc: Update k3s cilium installation to match k3s default podCIDR") which is in 1.14, I'm marking for backports for that release.

/test


Actually fix runnable order

Changes

to

Sorry for missing this one in original PR. This was an issue before https://github.com/rust-analyzer/rust-analyzer/pull/7596/commits/ee049b256a7718fb346a7172a34f0fc324b3269b, and I fixed it, yet somehow order got reversed (to a logical one)

bors r+

I think that was https://github.com/rust-analyzer/rust-analyzer/commit/9852537809a78b73a0e17eb2dfdf2ab5b07f5eae#diff-5432b98d7968fe0154258257763520ecf07ecc2769c9e2a55db02389f183c906R876-R881.
Before that, debug was created for !debug, which felt odd


use %?RESOURCES to access resources files

Seems like the test job failed here because of a docker error, This job passes. maybe re-run the action ?

Should have seen this. Thanks!


Exception ignored in: <function Driver.del at 0x11b9fd4c0> when using click

Hi
I am having this exception thrown when i use click with neo4j
Exception ignored in: <function Driver.del at 0x11b9fd4c0>
Traceback (most recent call last):
File "/opt/homebrew/lib/python3.9/site-packages/neo4j/_sync/driver.py", line 485, in del
File "/opt/homebrew/lib/python3.9/site-packages/neo4j/_meta.py", line 226, in unclosed_resource_warn
TypeError: 'NoneType' object is not callable
My code is as follows:
if __name__ == "__main__":
    driver = GraphDatabase.driver(os.environ.get('NEO4J_DB'),
                                  auth=(os.environ.get('NEO4J_USER'), os.environ.get('NEO4J_PWD')))
    driver.session()
    run_cli_scan(['test', 'test.json'])
    driver.close()

run_cli_scan is a method that ingests the data in neo4j, as this:
@click.command()
@click.argument('project', required=True)
@click.argument('file', required=True)
def run_cli_scan(project, file):
         result = driver.execute_query('''
    MERGE (n:project {project_name: $project, timestamp: $timestamp})
    RETURN n
    ''', project=project.get('name'), timestamp=project.get('timestamp'))

However if i use sysargv instead of click:
if __name__ == "__main__":
    driver = GraphDatabase.driver(os.environ.get('NEO4J_DB'),
                                  auth=(os.environ.get('NEO4J_USER'), os.environ.get('NEO4J_PWD')))
    try:
        project = sys.argv[1]
        file = sys.argv[2]
        run_cli_scan(project, file)
    except IndexError:
        print ("Arguments missed")
    driver.close()

and i change my run_cli_scan to this:
def run_cli_scan(project, file):
         result = driver.execute_query('''
    MERGE (n:project {project_name: $project, timestamp: $timestamp})
    RETURN n
    ''', project=project.get('name'), timestamp=project.get('timestamp'))

It works fine
Is there any known incompatibility with click? This behaviour is quite weird to me
Thanks

Hi and thanks for reaching out.
What driver version are you using?
As a first a step, I suggest you use the driver and sessions as context manager:
with GraphDatabase.driver(...) as driver:
    with driver.session(...) as session:
        ...  # do things with the session
    # or if you don't need a session, skip the two lines above
    driver.execute_query(...)

Furthermore, it would really be helpful if you could create a single snipped of Python code that runs and shows the issue as I was unable to reproduce it. I couldn't quite string together your snippets to something that executes without modifications.

Hi
it happens with click 3.1.0. Once I use the lates click version it works fine
the code is here https://github.com/javixeneize/neo4cyclone/blob/main/ingest_data_neo4j.py


Display for conformance profiles results

As discussed in the Gateway API meeting, post Kubecon NA 2023, there will be an implementations reorganization by profiles. Since it seems open ended, wanted to open an issue to see what folks would like to be captured.
Data displayed will be pulled from the conformance reports, how to display it meaningfully for an end-user trying to decide on which implementation suits their requirements
Some things that come to mind:

Table that can be filtered by certain features (for ex  version 0.8.0, HTTPRouteMethodMatching), category of implementations (L7, L4, Mesh), maybe stats of passing/failings/skipped tests?
Page headers based on use cases that a user can jump to to see implementations (use cases as in the supported features)
All implementations listed out, and having filters for selecting use case scenarios (similar to online shopping haha)

Some questions:

what else from the conformance reports should be pulled and displayed?
there seemed to be some small discussions on other displays besides badges for implementation projects?

This is open for suggestions, please leave any thoughts/questions/feedback
Related issues:
https://github.com/kubernetes-sigs/gateway-api/issues/1709

seeing as there are no strong opinions from the meetings/on this issue, will work on a poc and present to the group  👍

Thank you for taking this one on! Let us know if you run into any trouble, or need any help! :vulcan_salute:

thanks @shaneutt! currently been exploring the use of rmarkdown. Not sure if other k8s sites use it, but the ability to plot make more "dynamic" code would be great - do you know if maintainers would be opposed to using a flavour of markdown?( ideally I don't want to overly complicate the build for the site)

I for one am not opposed, my co-maintainers @youngnick and @robscott can chime in if they have thoughts :thought_balloon:

Hey @xtineskim, I've never seen k8s use it, but that looks promising to me, worth a shot.

currently been exploring the use of rmarkdown

Would this necessitate switching to a different static site generator, or would it be possible to integrate with the existing MkDocs configuration somehow?
MkDocs appears to use the Python Markdown library with some support for extensions but from a cursory glance I haven't noticed a way to integrate alternative Markdown processing libraries...

@mikemorris  ah good point, thanks for linking the resource 👍 i'll tal into that :)

@xtineskim how are things going with this?

I think given #2874 we can consider this one resolved. We can always open new issues for new UI improvements as well.


Refac bitcoind request retry after startup

Code can not prevent Connection and I/O Errors to happen, in that case
the bitcoind client can retry multiple time its request in the given limit.
This commit implements the try limit and make the request try again
in case of i/o or timeout errors.
In order to fail faster at startup the try limit is set after the
bitcoind sanity checks.
close #372
closes https://github.com/revault/revaultd/issues/95

Updated the OP as it solves https://github.com/revault/revaultd/issues/95 too.

ACK dad2c526352134b368045e993e620c47225dcc8b


如何最大化pipline batch infer的吞吐量/速度[Bug]

Checklist

[X] 1. I have searched related issues but cannot get the expected help.
[X] 2. The bug has not been fixed in the latest version.
[X] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug
在使用pipline 进行batch_infer时，分别用4卡a100合和8卡a100，发现速度没区别，GPU利用率也存在很长的空窗，batch_size为32,应该如何设置尽可能加快这个速度
Reproduction
rt
Environment
rt

Error traceback
No response

Feed a list of prompts to the pipeline, the more, the better.
And I suggest using the latest nightly built version from here, since we fixed an issue in multi-GPUs scenario.


boards: 96b_nitrogen: standardize scratch and storage partitions

Most of the other nRF52832 boards have the following settings for
scratch and storage partitions:
scratch_partition: partition@70000 {
label = "image-scratch";
reg = <0x00070000 0xa000>;
};
storage_partition: partition@7a000 {
label = "storage";
reg = <0x0007a000 0x00006000>;
};
Let's adjust the scratch size to align with the others and add the
storage partition so that settings and FS samples will work.
Signed-off-by: Michael Scott<EMAIL_ADDRESS>

Codecov Report

Merging #13652 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #13652   +/-   ##
=======================================
  Coverage   52.41%   52.41%
=======================================
  Files         322      322
  Lines       46596    46596
  Branches    10772    10772
=======================================
  Hits        24421    24421
  Misses      17267    17267
  Partials     4908     4908

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b1a7e71...46650e1. Read the comment docs.


Typos: SwiftCompilerSources

Fix typos in SwiftCompilerSources
This is one batch of many PRs fixing typos, see the tracking issue.

@AnthonyLatsis I've update to main branch, solve a merge conflict and squashed into a single commit. So re-review ready!

@swift-ci please smoke test


feat: Import Model subcommand

Notes for Reviewers
This PR fixes #

Adds new subcommand for model Command to import model

Signed commits

[x] Yes, I signed my commits.

@leecalcote @alphaX86 @MUzairS15 I've updated the PR. Now we can import the model from local file system, github url and from any public OCI Registry.
Here below i am attaching a demo video of it.
https://github.com/meshery/meshery/assets/107461226/e566369c-66cb-490a-a592-f58118928806

All errors and logs should be using MeshKit.

@RipulHandoo ??


Fix for TypeError (crashes server)

TypeError: Cannot read property 'access-control-allow-origin' of null. It's related to this thread: https://github.com/meteorhacks/inject-data/issues/10.

I'm affected by this too, my app in galaxy keeps crashing. Please merge! Is there any quick workaround in the meanwhile? Thanks!

I created my own fix, you can use this meteor add flean:fast-render it will load flean:inject-data that contains the fix. It fixed my problem

Awesome :), thanks.
On Sat, Dec 17, 2016, 16:32 Rey<EMAIL_ADDRESS>wrote:

I created my own fix, you can use this meteor add flean:fast-render it
will load flean:inject-data that contains the fix. It fixed my problem
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/meteorhacks/inject-data/pull/14#issuecomment-267782263,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AArP2XoLXpdfezQArromRCoSX0g8iUjpks5rJDjOgaJpZM4LKlDT
.

--
Tomás Hayes.

Hi guys, will this be merged any time soon? I am using @flean's fix in the meantime.


oAuth2 Support + Public links

Is your feature request related to a problem? Please describe.
I'm looking to have a copyparty instance where upload/list files/etc... is protected via an authentication provider (Authentik in my case) but that it still allows logged-in users to create public links to allow access to specific files to unregistered/not logged-in users.
Describe the idea / solution you'd like
oAuth2 would obviously be great for my use case, but LDAP is also a very good option if it's easier to implement.
Describe any alternatives you've considered
I've considered the setup that is "suggested" in #62, but that wouldn't allow the public links, as all traffic has to go through the auth service. (Yes, I'm using Traefik and Authentik)
Maybe these two features can be implemented alongside each other?
User "creation" could be handled similarly.

Thanks for the input :> I believe this should come for free through #62 but it's good to keep this in mind just to be sure. Basically it would be safe to grant all users access to the copyparty instance, even if authentik gives the thumbs-down so to say, as the request would not have an associated username/group and copyparty would assume an unauthorized user. Then in the copyparty config it should be enough to enable filekeys to grant read-access to unauthorized users. But we'll have to come back to the details once I've gotten familiar with traefik and authentik 👍

I'm not super familiar with copyparty yet, but I imagine that it would work. I'd only prefer if it was possible to dynamically create public links instead of manually editing a config file.
Lastly, this is something that I didn't specify in the first message, but an addition to the public links would be a "static link" that, for example, always redirects to the latest upload in the directory.
I ask about this because I was hoping to use copyparty in some CI/CD workflows, where I need to update files frequently but I don't realistically want to change a download URL in the workflow config.
(Sorry if I keep adding to this feature request)

I'd only prefer if it was possible to dynamically create public links instead of manually editing a config file

absolutely 👍 depending on usecase you can choose between two different algorithms for generating filekeys; one would generate a new passphrase if the file is modified, and the other one only looks at the absolute filepath when choosing the key.

a "static link" that, for example, always redirects to the latest upload in the directory

oh think I have some good news then, because that should be possible to do through the use of handlers (although a bit hacky) -- you can define custom behavior whenever someone hits a 404, which would invoke a python script that you provide instead of returning the usual 404. So for example if you link someone to folder/latest.exe, you could have a small bit of logic to return the latest exe in the folder.
still not sure if these are a perfect fit for your usecase, but it's a start at least!

depending on usecase you can choose between two different algorithms for generating filekeys; one would generate a new passphrase if the file is modified, and the other one only looks at the absolute filepath when choosing the key.

This should be perfect.

that should be possible to do through the use of handlers

I guess it's time to get back to python programming.
However, I just realized that this

Basically it would be safe to grant all users access to the copyparty instance, even if authentik gives the thumbs-down so to say

While I like this approach, it may not work. In a "classic" setup authentik will intercept a connection to the service behind it, and only passes through the connection if the authentication succeeds (at least that's how my setup works right now).
It may be solvable using some complex traefik configuration, but I'm not sure about that.
I've made a quick diagram of why I think it wouldn't work:

However I'm also not an expert of Traefik or Authentik, so if you find a way to make it work on your end, I should be able to replicate it on mine without a lot of issues.

right... yeah maybe two subdomains could work, auth.site.com and guest.site.com, where only the first one is configured to go through authentik and the other one is a straight relay to copyparty? although that would require users to modify the url and replace auth with guest when sharing a link, so bit of a drawback in that sense.
I've been a bit apprehensive about native support for oauth2 and such; at first glance they appear to be a good handful of things to understand, and hard to get right -- but maybe some of the python libraries would take care of some of that. And there's also been some interest for authenticating with Fido2/passkeys, so maybe these could be combined somehow.

oAuth2 definitely needs to be properly secured, but it looks like you already did that for username-password combinations, so it shouldn't be super difficult.
Having said that, I mainly use oAuth2 because I'd want a "safe" second factor (Fido2 key with biometrics in my case), and Authentik allows me to just log in once using the key for all (or almost all) the apps on my servers. It's both a security thing and a convenience thing.
I'll be patently waiting for some sort of implementation, if you need a betatester, hit me up.

Hey,
copyparty looks like the tool I am looking for after fiddling with other "blown up" solutions.
I tried to cope the code but am not sure if handlers might work. Guess a http 403 handler conducting the actions below with being able to handle ENV variables (make it docker friendly)
The idea with the auth method "proxy" defining a user header seems to be very flexible.
Pseudo setting/env variable could be:
Auth Method = "proxy-header"   // somehow name the auth method
Auth Header = "X-Forwarded-User"  // other values are "Remote-User", ...
Auth Auto Create Users = true     // if no user exists it will be created with group below or the http403 is valid if not present
Auth Auto Create Group = User  // default access rights for those users
Auth Auto Admin User =   // define a user with admin rights, maybe a list
Maybe this can all be handled on the fly.
This way it is possible to offload auth to other components like traefik and foward auth, authelia, ...
Hope this hits the topic

This is really an authentik configuration thing, see https://docs.goauthentik.io/docs/providers/proxy/#allowing-unauthenticated-requests
We just need share links to have a unique route (/share or something) that would allow public access on just that path.

I think this has been solved by https://github.com/9001/copyparty#shares, which makes it possible to create public links to a folder, which will appear below /share or another predefined location.
there is one more limitation that I've forgotten to mention in the readme, regarding IdP volumes and specifically the fact that they're forgotten on shutdown. If a user creates a share by pointing into a such a volume, then the share will temporarily "disappear" after a server restart, and then start working again after the associated user makes their first request to the server.
gonna close this , but let me know if I've missed something :>


purposed fix for issue #4535.

this is my purposed fixed for issue #4535.

Thanks for the PR! However, I am closing this as the seed code already contains the anchor tag with the appropriate link URL.


Write more robust updateBuilder test

The current test assumes a looping order of Objects (but the underlying method being tested does not).
This works fine because V8 will likely continue to loop Objects predictably, but obv. there's no guarantee so I should update that sooner rather than later.

These are gone, so this doesn't matta anymore


fix(jazzy): correct backward_ros tag and release rc2

Description

With tag 1.0.4 and upwards, backward_ros is not being detected by ROS so I had to roll back to 1.0.3.
This should be released as v2.0.0-rc1-jazzy.

Review guidelines

Estimated Time of Review: 5 minutes

Checklist before merging:

[ ] Confirm that the relevant changelog(s) are up-to-date in case of any user-facing changes

But you mark rc2 in the PR title and code, so I assume that's a typo in the description.
Typo yes.

Also, I guess the existing v2.0.0-rc1-jazzy was tagged and pushed manually since #113 was merged with 0.0.1 as the version tag for jazzy?
That's correct yeah, the action was triggered from a branch


A certain package is crashing Atom

Description
When uninstalling the pigments package and install it again (right away) Atom is crashing
Steps to Reproduce

go to install package window
uninstall pigments package
install pigments package again

Actual behavior:
Atom is crashing
Reproduces how often:
Reproduces all the time

100%
### Versions
Atom version 1.58.0X64
pigments version 0.40.6
OS version -Edition	Windows 10 Home
Version	21H1
Installed on	3/16/2021
OS build	19043.1110
Serial number	PF10DYR7
Experience	Windows Feature Experience Pack 120.2212.3530.0

Reproduced with latest Atom and Win10. This is certainly a bug.

Could you report this issue in the pigment repo? We do not maintain that package

Yes of course

Thanks!


Python version recommendation

Python 2.7 is recommended against by the python maintainers for new stuff, and has an end of life in 2020.
Python 3.6 is the latest release and should probably be used for new stuff.
Upgrading from 2.7 to 3.x can be tricky.
We should make a clear recommendation about python version.

This has been documented and resolved.


check test case

What type of PR is this?
/kind feature

What this PR does / why we need it:
check completed E2E Case
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

/approve


allow reject/select to receive arrays

Super useful in the case where you are programmatically generating an array to pass to reject/select as in a hash diff method:
def modified(one : Hash, two : Hash)
  diff_keys = one.keys - two.keys
  one.select(diff_keys)
end

The above code doesn't work with current Crystal as select takes a tuple only.

This looks good, thank you!

:metal:

I tried that since it was suggested elsewhere (another issue maybe?) and it gave issues elsewhere. I forget why offhand.


Changing values on Materials/Codes always crashes game.

Hello, I used this excellent trainer for a little while and everything worked fine. After playing the game for some time and progressing forward, the trainer and the game/wii u both have started to crash every single time I try to edit any Material amounts and now any Codes as well. Everything else seems to work fine after an edit and the game wont crash.
Any ideas? I am using your very latest version 2.3.1 and have played with previous versions (which all crash as well now). I am playing the US version and it's installed via the Wii U Nand and I don't have a physical disc copy.
Thank you for your hard work and time and attention ;)

Both my game and the program crashes whenever I try to edit anything now:
http://oi65.tinypic.com/zur3i1.jpg

Ive added some error catching to log whats happening

Where can I see the log to post it?

Error tab

Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.
And before you ask, no... it didn't disconnect.
No I didn't open any other trainers or gecko or anything before or after this.

Exception of type 'BotwTrainer.ETCPGeckoException' was thrown. once again trying to modify armor to something else.
It's changing the durability of my weapons, but it's not changing them to the item I tell it to change to.

I also get the "Exception of type 'BotwTrainer.ETCPGeckoException' was thrown." message, though I don't seem to have as many problems as Boldboro does. I only have issues when I edit the Materials and the Codes, everything else works without crashing. In fact All I have to do is just click on either the materials or codes tabs and not even change anything and just click the save button and pow! Crash!
In case this helps, I used to be able to use the Shinzuya/GamePil trainer perfectly here: https://gbatemp.net/threads/the-legend-of-zelda-breath-of-the-wild-trainer-eshop-eu-beta.463066/ and for reasons that are unknown it also crashes the game now as well and can no longer load items in the list (but the other cheats like freezing health and stamina still work and always have).
It's strange but both trainers have starting giving me issues around the same time all of the sudden both wouldn't edit and both would crash, I'm wondering now if it is "linked" (sorry for the pun) to a point in the game where something had changed along the way. Maybe using either trainer to edit things can mess up the state of which the saved game is affected and can cause a cascade on future saves and trainer edits that are expected to work... Just a guess though.
The other error I can report is what shows up on the screen once a crash has happened it says:
Illegal command byte received: 0x7b
Server Version: 02/25/2017
Hope that helps, I will update if I find a different error from the trainer or output from the Wii U error screen.

Thanks this is very helpful. I also think this is the issue in some way. I am early in the game so perhaps when I get to the same point as you I will face the issue. Also the gecko code is not mine, it's from the ported tcpgecko so perhaps it has bugs. Over the next week I'm  going to try to add as much detail to error log as possible to see what it is.

I am glad to be of service good sir ;) I edited my previous post and added some more info if you haven't read it yet -thank for your hard work!

Ok so with your new trainer and more error codes available I now get this:
Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()
at BotwTrainer.MainWindow.SaveClick(Object sender, RoutedEventArgs e)
These are the exact same error codes that the user Sapphireflames has reported (but he did not report if he is using the PAL or US version and if his game is up to date or not). Though as usual, I can still use everything like before except any material changes or code/cheats. those still crash as usual.
There is one difference though, I reported that the items would appear from last to first top to bottom in the trainer after a refresh, but now they appear as first is first and last is last top to bottom in the trainer display :) Now maybe you just a made a cosmetic change but I thought I should tell you ;)
Oh and speaking of cosmetic changes, I forgot to mention that I really like the simple dotted lines around all boxes, it was a nice touch and does a great job of making it easier to see where the line match up, thanks for that simple yet very effective visual enhancement :P
Oh and in case it has been forgotten, I am using the US version with the latest 1.1 (v33) update of the game and no DLC, and always your latest trainer version (in this case 2.3.3). The error on screen after the usual crash is still the same one I posted about earlier:
Illegal command byte received: 0x7b
Server Version: 02/25/2017
No change there. Will report back if I find anything else, keep up the good work ;)

New version which should help identify where exactly this happens

I was able to edit my apples to x10 and it stuck and I was getting very excited!
But alas, it seems to be random if the game will crash. This is funny part, the new trainer (2.3.4) NEVER crashed through all my testing. I never got one error code to report from the trainer itself, instead the game always crashed and I do have a few different codes displaying now here they are:
Illegal command byte received: 0xXX <---This would change to these values below:
Server Version: 02/25/2017
0x7d Just the apple edit ok with x10 then tried to edit a second material to x10 then crash :(
0x2a All materials edited at one time to x990 each, crash
0x2a Just the apple edited to x99 apples, crash
0x7b Just the apple edited to x10 apples, crash
0x12 Edited Fairies to just x1 from x897 (Saved that way previously), crash
Sadly this was starting to seeming appear random and hard to track down, I was really hoping to get something back from the trainer itself but not one time did it report anything in the error tab. Also, all other items are still able to be edited in other tabs, again it's just the Materials and the Codes that are still affected for me.
We are getting closer now, I can feel it, since the apple edit worked more than a few times already, but for the most part always crashed anyways. I did try to refresh the data after every edit to be sure, and I also tried without refreshing just to see if anything would show a better result. In all cases mostly a crash and very few stable edits that stuck.
I'm going to see if I can get an edit to stick and quickly save and then try to edit again from that save, it's hard getting one to stick, but if it does I'm hoping maybe after loading a successful edit from previously maybe I can get it to work properly. Just a wish and a dream but worth a shot right now.

Lol just saw your 2.3.5 version posted right after I made my last post, I will try it right now! :)

OK! I got a crash again, but this time I got an error code to give to you from the trainer itself :)
Shield Slot CodeException of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()
The on screen error code is as follows:
Illegal command byte received: 0x7b <-- This seems to be the most common code for some reason
Server Version: 02/25/2017
Will try some more testing now :P

Nice one, can you screenshot the codes tab for me please? How many shield slots have you got?

Sorry for the delay, was busy.
I have only 4 shield slots, and here is the requested screen shot for you:
https://1drv.ms/i/s!AtPipJ44eLuQjRAD8YL4HGsm_zKM
I am now testing your latest release (2.5.2) and will report back when I get a chance ;)

Wow, the materials no longer crash!! GOOD WORK!! ;)
Also, I then did try to change a code in the codes tab, I keep using the rupees as a test only. Unfortunately it did crash :(
Here is the error output from trying to edit rupees to 999,999 and freezing them with the check box:
Stamina Code
Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()
Health Code
Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()
Rupee Code
Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()
Mon Code
Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()
Run Code
Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()
Speed Code
Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()
Weapon Slot Code
Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()
Bow Slot Code
Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()
Shield Slot Code
Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()

Here is another error output from me simply freezing the hearts with the check box and clicking save:
Stamina Code
Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()
Health Code
Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()
Rupee Code
Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()
Mon Code
Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()
Run Code
Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()
Speed Code
Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()
Weapon Slot Code
Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()
Bow Slot Code
Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()
Shield Slot Code
Exception of type 'BotwTrainer.ETCPGeckoException' was thrown.   at BotwTrainer.TCPGecko.Dump(UInt32 startdump, UInt32 enddump, Stream[] saveStream)
at BotwTrainer.TCPGecko.peek(UInt32 address)
at BotwTrainer.MainWindow.DebugData()

I have changed nothing since the screen shot I uploaded just so you know, also I figure what has made the edit on materials work properly now is since version 2.5 is that the changed values don't save on a global scale, I guess my real problems stem from only the codes on the code tab and how they are handled.
I think you already know all that though lol, but I wonder why they are so problematic then? Anyways, I will continue to test ;)

Can you test this please:
https://joffnerd.s3.amazonaws.com/app.publish.test.zip

Tried your newest release, got a game crash when trying one edit rupees, then tried again but with hearts and crashed again. Always only in the codes tab it crashes now, also, got no error in the error log both times :(
All I get is the on screen error of for both tries:
Illegal command byte received: 0x2a
Server Version: 02/25/2017
All other fixes still work and I found out that editing too many materials at once wont allow them to save,
so editing in small batches are ok, plus maybe triggering some use of the materials (doesn't matter which one) and a fast travel all seem to make the values stick.
Simply doing an edit, save, then load, doesn't seem to always work for some reason and other times doing an edit, save, then load works just fine and saves the value, not a big deal just weird some times.
Really appreciate your hard work man, you really have come far very fast ;)

As far as I can tell its something to do with your wifi connection. Do you have wifi issues outside of this app?
I don't understand the deep underlying code enough to help i'm afraid :(

Wifi works great, I do have a complex home network though, but its never given me trouble. I don't see how the connection is always the problem. The other trainer from Shinzuya/GamePil can freeze the health and/or stamina no problem.
Are you using the same code for those values?

Testing your latest version atm... will report back.

The problem is THE NETWORK CARD!!
I disabled my 10gb NIC and used my 1gb nic instead and THE CODES TAB NOW WORKS PERFECTLY!
All Codes in the code tab are perfect and have no issue what so ever :))))
I have an intel 1gb nic and a Mellanox 10 gb card. There is a HUGE difference between them driver and hardware wise, so it all makes sense.
The only thing is I feel so bad putting you through so much uneeded debugging!!! IM SO SORRY FOR THAT!!!! :S
Your trainer has always been great, I'm now going to call it AWESOME! This whole time it was my end wtf!
I will report anything unusual if I find anything. Thank you SO MUCH for your efforts! You deserve a pat on the back and beer good sir! :)

One odd thing I have noticed now that it works fine, when I click on the save tab, the button is slow to respond like there's a delay before the save button returns to the unclickable state it is when there are no changes to commit. It's like that every time I hit the save button now.
Before it was almost INSTANT when I had the 10gb nic in use, but now it almost seems to wait/take longer after every click I do on the the save button. That may have some indication as to why it would always fail before ;)

Awesome. Glad it's working for you :-) This is useful to know for others who have similar problems. So many factors with computing.
So the save button seems slow because the app is 'busy' writing to the wii u and so it freezes while doing so. I should make it async really. It basically takes time for the data to be sent and written to memory so until that competes the button is frozen. When the action finally competes the button is then disabled so it is a very slow to respond feel about it. Thanks for persisting and using the trainer.

When I get time I'm going to see what network setting could be affecting me and share it with you. I'm sure I can get this working with my 10gb nic with enough playing around, while maybe also finding a common setting that can help others with their connection as well.
If I find out what could be a root cause for some/most people and how to correct it, I will close this thread. If I can't find the exact reason in a reasonable time I will still come back and close it, the issue is fixed for the most part for me, but I would like to try and help a little further for other people first if I can.
Just a quick comment, I love how your trainer allows the stamina to be infinite but it doesn't show the ugly stamina circle/meter to do so, that's great ;) Oh and don't lock your rupees when talking to the large Fairies for the first time, the game will freeze lol. Stay awesome good sir, until next time....

Thanks dude, I think its just poor connection code that i borrowed. I may be way off but i think your 10gb nic is too quick and the Wii U can't cope with the overload due to my implementation. I plan to migrate it over to something cleaner reflecting JGecko U but they may take some time.
Just out of interest, can you connect to to a game with JGecko U with both your network cards?

So busy atm, but as soon as I get the chance I will try out JGecko U some time later today 😉

JGecko U connects just fine with either network card but I need to add some code(s) to try to see if it will actually work. Have any that work on JGecko U? I'm new to using it and played with it before but never really got into it, I have to figure out where to enter the code properly lol :P

Can someone please explain what I am supposed to disable and enable to get this to work for me.
I don't know much on network cards or how they even work.

Un-Equip everything and then pause your game.

If its both the trainer and the game, you should get a fw.img file and use the CFW booter from the homebew store, launch that then load the homebrew launcher from sthetix.info/wiiu rather than Loadiine. Boot into the CW Launcher, then after you load that then go back to sthetix.info/wiiu and install TCPgecko then open Zelda. Should work there.

Unequipping didn't fix my error.

@Bolboro I haven't disabled anything on my intel network card but it works using that card with no crashes at all now. My problem was that I was using a 10gb network card (which I have now disabled/stopped using), it's 10x the speed of regular networking adapters and has different drivers that run it with a ton of totally different settings and features which all contributed to my crashing.
Joffnerd thinks that the raw speed of my 10gb network card was causing my issues as the wii u probably couldn't keep up, or at least the networking code for either TCPGecko or the trainer itself couldn't keep up. In any case switching to a standard, decent quality networking car (in my case an intel) seems to have fixed the issues that were plaguing me.
I'm right now in the process of trying to figure out if I can still use my other faster 10gb networking card with different settings instead, and they may be common settings that other networking cards share. I'm hoping to solve the problem a lot of people are having with a simple networking tweak. No idea if it will work and so far I haven't had any luck.
I have a feeling that generic networking cards that are built into most systems that are not an intel may have the same issue as my 10gb one did as they are off the beaten path so to speak in terms of quality and compatibility (except my 10gb card it's just waaay different I think, cuz it's great quality).

So there's absolutely no way I can use this trainer whatsoever is what you're saying unless I buy a new network card for my laptop...

I honestly don't know. As I keep saying, it could be one of so many things.
Is your AV running?
Did you update your version of .NET?
Have you used the latest code handler?
Outside of this I can't help, i'm sorry. I'm not a professional developer, just doing in this in my spare time and learning as I go.

I have no Antivirus at all.
My .NET is 4.0 I think. How do I check? Do I need one that's better?
I use the codehandler that came from the Gecko U Updater.

It's in the readme

I have not seen a readme in the files I've downloaded.

http://oi63.tinypic.com/otpu84.jpg

Sorry man but something about your setup doesn't work.

What's not fair is that everone else can get it to work and not me and can't seem to either find a solution or at least help me figure out how to do the same thing in this trainer through the memory viewer of JGecko U.
I just want to edit as well and seems like because others got it to work they don't care to try and help someone to get the same benefits as them.
I also don't know what "setup" I need as I'm not doing anything different than anyone else to try and simply get this to work.

@Boldboro Don't sweat it too much man, the truth is that all trainers that are out there for Zelda BoTW are kinda flakey at this stage. They are so new that they have issues, but the creators are working hard to try and make them more compatible and work for everyone. Joffnerd doesn't have extensive knowledge of the network code used to make his trainer super uber compatible with everyone's setup.
Just look at what I went through, I am still testing his trainer even though I got LUCKY and got it to work with a different network card. That's all, I did everything else under the sun to get it to work before and had no luck also. I had to use Jofferd's trainer and the other popular trainer out there to get anywhere in combination. I would edit weapons with Joffnerds trainer and use the stamina code and inf health with another and after an edit, I was always disconnecting and reconnecting between them.
If Joffnerd or any other coder went ahead and made the network code and trainer all together from the ground up I'm sure he could help you better... Just don't feel like no one is helping you bud, cuz there are a lot of others in your boat, the hacks just aren't that mature and may never be, there's no way to know since these things as they take time and they are done for FREE and are just personal projects. If you want to help to speed up the fixes and progress then beta test like I did, and with enough playing around you may also find a fix that works for you.
I'm not some uber amazing programmer or anything, I just have general computer knowledge that gets me by when it comes to this stuff, it helps a lot if you want to make things work... it's computers after all, not a game console, it's not always 1-2-3 and you need to be patient and bang away at the problem until you can find a fix or workaround, that's just how these things go. That's how I did it, that's how anyone should do it, instead of calling it unfair -work the problem, learn and contribute and then maybe you will get somewhere ;)

@Joffnerd  I have been testing all of your releases up to 3.1 as of present and everything is going fine and all codes are working as you add them. I am of course having no luck on finding a network tweak or setting that may allow others to use the trainer when using a different network card like my 10gb one.
I am about to try the latest trainer on my simple file server as it's always on and it's best I run your trainer(s) from that machine, and that machine has a simple crappy realtek NIC in it. I am wondering what will happen over there, it also has the EXACT SAME 10gb nic as my main machine and I will be trying again with that one as well just to see what happens. I will report back as soon as that's tested ;)

@rEd2k
There's more than one other than breath.exe? The creator of breath.exe stopped working on it long ago. What othet BotW Trainers are out there as I just want to modify a few things. That's all.
Breath.exe never failed me and I use that for my codes like moonjump and such. I just want an item editor.

@ Boldboro

How do I tell what is my network card so I can see if Windows 8 can update the drivers?

You tube bud.... you tube.

I can't use wired as my modem is far away from me....

If you want to try and solve the problem you have to at least try and test with using your ethernet/wired connection and since it is a DIFFERENT NETWORKING CARD that you would be using for that type of connection, chances are it might actually work for you, like it did for ME by CHANGING THE NETWORK CARD USED. Wifi is a networking connection just like an ethernet port is and this means it is considered a network card/interface just like the wired kind everyone uses, it's just that its wiresless <-(which could be the root of your problem).

I don't think I ever seen an ethernet port on my Wii U before.

That's right, the wii u doesn't have a wired connection, hence why I stated previously to at least WIRE THE COMPUTER to a wireless router to eliminate potential problems. The Wii U will have to be wireless, you can't get away from that unless you buy a usb to ethernet adapter specifically made for the Wii U, but it's not needed.
Unless you have a computer tower or a huge beast of a machine that you cannot move close to a router for a wired connection, there's no reason you can't take a laptop over to your router and run the trainer on a wired connection just for a simple test. Also don't forget to TURN OFF YOU WIFI on the computer your testing with to MAKE SURE you are using the WIRED connection to the router.
Once you get everything up and running, try a stamina or health cheat code and then go back to the game and see if your invincible now or can run forever. If it failed you will  get an error right away and then you know it didn't fix the problem and you may have to look for another solution. Good luck!

I would just like to add that I just tried your latest version of the trainer (3.6.1) and it totally fixes ALL and ANY of the issues discussed previously and works perfectly with ALL of my various network cards/interfaces now despite what driver version they use. GREAT JOB JOFFNERD! :) Thank you AGAIN for your amazing talent and hard work on this. Your trainer was already great, but you have made it the BEST trainer I have the pleasure to use for one of my now favorite games.  :) ;) XD


reactions missing on imported model

Describe the bug
Importing this analytic geometry model works:
https://github.com/sbmlteam/sbml-spatial-models/blob/master/Analytic models/fish_300x300.xml
but the reactions are missing
Likely the cause is that the reaction math contains piecewise functions, which we don't support for reaction terms.
Currently also no plan to support this, as we would need to translate them to infix that muparser can parse for dune to evaluate them, which looks hard to do reliably.
We should at least display a warning to the user that the reactions are being ignored though... (and check that they don't cause any issues if the model is modified and simulated)

Didn't see this one before! if it is not something that could be exchanged reliably, then of course its absolutely fine to leave out. I can rewrite the example different reactions in that case. In general though, i could see piecewise expressions being used for non homogeneous initializations. So it would be nice if SME could support it, and then pass it on as sampledfield to the dune solver or something like that.

Ultimately I think it would be nice to cover the full instruction set of equations that we can generate from COPASI. That way we could be sure to be able to 'just use' those models in SME. But it is a nice-to-have. So definitely not a priority.

initial cause of missing reactions was them not having a compartment, fixed in #522
support for piecewise function (and all other L3 sbml math) in reactions added in #572
model is now loaded correctly and can be simulated with pixel
dune does not yet support piecewise, but it will once it switches from muparser to symengine, without any further changes required here


NOC certificate questions (CON-416)

How many entries can the NOC certificate chain have? Where is the root of that chain?

Should the chain look like this?
RootCA -- company owned (cloud)
IAC -- One per house (cloud)
IAC -- One per controller (stored in controller, shares CASE Authenticated Tag)
NOC - final certificates in the nodes
or like this?
RootCA -- One per house (cloud)
IAC -- One per controller (stored in controller, shares CASE Authenticated Tag)
NOC - final certificates in the nodes
or is the root of this chain a CSA issued certificate?
This model allows people to delete the controller app off from their phones, and then when they reinstall it a new IAC will be generated from the cloud-base house IAC.

Matter NOC certificates need MatterRCACId, MatterFabricId, MatterICACId values encoded into the Issuer and Subject fields. Where do these IDs come from?

So when instantiating a controller it needs parameters:
setRootCertificate(ByteArray) -- how do you it build this ByteArray from a CRT file?
setIntermediateCertificate(ByteArray)
setFabricId(id) - this ID is also in the certificate, where does it come from?
setIpk() -- it needs this, what is it, and how do I set it?

What do you do with a CASE Authenticated Tag? I think it is for doing this, am I right?  I want to have multiple controllers on the same fabric. When a node is commissioned, the nodeId of the commissioner gets written into the ACL.  So now if I add a second commissioner, how do I get its nodeID into all of the nodes? I think the answer to this is to use a CASE Authenticated Tag. The first commissioner would update the nodes ACL entry to use the CASE Authenticated Tag instead of a normal  nodeID. Then both commissioners would have the same CASE Authenticated Tag. How do you use this feature since the CASE Authenticated Tag has to be inside the NOC certificate? Page 327 of the spec.

Each fabric has one RCAC, so the chain can look like this:

RCAC&RootKey -- One per house(cloud)

RCAC&ICAC(Optional)&NOC&NOCKey -- Devices(both the controller and end devices, can share CATs)
Then the controller and end devices  have the same Root cert so they can establish CASE Session.

Devices send CSR to the issuer (clould), and the issuer will use the RCAC and its key to issue ICAC and NOC, and add these IDs to the certs when issuing them.
The RootCert is self-issued in cloud and sent to device in ByteArray format after the could has issued the ICAC and NOC; The ICAC is also sent to device in ByteArray format; The Fabric ID is generated randomly in clould before the RootCert is self-issued; IPK is in SPEC <IP_ADDRESS>.1
Yes, you are right. I have added a method to send the NOC with CATs when pairing devices in PR, you can follow the test steps in that PR.

Is this sequence accurate? Can you help fill in exact commands?

Generate MatterRCACId, MatterFabricId, a CASE-Authenticated-Tag (how?), add to root cert

Generate self-signed root key in cloud for house
chip-cert gen-cert --type r --subject-chip-id CACACACA00000001 --valid-from "2020-10-15 14:23:43" --lifetime 7305 --out-key Chip-Root-Key.txt --out Chip-Root-Cert.txt --out-format chip-hex
// What is exact chip-cert command?

Controller asks cloud to send it RootCert

Controller gets MatterRCACId, MatterFabricId, CASE-Authenticated-Tag out of RootCert

Generate intermediate key on controller,
openssl ecparam -genkey -name prime256v1 -noout -out $1/private/intermediate.key
// What is CHIP API to do this?

Send CSR off to cloud

Use Root key in cloud to sign CSR
openssl ca -batch -config root-ca/openssl.conf -keyfile root-ca/private/ca.key -cert root-ca/certs/ca.crt -extensions v3_req -notext -days 900000 -md sha256 -in $1/certs/intermediate.csr -out $1/certs/intermediate.crt
// What is exact chip-cert command to sign CSR?

Send intermediate cert back to controller

Use RootCA and IntermediateCA to start controller.

I think you are not so clear about the NOC chain generation. There is a example for an NOC Chain generation.

A user creates a house in his phone app below his account, the cloud will generate a self-signed RCAC&key and ICAC&key(issued by the RCAC) , save them and bind them to the house created by the user. The API is IssueX509Cert. The DNs in RCAC are RCACId and FabricId. Both of them are generated randomly, saved in cloud and bound to the house. The CATs can also be generated randomly, saved and bound to the house.

The user adds a controller(commissioner) to the house, the controller will generate a NOC keypair and use the public key in the key pairing to generate a CSR(APIs are in emberAfOperationalCredentialsClusterCSRRequestCallback), send the CSR Elements to the could. after obtaining the CSR Elements, the cloud will decode it(https://github.com/project-chip/connectedhomeip/blob/master/src/controller/ExampleOperationalCredentialsIssuer.cpp#L343-L362) and use the public key to issue a NOC for the controller IssueX509Cert. The DNs in NOC include NodeId, FabricId and NOCCATs, The nodeId should be sent by the controller, and the other two are generated before.

After issuing the NOC, the cloud will send the RCAC cert, ICAC cert and NOC back to the controller. and the controller will save what it receives.

When the user tries to pair a matter end device, After PASE Session establishment and device attestation verification, the device will generate CSR and send it to the commissioner. The commissioner will forward the CSR to the cloud and the could will generate the NOC, send it back to the commissioner, and then the commissioner will send this NOC to the device with the RCAC and ICAC and set the device's ACL with the CATs.

To ensure the security, the RCAC private key and ICAC private key are saved in cloud and will never be sent to the controller or end devices. Each devices will have a NOC with the NOC private key, also the NOC private key will never leave the device. So the NOC generation should be in cloud.
The cert chain: RCAC(self-signed) ->ICAC(signed by RCAC private key) ->NOC(signed by ICAC private key)
Also the ICAC is optional for NOC chain, so if there is no ICAC, the NOC is sighed by the RCAC private key. And in this situation, the cloud will save only the RCAC and its priv-key with the RCACId, fabricId and CATs.

Why can't the ICAC/key be generated and stored on the controller? Now the controller can commission devices without needing the cloud.

Can multiple controllers have different ICAC (with common Root) and be on the same fabric?

Why can't the ICAC/key be generated and stored on the controller?

ICAC's key pair can be generated and store on the controller and the cert generation should be in cloud, because ICAC is signed by the RCAC private key and the private key should never leave the cloud.

Now the controller can commission devices without needing the cloud.

Yes, because the test controller will store the RCAC&private-key, but it cannot control the devices once it is factory-reset, so it is only for test. Storing the RCAC&key in the cloud can make the controller be able to control the devices even if it is factory-reset because it can re-obtain the ICAC&NOC signed by the original RCAC again. You can backup the RCAC&key generated by the controller in the cloud, but it is not so safe because the RCAC private key might be leaked during the backup.

Can multiple controllers have different ICAC (with common Root) and be on the same fabric?

Yes, they can.

@jonsmirl Please close this issue if your questions are answered

Hello，@jonsmirl ，Does the private keys of ICAC and RCAC must be stored in the cloud? Can ICAC be stored in an APP or matter controller（like hub）? It semms like the RCAC and ICAC certificate perpetually valid unless recreate the house. How to handle if the private key leaked? It seems like there is no RCAC certificate revocation mechanism in matter.

If you store the ICAC in the app you can commission devices without needing to access the cloud. This is a way to support multiple users. For each user a different ICAC is generated which they store locally in their phone.
If the RCAC private key is leaked, you are doomed and have the reset all of the devices. There is no way to revoke it. That makes sense, because if you revoked the RCAC then you'd still have to reset all of the devices to get the old RCAC out of them. You could also remove all of the entries for the old fabric and then make a new one using a new RCAC.
If the ICAC private key is leaked, you don't revoke it. Instead you create new ICAC for everyone still trusted with a different CAT tag. Now upgrade all of the device's ACL to use the new CAT tag.  This will render the leaked ICAC useless since they won't be able to generate a new ICAC with the new CAT tag in it. You can leave both CATs in the devices durring a window while you are upgrading the controllers.
There is no solution for losing an RCAC or ICAC private which does not involve touching all of the devices. CRL (certificate revocation lists) only work because when you verify a certificate you get the CRL from the cloud and Matter devices don't access the cloud on there own.
One thing for sure not to do -- don't store the RCAC in a phone. Because if the person deletes and reinstalls the app, they will lose the RCAC.  And people delete apps all of the time when their phones run out of space.

If you store the ICAC in the phone app you can commission devices without needing to access the cloud. This is a way to support multiple users. For each user a different ICAC is generated which they store locally in their phone.
If the RCAC private key is leaked, you are doomed and have the reset all of the devices. There is no way to revoke it. That makes sense, because if you revoked the RCAC then you'd still have to reset all of the devices to get the old RCAC out of them. You could also remove all of the entries for the old fabric and then make a new one using a new RCAC.
If the ICAC private key is leaked, you don't revoke it. Instead you create new ICAC for everyone still trusted with a different CAT tag. Now upgrade all of the device's ACL to use the new CAT tag. This will render the leaked ICAC useless since they won't be able to generate a new ICAC with the new CAT tag in it. You can leave both CATs in the devices durring a window while you are upgrading the controllers. This is how you get rid of a user you gave an ICAC to.
Also note, giving another user an ICAC lets them commission new devices, alter ACLs, etc. You don't have to give other users an ICAC instead you can just give them normal certificates. With a normal certificate they can still do anything allowed by the ACL, they just aren't admins. This is a solution to something like guests in an AirBnB. But why bother? Instead use Google/Apple/Amazon to handle the guests.
There is no solution for losing an RCAC or ICAC private key which does not involve touching all of the devices. CRL (certificate revocation lists) only work because when you verify a certificate you get the CRL from the cloud and Matter devices don't access the cloud on their own. CRLs don't really gain you anything that you can't get from ACLs and changing the CAT tag. If you revoked the RCAC the whole fabric would stop working which is equivalent to remove it or reseting the device.
One thing for sure not to do -- don't store the RCAC in a phone. Because if the person deletes and reinstalls the app, they will lose the RCAC. And people delete apps all of the time when their phones run out of space.

Thank you.  You're right, if the ICAC private key is stored in the mobile app, you can commission devices without needing to access the cloud. However, if the private key is stored in the app, and I lose my phone, anyone who knows my Wi-Fi password could potentially control my devices. In that case, I would have to reset all the devices, which seems quite troublesome when there are a lot of devices involved.

Thank you. You're right, if the ICAC private key is stored in the mobile app, you can commission devices without needing to access the cloud. However, if the private key is stored in the app, and I lose my phone, anyone who knows my Wi-Fi password could potentially control my devices. In that case, I would have to reset all the devices, which seems quite troublesome when there are a lot of devices involved.

When you make keypair on a phone using the OS API the private key is stored in the ARM trust zone which is inaccessible. Also, if you lose your phone your phone will have the normal IOS/Android locks on it which are quite difficult to defeat. In general only law enforce can gain access to a lost phone. Note that if you don't store an ICAC in the phone you have to store a normal certificate which can still access all of the devices, it just can't make new ones.
I see little downside to storing an ICAC in the phone. If you are really worried when you lose the phone you can use the process of changing the CAT tag to render the phone useless.

谢谢。你说得对，如果 ICAC 私钥存储在移动应用程序中，你可以无需访问云就可以调试设备。但是，如果私钥存储在应用程序中，而我丢失了手机，那么任何知道我的 Wi-Fi 密码的人都可能控制我的设备。在这种情况下，我必须重置所有设备，当涉及很多设备时，这似乎相当麻烦。

当您使用 OS API 在手机上创建密钥对时，私钥存储在无法访问的 ARM 信任区域中。此外，如果您丢失了手机，您的手机上将有正常的 IOS/Android 锁，这些锁很难破解。一般来说，只有执法人员才能访问丢失的手机。请注意，如果您没有在手机中存储 ICAC，则必须存储仍然可以访问所有设备的普通证书，只是无法创建新设备。
我认为将 ICAC 存储在手机中没有什么坏处。如果你真的担心丢失手机，你可以使用更改 CAT 标签的过程使手机变得无用。

Got it. Thanks very much.


Using toast.promise with use:enhance

Hi! Forgive me if it seems like a basic quesiton, I'm still learning Svelte(kit).
I have a Form component with use:enhance attached, sending data to the +page.server.js with POST and returning new data. I wanted to somehow use toast.promise before and after the form submission, as per the svelte docs , with the form property as promise but I'm not quite sure how and where to implement it. Could you give me some pointers?
+page.server.js looks like this
import { fail } from '@sveltejs/kit'

export const actions = {
  default: async ({ request }) => {
    const form = await request.formData()
    const newGame = form.get('new-game')
    const newPlatform = form.get('new-platform')

    if (!newGame) {
      return fail(400, { newGame, missing: true })
    }
    return { success: true, newGame, newPlatform }
  }
}

+page.svelte

<Form />

{#if form?.success}

Added {form.newGame} on {form.newPlatform}

{/if}

Form.svelte

<form method='POST' use:enhance>
  <input type='text' name='new-game'>
  <input type='text' name='new-platform'>
  <button type='submit'>Create</button>
</form>

Hey
So, do you want to show a toast, when the user logs in?
Code form:

<form
	method="POST"
	use:enhance={() => {
		return async ({ result, update }) => {
			toast.success("Successfully logged in!", {
				position: "bottom-right",
			})
		};
	}}
/>

in your layout file:

<Toaster />

@babakfp It seems the loading state in the toast.promise callbacks are not being triggered when using use:enhance promise structure
import { applyAction, enhance } from '$app/forms';
<form
	method="POST"
	action="?/example"
	use:enhance={() => {
		return async ({ result }) => {
		      toast.promise(applyAction(result), {
			      loading: 'Saving...',
			      success: 'Settings saved!',
			      error: 'Could not save.'
		      });
		};
	}}
>

@babakfp @cassianodaniel The way you ate doing it is starting toast in the page data invalidation, not on form submission.
The code inside return async ({ result, update }) => { is runing after the action execution. It will create an initial delay. What toast is really waiting is the page data loading, not the form submission.
<form
	method="POST"
	use:enhance={() => {
                // START OF FORM SUBMISSION
		return async ({ result, update }) => {
			// SUBMISSION HAS DONE
                        // BEFORE START UPDATING PAGE DATA
                       await update()
                        // PAGE DATA UPDATED
		};
	}}
/>

So, the best way to do this is something like:
<form
	method="POST"
	use:enhance={() => {
		// START OF FORM SUBMISSION
		const toastId = toast.loading('Doing something..')
		return async ({ result, update }) => {
			// SUBMISSION HAS DONE
            // BEFORE START UPDATING PAGE DATA
           await update()
            // PAGE DATA UPDATED
           toast.success('Done!', { id: toastId });
		};
	}}
/>

Apologies for the late reply, I ended up doing a new Promise similar to what @cassianodaniel suggested.
<form method="POST" action="?/insert" use:enhance={() => {
  let toastResolve, toastReject
  let toastPromise = new Promise(( resolve, reject ) => {
    toastResolve = resolve
    toastReject = reject
  })
  toast.promise(toastPromise, {
    loading: "Saving...",
    success: (e) => `${e}`,
    error: (e) => `${e.message}`,
  })

  return ({ result, update }) => {
    if ( result.data === "no game found" ) {
      toastReject(new Error( "no game found" ))
    } else {
      toastResolve("OK !")
      update()
    }
  }
}}>


Simple Rmd document to clean-up and plot the data with R

Hi,
I messed-up #246 by including additional files in PR.  So I have created a new one with the Rmd file only,
Robert

#268 Make sense. I close this.

@ivanek we're providing simple examples of uses of this data here: https://renkulab.io/projects/covid-19/covid-19-public-data - the data is pulled once a day from this repo and pipelines are run to update various notebooks and figures. Would be great if you could contribute this R example to the project!

Dear @rokroskar, I will be happy to do it.


global: refactored CFG variables

Replaces custom path concatenations with os.path.join

Refactors commonly used paths to use config variables.

Renames CFG_PREFIX to _CFG_PREFIX. updated by @jirikuncar

I would like some input on this in order to drop the WIP tag quicker:
Some references to _CFG_PREFIX remain and perhaps some legacy/ files were better off remaining similar to their master equivalents.
If accepted, supersedes  #2818

@dset0x it looks good. Please remove WIP and fix kwalitee except in invenio/legacy/* files.

@dset0x it looks fine. Can you polish commit messages (^(bib|web).* -> legacy, testsuite -> tests)?

ping @dset0x
(cc @jalavik @jmartinm)

So there is a few different things here, but it looks all fine for me. However I did not yet try the new demosite populate or other refactored commands - so will need to do that.

There are still some problems with tests. Can you check this one?
======================================================================
ERROR: test_usage (invenio.testsuite.test_legacy_docextract_convert_journals.ScriptTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/inveniosoftware/invenio/invenio/testsuite/test_legacy_docextract_convert_journals.py", line 61, in test_usage
    out = run_py_func(["convert_journals", "-h"], main).out
  File "/home/travis/build/inveniosoftware/invenio/invenio/base/utils.py", line 307, in run_py_func
    manager_run()
TypeError: 'list' object is not callable

----------------------------------------------------------------------

The rest should be solved after you rebase.

@dset0x please improve commit messages (kwalitee check message master..upstream/pr/2829).

base -> global
(bib|web)* -> legacy

@dset0x can you rebase and amend the component names? Thanks


Run the server without root rights

Running code as root is a redonculous security risk, and for me, a bluetooth server it's a little hard to justify.
What are the current causes for needing root rights, and could you remove the necessity for root?

Hi @jjpe, Last time I checked / used this. You had to run as root to be able to read the bluetooth streams. I am not sure whether that is still a requirement. And not sure what running this as root will give you?
What the security risk is, it's pretty dependant on pairing the device. Unless you are able to get in somehow via a snoop or something.
You can try and run it without the sudo command and seeing whether it still works. If it does, then that solves your concern

Ahh just looked at it, it's to be able to bind to the bluetooth socket. Like I said it used to be an issue, lemme know if it doesn't faceplant without that anymore


Print debug output in minified form if specified

This adds a config variable ASK_MINIFY_DEBUG_LOGS which, if set to True, minifies the request and response JSON dumps printed to the debug log. Default value is False.
This is useful when logging to CloudWatch, as it prevents the output from being broken up into multiple log events. CloudWatch automatically prettifies single-line JSON objects, and splits multi-line ones into one event per line.

This looks useful! I'll test it out today and give you feedback if any. It might be a better, more sensible default vs. the full pretty-printed form.

should this be default behavior with pretty printing as an option? That seems to be more inline with other apis, where you have to pass a prettyprint=True option.

TBH I actually thought the same thing, but erred on the side of conservatism for the initial PR. I'll make another commit when I get to work (3 hours) and flip it around.

Change made.

@sodle I think you're missing a change in models.py as I see the response being rendered using the old method call. Can you check that?
@johnwheeler what do you think about putting the if/else logic and config flag checking into the core._dgbdump() function? I think it makes the _dbgdump() calls cleaner and can remove the need for the class property that evaluates the Flask app config setting. (The app config value isn't unique to the instance of the Ask object, so I think it still makes sense if the lookup via current_app is done in _dbgdump().) Make sense?

@voutilad Yeah you're right... I just fixed that.
I second your suggestion to @johnwheeler. I also notice that the _dbgdump function from core.py is duplicated in models.py. I think we should delete the copy in models, instead importing the one from core. The function in core should then be modified to not accept an indent parameter, instead evaluating the ASK_PRETTY_DEBUG_LOGS config variable.

Change made.

@johnwheeler Both changes have been made. I should probably update the docs, since a configuration flag has been added. How do I do that (presumably using docs\make.bat)?

Where do we stand on merging this?

Merged. Thank you!

Thank you as well!


Model viewer vs Aframe AR.js

Description
Please provide a detailed description of the bug, how to reproduce it, and the
expected behavior.  Always include a code snippet, screenshot, any errors
reported in the console, and the model to help us understand and fix the
problem.
Live Demo

https://glitch.com/edit/#!/model-viewer

Browser Affected

[ ] Chrome
[ ] Edge
[ ] Firefox
[ ] Helios
[ ] IE
[ ] Safari

OS

[ ] Android
[ ] iOS
[ ] Linux
[ ] MacOS
[ ] Windows

Versions

model-viewer: vX.X.X
three.js: rXX

@Jibin-John This is the second bug you've left here with no description at all; is Github not saving your edits or something? If you can leave a description of what you're asking, I'd be happy to respond.


The basedir of brewed MySQL can't be initialized

MySQL couldn't be installed correctly, stuck in postinstall. I have tested brew install mysql and brew install --build-bottle mysql, both warn:
==> Installing mysql
==> Downloading https://cdn.mysql.com/Downloads/MySQL-5.7/mysql-boost-5.7.18.tar.gz
######################################################################## 100.0%
==> cmake . -DCMAKE_C_FLAGS_RELEASE=-DNDEBUG -DCMAKE_CXX_FLAGS_RELEASE=-DNDEBUG -DCMAKE
==> make
==> make install
==> ./mysql-test-run.pl status --vardir=/tmp/d20170517-3434-gap9rz
==> /home/dyhu/.linuxbrew/Cellar/mysql/5.7.18_1/bin/mysqld --initialize-insecure --user
Last 15 lines from /home/dyhu/.cache/Homebrew/Logs/mysql/post_install.01.mysqld:
2017-05-17 13:09:23 +0800

/home/dyhu/.linuxbrew/Cellar/mysql/5.7.18_1/bin/mysqld
--initialize-insecure
--user=dyhu
--basedir=/home/dyhu/.linuxbrew/Cellar/mysql/5.7.18_1
--datadir=/home/dyhu/.linuxbrew/var/mysql
--tmpdir=/tmp

Warning: The post-install step did not complete successfully
You can try again using `brew postinstall mysql`
==> Caveats
We've installed your MySQL database without a root password. To secure it run:
    mysql_secure_installation

MySQL is configured to only allow connections from localhost by default

To connect run:
    mysql -uroot

A "/etc/mysql/my.cnf" from another install may interfere with a Homebrew-built
server starting up correctly.

==> Summary
🍺  /home/dyhu/.linuxbrew/Cellar/mysql/5.7.18_1: 321 files, 417.9MB, built in 9 minutes 49 seconds

The postinstall didn't success.
A gist-logs is posted here
As you @sjackman  suggested, I tried:
dyhu@mit232:~$ /home/dyhu/.linuxbrew/Cellar/mysql/5.7.18_1/bin/mysqld --initialize-insecure --user=dyhu --basedir=/home/dyhu/.linuxbrew/Cellar/mysql/5.7.18_1 --datadir=/home/dyhu/.linuxbrew/var/mysql --tmpdir=/tmp

No errors return, but mysql's related dir still seems uncorrectly:
dyhu@mit232:~$ mysql.server start
Starting MySQL
 * Couldn't find MySQL server (/usr/bin/mysqld_safe)
dyhu@mit232:~$ mysql_secure_installation

Securing the MySQL server deployment.

Enter password for user root:
Error: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

What's wrong here? Thanks

It's running your host's /usr/bin/mysqld_safe rather than the brewed daemon. Try
hash -r
which -a mysql.server
PATH=$(brew --prefix)/bin:$PATH
which -a mysql.server
mysql.server start

@sjackman Not work... Brew's bin path has already included.
dyhu@mit232:~$ hash -r
dyhu@mit232:~$ which -a mysql.server
/home/dyhu/.linuxbrew/bin/mysql.server
dyhu@mit232:~$ PATH=$(brew --prefix)/bin:$PATH
dyhu@mit232:~$ which -a mysql.server
/home/dyhu/.linuxbrew/bin/mysql.server
/home/dyhu/.linuxbrew/bin/mysql.server
dyhu@mit232:~$ mysql.server start
Starting MySQL
 * Couldn't find MySQL server (/usr/bin/mysqld_safe)

That's a bug. I'm afraid that I'm not terribly familiar with MySQL. Try running
/home/dyhu/.linuxbrew/bin/mysqld_safe

See https://dev.mysql.com/doc/mysql-startstop-excerpt/5.7/en/mysqld-safe.html


Adding support for configuring modulepath from environment.conf.

Hey Jethro - Pull request with regard to our conversation about support for environment.conf.

Nice work @ljohnston, looks like a good quality commit - thanks for submitting!
I've merged it in and validated it works correctly for existing users. I've now gone and built a new gem and pushed it to rubygems so any new installs will include this contribution.


crash when targeting merchant by id in gh using hotkey

works fine in doa outpost

Initially reported by @Hour-of-the-Owl
Looked into this. Merchants have different ids, so the only way to make this work would be to create a lot of hotkeys. I was also unable to reproduce the crash, the hotkeys behaves as expected (if the target exists it's selected, otherwise nothing happens). I tried in doa and a few guild halls.

@Hour-of-the-Owl can't reproduce any more, closing


adds freezeCoreModules configuration option to mitigate memory leaks

resolves #6399, resolves #6814
Summary
Some commonly used libraries like agent-base and graceful-fs override node core modules' (a.k.a built-in modules) methods.
This causes a memory leak in jest since the long lived node core modules' state is now bound to the short lived test execution context.
Each time a new test runs, jest creates a new context, the context is then bound to the long lived core module state, thus increasing the leak.
Test plan
Tests were added here:
https://github.com/lev-kazakov/jest/blob/25fb8ba198878bef6f8b25baac0b79ce57f33fda/packages/jest-runtime/src/__tests__/runtime_require_module.test.js#L186-L211
For an e2e test, you can run the following test suite with this branch linked:
https://github.com/lev-kazakov/jest-leak-fixer/tree/master/test

@scotthovestadt thoughts on this one?

@SimenB , @scotthovestadt
another approach would be to allow users run tests without isolation.
i.e use native node require instead of jest's.
this would probably require creating a new environment + other refactoring.

That's not really an option - if you want that Jest is not for you :) Almost all of the features in jest depends on the isolation (mocks, transformations, fake timers etc).

Codecov Report

Merging #8331 into master will decrease coverage by <.01%.
The diff coverage is 62.5%.

@@            Coverage Diff             @@
##           master    #8331      +/-   ##
==========================================
- Coverage   62.29%   62.29%   -0.01%
==========================================
  Files         266      266
  Lines       10729    10744      +15
  Branches     2609     2613       +4
==========================================
+ Hits         6684     6693       +9
- Misses       3460     3463       +3
- Partials      585      588       +3

Impacted Files
Coverage Δ

packages/jest-config/src/Defaults.ts
100% <ø> (ø)
:arrow_up:

packages/jest-config/src/normalize.ts
77.85% <ø> (ø)
:arrow_up:

packages/jest-config/src/Descriptions.ts
100% <ø> (ø)
:arrow_up:

packages/jest-config/src/index.ts
12.5% <ø> (ø)
:arrow_up:

packages/jest-config/src/ValidConfig.ts
100% <ø> (ø)
:arrow_up:

packages/jest-runtime/src/index.ts
69.26% <62.5%> (-0.37%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4d3c1a1...105fc9b. Read the comment docs.

I've made a patch-package of this pull request if more people wants to test this tests against 24.7.1
https://gist.github.com/sibelius/f98e62dd9346ddc97f07fe3814dc1b6e
https://github.com/ds300/patch-package

More me the main aggressor is agent-base
Error: Trying to override method 'request' of a frozen core module 'https'
    at Object.<anonymous> (/app/node_modules/@slack/client/node_modules/agent-base/patch-core.js:45:15)

using this pull requests keeps my tests below 200MB

@SimenB , @scotthovestadt
ping

any updates on this PR?
It's a pain to debug memory leaks due to hotpatching in some deep dependency and it only affects us when testing.

Is things going in? We'd really benefit a lot from this PR over here.

@SimenB , @scotthovestadt any thoughts on this one?

@lev-kazakov can you rebase?

@sibelius done.

@SimenB , @scotthovestadt is there a chance we get this thing reviewed/merged/denied/whatever? :)

Ping @SimenB , @scotthovestadt any time to look at this PR?
We have reached the point at which we are trying to avoid upgrading packages in our main app because of the potential leaks that might happen during testing 😬

This needs to merged cuz it's really important

Please!

Or, can someone make a fresh patch / instruction how to patch/fork Jest to have this option? We're spending tons of money jumping between CI solutions only because Jest leaks memory...

We ended up biting the bullet and migrating all our tests to https://github.com/avajs/ava. I wanted to stick with jest, but there really wasn't an acceptable work around for this issue for us.

try this https://github.com/facebook/jest/pull/9443

I also experiencing memory leaks. I opened a new ticket (https://github.com/facebook/jest/issues/9697) but I'd like to know if you think it's the same issue. I can't really figure that one out. Memory leaks are tricky! Thanks!

The way Jest achieves sandboxes and loads modules definitely causes a leak in graceful-fs, and I've opened up a PR there now which (from my quick testing) fixes it.
Beyond that specific module, I still think the correct solution is to fix other modules to make sure they don't leak rather than trying to work around it in jest. I might be convinced to add this flag to jest since it's rather hard to track down, and sometimes triggered specifically by how Jest works

Can you link the PR here? So we can have as a baseline to fix other leaking modules
This flag could be useful when some package got some regression and start leaking

https://github.com/isaacs/node-graceful-fs/pull/184

Meanwhile, in case someone needs - I've just adapted this code to the recent<EMAIL_ADDRESS>to be applied with npx patch-package: https://gist.github.com/kirillgroshkov/d312f757da432b800bff5e2489a7e749
Without this patch our CI tests hang 90% of the time (CircleCI gives "Too long with no output (exceeded 10m0s): context deadline exceeded"). This patch fixes it.

Jest 25.5 has been released with the graceful-fs fix

@SimenB Thank you! I had to delete my package-lock because some of my dependencies (such as jest-junit) depend on<EMAIL_ADDRESS>but now everything is working without leaks!

Is the only guidance on debugging these issues in this GH issue and the related ones? I ended up being able to get our test suite to run (84 files, 520 total tests) by using<EMAIL_ADDRESS>and applying the patch from above.
When running locally with --runInBand --logHeapUsage, I can see it increasing consistently, up until about 1500MB then it somehow "resets" back down to 100MB.
Without the patch from above, the CI build always crashes with OOM.
I also tried the newest<EMAIL_ADDRESS>and the latest of graceful-fs as well as a newer version of agent-base, all with no success.
Again, even now, the memory balloons, but is able to at least get through CI.
Are there any other resources for this?

I'm not super opposed to landing this PR, I'd just prefer to fix the underlying issues rather than applying a band-aid and move on. Which modules leak for you?

then it somehow "resets" back down to 100MB.

That's GC running.

Thanks for the reply! Unfortunately, I haven’t been able to track down which modules are leaking. There are no console warning or errors when using the patch. Using —detectOpenHandles doesn’t surface anything, and —detectLeaks says that my tests are leaking but no other information.

try to profile both cpu and memory like this
node --cpu-prof --heap-prof ./node_modules/.bin/jest /path/to/test/withLeak

I think this option can dramatically improve experience in finding modules, that are leaking.

This has helped me find then-fs, and docker-modem (which has a copy-paste of follow-redirects, which also isn't fixed), and various new, exciting ways to reach agent-base, including axios and gaxios and nodemailer and google cloud's tooling.
I feel that this useful contribution allows me also to add a "plus one", which follows.
I spent probably days trying to find this with a heap debugger. These memory leaks are a killer for us; with ts-jest you start out at a leak of about 450MB, and we're seeing >50MB per test. We have approaching a thousand tests we'd like to run.
We work around this by splitting the test run manually, completely negating most of the benefits of jest.
A "quick fix" for this problem for us is to use the above Jest patch to find places where we're importing something, and change it to a lazy require(), so at least most tests don't fail.
import { foo } from 'dodgy-module';

function a() { /* not foo */ }
function b() { foo(); }

becomes
function a() { /* not foo */ }
function b() { require('dodgy-module')(); }

...totally ruining our TS and linting rules and, in some places, the code structure.

I think this could be useful to find leaks

<EMAIL_ADDRESS>does not have more memory leaks for our codebase
this patch was breaking ws package tests, so we removed it for now

latest version with node 14 still leaking
I think it is graceful-fs again

can we rebase this once again?

can we rebase this once again?

hi, sorry, no capacity.
if jest people agree to merge this, i'll rebase once again.

Jest 25.5 has been released with the graceful-fs fix

@lev-kazakov  Just be clear, the jest25.5 or ts-jest 25.5 or jest-core25.5. I will fix my verison

Jest 25.5 has been released with the graceful-fs fix

@lev-kazakov Just be clear, the jest25.5 or ts-jest 25.5 or jest-core25.5. I will fix my verison

i think you meant to tag @SimenB. i'm not part of the jest team.

Jest 25.5 has been released with the graceful-fs fix

@lev-kazakov Just be clear, the jest25.5 or ts-jest 25.5 or jest-core25.5. I will fix my verison

i think you meant to tag @SimenB. i'm not part of the jest team.

I like using your branch as a temp solution, till that merges into master. But still not very understand why

override node core modules' (a.k.a built-in modules) methods
case memory leak

Dear github bot, please don't close this useful PR, until we finally have a better solution for the huge memory leaks in jest or ts-jest
Sincerely,
a dev


refresh imports from 24Nov2024

closes https://github.com/monarch-initiative/mondo/issues/8437
Need to sort out if it's ok to merge this based on what changes need to be added from Mondo SOP and mondo-ingest pipelines. The refresh imports only takes ~20 min to run now with 40GB Memory so this can easily be re-done if needed.

@matentzn do you need this PR to be merged before any other pipeline PRs you're working on? If yes, then this can be merged now. I left this as draft to make sure it would not interfere with other pipeline PRs you're working on, but if this one is needed first then it's fine to merge it then.