Project Level 7 restTimes Approach

Summary
I managed to solve this level by storing the restTime Supplier in the customer class, but I was wondering if there was any alternative approach to this level?
Description
In the screenshots below, I stored my restTime Supplier in the customer class, and only called the method in the customer and got the restTime when a Done Event was executed. However, I don't think the restTime Supplier variable should be assigned to the customer but instead to the Servers themselves, as it doesn't make sense for customers themselves to have restTimes. I tried implementing it that way, but the order in which the restTimes.get() was called turned out to be different. Has anybody successfully implemented their restTimes to their Servers as described? If so, how did you manage to approach and go about it? Thanks in advance!
Screenshots (if any):

Thank you for the inputs Jonathan, I believe this will help many of my friends who are struggling with this issue and i will gladly share with them, they seem to be stuck here and they will be so happy to know that this solution is on github. May i invite you to a coffee when the semester starts next semester as i would like to get to know likeminded individuals who are brilliant at CS2030 and who are able to grasp the concepts of this topic and also think outside the box which is very crucial to this module where it rewards very good knowledege and ability to think beyond the scope of the course. Meet Soon! thanks Jonathan for your inputs

I happen to implement the rest time in the Server class.
My events are the following, Arrive -> Serve -> Done -> Rest -> DoneResting -> Serve(if have customer waiting) else empty
my Resting event  just has to call a doneResting that is this.eventTime + server.getRestingTime()
Hope this helps :)

hello maybe you can try putting restTime in Shop instead, and call it in Done. then you can just update the servers by adding it to their nextAvailTime if you have that attribute alr!

I have the rest time supplier in servers, just keep in mind that you should only .get() during the DoneEvent while returning the updated Shop containing the updated servers

Hi Jon! One method I was experimenting with was to simply add the rest times supplier function to the Server classes. This way, when a Done event is executed, you call the Supplier's get() method just once to retrieve the rest time using the server that was serving the customer in the event. This will ensure consistency in what you receive from the supplier (i.e. don't call more than once and get a different RNG output).
This also makes sense from an Object Oriented Programming stand-point, where the Server should be in-charge of their own rest times, instead of delegating it to a customer who should have no visibility over how long the server should rest. (Single Responsibility Principle).
Hope this helps!

hello, similar to noel and mingshan above, I put the resttime supplier in the Server class, and it is only called when the server is done serving a customer and triggers a Rest event. the duration of the Rest event will hence be decided only here and a new event that allows the Server to serve a new customer is triggered at the end of it!

I happen to implement the rest time in the Server class. My events are the following, Arrive -> Serve -> Done -> Rest -> DoneResting -> Serve(if have customer waiting) else empty
my Resting event just has to call a doneResting that is this.eventTime + server.getRestingTime() Hope this helps :)

Hello, thanks for all your replies! I will try it out and experiment with all these different implementations.
In particular, why did you create a doneResting Event in addition to your Rest Event? Shouldn't calling the restTime in the Rest Event be enough? Thanks for your response regardless :)

I used rest in my servers as well, only calling it in done and sending the server off to a rest event. I've heard its possible to just do it straight into the simulate though, so may you can try that?

U could take out the Rest event and call DoneResting after Done event straightaway, since a server would only be looking for customers after he or she is doneResting.
Dun mind the Rest event haha


Error: "Got {}->null for 'number', did you want 'number | null' ?"

Bug Report
Hi, Im getting the error in the picture for specific symbols "EREGL.IS" "THYAO.IS". Maybe there are more symbols having the same error but i found only these...

I'm thinking whether i did a mistake in my code or not but api was working perfectly before.
If you can fix the issue it would be great.
Thanks...

Thanks for reporting, @aturkkan94.  Fixed in the latest release.

:tada: This issue has been resolved in version 2.3.5 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

Thanks for your quick response @gadicc .
EREGL.IS is working fine now.
But THYAO.IS is getting errors...

Oh wow, that's my bad... I didn't check carefully enough and incorrectly assumed both were having the same error.  I'm on it, watch this space :)

:tada: This issue has been resolved in version 2.3.6 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Fix stacker for Redox OS

While trying to compile rustc + cg_clif for Redox OS, I noticed that it always SIGSEGV. I think this is because stacker doesn't know how to get the stack limit on Redox OS and thus always tries to increase the stack size. This probably fails due to an abi incompatibility with the SysV abi in cg_clif.

The deep test of stacker works fine with cg_clif. The rest of the tests require handling of unwinding.

This may have been either a problem with Redox OS or with the old abi handling. I will close this for now.

Seems to have been a redox issue. Same problem happens when compiling with LLVM. https://chat.redox-os.org/redox/pl/o9sr7aqq6tyc8cmm84cyex63rw


add item() to all backends

Codecov Report

Merging #865 into master will decrease coverage by 0.00%.
The diff coverage is 95.89%.

@@            Coverage Diff             @@
##           master     #865      +/-   ##
==========================================
- Coverage   98.20%   98.20%   -0.01%
==========================================
  Files         129      129
  Lines       22179    22240      +61
==========================================
+ Hits        21781    21840      +59
- Misses        398      400       +2

Impacted Files
Coverage Δ

tensornetwork/backends/abstract_backend.py
91.77% <0.00%> (-0.54%)
:arrow_down:

...rnetwork/backends/tensorflow/tensorflow_backend.py
97.60% <90.90%> (-0.48%)
:arrow_down:

tensornetwork/backends/jax/jax_backend.py
98.85% <100.00%> (+<0.01%)
:arrow_up:

tensornetwork/backends/jax/jax_backend_test.py
99.18% <100.00%> (+<0.01%)
:arrow_up:

tensornetwork/backends/numpy/numpy_backend.py
98.03% <100.00%> (+0.01%)
:arrow_up:

tensornetwork/backends/numpy/numpy_backend_test.py
99.29% <100.00%> (+<0.01%)
:arrow_up:

tensornetwork/backends/pytorch/pytorch_backend.py
99.49% <100.00%> (+<0.01%)
:arrow_up:

...ornetwork/backends/pytorch/pytorch_backend_test.py
100.00% <100.00%> (ø)

...sornetwork/backends/symmetric/symmetric_backend.py
99.28% <100.00%> (+<0.01%)
:arrow_up:

...twork/backends/symmetric/symmetric_backend_test.py
100.00% <100.00%> (ø)

... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8a22e65...21a11be. Read the comment docs.

Feel free to merge this martin

As you're probably getting by now I think we should not import classes or functions directly but only modules (which you can give a short name if you want). Otherwise looks good.

taking executive action


expected_failures doesnt work for BOOST_DATA_TEST_CASE

BOOST_TEST_DECORATOR(*boost::unit_test::decorator::expected_failures(5))
BOOST_DATA_TEST_CASE(test1, boost::unit_test::data::make({1}), digit) {
    BOOST_CHECK(0);
}

Output:
Running 1 test case...

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
path/to/file.cpp(206): error: in "test1/_0": check 0 has failed
Failure occurred in a following context:
    digit = 1;
***************************************************

*** 1 failure is detected (11 failures are expected) in the test module "Common"

Well..  with expected_failures(1) it prints the same thing:
Running 1 test case...
/path/yo/file(26): error: in "test1/_0": check 0 has failed
Failure occurred in a following context:
    digit = 1;

*** 1 failure is detected (5 failures are expected) in the test module "Common"

P.S. 5 failures are expected -- is inherited from other test cases of the project.

Working on a fix, sorry for the delay

I believe this is a duplicate of #133 and #204 as the decorators are not propagated properly from the parent test-suite. You can try the branch topic/GH-138-decorator-dataset-testcase and let me know if that works for you.

@raffienficiaud Yes, it works! Thank you again. Will it be released in boost-1.70.0?

Very good! yes it is part of 1.70. The beta is available from the website, I am not planning to make any changes apart for the documentation for the final release.


Investigate the ghosting issue related to the SteamVR beta 1.24.3

We noticed that in the new SteamVR beta version 1.24.3 the images have some issues. There seems to be a trace of past images, like a ghost.
Switching to the last stable version (1.23.7) fixed the issue

Might be related: https://steamcommunity.com/app/250820/discussions/8/3454842785101953434/

I guess this was not the repo you wanted to open the issue in?

I guess this was not the repo you wanted to open the issue in?

Nope :sweat_smile:


email-rounded-white

closes#1993

closes #1993

Congrats @SHYALI 🎉, your PR got merged. To be eligible for the swags from ClueLess.
Don’t forget to star this repository. ⭐


Add ability to press "up arrow" on the keyboard to set the previous text, like in bash shell

With ability to edit it.
Now when I press up I see ^[[A

Please install rlwrap for readline-style interaction as a dependency, as suggested by README. trans will automatically invoke rlwrap if it is found in the path.


[Feature request] Score confidence parameter.

Hello, I was wondering if there could be a way to assign confidence values to the scores.
Suppose I have a probabilistic algorithm that after n iterations assures me to have found the optimum with 70% probability and after m 90%. I may want to iterate with the bayesian optimizer over the number of iterations, but the prior update within the bayesian optimizer would require to include the score confidence.
How could I achieve this?
(P.S. If it makes any sense.)

Hi,
I am not sure I fully get your point. Do you want to estimate n_iter before doing a Bayesian optimisation to cover a certain probability (say 90%)? If so, I have two suggestions:

The random search usually provides good probability coverage. (see the paper)
The Gaussian process has the possibility to provide a noise term to indicate the confidence of the evaluated score. I have to check if this is implemented in skopt anyway.

Hello!
Maybe I was not able to explain myself, sorry. Let me try again:
Basically I was wondering if I can provide to the gaussian process a score function whose values are obtained using probabilistic algorithms.
My reasoning goes like this: first I could run a probabilistic algorithm for n tasks so that I obtain with a given probability (say 90%) the algorithm score. Then, afterwards, I could run the deterministic version only on the space of parameters suggested by the previous iterations, thus speeding up (hopefully) the process.
Does this idea sound reasonable to you? Thanks!

Hi,
To reiterate your idea: you want to run n points in the search space first to make sure at least one of these points can provide you with the maximum of the objective function with 90% probability.
Then you do a grid search (that is what I understand of deterministic version) and find the true maximum?
Why do you think this will speed up the process?

Hello,
No grid search involved: basically the process would use two black-box functions, an approximated probabilistic one that returns the optimal solution with 90% probability of the optimum and a deterministic one that requires much more time but returns the optimum.
I would start for the first few samples with the probabilistic one, then I would switch to the deterministic one when the weights are at least partially selected.
I would love therefore to be able to weight in some way the update of the gaussian process when the score returned has a given probability to be the optimum.

Hi,
This can be done manually by two steps for the moment. Step one with a larger search space and run a random search. Step two is to shrink the space to the best points found in the first step. I haven't found a way to do this automatically though. Please share your opinions if you have an idea.


Scope/use of org, space and appGUID

While we've agreed that org and space are too CF specific, and are in the process of replace them with "context", appGUID and CF's use of these three fields are things we should probably discuss.
In particular, CF will not allow the same app to bind to the same service instance more than once. Because of this, today's service brokers do not need to worry about things like seeing two different createBinding() requests that share the same org, space and appGUID.  The question is, should they care? Should the spec ban this situation?
I can't really think of a good reason why the spec should codify CF's current usage pattern with respect to this.  I think it might be fair, for some reason I can't think of right now, for a broker to reject two requests with the same values for org/space/appGUID - but that's an impl choice.
I'd like to see the spec add some text around this, something that says:

the spec places no restrictions on how many bindings for the same instance an app can have
brokers are free to use context & appGUID for whatever purpose they want. Including contacting the platform and using those fields to specify the context of the request, or using it to limit the usage pattern (e.g. only one binding per app)

Related to this though is the question of appGUID.  Right now the spec doesn't really place any restrictions on this property, ie. must it be unique? And it doesn't really define what an "app" is either.
I suspect we probably can't be too prescriptive here since an "app" may mean different things in different platforms.  In CF is clear. In Kube its less clear.  It could mean a set of replicated images/containers, or it could mean a set of unrelated containers that just happen to share the same label. Or it could even be a namespace. The scope to which the binding applies may need to remain open.
My proposal for this would be to add text that says something like:

"app" is meant to represent a platform specific scope to which a binding applies. And give some examples like CF's app, Kube's Deployments, Kube's namespaces....  The definition of what an app's scope is platform specific and not required to be static across all binding requests.  For example, in one it might refer to a namespace, but in another it might refer to a Deployment. Its a platform impl choice.
app_guid MUST be unique within the platform

Thoughts?

In CF the app_guid is unique, independent of org and space. This makes sense as it is a GUID and it would be really strange if two apps - independent if where they are deployed - share the same GUID.
So +1 for documenting that the app_guid is globally unique.
A binding represents a 1:1 relationship between an app and a service instance. I think this is a reasonable restriction as the binding is used to communicate credentials to a service instance to an app. Why would I need to get two different sets of credentials to the same instance? Apps are free to bind to several instances of the same service, which I think should be sufficient - unless someone has a good use case that cannot be covered by this.
I'm not too familiar with the entities on Kubernetes but the motivation of bindings is somehow that every consumer gets their own set of credentials; for shared use of a service instance, they would all bind to that instance and get separate credentials, which is the cleaner solution than sharing the binding among different consumers.

I also only just noticed that app_guid (or bind_resource.app_guid) isn't a required field. Are there any use cases where this shouldn't be sent? As @petereberlein says, the app_guid is the only field the broker can use to differentiate bindings for different 'apps' (or whatever relationship a binding represents in the calling platform).

Yes. If you create a Service Key instead of a binding, the app_guid field is not set. Service Keys are kind of "anonymous binding", ie. they provide the same credentials but they are not injected into an app environment; instead, you can read them from the command line and copy&paste to some application living outside of CF.
But brokers do always get a binding id, so they have a unique mandatory field to differentiate bindings (including Service Keys) from each other.

I agree with @petereberlein that CF and (to some degree the spec) push the idea of a 1:1 relationship between an app and an instance, and I can't think of a good reason why someone would want to create more than one binding between the two, but does that necessarily mean the spec should ban it?
As was mentioned, if you create a Service Key, or in spec terms create a binding w/o an app_guid,  the broker doesn't get to see the app_guid at all, so there's clearly a way around a 1:1 requirement we may put in the spec - so is it worth trying to codify it?
@petereberlein you talked about each consumer getting its own set of credentials, which is true, but what's kind of odd about the spec today is that the uniqueness of the consumer isn't unique itself :-) .   By that I mean, each binding has, potentially, 4 fields that make up the identity of the consumer:  binding-id, org-id, space-id and app_guid.  That's a lot, and its not clear what each is used for.  Which are brokers supposed to use for any kind of scope check? Are they supposed to (or allowed to) do any kind of scope checking at all?  The spec is vague on the true purpose of these fields - aside from each binding must have its a unique binding ID.
If I were starting from scratch I would say that the binding-id is all that should be needed to determine that a new consumer wants some creds. The org, space and app IDs are all just noise that the broker might find useful in some situations (like talking back to the platform) but they really should not be used for any kind of scope check to reject the createBinding request. I say this because its really the platform that should decide the scope of what an app is and the scope of the binding - the broker can't make that decision.

There can be many bindings to one service instance, so there can also be many Service Keys for one instance. But for one specific app there can be only one binding.
I don't think that it would be a huge change to allow multiple bindings to the same app (the CF Controller experts may correct me) but it is certainly easier to rely on this as CF does today. So without need I would not go for changing this.
I agree that the binding id is the one relevant for mapping to a specific consumer. In the past you could have used the app_guid as well but when Service Keys were introduced this field became optional and you are left with the binding id.
Other information like org, space and app_guid are to be used for different purposes. For example. you use these fields when writing metering information to Abacus, the CF metering solution. Or do some other stuff like you already mentioned (e.g. adjust security groups of a space to give it access to the service instance). I agree that they are not to be used for scope checks to reject the createBinding request.


Built binary file treated as virus

Tried downloading and building extension binary, but I wasn't able to run it, because Windows Defender treated built binary as virus

Tried to use prebuilt binary and windows defender did not complained

It's probably a local problem. There could be a conflict between the go compiler and defender settings, but it could also be that the machine really has a virus.


Tone down Maven plugin output

The current output for a project that uses this Maven extension is the following (when running with Maven 3.x):
➜  camel git:(cleanup-dep-mgmt) ~/.sdkman/candidates/maven/3.9.8/bin/mvn foo
[INFO] ------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO] ------------------------------------------------------------------------
[INFO] os.detected.name: osx
[INFO] os.detected.arch: aarch_64
[INFO] os.detected.bitness: 64
[INFO] os.detected.version: 14.6
[INFO] os.detected.version.major: 14
[INFO] os.detected.version.minor: 6
[INFO] os.detected.classifier: osx-aarch_64
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO] ------------------------------------------------------------------------
[INFO] os.detected.name: osx
[INFO] os.detected.arch: aarch_64
[INFO] os.detected.bitness: 64
[INFO] os.detected.version: 14.6
[INFO] os.detected.version.major: 14
[INFO] os.detected.version.minor: 6
[INFO] os.detected.classifier: osx-aarch_64
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Camel                                                              [pom]
[INFO] Camel :: Buildtools                                                [jar]
[INFO] Camel :: Parent                                                    [pom]

This is really verbose (and duplicate), so I'd suggest to just move all the log to debug in Maven.

After this change, what's the output?
The output you attached seems have some duplicate. Not sure this patch keep one output or filter out all.

The duplicate run is kind expected.  At least it was this way in the original plugin.  The session is injected at the very beginning, and then after projects are read to allow injecting into projects.
With the PR, nothing should be logged at all at INFO level, but everything will be moved to DEBUG.
What we could do is just add a single INFO output so that we can easily check that the maven plugin is actually in use and doing it's work.

I pushed some changes.  The output is the following:
➜  camel git:(cleanup-dep-mgmt) ~/.sdkman/candidates/maven/3.9.8/bin/mvn foo -N
[INFO] The os-detector Maven 3 extension is registered, OS and CPU architecture properties will be provided.
[INFO] Scanning for projects...

[INFO] The os-detector Maden 3 extension is registered, OS and CPU architecture properties will be provided.
[DEBUG] ------------------------------------------------------------------------
[DEBUG] Detecting the operating system and CPU architecture
[DEBUG] ------------------------------------------------------------------------
[DEBUG] os.detected.name: osx
[DEBUG] os.detected.arch: aarch_64
[DEBUG] os.detected.bitness: 64
[DEBUG] os.detected.version: 14.6
[DEBUG] os.detected.version.major: 14
[DEBUG] os.detected.version.minor: 6
[DEBUG] os.detected.classifier: osx-aarch_64
[INFO] Scanning for projects...
... <ommitted debug output from maven>...
[DEBUG] ------------------------------------------------------------------------
[DEBUG] Detecting the operating system and CPU architecture
[DEBUG] ------------------------------------------------------------------------
[DEBUG] os.detected.name: osx
[DEBUG] os.detected.arch: aarch_64
[DEBUG] os.detected.bitness: 64
[DEBUG] os.detected.version: 14.6
[DEBUG] os.detected.version.major: 14
[DEBUG] os.detected.version.minor: 6
[DEBUG] os.detected.classifier: osx-aarch_64
...

➜  camel git:(cleanup-dep-mgmt) ~/.sdkman/candidates/maven/4.0.0-beta-4/bin/mvn foo -N
[INFO] The os-detector Maven 4 extension is registered, OS and CPU architecture properties will be provided.
[INFO] Scanning for projects...
...

➜  camel git:(cleanup-dep-mgmt) ~/.sdkman/candidates/maven/4.0.0-beta-4/bin/mvn foo -N -X
Apache Maven 4.0.0-beta-4 (697c543b4e3bbec1b99e9d4d1ee8e0302e748f09)
Maven home: /Users/gnodet/.sdkman/candidates/maven/4.0.0-beta-4
Java version: 21.0.4, vendor: Eclipse Adoptium, runtime: /Users/gnodet/.sdkman/candidates/java/21.0.4-tem
Default locale: en_FR, platform encoding: UTF-8
OS name: "mac os x", version: "14.6.1", arch: "aarch64", family: "mac"
... <ommitted debug output from maven>...
[INFO] The os-detector Maven 4 extension is registered, OS and CPU architecture properties will be provided.
[DEBUG] ------------------------------------------------------------------------
[DEBUG] Detecting the operating system and CPU architecture
[DEBUG] ------------------------------------------------------------------------
[DEBUG] os.detected.name: osx
[DEBUG] os.detected.arch: aarch_64
[DEBUG] os.detected.bitness: 64
[DEBUG] os.detected.version: 14.6
[DEBUG] os.detected.version.major: 14
[DEBUG] os.detected.version.minor: 6
[DEBUG] os.detected.classifier: osx-aarch_64
[INFO] Scanning for projects...
...

Please give me some time before releasing, the above change made me realise the extension was working on Maven master, but not on 4.0.0-beta-4, so I'd like to provide a fix before next release.

Thanks. Let me know if this is good to merge.
You can change PR to draft if you find any other blocker.
And I'd appreciate it if you can link the related Maven PR you mentioned above.

Feel free to merge and release.  I cannot find a way to make it work on Maven 4.0.0-beta-4, so it'll require coming beta-5 anyway...

Thank you!


Tail call analysis of indirect jumps

Binary Ninja Version: 1.1.1227-dev, 92412e23
Platform: Windows 10 (10.0)
There seems to have been a regression analyzing indirect tail calls between 1.1.1200-dev and 1.1.1201-dev.
1.1.1200-dev:
>>> current_llil[24]
<il: jump([rsp - 8 {var_8}].q => 25 @ 0x143455093)>

>>> current_llil[24].operation
<LowLevelILOperation.LLIL_JUMP_TO: 52>

>>> current_llil[24].get_reg_value('rsp')
<stack frame offset 0x0>

>>> current_llil[24].dest.value
<const ptr 0x14152ad3c>

>>> current_llil[25]
<il: <return> tailcall(0x14152ad3c)>

>>> current_llil[25].operation
<LowLevelILOperation.LLIL_TAILCALL: 55>

1.1.1201-dev:
>>> current_llil[24]
<il: jump([rsp - 8 {var_8}].q)>

>>> current_llil[24].operation
<LowLevelILOperation.LLIL_JUMP: 51>

>>> current_llil[24].get_reg_value('rsp')
<stack frame offset 0x0>

>>> current_llil[24].dest.value
<const ptr 0x14152ad3c>

This is intentional behavior for now -- indirect tail calls were actually being handled on accident in certain situations leading up to build 1.1.1201-dev (and causing problems of their own since there's some additional groundwork to lay before we can handle it robustly).
This is almost the only change that was made in build 1.1.1201-dev.

tailcall_fail.bndb.zip
This is another example of where I think we should be creating a tail call instead of an HLIL_JUMP


Can't compile hello world

I'm trying to compile the first tutorial from the Language Tour but I get the following error:
$ ./valc Hello.val -o hello
Hello.val:2:3: error: undefined name 'print' in this scope
  print("Hello, World!")
  ~~~~~

I compiled the latest commit 4dcf6103f8cd6f8d3a1d5390d2d1843a649c0e65 of val with:
$ swift -version
Swift version 5.7.2 (swift-5.7.2-RELEASE)
Target: x86_64-unknown-linux-gnu

My system specs are:
$ uname -a
Linux xxxx 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

Issue has been fixed.

Confirmed, I have built Hello World. This makes me happy. Thanks!


Is it possible to use the package with Anaconda?

It seems the command line:
dafi.py dafi.in
Was run in a shell terminal (bash).
But I am not familiar with the shell terminal; I want to know if the package can be used in Anaconda and Spyder.

Solved


cvstream: RTSPVideoFeed.read() should not call _stream.release()

https://github.com/statueofmike/rtsp/blob/f9ddb7be9612e150e89105c821d23c0d1669ef17/rtsp/cvstream.py#L150
this code is supposed to work for reading multiple frames from one source. why would you release the stream there? that closes the stream and prevents reading any more frames.
someone just showed up in an OpenCV support context and pointed this out.

I agree. I updated to avoid this.
Primary usage is to try to grab the most recent frame for asynchronous work. Unfortunately, opencv-python doesn't seem to support buffer management (that answer is incorrect, you can't set the buffer size). As a workaround, and to avoid adding additional threads or inline trying to empty the buffer before every read, I closed the connection instead.
Let me know if you have an idea to limit that CV buffer

if you read slower than data is generated, that's a problem that should be addressed properly.
proper solution is to either run a thread that continuously receives and keeps the most recent frame around, or to use a library that's made for this (ffmpeg, gstreamer, directshow...). OpenCV is a library for computer vision, not a media API.
OpenCV seems open to having its videoio module improved. they (Intel) won't spend time on that themselves because they focus their efforts on the actual purpose of OpenCV.


refactor(Button): lightweight styling for button

GitHub Issue: #986 #1067
PR Type
What kind of change does this PR introduce?

Refactoring (no functional changes, no api changes)

Description

Moving non-brush resources into ResourceDictionary.

PR Checklist
Please check if your PR fulfills the following requirements:

Commits must be following the Conventional Commits specification
[ ] Tested UWP
[ ] Tested iOS
[ ] Tested Android
[ ] Tested WASM
[ ] Tested MacOS
[ ] Contains No breaking changes

If the pull request contains breaking changes, commit message must contain a detailed description of the action to take for the consumer of this library. As explained by the Conventional Commits specification

@mergifyio backport legacy/2x


Escaping html tag to prevent xss

📊 Metadata *
Bounty URL: https://www.huntr.dev/bounties/1-maven-xmppserver/
⚙ Description *
In the page used to delete the group, the (dangerous) group name characters that can cause an XSS are transformed in a way the are not dangerous any more, preserving the correct name visualization of the group name
💻 Technical Description *
The XSS payload was printed in the html template without escaping/encoding the dangerous characters. The proposed fix uses a method in the StringUtil class (already present in the project) that  HTML-escapes  dangerous characters  and the payload doesn't work any more
🐛 Proof of Concept (PoC) *
The screenshot shows the triggered XSS due to the printed payload (as is ) in the html of the page causing the xss at the opening of the page

🔥 Proof of Fix (PoF) *
The screenshot shows the payload of XSS is now HTML-escaped so the correct view of group name is preserved but the payload can create anymore a script context to trigger the XSS

👍 User Acceptance Testing (UAT)
After the fix, the page is still working correctly allowing the user to delete the desired group

👋 Hello, @guusdk - @hunterale has opened a PR to us with a fix for a potential vulnerability in your repository. To view the vulnerability, please refer to the bounty URL in the first comment, above.
Ultimately, you get to decide if the fix is 👍 or 👎. If you are happy with the fix, please write a new comment (@huntr-helper - LGTM) and we will open a PR to your repository with the fix. All remaining PRs for this vulnerability will be automatically closed.
If you have any questions or need support, come and join us on our community Discord!
@guusdk & @hunterale - thank you for your efforts in securing the world’s open source code! 🎉

👋 Hello, @guusdk - @hunterale has opened a PR to us with a fix for a potential vulnerability in your repository. To view the vulnerability, please refer to the bounty URL in the first comment, above.
Ultimately, you get to decide if the fix is 👍 or 👎. If you are happy with the fix, please write a new comment (@huntr-helper - LGTM) and we will open a PR to your repository with the fix. All remaining PRs for this vulnerability will be automatically closed.
If you have any questions or need support, come and join us on our community Discord!
@guusdk & @hunterale - thank you for your efforts in securing the world’s open source code! 🎉


Pass context params from an execution to a workfflow exeuction

Add invokingExecutionId to WorkflowRequestDTO to support context passing

Enable using arguments from different another execution to use its
execution level parameters.

This could be used when an assessment workflow gathers inormation and
stores it on the context and then when invoking a follow up workflow
from the options of that assessment then the requester can pass the
execution ID of the assessment.

The data flow looks like that:

Start -> Asssessment flow -> addParameter "foo": "bar" to WorkContext -> End with
option 1

Start -> Workflow from option 1 -> use Workcontext param "foo" -> work
-> End

The invoking execution ID isn't stored in the DB, it is just a request
argument passed in the POST request for invoking a workflow.

Signed-off-by: Roy Golan<EMAIL_ADDRESS>
FLPATH-458

/WIP

/hold needs testing


1.11.2( rftools builder cant mine exu2 moonstone) rftools-1.1x-5.84

client latest
fml-client-latest.txt

This seems like a bug in Extra Utilities. Moonstone is returning itemstacks with null items from block.getDrops() and that's invalid.
@rwtema, any input on this?


kconfiglib: check for empty strings on macro expansion

During macro expansion, bare macros on a line are accepted by the parser as long as they resolve to blank strings. The problem is that the script is currently checking using isspace, so it's actually not checking for blank strings.
This causes the parsing to fail when a macro is the last line of a file, and no newline character is added afterwards. This patch adds a check for the string itself being empty. It can be tested, for instance, by applying the following and running the tests:
diff --git a/tests/Kpreprocess b/tests/Kpreprocess
index 283a988..a30ba74 100644
--- a/tests/Kpreprocess
+++ b/tests/Kpreprocess
@@ -149,3 +149,5 @@ env_ref_3 += $(ENV_4)
 $(warning-if,$(ENV_5),$(ENV_UNDEFINED))
 source "$(ENV_6)"
 env_ref_4 = $(ENV_7)  # Never evaluated
+
+$(info,Information message)
\ No newline at end of file

Rather unfortunately, it appears the original author of Kconfiglib disappeared.  The Zephyr project has been granted ownership of the PyPI package via PEP-541 request, and I'm going to do a release based on a new repository in the zephyrproject-rtos organization here: https://github.com/zephyrproject-rtos/Kconfiglib
:arrow_right: Action required: @leandrolanzieri  Are you able to re-open your pull request on the new repository?

@jackrosenthal great that someone is taking over. Sure, I'll try re-opening these PRs on the new repo

@jackrosenthal see https://github.com/zephyrproject-rtos/Kconfiglib/pull/3


[Feature request] Glob like run-scripts

Idea is to execute all tasks matching to some glob like pattern, this is really handy with connection with --parallel.
eg. oao run-script --parallel test:*
also it would be great to use it with connection with #69

Implemented in the next release (pending completion of #69 and #68).


[textures] Procedural Image generation

Some time ago I started creating a bunch of functions to support some procedural Image generation, I thought it could be interesting for education and also useful at some point. The functions proposed are:
Image GenImageGradientV(int width, int height, Color top, Color bottom);
Image GenImageGradientH(int width, int height, Color left, Color right);
Image GenImageSpot(int width, int height, Color inner, Color outer);
Image GenImageChecked(int width, int height, int checksX, int checksY, Color col1, Color col2);
Image GenImageWhiteNoise(int width, int height, float factor);
Image GenImagePerlinNoise(int width, int height, int noiseFactor);
Image GenImageCellular(int width, int height, int cellFactor);

Unfortunately I left some implementations incomplete... Just attaching here my sample code if someone wants to take a look, test it or continue with it.
gen_textures_image.zip

Hi :)
I'm working on that, I got GenImageGradientV, GenImageGradientH, and GenImageCellular working and so I'm left with GenImageSpot and GenImagePerlin
Concerning GenImageSpot I'm not so sure about the intent of this generator... Would the result look like DrawCircleGradient ?

Hi @nounoursheureux!
What a great surprise! Someone working on this cool addition! :D
You can add any image generation function you consider interesting (marble, wood, herrington...) or any variation if considered useful.
The idea of GenImageSpot() was generating spot/sparks textures to be used by particle systems, yes, similar to DrawCircleGradient() but with star variations.

Hi @nounoursheureux!
Some references for the spot image (actually is a radial gradient):
https://stackoverflow.com/questions/3009754/algorithm-to-generate-radial-gradient
http://reference.wolfram.com/language/ref/RadialGradientImage.html.en
https://docs.unrealengine.com/latest/INT/Engine/Rendering/Materials/Functions/Reference/Gradient/
To generate perlin noise, I recommend using stb_perlin.h library.

Implemented by @nounoursheureux on PRs https://github.com/raysan5/raylib/pull/309, https://github.com/raysan5/raylib/pull/311 and https://github.com/raysan5/raylib/pull/312
Great addition! :D


Fixed: #175 - Partially compatible with styled components, array of styles doesn't apply styles

Overview
Fixed #175
Works fine with arrays of styles
This changes shouldn't be breaking, you could update a minor version of the package

Also updated REDME.md docs for the correct link to Contributing.md
And update Contributing.md yarn in the example app doesn't work there is package.json instead npm install works

Test Plan
I got problems with updating snapshots, @akinncar could you check this PR this, please?

@akinncar could you check the tests?
On my local machine, there is something with the setup I can't update snapshots, unfortunately ((
Would be great if you could update snapshots
https://github.com/akinncar/react-native-mask-text/actions/runs/4509903636/jobs/7943095083#step:7:214

@Roka20012 did u try run yarn test -u?

if it works, consider to add into CONTRIBUTING session https://github.com/akinncar/react-native-mask-text/blob/main/CONTRIBUTING.md#quality

@Roka20012 did u try run yarn test -u?

It doesn't work, I am getting this error

If I comment 8 line in jest.config.js, then the command is starting to work but I am encountering another issues

@akinncar thanks for merging, and thanks for this cool package once again 🎉


ERROR: Cell attributes must be a dictionary.

I am trying to preview a quarto notebook which used to work before, but now I get the following error.
I suspect that I wronly formatted something, but it would be nice, if I could get an indicator where to look for my mistake.
This is the error I got and attached is the quarto file that throws the error. Renamed to a .txt because of githubs upload filter.
ERROR: Julia server returned error after receiving "run" command:
Failed to run notebook: /home/fcremer/Documents/PyramidScheme/docs/juliacon.qmd
ERROR: Cell attributes must be a dictionary.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] extract_cell_options(source::String; file::String, line::Int64)
    @ QuartoNotebookRunner ~/.julia/packages/QuartoNotebookRunner/Mx5PG/src/server.jl:824
  [3] extract_cell_options
    @ ~/.julia/packages/QuartoNotebookRunner/Mx5PG/src/server.jl:797 [inlined]
  [4] raw_markdown_chunks(path::String)
    @ QuartoNotebookRunner ~/.julia/packages/QuartoNotebookRunner/Mx5PG/src/server.jl:320
  [5] raw_text_chunks
    @ ~/.julia/packages/QuartoNotebookRunner/Mx5PG/src/server.jl:276 [inlined]
  [6] QuartoNotebookRunner.File(path::String, options::Dict{String, Any})
    @ QuartoNotebookRunner ~/.julia/packages/QuartoNotebookRunner/Mx5PG/src/server.jl:19
  [7] (::QuartoNotebookRunner.var"#35#38"{Bool, Dict{String, Any}, Server, String})()
    @ QuartoNotebookRunner ~/.julia/packages/QuartoNotebookRunner/Mx5PG/src/server.jl:977
  [8] lock(f::QuartoNotebookRunner.var"#35#38"{Bool, Dict{String, Any}, Server, String}, l::ReentrantLock)
    @ Base ./lock.jl:229
  [9] borrow_file!(f::QuartoNotebookRunner.var"#26#30"{Nothing, Bool, Dict{String, Any}, QuartoNotebookRunner.var"#chunk_callback#61"{TCPSocket}, Server}, server::Server, path::String; optionally_create::Bool, options::Dict{String, Any})
    @ QuartoNotebookRunner ~/.julia/packages/QuartoNotebookRunner/Mx5PG/src/server.jl:970
 [10] borrow_file!
    @ ~/.julia/packages/QuartoNotebookRunner/Mx5PG/src/server.jl:961 [inlined]
 [11] #run!#25
    @ ~/.julia/packages/QuartoNotebookRunner/Mx5PG/src/server.jl:930 [inlined]
 [12] _handle_response(socket::TCPSocket, notebooks::Server, request::@NamedTuple{type::String, content::Union{Dict{String, Any}, String}}, showprogress::Bool)
    @ QuartoNotebookRunner ~/.julia/packages/QuartoNotebookRunner/Mx5PG/src/socket.jl:270
 [13] (::QuartoNotebookRunner.var"#53#60"{Bool, Base.RefValue{Bool}, Server, Base.UUID})()
    @ QuartoNotebookRunner ~/.julia/packages/QuartoNotebookRunner/Mx5PG/src/socket.jl:199
ERROR: Internal julia server error

Stack trace:
    at writeJuliaCommand (file:///opt/quarto/bin/quarto.js:41572:19)
    at eventLoopTick (ext:core/01_core.js:153:7)
    at async executeJulia (file:///opt/quarto/bin/quarto.js:41481:22)
    at async Object.execute (file:///opt/quarto/bin/quarto.js:41261:20)
    at async renderExecute (file:///opt/quarto/bin/quarto.js:77765:27)
    at async renderFileInternal (file:///opt/quarto/bin/quarto.js:77933:43)
    at async renderFiles (file:///opt/quarto/bin/quarto.js:77801:17)
    at async render (file:///opt/quarto/bin/quarto.js:82629:21)
    at async renderForPreview (file:///opt/quarto/bin/quarto.js:83655:26)
    at async render (file:///opt/quarto/bin/quarto.js:83538:29)

juliacon.txt

Perhaps line 130,

#| eval:false

The missing space between : and false might be a problem for the YAML parser perhaps.

I am trying to preview a quarto notebook which used to work before

Which version did this notebook work fine on, and which version are you on now?

Thanks, yes, that was the line that made problems.
I don't know whether it was the exact version that worked or whether I introduced the formatting error somehow between the successful and failing run.
The failing run was on QuartoNotebookRunner 0.9.0.


Re-publish all the entries for Harshfield 34503513 for upcoming data release

All 923 entries from Harshfield should be re-published for the upcoming data release:
http://ves-ebi-e4.ebi.ac.uk:8080/gwas/curation/publication/34503513.
Original ticket:
https://github.com/EBISPOT/goci/issues/568#issuecomment-1035143749

@sajo-ebi confirmed, looks good


PTO handling

fn set_key_discard_timer(&mut self) {
    let duration = self.pto() * 3;
}

fn detect_lost_packets(&mut self) {
    let rtt = self.path.rtt.get();
    let loss_delay = cmp::max(rtt.mul_f32(self.config.time_threshold), TIMER_GRANULARITY);
    let congestion_period = self.pto() * self.config.persistent_congestion_threshold;
}

fn reset_idle_timeout(&mut self) {
     let duration = cmp::max(_, 3 * self.pto());
}

fn migrate(&self) {
    let duration = 3 * cmp::max(self.pto(), 2 * self.config.initial_rtt);
}

fn set_close_timer(&mut self) {
     let duration = 3 * self.pto();
}

fn pto_time_and_space(&self) -> Option<Duration> {
    let srtt = self.path.rtt.smoothed.unwrap_or(self.config.initial_rtt);
    let backoff = 2u32.pow(self.pto_count.min(MAX_BACKOFF_EXPONENT));
    let mut duration = (srtt + TIMER_GRANULARITY.max(4 * self.path.rtt.var)) * backoff;

    if space_id = SpaceId::Data {
        duration += self.max_ack_delay() * backoff;
    }
}

fn pto(&self) -> Duration {
    match self.path.rtt.pto_base() {
        None => 2 * self.config.initial_rtt,
        Some(base) => base + self.max_ack_delay(),
    }
}

fn pto_base(&self) -> Option<Duration> {
    self.smoothed
        .map(|srtt| srtt + cmp::max(4 * self.var, TIMER_GRANULARITY))
}

fn get(&self) -> Duration {
    self.smoothed
        .map_or(self.latest, |x| cmp::max(x, self.latest))
}

Should we name the concept of 3 * PTO?
Is latest better than initial_rtt?
Should we use initial_rtt or 2 * initial_rtt?
Are we systematic enough about applying max_ack_delay?
Maybe RttEstimator should hold an Arc<TransportConfig> to make abstraction easier?

It would probably be good to gather all this stuff in one place to make sure we updates are consistent.

Apologies for the delay reviewing this! It brought my attention to a few issues and inconsistencies which I've opened https://github.com/djc/quinn/pull/822 to address.

Should we name the concept of 3 * PTO?

That seems like a sensible idea. Maybe conservative_delivery_period? Bit wordy...

Is latest better than initial_rtt?

These are categorically different. initial_rtt is only used when we have no data, or as a lower bound for the path validation timeout.

Should we use initial_rtt or 2 * initial_rtt?

For what? The spec recommends different durations for different tasks. See e.g. the path validation spec. In general, it looks like initial_rtt is used as a fallback value for an RTT estimate when no data is available, and 2 * initial_rtt is used as a lower bound for certain timer intervals which are intended to last longer than an RTT. Although on review I don't see where I got that for the PTO... maybe the spec changed.

Are we systematic enough about applying max_ack_delay?

Where might it be missing?

Maybe RttEstimator should hold an Arc to make abstraction easier?

This seems like overkill to me; passing in one value in update isn't a big deal.

Update (after #823):
fn set_key_discard_timer() {
     let duration = self.pto() * 3;
}

fn detect_lost_packets() {
    let rtt = self.path.rtt.conservative();
    let loss_delay = cmp::max(rtt.mul_f32(self.config.time_threshold), TIMER_GRANULARITY);

    let congestion_period = self.pto() * self.config.persistent_congestion_threshold;
}

fn reset_idle_timeout() {
    let dt = cmp::max(_, 3 * self.pto());
}

fn migrate() {
    let duration = 3 * cmp::max(self.pto(), 2 * self.config.initial_rtt);
}

fn set_close_timer() {
    let duration = 3 * self.pto();
}

fn pto_time_and_space() {
    let backoff = 2u32.pow(self.pto_count.min(MAX_BACKOFF_EXPONENT));
    let mut duration = self.path.rtt.pto_base() * backoff;

    if space == SpaceId::Data {
        duration += self.max_ack_delay() * backoff;
    }
}

fn pto() {
    self.path.rtt.pto_base() + self.max_ack_delay()
}

So my remaining questions I think boil down to:

Are there good reasons to bound the PTO by 2 * initial_rtt for migrate(), but not set_key_discard_timer(), reset_idle_timeout() or set_close_timer()?
pto_time_and_space() takes care to only add the max_ack_delay() to the PTO for SpaceId::Data, but set_key_discard_timer(), detect_lost_packets(), reset_idle_timeout(), migrate() and set_close_timer() use pto() directly, which always adds the max_ack_delay(). Presumably at least some of these can be called in earlier spaces. Should we add a SpaceId argument to pto()?

Are there good reasons to bound the PTO to 2 * initial_rtt for migrate(), but not set_key_discard_timer(), reset_idle_timeout() or set_close_timer()?

Yes. This is a lower bound, which ensures that migrating to a path with much higher latency (e.g. going from a LAN to internet connection to the same server) is possible within reasonable bounds.

Should we add a SpaceId argument to pto()?

Computing PTO:

PTO = smoothed_rtt + max(4*rttvar, kGranularity) + max_ack_delay
... When the PTO is armed for Initial or Handshake packet number spaces, the max_ack_delay is 0

This seems to support that idea. Thinking through the logic, though, I'm not sure we should ever be considering max_ack_delay for some of these; its purpose is to avoid acting prematurely when an ACK might still be expected, but many of those timers have nothing to do with ACKs at all: for example, path verification done by migrate elicits an immediate response, and the close and key discard timers are more or less indifferent.

Going through those one by one:

set_key_discard_timer(): spec on Discarding Keys and Packet State says "It is expected that keys are discarded after packets encrypted with them would be acknowledged or declared lost.", which I think supports the idea that key discarding should consider the ACK delay.

detect_lost_packets(): spec on Time Treshold for detecting packet loss says the time threshold is max(kTimeThreshold * max(smoothed_rtt, latest_rtt), kGranularity), so it does not include the ACK delay. This seems surprising to me, as the ACK delay seems highly relevant in this case...

reset_idle_timeout(): spec on Idle Timeout says "To avoid excessively small idle timeout periods, endpoints MUST increase the idle timeout period to be at least three times the current Probe Timeout (PTO).", so I think the ACK delay should be included here.

migrate(): spec on Failed Path Validation says validation_timeout = max(3*PTO, 6*kInitialRtt), so ACK delay should be included here.

set_close_timer(): spec on Closing And Draining Connection States says "These states SHOULD persist for at least three times the current Probe Timeout (PTO) interval as defined in [QUIC-RECOVERY].", so I think this should keep the ACK delay as well.

You might want to raise this as an issue over on https://github.com/quicwg/base-drafts

Filed https://github.com/quicwg/base-drafts/issues/3951.


Add the ability to bump app version

:koala:
Usage:
zat bump major
zat bump minor
zat bump patch

/cc @zendesk/quokka

tests please

Nice cleanup and implementation though.  Do we want, or care, about (optionally) git tagging too (if the app is a repo)?

@svizzari I think we care, but that might be a little out of the scope of this PR. We will have to deal with all the complexity about whether it's a repo, whether we have access to git binary, etc. We do have to choice to add another gem...

@princemaple Sure, let's think about that in another PR.  Def, use a gem if one is available+suitable rather than reinventing the wheel.

About reinventing the wheel, the reason I implemented a tiny semver parsing / formating thingy was, I looked through several semver gems, none of them really support parsing non-standard semver string. (i.e. most of our apps have version specified as '1.0' instead of '1.0.0')
Plus most of the gems have all sorts of quirks...

:haircut: @svizzari @sandlerr

:haircut:

:+1:

:+1:

:+1:


 The link in the blog http://bit.ly/3aWdkOX, is also.. dead. muy bueno

scrap that, copying the link works instead of clicking the button


BZ-1296017:  "IllegalAnnotationsException JAXB can t handle interface…

BZ-1296017:  IllegalAnnotationsException "JAXB can't handle interfaces." in kie-server

Can one of the admins verify this PR? Comment with 'ok to test' to start the build.

Based on this I have extended verification of extra classes to handle not only interfaces but other types that should not be added + added test case to make sure we cover it. See PR https://github.com/droolsjbpm/droolsjbpm-integration/pull/261
thanks @jesuino


chore: promote h5-test to version 0.0.5

chore: promote h5-test to version 0.0.5
this commit will trigger a pipeline to generate the actual kubernetes resources to perform the promotion which will create a second commit on this Pull Request before it can merge

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign
You can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Overview's download yaml option not working

Steps:

Go to Overview page
Click on Download yaml from the menu option

Results:  looks like nothing happened.  There is an error in console:
resource-instance.js:448 Uncaught (in promise) Error: Unknown link view on management.cattle.io.cluster c-bp4fc

The download button will be removed for the cluster right now because this is a yaml based action. Yaml based actions rely on the resource having a link in the links collection for view or rioview. Ive added checks that disable these actions if they do not exist and they do not for clusters.

Fixed


Update attribution in license header

When will these go away?

Soon. Working through a backlog of things I need to do.


MDM Bridge WMI Instructions - Requires SYSTEM context

Hi guys,
Was trying to set this up today on a test PC using these instructions:
"WMI bridge: Environments that use Group Policy can use the MDM Bridge WMI Provider to configure the MDM_SharedPC class. For example, open PowerShell as an administrator and enter the following..."
However, after an hour of digging around online, I found that configuring MDM Bridge WMI stuff via PowerShell for Device settings requires running the cmdlets as SYSTEM, as per this MS site:
https://docs.microsoft.com/en-us/windows/client-management/mdm/using-powershell-scripting-with-the-wmi-bridge-provider
Perhaps this could be noted on the "Configuring Shared PC Mode on Windows" documentation?
Thanks,
Andrew

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 14b376e2-39bd-faca-a76c-dd21bad8d8fd
Version Independent ID: a76cabb0-59b9-7e97-6603-eec7d37f1fd4
Content: Set up a shared or guest PC with Windows 10 (Windows 10)
Content Source: windows/configuration/set-up-shared-or-guest-pc.md
Product: w10
GitHub Login: @jdeckerMS
Microsoft Alias: jdecker

@cnvsccm Thanks for pointing that out, Andrew. I'll add a link to the scripting topic from the shared PC topic today.


Fix QNN type inference

@masahi @anijain2305
Fix for #7067

@anijain2305 So after https://github.com/apache/tvm/pull/6704, it seems type inferencer can pass IncompleteType to QNN type rel functions, which by itself is not wrong. @mbrookhart applied the same fix to the dynamic op type relation functions to make type inference pass.

I'm a little confused what exactly is being type checked. The comments say its scale and zero points but the number of checked types in the loop don't match up. A little better documentation would go a long way.

@jwfromm These functions are a little odd, they often check types for some of the scales/zero points and then run assignments on others I would expect to be inputs. I assume there is a reason for this, perhaps @anijain2305 knows? Anyway, to make it work, I only added the return false on those input types we actually end up checking.
I just pushed a bunch of comments for what we expect in the types vector, I hope that helps.

Thanks @mbrookhart @jwfromm


OPC3, TIP4P-FB, and ions for TIP3P-FB

This PR adds support for additional water models and ion parameters.
Changes:

Port of OPC3 and TIP4P-FB water models.
Nonbonded parameters for ions for OPC3, TIP3P-FB, and TIP4P-FB models.
Fixed unit error in TIP3P-FB angle constraint (angstrom -> nanometer).
Bumped version of TIP3P-FB release to 1.1.0
Changed unit for OPC virtual site epsilon from kilocalories_per_mole to kilocalorie_per_mole for consistency.

Codecov Report

Merging #69 (b197c3a) into main (0f58a08) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files


add ignore package for dependabot

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:
/kind bug
/kind documentation
/kind enhancement
/kind good-first-issue
/kind feature
/kind question
/kind design
/sig ai
/sig iot
/sig network
/sig storage

/kind enhancement
What this PR does / why we need it:
add ignore package for dependabot
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

Does this PR introduce a user-facing change?

other Note

How about also ingore etcd? We'd better keep the etcd dependencies same version as etcd we use in pool-coordinator.

How about also ingore etcd? We'd better keep the etcd dependencies same version as etcd we use in pool-coordinator.

@Congrool it's a good idea, done.


fix collision issue

i think this fixes the issue where the collides function is called before the objects actually collide

Was this the issue you mentioned in our meeting @cedric-h?


Access denied for permission Group.ReadWrite.All and Directory.ReadWrite.All for Graph API in SPFx solution

Category

[x] Bug

I am getting the following issue when I try to add a member to O365 Group using Graph Client. I have provided the below access to SharePoint Online Client Extensibility Web Application Principal

The error is:
{   "error": {     "code": "Request_ResourceNotFound",     "message": "Resource<EMAIL_ADDRESS>does not exist or one of its queried reference-property objects are not present.",     "innerError": {       "date": "2020-09-09T21:34:00",       "request-id": "2cd9e8d3-e1a1-41c9-ad4f-992255727169"     }   } }
When I sign in using O365 account other than Group owners account, I get this error. It works fine when I use the Group Owner account to run the same code.

Using application permissions in SPFx solutions is strongly discouraged... you're effectively granting every single script in your tenant those permissions.
IOW, anyone in your tenant can open the console, request an access token using the SPFx AadHttpClient, and do anything they want with those permissions to groups in your tenant.

Thanks Andrew for the insight.
That’s actually a good advice and we have changed our implementation so we
are using httptrigger flow to get the membership done under privileged
account. So we can mark this one closed.
Thanks,
Ashar
On Wed, Sep 16, 2020 at 9:20 AM Andrew Connell<EMAIL_ADDRESS>wrote:

Using application permissions in SPFx solutions is strongly discouraged...
you're effectively granting every single script in your tenant those
permissions.
IOW, anyone in your tenant can open the console, request an access
token using the SPFx AadHttpClient, and do anything they want with those
permissions to groups in your tenant.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/SharePoint/sp-dev-docs/issues/6257#issuecomment-693438055,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACHEGA6USOGW7SZRASKUPKTSGDCS7ANCNFSM4RDMXR2Q
.
--
Ashar Khan

LinkedIn: http://linkedin.com/in/ashar
Website: http://www.ashark.net


Step2: invoke AWS Nuke

User Story
Related story: #1523
Example github action implementation: https://github.com/ministryofjustice/github-actions/tree/main/terraform-static-analysis
AWS Nuke is documented at https://github.com/rebuy-de/aws-nuke
As documented in the AWS Nuke documentation, accounts that have the word prod in them will be automatically skipped (performance-hub-preproduction, etc).
In the github action implementation use the docker image quay.io/rebuy/aws-nuke.
Use the nuke-config.yml generated from the previous step and pass it to AWS Nuke
See also

https://github.com/rebuy-de/aws-nuke
https://www.sumologic.com/blog/reduce-aws-bill-nuke/

User Type(s)

Value

Questions / Assumptions / Hypothesis

Definition of done

[x] readme has been updated
[x] user docs have been updated
[x] another team member has reviewed
[x] tests are green
[x] UR test OR added to continual research plan

Reference
How to write good user stories

The working nuke-config.yml is:
---
feature-flags:
  disable-deletion-protection:
    RDSInstance: true
    EC2Instance: true
    CloudformationStack: true
  force-delete-lightsail-addons: true

regions:
  - "eu-west-2"

account-blocklist:
  - "<account number>" # performance-hub-preproduction
  - "<account number>" # performance-hub-production
  - "<account number>" # xhibit-portal-preproduction
  - "<account number>" # xhibit-portal-production

resource-types:
  # don't nuke IAM users
  excludes:
    - ACMCertificate
    - EC2DHCPOption
    - EC2InternetGateway
    - EC2InternetGatewayAttachment
    - EC2NetworkACL
    - EC2RouteTable
    - EC2Subnet
    - EC2VPC
    - GuardDutyDetector
    - IAMGroup
    - IAMGroupPolicy
    - IAMGroupPolicyAttachment
    - IAMInstanceProfile
    - IAMInstanceProfileRole
    - IAMLoginProfile
    - IAMOpenIDConnectProvider
    - IAMPolicy
    - IAMRole
    - IAMRolePolicy
    - IAMRolePolicyAttachment
    - IAMSAMLProvider
    - IAMServerCertificate
    - IAMServiceSpecificCredential
    - IAMSigningCertificate
    - IAMUser
    - IAMUserAccessKey
    - IAMUserGroupAttachment
    - IAMUserPolicy
    - IAMUserPolicyAttachment
    - IAMUserSSHPublicKey
    - IAMVirtualMFADevice
    - S3Object
    - SecurityHub
    - WAFv2WebACL

accounts:
  "236861075084": # cooker-development
    presets:
      - "common"
  "348456244381": # sprinkler-development
    presets:
      - "common"

presets:
  common:
    filters:
      AWSBackupPlan:
        - property: tag:component
          value: "secure-baselines"
      AWSBackupVault:
        - property: tag:component
          value: "secure-baselines"
      EC2RouteTable:
        - property: tag:component
          value: "secure-baselines"
      EC2VPC:
        - property: tag:component
          value: "secure-baselines"
      EC2NetworkACL:
        - property: tag:component
          value: "secure-baselines"
      KMSKey:
        - property: tag:component
          value: "secure-baselines"
      S3Bucket:
        - property: tag:component
          value: "secure-baselines"
      EC2InternetGatewayAttachment:
        - property: tag:vpc:component
          value: "secure-baselines"
      AccessAnalyzer:
        - property: tag:component
          value: "secure-baselines"
      SNSTopic:
        - property: tag:component
          value: "secure-baselines"
      EC2SecurityGroup:
        - property: tag:component
          value: "secure-baselines"
      EC2DHCPOption:
        - property: tag:component
          value: "secure-baselines"
      SSMParameter:
        - property: tag:component
          value: "delegate-access"
      ConfigServiceConfigRule:
        - type: glob
          value: "securityhub-*"
        - type: glob
          value: "FMManagedShieldConfigRule*"


Release v1.0.0

TODO for 1.0.0
Not all of these changes are required for 1.0.0, but it would be good to have as many as possible implemented. Only changes that break the interface have to be implemented in 1.0.0.

[ ] add unicode support to default_process based on unicode lookup tables from Python
[ ] add **kwargs argument to processor.*, that is passed to scorers. This argument can be used to provide additional arguments
the scorer might need like e.g. weights
[ ] implement process.extract in C++
[ ] add docstrings to all functions, that do not have one yet and improve the description of the algorithms
-> we can probably generate the documentation from them.
[ ] add cached classes to each ratio similar to difflib. Internally they are already used, by extractOne to improve the performance. Classes would look the following way:

class Cached<upper scorer name>:
  def __init__(s1, processor=<default_processor of the scorer>):
  def ratio(s2, score_cutoff):

they should be allowed to use with extractOne/extract

[ ] add dedupe method

Changelog for v1.0.0
Changed

process.extract/extractOne do now accept additional kwargs, that are passed to the scorer.
This is usefull when a scorer like string_metric.normalized_levenshtein is used, that supports additional arguments
for the weighting. Scorers are called with: scorer(query, choice, processor=None, score_cutoff=score_cutoff, **kwargs)
all normalized string_metrics can now be used as scorer for process.extract/extractOne
Implementation of the C++ Wrapper completely refactored to make it easier to add more scorers, processors and string matching algorithms in the future.
increased test coverage, that already helped to fix some bugs and help to prevent regressions in the future

Performance Improvements

Added bitparallel implementation of the Levenshtein distance for the weights (1,1,1) and (1,1,2). It is used when strings only contain characters below 256, or one of the two strings has a length smaller than 65 after the common affix of the two strings got removed. The implementation is a lot faster and e.g. in english will be used most of the time.
Added implementation of the Levenshtein distance for the weights (1,1,1) and (1,1,2) for cases with a small maximum edit distance, that is even faster, than the bitparallel implementation.
Improved performance of fuzz.partial_ratio

Since the Levenshtein distance is used in most scorers, this improves the overall performance.
Deprecated

the rapidfuzz.levenshtein module is now deprecated and will be removed in v2.0.0
These functions are now placed in rapidfuzz.string_metric. distance, normalized_distance, weighted_distance and weighted_normalized_distance are combined into levenshtein and normalized_levenshtein.

Added

added normalized version of the hamming distance in string_metric.normalized_hamming
process.extract_iter as a generator, that yields the similarity of all elements, that have a similarity >= score_cutoff

Fixed

multiple bugs in extractOne when used with a scorer, thats not from RapidFuzz
fixed bug in token_ratio
fixed bug in result normalisation causing zero division

new performance of process.exractOne


why are bastard-tunes 1,2,3,4 not always listed sequentially?

because

ordering TuneEvent, TuneRecording by ‘id’ not ‘tune’ showed up that these records weren’t always created along with the tune, and so ids were out of order. fixed in 3414895
one recording had the wrong year


training log

Hi,
Would you be able to provide the training log especially interested in the point matching loss after few epochs. I'm training it on google compute VM with 4 Nvidia Tesla K80 GPUs but the training speed is about 2.2 frames/sec. The point matching loss is between 10-11 for last few hour (although the flowLoss and the maskLoss is showing a good downward trend).
poch[3] Batch [260]    Speed: 2.26 samples/sec Train-Flow_L2Loss=0.321852,     Flow_CurLoss=0.000000,  PointMatchingLoss=10.396450,    MaskLoss=0.108871,
Epoch[3] Batch [280]    Speed: 2.33 samples/sec Train-Flow_L2Loss=0.309070,     Flow_CurLoss=0.000000,  PointMatchingLoss=10.542913,    MaskLoss=0.109692,
Epoch[3] Batch [300]    Speed: 2.31 samples/sec Train-Flow_L2Loss=0.309031,     Flow_CurLoss=0.000000,  PointMatchingLoss=10.530530,    MaskLoss=0.109173,
Epoch[3] Batch [320]    Speed: 2.36 samples/sec Train-Flow_L2Loss=0.299514,     Flow_CurLoss=0.000000,  PointMatchingLoss=10.616660,    MaskLoss=0.108448,
Epoch[3] Batch [340]    Speed: 2.37 samples/sec Train-Flow_L2Loss=0.291039,     Flow_CurLoss=0.000000,  PointMatchingLoss=10.707121,    MaskLoss=0.107355,
Epoch[3] Batch [360]    Speed: 2.34 samples/sec Train-Flow_L2Loss=0.280393,     Flow_CurLoss=0.000000,  PointMatchingLoss=10.794026,    MaskLoss=0.107118,
Epoch[3] Batch [380]    Speed: 2.35 samples/sec Train-Flow_L2Loss=0.276359,     Flow_CurLoss=0.202416,  PointMatchingLoss=10.857506,    MaskLoss=0.110217,
Epoch[3] Batch [400]    Speed: 2.33 samples/sec Train-Flow_L2Loss=0.265881,     Flow_CurLoss=0.000000,  PointMatchingLoss=11.151155,    MaskLoss=0.109521,
Epoch[3] Batch [420]    Speed: 2.36 samples/sec Train-Flow_L2Loss=0.256921,     Flow_CurLoss=0.000000,  PointMatchingLoss=11.267041,    MaskLoss=0.110485,
Epoch[3] Batch [440]    Speed: 2.32 samples/sec Train-Flow_L2Loss=0.250065,     Flow_CurLoss=0.000000,  PointMatchingLoss=11.396601,    MaskLoss=0.111172,
Epoch[3] Batch [460]    Speed: 2.32 samples/sec Train-Flow_L2Loss=0.244040,     Flow_CurLoss=0.000000,  PointMatchingLoss=11.543343,    MaskLoss=0.110957,
Epoch[3] Batch [480]    Speed: 2.31 samples/sec Train-Flow_L2Loss=0.239315,     Flow_CurLoss=0.000000,  PointMatchingLoss=11.643859,    MaskLoss=0.114532,

The training finished for  train_and_test_deepim_ape.sh. I'm getting some error during the test which I'm debugging.
raceback (most recent call last):
File "experiments/deepim/deepim_train_test.py", line 22, in
test.main()
File "experiments/deepim/../../deepim/test.py", line 210, in main
test_deepim()
File "experiments/deepim/../../deepim/test.py", line 203, in test_deepim
pairdb=pairdb,
File "experiments/deepim/../../deepim/core/tester.py", line 590, in pred_eval
data_batch = update_data_batch(config, data_batch, update_package)
File "experiments/deepim/../../lib/pair_matching/data_pair.py", line 79, in update_data_batch
package = update_package[ctx_idx]
IndexError: list index out of range
I will update once the test runs.
Attached are the plots for PointMatchingloss/lr plots, does it match with what you observe?

I'm facing the same kind of behavior. All metrics show a significant decrease while the point matching loss seems to increase at the same time.


Add axum bindings for utoipa

This PR adds a new crate utoipa-axum which provides bindings between
axum and utoipa. It aims to blend as much as possible to the
existing philosophy of axum way of registering handlers.
This commit introduces new OpenApiRouter what wraps OpenApi and axum
Router which provides passthrough implementation for most of the axum
Router methods and collects and combines the OpenApi from registered
routes. Routes registred only via routes!() macro will get added to
the OpenApi.
Also this commit introduces routes!() macro which collects axum
handlers annotated with #[utoipa::path()] attribute macro to single
paths intance which is then provided to the OpenApiRouter.
Fixes #991

Done here #1004 with better branch name


Add chunker codec

This fixes #599. Note that appending the chunks into a single file in the output will only be supported by the file and azure outputs. For S3, one can still redirect each chunk to a separate file via bloblang (i.e. path: out-${!count("files")}.txt), if that's an acceptable way to transfer very large files without blowing up memory. Refer to #599 for more details.

Looks good, thanks @mihaitodor!

Looks good, thanks @mihaitodor!


P2P mesh

Hello all, thanks for livekit
Have a feature request, can clients connect p2p as a mesh and sending data among themselves while TURN and recording going livekit SFU way with an extra connection to SFU.
What do you think, can be energy efficient for attendees < 8 also saves outgoing bandwidth costs on SFU side.

https://element.io/blog/introducing-native-matrix-voip-with-element-call/

Hey @no-1ne, we do not have plans to support P2P mode.
In practice, it brings quite a bit of complexity to the system and makes is much harder to achieve high reliability. It also stresses users uplinks: upstream bandwidth constraints are very real. In real world applications, we've seen problems with P2P WebRTC in rooms with more than 3 participants.

I'll just add that 3 participant realtime, or perhaps more if the data was not video (say, realtime comments in text), is still very useful. Having all user data go through servers is a big liability and privacy concern. I'll track this in lieu of that https://github.com/livekit/livekit/issues/303

You are absolutely right that you can get WebRTC to work well with mesh where it's only sending data / low-bitrate audio. In that case using plain WebRTC would be the path forward, you wouldn't need LiveKit there.
E2EE is pretty important in a SFU topology, we'll use https://github.com/livekit/livekit/issues/303 to keep track of that request.

In that case using plain WebRTC would be the path forward

Are you saying this is readily doable with plain WebRTC without a lib? Or that it's in as much a need of a lib and out of scope for LiveKit

It might be complex, but it's definitely possible to build it with WebRTC. The question is whether that complexity is handled by a library or within your own software.
And yes, this is out of scope for LiveKit.

I agree that LiveKit doesn't need the addition of p2p mesh. If you are looking for a good library to support that, I had good luck with  https://github.com/feross/simple-peer
Their library is easy to work with and very lightweight. However, I ultimately went with LiveKit because p2p mesh wasn't a good fit for my use case.

and having a P2P mesh optimisation within that blackbox called livekit

Can you elaborate what you mean by this?


Update delete_default_argocd_instance function in bootstrap.sh

The current delete_default_argocd_instance function will delete the instance of gitopsservice and argocd CR. This is because the deletion of the gitopsservice instance did not delete the instance of argocd.
delete_default_argocd_instance () {
echo "Delete the default ArgoCD instance"
pushd ${OUTPUT_DIR}
oc delete gitopsservice cluster -n openshift-gitops
oc delete argocd openshift-gitops -n openshift-gitops
popd
}
This has been fixed by the Red Hat team now so the deletion of the gitopsservice will also delete the default instance of ArgoCD. The second delete command can be removed.

This is fixed now via #112


Update support for idf5.0 (IDFGH-7407)

When I compiled using the master branch of ESP-IDF, I found it had a lot of errors, I thought it was necessary to gradually start supporting IDF5.0

sorry i posted in the wrong place


pull request for e2e test at 12/11/2016 21:15:02

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

optest/docs/updated_at.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


docs: Missing HMR config for React

Documentation Is:

[X] Missing
[X] Needed
[ ] Confusing
[ ] Not Sure?

Please Explain in Detail...
Since some hmr webpack's options are under internals it's not clear how to configure HMR for React.
I have switched hot: true. Although the changes come to the browser (I can see console log in browser) they do not modify the page.
Your Proposal for Changes
Add some words in docs how to configure HMR for React

Thanks for the issue :beer:
This project caters to more than just React, and as such we don't have documentation for each framework that it's possible to use this module with. Think about that - we'd have a million documentation files for every framework that was in use today. There was a recent tweet with a link to a blog post highlighting 32 frameworks in active use by a significant number of users. Documentation belongs in a project's docs, meaning that the React docs should have some info on how to use this module.
That said, hot: true isn't doing anything, because true is the default value. I would recommend reading about how to setup HMR with webpack first, and then trying webpack-serve. HMR should work identical to webpack-dev-server as well, since they both use the same module to enable that (webpack-dev-middleware).


Add few tweaks

Signed-off-by: Nitin Goyal<EMAIL_ADDRESS>

/override ci/prow/ci-bundle-odf-operator-bundle
/override ci/prow/odf-operator-e2e-aws
/override ci/prow/images


update home page blocks

should be done post Jan 1st launch of new publication.
DEV-376

Group changed their mind and wants this pushed live before the end of the year


Handling 204 response codes

Where an OpenAPI operation only includes 204 in its successful responses, the generated method should be declared to effectively return void.
'/DoTheThing':
    POST:
      responses:
        '204':
          description: An empty response
      x-csharp: Task DoTheThing.PostAsync(...);

If at runtime, the server returns a different 2XX with a body, then we should read the body and then discard it.  Failing to read the body will hang the server.
When a operation is expected to return a value, but instead returns a 204 at runtime (whether described or not), the returned value should be null.
'/nullableSimpleObject':
    get:
      responses:
        '200':
          description: A response with an object type response
          content: { "application/json": { schema: { $ref: "#/components/schemas/simpleObject" }}}
        '204':
          description: A response with an object type but with null value
      x-csharp: Task<SimpleObject> GetAsync(...);

fixed with #1457 and other pull requests. interestingly enough Java requires you to close the response but not to read it. Dotnet and Go to read and close. JavaScript to read without closing.


Load RSEI industrial pollution data to make available in the tool database

Description
RSEI's data is available crosswalked to census tracts:

RSEI Geographic Microdata provides results at the grid cell level. Crosswalks to translate data from the RSEI grid cell system to U.S. census block geographies are provided so that users can combine RSEI results with other data and aggregate the results to any census geography, like block, block group, or tract.

https://www.epa.gov/rsei/rsei-grid-and-locational-information
This could be a valuable resource for information on pollution sources. It is updated on a yearly basis with a deep historical record, and because the data collection is legislatively mandated, it's very comprehensive.

Data dictionary
Methodology documentation
Selecting data sources

Sample instructions from the documentation:
For example, if you filter your data by selecting "325 Chemicals" and "326 Plastics and Rubber" from the Industry Sector field, the subset of data will be any records where "325 Chemicals" or "326 Plastics and Rubber" is the value in the Industry Sector field - "325 Chemicals" and "326 Plastics and Rubber" are both in the selected state. All other values in the Industry Sector field are alternative states. Within "325 Chemicals" and "326 Plastics and Rubber" are many subsectors that are denoted as possible states because, though not selected, they are possible given the current selection state.
This begs several questions for further investigation and for clarification:

What is the question one is trying to answer with this dataset. For example which sectors and/or facilities contribute to the most "X" type of particulate or carcinogen?
It's not immediately clear how to access the datasets programmatically without going through the reporting tool (please see item (3) above)
Do we want modeled data? This includes include identifiers, locational data, water modeling data, air modeling data, and other information about the facility.

Looks like there's a single aggregate score they produce: https://www.epa.gov/rsei/understanding-rsei-results#what.

Hi Lucas,
As we discussed the three questions that I raise - and that are manifested in the SEI score and some additional notes that address (2) above:

RSEI Scores do not describe a level of risk (such as the number of excess lung cancer cases). RSEI does not represent risk-assessments; rather to indicate area of further analysis - which is why I raised (1), above.

From the documents: RSEI Scores are designed to be compared to each other. A RSEI Score 10 times higher than another RSEI Score suggests that the potential for risk is 10 times higher. Relatively small chemical releases may lead to high RSEI Scores if the toxicity weight is particularly high or if the modeled exposed population is large. Conversely, large chemical releases may lead to low RSEI Scores if the toxicity weight is low or if the modeled exposed population is small. High RSEI Scores identify areas that may warrant further investigation, and do not conclusively demonstrate sources of risk. A facility with a high RSEI Score may not be the primary source of chronic human health risk in an area. RSEI Scores point to chemical releases that may warrant further investigation.

There seems to be be an FTP server that requires authentication for the census tract microdata at "newftp.epa.gov"

This provides details on how RSEI should be used: https://www.epa.gov/rsei/how-rsei-should-be-used.

Below is the summary of basic exploration of the reporting year 2019. According to the associate that I was in contact with they have not produced census tract files for any year but 2019.
According to the associate, in the future they will have 2020 data, which will have all the years of data by census tract up to 2020. They will host this data on another production server - this dataset is too large for the FTP host.  Generally, the way to access the data is to contact us, and they will provide download links.
import pandas as pd
import os
import ftplib

ftp = ftplib.FTP('newftp.epa.gov')
ftp.login()

'230 User logged in, proceed.'

Summary of findings from searching FTP server
In order to ensure alignment with our unit of measurement, we seek two distinct sources:

Census Tract Aggregated Microdata files
Disaggregated Census Tract Microdata files

Across many of tables it appears the foreign key is the DCN (Unique identifier assigned by TRI to each facility submission - otherwise known as the document control number).
The source documentation with the associated tables is found here, page 20.
def ftp_as_zip(file_name, file_path, host, data_location = '.'):
    '''This is the generic function for accessing the ftp server and saving a designated
    file from the host server to a local machine
    '''
    print("Connecting to: ", host)
    ftp =  ftplib.FTP(host)
    ftp.login()
    ftp.cwd(file_path)
    print("Data will be peristed to: " + data_location)
    if not os.path.exists(data_location):
        os.makedirs(data_location)
    print("writing file: " + file_name)
    with open(os.path.join(data_location, file_name), "wb") as f_handler:
        # this is weird, I know. The second argument is a callback
        # function - which in this instance to call the file handler
        ftp.retrbinary('RETR %s' % file_name, f_handler.write)
        ftp.quit()

# Example Usage - Note this is just an example and does
# not represent the current files we seek
file_name = 'poly_gc14_conus_810m_top.dbf' #downloading from EPA public FTP site
file_path='RSEI/Shapefiles/810m_Standard_Grid_Shapefiles/' # path on host server
host = 'newftp.epa.gov' # host
ftp_as_zip(file_name, file_path, host)

Connecting to:  newftp.epa.gov
Data will be peristed to: .
writing file: poly_gc14_conus_810m_top.dbf

We would want the 2019 version of the release, correct? This appears to be the latest version of the release
 ftp.cwd("RSEI/Version239_RY2019/")

'250 Directory changed to /RSEI/Version239_RY2019'

4 directories
files = ftp.nlst()
len(files)

4

Each is a specified directory.....let''s take a look at each
files

['Aggregated_Grid_Cell_Data',
 'Disaggregated_Microdata',
 'Public_Release_Data',
 'RSEI_Queries']

Below is an example of what is available
Example Shape files
RSEI crosswalks and shapefiles. RSEI crosswalks allow you to transpose data from the RSEI grid to Census blocks, and shapefiles provide grid geometry for mapping microdata
import geopandas
import pandas as pd

/Users/sarahluw/.pyenv/versions/3.6.2/envs/my-virtual-env-3.6.2/lib/python3.6/site-packages/geopandas/_compat.py:110: UserWarning: The Shapely GEOS version (3.9.1-CAPI-1.14.2) is incompatible with the GEOS version PyGEOS was compiled with (3.10.1-CAPI-1.16.0). Conversions between both will be slow.
  shapely_geos_version, geos_capi_version_string

gdf = geopandas.read_file("poly_gc54_guam_810m.shp")

gdf.head()

CELLX
CELLY
CLAT
CLONG
CX
CY
geometry

0
-1400
1814
13.181812
144.532682
-1134000
1469340
POLYGON ((144.52879 13.18531, 144.53626 13.185...

1
-1399
1814
13.182124
144.540150
-1133190
1469340
POLYGON ((144.53626 13.18562, 144.54372 13.185...

2
-1398
1814
13.182436
144.547618
-1132380
1469340
POLYGON ((144.54372 13.18594, 144.55119 13.186...

3
-1397
1814
13.182747
144.555085
-1131570
1469340
POLYGON ((144.55119 13.18625, 144.55866 13.186...

4
-1396
1814
13.183058
144.562553
-1130760
1469340
POLYGON ((144.55866 13.18656, 144.56613 13.186...

DBF Details Source
A DBF file is a standard database file used by dBASE, a
database management system application. It organizes data
into multiple records with fields stored in an array data type.
DBF files are also compatible with other "xBase" database programs,
which arose because of the file format's popularity.
import dbf

table = dbf.Table('poly_gc14_conus_810m_top.dbf')

# these do not reflect the level of geographic measurement
table.field_names

FieldnameList(['CELLX', 'CELLY', 'CLAT', 'CLONG', 'CX', 'CY'])

Aggregate - this is what I thought is what we sought (page 20 of this reference)
However, it is in fact this
import pandas as pd
# hmmmm, okay, so no header -
# no problem, let's see if we just use the file handler from gzip (further down)
data = pd.read_csv('/Users/sarahluw/Downloads/aggmicro2019_2019_gc14.csv.gz',
                   nrows=10, compression='gzip', header=1,
                   error_bad_lines=False)
data.head()

   1745  311  1   7  4  6.03996E-003  0.00000E+000  0.00000E+000.1  \
0  1746  311  1   7  4      0.006368           0.0             0.0
1  1747  311  1   7  4      0.006731           0.0             0.0
2  1748  311  1   7  4      0.007100           0.0             0.0
3  1749  311  1   7  4      0.007324           0.0             0.0
4  1750  311  1   7  4      0.007367           0.0             0.0
5  1751  311  1   7  4      0.007427           0.0             0.0
6  1752  311  1   7  4      0.007650           0.0             0.0
7  1753  311  1   7  4      0.007895           0.0             0.0
8  1754  311  2  11  6      0.020016           0.0             0.0
9  1755  311  2  11  6      0.021184           0.0             0.0

   0.00000E+000.2  0.00000E+000.3
0             0.0             0.0
1             0.0             0.0
2             0.0             0.0
3             0.0             0.0
4             0.0             0.0
5             0.0             0.0
6             0.0             0.0
7             0.0             0.0
8             0.0             0.0
9             0.0             0.0

import gzip

with gzip.open('/Users/sarahluw/Downloads/aggmicro2019_2019_gc14.csv.gz', 'rb') as f:
    data = pd.read_csv(f)

# easier to read
data.head()

1744
311
1
7
4
5.71468E-003
0.00000E+000
0.00000E+000.1
0.00000E+000.2
0.00000E+000.3

0
1745
311
1
7
4
0.006040
0.0
0.0
0.0
0.0

1
1746
311
1
7
4
0.006368
0.0
0.0
0.0
0.0

2
1747
311
1
7
4
0.006731
0.0
0.0
0.0
0.0

3
1748
311
1
7
4
0.007100
0.0
0.0
0.0
0.0

4
1749
311
1
7
4
0.007324
0.0
0.0
0.0
0.0

These tables are avialable in RSEI_Queries. Here is an example. The references are here
Foreign keys that relate Releases with Submissions

ReleaseNumber - Unique internal identifier.
SubmissionNumber - Unique internal identifier

import jaydebeapi
import pandas as pd
import pandas_access as mdb

db_path = "rsei_data3_v239.accdb"

# Listing the tables from Microsoft Access tables
for tbl in mdb.list_tables(db_path):
    print(tbl)

Releases
Submissions

# Read a huge table
accumulator = []
for chunk in mdb.read_table(db_path, "Submissions", chunksize=10000):
    accumulator.append(chunk)

accumulator[0].head()

DCN
SubmissionNumber
FacilityID
FacilityNumber
CAS
ChemicalNumber
MPOUse
MaxOnsite
Entire_Fac
One_Time_Release_Qty
One_Time_Release_Qty_NA
Partial_Fac
Trade_Secret_Ind
SubmissionYear

0
1300140000011
1
15902CCKRN75BRI
8527
N982
595
111
04
1
0
0
0
0
2000

1
1300140000023
2
15902CCKRN75BRI
8527
007439921
346
111
02
1
0
0
0
0
2000

2
1300140000035
3
70363GLFSL583TH
45099
000108883
549
011
03
1
NaN
1
0
0
2000

3
1300140000047
4
70363GLFSL583TH
45099
001330207
592
011
03
1
NaN
1
0
0
2000

4
1300140000050
5
70363GLFSL583TH
45099
007440666
594
010
03
1
NaN
1
0
0
2000

db_path_2 = "rsei_data1_v239.accdb"

# First Table
for tbl in mdb.list_tables(db_path_2):
    print(tbl)

Category
NAICSCode
Sictable
Chemical
Media
Facility
Offsite

db_path_3 = "rsei_data2_v239.accdb"

# Second Table
for tbl in mdb.list_tables(db_path_3):
    print(tbl)

Elements

Public Release data (RSEIv239_Public_Release_Data directory)
from glob import glob

list_of_files = list(glob(str("RSEIv239_Public_Release_Data/") + "/*.csv"))

dfs_list = [
    pd.read_csv(f)
    for f in list_of_files
]

/Users/sarahluw/.pyenv/versions/3.6.2/envs/my-virtual-env-3.6.2/lib/python3.6/site-packages/IPython/core/interactiveshell.py:3263: DtypeWarning: Columns (3,24,39,43) have mixed types.Specify dtype option on import or set low_memory=False.
  if (await self.run_code(code, result,  async_=asy)):
/Users/sarahluw/.pyenv/versions/3.6.2/envs/my-virtual-env-3.6.2/lib/python3.6/site-packages/IPython/core/interactiveshell.py:3263: DtypeWarning: Columns (19,30,31) have mixed types.Specify dtype option on import or set low_memory=False.
  if (await self.run_code(code, result,  async_=asy)):
/Users/sarahluw/.pyenv/versions/3.6.2/envs/my-virtual-env-3.6.2/lib/python3.6/site-packages/IPython/core/interactiveshell.py:3263: DtypeWarning: Columns (8) have mixed types.Specify dtype option on import or set low_memory=False.
  if (await self.run_code(code, result,  async_=asy)):
/Users/sarahluw/.pyenv/versions/3.6.2/envs/my-virtual-env-3.6.2/lib/python3.6/site-packages/IPython/core/interactiveshell.py:3263: DtypeWarning: Columns (6) have mixed types.Specify dtype option on import or set low_memory=False.
  if (await self.run_code(code, result,  async_=asy)):

# none of these tables contain the unit of measurement we are after.....
# although they do offer ample and comprehensive results

[df.columns for df in dfs_list]

[Index(['CASNumber', 'CASStandard', 'ChemicalNumber', 'Chemical', 'MCL1988',
        'MCL1989', 'MCL1990', 'MCL1991', 'MCL1992', 'MCL1993', 'MCL1994',
        'MCL1995', 'MCL1996', 'MCL1997', 'MCL1998', 'MCL1999', 'MCL2000',
        'MCL2001', 'MCL2002', 'MCL2003', 'MCL2004', 'MCL2005', 'MCL2006',
        'MCL2007', 'MCL2008', 'MCL2009', 'MCL2010', 'MCL2011', 'MCL2012',
        'MCL2013', 'MCL2014', 'MCL2015', 'MCL2016', 'MCL2017', 'MCL2018',
        'MCL2019'],
       dtype='object'),
 Index(['OffsiteID', 'FacilityNumber', 'TRIFID', 'DropIncinerator',
        'POTW_Incin', 'Name', 'Street', 'City', 'State', 'ZIPCode', 'ZIP9',
        'Latitude', 'Longitude', 'GridCode', 'Country', 'X', 'Y',
        'RadialDistance', 'StackHeight', 'StackVelocity', 'StackDiameter',
        'StackTemperature', 'StackParameterSource', 'NAICS', 'FRSID', 'FRSYear',
        'HEM3ID', 'DistanceToHEM3', 'WBANID', 'DistanceToWBAN', 'WaterReleases',
        'OutfallLongitude', 'OutfallLatitude', 'NearReach', 'NearComID',
        'DistanceToReach', 'AssignedReach', 'AssignedComID', 'ReachSource',
        'ReachNotes', 'LatLongSource', 'LatLongYear', 'LockLL',
        'CentroidAdjustment', 'Notes_on_coordinates',
        'AdditionalSourcesForLocation', 'LocationConfidence', 'Foreign'],
       dtype='object'),
 Index(['FacilityID', 'FacilityNumber', 'FRSID', 'Latitude', 'Longitude',
        'LLOverridden', 'GridCode', 'X', 'Y', 'LatLongSource', 'LLYear',
        'LLNotes', 'RadialDistance', 'FacilityName', 'Street', 'City', 'County',
        'State', 'ZIPCode', 'ZIP9', 'FIPS', 'STFIPS', 'DUNS', 'ParentDUNS',
        'ParentName', 'StandardizedParentCompany', 'Region',
        'FederalFacilityFlag', 'FederalAgencyName', 'PublicContactName',
        'PublicContactPhone', 'PublicContactPhoneExtension',
        'PublicContactEmail', 'StackHeight', 'StackVelocity', 'StackDiameter',
        'StackTemperature', 'StackHeightSource', 'StackVelocitySource',
        'StackDiameterSource', 'StackTemperatureSource', 'StackHeightNEIYear',
        'StackVelocityNEIYear', 'StackDiameterNEIYear',
        'StackTemperatureNEIYear', 'ChromHexPercent', 'ChromSource',
        'NewIndustryFlag', 'ModeledNAICS', 'NAICSCode3Digit', 'NAICSCode4Digit',
        'NAICSCode5Digit', 'NAICS1', 'NAICS2', 'NAICS3', 'NAICS4', 'NAICS5',
        'NAICS6', 'SIC1', 'FinalReach', 'FinalCOMID', 'ReachSource',
        'DistanceToReach', 'HEM3ID', 'DistanceToHEM3', 'LLConfirmed',
        'WaterReleases', 'ModeledReleases', 'ModChromReleases'],
       dtype='object'),
 Index(['DCN', 'SubmissionNumber', 'FacilityID', 'FacilityNumber', 'CAS',
        'ChemicalNumber', 'SubmissionYear', 'MPOUse', 'MaxOnsite', 'EntireFac',
        'OneTimeReleaseQty', 'OneTimeReleaseQtyNA', 'PartialFac',
        'TradeSecretInd'],
       dtype='object'),
 Index(['2017NAICSCode', 'LongName', 'Changed2017', 'TRIIndustrySector',
        'IndustrySubsector', '4DigitNAICS', 'NewIndustry'],
       dtype='object'),
 Index(['ReleaseNumber', 'SubmissionNumber', 'Media', 'PoundsReleased',
        'OffsiteNumber', 'TEF', 'TEFSource', 'DCN', 'TransferLocNum',
        'OffsiteAmountSequence', 'WaterSequenceNum'],
       dtype='object'),
 Index(['ElementNumber', 'ReleaseNumber', 'PoundsPT', 'ScoreCategory', 'Score',
        'Population', 'ScoreA', 'PopA', 'ScoreB', 'PopB', 'ScoreC', 'PopC',
        'ScoreD', 'PopD', 'ScoreE', 'PopE', 'NCScore', 'CScore', 'Hazard',
        'HazardC', 'HazardNC'],
       dtype='object'),
 Index(['Media', 'MediaText', 'TRIEnvironmentalMedium', 'Itw', 'Otw', 'Mtw',
        'MediaCode', 'TRICode', 'TRICategory', 'LongDescription',
        'AggDescription'],
       dtype='object'),
 Index(['ScoreCategory', 'Category', 'Model', 'InhaleTox'], dtype='object'),
 Index(['CASNumber', 'CASStandard', 'ChemicalNumber', 'TRIChemID',
        'MetalCombinedChemNum', 'Category', 'SortCAS', 'SortName',
        'FullChemicalName', 'Chemical', 'Added', 'ToxicitySource', 'RfCInhale',
        'RfCUF', 'RfCMF', 'RfCConf', 'RfCSource', 'RfCListingDate',
        'RfCToxWeight', 'RFDOral', 'RfDUF', 'RfDMF', 'RfDConf',
        'RfDListingDate', 'RfDSource', 'RfDToxWeight', 'UnitRiskInhale',
        'QSTAROral', 'WOE', 'UnitRiskListingDate', 'UnitRiskSource',
        'IURToxWeight', 'QStarListingDate', 'QStarSource', 'OSFToxWeight',
        'WOEListingDate', 'WOESource', 'ITW', 'OTW', 'ToxicityClassOral',
        'ToxicityClassInhale', 'ToxicityCategory', 'AirDecay', 'Koc',
        'H2ODecay', 'LOGKow', 'Kd', 'WaterSolubility', 'POTWPartitionRemoval',
        'POTWPartitionSludge', 'POTWPartitionVolat', 'POTWPartitionBiod',
        'IncineratorDRE', 'BCF', 'Henrys', 'MCL', 'MolecularWeight', 'HAPFlag',
        'CAAFlag', 'PriorityPollutantFlag', 'SDWAFlag', 'CERCLAFlag',
        'OSHACarcinogens', 'ExpansionFlag', 'Core88ChemicalFlag',
        'Core95ChemicalFlag', 'Core98ChemicalFlag', 'Core00ChemicalFlag',
        'Core01ChemicalFlag', 'HPVFlag', 'HPVChallenge', 'PBTFlag', 'Metal',
        'HasTox', 'MaxTW', 'MaxNC', 'MaxC', 'Notes'],
       dtype='object')]

Disaggregated Microdata
Values represent columns here
# Lazily reading files into single Dask DataFrame (~21 GB)
import dask.dataframe as dd

ddf_direct = dd.read_csv(
    "/Users/sarahluw/Downloads/micro2019_2019.csv.gz",
    blocksize=None,
    compression='gzip',
    sample=100
)

# first one is the GridCode
# second X-coordinate of grid cell
# third Y-Coordinate of grid cell
ddf_direct.columns

Index(['44', '-58', '-4', '6457356', '519', '16', '1', '2.17067E-05',
       '7.59735E-05', '0.00000E+00', '0.00000E+00.1', '0.00000E+00.2',
       '0.00000E+00.3'],
      dtype='object')

CSV Aggregate for Tract Level Geographic Unit - after exchange with contact at EPA
import pandas as pd

url = "http://abt-rsei.s3.amazonaws.com/microdata2019/census_agg/CensusMicroTracts2019_2019_aggregated.zip"
df = pd.read_csv(url, header=0)

columns = [
        "GEOID10",
        "NUMFACS",
        "NUMRELEASES",
        "NUMCHEMS",
        "TOXCONC",
        "SCORE",
        "POP",
        "CSCORE",
        "NCSCORE"
]
# the columns are actually a census tract's data at this point
# The subsequent step make the current columns the first row
# of the database
df.columns

Index(['36047023200', '193', '904', '96', '4.85177E+002', '7.59714E+002',
       '6.19922E+003', '6.10315E+002', '1.54848E+002'],
      dtype='object')

first_row = dict(zip(columns, df.columns))

# add first row to dataframe
row_df = pd.DataFrame(first_row, index=[0])
df.columns = columns
# simply concatenate both dataframes
df = pd.concat([row_df, df]).reset_index(drop = True)
df.head(5)

GEOID10
NUMFACS
NUMRELEASES
NUMCHEMS
TOXCONC
SCORE
POP
CSCORE
NCSCORE

0
36047023200
193
904
96
4.85177E+002
7.59714E+002
6.19922E+003
6.10315E+002
1.54848E+002

1
36029007202
84
376
81
227.366
79.5029
1601.87
63.0466
14.8128

2
42003429201
144
1624
251
2705.16
2553.95
4527.6
2447.72
113.751

3
17201004100
80
365
57
3626.62
3984.97
5766.72
3969.13
27.0562

4
47065011203
79
358
79
1133.4
1885.51
7820.66
1410.31
447.775


Update source code link for Socket.IO package

#46364
Contributing to the Azure SDK
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

/check-enforcer override