Switch to pynput module

This PR switches to the pynput module to handle hotkeys. This will enable to run AlwaysReddy as a non-root user.
For issue https://github.com/ILikeAI/AlwaysReddy/issues/11
Limitations:

pynput cannot handle space and tab for hotkeys.
The default Ctrl-Alt-F12 in linux is captured to switch console and cannot be used.
It only works for X. XWayland can only handle the hotkeys in it´s own window. Otherwise on Wayland the application needs to be run as root either way.

And it also adds my venv to .gitignore, which I thought I kept outside of the feature branch... It is a good practice to use python venv's for your (dev) environments anyway.

Thanks for the PR this looks great! I am hoping to get linux set up so i can test it locally asap.
Im a fan of the current Ctrl+Shift+Space hotkey so i may not use this new library as the default hotkey handler but instead try to set it up in a way so that Linux users can use that library instead. Unfortunately I have a lot on my plate for the next few days so I will see how I go making these modifications, Ill try to get it setup asap.
Yep I was thinking about Venvs last night, good call

Just letting you know that I have been playing with your code changes on my own branch. Check out the WIP branch to have a look. I've set up a separate file as a keyboard handler that can identify which operating system a user is on and will select a different hotkey handler depending. Let me know what you think of this. Also, sorry I'm a bit of a noob to GitHub, so I'm not sure if this is the proper protocol. If I want to modify someone else's pull request, let me know if I've done anything wrong here.
Here are some thoughts though:
It appears that some Linux users will need to run as root anyway, depending on their display protocol. This makes me wonder if maybe it's best to have all users just use their default keyboard library with root permissions so that then we could at least have the same keyboard hotkeys across the different platforms, meaning we can still use the default Control+Shift+Space. I am a bit of a fan of that hotkey, haha.
I'm not sure though, let me know what you think of this. It's possible that there's somewhere in the middle where people can use the version without root privileges if they prefer, but they'll have to deal with different hotkeys. But by default, you need to give it root privileges and deal with the standard hotkeys... Or maybe im just way too attached to the current hotkey setup becuase im used to it lol
Let me know what you think. Either way I doubt the current content of WIP will be the final verison

Side note: looks like the clipboard functionality does not work properly for me on linux, did you get that too?
An error occurred while getting completion:
    Pyperclip could not find a copy/paste mechanism for your system.
    For more information, please visit https://pyperclip.readthedocs.io/en/latest/index.html#not-implemented-error

Pynput is now integrated


Windows 11 DFE using this URL v20.events.data.microsoft.com but it's not mentioned in the spreadsheet.

Windows 11 DFE using this URL v20.events.data.microsoft.com but it's not mentioned in the spreadsheet.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 28a4997c-0029-be57-24a3-7aa10dadda53
Version Independent ID: 98227b02-39e2-03f6-c947-23317abb818f
Content: Configure device proxy and Internet connection settings
Content Source: microsoft-365/security/defender-endpoint/configure-proxy-internet.md
Product: m365-security
Technology: mde
GitHub Login: @mjcaparas
Microsoft Alias: macapara

@luutuananh Please confirm Thanks

@luutuananh
us-v20.events.data.microsoft.com And us4-v20.events.data.microsoft.com Both are mentioned in Excel sheet

I check my proxy in my lab, it's also use both url v20.events.data.microsoft.com and us-v20.events.data.microsoft.com together.


Add unit tests for Administration Rules section

Refactored code for testability:

Moved bulk of the view control code into new rulesListDirective
Changed RuleState methods to return the promise from the CollectionsAPI
rather than creating a new promise.

Coverage Before:
====================== Coverage summary ======================
Statements   : 30.83% ( 1493/4842 )
Branches     : 8.22% ( 128/1557 )
Functions    : 25.71% ( 326/1268 )
Lines        : 30.83% ( 1493/4842 )
Coverage After:
====================== Coverage summary ======================
Statements   : 36.19% ( 1708/4719 )
Branches     : 10.88% ( 163/1498 )
Functions    : 30.91% ( 387/1252 )
Lines        : 36.19% ( 1708/4719 )

@AllenBW Thanks for the comments. Addressed your concerns.

Looks like still a few CodeClimate issues that need to be addressed.

@chriskacerguis Any pointer on how to figure out what it is that CodeClimate is eluding to?

@jeff-phillips-18 click on "details",
https://codeclimate.com/github/ManageIQ/manageiq-ui-self_service/pull/191

@chriskacerguis Sorry, I followed the details link, on that page I've clicked every link I could think might help but they all seem to lead to the same page 😟

@jeff-phillips-18 do you see this page?

@chriskacerguis Yes. I don't understand the issues and I'm not finding the links to be helpful.

@jeff-phillips-18 on each issue there is a read more info.  Take a look there, it seems that there is a lot of duplicate code, that can probably be streamlined.  Take a look and let me know if you still need help and @AllenBW or I can help

@chriskacerguis Still not that useful but I *think I know what it's getting at. This will likely make the code less readable in places. I've installed Docker and codeclimate so that I can run and check for issues prior to pushing. The default of 15 minutes to run the analysis is too short and I've updated it to 30 minutes...

I believe all code climate issues have been fixed now


[Talessya] Sheet merged but not added to approved.yaml

Following this discussion on the PR https://github.com/Roll20/roll20-character-sheets/pull/9090, the Talessya sheet was merged into the master branch but hasn't been added to the approved.yaml so we cannot use it yet on the app. What would be the next step for this?

Hello @nmbradley , is there anything I can do to see this published?

My apologies, I must have overlooked this during the original merge. I will ensure this gets done today.

@jorsermar I've added this to the approved.yaml, and should be appearing sometime in the next 15 minutes or so. I'll check in a bit to make sure everything has gone through.

It appears something in this sheet is breaking the sheet select. My experience with this is that it is normally because the CSS file confuses the CSE. I've made some changes on #9200 that I'd like you to review to see if we can resolve this issue

Hi, sorry for not replying before. I´ve been disconnected during summer. Thanks for fixing it, we can see it's working now.
You can close the ticket, and again, thanks for all your help.

Hi, sorry for not replying before. I´ve been disconnected during summer. Thanks for fixing it, we can see it's working now.
You can close the ticket, and again, thanks for all your help.


Add apigw execution ARN (non stage) to outputs - used by Lambda invok…

When creating permissions for lambda invocation, we need to use the REST API's execution ARN and not the stage ARN (one level broader). This PR adds the REST API's exec ARN to outputs for ease of use by the module user.

@pkailasu do you have any idea about the security failure regarding CKV_AWS_338: "Ensure that CloudWatch Log Group specifies retention days"? Seems clearly a false positive.

@pkailasu do you have any idea about the security failure regarding CKV_AWS_338: "Ensure that CloudWatch Log Group specifies retention days"? Seems clearly a false positive.

@paul-ylz yeah noticed someone else faced the same issue in another repo for the same. Can skip first,


Incorrect docs/animation

Its listed as; :animation="animation", however it should be :animation="ani-none'" does not work an error is thrown, but :animation="'ani-none'", with single quotes does.

Just noticed this is based on how I was implementing the properties, closing this.


Automatic tag closing sometimes wrongfully adds  behind

Description
The tag autocompletion that adds the corresponding closing tag after you type the opening tag sometimes mistakenly adds  when you type . It does not do this in every instance, only in specific places in the code. When typing  in other places nothing happens. Removing the  in places where Brackets adds the  and typing it again results in it being changed to "" again.
Steps to Reproduce

Create a new file in Brackets and save it as .html
Copy the following code:

<!DOCTYPE html>

The quick brown
fox jumps over the lazy dog

Remove the
 and type
 at the same position the removed code used to be (right before the "f" of fox, after the space).
Brackets automatically adds  to the right of the cursor.

Expected behavior:
Brackets should not add a closing tag, since the tag is a standalone tag and is already closed and the tag it adds does not fit.
Actual behavior:
Brackets attempts to close the already closed  tag with a not matching  tag and thus it and all subsequent closing tags are marked red, since Brackets notices the wrong syntax.
Versions
Windows 10, Brackets 1.14 (Build 1.14.0-17740; no extensions)

Same issue appears on macOS (same build).


There is no any status indication in the Console output whether an attack is successful or not

Expected Behavior
There is some status indication in the Console output whether an attack is successful or not.
At least, this could be fixed to the indication as it was in the prev version db1000n-v0.8.2-windows-amd64:
"Атака проводиться успішно! Руський воєнний корабль іди нахуй!"
Actual Behavior
There is no any status indication in the Console output whether an attack is successful or not:

Steps to Reproduce the Problem

Launch db1000n.exe
Wait a few minutes

Specifications

Version:     0.8.8
Platform:   Windows 10 Home, 64-bit
Subsystem:

Fixed in #436


Unnecessary yaFyaml dependency in GOCART build system

yaFyaml is not needed to build GOCART. However, the following dependency is included in the build system:
https://github.com/GEOS-ESM/GOCART/blob/e59ddd490f3bc413465317a89e5684cee68de8e8/CMakeLists.txt#L64

Ah. Nuts. Yeah. We probably need a "build like UFS" build test here as well.
@rmontuoro In MAPL we handle this via USE_EXTDATA2G. I'm guessing you are building MAPL with -DUSE_EXTDATA2G=NO?

@mathomp4 - yes, we build MAPL with -DUSE_EXTDATA2G=NO in the UFS.
The yaFyaml dependency is required by MAPL though, not GOCART.

True. Indeed, there seem to be a lot of unneeded find_package in GOCART:
https://github.com/GEOS-ESM/GOCART/blob/4c941791d3d1fbea65dd5d2be330ada8c4951036/CMakeLists.txt#L61-L64
Those are needed by MAPL, not GOCART. @rmontuoro if I make up a hotfix patch, can you test it?

@mathomp4 - sure. Thanks!

@rmontuoro Can you actually tell me exactly how you are building MAPL now? That is what CMake options? I should capture that.

Here they are:
https://github.com/NOAA-EMC/hpc-stack/blob/74a20edac33e66b59aa82fed47c7b8189c14363c/libs/build_mapl.sh#L69-L78

@rmontuoro Okay can you try the hotfix/mathomp4/184-ufs-cmake-fix? (see #185)

@rmontuoro Actually, another question: what version of GOCART are you using? My draft PR is based on our main which is 2.1.0, but if you still use an older version, I'll need to hotfix that too.

@mathomp4 - Thank you.
We are currently using rev. a342b1b215 in the ufs-weather-model

@mathomp4 - Thank you. We are currently using rev. a342b1b in the ufs-weather-model

Ah. Good then we can (for now) just hotfix 2.1.0 since you are (essentially) on that. I'm doing testing now and will coordinate with @amdasilva about getting a v2.1.1 out for you.

@mathomp4 - hotfix/mathomp4/184-ufs-cmake-fix built successfully in the UFS after minor modifications listed in https://github.com/GEOS-ESM/GOCART/pull/185#issuecomment-1247314269


Determine images registry

@JeeWeetje and @sandervandevelde,
My initial docker registry was github.
Should I move the output to Docker Hub ?

Keep it on github for now, sering the license policy change at Docker. We might move all public images to github, because of it.


Add unit tests

This commit adds more testing!
A few of them I couldn't get to work correctly....

This fixes https://github.com/chef-cookbooks/audit/issues/128


Add decoded-cipher's file

No description provided

created a branch, created a file and made a commit, and opened a pull request


Row Level Security on populated associations

Is it currently possible for row level security to be applied to populated associations? It works on find and findOne but not in populations.

Could you give an example of what you are trying to do?  From what you say, I assume it is something like , in a query like this: Store.find(3).populate('widgets')  -only retrieve widgets that you have permissions to read.  If that is correct, this is not currently supported.

I referring to when you blacklist an attribute on a model. For example if I was to blacklist the title attribute on ModelB and did ModelB.findOne(1) the title attribute is removed, but if I do ModelA.findOne(1).populate('modelb')``` the title attribute is not removed.


README.md: Add table row for alloc-fmt crate

Closes #116 🙂

Hey @hiseni , thanks for the PR and welcome!


fix: add dependencies in pyproject.toml

Fix #82

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Change /policies/{tierLevel} resource in the Publisher API to /policies/{type}?level={level}

The current resource to retrieve rate limiting policies on the publisher API is as /policies/{tierLevel}. Examples are /policies/api, /policies/subscriptions, etc. This resource has been designed to support rate limiting policies only. If later we introduce policies of other types, such as threat protection policies we will not be able to use this resource. In order for the resource to be future compatible, I suggest that we change this resource to /policies/{level}.
Example to query rate limiting policies by API would then be like /policies/rate_limits?level=api

With new Rest API version for APIM 3.0.0 , a separate resource is introduced for throttling (ex: /throttling/policies/application ).
[1] https://github.com/wso2/carbon-apimgt/blob/master/components/apimgt/org.wso2.carbon.apimgt.rest.api.admin/src/main/resources/admin-api.yaml#L112
[2] https://github.com/wso2/carbon-apimgt/blob/master/components/apimgt/org.wso2.carbon.apimgt.rest.api.admin/src/main/resources/admin-api.yaml#L390


Add debug logging to the compact index client

\c @indirect

@indirect r?

@homu r+

:pushpin: Commit 980495a has been approved by indirect

:hourglass: Testing commit 980495a with merge 7607a61...

:sunny: Test successful - status


fcitx5-mozc: wrapQtApp for mozc_tool

Description of changes
fixes #221958
I found that this derivation has no runHook, but it is not related to this PR.
It is better to create a separate PR to add runHook, right?

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Thank you for reviewing.
I have added runHooks in another commit. If it seems inappropriate, I will remove it.


[dv, util] Make poll_for_stop() opt-in

The poll_for_stop() method in dv_utils_pkg provides a mechanism to
kill the simulation more gracefully on hangs. It looks for the presence
of a file called dv.stop every 1us in the rundir which the user can
'touch' if it needs to be killed.
This mechanism was added because historically, it was not always
reliable to kill a sim by hitting CTRL-C, especially with waves which
would get corrupt on CTRL-C. The other reason was to be able to kill a
specific sims if they are deemed broken already while allowing others to
continue to run.
In any case, by allowing that logic by default introduces a pretty big
bottleneck - it consumes a whopping 54% of simulation runtime,
especially for UART which has periods of idle due to slowness of the
baud rate. In this commit, the logic by default is turned off. If this
feature is needed, it can be turned on by passing --run-opts "+poll_for_stop=1" on the command line.
The other change is to increase the polling interval to 10us.
Signed-off-by: Srikrishna Iyer<EMAIL_ADDRESS>

Yikes! How much speedup do we get by just upping the polling interval? I'm wondering whether it makes sense to keep the unconditional enabling but just running the "am I done" task less often.

Yikes! How much speedup do we get by just upping the polling interval? I'm wondering whether it makes sense to keep the unconditional enabling but just running the "am I done" task less often.

I remember this was added because the wave may be broken, when we kill the simulation with ctrl-C.
If the overhead is over 10%, I would rather to enable it when it's needed.

Yikes! How much speedup do we get by just upping the polling interval? I'm wondering whether it makes sense to keep the unconditional enabling but just running the "am I done" task less often.

I made the interval 10us which barely made any difference (still consumed 52% of the runtime). It's the $system call that consumes too much time compared to the sim. Increasing the interval any more than that may not be very useful.
We should really enhance dvsim's ctrl-c mechanism to prompt the user for input (what would they like to kill - specific set of tests, or everything). As for the waves getting corrupt - I have not seen that happen lately so we are probably good there.

Regarding Ctrl-C and waves, I think there is a difference in behavior between one and more Ctrl-C. Perhaps sending more than one could unconditionally kill the simulator and bypass completing the fsdb file flush.


Proxy list import

Apparently, at the moment we have to enter them manually.
Any plans to implement a list import?
Though it's been said that 1-2 should be enough, the input form is poorly made and doesn't allow copypaste.

Yep, I understand that current proxy input has quite rough implementation. I defenitely have plans to improve it and then consider option with import from file. It would be useful, thanks!


Temporarily ignore RuntimeWarning in test_setitem_extended_API_2d_mask

This appears to be coming from numpy directly. I've opened https://github.com/numpy/numpy/issues/23000 upstream to see if numpy devs have any thoughts on why this may be happening. I'll suggest we ignore the warning in this test for now.
xref https://github.com/dask/dask/issues/9793
cc @hendrikmakait

Thanks @hendrikmakait


Added traits of a good social engineer

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
Closes #XXXXX

The PR does not add anything fruitful. CLOSING


ORC-1255: Fix ORC website navbar highlight

What changes were proposed in this pull request?
This PR aims to fix the ORC website navbar highlight issue.
Why are the changes needed?
BEFORE

AFTER

How was this patch tested?
Manually build and check.

cc: @dongjoon-hyun


Overhaul

Reorganized the existing project to get it working. The only major change was moving the setting of the tenant from a callback to the deserialize method, since deserialization takes place before callbacks are triggered.

I Don't know much about ActiveJob but the only thing I Might suggest is you use
Apartment::Tenant.switch(job_data['tenant']) do
  super
end

Which ensures that the job is run in the current tenant, but reset after it's done (so it doesn't leave you in a weird state)

Will this get merged in ? What's the current status of this gem ?

@LaurensN just merged and made some cleanup, but I haven't personally used this at all.
@desheikh have you tried using this? I'm a bit wary of the installation instructions that monkeypatch ActiveJob. Is there no official way to tap into the processing stack?

Great! Thanks for the quick reply.

@LaurensN thanks for the pull request!
I'm going to look into this today and try to get a release out.
I'm leaning towards an inheritance based solution though, since that's closer to how ActiveJob classes are declared.

Don't thank me @desheikh , thank @dgrouiux ;). Awesome, and thanks.

Oops my bad: @dgroulx


Add @jupyterlab/shared-models to the package.json

The @jupyterlab/shared-models was being referenced here without having its dependency listed in the package.json:
https://github.com/QuantStack/jupytercad/blob/f4d7bbf1e4382fa6a110656790f6f2b5e7e95292/src/model.ts#L3

Thanks @jtpio!


Make LanguageDetector class thread-safe

According to issue #4 , the LanguageDetector class should be thread-safe. This is definitely not the case in the current version, as the languagesSequence variable can be mutated by multiple threads at the same time. This is especially critical combined with the isExcludedFromDetection field of Language. It is hard to reproduce in general because it only seems to happen rarely and with languages with different alphabets. Nevertheless, I think this is a critical issue.

Codecov Report

Merging #9 into v0.5.0-dev will decrease coverage by 0.23%.
The diff coverage is 66.66%.

@@               Coverage Diff                @@
##             v0.5.0-dev       #9      +/-   ##
================================================
- Coverage         55.79%   55.55%   -0.24%
+ Complexity          124      123       -1
================================================
  Files                14       14
  Lines               803      792      -11
  Branches            203      203
================================================
- Hits                448      440       -8
+ Misses              288      285       -3
  Partials             67       67

Impacted Files
Coverage Δ
Complexity Δ

...kotlin/com/github/pemistahl/lingua/api/Language.kt
100% <100%> (ø)
6 <1> (ø)
:arrow_down:

...om/github/pemistahl/lingua/api/LanguageDetector.kt
68.44% <65.21%> (-0.24%)
52 <1> (-1)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c5ad04e...4243e4f. Read the comment docs.

Hi @bgeisberger, you say it is hard to reproduce. How did you reproduce it? Can you please provide a minimal example setup illustrating the problem so that I'm able to understand it better?

Hi,
I reproduced it by detecting the same three texts (for which I know the library should return the correct language) a lot of times (5000+ times) in 8 threads. In some cases the detection returned UNKOWN.
Unfortunately I can't provide a code sample because I ran this test within an internal processor in our NiFi streaming environment.
But I can describe you the problem more in detail:
Imagine one instance of LanguageDetector used by two threads, let's call them T1 and T2.
T1 calls detector.detectLanguageOf(string1), where string1 is some english text.
The method runs until (including) line 104 filterLanguagesByRules(words). In this call, all languages that are not using the latin alphabet are filtered out from languagesSequence. Additionally, for all non-latin languages lang.isExcludedFromDetection = false is set.
At this point T1 loses its CPU slot for a moment and T2 gets active.
T2 also calls detector.detectLanguageOf(string2), where string2 is some russian text. The problem occurs now within filterLanguagesByRules: The method will filter out all languages that are not using the cyrillic alphabet. It will also set lang.isExcludedFromDetection = false for all non-cyrillic languages. As a result, languagesSequence will be empty after line 104, because the detector did not get reset after T1's actions, as T1 has not finished yet either. At least one of those two threads will now return UNKNOWN.
My solution is just to keep languagesSequence local for each call of detectLanguageOf and to not use lang.isExcludedFromDetection, as the Language objects are all singletons. This should fix this problem.

Thank you for the explanation. Now I see the problem. You could then have removed the attribute isExcludedFromDetection from the Language enum altogether since it is not needed anymore with your approach. I will do it after merging your pull request.
Thanks again for your time and effort. 👍

Codecov Report

Merging #9 into v0.5.0-dev will decrease coverage by 0.23%.
The diff coverage is 66.66%.

@@               Coverage Diff                @@
##             v0.5.0-dev       #9      +/-   ##
================================================
- Coverage         55.78%   55.55%   -0.24%
+ Complexity          125      123       -2
================================================
  Files                14       14
  Lines               812      792      -20
  Branches            207      203       -4
================================================
- Hits                453      440      -13
+ Misses              288      285       -3
+ Partials             71       67       -4

Impacted Files
Coverage Δ
Complexity Δ

...kotlin/com/github/pemistahl/lingua/api/Language.kt
100% <100%> (ø)
6 <1> (ø)
:arrow_down:

...om/github/pemistahl/lingua/api/LanguageDetector.kt
68.44% <65.21%> (+0.46%)
52 <1> (-1)
:arrow_down:

...ub/pemistahl/lingua/api/LanguageDetectorBuilder.kt
100% <0%> (+4.54%)
4% <0%> (-1%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fe1dee9...4243e4f. Read the comment docs.

By the way, @bgeisberger, your changes make running the detector faster. Classifying all test data with one CPU core needs approximately 40 seconds less now on my already very potent machine. Awesome! :-)

Hi @pemistahl !
So, is LanguageDetector finally thread safe in the 0.6 version?

Yes, it is @davidecaroselli.


Update NFRs and Glossary

Added relevant non-functional requirements and defined key terms.

Codecov Report

Merging #47 (e6760aa) into master (08fc81f) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head e6760aa differs from pull request most recent head 6d73270. Consider uploading reports for the commit 6d73270 to get more accurate results

@@            Coverage Diff            @@
##             master      #47   +/-   ##
=========================================
  Coverage     72.15%   72.15%
  Complexity      399      399
=========================================
  Files            70       70
  Lines          1232     1232
  Branches        125      125
=========================================
  Hits            889      889
  Misses          311      311
  Partials         32       32

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 08fc81f...6d73270. Read the comment docs.


Query objects are not supported as input to the query method

The query method of the OxigraphStore does not accept rdflib Query objects but only strings and otherwise fails with a TypeError.
Example:
import rdflib
from rdflib.plugins.sparql import prepareQuery
from rdflib.namespace import DCTERMS

graph = rdflib.Graph(store='Oxigraph')
graph.add((rdflib.URIRef('http://localhost/example'), DCTERMS.title, rdflib.Literal('An arbitrary example')))

query_string = f'SELECT ?title WHERE {{?obj <{DCTERMS.title}> ?title}}'
query_object = prepareQuery(query_string)

# This works:
result = graph.query(query_string)
print(list(result)[0][0])

# This does not work:
result = graph.query(query_object)
print(list(result)[0][0])

Maybe the line https://github.com/oxigraph/oxrdflib/blob/b8008f26209e363a79a4a386bf065564f3a181bc/oxrdflib/init.py#L110 should rather read if isinstance(query, Query) or kwargs for the time being?
In the long term, it would be great if prepared Query objects could be supported. Maybe this could result in even further speed ups? Probably this is related to https://github.com/oxigraph/oxrdflib/issues/11 as well.
Thanks a lot for your efforts!

Thank you for this nice bug report! I have fixed the condition in 85ce7d4ec0b0fbf617e53fdf48d2a696ddd29b12 and released v0.3.4 with the fix.

In the long term, it would be great if prepared Query objects could be supported. Maybe this could result in even further speed ups?

Yes, indeed. Not sure about the speed up because there is the conversion cost between Python and Rust and I am not sure if it's much faster than SPARQL parsing. I see two ways of implementing this:

write code to convert the rdflib SPARQ algebra objects to Rust spargebra objects.
Serialize the rdflib SPARQL algebra object back to a SPARQL string and give this string to Oxigraph.

Approach 2 might be slightly slower but might be easier to implement and having a serializer for SPARQL algebra objects might be a relevant feature for rdflib independently of Oxigraph.

Some food for thought:
Rdflib is often perceived as rather slow. I think that the main reasons for this are either:
a) Performance limitations of the underlying store
b) Performance limitations of the SPARQL query parsing
For the project I am working on I first switched to the Oxigraph store because of a hint on Stack Overflow (https://stackoverflow.com/a/71008345). I was really surprised by how much faster my program ran.
Only later I found out that I can achieve an even better runtime by using the default rdflib memory store and preparing & caching rdflib Query objects.
I can imagine that quite some people are using Query objects instead of SPARQL strings just for performance reasons.
While a serialization from rdflib Query objects to SPARQL strings might not be very costly computationally, the generation of the rdflib Query objects (e.g. using the rdflib.plugins.sparql.prepareQuery function) in the first place is computationally expensive.
So maybe instead of going with approach 2 it might even be more sensible to not change anything and rather educate people to use only SPARQL strings and to not even prepare Query objects when using the Oxigraph store.
Side note: How efficient is the creation of Rust spargebra objects from SPARQL strings? Maybe it makes sense to cache/memoize this somehow in Oxigraph?

I ran into a similar problem ages ago, and I submitted a PR with a "temporary" patch, which apparently is still there :laughing: :
a Query object carries with it the arguments used to build it, i.e. the query text, the namespaces, and the base IRI.
I monkeypatched oxrdflib to take advantage of this, and it seems to work fine. It boils down to the following preprocessing lines:
if isinstance(query, rdflib.plugins.sparql.sparql.Query) and hasattr(query, "_original_args"):
    query, initInitNs, base = query._original_args
    initInitNs.update(initNs)
    initNs = initInitNs
    if base:
        kwargs.setdefault('base', base)
base = kwargs.pop('base', None)
if base != None:
    query = f'BASE <{base}>\n{query}'

@pchampin Nice! Thank you!
I am a bit reluctant to use it in oxrdflib. I guess that if the Query object gets mutated after parsing but before being given to the Store.query method, the mutations will get ignored with this code. This looks like a significant footgun to me. Am I correct?

I guess that if the Query object gets mutated after parsing but before being given to the Store.query method, the mutations will get ignored with this code.

I would never have considered mutating an instance of Query, but it is true that the attributes are apparently "public" (per the naming convention), so in principle "open for mutation"... but I would consider that (mutating the attribute of a Query instance) as a footgun...
Would you be more comfortable to apply the strategy above if all attributes of Query were "protected" in RDFlib? If so, it might be worth asking what the intention of the authors are, and proposing a patch.


[Openfoam] Clean Simulator and Scenarios

This PR follows the same idea of #587 and simplifies the simulator name to Openfoam and cleans up the respective scenarios.
To test the changes I ran the following script for the WindTunnel - moreover I also perform tests in the WindOverTerrain and HeatSink eventhough they won't be documented for the release:
scenario = inductiva.fluids.WindTunnel(
        flow_velocity=[20, 0, 0],
        domain={"x": [-7, 12], "y": [-5, 5], "z": [0, 8]})

task = scenario.simulate(
    num_iterations=100,
    object_path="meta_object.obj",
    resolution="low",
    n_cores=2)

# Get all the WindTunnel simulation files
output = task.get_output(all_files=True)

Quick question. I run the example and got this:
(base) lsarmento@Luiss-MacBook-Air swash % python test2.py
Traceback (most recent call last):
  File "test2.py", line 8, in <module>
    task = scenario.simulate(
  File "/usr/local/anaconda3/lib/python3.8/site-packages/inductiva/fluids/wind_tunnel/wind_tunnel.py", line 137, in simulate
    task = super().simulate(simulator,
  File "/usr/local/anaconda3/lib/python3.8/site-packages/inductiva/scenarios.py", line 51, in simulate
    self.create_input_files(simulator, input_dir)
  File "/usr/local/anaconda3/lib/python3.8/functools.py", line 912, in _method
    return method.__get__(obj, cls)(*args, **kwargs)
  File "/usr/local/anaconda3/lib/python3.8/site-packages/inductiva/fluids/wind_tunnel/wind_tunnel.py", line 214, in _
    shutil.copy(self.object_path, os.path.join(object_dir, "object.obj"))
  File "/usr/local/anaconda3/lib/python3.8/shutil.py", line 418, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/local/anaconda3/lib/python3.8/shutil.py", line 264, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/lsarmento/swash/meta_object.obj'

Was this object supposed to be in the release?
How do we always have a default object (like the Utah Teapot)?

Quick question. I run the example and got this:
(base) lsarmento@Luiss-MacBook-Air swash % python test2.py
Traceback (most recent call last):
  File "test2.py", line 8, in <module>
    task = scenario.simulate(
  File "/usr/local/anaconda3/lib/python3.8/site-packages/inductiva/fluids/wind_tunnel/wind_tunnel.py", line 137, in simulate
    task = super().simulate(simulator,
  File "/usr/local/anaconda3/lib/python3.8/site-packages/inductiva/scenarios.py", line 51, in simulate
    self.create_input_files(simulator, input_dir)
  File "/usr/local/anaconda3/lib/python3.8/functools.py", line 912, in _method
    return method.__get__(obj, cls)(*args, **kwargs)
  File "/usr/local/anaconda3/lib/python3.8/site-packages/inductiva/fluids/wind_tunnel/wind_tunnel.py", line 214, in _
    shutil.copy(self.object_path, os.path.join(object_dir, "object.obj"))
  File "/usr/local/anaconda3/lib/python3.8/shutil.py", line 418, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/local/anaconda3/lib/python3.8/shutil.py", line 264, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/lsarmento/swash/meta_object.obj'

Was this object supposed to be in the release? How do we always have a default object (like the Utah Teapot)?

This object is available in the resources of the github, but it does not go with the package itself. In the docs it explains how to get it. However, I agree that we can now do an interface similar to what Cristiana is doing for some of this samples.


fix #2140 by ensuring the access token flows to WebSocketTransport

Right now I do it by rewriting the access token factory on the HttpConnectionOptions. I'm open to other options, we could flow it through the factory but it gets a bit funky because we either have to deal with the fact that the access token provider is async but ITransportFactory is not (currently).

Doesn't this break Stop then Start?

Doesn't this break Stop then Start?

Yeah, it probably would. I'll plumb a delegate that pulls the right field off the HttpConnection through instead then

:up: :date: Now I flow a delegate to HttpConnection.GetAccessTokenAsync() down through to the transport so it can invoke and await it right before starting the WebSocket transport. The other transports use the HttpMessageHandler, so they're good.


Unable to invite users

The steps in the section Invite a guest user to an app from the Access Panel through "Manage your application" redirects to Microsoft Entra user management, which has no option to add/invite email addresses. It's only possible to select users who already exist in the directory. This conflicts with the documentation where its stated that "The guest user doesn't necessarily need to be added to Azure AD in advance.".
The steps in the section Invite someone to join a group that has access to the app redirect to a different looking "My Groups" interface, than the pictures shown. With the Add option, it's only possible to "search by name or alias", and the Add button remains greyed out if you put in an email address. This again makes it impossible to invite someone to the group without adding them to the directory in advance.
Both were tried as admin users with Global Administrator, Guest Inviter, application owner, and group ownership privileges.
Setup was done following the steps in this documentation.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 7fbdc24d-5bc5-54f0-208c-8574b19dfb40
Version Independent ID: 09e9c6cf-8626-532b-8a59-640ec5d4798a
Content: Add B2B collaboration users as an information worker - Microsoft Entra
Content Source: articles/active-directory/external-identities/add-users-information-worker.md
Service: active-directory
Sub-service: b2b
GitHub Login: @csmulligan
Microsoft Alias: cmulligan

@andrasf12
Thanks for your feedback! We will investigate and update as appropriate.

Hi @andrasf12 You invite a guest user to an app from the Access Panel through "Manage your application" and inviting someone to join a group that has access to the app
you can invite a guest user to the directory, to a group, or to an application. To add a guest user to a group, you can follow these steps:
Sign in to the Azure portal as an Azure AD administrator.
Search for and select Azure Active Directory from any page.
Under Manage, select Groups.
Select a group (or select New group to create a new one). It's a good idea to include in the group description that the group contains B2B guest users.
Under Manage, select Members.
Select Add members.
Complete one of the following set of steps:
If the guest user is already in the directory:
On the Add members page, start typing the name or email address of the guest user.
In the search results, choose the user, and then choose Select.
If the guest user isn't currently in the directory:
On the Add members page, type the user's full email address in the search box, type a Personal message (optional), and then choose Invite to send an invitation.
Choose Select.
Now add the user to the group: On the Members page, choose Add members. Type the user's name or email address in the search box, choose the user in the results, and then choose Select.
If you are still having trouble, you can try to follow the steps in the documentation for adding a guest user and sending an invitation.
I'd recommend working closer with our support team via an [Azure support request] (https://docs.microsoft.com/en-us/azure/azure-portal/supportability/how-to-create-azure-support-request).

@andrasf12
We are going to close this thread as resolved but if there are any further questions regarding the documentation, please tag me in your reply and we will be happy to continue the conversation.

I wouldn't consider this thread to be "resolved."  The steps for "Invite someone to join a group that has access to the app" do not match the current UI in the portal. Now group owners that used this feature to invite external users and add to the group at the same time, through a mostly friendly interface, are suddenly no longer able to.  I understand wanting to update the UI every so often, but not at the expense of reducing functionality.
Also, I don't feel that closing the thread as resolved is a good response when the article no longer provides accurate information.

@andrasf12
Sorry for in convince, so I will make sure you will get accurate answer for your query. I'm going to assign this to the document author so they can take a look at it accordingly.

@csmulligan
Please review it.

Hi @SonicSteel thank you so much for pointing this out. I've added this to an internal work item for an update. Since we're tracking it there, I'll go ahead and close this issue. Feel free to reopen it if you have anything to add. Thanks again!
#please-close


Question: Is CppuTest usable for Bare Metal embedded development?

Hi,
I would like to use a test framework with which we can run test on the hardware. We have no OS. So my question is CppuTest can be run on a bare metal embedded device?
Regards Stefan

Yes, although there might be a problem with memory usage of CppUTest itself.
You will need to create a new "Platform" and implement all of the dependencies to Standard C (they are all separated out). We tried to minimize these, so it shouldn't be a huge amount of work.

ok thanks for the answer. We try to get cpputest running on our https://www.st.com/en/evaluation-tools/nucleo-f411re.html
let me know if you are aware of some helpfull tips.

I'd check what is there already. IAR and Keil support was implemented earlier, I believe. You'll mostly ought to check the "Platform" directories which is where all the dependencies to platform and Standard C are.

I updated trunk last week to move a few standard C functions that had sneaked into he codebase to the PlatformSpecific functions. So the current trunk ought to be better than 2 weeks ago.


docs: fix typos

This PR fixes a few typos in CHANGELOG.md. It was created using https://github.com/tbroadley/github-spellcheck-cli.

          1 Message

:book:
@tbroadley is not a contributor yet, would you like to join the Danger org?

  Generated by :no_entry_sign: Danger

Ace 👍


Need example for schedule-yaml (there are other yaml files that need to be documented.)

We need an example for the schedule-yaml  for:
az ml pipeline create-schedule --experiment-name
                               --name
                               --pipeline-id
                               [--only-show-errors]
                               [--resource-group]
                               [--schedule-yaml]
                               [--workspace-name]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 90bc921c-f7f2-0c82-9540-b3ee2dd471ba
Version Independent ID: 00d7a81a-5be2-345d-200f-f89e9e8a7754
Content: az ml pipeline
Content Source: latest/docs-ref-autogen/ext/azure-cli-ml/ml/pipeline.yml
GitHub Login: @rloutlaw
Microsoft Alias: routlaw

ml pipeline


Is it possible to provide a command to shut down gke-local.

Now we can only find related processes through the keyword "gke-local" and forcefully kill them.

Hi @shijian-git! We just merged a weaver gke-local purge command into main that shuts down all gke-local processes and purges all gke-local data. It should be available on the next release. Thanks again for your suggestion!


Availability problem when trying proxing too many servers.

We had next infrastructure:
3 servers: each server had 2 redis instances run, and each server had own twemproxy instance, which looked on all 6 redis instances on all 3 serevers. And everything worked fine.
Now, we rich limit of this infrastructure, and try increase amount of redis instances:
1 server: 6 redis instances (by amount of available CPU cores)
2 server: 6 redis instances (by amount of available CPU cores)
3 server: 12 redis instances (by amount of available CPU cores)
And we found a problem:
50% of requests to twemproxy return all available data from all redis instances
50% of requests to twemproxy return only data from 30-60% redis instances available, and there is no errors on twemproxy logs.
Our new config and old config the same, just differe list of servers 6 vs 24:
listen: <IP_ADDRESS>:22101
hash: fnv1a_64
distribution: ketama
auto_eject_hosts: true
redis: true
server_connections: 32
preconnect: true
servers:

<IP_ADDRESS>:6380:1
<IP_ADDRESS>:6381:1
<IP_ADDRESS>:6382:1
<IP_ADDRESS>:6383:1
<IP_ADDRESS>:6384:1
<IP_ADDRESS>:6385:1
<IP_ADDRESS>:6386:1
<IP_ADDRESS>:6387:1
<IP_ADDRESS>:6388:1
<IP_ADDRESS>:6389:1
<IP_ADDRESS>:6390:1
<IP_ADDRESS>:6391:1
<IP_ADDRESS>:6380:1
<IP_ADDRESS>:6381:1
<IP_ADDRESS>:6382:1
<IP_ADDRESS>:6383:1
<IP_ADDRESS>:6384:1
<IP_ADDRESS>:6385:1
<IP_ADDRESS>:6380:1
<IP_ADDRESS>:6381:1
<IP_ADDRESS>:6382:1
<IP_ADDRESS>:6383:1
<IP_ADDRESS>:6384:1
<IP_ADDRESS>:6385:1

Where could be a problem?  Servers had private internal network with 1Gb, and high quality hardware.

Try to compile with ./configure --enable-debug=full and increase the  --verbose parameter (e.g.: --verbose=7) first. You should see detailed logs after that.
I guess Hosts-ejection may be blamed since auto_eject_hosts is true in your conf file. If it happened, your should see server_ejects > 0 in the stats(tcp://twemproxy:2222 by default).

We already compile with verbose --11, in log we have just:
[2015-07-02 11:32:37.774] nc.c:192 run, rabbit run / dig that hole, forget the sun / and when at last the work is done / don't sit down / it's time to dig another one
[2015-07-02 11:50:57.642] nc_response.c:118 s 12 active 0 is done
[2015-07-02 11:50:57.642] nc_response.c:118 s 43 active 0 is done
[2015-07-02 11:50:57.643] nc_response.c:118 s 83 active 0 is done
[2015-07-02 11:50:57.643] nc_response.c:118 s 93 active 0 is done
Last rows was appeared in moment, when we restart redis instances manually.

I tried to with ./configure --enable-debug=no and run via  ./src/nutcracker -c conf/local.yml --verbose=11
[2015-07-02 18:18:31.796] nc.c:187 nutcracker-0.4.0 built for Linux 3.8.13-gentoo-r1 x86_64 started on pid 13742
[2015-07-02 18:18:31.796] nc.c:192 run, rabbit run / dig that hole, forget the sun / and when at last the work is done / don't sit down / it's time to dig another one
[2015-07-02 18:18:41.556] nc_response.c:118 s 9 active 0 is done

[2015-07-02 18:18:41.556] nc_response.c:118 s 9 active 0 is done This line occurs when I shutdown one of my backend server(memcached) deliberately. Are you sure you've compiled correctly?

Yep, we have same behavier, at moment we had done one redis instances, we got same log row in log file, as u described. Can be problem in backlog parameter?

Your might missed that I compiled with ./configure --enable-debug=no(no, not full). If I compile with full, I can see lines of verbose log.
...
[2015-07-02 18:42:14.517] nc_stats.c:1151 incr field 'out_queue' to 1
[2015-07-02 18:42:14.517] nc_stats.c:1134 metric 'out_queue_bytes' in pool 0 server 11
[2015-07-02 18:42:14.517] nc_stats.c:1181 incr by field 'out_queue_bytes' to 28
[2015-07-02 18:42:14.517] nc_stats.c:990 skip swap of current 0x1a690d0 shadow 0x1a5b4c0 as aggregator is busy
[2015-07-02 18:42:14.518] nc_epoll.c:254 epoll 0001 triggered on conn 0x1a978b0
[2015-07-02 18:42:14.518] nc_core.c:324 event 00FF on s 13
[2015-07-02 18:42:14.518] nc_util.c:228 malloc(32) at 0x1ab4ac0 @ nc_array.c:29
[2015-07-02 18:42:14.518] nc_util.c:228 malloc(16) at 0x1a6d290 @ nc_array.c:34
[2015-07-02 18:42:14.518] nc_message.c:319 get msg 0x1a8ad10 id 578 request 0 owner sd 13
[2015-07-02 18:42:14.518] nc_mbuf.c:99 get mbuf 0x1ac5140
[2015-07-02 18:42:14.518] nc_mbuf.c:182 insert mbuf 0x1ac5140 len 0
[2015-07-02 18:42:14.518] nc_connection.c:362 recv on sd 13 2059 of 16336
[2015-07-02 18:42:14.518] nc_memcache.c:1186 parsed rsp 578 res 0 type 19 state 0 rpos 2059 of 2059
00000000  56 41 4c 55 45 20 74 65  73 74 5f 6b 65 79 5f 33   |VALUE test_key_3|
00000010  20 30 20 31 30 30 30 0d  0a 69 69 69 69 69 69 69   | 0 1000..iiiiiii|
00000020  69 69 69 69 69 69 69 69  69 69 69 69 69 69 69 69   |iiiiiiiiiiiiiiii|
00000030  69 69 69 69 69 69 69 69  69 69 69 69 69 69 69 69   |iiiiiiiiiiiiiiii|
00000040  69 69 69 69 69 69 69 69  69 69 69 69 69 69 69 69   |iiiiiiiiiiiiiiii|
00000050  69 69 69 69 69 69 69 69  69 69 69 69 69 69 69 69   |iiiiiiiiiiiiiiii|
00000060  69 69 69 69 69 69 69 69  69 69 69 69 69 69 69 69   |iiiiiiiiiiiiiiii|
...

Use the timeout directive in your config. Have you read the documentation in recommendation.md

@manjuraj can you please describe how in particular timeout will help here? as we see all servers can be reached by twemproxy, so there is no problem in timeouts.

@manjuraj can you please also explain what does following lines means?
[2015-09-30 16:20:39.953] nc_stats.c:671 skip aggregate of shadow 0x1636160 to sum 0x1639e90 as generator is slow
[2015-09-30 16:20:56.685] nc_stats.c:996 skip swap of current 0x16267d0 shadow 0x1636160 as there is nothing new
[2015-09-30 16:21:09.976] nc_stats.c:671 skip aggregate of shadow 0x1636160 to sum 0x1639e90 as generator is slow
[2015-09-30 16:21:26.711] nc_stats.c:996 skip swap of current 0x16267d0 shadow 0x1636160 as there is nothing new
[2015-09-30 16:21:40.002] nc_stats.c:671 skip aggregate of shadow 0x1636160 to sum 0x1639e90 as generator is slow
[2015-09-30 16:21:56.731] nc_stats.c:996 skip swap of current 0x16267d0 shadow 0x1636160 as there is nothing new
[2015-09-30 16:22:10.026] nc_stats.c:671 skip aggregate of shadow 0x1636160 to sum 0x1639e90 as generator is slow

generator is slow

what this line is about in particular? is it about generator for consistent hashing for keys?

generator is slow is about stats; nothing pertinent here

@Ragazzo open a new issue with the problem you are facing

no, actually i am currently working on solving this problem that have been described in this issue. Currently as i see it works fine on 12 and 18 instances when there is no high traffic, so it could be that it is timeouts issue when, will see.


Write a function

As the Solution is updated to the repo already : https://github.com/jarvismayur/Hackerrank-Problem-Solutions/blob/main/Python/Write a function/Write_a_function.py


fatal error on androidx.annotation

Execution failed for task ':app:licenseProdReleaseReport'.

org.xml.sax.SAXParseException; systemId: file:///Users/ciprian.grigor/.gradle/caches/modules-2/files-2.1/androidx.annotation/annotation/1.3.0-alpha01/bead8516d0fc24d4b39d3cdcf05e60433d9ae660/annotation-1.3.0-alpha01.jar; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.

Do you have an example? I have this working here: https://github.com/jaredsburrows/android-gif-example.

Having the same issue.


Stats are wrong when period > 24 hours

Default 24 hours looks fine:
[midas@localhost ~]$ minotaur --stats 24

stats for the last 24 hours:

 from: 2018-02-12 19:36:48
   to: 2018-02-13 19:30:58

total:    2.50 mBTC
        130.52  DKK

 rate:    2.51 mBTC/day
        131.05  DKK/day
       3931.65  DKK/month

income by algorithm:

equihash: 2.4200 mBTC  126.21 DKK
 x11gost: 0.0827 mBTC  4.31 DKK

Something is fishy when viewing 48 hour stats though. If total earnings in the last ~48 hours is 3.93 mBTC then surely the daily rate should be half of that, and not 3.95 mBTC.
[midas@localhost ~]$ minotaur --stats 48

stats for the last 48 hours:

 from: 2018-02-11 19:37:48
   to: 2018-02-13 19:30:58

total:    3.93 mBTC
        205.00  DKK

 rate:    3.95 mBTC/day
        205.98  DKK/day
       6179.35  DKK/month

income by algorithm:

       equihash: 3.2219 mBTC  168.03 DKK
        x11gost: 0.3280 mBTC  17.10 DKK
      neoscrypt: 0.2101 mBTC  10.96 DKK
daggerhashimoto: 0.1504 mBTC  7.84 DKK
        blake2s: 0.0119 mBTC  0.62 DKK
      lyra2rev2: 0.0085 mBTC  0.44 DKK

fixed in master


Replace asserts that might fail

with an explicit exception raising.
I'm expecting a Codecov failure.

Codecov is complaining.  I'll try with patch.object to touch the diff.

Codecov is complaining.  I should be able to touch the line.  https://stackoverflow.com/questions/39183741/how-to-mock-components-within-unit-tests-for-click-based-cli-applications

The coverage improved a bit.  Maybe private_key_to_address is always failing?

The coverage increased further.

I'll do that in this PR.


Is there a way to pass state with route method?

Hi there,
I guess that title of this issue sums up my question pretty well, but to expand a little bit.
I want to pass state of one component to another which is in this case iframe.
I would like to be able to do this:
route("/preview", this.state);

I've just started with Preact and probably I'm not doing it inline with communities best practices but still I would love if someone could point me to some resource or technique for resolving this problem.
Thanks!

Nope - the issue there would be that the URL no longer represents your routing state - that transient object you passed is the real state.  You can serialize the state to a URL querystring (or as JSON), or bypass routing to set the state.

Thanks for the answer! I've done it with querystring, good thing is that I can now share that URL and load preview of page with that configuration, but at the end it feels kinda dirty.

This is good to close, right Jason?

Yup!

Navigating with window.history.pushState(state, title [, url]) one may store state within the globally accessible window.history object.
https://developer.mozilla.org/en-US/docs/Web/API/History/pushState
I don't see why this couldn't be implemented in preact-router.
react-router already is utilising this.
https://reactrouter.com/web/api/history

@denysonique yup, you can pass the same custom history object to preact-router too. See this section in the README: https://github.com/preactjs/preact-router#custom-history


New errors checks on ConfigFile::load() generate unnecessary error prints when file doesn't exist yet

Godot version:
3.2 master (86371b7), regression from #31077 (@qarmin)
OS/device including version:
Any
Issue description:
PR #31077 added missing return value checks for ConfigFile::load(), but missed the fact that this API can be used successfully even if err != OK, as it will return ERR_CANT_OPEN if the file can't be opened, which also occurs if the file doesn't exist.
Code that uses ConfigFile typically attempts to load files even if not present, so that they can be either edited or created and then re-saved. So in this case no error should be raised.
Current code raises errors all the time when you e.g. save a scene, as the project metadata for the scene doesn't exist yet. And the metadata is not created at all due to this early exit.
The ConfigFile API could maybe be changed to use a more specific error when the file can't be loaded because it doesn't exist, so that it can be treated as a harmless error (e.g. ERR_FILE_NOT_FOUND).
Steps to reproduce:

Create a new project
Create new scene, save it
Check terminal

Stumbled upon this while testing #31111, project metadata was used for saving last language and template used, so new projects would emit this error but only once, which got me confused.


Add in functionality for dashboard page buttons

Make sure the dashboard isn't static and is actually getting information from the database

Interested

Interested

interested

Interested


Input Normalization

I'm not sure if my mixing is exactly same as yours, but does your torchaudio read the wav files to int(value is typically around a couple hundred) or to float values between [-1, 1]?
I started with scipy which loads to int, and it caused loss going to NaN at a point. So I switched to librosa which loads to float

float values between [-1, 1]. if you started with scipy which loads to int, you will norm audio between [-1,1]

how long did it take for you to train? I'm using 4x1080Ti and each epoch takes around 2 hrs. I use wsj0-2mix, and take 4s windows with 2s skip size, with 2s least length. Is this speed normal?

Also, just to confirm, norming between [-1, 1] is just dividing int by 2**15, right? Or is it minmax normalization?


file does not exist

Hi, the inference.py file on which the work is based, does not exist. Could you please fix this?

Hey Vijeth! I'm sorry for the inconvenience. I have added the inference.py file


Tenant id must be accessed from RM context with Tenant.TenantId

@iansul,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by Microsoft. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot


switch relative path when host doesn't include 's3.' in url

I want to use below library on my local for development.

https://github.com/jamhall/s3rver

Hey!
Thanks for the PR. My initial reaction here is :-1: because it seems like this change only exists to help compatibility with one particular fake s3 implementation. It seems like some /etc/hosts entries are enough to make the existing functionality work, and even if that isn't enough for this particular implementation, that strikes me as a problem with s3rver for not handling the AWS S3 spec.

@benwilson512
Thank you for replying, I see :smile:
I will see more about s3rver.
Thank you.


Enhancement: Update friendsofphp/php-cs-fixer

This PR

[x] updates friendsofphp/php-cs-fixer

💁♂ For reference, see https://github.com/FriendsOfPHP/PHP-CS-Fixer/compare/v2.13.1...v2.15.0.

Build failure is unrelated and fixed with #40.

Build failure is unrelated and fixed with #45.

Thank you, @Jan0707!


[FLINK-35378] [FLIP-453] Promote Unified Sink API V2 to Public and Deprecate SinkFunction

What is the purpose of the change
Promote Unified Sink API V2 to Public and Deprecate SinkFunction
Brief change log
da30c2faaa487ca03304b90553326ab189fba699 Deprecate SinkFunction, SocketClientSink and TwoPhaseCommitSinkFunction
aa94e3822323bbb49bbac81a06a7535aa55c625b Promote SinkV2 interfaces
Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): yes
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

@flinkbot run azure


Virtualbox builds failing on latest update

Description
Starting September 18, my Virtualbox image builds have failed with the initial configuration being left in 'gurumeditation' state. I tried updating Virtualbox, Vagrant, Packer, and all underlying software in the build script but none of it worked. I believe the issue is due to the kernel and/or hypervisor configuration in the Github Actions runner itself.
Platforms affected

[ ] Azure DevOps
[X] GitHub Actions - Standard Runners
[ ] GitHub Actions - Larger Runners

Runner images affected

[ ] Ubuntu 20.04
[X] Ubuntu 22.04
[ ] Ubuntu 24.04
[ ] macOS 12
[ ] macOS 13
[ ] macOS 13 Arm64
[ ] macOS 14
[ ] macOS 14 Arm64
[ ] Windows Server 2019
[ ] Windows Server 2022

Image version and build link
Current runner version: '2.319.1'
Operating System
Ubuntu
22.04.5
LTS
Runner Image
Image: ubuntu-22.04
Version: 20240915.1.0
Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240915.1/images/ubuntu/Ubuntu2204-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240915.1
Runner Image Provisioner
2.0.384.1
GITHUB_TOKEN Permissions
Contents: read
Metadata: read
Packages: read
https://github.com/ssplatt/vagrant-box-centos-stream-9/actions/runs/10933205667/job/30351289562
Is it regression?
https://github.com/ssplatt/vagrant-box-centos-stream-9/actions/runs/10914285276
Expected behavior
virtual machine starts, runs, builds, and shuts down.
last good build:
Current runner version: '2.319.1'
Operating System
  Ubuntu
  [2](https://github.com/ssplatt/vagrant-box-centos-stream-9/actions/runs/10914285276/job/30292164221#step:1:2)2.04.4
  LTS
Runner Image
  Image: ubuntu-22.04
  Version: 20240908.1.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240908.1/images/ubuntu/Ubuntu2204-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240908.1
Runner Image Provisioner
  2.0.[3](https://github.com/ssplatt/vagrant-box-centos-stream-9/actions/runs/10914285276/job/30292164221#step:1:3)84.1
GITHUB_TOKEN Permissions
  Contents: read
  Metadata: read
  Packages: read
 ...
 Run make build
  make build
  shell: /usr/bin/bash -e {0}
./scripts/build_box_virtualbox.sh
Installed plugin github.com/hashicorp/vagrant v1.1.5 in "/home/runner/.config/packer/plugins/github.com/hashicorp/vagrant/packer-plugin-vagrant_v1.1.5_x5.0_linux_amd64"
The configuration is valid.
==> vagrant.box: Creating a Vagrantfile in the build directory...
==> vagrant.box: Adding box using vagrant box add ...
    vagrant.box: (this can take some time if we need to download the box)
==> vagrant.box: Calling Vagrant Up (this can take some time)...
==> vagrant.box: Using SSH communicator to connect: 1[2](https://github.com/ssplatt/vagrant-box-centos-stream-9/actions/runs/10914285276/job/30292164221#step:5:2)<IP_ADDRESS>
==> vagrant.box: Waiting for SSH to become available...
==> vagrant.box: Connected to SSH!
==> vagrant.box: Provisioning with shell script: provision.sh
...
    vagrant.box: Complete!
==> vagrant.box: Created symlink /etc/systemd/system/multi-user.target.wants/vboxservice.service → /usr/lib/systemd/system/vboxservice.service.
==> vagrant.box: Removed "/etc/systemd/system/default.target".
==> vagrant.box: Created symlink /etc/systemd/system/default.target → /usr/lib/systemd/system/graphical.target.
==> vagrant.box: Packaging box...
==> vagrant.box: destroying Vagrant box...
Build 'vagrant.box' finished after 13 minutes 52 seconds.

==> Wait completed after 13 minutes 52 seconds

==> Builds finished. The artifacts of successful builds are:
--> vagrant.box: Vagrant box 'package.box' for 'virtualbox' provider

Actual behavior
Run vagrant --version
  vagrant --version
  packer --version
  vboxmanage --version
  shell: /usr/bin/bash -e {0}
Vagrant [2](https://github.com/ssplatt/vagrant-box-centos-stream-9/actions/runs/10933205667/job/30351289562#step:6:2).3.7
Packer v1.11.2
6.1.50_Ubuntur1610[3](https://github.com/ssplatt/vagrant-box-centos-stream-9/actions/runs/10933205667/job/30351289562#step:6:3)3

./scripts/build_box_virtualbox.sh
Installed plugin github.com/hashicorp/vagrant v1.1.[5](https://github.com/ssplatt/vagrant-box-centos-stream-9/actions/runs/10933205667/job/30351289562#step:7:6) in "/home/runner/.config/packer/plugins/github.com/hashicorp/vagrant/packer-plugin-vagrant_v1.1.5_x5.0_linux_amd64"
The configuration is valid.
==> vagrant.box: Creating a Vagrantfile in the build directory...
==> vagrant.box: Adding box using vagrant box add ...
    vagrant.box: (this can take some time if we need to download the box)
==> vagrant.box: Calling Vagrant Up (this can take some time)...
==> vagrant.box: Vagrant error: The guest machine entered an invalid state while waiting for it
==> vagrant.box: to boot. Valid states are 'starting, running'. The machine is in the
==> vagrant.box: 'gurumeditation' state. Please verify everything is configured
==> vagrant.box: properly and try again.
==> vagrant.box:
==> vagrant.box: If the provider you're using has a GUI that comes with it,
==> vagrant.box: it is often helpful to open that and watch the machine, since the
==> vagrant.box: GUI often has more helpful error messages than Vagrant can retrieve.
==> vagrant.box: For example, if you're using VirtualBox, run `vagrant up` while the
==> vagrant.box: VirtualBox GUI is open.
==> vagrant.box:
==> vagrant.box: The primary issue for this error is that the provider you're using
==> vagrant.box: is not properly configured. This is very rarely a Vagrant issue.
==> vagrant.box:
==> vagrant.box: Step "StepUp" failed, aborting...
==> vagrant.box: aborted: skipping cleanup of step "StepAddBox"
==> vagrant.box: aborted: skipping cleanup of step "StepCreateVagrantfile"
==> vagrant.box: aborted: skipping cleanup of step "StepOutputDir"
Build 'vagrant.box' errored after 54 seconds 54[7](https://github.com/ssplatt/vagrant-box-centos-stream-9/actions/runs/10933205667/job/30351289562#step:7:8) milliseconds: Vagrant error: The guest machine entered an invalid state while waiting for it
to boot. Valid states are 'starting, running'. The machine is in the
'gurumeditation' state. Please verify everything is configured
properly and try again.

If the provider you're using has a GUI that comes with it,
it is often helpful to open that and watch the machine, since the
GUI often has more helpful error messages than Vagrant can retrieve.
For example, if you're using VirtualBox, run `vagrant up` while the
VirtualBox GUI is open.

The primary issue for this error is that the provider you're using
is not properly configured. This is very rarely a Vagrant issue.

==> Wait completed after 54 seconds 547 milliseconds

==> Some builds didn't complete successfully and had errors:
--> vagrant.box: Vagrant error: The guest machine entered an invalid state while waiting for it
to boot. Valid states are 'starting, running'. The machine is in the
'gurumeditation' state. Please verify everything is configured
properly and try again.

If the provider you're using has a GUI that comes with it,
it is often helpful to open that and watch the machine, since the
GUI often has more helpful error messages than Vagrant can retrieve.
For example, if you're using VirtualBox, run `vagrant up` while the
VirtualBox GUI is open.

The primary issue for this error is that the provider you're using
is not properly configured. This is very rarely a Vagrant issue.

==> Builds finished but no artifacts were created.
make: *** [Makefile:4: build] Error 1
Error: Process completed with exit code 2.

Repro steps
start build, fail.

Hey!
I also encountered this problem after updating the images to 20240915.1.0
On images 0240908.1.0 all was fine

Hi @ssplatt ,Thank you for bringing this issue to us. We are looking into this issue and will update you on this issue after investigating.

rolling back to ubuntu-20.04 does work, btw.

seeing this issue on ubuntu-latest (virtualbox 6) and ubuntu-24.04 (virtualbox) ... enabling video capture in packer via:
vboxmanage = [
    ["modifyvm", "{{.Name}}", "--recordingscreens", "all"],
    ["modifyvm", "{{.Name}}", "--recordingmaxsize", "500"],
    ["modifyvm", "{{.Name}}", "--recordingfile", "capture.webm"],
    ["modifyvm", "{{.Name}}", "--recording", "on"],

shows how the kernel boots, but then everything gets stuck with no error shown.

I'm facing the same issue since the release of runner image 20240915 . Unfortunately reverting to ubuntu-20 is not a solution in my case.

HI @ssplatt , @benrobot @danilapog
Could you try with the new image version (20241006.1) . it might fix your issue.

Thank you for your confirmation , we're closing the issue as completed.


basline_anchors

Hello，
How did you get your “basline_anchors.txt”?
                                       Thank you

In Yolo, i think the shapes of the anchors are calculated by using K-Mean Clustering on the bounding box of the training images.

在Yolo中，我认为锚的形状是通过在训练图像的边界框上使用K均值聚类来计算的。

Are you using basline_anchors.txt or your own anchor file?
How was the final training?

@HtutLynn How can i gen it for my data bro ?

Anyone figured this out? Pls help me out. THX

@cong235 , The author has already mentioned it on README. You can get it by referencing this code - https://nbviewer.jupyter.org/github/YunYang1994/tensorflow-yolov3/blob/master/docs/Box-Clustering.ipynb

Anyone

Hello，
How did you get your “basline_anchors.txt”?

Same question, anyone got it?


Vulnerability roundup 109: vault-1.7.4: 1 advisory [9.1]

search, files

[ ] CVE-2021-43998 CVSSv3=9.1 (nixos-21.05)

CVE details
CVE-2021-43998
HashiCorp Vault and Vault Enterprise 0.11.0 up to 1.7.5 and 1.8.4 templated ACL policies would always match the first-created entity alias if multiple entity aliases exist for a specified entity and mount combination, potentially resulting in incorrect policy enforcement. Fixed in Vault and Vault Enterprise 1.7.6, 1.8.5, and 1.9.0.

Scanned versions: nixos-21.05: 3b422991781.
Cc @Chili-Man
Cc @lnl7
Cc @offlinehacker
Cc @pradeepchhetri
Cc @rushmorem

See also: #143008, #138697

NixOS 21.05 is EOL


Investigate collations potentially messing with DETERMINISTIC stored functions

From gitter:

From what I understand, if I have a function CMP("a", "b") that returns the sort order of "a" and "b" that function cannot be marked as IMMUTABLE, because a change to the collation may change its output (since the sort order of those two characters may have changed)
IMMUTABLE functions need to be IMMUTABLE across database versions, too.
Or at least, in MySQL, they do (I forget how mysql marks immutable procedures)

http://thewindowsupdate.com/2020/12/13/dont-let-collation-versions-corrupt-your-postgresql-indexes/

https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-sets.html

MySQL is probably not affected


adding new APIs under System.Linq.* namespaces

@VSadov, @OmarTawfik who should review these new methods under the Enumerable class?

This is a documentation improvement.
The APIs themselves are not very new now. I think we have reviewed them long time ago.

Yeah, API review is needed only for new API in the code, not for docs.

Yes, we're only catching up with the docs now. @OmarTawfik @VSadov
Thank you for your review!


graphviz rendering of class diagram no longer working

The class diagram at https://docs.ros.org/en/galactic/Concepts/About-Executors.html#types-of-executors is no longer rendered. In a local setup however, it works for me. Is this a general issue with graphviz?

@ralph-lange Good call.  I think #2253 will fix this.


Event Ontology

Hi,
I have been using this library for a project. I am using it for event detection, but I have not found the exact event ontology used to train the model.
Does this ontology comprise event types from both ACE and MAVEN? Or is there any custom event ontology for the Event Detection model? Where can I access the ontology file?
Thank you.

The ontology file will be stored in ./data/processed/<dataset_name> after data processing.


Documentation discussion: invoking connect function as middleware leads to event listener accumulation

I didnt find anywhere to open a discussion so Im mentioning this here.
In the documentation for this project I found that the connect function is called as a middleware like so:
import { session } from 'next-session';

const handler = nextConnect()
  .use(session({ ...options }))

I found that when calling it that way, instead of saving what the function returns and using that, .on('connect') and .on('disconnect') event listeners start piling up every time the middleware function is called, leading to a maxListenersExceeded warning. This doesn't:
import { session } from 'next-session';

const mySession = session({ ...options });

const handler = nextConnect()
  .use(mySession)

Not sure if this is fine, Im interested in your feedback on this. If Im making a good point maybe it would be a good idea to update the ReadMe to avoid having to dig in the code to find out where the event listeners get added.
Thanks

This is definitely a really bad issue and should have been noted! Thanks for pointing it out.
Would you like to open a PR to update the README?

Yes dude Id love to. Please do correct my technical usage of the terms when I do as Im not sure of how rigorous I will be


Show image that introduces concept of splits/configs

also: fixes #1212

It looks like you've updated documentation related to model or dataset cards in this PR.
Some content is duplicated among the following files. Please make sure that everything stays consistent.

modelcard.md
docs/hub/model-cards.md
docs/hub/model-card-annotated.md
src/.../modelcard_template.md (huggingface_hub repo)
src/.../datasetcard_template.md (huggingface_hub repo)
src/.../repocard.py (huggingface_hub repo)
datasetcard.md
docs/hub/datasets-cards.md

ok bot

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

you could add a section with a screenshot of parquet-converter profile so it gives it "legitimacy"

done


How to calculate the palm size

The paper "MediaPipe Hands: On-device Real-time Hand Tracking" side the MSE normalized by palm size is 13.4%.
Can you tell me, how to calculate the palm size, and do you test the method using the error in mm?
Best wishes

Hi @ZYX-MLer,
Could you please elaborate your query with complete details and use case to calculate the palm size using the Handtracking solution. Thank you!


Unable to read ntuple with std::vector after doing 'hadd'

I ran into a strange case where I could not read a ntuple with std::vector after doing 'hadd'. I created a test ntuple to reproduce the issue. It contains one event and has several branches:
- v_int16 : std::vector<int16_t>
- v_int32 : std::vector<int32_t>
- v_int64 : std::vector<int64_t>
- v_uint16: std::vector<uint16_t>
- v_uint32: std::vector<uint32_t>
- v_uint64: std::vector<uint64_t>
- v_bool  : std::vector<bool>
- v_float : std::vector<float>
- v_double: std::vector<double>

In uproot 2.5.10, I tried the following and it worked:
>>> f = uproot.open('stlvector.root')
>>> print f.keys()
['ntupler;1']
>>> tree = f['ntupler/tree']
>>> print tree.keys()
['v_int16', 'v_int32', 'v_int64', 'v_uint16', 'v_uint32', 'v_uint64', 'v_bool', 'v_float', 'v_double']
>>> print tree.array('v_int16')
[[1 2 3]]
>>> print tree.array('v_int32')
[[1 2 3]]
>>> print tree.array('v_int64')
[[1 2 3]]
>>> print tree.array('v_uint16')
[[1 2 3]]
>>> print tree.array('v_uint32')
[[1 2 3]]
>>> print tree.array('v_uint64')
[[1 2 3]]
>>> print tree.array('v_bool')
[[False  True]]
>>> print tree.array('v_float')
[[ 999. -999.]]
>>> print tree.array('v_double')
[[ 999. -999.]]

However, if I use hadd [1], I got errors (like [2] for std::vector<int16>) for doing the same thing as above:
>>> f = uproot.open('stlvector_after_hadd.root')
>>> print f.keys()
['ntupler;1']
>>> tree = f['ntupler/tree']
>>> print tree.keys()
['v_int16', 'v_int32', 'v_int64', 'v_uint16', 'v_uint32', 'v_uint64', 'v_bool', 'v_float', 'v_double']
>>> print tree.array('v_int16')
ValueError: cannot interpret branch 'v_int16' as a Python type
>>> print tree.array('v_int32')
ValueError: cannot interpret branch 'v_int32' as a Python type
>>> print tree.array('v_int64')
ValueError: cannot interpret branch 'v_int64' as a Python type
>>> print tree.array('v_uint16')
ValueError: cannot interpret branch 'v_uint16' as a Python type
>>> print tree.array('v_uint32')
ValueError: cannot interpret branch 'v_uint32' as a Python type
>>> print tree.array('v_uint64')
ValueError: cannot interpret branch 'v_uint64' as a Python type
>>> print tree.array('v_bool')
ValueError: cannot interpret branch 'v_bool' as a Python type
>>> print tree.array('v_float')
ValueError: cannot interpret branch 'v_float' as a Python type
>>> print tree.array('v_double')
[[ 999. -999.], [ 999. -999.], [ 999. -999.]]

I also tried
>>> print uproot.interpret(tree['v_int16'])
None
>>> print tree['v_int16']._streamer
None

So it appears that the Streamer info is missing somehow after doing 'hadd'? It's strange... I attach the root files 'stlvector.root' and 'stlvector_after_hadd.root'. I'd appreciate it if you could look into this. Thank you!
Best regards,
Jia Fu
[1]
hadd -f stlvector_after_hadd.root  stlvector.root stlvector.root stlvector.root

[2]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/jiafu/muon_work/venv/lib/python2.7/site-packages/uproot/tree.py", line 347, in array
    return self.get(branch).array(interpretation=interpretation, entrystart=entrystart, entrystop=entrystop, cache=cache, basketcache=basketcache, keycache=keycache, executor=executor, blocking=blocking)
  File "/tmp/jiafu/muon_work/venv/lib/python2.7/site-packages/uproot/tree.py", line 933, in array
    interpretation = self._normalize_interpretation(interpretation)
  File "/tmp/jiafu/muon_work/venv/lib/python2.7/site-packages/uproot/tree.py", line 717, in _normalize_interpretation
    raise ValueError("cannot interpret branch {0} as a Python type".format(repr(self.name)))
ValueError: cannot interpret branch 'v_int16' as a Python type

[3]
# My setup
virtualenv venv
source venv/bin/activate
pip install --upgrade pip
pip install uproot

The root files (they are zipped because GitHub doesn't allow .root file type):
stlvector.root.zip
stlvector_after_hadd.root.zip

Thanks! I'll try to solve this before the end of the day tomorrow.

It's not urgent. But thank you in advance!

Indeed, hadd does drop all STL streamers except for vector<double> and vector<string> (which are built into ROOT, I believe— these are two that you never have to manually build dictionaries for, unlike the others). I'd call that a bug in hadd and maybe it should be raised on the ROOT forum.
Nevertheless, files with this issue exist and we need to be able to read them. We can see the lack of most STL streamers in PyROOT:
>>> import ROOT
>>> tfile = ROOT.TFile("stlvector_after_hadd.root")
>>> tfile.ShowStreamerInfo()
... # (no STL streamers except vector<double> and vector<string>; compare to the other file)

as well as in uproot:
>>> import uproot
>>> f = uproot.open("stlvector_after_hadd.root")
>>> f.showstreamers()
... # (same issue)

The difference is also apparent when you "show" the branch info (new, undocumented method):
>>> t = f["ntupler/tree"]
>>> t.show()
v_int16                    (no streamer)              None
v_int32                    (no streamer)              None
v_int64                    (no streamer)              None
v_uint16                   (no streamer)              None
v_uint32                   (no streamer)              None
v_uint64                   (no streamer)              None
v_bool                     (no streamer)              None
v_float                    (no streamer)              None
v_double                   TStreamerSTL               asstlvector(asdtype('>f8'))

Before this bug-fix, the algorithm for assigning interpretations for branches depended on having an associated streamer, and after hadd, these branches simply don't. (I don't know how to check that in ROOT as none of the method names look like they'll tell me the branch-streamer association.)
Somehow, however, ROOT is still able to interpret these branches:
>>> ttree = tfile.Get("ntupler/tree")     # (back in PyROOT now)
>>> for x in t:
...   print list(x.v_int16)
...
[1, 2, 3]
[1, 2, 3]
[1, 2, 3]

So I searched through 100% of the data available in the file associated with this branch, and the only type-indication that ROOT might be using is the fact that the TBranch.fClassName is "vector<short>". I wasn't willing to use this string at first because if a class is split among many branches, all of the branches would have the name of the top-level class name in this field, rather than the specific data they contain (e.g. they'd all say "Event" or something). To get your TTree to work, which is a requirement because it works in ROOT, I added interpretation rules
# desperation cases, usually because streamer info was dropped by hadd
elif branch.fClassName == b"vector<bool>":
    return asstlvector(asdtype("bool"))
elif branch.fClassName == b"vector<char>":
    return asstlvector(asdtype("i1"))
elif branch.fClassName == b"vector<unsigned char>":
    return asstlvector(asdtype("u1"))
elif branch.fClassName == b"vector<short>":
    return asstlvector(asdtype("i2"))
elif branch.fClassName == b"vector<unsigned short>":
    return asstlvector(asdtype("u2"))
elif branch.fClassName == b"vector<int>":
    return asstlvector(asdtype("i4"))
elif branch.fClassName == b"vector<unsigned int>":
    return asstlvector(asdtype("u4"))
elif branch.fClassName == b"vector<long>":
    return asstlvector(asdtype("i8"))
elif branch.fClassName == b"vector<unsigned long>":
    return asstlvector(asdtype("u8"))
elif branch.fClassName == b"vector<float>":
    return asstlvector(asdtype("f4"))
elif branch.fClassName == b"vector<double>":
    return asstlvector(asdtype("f8"))

but for the reason given above, they should only work if the branch is manually declared to have type vector<short>, etc., not if the branches are created by splitting a class that contains a vector<short>.
Ordinarily, I'd be against adding code that makes a codebase more complex while only stamping out an individual case, moreover, a case generated by another bug elsewhere. (I can't believe that behavior in hadd is intentional.) However, this one is okay. The interpretation rules are supposed to be an ever-growing list, borne out by examples of ROOT files found in the wild, and besides, they're only defaults that can be manually overridden. Moreover, the effect of this hack/fix is not hidden: it can be seen in tree.show():
>>> t.show()
v_int16                    (no streamer)              asstlvector(asdtype('>i2'))
v_int32                    (no streamer)              asstlvector(asdtype('>i4'))
v_int64                    (no streamer)              asstlvector(asdtype('>i8'))
v_uint16                   (no streamer)              asstlvector(asdtype('>u2'))
v_uint32                   (no streamer)              asstlvector(asdtype('>u4'))
v_uint64                   (no streamer)              asstlvector(asdtype('>u8'))
v_bool                     (no streamer)              asstlvector(asdtype('bool'))
v_float                    (no streamer)              asstlvector(asdtype('>f4'))
v_double                   TStreamerSTL               asstlvector(asdtype('>f8'))

They don't claim (wrongly) to have a streamer, but they nevertheless have an interpretation (right column) for reasons unrelated to streamers. It's something that I'll be able to ask users to print out to diagnose future bugs.

The fix is released in GitHub and PyPI (pip install) as 2.5.11.

Thanks for the detailed explanations and fixing it so quickly! :open_mouth:
I can confirm uproot 2.5.11 is able to read my analysis ntuple (after hadd) correctly. And it's reeaaally fast compared to rootpy, which is what I have been using.
It does seems like hadd is optimizing away the streamer info for other types of std::vector , because they are not needed by ROOT. In any case, I'm happy with your solution.

I'm glad it's working for you! Large performance improvements with respect to PyROOT or rootpy are expected, so that's good (I'd worry if the difference wasn't evident).
Labeling something as a bug or a feature is subjective, but I'd say this behavior in hadd is a bug. Since the streamer info is small compared to and doesn't grow with event data, dropping parts of this information is not really an optimization, especially since it must be ambiguous in the case of vectors in a split class (what I described above).
In fact, hadd has a difficult job: it has to merge two potentially different sets of streamer infos (no duplicates) and putting them into mutual dependency order. It looks like they got that wrong when STL collections are concerned.

Oh, and since you're working with STL vectors, consider installing Numba if you haven't already. When we read STL vectors, we have to touch each event individually (to remove a header), which is done in a Python for loop if you don't have Numba and in compiled code if you do. For large ntuples (where the time spent processing the ntuple exceeds the half second or so spent compiling the code), you should see an additional speedup.
But if it's fast enough and installing Numba is a bother, don't bother.

Ok, I'll make a bug report regarding hadd to the ROOT team and see what they would say. But I won't insist that they fix it.
Regarding Numba, what do I have to do to activate it? Do I simply put import numba at the beginning and that's it? I'm doing this on the LPC machine, and it seems Numba has already been installed.
>>> import numba
>>> print numba.__version__
0.33.0

My next question is off-topic. But I'm trying to port my rootpy script to uproot. I'm analyzing a list of hits, and each hit has its variables like endcap, station, ring. phi, theta, etc. The way I store them in my ntuple is the following:
- vh_endcap    : std::vector<short>
- vh_station   : std::vector<short>
- vh_ring      : std::vector<short>
- vh_sim_phi   : std::vector<float>
- vh_sim_theta : std::vector<float>
etc

In the example you gave at LPC about nested structures, you did the following to analyze a list of electrons:
    for event in events:
        mht_px = 0.0
        mht_py = 0.0
        mht_pz = 0.0
        for electron in event.Electron:
            mht_px += electron.pt * cosh(electron.eta) * sin(electron.phi)
            mht_py += electron.pt * cosh(electron.eta) * cos(electron.phi)
            mht_pz += electron.pt * sinh(electron.eta)

I would like to do the same, but unfortunately I'm using std::vector instead of TClonesArray, so I don't know how to "zip" and loop through all the hit variables together (unless I explicitly call zip() on all of my std::vector's).
What is the closest thing I can do? In rootpy, I use this nice function tree.define_collection which allows me to do:
tree.define_collection(name='hits', prefix='vh_', size='vh_size')

for evt in tree:
  for hit in evt.hits:
    print hit.endcap, hit.station, hit.ring, hit.sim_phi, hit.sim_theta

I would appreciate any suggestion, thanks!

Regarding Numba, there's no explicit activation. If import numba does not result in an ImportError, then uproot is already using it to accelerate the branch-reading. So that's contributing to the speedup you already see.
Actually, it's CMSSW that's bundled with Numba; it's not installed by default on CMSLPC.

The nested structures thing wasn't a difference between TClonesArray and vector. Both of these types of branches are interpreted as JaggedArrays, so they behave exactly the same way.
The nested structures talk was actually using two codebases together: uproot to read ROOT data and oamap (object-array mapping). Since then, I've rewritten uproot (so that you can use it with vector now) and I'm in the process of rewriting oamap as well. There isn't a combination of versions of the two packages that (a) interface with one another and (b) give you the vector that you want.
rootpy's define_collection function looks nice; I may implement something similar for uproot (but without modifying the tree object). For now, you'd have to create a big zip of them all.
However, that doesn't have to be a chore: it looks like you want to combine all the branches that start with "vh_". Okay, there's at least one difference between TClonesArray and vector: a collection of vector branches do not have an external size branch like "vh_size", since they can in principle have different sizes. In fact, you didn't have to make a TCloneArray to get branches with a common size branch: you can make them like
tree->Branch("vh_size", &vh_size, "vh_size/I");
tree->Branch("vh_endcap", &vh_endcap, "vh_endcap[vh_size]/F");
// etc...

But that's a digression. You could get a bunch of branch names starting with "vh_" with something like
hitnames = tree.allkeys(filtername=lambda x: x.startswith("vh_"))

And then extract only these arrays with
arrays = tree.arrays(hitnames, outputtype=tuple)

where the outputtype=tuple ensures that arrays is a tuple in the same order as hitnames. Then to zip over all of them, do
zip(*arrays)

which effectively makes each array in arrays an argument in zip. Like this:
>>> for event in zip(*arrays):
...   print "new event"
...   for name, data in zip(hitnames, event):
...     print name, data
...
new event
v_int16 [1 2 3]
v_int32 [1 2 3]
v_int64 [1 2 3]
v_uint16 [1 2 3]
v_uint32 [1 2 3]
v_uint64 [1 2 3]
v_bool [False  True]
v_float [ 999. -999.]
v_double [ 999. -999.]
new event
v_int16 [1 2 3]
v_int32 [1 2 3]
v_int64 [1 2 3]
v_uint16 [1 2 3]
v_uint32 [1 2 3]
v_uint64 [1 2 3]
v_bool [False  True]
v_float [ 999. -999.]
v_double [ 999. -999.]
new event
v_int16 [1 2 3]
v_int32 [1 2 3]
v_int64 [1 2 3]
v_uint16 [1 2 3]
v_uint32 [1 2 3]
v_uint64 [1 2 3]
v_bool [False  True]
v_float [ 999. -999.]
v_double [ 999. -999.]

In the future, let's move these discussions to the uproot users Google Group.

Later, when you need more speed (because you've got your process set up— all exploration done), you'll probably want to put this loop inside of a compiled Numba function, which might complain about the use of zip(*arrays). At that point, you'll have to switch it to longhand, but one-line Python commands can spit out the Python code you'd want to copy-paste into such a script.

Thanks a lot for answering all my questions. I'll use the Google Group in the future.
I thought the fact that you could loop over for electron in event.Electron and access electron.pt, electron.eta, etc was because of TClonesArray. But apparently it was something else (oamap). I think that would be a really nice feature to have.
Meanwhile, the snippet that you posted will do the job. That's much better than me writing a really long zip statement. So, thanks again for all your help!


need to allow users to specify source path

My cee.studio backend uses containers to mount source code to /home/jail/ to build executables.
When a user run the executable, the executables and source code are mount to  /home/usr/

This cause pbd cannot find the source files. The following is the error message.
pbd a.out main -s
PBD: Source code/theme file /home/jail/prog.c not found, please
check if the file exists in your system!

PBD still looks for the source path from which the executables are built.  Is it possible to allow users to truncate "/home/jail/" by specifying env TRUCATE=/home/jail/  so PBD can search source files from the current folder.

never mind, I fixed it by using the same path.


Revert "feature: add dist folder"

Reverts devlint/gridlex#60

What was the issue?

Hello @eljefedelrodeodeljefe
I'm ok for the /dist/ folder, but I not sure to change node script name.
Why a version script by the way?
I'll back next week with a resolution.
Thx for the PR

@eljefedelrodeodeljefe Sorry for the time and thx for the PR.
Note that from version 2.4.2 I got your change about dist folder.


Backend test suite

Currently whenever any changes are made to the backend code, manual testing is needed to validate it. That slows down development and makes it more likely that regressions will be introduced. There should be a test suite for the python backend to automatically validate changes.
Since Kolibri is a Django application, familiarity with Django testing is required. However, since Kolibri initializes Django, the Django testing infrastructure can't be used directly without adding a custom test runner. Furthermore, the Django testing tools are based on unittest, which I personally find to be unsatisfactory. Kolibri itself uses pytest-django, which bridges Django to pytest. Even then a bit of custom setup code is needed to get Kolibri and Django to initialize in a way that resembles Kolibri at runtime.
I worked on this a few weeks ago and have some minimal tests that can serve as a starting point.

#763 looked excellent, so I merged it. Are you intending for more tests to be written imminently as part of this issue (in which case I should move it back to In Progress), or is #763 it for now (in which case I should move the issue to Verify)?

Thanks! I'm going to try to get the collection downloader tests landed since I'm close and that we be a big win. If it seems like that's taking too long, I'll split that out to a separate task and close this one.

I was working on several other tasks and the collection downloader tests got pushed down. I opened #778 to hopefully fix in the next cycle.


Code

Hi! Congratulations on the paper. What are the timelines for releasing the code?
Thanks!

Hi,
Sorry for the delay. We will share it by Aug 15.
Thanks.

Thank you for your work and contribution to the community. Are you guys still planning on releasing the code?


overleaf link

I tried to clone the repo but it was over 300MB and not yet complete so I aborted it. I would like to setup the neurodata paper template for mgc - is there an overleaf link I can use instead?

https://git.overleaf.com/5045839tctygn
On Sun, Dec 18, 2016 at 8:12 PM, Greg Kiar<EMAIL_ADDRESS>wrote:

Assigned #172 https://github.com/neurodata/MGC/issues/172 to @jovo
https://github.com/jovo.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/neurodata/MGC/issues/172#event-898013924, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AACjco2MfsyA0PapXxav8M5xmSfEmYM0ks5rJdnogaJpZM4LQSut
.

--
the glass is all full:  half water, half air.
neurodata.io, jovo calendar
https://calendar.google.com/calendar/embed?src=joshuav%40gmail.com&ctz=America/New_York
starting in 2017, i will be responding to emails approximately weekly,
so that i can focus more on the people around me.
thank you for your understanding.

yah I did a quick weight-reduction, but the repo size is still 2 gb....
code is 0.5MB, data is 50MB, draft and others is 50 MB, history files takes up the other 1.9GB....
not sure what to do with the large repo size though....
regarding arxiv formatting, I tried to re-format a few days ago, but cannot achieve in an hour, because there seem to be many formatting conflicts in the new format and random other formats we use in old papers....

the overleaf repo is only 35 MB.
On Sun, Dec 18, 2016 at 8:46 PM, joshua vogelstein<EMAIL_ADDRESS>wrote:

https://git.overleaf.com/5045839tctygn
On Sun, Dec 18, 2016 at 8:12 PM, Greg Kiar<EMAIL_ADDRESS>wrote:

Assigned #172 https://github.com/neurodata/MGC/issues/172 to @jovo
https://github.com/jovo.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/neurodata/MGC/issues/172#event-898013924, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AACjco2MfsyA0PapXxav8M5xmSfEmYM0ks5rJdnogaJpZM4LQSut
.

--
the glass is all full:  half water, half air.
neurodata.io, jovo calendar
https://calendar.google.com/calendar/embed?src=joshuav%40gmail.com&ctz=America/New_York
starting in 2017, i will be responding to emails approximately weekly,
so that i can focus more on the people around me.
thank you for your understanding.

--
the glass is all full:  half water, half air.
neurodata.io, jovo calendar
https://calendar.google.com/calendar/embed?src=joshuav%40gmail.com&ctz=America/New_York
starting in 2017, i will be responding to emails approximately weekly,
so that i can focus more on the people around me.
thank you for your understanding.

@gkiar can you get everything but the font working?
On Sun, Dec 18, 2016 at 10:11 PM, joshua vogelstein<EMAIL_ADDRESS>wrote:

the overleaf repo is only 35 MB.
On Sun, Dec 18, 2016 at 8:46 PM, joshua vogelstein<EMAIL_ADDRESS>wrote:

https://git.overleaf.com/5045839tctygn
On Sun, Dec 18, 2016 at 8:12 PM, Greg Kiar<EMAIL_ADDRESS>wrote:

Assigned #172 https://github.com/neurodata/MGC/issues/172 to @jovo
https://github.com/jovo.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/neurodata/MGC/issues/172#event-898013924, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AACjco2MfsyA0PapXxav8M5xmSfEmYM0ks5rJdnogaJpZM4LQSut
.

--
the glass is all full:  half water, half air.
neurodata.io, jovo calendar
https://calendar.google.com/calendar/embed?src=joshuav%40gmail.com&ctz=America/New_York
starting in 2017, i will be responding to emails approximately weekly,
so that i can focus more on the people around me.
thank you for your understanding.

--
the glass is all full:  half water, half air.
neurodata.io, jovo calendar
https://calendar.google.com/calendar/embed?src=joshuav%40gmail.com&ctz=America/New_York
starting in 2017, i will be responding to emails approximately weekly,
so that i can focus more on the people around me.
thank you for your understanding.

--
the glass is all full:  half water, half air.
neurodata.io, jovo calendar
https://calendar.google.com/calendar/embed?src=joshuav%40gmail.com&ctz=America/New_York
starting in 2017, i will be responding to emails approximately weekly,
so that i can focus more on the people around me.
thank you for your understanding.

I have not tried the figure boxes as some of the figures+captions are very large and spill over page margins, but the footer logo and page number are setup, ya

@gkiar let me know when you replace the existing "preamble/style" with neurodata

i think the best way is to wait til the draft is finalized by jovo, then we can spend a while to fix all styles & upload to arxiv.
PS: although it works well on overleaf, I am having trouble compile the draft locally by XeLatex....probably because my MikTex needs update?

@cshen6 agreed, @gkiar and i will do it when we are done :)

@gkiar given the new fig 1, i think we are well situated to get the top to lines of mgc.tex to be "neurodata" this week.  let's put it in our thingy for the week?

@gkiar let's commit to getting this done this week.  shall we schedule a time? 4pm today (monday) works for me. if you too, put in gcal?

Yup, let's do that.
On Jan 30, 2017 00:12, "joshua vogelstein"<EMAIL_ADDRESS>wrote:

@gkiar https://github.com/gkiar let's commit to getting this done this
week. shall we schedule a time? 4pm today (monday) works for me. if you
too, put in gcal?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/neurodata/MGC/issues/172#issuecomment-275981545, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AEqDWHAFBL9OqY6WqRC_curA_D30-zXlks5rXXFCgaJpZM4LQSut
.