fix: Heap usage request throttling

fixes #1439 - Heap usage code now guarantees that new requests won't be scheduled until all previous requests are complete.
Testing
I tested by shutting my Mac for ~2 hours and returning. Timings looks something like this:
Summary

It seems that requests still happen occasionally while sleeping. On Wake up, there were a few ticks resulting in "Unable to get heap usage" console errors. Then a prompt showed up indicating credentials had expired at which point requests start succeeding again.

HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 0
useAsyncInterval.ts:37 [useAsyncInterval] tick #4. 19 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 10000
useAsyncInterval.ts:37 [useAsyncInterval] tick #5. 537820 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 0
useAsyncInterval.ts:37 [useAsyncInterval] tick #6. 16 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 10000
useAsyncInterval.ts:37 [useAsyncInterval] tick #7. 191633 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 0
useAsyncInterval.ts:37 [useAsyncInterval] tick #8. 18 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 10000
useAsyncInterval.ts:37 [useAsyncInterval] tick #9. 23720 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 0
useAsyncInterval.ts:37 [useAsyncInterval] tick #10. 18 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 10000
useAsyncInterval.ts:37 [useAsyncInterval] tick #11. 886367 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 0
useAsyncInterval.ts:37 [useAsyncInterval] tick #12. 17 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 10000
useAsyncInterval.ts:37 [useAsyncInterval] tick #13. 1010951 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 0
useAsyncInterval.ts:37 [useAsyncInterval] tick #14. 6 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 10000
useAsyncInterval.ts:37 [useAsyncInterval] tick #15. 650011 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 0
useAsyncInterval.ts:37 [useAsyncInterval] tick #16. 13 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 10000
useAsyncInterval.ts:37 [useAsyncInterval] tick #17. 731687 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 0
useAsyncInterval.ts:37 [useAsyncInterval] tick #18. 19 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 10000
useAsyncInterval.ts:37 [useAsyncInterval] tick #19. 120545 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 0
useAsyncInterval.ts:37 [useAsyncInterval] tick #20. 16 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 10000
useAsyncInterval.ts:37 [useAsyncInterval] tick #21. 446345 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 0
useAsyncInterval.ts:37 [useAsyncInterval] tick #22. 13 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 10000
useAsyncInterval.ts:37 [useAsyncInterval] tick #23. 10999 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 8987
useAsyncInterval.ts:37 [useAsyncInterval] tick #24. 9020 ms elapsed since last tick.
HeapUsage.tsx:68 [HeapUsage] Unable to get heap usage Error: Authentication details invalid
useAsyncInterval.ts:56 [useAsyncInterval] adjusted minIntervalMs: 10000
useAsyncInterval.ts:37 [useAsyncInterval] tick #25. 10030 ms elapsed since last tick.


[FEATURE] Add Translation Tool

I want to add a Translation Tool(all language) properly.
@SurajPratap10  PA please assign me this task.

already assigned to someone


Mirror settings -> interval: -1

Gitea version (or commit ref): Not relevant
Git version: Not relevant
Operating system: Not relevant
Database (use [x]):

[x] PostgreSQL
[x] MySQL
[x] MSSQL
[x] SQLite

Can you reproduce the bug at https://try.gitea.io:

[ ] Yes (provide example URL)
[ ] No
[x] Not relevant

Log gist: Not relevant

Description
Maybe I'm the only one needing this feature, I don't know. The mirror interval setting is taking a value in "h", "m", "s" (with 10 minutes as the minimal value on try.gitea.io which is not printed but anyway).
Imagine I want a repo to be a mirror but without automatic syncing. This way, when a click to the Sync Now button, the equivalent of a git pull would be done. However, I don't want the sync to be automatic (just for my case, the branch would be deleted for example).
So the idea is that the mirror interval setting would take -1 as a value and this would mean no automatic syncing. Is this a too specific and personal functionality or can it be useful to other people too ? (maybe I'm using the tools is a bad way ..)
Screenshots
Not relevant

Have you tried 0?

Already tried with 0, -1 etc .. Same result : Mirror interval is not valid.
True on try.gitea.io.


Trailing slash leads to IssuerUrlsDontMatch

Example:
The expected issuer is https://example.com, but the well-known/openid-configuration reports https://example.com/ with a trailing slash.
Currently, this leads to the Err(ProviderMetadataVerificationError::IssuerUrlsDontMatch) because of this exact string match: https://github.com/matrix-org/matrix-authentication-service/blob/3d90d0861ccf87d10e331a7fccd47c6290887bec/crates/oauth2-types/src/oidc.rs#L504-L506
Should the comparison ignore trailing slashes or is this a misconfiguration by the user?

This is a misconfiguration by the user IMO. We're intentionally being strict as the various specs (1, 2) want you to do strict simple string matches for the issuer.

As far as I understand, they want you to do simple string matching of the URL after the URL has been normalized:

This comparison MUST use simple string comparison as defined in Section 6.2.1 of [RFC3986].

Then if we look at Section 6.2.1:

False negatives are caused by the production and use of URI aliases.
Unnecessary aliases can be reduced, regardless of the comparison
method, by consistently providing URI references in an already-
normalized form (i.e., a form identical to what would be produced
after normalization is applied, as described below).

Finally we look at the ways a URI can be normalized in section 6.2.3:

In general, a URI that uses the generic syntax for authority with an
empty path should be normalized to a path of "/".

It even lists that these two are equivalent:

http://example.com
http://example.com/


fix(typescript-generator): remove useless path from import statements

Updates the TypeScript generator not to prepend the useless path of ./ in front of all the imports. This is causing linter errors when using mocks in live projects.

Hmm, I think that's the "correct" way to import a file that's in the same directory. Without the ./ I believe the module resolution algorithm will look in node_modules. Which ESLint rule is complaining?

Hmm, I think that's the "correct" way to import a file that's in the same directory. Without the ./ I believe the module resolution algorithm will look in node_modules. Which ESLint rule is complaining?

Useless path segments for "./../path-types/AccountTransactions.types.js", should be "../path-types/AccountTransactions.types.js"eslint[import/no-useless-path-segments](https://github.com/import-js/eslint-plugin-import/blob/v2.28.0/docs/rules/no-useless-path-segments.md)

Oh, I didn't know it was putting ./ in front of ../.

Good catch, but we need to make sure all relative paths that don't start with ../ still have ./.

Ok, I'll get it updated.

I think the simplest fix be to .replaceAll("././", "./").

Hmm, I just saw this after I'd pushed my proposed fix. Do you want to review what I have first and if you'd prefer the replaceAll logic I can do that instead?

I don't see any new pushes?

messed up PR


issue sending emails

I am having an issue sending the emails.  I am using the default.cfg.  Should I change some of the default values for the SMTP server?

short answer is YES/Maybe.
First, are you specifying the "-s" option when you run SPF?  The "-s" option for to tell SPF to send emails.  Additionally, have you allowed SPF to start any webservices/websites?  In its current configuration, SPF requires there to be a web site started before it will send the emails.  This can be accomplished by using the "-w" flag.
Now, if you have done those things and still no luck, try this...
if you take a look at default.cfg, you will see the following section:
[SMTP]
DETERMINE_SMTP: 1
USE_SPECIFIC_SMTP: 0
SMTP_SERVER: <IP_ADDRESS>
SMTP_USER: XXXX
SMTP_PASS: XXXX
SMTP_FROMADDR<EMAIL_ADDRESS>SMTP_DISPLAYNAME: SUPPORT
SMTP_PORT: 25
Now you have to decide how you want SPF to send the emails.  It is configured by default to attempt to connect to and send the emails directly to the target's SMTP server.  This will usually work, but not in every case.  You can specify a SMTP server to use if you wish..  something like "smtp.com".  All you have to do is change "DETERMINE_SMTP" to 0 and set "USE_SPECIFIC_SMTP" to 1 then update "SMTP_SERVER", "SMTP_USER", and "SMTP_PASS" as needed.
Let me know if none of this works for you.

I ended up having to edit the code in emails.py to get it to work even after trying to set the variables in the default.config.
One more thing that would be a nice enhancement is to further obfuscate the text of the hyperlink that is displayed to the end user in the email message vs what the link is in the a href tag.  Is there a way to accomplish that - and if not easy to do what code module should I look to change so I can alter [TARGET]?

From: Adam Compton<EMAIL_ADDRESS>Sent: Thursday, October 26, 2017 9:40:44 AM
To: tatanus/SPF
Cc: John Dickerson; Author
Subject: Re: [tatanus/SPF] issue sending emails (#23)
short answer is YES/Maybe.
First, are you specifying the "-s" option when you run SPF? The "-s" option for to tell SPF to send emails. Additionally, have you allowed SPF to start any webservices/websites? In its current configuration, SPF requires there to be a web site started before it will send the emails. This can be accomplished by using the "-w" flag.
Now, if you have done those things and still no luck, try this...
if you take a look at default.cfg, you will see the following section:
[SMTP]
DETERMINE_SMTP: 1
USE_SPECIFIC_SMTP: 0
SMTP_SERVER: <IP_ADDRESS>
SMTP_USER: XXXX
SMTP_PASS: XXXX
SMTP_FROMADDR: support<EMAIL_ADDRESS>SMTP_DISPLAYNAME: SUPPORT
SMTP_PORT: 25
Now you have to decide how you want SPF to send the emails. It is configured by default to attempt to connect to and send the emails directly to the target's SMTP server. This will usually work, but not in every case. You can specify a SMTP server to use if you wish.. something like "smtp.com". All you have to do is change "DETERMINE_SMTP" to 0 and set "USE_SPECIFIC_SMTP" to 1 then update "SMTP_SERVER", "SMTP_USER", and "SMTP_PASS" as needed.
Let me know if none of this works for you.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/tatanus/SPF/issues/23#issuecomment-339669323, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Afk4DkvQeSRVzSnE4mwCHV5fHoKs5O4eks5swIvcgaJpZM4QG8If.

can you tell me what you changed in emails.py to get it to work for you?  possibly a PULL REQUEST?

I do not think the link can be changed currently via any standard mechanism.  If you want to change it in the code, it would be somewhat difficult to do it in a way that is still dynamic and universal.  But if you wish to take a look, it would in around line 885 and 893 of spf/core/framework.py

Item 1 - I had to delete all of the SMTP checks prior to the section that does the sending of the email.
Item 2 - I will take a look at the code you suggested to see if there's anything that can be done...
Thanks for the quick feedback!!!

What do you mean, "...had to delete all of the SMTP checks prior...".  what checks and and in which section.  There are 3 different functions that can send emails.  which of the three was this in?

I believe in emails.py there appear to be tests near the top of the code that try to detect the remote email server (depending on the settings in the default.config) and then there is code to try to use the email server you specify in the default.config file.  For some reason, I was not getting past the remote server test, so I got rid of that code first.  Then I also deleted the section that pulls the smtp server settings from default.config.  I then hardcoded the smtp settings in the code that remained and then it worked as expected.  I'm not sure the reason I had to delete the section that pulls the smtp settings from the default.config file, I am thinking it might had to do with the fact I was using an IP Address for the SMTP server rather than a fully qualified domain name.  Again, I'm not sure what the issue was, I just know removing those 2 sections and hardcoding the SMTP settings directly in emails.py allowed the emails to be sent successfully...
I haven't had a chance to look at the [TARGET] code you pointed me to.
Hope this helps.

thank you for your response.  I will look into this and see if I can see a way to accommodate situations like your and see if there is a way to clean it all up.


Fix/skip sent 2 vec test

This is a workaround https://github.com/wellcometrust/WellcomeML/issues/89 to enable our pipelines to pass until we fix the speed of the tests.
It also adds profiles to test so we can partially address https://github.com/wellcometrust/WellcomeML/issues/89.

Codecov Report

Merging #92 into master will decrease coverage by 0.32%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master      #92      +/-   ##
==========================================
- Coverage   76.35%   76.03%   -0.33%
==========================================
  Files          37       37
  Lines        1552     1552
==========================================
- Hits         1185     1180       -5
- Misses        367      372       +5

Impacted Files
Coverage Δ

wellcomeml/ml/sent2vec_vectorizer.py
55.00% <0.00%> (-25.00%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9770d06...ba58456. Read the comment docs.

@nsorros are you ok with this temporary workaround? It will decrease overall coverage by just 0.32%. We can still run the skipped test locally.


[Home] Issues based on review

Reviewing: https://developer-stage.adobe.com/client-sdk/documentation/

[ ] Missing subtitle below page heading Welcome to the home of Adobe Experience Platform Mobile SDK documentation.

[ ] Good to go for iOS 15 section

Heading and content needs to be changed for iOS 16 and Android 13.
Link for current SDK versions page is wrong

[ ] Project Griffon will be Assurance box is missing

[ ] Need to remove following boxes

The Swift versions of the iOS SDKs have been released!
Support for Adobe Experience Platform Edge Network

[ ] Are you interested in Kotlin? section

Heading and content needs to be changed. Check with Chetan if we still need it as we are already doing a refactor in Kotlin.
Needs to be moved after `Supported Platforms section.
following survey link opens a 404 form. Need to check with Chetan for the updated form link.

[ ] Bring the Adobe Experience Cloud to your apps section - Link for variety is incorrect, should point to Supported Platforms section in same page

[ ] Embedded video needs to be removed (maybe it was removed because it was outdated?)

[ ] Supported platforms section

Missing some entry for React Native AEP 1.x.
Links and texts are incorrect, should point to corresponding section in current SDK versions page.

[ ] Resources section

Should be places above Get Help section
Should be renamed as Quick links

[ ] Get help

Needs to be the last section on the page
Needs to renamed to Further help
Text need to be changed

These are being tracked in #115!
I'll close this issue once that PR is merged in.


ERROR - Expecting value: line 1 column 1 (char 0)

Today I got an error like this, even though the user is live.

any help?

I think my IP has been blocked or something, I tried using another internet provider, worked fine.

yeah i get that issuee too, i think you can raise wait to 5 minute or 300 sec to avoid ip blacklist, you can edit wait time in enum.py

yeah i get that issuee too, i think you can raise wait to 5 minute or 300 sec to avoid ip blacklist, you can edit wait time in enum.py

did this work for you? I tried it and saw no diference

yeah this is work for me, i didn't get ip blacklist anymore, for 48h running, i didn't get blacklist

Should the default long wait time be increased to 5 minutes instead of 2 minutes? I have not encountered this error before. Maybe it only affects users from certain countries/locations?


docUpdate index.html

Issue:

Hello @shubh907 if you would like to contribute please contribute with something useful. Check out the issues section to see how you can contribute


Prod mode Error: Cannot match any routes. URL Segment: 'connect/authorize'

First off, I apologize if I'm in the wrong place, so thanks for your patience.
I'm using .NET Core 2.2 with Identity Server4 with Angular 7 (CLI) and v5.0.2 of this library and everything works great in development mode. I hit a login button, I get redirected to a static page on the same server (the angular SPA and the ID server are currently one and the same), I log in, and the site redirects me back to my SPA and angular takes over and everything is good 👍 .
When I build in production mode, though (and I'm not sure where the disconnect is, which is why I'm posting), when I go to login, I get the following error in the console and my MVC controller for the login page is never hit (it should be):
ERROR Error: Uncaught (in promise): Error: Cannot match any routes. URL Segment: 'connect/authorize'
Error: Cannot match any routes. URL Segment: 'connect/authorize'

I receive this error because I do not currently have a fallback route setup. If I setup a fall back route, then the error goes away but I'm also never actually directed to the login page; the app just reloads itself and does not navigate to the static site page.
The last piece of code that I see the app hitting before refreshing is this.oauthService.initImplicitFlow();, then the app reloads with something like this in the request URI: connect/authorize?response_type=id_token%20token&client_id=spaweb&state=pX1XiyN03w2yic3yGDDkMdnGQyfXJivUh9EwOY5G&redirect_uri=https%3A%2F%2Flocalhost%3A44350%2Fhome&scope=spaweb%20openid%20profile&nonce=pX1XiyN03w2yic3yGDDkMdnGQyfXJivUh9EwOY5G, and app.component starts doing its normal loading process. It appears that Angular is refusing to let the server handle the request at all, but I'm at a loss as to how that can happen.
Any thoughts on what could be going on here that I'm missing? I'd greatly appreciate any pointers you could give!

With the current info, I'd guess that it's not a bug/issue with the library, but a problem with your specific setup. So I suggest asking a question on Stack Overflow, providing steps for a minimal repro to take out the guesswork (which we'd also need to reliably help here if it does turn out to be an issue with the library).

Thanks for the followup, I think you're right. I finally managed to find this issue on the angular CLI that looks to be what is going on for me: https://github.com/angular/angular-cli/issues/6757. I'll continue my troubleshooting there. Thanks!

I faced the same issue I changed the routing to hash, and tried all the possibles solutions on https://github.com/angular/angular-cli/issues/6757, in my case I have the routes on a separated file called routes, and in the router module I did the import and some clean. In dev mode all works fine but the issue comes in prod mode, I think it has to be for the aot compilation it compiles the routes before the import of the routes so the main router has no routes, I solver putting all my routes inside the router module and it works for dev and prod builds.

I'm creating a new issue

I am also having this issue using dotnet core 5 with angular 10. the routes works fine in development but not in production :(

Hi, have you solved ur issue? I have tested both development and production mode, it is working fine. Once I publish and deploy it to IIS with the folder name "WorldCities2" then modify the base href from / to /WorldCities2/. The error "ERROR Error: Uncaught (in promise): Error: Cannot match any routes. URL Segment: 'worldcities2/Home'" occur.


intr: Store eflags

If I'm reading the intel manual (iret starts at page 603) correctly it looks like iret already restores eflags. Other sources suggest the same. Am I mistaken?

sounds reasonable, will undo. thanks for the tip :+1:


add logging facade

Purpose
Fix #181
Change
Add the logger functionality to the domain module through a logger facade

 Fix issue #181 (Add logger facade to enable configurable logging in business domain classes)

 Fix issue #181 (Add logger facade to enable configurable logging in business domain classes)

Thanks for the suggestions, I like the way you implemented the logger 👍🏼
Still, I think you need to approve since I created this PR and cannot approve it myself.

Thanks for the suggestions, I like the way you implemented the logger 👍🏼
Still, I think you need to approve since I created this PR and cannot approve it myself.


Combining regular numba.jit / njit with CUDA array interface?

Feature request
It is currently a bit unclear if compiling to a non-CUDA target, i.e., using the regular @numba.jit or @numba.njit decorator is supposed to work with the CUDA array interface. A minimal example would look like.
import numba
import numba.cuda
import torch

@numba.njit
def numba_test_func(tensor):
    # Can we work with CUDA tensors from CPU?
    ...

# Create some tensor on GPU
device = torch.device("cuda:0")
tensor = torch.tensor([1.0, 2.0, 3.0], device=device)

# Convert it to the underlying CUDA array interface, and try to pass to numba function:
tensor_view = numba.cuda.as_cuda_array(tensor.detach())
numba_test_func(tensor_view)

This currently fails at runtime with:
TypeError: can't unbox array from PyObject into native value.  The object maybe of a different type

I not really sure if this is expected to work, or whether the CUDA array interface is only supposed to work with @numba.cuda.jit.
In the pytorch context it would be convenient if this was possible, because pytorch in general tries to be agnostic whether a tensor resided on CPU or GPU. It would be nice to just give some Python code a boost without having to write two numba jit versions, one for CPU and a true CUDA kernel for GPU. I came across this issue when trying to answer this StackOverflow question, which also demonstrates a suggested work-around if this wasn't possible.
If this is not supposed to work, it would be great if this could be clarified in the docs, perhaps also with a slightly more explicit error message at runtime.

@bluenote10 As far as I know unboxing of a PyTorch tensor into a Numba array type should not be possible. The Numba array type is specifically tied to numpy.ndarray and the error you are getting is basically telling you that PyTorch tensor has a different type and Numba does not know how to convert from a tensor to an array.
Does you example work if you change the tensor to be defined on a cpu device?
I will also like to point you to a new numba extension numba-dpex that my team in Intel is developing with guidance from the core Numba developers. One of our goals is to support the exact use case you are describing.
Numba-dpex extends numba.njit to compile dpnp. Dpnp is a proto-drop-in NumPy replacement based on Intel oneAPI and the SYCL language. Dpnp supports multiple types of devices that for now are restricted to Intel CPUs, Intel discrete GPUs, Intel integrated GPUs. In future, CUDA and AMDGPU devices may be supported as well once the support for these types of devices becomes more mature in oneAPI (reference: ongoing work to add CUDA and AMDGPU support to oneAPI).
Dpnp already supports the dlpack protocol and it is possible to "zero-copy" (i.e., take ownership of the underlying memory without copying data) a torch.tensor into a dpnp.ndarray.
Thus, to solve your example first the torch.tensor will have to be converted to a dpnp.ndarray using dlpack and then passed to a numba-dpex's JIT decorated function. Numba-dpex will then automatically detect the device of the dpnp.ndarray and compile the operations on the dpnp.ndarray to device-specific code. The approach will work not just for dpnp ufuncs, but even prange loops.
Do note that numba-dpex and dpnp are still pre-beta and lots of features are missing, but both projects are under heavy development and rapidly gaining features. If you are interested, feel free to come and discuss with us on the numba-dpex repo.

@gmarkall I'm still rather confused how it is supposed to work in general. In the linked example the @njit function is basically a "scalar only" function. However, if the @njit function also needs to operate on full arrays, just calling the @njit function for the CUDA kernel doesn't seem to work. For instance, the following will crash:
import numba.cuda
import numpy as np

@numba.njit
def f_cpu(array):
    array[:, 1] += 1.0

@numba.cuda.jit
def f_gpu(array):
    f_cpu(array)

if __name__ == "__main__":
    array_cpu = np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float32)
    array_gpu = numba.cuda.to_device(array_cpu)

    print("\nRunning on CPU:")
    f_cpu(array_cpu)
    print(array_cpu)

    print("\nRunning on GPU:")
    f_gpu[1, 1](array_gpu)  # type: ignore
    print(array_gpu)

I've raised a separate issue https://github.com/numba/numba/issues/9324 to clarify if this is just a bug.


Add documentation/repos on docker images

The latest docker images built by brightway-lca deserve a place in the documentation.

I suggest a sub-page of the Setup page.

@tngTUDOR gimme gimme gimme the docs for the docker 🎶🎉

issue-65 branch will include fix for this

There will only be "documentation" added, not the full repo(s) of the docker images for the moment.

@michaelweinold  please wait to merge until I figure out if we a "autobuild" API like documentation for the docker images is useful / necessary

Sure, I'll wait for your go-ahead. Please just check with me first if you end up deciding you want to autobuild (we are currently using sphinx-autobuild=2021.3.14) as you likely already know.

You can go ahead with the merge @michaelweinold


FreshIOC.Container.Register() no constructor takes 1 argument

In the newest version did this change? I had that line in previous versions without any complaints but now it won't compile.
I have switched it to FreshIOC.Container.Register(typeof(HttpClient)); but not sure that is the proper way.

Yeah, I believe it's now Register<InterfaceClass,ImplementationClass>(). So this should probably be how you want to do it now:
FreshIOC.Container.Register<HttpClient, HttpClient>();


Add a isVariableSubstitutedAsLabels prop

Ref: https://github.com/bigbinary/neeto-form-web/issues/8982
https://abhay-ashokan.neetorecord.com/watch/875c5140-ba76-4172-8fec-19ce971b1506

[x] Add a isVariableSubstitutedAsLabels prop (false by default). If this variable is enabled, the variables should be rendered using labels instead of ids. This can be implemented by writing a NodeView to render the Editor content.
[x] In case of longer labels/questions, we can truncate it to 15 characters and then suffix it with ...

@deepakjosp _a
enhancement _t

@AbhayVAshokan I had a call with @subins2000 and they are okay with showing the variable's label in NeetoCal. Hence I'm not adding a variable to control whether the Editor should display a label or id for the variables.


Support non exception errors from fetcher

Non-working PR to show the proposed way of adding support for allowing the fetcher to communicate errors not via exceptions.
The main complication here is in the builder API. we already have 2 builders because we support a builder with and without source of truth (to change to generic type). If we went this way we would either need to support 4 builder or we will need to change the builder API to force providing the source of truth (and the new translator) in the builder's constructor

Sorry this was open for so long. finally got around to finish it. Note that this would be a major API change for how we build store (but no change to Store's API itself).

the tl;dr of the api change is that you have to pass a source of truth when you create a builder rather than later:
        StoreBuilder
            .fromNonFlow<Pair<String, RedditConfig>, List<Post>, List<Post>>(
                fetcher = { (query, config) ->
                    provideRetrofit().fetchSubreddit(query, config.limit)
                        .data.children.map(::toPosts)
                },
                sourceOfTruth = SourceOfTruth.from(
                    reader = { (query, _) -> db.postDao().loadPosts(query) },
                    writer = { (query, _), posts -> db.postDao().insertPosts(query, posts) },
                    delete = { (query, _) -> db.postDao().clearFeedBySubredditName(query) },
                    deleteAll = db.postDao()::clearAllFeeds
                )
            )
            .build()

(still need to add a couple of tests here, will add soon hopefully)

@yigit @digitalbuddha this is now ready for review - it looks bigger than it is because of the changes to the builder's interface (and exposing sourceOfTruth through the interface)

Sorry i didn't have time to check this, planning to do it now.
i did a quick through and i have a question.
Does it make sense to also make the fetcher a class instead of a function?
That could help w/ two things:
a) we can have future changes there w/o going into this builder API issues
b) different ways to create a Fetcher would be isolated in the Fetcher itself, instead of leaking into the store builder API.
I just wanted to ask in case you've already evaluated that option.

created https://github.com/dropbox/Store/pull/139 to make discussions easier. unlike this one, it is not a complete PR.

Sorry for dropping off here. It's been hard to find OSS time in the last few weeks with the shelter in place and all. #139 seems good to me. I put a comment around the fetcher's API.
I'll try to integrate those changes into this revision over the next week of so, but feel free to finish up #139 you feel like it and if I'm taking too long.

@yigit Thanks for all the great feedback. This should be ready to go now.

reverting #159 for now because it makes it impossible to update both the source and the sample in the same diff. Let's discuss bringing it back in an issue. there are pros and cons for it, but for now, while the API is unstable it's just to block us

Just tagging @digitalbuddha in this, since he drove the change.

reverting #159 for now because it makes it impossible to update both the source and the sample in the same diff. Let's discuss bringing it back in an issue. there are pros and cons for it, but for now, while the API is unstable it's just to block us. FYI @DYJParker

opened #161 to continue the discussion over #159


Keep app state in url query

for example, left panel closed => ?leftClosed=1
what we want to keep:

panel state
tab state, which is active
search filter

So this is handled at the router level, right? We could have a queryParam model that uses getters from the other models to build itself up and pass that as a query argument every time we navigate. Something like that.


data structures, or what events can happen?

this is really great @agentlewis, thanks for starting this.
i'm keen to start a discussion about data structures, or what events happen in the system.
obviously i'm keen to for us to leverage the Value Flows vocabulary, since it's already based around events and i think nanox is a great example of a "cash flow" app.
some references:

classes and properties :star2:
UML diagram
OWL diagram

to get started i'm interested in focusing on only accounting for what just happened. so no plans, projections, events we know are recurring, etc.
or in Value Flows terms:

EconomicEvent
action
Action

so, what are some examples of incoming (https://github.com/enspiral-root-systems/nanox/issues/1) and outgoing (https://github.com/enspiral-root-systems/nanox/issues/2) events do we want to track? then i'd be keen to model them in Value Flows.

first example, being paid!
type: event
provider: client
receiver: Root Systems
affectedQuantity:
  value: 100
  unit: NZD
actions:
  receive:
    affects: RS bank account
  provide:
    affects: client's bank account

i used this example for help.
or maybe
type: event
quantity:
  value: 100
  unit: NZD
actions:
  receive:
    agent: Root Systems
    affects: RS bank account
  provide:
    agent: client
    affects: client's bank account

tracking hours!
type: event
resource: client's project
work:
  - agent: Alice
    quantity:
      value: 15
      unit: hour
  - agent: Bob
    quantity:
      value: 25
      unit: hour
  - agent: Carol
    quantity:
      value: 10
      unit: hour

other example from back in the day: here.

invoicing clients!
version: 1
id: 1
agent: Team
type: plan
action:
  - agent: Team
    type: consume
    resource: Team time
    quantity: -50
    unit: hour
  - agent: Client
    type: produce
    resource: Client time
    quantity: 50
    unit: hour
 - agent: Client
    type: give
    resource: Client bank account
    quantity: -100
    unit: NZD
  - agent: Team
    type: take
    resource: Team bank account
    quantity: 100
    unit: NZD

confirmation:
version: 1
id: 2
agent: Client
type: plan
plan: 1

or more negotiation:
version: 1
id: 3
agent: Client
type: plan
action:
  - agent: Team
    type: consume
    resource: Team time
    quantity: -25
    unit: hour
  - agent: Client
    type: produce
    resource: Client time
    quantity: 25
    unit: hour
 - agent: Client
    type: give
    resource: Client bank account
    quantity: -50
    unit: NZD
  - agent: Team
    type: take
    resource: Team bank account
    quantity: 50
    unit: NZD

then confirmation:
version: 1
id: 4
agent: Team
type: plan
plan: 3


definition of closed monoidal category is mistaken

I found the definition is weird and realized that the adjoint functors need to be related by mates, so that the adjoint functors are actual "natural".

this is fixed by https://github.com/agda/agda-categories/commit/1fd80457f764209c59886399fa14a14185d1afe0


fix: standardizing the TypeScript name in Portuguese documentation

Standardizing the TypeScript name in Portuguese documentations.

It seems we're having a file misplaced here, I'll take a look

@yuriwithowsky can you please rebase the master branch onto yours and re-push it so we can run the checks again?

@yuriwithowsky can you please rebase the master branch onto yours and re-push it so we can run the checks again?

Done!

broken again 😞

Hrm, I don't have merge rights anymore - might be worth giving it another try as it seems to be pretty arbitrary https://github.com/OSS-Docs-Tools/code-owner-self-merge/issues/35

Ok!
🤞
LGTM


limits file not found

Hi,
When running the "AI Benchmark" form https://ai-benchmark.com/ranking_deeplearning.html for the first time, I got a crash with the following info:
MIOpen(HIP): Error [Compile] 'hiprtcCompileProgram(prog.get(), c_options.size(), c_options.data())' naive_conv.cpp: HIPRTC_ERROR_COMPILATION (6)
MIOpen(HIP): Error [BuildHip] HIPRTC status = HIPRTC_ERROR_COMPILATION (6), source file: naive_conv.cpp
MIOpen(HIP): Warning [BuildHip] /tmp/comgr-e64065/input/naive_conv.cpp:39:10: fatal error: 'limits' file not found
#include <limits> // std::numeric_limits
         ^~~~~~~~
1 error generated when compiling for gfx908.
terminate called after throwing an instance of 'miopen::Exception'
  what():  /long_pathname_so_that_rpms_can_package_the_debug_info/data/driver/MLOpen/src/hipoc/hipoc_program.cpp:304: Code object build failed. Source: naive_conv.cpp
Aborted (core dumped)

It seems like I resolved it by installing the libstdc++-12-dev on ubuntu 22.04.2 LTS .
I have rocm 5.4.3 installed through packages.
I guess one of your packages should declare that one or the right one as dependency.
Best regards,
Epliz

@Epliz Thanks for reporting. It seems like you do not have the binary kernel cache package installed.
This is HIPRTC-specific issue. naive_conv.cpp needs to be fixed like this:
https://github.com/ROCmSoftwarePlatform/MIOpen/blob/d3ee8a87fa6e7b0a9db1b41f1d7c11acb529786a/src/kernels/static_composable_kernel/include/utility/static_kernel_reduction_operator.hpp#L29-L31
More details can be found at #627.
[Attribution] @junliume @johnny-keker https://github.com/ROCmSoftwarePlatform/MIOpen/labels/bug https://github.com/ROCmSoftwarePlatform/MIOpen/labels/urgency_normal, Proposed assignee: @carlushuang

@Epliz @atamazov this is a duplicated issue of https://github.com/ROCmSoftwarePlatform/MIOpen/issues/1921 at least with similar root cause

thank you for looking into this.
Regarding:

It seems like you do not have the binary kernel cache package installed.

I have miopenkernels-gfx908-120kdb installed as mentioned at https://docs.amd.com/bundle/MIOpen_gh-pages/page/install.html . Anything else I should make sure I have installed?
Any environment variable that I should make sure is set?

@Epliz Do you use MI100 with 120 compute units?

@junliume

@Epliz @atamazov this is a duplicated issue of #1921 at least with similar root cause

Well, this is actually related to #1926.

@Epliz Do you use MI100 with 120 compute units?

Yes, as shown by the rocminfo output:
$ rocminfo
ROCk module is loaded
=====================
HSA System Attributes
=====================
Runtime Version:         1.1
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:<PHONE_NUMBER>3709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE
System Endianness:       LITTLE

==========
HSA Agents
==========
*******
Agent 1
*******
  Name:                    AMD Ryzen 7 1700 Eight-Core Processor
  Uuid:                    CPU-XX
  Marketing Name:          AMD Ryzen 7 1700 Eight-Core Processor
  Vendor Name:             CPU
  Feature:                 None specified
  Profile:                 FULL_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        0(0x0)
  Queue Min Size:          0(0x0)
  Queue Max Size:          0(0x0)
  Queue Type:              MULTI
  Node:                    0
  Device Type:             CPU
  Cache Info:
    L1:                      32768(0x8000) KB
  Chip ID:                 0(0x0)
  ASIC Revision:           0(0x0)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   3000
  BDFID:                   0
  Internal Node ID:        0
  Compute Unit:            16
  SIMDs per CU:            0
  Shader Engines:          0
  Shader Arrs. per Eng.:   0
  WatchPts on Addr. Ranges:1
  Features:                None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    32792528(0x1f45fd0) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 2
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    32792528(0x1f45fd0) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 3
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    32792528(0x1f45fd0) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
  ISA Info:
*******
Agent 2
*******
  Name:                    gfx908
  Uuid:                    GPU-6e83bfb5727a9272
  Marketing Name:          AMD Instinct MI100
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          64(0x40)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    1
  Device Type:             GPU
  Cache Info:
    L1:                      16(0x10) KB
    L2:                      8192(0x2000) KB
  Chip ID:                 29580(0x738c)
  ASIC Revision:           2(0x2)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   1502
  BDFID:                   3840
  Internal Node ID:        1
  Compute Unit:            120
  SIMDs per CU:            4
  Shader Engines:          8
  Shader Arrs. per Eng.:   1
  WatchPts on Addr. Ranges:4
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      TRUE
  Wavefront Size:          64(0x40)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Max Waves Per CU:        40(0x28)
  Max Work-item Per CU:    2560(0xa00)
  Grid Max Size:<PHONE_NUMBER>(0xffffffff)
  Grid Max Size per Dimension:
    x<PHONE_NUMBER>(0xffffffff)
    y<PHONE_NUMBER>(0xffffffff)
    z<PHONE_NUMBER>(0xffffffff)
  Max fbarriers/Workgrp:   32
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    33538048(0x1ffc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 2
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Alignment:         0KB
      Accessible by all:       FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx908:sramecc+:xnack-
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:<PHONE_NUMBER>(0xffffffff)
      Grid Max Size per Dimension:
        x<PHONE_NUMBER>(0xffffffff)
        y<PHONE_NUMBER>(0xffffffff)
        z<PHONE_NUMBER>(0xffffffff)
      FBarrier Max Size:       32
*******
Agent 3
*******
  Name:                    gfx1031
  Uuid:                    GPU-XX
  Marketing Name:          AMD Radeon RX 6700 XT
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          64(0x40)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    2
  Device Type:             GPU
  Cache Info:
    L1:                      16(0x10) KB
    L2:                      3072(0xc00) KB
    L3:                      98304(0x18000) KB
  Chip ID:                 29663(0x73df)
  ASIC Revision:           0(0x0)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   2855
  BDFID:                   4608
  Internal Node ID:        2
  Compute Unit:            40
  SIMDs per CU:            2
  Shader Engines:          4
  Shader Arrs. per Eng.:   2
  WatchPts on Addr. Ranges:4
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      TRUE
  Wavefront Size:          32(0x20)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Max Waves Per CU:        32(0x20)
  Max Work-item Per CU:    1024(0x400)
  Grid Max Size:<PHONE_NUMBER>(0xffffffff)
  Grid Max Size per Dimension:
    x<PHONE_NUMBER>(0xffffffff)
    y<PHONE_NUMBER>(0xffffffff)
    z<PHONE_NUMBER>(0xffffffff)
  Max fbarriers/Workgrp:   32
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    12566528(0xbfc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 2
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Alignment:         0KB
      Accessible by all:       FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx1031
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:<PHONE_NUMBER>(0xffffffff)
      Grid Max Size per Dimension:
        x<PHONE_NUMBER>(0xffffffff)
        y<PHONE_NUMBER>(0xffffffff)
        z<PHONE_NUMBER>(0xffffffff)
      FBarrier Max Size:       32
*** Done ***

I typically disable the RX 6700XT that I use for display output whenever using tensorflow by setting ROCR_VISIBLE_DEVICES accordingly.

@Epliz

I typically disable the RX 6700XT that I use for display output whenever using tensorflow by setting ROCR_VISIBLE_DEVICES accordingly.

This is good, because MIOpen currently requires all GPUs to be identical.

@Epliz

I have miopenkernels-gfx908-120kdb installed as mentioned at https://docs.amd.com/bundle/MIOpen_gh-pages/page/install.html . Anything else I should make sure I have installed? Any environment variable that I should make sure is set?

No, you seem to have done everything right. But the library should read the naive kernel from the precompiled binary cache because you have it installed. I am wondering why it even tries to build it. Can you attach a text file with log captured with export MIOPEN_LOG_LEVEL=6?
CC @JehandadKhan

I can't reproduce anymore even after installing the libstdc++12 package, so I think it was most likely a mistake on my side. I probably had not yet installed properly the kernels.
I don't think it is worth wasting your time on this.
If you are interested in seeing the logs, I am still attaching them. But they seem to show that the kernel database was found properly.
aibenchmark_miopen_log.txt

@Epliz Please remove user kernel cache and try (.ukdb files somewhere in ./cache/miopen).

Here is with deleting the cache before
aibenchmark_miopen_log2.txt

@Epliz it does not try to load or build the naive kernel, so there is no error either ;) Anyway, we have PR #2050 with fix, and this issue can be closed.

@Epliz Thanks, I think we already have this feature in our backlog and the ticket is not necessary.
CC @JehandadKhan

The pr for the fix has been merged, so closing the ticket.
Thank you for your quick action and your help was much appreciated!

Latest nightly (from yesterday) seems still affected.
pip install --pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/rocm5.5
shows
MIOpen(HIP): Error [Compile] 'hiprtcCompileProgram(prog.get(), c_options.size(), c_options.data())' naive_conv.cpp: HIPRTC_ERROR_COMPILATION (6)
MIOpen(HIP): Error [BuildHip] HIPRTC status = HIPRTC_ERROR_COMPILATION (6), source file: naive_conv.cpp
MIOpen(HIP): Warning [BuildHip] /tmp/comgr-d809c6/input/naive_conv.cpp:39:10: fatal error: 'limits' file not found
#include <limits> // std::numeric_limits
         ^~~~~~~~

on use.

@Mershl  could you help to provide more detailed log? Could you provide system info (which GPU) and try if the above suggestions might help? i.e. install KDB, and/or install the libstdc++-12-dev on ubuntu 22.04.2 LTS.
CC: @jeffdaily  since this is PT wheel nightly related issue.

try if the above suggestions might help? i.e. install KDB, and/or install the libstdc++-12-dev on ubuntu 22.04.2 LTS.

Providing the libstdc++-12-dev equivalent (libstdc++-devel) on Fedora alone did not fix the issue. Installing the "Development Tools" "Development Libraries" group (similar to the build-essential group on Ubuntu) as well fixes the issue. Generation is completing successfully on latest torch nightly.
Interesting though that I can now remove stable-diffusion-webui + its venv + libstdc++-12-dev + the dev tools + do a reboot
and a freshly checked out stable-diffusion-webui will still generate successfully. Is the built kernel / openmi lib cached somewhere?
EDIT: it fails again when using previously never used features of stable-diffusion-webui (like enabling HiRes fix) and is again fixed when providing the dependencies listed above.

Built kernels are cached at ~/.cache/miopen, and #2050 is only recently merged into PT wheel nightly and expect to exist there in the next build https://download.pytorch.org/whl/nightly/rocm5.5


Kickoff on scoping this Epic

have a meeting to scope NFT requirements
listen from @chenzongxiong on faced challenges
talk about next step

Let's have initial discussion within @bdmason @chenzongxiong @markdavison and @jdomingos

Some notes on indexing NFTs on EVM
1. Detecting all NFT contract creations
We have the logic for how to do this using the EVM transaction input, the issue here is how we listen for transactions. For Moonbeam EVM we are listening to Ethereum.Executed events on the substrate network using Subsquid. In The Graph you normally to specify the contracts you are interested in, but you can omit this and watch the blocks. See the docs.
2. Indexing NFT activity
This shouldn't be too hard, we've already done some of this here.
3. Indexing metadata links that work
This is a highly annoying task. Each contract (and probably each NFT from each contract) can store the metadata URI (if they have one) differently, sometimes just an IPFS hash, sometimes an actual URL, sometimes something in between... @chenzongxiong has done something in the past that we can adapt. Get the IPFS hash and hit all known IPFS hosts then use the first that gives a 200. This has been done when the user requests NFT content, and can take a while so it's not ideal. We could run scripts to do this in the background and store the useful URL. We could also consider creating our own IPFS node and storing the data ourselves, though I imagine the volume will be ludicrous.
Notes
BSC and other EVM networks
This should be straightforward. We just run the same code on other networks
The Graph - should we use it?
When indexing specific contracts The Graph is great, you get graphql schema to postgres scaffolding out the box with codegen, as well as automatically generated types from the contracts (via the ABI -> something we won't always have).
When developing on the graph the current flow is: make changes -> build -> deploy (to test environment on their hosted service) -> look at the results.  Also, when switching networks you need a new graph instance, it's not as simple as switching an environment variable.
I think The Graph is going to be more of a hinderance to us, and as we have very focused requirements simply looping through the blocks looking for what we want using our own custom code is more preferable. I did this on Substrate last year and it was trivial. We'd need to implement graphql and postgres ourselves, but that's also quite trivial. The advantage is complete flexibility in our approach.

Regarding other networks
EVM is the original smart contract platform, so it suffers from legacy data & standards issues a lot. If we can crack EVM I think most of the hard work is done.
On Solana you can see your NFTs in all your wallets, so unless all those wallets are hooked up to indexed data somewhere that means Solana stores NFT ownership data differently to EVM (as you can't just query NFTs by account on EVM). So in terms of knowing who owns what I expect this to be easy without any indexing required. In terms of NFT analytics, that's a whole different issue, and the volume of data on Solana (due to it being a high throughput network) will be the issue. Nevertheless, it's a newer network so we might find the design makes things a lot easier for us to work with.
I (optimistically hope) that the same will hold true for other NFT networks, that being newer the implementation of NFTs on the networks might make our life a hell of a lot easier than it has been on EVM.


Visibility: Visibility not functional inside of a Modal

Bug Report
Semantic-UI React Visibility does not seem to be functioning properly inside of modals.
Steps
Take any element with visibility and place it inside of a modal, some calculations for visibility will not function properly, notable with bottomVisible/topVisible.
Expected Result
When the bottom/top of the page is scrolled to, visibility should recognize that and adjust their respective booleans according
Actual Result
The booleans don't adjust themselves in a logical manner when the visibility is placed inside of a modal, it is potentially focused on the screen behind the modal.
Version
^0.82.5
Testcase
https://codesandbox.io/embed/vy83jjwz20?fontsize=14

Yeah I'm also having this issue with Visibility inside Modals... It seems when the view is in a Modal, the scroll doesn't trigger the Visibility at all except at the initialization.

I believe this may be related to this PR that's just been merged to master:
https://github.com/Semantic-Org/Semantic-UI-React/pull/3400

Created simplified version of CodeSandbox: https://codesandbox.io/s/semanticuireactexample-j217f
Behaves as expected, if I missed something please ping me.


Node.Process

The Node.Process dependency is placing a hard dependency where you have to run the tests on nodejs. The issue with that is that you can't use something like phantomjs to run tests and test libraries that interact with the DOM.
How do you feel about going back to defining a custom PROCESS effect?

Yeah, that is unfortunate. We could basically revert 9ce8ebf76f4c2a1a7802b93fa4456831db174ff8. Something you'd like to send a PR for?

sure! :)

can we keep using the PROCESS row label though? That was my initial motivation for submitting #23.

@felixSchl Yeah, sure!


AttributeError at /auth/token

Client object has no attribute id with post request method

Fixed this. Was caused by django's jwt engine looking for id. I changed it such that it looks for the username as a unique field. Works now. Merged into my branch frontend/amay. Creaing a pull request. please accept it.

Changes are visible - closing the issue


Error: This integration doesn't support provided entity!

Describe the bug
I get the fallowing error when adding the card to my lovelace dashboard:
Error: This integration doesn't support provided entity!
type: custom:meteoalarm-card
entity: binary_sensor.meteoalarm
integration: meteoalarm
override_headline: false

To Reproduce

Choose the lovelace dashboard
clock on: +add card
add the meteoalarm-card
chose the entity: binary_sensor.meteoalarm

Attributes of the entity: binary_sensor.meteoalarm
language: en-GB
category: Met
event: Yellow Rain Warning
responseType: None
urgency: Expected
severity: Moderate
certainty: Likely
effective: 2021-07-15T00:00:00+02:00
onset: 2021-07-15T00:00:00+02:00
expires: 2021-07-16T21:00:00+02:00
senderName: MeteoSwiss
headline: Yellow Rain Warning for Switzerland - Wohlen-Albis
description: - Expected amounts: 40-60 mm
- Snowfall limit: 2700-3000 m
web: http://www.meteoswiss.admin.ch
contact<EMAIL_ADDRESS>attribution: Information provided by MeteoAlarm
friendly_name: meteoalarm
device_class: safety

Screenshot

System:

Card version: v1.9.0
Integration type: Core
Installation type: HACS

Hey, it seams like you are trying to use custom xlcnd/meteoalarmeu while selecting integration: meteoalarm which is a mode for core integration and that's the issue.
The next this is you shouldn't use xlcnd integration anymore see: #38

Thanks for your response, but I did not install the custom integration. To be honest I have no clue from which source HA has the attributes of the binary sensor. Because on meteoalarm.org are no alerts listed for my region. 🤨
Seems like I need to investigate this by myself. Thank you nevertheless for making this nice custom card.

Hi! I am having the same issue. I use the core meteoalarm integration but get the same error than @gartenbauer. I also don't use the deprecated custom integration. Is there any way to get rid of this error?

Hi! I am having the same issue. I use the core meteoalarm integration but get the same error than @gartenbauer. I also don't use the deprecated custom integration. Is there any way to get rid of this error?

Hey, please create new issue and attach screenshots, configuration and console logs


-1 for TLS 1.2

I'm seeing that -1 is listed for a high number of clients under TLS 1.2 Client Enabled setting. Am I right in assuming that thi should be NULL or 0 and is somehow gotten missed confirmed during the Windows Setup?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: dd180142-4c19-8abc-ecce-e4c7e5ac8951
Version Independent ID: fc5bba87-afad-1729-8f7e-333c73188b73
Content: Transport Layer Security (TLS) registry settings
Content Source: WindowsServerDocs/security/tls/tls-registry-settings.md
Product: windows-server
Technology: security-tls-ssl
GitHub Login: @Justinha
Microsoft Alias: justinha

Please note the following:
You may want to look elsewhere for answers, unless you have suggestions for improving one or more specific pages in the MS Docs repository.
Disclaimer: I am not affiliated with Microsoft or the Microsoft Docs teams here on GitHub.
The intention of this repository is to improve on the existing documentation by providing the users a way to report factual errors, improvement suggestions and request updates for new versions.
The MS Docs team members might not have time to answer your question, both due to the small size of the team and the big workload.
Try the following resources to get in touch with users who may be able to help you:

TechNet forums: https://social.technet.microsoft.com/Forums/en-US/home
MSDN: https://social.msdn.microsoft.com/Forums/en-US/home
Microsoft Community: https://answers.microsoft.com/

@Garthmj GitHub issues for this repo are for reporting problems with documentation.  Please open a support ticket or post your question at Microsoft Q&A.


Considering moving ymls that only run against the VMR

We should consider moving ymls that only run against the VMR into src/SourceBuild/content/eng/pipelines.
Currently, there are a few ymls in installer/eng/common/pipelines that we should consider moving, including vmr-license-scan.yml and source-build-sdk-diff-tests.yml
This was originally brought up in https://github.com/dotnet/installer/pull/19201#discussion_r1539852807

Changes mirrored, switched the source in the pipelines, waiting for the builds to finish to verify the move

Both builds passed without visible regressions, closing


NLTK Documentation added

Related Issue : #439

@shreya024 your code is not readable, maintain the proper headings in middle as well...

have a look and try to maintain the similar- https://buildmedia.readthedocs.org/media/pdf/jupyter-notebook/latest/jupyter-notebook.pd

I am not able to open this file

@shreya024 your code is not readable, maintain the proper headings in middle as well...

have a look and try to maintain the similar- https://buildmedia.readthedocs.org/media/pdf/jupyter-notebook/latest/jupyter-notebook.pd

@Niketkumardheeryan I am not able to open this file...it is showing this ...
This XML file does not appear to have any style information associated with it. The document tree is shown below.
<Error>
NoSuchKey
<Message>The specified key does not exist.</Message>
<Key>pdf/jupyter-notebook/latest/jupyter-notebook.pd</Key>
<RequestId>PDP5QXTDBK0DCY5J</RequestId>
<HostId>p3jWFOWpMEqH9COHyU1v/FUPApGPBAl/9i6tSK8+Iidyv3AlRLkQ3lmRy84C5zpAKBUn4DbJ1rk=</HostId>
</Error>

you can refer other notebooks ...

@shreya024 your code is not readable, maintain the proper headings in middle as well...

have a look and try to maintain the similar- https://buildmedia.readthedocs.org/media/pdf/jupyter-notebook/latest/jupyter-notebook.pd

@Niketkumardheeryan I am not able to open this file...it is showing this ...
This XML file does not appear to have any style information associated with it. The document tree is shown below.
<Error>
NoSuchKey
<Message>The specified key does not exist.</Message>
<Key>pdf/jupyter-notebook/latest/jupyter-notebook.pd</Key>
<RequestId>PDP5QXTDBK0DCY5J</RequestId>
<HostId>p3jWFOWpMEqH9COHyU1v/FUPApGPBAl/9i6tSK8+Iidyv3AlRLkQ3lmRy84C5zpAKBUn4DbJ1rk=</HostId>
</Error>

you can refer other notebooks ...

I have added proper headings in the middle as well


Upgrade parquet to 1.12.0-palantir.7

Take fix for PARQUET-1550.

👍

The DataFrame reuse test in StreamSuite is failing because we're hitting PARQUET-1746.

PARQUET-1746 was closed because it's not a Parquet issue. Now sure what the resolution is.

Upstream ticket to track getting Parquet 1.11.0 into Spark: SPARK-26346
They identified issues with 1.11.0 tracked in PARQUET-1774. Don't think we have all of 1.11.1, e.g. https://github.com/apache/parquet-mr/pull/747, but don't see how that'd be related to tests failing here.


programmatic usage

php-ipfs-api is introduced through autoload.php
test.php Code:
//$driver = $container[IPFS\Driver\Cli::class];
$driver = $container[IPFS\Driver\Http::class];
$client = new IPFS\Client($driver);
$response = $client->execute((new \IPFS\Api\Basics())->version());
var_dump($response);

error:
Fatal error: Uncaught TypeError: Argument 1 passed to IPFS\Client::__construct() must implement interface IPFS\Driver\Driver, null given, called in /php/phalapi/public/1.php on line 12 and defined in /php/phalapi/vendor/ipfs/php-client/src/Client.php:26 Stack trace: #0 /php/phalapi/public/1.php(12): IPFS\Client->__construct(NULL) #1 {main} thrown in /php/phalapi/vendor/ipfs/php-client/src/Client.php on line 26

try it like this:
you probably didnt imported the Pimple DIC:
see https://github.com/digitalkaoz/php-ipfs-api/blob/master/src/Container/ServiceProvider.php#L96 how to create the CLI Client by hand
// cli driver
$driver = new \IPFS\Driver\Cli(new Process([]), new AnnotationReader(
                new \Doctrine\Common\Annotations\AnnotationReader(),
                \phpDocumentor\Reflection\DocBlockFactory::createInstance()
            ), getenv('IPFS_BINARY') ?: 'ipfs');

$client = new \IPFS\Client($driver);
$response = $client->execute((new \IPFS\Api\Basics())->version());
var_dump($response);

i strongly advice to use Pimple
then it would be this easy:
use Pimple\Container;

//container
$container = new Container();
$container->register(new \IPFS\Container\ServiceProvider());

// setup api client
$driver = $container[\IPFS\Driver\Cli::class];
//$driver = $container[\IPFS\Driver\Http::class];
$client = new \IPFS\Client($driver);

//use it
$response = $client->execute((new \IPFS\Api\Basics())->version());
var_dump($response);

Thank you very much for solving my problem. The epidemic is serious
recently. I hope everything is OK!


Team Challenge

Define a challenge for which 2-3 teammates can join forces to complete. This is for milestone 1

@BlaiseGratton Please help me come up with ideas for this challenge

@stevebrownlee I'd like to propose something involving implementing an interface of behavior via different data structures... but I'm noticing that the three milestone charts of topics don't have an explicit section for data structures. We usually introduce LINQ in the first part of the curriculum, and it's only officially declared in milestone 2 ("Using LINQ in MVC"). Is it fair game to assume Milestone 1 includes a cursory knowledge of at least the built-in collections and data structures (Lists, Dictionaries, Stack, etc.)?

@BlaiseGratton  For a team challenge - and for the course, in general - it needs to be something less esoteric than that. Give them an actual problem to solve, an app to build, or a game. Something that they can relate to and have fun building. Less math, less data structures, more relatable subjects.

For example, in cohort 12, Lynn and Caitlin added some group projects that had students building an application that helped a veterinarian clinic manage the scheduling and care of animals, or let an art dealer manage their inventory of art, who the artist was, etc.

@stevebrownlee Gotcha, my bad. And these would be console apps with no database?

@BlaiseGratton Well, if it was for the first milestone, it would be a command line application, but with a database. Challenges are truly that. It takes what has been covered so far, and then suggests that they use something that either (a) we don't teach and they will have to research themselves, or (b) will be covered in the next milestone so they can read ahead and get some hints on how to solve it

Okay, couple ideas here:
A console interface for a personal blog/journal. Each entry contains a timestamp, title, content, and tags. At entry time, the user has the option to enter an alternate date and time, otherwise it defaults to the current timestamp. There is a console interface for listing out entries and viewing items of a given entries list. The user can search by time categories (morning, afternoon, day, month, year) as well as tags (and both).
A console application which tracks progress data for a list of Treehouse profiles. Database contains users and their corresponding totals in each language over time. Either manually or as a scheduled task, the application runs through the list of user profiles and updates their current stats in the db (this is hitting a url from Treehouse and parsing the JSON response data). The user data over time is displayed graphically in the console.
A console game consisting of a bird's eye view of a room (a 2-d matrix) showing your character's location. You navigate up, down, left, and right (within the bounds) and encounter either empty space, items, monsters, or the escape door (none of this is displayed, goal is to escape the room). The items, monsters, and escape door are randomly generated each game. You can use items when battling monsters. When battling a monster, it is a turn-based interface where you and the monster take turns attacking until someone's health stats run out. (I believe I did a writeup of this a while ago, only thing it's missing is a datastore. What would be stored? Game state?)

I like ideas 3 and 1.
If you've got some initial content for 3, let's flesh that out some more. As for what can be stored in a database, it could be as simple as...

Character (health, name, rooms escaped count)
Inventory
Items held (2 out of the inventory at any time)

Character could pick up any item, but can only hold two at a time.


Added a Fish class under jrcryan directory

A class to draw Fish w/ a subclass to draw FishWithFins

can't run code, make sure you complete lab and turn in all necessary .java files

Are there more necessary files besides Fish.java, FishWithFins.java, and AllMyDrawings.java?

I need your simple directory as well


Changes to Bowling doc or test

In the documentation an atom : ok is required as a return value for roll. But this will break the test at line 12.

Good catch! I think it would be preferable to remove that line from the function documentation rather than break the test. Do you want to update it? If not then I'll take care of it.


SSH host key changing on every boot

Describe the bug
Hi!
I have recently installed nixos on my raspberry pi 4B (aarch64-linux) and I have noticed that everytime I try to ssh to it its ssh host key has changed and i have to edit my known_hosts. After "debugging" it, I have confirmed that everytime the pi boots the ssh host key generation produces a new key.
This has not happend on any other of my nixos installations (all of them are x86_64-linux), so I am maybe wondering if this is a bug with any aarch64-linux nixpkgs/ssh implementation or its maybe a hardware specific issue with the pi 4B.
Thank you for your help!
Steps To Reproduce
Steps to reproduce the behavior:

Install nixos-22.05 on aarch64-linux
Enable openssh server
Profit?

Expected behavior
ssh host key stays the same across reboots
Actual behaviour
ssh host key changes on each reboot
Additional context
Running nixos 22.05 on the raspberry pi 4B
Configuration: https://github.com/baitinq/nixos-server
Notify maintainers
@edolstra
Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-linux"`
 - host os: `Linux 5.15.32, NixOS, 22.05 (Quokka), 22.05.20221001.9cac458`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.8.1`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/ssh-host-key-changing-on-every-boot/22316/1

Is your root filesystem a tmpfs?

Is your root filesystem a tmpfs?

This.
https://github.com/Baitinq/nixos-server/blob/587e04cb3a48d3e99f5ee15e225b1b5e831d99a1/hardware/pi/disks.nix#L53-L57
You don't have a persistent storage for the generated ssh keypair below /etc/ssh, so they vanish when you reboot.

Hi @rapenne-s @mweinelt , yes my / is a tmpfs. I figured this might be part of the issue but this doesnt happen on any other hosts which also have a tmpfs / (https://github.com/baitinq/nixos-config).
Do you have any idea of what could cause this different behaviour?

You may want something like this
 environment.etc."ssh/ssh_host_rsa_key".source
    = "/nix/persist/etc/ssh/ssh_host_rsa_key";
  environment.etc."ssh/ssh_host_rsa_key.pub".source
    = "/nix/persist/etc/ssh/ssh_host_rsa_key.pub";
  environment.etc."ssh/ssh_host_ed25519_key".source
    = "/nix/persist/etc/ssh/ssh_host_ed25519_key";
  environment.etc."ssh/ssh_host_ed25519_key.pub".source
    = "/nix/persist/etc/ssh/ssh_host_ed25519_key.pub";

from https://elis.nu/blog/2020/05/nixos-tmpfs-as-root/

You may want something like this

Thank you, I will try that asap.
Should I leave this issue open in case there is further info/research on what causes the different behavior between my hosts? I am still quite baffled by that.

If you have / on tmpfs you better be using some tool to diff the state before shutting down the system, otherwise you risk loosing data that hasn't been made persistent.
A simple solution is this:
{ lib, pkgs, ... }:

with lib;

let
  # Utility to save a snapshot of the root tree
  save-root = pkgs.writers.writeDashBin "save-root"
    ''
      ${pkgs.findutils}/bin/find \
        / -xdev \( -path /tmp -o -path /var/tmp -o -path /var/log/journal \) \
        -prune -false -o -print0 | sort -z | tr '\0' '\n' > "$1"
    '';

  # Utility to compare the root tree
  diff-root = pkgs.writers.writeDashBin "diff-root"
    ''
      export PATH=${with pkgs; makeBinPath [ diffutils less ]}:$PATH
      current="$(mktemp current-root.XXX --tmpdir)"
      trap 'rm "$current"' EXIT INT HUP
      ${save-root}/bin/save-root "$current"
      diff -u /run/initial-root "$current" --color=always | ''${PAGER:-less -R}
    '';

in

{
  environment.systemPackages = [ diff-root ];

  systemd.services.save-root-snapshot =
    { description = "save a snapshot of the initial root tree";
      wantedBy = [ "sysinit.target" ];
      requires = [ "-.mount" ];
      after = [ "-.mount" ];
      serviceConfig.Type = "oneshot";
      serviceConfig.RemainAfterExit = true;
      serviceConfig.ExecStart = ''${save-root}/bin/save-root /run/initial-root'';
    };

}

You can see a list of all files that have been created since boot with sudo diff-root.

save-root

Thank you! Ill definitely implement that to my config.

Closing since configuration error and all the questions where answered.


WCF Client Generator

I was looking at this article and it got me thinking. It shouldn't be too hard to add a WCF Client generator to this tool. This would give us a much simpler code than the current HttpClient based client and we'd gain the other advantages that WCF has to offer.
https://blogs.msdn.microsoft.com/pedram/2008/04/21/how-to-consume-rest-services-with-wcf/

Interesting, didnt know that wcf is still a thing... Looks interesting. But why do you care about the actual client implementation?


[server] posting mechanism

Reopening this cause of post creation do not wotk with tags yet. It was planned in second PR related to this task


Nightly Flake : Gather code run from input box

Error message
'# This file contains only the code required to produce the results of the gathered cell.\n' == '# This file contains only the code required to produce the results of the gathered cell.\n# %%\na=1\na\n\n'
Stack trace
AssertionError [ERR_ASSERTION]: '# This file contains only the code required to produce the results of the gathered cell.\n' == '# This file contains only the code required to produce the results of the gathered cell.\n# %%\na=1\na\n\n'
    at interactiveWindowTestHelpers_1.runMountedTest (out/test/datascience/interactiveWindow.functional.test.js:491:20)
    at process._tickCallback (internal/process/next_tick.js:68:7)

This is either working or disabled at the moment.


shoutbox dont work

get_msgs.php work but in index.php shuotbox always clear

Provide more info?
It works perfectly fine for me.

mainpage

get_msgs.php output

index.php code i chance only config and db connect

and shoutbox id is result

Try on a different server.
Pretty sure ajax is not working properly for you.


Unhandled command event type 3

Steps to reproduce

Log in with two clients.
Select another player
Right click another player
Select Info
Right click same player
Select Info again.

Every subsequent "Info" request results in the error message Unhandled command event type 3. Which I assume is some sort of "Refresh Windows" command.

Closing this as it appears to have been resolved and was probably unsent bits causing unrelated messages to appear.


Verify zpaq files after compressd by zpaqfranz 59.1 with -chunk

After run the command : zpaqfranz a test??.zpaq -chunk 1m
I got 13 files : test01.zpaq ~ test13.zpaq
if run the command : zpaqfranz v test01.zpaq
I got error ： zpaqfranz error: archive not found

if run the command : zpaqfranz v test02.zpaq
I got ask me to input password ??

If i modify test08.zpag content just 1 bit and run zpaqfranz v test??.zpaq to verify then show all ok

If run zpaqfranz t test??.zpaq to test then show error

Wow, a single question that raise 4 different answers 😄
First of all, thanks for you report.
I'll explain all of your concers (no keyboard now)
The differences of t test and v verify (2 different things)
Why chuncked archives cannot be used like not chuncked
Why they get the same extension (zpaq)
And how to use backup (eventually) instead
"When" use chunk, and "why"

Now the answers, broken down by topic
Let's start with chunked archives
Multipart .zpaq archives are composed of whole zpaq files
zpaqfranz a z:\full_?? *.cpp
zpaqfranz a z:\full_?? *.exe

This will create TWO zpaqs
17/01/2024  11:39         1.253.507 full_01.zpaq
17/01/2024  11:39        12.304.926 full_02.zpaq

Of different size.
 You can replace test_27.zpaq with a new zpaq and the program will not notice it
Similarly you can have "holes" (test_01...02...03...05...06...) with a missing 04
The problem increases more if the files (which make up the .zpaq file sequence) have different passwords

CHUNKED Multipart .zpaq archives are NOT composed of whole zpaq files
Now split @ 1MB the same example
C:\zpaqfranz>zpaqfranz a z:\piece_?? *.cpp -chunk 1m
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-chunk                            1.000.000
(...)INFO: The last chunk is: z:/piece_02.zpaq

C:\zpaqfranz>zpaqfranz a z:\piece_?? *.exe -chunk 1m
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-chunk                            1.000.000
(...)
INFO: The last chunk is: z:/piece_14.zpaq

Here's the files. Note that "piece_01" and 02 are the FIRST version, the other the 2nd
17/01/2024  11:47         1.048.576 piece_01.zpaq
17/01/2024  11:47           204.931 piece_02.zpaq
17/01/2024  11:47         1.048.576 piece_03.zpaq
17/01/2024  11:47         1.048.576 piece_04.zpaq
17/01/2024  11:47         1.048.576 piece_05.zpaq
17/01/2024  11:47         1.048.576 piece_06.zpaq
17/01/2024  11:47         1.048.576 piece_07.zpaq
17/01/2024  11:47         1.048.576 piece_08.zpaq
17/01/2024  11:47         1.048.576 piece_09.zpaq
17/01/2024  11:47         1.048.576 piece_10.zpaq
17/01/2024  11:47         1.048.576 piece_11.zpaq
17/01/2024  11:47         1.048.576 piece_12.zpaq
17/01/2024  11:47         1.048.576 piece_13.zpaq
17/01/2024  11:47           770.590 piece_14.zpaq

You cannot list, for example, the piece_12.zpaq as before: zpaqfranz cannot recognize the "piece" (as a zpaq archive), then "thinks" it is an encrypted one, then you must try a password
C:\zpaqfranz>zpaqfranz l z:\piece_12.zpaq
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-hw
Archive seems encrypted (or corrupted)
Enter password :

This, of course, don't work
You must use the wildcard, as standard for zpaq, to list (or extract or whatever)
C:\zpaqfranz>zpaqfranz l z:\piece_??.zpaq
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-hw
z:/piece_??.zpaq:
2 versions, 20 files, 13.558.433 bytes (12.93 MB)

- 2023-11-10 16:38:26           3.263.488 A     12.exe
- 2024-01-15 14:26:16           3.331.584 A     13.exe
- 2023-04-21 17:28:19           3.148.288 A     5.exe
- 2024-01-15 16:19:22           3.492.792 A     59_1.cpp
- 2023-04-21 17:28:19           3.148.288 A     6.exe
- 2018-12-30 21:56:20             968.192 A     CppCommentRemover.exe
- 2017-10-05 08:27:44              86.546 A     dd.exe
- 2023-10-04 14:43:51           3.244.032 A     exp01.exe
- 2023-12-16 14:38:40           5.859.195 A     exp03-Wunused-parameter.exe
- 2023-12-16 15:01:40           3.014.144 A     exp03.exe
- 2023-10-25 15:53:54             307.200 A     unz.exe
- 2024-01-15 18:03:00           3.494.187 A     va01.cpp
- 2023-12-09 18:44:23             635.392 A     zpaq715.exe
- 2016-08-17 23:07:33             984.576 A     zpaqd.exe
- 2024-01-16 14:54:03           2.513.087 A     zpaqfranz-linux.cpp
- 2024-01-16 14:53:45           3.494.167 A     zpaqfranz.cpp
- 2024-01-16 14:52:40           6.109.764 A     zpaqfranz.exe
- 2024-01-16 14:57:46           3.074.048 A     zpaqfranz32.exe
- 2024-01-16 14:58:55           3.120.128 A     zpaqfranzhw.exe
- 2023-07-23 11:40:15             999.264 A     zpaqlist.exe

           54.288.362 (51.77 MB) of 54.288.362 (51.77 MB) in 20 files shown
           13.558.433 compressed  Ratio 0.250 <<z:/piece_??.zpaq>>
0.078 seconds (000:00:00) (all OK)

The key-point is that backward compatibility with zpaq 7.15 is maintained
Look at this example
C:\zpaqfranz>zpaq64 l z:\piece_??
zpaq v7.15 journaling archiver, compiled Aug 17 2016
z:/piece_??.zpaq: 2 versions, 20 files, 577 fragments, 13.558433 MB

- 2023-11-10 15:38:26      3263488 A     12.exe
- 2024-01-15 13:26:16      3331584 A     13.exe
- 2023-04-21 15:28:19      3148288 A     5.exe
- 2024-01-15 15:19:22      3492792 A     59_1.cpp
- 2023-04-21 15:28:19      3148288 A     6.exe
- 2018-12-30 20:56:20       968192 A     CppCommentRemover.exe
- 2017-10-05 06:27:44        86546 A     dd.exe
- 2023-10-04 12:43:51      3244032 A     exp01.exe
- 2023-12-16 13:38:40      5859195 A     exp03-Wunused-parameter.exe
- 2023-12-16 14:01:40      3014144 A     exp03.exe
- 2023-10-25 13:53:54       307200 A     unz.exe
- 2024-01-15 17:03:00      3494187 A     va01.cpp
- 2023-12-09 17:44:23       635392 A     zpaq715.exe
- 2016-08-17 21:07:33       984576 A     zpaqd.exe
- 2024-01-16 13:54:03      2513087 A     zpaqfranz-linux.cpp
- 2024-01-16 13:53:45      3494167 A     zpaqfranz.cpp
- 2024-01-16 13:52:40      6109764 A     zpaqfranz.exe
- 2024-01-16 13:57:46      3074048 A     zpaqfranz32.exe
- 2024-01-16 13:58:55      3120128 A     zpaqfranzhw.exe
- 2023-07-23 09:40:15       999264 A     zpaqlist.exe

54.288362 MB of 54.288362 MB (20 files) shown
  -> 42.161910 MB (813 refs to 577 of 577 frags) after dedupe
  -> 13.558433 MB compressed.
0.031 seconds (all OK)

Changing the extension of the chunks (e.g., from .zpaq to .zcp) so that zpaqfranz would recognize them for what they are (multipart chunked and not multipart) would have prevented use with zpaq 7.15 (incidentally costing me a lot of time and effort)
Short version => use the "?" when dealing with multipart archive, chunked or standard

From multipart to monolithic
It is possible to convert a multipart archive to a single .zpaq file in essentially two ways.
The first is to use the m (merge) command
C:\zpaqfranz>zpaqfranz m z:\full_?? z:\full-one-piece.zpaq
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-hw
*** Merge (consolidate) ***
Chunk 00000000            1.253.507 <<z:/full_01.zpaq>>
Chunk 00000001           12.304.926 <<z:/full_02.zpaq>>
--------------------------------------------------------------------------------------------------------------------
Total 2           13.558.433 (12.93 MB)
Outfile    <<z:/full-one-piece.zpaq>>
Free space       85.772.365.824
Needed               13.558.433
(...)
Done
Written            13.558.433
Expected           13.558.433

0.062 seconds (000:00:00) (all OK)

Same thing for the chunked
C:\zpaqfranz>zpaqfranz m z:\piece_?? z:\piece-one-piece.zpaq
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-hw
*** Merge (consolidate) ***
Chunk 00000000            1.048.576 <<z:/piece_01.zpaq>>
Chunk 00000001              204.931 <<z:/piece_02.zpaq>>
Chunk 00000002            1.048.576 <<z:/piece_03.zpaq>>
Chunk 00000003            1.048.576 <<z:/piece_04.zpaq>>
Chunk 00000004            1.048.576 <<z:/piece_05.zpaq>>
Chunk 00000005            1.048.576 <<z:/piece_06.zpaq>>
Chunk 00000006            1.048.576 <<z:/piece_07.zpaq>>
Chunk 00000007            1.048.576 <<z:/piece_08.zpaq>>
Chunk 00000008            1.048.576 <<z:/piece_09.zpaq>>
Chunk 00000009            1.048.576 <<z:/piece_10.zpaq>>
Chunk 00000010            1.048.576 <<z:/piece_11.zpaq>>
Chunk 00000011            1.048.576 <<z:/piece_12.zpaq>>
Chunk 00000012            1.048.576 <<z:/piece_13.zpaq>>
Chunk 00000013              770.590 <<z:/piece_14.zpaq>>
--------------------------------------------------------------------------------------------------------------------
Total 14           13.558.433 (12.93 MB)
Outfile    <<z:/piece-one-piece.zpaq>>
Free space       85.758.803.968
Needed               13.558.433
(...)
Done
Written            13.558.433
Expected           13.558.433

0.110 seconds (000:00:00) (all OK)

Now
C:\zpaqfranz>dir z:\*one*.zpaq
 Il volume nell'unità Z è RamDisk
 Numero di serie del volume: 40FE-1005

 Directory di z:\

17/01/2024  11:59        13.558.433 full-one-piece.zpaq
17/01/2024  12:00        13.558.433 piece-one-piece.zpaq

You can also do the merge with normal commands, such as copy /b, I won't go into it here

Now on v (verify) against t (test)
The v command has a different purpose than the one for which I think you want to use it
The goal is NOT to check that the .zpaq archive is correct, or even that its contents match the files that were archived, once extracted
That is you do with the t command (or p or w)
Verify compares, possibly in parallel on SSD drives (with the -ssd switch) that the hashes of the files that zpaqfranz has processed match those of the files in the original folders (or wherever you want with the -to, -find, -replace switches)
The point is related to the possibility of changes DURING the procedure.
I give an example that I hope will clarify it (!)
Suppose we have a Windows file server with a number of users accessing files from their clients
Suppose we want to back them up somewhot
Let's run it and get a .zpaq file, perhaps with a 15-minute time frame
What if a user had opened, say, the foo.doc file (while the backup was running), modifies it, and saves it while the backup was running?
The foo.doc file, currently in the file server folder, may be DIFFERENT from the one inside the .zpaq file. Not because of some kind malfunction, but precisely because the compression operation is not atomic, it takes time. You start with a 100 bytes-long foo.doc, but you store a 200 bytes-long foo.doc (just an example)
zpaqfranz deals with this possibility in two ways: the first is with the zfs support (on non-Windows machines), where you can be assured that the files will NOT be modified during execution (taken from a snapshot)
The second, on Windows, is to do a v (verification) at the end of archiving. On SSD drives this is very fast (even more than 1GB/s in the real world).
Short version: v checks that the hashes created (during the compression phase) of the files match with the current one. This is a very quick check, but it says nothing about the integrity of the .zpaq file, for which the other command (t) is used
To do this you would have to extract the archive (somewhere) and then check the hashes there. That's exactly the t command with the -paranoid switch

The t command (and the -paranoid)
First of all I suggest to... read the wiki, or the embedded manual
zpaqfranz h t

looking at the examples
Last version:                        t z:\1.zpaq
All versions:                        t z:\1.zpaq -all
Compare against filesystem:          t z:\1.zpaq -verify
Against filesystem, 4 threads:       t z:\1.zpaq -verify -ssd -t4
Real paranoid: extract all           t z:\1.zpaq -to z:\knb -paranoid
Fast-SHA1 (nz the source dir):       t z:\1.zpaq c:\nz
Cnk-SHA1+hash (nz the source dir):   t z:\1.zpaq c:\nz -checksum
Multiple paranoid check (Win):       t *.zpaq -to z:\temp\ -paranoid -longpath -big
Multiple test (*NIX):                t "/copie/*.zpaq"
All version SHA-1 collisions:        t z:\1.zpaq -collision -all

Lets pick some of them
To test that the .zpaq file IN THE LAST VERSION is not corrupted, you use this
C:\zpaqfranz\spaz>zpaqfranz t intermedie.zpaq
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-hw
intermedie.zpaq:
7350 versions, 7.631 files, 431.966.159 bytes (411.95 MB)
To be checked 351.508.130 in 102 files (32 threads)
7.15 stage time       0.86 no error detected (RAM ~130.07 MB), try CRC-32 (if any)
Checking             4.592 blocks with CRC-32 (351.508.130 not-0 bytes)

CRC-32 time           0.12s
Blocks         351.508.130 (       4.592)
Zeros                    0 (           0) 0.000000 s
Total          351.508.130 speed 2.492.965.460/sec (2.32 GB/s)
GOOD            : 00000102 of 00000102 (stored=decompressed)
VERDICT         : OK                   (CRC-32 stored vs decompressed)
1.000 seconds (000:00:01) (all OK)

Then you know that the very last version (7350) is good, with (about) 351MB
If you want to test EVERY part of the file, you do -all
C:\zpaqfranz\spaz>zpaqfranz t intermedie.zpaq -all
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-all                                      4
franz:-hw
intermedie.zpaq:
7350 versions, 7.631 files, 431.966.159 bytes (411.95 MB)
To be checked 25.314.671.936 in 7.629 files (32 threads)
7.15 stage time      15.47 no error detected (RAM ~258.07 MB), try CRC-32 (if any)
Checking           303.287 blocks with CRC-32 (25.314.671.936 not-0 bytes)
Block 00275K         21.54 GB
CRC-32 time           8.89s
Blocks      25.314.671.936 (     303.287)
Zeros                    0 (           0) 0.000000 s
Total       25.314.671.936 speed 2.846.904.176/sec (2.65 GB/s)
GOOD            : 00007629 of 00007629 (stored=decompressed)
VERDICT         : OK                   (CRC-32 stored vs decompressed)
24.359 seconds (000:00:24) (all OK)

In this example about 25GB of data is checked, two times
The first is fragment-SHA1 test (just like zpaq 7.15)
The second stage is global (file level) CRC-32 (zpaqfranz's)
Now you know that the archive contains valid data, however, you DO NOT KNOW if it matches the original data, and you DO NOT know if it is restorable

Restorability issue (and test)
Can you, concretely, recreate all the files that are present, and do these files contain exactly the data they should contain (i.e., are the hashes OK)?
If you are really paranoid, that is, you want to make sure that really everything is okay, you need to extract the contents of the archive, in some temporary folder, and then check each file. This will be a lengthy test (depending on the power of your computer of course) and is normally done on RAMDISK or solid-state disks
Now extract everything (from the intermedie.zpaq) because of -all, inside the temp folder z:\knb, and test everything -paranoid
C:\zpaqfranz\spaz>zpaqfranz t intermedie.zpaq -to z:\knb -all -paranoid
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
franz:-all                                      4
franz:-hw -paranoid
intermedie.zpaq:
7350 versions, 7.631 files, 431.966.159 bytes (411.95 MB)
Extract 25.314.671.936 bytes (23.58 GB) in 14.981 files (0 folders) / 32 T
        97.27% 00:00:00  (  22.93 GB)=>(  23.58 GB)  670.94 MB/sec

FULL-extract hashing check (aka:paranoid)

Total bytes                 25.314.671.936 (should be 25.314.671.936)
Bytes checked               25.314.671.936 (should be 25.314.671.936)
Files to be checked                  7.629
Files ==                             7.629 (should be 7.629)
Files !=                                 0 (should be zero)
Files deleted                        7.629 (should be 7.629)
52.156 seconds (000:00:52) (all OK)

If, for some reason, the temp folder is NOT EMPTY (after the execution) THEN some file will not match, and this is very bad

If you don't want to extract the data to a physical drive (this slows down and also makes tear), but you have large amounts of free RAM, you can use the w command (call help with zpaqfranz h w for some examples)
In this case we want to extract to RAM "fake -ramdisk" and -test with -checksum in multithread (-ssd) -all files with -verbose
You cannot operate this way if you have a file larger, in the archive, than the available RAM (or rather than what zpaqfranz will use)
TRANSLATION
Suppose you have 16GB of free RAM, and every file in the archive is <10GB. Then you can use (zpaqfranz will use 16*0.75=12GB at most).
If, on the other hand, you have a 20GB file in the archive, you will have to operate on a disk drive (i.e. t -paranoid on temporary folder) since there is not enough RAM.
You can use the -frugal switch (of the w command) to minimize RAM usage, or leave it to zpaqfranz (which will take 75% of that reported free by the operating system). Obviously the more RAM, the faster
The free RAM in my PC is ~32GB, we can continue (minimum ram needed is 5.411.726 bytes)
C:\zpaqfranz\spaz>zpaqfranz w intermedie.zpaq -ramdisk -test -checksum -ssd -all -verbose
zpaqfranz v59.1a-JIT-GUI-L,HW SHA1/2,SFX64 v55.1,(2024-01-16)
DETECTED SHA1/2 HW INSTRUCTIONS
franz:-all                                      4
franz:-checksum -hw -ramdisk -ssd -test -verbose
**** CHUNKED EXTRACTION/TEST ****
intermedie.zpaq:
7350 vers, 7.631 files, 9.715 frags, 21.805 blks, 431.966.159 bytes (411.95 MB)
Minimum needed  (+10%)             5.411.726 0763/zpaqfranz.cpp
Free RAM (-25%)               32.891.433.984 (as reported by OS)
Chunks 0001 x                 25.314.671.936 (total decompressed size 25.314.671.936)
====================================================================================================================
Chunk 001/001        25.314.671.936 bytes (23.58 GB) in 7.628 files by 32 threads
RAMDISK              25.314.671.936 bytes (23.58 GB) time 9.16 s @ 2.764.214.013 (2.57 GB/s)
Running 32 threads on 7.628 files
CRC-32        25.314.671.936 [OK] (   36.60 s) @ 659.67 MB/s  UNKN 0 [OK]  GOOD 7.628 [OK]  BAD 0 [OK]
HASHes        25.314.671.936 [OK] (   22.12 s) @  1.07 GB/s  UNKN 0 [OK]  GOOD 7.628 [OK]  BAD 0 [OK]
--------------------------------------------------------------------------------------------------------------------
Bytes expected            25.314.671.936 (  1.88 s) @ 0 B/s
RAMDISK releasing         25.314.671.936 bytes  to 0 (should be zero)
Stage VEF 0001 : errors  0 (0=good)
====================================================================================================================
12.312 seconds (000:00:12) (all OK)

Then there are numerous other verification mechanisms, to be adopted if the original files are available, or using other algorithms etc.
You can find a good part of the explanation in the wiki, or you can ask me directly

Last, but not least, to merge a multipart archive the 2nd way, the repack
In this example
x means... extract
"*" means everything (you want only cpp? do *.cpp or whatever)
-to pippo is a dummy (necessary to do not overwrite locally!)
-space is do-not-care-for-pippo
zpaqfranz x z:\piece_?? * -repack z:\all-in-one.zpaq -to pippo -space

Actually, this repack requires an understanding and familiarity with zpaq archives, let's say at the "poweruser" level, so I don't suggest it for the novice


[opentitantool] Multi-byte read()/write() in serial console

The serial console library suffers from the need not to consume any additional bytes, if one of the regular expressions matches, for which we have found no better solution than to read a single byte at a time.
At Google, we often use opentitantool console and pipe the output into other test scripts (which may do regular expressions).  In such cases, the opentitanlib is not performing any regular expression matching, but it still reading a single byte at a time, and writing each byte using a separate write() operating system call.  This is highly inefficient, when the data goes through anonymous pipes, and is often further processed just one or a few bytes at a time by other processes.
This PR modifies the logic in uart/console.rs to recognize the case of there not being any regular expressions to match, and in that case read up to 1024 bytes at a time from the transport USB device.
I plan on combining this with below enhancements to HyperDebug firmware, to make it preferably compose USB packets with multiple characters captured on its UARTs.
https://chromium-review.googlesource.com/c/chromiumos/platform/ec/+/6020315

Backporting to earlgrey_es_1.0.0 is currently blocked by this other cherry-pick:
https://github.com/lowRISC/opentitan/pull/25498