[YUNIKORN-2719]Insert invalid group name check(draft)

What is this PR for?
Assert invalid group name in Get Group REST API
Here only add the invalid group name check in handlers.go. But there is some issue here. Make it as a draft first.
What type of PR is it?

[ ] - Bug Fix
[x] - Improvement
[ ] - Feature
[ ] - Documentation
[ ] - Hot Fix
[ ] - Refactoring

Todos

[ ] - Task

What is the Jira issue?
https://issues.apache.org/jira/browse/YUNIKORN-2719
How should this be tested?
It should pass all unit tests.
Screenshots (if appropriate)
Questions:

[ ] - The licenses files need update.
[ ] - There is breaking changes for older versions.
[ ] - It needs documentation.

The failed test is related to https://github.com/apache/yunikorn-core/pull/905/files#r1678723246

Finally find， this commit can't pass the unit test, because the TestSpecificGroupResourceUsage failed.
And the error message is YUNIKORN/yunikorn-core/pkg/webservice/handlers_test.go:1785: assertion failed: 404 (http.StatusNotFound int) != 400 (resp.statusCode int): Incorrect Status code.
Then dig into it, find in TestSpecificGroupResourceUsage, there is a test case as below:
``// Test group name with special characters escaped properly
validGroup := url.QueryEscape("test_a-b_c@#d@do:mai/n.com")
req, err = createRequest(t, "/ws/v1/partition/default/usage/group", map[string]string{"user": "testuser", "group": validGroup})
assert.NilError(t, err)
resp = &MockResponseWriter{}
getGroupResourceUsage(resp, req)
assertGroupNotExists(t, resp)
this group name `test_a-b_c@#d@do:mai/n.com` can't pass the InvalidGroupName test, so it returns 400 bad request error, which is different with the GroupNotExists error 404.

this group name test_a-b_c@#d@do:mai/n.com can't pass the InvalidGroupName test, so it returns 400 bad request error, which is different with the GroupNotExists error 404.

Yep, maybe we can assume test_a-b_c@#d@do:mai/n.com is a test case which should fail on RegExp. Hence, please try to update assertGroupNotExists to make test pass

Many thanks for your suggestion. Update a new commit to fix the issue. Now it works as expected.

@SophieTech88 please fix the conflicts!


V0.6.1

hey @BenFradet , I needed to include additional work and update some existing commits after your review. Do you mind taking another look at the PR? I also published 0.6.1-rc1 to both AWS & GCP.

Thanks @BenFradet , yes, we've changed the way we publish artifacts, the process is moved to Travis and BIN.enc is an encrypted file which is decrypted during build process and used by Packer when publishing to GCP.


Frozen screen McAIRPOS

Hi,
I tried to install McAirpos on different configurations (PI zero, Pi4 etc.) and on recalbox 7 and 8 also Retropie and I've the same issue on all different cinfigurations with a frozen screen when i launch a Microsoft Arcade game.
Any ideas - it's probaly an issue with the way Makecode compiles the code ?
I'm downloading the ELF files with  https://vegz78.github.io/McAirpos/
Thank your for your help.
You can find the image of the screen
https://ibb.co/jgk5Cnr
Yann

It might be, as you suggest, a regression of MakeCode Arcade vs ARM Linux, given ARM Linux is not a major concern anymore for MakeCode development, and Linux kreps evolving in its own direction independently.
I have not tried MakeCode Arcade with McAirpos in a while, but could test during the summer if I can reproduce.
Could you please also post here a copy of your /tmp/McAirpos.log after this freeze occurs?
Mvh
Vegard

jun. 2023 kl. 16:38 skrev yannour @.***>:

Hi,
I tried to install McAirpos on different configurations (PI zer, Pi4 etc.) and on recalbox 7 and 8 also Retropie and I'm the same issue. I het a frozen screen when i launch the game.
Any ideas - it's probaly an issue with the way Makecode compiles the code ?
I'm downloading the ELF files with https://vegz78.github.io/McAirpos/
Thank your for your help.
You can find the image of the screen
https://ibb.co/jgk5Cnr
Yann
[IMG-0374]https://user-images.githubusercontent.com/56848737/246616191-cb00627d-cf92-40a1-be15-a1bde6e165db.jpg
—
Reply to this email directly, view it on GitHubhttps://github.com/Vegz78/McAirpos/issues/49, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALWCWWIKBWKTZS76UNT6BALXLW6NDANCNFSM6AAAAAAZKITU6I.
You are receiving this because you are subscribed to this thread.Message ID: @.***>


Naming: radius prop of MatRipple is actually not radius

It is actually diameter. May be it should be size instead? Is it kept for historical reasons? Also what can be done with this problem?

oh very sorry for taking your precious time


Remove 'volunteering' and 'registration' pages from FreeCEN1

Remove Brenda's email address as well as the these pages.

Is there any more to do with this story?

Is there any more to do with this story?

@DeniseColbert to tell Vino what to remove

Brenda's email needs to be removed:

Will look at County pages and have a think

Denise to provide text re data Problem directing to FC2

Denise to provide text and test changes already made on test.freecen.org.uk

https://test.freecen.org.uk/project.htm basically needs replacing wholesale, I will write the text. We must preserve the page via archiving or such.

Done so closing


Please, add check-ansible-centos-centos10-action

The CentOS 10 stream (beta) is out and it would be good to have the role running against the newer system.
I think it should be mostly change of stream9 -> stream10, but I will double-check.

The WIP branch:
https://github.com/roles-ansible/check-ansible-centos-centos9-action/compare/main...Jakuje:check-ansible-centos-centos9-action:centos10
Testing the action in willshersystems/ansible-sshd#304 now.

Sounds like this worked ok:
https://github.com/Jakuje/ansible-sshd/actions/runs/12353611673/job/34473088916
Feel free to use my branch for the new GH action (the icons were not updated though) when you will have time or let me know how I can help there.

thanks a lot for your work @Jakuje
https://github.com/marketplace/actions/check-ansible-centos-centos10


Update proof-of-stake.md

@randall can you include the link to the new "Tallying Rule" anchor you've built into the text Tallying Rule in this sentence: "The $\mu_v$ variable captures the output of the Tallying Rule ...."

This will require some conflict resolution, but also the title was a bit confusing as I originally expected to see documentation changes only (maybe rename title).

Not sure I totally understand this comment. We had to force merge as content updates came in late and were not included in the release pushed to Devnet. Changes are reflected on main https://github.com/MystenLabs/sui/commits/main/doc/src/learn/tokenomics/proof-of-stake.md
Shouldn't be a conflict as the merge was from main so the branches are now in sync.


Mixing Required + Non-Required Options in an ArgGroup -- required options not validated

I have a use case similar to other issues -- options with sub-options. For example:
cli --group=alpha --opt1=1 --group=bravo --opt2=foo
I'm trying something like this:
class Group {
    @Option(names = {"--group"}, required = true) String name;
    @Option(names = {"--opt1"}) int opt1;
    @Option(names = {"--opt2"}) String opt2;
}

@ArgGroup(exclusive = false, multiplicity = "0..*") List<Group> groups;

This almost works:

cli --group=alpha --opt1=1 --group=bravo --opt2=foo
results in
groups = [{name=alpha, opt1=1}, {name=bravo, opt2=foo}]
cli --opt1=1 --group=alpha --opt2=foo (weird that --opt1 comes before --group, but allowed)
results in
groups = [{name=alpha, opt1=1, opt2=foo}]

However, calling cli --opt1=1 results in [{name=null, opt1=2}], with no error. I'd expect an error since the group name is required but not provided, but that doesn't seem to be the case. Is this supported?

Thank you for raising this. That does look odd. I will take a look when I get to my PC.

Sorry, should have mentioned I'm using 4.0.4.

Also, to give some context, I'm trying to build a Java equivalent of a CLI previously implemented in python via Click. It uses command chaining and the command group callback feature to accomplish repeated subcommands. AFAICT, repeated subcommands aren't possible in picocli, but this seems like the closest approach.

@wjohnson5  Thanks for that background. A feature like this ("repeatable subcommands" https://github.com/remkop/picocli/issues/454 ) is on the TODO list but has been shelved until a clear use case comes up. On the surface it does not look that hard to implement but I haven't spent the time to analyse in depth. The Click documentation is certainly helpful.

I have been able to reproduce the issue and found the cause: a bug in the validation logic.

Fixed in master. Can you confirm?
git clone https://github.com/remkop/picocli.git
cd picocli
gradlew clean build publishToMavenLocal

Then test your project with picocli-4.0.5-SNAPSHOT. You should get an error for input like cli --opt1=1.

Confirmed fixed!
Some other comments:

Using exclusive = True with both optional and required parameters probably doesn't make sense as it is right now -- if you change the example to have exclusive = True, calling cli --opt1=1 works and gives [{name=null, opt1=1}], and cli --group=foo --opt1=1 gives [{name=null, opt1=1},{name=foo}]. I'd think, either (1) only the required arguments are exclusive, and it's still an error to specify an ArgGroup option without exactly one of the required args, or maybe (2) picocli should error out if using non-required options w/ exclusive = True.

Any thoughts on changing some error messages?

If num groups > max(multiplicity) (whether the args are optional or required), it's currently:

Error: expected only one match but got (--group=<name> [--opt1=<opt>])={--group=foo --opt1=1} and (--group=<name> [--opt1=<opt>])={--opt1=1}

I like that this shows you the matched groups, but (1) it always says "expected only one match", even if the minimum multiplicity > 1, and (2) it seems unnecessarily to repeat the group spec.
If num groups < min(multiplicity), it's currently:

Error: Group: (--group=<name> [--opt1=<opt>]) must be specified 2 times but was matched 1 times

For consistency, I think both cases should be something like:

Error:  (--group=<name> [--opt1=<opt>]) must be specified [X / at least X / at most X] time(s) but was matched 2 times: {--group=foo --opt1=1}, {--opt1=2}

If missing a required option for a group, it's currently:

Error: Missing required argument(s): --group=<name>

IMO something like

Error: --group=<name> is required in (--group=<name> [--opt1=<opt>]), got: {--opt1=1}

This clarifies that --group=<name> is only needed when specifying --opt1. This also applies for mutually dependent args:

Error: --req-arg1=<foo>, --req-arg2=<bar> are all required in (--req-arg1=<foo>, --req-arg2=<bar> [--opt1=<opt>]), got: {--req-arg1=foo}

I noticed a bug with exclusive args and multiplicity (I can open this as a separate issue). For exclusive=true, multiplicity="2", with a group containing required options --req1 and --req2, passing cli --req1 --req2 fails: must be specified 2 times but was matched 1 times

Thanks for the awesome support! I really appreciate the quick responses and turn around time. (I even searched for an option to donate some amount in appreciation -- if you're open to the idea of donations!)

Thanks for confirming the fix and thanks for the kind words!
Thanks also for letting me know about the other issues. Would it be possible to raise separate tickets for them? That makes it easier for me to track.

Will do. :)


[ShadowrunAnarchy] FR Translation errata

Changes / Comments
As per Shadowrun Anarchy French Edition Manual.

Hello @tanji ! Your Pull Request includes changes to translation files. We will not deny your Pull Request, but we would prefer sheet contributors use Crowdin for translations. To gain access to Crowdin, please submit a translation approval request through our Help Center. Be sure to include the languages you would like to translate, and please use Crowdin for future translation updates! Additionally, we will be changing the way translations work on 11/06/2020, so check out our forum post on those changes for the full breakdown. Thanks for all you do!

Thank you Miguel, I'll check out Crowdin for sure, looks promising!


Update CI URLs to point to new caching infrastructure

Hello there!
This is an automated pull request submitted by @staticfloat to help package authors transition their Julia installation CI setups to the new binary provider URL.  Please take a look at this PR, and if there is a problem or it doesn't download correctly when CI runs, feel free to ping @staticfloat

do you want me to tag a new release with this change?


chore(dep): Upgrade react-dev-utils to support webpack5 (#67)

@chang-ke thanks for updating this!
@gregberge or @jsfez - would love to see this PR get merged when you have time 😀

any updates on this? would love to give this a go on my current setup

Bumping this for visibility, still would love to see this merged

Thanks!

Thank you very much!


comparing objects with is and ==

Noticed that in the snippet below objects are compared using is, not ==.
Though the exact code works wanted to put it out there that it used to lead to issues.
It's seems better now but haven't seen Blender mentioning anywhere that it was resolved completely so it could be still more safe to use == to make sure it's the same object, not is.
https://github.com/PacktPublishing/Python-Scripting-in-Blender/blob/a9aabf003078baa311c8d25890702ef4e6fe9f33/ch2/0500_understanding_user_context.py#L44-L48

Hi, thanks for pointing that out and apologies for coming back so late: I wanted to write a proper reply. I also apologise in case you are already knowledgeable about the subject and find my explanation redundant.
The difference between the two statements is that == evaluates to True when both sides have the same value, is evaluates to True when the both sides are the same thing. A quick example
>>> a = "Hello"
>>> b = "Hello World"
>>> a + " World" == b
True

>>> a + " World" is b
False

But as you wrote, blender 2.78 presents this weird behavior
>>> C.object == C.object
True

>>> C.object is C.object
False

That happens because those python objects are non-unique containers. The actual blender objects are part of the C core app instead. The bpy module consists of "wrappers" that do the following:

take the C object
put it in a python box
give the box to the python interpreter

Wrappers are written C++ and have access to blender internal entities, but they are derived from python C++ classes so they can be accessed in python scripts.
When we type C.object in the console and press enter, blender takes the active object, puts it inside a python box and gives it to the interpreter. Now it should be clear how come the two results used to differ. When asked if
>>> C.object == C.object
python is faced with a box on the left and a box on the right. Will look in one box and then in the other: "hey they contain the same C object, so these boxes are equivalent". It returns True.
But confronted with
>>> C.object is C.object
python doesn't bother to look inside: it's either the same box or not.
The function id() returns the memory address of a python object. Asking for the memory address of C.object repeatedly in blender 2.79 gives these results (every time different numbers, not the same as in this example)
>>> id(C.object)
1955119776328

>>> id(C.object)
1955119776136

>>> id(C.object)
1955119776904

Those would fail the is condition, because it's every time a different box stored at a different memory address.
At a certain release, perhaps 2.8, they improved that policy: blender keeps returning the same box.
Asking for the id repeatedly in blender 3.x returns a consistent memory address (you will get a different number)
>>> id(C.object)
2300011633600

>>> id(C.object)
2300011633600

>>> id(C.object)
2300011633600

I could not find the exact change behind that in the source, but I have talked with the developers and they assured me that, from blender 3 on, the identity between selected_object[i] and C.object holds.
I admit that using identity in my example would have required such explanation, and in the hindsight I should have rather used ==.
Now I would rather not change the accompanying code in order not to confuse the reader, but I'll mention the alternative in a comment. In the meantime, I hope that this explanation can be somehow beneficial
Kind Regards,
Paolo

Yeah, makes sense, thank you.
Some benefit of is that it could be slightly faster too since it's just comparing python objects ids instead of looking up original objects. But it's probably barely noticeable.


Support string type inputs from Python application code.

This PR supports string type inputs from Python application code.

This PR supports utils/RunONNXModel.py and test/backend/inference_backend.py except the JNI cases. we assume that numpy.arrays of strings (e.g. numpy.array([["abc", "d"], ["e", "fghi"]], dtype=np.str_)).
This PR enables two backend tests with string inputs ("test_equal_string_cpu test_equal_string_broadcast_cpu") by "cmake check-onnx-backend." However they are skipped with the emit JNI option.
This PR is similar to PR#2478. The objective might be the same, but the approach to use “object” type for string is different.

The basic strategies are as follows.

At Python side, use dtype=object instead of dtype=np.str_ to handle string values as pointers in order to avoid that they are handled as an array of constant-length strings. No limitations with string length.
At C++ runtime side,

Investigate if the dtype is object in order to check the original data type is string or not. (c.f. https://github.com/pybind/pybind11/issues/1538 )
Convert the array to a vector of strings (e.g.  by  inputPyArray.cast<std::vector<std::string>>(); for 1D array).
Generate OMTensor for each argument by allocating buffers for the strings and copying the strings to the allocated buffer (c.f. strdup is used in the code.) in order to convert c++ vector type multi-dimentional array into c-type one-dimentional array (used in the OMTensor structure).
Invoke the compiled code with the generated OMTensor(s) as input arguments.
Free buffers for the strings in omTensorDestroy.

Confirmed that onnx-mlir can run bidaf-9 model in the model zoo correctly with this PR.
C.f. Input files in the model zoo is not correct, input files of the second and third arguments (e.g.  test_data_set_/input_{1,2}.pb) need to be swapped.
Comments on the basic strategy and testing policies are very welcome.

@gongsu832 This PR does not support JNI. Can you kindly support to enable the JNI part? Or can you create another PR to support string inputs with JNI?

@gongsu832 This PR does not support JNI. Can you kindly give me comments to support to enable the JNI part, or create another PR to support string inputs with JNI?

Let's first decide how users would typically input a tensor of strings for the languages we support. For example, let's say a model specifies an input tensor [d1, d2, d3] of strings, how would a user input it?

for C/C++, it could be a multi-dimensional array of pointers, each pointing to a string, e.g., char *tensor[d1][d2][d3];
for Java, it could be a multi-dimensional array of strings, e.g., Strings[d1][d2][d3] tensor;
for Python, I'm not very familiar with pybind11, how would a user input such a string tensor in Python?

Regardless of how the string tensor is inputted in various languages, ultimately they will all be converted to the OMTensor expected input format, which is an one-dimensional d1*d2*d3 array with shape info. For C/C++, multi-dimensional and one-dimensional arrays don't really differ much since the array elements are all stored contineously in memory in both cases. So conversion between the two is straightforward. But for Java (and for Python also I suspect) the memory layout for multi-dimensional and one-dimensional arrays are very different. So in order to minimize copying, the choice of the input format matters quite a bit. In Java, so far for non-string data, we have used one-dimensional array input, combined with bytebuffer backing, to avoid having to copy input data when crossing Java/native boundary. But I'm not yet sure if we can do the same for string data so I'll have to think a bit. I think we need to do the same for Python, i.e., carefully consider the input format to minimize copying when crossing Python/native boundary.
Also, I would not change the meaning of the _owning flag because of the string tensor. When _owning is true, it means the API will free the tensor data. For string tensor, data means the pointer array plus the strings themselves. So freeing tensor data means freeing both. There shouldn't be a case where the pointer array is freed but the strings are not, or the strings are freed but the pointer array is not.

@gongsu832 Thanks for your variable comments.

Let's first decide how users would typically input a tensor of strings for the languages we support. For example, let's say a model specifies an input tensor [d1, d2, d3] of strings, how would a user input it?
* for C/C++, it could be a multi-dimensional array of pointers, each pointing to a string, e.g., char *tensor[d1][d2][d3];
* for Java, it could be a multi-dimensional array of strings, e.g., Strings[d1][d2][d3] tensor;
* for Python, I'm not very familiar with pybind11, how would a user input such a string tensor in Python?

Yes. Although this PR focuses to support Python, but other languages should be considered.
For Python, we assume that numpy.arrays of strings (e.g. numpy.array([["abc", "d"], ["e", "fghi"]], dtype=np.str_)).

Regardless of how the string tensor is inputted in various languages, ultimately they will all be converted to the OMTensor expected input format, which is an one-dimensional d1d2d3 array with shape info.

For Python, "pybind11" converts muti-dimentional Python arrays into the multi-dimentional std::vector types  in c++(e.g. std::vector<std::vector<std::string>> for 2D), which are different from the OMTensor style.
And this PR uses pointers of strings directly for string data themselves to when crossing Python/native boundary.

So conversion between the two is straightforward. But for Java (and for Python also I suspect) the memory layout for multi-dimensional and one-dimensional arrays are very different.

The code copies strings in the C++ vector types into one-dimentional array with shape info (1) in order to change the array style (from C++ multi-dim vector to C one-dim array) and (2) duplicate buffers for the string data for keeping the string buffers until "omTensorDestroy."  This is only one copy for each string.  I agree that copies should be minimized, and I suppose that this copy in PR are minimum and mandatory.

Also, I would not change the meaning of the _owning flag because of the string tensor. When _owning is true, it means the API will free the tensor data. For string tensor, data means the pointer array plus the strings themselves. So freeing tensor data means freeing both. There shouldn't be a case where the pointer array is freed but the strings are not, or the strings are freed but the pointer array is not.

Yes. exactly. I suppose that there are no such cases now, but I am not sure especially about other language cases.
It might to be better to use bitmasks to manage buffers for pointer arrays and strings independently.
Anyway, we need to manage string data themselves by the OMTensor structure to keep them until "omTensorDestroy", I suppose.
Thank you for your useful comments. I am not sure my comments answers all of your questions, but more comments and suggestions are very welcome!

I think we should make the string tensor behave as similar to non-string tensor as possible. Currently, C/C++ and Java APIs use the combination of an one-dimensional data array with a shape array instead of a multi-dimensional array for the input/output tensors . This has the advantage that the one-dimensional data array is tightly packed in contiguous memory and therefore they can be passed into the model runtime via OMTensor struct without having to copy the data array. As I mentioned before, one-dimensional vs multi-dimensional array doesn't make much difference in C/C++ but matters quite a bit in Java. I I suspect it matters too in Python. I don't know why we used a multi-dimensional array with the Python API. If I were doing it, I would have used one-dimensional data array + shape array as well for Python, if only just for being consistent with other languages.
So back to the string tensor, I think it should also use a combination of one-dimensional string array plus a shape array for input/output string tensors, i.e., something like [ "foo", "bar", ... ] (string data) + [ 2, 3, 4 ] (shape). Internally, this is represented as an one-dimensional array of pointers to string data (with a total of 2*3*4=24 pointers and strings). This makes a string tensor behave "almost identical" to an int64 tensor, with the difference being that after you get the int64 element you need to dereference it to get the actual string data. This representation of string tensor should in most cases allow passing it by manipulating the pointers therefore avoid having to copy the actual string data.
Regarding the _owning flag, I don't think it has anything to do with the frontend language being used. It's a flag for the model runtime API, which is in C.

Thank you for the comments!

I think we should make the string tensor behave as similar to non-string tensor as possible. Currently, C/C++ and Java APIs use the combination of an one-dimensional data array with a shape array instead of a multi-dimensional array for the input/output tensors . This has the advantage that the one-dimensional data array is tightly packed in contiguous memory and therefore they can be passed into the model runtime via OMTensor struct without having to copy the data array. As I mentioned before, one-dimensional vs multi-dimensional array doesn't make much difference in C/C++ but matters quite a bit in Java. I I suspect it matters too in Python. I don't know why we used a multi-dimensional array with the Python API. If I were doing it, I would have used one-dimensional data array + shape array as well for Python, if only just for being consistent with other languages.

I suppose that this PR uses one-dementional data array (of pointers to strings), but does not use multi-dimentional array.
In order to keep the one-dimentional data array structure, this PR copies string data pointers in from the multi-dimentional data array (generated by "pybind11") to the one-dimentional array (used in the original OMTensor structure).
(I inserted some words in the description, I am sorry if my descriptions were not enough.)
In this PR, string data are kept in the one-dimentional data array, and shapes are kept in the shape structure. It is same to  numerical data cases. The only difference is that the one-dimentional data array keeps pointers instead of real values, since the data lengths of string elements in an array are different unlike numerical data cases.

So back to the string tensor, I think it should also use a combination of one-dimensional string array plus a shape array for input/output string tensors, i.e., something like [ "foo", "bar", ... ] (string data) + [ 2, 3, 4 ] (shape). Internally, this is represented as an one-dimensional array of pointers to string data (with a total of 2*3*4=24 pointers and strings). This makes a string tensor behave "almost identical" to an int64 tensor, with the difference being that after you get the int64 element you need to deference it to get the actual string data. This representation of string tensor should in most cases allow passing it by manipulating the pointers therefore avoid having to copy the actual string data.

I suppose that this PR keeps the one-dimentional array structure. A string tensor behaves "almost identical" to an int64 tensor as described in this comment.

Regarding the _owning flag, I don't think it has anything to do with the frontend language being used. It's a flag for the model runtime API, which is in C.

I understand the idea, but onnx-mlir runtime code needs to hold the string data (not pointers) in the OMTensor structure until "omTensorDestroy" is called, because Python applications free string data used for an OMTensor for the first argument before generating another OMTensor for the next argument.
Thank you very much for the comments, and I am sorry if my explanations were not enough or I do not understand your comments. Your comments are important and very welcome to me.

I suppose that this PR uses one-dementional data array (of pointers to strings), but does not use multi-dimentional array. In order to keep the one-dimentional data array structure, this PR copies string data pointers in from the multi-dimentional data array (generated by "pybind11") to the one-dimentional array (used in the original OMTensor structure). (I inserted some words in the description, I am sorry if my descriptions were not enough.)
In this PR, string data are kept in the one-dimentional data array, and shapes are kept in the shape structure. It is same to numerical data cases. The only difference is that the one-dimentional data array keeps pointers instead of real values, since the data lengths of string elements in an array are different unlike numerical data cases.

What I meant is that the Python interface, instead of using a multi-dimensional numpy array, should have used an one dimensional array plus a shape array, like the other languages.

I understand the idea, but onnx-mlir runtime code needs to hold the string data (not pointers) in the OMTensor structure until "omTensorDestroy" is called, because Python applications free string data used for an OMTensor for the first argument before generating another OMTensor for the next argument.

I'm not sure I understand this. Can you provide an example?

What I meant is that the Python interface, instead of using a multi-dimensional numpy array, should have used an one dimensional array plus a shape array, like the other languages.

Thanks. Now, I understand your points. I just wanted to treat multi-dimentional string input data same to multi-dimentional numerical input data.
I am investigating how the current Python test codes (especially utils/RunONNXModel.py and test/backend/inference_backend.py) treats mumti-dimentional (numerical) data.
I suppose that shape array is already passed by the test codes,
and logic for receiving multi-dimentional data array from applications may not be necessary.
I do not intend to change treatment of multi-dimentional array for supporting string data.
I just want to follow the way for numerical data cases, and shape info should be given
We need to manage multi-dimentional
For numerical numbers in Python applications, multi-dimentional numpy array
Current onnx-mlir for numerical numbers compiler uses
In the Python

I understand the idea, but onnx-mlir runtime code needs to hold the string data (not pointers) in the OMTensor structure until "omTensorDestroy" is called, because Python applications free string data used for an OMTensor for the first argument before generating another OMTensor for the next argument.

I'm not sure I understand this. Can you provide an example?

What I meant is that the Python interface, instead of using a multi-dimensional numpy array, should have used an one dimensional array plus a shape array, like the other languages.

Thanks. I have got the point now. My comments are as follows.
(1) The current implementation uses multi-dimensional numpy for input/output Tensors, that is a standard array format in Python.  This PR simply follows the current one for the numerical data cases, does not change the policy including multi-dimensional array treatment.
(2) "numpy" in Python originally has the shape attribute for managing array shape explicitly as same to the OMTensor. The shape attribute defines the shape of numpy. Users gets the array's shape by referring the shape attribute, and changes the shape by modifying the shape attribute.
(3) The "pybind" module and current onnx-mlir runtime module use the numpy shape attribute of input/output arrays to pass their shapes. To utilize them, we need to use a standard numpy shape attribute for multi-dimensional arrays.

What I meant is that the Python interface, instead of using a multi-dimensional numpy array, should have used an one dimensional array plus a shape array, like the other languages.

Thanks. I have got the point now. My comments are as follows.
(1) The current implementation uses multi-dimensional numpy for input/output Tensors, that is a standard array format in Python. This PR simply follows the current one for the numerical data cases, does not change the policy including multi-dimensional array treatment. (2) "numpy" in Python originally has the shape attribute for managing array shape explicitly as same to the OMTensor. The shape attribute defines the shape of numpy. Users gets the array's shape by referring the shape attribute, and changes the shape by modifying the shape attribute. (3) The "pybind" module and current onnx-mlir runtime module use the numpy shape attribute of input/output arrays to pass their shapes. To utilize them, we need to use a standard numpy shape attribute for multi-dimensional arrays.

I understand you are trying to follow the current way of using numpy multi-dimensional array for the Python wrapper. But that's exactly what I'm questioning, that whether this is the best way of doing it. I think using a multi-dimensional numpy array actually makes things unnecessarily complicated because you have to "flatten" it to what OMTensor expects, an one-dimensional array. If you had used an one-dimensional array (plus a shape array) to begin with, pybind11 probably would have converted it to a vector instead of vector of vector. So there is no need to "flatten" it further. In addition, you would also be able to handle arbitrary number of dimensions instead of hardcode a maximum 4 dimensions in PyExecutionSessionBase.cpp, which looks rather hacky.

I understand you are trying to follow the current way of using numpy multi-dimensional array for the Python wrapper. But that's exactly what I'm questioning, that whether this is the best way of doing it.

Because "pybind11" uses shape defined the shape attribute of numpy, we cannot flat the multi-dimensional arrays in Python code to keep the original shape in order to utilize the pybind code.  According to my understanding, all special treatments for numpy is done in runtime code for Python in "src/RuntimePyExecutionSessionBase.cpp".
In the case of Python, in comparing with other languages, it is considered that the runtime code and pybind11 module are a part of Python application to prepare shape and uni-dimensional arrays.

I think using a multi-dimensional numpy array actually makes things unnecessarily complicated because you have to "flatten" it to what OMTensor expects, an one-dimensional array. If you had used an one-dimensional array (plus a shape array) to begin with, pybind11 probably would have converted it to a vector instead of vector of vector.  So there is no need to "flatten" it further. In addition, you would also be able to handle arbitrary number of dimensions instead of hardcode a maximum 4 dimensions in PyExecutionSessionBase.cpp, which looks rather hacky.

Yes, exactly. As described in your comments, for the numerical data cases, we need NOT to flatten multi-dimensional array manually, I tried to follow the same way for string data cases, but I could not follow the way for string data because of a pybind11 issue.  I will update the code if we find better way to avoid the issue or pybind11 fixes the issue.
I appended the following comments in the code to explain the situation.
      //
      // Convert multi-dimensional array (for string data pointers) to
      // one-dimensional array to mange multi-dimensional array in an
      // integrated way.
      //
      // For numerical array, pybind11 can convert multi-dimensional array into
      // one-dimensional array without manual conversion, but pybind11 has
      // an issue about "dtype caster does not accept strings or type objects"
      // (https://github.com/pybind/pybind11/issues/1538). The issue page
      // shows a solution to avoid the bug, but it does not work for our case.
      // The following part solves the issue temporally, and will be updated
      // if we find better way to avoid the issue or pybind11 fixes the issue.
      //

The current PyExecutionSessionBase::pyRun function
std::vector<py::array> PyExecutionSessionBase::pyRun(const std::vector<py::array> &inputsPyArray)

takes a vector of py::array, which is converted from numpy.ndarray by pybind11. What I was trying to say is that there is nothing that says we have to do this. We could always make the function take two vectors, one is the inputsPyArray like we have now but the original numpy.ndarray is flattened, and the other a new shapesPyArray that carries the shape of the original unflattened numpy.ndarray.
This way you will be able to handle arbitrary dimension without having to hardcode a maximum supported dimension. That part of the code is really ugly. Yes this means that more test code will have to be changed, e.g., code like
outputs = session.run(inputs)

will need to be changed to something like
outputs = session.run(map(lambda t: t.flatten(), inputs), map(lambda t: t.shape, inputs))

But IMO this is the right way to do it.

A side question, for an integer or float input tensor, do we have to allocate a new buffer and copy the data if flattened? I mean to use the data pointer from np.array. Is there any copy when py.array is converted to np.array by pybind? The reason I am asking these question is that the input tensors for KV caching may be quite large.

A side question, for an integer or float input tensor, do we have to allocate a new buffer and copy the data if flattened? I mean to use the data pointer from np.array. Is there any copy when py.array is converted to np.array by pybind? The reason I am asking these question is that the input tensors for KV caching may be quite large.

Good question. When you flatten a numpy.ndarray with flatten(), it always makes a copy. So a better function is ravel(). According to its doc, it returns an one-dimensional "view" of the numpy.ndarray and only makes a copy if needed. Although it's not clear from the doc what condition makes the copy needed, my guess is that as long as the multi-dimensional array is contiguous in memory, ravel() doesn't make a copy.
When pybind11 converts the numpy.ndarray to a native array, it doesn't make a copy unless the array is immutable, according to the code in PyExecutionSessionBase::pyRun (https://github.com/onnx/onnx-mlir/blob/552c2c1cc0b3674f8ba3d7a836d2be7e3a1449f6/src/Runtime/PyExecutionSessionBase.cpp#L78-L89)

@jenkins-droid test this please

@gongsu832 > This way you will be able to handle arbitrary dimension without having to hardcode a maximum supported dimension. That part of the code is really ugly. Yes this means that more test code will have to be changed, e.g., code like outputs = session.run(inputs)  will need to be changed to something like outputs = session.run(map(lambda t: t.flatten(), inputs), map(lambda t: t.shape, inputs))
Thanks for the comments. I fixed this PR according to your comments, so that this PR does not have hardcoded maxmum supported dimension now.  I changed codes for calling session.run with the following three arguments. All arguments are numpy.ndarray to utilize the pybind11 module, which supports numpy.ndarray (not list of Python).

"data": numpy.ndarray of the flatten input data.
"shape": numpy.ndarray of the original input shape.
"strides": numpy.ndarray of the original input stride.

run_inputs = []
run_shapes = []
run_strides = []
for inp in inputs:
    run_inputs.append(inp.flatten())
    run_shapes.append(np.array(inp.shape, dtype=np.int64))
    run_strides.append(np.array(inp.strides, dtype=np.int64))
outs = sess.run(run_inputs, run_shapes, run_strides)

I hope that updated PR resolves your concerns. Thanks.

@negiyas Thanks for the updates. It's much nicer without the hardcoded dimension limit. In order to use this new Python API, I think py_om_execution_session.py should become part of the src/Runtime (perhaps renamed to something like PyOMExecutionSession.py) and gets installed along with other runtime libraries. Although it should probably get installed into python package directory instead of /usr/local/lib using the python setuptools (we can address that in a separate PR). So users will just do
from PyOMExecutionSession import PyOMExecutionSession

in order to use the Python API. And more Python APIs can be added to it in the future.

@gongsu832 Thanks for your useful comments.

Although it should probably get installed into python package directory instead of /usr/local/lib using the python setuptools (we can address that in a separate PR).  in order to use the Python API. And more Python APIs can be added to it in the future.

I agree that 'PyOMExecutionSession.py' should be installed as a python package to enable users to use  import PyOMExecutinoSession simply.
If you are O.K., I would like to keep this PR without using python setuptools, and will create another PR to install Python packages. Accordingly more Python API can utilize the new APIs in the future.
Your comments are very welcome.

@jenkins-droid test this please

@gongsu832 Thanks for your useful comments.

Although it should probably get installed into python package directory instead of /usr/local/lib using the python setuptools (we can address that in a separate PR).  in order to use the Python API. And more Python APIs can be added to it in the future.

I agree that 'PyOMExecutionSession.py' should be installed as a python package to enable users to use import PyOMExecutinoSession simply.
If you are O.K., I would like to keep this PR without using python setuptools, and will create another PR to install Python packages. Accordingly more Python API can utilize the new APIs in the future.
Your comments are very welcome.

@negiyas It's fine to have another PR to deal with the setuptools part. But I think PyOMExecutionSession.py should be under a more logical directory such as src/Runtime/python. The current directory structure is such that src/Runtime has the C/C++ APIs, src/Runtime/jni has the Java APIs (I'm thinking src/Runtime/java may be a better name). So it seems logical to put Python APIs under src/Runtime/python.

@jenkins-droid test this please

@gongsu832 > It's fine to have another PR to deal with the setuptools part. But I think PyOMExecutionSession.py should be under a more logical directory such as src/Runtime/python. The current directory structure is such that src/Runtime has the C/C++ APIs, src/Runtime/jni has the Java APIs (I'm thinking src/Runtime/java may be a better name). So it seems logical to put Python APIs under src/Runtime/python.
Thanks for your your comments. I am moving PyOMCompileExecutionSession.{hpp,cpp} and other existing Python-related codes (e.g. PyExecutionSession.{hpp,cpp}, PyExecutionSessionBase.{hpp,cpp}) under src/Runtime to src/Runtime/python.

@gongsu832 I am sorry, but I cannot understand your comments about python package fully.
gongsu> In order to use this new Python API, I think py_om_execution_session.py should become part of the src/Runtime (perhaps renamed to something like PyOMExecutionSession.py) and gets installed along with other runtime libraries. Although it should probably get installed into python package directory instead of /usr/local/lib using the python setuptools (we can address that in a separate PR). So users will just do from PyOMExecutionSession import PyOMExecutionSession in order to use the Python API. And more Python APIs can be added to it in the future.
I have questions about the comments.
(1) Does "python package" mean so-called python package installed by pip.
(2) What does "python package directory" mean?
I am happy if you kindly give me more information about what should we do to apply your comments.

Imagine you do a "make install" then delete the onnx-mlir source repo. In order to be able to use the python API, your python wrapper and its related pybind11 .so need to be installed into a directory searched by python. When you install them using setuptools (on top of which pip runs), they are typically installed under /usr/local/lib/python3.xx/dist-packages.

@gongsu832 Thanks for comments about python installation issue.
Since we agreed that the issue will be managed by another PR, I hope that this PR can be merged at first.
Can you kindly teach me what are necessary items for this PR to be merged?

@negiyas I think the last thing left to be done for this PR is to handle py_om_execution_session.py a bit more gracefully without a "make install".
Currently you have py_om_execution_session.py in two places, backend and utils. Basically, you need py_om_execution_session.py in every directory where you use it. That's a bit cumbersome. Instead, I think you should have a single copy of py_om_execution_session.py under src/Runtime/python and have it copied to CMAKE_LIBRARY_OUTPUT_DIRECTORY (which is set to ${CMAKE_BINARY_DIR}/$<CONFIG>/lib) during cmake. That directory is search by most of our python code by doing a sys.path.append.

@gongsu832 Thank you so much for your very helpful comments.
I updated this PR to apply your comments as follows.

Integrated two py_om_execution_session.py files into "src/Runtime/python/py_om_execution_session.py".
Add the following lines in "src/Runtime/python/CMakeLists.txt" to copy the python code in the target machine.

file(GENERATE
  OUTPUT ${CMAKE_LIBRARY_OUTPUT_DIRECTORY}/py_om_execution_session.py
  INPUT ${CMAKE_CURRENT_SOURCE_DIR}/py_om_execution_session.py
  )

@negiyas Thanks for the updates. Nitpicking a bit...can we rename py_om_execution_session.py to PyOmExecutionSession.py to be consistent with the naming of other files?

@gongsu832 Thanks. I renamed py_om_execution_session.py to PyOMExecutionSession.py.

@AlexandreEichenberger I have a bigger question, which I would like to have addressed.
Thanks for your comments. I am changing

2 Questions:
(Q1) Why is there not also a PyOMCompileExecutionSession? It feels just as needed as for the execution alone?
(Q2) Do you ever anticipate the user to need to access the old classes (the one with the 3-argument run class)? To me, it looks like that is not the case. Users will want to use the PyOMExecutionSession (and PyOMCompileExecutionSession) because they are more convenient.

The answers are yes and yes.

So if the answer is yes and yes to these 2 questions, I would recommend to rename the PyOM... class to drop the Py so as to preserve the old name; and rename the OM... class as OMInternal... as they are only used internally to provide our support and are not meant to be used directly.

Yes. I agreed with @gongsu832 to change the argument number of "OMExecutionSession" function in the "PyRuntime" module from one to three in order to keep language-independent interface consistent to other languages.
To implement the agreement,

At first, I changed the "OMExecutionSession" function.
Then, I introduced "PyOMExecutionSession" function for Python programmers' convenience.

However, I realized that
"OMExecutionSession" module implementation at first, then I introduced "PyOMExecutionSession" for user convenience later.
Since I kept the old module name("PyRuntime") and function name("OMExecutionSession"), and added a new module to hide the API change
As a result of that, there is no need to change any of the existing code base as we preserve the same (external) interfaces of having only 1 parameter for the run operation.

Quick question: does this new 2 step interface have an impact on performance? E.g. does it increase the number of data copy (as they are possibly very large), especially for the NON-String inputs?

Please respond to these questions, and depending on the answer, change the code accordingly.

@AlexandreEichenberger Thanks for your thorough comments.

I have a bigger question, which I would like to have addressed.

Let me answer this bigger question at first. I will answer other questions next.
Basically speaking, I updated this PR, which preserves the external Python interfaces and still has the same internal("pybind") interface with three arguments.

Before: we had OMExecutionSession and OMCompileExecutionSession python classes to perform execution and compilation&execution.
Then you also added a new python class, PyOMExecutionSession that provides back the 1 parameter run method.

Thanks for the summary. It is correct.

2 Questions:
Why is there not also a PyOMCompileExecutionSession? It feels just as needed as for the execution alone?
Do you ever anticipate the user to need to access the old classes (the one with the 3-argument run class)?

The answers are yes and yes.

As a result of that, there is no need to change any of the existing code base as we preserve the same (external) interfaces of having only 1 parameter for the run operation.
Now you have added (for reasons that @gongsu832 and you agreed on, I assume) to modify the run method of these 2 python class to take 3 parameters instead of 1.

Yes. I agreed with @gongsu832 to have class with 3-argument run class for execution, and developed this PR based on your summary.  Now I have realized that we can develop this PR, and have updated it, which preserves the external Python interfaces, and still has the same internal("pybind") interface with three arguments.

Change the PyRuntime/PyCompileAndRuntime module names to PyRuntimeC/PyCompileAndRuntimeC, which have
OMExecutionSession/OMCompileExecutionSession class with 3-argument "run" function.
Introduce PyRuntime.py/PyCompileAndRuntime.py, which has OMExecutionSession/OMCompileExecutionSession class with 1-argument run function, which preserve the existing Python interfaces fully.

I hope that the latest PR answers most comments of @AlexandreEichenberger  and @gongsu832.
Further comments on the latest PR are very welcome!

Quick question: does this new 2 step interface have an impact on performance? E.g. does it increase the number of data copy (as they are possibly very large), especially for the NON-String inputs?

I suppose that this PR has a negligible impact on performance. Although we need to pass two additional arguments (shapes and strides) though the "pybind" module, the arguments are quite small (1D with input rank size) and are passed only once at whole module execution.

PS: I see the python format failing for other PRs as well, so no need to worry about this.\
I asked @gongsu832 to look into why it reports a failure

LGTM, thanks for the changes, I am grateful that you found a way to not change the external interface while satisfying the needs for the 3-arguments run method. Please confirm that it is indeed the case.

I think there might be some misunderstanding. The external interface will change to be 3 arguments. The original 1-argument interface uses shape info embedded in the numpy array and the python-to-native conversion is done by pybind11 directly. But it has trouble supporting string data type because the multi-dimensional array must be "flattened" to 1-dimension by copying each dimension manually in C/C++ since pybind11 currently does not directly support string data type. With the 3-argument interface, the numpy array is "flattened" in the new python wrapper (and hence the need for additional arguments to carry shape info) and this "flatten" typically is a simple "view change" without copying. This 3-argument interface is also more similar to C/C++ and Java interfaces.

PS: I see the python format failing for other PRs as well, so no need to worry about this.\
I asked @gongsu832 to look into why it reports a failure

The problem is that MacOS is now using a newer version of black 24.1.0. So on your local machine you should upgrade black to that version.

@jenkins-droid test this please

@gongsu832 I updated the black command, and also updated format of utils/analyze-simd.py and utils/pre-onnx-mlir.py to fix the black format errors. Thanks.

@gongsu832 The external interface will change to be 3 arguments. The original 1-argument interface uses shape info embedded in the numpy array and the python-to-native conversion is done by pybind11 directly. But it has trouble supporting string data type because the multi-dimensional array must be "flattened" to 1-dimension by copying each dimension manually in C/C++ since pybind11 currently does not directly support string data type.

Yes. Exactly. Thanks for pointing out this issue. I remember this issue again now:-).
@gongsu832 Thank you again for reviewing this complicated PR!

Jenkins Linux amd64 Build #13930 [push] Support string type inpu... started at 22:09

Jenkins Linux s390x Build #13957 [push] Support string type inpu... started at 23:09

Jenkins Linux ppc64le Build #12954 [push] Support string type inpu... started at 23:18

Jenkins Linux s390x Build #13957 [push] Support string type inpu... passed after 1 hr 27 min

Jenkins Linux amd64 Build #13930 [push] Support string type inpu... passed after 1 hr 31 min

Jenkins Linux ppc64le Build #12954 [push] Support string type inpu... passed after 1 hr 51 min

After pulling in this change, I'm noticing that the PyRuntime files are coming out as PyRuntimeC.cpython-*-linux-gnu.so Also to get our Python examples to work, I have to do from PyRuntimeC import OMExecutionSession
As is, that would then break existing client applications. Their programs would need to be updated for the change. Also I haven't tried it yet, but I'm guessing the onnx-mlir examples at https://github.com/onnx/onnx-mlir/blob/main/docs/mnist_example/mnist-runPyRuntime.py#L4 are broken because it still has from PyRuntime instead of from PyRuntimeC
If it breaks clients, then that should be a major version change (ie 0.5.0) for onnx-mlir. Is that desired? For zDLC, after generating the PyRuntimeC.cpython-*-linux-gnu.so, I can probably rename the PyRuntime.cpython-*-linux-gnu.so to avoid breaking exploiters but I wonder if we should revert the name back in onnx-mlir as a whole.

Sorry I didn't notice this when reviewing the PR. I think the change is due to the original PyRuntime generated by pybind11 conflicting with the new PyRuntime.py wrapper. @negiyas I think we should keep the original PyRuntime in order not to break the existing python clients. We can perhaps name the new python wrapper PyOMRuntime.py.
@cjvolzka keep in mind that the original PyRuntime will not support string data type. To use string data type, you must switch to PyOMRuntime.

I can probably rename the PyRuntime.cpython-*-linux-gnu.so to avoid breaking exploiters but I wonder if we should revert the name back in onnx-mlir as a whole.

Actually I was wrong on this. I missed that the .run() interface changed. So that wouldn't actually work to avoid breaking exploiters. See comment at https://github.com/onnx/onnx-mlir/pull/2629#discussion_r1476594493


Dictyostelium discoideum missing from Organism facet, apparently no dicty annotations

Hi Seth,
I'm always annoyed that all kind of fungus and stuff is in the list right away, but always had to search for Dicty. Now even that search returns nothing. what happened?
Thanks,
Petra

Is this still happening? Does the following link work for you?
http://amigo.geneontology.org/amigo/search/annotation?q=dicty*&fq=taxon_subset_closure_label:"Dictyostelium discoideum"&sfq=document_category:"annotation"

I would swear that I was able to duplicate this about twenty minutes ago, then, when checking again to write the ticket (and possibly connect it to #400 ), I things worked cirrectly and I could find the annotations.
My guesses at this point are along the lines that somehow a page got cached from during a load when dicty was not yet in the index, and we managed to get it; the page/cache is not refreshed.

Ah, sorry, yes, the link works. After some trying, I noticed that I used a child term of the term Dicty has annotations to. Only, if there are no annotations to the term, a message to that affect could help? I was first convinced we must have annotations and saw blank screen, no message in the pop up where you start typing your organism name.
Also, is the ready list when you open the Organism tab by usage? Now Dicty is in the list right away and wondered why I always need to search. Maybe don't do it often enough

Dicty, at this point, is cut off as the list is ordered by number of annotations. You can bookmark various searches to start with, so you can jump right to the relevant data for your organism. For example:
annotations: http://amigo.geneontology.org/amigo/search/annotation?q=:&fq=taxon_subset_closure_label:%22Dictyostelium%20discoideum%22&sfq=document_category:%22annotation%22
gps: http://amigo.geneontology.org/amigo/search/bioentity?q=:&fq=taxon_subset_closure_label:%22Dictyostelium%20discoideum%22&sfq=document_category:%22bioentity%22
I do, however, think something was going wrong there temporarily at production--I would swear that I found no dicty annotations for a short period.


Problem with default locations

Flyway#locations property cannot be empty.

Flyway.java#L81

private Locations locations = new Locations("db/migration");

Flyway.java#L608

public void setLocations(String... locations) {
    this.locations = new Locations(locations);
}

Therefore, CompositeMigrationResolver always create default migration resolvers.

Flyway.java#L1219
CompositeMigrationResolver.java#L72

for (Location location : locations.getLocations()) {
	migrationResolvers.add(new SqlMigrationResolver(dbSupport, classLoader, location, placeholderReplacer,
......

If I use only custom migration resolvers, I delete default locations forcedly.

Example

// create flyway
Flyway flyway = new Flyway();
flyway.setSchemas("test");
flyway.setDataSource(new DataSource());

// clear default locations
Locations locations;
try {
	Field field = flyway.getClass().getDeclaredField("locations");
	field.setAccessible(true);
	locations = (Locations) field.get(flyway);
}
catch (Exception e) {
	throw new IllegalStateException("Failed to get locations", e);
}
locations.getLocations().clear();

// set custom resolvers
flyway.setResolvers(new CustomMigrationResolver(......));

// start migration
flyway.clean();
flyway.migrate();

To use custom migration resolvers quickly, is there an easy way to delete default locations? Or default locations should be empty?

locations cannot be null but can well be empty.
flyway.setLocations();

will effectively replace your
// clear default locations
Locations locations;
try {
    Field field = flyway.getClass().getDeclaredField("locations");
    field.setAccessible(true);
    locations = (Locations) field.get(flyway);
}
catch (Exception e) {
    throw new IllegalStateException("Failed to get locations", e);
}
locations.getLocations().clear();

Thus not a bug IMHO.

@mgurov Thanks for the analysis. I also wasn't able to reproduce this.
@hkz-dev Please reopen with an exact test case to demonstrate the problem should it reappear.

Sorry for mistaken issue. It was my oversight about varargs.
And configuration 'skipDefaultResolvers' in Flyway 4.0 cleared up this issue without regard of locations.
Thank you!!


📖 Added CODE_OF_CONDUCT.md

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hello @derekkramer,Thanks again for the PR! We want to show our appreciation by sending                                              you some swag. Could you please fill out                                              this form so we can send it to you? Thanks!                                             Team SendGrid DX

@derekkramer
If you are interested, we would love the opportunity to talk to you about Hacktoberfest and our API libraries.
Our agenda would be: Explore what you liked and is there anything we can do to improve?
You can grab a time on my calendar that works for you and we can have a chat on Google Hangout or Skype. If you prefer, you can email me using my GitHub username at my company’s domain.
Thank you so much,
Matt Bernier - @mbernier - SendGrid Developer Experience Product Manager
Elmer Thomas - @thinkingserious - SendGrid Developer Experience Engineer
@SendGrid

Team SendGrid DX,
I’m emailing regarding the special edition swag that you offered to contributors during Hacktoberfest 2017. I contributed to one of your repos on GitHub and received notice that my pull request was merged. I haven’t yet received the swag however, I was wondering if that was still in the process of being sent or if I may have been missed in the list of contributors. Thanks!
Best,
Derek

Derek Kramer
(303) 506-8669
<EMAIL_ADDRESS>derekkramer.co
linkedin.com/in/derek-kramer
github.com/derekkramer

On Oct 16, 2017, at 7:38 PM, Elmer Thomas<EMAIL_ADDRESS>wrote:
Hello @derekkramer https://github.com/derekkramer,
Thanks again for the PR!
We want to show our appreciation by sending you some swag. Could you please fill out this form https://dx.sendgrid.com/swag so we can send it to you? Thanks!
Team SendGrid DX
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/sendgrid/nodejs-http-client/pull/12#issuecomment-337091769, or mute the thread https://github.com/notifications/unsubscribe-auth/AYdn6wi4jZ4k-qlE5GuMETi8QjeD97EAks5stAUUgaJpZM4P0nht.

@derekkramer Did you fill out the swag form on your PR? If so, the swag is coming - we had such a large response that we had to do another order and get it to our fulfillment house, so that we could send out what was promised.


currency converter,rock paper scissor and algorithm visualizer

I would like to add some basic javascript projects (like currency converter ,rock paper scissors) and also algorithm visualizer which was build using React js to web development folder.Please assign this issue to me.
Please assign this issue to me

Hey I am the mentor of this project
@bhavanamn  Hello Thank you for opening the pull issue and you can opening new issue by creating  the new concept those are not  exist in the project !!
@Kushal997-das kindly close the issue since it already exist in our project.


Custom templates: "No templates available."

Description
I'm using custom templates, and i can see that the request to /api/templates?key=containers returns my JSON, but the App Templates page just says "No templates available."

Set custom templates URL in settings
Go to App Templates page

Technical details:

Portainer version: 1.13.2
Target Docker version (the host/cluster you manage): 17.05.0-ce
Platform (windows/linux): linux
Command used to start Portainer: /usr/bin/docker run --name portainer -v "/var/run/docker.sock:/var/run/docker.sock" -v "/srv/portainer:/data" -p 6000:9000  portainer/portainer
Target Swarm version (if applicable): -
Browser: Firefox 53.0.3 64bit

It doesn't work with 1.13.1 and 1.13.0 either (when using --templates), but works fine with 1.12.4

@jangrewe could you share your JSON definitions?

Here's an abbreviated version which doesn't work:
[
  {
    "title": "Registry",
    "description": "Docker image registry",
    "logo": "https://faked.org/portainer/logos/registry.png",
    "image": "registry:latest",
    "ports": [
      "5000/tcp"
    ],
    "volumes": ["/var/lib/registry"]
  }
]

I had a similar issue when setting up custom templates.
For me adding the platform of the template to the JSON def seemed to fix this and my templates appeared without issue.

I've been able to reproduce this, indeed it seems that adding  the platform field solves the problem. Will fix as this field should be optional.

Thanks, adding this field fixes it!

I'll keep this issue open as the platform field is documented as optional. Will be fixed in the next release.


Corrections for Utah Addresses

Utah Addresses have several core fields missing, rendering grid addresses in Salt Lake City unsearchable. This means we were missing out on ~ 475,000 addresses for the state.
This cleanup adds:

Address number suffix (123 1/2 fake street)
Prefix Street directionals (123 S fake street)
Suffix Directionals (123 S Fake Street E)

Bonus points:

City
State
Parcel ID's

Huge shoutout to @stephenkhess for actually finding this issue, who along with @orangejulius helped write the importer with me.

Looks like it's passing now.
We're currently missing 36 addresses that are just "numeric housenumber", "numeric streetname". Some of those are in there, but as fulltext. Maybe worth writing a regex for em?


how to set proxy on before plugin install/update

I can export http_proxy, but I need type that every time I update my plugins.
I workded in a internal net environment, It's not able to switch proxy on all the time.

let g:plug_url_format

Thanks, I've solved it by a small script which toggle proxy before and after plug installing.

哪个脚本？

@maxrisk You can put the following in your vimrc/init.vim which has worked for me.
call system('export http_proxy=')
call system('export https_proxy=')


Make "fields" in FormState not a private member

Use case

I want to implement feature that when form validated fail scroll to that FormField.
currently I implement with copy form.dart file  and change _field to field to make it public
and do some validation like document said
and scroll to specify field with
formKey.currentState.validate();
Scrollable.ensureVisible(formKey.currentState.fields.first.context,
        duration: Duration(milliseconds: 500));

Proposal

form.dart
...
class FormState extends State<Form> {
  int _generation = 0;
  final Set<FormFieldState<dynamic>> fields = <FormFieldState<dynamic>>{};
...

What do you think ?
Thank you.

Hi @panudetjt
It's not clear if you're asking a question or proposing a change  in  form validation
Can you please explain in detail and provide some uses for this proposal
Thank you

Thank you for respond @TahaTesser and sorry for unclear detail.
When validate the form I cannot access to the Set<FormFieldState> _fields from the GlobalKey<FormState> because it's private member of FormFieldState.
I need to access that variable because when form validated in _fields they have FormFieldState.hasError and BuildContext that Scrollable.ensureVisible required to perform scroll to that field.
I don't know if the flutter have a better solution.
I'm sorry for very bad english.
Thank you 😊

If anyone has a need for this, a public fields property of FieldState, please give a 👍 to the issue above.  Also be aware that you can probably find the field you need by maintaining its key instead, as in this example.
I closed the PR https://github.com/flutter/flutter/pull/67517 but wanted to keep this issue open for awhile to gauge interest for this feature.


Prepend arg for defaults-file

Overview
The mysql CLI depends on the --defaults-file=foo argument being the very first argument.  This change provides a new 'PrependArgs' method and uses it for the mysql plugin.
From the mysql CLI help text:
The following groups are read: mysql client
The following options may be given as the first argument:
--print-defaults        Print the program argument list and exit.
--no-defaults           Don't read default options from any option file,
                        except for login file.
--defaults-file=#       Only read default options from the given file #.
--defaults-extra-file=# Read this file after the global files are read.
--defaults-group-suffix=#
                        Also read groups with concat(group, suffix)
--login-path=#          Read this path from the login file.

Type of change

[ ] Created a new plugin
[ ] Improved an existing plugin
[x] Fixed a bug in an existing plugin
[ ] Improved contributor utilities or experience

Related Issue(s)

Resolves: #406

How To Test
Build the plugin locally and run op plugin run mysql -- --host=foo using a credential with no host field.  Before this PR, it results in the error: mysql: [ERROR] unknown variable 'defaults-file=... because the MySQL treats unknown arguments in the form --variable-name=foo as MySQL configuration variable. After the PR, additional command-line arguments are appended after --defaults-file that is injected by the mysql plugin.

Changelog

The mysql CLI depends on the --defaults-file=foo argument being the very first argument

Added my GPG key to Github -- commits are signed

Any chance to get this approved and released ?  The plugin is otherwise broken for all but the simplest usecases.

@AndyTitu can you help with getting a second approver?  Any chance you could add a CODEOWNERS or some other mechanism to help find the right reviewers in the future?

Any maintainers able to merge to get this officially into next 1password-cli release ?

@SimonBarendse this seems ready to go. Can you provide guidance if there are any steps missing?

@mrjones2014 I see you are a recent committer/merger -- can you approve and merge this please?


Issue in cmd

Problem encountered on https://dotnet.microsoft.com/en-us/learn/aspnet/hello-world-tutorial/create
Operating System: windows
Provide details about the problem you're experiencing. Include your operating system version, exact error message, code sample, and anything else that is relevant.
Error: Invalid parameter(s):
-f net7.0
'net7.0' is not a valid value for -f (Framework).

@devAshShahi do you have .NET 7.0 installed? What does dotnet --info print?


Syntax error in module 'sync-catalog'

Syntax error in module 'sync-catalog': invalid syntax (init.py, line 399)

I got the same error. The reason was that when I executed the .bin/deploy.sh, it ran this line using my default pip for Python 3.8-
pip install -r requirements.txt -t "$PWD" --upgrade
The code in the blog assumes that you are using pip for Python 2.7.
So be sure that your pip version is for Python 2.7 when you run execute the .bin/deploy.sh file.
I did this on my mac since I had both Python2.7 and Python3.8 installed-
pip install virtualenv
virtualenv -p /usr/bin/python2.7 venv
source venv/bin/activate

When virtualenv of Python2.7 was activated, it used Pip from python 2.7 and installed the correct python packages.
I hope this helps!


Remove requirement for macOS 10.13

The runtime requires macOS 10.13, which is quite inconvenient because all consuming packages now also need to explicitly require 10.13 in the package manifest. It is especially annoying if one tries to conditionally implement Lambda support in a 10.10 package via #canImport, e.g.: https://github.com/Macro-swift/Macro/blob/feature/lambda-1/Sources/http/Lambda/lambda.swift#L9
Slack says this requirement only exists for the ISO date formatter, not sure whether you use it for rendering only or also for parsing. In any case, it should be quite easy to replace w/ a custom parser/renderer. Should be reasonable because the format is fixed and never changes.
It could be built on top of timegm and strptime, though it needs an extra processing step to capture the milliseconds:
import func   Foundation.strptime
import func   Foundation.timegm
import struct Foundation.tm

let s          = "20180905T140903.591680Z"
var parsedTime = tm()

s.withCString { cstr in
  strptime(cstr, "%Y%m%dT%H%M%S.%fZ", &parsedTime) // this needs to extract the ms
}
let time = timegm(&parsedTime)
let ti   = TimeInterval(time) // + milliseconds

You could also provide this, and still use the ISO formatter if available (via if #available(macOS 10.13, ...)), which is what I do in Macro.

I'm a fan of this and think that we should address it. @tomerd wdyt?

+1 conceptually - I dont like the 10.13 constraint either.
do we know  if this can be done in all formats coming form AWS? I think foundation used ICU for some timezone localization aspects which may be more complicated than the one demonstrated here, but it may not be used here

? It’s ISO, there are only really two variants, with ms and without

Oh, timezones, this format doesn’t support timezones (anything locale bound requiring tzdata) , max gmt offsets, though it probably is Z always

right, but IIRC there are other date formats we deal with in the library (unfortunately different AWS events have different date formats) so trying to understand if we can get away with this in all cases

neither dateformatter nor locale require 10.13, it is just the specific isodateformatter, and iso is fixed

works for me!

@tomerd We can go full strptime here or we just create a DateFormatter that has the iso as the dateFormat string like here:
    let formatter = DateFormatter()
    formatter.dateFormat = "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
    formatter.timeZone   = TimeZone(secondsFromGMT: 0)
    formatter.locale     = Locale(identifier: "en_US_POSIX")
    return formatter

https://github.com/fabianfett/swift-lambda-runtime/blob/c306ac8b32a79a42ef8ada855a79930f3d368a2a/Sources/LambdaEvents/SNS.swift#L94

date formatters are really really slow, i kicked them out of shrugs because they have been taking like 40% of the startup time. they are locale aware which is zarro necessary here

i think strptime is fine, but the ms need to be handled separately

ugh, looks like strptime not avail on through Glibc. I guess I could make this Darwin only since we will use the formatters in linux anyways


How thread safety is handle by Celluloid-ZMQ with ffi-rzmq

Fail to find a google group or an irc channel had resisted me to post the query over here.
Looking through the README code. I'm trying to understand how does Celluloid-zmq handle Thread Safety when used with ffi-rzmq.
Googling around I found this https://stackoverflow.com/questions/38773624/thread-issue-in-zeromq-ruby
But no concrete answer for it.
I see the Celluloid-zmq is moved to CZMQ but we are using the version that still uses ffi-rzmq support.

Don't share sockets between threads, thats all there is to it.


:lipstick: Change _App.vue

Little change in the UI
Vue ❤

Thanks a lot for the PR, we will do a proper redesign of the starter when it's closer to 1.0.


CalendarView isn't pagable

After taking a look at the CalendarView and ItemView, I noticed the difference of CalendarView only inheriting from ViewBase while ItemView is a child of PagedView.  I get the following exception for large queries of calendar data:
microsoft.exchange.webservices.data.ServiceResponseException: You have exceeded the maximum number of objects that can be returned for the find operation. Use paging to reduce the result size and try your request again.

And line 141 in CalendarFolder.java seems to always call the initial offset of 0 for any view query.
If there is something I am missing, please let me know, otherwise I wanted to flag this as a potential issue for anyone querying large amounts of calendar data.

We're using service.findItems() for this, not sure what the advantage of CalendarFolder is?

Well if I want to get a set of Appointment objects from a given start and end date, but the number of returned results exceeds the maximum items to be returned.  I would be unable to query this using service.findItems() since the CalendarView doesn't extend PagedView.  Unless I am missing something?

I'm just guessing here, but can't you just cast the Items from service.findItems() to an Appointment?
I agree the findAppointments() method should probably be fixed (well, the CalendarView should), but maybe there are already Appointment objects returned.

I could cast, but the call to service.findItems() would not allow me to filter based off date (as a CalendarView would).

Yes it does, something like this should work: (don't ask me why we're using java.sql.Date here, I'm just copying the code as is and still need to figure this out)
            SearchFilter filter = new SearchFilter.IsGreaterThanOrEqualTo(AppointmentSchema.Start,
                    new java.sql.Date(new Date().getTime()));
            ItemView view = new ItemView(PAGE_SIZE);
            view.setPropertySet(PropertySet.IdOnly);

            FindItemsResults<Item> items;
            do {
                items = service.findItems(getDelegatedFolderId(), filter, view);
...
                if (items.isMoreAvailable()) {
                    view.setOffset(items.getNextPageOffset());
                }
            } while (items.isMoreAvailable());

@dotCipher do you want to create a PR on the CalendarView issue?
@MikeN123 hopefully we dont force the use of java.sql.Date :smile: here.

If you pass an Object into the SearchFilter value, it gets converted to a String when writing the request XML. That's done by calling the toString() method, which returns a yyyy-MM-dd date string in case of a java.sql.Date.
So using a java.sql.Date here is kind of a hack, but it would work. Other formats may or may not be possible and you may or may not need to specify a timezone, depending on your situation.

@MikeN123 I stand corrected, but it might be more concise to have this part of the CalendarView.
@serious6 If it makes sense and agreed on, I would gladly add it.

hmm maybe we can have a default parsing pattern for the dates and also add a constructor with a SimpleDateFOrmat as parameter to enable users to add their own.

@serious6 +1 that seems like a good idea

I don't think we need a constructor with an SDF, it should be enough to overload the method with a Date instead of a Object parameter and do the correct formatting on that object.

Either way, would it make sense to allow CalendarView to contain a page offset?

That 0 is not the one you would need to edit, as the paging is done serverside. CalendarView probably needs to extend PagedView instead of ViewBase for that to work.

I will have some tests on that in the next few days. Besides PR are welcome.

I am fairly certain this is simply correct behavior. The SERVER SIDE of this will not support paging for CalendarView. You have to "fake it", which is a little tricky and error prone

For those looking for a workaround/ faking of the paging, you use do a do-while like this
boolean hasMoreAppointments = true;
do {
    FindItemsResults<Appointment> results = exchangeService.findAppointments(folderId, calendarView);

    // TODO Save results for processing after do-while

    if (results.isMoreAvailable()) {
        Appointment appointment = results.getItems().get(results.getItems().size() - 1);
        calendarView.setStartDate(appointment.getStart());
    } else {
        hasMoreAppointments = false;
    }
} while (hasMoreAppointments);

That is jnfact exactly what I meant by the tricky comment I made :;


Add other parliamentary business

Publish the petitioner emails as items of other parliamentary business.
https://www.pivotaltracker.com/story/show/103804678

LGTM, works on preview


Get player id from player name

Has anyone been able to get a player's id from just their name? I haven't finished it yet but the only way I seem to be able think this will work is look up the player's team, grab the roster and then loop through all the players until I find them. Is there another API available that makes this easier?

That's the best way using this api I believe. That's how the old nhl website used to handle it. They do have a different api that is used on the player search input on the players page I believe, that suggests names based off of what is typed in that will also return the corresponding id. I think it is not allowed from all sources though so it won't work once deployed.

Thanks Daniel


Accessibility in Emails

How can we reproduce this bug?
Any component that generates a <table> element.
What did you expect to happen?
Add role="presentation" to the <table>.
What happened instead?
Without this ARIA Role, screen readers try to read the table as tabular data, which is incorrect.
What email clients does this happen in?
All, when a screen reader is being used.

Most of the time you can add a role="presentation" to the Inky component, in the case of <center>, it does not get applied correctly. Ideally, that role would automatically get applied during the build system to keep the working markup cleaner. I don't think there are any drawbacks to this because if you really wanted to display a <table> as tabular data, you would just build the table using the <table> element, not <container>.
Further Reading: http://blog.rebelmail.com/accessibility-in-email-part-ii/

I'm happy to submit a PR for this, but unsure of the best approach.

I've made a PR in the Inky repo. It might take some time before it gets approved and bubbles all the way into foundation-emails. If there's anyone who can help, that would be splendid, cause these repos seem dead.
Until then...
...since I am not using any tables in my emails other then the generated ones by Inky, for now I just go with a simple task, that I call everywhere after inline. It appends the role="presentation" to each table. I hope this hack helps someone:
return gulp.src('dist/**/*.html')
  .pipe($.if(PRODUCTION, $.replace('<table', '<table role="presentation"')))
  .pipe(gulp.dest('dist'));

Any updates?

Any updates?

No as you can see =)
Any help is very welcome.

I've made a PR in the Inky repo.

Not sure wich one. In such cases please always link it.

Also relevant: https://github.com/foundation/inky/pull/114/files

Also relevant: https://github.com/foundation/inky/pull/114

There it is: https://github.com/foundation/inky/pull/102/files
Also relevant: https://github.com/foundation/inky/pull/114

I've made a PR in the Inky repo.

Not sure wich one. In such cases please always link it.

Linked few lines above. But here you go again. ;)
https://github.com/foundation/inky/pull/102

Bump because this would be an improvement still  💯

+1


Use the real sort order for 'name'

As noted in https://github.com/googlefonts/fontc/pull/314#pullrequestreview-1455005475, apparently I invented a sort order for 'name'. Turns out the spec says something else; do that instead.

Hmm, I thought that write-fonts handled sorting when it generated the table? Is this skipping that somehow?

@cmyr the output changes when I change my sorting so evidently write-fonts is not handling it. To your point, it probably should so perhaps we can fix that then remove sorting here?

Huh I'm very confused, Name stores the records using a BTreeSet which should have a consistent ordering by definition? Something funny is going on, will take a look


iOs FIRMessaging didReceiveMessage not implemented

Hello together, i implemented messaging on iOs but when i send a message in the firebase console
and the device gets the message it gives me this output:
2017-07-17 12:42:24.803 App[1608]  [Firebase/Messaging][I-FCM002019] FIRMessaging received data-message, but FIRMessagingDelegate's-messaging:didReceiveMessage: not implemented
Is it possible that there is a missing function in the c++/ios part?

Thanks for the report. It looks like there was indeed an oversight. I'm working on a fix and I'll leave a comment here when it's available.

Version 4.0.3 of the Firebase C++ SDK has been released, which fixes this issue. You can view the release notes here and download the release from https://firebase.google.com/docs/cpp/setup.

Use Dispatch que
DispatchQueue.main.async(execute: {                         UIApplication.shared.registerForRemoteNotifications()                     })

@saqibomer are you saying you still have issues?

My bad. I had not added FirebaseAppDelegateProxyEnabled and was having that issue. Deleted comment. @stewartmiles


Error when launching from terminal

When launching the application from cmd, the following errors are printed:
std::filesystem fatal error.
what():  canonical: unknown error: "bspguy"
path1(): "bspguy"
path2(): ""
code().value():    2
code().message():  unknown error
code().category(): system

I'm unable to use commands because of this

Fixed in latest build ?
https://github.com/UnrealKaraulov/newbspguy/releases/tag/NightBuild_2024.01.19_09-53

That error no longer happens, but when trying to use the merge command to combine bsps, the application just crashes. Nothing is printed in terminal.

And GUI not showing?

GUI still works just fine

@Outerbeast fixed in latest build https://github.com/UnrealKaraulov/newbspguy/commit/9ad3f7454d9f8f72d46943e63b224708b17012fd ?

Thanks, this is fixed. I can merge maps now.
One remaining issue is that there is no console output during/after the merge process, for instance the new merged bsp stats won't be displayed https://github.com/wootguy/bspguy/wiki/Merging-workflow#first-step

@Outerbeast very strange because I can see this info


add bower.json

It would be nice to include bower.json in order to specify which files bower install ramda should copy to bower_components.

seems to be done


Fastsafetensors w/ GDS performance is much slower than safetensors

Setting:
I'm currently trying to load the Huggingface Zephyr model but the fastsafetensors model loading is much slower.
Please first download the huggingface file first and then copy the filename into the ZEPHYR_MODEL_PATHS location.
huggingface-cli download HuggingFaceH4/zephyr-7b-beta --include "*.safetensors"

Reproduction Script:
import time

import torch
from safetensors.torch import safe_open
from fastsafetensors.loader import fastsafe_open

ZEPHYR_MODEL_PATHS = [
    f"/mnt/local_storage/data/cache/huggingface/hub/models--HuggingFaceH4--zephyr-7b-beta/snapshots/b70e0c9a2d9e14bd1e812d3c398e5f313e93b473/model-0000{i}-of-00008.safetensors" for i in range(1, 9)
]

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
state_dict = {}
cloned_state_dict = {}
start = time.perf_counter()
with fastsafe_open(ZEPHYR_MODEL_PATHS, device="cuda:0", nogds=False, debug_log=True) as f:
    for k in f.get_keys():
        state_dict[k] = f.get_tensor(k)

    end = time.perf_counter()

    f.bufs.close() # this releases memory... so my call to the safetensor run works or else OOM
print(f"fastsafe_open: {end - start} seconds")

start = time.perf_counter()
state_dict = {}
for model_path in ZEPHYR_MODEL_PATHS:
    with safe_open(model_path, framework="pt", device=0) as f:
        for k in f.keys():
            state_dict[k] = f.get_tensor(k)
end = time.perf_counter()
print(f"safe_open: {end - start} seconds")

Results:
fastsafe_open: 14.059521514000153 seconds
safe_open: 2.213277554999877 seconds

Thank you for reporting the issue! To be honest, I have not tried the model with fastsafetensors, but I will analyze it and discuss what we should do later (hopefully this week).

Thank you for the prompt response and your great work!

@BabyChouSr
One thing should be noted here is that safe_open's get_tensor just does mmap, while fastsafe_open actually executes I/O. Please take a look at memory usages around page cache. or do some calculation on the loaded tensors.

I performed a couple matrix multiplications and there was not much performance difference. In terms of memory usage: here are some stats using the tensorizer.utils function from coreweave repo:
Fast safe open
Memory usage before:  CPU: (maxrss: 665MiB F: 8,089MiB) GPU: (U: 192MiB F: 22,299MiB T: 22,491MiB) TORCH: (R: 0MiB/0MiB, A: 0MiB/0MiB)
Memory usage before:  CPU: (maxrss: 924MiB F: 7,842MiB) GPU: (U: 15,062MiB F: 7,429MiB T: 22,491MiB) TORCH: (R: 14,852MiB/14,852MiB, A: 8MiB/14,836MiB)

Safe open:
Memory usage before:  CPU: (maxrss: 666MiB F: 8,170MiB) GPU: (U: 192MiB F: 22,299MiB T: 22,491MiB) TORCH: (R: 0MiB/0MiB, A: 0MiB/0MiB)
Memory usage after:  CPU: (maxrss: 2,560MiB F: 8,173MiB) GPU: (U: 14,134MiB F: 8,357MiB T: 22,491MiB) TORCH: (R: 13,942MiB/13,942MiB, A: 13,812MiB/13,812MiB)

It seems like safe_open does move the tensors to the GPU?

@BabyChouSr
With my A100 GPU, fastsafetensors outperformed with your script (I used GDS compat mode, though).
fastsafe_open: 2.685559306293726 seconds
safe_open: 3.209021440707147 seconds

Like you said, yes, safe_open actually loaded tensors to a GPU. I overlooked the parameter device=0 for safe_open.
Please check your storage maximum throughput (FIO or something) and make sure dropping page cache before your tests for apple-to-apple comparison. Also, the PCI topology may affect the GDS performance as well. Peer-to-peer DMA could degrade performance very much on two devices on different PCI roots.
Another possibility is that you may use different GPUs with slow DtoD cudaMemcpy. For the files, fastsafetensors fixed up misaligned headers for some files, which cause memcpy inside the GPU memory.

This issue was closed because it has been inactive.


[ReadMe] remove "should" from FAQ question

Replaces "should strive" below with "strives" since "should" could be construed to have specific meaning per RFC2119

Q: What happens to AppC or Docker Image Formats?
A: ... The OCI Image Format project should strive to provide a dependable open specification ...

Signed-off-by: Rob Dolin<EMAIL_ADDRESS>

LGTM

lgtm


Make a configuration management tool based on Nix to manage User Profiles.

This pull request is made to bootstrap a variant of NixOS, which is dedicated at managing the home directory of users.  This new project is called NixUP (Nix User Profile), and the goals are the following:

[✓] Generate an activation script.
[✓] Abstract the creation of resource files. (similar to environment.etc)
[✓] Option to list installed packages. (similar to system.packages)
[✗] Command line tool to manage profiles generations and updates.  (similar to nixos-rebuild)
[✗] Command line tool to write / read option definition without using an editor.  (no legacy yet)
[✗] Mock compatibility modules to use some NixOS modules in NixUP.
[✗] Include NixUP modules as submodules of NixOS option users.extraUser.<name?>

Ping? :)

I am currently doing some clean-up on NixOS, mostly as a way to try a way to include similar work as this one in NixOS (such as within extraUsers).
So far you can already use this branch, but the commands might be a bit raw as you would have to basically type manually what is done by nixos-rebuild — without the nix update.
I will have to get back to it, and think more on how to make a command line tool which is both similar to nix-env and also compatible with the module system.  The idea would be to have something like:
$ nixup extend config ssh.authorizedKeys ./id.pub

:+1:

Does this actually need NixOS? It seems like anything that can run Nix could have its user profiles managed this way.

@copumpkin good point

Closing: This work is taken over by @ts468 in #9250.


hook not compatible with global gem install on windows

I am seeing an issue as someone who installed lefthook globally as recommended if you follow the instructions blindly...
I can see in a procmon trace that the hook tries to execute lefthook through bundler, which results in a command line like C:\Ruby31-x64\bin\ruby.exe -x C:/Ruby31-x64/bin/bundle exec lefthook -h, but I have no such gem installed in the project. I can run lefthook manually with no problem, its in my PATH.
Very annoyingly, if I run git push, it gets to echo "Can't find lefthook in PATH", but my console just hangs, nothing happening until I break into it (which exits). I have no idea why that happens.
I have no Lefthook.exe, only a lefthook.bat at "C:\Ruby31-x64\bin\lefthook.bat", so that's why it skips the first line.
I presume the easiest fix for me will be to add lefthook in my bundle. Someone else will surely hit this.

Related: https://github.com/evilmartians/lefthook/issues/212

Hmm, I'm not even sure where lefthook.bat came from. Did RubyGems generated it for us? :thinking:
Thanks for detailed description of the problem!
@mrexox, any thoughts?

Hey @ariccio! The problem is that we assume that on Windows lefthook has actually lefthook.exe file as executable. But is seems like Ruby creates a wrapper lefthook.bat which executes a binary from the gem.
I would suggest installing lefthook with go install github.com/evilmartians/lefthook@latest instead of globally installed gem. This should fix the issue.
I will add a check for lefthook.bat for Windows also, and it should fix the problem.

Will release it in about a week as version 1.1.0