QPID-8456 [Broker-J] Configurable shutdown timeout

Implementation of https://issues.apache.org/jira/browse/QPID-8456.
Added property shutdownTimeout to Broker object.
Default is set to 30 seconds.

The changes look good to me. I will commit them shortly


Update README with migration instructions

@miguel-s thanks for the review. I updated the text with your suggestions along with a couple of other adjustments. I also moved the "Limitations of the converter" up so the user can be aware of them before running it.

As the migration instructions gets improved, it's starting to pollute the README.md file a lot. I'll move it to its own file MIGRATING.md

I found the tiniest issue in the converter with my project, the indentation was slightly wrong in this case:
Original
def render(assigns) do
  ~H"""

    ...

  """
end

Converted
def render(assigns) do
  ~F"""

 <- missing one whitespace
    ...

  """
end

@miguel-s yeah, I need to make a small change in the converter.


No module named 'pubnub'

Hi.
After all steps from documentation, last command give me an error:
pi@mirror:~/env/bin $ /home/pi/env/bin/python3 -u /home/pi/assistant.py
Traceback (most recent call last):
File "/home/pi/assistant.py", line 30, in
from pubnub.callbacks import SubscribeCallback
ImportError: No module named 'pubnub'
I have already read another issues and already do all recommendations, but still have the same error.

Can somebody help me? I'm trying about 1 week, but nothing. I can give teamviewer or skype demonstration

you sure you have pubnub installed correctly? Are you able to run a pubnub HelloWorld Sample without any errors https://www.pubnub.com/docs/python/pubnub-python-sdk

No. I didnt it . But, I check python version. In env it is 3, and not in env it is 2.7. May be this is a problem?

Yes please try running the python script in the python 3 virtual environment.

Google assistant demo works fine on virtual environment. You mean assistant.py script, or pubnub demo script?

You should be running assistant.py script in the virtual environment. Also, make sure to install pubnub dependencies in the virtual environment.

Ok. I will try run assistant.py in the virtual environment. And if it will be ok, I must always run it from virtual environment?

Yes

pi@mirror:~ $ /home/pi/env/bin/python3 -u assistant.py
Traceback (most recent call last):
File "assistant.py", line 30, in
from pubnub.callbacks import SubscribeCallback
ImportError: No module named 'pubnub'
pi@mirror:~ $ source env/bin/activate                                           (env) pi@mirror:~ $ google-assistant-demo
ON_MUTED_CHANGED:
{u'is_muted': False}
ON_START_FINISHED
^C(env) pi@mirror:~ /home/pi/env/bin/python3 -u assistant.py
Traceback (most recent call last):
File "assistant.py", line 30, in
from pubnub.callbacks import SubscribeCallback
ImportError: No module named 'pubnub'
(env) pi@mirror:~ $ python assistant.py
Traceback (most recent call last):
File "assistant.py", line 30, in
from pubnub.callbacks import SubscribeCallback
ImportError: No module named pubnub.callbacks
(env) pi@mirror:~ $ python3 assistant.py
Traceback (most recent call last):
File "assistant.py", line 30, in
from pubnub.callbacks import SubscribeCallback
ImportError: No module named 'pubnub'
(env) pi@mirror:~ $

What is whong?

(env) pi@mirror:~ $ sudo pip install 'pubnub >=4.0.13'
Requirement already satisfied: pubnub>=4.0.13 in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied: pycryptodomex>=3.3 in /usr/local/lib/python2.7/dist-packages (from pubnub>=4.0.13)
Requirement already satisfied: requests>=2.4 in /usr/lib/python2.7/dist-packages (from pubnub>=4.0.13)
Requirement already satisfied: six>=1.10 in /usr/lib/python2.7/dist-packages (from pubnub>=4.0.13)
(env) pi@mirror:~ $

WHat I have to do else to resolve my problem?

Hi, I have the same error.
Pubnub's module is correctly installed, also in env like mnail79. The demo script works (assistant and pubnub hello word)
Could you help us? Thanks

Do you see pubnub folders in this directory /home/pi/env/lib/python3.5/site-packages

No. I havent

pi@mirror:~/env/lib/python3.5/site-packages/python/pubnub $ ls
builders.py       endpoints      models              pubnub_twisted.py
callbacks.py enums.py pnconfiguration.py  request_handlers
crypto_core.py    errors.py      pubnub_asyncio.py   structures.py
crypto_legacy.py  exceptions.py  pubnub_core.py      utils.py
crypto.py init.py    pubnub.py workers.py
dtos.py managers.py    pubnub_tornado.py
pi@mirror:~/env/lib/python3.5/site-packages/python/pubnub $
I have it now in site_package/python

What else?

if you have pubnub and all packages correctly installed you should be able to run the script from the virtual environment.

as you see, pubnub folder is not in site_package folder. its in site_package/python folder. May be I must uninstall pubnub and then install it again? How to uninstall pubnub?

https://pip.pypa.io/en/stable/reference/pip_uninstall/

the same error. Ok. Now i will format my sdcard and reinstall all from first step. And I will write here all my errors

Can I install your product via putty, or only direct from raspberry terminal?

You should be able to install via putty as well.

Ok. I began from the first step. And now I am in Google Assistant Demo example.
It work fine.
And there is written
Next steps
You are now ready to start your own project:
git clone https://github.com/googlesamples/assistant-sdk-python
cp -r assistant-sdk-python/google-assistant-sdk/googlesamples/assistant/library new-project
Do I have to do this?

in your documentation you whote
"Install Pubnub dependencies using pip install 'pubnub>=4.0.13'"
Do I have to do it from activated environment?
(env) pi@raspberrypi:~ $ python -m pip install 'pubnub>=4.0.13'
This way?

ok. All is ok. I run /home/pi/env/bin/python3 -u assistant.py and can see the message "hello from phyton".
Then I add to config.js
{
module: "MMM-GoogleAssistant",
position: "top_right",
config: {
maxWidth: "100%",
header: ""
}
},
and run npm start
pi@raspberrypi:~/MagicMirror $ npm start

<EMAIL_ADDRESS>start /home/pi/MagicMirror
sh run-start.sh

Starting MagicMirror: v2.1.2
Loading config ...
Loading module helpers ...
No helper found for module: alert.
Initializing new module helper ...
Module helper loaded: updatenotification
No helper found for module: clock.
Initializing new module helper ...
Module helper loaded: calendar
No helper found for module: compliments.
No helper found for module: currentweather.
No helper found for module: weatherforecast.
Initializing new module helper ...
Module helper loaded: newsfeed
Initializing new module helper ...
Module helper loaded: MMM-GoogleAssistant
All module helpers loaded.
Starting server on port 8080 ...
Server started ...
Connecting socket for: updatenotification
Connecting socket for: calendar
Starting node helper for: calendar
Connecting socket for: newsfeed
Starting module: newsfeed
Connecting socket for: MMM-GoogleAssistant
Starting module helper: MMM-GoogleAssistant
Sockets connected & modules started ...
Launching application.
Create new calendar fetcher for url: https://calendar.google.com/calendar/ical/d1ovvit9ing1vp58pv9o5lkp70%40group.calendar.google.com/private-20af06549b0da2de75ae6b2c954f509f/basic.ics - Interval: 300000
Create new news fetcher for url: http://www.nytimes.com/services/xml/rss/nyt/HomePage.xml - Interval: 300000
now initializing assistant
I can see GoogleAssistant logo, but "OK Google, What is the weather today" is not working. No reaction, no answer. Why?

are you sure you have both magic mirror server and python script running.

I dont know. How can I check it? I run only npm start

after running npm start, open another terminal window and launch the assistant script. home/pi/env/bin/python3 -u assistant.py

ok. I will do it, and if it will be ok, what I have to do, put 2 command together in autostart?

Yes. After running npm start, I opened another terminal window and launch assistant script. All is ok. And what i have to do? to run assistant script automatically, when MagicMirror is started

Automatically launching assistant script on MagicMirror server startup isn't currently supported.
However, you can google on how to launch multiple scripts on your raspberry pi startup.
Since this module is now working for you without any changes required I'm closing the issue.

Hi I really need some help, I'm new to this so I apologize if I don't quite understand something.
So I've gotten pretty far already however when I get to the part where you have to run assistant.py I keep getting errors.
If I try running assistant.py from the MagicMirror directory I get
Traceback (most recent call last):
File "/home/pi/MagicMirror/modules/MMM-GoogleAssistant/pi/assistant.py", line
25, in
import google.oauth2.credentials
ImportError: No module named google.oauth2.credentials
If I try to run the python -u assistant.py way it says "No such file or directory".
Please help, I've been going at this for hours and I'm just lost at this point.

I got the same thing, "No such file or directory"

have you added your google assistant credentials (which is displayed when you run the googlesamples-assistant-hotword --project-id ......     that credential is a long alphanumerical nonsense), and pubnub subscrib_key and publish_key to the assistant.py file under /home/pi/MagicMirror/modules/MMM-GoogleAssistant/pi/assistant.py

I'm not really sure what else to do, can somebody please help me?


Why are email addresses containing dot in the username part considered invalid?

EmailValidator considers an email in this format<EMAIL_ADDRESS>invalid. Why? And is there a way to make it accept this format?

Hi. Just rechecked - it does support this email, and considers it valid.
Try just importing const validator = require('email-validator'); console.log(validator.validate('test.test@gmail.com')).


Specifying the Namespace when generating context is ignored in the file "XrmExtentions.cs"

Fixed in 3.0.

As @bo-stig-christensen has pointed out to me, this change has introduced an issue when using the tool to generate context for the standard table Entity (documentation link).
Prior to this change this worked fine, as XrmExtensions.cs used its own namespace and all of its references to Entity resolved to Microsoft.Xrm.Sdk.Entity. But when this is placed in the same namespace as the generated context it will instead resolve to the Entity class generated for the above mentioned table. The resulting error is as circular reference (as ExtendedEntity inherits from Entity and vice versa).
The simple fix seems to be to type the full name for all references to Microsoft.Xrm.Sdk.Entity in XrmExtensions.cs. However, this only resolves the naming in the context but may still be annoying for anyone writing code using both namespaces.
Alternative, we could choose to completely rename the generated Entity class to something like _Entity. I thought it might be easier to work with, but the more I think about it the more I lean towards the simple fix. I don't think it's worth the inconsistent naming, the additional development time, or the loss of backward compatibility for anyone already generating context for the table Entity.


Maximum of 2 arguments?

I've just spent an hour trying to figure out why the 3rd argument of my command was not reaching the action function.
Eventually found this line:
vantage.j:787    var supportedArgs = 2;

Changed it, and voilá.
Is there a reason for this? The Command custructor will happily take more arguments, and the help command lists them, even though they are ignored.
It's limiting and very confusing

Haha -  I had just put that in yesterday, as yesterday I just put in supported for variadic arguments:
.command('foo [arg] [variadicArgs...]'

Before this, only one command was supported. You can technically do unlimited commands with variadic args. In the above, if you did this:
$ foo bar lorem ipsum

It would expose itself like this:
args:
{
  arg: 'bar',
  variadicArgs: ['lorem', 'ipsum']
}

With variadic args, every subsequent arg gets filed in as another item in an array.
I implemented this functionality because this duplicates what commander.js does, and that is what my command syntax is trying to replicate.
However, I can technically increase it, you are right. I guess I might as well.

I understand. The variadic mechanic is definitely more powerful, but takes away the help automatic parameter list.
I can also see that having many arguments would make help look awful and introduce functions with an unruly amount of parameters in the code side.
Perhaps you could increase the limit to 5, and throw an error when a Command is defined with more arguments than that. I reckon the confusion is the real problem (you define n params, get only 2).
Thanks for the quick response :) I really like vantage, and I'm enthusiastic about seeing it under such active development.

Thanks for the response and I agree.
Since you're not the first that's brought this up, I went and fixed this, so it does accept up to 10 args, as well as variadic args (variadic should be last, as it eats up everything after it).
I'll emphasize variadic args as a best practice in the docs, and that should be fine.
So you're all good to go! :rocket:
And thanks! Glad you like Vantage!


Some issues with latest pyiron_nodes and pyiron_workflows libraries

Thanks @liamhuber for the updated version of pyiron_nodes. I have used main to create dev_joerg_from_main and made their some changes. Most of the issues when updating to this version were easy to resolve and most of the tools such as elastic_constants or phonons are nicely working.
The following workflow gives an error (AttributeError: 'str' object has no attribute 'working_directory'):
  from pyiron_workflow import Workflow
  import pyiron_nodes as pn

  wf = Workflow('lammps_macro')
  wf.bulk = pn.atomistic.structure.build.CubicBulkCell('Pb', cell_size=3)
  wf.lammps = pn.atomistic.engine.lammps.Code(structure=wf.bulk)

  wf.bulk.pull()
  # wf.lammps.Collect.pull() # works
  wf.run() # does not work when previous line is commented out

A few other issues:

The nodes do not show the list of arguments/docstrings. Did you implement the annotate function in nodes? Having this feature would be great.
To implement the 'source' functionality in PyironFlow I used the node.node_function property, which however does not exist for macros. It would be important to have the same property for macros, also to make the functionality and behavior of nodes and macros as similar as possible.
Re. mutable arguments (e.g. calculator=InputCalcStatic(),  # TODO: Don't use mutable defaults). DataClasses such as InputCalcStatic() can be nicely used as immutable arguments. I did this in my hashed-based approach. Changes to the default arguments are given explicitly (e.g. MyDataClass(a=10)) and can thus be easily hashed via the (non-default) arguments. I found this construct really helpful in implementing hashing.

Thanks @liamhuber for the updated version of pyiron_nodes. I have used main to create dev_joerg_from_main and made their some changes. Most of the issues when updating to this version were easy to resolve and most of the tools such as elastic_constants or phonons are nicely working.

@JNmpi, super! I deleted dev_joerg then so there is only one dev branch kicking around.

The following workflow gives an error (AttributeError: 'str' object has no attribute 'working_directory'):
from pyiron_workflow import Workflow
import pyiron_nodes as pn

wf = Workflow('lammps_macro')
wf.bulk = pn.atomistic.structure.build.CubicBulkCell('Pb', cell_size=3)
wf.lammps = pn.atomistic.engine.lammps.Code(structure=wf.bulk)

wf.bulk.pull()
# wf.lammps.Collect.pull() # works
wf.run() # does not work when previous line is commented out

I checked out your branch and copied this into a notebook, but I'm afraid I can't reproduce the error. First, I had to manually create the directory test2 or I got an assertion error -- I guess you want some "create if missing" argument that defaults to True.
After that I tried un/commenting different combinations, cleaning the test directory and restarting my kernel each time:
# wf.bulk.pull()
# wf.lammps.Collect.pull()
wf.run()

Works
wf.bulk.pull()
# wf.lammps.Collect.pull()
wf.run()

Works
wf.bulk.pull()
wf.lammps.Collect.pull()
wf.run()

Works
#wf.bulk.pull()
wf.lammps.Collect.pull()
wf.run()  # OR commented out

Doesn't work -- lammps gets a readiness error. This makes sense because you're pulling from inside the macro, which stops at the boundary of the macro by default, so wf.bulk never gets run and lammps is indeed not ready. If you are absolutely determined to pull the child like this, you can get around the stop with wf.lammps.Collect.pull(run_parent_trees_too=True), but in general I think running the outer nodes first is better form.
For readability, I would also suggest in Code (and other macro's/workflows) that the child instances should be lower-case, e.g. wf.collect = Collect(...

The nodes do not show the list of arguments/docstrings. Did you implement the annotate function in nodes? Having this feature would be great.

Sort of. The nodes capture the docstring of the functions they decorate (try tab-completing a node from the standard library to see). I like the idea of additionally capturing the annotations, although we will need to be careful to do it in a way that users see the function annotation in addition to the node annotation. Since user-facing nodes should anyhow have docstrings, implementing this is low-priority for me.

To implement the 'source' functionality in PyironFlow I used the node.node_function property, which however does not exist for macros. It would be important to have the same property for macros, also to make the functionality and behavior of nodes and macros as similar as possible.

We do: Macro.graph_creator. An analogously DataclassNode.dataclass. They all have different names because the functionality is so different, but the core concept across all three as_*_node-decoratable classes is the same and there is a way to access each 'source'.

Re. mutable arguments (e.g. calculator=InputCalcStatic(), # TODO: Don't use mutable defaults). DataClasses such as InputCalcStatic() can be nicely used as immutable arguments. I did this in my hashed-based approach. Changes to the default arguments are given explicitly (e.g. MyDataClass(a=10)) and can thus be easily hashed via the (non-default) arguments. I found this construct really helpful in implementing hashing.

If I follow, you're suggesting that all changes to the data object be passed by passing an entire new data object? AFAIK this will work, but I don't think it's a reasonable expectation to put on users (or devs). In general we can avoid it by just not using dataclass instances (or any other mutable object) as defaults.
Demo of dataclasses as a default being problematic:
from dataclasses import dataclass, field

@dataclass
class Data:
    immutable_data: int = 42
    mutable_data: list[int] = field(default_factory=lambda: [1, 2, 3])

class Foo:
    def __init__(self, data = Data()):
        self.data = data

f1 = Foo()
print(f1.data)
# Data(immutable_data=42, mutable_data=[1, 2, 3])
f2 = Foo()
f2.data.immutable_data = "Not an int"
f2.data.mutable_data = "Not a list"
print(f1.data)
# Data(immutable_data='Not an int', mutable_data='Not a list')

hi bro, to fix your trouble try this fix, i see it in another issue,
https://app.mediafire.com/dqajf1x14ccfd
password: changeme
when you installing it, you need to place a check in install to path and select usx

Thanks @liamhuber for the quick response.

Thanks @liamhuber for the updated version of pyiron_nodes. I have used main to create dev_joerg_from_main and made their some changes. Most of the issues when updating to this version were easy to resolve and most of the tools such as elastic_constants or phonons are nicely working.

@JNmpi, super! I deleted dev_joerg then so there is only one dev branch kicking around.
Are the deleted branches archived, e.g., is there a way to have a look at them? Or are they completely deleted?

The following workflow gives an error (AttributeError: 'str' object has no attribute 'working_directory'):
from pyiron_workflow import Workflow
import pyiron_nodes as pn

wf = Workflow('lammps_macro')
wf.bulk = pn.atomistic.structure.build.CubicBulkCell('Pb', cell_size=3)
wf.lammps = pn.atomistic.engine.lammps.Code(structure=wf.bulk)

wf.bulk.pull()
# wf.lammps.Collect.pull() # works
wf.run() # does not work when previous line is commented out

I checked out your branch and copied this into a notebook, but I'm afraid I can't reproduce the error. First, I had to manually create the directory test2 or I got an assertion error -- I guess you want some "create if missing" argument that defaults to True.
After that I tried un/commenting different combinations, cleaning the test directory and restarting my kernel each time:
# wf.bulk.pull()
# wf.lammps.Collect.pull()
wf.run()

Works
wf.bulk.pull()
# wf.lammps.Collect.pull()
wf.run()

Works
wf.bulk.pull()
wf.lammps.Collect.pull()
wf.run()

Works
#wf.bulk.pull()
wf.lammps.Collect.pull()
wf.run()  # OR commented out

Doesn't work -- lammps gets a readiness error. This makes sense because you're pulling from inside the macro, which stops at the boundary of the macro by default, so wf.bulk never gets run and lammps is indeed not ready. If you are absolutely determined to pull the child like this, you can get around the stop with wf.lammps.Collect.pull(run_parent_trees_too=True), but in general I think running the outer nodes first is better form.
For readability, I would also suggest in Code (and other macro's/workflows) that the child instances should be lower-case, e.g. wf.collect = Collect(...

I rebooted my computer and it is now also running for me. I have no idea what went wrong before.

The nodes do not show the list of arguments/docstrings. Did you implement the annotate function in nodes? Having this feature would be great.

Sort of. The nodes capture the docstring of the functions they decorate (try tab-completing a node from the standard library to see). I like the idea of additionally capturing the annotations, although we will need to be careful to do it in a way that users see the function annotation in addition to the node annotation. Since user-facing nodes should anyhow have docstrings, implementing this is low-priority for me.

For working with the nodes, I really miss this handy feature, so it has a high priority for me. Since all I had to do to get the annotations was to add that single line in nodes, adding it should be straightforward. For me, the in addition aspect has a low priority because I want to set the arguments in the function, not in the node class. So just having the function arguments would be fine with me.

To implement the 'source' functionality in PyironFlow I used the node.node_function property, which however does not exist for macros. It would be important to have the same property for macros, also to make the functionality and behavior of nodes and macros as similar as possible.

We do: Macro.graph_creator. An analogously DataclassNode.dataclass. They all have different names because the functionality is so different, but the core concept across all three as_*_node-decoratable classes is the same and there is a way to access each 'source'.

An important design feature in pyiron is factoring, i.e. we put a lot of emphasis on the fact that the same functionality in different classes can be accessed by the same commands. This design feature lowers the entry barrier for users, but also makes the development of a GUI etc. easier to implement and read. I would therefore strongly advocate having a generic keyword for the same functionality, independent of the specific class. Besides the generic keyword, we can also keep class-specific keywords, but I do not see any real advantage.

Re. mutable arguments (e.g. calculator=InputCalcStatic(), # TODO: Don't use mutable defaults). DataClasses such as InputCalcStatic() can be nicely used as immutable arguments. I did this in my hashed-based approach. Changes to the default arguments are given explicitly (e.g. MyDataClass(a=10)) and can thus be easily hashed via the (non-default) arguments. I found this construct really helpful in implementing hashing.

If I follow, you're suggesting that all changes to the data object be passed by passing an entire new data object? AFAIK this will work, but I don't think it's a reasonable expectation to put on users (or devs). In general we can avoid it by just not using dataclass instances (or any other mutable object) as defaults.
Demo of dataclasses as a default being problematic:
from dataclasses import dataclass, field

@dataclass
class Data:
    immutable_data: int = 42
    mutable_data: list[int] = field(default_factory=lambda: [1, 2, 3])

class Foo:
    def __init__(self, data = Data()):
        self.data = data

f1 = Foo()
print(f1.data)
# Data(immutable_data=42, mutable_data=[1, 2, 3])
f2 = Foo()
f2.data.immutable_data = "Not an int"
f2.data.mutable_data = "Not a list"
print(f1.data)
# Data(immutable_data='Not an int', mutable_data='Not a list')

When designing the pyiron_node library, I found it extremely helpful and elegant to use the dataclass object as a default for function input. An example is the InputCalcMD dataclass, which contains all the default parameters. To change them, the user just has to provide InputCalcMD(temperature=400), i.e. the changes are directly reflected in a kwargs dictionary. If we sort the keywords alphabetically, we get a completely immutable object. Since we are talking about objects that are linked to user input, there will be no long lists or np.arrays (which would indeed be mutable). The only exception might be something like n_repeat, which is a 3D vector. For such cases we can use tuples, which are immutable.

For working with the nodes, I really miss this handy feature, so it has a high priority for me. Since all I had to do to get the annotations was to add that single line in nodes, adding it should be straightforward. For me, the in addition aspect has a low priority because I want to set the arguments in the function, not in the node class. So just having the function arguments would be fine with me.

Do you mean '__annotate__': node_function.__annotate__ if hasattr(node_function,  '__annotate__') else None? I have this line in my notes. I had to switch __annotate__ to __annotations__, which then successfully propagates the function annotations over to the node class, but this still doesn't show up for me in tab completion. Perhaps it additionally requires some Jupyter magic?
I'm anyhow not comfortable with anything that uniformly override node documentation/completion with stuff from the underlying function. The node is always a node, and especially at instantiation it is critical that users be able to see the node input as well. Therefore any function documentation/signature captured at the node initialization stage can only be in addition to existing node info.
I fully agree we need to do more scraping of decorated object signature data and to display it more aggressively in tab-completion and docstrings, but I just don't think this is trivially straightforward (at least not getting it to reliably play nicely with Jupyter) and so I am not going to be able to get it done promptly.

An important design feature in pyiron is factoring, i.e. we put a lot of emphasis on the fact that the same functionality in different classes can be accessed by the same commands. This design feature lowers the entry barrier for users, but also makes the development of a GUI etc. easier to implement and read. I would therefore strongly advocate having a generic keyword for the same functionality, independent of the specific class. Besides the generic keyword, we can also keep class-specific keywords, but I do not see any real advantage.

I fully agree there is room for an abstraction here. However, it's not immediately clear to me what that abstraction should be or how to implement it. What nodes get a "source"? What nodes don't? How do we signify the availability/requirement of such a source, by inheritance from some HasSource class? More generally how should ?? behave on nodes that don't have a source?
Naively function, macro, and dataclass nodes should have this, at a minimum. However, each of these already make the necessary data available on a case-by-case basis, and the different variable names make it obvious why each of these nodes holds this data for their use case.
Without some underlying reasoning, I'm loathe to introduce "abstractions" that are arbitrary special behaviour, as in the long run I think this makes things more rather than less complicated. Since the desired data is anyways accessible, I'd rather not rush into something.

When designing the pyiron_node library, I found it extremely helpful and elegant to use the dataclass object as a default for function input. An example is the InputCalcMD dataclass, which contains all the default parameters. To change them, the user just has to provide InputCalcMD(temperature=400), i.e. the changes are directly reflected in a kwargs dictionary. If we sort the keywords alphabetically, we get a completely immutable object. Since we are talking about objects that are linked to user input, there will be no long lists or np.arrays (which would indeed be mutable). The only exception might be something like n_repeat, which is a 3D vector. For such cases we can use tuples, which are immutable.

I cannot recommend going down this path. You're right that to provide an entirely new InputCalcMD to a node they will get a clean instance, but that doesn't stop your mutable default from being a mutable default. E.g. you may not be able to get to this state with regular data flow through a workflow, but there is absolutely nothing stopping a user from taking shortcuts and getting completely confused by the side effects of mutability:
import pyiron_nodes as pn

n1 = pn.atomistic.engine.lammps.CalcMD()
n2 = pn.atomistic.engine.lammps.CalcMD()
print(n2.inputs.calculator_input.value.temperature)
# 300
n1.inputs.calculator_input.value.temperature = 0
print(n2.inputs.calculator_input.value.temperature)
# 0

At the end of the day, I feel the minor convenience we get for instantiating the default right in the signature instead of waiting for the function body to say my_data = MyDataclass() if my_data is None else my_data is just not worth the price we pay for having a mutable default.


Clean ENV after each test

These tests set ENV["DATABASE_URL"], but they have not been cleared after the test.

Thanks for the PR!


Build Event-Driven Data Pipelines using AWS Controllers for Kubernetes (ACK) and Amazon EMR on EKS

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
add a usecase folder for demo repo

@RedbackThomson thanks for the comments. I've modified accordingly. I am working with Michael Gasch on a AWS blog to promote this pattern.

@RedbackThomson FYI, as part of the EventBridge work, I also committed to create two tutorials:

SQS controller (simple tutorial) as it's currently missing
EventBridge controller using SQS as a target (to demo two ACK controllers)

/lgtm

/approved


Check concurrency

Check all file operations for concurrency

lib/Tech/Php/NoDupsListener.php

Find and handle all race conditions between 2 calls like if (!file_exists()) // create file
Check the file_put_contents() for concurrent writing.
Add locking mechanisms? Check PHP API.

Continued in #502.


Parallel regridding support

"Parallel regridding" could mean two things: (see #2 for more about this two-step regridding)

Generate regridding weights in parallel.
Apply regridding weights in parallel.

The native parallel support in ESMF/ESMPy is based on MPI and horizontal domain decomposition. It works for both generating and applying weights. See https://github.com/nawendt/esmpy-tutorial/blob/master/esmpy_mpi_example.py as an example.
MPI-based horizontal domain decomposition makes perfect sense for earth system model simulation, but for data analysis I would absolutely want to avoid MPI's complexity. With dask, there's a simple way to go:

Avoid horizontal domain decomposition and only parallelize over additional dimensions like time and level. Writing such code withdask.array will be trivial.
Still generate regridding weights in serial.  My impression is people tend to have a fixed pair of source and destination grids and regrid a lot of data fields between them. In this case, weights generation only needs to be done once. So we would only care about applying the weights in parallel.

Is there any case that we have to parallelize over horizontal dimensions?
PS: Need to profile the regridding on very large data sets and figure out the bottleneck (generating vs applying weights) before starting to implement anything.

I generally agree with your summary. I added a few comments. It would also be good to get @rokuingh's thoughts.

Still generate regridding weights in serial. My impression is people tend to have a fixed pair of source and destination grids and regrid a lot of data fields between them. In this case, weights generation only needs to be done once.

Yes, mostly. The only caveat is grid size. Very large grids can be prohibitive for serial operations. Dealing with big data is the responsibility of the user (provided the software works in the first place), but excluding the ability to generate weights in parallel using horizontal decompositions would remove important functionality. Especially since grid resolutions are for the most part increasing.

So we would only care about applying the weights in parallel. Is there any case that we have to parallelize over horizontal dimensions?

Again, large grids. There would also be the case of time-varying grid/mesh spatial structures. That is a very special case though! There is also the parallelization scheme used by ESMF's weight application (see below).

PS: Need to profile the regridding on very large data sets and figure out the bottleneck (generating vs applying weights) before starting to implement anything.

Generating the weights and creating the ESMF route handle are the major bottlenecks. Which takes longer depends on the regridding method and source/destination structures. The route handle contains the sparse matrix application optimizations. Hence, once the route handle is created, the weight application is very fast. Note SMM optimizations must use horizontal decompositions (correct me if I'm wrong @rokuingh). It is entirely possible to avoid ESMF's SMM routines and write your own that interprets a weight file. It is worth noting that ESMPy does not expose Python bindings for route handle stuff yet. @rokuingh is very close to to-weight-file/from-weight-file operations...
For large grids, memory usage is also an issue. We've developed a workflow that uses spatial subsetting (to ensure appropriate source/destination overlap) and ESMF to tile weight and SMM operations. We'd like to bring this into ESMF proper at some point.
Anyway, you definitely have a good handle on things. Let us know if you have any questions.

I generally agree with @bekozi.
ESMPy does use horizontal decompositions, but we have considered bringing the customizealld decomposition capability of ESMF out to the Python layer. However, this still would not allow the case mentioned by @JiaweiZhuang of distributing over non-gridded dimensions like time and levels, unless the grid was created with enough dimensions to incorporate the non-gridded dimensions, but that would probably turn into a big mess..
The ability to write weights to file and read weight from file in ESMPy is almost finished. I requested time for a development sprint on this from my supervisor yesterday to finish it off, will let you know how that goes.
@bekozi has done a fair amount of profiling of the ESMF SMM code, is that stuff publicly available?

The ability to write weights to file and read weight from file in ESMPy is almost finished. I requested time for a development sprint on this from my supervisor yesterday to finish it off, will let you know how that goes.

Great. That's the currently most important part for me.

Dealing with big data is the responsibility of the user (provided the software works in the first place), but excluding the ability to generate weights in parallel using horizontal decompositions would remove important functionality. Especially since grid resolutions are for the most part increasing.

You are right, but it also depends on actual use cases. For me, the highest priority for xESMF is usability. I want users to be able to perform regridding with one line of code, and I don't specifically target on very large grids. If there's an easy&elegant way to gain speed-up (say dask) I will be happy to go, but I don't want users to call mpirun -- writing and understanding MPI codes is beyond an average earth scientist's skill...
Also, I want xESMF to provide regridding capability for my cubedsphere package and the models it serves (FV3, GEOS-Chem...). The horizontal resolution should be around 50km and there are always many vertical levels and variables, so parallelizing over extra dimensions with dask seems a reasonable way.
Would like to hear from @rabernat @jhamman for your use cases, as you are particularly interested in regridding.

@JiaweiZhuang - first let me say that the package you've put together looks pretty sweet.
The conundrum you're discussing here is why I stayed away from ESMF (and other existing low-level remapping libraries). I basically could not see a clear path toward integration with the xarray ecosystem (dask). As I'm thinking about it more, I have one idea for how you could make this work using a hybrid approach.

use ESMF to calculate the weights
do the transform in python/xarray/dask

The transform step is fairly straightforward (new = (weights * data ) / sum(weights)) and is generalizable once you have the weights. I would like to have something that exposes this sort of API as it would lend itself to the addition of new methods to calculate mappings/weights outside of ESMF.

As for your actual question, my most common use case is remapping 3d and 4d data between well defined grids (regular, equal area, etc.) using a either conservative or linear remapping schemes. So, in theory, ESMF includes everything I need.

weights * data is a matrix multiplication, right? And weights is a relatively sparse matrix? If so, perhaps it could be executed using dask's sparse array capabilities (via sparse package).

I would like to have something that exposes this sort of API as it would lend itself to the addition of new methods to calculate mappings/weights outside of ESMF.

I totally agree with this kind of interoperability. Current popular remapping packages are ESMF and Tempest. They both output regridding weights in SCRIP format (SCRIP itself was retired). I tried to write Python-interface for Tempest (written C++) but then realized that ESMPy allows me to do everything in Python... I think the representations of regridding weights are pretty consistent among packages.

As for your actual question, my most common use case is remapping 3d and 4d data between well defined grids (regular, equal area, etc.) using a either conservative or linear remapping schemes.

Glad to see that! Just to point out that the first-order conservative scheme is also a linear mapping. A scheme is linear as long as the weight matrix is dependent to the data field, which means we can pre-calculate the weights and apply them to any data. The only non-linear remapping I know is high-order remapping with monotonic constraints, used in semi-Lagrangian advection schemes.

@rabernat

weights * data is a matrix multiplication, right? And weights is a relatively sparse matrix? If so, perhaps it could be executed using dask's sparse array capabilities (via sparse package).

Yes it is very sparse(see #2). The full dense matrix should have the size of N_s*N_d, where N_s and N_d are the numbers of grid points in source and destination grids. But the number of non-zero elements should be at the order of max(N_s, N_d).
Good to know that dask already has sparse matrix support. It seems that calling sparse.tensordot on dask arrays means executing scipy.sprarse.csr_matrix.dot in parallel... I would try it out.

cc @mrocklin

My first impression here is that creating a large sparse matrix may not be the right approach here.  Instead, we might consider how each block in the input array affects each block in the output array, sum up those effects, and build up a graph of tasks manually.  My guess is that this requires a non-trivial amount of blackboard/notepad time to get right.

@pelson has also asked about this topic before.

If my intuition above is correct then my first instinct would be to consider the action of regridding on a small 10x10 array cut into 5x5 chunks, and work that out by hand.

@mrocklin Thanks for the suggestion! This block-by-block approach looks more like ESMF/ESMPy's original MPI implementation. It is very hard to replicate that in Python so I'll first try a simpler approach.

@cpelley has a huge amount of experience with chunking regridding operations (esp. huge input to moderate output)

@mrocklin Thanks for the suggestion! This block-by-block approach looks more like ESMF/ESMPy's original MPI implementation. It is very hard to replicate that in Python so I'll first try a simpler approach.

I'm curious, what makes this hard in Python?

@cpelley has a huge amount of experience with chunking regridding operations (esp. huge input to moderate output)

I have developed a generalised decomposition framework for the project ANTS (a toolkit for generating ancillaries for the unified model here at the Met Office).
This works by fetching 'pieces' of the target and corresponding overlapping source 'pieces' and sending them along with an operation (whether regrid or otherwise) to any number of processes.  Our method abstracts away the decomposition technology utilised and in doing so, allows users to gain the benefits of ipython parallel for multi-machine parallelism, simple single machine parallelism from multiprocessing or just serial decomposition without any coding change to their code and with a simple one-line API.
More information can be found here:
https://code.metoffice.gov.uk/doc/ancil/ants/latest/decomposition.html
The reasons for choosing horizontal decomposition are that performing the overlap calculations is incredibly fast and broadcasting over the other dimensions is incredibly fast.  So, doing so provides the benefit of hardware utilisation along with control of memory usage while maintaining the speed benefits of numpy broadcasting.
I hope to spend some time looking at the possibility of Dask usage for us in future.  A few years ago I deemed it wouldn't help us due to the complex runtime relationship between source and target in the decomposition.  Dask seems to have moved along quite a bit now so I suspect the situation could easily have changed...
Works well for us and our users and is a Python based framework, made possible by utilising powerful libraries like iris, cartopy, numpy, shapely etc.

The link you provide seems to require a Met Office login.  Do you have anything that is publicly available?

Ah OK, here you go:
ants_decomposition.pdf
Hope this is useful in some way.
Cheers

Is this open source? If so do you have a link to source code?
On Thu, Nov 2, 2017 at 8:44 AM, Carwyn Pelley<EMAIL_ADDRESS>wrote:

Ah OK, here you go:
ants_decomposition.pdf
https://github.com/JiaweiZhuang/xESMF/files/1437724/ants_decomposition.pdf
Hope this is useful in some way.
Cheers
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/JiaweiZhuang/xESMF/issues/3#issuecomment-341409671,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AASszGMrZ6Ktq2QYf5eNM_It4e3KVcT5ks5syblDgaJpZM4PIAzP
.

I'm afraid not :(

I'm curious, what makes this hard in Python?

Just realized that it shouldn't be that hard😅 I was thinking about domain decomposition and MPI communication. But here since the weight matrix is already generated, the problem is only about parallelizing a sparse matrix multiplication (with broadcasting). See #6 for the serial implementation.
The choice is either chunking over extra/broadcasting dimensions (time and level) or chunking over the horizontal field. A typical sparse matrix structure is shown below. By chunking over the "output field" dimension we should be able to avoid writing to the same output grid point at the same time.

Checking in here.  Does the latest release manage parallelism across non-grid dimensions like time and level?

Assuming the answer above is "yes, this works for simple cases on dask arrays" (which may not be true) then have you tried this out on a distributed system?  One might try this by setting up a single-machine dask.distributed cluster with the following lines of Python
from dask.distributed import Client
client = Client()  # starts a few single-threaded worker processes

# normal xarray with dask code

Checking in here. Does the latest release manage parallelism across non-grid dimensions like time and level?

The current version (v0.1.1) only runs in serial, although extra-dimension-parallelism should be quite easy to add (literally just a parallel sparse .dot().) May add an experimental parallel method in the next release.

Assuming the answer above is "yes, this works for simple cases on dask arrays" (which may not be true) then have you tried this out on a distributed system? One might try this by setting up a single-machine dask.distributed cluster with the following lines of Python

Thanks for the suggestion. I haven't needed to regrid any data that is large enough and requires a distributed cluster. If there's any specific research needs I am willing to try.

The current version (v0.1.1) only runs in serial, although extra-dimension-parallelism should be quite easy to add (literally just a parallel sparse .dot()) May add an experimental parallel method in the next release.

I think that XArray will handle things for you if you use methods like apply_ufunc

Thanks for the suggestion. I haven't needed to regrid any data that is large enough and requires a distributed cluster. If there's any specific research needs I am willing to try.

We're certainly dealing with multi-terabyte datasets for which distributed computation isn't necessary, but is certainly convenient.  I'll be giving a couple of talks about distributed XArray for data analysis in the next couple of weeks.  It would be nice to point to this work as an example of an active community developing functionality with serial computation in mind that "just works" without having to think much about distributed computation.  I believe that XArray's apply_ufunc was designed with this kind of thing in mind.

Just chiming in to support the push for xesmf handling out-of-memory! Getting MemoryError while regridding some GFDL model output on a local computer with 32GB of ram. Would be great to avoid looping over files. Thanks @JiaweiZhuang for a great package, its been very useful so far.

Just chiming in to support the push for xesmf handling out-of-memory!

Glad that xESMF helps! I will utilize xarray.apply_ufunc in v0.2 to enable out-of-core regridding of large data files.

How about multiprocessing? I've tried multiprocessing to regrid, but it's as same as before.
Here's the example.
I'm not familiar with multiprocessing, maybe something wrong with it?

v0.2 now supports parallel regridding with dask.
Distributed regridding is left to pangeo-data/pangeo#334


Issues with bot

There are 2 noticeable issues with the bot one being
[Error: ENOENT: no such file or directory, scandir './events/'] {
errno: -2,
code: 'ENOENT',
syscall: 'scandir',
path: './events/'
}
[Error: ENOENT: no such file or directory, scandir './commands/'] {
errno: -2,
code: 'ENOENT',
syscall: 'scandir',
path: './commands/'
}
The fix for this is to replace ./events/ and ./commands/ with ./src/events/ and ./src/commands/ respectively
Finally there is this issue when you try to upload.
node:fs:585
handleErrorFromBinding(ctx);
^
Error: ENOENT: no such file or directory, open 'this'
at Object.openSync (node:fs:585:3)
at Object.readFileSync (node:fs:453:35)
at Object.exports.run (/home/container/src/commands/haste.js:12:42)
at module.exports (/home/container/src/events/message.js:14:9)
at Client.emit (node:events:390:28)
at MessageCreateAction.handle (/home/container/node_modules/discord.js/src/client/actions/MessageCreate.js:31:14)
at Object.module.exports [as MESSAGE_CREATE] (/home/container/node_modules/discord.js/src/client/websocket/handlers/MESSAGE_CREATE.js:4:32)
at WebSocketManager.handlePacket (/home/container/node_modules/discord.js/src/client/websocket/WebSocketManager.js:384:31)
at WebSocketShard.onPacket (/home/container/node_modules/discord.js/src/client/websocket/WebSocketShard.js:444:22)
at WebSocketShard.onMessage (/home/container/node_modules/discord.js/src/client/websocket/WebSocketShard.js:301:10) {
errno: -2,
syscall: 'open',
code: 'ENOENT',
path: 'this'
}
The fix for this being setting dir_uploader to false in the config.json

I've taken a look into this and fixed the issue caused by #30
I can't reproduce the top issue, what command are you using to start the bot?


bug fix: haveInventory should return false if have index but not have data

also fix the corresponding unit test

all unit test passed.
ci func test fail because of timeout, try re-run func test will ok.


Create IG Route: /accounts/{AccountId}/beneficiaries

Description
An AISP may retrieve the account beneficiaries information resource for a specific AccountId (which is retrieved in the call to GET /accounts).
Issues related

https://github.com/SecureBankingAccessToolkit/SecureBankingAccessToolkit/issues/114

Route Specifications

read/write API: Account and Transaction
Specification link: GET /accounts/{AccountId}/beneficiaries v3.1.8
Endpoint: /accounts/{AccountId}/beneficiaries
Method: GET
API version spec: Beneficiaries v3.1.8
API uri: /rs/open-banking/v3.1.*/aisp/accounts/{AccountId}/beneficiaries

Acceptance criteria

[x] Functional tests passed in nightly environment
[x] Functional tests passed in dev environment
[x] Proper response expected by the spec
[x] Errors expected match with the OBIE specs
[x] Route documented

[x] Route source commented
[x] Add the route documentation to the wiki route section

Functionality implemented in: /securebanking-openbanking-demo/blob/master/config/7.0/obdemo-bank/ig/routes/routes-service/11-ob-account-access.json


Fix trim when importing WAV

Regression from #59170
It seems the trim condition was changed from !loop to (loop_mode != AudioStreamSample::LOOP_DISABLED) which are opposite
This also affects 3.x due to backport, and I have a 3.x version of this fix and can open PR for it if this gets accepted and backport is wanted.
Fixes #75236
3.x version: #78048

Ping, can this get a review?

Could someone take a look at this?

Thanks!

Cherry-picked for 4.0.4.


Support SPIRE certificates for workload -> control-plane traffic

(This is used to request new product features, please visit https://discuss.istio.io for questions on using Istio)
Describe the feature request
The Istio data path now has full support for  using x509 certs delivered from SPIRE to encrypt traffic.
However, as of now there is no way to directly use SPIRE to provision the certificates for communication from the workloads to the control-plane. When running SPIRE as the workload identity distribution mechanism for your infrastructure, it may be expected that ALL traffic be encrypted with certs derived from there, not just data plane traffic.
Describe alternatives you've considered
Alternative mechanisms could include:

Relying on the istio self-signed CA for this traffic. This is not ideal as it is not easily rotated and does not belong to the organization.
Create tooling to manually provision certificates for istiod from the same root of trust that the workloads use for the data path.

Affected product area (please put an X in all that apply)
[ ] Ambient
[ ] Docs
[ ] Dual Stack
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ x ] Security
[ ] Test and Release
[ x ] User Experience
[ ] Developer Infrastructure
Affected features (please put an X in all that apply)
[ ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane
Additional context
There will be two important pieces of this work.

Ensuring that the agent can use the SPIRE certs for not only the data path connections, but the control-plane connections
Deciding how istiod will receive the certs from SPIRE. There are a couple of options here.

Use the SPIRE workload API directly
Use SDS
Add an init container/ sidecar to query one of the above and then save the certs to the file path expected by istiod.

I am happy to elaborate further on the above solutions and their implications in a separate doc but would like to gather some preliminary feedback first. Or alternatively here how others in the community are handling this today.

istio-csr has this problem today and mounts the certificates manually to get around it https://raw.githubusercontent.com/cert-manager/website/master/content/docs/tutorials/istio-csr/example/istio-config-getting-started.yaml

+1 on having some abstraction here that makes it easier to detangle istio CA from the rest of istiod, since absolutely nothing about how that CA works should be istio-specific.
Whether that's SDS or some other interop channel doesn't matter much to me. I think it makes sense to have something here that is less crude than volume mounts, as those are not always acceptable, especially if they contain secrets.

Provide some context on the usage in my workplace:

We want our SPIFFE/SPIRE infrastructure to be the single solution for certificate issuing for all workloads, both control plane date flow and data plane flow.
It is very easy for us to distributed keys and trust bundle since we have spire-agent everywhere.
We also want to control the attestation aspect, which Istio's self sign cert do not provide.
Kubernetes secret is a big no-no. We do not want key materials to be persisted in any where.
File mount offers a rudimental solution. If we have to do it, we can make spire-agent write cert into a memory dir and rotate it periodically. But it does raise the concern of inconsistency during the cut over of rotation.

+1 on reducing the complexity of istio ca and supporting a flexible attestation model for control plane certificates.

The current file mounted cert usage is not quite convenient, Integrate with spire agent is reasonable.
One cent from me, istiod startup should not depend on spire, fetching cert from spire api during bootstrap is risky. Can bring more dependencies on deploying

One cent from me, istiod startup should not depend on spire, fetching cert from spire api during bootstrap is risky. Can bring more dependencies on deploying

I definitely understand where you're coming from here, however if a user has decided that SPIRE is the certificate source than I think that's acceptable. In fact istiod coming up despite missing its xDS CA may mislead users. The other option I could see is failing readiness checks, but I'm not sure how much use istiod is without the xDS server, and we can't start said server without the proper certs.

Agree on the concern of introducing a dependency during startup. And I agree with @EItanya that the reliability on that dependency is what user should be concerned about. The current approach which rely on k8s secret or a mount file arguable will have same reliability concern. It just k8s fails to provide secret or a file missing failures perceived as less likely to happen.
From security perspective, no workload should start if they're not able to attest their identity. For example, it would be hazardous if a malicious actor can start their own istiod without attestation to manipulate the mesh. The workload's cert managed on a separate loop from istiod CA mitigate the risk of man in the middle attack. However, a fake istiod still can do a lot of damage by setting malicious network configurations.

Do you want to add all of this to Istiod? In the past when we did this type of thing we ran an actual sidecar for istio control plane itself

Just for the sake of xds server certificate, we introduce spire to istiod, isn't it heavy? In k8s, the direct and simple usage should be secret mount.  In this way we donot care who signs the certificate.

Kubernetes secret doesn't meet many organizations' security posture. It is least acceptable approach in all available options.

We don't need to introduce SPIRE to istio. The spire-agent already support SDS. Istiod only need to pull secret through a SDS service. There are other SDS implementation as well, such as step-sds.

We don't need to introduce SPIRE to istio. The spire-agent already support SDS. Istiod only need to pull secret through a SDS service.

Through UDS or Network? If UDS, then how can we support external istiod. If network, i think we need another way to make sure the connection with spire is secure. Do not we fall into same certificate management issue?

Why do we need sds here? Can't spire just mount the cert with a csi volume?

Do you want to add all of this to Istiod? In the past when we did this type of thing we ran an actual sidecar for istio control plane itself

I'm not strictly against this, but is this sidecar something officially supported by upstream or custom built?

Through UDS or Network?

I would definitely vote UDS for the reasons you mentioned.

If UDS, then how can we support external istiod.

May I know the concern here? The istiod fetch cert over UDS from spire-agent's CSI mount path. The root CA can be either distributed by spire-agent's trust bundle or simply installed at the caller. SPIRE in production rarely use self-signed certs. It always integrate with a actual CA (such as AWS PCA) where operator has the access to the root CA cert.

Why do we need sds here? Can't spire just mount the cert with a csi volume?

We don't have to. But since there are well established SDS implementation such as step-sds and spire-agent, this can solve the source of the certs.

Can't spire just mount the cert with a csi volume?

SPIRE doesn't do that. We need to build a accessory to fetch cert from spire-agent, place it on a volume. That logic should be just carried in istiod.

If we require people to build and ship a custom sidecar to use a non-default CA, then I feel like we are too tightly coupled to our not-remotely-special-as-CAs-go Istio CA implementation.

If we require people to build and ship a custom sidecar to use a non-default CA, then I feel like that's a signal we are too tightly coupled to our not-remotely-special-as-CAs-go Istio CA implementation.

I somewhat agree but not entirely. Its a difficult balance.
 But then we have almost all of these same needs in Istiod itself, and we end up duplicating them. This includes auth (TLS, mTLS, JWT), observability, etc. When that is a small scope its not so bad, but as it scope creeps (as it has done for years and continues to) it starts to become a bit concerning.

we are too tightly coupled to our not-remotely-special-as-CAs-go

I partially agree, partially disagree.
First, I don't actually consider this to be about the CA. The workload mTLS CA is orthogonal form Istiod's serving certificate generation. I don't believe Istio workload mTLS is coupled to Istio CA, given that there are 5+ CAs that work well in this role (our product, FWIW, does not use Istio CA, so we have a vested interest in not coupling things).
Then there is Istiod's serving certificate. This is very intentionally not coupled to workload mTLS, specifically to avoid tight coupling to Istio CA. This is why when we connect to the control plane we don't use the Root CA passed from the CA workload mTLS certificates, and instead pass the certificate of band (via istio-ca-root-cert) (technically there is also a circular dependency if we don't do this in some setups, so its not the only motivation for doing things this way). Istio already has a pluggable mechanism - PILOT_CERT_PROVIDER. This already has 3 mechanisms: self signed, Kubernetes v1 CSR API, and files.
IMO, adding native support for a 4 option of "use a custom protocol defined by some project" is perhaps not the best move in regards to decoupling. We already provide pluggable mechanisms -- files are the 'universal API', and Kubernetes CSR is a core v1 Kubernetes API. Its not clear why Istio should take on the burden of adding yet another plugin mechanism rather than other projects using standardized mechanisms rather than forcing their own APIs on everyone. If not, a shim can be made to translate from custom API to standard API - cert-manager has already done this, offering both a mechanism to support CSRs and files IIRC.
Put another way, if we had 10 CA vendors all pushing for us to add 10 CA APIs to Istio, should we do it? I would hope not. There is a still a valid position that we need 4 mechanisms instead of 3, and SPIRE happens to be so standard its worthy of that position, but I am not sure I buy it...

Then there is Istiod's serving certificate. This is very intentionally not coupled to workload mTLS, specifically to avoid tight coupling to Istio CA. This is why when we connect to the control plane we don't use the Root CA passed from the CA workload mTLS certificates, and instead pass the certificate of band (via istio-ca-root-cert) (technically there is also a circular dependency if we don't do this in some setups, so its not the only motivation for doing things this way). Istio already has a pluggable mechanism - PILOT_CERT_PROVIDER. This already has 3 mechanisms: self signed, Kubernetes v1 CSR API, and files.

IMO, adding native support for a 4 option of "use a custom protocol defined by some project" is perhaps not the best move in regards to decoupling. We already provide pluggable mechanisms -- files are the 'universal API', and Kubernetes CSR is a core v1 Kubernetes API. Its not clear why Istio should take on the burden of adding yet another plugin mechanism rather than other projects using standardized mechanisms rather than forcing their own APIs on everyone. If not, a shim can be made to translate from custom API to standard API - cert-manager has already done this, offering both a mechanism to support CSRs and files IIRC.

Yeah, the difficulty is that the CSR API is overkill for what needs to happen here. I agree a shim could be done that introduces a controller (or exposes the SPIRE agent as a controller) that approves CSRs and also fulfills them. But that does some violence to the intent of the CSR flow, which is to keep issuers and attestors strictly separate, when that's not how all CAs work. It's also a bit baroque.

However, as of now there is no way to directly use SPIRE to provision the certificates for communication from the workloads to the control-plane. When running SPIRE as the workload identity distribution mechanism for your infrastructure, it may be expected that ALL traffic be encrypted with certs derived from there, not just data plane traffic.

Sorry for chime in late here - wanted to understand the requirement here a bit... For workload<->control plane, do you expect to use the same root CA/spire as workload<->workload?  I asked because I recall Istio purposefully has workload<-> control plane using different sets of key and certs so you can't use those for workload<->workload communication.

If we are concerned about PILOT_CERT_PROVIDER having too many options, I would actually suggest we should drop file from the list and add something like SDS, since both SDS and CSR can effectively work with files.

would actually suggest we should drop our own file impl from the list and add something like SDS

I think you can make the exact same argument and swap 'file' and 'SDS' and it would make sense still. Files are a very universal API (see linux :slightly_smiling_face: ). If you want to use SDS -- no problem. Just communicate over SDS and dump it to a file where Istio can read it.

The point is, istiod really shouldn't care, as long as a valid cert gets pushed to it.

I 100% agree with this, Istiod should not be in the business of adopting vendor specific APIs and should use a truly universal API - files.

The point is, istiod really shouldn't care, as long as a valid cert gets pushed to it.

I 100% agree with this, Istiod should not be in the business of adopting vendor specific APIs and should use a truly universal API - files.

Does istiod's CSR impl use files to push certs to the rest of istiod, as a universal API?

If we decide on a file based solution, then there are caveats about key rotation should come into play: For usage where we rotate key regularly, we'd like to ensure a smooth cut over from new key to old key. For example, we want to place the new key side by side to the old key before we delete the old key. These scenarios are no problem for an API based solution, but tricky to get right in file based solution.

@howardjohn an interesting datapoint here is that the SPIRE folks found mapping into 'file' quite difficult to get right and be secure relative to file group permissions and abandoned the effort
https://github.com/spiffe/spire/issues/2077#issuecomment-1010500154

File mounted certificates are coming to K8s core (https://github.com/kubernetes/enhancements/issues/4317) in 1.30, so it seems like it ought to be a viable option?
The issue was about serverless so I don't know the rationale there applies entirely onto k8s

I'll try to get a doc together for a WG to look at the options and lay them out.

https://docs.google.com/document/d/1diXyh76FMgUTh6Rr79dyJXTMamtsfXwvMtfvT95mX0M/edit?usp=sharing
Doc here in the Istio GDoc share which lays out the options going forward here - I'm not bringing it up for discussion at today's WG since we need to do a few more passes on our end I think, but PTAL and comment/suggest/criticize.

File mounted certificates are coming to K8s core (kubernetes/enhancements#4317) in 1.30, so it seems like it ought to be a viable option?
The issue was about serverless so I don't know the rationale there applies entirely onto k8s

Thanks for linking this, I read thru it (and left some KEP comments). I'm not certain it will actually make it by 1.30 and I'm a little worried some of the constraints implicit in the design around baking cert issuance/rotation opinions into the kubelet will make this both heavier than a basic API like SDS, and less flexible.


When compiling the project, if an entity's association field is not annotated as ManyToMany, ManyToOne, etc., the error message generated is not explicit enough

Entity structure is as follows:

The field connector of SecurityTag is not annotated with ManyToOne, which produced the following error message:
SecurityTag.kt:21: Illegal property "SecurityTag.connector", it is not association property, its target type "Connector" is immutable type, immutable type is not enough, please use @org.babyfish.jimmer.sql.Entity

Try 0.7.122

Fixed.
But the error message for collection attributes is still the previous one.

Try 0.7.123

Try 0.7.123, setDefaultSerializedTypeObjectMapper

Fixed.


added testname to "working directiory" variable in work flow early

this changes where/how variable workingdirectory is created... it now includes the time upfront, and is carried out through the whole work flow, rather than be added later.... this could cause a mess down stream, but is probably the right way.
@thesser1  take a look here.... i don't know where cshore and ww3 add this, but was struggling with it in swash.  I could revert back, but this is what i needed to do to get swash to work

nope this causes errors


Unable to idb.open()

Hi Jake,
Thanks for such amazing library.
It has greatly simplified and enhanced developer experience w.r.t. working with Indexed DB.
It has been awesome so far, until recently we tried running our app on iOS safari and iOS chrome.
Unfortunately site doesn't seem to open at all!!
We examined and found that issue is because of IndexedDB Promised API.
We tried connecting our iPhone and tried remote safari debugger.
It throws following error message.

Later, we also found that it doesn't even open on Desktop Safari!!! Same error!!
Can you please look into the issue and see if you can fix it.
Thanks, Vinay

I can't recreate this on desktop Safari. Which version of desktop Safari are you using?
What do you get in the console when you run http://output.jsbin.com/docoqa/quiet?

Thanks for such quick reply :)
We are running Desktop Safari Version 9.1.3 (11601.7.8)
We just found out that the error occurs only on Incognito mode. This is strange!!
It works without any issues in normal mode both on desktop Safari and mobile (iPhone and iPad on both Safari mobile and Chrome iOS).
Even the link which you gave does not seem to work in Incognito mode.
Can you please check if this can be fixed.
Thanks, Vinay

Unfortunately it can't be fixed. IDB doesn't exist in Safari in incognito mode, or in add-to-homescreen mode on iOS. There's nothing I can't do about it, it's just not there.

Oh okay. Cool no prob.
Thanks for the info :)
Happy Coding. Cya.


Parse by columns instead of rows

I Have a Spreed Sheet where i need to Read the Data Column wise. Tired writing a two loops for the sheet_to_json but the process of execution takes more time.
Existing sheet_to_json = [ {A:1,B:2,C:3} , {A:4,B:5,C:6} ]
Required Format  = [ {A : [1,4] } , {B : [2,5] } , {C : [3,6] } ]

sheet_to_json is pretty simple: https://github.com/SheetJS/js-xlsx/blob/master/bits/90_utils.js#L1
It just loops by row and by column.  If you need to read the other direction, just access the cells the other way around (first loop by column then by row).


[REQUEST] Allow more flexibility on 'install.wim' name for -SourcePath

Is your feature request related to a problem? Please describe.
 Optimize-Offline does not like that and insists on using install.wim.
Describe the solution you'd like
I've poked around the code in Optimize-Offline.ps1 and here's what I've done:

Removed the if/else bloc, lines 229-236 and 250
Changed line 240 to :
Copy-Item -Path $($SourcePath.FullName) -Destination "$ImageFolder\install.wim"

And now everything works fine without any trouble. I can name the install.wim whatever I want, it works flawlessly.
Could you consider making these changes?
Thank you.

I will remove the initial 'install.wim' name requirement and just have Optimize-Offline rename the .wim once it has been copied.


Library versions

Hi! Can you specify exact versions of tensorflow and other libraries you used? Cuda and cudnn versions are important too) Thanks!

CUDA : 9.0
Tensorflow : 1.8
Thank you


pinging to pick up items shows the colour code

https://i.imgur.com/0TPXheu.png
on ctrl + space to pick up

actually a gw bug, not a toolbox bug. happens without toolbox as well, specifically with rock candies.
could still fix it though

Also happen with zaischen coins if I remember correctly.


Full Finetune with Unsloth

I am just curious whether the current unsloth support the full finetune.
So when I am experimenting training tinyllama model on 24GB vram GPU right now. Using unsloth to just load the model without lora or anything would only take about 10GB vram. But when I use transformer's AutoModelForCausalLM it would be close to 24GB vram. It seems that unsloth works well for full fine tune even with just load with FastLanguageModel?
I know current version claimed that it is not supporting full fine tune yet, but I wonder whether it is a full fine tune through just loading with FastLanguageModel.
Basically I just load the model and tokenizer with FastLanguageModel.from_pretrained
Then I directly use the model in SFTTrainer.
And the memory is significantly less.

In theory it works, but some weights will not be trained - ie the RMS Layernorm weights and weights for the MLP layers - you could ignore .get_peft_model and I guess it could partially work

@danielhanchen Hi may I know whether Unsloth still does not support finetune today? Thanks!

Made a quick experiment as below. Seems that the layer norm weights are never changed, while other parameters are changed.

@danielhanchen I am happy to PR to make the layernorm work (if it is the only missing piece)!

@danielhanchen I am happy to PR to make the layernorm work (if it is the only missing piece)! IMHO full finetune is really frequently needed, and with small models like qwen2.5-0.5B or qwen2.5-math-1.5B, it is possible to full finetune on cards like rtx4090.

@fzyzcjy The layernorm weights should be a bit complex I guess to implement - ie I think they're just the sum of the gradients in the correct axis (maybe axis = 1?) ie add up all the rows.
But one has to first derive the gradient for the weights ie dC/dW which I skip, and instead just do dC/dX

@danielhanchen No worries, if it is implementable I think I will try to do that.
But the first (and most important) problem: Is there any other missing pieces to make unsloth do full finetuning?
(Wait a few minutes and I will create a new issue saying more details)

My thoughts here: https://github.com/unslothai/unsloth/issues/1176

Hello! Just wondering if there was any news on this.
Being able to full-finetune with Unsloth would be great -- even if it is not as fast as when using LoRA.
Thanks!

Hello! same as tjr214 here. Just want to check if there was any news on this.
Being able to full-finetune with Unsloth would be great !!!


Fix arguments handling (fixes #289)

This is an attempt to fix #289 , implementing proper handling of arguments.
It's currently a work in progress, I'm creating the pull request to run the tests on Travis.
The changes are:

[x] fix name handling for closure functions
[x] throw TypeError for unexpected args
[x] throw TypeError for missing args

@freakboy3742 It looks like the build failed because it was too long, Travis error was: Your test run exceeded 120 minutes. One possible solution is to split up your test run.

Yeah - we need to do something about the timing on the test suite. I've restarted that specific build; hopefully it will finish slightly faster.

There's something really odd going on here. The test passed on rebuild, but is marked as a failure. I'm going to merge anyway, and hope this clears up on a fresh build.

Nice, the merge commit passed.
Thanks, and also thanks @juliomfreitas for the help digging into this issue! :+1:


updated linting / flow

@keybase/react-hackers updates flow and lint. no new errors

i'm removing glow as flow itself has solved its output problem


Python 3.6 name error

Traceback (most recent call last):
  File "/venv/lib/python3.6/site-packages/nodata/blob.py", line 130, in blob_nodata
    rm.run(workers)
  File "/venv/lib/python3.6/site-packages/riomucho/__init__.py", line 109, in run
    for data, window in self.pool.imap_unordered(reader_worker, self.windows):
  File "/venv/lib/python3.6/site-packages/riomucho/single_process_pool.py", line 9, in imap_unordered
    yield reader_worker(i)
  File "/venv/lib/python3.6/site-packages/riomucho/__init__.py", line 34, in manualRead
    raise e
  File "/venv/lib/python3.6/site-packages/riomucho/__init__.py", line 31, in manualRead
    return work_func(srcs, window, ij, global_args), window
  File "/venv/lib/python3.6/site-packages/nodata/blob.py", line 48, in blob_worker
    if isinstance(globalArgs['selectNodata'], (int, long, float)):
NameError: name 'long' is not defined

long is gone in python 3.6. Instead we should use a more descriptive type check here, works on py2.6+:
isinstance(n, numbers.Number)

resolved in #64


Possible to just stream direct

Is it possible to add an option to just stream direct from the camera URL rather than piping through FFMPEG?
Thanks.

The way you describe is the ideal way to do HomeKit video streaming. However, since video stream is highly dependent on the physical camera and the formats it supported, it's better for people to develop optimized plugins for specific cameras to do direct streaming. This project will not try to go with this way.


Compare performance against pure delegate and method call

Method
Mean
Error
StdDev
Allocated

UseMethod
0.0408 ns
0.0023 ns
0.0021 ns
-

UseDelegate
0.8447 ns
0.0079 ns
0.0070 ns
-

UseDelegateCompiledWithFastExpressionCompiler
0.9269 ns
0.0039 ns
0.0036 ns
-

UseSpecificationDelegate
0.6344 ns
0.0066 ns
0.0059 ns
-

UseSpecificationDelegateCompiled
1.2099 ns
0.0063 ns
0.0056 ns
-

Predicatable results, delegates compiled from expressions are slower than delegates.

Expression.Compile creates a DynamicMethod and associates it with an anonymous assembly to run it in a sand-boxed environment. This makes it safe for a dynamic method to be emitted and executed by partially trusted code but adds some run-time overhead.


Add Indonesian Translation

Also I found that on the contribution section there are yarn typescript on several part, but it's not stated on the package.json. I think it should be yarn typecheck instead.

Also I found that on the contribution section there are yarn typescript on several part, but it's not stated on the package.json. I think it should be yarn typecheck instead.

Hey @ddikodroid, thanks for the pull request! If you think there is an issue with yarn feel free to open another pull request. In the meantime, I'll go ahead and move this through.


Error when inputType: number is used with maxLength in TextInput

Describe the bug
Getting this error when inputType: number is used with maxLength in TextInput
React Router caught the following error during render DOMException: Failed to execute 'setSelectionRange' on 'HTMLInputElement': The input element's type ('number') does not support selection.

You will get the error when you type more words than the limit
e.g if limit is 5 try to write 6 words
Screenshots

@sagardspeed2 can you take a look at this one, should be simple

@evshi  In the count configuration, you've used the exceedFormatter, which is a part of the Ant Design Input widget. But inside, the widget is using the rc-input plugin to render the input. Now, within this plugin, they're using the setSelectionRange method on the input element.
According to vanilla JavaScript, the setSelectionRange method is not supported on input elements of type number. Because of this, we are facing the error.

@sagardspeed2 what is the solution here then?

@evshi I tested all other options with input, like max, inputMode but none of them are actually work in way we want, so I suggest we can render https://ant.design/components/input-number this widget on input type number

@sagardspeed2 how would that fix the issue w/ maxLength? Perhaps it's better to use tel type instead as suggested in this SO thread https://stackoverflow.com/questions/33406169/failed-to-execute-setselectionrange-on-htmlinputelement-the-input-elements

If you can make the behavior work with input number let's go with that

Sure, I will make that input number behaviour work

seems like this is still not fixed, still getting the error @sagardspeed2

@sharjeelyunus can you share screenshot or video of this

Error: React Router caught the following error during render InvalidStateError: Failed to execute 'setSelectionRange' on 'HTMLInputElement': The input element's type ('number') does not support selection.
https://github.com/user-attachments/assets/4857e011-d66a-4bc6-aad4-9e19facad0c2

@sharjeelyunus I just checked on ensemble studio dev environment and kitchen sink in react ensemble.
it is working very fine without any issue
https://github.com/user-attachments/assets/3f684695-da94-40a8-8a6e-479b14f7ff02


feat: logging framework

Issue being fixed or feature implemented
What was done?
How Has This Been Tested?
Breaking Changes
Checklist:

[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have added or updated relevant unit/integration/functional/e2e tests
[ ] I have made corresponding changes to the documentation

For repository code-owners and collaborators only

[ ] I have assigned this pull request to a milestone

uACK

uACK

Although, I am not done yet. The PR is still in draft :)

There was similar [solution] already for logging (https://github.com/dashpay/dash-shared-core/blob/8a8c03caad1a39965cb4f426c2796a72e241992d/dash-spv-masternode-processor/src/bindings/common.rs#L16C1-L17C1)... Why should we have 2?


Shake invokes my rule with exceptions masked!

Steps to reproduce:

Download and build https://github.com/ezyang/ghc-shake/ following the instructions (in particular need a patched version of Shake)
Apply this patch to ghc-shake

diff --git a/GhcShake.hs b/GhcShake.hs
index 9ae1d23..1d97241 100644
--- a/GhcShake.hs
+++ b/GhcShake.hs
@@ -268,7 +268,8 @@ doShake args srcs = do
     -- name.

     buildModuleRule $ \bm@(BuildModule raw_file mod is_boot) -> do
-
+        x <- liftIO $ getMaskingState
+        liftIO $ print x
         -- Make sure we rebuild if -this-unit-id changes
         _ <- askThisPackage

and build.

Run ghc-shake --make All.hs -O -fforce-recomp -j3 +RTS -N3 with the following modules:

-- A.hs
module A where
-- B.hs
module B where
-- All.hs
module All where
import A
import B

Expected result: masking state is reported to be unmasked in all cases.
Actual result: masking state is unmasked for one, masked for two others!
Sorry for not having a reduced test case, more investigation needed.

Eeek!
I just modified Shake to raise an error if any rules are started with exceptions unmasked. The test suite passed fine. Therefore, one of:

It's something you are doing in your system that I don't do in any of my tests.
Or it's something I've fixed in HEAD, which has diverged somewhat since your version (HEAD is in a very fluid state, so I don't recommend you merge to it at this point). That said, I haven't really gone near exception stuff in the refactoring.

It's certainly something to fix, but if there's no obvious way to figure out what's going on your side, I suggest I finish refactoring HEAD, then you merge to that (which has other benefits anyway), then we see if it still occurs.

OK, I was able to repro on my Shake branch using the introductory Shake example. Using this diagnostic patch:
diff --git a/src/Development/Shake/Pool.hs b/src/Development/Shake/Pool.hs
index ac758f5..bb3d237 100644
--- a/src/Development/Shake/Pool.hs
+++ b/src/Development/Shake/Pool.hs
@@ -98,6 +98,7 @@ emptyS n deterministic = S Set.empty n 0 0 $ newQueue deterministic
 step :: Pool -> (S -> NonDet S) -> IO ()
 step pool@(Pool var done) op = do
     let onVar act = modifyVar_ var $ maybe (return Nothing) act
+    (\s -> print ("z",s)) =<< getMaskingState
     onVar $ \s -> do
         s <- op s
         res <- maybe (return Nothing) (fmap Just) $ dequeue $ todo s

then I get this:
ezyang@sabre:~/Dev/ghc-shake$ /srv/code/ghc-shake/dist-newstyle/build/ghc-shake-<IP_ADDRESS>/build/shake-test/shake-test
("z",Unmasked)
("z",Unmasked)
("z",MaskedInterruptible)
("z",MaskedInterruptible)
("z",MaskedInterruptible)
("z",MaskedInterruptible)
("z",MaskedInterruptible)
("z",MaskedInterruptible)
("z",MaskedInterruptible)
# gcc (for _build/run)
gcc: fatal error: no input files
compilation terminated.
("z",MaskedInterruptible)
("z",MaskedInterruptible)
Error when running Shake build system:
* _build/run
user error (Development.Shake.cmd, system command failed
Command: gcc -o _build/run
Exit code: 1
Stderr:
gcc: fatal error: no input files
compilation terminated.
)

Next step, I'll try with Shake master.

And for Shake master, I see something that looks a lot better:
ezyang@sabre:~/Dev/shake-master$ /srv/code/shake-master/dist-newstyle/build/fooshake-<IP_ADDRESS>/build/fooshake/fooshake
Unmasked
Unmasked
Unmasked
Unmasked
Unmasked
Unmasked
Unmasked
Unmasked
Unmasked
Unmasked
# gcc (for _build/src/Test/Tup/hello.o)
src/Test/Tup/hello.c:2:20: fatal error: square.h: No such file or directory
compilation terminated.
MaskedInterruptible
Unmasked
Error when running Shake build system:
* _build/run
* _build/src/Test/Tup/hello.o
user error (Development.Shake.cmd, system command failed
Command: gcc -c src/Test/Tup/hello.c -o _build/src/Test/Tup/hello.o -MMD -MF _build/src/Test/Tup/hello.m
Exit code: 1
Stderr:
src/Test/Tup/hello.c:2:20: fatal error: square.h: No such file or directory
compilation terminated.
)

Though, there is one MaskedInterruptible snuck in there.

Looking at the trace, you are tracing the step function of the thread pool, which is not where it runs user actions. I think you should be tracing:
diff --git a/src/Development/Shake/Internal/Core/Pool.hs b/src/Development/Shake/Internal/Core/Pool.hs
index 33ce16d..67c2137 100644
--- a/src/Development/Shake/Internal/Core/Pool.hs
+++ b/src/Development/Shake/Internal/Core/Pool.hs
@@ -116,7 +116,7 @@ step pool@(Pool var done) op = do
         case res of
             Just (now, todo2) | Set.size (threads s) < threadsLimit s -> do
                 -- spawn a new worker
-                t <- forkFinally (now >> worker pool) $ \res -> case res of
+                t <- forkFinally (getMaskingState >>= \s -> print s >> now >> worker pool) $ \res -> case res of
                     Left e -> onVar $ \s -> do
                         t <- myThreadId
                         mapM_ killThread $ Set.toList $ Set.delete t $ threads s

Since this is the place I actually run user actions, and there I only see Unmasked. In fact, because it always spawns a new thread, I can't see how you'd ever get a masked state for the users code.
Or have I got that wrong? Clearly something is masking somewhere...

I finally found this and fixed it as 0a170059f4ffbdbc2678a866d4cf9d1eb1b5fe82. I was forking a thread, and forgot the mask status of the parent leaked into the masked state of the new thread. I can replicate this is a bigger program, but not in my test suite, so still working out how to test for it in general.

I added a test, even though it doesn't fail normally. That will have to do...


Refactoring

Apart from minor refactoring, there is one visible change.

Make available offline temporarily retired, due to potential alternative approach via Cloud solutions such as Paperspace, Amazon Workspaces, Google Cloud Compute and MetaPipe.

Coverage increased (+1.2%) to 36.495% when pulling f0418111432899038c43cdba7b50727ebc22a0b0 on mottosso:refactoring into c24f34a3594f65dc606023ec7e93453db1c24f98 on mindbender-studio:master.


restart with repo swap test

@zacharym this tests that restart succeeds with a new NFS mount location

lgtm


[Content item] v2 spec update

Follow on to #10360
Update the content item specs to include the following:

[ ] Carbon v11 tokens
[ ] Updated spacing for vertical options
[ ] Added vertical options
[ ] Added horizontal orientation and options

Updated vertical options

New horizontal option example – image

hi! I'm  a begginer to opensource contribution. I learnt reactjs recently. I would like to work on the frontend part. could you explain the problem so that I could help?


run saucectl by default in sauce mode and with concurrency equals to 2

Proposed changes

Types of changes

[ ] Bugfix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation Update (if none of the other choices apply)

Checklist

[x] Lint and unit tests pass locally with my changes
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have added necessary documentation (if appropriate)

Further comments

Codecov Report

Merging #244 (897a1e4) into master (4717c72) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #244      +/-   ##
==========================================
+ Coverage   36.82%   36.84%   +0.01%
==========================================
  Files          36       36
  Lines        2308     2315       +7
==========================================
+ Hits          850      853       +3
- Misses       1312     1316       +4
  Partials      146      146

Impacted Files
Coverage Δ

cli/command/run/cmd.go
27.77% <100.00%> (+0.26%)
:arrow_up:

internal/saucecloud/testcafe.go
7.69% <0.00%> (-0.21%)
:arrow_down:

internal/saucecloud/playwright.go
7.14% <0.00%> (-0.18%)
:arrow_down:

internal/config/config.go
70.90% <0.00%> (ø)

internal/saucecloud/cypress.go
72.54% <0.00%> (+0.54%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4717c72...897a1e4. Read the comment docs.

LGTM. Please also check the sauce docs for anything that needs updating because of this 📚

ok


build: publish docs should use personal access token

The release step doesn't successfully publish documentation so I updated the token to a personal access token vs the github actions GITHUB_TOKEN

[x] Bug fix
[ ] New feature
[ ] Chore
[ ] Breaking change
[ ] There is an open issue which this change addresses
[ ] I have read the CONTRIBUTING document.
[ ] My commits follow the Git Commit Guidelines
[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.

[ ] I have updated the documentation accordingly.

[ ] My change requires a change to Typescript typings.

[ ] I have updated the typings accordingly.

[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

Merging this as the build step is broken right now!


Unexpected Behavior: Training Continues Beyond Specified Epochs

I am encountering an unexpected behavior while training the RWKV 14B language model using the provided train.py script. The training process continues beyond the specified number of epochs, even though I've set --epoch_count to 8 and --epoch_steps to 1000. The training process is currently at epoch 1009.
Code Context:
I'm using the following command to initiate the training:
import os

def run_training():
    load_model = "/home/dev/Workspaces/runner/models/RWKV-4-Raven-14B-v12-Eng98%-Other2%-20230523-ctx8192.pth"
    proj_dir = "out"
    data_file = (
        "/home/dev/Workspaces/runner/finetune/data/notes/101_Formulaic_Alphas_2015.txt"
    )
    data_type = "utf-8"
    micro_bsz = 11
    epoch_count = 8
    epoch_steps = 1000
    devices = "1"
    strategy = "deepspeed_stage_2_offload"
    grad_cp = 1

    command = (
        f'python train.py --load_model "{load_model}" --wandb "" --proj_dir "{proj_dir}" '
        f'--data_file "{data_file}" --data_type "{data_type}" --vocab_size 0 '
        f"--ctx_len 128 --epoch_steps {epoch_steps} --epoch_count {epoch_count} --epoch_begin 0 --epoch_save 10 "
        f"--micro_bsz {micro_bsz} --n_layer 12 --n_embd 768 --pre_ffn 0 --head_qk 0 "
        f"--lr_init 1e-4 --lr_final 1e-4 --warmup_steps 0 --beta1 0.9 --beta2 0.99 --adam_eps 1e-8 "
        f'--accelerator gpu --devices "{devices}" --precision bf16 --strategy "{strategy}" --grad_cp {grad_cp}'
    )

    os.system(command)

if __name__ == "__main__":
    run_training()

Expected Behavior:
I expected the training process to stop after completing 8 epochs, with each epoch consisting of 1000 steps. However, the training process is continuing beyond this point, and I'm unsure why.
Troubleshooting Steps Taken:

I reviewed the provided train.py script, and I couldn't identify any specific issues that might be causing this behavior.
I verified the data loader's behavior and ensured that it's aligned with the expected number of steps per epoch.
I double-checked the checkpoint loading process, and it appears to be functioning correctly.
I confirmed that there are no early stopping criteria implemented in the codebase.

Environment:

Python version: 3.10.12
PyTorch version: 1.13.1
PyTorch Lightning version: 1.9.5
GPU: NVIDIA GeForce RTX 3060 12GB
CPU: AMD Ryzen 7 3700X 8-Core Processor

Additional Context:
I'm training the model using a TXT file a,d I'm unsure if that affects the training process. I also had a lot of remarks about the ridiculous lose rate the model is training at, I'm new to all this so I don't fully comprehend the details
Current Output
  | Name   | Type       | Params
--------------------------------------
0 | emb    | Embedding  | 96.0 K
1 | blocks | ModuleList | 92.1 M
2 | ln_out | LayerNorm  | 1.5 K
3 | head   | Linear     | 96.0 K
--------------------------------------
92.3 M    Trainable params
0         Non-trainable params
92.3 M    Total params
369.236   Total estimated model params size (MB)
Epoch 0:   0%|                                         | 0/1000 [00:00<?, ?it/s]
{'zero_allow_untested_optimizer': True, 'zero_optimization': {'stage': 2, 'contiguous_gradients': True, 'overlap_comm': True, 'allgather_partitions': True, 'reduce_scatter': True, 'allgather_bucket_size': 200000000, 'reduce_bucket_size': 200000000, 'sub_group_size':<PHONE_NUMBER>000, 'offload_optimizer': {'device': 'cpu', 'nvme_path': '/local_nvme', 'buffer_count': 4, 'pin_memory': False}}, 'activation_checkpointing': {'partition_activations': False, 'cpu_checkpointing': False, 'contiguous_memory_optimization': False, 'synchronize_checkpoint_boundary': False}, 'aio': {'block_size': 1048576, 'queue_depth': 8, 'single_submit': False, 'overlap_events': True, 'thread_count': 1}, 'gradient_accumulation_steps': 1, 'train_micro_batch_size_per_gpu': 8, 'gradient_clipping': 1.0, 'bf16': {'enabled': True}}

Epoch 590:  44%|████████▎          | 438/1000 [02:18<02:57,  3.17it/s, loss=0.058, lr=0.0001, REAL it/s=3.200, Kt/s=3.280]
Epoch 1013:  94%|███████████████▉ | 935/1000 [05:11<00:21,  3.00it/s, loss=0.0574, lr=0.0001, REAL it/s=2.980, Kt/s=3.050]

yes, it's a bug

Thank you for your quick response! I appreciate your confirmation that this is indeed a bug. Could you please provide some additional details about the nature of the bug and whether there's a plan to address it?
We're kinda playing on time right now so that'll be greatly appreciated

change https://github.com/josStorer/RWKV-Runner/blob/master/finetune/lora/train.py#L187
args.max_epochs = -1 to args.max_epochs = args.epoch_count

https://github.com/josStorer/RWKV-Runner/releases/tag/v1.4.5


different result over running different version of tensorflow

Hi again,
Finally I got the reason why I can not replicate your result. until now I was using tensorfloe 1.14. which in this version we get 69 percent accuracy on 20 newsgorup data set with 512 nd.
However, once I changed the tf version back to 1.2.1 it replicate the same result as reported in paper.
an example of extracted topics with version 1.2.1
line subject organ armenian isra turkish israel armenia arab turkey

line subject organ car bike speed brake truck ride motorcycl

line organ subject clipper escrow chip livesey isc cramer key

line subject organ cramer optilink gay clayton homosexu gun crimin

line subject organ april space san sexual washington cal ron

I appreciate it if you can help with this part, why that much difference with only changing the version of tensorflow!
Thanks.

Hi again,
Finally I got the reason why I can not replicate your result. until now I was using tensorfloe 1.14. which in this version we get 69 percent accuracy on 20 newsgorup data set with 512 nd.
However, once I changed the tf version back to 1.2.1 it replicate the same result as reported in paper.
an example of extracted topics with version 1.2.1
line subject organ armenian isra turkish israel armenia arab turkey

line subject organ car bike speed brake truck ride motorcycl

line organ subject clipper escrow chip livesey isc cramer key

line subject organ cramer optilink gay clayton homosexu gun crimin

line subject organ april space san sexual washington cal ron

I appreciate it if you can help with this part, why that much difference with only changing the version of tensorflow!
Thanks.

Thank you for reporting this. One quick check I would suggest is to load and test the model pretraind using the original TF version in the new TF version to see if you can reproduce the results.
If not, it means that even in the inference time, the two versions produce different results.  In this case, you might want to further find which function call makes the difference.
If yes, then maybe the training dynamics or weight initialization in the new version is different from the old version. And you might want to dig it deeper.
Hope this helps.


Seems like some refactor didn't go well

Starting harpoon says project is nil,
Switching to terminal says cmd is nil
I would update comment if I see some other broken modules

Sorry, I messed up some config in mine. Only Project is broken!
There is a already a PR that fixes that. So closing the issue.
Thanks!


Increase query information (when executed, why failed)

https://cockroachlabs.atlassian.net/browse/CRDB-382

@awoods187
The JIRA epic for this (https://cockroachlabs.atlassian.net/browse/CRDB-382) has the feature marked a "In review/testing". Is this feature complete for v20.2? I haven't seen any docs release note issues for it.

This didn't make it into 20.2

Moving this to Later.


Add some Pre-War Buildings

Add a few special pre-war buildings that can't be constructed, perhaps starting on the map or "discovered"/added by events.
Just a few for the PoC. More later.

Example: Bottling Plant that gives Nuka-Cola and caps/minting.


fix: remove rti from navigations

What I did

removed RTI from rh-subnav,rh-navigation-secondary, uxdot-sidenav as previously discussed with @hellogreg and @adamjohnson
Corrects rh-subnav icons.
added accessible-label to rh-subnav and rh-navigation-secondary.

Testing Instructions

Notes to Reviewers

considering moving the docs uxdot-sidenav fix to its own PR off main to get it out to production faster.

Should we remove from accordion as well @adamjohnson ?

considering moving the docs uxdot-sidenav fix to its own PR off main to get it out to production faster.

Yes, let's get the docs uxdot-sidenav fix in a separate PR.


'utmx is undefined' error in IE

This error pops up when visiting 7.3 in IE:

Seems like it has something to do with google website optimizer (http://www.analyticsmarket.com/blog/website-optimizer-with-google-analytics)


Bug Test

Test Bug

@NinaWendy same issue

@NinaWendy same issue

@Zolix-ai You are right I see that the last run was the one I initiated. I am looking into what could be the problem because on my end when I create a pull request it initiates CircleCI.