PyActiveStorage and Reductionist: parallel performance, general aspects

Hi @markgoddard how's things, mate? Got a very quick question, and thought I'd open an issue since that may benefit of a discussion if the answer is not a simple "no". Can we run Reductionist on multiple threads, of which max threads be specified in the request_data dict? Funny thing is I actualy added num_threads: 100 in the request_data, and our chunks are being analyzed at an average speed 2.5x faster than in normal situations, but I get this message right at the very end of the run, before getting the result:
Traceback (most recent call last):
  File "/home/valeriu/run_gold_test_bigfile.py", line 46, in <module>
    gold_test()
  File "/home/valeriu/run_gold_test_bigfile.py", line 40, in gold_test
    result = active[:]
             ~~~~~~^^^
  File "/home/valeriu/PyActiveStorage/activestorage/active.py", line 158, in __getitem__
    return self._via_kerchunk(index)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/valeriu/PyActiveStorage/activestorage/active.py", line 300, in _via_kerchunk
    return self._get_selection(index)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/valeriu/PyActiveStorage/activestorage/active.py", line 348, in _get_selection
    return self._from_storage(stripped_indexer, drop_axes, out_shape,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/valeriu/PyActiveStorage/activestorage/active.py", line 397, in _from_storage
    result = future.result()
             ^^^^^^^^^^^^^^^
  File "/home/valeriu/miniconda3/envs/pyactive/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/valeriu/miniconda3/envs/pyactive/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/valeriu/miniconda3/envs/pyactive/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/valeriu/PyActiveStorage/activestorage/active.py", line 520, in _process_chunk
    tmp, count = reductionist.reduce_chunk(session,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/valeriu/PyActiveStorage/activestorage/reductionist.py", line 75, in reduce_chunk
    decode_and_raise_error(response)
  File "/home/valeriu/PyActiveStorage/activestorage/reductionist.py", line 203, in decode_and_raise_error
    raise ReductionistError(response.status_code, error)
activestorage.reductionist.ReductionistError: Reductionist error: HTTP 400: {"error": {"message": "request data is not valid", "caused_by": ["Failed to deserialize the JSON body into the target type", "num_threads: unknown field `num_threads`, expected one of `source`, `bucket`, `object`, `dtype`, `byte_order`, `offset`, `size`, `shape`, `order`, `selection`, `compression`, `filters`, `missing` at line 1 column 214"]}}
Command exited with non-zero status 1

Cheers very much in advance :beer:

Hi @valeriupredoi, nice to see you're trying to drive the performance up.
The short answer is that the exception is correct. You can't pass num_threads to Reductionist and the request will fail if you do. The API is described here. If you're still getting the correct result (and in less time), then I can only assume you're falling back to a local calculation.
The long answer is:
Reductionist uses the Tokio asynchronous runtime rather than threads. Each individual request is processed sequentially, but many requests may be processed concurrently. So it's important that the client is able to send many requests to the server concurrently in order to see good performance. I added parallel request support to PyActiveStorage some time ago, but it would be good to confirm that it's working.
In general with async runtimes a problem can be that there is too much concurrency, meaning that the server may accept too many requests and exhaust resources (e.g. memory) trying to fulfill them. I did some work on trying to limit the concurrency in a controlled way, but it is not enabled by default.
I'm happy to help you look into the performance if you think it's not working well.

Well, the one thing i would say is that we can't be failing back locally. I don't know what caused these results, but it wasn't that!

Hi @markgoddard - great explanation, very many thanks! Since I am here, let me pick your brains, when you have a bit of time to spare: consider two cases - one extremo wrt HDF5 chunking, and another fairly standard, let me explain (also have a look at the detailed test results we put together at https://github.com/valeriupredoi/testing_PyActiveStorage )
Extreme HDF5 chunking: 3400 tiny chunks (of about 70kb each)
We (PyActiveStorage/Active with Reductionist) use 150 threads to select, process, and eventually send all these 3400 tiny chunks to Reductionist; here is the timeseries of how long it takes Reductionist to process each of these chunks (note of course time is measured via Active, ie time for response = request(session, url, request_data) to arrive, so it includes network time to and from the remote Reductionist server, but that should be tiny, order miliseconds at wors):

and the same timeseries, but with Active running just one thread:

Running Active with 150 threads: these are tiny chunks, and Active is machine gunning them to Reductionist with 150 active threads: Reductionist seems to get oversaturated and for a large number of chunks Reductionist takes on average about 5s per chunk, and for the rest of them it takes about 1s per chunk
Running Active with just ONE thread (serial run): the timeseries for Reductionist is a lot cleaner and it shows a lot more stability, with most of the chunks taking about 0.2s each; problem is that the total runtime is a lot longer than with 150 threads since this is a serial run

So there is factor 5-10 variability for Reductionist's time to process a tiny 70 odd kb chunk, depending how much load it receives (and this is just from one client!) - in fairness, the 3400 chunks file with such tiny chunks is silly, but it serves as an excercise of testing the limits of our pipeline.
(I'll post the stuff about the other file, the 64 chunks one in the next comment so we don't overextend comments) :beer:

Decent chunking: 64 chunks (of about 4MB each)
Here things are relatively better in terms of how long it takes Reductionist to analyze one chunk ie there is no such distinct separation between populations of times, but yet still, running Active with 150 threads is very bad for Reductionst; here is the Reductionist time to process one chunk vs chunk size plot, when running with 150 threads:

and the same sort of plot but with 1 thread ie serial run:

Here too one sees about order 6 variability for average Reductionist times per chunk, depending how much load we send it at one given time.
Note that in the case of a serial Active run, it takes about the same 0.2-0.3s for Reductionist to process either a chunk of 70kB or a chunk of 4MB - which is very cool - but we need to think of ways to optimize its parallel processing since, as it is now, I don't think it's ready yet for a serious thrashing by a number of clients, each running silly small chunked data (OK not 70kB but 1MB default HDF5 chunks haha). What do you think? :beer:

just to be clear: we deployed Reductionist stock (the version from a month ago) - no tweaks or customization, so we (I) don't mess up anything I don't know how to fix :grin:

Hi @valeriupredoi, I've read through the milestone report and the data you've shared here. It's certainly interesting to see it used and the performance analysed. The bimodal chunk processing time distribution is very interesting, and definitely digging into further.
I think overall though it's definitely worth considering what we're trying to optimise for here. My assumption was something like this:

Use server resources to process requests and return responses as fast as possible
Try not to kill the server in the process (e.g. due to memory exhaustion)
Best effort for fairness when processing requests

Consider a simplified setup where the server processes requests sequentially. If the client concurrently sends 100 requests, the server will process them one by one, returning responses when ready. In this case the request latency seen on the server is proportional to the position of the request in the queue, giving a roughly linear plot. This is actually what we see in this graph.
Adding Reductionist's async processing into the setup makes it harder to reason about, especially without the resource management enabled. However we might simplify it as adding some parallelism into the queue processing. This would result in another linear plot with a different gradient.
If we consider the 3400 chunk plot, one key difference here is that the number of chunks exceeds the number of threads. So there is a new behaviour here that we did not see before - Python threads finishing their chunk, then picking up another. I wonder whether this accounts for the some of the bimodal nature?

Hi @markgoddard - hope you had a good Easter break! Sorry for the delay in responding, we all took a few days off around the time J. H. Christ decided to get on an American Airlines flight and take off on his annual trip :grin:
Very good points! We are now almost ready to test the following setup: @bnlawrence has created two more virtual machines (accessible via the main VM that we've been testing with, with the current remote Reductionist), so we are planning on deploying Reductionist on those two new ones, via a slightly adapted Ansible deployment. That way, we are hoping the chunks sent by the client will be distributed among the now three Reductionists. I am not Ansible-literate myself (Bryan is though), and wanted to ask you, how can I modify the inventory configuration so the deployment happens on two or more machines at the same time, please? Ie this file https://github.com/stackhpc/reductionist-rs/blob/main/deployment/inventory
Cheers muchos :beers:

To be specific, the configuration is that we have three nodes one of which has a public IP address. So, active is the public facing node (with two IP addresses), with active2 and active3 only having one each. They can all see the S3 server, but we want the public IP address to be the haproxy address.

Hi @valeriupredoi & @bnlawrence. Please see the example in the deployment docs. In your case it might look like this:
[haproxy]
active

[jaeger]
active

[prometheus]
active

[reductionist]
active
active[2:3]

[step:children]
reductionist

[step-ca]
active

[docker:children]
haproxy
jaeger
minio
prometheus
reductionist
step-ca

Thanks. But I think the issue is we were not sure what was going on with the terms that look like:
[haproxy]
localhost ansible_connection=local

what is the ansible_connection doing there?

The example inventory is for the simple case where you are running ansible on the same host that you want to deploy reductionist on. ansible_connection=local is used to force use of a local shell rather than SSH for simplicity.
If you're using multiple hosts then you shouldn't use ansible_connection=local for any remote hosts. If the hostnames are not resolvable you can set ansible_host= for them in the inventory. You'd need to specify each host on its own line in order to do that, rather than using the active[2:3] shorthand.

hang on - I think I know what's the hap here - proves out I don't have docker on this new activeh machine, wish it'd told me in a more straightforward way :rofl:

Hi @valeriupredoi. Looks like you missed this step when setting up the new host as an Ansible control host:
ansible-galaxy collection install -r deployment/requirements.yml

Make sure you've run all the steps from https://stackhpc.github.io/reductionist-rs/deployment/#ansible-control-host-setup too.

The inventory looks fine, although to be clear I might specify activeh rather than localhost for these groups:
[jaeger]
activeh

[minio]
activeh

[prometheus]
activeh

Regarding hostnames vs IPs, it doesn't matter, as long as Ansible can reach the host. If you have DNS, great. Otherwise you could add entries to /etc/hosts on activeh for active2 and active3. Or you can set ansible_host=<IP> in the inventory, e.g.
[haproxy]
activeh
active2
active3

[jaeger]
activeh

[minio]
activeh

[prometheus]
activeh

[reductionist]
activeh ansible_host=<IP_ADDRESS>
active2 ansible_host=<IP_ADDRESS>
active3 ansible_host=<IP_ADDRESS>

[step:children]
reductionist

[step-ca]
activeh
active2
active3

[docker:children]
haproxy
jaeger
minio
prometheus
reductionist
step-ca

many thanks @markgoddard - I had to do some pip magic, I replaced pip3 with pip and now all goes fine, well, all apart from deployment :laughing: I can only manage to have it attempt to deploy on only one of the three machines, usually active3 but it's flaky - sometimes it finds a reachable connection to active2 too, but in either case it fails with
TASK [Gathering Facts] *****************************************************************************************************************************
ok: [active3]

TASK [Ensure /etc/haproxy directory exists] ********************************************************************************************************
changed: [active3]

TASK [Ensure haproxy.cfg is templated] *************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible.errors.AnsibleUndefinedVariable: 'dict object' has no attribute 'default_ipv4'. 'dict object' has no attribute 'default_ipv4'
fatal: [active3]: FAILED! => {"changed": false, "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute 'default_ipv4'. 'dict object' has no attribute 'default_ipv4'"}

Note that I had trimmed the [step-ca] section to only one machine, otherwise I kept getting
TASK [Gathering Facts] *****************************************************************************************************************************
ok: [active3]

TASK [Assert that there is only one CA server] *****************************************************************************************************
fatal: [active3]: FAILED! => {
    "assertion": "groups['step-ca'] | length == 1",
    "changed": false,
    "evaluated_to": false,
    "msg": "Assertion failed"
}

Any ideas - also @bnlawrence

after yet another try it connected only to activeh now, went further than the other runs, but still died at the end:
TASK [Assert that there is only one Prometheus server] *********************************************************************************************
ok: [activeh] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Ensure /etc/prometheus directory exists] *****************************************************************************************************
changed: [activeh]

TASK [Ensure CA certificate is copied] *************************************************************************************************************
changed: [activeh]

TASK [Ensure prometheus.yml is templated] **********************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible.errors.AnsibleUndefinedVariable: 'dict object' has no attribute 'default_ipv4'. 'dict object' has no attribute 'default_ipv4'
fatal: [activeh]: FAILED! => {"changed": false, "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute 'default_ipv4'. 'dict object' has no attribute 'default_ipv4'"}

PLAY RECAP *****************************************************************************************************************************************
active2                    : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
active3                    : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
activeh                    : ok=33   changed=8    unreachable=0    failed=1    skipped=3    rescued=0    ignored=0

My inventory file:
[haproxy]
activeh
active2
active3

[jaeger]
activeh

[minio]
activeh

[prometheus]
activeh

[reductionist]
activeh
active2
active3

[step:children]
reductionist

[step-ca]
activeh

[docker:children]
haproxy
jaeger
minio
prometheus
reductionist
step-ca

and my /etc/hosts file:
<IP_ADDRESS>   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.171.169.zzz activeh
192.168.3.yyy active2
192.168.3.xxx active3

(masked off the last three digits for security purposes)

Nevermind me! I managed to get the connections done working :partying_face: - bit of jittery pokery with the ssh configuration file and that was that: I have activeh, active2 and active3 all seen by ansibles now, and the thing deploys, though I got this now:
fatal: [active2]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'stdout'. 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/home/vpredoi/reductionist-rs/deployment/site.yml': line 191, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n    - name: Bootstrap CA\n      ^ here\n"}
fatal: [active3]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'stdout'. 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/home/vpredoi/reductionist-rs/deployment/site.yml': line 191, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n    - name: Bootstrap CA\n      ^ here\n"}

What le grand crappe? :grin:

some update: I managed to step over that error by making a couple changes to site.yml:
diff --git a/deployment/site.yml b/deployment/site.yml
index e7d9b37..24dd776 100644
--- a/deployment/site.yml
+++ b/deployment/site.yml
@@ -185,14 +185,19 @@
       register: ca_fingerprint
       changed_when: false
       delegate_to: "{{ groups['step-ca'][0] }}"
-      run_once: true
+        #run_once: true
       when: not step_stat.stat.exists

+    - name:
+      debug:
+        var: ca_fingerprint
+
     - name: Bootstrap CA
       ansible.builtin.command: >
         step ca bootstrap
         --ca-url https://{{ hostvars[groups['step-ca'][0]].ansible_facts.default_ipv4.address }}:9999
         --fingerprint {{ ca_fingerprint.stdout }} --install
+      run_once: true
       changed_when: true
       when: not step_stat.stat.exists

@@ -343,6 +348,7 @@
         {{ reductionist_host }}
         {{ reductionist_remote_certs_path }}/cert.pem
         {{ reductionist_remote_certs_path }}/key.pem
+      run_once: true
       changed_when: true
       when: not reductionist_cert_stat.stat.exists

and, of course, by creating a couple root CA certificates on active2 and active3, but now reductionist is not pinging correctly:
TASK [Wait for reductionist server to be accessible via HAProxy] ***********************************************************************************
FAILED - RETRYING: [active2]: Wait for reductionist server to be accessible via HAProxy (3 retries left).
FAILED - RETRYING: [active3]: Wait for reductionist server to be accessible via HAProxy (3 retries left).
ok: [activeh]
FAILED - RETRYING: [active3]: Wait for reductionist server to be accessible via HAProxy (2 retries left).
FAILED - RETRYING: [active2]: Wait for reductionist server to be accessible via HAProxy (2 retries left).
FAILED - RETRYING: [active3]: Wait for reductionist server to be accessible via HAProxy (1 retries left).
FAILED - RETRYING: [active2]: Wait for reductionist server to be accessible via HAProxy (1 retries left).
fatal: [active2]: FAILED! => {"attempts": 3, "changed": false, "elapsed": 0, "msg": "Status code was -1 and not [200]: Request failed: <urlopen error TLS/SSL connection has been closed (EOF) (_ssl.c:1129)>", "redirected": false, "status": -1, "url": "https://<IP_ADDRESS>:8080/.well-known/reductionist-schema"}
fatal: [active3]: FAILED! => {"attempts": 3, "changed": false, "elapsed": 0, "msg": "Status code was -1 and not [200]: Request failed: <urlopen error TLS/SSL connection has been closed (EOF) (_ssl.c:1129)>", "redirected": false, "status": -1, "url": "https://<IP_ADDRESS>:8080/.well-known/reductionist-schema"}

I did open the 8080 ports so am not sure what to do next - will need some help from @bnlawrence methinks :beer:

I would say in general that the Ansible playbook is designed to be used with all hosts in the inventory. There's a certain amount of referring to facts of other hosts that are assumed to exist.
I think you should trim down the haproxy group to just activeh. HAProxy is not deployed with any support for failover (e.g. using keepalived), and the certificate is generated for the IP of only one of the HAProxy servers. I'll add a check for the group size similar to step-ca.
Regarding the step CLI bootstrap issues, I think it's a side effect of having configured the hosts separately. I'll update the condition to always grab the CA fingerprint so that it's available if any hosts require it. I'm a bit concerned about your addition of run_once to the bootstrap CA task - it should run against all hosts.

Some improvements: https://github.com/stackhpc/reductionist-rs/pull/90

Hi @markgoddard - legend, I'll grab that and run with it, cheers very much :beer:

I'm a bit concerned about your addition of run_once to the bootstrap CA task - it should run against all hosts.

That may explain the final error I reported, which smelled like an SSL issue, but lemme stash the changes I made and run with #90


ASGI: Add affordances and recipe for long-polling

The 3.0 ASGI implementation already affords long-polling, but it would be useful to provide a recipe to demonstrate the pattern in the context of a Falcon app.
Also, for the sake of efficiency, we may want to provide a req.wait_disconnect() or similar method that can be executed in a background task within a responder. The responder could then use this in order to detect if it should abandon a long-poll early in the case that the client bails out. This is similar to what asgi.App does when streaming SSE's (using watch_disconnect().

Having some way to signal http.disconnect to the responder might also be useful for handling HTTP streaming requests (where the incoming Content-Length is unknown in advance). And more generally, it might potentially make interaction with the request stream more elegant.
In addition, could we consider adding some request status attributes, as we have for WebSocket (such as asgi.WebSocket.closed etc).

Use cases we will need to address and document:

[ ] Reading from a request body when content-length header is present but the client disconnects early
[ ] Reading from a request body when content-length header is not present and the client disconnects (a) in a way that results in malformed data or (b) before a response can be sent.
[ ] Holding a client request while waiting for an event (long-polling) and the client disconnects before said event occurs

Potential affordances:

[ ] Request status attributes (e.g., closed, open) - but is this already covered by req.stream?
[ ] req.wait_disconnect() to be executed in a background task within the responder.
[ ] Raising a "client disconnected" error while reading from a stream

Note: 3.0 behavior is to truncate the request body in the case of a premature disconnect. Mitigation is for apps to always validate their inputs!

The current behaviour is somewhat awkward in the SSE scenario from the client perspective.
Even though asgi.App itself perfectly detects a disconnect event (via the mentioned watch_disconnect()), the event is not conveyed to the user's implementation.
Right now, it's possible to wrap the emitter's loop in a try.. block hoping to catch GeneratorExit when the async generator is torn down, and execute the cleanup code in the finally clause.
But not sure if it is a particularly elegant/recommended way to do it, and if it is, we should document recipes for that.

OK, so looks like we need to improve the SSE disconnect story, and overall we will need some recipes for SSE, WebSocket, and good-old-fashioned long polling.

When using as async generator in conjunction with resp.stream, we happen to check
            if hasattr(stream, 'close'):
                await stream.close()

Maybe the something similar could be supporting for the SSE generator too?


Creating Open AI Service container for API using GPT-3.5 Turbo Integration

As a developer working on the Code Whisperer AI application, I want to integrate GPT-3.5 Turbo into our the application, so that we can leverage its advanced natural language understanding capabilities to provide real-time coding assistance, recommendations, and optimizations to users.

I created the service container for the Open AI API making a post/get request method all in one.


Respect XDG basedir spec

Your proposal
I propose that debug respect the XDG Base Directory Specification such that files are written to locations controlled by user environment variables.
There are at least three relevant files:

rdbg_history
rdbgrc
unix_domain_socket_dir

Presently, the history file may be forced into a different location with the RUBY_DEBUG_HISTORY_FILE env var. While helpful, this does not adhere to the desire for single-configuration (setting XDG_* vars once and not needing to configure each of hundreds of different utilities). It should default to a file or directory in XDG_STATE_HOME (which itself defaults to $HOME/.local/state)
The unix_domain_socket_dir is already very nearly adhering to Xdg, as its first implicit location (after checking explicit config setting) uses XDG_RUNTIME_DIR.
The rdbgrc (and .rb variant) have no env var equivalents and so cannot even be set explicitly. However, they should be using files within XDG_CONFIG_HOME (which defaults to $HOME/.config). Perhaps $XDG_CONFIG_HOME/rdbg/config or $XDG_CONFIG_HOME/rdbg/config.rb ?
Additional context
The Xdg basedir spec is here: https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
Rationale

keeps user's homedirectory clean
allows versioning of configuration files cleanly (ie, dotfiles repo to version-control .config but not cache or data files that also typically end up in homedir)
allows single-configuration changes to move configuration to different directory tree (for testing, for simulated environments, and many other unknowable use-cases)

We need to make migration plan.
~/.rdbgrc (and so on) are already there so we need to respect current files.
On the https://github.com/vim/vim/issues/4275, it was closed because of duplication of https://github.com/vim/vim/issues/2034 which has huge comments and not accepted (I can't read all of comments) but #4275 proposes:
~/.vimrc
~/.vim/vimrc
$XDG_CONFIG_DIR/vim/vimrc (if the env var is unset, its default is ~/.config)

For rdbgrc, making searching order as
~/.rdbgrc
$XDG_RUNTIME_DIR/rdbg/config if `XDG_RUNTIME_DIR` is available

is one idea.
For ~/.rdbg_histroy there are two options:

search ~/.rdbg_history and $XDG_STATE_HOME/rdbg/history if $XDG_STATE_HOME is available
use $XDG_STATE_HOME/rdbg/history and ignore existing ~/.rdbg_history if $XDG_STATE_HOME is available
2.1. with warning

Could you survey another software migration plan?
BTW my Ubuntu 24.04 machines only have the following XDG envvals:
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
XDG_RUNTIME_DIR=/run/user/1000

Who set config dir and state dir?

BTW (2) I found NVIM_APPNAME to change the app name (and we can switch configurations at once). Should we support it with RUBY_DEBUG_APPNAME or something similar?

Some thoughts on this issue based on @ko1's comments above:

For rdbgrc, making searching order as
~/.rdbgrc
$XDG_RUNTIME_DIR/rdbg/config if `XDG_RUNTIME_DIR` is available

is one idea.

I'd expect to create a config file in $XDG_CONFIG_HOME (e.g. ~/.config) at a path like $XDG_CONFIG_HOME/rdbg/config. I believe that more closely follows patterns details on the XDG page on the Arch wiki.

For ~/.rdbg_histroy there are two options:

search ~/.rdbg_history and $XDG_STATE_HOME/rdbg/history if $XDG_STATE_HOME is available
use $XDG_STATE_HOME/rdbg/history and ignore existing ~/.rdbg_history if $XDG_STATE_HOME is available
2.1. with warning

I think $XDG_DATA_HOME (e.g. ~/.local/share) is the more appropriate place for history files. Similar to my note above, the XDG page on the Arch wiki cites a number of examples using $XDG_DATA_HOME for history files.

Thank you.
Next we need to define the search order.

I took a quick glance at the entries in sections 2.2 and 2.3 on the Arch wiki and, where noted, it looks like most apps prefer the "legacy" file (e.g. ~/.rdbg_history) if present. Otherwise, search for the $XDG_*-prefixed file path.
So, a restated version of @ko1's proposal above:

prefer ~/.rdbgrc if present,
else, $XDG_CONFIG_HOME/rdbg/config if $XDG_CONFIG_HOME is set and $XDG_CONFIG_HOME/rdbg/config is present,
else, no customized user configuration

For the history file:

prefer ~/.rdbg_history if present,
else, $XDG_DATA_HOME/rdbg/history if $XDG_DATA_HOME is set¹

¹ There'd need to be a check for this file path. If it exists, great! If not, create the path $XDG_DATA_HOME/rdbg and touch $XDG_DATA_HOME/rdbg/history.

Thank you. I'll make it (or PR is welecome)

@ko1 I just opened a draft PR for this: #1055.
I think I got the desired logic correct, though I'm open to feedback and could use some help with adding any tests that the core team may wish to see.


Wishlist: Make more IPFS options available

@TheDiscordian this sounds really great!
I really like to test this out, but I would need a couple of options, so I thought I could maybe post a wishlist 🥺
Ability to

set blake2b-256 as hash algorithm
select rabin as chunker (with min/max/avg)
use only CIDv1 (not sure if that's not already the case)
use raw-leaves

Additionally, I think I would need a way to trigger the whole sync process manually, as I would fetch a lot of data via rsync in irregular intervals. After the rsync process has completed its work I would like to notify ipfs-sync to start pushing the changes and terminate afterwards (or just pause the sync and wait for the next signal).
So maybe the easiest way would be to have it listen to SIGHUP or something like that. So I can send a signal vial kill to start syncing.

Sorry for my super late response, not sure how I missed this issue. I like all of this, more options the better!


fix aarch64 compile warning

thanks, this will be rebased into #1231 to merge at once.

Applied in #1231


[Application] Add LSTM examples using pytorch and tensorflow

Add LSTM examples using pytorch and tensorflow.

LSTM example using NNTrainer needs to be added.
We conduct memory benchmark test based on these codes.

Signed-off-by: Seungbaek Hong<EMAIL_ADDRESS>

:memo: TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #2101. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/.

:octocat: cibot: @baek2sm, The last line of a text file must have a newline character. Please append a new line at the end of the line in Applications/LSTM/README.md.

INFO: You can read if there are misspelled characters at our misspelling check report. Please read http://ci.nnstreamer.ai/nntrainer/ci/repo-workers/pr-checker/2101-202301301529000.24917006492615-ad5dd1cb6f21b7f6e18451388a04c7eaeecce1b0/report/misspelling_result.txt.

INFO: You can read if there are misspelled characters at our misspelling check report. Please read http://ci.nnstreamer.ai/nntrainer/ci/repo-workers/pr-checker/2101-202301301531360.32858109474182-d422c4b0941949be946fbb079f20512db7d2fc0b/report/misspelling_result.txt.


HTTP Error 401: Authorization Required -- Update authorization & bot-detection-evasion logic

Hello,
I am getting HTTP Error 401: Authorization Required every time I try to use the flurry data application. I have verified numerous times that my credentials are correct. Below is a snippet from the splunkd.log file. Has anyone else run into this, and is there a solution?
Thanks!
04-13-2015 10:57:39.082 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" Traceback (most recent call last):
04-13-2015 10:57:39.082 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/extract.py", line 236, in
04-13-2015 10:57:39.082 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" conn.login()
04-13-2015 10:57:39.082 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/extract.py", line 65, in login
04-13-2015 10:57:39.082 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" self.browser.open('https://dev.flurry.com/secure/login.do')
04-13-2015 10:57:39.083 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/mechanize/_mechanize.py", line 203, in open
04-13-2015 10:57:39.083 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" return self._mech_open(url, data, timeout=timeout)
04-13-2015 10:57:39.083 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/mechanize/_mechanize.py", line 255, in _mech_open
04-13-2015 10:57:39.083 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" raise response
04-13-2015 10:57:39.083 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" mechanize._response.httperror_seek_wrapper: HTTP Error 401: Authorization Required
04-13-2015 10:57:49.191 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" Traceback (most recent call last):
04-13-2015 10:57:49.191 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/extract.py", line 236, in
04-13-2015 10:57:49.191 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" conn.login()
04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/extract.py", line 65, in login
04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" self.browser.open('https://dev.flurry.com/secure/login.do')
04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/mechanize/_mechanize.py", line 203, in open
04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" return self._mech_open(url, data, timeout=timeout)
04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/mechanize/_mechanize.py", line 255, in _mech_open
04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" raise response
04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" mechanize._response.httperror_seek_wrapper: HTTP Error 401: Authorization Required

I am seeing a different error myself in splunkd.log:
04-15-2015 11:21:39.296 -0700 INFO  ExecProcessor - New scheduled exec process: python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py
04-15-2015 11:21:39.296 -0700 INFO  ExecProcessor - 	interval: 600000 ms
04-15-2015 11:21:39.296 -0700 INFO  ExecProcessor - New scheduled exec process: /Applications/splunk_621_flurry/bin/splunkd instrument-resource-usage
04-15-2015 11:21:39.296 -0700 INFO  ExecProcessor - 	interval: 0 ms
04-15-2015 11:25:55.092 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py" Traceback (most recent call last):
04-15-2015 11:25:55.093 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"   File "/Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py", line 241, in <module>
04-15-2015 11:25:55.093 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"     flurry_csv_stream = conn.download_log(year, month, day, offset)
04-15-2015 11:25:55.093 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"   File "/Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py", line 102, in download_log
04-15-2015 11:25:55.093 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"     resp = self.browser.open(url)
04-15-2015 11:25:55.093 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"   File "/Applications/splunk_621_flurry/etc/apps/flurry/bin/mechanize/_mechanize.py", line 203, in open
04-15-2015 11:25:55.094 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"     return self._mech_open(url, data, timeout=timeout)
04-15-2015 11:25:55.094 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"   File "/Applications/splunk_621_flurry/etc/apps/flurry/bin/mechanize/_mechanize.py", line 255, in _mech_open
04-15-2015 11:25:55.094 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"     raise response
04-15-2015 11:25:55.094 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py" mechanize._response.httperror_seek_wrapper: HTTP Error 404: Not Found

Regardless, both my error trace and your error trace indicate that either Flurry has changed their bot detection system or rate limiting system. The actual (private) API being accessed appears to still work, namely: https://dev.flurry.com/eventsLogCsv.do
I no longer have enough cycles to continue updating this app to adjust to Flurry's API changes. So I will mark this item as Community Support.

Hi,
How about setting "Next Session ID [Advanced]" as well as your credentials?
I got the same problem for many times without entering it, but the app seems to start again after I entered and saved bigger Session ID. I found the biggest Session ID on the time as below;
index="flurry" | stats max(Session)
And I entered a number next to the search result.
Thanks.


Installation

Dear clarkware please give more explanation about installation.
I don't understand what should I do with that.

Hi,
i'm not affiliated with 'clarkware' in any way. I just use the jDepend library.
To install the SonarQube plugin you can install it from the Update Centre within the SonarQube administration pages.
If you want to install it manually you can copy the .jar file from the release to the extensions/plugins/ directory within your SonarQube installation.

you could use as eclipse plugin:
http://andrei.gmxhome.de/jdepend4eclipse/index.html

I've had a trouble running it first time too. Basically what you can do is:

Unzip archive from Releases anywhere you want;
Open Terminal in jdepend-2.10\lib directory;
Run JDepend Swing GUI for your Maven project:
java -cp jdepend-2.10.jar jdepend.swingui.JDepend E:\JAVA\MyMavenProject\target\classes\.

If you have decided to create system environment variable JDEPEND_HOME as it is suggested in doc, then you can run JDepend from any directory, for example out of root directory of your maven project:

Powershell: java -cp $env:JDEPEND_HOME\lib\jdepend-2.10.jar jdepend.swingui.JDepend .\target\classes\
bash: java -cp $JDEPEND_HOME/lib/jdepend-2.10.jar jdepend.swingui.JDepend ./target/classes/


Must type the same letter twice when one letter is selected

VSCode Version: Code - Insiders 1.12.0-insider (0834f15e86ef8359da9fbcf95d49083001af65c3, 2017-05-03T15:02:36.792Z)
OS Version: Linux x64 4.4.0-75-generic
Extensions: none

Steps to Reproduce:

select a letter
type the selected letter (nothing happens)
type the selected letter again (the letter will finally be typed)

Real example - rename i to idx

select all three i variables
type idx (only dx will appear)
type iidx (first i won't appear)

for (var i = 0; i < length; i++) {
}

This works for me on Ubuntu.

@acls some questions:

What's in your settings.json?
What's your OS locale set to?
What Linux distro are you on?
What keyboard layout are you using?

What's in your settings.json?

{
    "window.zoomLevel": 1
}

What's your OS locale set to?

en_US

What Linux distro are you on?

Ubuntu 16.04

What keyboard layout are you using?

English (US)

I just booted up an old windows box and installed vscode. Issue still happens there.

It's happening on another Windows machine as well? Are you sharing the physical keyboard by chance?
Moving to editor owners.

Yeah, the windows box was just an old laptop, so different keyboards.

Duplicate of #16520 .
The fix should come out starting with today's or yesterday's insiders (i.e. anything that has 242cb936f0b703a92547c8884dc0c5609eab01e9 ). This might explain the difficulty in reproducing.

Talk about bad timing. The fix was made less than a day before I posted, so the latest insider's build didn't have it.
And sorry about the duplicate posting. I hadn't run into the issue until after I got the latest build an hour earlier. I had searched for doops, but not 6 months back.


Extending zod schema with branded type produces typescript error

I have a brand in a base schema, which I extend to add additional data, like so:
import { z } from "zod";

const agentBaseSchema = z.object({
  id: z.string().uuid().brand("Uuid"),
})

enum AgentType {
  CODE_FREE = "CODE_FREE",
  MACHINE_CODE = "MACHINE_CODE",
}

type AgentBase = z.infer<typeof agentBaseSchema> & {
  type: AgentType
}

const codeFreeAgentSchema = agentBaseSchema.extend({
  type: z.literal(AgentType.CODE_FREE),
  // Other specific things
})

const machineCodeAgentSchema = agentBaseSchema.extend({
  type: z.literal(AgentType.MACHINE_CODE),
  // Other specific things
})

// ERROR! Type 'string' is not assignable to type string & 'BRAND<"Uuid">'
const agentSchema = z.union([codeFreeAgentSchema, machineCodeAgentSchema]) satisfies z.ZodSchema<AgentBase>

type Agent = z.infer<typeof agentSchema>

TS playground
The agents are consumed as a discriminated union on the type, as you can see. I'm doing the satisfies bit to ensure that new schemas are created properly by other people in the team.
Is there a way to do this? Should I not be doing this?
It seems to be the same idea as https://github.com/colinhacks/zod/issues/2076, but I don't really understand how to apply the solution to my use case since I don't have input/output schemas.

type AgentBaseInput = z.input<typeof agentBaseSchema> & {
  type: AgentType
}

type AgentBaseOutput = z.output<typeof agentBaseSchema> & {
  type: AgentType
}

// ...

const agentSchema = z.union([codeFreeAgentSchema, machineCodeAgentSchema]) satisfies z.ZodSchema<AgentBaseOutput, any, AgentBaseInput>


Leaking connections if data was never read

The line below is never invoked if there were no reads on the file object. This causes connection pool saturation and eventually timeouts.
https://github.com/skshetry/webdav4/blob/a5ac69eb59405444958b13c148b24a1c74bb8186/src/webdav4/stream.py#L89
This gist demonstrates the issue:
https://gist.github.com/pauleikis/a9247f1e8fdbfb70c756cb89a3fd49e4

Thanks @pauleikis. What we probably need to do is, add response.close() on finally as well:
https://github.com/skshetry/webdav4/blob/a5ac69eb59405444958b13c148b24a1c74bb8186/src/webdav4/stream.py#L95-L96

Yes. I believe that will work.
That's an easy fix, I will make PR as soon as I have time to write a proper test.

Fixed by #106.


F2F-626 - Copy changes across the F2F journey

Per the findings of the F2F build review, copy has been updated to reflect the latest designs, alterations have been made to spacing and font size/weight etc.

F2F-626

Browser tests add/update


Feature Request :Add option to add months in Current date

Hi,
In Robot currently, we don't have the option to add months to the current data in the DateTime library.
We can now add Hours, Days, and, Years using Add Time to data.
${one_month_later}    Add Time To Date    ${current_date}    1 months
it we try to add the above it will not recognize as a keyword.

Month is problematic because it can e 28-31 days. The code for handling formats like "1 day 2 hours" is generic and cannot really handle months. In this particular context incrementing and decrementing months could probably be handled separately, though. I personally don't consider this too high priority, but if someone is interested to implement this I'm happy to review a pull request and discuss about some of the corner cases dealing with months would contain.


antigen not populating fpath correctly

For some reason, antigen isn't setting correct paths for installed bundles in fpath, resulting in zaw failing to start:
(anon):205: filter-select: function definition file not found
(anon):bindkey:206: no such keymap `filterselect'

This is not surprising, as fpath entry for zaw looks like this:
/home/yacoob/.antigen/repos/https-COLON--SLASH--SLASH-github.com-SLASH-zsh-users-SLASH-zaw.git//

(note the double slash at the end)
On a different Linux host, with same version of antigen and zaw, fpath gets set correctly:
/home/yacoob/.antigen/repos/https-COLON--SLASH--SLASH-github.com-SLASH-zsh-users-SLASH-zaw.git/functions

Is there any debug mode I can enable for antigen so I can see why is it making a bad decision when it comes to setting fpath?
Details: antigen 1.2.1, zsh 5.1.1, Linux (3.13.0)

This happens due to how zaw determines the current path in order to add functions to fpath, which works without antigen's cache involved, otherwise it fails.

Why is antigen able to set the path correctly on the other host then?

Not sure about that. May be running with caching disabled (_ANTIGEN_CACHE_ENABLED=false).
You should check ~/.antigen/.cache/.zcache-payload. It does contains the final fpath generated by antigen. If they differ then there's another problem.

I've checked that: on all of my boxes where I don't see the boxes, I had old version of antigen installed (+1year old), and no caching was enabled. After updating antigen and restarting zsh, I've started seeing same problems (wrong fpath) with the other bundle I use (zsh-users/zsh-syntax-highlighting). Disabling cache helped.
Please tell me if you need any other information.

I haven't seen errors with zsh-syntax-highlighting. What's the issue?

zsh-syntax-highlighting: highlighters directory '/Users/yacoob/.antigen/.cache/highlighters' not found.
zsh-syntax-highlighting: failed loading highlighters, exiting.

I'll try to reproduce it on zsh 5.1.1 but I can't see that error with latest zsh-users/zsh-syntax-highlighting, antigen v1.2.1 or above.

 Update single plugin

I'm having the same problem as @yacoob.  I try to install zsh-syntax-highlighters using antigen init and I get the same error.  However, I can successfully install the highlighters using antigen bundle zsh-users/zsh-syntax-highlighting from the command line.  I think it must have something to do with the cache, but I don't know.
I'm using zsh 5.2 and antigen v1.3.2

I'm gonna take a look.

I does seem to work as expected using both load methods: init and bundle.  What's the error you see?

I am getting the same error as @yacoob
zsh-syntax-highlighting: highlighters directory '/Users/bheussler/.antigen/.cache/highlighters' not found.
zsh-syntax-highlighting: failed loading highlighters, exiting.

This is what my antigen.zsh looks like
source /usr/local/share/antigen/antigen.zsh
antigen init /Users/bheussler/.dotfiles/zsh/antigenrc

which loads my antigenrc
# Load the oh-my-zsh's library.
antigen use oh-my-zsh

antigen bundle brew
antigen bundle bundler
antigen bundle docker
antigen bundle extract
antigen bundle gem
antigen bundle git
antigen bundle mvn
#antigen bundle npm
antigen bundle osx
antigen bundle vagrant
antigen bundle zsh-users/zsh-syntax-highlighting

# Load the theme.
antigen theme arrow

# Tell antigen that you're done.
antigen apply

Tried your set up and works as expected. Try: antigen reset, to reset cache.
Also what's the output of antigen version, just to check.

I think I found the issue.  I'm currently running Antigen v1.3.2 installed from Homebrew.  I did major upgrade from an earlier version (version 1?).  The fix was to wipe out the .antigen/ directory in my home and have antigen reinstall plugins.  Works great so far!

Good to hear. I'll investigate how to avoid issues between major upgrades.
I believe we currently reset cache on selfupdate.
Best regards


how to start developing?

Environment: unubtu 20.04, openwrt latest on proxmox
Description:
Hi
I installed openwrt on my proxmox and have access to openwrt by ip.
Now I want to test your package in my virtual openwrt also I developing with vscode on unubtu 20.04
I have a important question about running your repository where? in my local machine (Ubuntu 20.04) or my openwrt (proxmox)
please explain to me that how can I edit my codes in Ubuntu and show results in proxmox openwrt?
(sorry for my bad english)

https://zhaojh329.github.io/oui/guide/getting-started.html

With openwrt docs for developer I solved my issue.
thanks to you


code edited

t values are rounded to 2 sig fig

@yasmensarhan27  I think it is rounded up to two decimal place.


Remove the distinction between access and archive locations

Closes wellcometrust/platform#3502.

I'm not sure if this breaks what DDS/DLCS expect and hence whether we should merge it now though

It does; we should leave it until Tom and/or Adam are next in the office.

We can merge this, Tom said it's a relatively small change and he will need to fix other things soon.


Add multihead attention

This PR aims at supporting Multi-head attention including self attention in it in place of https://github.com/chainer/chainer/pull/7066 where I made a mistake while resolving conflicts.
I'm really sorry for this @himkt @hvy.
For the commit history, I made a copy of the branch https://github.com/crcrpar/chainer/pull/6.
Ref:

module: https://github.com/pytorch/pytorch/blob/76f847546b46890b9c028c7a98f36db8674bc0f5/torch/nn/modules/activation.py#L649-L783

function: https://github.com/pytorch/pytorch/blob/76f847546b46890b9c028c7a98f36db8674bc0f5/torch/nn/functional.py#L3126-L3378

Thanks for revising the PR!
https://github.com/chainer/chainer/pull/7066#issuecomment-518667004
The reason is that this PR is related to the WIP PR https://github.com/cupy/cupy/pull/2215.
To supporting cuDNN MHA APIs, it could break compatibility with the current interface.
There are a few use-cases of cuDNN MHA and we need time to

Sorry, I accidentally closed this PR while writing a comment. :bow:
Reopened.

Thanks for revising the PR!
https://github.com/chainer/chainer/pull/7066#issuecomment-518667004
The reason is that this PR is related to the WIP PR https://github.com/cupy/cupy/pull/2215.
To supporting cuDNN MHA APIs, it could break compatibility with the current interface.
(this is why we think MHA should be experimental)
We need more time to resolve the issue https://github.com/cupy/cupy/issues/2202.
However, MHA plays an important role in current NLP community.
(this is why we think we should merge this PR)

OK, I got it.
But just I want say that I changed the design of
‘chainer.functions.multihead_attention’ while working on this PR to be
compatible with the cudnn imple as possible and this is why there are two
parameters for softmax scaler exist.
On Wed, Aug 7, 2019 at 23:44 Makoto Hiramatsu<EMAIL_ADDRESS>wrote:

Thanks for revising the PR!
#7066 (comment)
https://github.com/chainer/chainer/pull/7066#issuecomment-518667004
The reason is that this PR is related to the WIP PR cupy/cupy#2215
https://github.com/cupy/cupy/pull/2215.
To supporting cuDNN MHA APIs, it could break compatibility with the
current interface.
(this is why we think MHA should be experimental)
We need more time to resolve the issue cupy/cupy#2202
https://github.com/cupy/cupy/issues/2202.
However, MHA plays an important role in current NLP community.
(this is why we think we should merge this PR)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/chainer/chainer/pull/7879?email_source=notifications&email_token=AD3Q7U32TQH4W6MZEOYMG5DQDLNWXA5CNFSM4IKAGMF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3YUQKA#issuecomment-519129128,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AD3Q7UYVQPLTQJR4L4U5JYLQDLNWXANCNFSM4IKAGMFQ
.

(I'm so sorry I failed submitting response comments. :bow:)

I changed the design of chainer.functions.multihead_attention while working on this PR to be compatible with the cudnn imple as possible
Sounds amazing! Thanks for letting me know. I remember this.

Only mac OS python2 test failed, meaning this PR itself is ready for review I think.

@hvy Please review it. :bow:

I left a couple of minors comments on details. I'm not confident enough to comment on the public interface so I asked @soskek for some feedback (from a researcher/user perspective).

I'll resume this.

Future follow-up PRs should access

implementing static_k and static_v
improving unit tests

As Chainer is now under the maintenance phase, unfortunately we decided not to merge this feature. Thanks again for your contribution.


Blogger API V3 Search Description

I can't find how to add the Search Description using the Blogger API V3. Is it possible to add search description using blogger API ?

Hi @jimmyxp01,
According to this community discussion, it's not possible to add a search description using the Blogger API.
You can create a feature request for the Blogger API by opening an issue here. You can read more about the issue tracker in general here.
In addition, you can send feedback to the Blogger team by clicking on the question mark icon, then clicking 'Send Feedback' button on the blogger site here.
I'm going to close this issue but please re-open it if you have additional questions.

Hi @jimmyxp01,
According to this community discussion, it's not possible to add a search description using the Blogger API.
You can create a feature request for the Blogger API by opening an issue here. You can read more about the issue tracker in general here.
In addition, you can send feedback to the Blogger team by clicking on the question mark icon, then clicking 'Send Feedback' button on the blogger site here.
I'm going to close this issue but please re-open it if you have additional questions.


Statefullness of the bot

Going thru the code, I am wondering if there is no work around for the bot being stateful
If that is the case, then are there any suggestions on how we can configure this statefulness when we have multiple VMs in an AWS ecosystem (Apart from Kubernetes).  Currently i have a Network load balancer and application load balancer on top of a single EC2 machine that does the job. But if i were to move to multiple machines,  statefulness will be a point to consider.

Calls are no request response type protocol.
App-hosted media bots are actively connected to the meeting and are receiving / sending media in real-time.
Do you expect to be able to receive bits and pieces of that communication on different machines?
Do you expect to be able to send bits and pieces of that communication from multiple machines?
You will need to expose each machine with it's own media port on your public facing IP, so that when your bot handles a call, each instance of your bot can be addressed uniquely from the Teams platform. You can still load balance the initial incoming call notification. That is HTTP. You can route that to your machine pool via round robin. The machine that is selected answers the call with a) a media platform object that has a public address port combination and b) an http endpoint for receiving subsequent notifications for that call.
a) is usually done with different ports on a public IP
b) is usually done with a reverse proxy that routes traffic based on a path segment

Also, EC2 is not officially supported. Running bots outside of Azure VMs can lead to issues. But that's beside the main question I guess.

Ahhh I was also thinking along the same line thanks. But curious abt the last comment tho, is it specified anywhere where running outside azure VM cud lead to problems or is this for windows servers in general.

An application-hosted media bot requires the Microsoft.Graph.Communications.Calls.Media .NET library to access the audio and video media streams, and the bot must be deployed on a Windows Server machine or Windows Server guest Operating System (OS) in Azure.

Source
While developing and debugging locally (outside Azure), I haven't seen issues with running the bot. However, I do believe you will not be receiving support, when you are running outside of Azure.
Other than that, I'm not sure what might happen, if there are ToS regarding that, or technical steps in place to enforce that kind of requirement, like IP banning. Maybe somebody from MSFT could also shine a light on that.

Hmmm I see , thanks for the info

@vigbav36 @InDieTasten
The bot media SDK does not have a hard, technical dependency on Azure; however, we do not officially support running the media SDK outside of Azure, and it is not a scenario we test.
Furthermore, the Teams meeting the media bot is connected to is most definitely hosted within Azure and real-time media is transmitted over a lossy RTP/UDP stream. So, we recommend the bot’s media endpoint be in Azure, and ideally co-located within the same Azure data center region as the meeting itself, to minimize round trip time and packet loss. Depending on the join call flow, the platform can inform the bot what Azure region the meeting is hosted in ("mediaHostedRegion"): bot apps can use that to co-locate their bot endpoint. Once the bot receives the real-time meeting audio/video media, it can do whatever it wants with it, including transmitting it outside of Azure using less real-time lossless transports.
Thanks.


Error handling

If there is an error in gopay (https://doc.gopay.com/en/#errors) the response does not contain the state parameter (https://doc.gopay.com/en/#return-errors), so the two methods below (getResultCode and getResultMessage) are throwing a notice, which is only logged in production, but the execution continues. And because the response status is not checked, the payment is marked as paid. I've added a simple error check, and return FAILED status, if an error occurs. I've also added the checkChargeStatus call before the charge return, which throws an exception if the payment failed.

Do you think we could store those errors without throwing them away and using only FAILED constant? I can't get the error on the sandbox environment, but I'm thinking this:
    public function getResultCode()
    {
        $err = $this->getError();
        if ($err) {
            return $err['error_code'];
        }
        return $this->response->getData()['state'];
    }

    public function getResultMessage()
    {
        $err = $this->getError();
        if ($err) {
            return sprintf("%s: %s", $err['error_name'], $err['message']);
        }
        return $this->response->getData()['state'];
    }

    protected function getError(): ?array
    {
        if (empty($this->response->getData()['errors'])) {
            return null;
        }
        return reset($this->response->getData()['errors']);
    }

The whole response is stored in the payment_logs table.. The result message (getResultMessage) gets saved to the approval field in recurrent_payments table, what was a bit odd, so I returned the failed state there too. You can try changing this line:
$response = new RecurrenceResponse($this, $purchaseResponseData);

to this:
$response = new RecurrenceResponse($this, [
            'date_issued' => (new DateTime())->format('c'),
            'errors' => [
                'scope' => 'G',
                'error_code' => 343,
                'error_name' => 'PAYMENT_RECURRENCE_TOO_OFTEN',
                'message' => 'Number of attempts to recure payment was exceeded',
            ],
        ]);

in the Omnipay\GoPay\Message\RecurrenceRequest class to test it with charge..

I managed to fail the payment after all. Yeah, the approval is not the best name, since we always store the error message there as well. Anyway, after the changes, it will look like this which should be OK.

I'll amend the changes to your commit and make a tagged release. It should be ready in 20 minutes, I'll ping you so you can test it right away.

Merged via https://github.com/remp2020/crm-gopay-module/commit/d0c2d9df505529a84baea36faf18af16053a4293. I'm not closing the issue yet, I'll wait for the confirmation it's working as expected after my additional changes.


check if all ports closed except ...

Hi @ all,
is there a way to check if all ports on a machine are closed except a few desired ones?
This would be a nice feature regarding a permanent security check of servers.
Thank you!

I was going to file an issue like this; better title would be: "Implement netfilter verification"

You're right, done!

Can you expand a bit more on how this syntax would look like? I haven't messed with Linux firewalls for a while, so not sure what's out there: iptables, firewalls, ufw.
Kind of undecided on this feature if there's a lot of variations. Curious on how Goss native would look vs using the command.


Subset worker refactor

Tested with my processing workflow (framerate resampling, resizing, cropping, cut finding, clipping).
Results are identical to before.
Merge after #275.

I merged #275 ; can you rebase here ?

Ok done

looks good on first glance

Can we merge the worker requests?

What do you mean by "requests" ?

Lgtm, this is not doing much, just moving the code around, a bit better


[release-1.6] Add version label to pilot_xds  metric

This is an automated cherry-pick of #25278

cc @mandarjog @douglas-reid Any concerns with porting this to release 1.6? I don't think cardinality is an issue.

@googlebot I consent

I don't have any concerns about cherry-picking this.


Markers visible only on one side of the international date line

Summary:
Markers are only visible on one side of the international date line at the same time.
Steps to reproduce:
Add some markers on the west side and east side of the international date line, where both should be visible on the screen at the same time.
Expected behavior:
All markers should be visible
Observed behavior:
Only markers on one side of the IDL are visible at the same time, whichever side is more on the screen at the time.
Device and Android version:
Pixelbook
Android 7.1.1
Google Play Services Maps 16.0.0
Screenshots:

I fixed this in https://github.com/googlemaps/android-maps-utils/pull/564.

Fixed by https://github.com/googlemaps/android-maps-utils/pull/564.


Add CashFlows and RatesCupons

This PR adds basic interest rate cash flows.

Codecov Report

Merging #17 (b6b4f33) into main (841c854) will decrease coverage by 0.16%.
The diff coverage is 95.83%.

@@            Coverage Diff             @@
##             main      #17      +/-   ##
==========================================
- Coverage   97.83%   97.68%   -0.16%
==========================================
  Files          27       29       +2
  Lines        1156     1252      +96
==========================================
+ Hits         1131     1223      +92
- Misses         25       29       +4

Files Changed
Coverage Δ

src/DiffFusion.jl
100.00% <ø> (ø)

src/products/Cashflows.jl
92.00% <92.00%> (ø)

src/products/RatesCoupons.jl
97.18% <97.18%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Better sky / atmosphere for global view

The built-in sky environment works nicely at ground level but isn't accurate at planetary scale.
Ground level

Planetary scale

Maybe see if we can integrate https://github.com/sebh/UnrealEngineSkyAtmosphere?


Keyset-based pagination

Needed for Jakarta Data

https://hibernate.atlassian.net/browse/HHH-17772

I'm thinking I'll merge this soon, even given its admittedly-incomplete state, so that I can keep working on the Jakarta Data side of this.
I've applied @Incubating rather liberally over the new APIs.


Support running with Java 21

Note: Compiling and testing on Java 21 will be supported as part of Gradle 8.4 using toolchains. This issue is only for being able to run Gradle itself on Java 21.
### Steps to complete
- [ ] https://github.com/gradle/gradle/issues/25573
- [x] Upgrade Groovy (`GroovyCoverage` as well)
- [ ] https://github.com/gradle/gradle/issues/26764
- [ ] https://github.com/gradle/gradle/pull/26913

Any ETA?

Still waiting on Kotlin 1.9.20 for the necessary version support in JvmTarget.

Still waiting on Kotlin 1.9.20 for the necessary version support in JvmTarget.

I remember reading somewhere that the next Kotlin version was going to be released in December did they mean 1.10/2.0 or did they mean, 1.9.20, because waiting until December for Java 21 would be long, I am so hyped 😅

I asked: https://youtrack.jetbrains.com/issue/KT-58987/Use-some-available-JVM-target-if-theres-no-JvmTarget-for-the-inferred-toolchain-version

As a compilation tool, Gradle should not rely on Kotlin's support for JDK 21, but should at least provide support through Groovy

As a compilation tool, Gradle should not rely on Kotlin's support for JDK 21, but should at least provide support through Groovy

The truth is, as long as Kotlin is there, it will throw exception to prevent Gradle from running on JDK 21.

On Macos im Combination with the Exec Command there is currently also the issue,
that the executable has to be fully qualified (e.g. /usr/local/bin/docker instead of "docker")
and jib will complain that it cannot pickup the credentials from a docker login.
On Linux everything seems to be fine.
a detailed descripted can be found here
https://github.com/gradle/gradle/issues/25574

As a compilation tool, Gradle should not rely on Kotlin's support for JDK 21, but should at least provide support through Groovy

Gradle uses Kotlin to run buildscripts written in the Kotlin DSL. It is necessary for Kotlin to have support for this in order to claim that Gradle can be run on Java 21. If all you need is compilation, that support is already landing in 8.4, see #25573.

Why gradle is always to late to release the latest version support. I am already running jdk 21 with maven since day one Sep 19. switching project form maven to gradle was not a good idea.

Just try it, it works for most of the use cases. We'been compiling and running tests with jdk21 since 8.3RC1, no problem at all

Silly question from me, but as far as we're using Groovy DSL only, isn't possible to deactivate the kotlin part and run 8.3 on JDK21?

Is using Kotlin beta an option for gradle?

Apache Kafka uses the Groovy DSL and we are able to compile and run with Java 21 using Gradle 8.3 - in fact this is enabled and passing in our CI right now.

Apache Kafka uses the Groovy DSL and we are able to compile and run with Java 21 using Gradle 8.3 - in fact this is enabled and passing in our CI right now.

I think it's because #25573 is closed as fixed :)

I have checked, and it is possible to build with JDK 21 using Gradle 8.4-rc-1.
https://docs.gradle.org/8.4-rc-1/release-notes.html

Silly question from me, but as far as we're using Groovy DSL only, isn't possible to deactivate the kotlin part and run 8.3 on JDK21?

In theory this should work without any changes to Gradle, i.e. as long as you don't have any Kotlin buildscripts, it should work as-is.

It seems like a big miss that a tool that many java projects depend on are dependent koltin compilation support before we can see java compilation support.
Regarding the kotlinscript comments, how is compiling the sources of your project's builds coupled to the sources of your built itself? I would think compiling the sources of your project would only depend on a lib for containing a version of javac.
Or is the issue here a release to build a project with only java sources is dependent on also supporting compiling kotlin project sources mixed koltin and java sources, similar to mixed scala support?

Regarding the kotlinscript comments, how is compiling the sources of your project's builds coupled to the sources of your built itself? I would think compiling the sources of your project would only depend on a lib for containing a version of javac.

It isn't, you can (and should) use toolchains. This is why we track compilation and testing separately. I previously stated this in https://github.com/gradle/gradle/issues/25574#issuecomment-1732842944.

It seems like a big miss that a tool that many java projects depend on are dependent koltin compilation support before we can see java compilation support.

I've never seen gradle seamlessly support the latest java without a new release of gradle. At this stage, those projects should know what they're getting into. There are other options.
Even once gradle releases, they'll be projects (looking at you spring) that force you to use old versions of gradle, so you're going to have to wait a while for those to update.
TLDR; If projects want seamless forward compatibility with future versions of Java, they should use Maven. Otherwise, commit to patiently waiting  ¯_(ツ)_/¯
Please, folks, correct me if I'm wrong here. But, my experience as always been that gradle doesn't do forward compatibility.... I'm stuck with gradle, so I'd love to be wrong. Is there a way to use Java 21 with Gradle 7.x ?

Can Gradle consider using a beta version of Kotlin? https://github.com/JetBrains/kotlin/releases/tag/v1.9.20-Beta2 this is out for a week and it should probably be okay if it's just the RC/Beta version of Gradle that depends on Kotlin Beta, and replace with Kotlin release once it's published.

TLDR; If projects want seamless forward compatibility with future versions of Java, they should use Maven.

I'll never use Maven. They don't even have a working nexus publication plugin for modern versions of Java. I've wasted hours on Maven and will never look at Maven ever again.
To me, this is all Kotlin's fault. They release slowly and passively fix bugs related to things they do not use internally.

We are solving this by running on JVM20 and building against JVM21, which works fine on 8.4-rc-1, and you can always build Gradle from trunk, which we did before release 👍🏻

jib + exec problems on macos are still present in 8.4 RC2
@goafabric: remindme

I have confirmed that it is possible to build JDK 21 with Gradle 8.4-rc-1. Even though it's a release candidate version, it can be used if necessary.
https://docs.gradle.org/8.4-rc-1/release-notes.html

Just tried this approach (using RC-2), unfortunately it is failing
Unsupported Java.
Your build is currently configured to use Java 21 and Gradle 8.4-rc-2.

Possible solution:
 - Use Java 20 as Gradle JVM: Open Gradle settings
 - Open Gradle wrapper settings, change `distributionUrl` property to use compatible Gradle version and reload the project

@ipsi-apant you probably have configured IntelliJ to run Gradle with the same JDK as the Project.

@ipsi-apant you probably have configured IntelliJ to run Gradle with the same JDK as the Project.

Thank you @steve-todorov 🙇♂
After configuring project SDK in intellij build settings, gradle build passed using Java 21 (for 8.4-rc-2).

I can confirm this is the way to go.
I have Java and Kotlin targets configured as Java 21 and project SDK as 21 but I need to set JDK 20 for Gradle 8.4-rc3 to work correctly.
Note for anyone else also trying to do this will Kotlin you will also need to change the Kotlin compiler version to 1.9.20-Beta2.

For me this approach is definitely not working .. at least not for native images with graalvm.
Refering to the plugin "org.graalvm.buildtools.native"
First and foremost, it has to be upgrade to the latest "0.9.27" which is not a big deal.
But then the plugin is confused with the different jdk handlings and complains about incompatible java classes.
Pretty sure this can also be configured .. somehow
But honestly all the "configure java21 but java20" is pretty tedious and mind boggling
and no where self explationary for anyone cheking out the project.
So i hope that with Kotlin supporting Java21 eventually, we will get a real fix, outside the toolchain config

If you have an issue with Gradle 8.4:
Unknown Kotlin JVM target: 21

Then update Kotlin to 1.9.20-Beta2, no need to have a snapshot gradle wrapper version

@artemptushkin How would you upgrade the Kotlin that Gradle uses to execute build scripts?

If you have an issue with Gradle 8.4:
Unknown Kotlin JVM target: 21

Then update Kotlin to 1.9.20-Beta2, no need to have a snapshot gradle wrapper version

I am using from intellij. how could i upgrade that?

How would you upgrade the Kotlin that Gradle uses to execute build scripts?

I just compiled (in terminal and Idea) a simple Gradle Kotlin DSL project too with 2 changes:

*

I am using from intellij. how could i upgrade that?

If you want to change this setting then it's possible, intellij inherits Gradle properties so it's just worked for me - no need to change the compiler even because this error is about Gradle

@artemptushkin Do these changes allow building a Gradle multi-module project with buildSrc folder containing *.kts files? If yes, could you please share your (or sample) project with these settings?

Same for me.
Was:
        java { sourceCompatibility = JavaVersion.VERSION_20; targetCompatibility = JavaVersion.VERSION_20 }
        withType<org.jetbrains.kotlin.gradle.tasks.KotlinCompile>().configureEach { kotlinOptions { jvmTarget = "20" } }

Now:
        java { toolchain { languageVersion.set(JavaLanguageVersion.of(21)) } }
        withType<org.jetbrains.kotlin.gradle.tasks.KotlinCompile>().configureEach { kotlinOptions { jvmTarget = "21" } }

Everything is working locally and in github actions with gradle 8.4 & kotlin 1.9.20-RC. And I don't set vendor.set(?).
Action:
      - name: 🪜 Set up JDK 21
        uses: actions/setup-java@v3
        with:
          java-version: 21
          distribution: temurin

And in docker I use FROM amazoncorretto:21

@izogfif Yes, I think it will work, .kts is the same Kotlin file as others. I'd not share, it's a trivial change that probably won't be needed when Gradle updates.

@artemptushkin how do you use that kotlinVersion? I assume that is in a toml file but what dependency does it apply to?

@namero999 i found this way works,
buildSrc/build.gradle.kts
plugins {
    kotlin("jvm") version "1.9.20-RC"
    `kotlin-dsl`
}

repositories {
    mavenCentral()
    gradlePluginPortal()
}

java {
    toolchain {
        languageVersion.set(JavaLanguageVersion.of(21))
    }
}

@namero999 i found this way works,
buildSrc/build.gradle.kts
plugins {
    kotlin("jvm") version "1.9.20-RC"
    `kotlin-dsl`
}

repositories {
    mavenCentral()
    gradlePluginPortal()
}

java {
    toolchain {
        languageVersion.set(JavaLanguageVersion.of(21))
    }
}

Do you understand why you need kotlin-dsl & gradlePluginPortal()? :-)
For any copy-paster dev: please, try different solution and try not to write extra code that is not needed 🤓

It is clear that those lines are unrelated to the question. What was missing from the original answer is to what plugin the version would apply, and how to define it.

i thought the question was “how to build kotlin dsl project with only jdk 21 rather than jdk17 to boot up gradle with java 21 toolchain”
the example snippet above is part of buildSrc script, that’s why it contains kotlin dsl and gradle plugin repo
generally
kotlin("jvm") version "1.9.20-RC"
is used to specify kotlin version for compiling gradle kotlin dsl script , which won’t be needed anymore once gradle released full java 21 support
for a completed working example, anyone can refer to
https://github.com/neowu/core-ng-project/blob/master/buildSrc/build.gradle.kts
java project gradle config is defined in
https://github.com/neowu/core-ng-project/blob/master/buildSrc/src/main/kotlin/project.gradle.kts
both use java 21 toolchain

只要尝试一下，它适用于大多数用例。我们从8.3RC1开始就用jdk21编译运行测试，完全没有问题

It's using Gradle+jdk21 to build the project, not compiling and running Gradle using jdk21

Does Gradle 8.4 work with Java 21 in conjunction with Spring? If it does, how can we make it work? Every time we test our code, we encounter unintended errors. Please assist with resolving this issue:
Unsupported Java.
Your build is currently configured to use Java 21.0.1 and Gradle 8.2.1.
Possible solution:
Use Java 19 as Gradle JVM: Open Gradle settings
Open Gradle wrapper settings, change distributionUrl property to use compatible Gradle version and reload the project

@Villa25698 I think you should go 19 and ask stackoverflow questions, it's impossible to determine what your issue and there might be so many options
Kotlin will release 1.9.20 soon with proper support and it will be straightforward

Your build is currently configured to use Java 21.0.1 and Gradle 8.2.1.

@Villa25698 If you want to use Java 21 and want to reduce the number of potential problems, you should update your Gradle version to the latest stable version (8.4 at the time of writing). There are two things to note, though:

Gradle will not officially support running on Java 21 until version 8.5, so you might still encounter problems when using Java 21 with Gradle 8.4.
Your problem doesn't have to be "Unsupported Java". Upon encountering problems, Gradle might say the Java version is not supported yet, but the actual problem might be another one. For example, if your Spring Boot application starts a Tomcat server and you terminate your application, Gradle will show the "Unsupported Java" error. Nothing went wrong, though. Similarly, other (actual) issues may be hidden behind the "Unsupported Java" error. Have a closer look at the build output or logs to determine the underlying issue.

we have the problem that Oracle has removed 19 and 20. they only have 17 and 21
Sendt fra Outlook for Androidhttps://aka.ms/AAb9ysg

From: Artem Ptushkin @.>
Sent: Tuesday, October 24, 2023 12:33:06 PM
To: gradle/gradle @.>
Cc: Villa25698 @.>; Mention @.>
Subject: Re: [gradle/gradle] Support running with Java 21 (Issue #25574)
@Villa25698https://github.com/Villa25698 I think you should go 19 and ask stackoverflow questions, it's impossible to determine what your issue and there might be so many options
Kotlin will release 1.9.20 soon with proper support and it will be straightforward
—
Reply to this email directly, view it on GitHubhttps://github.com/gradle/gradle/issues/25574#issuecomment-1776947471, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2UNWUMEXNAGHNPUG23W3FDYA6KOFAVCNFSM6AAAAAAZ2CT7NSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZWHE2DONBXGE.
You are receiving this because you were mentioned.Message ID: @.***>

that's the problem. it isn't the case we set it up incorrectly. earlier we did the same test without changing nothing and it said Gradle is configured java 21.0.1 to Gradle 8.3. therefore it cannot run
Sendt fra Outlook for Androidhttps://aka.ms/AAb9ysg

From: Maurits Silvis @.>
Sent: Tuesday, October 24, 2023 12:51:36 PM
To: gradle/gradle @.>
Cc: Villa25698 @.>; Mention @.>
Subject: Re: [gradle/gradle] Support running with Java 21 (Issue #25574)
Your build is currently configured to use Java 21.0.1 and Gradle 8.2.1.
@Villa25698https://github.com/Villa25698 If you want to use Java 21 and want to reduce the number of potential problems, you should update your Gradle version to the latest stable version (8.4 at the time of writing). There are two things to note, though:

Gradle will not officially support running on Java 21 until version 8.5, so you might still encounter problems when using Java 21 with Gradle 8.4.
Your problem doesn't have to be "Unsupported Java". Upon encountering problems, Gradle might say the Java version is not supported yet, but the actual problem might be another one. For example, if your Spring Boot application starts a Tomcat server and you terminate your application, Gradle will show the "Unsupported Java" error. Nothing went wrong, though. Similarly, other (actual) issues may be hidden behind the "Unsupported Java" error. Have a closer look at the build output or logs to determine the underlying issue.

—
Reply to this email directly, view it on GitHubhttps://github.com/gradle/gradle/issues/25574#issuecomment-1776976567, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2UNWUN2ELT5POI37GMAQV3YA6MTRAVCNFSM6AAAAAAZ2CT7NSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZWHE3TMNJWG4.
You are receiving this because you were mentioned.Message ID: @.***>

Still waiting on Kotlin 1.9.20 for the necessary version support in JvmTarget.

I remember reading somewhere that the next Kotlin version was going to be released in December did they mean 1.10/2.0 or did they mean, 1.9.20? Because waiting until December for Java 21 would be long, I am so hyped 😅

Kotlin 1.9.20 is released now.

I don't see it either on Github or on Kotlins release page?

I don't see it either on Github or on Kotlins release page?

It‘s currently being published. See for example:
https://repo1.maven.org/maven2/org/jetbrains/kotlin/kotlin-gradle-plugin/1.9.20/

Wow!

Kotlin 1.9.20 is released now.

Does that mean Gradle will support JDK 21 soon?

Why is this even an issue? There should be zero dependencies on the JDK from Gradle - unless you change the Gradle code to specifically use a JDK feature - which should be extremely rare.
This points to a less than great internal design of the Gradle codebase - which is a shame.

Why is this even an issue? There should be zero dependencies on the JDK from Gradle.

Gradle's Kotlin DSL depends on Kotlin, which in turn relies on a specific JDK version. As such, running Gradle is only fully supported for specific JDK versions (i.e., those that Kotlin supports).

Why is this even an issue? There should be zero dependencies on the JDK from Gradle.

Gradle's Kotlin DSL depends on Kotlin, which in turn relies on a specific JDK version. As such, running Gradle is only fully supported for specific JDK versions (i.e., those that Kotlin supports).

That is fine - so run Gradle with the required JDK - but have Gradle support whatever language version the build file specifies - there should be no dependency there. But that is not what happens according to:
https://github.com/oracle/javavscode/issues/29
Still, core infra that relies on a subsystem with similar strict dependencies is not the best. The same thing applies to the Kotlin DSL - there should be no reason that shouldn't run with the latest JDK - only if the DSL is changed to support some later feature should it ever be an issue.
It's like developers forgot the forwards compatibility mantra of Java. You've always been able to run compliant Java code on a future JDK release without changes.

You know that what you are asking is exactly what is described in this issue? You can already use the toolchains feature to compile and test your own code using JDK 21, it's just that you cannot yet run Gradle itself with JDK 21. I have lots of projects that are already on JDK 21 and those build just fine with Gradle 8.4 (and also with earlier versions). I just use JDK 20 to run Gradle itself.

I am sure it’s not gradle’s fault but the gradle language server extension is barging when the compiled class files are 65.
It is probably MS fault - but still. Clearly the ecosystem doesn’t understand javas forward principles. This should never have been an issue.

@robaho if you read this issue there's a link above on youtrack. Basically, kotlin has an enum with jdk versions, and if it doesn't find a suitable one, it throws an exception. Kotlin team fixed this bug but it takes like two to three months to release the fix.

Kotlin team fixed this bug but it takes like two to three months to release the fix.

Wierd 🤣

@robaho if you read this issue there's a link above on youtrack. Basically, kotlin has an enum with jdk versions, and if it doesn't find a suitable one, it throws an exception. Kotlin team fixed this bug but it takes like two to three months to release the fix.

That is my point. Given the current Java release cycle it is suboptimal for Gradle to couple itself to systems that do no adhere to Javas forward compatibility principles. Drop Kotlin and move on. Or at minimum let the Groovy and Kotlin versions diverge.

Drop Kotlin and move on

Like rewrite all these gigantic Kotlin codebase? I think rewriting Gradle completely without asm but with new JDK classfile API and without Kotlin/Groovy would be cheaper than that.

Guys, this is becoming an offtopic discussion. In a perfect world everything would be perfect, but the reality is we do not live in a perfect world. Deal with it, open a new ticket with your suggestions for improvements or move on to another tool that better fits your needs.

How is suggesting splitting gradle into two versions to address issues like this off topic?
it has to be very rare that a package needs both Kotlin and Groovy DSL support.

How is suggesting splitting gradle into two versions to address issues like this off topic?
it has to be very rare that a package needs both Kotlin and Groovy DSL support.

It is offtopic because "splitting gradle into two versions" and "rare that a package needs both Kotlin and Groovy DSL" diverge from "Support running with Java 21".
It's an entirely different discussion to be had. I suggest you create another issue and continue the discussion from there, whilst referencing the discussion in this issue here.

Filed #26944. Kotlin fanboys feel free to downvote.

Fixed by #26913

Fixed by #26913

Unsupported Java.
Your build is currently configured to use Java 21.0.1 and Gradle 8.5-rc-1.

@octylFractal you want me to open a new issue?

Fixed by #26913

Unsupported Java.
Your build is currently configured to use Java 21.0.1 and Gradle 8.5-rc-1.

This is intellij that's doing this. Upgrade to 2023.3 beta and you won't see it again

i'm using android-studio and it seems to be behind 2<IP_ADDRESS>

When will be 8.5 GA available?

Fixed by #26913

Unsupported Java.
Your build is currently configured to use Java 21.0.1 and Gradle 8.5-rc-1.

This is intellij that's doing this. Upgrade to 2023.3 beta and you won't see it again

Yeah this error is from intellij https://youtrack.jetbrains.com/issue/IDEA-326038 . A extremely annoying.
They are using Gradle's compatibility list for validation. https://docs.gradle.org/current/userguide/compatibility.html
Once, gradle updates compatibility list, I think it should work in intellij.

Fixed by #26913

Unsupported Java.
Your build is currently configured to use Java 21.0.1 and Gradle 8.5-rc-1.

In intellij settings, if we set JVM 20 for Gradle and project on Java 21. It works.

@ipsi-apant - when I set it to use Java 20 as you indicated, I get this error:
Cause: error: invalid source release: 21

Is the language level limited to the Java version used to run Gradle itself?

@ipsi-apant - when I set it to use Java 20 as you indicated, I get this error:
Cause: error: invalid source release: 21

Is the language level limited to the Java version used to run Gradle itself?

This was an interim solution until gradle 8.4. After release of Gradle 8.5 and updating intellij to the latest I am not seeing this error.
There was a ticket in YouTrack (can't recall) which described, for gradle based projects, intellij relies on Gradle's Java compatibility matrix.


[Bug] Event OnBeforeInsertReminderTextLine in Reminder Header procedure InsertLines

The call to event OnBeforeInsertReminderTextLine in procedure InsertLines in table "Reminder Header" seems to be a bug.
The event is not logical in the first place because there is no text line being inserted at that point. The code is inserting an additional fee line instead.
On top of that, the parameter ReminderHeader that is passed with the event will be empty in this case. It is not initialized in the function and it will not be initialized by earlier function calls either.
    procedure InsertLines()
    ........
                ReminderLine.Validate(Amount, AdditionalFee);
                ReminderLine."Line Type" := ReminderLine."Line Type"::"Additional Fee";
                OnBeforeInsertReminderTextLine(ReminderLine, ReminderText, ReminderHeader); // <-- incorrect event? ReminderHeader parameter will be empty
                OnBeforeInsertReminderLine(ReminderLine);
                ReminderLine.Insert();
    ........

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.


How to set rate limitation for multiple client IPs

How to set rate limitation for multiple client IPs?
Description:

I use remote_address action against descriptor to rate limit one specific client IP perfectly. But if I want to rate limit several client IPs, how should I do?

My config looks this:
rate_limits:
   - actions:
   - remote_address: {}
---
descriptors:
  - entries:
  - key: remote_address
    value: "<IP_ADDRESS>"
  token_bucket:
    max_tokens: 1
    tokens_per_fill: 1
    fill_interval: 60s

It works fine, and for client IP <IP_ADDRESS>, it can only request 1 time per minute.
Thanks

Presumably this is local rate limit, I think https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/local_rate_limit_filter#config-http-filters-local-rate-limit explains how to match multiple values for a given header.


tcp2: Implement a blocking connect

In order to implement a blocking connect, add a semaphore
and block on it in net_tcp_connect().
The semaphore is released when ESTABLISHED state is reached.
In case tcp_conn_unref() is called while waiting on the semaphore,
defer the unreference, tcp_conn_unref() will be called from
net_tcp_connect().
Fixes #26695

Why don't we use cb parameter of net_tcp_connect() (both in TCP and TCP2) to notify about established connection? What we are really missing in #26695 is that posix connect() should block by default, not necessarily net_tcp_connect(). Such behavior could be implemented by waiting on semaphore in zsock_connect_ctx().
In Linux userspace there is also a possibility to change socket file descriptor to non-blocking mode. Then you can call connect(), which won't block. poll() can be used on such file descriptor to get notification about connection being established. This gives userspace a mechanism for timeout, as file descriptor can be closed when userspace decides it takes too long to connect. If at some point we would like such mechanism in Zephyr, then it would be much easier to put all the logic in the zsock_connect_ctx() and decide whether to use semaphore (blocking mode) or signal poll() in cb implementation passed to net_context_connect().

This will be the better solution, but for now the module is unusable.
Temporarily solving it as currently suggested will be extremely helpful.
Thank you!

Why don't we use cb parameter of net_tcp_connect() (both in TCP and TCP2) to notify about established connection? What we are really missing in #26695 is that posix connect() should block by default, not necessarily net_tcp_connect(). Such behavior could be implemented by waiting on semaphore in zsock_connect_ctx().
In Linux userspace there is also a possibility to change socket file descriptor to non-blocking mode. Then you can call connect(), which won't block. poll() can be used on such file descriptor to get notification about connection being established. This gives userspace a mechanism for timeout, as file descriptor can be closed when userspace decides it takes too long to connect. If at some point we would like such mechanism in Zephyr, then it would be much easier to put all the logic in the zsock_connect_ctx() and decide whether to use semaphore (blocking mode) or signal poll() in cb implementation passed to net_context_connect().

Why don't we use cb parameter of net_tcp_connect() (both in TCP and TCP2) to notify about established connection? What we are really missing in #26695 is that posix connect() should block by default, not necessarily net_tcp_connect(). Such behavior could be implemented by waiting on semaphore in zsock_connect_ctx().

Indeed, that sounds actually a better solution. I think we need to revisit this fix anyway if I manage to figure out how to implement the user mode networking support. As we are understaffed atm and this fix works, I am merging this as is. Patches are welcome to implement this differently.


Add a log when marking shard as available

Coverage increased (+0.1%) to 77.982% when pulling 0820d4897cd631b7fff80928357a0c907be93632 on chao-log into 32aa0dcc8051afb957ded81d8b89cf8f8f3c6eed on master.

Coverage increased (+0.1%) to 78.008% when pulling 0820d4897cd631b7fff80928357a0c907be93632 on chao-log into 32aa0dcc8051afb957ded81d8b89cf8f8f3c6eed on master.


[Bug] scoreboard and click combo error

This is just a reference to the error I already showed you @Travja https://pastebin.com/wuxAF4si

Should be resolved in newest build. Close this when verified.