Does compleasm try to download something when run specifying lineage?

Hi,
I am trying to get compleasm running, but run into an issue. Specifically, I get this error:
Traceback (most recent call last):
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/urllib/request.py", line 1350, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/http/client.py", line 976, in send
    self.connect()
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/http/client.py", line 1443, in connect
    super().connect()
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/http/client.py", line 948, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/socket.py", line 728, in create_connection
    raise err
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/socket.py", line 716, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/bin/compleasm", line 10, in <module>
    sys.exit(main())
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/site-packages/compleasm.py", line 2534, in main
    args.func(args)
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/site-packages/compleasm.py", line 2425, in run
    mode=mode)
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/site-packages/compleasm.py", line 2092, in __init__
    self.downloader = Downloader(library_path)
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/site-packages/compleasm.py", line 85, in __init__
    self.lineage_description, self.placement_description = self.download_file_version_document()
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/site-packages/compleasm.py", line 127, in download_file_version_document
    urllib.request.urlretrieve(hash_url, hash_download_path)
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/urllib/request.py", line 247, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/urllib/request.py", line 1393, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/cluster/projects/nn8013k/programs/miniconda3/envs/compleasm/lib/python3.7/urllib/request.py", line 1352, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 110] Connection timed out>

The command is this: compleasm run -a default_filt.hic.hap1.p_ctg.fa -l aves -L /cluster/projects/nn8013k/opt/busco_dbs/lineages/ -t 10 -o compleasm_test
When running on the login nodes, it looks like it actually downloads aves and eukaryota, even though the folders existed in /cluster/projects/nn8013k/opt/busco_dbs/lineages/.
Am I doing something wrong?
Our computing nodes do not have internet access, and I don't think it is nice practice to download something without letting the user know that it is happening. How can I turn this off? That is, can I download something before submitting the job to the cluster?
Thank you.
Ole

I pointed compleasm to where I had downloaded all the lineages for BUSCO. Does it require something different than BUSCO, because it still downloaded the lineage. Can I use the compleasm downloaded lineages for BUSCO? I'd rather not have two sets of lineages laying around on the cluster.
What I did for BUSCO was to download everything, and then point to it. I guess I have to download each lineage independently in the case of compleasm?
When I got the lineage downloaded via compleasm, it ran successfully. So looking good so far. :)

@olekto
The organization of the lineage file downloaded by compleasm is different from that of BUSCO. So directly specifying the lineage directory downloaded by BUSCO doesn't work. We will consider making compleasm compatible with the lineage files downloaded by BUSCO in future versions.

is there a way to download lineage files manually instead of using compleasm download?

is there a way to download lineage files manually instead of using compleasm download?

I think you can directly download the data from here:
https://busco-archive.ezlab.org/data/lineages/
and can use it after unzip.

@olekto The organization of the lineage file downloaded by compleasm is different from that of BUSCO. So directly specifying the lineage directory downloaded by BUSCO doesn't work. We will consider making compleasm compatible with the lineage files downloaded by BUSCO in future versions.

I am waiting on this feature, too. Enabling the program to run with the lineage files downloaded by BUSCO will be really useful for my case.

Hi, are there any updates on this issue? The protein command works well in offline mode, but the run command is not working properly in offline mode

@olekto The organization of the lineage file downloaded by compleasm is different from that of BUSCO. So directly specifying the lineage directory downloaded by BUSCO doesn't work. We will consider making compleasm compatible with the lineage files downloaded by BUSCO in future versions.

Hi @lilinzhou,
The latest version of compleasm v0.2.5 has fix the problem. The bug is caused by the update of BUSCO related file format few month ago.

Hi @huangnengCSU
The latest version seems also request for a download by using run command. I found a newly empty file "file_versions.tsv.tmp" in the BUSCO database folder.
The command I use: python3 /path/to/software/compleasm_kit/compleasm.py run -a genome.fasta -l eukaryota_odb10 -o out_genome -L /path/to/software/BUSCO/lineages
see the error log at the last.
But everything goes well by using protein command.
The command I use: python3 /path/to/software/compleasm_kit/compleasm.py protein -a protein.fasta -l eukaryota_odb10 -o out_protein -L /path/to/software/BUSCO/lineages
Our nodes do not have internet access, I can only download the BUSCO database manually. Could you help to solve this problem.
Searching for miniprot in the path where compleasm.py is located
Searching for hmmsearch in the path where compleasm.py is located
miniprot execute command:
 /path/to/software/compleasm_kit/miniprot
Traceback (most recent call last):
  File "/path/to/software/Python-3.9.6/lib/python3.9/urllib/request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/path/to/software/Python-3.9.6/lib/python3.9/http/client.py", line 1257, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/path/to/software/Python-3.9.6/lib/python3.9/http/client.py", line 1303, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/path/to/software/Python-3.9.6/lib/python3.9/http/client.py", line 1252, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/path/to/software/Python-3.9.6/lib/python3.9/http/client.py", line 1012, in _send_output
    self.send(msg)
  File "/path/to/software/Python-3.9.6/lib/python3.9/http/client.py", line 952, in send
    self.connect()
  File "/path/to/software/Python-3.9.6/lib/python3.9/http/client.py", line 1426, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/path/to/software/Python-3.9.6/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/path/to/software/Python-3.9.6/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/path/to/software/Python-3.9.6/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/path/to/software/compleasm_kit/compleasm.py", line 2741, in <module>
    main()
  File "/path/to/software/compleasm_kit/compleasm.py", line 2737, in main
    args.func(args)
  File "/path/to/software/compleasm_kit/compleasm.py", line 2601, in run
    mr = CompleasmRunner(assembly_path=assembly_path,
  File "/path/to/software/compleasm_kit/compleasm.py", line 2114, in __init__
    self.downloader = Downloader(library_path)
  File "/path/to/software/compleasm_kit/compleasm.py", line 85, in __init__
    self.lineage_description, self.placement_description = self.download_file_version_document()
  File "/path/to/software/compleasm_kit/compleasm.py", line 127, in download_file_version_document
    urllib.request.urlretrieve(hash_url, hash_download_path)
  File "/path/to/software/Python-3.9.6/lib/python3.9/urllib/request.py", line 239, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/path/to/software/Python-3.9.6/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/path/to/software/Python-3.9.6/lib/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/path/to/software/Python-3.9.6/lib/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/path/to/software/Python-3.9.6/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/path/to/software/Python-3.9.6/lib/python3.9/urllib/request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/path/to/software/Python-3.9.6/lib/python3.9/urllib/request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error EOF occurred in violation of protocol (_ssl.c:1129)>

@lilinzhou,
I guess your problem should be caused by the network. During first run, compleasm will download some files. But since the network problem, the download did not finished and there will be some tag files having the name ending with .tmp in the download folder.
If the problem is from network, you may have to download the lineage files using compleasm download on a computer having access to database of lineage files. Then you can upload the download folder on the computer to work server. When performing compleasm run you can specify the download folder with option -L.
Why this problem only occurs in compleasm run and not in compleasm protein is because there is a process to check the lineage files in compleasm run. However, compleasm protein does not have this process (compleasm protein should have this process but I have not implemented it).


Refactor MAISI bundle to monai 1.4

Fixes # .
Description
refactor to monai1.4,
add quality check,
correct typo
add download script
Status
Ready/Work in progress/Hold
Please ensure all the checkboxes:

[x] Codeformat tests passed locally by running ./runtests.sh --codeformat.
[ ] In-line docstrings updated.
[ ] Update version and changelog in metadata.json if changing an existing bundle.
[ ] Please ensure the naming rules in config files meet our requirements (please refer to: CONTRIBUTING.md).
[ ] Ensure versions of packages such as monai, pytorch and numpy are correct in metadata.json.
[ ] Descriptions should be consistent with the content, such as eval_metrics of the provided weights and TorchScript modules.
[ ] Files larger than 25MB are excluded and replaced by providing download links in large_file.yml.
[ ] Avoid using path that contains personal information within config files (such as use /home/your_name/ for "bundle_root").

/build

/build

/build

/build

/build


Ungrouping works, grouping does not

It seems that a recent Sonos update made it so.

Works for me.

Controller: version 6.3.2 (build 321131051)
PLAY:5 version 6.3 (build 321130162a)
PLAYBAR: version 6.3 (build 321130071)

Works on my machine, although it can be improved for systems with many players.


support esm modules

Hi, thanks for the PR, I am not sure if we want to go with ESM for now, seems risky. I don't think the ecosystem is ready for this, I find it not working well with a lot of external dependencies


Squeezelite not closing when quiting companion

when quiting companion the squeezelite instance stays running so if you had music playing it wont close
so you have to quit using task manager

https://github.com/music-assistant/companion/blob/a5f80e5ce5e7d870868cfb65a25e81de9349e90b/src-tauri/src/lib.rs#L131

Hmm interesting, I would have thought that Tauri automatically cleaned up the child processes when running app.exit(1);. I can take a quick look at it

I saw this should be fixed in 0.0.74 but after using the companion (and quitting it) i still see the MA companion squeezelite process running.
Can you open this issue again @arctixdev or should i open a new one?

Hmm ok, I am not able to work on this at the moment but anyone is welcome to take a look at it!


WIP: Post release fixes

[x] Fix release command to work also under mac zsh
[ ] Fix doc build
[ ] Fix test roundtrip

Codecov Report

Merging #36 (d4e9ba9) into master (6358098) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #36   +/-   ##
=======================================
  Coverage   90.09%   90.09%
=======================================
  Files          16       16
  Lines         444      444
  Branches       52       52
=======================================
  Hits          400      400
  Misses         35       35
  Partials        9        9

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6358098...d4e9ba9. Read the comment docs.


failed to pull the webportal_plugin image

After I run the pull command, I get the following error:
openpai@dev-box:~$ docker pull docker.pkg.github.com/microsoft/openpaimarketplace/webportal_plugin:latest
Error response from daemon: unauthorized: Your request could not be authenticated by the GitHub Packages service. Please ensure your access token is valid and has the appropriate scopes configured

Hi, the docker image is published in (github packages)[https://github.com/microsoft/openpaimarketplace/packages/171126], so before you pull the image, you should firstly login github packages with docker.
Besides, we are working on a new release version of openpaimarketplace, it is highly recommended to use new version. I will update in this issue when new verison is released.

Thanks. I`ve already logged in but still get the same error.

Hi, the doc has been updated. Please refer to the new deployment doc or webportal doc for how to set up marketplace webportal.


New api endpoint

Change endpoint address from the domain ebanx.com to the domain ebanxpay.com to improve performance and security

Codecov Report

Merging #138 into master will not change coverage.
The diff coverage is 100%.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9709d46...50f34e1. Read the comment docs.


file owned by 'xst-git'

I'm not sure what the issue is here.  This is the output of any pacman related command with xst-git from AUR installed.
error: file owned by 'ncurses' and 'xst-git': 'usr/share/terminfo/s/st'
error: file owned by 'ncurses' and 'xst-git': 'usr/share/terminfo/s/st-256color'

The issue is ncurses provides st termcap definitions, which conflict with xst as it never changed the name.
Perhaps we can change the name (and then the TERM names) to xst-*..

I suppose in the meantime it's not causing any failures, it's just an eyesore.

is anyone going to work on that?
otherwise i could submit a PR (for this and #36)

Go for it, @actionless.


add fMinimizeConnections

The policy was not specified.

Hi @hankoh , congrats on your first PR into the driver docs repo! Thanks for pointing out the name of the policy. I moved the name from the section header to the body text just to be consistent with the other section headers, but otherwise I'll go ahead and merge this PR for you. It should be live later today.


Ensure S3 bucket has server-side encryption enabled

Control ID:
SC-13/SC-28
AWS Config Rule:
s3-bucket-server-side-encryption-enabled
Story:
As an AWS consultant
I would like to check if my S3 bucket configuration is compliant (has server-side encryption enabled)
So that I can check if my application conforms with Control IDs SC-13 and SC-28 within NIST 800-53.
Acceptance Criteria:
* Given a CDK stack with *a* *non-compliant S3* bucket
    * When NIST-503 Secure Aspects is run
    * Then the CDK stack does not deploy and the consultant receives an explanation about the non compliant S3 bucket for Control IDs SC-13 and SC-28 within NIST 800-53 .
* Given a CDK stack with *a* *compliant S3* bucket and other compliant resources
    * When NIST-503 Secure Aspects is run
    * Then the CDK stack deploys and the consultant does not receive an explanation about NIST 800-53.
* Given a CDK stack *without an S3* bucket and other compliant resources:
    * When NIST-503 Secure Aspects is run
    * Then the CDK stack deploys and the consultant does not receive an explanation about NIST 800-53

Closing as it's a duplicate of #104


Add instant submission mode

This adds a new task submission model: Instant submission. In instant submission, the entire existing scheduling and DAG construction logic is bypassed, and the node is instead immediately submitted to an inorder_executor.
This also allows us to circumvent the existing garbage collection and submission worker thread usage, which can make the behavior more predictable in scenarios with high CPU utilization due to user code.
The hope is that instant submission can reduce latencies in strong-scaling scenarios.
Instant submission is automatically activated when appropriate conditions are met:

task uses no buffers (USM-only)
task uses no SYCL 2020 reductions
task has no running dependencies on non-instant tasks
task is bound to a device
task is hard-wired to an in-order backend_executor, which happens e.g. when using a SYCL in-order queue.

Note that stdpar code always satisfies the conditions for instant submission.

[ ] The current implementation is NOT THREAD-SAFE, as in instant mode, multiple user threads can in principle call directly into runtime internals, which were developed under the assumption that only the runtime worker thread would be using them. Before merging, we need to add all the mutexes.
[x] We should also consider whether we want to enable this automatically by default as it is now, or whether it should - at least initially - be opt-in. In that case the user would need to set a macro to enable this optimization, if tasks are eligible. Introducing an entire new submission model makes it likely that some issues might pop up, so for stability reasons, it might be a good idea to not immediately expose this to all the code.

CC @al42and @pszi1ard

With 3056a88, the following code hangs in hipStreamSynchronize(); log attached below:
#include <sycl/sycl.hpp>

using T = char;

int main()
{
    for (const auto& dev : sycl::device::get_devices())
    {
        sycl::queue q{ dev, { sycl::property::queue::in_order() } };
        char*       arr_d = sycl::malloc_device<T>(123, q);
        char*       arr_h = sycl::malloc_device<T>(123, q);

        std::iota(arr_h, arr_h + 123, 42);

        q.submit(sycl::property_list{ sycl::property::command_group::hipSYCL_coarse_grained_events() },
                 [&](sycl::handler& cgh) { cgh.memcpy(arr_d, arr_h, 123 * sizeof(T)); });

        q.submit(sycl::property_list{ sycl::property::command_group::hipSYCL_coarse_grained_events() },
                 [&](sycl::handler& cgh) { cgh.memcpy(arr_h, arr_d, 123 * sizeof(T)); });

        q.wait_and_throw();

        sycl::free(arr_d, q);
        sycl::free(arr_h, q);
    }
}

verbose.log

Thanks, I've already seen this issue in CI. It seems to only happen for some backends. CUDA seems to be fine.

Seems like it incorrectly generates memcopies between the CPU device for some reason while dispatching the work to GPU, which cannot go well...

The CPU copies are only there because the test is first executed on the CPU device. If I only run on a GPU, no CPU->CPU copies are executed.

@al42and This is not what I meant; the issue I observed was that in certain cases a CPU->CPU copy was dispatched to the GPU queue. However, I could since fix this issue. It was limited to OpenCL and caused by a bug in the OpenCL backend's USM pointer query implementation, causing it to not correctly recognizing the device attached to the device pointer.
 This has caused the inorder_executor to assume that every task is from some unknown backend, and thus needs special synchronization...
Please try again :)

With this PR, for Lulesh with C++ standard parallelism offloading to NVIDIA in the CPU-bound regime, runtime goes down from 86s to 70s. It's now only 13s slower than nvc++ in this case, and most of these 13s can likely be attributed to SSCP JIT and runtime startup.

Whether instant submission is allowed is now controlled via the HIPSYCL_ALLOW_INSTANT_SUBMISSION macro.
In SYCL mode, instant submission is now opt-in, requiring that #define HIPSYCL_ALLOW_INSTANT_SUBMISSION 1 is present prior to including sycl.hpp. In stdpar mode, it is active by default.
Once things are more mature and tested, we should consider also enabling it by default in SYCL.

The preliminary results look good. All tests pass, and, for a small system (24k atoms), in GPU-resident mode with, we get noticeable speed-up:
On my workstation, RTX3060: 383 (374) -> 397 ns/day.
On Dardel, MI250X: 542 (472) -> 585 ns/day.
We usually set  HIPSYCL_RT_MAX_CACHED_NODES=0, but the numbers in () show the results without HIPSYCL_RT_MAX_CACHED_NODES=0. So, in "default" mode without any user interaction, the difference is much more dramatic. The CPU usage is also reduced.

@al42and Excellent, thanks for the feedback! How do we compare to HIP?

Looking at a more pathological case (6k atoms), the picture gets weirder.
For example, we have three kernels launched almost back-to-back (they are in different modules, so there is some host activity in-between, but not too much).
Here's how native HIP looks (the first hipLaunchKernel corresponds to the yellow kernel in the lower right corner); it takes ~13µs to launch all kernels:

If we look at the old code,  the version of hipSYCL without this patch with set HIPSYCL_RT_MAX_CACHED_NODES=0, it takes slightly longer, ~21µs:

Now, if we look at the instant submission case, things get much worse, 38µs:

Not sure what's causing the slowdown, but it also lowers the overall application performance, 1073.321 ns/day with HIP, 781.733 with Instant submission, 833.722 with old submission logic and no caching in this case (running 5 GROMACS threads bound to a 8c/16t CCX, no reserved cores, HSA_OVERRIDE_CPU_AFFINITY_DEBUG=0).
When the CPU is oversubscribed, though, this patch significantly improves performance (although, that's inadvisable way to run things to begin with).

Normally, the "new" (instant) behavior is indeed faster:
$ ROCR_VISIBLE_DEVICES=4 ./1128_instant_on
0.32079 seconds
$ ROCR_VISIBLE_DEVICES=4 ./1128_instant_off
0.395244 seconds

But if we bind all the cores to the CCX corresponding to the GPU, the "old" case gets much faster:
$ ROCR_VISIBLE_DEVICES=4 hwloc-bind --cpubind core:1-7 ./1128_instant_on
0.318388 seconds
$ ROCR_VISIBLE_DEVICES=4 hwloc-bind --cpubind core:1-7 ./1128_instant_off
0.27572 seconds

The "old" case also sometimes crashes when run with HIPSYCL_RT_MAX_CACHED_NODES=0, but that's besides the point:
$ HIPSYCL_RT_MAX_CACHED_NODES=0 ROCR_VISIBLE_DEVICES=4 ./1128_instant_off
double free or corruption (fasttop)free(): invalid pointer

Aborted
andreyal@nid002910:~$ HIPSYCL_RT_MAX_CACHED_NODES=0 ROCR_VISIBLE_DEVICES=4 gdb --args ./1128_instant_off
Reading symbols from ./1128_instant_off...
(gdb) r
(gdb) bt
#0  0x00000000006d51c0 in ?? ()
#1  0x00007ffff7fdebc9 in hipsycl::rt::dag_direct_scheduler::submit(std::shared_ptr<hipsycl::rt::dag_node>) () from /cfs/klemming/home/a/andreyal/OpenSYCL/build-5.3.3-instant-submission/install/lib/libhipSYCL-rt.so
#2  0x00007ffff7fe5725 in hipsycl::rt::dag_unbound_scheduler::submit(std::shared_ptr<hipsycl::rt::dag_node>) () from /cfs/klemming/home/a/andreyal/OpenSYCL/build-5.3.3-instant-submission/install/lib/libhipSYCL-rt.so
#3  0x00007ffff7fe6f65 in std::_Function_handler<void (), hipsycl::rt::dag_manager::flush_async()::$_0>::_M_invoke(std::_Any_data const&) () from /cfs/klemming/home/a/andreyal/OpenSYCL/build-5.3.3-instant-submission/install/lib/libhipSYCL-rt.so
#4  0x00007ffff7fecbd6 in hipsycl::rt::worker_thread::work() () from /cfs/klemming/home/a/andreyal/OpenSYCL/build-5.3.3-instant-submission/install/lib/libhipSYCL-rt.so
#5  0x00007ffff62773b4 in ?? () from /usr/lib64/libstdc++.so.6
#6  0x00007ffff54266ea in start_thread () from /lib64/libpthread.so.0
#7  0x00007ffff595aa8f in clone () from /lib64/libc.so.6

Ok... I cannot reproduce this behavior, even with hwloc-bind. What is the host CPU?
I also cannot reproduce the crash so far. Obviously, if something UB is going on there, then it might be difficult to compare performance. Is ROCm LLVM involved here? If so I'd suggest to try stock LLVM. Wouldn't be the first time ROCm LLVM miscompiles code.

What is the host CPU?

AMD EPYC 7A53

Is ROCm LLVM involved here?

Yes... Also, it's ROCm 5.3.3, which has some runtime latency issues that the newer versions do not.

If so I'd suggest to try stock LLVM. Wouldn't be the first time ROCm LLVM miscompiles code.

I can try Cray LLVM, but last time I checked (~1 year ago), it had troubles compiling hipSYCL, so I wouldn't hold it as a gold standard.
I can take a stab at building stock LLVM from scratch in the next days.

 AMD EPYC 7A53

Interesting, I can't find information on that one; is that a special LUMI CPU or did you mean 7453?
In any case, I don't have any Milan CPUs here for testing, but I can try to get access to some.

I can try Cray LLVM, but last time I checked (~1 year ago), it had troubles compiling hipSYCL, so I wouldn't hold it as a gold standard.
I can take a stab at building stock LLVM from scratch in the next days.

Is there perhaps any container solution available (e.g. singularity or charliecloud)? That would help with reproducability.

Interesting, I can't find information on that one; is that a special LUMI CPU or did you mean 7453?

It's the special LUMI (Frontier, Dardel) CPU. https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#simplified-node-layout
I can speculate that, with 128 virtual CPUs available, the OS scheduler might decide spread all the threads over different L3 domains hurting the communications between hipSYCL runtime threads.
Deliberately binding the test app on the edge of two domains (two cores from the first domain 0-7, two from the second 8-15) also kills the performance:
$ ROCR_VISIBLE_DEVICES=4 hwloc-bind --cpubind core:0-3 ./1128_instant_off
0.276253 seconds
$ ROCR_VISIBLE_DEVICES=4 hwloc-bind --cpubind core:7-10 ./1128_instant_off
0.413205 seconds
$ ROCR_VISIBLE_DEVICES=4 hwloc-bind --cpubind core:8-12 ./1128_instant_off
0.272617 seconds

Is there perhaps any container solution available (e.g. singularity or charliecloud)?

With a Cray system image? Not that I know of. We asked LUMI staff a while ago, and they could not suggest anything.

It's the special LUMI (Frontier, Dardel) CPU. https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#simplified-node-layout

I see, thanks!

I can speculate that, with 128 virtual CPUs available, the OS scheduler might decide to spread the threads over different L3 domains hurting the communications between hipSYCL runtime threads.
Deliberately binding the test app on the edge of two domains (two cores from the first domain 0-7, two from the second 8-15) also kills the performance:

I can totally see that this might be an issue for the old submission mechanism. Instant submission does not use any internal hipSYCL threads, so it should mostly be unaffected by binding (as your results confirm). But I'm surprised that the old mechanism outperforms instant submission. That would mean that for the little work that it still does, it still is beneficial to offload that to a worker thread.
We could try having a submission worker thread per queue that does nothing but submission for the instant submission case.

But I'm surprised that the old mechanism outperforms instant submission. That would mean that for the little work that it still does, it still is beneficial to offload that to a worker thread.

I don't understand how offloading stuff to a worker thread helps here. If we were doing something compute-intensive in the main thread that can overlap with submission, sure. But here we just submit a ton of kernels and then wait. As far as I can tell, the only thing we can do concurrently is checking the node properties and calling the HIP API. Is the former really so expensive that it makes sense to overlap the two?
Would it be more lightweight to use a per-submission/per-queue flag enforcing "instant" mode, and letting the user shoot themselves in the foot if they so desire?

I can totally see that this might be an issue for the old submission mechanism.

BTW, I can see how the same problem might happen on consumer devices with multiple L3 caches, like 7950X3D. Non-HPC users are unlikely to bother with manual affinity setting, so I feel like getting rid of threads and associated problems when running in a latency-sensitive mode might be the best way forward.

I don't understand how offloading stuff to a worker thread helps here. If we were doing something compute-intensive in the main thread that can overlap with submission, sure. But here we just submit a ton of kernels and then wait. As far as I can tell, the only thing we can do concurrently is checking the node properties and calling the HIP API. Is the former really so expensive that it makes sense to overlap the two?

It should not be that expensive, but I really don't know how the old mechanism might outperform instant submission otherwise. Instant submission just bypasses a lot of stuff; the only thing the old mechanism has that might change the equation is that it immediately offloads to another thread.

Would it be more lightweight to use a per-submission/per-queue flag enforcing "instant" mode, and letting the user shoot themselves in the foot if they so desire?

To be clear, I really don't think that what happens in handler and queue is expensive so that it might benefit from a worker thread. But there are more layers below until we get to the kernel launch, which are certainly more impactful than the couple of lines in handler:  inorder_executor, hip/cuda/ocl/..._queue, sscp/hiplike_kernel_launcher. What they do is mostly trivial too, but I don't know how to explain this otherwise at the moment.

BTW, I can see how the same problem might happen on consumer devices with multiple L3 caches, like 7950X3D. Non-HPC users are unlikely to bother with manual affinity setting, so I feel like getting rid of threads and associated problems when running in a latency-sensitive mode might be the best way forward.

Do I understand correctly that you would prefer instant submission in its current form, even if in certain circumstances the old mechanism might be faster? Instant submission does not currently rely on any threads.

Do I understand correctly that you would prefer instant submission in its current form, even if in certain circumstances the old mechanism might be faster? Instant submission does not currently rely on any threads.

Well, it complicated. If we look at our PME grappa systems, the behavior depends a lot on how we're scheduling things. In the fully GPU-resident mode, we just schedule tons of operations in advance and then just wait for them to complete, while in the partially GPU-resident mode, we compute some forces on the CPU and thus have to do a synchronization with host every ~10 kernels to copy them to the GPU. The table below shows the performance (ns/day) of the fastest configuration among those tested. "Old" runs are with  HIPSYCL_RT_MAX_CACHED_NODES=0; bold highlights the best performance for each mode:

 Size, katoms
Fully GPU-resident
Partially GPU-resident

old
new
old
new

6
936.0
916.4
837.8
781.7

12
796.2
812.1
751.7
709.9

24
616.8
624.4
605.1
597.0

48
456.7
464.0
448.1
460.0

96
287.6
291.7
286.8
292.8

192
160.0
162.0
158.1
159.9

384
80.5
80.8
81.0
81.6

768
41.1
41.2
41.1
41.2

So, the instant submission considerably improves the performance in many cases. As I mentioned, 6k is a pathological case, but it demonstrates an abnormal behavior in the runtime, so it might be worth examining further. As seen in the traces earlier, the submission is still much slower than native HIP.
How soon do you want a go/no-go decision on this PR? I'd rather spend some quality time with profiler here, but I don't think I'll have time for that until the last week of September at least.
BTW, here's where #719 could have helped (no judging, we're stalling on adding a proper support for tracing to GROMACS too).

The table below shows the performance (ns/day) of the fastest configuration among those tested. "Old" runs are with HIPSYCL_RT_MAX_CACHED_NODES=0; bold highlights the best performance for each mode:

Wow, thanks for compiling all this data! This is indeed very interesting.

So, the instant submission considerably improves the performance in many cases. As I mentioned, 6k is a pathological case, but it demonstrates an abnormal behavior in the runtime, so it might be worth examining further. As seen in the traces earlier, the submission is still much slower than native HIP.

I agree, I'd like to know what is going on too.

How soon do you want a go/no-go decision on this PR? I'd rather spend some quality time with profiler here, but I don't think I'll have time for that until the last week of September at least.

I don't see that there is much at stake since the instant submission mode as of this PR is disabled by default in SYCL. I have some interest in it going in for stdpar. So, one option could be for it to go in knowing that it might need further tweaking in the future. This is unless you tell me that the approach is fundamentally not what you want. In that case I'd rather not merge it than rip it out again later :-)
Once the name change is over (at last), we should urgently do a release. Let's say optimistically we do name change within a couple of days, then a couple of weeks for testing/stabilization, release could be within a month.
I think this feature would be interesting enough to be part of the release, if it does not break things for you. I have a use case where it noticably speeds up things.

@illuhad Thanks for the effort, this work looks quite promising!
I have just done some testing with more recent ROCm versions (>=5.5 which are N/A on Dardel/LUMI) and I observe significantly improved runtime performance, so I think we need totest with new ROCm too and use both e.g. 5.3 and 5.6 as our baseline. In particular, efficiency of the HIP/HSA stack improves, hiding runtime inefficiencies e.g. by async submission in a hipSYCL worker thread and +/- caching nodes become less useful. That might suggest that the observed gain of instant submission could increase, but the opposite is possible (and the above data seems to have caught such cases). One reason could be that inefficiencies in the SYCL runtime can become more pronounced and exposed to the application.
To this feature, @al42and's trace where the instant submission is showing much increased time between back-to-back launches is somewhat concerning (along the crash he encountered) so I think it would be best to understand what is happening there.
There can be a complex interaction between the hipSYCL, HIP and HSA runtimes and to make sure that we get broadly applicable real-world benefits in use-cases we want to see improvements for GROMACS, I think we should evaluate this feature on hardware and SW stack we target. We will also need time to assess stability for production and compare to the earlier RT-OPT, all as a function of ROCm version.
Therefore, how we / whether we can allocate time for such testing depends also on the release plans for the next 4-6 months. An imminent new stable release would mean that there will not be much time left for in-depth testing instant submission feature in addition to the release itself.  That's OK if there will be another stable release between now and ~Feb 2024 which can improve on instant submission if needed.

PS: tangentially related data, but here is some data on runtime overheads measured in the GROMACS application thread (i.e. total cost of invoking hipSYCL launches); unfortunately this is with pre RT-OPT I've not had time to re-test this with RT-OPT or instant submission.

HIPSYCL_RT_MAX_CACHED_NODES=100

HIPSYCL_RT_MAX_CACHED_NODES=5

HIPSYCL_RT_MAX_CACHED_NODES=0

wall t/iteration
GPU launch wall t
wall t/iteration
GPU launch wall t
wall t/iteration
GPU launch wall t

rocm 5.3.3
305.03
101.19
317.28
103.69
384.88
158.59

rocm 5.4.3
306.73
74.7
291.29
160.89
383.93
157.84

rocm 5.5.1
299.34
74.75
278.39
159.99
412.08
155.09

rocm 5.6.1
296.34
75.8
276.89
169.74
393.43
155.19

That's interesting: in the "instant" mode, with ROCm 5.3.3, the submission time increases as the kernels get longer. I  suspect that when there is a lot of work in the GPU queue, the HIP API calls might take longer on average, but I don't have fresh traces to prove it (but that API latency likes to randomly spike is established).
That's the same dataset as above, but with more detailed breakdowns (using Szilárd's script). "old" is run with HIPSYCL_RT_MAX_CACHED_NODES=0.

Size, katoms
Fully GPU-resident

old
new

perf, ns/day
wall t/iter, µs
GPU launch, µs
perf, ns/day
wall t/iter, µs
GPU launch, µs

6.0
939.9
184.7
46.6
917.5
222.1
187.0

12.0
792.3
218.3
48.0
804.4
244.3
194.0

24.0
609.7
283.5
48.1
618.8
291.4
208.0

48.0
447.4
386.6
49.6
450.8
382.5
221.6

96.0
281.6
613.7
49.0
284.1
606.6
267.7

192.0
156.9
1101.3
55.0
157.9
1107.1
286.0

384.0
78.6
2198.8
58.2
78.9
2171.6
287.1

768.0
39.7
4357.2
62.3
39.7
4319.9
283.9

Size, katoms
Partially GPU-resident

old
new

perf, ns/day
wall t/iter, µs
GPU launch, µs
perf, ns/day
wall t/iter, µs
GPU launch, µs

6
840.8
205.9
65.7
778.7
188.3
158.1

12
749.3
232.3
66.6
709.0
215.3
170.1

24
603.9
286.4
67.9
593.6
279.6
208.5

48
442.8
390.5
68.2
452.2
383.4
210.4

96
277.5
624.2
64.4
285.0
608.4
210.5

192
153.9
1122.9
64.9
156.1
1094.4
213.5

384
79.3
2179.9
70.6
79.6
2191.8
218.1

768
39.9
4333.6
74.8
40.0
4350.9
218.3

Note: the numbers are median over 5x8 runs with -ntomp 6; aggregation is done sliightly differently, so the application performance is slightly different from the one I posted earlier.

@al42and what would be relevant here is to also include a sensible HIPSYCL_RT_MAX_CACHED_NODES that gives close to best perf with "non-instant" submission.

For the rt_opt version (#1054), the HIPSYCL_RT_MAX_CACHED_NODES=0 is near optimal for the total performance (not for the GPU launch, of course). For a -ntomp 6 run, fully GPU-resident (I don't have -bonded cpu data there):
wall t/iter, µs | cache |   |   |   |   |   |   |
-- | -- | -- | -- | -- | -- | -- | -- | --
size | 0 | 1 | 2 | 5 | 10 | 50 | 100 | 200
6 | 202.93 | 196.54 | 194.99 | 196.1 | 197.71 | 210.15 | 219.54 | 240.99
12 | 224.54 | 227.88 | 227.77 | 226.6 | 229.43 | 239.65 | 259.37 | 272.48
24 | 290.59 | 285.43 | 294.09 | 292.98 | 295.48 | 307.21 | 322.26 | 338.15
48 | 402.48 | 404.03 | 404.7 | 404.31 | 402.87 | 412.14 | 425.31 | 444.03
96 | 632.24 | 628.02 | 637.08 | 632.91 | 633.91 | 647.41 | 665.13 | 677.07
192 | 1127.88 | 1129.99 | 1138.38 | 1135.94 | 1130.38 | 1140.55 | 1151.38 | 1183.27
384 | 2255.04 | 2256.37 | 2259.93 | 2255.15 | 2252.04 | 2264.6 | 2285.1 | 2311.54
768 | 4469.92 | 4468.2 | 4465.53 | 4459.53 | 4451.03 | 4472.31 | 4472.25 | 4502.36
GPU launch, µs | cache |   |   |   |   |   |   |
-- | -- | -- | -- | -- | -- | -- | -- | --
size | 0 | 1 | 2 | 5 | 10 | 50 | 100 | 200
6 | 74.4 | 67.0 | 67.4 | 57.8 | 55.6 | 53.9 | 55.9 | 43.4
12 | 62.9 | 60.8 | 63.6 | 58.0 | 52.3 | 50.8 | 55.0 | 44.0
24 | 71.3 | 65.9 | 61.8 | 51.9 | 57.2 | 51.4 | 51.5 | 41.9
48 | 58.1 | 52.8 | 52.0 | 42.3 | 57.6 | 52.1 | 49.8 | 41.2
96 | 43.3 | 42.5 | 49.9 | 41.6 | 53.9 | 48.5 | 43.8 | 39.7
192 | 40.7 | 41.2 | 47.9 | 45.1 | 34.8 | 39.6 | 48.3 | 38.2
384 | 51.2 | 42.5 | 44.8 | 36.5 | 36.6 | 39.6 | 43.7 | 38.7
768 | 43.8 | 53.0 | 43.8 | 38.5 | 40.4 | 41.0 | 41.2 | 38.6

@al42and Interesting, of course this is near-ideal from the point of view of runtime costs since we leave 2 full cores and hardware threads of the rest of the cores free.
What is clear is that the non-instant submission is effectively able to half to two thirds of the launch overheads exposed to the GROMACS application thread, a cost that becomes relevant when there is either CPU work or MPI communication on the critical path (so perhaps we should be measuring that separately by measuring the runtime overheads that delay CPU work?).
That would explain the slowdowns in cases where the CPU is on the critical path, but not the case where the CPU only does scheduling, so I'd still think there is some added overhead in the hipSYCL runtime, either new with instant submission or newly exposed to the caller application thread, that we should track down.
Do you have any native HIP data from recent benchmarks? Comparing that to the instant submission would help better understanding how much of what is exposed by instant submission is the HIP runtime's overhead and how much of it is added by the SYCL runtime.

PS: something odd I don't get is why does the launch overhead decrease with "partially GPU resident mode" when using instant submission (we are launching more work at least 3-4 extra API calls) but increases with RT_OPT?

Do you have any native HIP data from recent benchmarks? Comparing that to the instant submission would help better understanding how much of what is exposed by instant submission is the HIP runtime's overhead and how much of it is added by the SYCL runtime.

Yes, updated the table above. For small sizes, (walltime - GPU_launch) is approx. the same for "new" and "HIP", so it checks out that the main difference is just the submission cost.

For small sizes, (walltime - GPU_launch) is approx. the same for "new" and "HIP", so it checks out that the main difference is just the submission cost.

OK, that's good to know. For small sizes probably compute kernel differences become small, and the run is API launch-bound. To have a more reliable comparison across the whole range we'd have to make the kernels run at roughly equal speed (ideally close the perf gap ;)

(swapped GPU-resident and CPU-resident)

what's CPU resident, -update cpu?

what's CPU resident, -update cpu?

Yes. I meant "Patially GPU-resident". We need to come up with more convenient names.

Yes. I meant "Patially GPU-resident". We need to come up with more convenient names.

Nit: FWIW these are all "GPU resident".

A quick-and-dirty annotation with rocTX was not particularly enlightening, mostly due to 1µs resolution and, apparently, the fact that the overhead is spread between multiple locations. But perhaps @illuhad can gain some insights?
"Submit GPU Work" markers are right around the queue.submit(...) call, the others correspond to hipSYCL functions:

Interesting...
The gap between the end of hip_queue::submit_kernel and the end of inorder_executor::submit_directly is likely the coarse-grained event construction as really nothing else happens after the kernel submission. Due to SYCL's automatic memory management semantics this has to be a shared_ptr, so my guess is that the memory allocation on the heap is the culprit.
There's also memory allocation before the launch (to construct the DAG node). That might play a role in what is going on prior to submission inside create_task().
The queue will also copy the queue configuration into the handler (which is an std:.vector, which then gets processed and copied into the DAG node). I've wanted to experiment with small vector optimizations for a long time here...
Apart from that there's really not much that is happening.
For most cases this might not play a large role, but if the comparison is direct calls to hipLaunchKernel with trivial kernels, you might see all that.

The gap between the end of hip_queue::submit_kernel and the end of inorder_executor::submit_directly is likely the coarse-grained event construction as really nothing else happens after the kernel submission.

One "tick" along the X axis is 1µs = profiler resolution, so we should be careful when interpreting such single events. But there's clearly a lot happening before the HIP API is called. Here's the full trace:
8390_instant_on.json.gz

For most cases this might not play a large role, but if the comparison is direct calls to hipLaunchKernel with trivial kernels, you might see all that.

It will also be an issue if we have CPU work we want to overlap with the GPU. When we have a lot of cores, having a separate launcher thread would be a good solution, perhaps even better than native HIP. But for cases like LUMI, where we have 8c/8t per GPU, with one core reserved by the scheduler (presumably, for the HSA thread), the CPU compute tasks only have 7 cores, and yielding one of them to the worker thread would be not great; similarly, delaying the compute tasks by having an inefficient same-thread submit is bad, and delaying the communication can be even worse.
So, I agree that most application might not notice a difference. But it's not limited to 1µs-long kernels either.

It will also be an issue if we have CPU work we want to overlap with the GPU. When we have a lot of cores, having a separate launcher thread would be a good solution, perhaps even better than native HIP. But for cases like LUMI, where we have 8c/8t per GPU, with one core reserved by the scheduler (presumably, for the HSA thread), the CPU compute tasks only have 7 cores, and yielding one of them to the worker thread would be not great; similarly, delaying the compute tasks by having an inefficient same-thread submit is bad, and delaying the communication can be even worse.

Agreed. It sounds like we may need two different modes:

Synchronous instant submission, which requires optimizing the submission path (likely including moving to small vector classes instead of std::vector and similar optimizations)
Asynchronous instant submission where a worker thread is involved (Or at least we should investigate the benefit of this - maybe not much is to be gained once the submission path is sufficiently optimized)

Depending on the use case, one or the other might be chosen by the client application and/or the runtime.

The "old" case also sometimes crashes when run with HIPSYCL_RT_MAX_CACHED_NODES=0, but that's besides the point:

That happens even without this PR, so not a blocker here. I'll submit another issues once I get a better idea what happens there. Valgrind points at a long stack of recursive void hipsycl::rt::(anonymous namespace)::descend_requirement_tree calls, and randomness hints at a data race.

That happens even without this PR, so not a blocker here. I'll submit another issues once I get a better idea what happens there. Valgrind points at a long stack of recursive void hipsycl::rt::(anonymous namespace)::descend_requirement_tree calls, and randomness hints at a data race.

Thanks for the confirmation, I had expected this. This PR does not change the non-instant submission code paths at all, so I would have been very surprised if this had changed things.
The randomness could also be due to the DAG descent ignoring nodes that have been marked as completed. So it could also depend on whether an operation has already finished and the GC has noticed this.

I agree. I guess for simple cases we can have some autotuning code path where we make some measurements of the latency to offload to another thread versus just submitting synchronously, but there should also be some mechanism to let expert users enforce a particular behavior.
As I said, in any case I think we need to first examine how much performance we can extract from the synchronous submission code path to check whether it's even worth the effort to offload to another thread.

I haven't analyzed it in depth, but what if, instead of analyzing all the dependencies of a task, we track the possibility of the submission via a flag on the queue?
Task can be instantly submitted iff:

task uses no buffers (USM-only)
task uses no SYCL 2020 reductions
task has no running dependencies on non-instant tasks
task is bound to a device
task is hard-wired to an in-order backend_executor, which happens e.g. when using a SYCL in-order queue.
task has no explicit dependencies (handler::depends_on directly or via shortcuts)
task is submitted to an "instant" queue.

Conditions on a queue being "instant":

An out-of-order queue is always "non-instant";
An in-order queue starts "instant";
An in-order queue becomes "non-instant" if a non-instant task is submitted into it (i.e., the one with buffers, reductions, or dependencies)
An in-order queue becomes "instant" again after queue::wait() is called (there might be some edge-cases here, so perhaps it can never revert back to being instant).

This way, we spare ourselves the need to walk through all the dependencies before submission, hopefully cutting away some overhead.

I suspect that would not help much. For an in-order queue, the number of dependencies is typical 0 to 1 such that there won't be much difference to what you suggest. I really don't think that walking the dependencies itself is what is costing here.

A quick-and-dirty annotation with rocTX was not particularly enlightening, mostly due to 1µs resolution and, apparently, the fact that the overhead is spread between multiple locations.

Perfetto is to blame; the JSONs generated by roctrace have proper nanosecond resolution (note: profiler overhead is unknown): 8390_instant_on.json.gz
Below is not a proper analysis, but an average of three randomly-selected submission events. "Delay at start" is the delay between the start of this function and the start of the function below. "Delay at end" is the delay between the end of the function below and the start of this one. For "submit_kernel", the "function below" is the trio of HIP API calls. Times are measured by manually placed markers (see code linked above). A lot of variability in total submission times come from hipLaunchKernel (3-5 µs), but the overheads are more consistent.

Region
Delay at start
Delay at end

Submit GPU work
150
1112

queue::submit
778
201

queue::execute_submission
1216
137

handler::create_task
1757
93

submit_directly
547
969

submit_kernel
1149
284

So the conclusion still holds, the overhead is annoyingly spread out between multiple places :(

Some things I'd like to try:

Use bitmasks/bitmap in execution_hints for very efficiently querying whether a specific hint is there (very easy)
Use a small vector implementation instead of std::vector for execution hints and requirement_list and dag node dependencies (easy, but a bit more involved)
Spawn dag nodes and operations from object pools. (This is probably a larger change, so might be a project for later)

I'm officially on vacation until Oct 4, so my ability to do this at the moment is limited.

Spawn dag nodes and operations from object pools. (This is probably a larger change, so might be a project for later)

Perhaps using PMR with (un)synchronized_pool_resource would be easier? It should still cut down the overhead of memory allocations significantly without a need to change code much besides just passing the allocator around.

Now depends on #1178. On my system with gfx906, the instant submission test program above is now ~10% faster (it was however always substantially faster than non-instant submission for me so YMMV).
One thing I noticed is that if Gromacs sets the hipSYCL_coarse_grained_property per submission, I would recommend constructing the property list once and then reusing it, as constructing a property_list also requires dynamic memory allocation. This is harder to avoid, because the interface that SYCL has for property_list requires some form of type erasure. Once you have constructed it though, it can be passed to other submissions with little cost.

One thing I noticed is that if Gromacs sets the hipSYCL_coarse_grained_event property per submission, I would recommend constructing the property list once and then reusing it, as constructing a property_list also requires dynamic memory allocation. This is harder to avoid, because the interface that SYCL has for property_list requires some form of type erasure. Once you have constructed it though, it can be passed to other submissions with little cost.

Thanks for the suggestion, we'll look into that.
BTW, do you know any reason there is no constexpr constructor for property_list in the standard? Some properties, of course, might require runtime info, but currently the standard forbids constexpr construction of even empty property list. Here, we can live with static const, so I'm just curious.

BTW, do you know any reason there is no constexpr constructor for property_list in the standard? Some properties, of course, might require runtime info, but currently the standard forbids constexpr construction of even empty property list. Here, we can live with static const, so I'm just curious.

property_list was already there in SYCL 1.2.1, so I suspect it was originally designed before constexpr was a priority.
Performance-wise, an empty property list should not perform any operations either (or at least not more than default-constructing std::vector), but of course there are some additional semantic implications of constexpr.
Intel is working on adding support for compile-time properties to property_list, I assume constexpr support would also come as part of this.

Given that this is opt-in for now, I think we should merge given that for some use cases it will provide a performance-boosting option.


When/how to obtain imageFromCurrentFramebuffer (for use in a CVPixelBuffer)

Hi,
I'm capturing video with a AVCaptureSession utility of my own, that modifies the sample buffer's pixel buffer. I then add it into the GPUImage chain with this CVPixelBuffer GPUImageOutput variant:
https://github.com/YuAo/YUGPUImageCVPixelBufferInput
This works fine. And I'm able to take my capture session video frames through Dlib/OpenCV, mark the camera image with some face-tracking debug drawing, then it goes through GPUImage and I apply an effect filter or two. I also write this out to a GPUImageView so the user can see a preview. So my chain is:
pixelBufferInput.addTarget(effectFilter1)
effectFilter1.addTarget(effectFilter2)
effectFilter2.addTarget(myGpuImageView)
And as my camera frames come in, I have OpenCV/Dlib modify them directly, locking and unlocking the buffer. Then, they are fed to GPUImage with the following, via YUGPUImageCVPixelBufferInput:
pixelBufferInput.processCVPixelBuffer(myCapturedPixelBuffer)
The GPUImageView that's in my view hierarchy seems to filter the captured image in a timely manner, which is great, but:
I don't know how, or exactly when, to obtain a pixel buffer from the filtered GPUImage, and from which target.
I've tried using the imageFromCurrentFramebuffer method on the GPUImageView, and also the second effect filter, but those seem to always be nil at the point I try to access them. I have a method to convert that CGImage/UIImage to a pixel buffer, and I was hoping to then place that back into my original sample buffer.
I've seen many discussions on imageFromCurrentFramebuffer being nil, but they all seem to be around single image capture.
Can you provide any assistance, possibly on when/how I can reliably obtain an image from the current GPUImageOutput?

I'm now trying to use GPUImageRawDataOutput for this, but bytesPerRow is always 0 for me.
weak var output = rawDataOutput!
rawDataOutput?.newFrameAvailableBlock = {
guard let output = output else {
return
}
output.lockFramebufferForReading()
let bytesPerRow = output.bytesPerRowInOutput()
let dataLength = Int(bytesPerRow) * height
let nsData = NSData(bytes: output.rawBytesForImage, length: dataLength)
print("(bytesPerRow), (output.rawBytesForImage)")
output.unlockFramebufferAfterReading()
}
Any idea what I'm doing wrong here?

I realised eventually that if I query bytesPerRowInOutput before rawBytesForImage, then it's 0. If I first do something like:
let bytes = output.rawBytesForImage
let bytesPerRow = output.bytesPerRowInOutput
Both values are valid.


Enable kubectl gs login to workload cluster for non-admins

The command kubectl gs login ... --workload-cluster requires permission to list Organization resources, which are cluster scoped. A customer has been running into an issue with this.
Error: Organizations.security.giantswarm.io is forbidden: User "REDACTED" cannot list resource "organizations" in API group "security.giantswarm.io" at the cluster scope

The solution for non-admins is to add the --organization ORG flag, where ORG is the organization owning the workload cluster.
However we want to provide better usability by making this flag optional or dropping it entirely.
Source: Slack
Tasks

[ ] Verify if there are any additional queries that require cluster-admin permissions.
[ ] Change organization listing to use the SelfSubjectAccessReview method

@vvondruska This is the issue we just talked about.

I have just updated the priority of this issue from high to low. For the customer reporting the problem originally, we found that the root cause for not having admin permissions was a configuration problem, which is fixed now.
The remainder of this story is a general UX improvement for non-admins.

This issue was resolved when organization management logic was added to kubectl-gs.


native: add Ledger contract

But don't change the way we process/store transactions and blocks. Effectively
it's just an interface for smart contracts that replaces old syscalls.
Transaction definition is moved temporarily to runtime package and Block
definition is removed (till we solve #1691 properly).

Otherwise LGTM.


HIVE-27848 - TestCrudCompactorOnTez.secondCompactionShouldBeRefusedBe…

…foreEnqueueing fails

What changes were proposed in this pull request?

Change error message thrown when compaction is not successfully initiated
Why are the changes needed?

Regression of enhancement https://issues.apache.org/jira/browse/HIVE-27598
Does this PR introduce any user-facing change?

No
Is the change a dependency upgrade?

No
How was this patch tested?

Junit tests

@zabetak removed @ignore annotation, please help review & merge

@zabetak could you please help guide me, on the next steps with this pull request

Hey @tarak271 apologies for the delay. I think all the comments are addressed so I have to do a careful review of the refactoring to ensure that nothing broke and we should be good to merge this. I see that there are some merge conflicts in the InitiatorBase so if you can resolve these as well that would be helpful.

@zabetak resolved merge conflicts. Please help review and let me know if any changes are required

@zabetak addressed review comments and build is successful. Please help take a look


Problem Dealing with ScanNet Dataset

Dear friend,
I really appreciate your work and have spent a big chunk of my time testing and optimizing. I have already done some pruning of the network but need to finetune with the ScanNet dataset, where the problems come in.
Firstly, I notice that ScanNet occupies more than 1.2 Tera Bytes of disk, which is a pain to download. Since I am only finetuning the model, not like training from scratch, is it really necessary to download the entire dataset? If not, could you please give me suggestion on which part to download, or provide a slice of the processed ScanNet data (maybe two three gigabytes are good enough) for direct download if possible(like Google drive, Dropbox).
Second, since I see several export option for the ScanNet dataset, i.e. depth image, color image, pose, extrinsic, I would like to know if need to export all? And how do I arrange the data directory so that the training script will work without error? For example, arrange it like data/depth_img, data/color_img, etc. Again, it will be very nice if you can provide a slice of the data directly.
Third, a technical question. In your transformer skip connection (just like ResNet), I notice that you concatenate the input with the transformer attention output before feeding to feed forward network(so the dimension change from 256 channels to 512 channels). In a normal transformer setup, people will just add them instead of concatenation. May I know why you do that? From the optimization perspective, a linear layer of nn.Linear(512, 512) yields much more parameters and FLOPs than a nn.Linear(256,512). I am thinking that if all of the linear layer can be revised to 256 channel input, you may save like 500K parameters.
Final question. Is it possible to finetune(train) with one 1080Ti? I mean as a student, I can live with a longer training time. But I am not sure if the RAM of my GPU is enough. Will it crash in a minute and become totally untrainable?
Sorry for bringing up some many questions. It will definitely help me if you can give me some direction. Thanks.
Richard

Hi,
We really appreciate your efforts!

It makes sense to only use a subset of ScanNet when 'fine-tuning' LoFTR. My suggestion is to randomly sample some sequences from the train-set.
Yes. The data organization is the same as in scannet_test_1500.
We haven't done experiments to compare them. Intuitively, concat + linear might give better results than add + linear.
The major factor behind the number of GPUs is the batchsize. For training from scratch, the minimum recommended value is bs=8. In terms of fine-tuning, I'm afraid one 1080Ti is not enough.


Async PartitionsAssignedHandler

Currently PartitionsAssignedHandler in Consumer is a function Func<List<TopicPartition>, IEnumerable<TopicPartitionOffset>>.
When storing offsets in an external database this is where you'd query the database to get the offsets.
In any modern .NET codebase querying a database is an async operation, but because this handler function type isn't async the developer is forced to do sync-over-async, with the well-documented problems that causes.
I realise that this function is eventually bound to Librdkafka's RebalanceDelegate so it looks like making this async wouldn't be easy?

yes, this is tricky. also, required in order to implement ConsumeAsync completely correctly. i believe it's possible in theory (though I could be missing something). Without looking closely at the details, I think we'd need to make modifications to librdkafka to enable it to pass through Tasks/TasksCompletionSources or whatever is required as appropriate. I doubt we'll get to this anytime soon.
If you use Task.Run to run your consume loop, or your code is guaranteed to be running in many-threaded synchronization context (it probably is), that should pretty much get around any practical problem with sync-over-async here i think. That forces the consume loop to run in the default synchronization context (as well as the handler which runs on the same thread), and in this scenario there's no potential for deadlock due to thread exhaustion.
If there's something specific I'm missing that you're aware of, please let me know.


Support Button State in Precondition

Is your feature request related to a problem? Please describe.
When building multi-purpose devices, the input action or the output value depends on a certain switch position.
At the moment we can define a MobiFlight variable, and we can assign an input action to the switch, so that for a specific switch/button position a specific value is written to the variable.
The variable value can be used in a precondition and by doing so we can assign different input actions for the same input device or define different output values for the same output device.
Describe the solution you'd like
I would like to be able to evaluate a switch state directly in the precondition. This feels more intuitive than going the extra step through a variable.
Describe alternatives you've considered
The existing way.
Additional context
Using button state really only works for discrete state (on/off), e.g. latching switches. We definitely have more flexibility by using a MF variable, e.g. going through different pages by pressing a momentary switch and incrementing the variable, e.g. ($+1)%3

@Koseng this was the issue / idea I was talking about.

I actually believe that the benefit of having this is not so great. The context defined through a Mobiflight variable is a far more flexible and powerful option.


Unknow error for pwd

Hello, I have this error while attempt first connexion
Unknown error for pwd - 'data'; Response - {"errorcode":"no such callback","success":false}
The password is good, i have test a new password without spécial character too.
Archer C7
Thank you for your help

What is version of your router?

It's Archer C7 V4

Do you use router password to login or TP Link ID credentials?

i use my mail account tplink and password, i think it's the only login method on my  router

Router supports local authentication via password and via TP-Link ID cloud.
The integration supports only local router authentication

Ok thank you, i search how to connect to my router with local authentification because same if i turn off internet i can't have a local authentification to the web ui interface ....

Ok to have an local connexion, i have unlink my tplink cloud account but i lost my tplinkdns :(
I had test to connect the integration after but the error is the same, strange ...
So i reconnect my tplink cloud account, i 'll do without this HA integration but thank you anyway.


Update Makefile

basic_game - Removed glew from linker flags because on windows this fails to compile with the installed raylib package.

Hi Triangle345! Thanks for the fix! Could you push it to develop branch, it's the branch with latest changes. :)

sure!


ga_faster_x101 training NaN

When I try to train the ga_faster_x101, the loss has become very large, sometimes NaN. I only have one  1080Ti. So I have tried a lot of lr, and also tried to modify the warmup='exp',warmup_iters=1000, warmup_ratio=1.0/10. But the same result happened. (other detection network also like it).
Can you tell me why please? Thank you very much!

In my opinion, the small batch size might be the cause.

You may first search for a lr (e.g. 0.02/8 = 0.0025) which can be successfully adopted to the original Faster R-CNN with X-101 and FPN backbone. And then tries to train GA-Faster R-CNN with such a lr.

@chenzui1113  I have sloved the problem，by reducing the learn_rate，such as:
 retinanet_r50_fpn_1x_coco.py
_base_ = [
    '../_base_/models/retinanet_r50_fpn.py',
    '../_base_/datasets/voc0712.py',
    '../_base_/schedules/schedule_1x.py', '../_base_/default_runtime.py'
]
# optimizer
optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001)

# lr default is 0.01, I changed it to 0.001
 retinanet_r50_fpn_1x_coco.py
_base_ = [
    '../_base_/models/retinanet_r50_fpn.py',
    '../_base_/datasets/voc0712.py',
    '../_base_/schedules/schedule_1x.py', '../_base_/default_runtime.py'
]
# optimizer
optimizer = dict(type='SGD', lr=0.001, momentum=0.9, weight_decay=0.0001)


Warning: Duplicate HttpCacheWarmUpCommand

Not that big of an issue in src/Storefront/Framework/Command/
Warning: Ambiguous class resolution, "Shopware\Storefront\Framework\Command\HttpCacheWarmUpCommand" was found in both "/home/vagrant/shopware-dev/vendor/shopware/platform/src/Storefront/Framework/Command/HttpCacheWarmupCommand.php" and "/home/vagrant/shopware-dev/vendor/shopware/platform/src/Storefront/Framework/Command/HttpCacheWarmUpCommand.php", the first will be used.

Hey @tanmarpaul,
thanks for raising this issue, we've found the underlying problem and will fix it.
Kind regards
Jonas Elfering


Not a valid path value: 'null'

Description of the bug
Unable to complete any workflow except the "test" one.
Command used and terminal output
nextflow run nf-core/circrna -profile test_full,docker -r dev

Error executing process > 'NFCORE_CIRCRNA:CIRCRNA:CIRCRNA_DISCOVERY:CIRCEXPLORER2_REFERENCE (null)'

Caused by:  Not a valid path value: 'null'

Relevant files
nextflow.log
System information

N E X T F L O W  ~  version 22.10.4
Desktop
local
docker
Ubuntu 22 LTS
nf-core/circrna v1.0.0-g51abcba

This is because the iGenomes config is not up-to-date. The genome configuration which is used in the test_full example is not containing a gtf file for the configured genome R64-1-1. However, the version of the iGenomes configuration that is used in the rnaSeq-Pipeline contains the necessary files. I will update the iGenomes file and create a PR.


[BUG] Specification either not consistent or contradictory

Bug Description
Specification item PVL-30 contradicts with MEM-70

please elaborate why it is contradicting. one requirement is regarding the core interface, one regarding the coreplex interface

These two requirements has to be considered together as a common CVE32E20 specification, as they are tightly related to each other. They do not exist in their own vacuum, on the contrary: forcing one encoding different from the other is obviously resulting in additional logic that would not be needed if the implementation was not (explicitly) specified, but rather requested functionality.
I see a solution as to either:

relax encoding specification of the core interface (not explicit encoding), or

harmonize both interfaces encodings.


#94: #17  broke some functionalities (closes #94)

Issue #94
Test Plan
tests performed

issues page should activate extension issuesPageRegex

issue page should activate extension with issuePageRegex

new-issue page should activate extension with newIssuePageRegex

tests not performed (domain coverage)

At times not everything can be tested, and writing what hasn't been tested is just as important as writing what has been tested.

An example of partial test is a field displaying 4 possible values. If 3 values are tested, with screenshots, and 1 is not, then it should be mentioned here.}

🌵


Added a linter and formatter to the Github Actions workflow

@VT-AOE-DMAD-Drew-Bennett Just made a few small changes.

Updated the action to run on pull request and push to develop as well so we can catch errors earlier.
Added some arguments so black would show us what the difference in how it would format in the GH actions log.
Minor change to make sure our version of Python is the same across our CI/CD.

Black is intentionally uncompromising in its formatting so we could also skip the formatting or make it trigger just on a push to main so we don't deal with the formatter during development. We're never going to have the same level of formatting by hand as Black does, so I think our choice is really to either let Black autoformat on push to main or not use it.

@VT-AOE-DMAD-Drew-Bennett Just made a few small changes.

Updated the action to run on pull request and push to develop as well so we can catch errors earlier.
Added some arguments so black would show us what the difference in how it would format in the GH actions log.
Minor change to make sure our version of Python is the same across our CI/CD.

Black is intentionally uncompromising in its formatting so we could also skip the formatting or make it trigger just on a push to main so we don't deal with the formatter during development. We're never going to have the same level of formatting by hand as Black does, so I think our choice is really to either let Black autoformat on push to main or not use it.

Sounds good. I might comment out the yml and push so we can merge the two branches together once we figure out #65.

@VT-AOE-DMAD-Drew-Bennett Just made a few small changes.

Updated the action to run on pull request and push to develop as well so we can catch errors earlier.
Added some arguments so black would show us what the difference in how it would format in the GH actions log.
Minor change to make sure our version of Python is the same across our CI/CD.

Black is intentionally uncompromising in its formatting so we could also skip the formatting or make it trigger just on a push to main so we don't deal with the formatter during development. We're never going to have the same level of formatting by hand as Black does, so I think our choice is really to either let Black autoformat on push to main or not use it.

Sounds good. I might comment out the yml and push so we can merge the two branches together once we figure out #65.

Added a line so black only runs on a push to main so we should be good now.


feat(bench): add ec criterion benches for bls12_377 and bls12_381

TITLE
Description
Expand ec criterion benches.
Type of change

[x] New feature

Checklist

[x] Benchmarks added/run

Codecov Report

Merging #677 (7a563d2) into main (24c84ab) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #677   +/-   ##
=======================================
  Coverage   96.78%   96.78%
=======================================
  Files         124      124
  Lines       28876    28876
=======================================
  Hits        27949    27949
  Misses        927      927

:mega: Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in Chrome or Firefox today!

There are a lot of #[allow(dead_code)], they shouldn't be needed in public functions

@MauroToscano removed. clippy flagged the function headers for the individual *_elliptic_curve_benchmarks so I did not remove them.


[DATA] Update data preprocessing

PR types

New features
PR changes

APIs
Description

support megatron dataset.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Pip build includes test package

The pip package bundles the test package, which pollutes the global namespace.
In particular, the AUR package python-pylsp-rope occupies /usr/lib/python3.10/site-packages/test.

https://github.com/python-rope/pylsp-rope/blob/431415560779881b57048dc563802705f7556bca/setup.cfg#L23
According to the setuptools doc, adding
[options.packages.find]
exclude =
    tests*

should do the trick.

Hi @lukasjuhrich, thanks for taking the time to report and investigate this issue. I've updated the package on pylsp-rope==0.1.10 on PyPI which should fix this issue. It may take some time for the update to trickle to AUR and other distro repositories.
cc: @bageljrkhanofemus

Done.

Thanks @bageljrkhanofemus

Alright, thanks to both of you for the quick reaction!


release beta on master

i first want to test it in some apps to see if it contains a breaking change.
it fixes compatibility with prisma3 in typegrahql

:tada: This PR is included in version @ra-data-prisma/backend-v7.2.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version @ra-data-prisma/dataprovider-v6.7.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Org-capture

Hi, and congrats on your package.
I haven't used it yet, but I was wondering what the difference would be between org-journal and org-capture set to capture journal entries in a tree-fashion way. Thank you.

 It includes functions for encryption, searching, scheduling things in the future, and carrying TODO items with you until they are done.
Org-capture is a way of saving transient information into one or many org-mode files.
I don't know what more to say. You seem to imply that they are "competitors" in some way. But they are not. They are simply different things. Read the documentation on both, or try them yourself, to learn more.

Thanks, I ask because many users have implemented—as I understand—what org-journal does by using org-capture: you can get an subtree of dates and time with entries as long as in org-journal. That is why I ask.

BTW, I would disagree with "Org-capture is a way of saving transient information into one or many org-mode files.". How so "transient"? I very well keep muy diary using org-capture, and I can tell it's permanent.

Do you have an issue with org-journal?
This is not a forum for discussions about your preferred Emacs tools, sorry.

No, I'm not stating my preferred emacs tool, I am just asking what is superior in org-journal that org-capture cannot set to do as a way to avoid adding packages onto my init.e, thus slowing it down.


thanks

this fix is working for me
just wanted to say thanks

Thanks for letting me know, glad to hear that.


Send multiple messages with .send and messageRoom with one async function but still miss order.

I use async function and  await messageRoom.But I still can't correct my messages' order. Is there any way to solve this?
async function sendmultiple (robot, msg,msg2,msg3) {
    const roomID = roomSession.roomID;
    await robot.messageRoom(roomID , msg);
    await robot.messageRoom(roomID , msg2);
    await robot.messageRoom(roomID , msg3);
}

hi @PonnyZuo AFAIK the mattermost server does not guarantee order of messages if they are send in close proximity of each other


Fix streaming doesn't return finish reason

When stream == True the branch with the yield statement that contains the finish_reason isn't visited when remaining_tokens is empty.
Moved the yield statement containing finish_reason after the loop over remaining_tokens. This branch is always visited when stream == True.
Fixes #735

@gmcgoldr sorry for the delay on this, got swamped with work recently. It looks good but I need to re-enable the completion tests before I merge this. Should be able to do that soon.


Failed to set up

Describe the bug
Plugin says faiel to setup
Expected behavior
Screenshots
Environment details:

Environment HASSIO on a NUC
Core   2024.1.6
Supervisor   2024.01.1
Operating System   11.5
Frontend   20240104.0

Output of HA logs
Logger: homeassistant.config_entries
Source: config_entries.py:406
First occurred: 6 February 2024 at 09:58:39 (2 occurrences)
Last logged: 15:46:12
Error setting up entry LGE Devices for smartthinq_sensors
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/config_entries.py", line 406, in async_setup
result = await component.async_setup_entry(hass, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/smartthinq_sensors/init.py", line 215, in async_setup_entry
await cleanup_orphan_lge_devices(hass, config_entry.entry_id, client)
File "/config/custom_components/smartthinq_sensors/init.py", line 596, in cleanup_orphan_lge_devices
device_registry = await hass.helpers.device_registry.async_get_registry()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/loader.py", line 1075, in getattr
value = getattr(self._module, attr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/deprecation.py", line 258, in check_if_deprecated_constant
raise AttributeError(f"Module {module_name!r} has no attribute {name!r}")
AttributeError: Module 'homeassistant.helpers.device_registry' has no attribute 'async_get_registry'

Can you give detail about integration version and how you install? Looking the code seems that you are using an old version


Which is most reliable for locating a city

Several questions/issues.
I'm looking for the most reliable way to map an IP address to a city and I think the following is inconsistent.
(Using the GeoIP2 free version for cities)
Say you have let "<IP_ADDRESS>" as the ip. For this IP, maxminddb gives a city value of Santa Rosa, CA. This is off mark and its north of where I actually am, San Francisco. The zipcode produced, 95408, is also off mark and is about as far south of me as Santa Rosa is north of me.
So the zipcode and city are not agreeing. Is this because I'm using the free version? And even so, one would think the two would agree, zip code and city.
Also, which one is more reliable? My goal is to get the closest "big" city to the IP, I want to the functionality that websites like yelp offer when you first come to their site.
Also, I gave an IP that was from Russia but I got a postal code that looked like an American Code. Are postal codes only reliable for US based IPs?
(Coding note, I don't do too much C but I do love ocaml...which leads me to ask why aren't all the MMDB_DATA_TYPE_* just an enum rather than #defines. I want exhaustiveness from switch statements.)

I am not sure offhand why the zip code and city do not agree in that case. Our precision database has San Francisco, CA, 94103 for that IP. There may be a data quality issue with our source data causing the conflicting information. Generally, the postal code will be more specific than the city, but it should be in the appropriate city. You may have better luck contacting<EMAIL_ADDRESS>You can also submit a correction.
I am also not sure why macro definitions were used instead of an enum. We do use enums for the APIs in some other languages.


Logfile Management using Cumulocity

Is your feature request related to a problem? Please describe.
Currently thin-edge.io handles only its own log files. However Cumulocity could request all log files from a device.
It would be good if I could manage from Cumulocity side which log files I want to request and get them.
Describe the solution you'd like
Cumulocity already provides the supportedLogs Fragment for that. However thin-edge.io needs to push that to the inventory object, e.g. read out from an .toml file. Idealy this .toml would be managed by cumulocity (e.g. configuration files).

A good idea to be implemented along the line of the current work on configuration management.

Proposal

Update the c8y_configuration_plugin daemon to support c8y_LogfileRequest operations.
Use the same configuration file to list the paths both to the config and log files.
Both log and config files can be given a name (per default, the file path).
Remove the current log_request_plugin plugin.

Alternatives

Maintain distinct daemons for configuration and log requests, trading simplicity for flexibility.
Have two distinct configuration files to list the config and the log files.

The longer I think of that I personally prefer the alternative solution such that I do have a dedicated daemon for log files and one for configurations. In principle these two plugins are separated functions.
However a distinct configuration file could be used to list the log files.

@didier-wenzek

Use the same configuration file to list the paths both to the config and log files.

Assuming same configuration file means to use c8y_configuration_plugin.toml, what would it look like? At the moment it is something like this:

Would it look something like this?

files = [
    '/etc/tedge/tedge.toml',
    '/etc/tedge/mosquitto-conf/c8y-bridge.conf',
    '/etc/tedge/mosquitto-conf/tedge-mosquitto.conf',
    '/etc/mosquitto/mosquitto.conf',
    '/etc/var/log/tedge',
    '/etc/var/log/mosquitto',
  ]

Or something like this?
files = [
    '/etc/tedge/tedge.toml',
    '/etc/tedge/mosquitto-conf/c8y-bridge.conf',
    '/etc/tedge/mosquitto-conf/tedge-mosquitto.conf',
    '/etc/mosquitto/mosquitto.conf',
  ]

log_files = [
   '/etc/var/log/tedge',
   '/etc/var/log/mosquitto',
]

@didier-wenzek just an obervation.
Depending on directory permissions we may or may not require sudo to read logs.
On my machine, I can not read /etc/var/log/mosquitto/mosquitto.log unless I am sudo or mosquitto user.
But I can read /etc/var/apt/history.log as any user.

@initard

Assuming same configuration file means to use c8y_configuration_plugin.toml, what would it look like?

If we decide to keep a single configuration file, then we need to have two independent lists, a first one for the configuration files, a second ones for the log files.

The purpose of the new log file management is

To deliver configurable log file management plugin.
To add the configuration file to the existing log file management plugin so that users can choose which log files to monitor on c8y.

Solutions:

The log file plugin is independent from the config management plugin.
Change the log file plugin to daemon.
The plugin declares the supported log types to c8y. Currently, the c8y mapper declares them. At the daemon start-up and inotify. Use the config management plugin as a blue-print.
The log file plugin config file is separated from the config-management plugin config file.
The config file should be the list of glob patterns with names (key as log file type, value as actual glob file patterns), not just the list of paths.

# example to explain the second bullet point.
log_files = [
   {key = "tedge-log", value = "/var/log/tedge/*"},
]

Out of scope:

Open up tedge.toml file to keep also plugin config files.

How to work:

[ ] Write a short specification for the new log file management plugin like the one for config management.
[ ] Show it in the sprint demo.
[ ] Implementation.


Not responding after resume from suspend

Hi, thank you for this beautiful Neovim client!
I'm having a weird issue where Neovide does not respond anymore after my laptop resumes from sleep. I'm happy to help debugging, but I don't really know where to start.
Some basic info:
Neovide version: 0.6.0.r32.g0203d02
uname -or: 5.4.72-1-MANJARO GNU/Linux

Hmmm. I'm not sure what would be causing this. Is it consistent every time you resume from sleep? This seems like an upstream issue in Skulpin

Yup 100% consistent. When you responded I thought maybe the problem is that I prevent neovide from blocking compositing with a KWin rule, but the only difference is that the window is completely black instead of showing the contents from before suspend, so I don't think it had anything to do with it.
Anyway, I'll create an issue upstream.

I think it may be smart for us to create a minimum viable repro of the issue. For instance I'd be curious if the demos in skulpin exhibit the same issue when using sdl2 window

Cause if the answer is no, it may be something we are doing strangely on our side

Good idea! I just tried it and it happens with on of their examples too. (see my comment in the skulpin issue)

I think this has been fixed in main. Please reopen if you come back to this and the issue remains.

I checked and it's fixed! How did you do it?

We swapped to winit, reworked the renderer to use opengl, and reworked all of the window event management.

Wow, sounds like a lot of work. Thank you!


Added "Name" members filter

refs https://github.com/TryGhost/Team/issues/1408

switched to @tryghost/nql packages to get access to latest releases
updated GET /members mirage endpoint with a try/catch and explicit logging to make any errors from NQL more visible
added "Name" filter option

has is, contains, does not contain, starts with, ends with operators
uses a plain text field for the input value

added support for ~, -~, ~^, and ~$ operators when generating NQL queries from filter definitions

TODO:

[x] resolve NQL questions

[x] requirement for contains queries to use literals
[x] escaping of single quotes in strings

[x] add parsing of contains operator statements in query param to filter state

Codecov Report

Merging #2289 (84f6c96) into main (3f4c7af) will decrease coverage by 0.07%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #2289      +/-   ##
==========================================
- Coverage   40.81%   40.73%   -0.08%
==========================================
  Files         578      578
  Lines       22649    22668      +19
  Branches     4007     4013       +6
==========================================
- Hits         9244     9234      -10
- Misses      13405    13434      +29

Impacted Files
Coverage Δ

app/components/members/filter.js
85.71% <100.00%> (+1.42%)
:arrow_up:

app/components/gh-html-iframe.js
45.45% <0.00%> (-38.64%)
:arrow_down:

app/helpers/gh-price-amount.js
70.00% <0.00%> (-20.00%)
:arrow_down:

app/components/gh-site-iframe.js
46.00% <0.00%> (-4.00%)
:arrow_down:

app/components/gh-file-uploader.js
91.53% <0.00%> (-3.08%)
:arrow_down:

app/components/gh-image-uploader.js
94.00% <0.00%> (-1.34%)
:arrow_down:

app/controllers/offer.js
44.34% <0.00%> (-0.91%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3f4c7af...84f6c96. Read the comment docs.


Quick Question

How long will it take that my new merged LogoURL will show up in Solscan?

https://github.com/solana-labs/token-list#scannerwallet-hasnt-updated-yet


Don't generate a transaction for a details_for call

@pierre @daliwei

Could you add a test case to verify https://github.com/killbill/killbill-paypal-express-plugin/issues/26 is indeed fixed with this patch?

For the testing, you need to verify @plugin.get_payment_info returns a single payment PaymentTransactionInfoPlugin per payment transaction (for a typical auth/capture, only two should be returned).
What needs to be checked:

Plugin status is set correctly on a per transaction basis (e.g. ERROR for failed transactions)
Plugin properties returned are the ones from the actual payment API call, not the details_for one. You expressed interest in merging the two, i.e. augmenting the properties from the payment API call with the details_for ones to return additional information like payerEmail. This is absolutely fine but for conflicting ones (message, success, etc.), the ones from the payment API call take precedence.
Verify that the PaymentTransactionInfoPlugin returned by the initial API call (e.g. authorizePayment) and the one from getPaymentInfo are consistent (it isn't anymore in case of failure, which can confuse Kill Bill for PENDING and UNKNOWN).
I don't see many negative tests (insufficient funds, account closed, etc.) besides a double capture. It would be worth adding a new negative scenario (e.g. failed auth) since this issue mainly manifests itself for failed payments.

Also, all existing tests need to be updated, as they will probably fail with your patch (see https://github.com/killbill/killbill-paypal-express-plugin/blob/66c7d2bc3c4ed6c4efa2a9eed57aba959c814a93/spec/paypal_express/remote/hpp_spec_helpers.rb#L58 for example).
@daliwei has credentials to run the full suite (bundle exec rake test:remote:spec) if you don't have any.

@pierre updated and tested this PR

👍


depricarting "YoutubeDataApi" in favor of "YouTubeDataAPI"

As of v.0.0.21, we're depricating the class "YoutubeDataApi" in favor of "YouTubeDataAPI". These classes are functionally equivalent!

Note that the About of this repository seems affected:

A Python Client for collect and parse public data from the Youtube Data API

Source: https://github.com/SMAPPNYU/youtube-data-api


Realm accessed from incorrect thread

Hello and thanks everyone for this wonderful ORM!
From time to time I face this problem:
*** Terminating app due to uncaught exception 'RLMException', reason: 'Realm accessed from incorrect thread.'

Yes, I know that Realm-object needs to be instantiate through "defaultRealm" for every thread.
So, It's really hard to reproduce this bug, but if I leave my app running for a long time (f.e. 2 hours) I almost always get it.
The app crashes here:

Code example (Following method runs in background serial queue (gcd)):
+ (void)parseNewObjects:(<NSString*>NSArray *)objectNames //objects - array with strings
{
    RLMRealm *realm = [RLMRealm defaultRealm]; //here I get current instance of Realm

    NSMutableArray *parsedObjects = @[].mutableCopy;

    [objectNames enumerateObjectsUsingBlock:^(NSString *objectName, NSUInteger idx, BOOL *_Nonnull stop) {
        MyCustomObject *object = [MyCustomObject new];

        object.oName = objectName; //here I set object's name

        [parsedObjects addObject:object]; //here I push named object to mutable array
    }];

    [realm beginWriteTransaction]; // here I get the crash =(
    [realm addOrUpdateObjectsFromArray:parsedObjects];
    [realm commitWriteTransaction];
}

I already checked the app for memory and disk space leaks and found nothing serious. So, this crash happens in absolutely trivial situation:

Last days' crashes:

version of realm: 1.1.0
versions of ios: 8-9-10

I wonder if it's possible that the enumeration block is occasionally being called on a queue different from that which parseNewObjects: is running on. Can you try getting rid of the enumerateObjectsUsingBlock: call and using a for-loop instead, and seeing if you can reproduce the crash?

@austinzheng as you see, enumeration block does nothing with realm and crash happens in beginWriteTransaction method.
I try to avoid this crash right now, cause I think it may be my architecture fault.
So, new version of my app and crashlytics will show the results of this changes.

Okay, great. Let us know if the new version of your app fixes the problem, or if you need us to help you further!

The problem on my side, I think - I have some OOM's in my App, so it happens when user does not have enough memory.
Sorry for worrying.

No problem! Thanks for getting back to us, good luck fixing your memory issues, and feel free to open another ticket if you run into any Realm-related issues!


Compile in Delphi 11 and 10.4 fails

In both, get the following error
**Build FAILED.
"C:\Users\Neil\Documents\Skia4Delphi\Packages\RAD Studio 11.0 Alexandria\Skia.Package.RTL.dproj" (default target) (1) ->
C:\Users\Neil\AppData\Roaming\Embarcadero\BDS\22.0\environment.proj(7,9): error MSB5016: The name "asl.log" contains an invalid character ".". [C:\Users\Neil\Documents\Skia4Delphi\Packages\RAD Studio 11.0 Alexandria\Skia.Package.RTL.dproj]**
Did try running install program as administrator with same result

Duplicate of #39
Hi @nszmnsky, thank you to report the problem. This issue has already been reported. Today we will release a fix, but for now you can do this workaround: https://github.com/skia4delphi/skia4delphi/issues/39#issuecomment-1039378582