Cassandra Read Performance

Does the size of data list affect the performance in graphite ?
I have the following rollup:
"10s:24h"
"1m:30d"
"15m:1year"
Now lets say, if a datapoint comes every second to cyanite, then in 24 hours, the following rows will be created:
For "10s:24h" rollup
(246060/10 = 8640) rows will be created each row having 10 elements in the data list
For "1m:30d" rollup
(24*60 = 1440) rows will be created each row having 60 elements in the data list
for "15m:1year" rollup
(246060/900 = 96 rows will be created with each row having 900 elements in the data list.
Is the above information correct. If yes then lets extrapolate this to store 90 days of data. Now the issue is that the time taken to query cyanite (using Grafana) to get 90 days of data for a given metric is more than the time required to query 1 day of data for the given metric.
If you notice the # of rows returned by cassandra for both the query will be 8640. The only difference is the # of elements in each row, which will be more for 90 days data.
Also if I run a performance test with 10 users accessing 90 days of data simultaneously, then I start seeing cassandra timeouts in the backend, and it spends a lot of time in Garbage collection, but the same does not happen when I do the same by querying 1 day of data.
So I am concluding that the number of elements in the data list is affecting performance. Is there a way to limit that, or to avoid storing every element in each rollup. Can we not aggregate it before storing ?
Are you seeing similar performance issues in your cyanite/cassandra stack....?

Can we not modify cyanite to store averages of the data points in a row, instead of storing every data points in every retention period ? It is making the read on cassandra slow, because of frequent GC because of large size of the data list.

@vsonone this is now the case in #80, I'd love help with testing!

i am not sure this strategy will work.  specifically because when cyanite is run in a clustered environment, no single cyanite instance will know about all the input data. therefore the averages each instance stores should be weighted by the amount of traffic it receives. unfortunately, what i suspect will actually happen is that the average from each instance will be weighted equally when producing a final value to display in graphite since the correct weights are not stored in the record.
on my project we have a background task to pre-aggregate longer retention periods after cyanite is done writing to them.  generally we wait for either a later bucket to appear or a generous grace period to elapse (e.g. 5 minutes) before assuming that a bucket is complete and cyanite (carbon-c-relay) will not be writing to it any longer.
so, for example, if you were to receive the following data points in four consecutive minutes: [1,2,3,4], then the mean should be 2.5.  unfortunately, if [1,2,3] goes to one instance and [4] goes to another, then the averages stored in the db will be 2 and 4. so when cyanite goes to read the data back for graphite, it will return an average of 3, which is incorrect.
perhaps i am missing something?

Hi @tjamesturner. This is true and mentioned in the updated docs at http://cyanite.io.
However, it is trivial to ensure that points for a specific metric are routed to a single cyanite host.
I will also provide to alternative ways to make this even easier:

A proxy inside cyanite itself, provided its configuration contains the topology
Queue based inputs (Kafka & RabbitMQ)


Try Disallow: /?, as used in orangelight

fixes #811

Coverage remained the same at 100.0% when pulling 1aaa2a046e5b8d644297aa7cb5e5413749f5342a on 811-disallow-more into b24410464cadc4923979b0f7608f843aeeadf7c1 on master.


Explain what exception chains are

In reading and following the evolution of our function checks lesson, it's clear to me that some of what we need to do is explain exception chaining at a very high level and create a visual for students to understand.
my experience teaching Python has been that students are totally overwhelmed by the volume of output that Python provides. And I think helping them understand what's happening at a high level could ease some of that anxiety and make the concepts around try/except blocks easier to absorb. This relates to the recently merged #72 by @sneakers-the-rat

Completely fair. I was trying to avoid a full explanation in that lesson because I figured we have limited time in the workshops and wouldn't want to spend it all on exception handling, and just gave a link to further docs for people who wanted to read more on their own time. Following your lead focusing more on "how do i make this do what i want it to" than understanding the system design in complete detail. But yes! ideally these lessons would be self-sufficient after the festival, so then adding in details would be great.
Might be better in an "advanced" lesson, because I think all the permutations of exception handling and chaining really only come into play when you're designing more complex systems or systems intended for downstream use where you would want to be specific about the types of errors you raise so someone can catch them. Otherwise catch -> handle; catch -> re-raise (eg. with added note) handle most common needs? I think what would be nice to have here is like a dedicated lesson that's just exception handling patterns with not exactly a "pro/con" list but a discussion on when you would use those patterns and when you wouldn't.
I think as far as examples go, data validation might be a good one? like i'm thinking of what i'm doing right now in numpydantic, and i need to catch a whole range of exceptions and transform them into a hierarchy of exceptions that then are chained beneath a pydantic ValidationError - so we both get feedback about what caused the error, but enclosing code is able to catch different kinds of errors in a predictable way rather than needing to catch all the kinds of errors that could happen. pydantic is also a case where you need to be careful about the kinds of errors that you raise because the framework you're in expects a specific error type, so it might be a decent, familiar example that covers a range of patterns on the raising and handling side with good motivations for chaining/not chaining/etc.

This sounds great @sneakers-the-rat Yes, let's wait until after the workshop. We have a fairly short period of time to cover this stuff during.
And i think our main goal for the workshop is really to raise awareness of adding logic and checks to code and how powerful it can be for users and developers / scientists processing data!
I like the idea of adding this content to our lessons after!!

Python 3.13 should improve much of the error understanding: https://docs.python.org/3/whatsnew/3.13.html#improved-error-messages


Allow exporting to more outputs

Whilst templating works well for single files to be exported, they are not a good fit for exporting to file formats that have lots of other information in the same file. Update stylesync to allow exporting to:

[ ] Sketch
[ ] Figma
[ ] InVision DSM
[ ] Xcode 9 assets catalogue
[ ] Lona

When combined with #23, this should allow much more flexibility in the use of stylesync.

For Xcode 9 assets catalogue, a convenience file should also be exported, e.g.:
extension UIColor {
	public static let testColor = UIColor(named: "TestColor")!
}

This will allow you to reference the same colour in code as you do in IB. This will ned to be in the same location as the assets catalogue.


unable to compose ostree-based disk images from ostree repo with multiple commits

Describe the bug
When composing an ostree-based disk image (i.e. edge-raw-image, edge-installer) from an ostree repo that has multiple commits, the compose fails.
Environment

OS version (/etc/os-release and /etc/redhat-release):

$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="9.2 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.2"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.2 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.2
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.2"

osbuild-composer version (rpm -qi osbuild-composer)

$ rpm -qi osbuild-composer
Name        : osbuild-composer
Version     : 76
Release     : 2.el9_2.2
Architecture: x86_64
Install Date: Tue Aug 22 14:48:52 2023
Group       : Unspecified
Size        : 17587
License     : Apache-2.0
Signature   : RSA/SHA256, Wed Apr 26 06:04:11 2023, Key ID 199e2f91fd431d51
Source RPM  : osbuild-composer-76-2.el9_2.2.src.rpm
Build Date  : Wed Apr 26 04:29:29 2023
Build Host  : x86-64-02.build.eng.rdu2.redhat.com
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : https://github.com/osbuild/osbuild-composer
Summary     : An image building service based on osbuild
Description :

A service for building customized OS artifacts, such as VM images and OSTree
commits, that uses osbuild under the hood. Besides building images for local
usage, it can also upload images directly to cloud.

It is compatible with composer-cli and cockpit-composer clients.

To Reproduce

Create empty blueprint in order to build edge-commit
Start edge-commit compose:  composer-cli compose start-ostree <blueprin> edge-commit
Grab tarball and explode to disk
Copy repo directory to /usr/share/nginx/html (or similar) to be served via HTTP: cp -r ./repo /usr/share/nginx/html/
Modify existing blueprint to have additional package, in order to build new ostree commit
Start new edge-commit compose
Grab new tarball, use ostree pull-local to copy new commit to existing ostree repo:  ostree --repo=/usr/share/nginx/html/repo pull-local ./repo
Create empty blueprint to compose disk image
Start edge-raw-image/edge-installer compose: composer-cli compose start-ostree --ref rhel/9/x86_64/edge --url http://<IP_ADDRESS>/repo <blueprint> edge-installer

Expected behavior
Disk image compose is successful
Additional context
The error in the journal looks like it is failing on an internal use of ostree pull-local trying to pull data into a temporary ostree repo:
Aug 22 16:17:08 localhost.localdomain osbuild-composer[5105]: 2023/08/22 16:17:08 POST /api/v1/compose
Aug 22 16:17:15 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:17:15-04:00" level=info msg="Running job 'b5b21628-80c5-4670-9027-422b37d4df1b' (osbuild)\n"
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]: Traceback (most recent call last):
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:   File "/usr/bin/osbuild", line 33, in <module>
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:     sys.exit(load_entry_point('osbuild==81', 'console_scripts', 'osbuild')())
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:   File "/usr/lib/python3.9/site-packages/osbuild/main_cli.py", line 165, in osbuild_cli
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:     r = manifest.build(
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:   File "/usr/lib/python3.9/site-packages/osbuild/pipeline.py", line 467, in build
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:     res = pl.run(store, monitor, libdir, stage_timeout)
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:   File "/usr/lib/python3.9/site-packages/osbuild/pipeline.py", line 371, in run
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:     results = self.build_stages(store,
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:   File "/usr/lib/python3.9/site-packages/osbuild/pipeline.py", line 344, in build_stages
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:     r = stage.run(tree,
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:   File "/usr/lib/python3.9/site-packages/osbuild/pipeline.py", line 212, in run
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:     data = ipmgr.map(ip)
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:   File "/usr/lib/python3.9/site-packages/osbuild/inputs.py", line 91, in map
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:     reply = client.call("map", args)
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:   File "/usr/lib/python3.9/site-packages/osbuild/host.py", line 348, in call
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:     ret, _ = self.call_with_fds(method, args)
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:   File "/usr/lib/python3.9/site-packages/osbuild/host.py", line 384, in call_with_fds
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:     raise error
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]: osbuild.host.RemoteError: CalledProcessError: Command '['ostree', 'pull-local', '/var/cache/osbuild-worker/osbuild-store/sources/org.osbuild.ostree/repo', 'd289203132531644640301eb930549e3f19aae2dc0067908b2acb65b2f41e21e', '--repo=/var/cache/osbuild-worker/
osbuild-store/tmp/buildroot-tmp-mtwe1add/inputs/commits/repo']' returned non-zero exit status 1.
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:    File "/usr/lib/python3.9/site-packages/osbuild/host.py", line 268, in serve
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:     reply, reply_fds = self._handle_message(msg, fds)
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:   File "/usr/lib/python3.9/site-packages/osbuild/host.py", line 301, in _handle_message
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:     ret, fds = self.dispatch(name, args, fds)
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:   File "/usr/lib/python3.9/site-packages/osbuild/inputs.py", line 121, in dispatch
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:     r = self.map(store,
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:   File "/usr/lib/osbuild/inputs/org.osbuild.ostree", line 141, in map
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:     reply = export(refs, source, target)
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:   File "/usr/lib/osbuild/inputs/org.osbuild.ostree", line 108, in export
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:     ostree("pull-local", repo_cache, commit,
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:   File "/usr/lib/osbuild/inputs/org.osbuild.ostree", line 90, in ostree
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:     subprocess.run(["ostree"] + args,
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:   File "/usr/lib64/python3.9/subprocess.py", line 528, in run
Aug 22 16:17:47 localhost.localdomain osbuild-worker[87077]:     raise CalledProcessError(retcode, process.args,
Aug 22 16:17:47 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:17:47-04:00" level=error msg="osbuild job failed: osbuild build failed" jobId=b5b21628-80c5-4670-9027-422b37d4df1b
Aug 22 16:17:47 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:17:47-04:00" level=warning msg="Job 'b5b21628-80c5-4670-9027-422b37d4df1b' (osbuild) failed: error decoding osbuild output: unexpected end of JSON input\nthe raw output:\n"

If I try to compose an ostree-based disk image from a repo with only a single commit (i.e. after step 4 above), it is successful.

Successful compose:

Aug 22 16:11:08 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:11:08-04:00" level=info msg="Running job 'a02eedd8-dca5-4cc3-878a-6450596652b6' (osbuild)\n"
Aug 22 16:11:48 localhost.localdomain ostree[85491]: libostree pull from '74e00493-9967-44f8-b781-1739d3b41ffb' for 0 refs complete
                                                     security: GPG: disabled
                                                     security: SIGN: disabled http: TLS
                                                     non-delta: meta: 3480 content: 26858
                                                     transfer: secs: 39 size: 841.9 MB
Aug 22 16:12:38 localhost.localdomain kernel: loop0: detected capacity change from 0 to 20971520
Aug 22 16:12:38 localhost.localdomain kernel: loop0: detected capacity change from 0 to 20971520
Aug 22 16:12:38 localhost.localdomain kernel: loop0: detected capacity change from 20971520 to 19920863
Aug 22 16:12:38 localhost.localdomain kernel: loop0: detected capacity change from 0 to 20971520
Aug 22 16:12:38 localhost.localdomain kernel: loop0: detected capacity change from 20971520 to 19920863
Aug 22 16:12:39 localhost.localdomain kernel: loop0: detected capacity change from 0 to 20971520
Aug 22 16:12:39 localhost.localdomain kernel: loop0: detected capacity change from 20971520 to 19920863
Aug 22 16:12:39 localhost.localdomain kernel: loop0: detected capacity change from 0 to 20971520
Aug 22 16:12:39 localhost.localdomain kernel: loop0: detected capacity change from 20971520 to 260096
Aug 22 16:12:40 localhost.localdomain kernel: loop0: detected capacity change from 0 to 20971520
Aug 22 16:12:40 localhost.localdomain kernel: loop0: detected capacity change from 20971520 to 786432
Aug 22 16:12:40 localhost.localdomain kernel: loop0: detected capacity change from 0 to 20971520
Aug 22 16:12:40 localhost.localdomain kernel: loop0: detected capacity change from 20971520 to 19920863
Aug 22 16:12:41 localhost.localdomain kernel: loop0: detected capacity change from 0 to 20971520
Aug 22 16:12:41 localhost.localdomain kernel: loop0: detected capacity change from 20971520 to 786432
Aug 22 16:12:41 localhost.localdomain kernel: loop1: detected capacity change from 0 to 20971520
Aug 22 16:12:41 localhost.localdomain kernel: loop1: detected capacity change from 20971520 to 260096
Aug 22 16:12:41 localhost.localdomain kernel: loop2: detected capacity change from 0 to 20971520
Aug 22 16:12:41 localhost.localdomain kernel: loop2: detected capacity change from 20971520 to 19920863
Aug 22 16:12:41 localhost.localdomain kernel: XFS (dm-4): Mounting V5 Filesystem
Aug 22 16:12:41 localhost.localdomain kernel: XFS (dm-4): Ending clean mount
Aug 22 16:12:41 localhost.localdomain kernel: XFS (loop0): Mounting V5 Filesystem
Aug 22 16:12:41 localhost.localdomain kernel: XFS (loop0): Ending clean mount
Aug 22 16:12:48 localhost.localdomain kernel: XFS (loop0): Unmounting Filesystem
Aug 22 16:12:49 localhost.localdomain kernel: XFS (dm-4): Unmounting Filesystem
Aug 22 16:12:49 localhost.localdomain kernel: loop0: detected capacity change from 0 to 20971520
Aug 22 16:12:49 localhost.localdomain kernel: loop0: detected capacity change from 20971520 to 19920863
Aug 22 16:12:49 localhost.localdomain kernel: loop0: detected capacity change from 0 to 20971520
Aug 22 16:12:49 localhost.localdomain kernel: loop0: detected capacity change from 20971520 to 19920863
Aug 22 16:13:50 localhost.localdomain NetworkManager[764]: <info>  [1692735230.1187] dhcp4 (enp1s0): state changed new lease, address=<IP_ADDRESS>
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="build pipeline results:\n" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.rpm success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.selinux success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="image-tree pipeline results:\n" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.ostree.init-fs success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.ostree.pull success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.ostree.os-init success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.mkdir success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.ostree.deploy success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.ostree.remotes success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.ostree.fillvar success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.ostree.config success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.fstab success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.ignition success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.users success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.keymap success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.locale success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.grub2 success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.ostree.selinux success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="image pipeline results:\n" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.truncate success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.sfdisk success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.luks2.format success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.clevis.luks-bind success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.lvm2.create success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.mkfs.fat success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.mkfs.xfs success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.mkfs.xfs success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.copy success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.lvm2.metadata success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.luks2.remove-key success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.grub2.inst success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="xz pipeline results:\n" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:06 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:06-04:00" level=info msg="  org.osbuild.xz success" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:07 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:07-04:00" level=info msg="osbuild job succeeded" jobId=a02eedd8-dca5-4cc3-878a-6450596652b6
Aug 22 16:15:07 localhost.localdomain osbuild-worker[5106]: time="2023-08-22T16:15:07-04:00" level=info msg="Job 'a02eedd8-dca5-4cc3-878a-6450596652b6' (osbuild) finished"

In step 6

Start new edge-commit compose

is the new commit built as a child of the first one or is it completely separate?

is the new commit built as a child of the first one or is it completely separate?

The available command history + journal on my RHEL 9.2 system isn't sufficient to determine the exact commands used.
I'll need to experiment some more to see if I can recreate this behavior.

I've not been able to reproduce this with multiple commits in a repo.  I've tried with commits that have proper parent commits and with commits that are "independent".
If I encounter this again, I'll reopen.


Engine graph in StatisticsOverlay is always infinity fps.

I know you're good... but you're not that good...

I can't repro this. Is this specifically for when running under mojo maybe?

This was a mojodroid issue.


Softmax fwd optimization for GPU

Description
This PR optimizes Softmax implementation (fwd only for now, I will add bwd in the future PR) for cases where stride is 1 and the leading dimension is small (up to 20kB of data in that dimension).
There are 2 optimizations in this kernel compared to the previous one:

using of the longer datatypes for loading/writing data (so using up to 8B per read/write instead of e.g. 2B in case of fp16 I/O)
using persistent storage to reduce the number of memory accesses (previous implementation used 3 reads and 1 write, this implementation uses 1 read and 1 write).

Compared to the previous implementation on fp16 I/O the new kernel is up to 4x faster.
@eric-haibin-lin
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[x] Changes are complete (i.e. I finished coding on this PR)
[x] All changes have test coverage:
[x] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

@mxnet-label-bot add [Operator, pr-awaiting-review]

@ptrendx Do you have any data on how much performance boost this change is introducing on applicable example workloads?

Great! I'm considering to use this kind of warp-level primitives to accelerate our current reduce.

@haojin2 For the perf improvement on end to end training - I did not measure yet, measured just the kernel speedup for now. BTW - how important do you think safe accumulation option is (as in, having an option to NOT do it instead of just always doing safe accumulation)? Personally I don't see value in not using it in softmax as it would most probably affect accuracy, and having 3 TYPE_SWITCH makes the compilation time quite big. You did not put the ability to skip it in softmax_with_length, should I remove it from regular softmax as well?

@haojin2 I'm not sure how to progress with this PR - it seems that Windows CI instances do not have enough RAM to process all the templates here. The problem exists even with CPU compilation, where I just fixed your omission of MXNET_SAFE_ACCUMULATION=0 case in softmax with length.

@sxjscience totally agree.  This would provide a lot of benefit across the framework (for example the layernorm op).
@ptrendx
I see what you mean "fatal error C1002: compiler is out of heap space in pass 2".  The CI windows machines should have a fair amount of RAM so this is a little strange.

@marcoabreu Could you give some advice on that Windows CI problem - are the Windows builder instances much different than the unix ones?

Yeah they are. but the heap space error is the same old problem that we just have too many macros I think. Basically we can't add any more operators because the file just grew too large.
Try compiling locally on Unix with optimizations disabled and enable intermediary output. You will see some intermediary files being multiple gigabyte in size.

@haojin2 had the same problem btw

But the windows instances are c5.18xlarge. they have plenty of ram. We're literally running into the limitations of the compilers

Would it help if we split the fwd and backward into different files? Those limitations are per file, right?

I'd rather split operators into different files. Forward and backwards path kinda belong together, right?
This could also lay the base for dynamic loading of operators where we could ship operators selectively.

Thanks for the info Marco, appreciate you helping with this.
I remember the MSVC team was very slow in moving their compiler process to 64 bit.  Just to be clear they supported compiling 64 bit programs very early on, but the compiler process itself didn't need much ram so it stayed 32-bit.  Apparently now they do have a 64 bit compiler process, but 32-bit is still the default.  Is there any way we could check any of the windows hosts, and see if the compiler process they're using (CL.exe) is running in 32-bit mode in the task manager?  Are any of the compilation processes actually using more than 4GB of ram?

I remember that I checked it on Unix and that I saw high ram usage there. I'm currently not able to get data from windows, but the operator compilation has always been a bottleneck.

There really is a pattern that these kinds of errors always come when people try to add new operators

https://developercommunity.visualstudio.com/content/problem/284930/error-c1002-compiler-is-out-of-heap-space-in-pass.html We should update to VS 2017

(or Newer)

Ok, it seems that splitting softmax.cc into 3 files, 1 for each operator (softmax, softmin and log_softmax) did the trick fortunately.

Thanks for refactoring the Softmax functions to make it into one.


Simplify Package Name, Add JSON config option

I appreciate this package! It saved me a lot of time.
I'm proposing two improvements:

Simplify the package name pkce_client, its a simpler, more memorable import.  It is still open on PyPI last I checked(8/20/24).  I think you should add this as a package in PyPI. Your solution is a top hit in google.
PkceLoginConfig.from_config_file() This adds the ability to load a JSON config (format in README.md) that saves time when you are using the same PKCE endpoint frequently.

I think you should deploy your package to PyPI!

Hi @HungryGnu, thanks for this PR. This package is on PyPI already, you can find it here: https://pypi.org/project/TerevintoSoftware.PkceClient/
I don't mind the changes in this PR that add support for the JSON config, but:

I'm not open to updating the package's namespace, all my packages have the same prefix.
There are a number of cache files that need to be excluded in the .gitignore and removed from the branch.
The changes to pyproject.toml removed [project.urls], which would remove the GitHub URLs from PyPI so it's not acceptable.

FYI, I stopped using this package myself when I found this one: https://github.com/AzureAD/microsoft-authentication-library-for-python

For what it's worth, I stopped using this package myself when I found this one: https://github.com/AzureAD/microsoft-authentication-library-for-python

I updated the pull request. I restored the URL's and the 'terevintosoftware' namespace.
Poetry doesn't allow the dot in the namespace, not sure if that is a dealbreaker for that particular change.
I'll take a look at MSAL. Thanks for the tip!


[RCD-32] reduce logging verbosity

This commit amends the daedalus.yaml logger config file by tuning down
the verbosity of the 'syncWalletWorker', which is the component
responsible for logging wallet restorations.
It spits out lots of DEBUG output and this was saturating rapidly the
end user's filesystem.
This commit reduces the verbosity to Error+.
Description

Linked issue

Type of change

[X] 🐞 Bug fix (non-breaking change which fixes an issue)
[ ] 🛠 New feature (non-breaking change which adds functionality)
[ ] ⚠ Breaking change (fix or feature that would cause existing functionality to change)
[ ] 🏭 Refactoring that does not change existing functionality but does improve things like code readability, structure etc
[ ] 🔨 New or improved tests for existing code
[ ] ⛑ git-flow chore (backport, hotfix, etc)

Developer checklist

[X] I have read the style guide document, and my code follows the code style of this project.
[X] If my code deals with exceptions, it follows the guidelines.
[X] I have updated any documentation accordingly, if needed. Documentation changes can be reflected in opening a PR on cardanodocs.com, amending the inline Haddock comments, any relevant README file or one of the document listed in the docs directory.

Testing checklist

[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

QA Steps

Screenshots (if available)

this doesn't reduce the logging in cardano-node.log which is where the problem is.
it seems this is going to take several days to make a one line fix in our code so lets abandon it for this release, i will bring this up in retro because its a sorry state of affairs


Getting struck at "waiting setconf dlci thread"

I have successfully compiled the poc on kernel version "6.5.0-28-generic" but it get stuck at waiting setconf dlci thread
Do we have to enable something from the config ?

Thanks, It work after trying multiple times


When importing a wallet, no extra FreeTFT trustline should be created

hey @robvanmieghem does this still happen in production? We thought this trustline was only created on the staging version. All new wallets in production are only created with TFT Trustline.

Now they don't even add anymore on staging.


Release build accessible from chrome://inspect

Short description of the problem:
Release build of my Android app is accessible from chrome://inspect.  I am able to see console logs and HTTPS network traffic.
What behavior are you expecting?
Release build should not be accessible.
Steps to reproduce:

ionic start -v2 testblank blank
ionic plugin remove cordova-plugin-console
add some console logs and HTTPS network requests.
ionic build android --release --prod -- --keystore="..." --alias=...
install on Genymotion and inspect with Chrome

Which @ionic/app-scripts version are you using?
1.0.0
Other information: (e.g. stacktraces, related issues, suggestions how to fix, stackoverflow links, forum links, etc)
May be related to #671.
$ ionic info
******************************************************
 Dependency warning - for the CLI to run correctly,
 it is highly recommended to install/upgrade the following:

 Please install your Cordova CLI to version  >=4.2.0 `npm install -g cordova`

******************************************************

Your system information:

 You have been opted out of telemetry. To change this, run: cordova telemetry on.
6.4.0

Ionic Framework Version: 2.0.0-rc.5
Ionic CLI Version: 2.2.1
Ionic App Lib Version: 2.2.0
Ionic App Scripts Version: 1.0.0
ios-deploy version: Not installed
ios-sim version: Not installed
OS: Windows 8.1
Node Version: v6.9.2
Xcode version: Not installed

******************************************************
 Dependency warning - for the CLI to run correctly,
 it is highly recommended to install/upgrade the following:

 Please install your Cordova CLI to version  >=4.2.0 `npm install -g cordova`

******************************************************

$ node -v
v6.9.2

$ npm -v              **this takes about 30s**
4.0.5

$ cordova -v
You have been opted out of telemetry. To change this, run: cordova telemetry on.
6.4.0

$ ionic platform                   **this also takes about 30s***
******************************************************
 Dependency warning - for the CLI to run correctly,
 it is highly recommended to install/upgrade the following:

 Please install your Cordova CLI to version  >=4.2.0 `npm install -g cordova`

******************************************************

You have been opted out of telemetry. To change this, run: cordova telemetry on.

Installed platforms:
  android 6.1.0
Available platforms:
  amazon-fireos ~3.6.3 (deprecated)
  blackberry10 ~3.8.0
  browser ~4.1.0
  firefoxos ~3.6.3
  webos ~3.7.0
  windows ~4.4.0
  wp8 ~3.8.2 (deprecated)

As I mentioned in the other issue, it's very likely you're not doing a true prod build.
Thanks,
Dan

@danbucholtz, here is the output of my build:

ionic-app-scripts build "--release" "--prod" "--" "--keystore=" "--alias="

[09:12:24]  ionic-app-scripts 1.0.0
[09:12:24]  build prod started ...
[09:12:24]  clean started ...
[09:12:24]  clean finished in 47 ms
[09:12:24]  copy started ...
[09:12:24]  ngc started ...
[09:12:43]  ngc finished in 19.11 s
[09:12:43]  webpack started ...
[09:12:45]  copy finished in 21.30 s
[09:12:51]  webpack finished in 7.82 s
[09:12:51]  uglifyjs started ...
[09:12:51]  sass started ...
[09:12:55]  sass finished in 3.92 s
[09:12:55]  cleancss started ...
[09:12:57]  cleancss finished in 2.28 s
[09:13:03]  uglifyjs finished in 12.34 s
[09:13:03]  build prod finished in 39.41 s
It looks like it is at least thinks it is building prod.  Is there any other way I can confirm this?

@danbucholtz
Like with the other issue I logged (#671), I am also able to debug the release build of the blank app using chrome://inspect, using the signed release apk that I built and ran in Genymotion on my Mac.

@danbucholtz This is still an issue using latest app-scripts nightly build. Please reopen.

@danbucholtz can you please reopen and have a look at this. This seems like a serious security bug. Problem wasn't resolved with changes that fixed #671

I'm sorry, perhaps I'm not understanding this correctly. Why would you expect not to be able to see that data?

Probably because I am new to this kind of development :-) If someone
downloads my app and installs it on the simulator, they will then be able
to see my private api between the app and the server. This could open up
the server to misuse or abuse.
On Sun, 05 Feb 2017, 09:19 Dan Bucholtz<EMAIL_ADDRESS>wrote:

I'm sorry, perhaps I'm not understanding this correctly. Why would you
expect not to be able to see that data?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/driftyco/ionic-app-scripts/issues/672#issuecomment-277502249,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AVoOPy3wkyFG2VWNKec4pZZaD9uCyL-9ks5rZXgagaJpZM4Lpv-Z
.

This is because of Genymotion, they do something to the apk when you install it and becomes debuggable.


Share plugin fails partially in Windows RT and WindowsPhone RT

CrossShare.Current.Share - FAIL
CrossShare.Current.ShareLink - OK
CrossShare.Current.OpenBrowser - OK

Can you please report: https://github.com/jguertl/SharePlugin
Also what the failure is and perhaps a crash log.


Asynced credits and more

Im backkk
This is gonna mainly be for the asynced credits menu rework as you could see from the title but also for other general improvement / fixes (as you can for example see what i did libs.xml)
Im planning on making more kind of credits displayer types too :p

yes ik i still gotta do the offline part and more, its 5am i'll continue tmr💀

oh and btw for that thing in libs.xml, i asked it to @NeeEoo first and he said to do it soo

okay i had to do .persistent = true, im so dumb lmfaoo
i thought just because i was working with bitmapdatas it wouldnt work since .persistent was about flxgraphic (but the add function actually RETURNS a flxgraphic💀💀)

ah shit: haxe.ds.Either<GitHubUser, GitHubContributor>
imma add it later

nvm both either class and oneoftwo class are bullshits💀

for now ig its good enough like this
man if someone checks this chat is gonna think im schizo💀

okay, as suggested by @NeeEoo im gonna make the non api method primary
im gonna push the commit later

lemme try to reduce those tries cause theyre many

ready to be merged


[FIXED] Feat: Adding searching and sorting functionality in Dubai page #1864

Related Issue
#1864
Fixes:  #1864
Description
#1864
On the Dubai page where the hotels are displayed there is no option to search hotels by their name or sort the hotel by their price such as 'high to low' or 'low to high' or 'featured'.
I added the following functionalities to the page:

Sorting
Searching

Type of PR

[ ] Bug fix
[X] Feature enhancement
[ ] Documentation update
[ ] Other (specify): _______________

Screenshots / videos (if applicable)
The feature looks as follows:

The feature looks like this on the page:

The new feature works as follows:
https://github.com/user-attachments/assets/2416e4ed-a17d-4676-8cba-6179221c2075
Checklist:

[X] I have made this change from my own.
[X] My code follows the style guidelines of this project.
[X] I have performed a self-review of my own code.
[X] I have commented my code, particularly in hard-to-understand areas.
[X] I have made corresponding changes to the documentation.
[X] My changes generate no new warnings.
[X] I have tested the changes thoroughly before submitting this pull request.
[X] I have provided relevant issue numbers and screenshots after making the changes.

@PriyaGhosal check this out


Non-matching types for same id

On line 9168 of AvaTaxClient
 public Future<FundingStatusModel> activateFundingRequestAsync(Long id) {
        AvaTaxPath path = new AvaTaxPath("/api/v2/fundingrequests/{id}/widget");

 public FundingStatusModel fundingRequestStatus(Integer id) throws Exception {
        AvaTaxPath path = new AvaTaxPath("/api/v2/fundingrequests/{id}");

Once funding request id is Int, once it's Long. Which one is it ? :-)

This is resolved in version 24.6.4


Length Aware Paginator

I think it's important for "new to laravel" folks to understand slice is not the only way to use the Length Aware Paginator.  Please see https://github.com/laravel/docs/issues/4997

Hey, you've already created an issue within docs. Please close as this doesn't have any bugs/issues within the framework itself.


HNC: Test what if two parents have the same subnamespace anchor

This could be a race condition when two parents both create a subnamespace anchor with the same name simultaneously. The webhook could be bypassed if the second anchor is created before the forest is updated.
Test and see what would happen if we delete the bad anchor in one of the parent. The good anchor is the one in the namespace that the subnamespace points to as the parent.

/assign @yiqigao217
cc @adrianludwin

FTR the current behavior is that the subnamespace of the good anchor will also be deleted when we try to delete the bad anchor, which is very dangerous. The change in #799 will prevent this.

Note that this bug, while scary, is doubly-unlikely to occur in HNC v0.4:

The validation controller will attempt to prevent you from creating a bad anchor. A bad anchor can only be created if the webhook is disabled, or in extremely rare circumstances if there's a race condition.

In HNC v0.4, you cannot delete any subnamespace unless allowCascadingDelete is set to true in that subnamespace (this restriction is relaxed in v0.5). Since this field should very rarely be set, it will typically not be possible for the owner of a bad anchor to delete someone else's namespace.

Note that this is a good validation of our defence-in-depth strategy to make sure it's very difficult to delete subnamespaces.


Unclear / misdirecting wording in step-by-step guide to setup Marketing Analytics

The wording of Step 8 of the Guide is unclear / misdirecting, with the sentence:
“Sign into Dynamics 365 Marketing and go to the Settings area in the area switcher, then go to Data management > Analytics configuration.”
Both myself, and my client made the same mistake of going to Advanced Settings, finding the Data Management page there, and getting confused why we couldn't see Analytics configuration there.
Perhaps a few screenshots could help here?

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 06d56767-ae95-eb6b-353e-af2ccb415d00
Version Independent ID: 5100476a-5a38-b8b3-6ba7-b61869ba3383
Content: Prepare for analytic reporting with Power BI (Dynamics 365 Marketing)
Content Source: ce/marketing/custom-analytics.md
Service: dynamics-365-marketing
GitHub Login: @alfergus
Microsoft Alias: alfergus

Thank you for the heads up on this! I'll add a screenshot to clarify. In the meantime, make sure you go to the area switcher in the lower left and select Settings there:

The Analytics configuration is under Data management on that page:


base_convert precision

Consider the following hexadecimal bitmap: f23e4491a8e08020
The following binary bitmap is the correct representation of the above hexadecimal one: 11110010 00111110 01000100 10010001 10101000 11100000 10000000 00100000
The following binary bitmap is the outcome of using PHP's base_convert function to represent the above hexadecimal one:  11110010 00111110 01000100 10010001 10101000 11100000 10000000 00000000
Notice how bit 59 is ON in the correct bitmap, and OFF in PHP's.
This is due to PHP's base_convert function losing precision when working with data of a length greater than 32 bits (see the docs).
I've created a work around in PR #1

Closed by your PR, thank you!
Awesome job!


Model Processing Integration

I am working on generation of voxel scenes for my game.
I fact what I did is generated a big vox file with my ground composed of tiles of 16x16x16 voxels. The actual size is 16x16 tiles. So I go around 450.000 voxels and the optimisation and export run fast. That is good.
I'll create a complete scene and see the number of voxels and if I run into issue.
In fact this is the process I would like : Unity Scene Editor -> Generate a scene description in voxels tiles -> Loaded in Voxelshop -> optimized export -> Blender for baking (optional) -> Unity integration as new scene or dynamic loading.
So I need to have Voxelshop able to make this conversion:

Having Voxelshop listening to some files change in a directory and then load it and convert it based on a description file
Other idea would be to launch Voxelshop with params and doing batch conversion and then stop working.

In any case I'll put in this issue feedbacks about my progress and some metrics.

It seems what you want to use VoxelShop for I mesh optimization. I'm seriously considering separating file conversions out and making them available as a web api...

As for your problem: Since you can already compile voxelshop locally, I'd recommend that you create script main Java file that does this for you. Should be easiest and fastest solution.

It could be also a good start for plug-in integration. If I do things with VoxelShop it seems to me better to share things than doing only for me...

Please continue discussion here: https://github.com/simlu/voxelshop/issues/38


Fix CI tests cl/only-use-dart-ss58-encoding

Fix CI tests cl/only-use-dart-ss58-encoding

Thanks for the help! I will close the PR as I could fix the error in my PR. :)


improve image file validation

Description:
improve image file validation, check whether CDI or QEMU have tools to validate the correct image type and content

Close as stale. Feel free to reopen if needed.


The word "Pending" in FOX Yeildy modal too faint to see when in Light Mode

Overview
When looking at the Overview for FOX Yeildy in the Defi vault, if you have a withdrawal that is pending, the word "Pending" is coloured a faint yellow colour and is very hard to read.
References and additional details

Log into app.shapeshift.com
Connect to a wallet of preference.
Under Settings change from dark mode to light mode.
Select the Fox Token Staking card from the DeFi vault
Notice if there is a withdrawal transaction that the word pending is very hard to read.

Acceptance Criteria
The word pending should be readable in both light and dark modes
Need By Date
No response
Screenshots/Mockups
No response
Ownership

[X] If my bounty needs engineering or needs product I have added the respective labels on the right
[X] As the sponsor of this bounty I will review the changes in a preview environment (ops/product) or review the PR (engineering)

Estimated effort
1 hour
Sponsor / Stakeholder
@0xdef1cafe
Bounty Hunters

Join our discord
Include an expected timeline for you to complete work in the work plan when you apply for this bounty!
Please refer to this link for some basic info
Please do not start work on this issue until you are approved in Gitcoin.

Looks like its a tint of the yellow. Likely we'll just need to use that solid color at 100%. @reallybeard to confirm

Looks like its a tint of the yellow. Likely we'll just need to use that solid color at 100%. @reallybeard to confirm

Yah we should be using yellow.500 for light mode and yellow.200 for dark.

@Lychbot yes plz

Is this Bounty still assigned to someone? I'd love to take a crack at it if it isn't.

If you don’t get an update soonish, I’d like to apply, seems straightforward


[cfe] Crash with cyclic dependency in type alias

Consider the following program:
typedef F = Function(Function<X extends F>());

void main() {}

This program crashes the front end with a fresh dart from 342f9f0d3cac0ae86b38e78aa9e6ccf9a94276a3, with a stack trace that indicates an infinite loop. Hence, some extra checks for cyclic dependencies in a type alias seem to be needed.
Crash when compiling null,
at character offset null:
Stack Overflow
#0      FunctionType.visitChildren (package:kernel/ast.dart:7487:3)
#1      RecursiveVisitor.defaultNode (package:kernel/visitor.dart:603:10)
#2      Visitor.defaultDartType (package:kernel/visitor.dart:518:39)
#3      Visitor.visitFunctionType (package:kernel/visitor.dart:525:45)
#4      FunctionType.accept (package:kernel/ast.dart:7481:42)
#5      FunctionType.visitChildren (package:kernel/ast.dart:9504:14)
#6      RecursiveVisitor.defaultNode (package:kernel/visitor.dart:603:10)
#7      Visitor.defaultDartType (package:kernel/visitor.dart:518:39)
#8      Visitor.visitFunctionType (package:kernel/visitor.dart:525:45)
#9      FunctionType.accept (package:kernel/ast.dart:7481:42)
#10     TypeParameter.visitChildren (package:kernel/ast.dart:8552:11)
...
#14170  Visitor.defaultTreeNode (package:kernel/visitor.dart:603:10)
#14171  TreeVisitor.visitLibrary (package:kernel/visitor.dart:246:35)
#14172  List.forEach (dart:core-patch/growable_array.dart:313:8)
#14173  transformLibraries (package:vm/transformations/call_site_annotator.dart>
#14174  VmTarget.performModularTransformationsOnLibraries (package:vm/target/vm>
#14175  KernelTarget.runBuildTransformations (package:front_end/src/fasta/kerne>
#14176  KernelTarget.buildComponent.<anonymous closure> (package:front_end/src/>
#14177  _rootRunUnary (dart:async/zone.dart:1198:47)
#14178  _CustomZone.runUnary (dart:async/zone.dart:1100:19)
...

Variants like typedef F = List<Function<X extends F>()> Function(); and typedef F = Function(List<Function<X extends F>()>); encounter the same issue.

FYI, https://dart-review.googlesource.com/c/sdk/+/167720 introduces a non-function type alias that exhibits a similar crash.  The test is cfe-strong-linux:language/nonfunction_type_aliases/cyclic_expansion_error_test and the stack trace looks like this:
#0      Visitor.visitFunctionType (package:kernel/visitor.dart:525:3)
#1      FunctionType.accept (package:kernel/ast.dart:7481:42)
#2      InterfaceType.visitChildren (package:kernel/ast.dart:9504:14)
#3      Visitor.visitInterfaceType (package:kernel/visitor.dart:603:10)
#4      InterfaceType.accept (package:kernel/ast.dart:7382:42)
#5      TypeParameter.visitChildren (package:kernel/ast.dart:8552:11)
#6      Visitor.defaultTreeNode (package:kernel/visitor.dart:603:10)
#7      TreeVisitor.visitTypeParameter (package:kernel/visitor.dart:251:47)
#8      TypeParameter.accept (package:kernel/ast.dart:8548:38)
#9      visitList (package:kernel/ast.dart:9504:14)
#10     FunctionType.visitChildren (package:kernel/ast.dart:7489:5)
#11     Visitor.visitFunctionType (package:kernel/visitor.dart:603:10)
#12     FunctionType.accept (package:kernel/ast.dart:7481:42)
#13     FunctionType.visitChildren (package:kernel/ast.dart:7493:16)
#14     Visitor.visitFunctionType (package:kernel/visitor.dart:603:10)
#15     FunctionType.accept (package:kernel/ast.dart:7481:42)
#16     InterfaceType.visitChildren (package:kernel/ast.dart:9504:14)
#17     Visitor.visitInterfaceType (package:kernel/visitor.dart:603:10)
#18     InterfaceType.accept (package:kernel/ast.dart:7382:42)
#19     TypeParameter.visitChildren (package:kernel/ast.dart:8552:11)
#20     Visitor.defaultTreeNode (package:kernel/visitor.dart:603:10)
#21     TreeVisitor.visitTypeParameter (package:kernel/visitor.dart:251:47)
#22     TypeParameter.accept (package:kernel/ast.dart:8548:38)
#23     visitList (package:kernel/ast.dart:9504:14)
#24     FunctionType.visitChildren (package:kernel/ast.dart:7489:5)
#25     Visitor.visitFunctionType (package:kernel/visitor.dart:603:10)
#26     FunctionType.accept (package:kernel/ast.dart:7481:42)
...

I'm assuming it's the same root cause; if it isn't, feel free to split this out to its own issue.

@eernstg is this still an issue?

Closing: This is no longer crashing, neither the initial example here nor the tests introduced by https://dart-review.googlesource.com/c/sdk/+/167720.


New package: ConstructiveGeometry v0.1.0

Registering package: ConstructiveGeometry
Repository: https://github.com/plut/ConstructiveGeometry.jl
Created by: @plut
Version: v0.1.0
Commit: 2f4020a9155d6bacbadef08cac2b053588f7bcc2
Git reference: master
Release notes:

Package is right now at a “proof-of-concept” stage. Code and interface is nowhere near stable yet.

@plut thanks for the package!  Note the issue above though.  After fixing it, you can retrigger registration keeping the same version number
[noblock]


Refactor/decorator2

PR Type

[ ] Bugfix
[ ] Feature
[x] Refactoring

Context

[x] Related with #4750
[x] Related with #6392
[ ] Requires deployment <snek/rubick/worker>

Before submitting pull request, please make sure:

[x] My contribution builds clean without any errors or warnings
[x] I've merged recent default branch -- main and I've no conflicts
[x] I've tried to respect high code quality standards
[x] I've didn't break any original functionality

Optional

[ ] I've tested it at </ksm/collection>
[ ] I've tested PR on mobile
[ ] I've written unit tests 🧪
[ ] I've found edge cases

Did your issue had any of the "$" label on it?

[x] Fill up your DOT address: Payout

Community participation

[x] Are you at KodaDot Discord?

Screenshot 📸

[ ] My fix has changed something on UI; a screenshot is best to understand changes for others.

Copilot Summary

🤖 Generated by Copilot at 769abca
Refactored the code for opening the shopping cart modal from the navbar. Moved the logic and configuration from ShoppingCartModalConfig.ts and ShoppingCartButton.vue to Navbar.vue to improve code organization and performance.

🤖 Generated by Copilot at 769abca

ShoppingCartButton
Simpler, no more emit -
Autumn leaves falling

Thank you @vikiival & @roiLeo for reviewing my PR.

@prury can I ask for a recheck one more time if validation works?

@prury can I ask for a recheck one more time if validation works?

on it!

if it does what i think it does (refactor the part related to bsx assets and its balances and also the button to change asset fee on bsx) then it works 😄

if it does what i think it does (refactor the part related to bsx assets and its balances and also the button to change asset fee on bsx) then it works 😄
https://basilisk.subscan.io/tx/0xd3a89e8a289a56677972b5c8fe600cead5e68df63c9fe7ad3f2cfdbc11821bdc

I don't understand the basilik scan btw. I just see a couple of transactions being made. I just recently started on this project so need more help @prury, would greatly appreciate if you can explain in more detail
Thank you

pay 50 usd

😍 Perfect, I’ve sent the payout
💵 $50 @ 4.78 USD/DOT ~ 10.46 $DOT
🧗 16UcV9V6nVvPYdHz98ymUKmNLkzjCEU5sbKJMi7hxYyTHjzR
🔗 0xe4c12d2e443dc979709e19365f6182ee89fa40501745a12d76d3b815b612c182
🪅 Let’s grab another issue and get rewarded!
🪄 github.com/kodadot/nft-gallery/issues


use "schema" as "search_path" connection option

This change adds an alias currentSchema for schema configuration parameter; and uses this value as a value for "search_path" connection option (instead of executing SET SCHEMA TO command upon connection)
[resolves #271 ]

Make sure that:

[x] You have read the contribution guidelines.
[x] You have created a feature request first to discuss your contribution intent. Please reference the feature request ticket number in the pull request.
[ ] You use the code formatters provided here and have them applied to your changes. Don't submit any formatting related changes.
[x] You submit test cases (unit or integration tests) that back your changes.

Issue description

Closes #271
New Public APIs

Additional context

hi @mp911de is to too bad for review? just to clarify: I've left one action (about formatters) unchecked, because when I applied (or thought to have applied) the recommended style and ran the Code Reformat action in IntelliJ, the code was re-formatted heavily so I had to manually adjust it.
It still feels that making something like
Map<String, String> options;
if (connectionFactoryOptions.getValue(OPTIONS)) instanceof String) {
   options = tryParseOptions(connectionFactoryOptions.getValue(OPTIONS));
} else {
   options = connectionFactoryOptions.getValue(OPTIONS);
}

might be a more generic way to address this (and similar) issues. What do you think ?

No worries. We weren't able yet to review this PR as we're busy with other projects.

Thank you for your contribution. That's merged, polished, and backported now.


Fixes memory leak in Image.cpp

I seem to have found a memory-leak that occurs when an image is loaded from disk.

I had the same problem and made the same fix in my project. Definitely good pull request, +1 from me

I seem to have found a memory-leak that occurs when an image is loaded from disk.

I had the same issue, thanks for the fix it helped met! 👍


blog post to announce arcade changes

[ ] document new features and field editors
[ ] document tilemap api changes
[x] Introduce new sprite editor, tools, controls.
[x] mention GitHub integration
[x] talk about the new look of the Sim

more changes, not sure which ones should make the post:

settings api - can persist settings, save games (JS only), and Hi Score is persisted on hardware and in sim
a bunch of new sprite assets
experimental WiFi support [not sure here; I think the packages are hidden]
serial port support on Linux/RPi

Closed in favor of #6417.


Bumblekite: Machine learning summer school in healthcare and biosciences in Zurich, Switzerland

Please add this new MLSS that is coming up in 2020: Machine learning summer school in healthcare and biosciences
19th July - 28th July 2020
ETH Zurich, Zurich, Switzerland
https://four-corp.com/bumblekite

Thank you very much. More details are available now:

Name/URL: bumblekite – Machine learning summer school in healthcare and biosciences
Location: Zurich, Switzerland
Organiser: Four
Dates: 19th July – 28th July 2020
Deadline: March 15th 2020
Fee: $600~$2900
Aid (Travel Grants etc): Full and partial scholarships covering part or the entire cost of participation

Thanks for adding :+1:

Added :+1:


Duplicate content when no launches are found

Current example: https://api.spacexdata.com/v1/launches/upcoming?start=2017-10-30&final=2017-11-06
Output:
{
  "error": "No Matches Found"
}{
  "error": "No Matches Found"
}[]

Weird, looking into this currently

Cache would certainly be an explanation. I was looking into it and didn't see anything obvious, but one thing that stuck out was it wasn't consistent.


[client-v2] Underscore in column name is removed

Describe the bug
According to the documentation, underscore is a valid character in a column name:
Non-quoted identifiers must match the regex ^[a-zA-Z_][0-9a-zA-Z_]*$ and can not be equal to [keywords](https://clickhouse.com/docs/en/sql-reference/syntax#keywords). Examples: x, _1, X_y__Z123_.

But when I try to register an POJO for a table in the Java client-v2, an error occurs for any column that contains an underscore in the name:
Exception in thread "main" java.lang.IllegalArgumentException: No serializer found for column 'ORA_ROW_ID'. Did you forget to register it?
at com.clickhouse.client.api.Client.insert(Client.java:1178)
at com.clickhouse.client.api.Client.insert(Client.java:1124)
at solutions.a2.oracle.expimp.pipe.DtoTest.main(DtoTest.java:84)
Steps to reproduce

Create project with Java client-v2, i.e. for Gradle add dependency:

	def clickHouseVersion = "0.7.0"
	// https://mvnrepository.com/artifact/com.clickhouse/clickhouse-client
	implementation("com.clickhouse:client-v2:$clickHouseVersion")

Create table

  CREATE TABLE TEST4CH(
    ORA_ROW_ID String,
  	OWNER String,
	NAME String,
	TYPE String,
	LINE UInt32,
	TEXT String,
	ORIGIN_CON_ID Nullable(UInt32)) engine = MergeTree order by ();

Try to register POJO for this table and then insert to this table

Expected behaviour
Successful execution
Code example
		client.register(dynamicClass, client.getTableSchema("TEST4CH"));
		InsertSettings settings = new InsertSettings().compressClientRequest(true);
		try (InsertResponse response = client.insert("TEST4CH", list).get()) {
			System.out.println("rows written=" + response.getWrittenRows());
		} catch (InterruptedException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (ExecutionException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
		client.close();

Error log
6:50:01,415  INFO main Client:build:886 - Using server timezone: UTC
06:50:01,712  INFO main Client:poolConnectionManager:201 - Connection reuse strategy: FIFO
06:50:01,745  INFO main Client::110 - client compression: true, server compression: true, http compression: false
06:50:01,746  INFO main Client::168 - Using new http client implementation
06:50:03,499  WARN main Client:register:1087 - No getter method found for column: orarowid
06:50:03,499  WARN main Client:register:1095 - No setter method found for column: orarowid
06:50:03,509  WARN main Client:register:1087 - No getter method found for column: originconid
06:50:03,509  WARN main Client:register:1095 - No setter method found for column: originconid
Exception in thread "main" java.lang.IllegalArgumentException: No serializer found for column 'ORA_ROW_ID'. Did you forget to register it?
at com.clickhouse.client.api.Client.insert(Client.java:1178)
at com.clickhouse.client.api.Client.insert(Client.java:1124)
at solutions.a2.oracle.expimp.pipe.DtoTest.main(DtoTest.java:84)
Configuration
Environment

Client version: 0.7.0
Language version: N/A
OS: Linux

ClickHouse server
24.6.2

CREATE TABLE statements for tables involved:
CREATE TABLE TEST4CH(
ORA_ROW_ID String,
OWNER String,
NAME String,
TYPE String,
LINE UInt32,
TEXT String,
ORIGIN_CON_ID Nullable(UInt32)) engine = MergeTree order by ();

Proposed PR - register(): make underscore legal in column name

@averemee-si thank you for reporting the issue!
Would you please give us an example of your POJO?
If it doesn't have getters please let us know.
Thank you!

Sergey,
I'm not using a static POJO because we need to build a bootstrapped Oracle database with thousands of tables, each table with up to five hundred columns, and each name with an underscore. Due to these limitations, we use dynamic construction of the POJO

You need to add ByteBuddy to project dependencies - below Gradle snippet

	def byteBuddyVersion = "1.15.3"
	// https://mvnrepository.com/artifact/net.bytebuddy/byte-buddy
	implementation("net.bytebuddy:byte-buddy:$byteBuddyVersion")

Java code snippet (of course this is an example - we plan to download data from Oracle Data dictionary

		DynamicType.Unloaded<?> dynamicType = new ByteBuddy()
				.subclass(Object.class, ConstructorStrategy.Default.DEFAULT_CONSTRUCTOR)
				.name("DEPT.SCOTT.PRD.ora")
				.defineProperty("ORA_ROW_ID", String.class)
				.defineProperty("OWNER", String.class)
				.defineProperty("NAME", String.class)
				.defineProperty("TYPE", String.class)
				.defineProperty("LINE", int.class)
				.defineProperty("TEXT", String.class)
				.defineProperty("ORIGIN_CON_ID", int.class)
				.make();

		Class<?> dynamicClass = dynamicType
				.load(dynamicType.getClass().getClassLoader(), ClassLoadingStrategy.Default.WRAPPER)
				.getLoaded();

		// Print list of getters/setters
		for (Method method : dynamicClass.getDeclaredMethods()) {
			System.out.println("++>" + method.getName() + "=" + Modifier.isPublic(method.getModifiers()));
		}

		Constructor<?> dynamicConstr = dynamicClass.getConstructor();
		Object dtoElement = dynamicConstr.newInstance();

		dynamicClass.getMethod("setORA_ROW_ID", String.class).invoke(dtoElement, "AAAXCiAADAAAb2TAAA");
		dynamicClass.getMethod("setOWNER", String.class).invoke(dtoElement, "SYS");
		dynamicClass.getMethod("setNAME", String.class).invoke(dtoElement, "STANDARD");
		dynamicClass.getMethod("setTYPE", String.class).invoke(dtoElement, "PACKAGE");
		dynamicClass.getMethod("setLINE", int.class).invoke(dtoElement, 1);
		dynamicClass.getMethod("setTEXT", String.class).invoke(dtoElement, "package STANDARD AUTHID CURRENT_USER is              -- careful on this line; SED edit occurs!");
		dynamicClass.getMethod("setORIGIN_CON_ID", int.class).invoke(dtoElement, 1);

		List<Object> list = new ArrayList<>();
		list.add(dtoElement);

		PipePool pool = PipePool.get("ch-1", "clickhouse:XXXXX", "YYYYYYY", "ZZZZZZZZ");
		Client client = pool.getChClient();
		client.register(dynamicClass, client.getTableSchema("TEST4CH"));
		InsertSettings settings = new InsertSettings().compressClientRequest(true);
		try (InsertResponse response = client.insert("TEST4CH", list).get()) {
			System.out.println("rows written=" + response.getWrittenRows());
		} catch (InterruptedException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (ExecutionException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
		client.close();

Thanks/ Заранее признателен!
Aleksei

@averemee-si Thank you for explaining!
This is fine - I will think on the solution to make it more general.  Something like settings where exact mapping or strategy can be passed. Hopefully this week I can provide the solution.

Good day, @averemee-si!
Have you a chance to test the nightly build?
Thanks!

Hi Sergey,
Thanks for the update.
I'll check it tomorrow.
Regards,
Aleksei

Hi Sergey,
I started checking and realized that I can't download the nightly build. Please tell me where I can find it.
Thanks.
Regards,
Aleksei

@averemee-si here is a gist with instructions https://gist.github.com/chernser/b4eacfde70093847449e3aef8e51ae8e

Thanks!
Cool.
Tested with build 20241022-155731-21
Regards,
Aleksei

@averemee-si
is it resolve the issue?

Yes, thanks.
that's what Cool meant
Regards,
Aleksei


Update UI Constraint Schema to Align with Backend Changes

There is a PR open in the backend to correct a schema mismatch between the cached constraint results and the active constraint check response https://github.com/NASA-AMMOS/aerie/pull/1156
We need to update the UI to ensure we're aligned with these schema adjustments.

No fix needed. The constraint_run table was changed to match the constraint action response, and we are not currently reading anything out of constraint_run. If we start using the cached run results, it looks like the schema will match the shape we are already expecting for constraint check results.


DP-697 Two-way transformer between list and list<Property>

Compared to previous request: all too specific things are dropped, only transformer left.

Good job. You finally did it.


add-sequence command

Rename insert-sequence to add-sequence to be consistent

Also, modify it so multiple sequences can be added simultaneously. Thus, the following command would be valid: ./proton add-sequence key.pub testproj 1 2 3 4 5 --index=1. All added sequences will be inserted at the index as a chunk. If the playlist was [7, 8, 9], then the above command will cause the resulting playlist to be [7, 1, 2, 3, 4, 5, 8, 9]


Add gelf input to logstash

@SimonSchneider thanks for your contribution, but the correct way to customize your stack is to fork this repo and implement your configuration in your own repository.
We can't accept configuration changes otherwise nobody would be able to agree on what's required or not (and we have a lot of users). Instead, we ship a minimal configuration that everyone can use as a base.


[vcpkg] Refactor Commands: Part 1

This is simply a code cleanup, in preparation for the coming refactoring :) This is a simple change that allows us to go forward :)

/azp run


Coverage on instrumented app crash

Hi, is there any way (or will there any way in the future) for acvtool to obtain code coverage for an app even when the app crashes?

Hi Miguel, this functionality is implemented.
Some details are here https://github.com/pilgun/acvtool/issues/8#issuecomment-495898872
I remember there was a known problem in Android when handling exception by Instrumentation class. As of now it's not in my head, but I consider to come back to it later.

In the comment its said that crashes are logged into a errors.txt file but i cant seem to find it, where is the file created?
This is acv's output in the command line
INSTRUMENTATION_RESULT: shortMsg=Process crashed.
INSTRUMENTATION_CODE: 0

Acvtool is about coverages rather then crashes. Catching them through
Instrumentation process isn’t reliable. I.e., if instrumentation process
crashes itself, it will be impossible to catch anything. Yet, it’s
implemented to give as much info as possible. Always use logcat to reveal
crashes reliably.


work around EOS auth problems

There is a problem with the gsi authentication plugin in xrootd. Effect is that jobs running at CERN loose their EOS privileges after a while and the job dies as it can't read any input anymore.
This patch works around this problem by increasing an internal xrootd timeout.
This has been open for a while, the xrootd team didn't really provide a real fix apart from "please upgrade to the latest xrootd client and check if this is still an issue". Even if the latest xrootd client fixes this, it'll take a while until we stop using CMSSW releases that use older clients.
Making a pull request because a) don't want to run with this custom patch in the Tier0 forever and b) CompOps production jobs at CERN should also be affected by this.

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/797/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/799/ for details

Summary of pylint changes for pull request 6325:
Abbreviated pylint report for src/python/WMCore/WMSpec/Steps/Executors/CMSSW.py follows:

Line 29 in analysisFileLFN W0612 Unused variable 'junk'
Line 68 in CMSSW.pre W0612 Unused variable 'stepHelper'
Line 220 in CMSSW.execute W0612 Unused variable 'applicationStart'
Line 13 W0611 Unused import select
plus 6 total warnings
plus 65 comments on code style

No changes to unit tests for pull request 6325

Ok, addressing these issues too...

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/801/ for details

Dirk, is this 86400 the maximum length a job can run at CERN without facing these xrootd disconnections? If so, then we should increase it to 2 days at least, no?

Summary of pylint changes for pull request 6325:
Abbreviated pylint report for src/python/WMCore/WMSpec/Steps/Executors/CMSSW.py follows:

Line 14 W0611 Unused import time
plus 3 total warnings
plus 65 comments on code style

No changes to unit tests for pull request 6325

24h is what was recommended, I actually don't know exactly what this timeout does...
Also fixed the last spurious import.

Btw, I feel a bit uncomfortable merging this, but I also feel uncomfortable leaving it out as it seems to ne absolutely required for Tier0 operations (or also CompOps operations) at CERN. I started a HN thread about whether this is safe to merge
https://hypernews.cern.ch/HyperNews/CMS/get/edmFramework/3572.html
maybe we should push a bit more to get an answer on this...

Yeah, to be honest I'm not really confident this problem happens in central production at CERN. Is Tier0 being locally mapped to cmsprd or cmsprod?

Maybe on the timeout question, ask on the wan-access HN.

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/804/ for details

Maybe it doesn't happen to CompOps because they don't run much on the CERN cloud resources. But once you do you'll be inside the same pilots as the Tier0 jobs, I very much doubt that you won't be affected by this. CompOps using T2_CH_CERN/LSF is a different question, but that use should go down to almost zero over time.

Anyways, the way this was explained to me was a problem between CMSSW, the gsi plugin authentication handling inside xrootd and EOS, so to first order everyone running at CERN should be affected.

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/808/ for details

No pylint warnings for pull request 6325.

Unit test changes for pull request 6325:

WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testProductionMultiQueue changed from success to failure

Previously working unit tests have failed!

retest this please

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/814/ for details

No pylint warnings for pull request 6325.

Unit test changes for pull request 6325:

WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testSetBlock was added. Status is success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testXSetBlock was deleted. Prior status was success
WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testProductionMultiQueue changed from success to failure

Previously working unit tests have failed!

retest this please

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/817/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/820/ for details

No pylint warnings for pull request 6325.

Unit test changes for pull request 6325:

WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testSetBlock was added. Status is success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testXSetBlock was deleted. Prior status was success
WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testProductionMultiQueue changed from success to failure

Previously working unit tests have failed!

test this please

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/846/ for details

No pylint warnings for pull request 6325.

Unit test changes for pull request 6325:

WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testConflicts changed from success to failure

Previously working unit tests have failed!

test this please

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/848/ for details

No pylint warnings for pull request 6325.

Unit test changes for pull request 6325:

WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testConflicts changed from success to failure

Previously working unit tests have failed!

test this please

Modify the patch to only implement the workaround at CERN.

@ticoann @ericvaandering , this is ready to be merged, can you review it please ?

patch looks fine to me.


Inverted joystick enumeration

Linux Mint 17.2 (kernel 4.1.6).
Having a problem where all the joystick devices provided by udev are enumerated from last-to-first by GLFW.  So if I have 4 joystick devices, GFWL will expose the highest enumerated /dev/input/js* (or /dev/input/event*) device provided by udev as joy[0]/GLFW_JOYSTICK_1.
Consistently able to produce this output:
/dev/input/js0  ->  /dev/input/event19  ->  GLFW_JOYSTICK_4 (joy[3])
/dev/input/js1  ->  /dev/input/event20  ->  GLFW_JOYSTICK_3 (joy[2])
/dev/input/js2  ->  /dev/input/event21  ->  GLFW_JOYSTICK_2 (joy[1])
/dev/input/js3  ->  /dev/input/event22  ->  GLFW_JOYSTICK_1 (joy[0])
Reproduced on three machines running Mint 17.2, using both Hidromancer and the kernel xboxdrv.  Built GLFW from source using master; BUILD_SHARED_LIBS ON, RelWithDebInfo.
I haven't experience this behaviour with any other application.  evtest, jstest-gtk and every other game and API (SD2/SFML/libudev) seems to address them in the correct order.  I verified GLFW_JOYSTICK_n macros are all correct.

So GLFW doesn't guarantee any ordering of joysticks, and it looks like SDL mostly returns them in the order you desire by accident. SDL guesses a bunch of file names in ascending order while GLFW uses readdir and regexec (and it's a todo in SDL to do it the GLFW way).
Are you suggesting that the indices should always match, even if it leaves holes in the index space on init?

Well, I think making a case for ordering is easy enough.  Joystick[0]/GLFW_JOYSTICK_1 should always represent the first joystick enumerated by the system.  With hidromancer, xpad (kernel) and xboxdrv, the driver reserves a block of devices and assigns the each physical device that connects to it the next available js*/event* device within the block in ascending order.
So, for example, if you connect a usb xbox receiver, 4 devices become exposed to a GLFW application.  As far as GLFW can see all the devices are valid, but the user probably only has one controller connected.  If the enumeration is inverted and the application assigns GLFW_JOYSTICK_1 for, say, menu navigation.  The user would then have to connect 3 additional controllers to the receiver, using the 4th, in order to 'be' player one.
Making a case for literal mapping is a little trickier, but I think it's best that input API's expose the functions of the driver/hw as directly as possible with regard to crossplatform uniformity.  This is one of the reasons that I prefer GLFW to SDL and SFML.  SDL makes some ridiculous assumptions and implements them with dirty hacks (axis remapping and obscuration) which break a the uniform model.  SFML obtusely limits all hardware on all platforms to the worst joystick api on them all (mmsystem.)
If I create a udev joystick device under linux with axes 0-20 enabled, GLFW exposes all the axes without trying to second-guess the intent of the driver.  Through GLFW all axes are available and enumerated just as the driver provided them.  To me this is the only sane behaviour, since only the driver knows how the hardware works.  It's not much of a jump to apply the same principals to device enumeration as to axis/button enumeration.
Another good case for literal mapping of OS enumerated devices might be hotplugging.  The onus is already on the OS to handle the enumeration of devices as they are connected and lost in no predictable order.  We already have to consider this a reliable subsystem, and in the case of linux, the I've found the behaviour of udev to be simple and reliable.
I think the case of having index 'holes' on init will be somewhat uncommon.  This is only expected to happen when, say, player1 disconnects, but player2 remains.  This might be because batteries are being swapped out, in which case, reconnecting the device will restore it to the first available index anyway.  Hidromancer has a mechanism to selectively change to any available udev vport with a block, but otherwise, the behaviour would be consistent with OS enum and most (if not all) consoles.

Ah, you want it because the USB Xbox receiver driver lies. Thank you.

Not really, no.
Please do me the dignity of reading the entirety and conversing with me about this.  Giving me a condescending brush-off is inappropriate.
Anyway, the conclusion you've reached about this bug/proposal is incorrect.
If I need to qualify my position further, what M$ has done the landscape of input devices is cancerous.  By no means am I looking for a hacky xbox-specific rectification so I can directly support a terrible way of handling input.  I'm interested in simple, crossplatform unified support of every reasonably-possible joystick that someone might connect to a computer.
Hidromancer, which I mentioned, is a bluetooth DS3 driver which I have written from the ground up.  I've looked at the source for xpad and xboxdrv.  I've also written [linux] input drivers for the wiimote, wiiu-pro, DS4 (partially), and a few random old usb/db15 joysticks that I just cant stop loving, heh.
  This bug, however, seems to be affecting every joystick in my system, as long as there are multiple joysticks, GLFW seems to be inverting the OS recognized order of them.
I can probably even find time to fix this, provided we can agree on a consistent and rational behavioural definition for device enumeration.
Cheers

I didn't brush you off; I summarised the cause of this bug report, which is that the driver exposes device nodes for non-existent joysticks.
It's easy to fix. I'll just sort the joysticks after initial enumeration. The ordering of inotify events should ensure that hotplugged receivers will get their fake device nodes in the desired order as well.
You might want to know that I will be adding axis/button remapping in this release so that Xbox controllers and compatible devices will have the same number of buttons and axes in the same order across supported platforms.

Great, sorting the identified devices should resolve the ordering for the most part.  Thanks!
I would like to point out again, however, this is not endemically an xbox receiver issue.  It has nothing to do with drivers or hardware lying, per se.  I have 3 Logitech  F710 Gamepads, each connects to their own USB port.  If I connect them one at a time, the third one I connect (which is /dev/input/js2) still ends up being recognized by GLFW as GLFW_JOYSTICK_1.  In this case there are no device nodes being block-reserved -- just one gamepad, one USB port, one driver instance, one device node each.
There always exists the possibility of holes in the enum, depending on when a device is lost.  Since, again, the OS bears the onus of enumerating, it still seems to me to best to trust the OS to handle it, rather than reimplement the wheel.
I actually really like using GLFW for input becuase it properly exposes the driver as a flat list of buttons and axes without performing any dirty hacks.  Now... am I misunderstanding this, or is the next version you mentioned going to essentially overwrite the driver defined axis/button enum and break the flat model?

I understand that it's not specific to your receiver device and that the bug applied to any set of more than one joystick.
I know that holes are unavoidable once you start plugging things out, but due to a sentence in the GLFW API spec I need the initial set to be packed.
The remapping will only apply to a specific list of devices that I know will be expected to behave like an Xbox controller. Anything not in that list will not be affected and will work exactly as before. Initially, if anything, the list will be too short and we will have to crowd-source the necessary IDs.

The reason for this remapping is so that games written for Xbox-like controllers on one platform will behave as expected when compiled on another. Actual joysticks and wheels have such variance that any game making use of them need to be able to handle that anyway.


Navigation to other View Controllers using Segues

I implemented the project into mine. Set everything up but could not get the menu controller to navigate to any other view controllers except for the first one in my list which by following the example would be similar to the "Second View Controller". I then created segues in the menu vc class and used the performSegueMethod. This method worked except once the other view Controllers were presented I could not use the swipe gesture nor could I bring up the menu by tapping the "menu" nav bar button.
Please let me know any suggestions you might have to get this to work. Thanks!

It is hard coding something like
    DEMOSecondViewController *secondViewController = [self.storyboard instantiateViewControllerWithIdentifier:@"secondController"];
    navigationController.viewControllers = @[secondViewController];
}

self.frostedViewController.contentViewController = navigationController;
[self.frostedViewController hideMenuViewController];

@MarlowCharite, will it cause some memory/state change issues, since we already initialized the view controllers, but when we a clicking on the side menu, it is not using other existing controllers, but create a new one and present it?


Workflow lifecycle hook plugin

Hi Argo maintainers! Thank you so much for building such an awesome workflow solution for Kubernetes!
I'm a maintainer of Kubeflow Pipelines (KFP).
Kubeflow Pipelines uses argo workflows under the hood as the workflow engine, see https://github.com/kubeflow/pipelines#acknowledgments.
Use Cases
TLDR
Realized I wrote too long, so TL.DR here
We want a mechanism that allows a external plugin service way to hook deeply into argo workflow task or probably pod lifecycle:

offload caching decision to an external system
allow extra placeholders using data from an external system
block workflow execution until an external system has finished logging some metadata for one task

The external system generates argo workflow spec, so it works for us if above features require changes in workflow spec.
Details
Kubeflow Pipelines' mission is building a platform for reusable end to end ML workflows.
What's special about ML workflows is that people care a lot about data, e.g. where the data come from, what hyper parameters we used for a model, how good is a model? These are questions we care about.
Therefore, KFP has standardized on using https://github.com/google/ml-metadata (MLMD) as its metadata store. Additionally, we do not only want to support users writing code in their pipelines to record those metadata. We also want to auto-log any metadata information the workflow system already knows (like artifact url, name...).
We have already achieved this goal by building metadata writer -- a service that watches argo workflows and pods, then auto log those information into MLMD.
Continuing on this path, we further want to allow using data from MLMD for workflow orchestration. For example, tasks can use placeholders that refer to information in MLMD.
We can insert tasks into user workflows to read from MLMD and expose them to argo, but given the fact that our metadata-writer watches argo workflows to record metadata, there's no guarantee any data is available in MLMD before the task that requires it.
When thinking about building a future KFP with full integration of MLMD on top of argo, we'd want a mechanism that allows an external plugin service to hook deeply into argo workflow task lifecycle:

offload caching decision to the plugin
allow extra placeholders
block workflow execution until the plugin has finished logging some metadata for this task

For clarification, KFP's current architecture is like:

Users write workflow definition in KFP DSL
KFP Compiler compiles the DSL code to argo workflow spec
User uploads the spec to KFP DB and triggers runs by KFP API, the runs will be created as argo workflows in the cluster.

Summary
KFP needs two type of lifecycle hooks:

prehook - before a task runs, argo workflow controller sends a request to the plugin service. The request should contain all information related to this task.  The plugin service should return

information about additional placeholders specific to this plugin service
caching decision and what artifacts/parameters to reuse

posthook - after a task completes, argo workflow controller sends a request to the plugin service. The request should contain task spec and results. The plugin service should return when it finishes metadata logging. Note, argo workflow controller should block on the request until it proceeds with tasks depending on the current one, because the plugin service could be doing sth that affects the next steps.

Potentially, to avoid disrupting non KFP argo workflows, we can configure argo that only workflows with a certain label will be sent to the lifecycle hook.
This is a very early discussion on one primitive feature that can enable us to build what KFP wants on top of argo. I am willing to contribute if we can get consensus on the enhancement proposal.
What do you think about this? Do you have alternatives (existing features, different mechanism to integrate?) you'd suggest instead?
Looking forward to your response! : )

Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

My first thought is how would you do lifecycle hooks today:

We emit Kubernetes such as WorkflowRunning which another controller could pick up on and perform processing based on this.
Any workflow could have an initial step and an exit handler to do the processing.

I'm guessing you've considered these already - so I'm interested to know how they don't fit the bill, or how you want to do things differently.

Hi @alexec, maybe I wasn't clear enough. What we want is lifecycle hooks for tasks, not workflow.
Above mentioned feature request is an early strawman proposal, if there are any way existing today or any other way you may design that can let us manage task lifecycle with our requirements, we are okay to them too.

I just built a POC of one type of interface I'd want for caching: Allow an init container in a task to decide whether the Pod should be cached, if it should be cached, the init container can mark the Pod as skipped and emit previous artifacts loaded from a database.
https://github.com/kubeflow/pipelines/blob/mlmd2/contrib/ir/samples/05_caching.yaml
I used a few hacks to achieve this with argo today:

Let init container emit outputs by adding workflows.argoproj.io/outputs annotation to its own pod using data saved previously.
Let the init container exit with 1, so main containers are skipped.
Configure this step to continueOn: { error: true }, so that the error doesn't stop the entire workflow.

This seems to be enough for my caching requirement. What do you think about this hack?
Can we adjust it to make an official interface in argo?

Regarding Pod exit handler, it probably works for me simply generating two argo steps for each user step, so that we put exit handler into a single argo step.

I think the kind of caching you're talking about could be done using the new memorization feature.
Kubernetes events are also emitting for node completion. If they were also emitted on start, would that help?

I think the kind of caching you're talking about could be done using the new memorization feature.

I am aware of the new memoization feature in argo, but we are opinionated for KFP caching behavior:
caching data should come from MLMD and we need some lifecycle handling even when a step is cached. I don't think we will ever adopt argo memoization.
/cc @Ark-kun
What's your opinion about this?

Kubernetes events are also emitting for node completion. If they were also emitted on start, would that help?

Rethinking over this, I don't think that will be helpful, the primary reason we ask for a lifecycle hook for node start event is to handle caching. Readonly event doesn't work for us.
Actually, thinking carefully about features I want to implement in KFP, I think this argo behavior in this POC: https://github.com/kubeflow/pipelines/blob/mlmd2/contrib/ir/samples/05_caching.yaml is enough for us to build on top. We do not have any feature requests beyond that.
Do you have any concerns with this HACK to implement third party caching behavior in initContainers? Can we rely on it?

Can we rely on it?

Short answer - no. These reads an annotation which is an implementation detail which we plan to remove to improve our ability to have larger inputs and outputs.
I think we want to support lifecycle hooks in Argo Workflows that can be configured at a namespace-level and controller-level.
I'm going to assume you need to run a container to do the processing. This would be both simple and flexible.
We already kind of have hooks: the init container runs before the main container and the wait container runs afterwards.
What if you could configure a preTask and postTask images at namespace or controller-level. These run before the init container, and have the inputs and outputs mounted as read/write volumes?
More broadly, I think you, me, @jessesuen and @Ark-kun should get on a Zoom so we can learn more about KFP roadmap and get alignment.

Short answer - no. These reads an annotation which is an implementation detail which we plan to remove to improve our ability to have larger inputs and outputs.

Understood, makes sense for supporting larger inputs and outputs.
Let me think about if we need an official supported interface in argo, what would be best for us.
Off the top of my head, lifecycle containers align with what I try to do.

More broadly, I think you, me, @jessesuen and @Ark-kun should get on a Zoom so we can learn more about KFP roadmap and get alignment.

Yes, agree we should proceed with zoom. I'll organize KFP side people who should attend.

Hi @alexec!
I just came up an idea with an alternative feature that can enable what we want to achieve. I want to get some early feedback from you.
Feature request: Tasks can consume and return artifacts by json metadata with its url.
To be specific, a task can consume a previous task’s artifacts by json representation of the artifact, e.g. for a minio artifact, it can look like (whatever format argo is emitting today)
{
    'name': 'hello-world',
    'bucket': 'ml-pipeline',
    'key': 'path/in/the/bucket/object-name',
    'secret': { 'name': 'pipeline-minio-secret' }
}

Similarly, a task can return artifacts by returning this json representation.
This basic primitive feature will enable us to fully encapsulate a normal argo task by control steps.
A KFP step can be translated to

Driver step that reads artifact metadata from MLMD, make caching decision, and outputs argo artifacts by artifact metadata.
Executor step that runs the KFP step's user container.
Publisher step that consumes executor step's outputs by metadata and publishes them to MLMD

In this way, we can control exact runtime semantics we want to expose to KFP users, while utilizing argo for workflow orchestration.
Current status: I'm discussing my design with my teammates, once we reach an agreement, I'll reach out to you for further discussion.

I'm not sure I fully understand your idea, but is it similar to #4341 ?

@alexec Definitely, that sounds like a step in the right direction.
My feature request will be supporting placeholders like {{inputs.artifacts.name.key}}.
And output an artifact by key in a container
outputs:
  artifacts:
  - name: hello-param
    keyFrom:
      path: /tmp/hello_world.txt

Let's get a meeting with @Ark-kun . Do you need me to drive this?

@alexc Thanks! I'll get my design reviewed internally first next week. I can drive the meeting with you after that.

I've got a round of feedback internally, I'll soon update the feature requests to reflect latest discussion.

A brief update,
We are designing and prototyping KFP v2 and our current design heavily relies on argo existing features to add the lifecycle hooks.
The gist is that

Inject a Control Pod before each KFP task/sub-graph
The control pod can do things like, reading data from an external service; publishing data to an external service; making cache decisions; data aggregation after parallel for, etc.
Each KFP task will be controlled by the control pod using podSpecPatch, so a lot of behavior can be dynamic and decided by the control pod. To clarify, "controlled" can mean that we could let the control pod decide things like nodeSelector, resource requests etc, so these information must be decided before the Pod starts, thus control logic cannot run inside the same Pod via hacks like entrypoint rewrite.

I've made some design and POCs, now we are more sure this approach can let us implement our desired features on top of argo.
The only problem left is that, the extra pods are a heavy overhead. Each Pod will take a few seconds to be scheduled, started and finished.
So the feature request becomes two options I can think of:

https://github.com/argoproj/argo/issues/795
https://github.com/tektoncd/community/blob/main/teps/0002-custom-tasks.md

I understand the concerns about http template. I think it's a nice fit as an integration point for extending argo workflow functionality use-case, but there are some challenges making it useful/non-problematic for other use-cases.
So what do you think about a very different feature in Tekton that would also allow us to plug in an operator to implement what we want?
/cc @alexec @jessesuen @Ark-kun

@jessesuen I’ve proposed non-pod tasks as part of multi cluster. But I think we might want to consider a plugin system instead. That would allow people to write Golang plug-ins that becomes part of the workflow lifecycle, and implement things like template executors.  I believe that pod, http, and data templates could be implemented this way.

With https://github.com/argoproj/argo-workflows/issues/795 implemented, I think we can close this.

I think this has been completed by #7582 and #7256