findBySecondaryIndex returns array not ordered by versionstamp

My expectation was that using findBySecondaryIndex would return an array ordered by versionstamp, but it is not the case in my test on local Deno KV (untested on Deploy)
This:
const db = kvdex(kv, {
    apps: collection(GroupsModel, {
        indices: {
            urlid: "secondary",
            timestamp: "secondary",
        },
        serialized: true
    })
});

  const allGroups = await db.apps.findBySecondaryIndex(
      'urlid',
      urlid,
      {
          reverse: true
      });

returns the groups in this order (not the latest group is in the middle):
[[
   {
    id: "fabf1625-9fb4-4d8a-adb6-269b4b3278ab",
    versionstamp: "00000000000002330000",
    value: {
      groups: [],
      urlid: "gt3h48wks7dc1m",
      timestamp: "2023-11-26T09:37:17.791Z"
    }
  },
   {
    id: "97416ee8-921b-4b3d-8239-ac786d5321e2",
    versionstamp: "00000000000001ee0000",
    value: {
      groups: [ ],
      urlid: "gt3h48wks7dc1m",
      timestamp: "2023-11-26T09:37:08.877Z"
    }
  },
   {
    id: "9449716e-d7a8-4659-9d9d-2ad14c3bac23",
    versionstamp: "00000000000001f70000",
    value: {
      groups: [],
      urlid: "gt3h48wks7dc1m",
      timestamp: "2023-11-26T09:37:12.372Z"
    }
  },
   {
    id: "81fdf6e0-a55d-4eab-8001-55ec37d4987b",
    versionstamp: "00000000000002270000",
    value: {
      groups: [ ],
      urlid: "gt3h48wks7dc1m",
      timestamp: "2023-11-26T09:37:16.462Z"
    }
  },
   {
    id: "81831af8-4f1e-4699-adfc-31b2a8bd9d77",
    versionstamp: "000000000000024a0000",
    value: {
      groups: [],
      urlid: "gt3h48wks7dc1m",
      timestamp: "2023-11-26T09:52:14.424Z"
    }
  },
   {
    id: "2fe987e3-512f-4960-ab69-8c0bacc8aeef",
    versionstamp: "000000000000021a0000",
    value: {
      groups: [  ],
      urlid: "gt3h48wks7dc1m",
      timestamp: "2023-11-26T09:37:15.293Z"
    }
  },
   {
    id: "2d2bbea6-47e0-481e-b131-e6e80175b902",
    versionstamp: "00000000000002090000",
    value: {
      groups: [ ],
      urlid: "gt3h48wks7dc1m",
      timestamp: "2023-11-26T09:37:14.091Z"
    }
  }
]

Each entry is added with add().
This makes it impossible to use limit: 1, reverse: true consistently as the order is not (as far as I can see) predictable.
Is there something I'm missing?
I will use the workaround to retrieve everything and filter compare the timestamps, but it will be costly when users have created many versions.

This is expected behaviour, and I'll explain why:
Almost all methods that handle multiple documents use kv.list under the hood (except for findMany()). This means the documents you get back when using getMany or findBySecondaryIndex() are ordered exactly like they are stored in KV. Values in KV are not ordered by versionstamp, but by their key. The default id setter for kvdex when using add() is crypto.randomUUID(). So the documents are actually ordered by random at default. If you want to have them ordered by insertion timestamp you can easily set a custom id generator in the collection options. I recommend using ulid, which you can find in the standard library for Deno. I'll provide an example:
import { collection, kvdex, model } from "https://deno.land/x/kvdex/mod.ts"
import { ulid } from<EMAIL_ADDRESS>
const db = kvdex(await Deno.openKv(), {
  strings: collection(model<string>(), {
    idGenerator: () => ulid(),
  }),
})

Thanks a lot for the explanation and detailed solution! I now recall seeing ulid() used in examples 😅
It would good to add this expected random ordering behavior to the readme, as it might surprise others.


Have a configurable timeout for connecting to snowflake before retrying

add retry logic for the connection promise
better retry logic handling
Add timeout to the connection attempts

wrong repo


Merge conflict fix

Done a reverse merge from the Roll20 Master, and then reapplied my updates. Seems to have resolved the conflicts (which seem to have come about with something to do with #1858.  All good now.

Please can this pull request be merged asap.  It replaces #1857, and #1853, #1854, #1855, & #1856 are to a degree dependent upon it (this is the library release for the rest of the RPGMaster suite...).


[SPARK-47085][SQL] reduce the complexity of toTRowSet from n^2 to n

What changes were proposed in this pull request?
reduce the complexity of RowSetUtils.toTRowSet from n^2 to n
Why are the changes needed?
This causes performance issues.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Tests + test manually on AWS EMR
Was this patch authored or co-authored using generative AI tooling?
No

Could we share some performance results in this section? It appears that replacing the while-loop with map/foreach is the only change made.

@yaooqinn yes this reduce the complexty to O(n) as Seq.apply is O(n)

Thanks, @igreenfield, merged to master.
Can you make backport PRs to the 3.5 and 3.4 branches?

@yaooqinn Yes I will create.

Thank you @dongjoon-hyun.
Hi @igreenfield, can you help update the jira side as @dongjoon-hyun suggested?

Hi @yaooqinn @dongjoon-hyun Done.

Unfortunately, this part is still missing, @igreenfield .

provide some background why this is a regression at Apache Spark 3.4.0.

Does this PR actually result in an improvement ? Seq.apply is expensive only if it is not an indexed seq.
The change itself is reasonable, but it looks like current usages pass in an IndexedSeq - and so should not be expensive ?
If yes, let us not backport it.

So, we are good, @mridulm ? 😄


Unable to save/sync while connected to server in debug mode

When I start my server I can save files locally and they are synced to the server's JCR just fine.
When I put a break point into a JSP and enter debug mode, run a page that uses the component, my debug view in Eclipse works just fine.
When I am connected in debug mode, and make changes to a file, then hit save, it attempts to sync but then fails the operation and gives me this message:
Could not publish to the server.
java.lang.NullPointerException
I am using Luna. If you need any other information let me know.

Thanks for the report. I've seen this as well, recently filed https://issues.apache.org/jira/browse/SLING-4374 .

The content sync issue has been fixed in the Sling IDE Tooling project ( see SLING-4374: Intermittent error when publishing content ) and will be incorporated in the next AEM Developer Tools public release.
If you'd like to build a fix yourself, please follow the instructions from Sling IDE Tooling - building from source.


Buy plan page

Notion card
https://www.notion.so/saeloun/Develop-the-Plan-details-page-5dae88a09c1f48b0a9d22707161ac81a
Summary

Designed Buy plan page
Functions for calculating amount added

Preview

https://www.loom.com/share/43341e2b03b64265a6a469053847759d
Type of change

[ ] New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Checklist:

[ ] I have manually tested all workflows
[ ] I have performed a self-review of my own code

Becasue of multiple files changed (rebasing issue) creating a new PR.


feat: Support for running a single integration test case

Feature Request
Describe the Feature Request
Support for running a single integration test case
Example:

Select this use case to run directly

Since the integration test a move file is a test case, there is no need to support running a certain step


opening empty text field gives error

When the text field is empty, you open the code editor and press 'OK'  without giving input, the editor throws an error and places a � in the input area. It also throws an error
tinymce.min.js:6 Uncaught IndexSizeError: Failed to execute 'setStart' on 'Range': There is no child at offset 2.

it occurs the tinymce option forced_root_block : false is set

Hey @sowirepo
i can reproduce.. please provide more details, code or anything else and steps to reproduce.
try also use: forced_root_block : '' in your tiny_mce config option.
thanks
thomi

Good morning @thomich
We had some problems too with forced_root_block:'p'
(When pressing ok while not being in a node it has every time created a new p node)
But we found a little hotfix:
In your function submit() block we added (at the beginning):
var forced_root_block_orig = editor.settings.forced_root_block;
editor.settings.forced_root_block = "";

and at the end of that function:
editor.settings.forced_root_block = forced_root_block_orig;

forced_root_block is recommended by tinymce and we like it for the non-dev users!


Building an openwrt factory image

Hi Ron,
Thanks for sharing your findings; its clear you put a lot of time and effort it in - and its saved a lot of other engineers and hackers a lot of time.
I'm trying to build a factory image for another device; with identical hardware to the HooToo. Its the RAVPower RP-WD01. It does not have an Ethernet port, so I need to flash it and have Wireless work straight away.
I can do this if I can build a 'factory' image with a sysupgrade image with wireless ready to go. I have a HooToo TM02 to practice on. I have pulled revision r42649 and applied your patch; I have baked in a wireless config.
You mentioned in your posts on openwrt forums:
"Factory firmware update mechanism is not compatible with standard OpenWRT images. I’ve decoded the factory image format..."
"This is the "factory" upgrade image to be used by the factory HooToo software to do the initial upgrade to OpenWRT.It is based on the same binary image as the normal OpenWRT sysupgrade image but has been re-packaged with a loader, CRC, etc. such that the factory HooToo software will accept it. The factory HooToo software is filename sensitive and will reject the file if it's too different from what it expects. If the TM02 rejects this name try changing it to "fw-WiFiPort-HooToo-TM02-2.000.018"
Can you offer any direction on how to develop the loader and the CRC required, also the filename check?
Thanks
Ian

The HooToo uses a structure that's very different than most. It has binary images embedded with a shell script that's very specific to the HooToo Linux configuration. I'm 100% sure it's totally incompatible with anything else. I deduced all this with a hex editor and taking a standard HooToo image apart and analyzing what it all does. Then I wrote a new script and after a bunch of hours debugging got it to work. I would start by getting the hexeditor out and looking at the RAVPower images. For a one off, it's not worth all the work in my opinion. I recommend finding the serial port connections and getting into both it's uboot (if it uses that) and also the Linux console. Nearly every embedded Linux I've ever looked at already has the software tools on-board to reprogram the FLASH with a new Linux image - get into the console and find those and you'll be set.
Cheers,
Ron

Hi Ron,
Thanks for the tip; I didn't want to jam a screwdriver into it just yet. I dug around in a hex editor and took a look. Sure enough, its a shell script stub which checks the CRC, performs a backup, quiesces the system and then starts the flash:
CRCSUM=1244231419
OEM=POWER7
VERSION=2000030
VENDOR=RAV
PRODUCTLINE=SD
SKIP=173
TARGET_OS="linux"
TARGET_ARCH="arm"
DEVICE_TYPE=WD01
CPU=5350
Once I got the CRC at the start figured out it just worked!!...
I have since added SDHCI support, then cloned and build an exfat kernel driver based on Yuri's driver over at https://github.com/dorimanx - it'll mount and r/w 64GB SD Cards. Now I can finally backup my DSLR photos to USB on the move. I'm still sorting out the LED's, but I'll write this up and share the scripts later in the week.
Thanks for the tip
Ian

Hi Ian,
Well, that’s a first for me! I haven’t seen any other 5350 based devices using a shell script like these do. Sounds like they may be using the same software contractor since it’s very similarl. I am looking at writing a “make image” tool to automatically make these “factory” images as part of the OpenWRT builder but at the moment my time is consumed with higher priority stuff (like looking for a new job). I do have a shell script though that does the job. I don’t mind sharing it. It’s attached. It’s under the “Open Commons” license so please share any progress you make with it.
Btw, I just got a TM04 which is really an awesome unit and it has an SD card, a built-in battery, etc and there is a sticker inside that says RAVxxxx so I’m thinking that your unit and the TM04 may be made by the same company.
Cheers,
Ron

On Mar 23, 2015, at 2:45 PM, Ian Clegg<EMAIL_ADDRESS>wrote:
Hi Ron,
Thanks for the tip; I didn't want to jam a screwdriver into it just yet. I dug around in a hex editor and took a look. Sure enough, its a shell script stub which checks the CRC, performs a backup, quiesces the system and then starts the flash:
CRCSUM=1244231419
OEM=POWER7
VERSION=2000030
VENDOR=RAV
PRODUCTLINE=SD
SKIP=173
TARGET_OS="linux"
TARGET_ARCH="arm"
DEVICE_TYPE=WD01
CPU=5350
Once I got the CRC at the start figured out it just worked!!...
I have since added SDHCI support, then cloned and build an exfat kernel driver based on Yuri's driver over at https://github.com/dorimanx https://github.com/dorimanx - it'll mount and r/w 64GB SD Cards. Now I can finally backup my DSLR photos to USB on the move. I'm still sorting out the LED's, but I'll write this up and share the scripts later in the week.
Thanks for the tip
Ian
—
Reply to this email directly or view it on GitHub https://github.com/wingspinner/HooToo-Tripmate-HT-TM02-OpenWRT/issues/6#issuecomment-85210114.

Ian,
A follow-up question in addition to my previous reply…
Isn’t there already a exFAT and VFAT kernel driver? I’m curious as to why you didn’t use that. As well, on the SDHCI support did you use the OpenWRT kernel driver or is this something you developed on your own?
Lastly, I’m curious about what you did with the loader script. I found a few things, first, the existing UBoot partition isn’t compatible with OpenWRT (at least if you don’t want to sacrifice a substantial amount of the FLASH. As well, I found that I had rewrite the secondary script for various reasons. Are you saying you just plopped in a sys upgrade image into the HooToo image, changed the CRCSUM and off it went or that you started with my factory image.
Just curious and interested.
Thanks for the note!
Cheers,
Ron

On Mar 23, 2015, at 2:45 PM, Ian Clegg<EMAIL_ADDRESS>wrote:
Hi Ron,
Thanks for the tip; I didn't want to jam a screwdriver into it just yet. I dug around in a hex editor and took a look. Sure enough, its a shell script stub which checks the CRC, performs a backup, quiesces the system and then starts the flash:
CRCSUM=1244231419
OEM=POWER7
VERSION=2000030
VENDOR=RAV
PRODUCTLINE=SD
SKIP=173
TARGET_OS="linux"
TARGET_ARCH="arm"
DEVICE_TYPE=WD01
CPU=5350
Once I got the CRC at the start figured out it just worked!!...
I have since added SDHCI support, then cloned and build an exfat kernel driver based on Yuri's driver over at https://github.com/dorimanx https://github.com/dorimanx - it'll mount and r/w 64GB SD Cards. Now I can finally backup my DSLR photos to USB on the move. I'm still sorting out the LED's, but I'll write this up and share the scripts later in the week.
Thanks for the tip
Ian
—
Reply to this email directly or view it on GitHub https://github.com/wingspinner/HooToo-Tripmate-HT-TM02-OpenWRT/issues/6#issuecomment-85210114.

Hey Ron,
openwrt trunk has kernel mode drivers for ntfs and vfat, but not exfat. I'm pretty sure there is a user land fuse based driver, but I didn't fancy the performance hit on a device like this if a kernel mode implementation is available elsewhere. Patching it into openwrt only took half an hour - yuri's driver is based on the GPL'd samsung code. I ran an overnight read/write test and got a sustained 4.7MB/sec transfer rate from SD to USB - comparable with Tuxera's driver which ships with the RAVPower.
It wasn't as straight forward as fixing up the crc and replacing HooToo's image; but not too far off. I didn't know there were any issues with uBoot and openwrt - so the quickest thing for me to try was writing the openwrt sysupgrade image into the firmware partition starting at 0x50000 - just as you might normally do. Since the mtd tool on the device doesn't seem to be able to target an offset - the simplest approach was to write out the first 3 partitions and concatenate them with my sysupgrade image - then write the whole lot back.
[    0.490000] 0x000000000000-0x000000030000 : "u-boot"
[    0.500000] 0x000000030000-0x000000040000 : "u-boot-env"
[    0.520000] 0x000000040000-0x000000050000 : "factory"
[    0.520000] 0x000000050000-0x000000800000 : "firmware"
[    0.550000] 0x000000150430-0x000000800000 : "rootfs"
cross you fingers, wait... wait.. and it booted :+1:
You mentioned wasting flash: so are all of those first 3 partitions needed? I assume your approach did away with them?

Ah, got it. I’d love to get hold of your exfat driver!
On the the upgrade. There are no issues with OpenWRT and uboot. However, every uboot has the MTD partitioning hard-coded into it and the HooToo devices use a modified version with different partitioning. It so happens that their kernel image partition is at 0x50000 but it’s a very short partition because after that is a “parameters” partition, then another partition, THEN the rootFS. It’s kind of messed up for OpenWRT purposes.  RAVPower may not be using the same hacked up UBoot that HooToo is though.
However, the listing you show appears to be the Linux boot kernel messages and that will always show the partitioning Linux is using which isn’t necessarily the same as what the firmware is using. In fact, unless you pass some partitioning info in the boot command line Linux/OpenWRT will pretty much ignore uboot partitioning. As long as you aren’t overwriting the “factory” partition” n some ways it’s a don’t care. Where it can be a problem is if you try to upgrade the kernel image via TFTP. In the case of the TM02 the kernel (firmware) partition is quite small because they separated the rootFS from it and that partition is only big enough for the bare kernel. HooToo uboot  will reject an OpenWRT sys upgrade image more often than not for that reason.
You may know this but you have a unit that still has the stock firmware you can find out what partitioning it uses via “cat mtd” in the proc directory.
In anyway, what you did is pretty much how I got OpenWRT on the TM02 the first time except I swapped out the uboot and used TFTP.
On another note, it appears that the RAVPower device and the TM04 may be the same unit or at least very similar. I just got one a couple of days ago and when I removed the RF shielding I found a sticker labeled “RAV-XXXXX” inside the daughter-card module. I found the my factory images and the trunk image  work without modification on it. That’s good news because it’s a very nice little unit with built-in 3000mah battery and both SD and USB ports for only $29.
In any case, I think I mentioned that I released my factory image builder tool on my GitHub repository. You may be able to use that in the future if you buy more devices. It generates HooToo compatible firmware upgrade images and replaces the uboot with one that has identical partitioning to OpenWRT as well as installing OpenWRT.
Thanks for the interesting discussion!
Best,
Ron

On Mar 25, 2015, at 7:47 AM, Ian Clegg<EMAIL_ADDRESS>wrote:
Hey Ron,
openwrt trunk has kernel mode drivers for ntfs and vfat, but not exfat. I'm pretty sure there is a user land fuse based driver, but I didn't fancy the performance hit on a device like this if a kernel mode implementation is available elsewhere. Patching it into openwrt only took half an hour - yuri's driver is based on the GPL'd samsung code. I ran an overnight read/write test and got a sustained 4.7MB/sec transfer rate from SD to USB - comparable with Tuxera's driver which ships with the RAVPower.
It wasn't as straight forward as fixing up the crc and replacing HooToo's image; but not too far off. I didn't know there were any issues with uBoot and openwrt - so the quickest thing for me to try was writing the openwrt sysupgrade image into the firmware partition starting at 0x50000 - just as you might normally do. Since the mtd tool on the device doesn't seem to be able to target an offset - the simplest approach was to write out the first 3 partitions and concatenate them with my sysupgrade image - then write the whole lot back.
[ 0.490000] 0x000000000000-0x000000030000 : "u-boot"
[ 0.500000] 0x000000030000-0x000000040000 : "u-boot-env"
[ 0.520000] 0x000000040000-0x000000050000 : "factory"
[ 0.520000] 0x000000050000-0x000000800000 : "firmware"
[ 0.550000] 0x000000150430-0x000000800000 : "rootfs"
cross you fingers, wait... wait.. and it booted
You mentioned wasting flash: so are all of those first 3 partitions needed? I assume your approach did away with them?
—
Reply to this email directly or view it on GitHub https://github.com/wingspinner/HooToo-Tripmate-HT-TM02-OpenWRT/issues/6#issuecomment-86062605.

I'll put together an exfat patch - it didn't need code changes, just the right makefiles to build. Regarding the flash partitions, did your loader blow away the the 'u-boot-env' and 'factory' partitions?

One last thing about exfat (and ntfs for that matter) - they are not automounted on openwrt. I looked into this and it looks like the project moved over to using ubox for hotplug automouting; this in turn uses a custom version of libblkid (called libblkid-tiny) that probes the device and uses a set of heuristics to establish the file system. The logic for exfat and ntfs are not present - but again it looks pretty straight forward to add. The full fat version of libblkid has them. I'll prod it a bit and see what happens

I missed the last paragraph of your last post, yer I was being lazy and posted the linux boot messages - so it looks like I was lucky with the flashing. Thanks for the tips!

No, it retains those. My loader installs a new uboot with the correct OpenWRT partitioning along with a sysupgrade image while retaining those partitions that are required. It also writes out a full backup of the original firmware to a USB Flash stick in case you ever want to go back. The u-boot-env partition is required by uboot and the factory partition contains parameters and calibration info for the wifi chip. If that get’s blown away there’s a good chance the wifi won’t work very well or not work at all. Or so RALink tells me. I’ve not actually done it to test it ;-) .
Cheers,
Ron

On Mar 25, 2015, at 12:27 PM, Ian Clegg<EMAIL_ADDRESS>wrote:
I'll put together an exfat patch - it didn't need code changes, just the right makefiles to build. Regarding the flash partitions, did your loader blow away the the 'u-boot-env' and 'factory' partitions?
—
Reply to this email directly or view it on GitHub https://github.com/wingspinner/HooToo-Tripmate-HT-TM02-OpenWRT/issues/6#issuecomment-86182611.


Reformat the whole code

reformat the code. Make it PEP8 compliant, good naming conventions

I am on it! :+1:


Started on multi-opcode functions

FILL OUT ITALICIZED SECTIONS
Description
This PR adds the base for multi-opcode arithmetic functions
Type of change

[ ] Bug fix
[X] New non-breaking change
[ ] New breaking change

How Has This Been Tested?
It has been thoroughly tested on my computer
OS : Arch Linux
Java Version: Unsure

@MH15 I believe I'm done with the commenting


[Do not merge] Add Repro fro #405

Code to replicate issue #405.
The repro crashes in 5.8.1 after 100 iterations:
Backtrace
    frame #0: 0x0000fffff696c584 libc.so.6`___lldb_unnamed_symbol3383 + 1348
libc.so.6`___lldb_unnamed_symbol3383:
->  0xfffff696c584 <+1348>: ldr    x8, [x6, #0x10]
    0xfffff696c588 <+1352>: cmp    x8, x2
    0xfffff696c58c <+1356>: b.ne   0xfffff696c888            ; <+2120>
    0xfffff696c590 <+1360>: ldr    x8, [x2, #0x10]
Target 0: (Repro) stopped.
(lldb) bt
* thread #2, name = 'NIO-ELT-0-#0', stop reason = signal SIGSEGV: invalid address (fault address: 0xfffdf00000a0)
  * frame #0: 0x0000fffff696c584 libc.so.6`___lldb_unnamed_symbol3383 + 1348
    frame #1: 0x0000fffff696e548 libc.so.6`__libc_calloc + 216
    frame #2: 0x0000fffff75c05b8 libdispatch.so`_dispatch_continuation_alloc_from_heap + 20
    frame #3: 0x0000fffff75d2b88 libdispatch.so`_dispatch_continuation_redirect_push + 316
    frame #4: 0x0000fffff7336690 libswift_Concurrency.so`resumeTaskAfterContinuation(swift::AsyncTask*, swift::ContinuationAsyncContext*) + 380
    frame #5: 0x0000aaaaaace1358 Repro`partial apply for specialized closure #1 in closure #1 in EventLoopFuture.get() at <compiler-generated>:0 [opt]
    frame #6: 0x0000aaaaaad1be70 Repro`merged generic specialization <()> of closure #1 @Sendable () -> NIOCore.CallbackList in NIOCore.EventLoopFuture._publicWhenComplete(@Sendable (Swift.Result<τ_0_0, Swift.Error>) -> ()) -> () + 92
    frame #7: 0x0000aaaaaad1bd94 Repro`merged generic specialization <()> of closure #1 @Sendable () -> () in NIOCore.EventLoopFuture._internalWhenComplete(@Sendable () -> NIOCore.CallbackList) -> () + 68
    frame #8: 0x0000aaaaaadc1390 Repro`SelectableEventLoop.run() [inlined] closure #4 () -> () in NIOPosix.SelectableEventLoop.run() throws -> () at SelectableEventLoop.swift:518:30 [opt]
    frame #9: 0x0000aaaaaadc138c Repro`SelectableEventLoop.run() [inlined] generic specialization <()> of NIOPosix.withAutoReleasePool<τ_0_0>(() throws -> τ_0_0) throws -> τ_0_0 at SelectableEventLoop.swift:29:16 [opt]
    frame #10: 0x0000aaaaaadc138c Repro`SelectableEventLoop.run(self=0x0000fffff0001170) at SelectableEventLoop.swift:517:21 [opt]
    frame #11: 0x0000aaaaaada60fc Repro`closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:parentGroup:selectorFactory:initializer:) at MultiThreadedEventLoopGroup.swift:97:22 [opt]
    frame #12: 0x0000aaaaaada60d8 Repro`closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(t=<unavailable>, parentGroup=<unavailable>, selectorFactory=0x0000aaaaaada631c Repro`implicit closure #1 () throws -> NIOPosix.Selector<NIOPosix.NIORegistration> in NIOPosix.MultiThreadedEventLoopGroup.init(numberOfThreads: Swift.Int) -> NIOPosix.MultiThreadedEventLoopGroup, initializer=<unavailable>, lock=NIOConcurrencyHelpers.NIOLock @ x19, loopUpAndRunningGroup=0x0000aaaaab16d7c0) at MultiThreadedEventLoopGroup.swift:118:41 [opt]
    frame #13: 0x0000aaaaaada8d64 Repro`partial apply for closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:parentGroup:selectorFactory:initializer:) at <compiler-generated>:0 [opt]
    frame #14: 0x0000aaaaaadaab38 Repro`thunk for @escaping @callee_guaranteed (@guaranteed NIOThread) -> () at <compiler-generated>:0 [opt]
    frame #15: 0x0000aaaaaaddbe2c Repro`closure #1 in static ThreadOpsPosix.run($0=0xaaaaab16d950) at ThreadPosix.swift:112:13 [opt]
    frame #16: 0x0000fffff695d5c8 libc.so.6`___lldb_unnamed_symbol3291 + 724
    frame #17: 0x0000fffff69c5d1c libc.so.6`___lldb_unnamed_symbol3659 + 12

Codecov Report

Merging #406 (8f9a39d) into main (d89a723) will decrease coverage by 0.13%.
The diff coverage is n/a.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #406      +/-   ##
==========================================
- Coverage   49.13%   49.01%   -0.13%
==========================================
  Files         108      108
  Lines        8841     8841
==========================================
- Hits         4344     4333      -11
- Misses       4497     4508      +11

see 2 files with indirect coverage changes


Add support for CVMFS_REPOSITORY_DATE

Please add support for CVMFS_REPOSITORY_DATE as an alternative to CVMFS_REPOSITORY_TAG in the mount parameters.
https://github.com/cernops/cvmfs-csi/blob/e96e011380031690d9667706f7a9783915d8d52c/pkg/cvmfs/cvmfsconf.go#L43-L45

Hi @innovate-invent, thanks for the suggestion! As I've already mentioned in couple of other issues/PRs you've created, there's #39, which updates and fixes a lot of issues.
The new version provides two config maps:

cern-cvmfs-csi-default-local ConfigMap is mounted as a subPath to /etc/cvmfs/default.local file, and
cern-cvmfs-csi-config-d ConfigMap is mounted to /etc/cvmfs/config.d directory.

The second one is per-repository configuration, where you could add the CVMFS_REPOSITORY_DATE parameter. Would this be sufficient for your use case?
Note that the new version drops support for specifying hash and tags in StorageClasses; for now all configuration is done via the mentioned ConfigMaps only.

It looks like I won't be able to use this version if this is the case. I need to specify different PVC for different application with different repo configs. I can't redeploy my entire CSI driver every time a different config is needed, that would break every running application.

It looks like I won't be able to use this version if this is the case.

In the following iterations we will try to get this per-PVC config feature back.

I can't redeploy my entire CSI driver every time a different config is needed, that would break every running application.

Sure, that's not what I'm suggesting at all.

How do you update the config without restarting the CSI driver?

@innovate-invent In v2, the ConfigMap that contains repo configs is mounted as a directory, so it can be changed even without re-deploying the driver.
https://github.com/cernops/cvmfs-csi/blob/03319490290e83da434948041e572516ab6bf719/docs/how-to-use.md#adding-cvmfs-repository-configuration.
For per-PVC config, it would be sourced in via spec.csi.volumeAttributes in PersistentVolume. In this case the config could be changed only by recreating the PV+PVC.

Ah, the driver watches the folder for configs?
I guess the issue at that point is that the ConfigMap is owned by the cluster and isn't available to the application deployment.

Ah, the driver watches the folder for configs?

Not the driver itself, it's actually just the regular Kubernetes' ConfigMap-as-a-directory machinery that is able to update the contents. The CVMFS client then picks them up by itself when mounting a repo.
In any case, this is a per-repository configuration, so I don't think this would fit your use-case since you want to have multiple mounts of the same repo with different CVMFS_REPOSITORY_DATE if I understood correctly?

I guess the issue at that point is that the ConfigMap is owned by the cluster and isn't available to the application deployment.

Well this really depends on how the driver is deployed, but the only requirement is that ConfigMap(s) must live in the same namespace as the driver itself.
N.B. we'll probably aim to have per-PVC config available in the next point release.

you want to have multiple mounts of the same repo with different CVMFS_REPOSITORY_DATE if I understood correctly?

Yes, or TAG.

N.B. we'll probably aim to have per-PVC config available in the next point release.

Thanks!

Hi @innovate-invent, this is now done in v2.2.0. Please check release notes in https://github.com/cvmfs-contrib/cvmfs-csi/releases/tag/v2.2.0. Closing.


Add Mode connector

Is your feature request related to a problem? Please describe.
Mode is one of the popular dashboard/analytics tool. More and more users are using it and requesting OpenMetadata to add that connector.
Describe the solution you'd like
Add Mode connector to OpenMetadata.
Mode has good REST API with docs here https://mode.com/developer/api-reference/introduction/
Describe alternatives you've considered
None

Test the full stack -> DBT + BigQuery (adding more data to the sources) + Mode dashboards + airflow lineage


Add Node 10.x to appveyor

The README says that it requires node 6, but I see 4.x in the appveyor config.


Add check for chws/vchw feature presence

Description
This pull request addresses the problems described at issue #3363, adding a check com.google.fonts/check/cjk_chws_feature.
To Do

[x] update CHANGELOG.md
[x] wait for all checks to pass
[x] request a review

Please see notes on #3363. We should confirm with @kojiishi before merging.

@simoncozens, I've converted this PR to a draft while we wait for feedback from @kojiishi

We should confirm with @kojiishi before merging.

@simoncozens, can you talk to Koji Ishii about this?

I don't know Ishii-san other than he was the original requester of the check. I'm happy with my implementation; I wasn't really checking its correctness, just wanted to make sure it did what the original requester wanted.

would you like to merge this without his feedback?

Yes, please. I’ll fix up the Changelog conflict.

Sorry I was slow to respond, but this looks great. Thank you for working on this!


BDS-60 Create/Build  NICC API - Draft Part 3

Added unit tests to service and cleaned up unnecessary code.

Test this please.


Feature/facet help

@NYULibraries/finding-aids Creating popover help here for the facets from locales.
Finishes #89950316

Coverage remained the same at 97.13% when pulling 83cfe8de3f567a324015ca54bb264510a1d9c2ac on feature/facet_help into 8bdc8be6668799c9dc2801f0553f3708cd10379b on development.

Coverage remained the same at 97.13% when pulling 83cfe8de3f567a324015ca54bb264510a1d9c2ac on feature/facet_help into 8bdc8be6668799c9dc2801f0553f3708cd10379b on development.

Coverage remained the same at 97.13% when pulling 83cfe8de3f567a324015ca54bb264510a1d9c2ac on feature/facet_help into 8bdc8be6668799c9dc2801f0553f3708cd10379b on development.

Coverage remained the same at 97.13% when pulling f6eff7406938eb6c6e6defba2f6d2d06cd0de4b5 on feature/facet_help into 8bdc8be6668799c9dc2801f0553f3708cd10379b on development.

Coverage remained the same at 97.27% when pulling 3ddd5fc8dc34ca27a6dc89ec8d3b0d68bc5ca009 on feature/facet_help into 69745aee38da4ede63aac4114e5131b70af45b53 on development.


Test shearing system

Draft PR, only to test the new shearing system.

@meetcshah19 I guess this can also be closed, can it?


fix(filters):  fix clean up of filter files being incomplete

"Filter records" are Visual Studio's name for "folders", and this project cleverly creates these folders to match the file LVGL pjt directory structure.
The problem:
LvglProjectFileUpdater project has 3 places where .vcxproj.filters files are loaded, records removed before a new set of filter records are added. However, for all 3 projects that it maintains, it is not removing all the filter records first as intended. The result of this is that the records that are not removed remain in the file and accumulate when the new filter records are added.
This PR fixes the source of the problem, as well as updates the .vcxproj.filters files that had duplicate filter records removed, and updated LvglWindowsSimulator.vcxproj since new files have been added into the LVGL repository since it was last updated.
It is confirmed that the new files result in a normal build.  (With the VS-emitted code warnings that have been around for a while.)
Resolves #91

Using "lvgl" instead of "lvgl" is by design for make sure we capture the folder, not the file with the same name.
So, we need to change the other logic in the generator to achieve that goal.
Kenji Mouri

Using "lvgl" instead of "lvgl" is by design for make sure we capture the folder, not the file with the same name.
So, we need to change the other logic in the generator to achieve that goal.

Good point!  Done!


docs: Added Component Review Template

Description
This is an updated template document for Component Reviews. With more detail around adding screenshots and checklists.
Type of change

[X] Documentation (update or new)

How Has This Been Tested?
Tested by auto test in GitHub for formatting.
Testing Checklist:

[X] Tested in latest Chrome

Checklist:

[X] My code follows the style guidelines of this project
[X] I have performed a self-review of my own code
[X] I have commented my code in hard-to-understand areas
[X] I have made corresponding changes to the documentation
[X] My changes generate no new warnings

Changes approved


Bug:TabNine and VIM extension may conflict

please complete the following information:

OS version: [e.g. Win 10, Mac Catalina]
Win 11
Editor: [e.g. vscode, intelij, sublime]
vscode
Editor version:
Version: 1.67.0 (user setup)
Commit: 57fd6d0195bb9b9d1b49f6da5db789060795de47
Date: 2022-05-04T12:06:02.889Z
Programming language: [e.g. js, ts, c++]
cuda
TabNine extension version:
v3.5.45
Engine version (could be located within Tabnine Hub):
4.4.6-49

Issue Details:
I install vim plugin on vscode. So when I input "glob..." waiting Tabnine complete it to "__global__", I push "Tab" and the word completed successfully but vim turns to VISUAL model. But if I push "ENTER", it still stays in INSERT model.
gz#14046

Hey @AutoColin , thank you for contacting us! @LeshemNoa is working on a solution, and we would love to have you validate it as soon as possible.

Hey @AutoColin! My name's Noa and I'm part of the Tabnine developer team.
I just released a fix for Vim's mode switching and incorrect cursor positioning after accepting an inline completion from Tabnine. The fix is available in version number 3.5.49.
I'd appreciate it if you could try it out and let us know what you think.
Thanks again for reaching out!
Noa
°°°

k.

Hi Noa, thank for your effort. I have tried the new release and I think the bug before has been corrected.
And also i wonder if your team will make Cuda support of Tabnine. For example, "cufftComplex" is a variable type of cufft library and this variable wont show auto-completion in Tabnine.

Hey,
Thanks for the useful feedback, we'll definitely take Cuda into consideration for additional technologies to support.
Have a great day!
Noa
°°°


Support RSA

available algorithms: rsa-sha1, rsa-sha256, rsa-sha384 and rsa-sha512
Hello, I need to support RSA algorithms like https://github.com/tomitribe/http-signatures-java,
so implemented it.

Any word on when this will be implemented? I'd like to use this gem in my own project but I need RSA support.

Here's an example test case to verify an invalid signed message.
diff --git a/spec/rsa_spec.rb b/spec/rsa_spec.rb
index fa41581..01763d7 100644
--- a/spec/rsa_spec.rb
+++ b/spec/rsa_spec.rb
@@ -14,6 +14,12 @@ RSpec.shared_examples_for "verifier" do
     context.signer.sign message
     expect(context.verifier.valid? message).to eq(true)
   end
+
+  it "rejects if a signed header has changed" do
+    context.signer.sign message
+    message["Date"] = "Thu, 12 Jan 2012 21:31:40 GMT"
+    expect(context.verifier.valid? message).to eq(false)
+  end
 end

 RSpec.describe "Using RSA" do

Thanks @hamakn and @evanminto — and sorry this took literally years to get to.


Fix typo in configuration example for Netlify Blob sessions

Description (required)
Netlify blob example had adapter incorrectly nested in the experimental  property instead of at the root of the configuration object.
Discord: sledsworth

Related issues & labels (optional)

Suggested label: typo/link/grammar - quick fix!

Lunaria Status Overview
🌑 This pull request will not trigger status changes.

Learn more
Lunaria automatically ignores changes on specific PRs by adding a ignored keyword in its title. Found: typo.
You can change this by either removing the keyword above from the PR's title, or modifying the ignoreKeywords property in your Lunaria configuration file.

Tracked Files

[!NOTE]
The notes below indicate what would happen if the pull request is merged when triggering status changes. Since a ignored keyword was found in the PR's title, the status changes indicated below won't be applied.

File
Note

en/reference/experimental-flags/sessions.mdx
Source changed, localizations will be marked as outdated.

Warnings reference

Icon
Description

🔄
The source for this localization has been updated since the creation of this pull request, make sure all changes in the source have been applied.

Hello! Thank you for opening your first PR to Astro’s Docs! 🎉
Here’s what will happen next:

Our GitHub bots will run to check your changes.
If they spot any broken links you will see some error messages on this PR.
Don’t hesitate to ask any questions if you’re not sure what these mean!

In a few minutes, you’ll be able to see a preview of your changes on Netlify 🥳.

One or more of our maintainers will take a look and may ask you to make changes.
We try to be responsive, but don’t worry if this takes a few days.


Color support detection problem in git bash on windows

I recenlty update msysgit (windows version of git) from version 1.9.5 to 2.5.3. This updated the terminal it brings along (git-bash) to "GNU bash, version 4.3.42(2)-release (x86_64-pc-msys)".
I since then lost all colors from Gulp output.
Using --color as a parameter brings color back.
I assume something have changed on the way git-bash exposes its color support, and Gulp detection is not finding out the correct answer anymore.
I don't know how color support detection works in Gulp, but I can say that $TERM is exposing "xterm-256", which should be enough to assume color support.

@sindresorhus this might be something that chalk needs to know about?

That should be enough, yes. You can see the test here: https://github.com/chalk/supports-color/blob/9434c93918301a6b47faa01999482adfbf1b715c/index.js#L61

@Qix-

@mlarcher would you mind doing
$ npm i supports-color
$ node
> var s = require('supports-color');
> console.log(require('util').inspect(s));

in the updated console?

@Qix: I confirm I only updated git, gulp didn't change version.
Here's the output you requested:
$ node
> var s = require('supports-color');
> console.log(require('util').inspect(s));
{ level: 1, hasBasic: true, has256: false, has16m: false }
> console.log(process.env.TERM.length, process.env.TERM);
14 'xterm-256color'

has256: false seems odd, I must say...

Hmm interesting. I'll take a look at the 256 though chalk as of now doesn't even support 256 colors. That should be enough for gulp to be outputting color via chalk...
@sindresorhus what do you think?

Probably related to: https://github.com/nodejs/node/issues/3006

@sindresorhus, it may be, but the issue in that thread seems to be due to changes in nodeJS 4, while I'm still using node 0.12 and triggered the issue by updating only git bash. Both could have the same underlying cause, though.

@mlarcher @sindresorhus what was the outcome for this issue?

@phated It's still ongoing afaik. Probably not on top of priorities, though.
For my part I still use --color after each gulp command for now :|

I can confirm that this is an issue.

This comment mentions the same issue. This is a node problem, not a Chalk problem.
What appears to be happening is supports-color, the module we wrote to check to see if the terminal supports color, is seeing that isatty() is returning that the terminal session is not actually a terminal session, but a pipe or something. That's Node's fault.
From there, chalk is told not to use colors, hence not getting any colored output.
Unfortunately there isn't much we can do other than to advise using --color or the environment variable FORCE_COLOR to run what you need inside of MSys or Cygwin (not sure if this affects Cygwin but it very well might).

@Qix- what would happen if we set process.env.FORCE_COLOR = true for every instance inside the CLI?

Well if you have any processes piping to each other that don't take into account ANSI output you might have a problem there. Build systems and ANSI output from child processes can get tricky pretty quickly.

Looks like this is a Git for Windows problem and can be resolved by following the comment at https://github.com/nodejs/node/issues/3006#issuecomment-171129702
Closing this as it is something we can't control.

Was helped in another thread and set a windows system variable FORCE_COLOR to true, did not restart, and now I have colors in Git Bash (windows 7)

In windows 8.1 a system environment variable did not work, but adding export FORCE_COLOR=true to .bashrc did the trick.

Btw, with changes happening in gulp-cli, there will no longer be a check for supports-color and if color is not supported, the user will have to add --no-color to their command.

For what it's worth, FORCE_COLOR value should be numeric string, not true.
For shell users, if you export the value (as below), all commands run by the shell will be colorized
$ export FORCE_COLOR="1"    # Force terminal escape colors for ALL commands to follow...
$ gulp

You can also enable colors for a single command à la
$ FORCE_COLOR="1"  gulp

Disabling colors works similarly, here disabling color support for a single command.
$ FORCE_COLOR="0" gulp

is this still the problem on git bash?

I have this problem, using git bash on Windows Terminal...
←[032mâo" Generated

etc...


NLP Service:: read and ask for help about MLQA

facebook: https://github.com/facebookresearch/MLQA?fbclid=IwAR3klBDdCAF8WPK9e6xcxCezDgl5nPiMBFlFbKCnX6uo_2Btc1sU5hzme08

https://ai.facebook.com/blog/mlqa-evaluating-cross-lingual-extractive-question-answering/


Handle task scheduling in ResumableSimulationDriver

Tickets addressed: Fixes #1256
Review: By commit
Merge strategy: Merge (no squash)

Description

Here is an example of how the issue has manifested:

The scheduler has to simulate a plan with activities A and B with B anchored to A.
After simulation, the scheduler tries to query the duration of task B, it fails
Indeed, the resumable simulation driver does not have a direct reference to task B because of how anchored activities were managed, that is, by creating an anonymous task that would include the A, a delay, then B. The driver would remember the id of this anonymous task but not the id of task B precisely. The id of a task is given when the task is scheduled and in this case, the simulation engine would schedule task A and B, not the driver.

The solution consists in replacing this anonymous task creation with a mechanism that lets the driver schedule dependent tasks as soon as their dependent is finishing. Handling task creation allows to maintaining references to task ids and solve the above issue.
Verification

A simple test that was not passing before has been added.
Documentation

N/A
Future work

N/A

Good catch, I forgot this, I have updated the GrowBanana of one test to a PickBanana.

I'd like to point out that this PR is strictly removing code, which should be celebrated somehow :p


ARROW-16076: [R] Bindings for the new TPC-H generator

Expose TPC-H data generation in R.
One can generate one table with tpch_dbgen("lineitem", 1) (and then collect(), or compute() if you want a data.frame or an Arrow table) or can write a full set of TPC-H data out to a set of datasets (one for each table) with tpch_dbgen_write(1, "path/to/").

Based on ARROW-16100 I think we are pausing this PR.  @save-buffer Can we pull these C++ fixes into their own dedicated PR?


Change creation date to toggle with checkbox.

closes #2280
Why was this change made? 🤔
How was this change tested? 🤨
⚡ ⚠ If this change involves consuming from or writing to another service (or shared file system), run integration test create_object_h2_spec.rb and/or test manually in [stage|qa] environment, in addition to specs. ⚡
Cypress, Amy

@amyehodge approved.


Update EdDSA to last circomlib version

Update EdDSA to last circomlib version

Compatible with circomlib version 0.3.0 at
https://github.com/iden3/circomlib/commit/5935ac69df760437da84cc7fe9f4db3ead56d47b
Use blake hash precandidate (non-blake2b) for circomlib
compatibility
Add circomlib testvector

Change PrivateKey from BigInt to [u8;32], which lead to faster
signature computation:
(Benchmarks on a Intel(R) Core(TM) i7-8705G CPU @ 3.10GHz, with
32 GB of RAM)

Old:sign                    time:   [559.84 us 568.41 us 576.26 us]
verify                  time:   [376.59 us 376.68 us 376.78 us]

New:sign                    time:   [383.85 us 390.31 us 396.44 us]
verify                  time:   [371.33 us 376.06 us 381.19 us]

Merged directly through git


updated index.md

Docs Build status updates of commit 65fdda5:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

mixed-reality-docs/index.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Add endpoint to fetch individual by email

Description
To complete the ticket about fetching a person record by email, I also added the endpoint to retrieve an Individual.
Ticket
Find a person record by email
Video
https://github.com/taylorbrooks/virtuous/assets/39068923/f56a26c7-77a5-4545-a22b-7bd13f646c18

Me could maybe merge it after #2 to run the tests


Google Photo album error code

When I try to look at albums I get this error: "An error Occurred in 'handleGetAlbumList'. Code: 404- The requested URL returned error: 404 Not Found"

@sjsj69 -- Are you seeing this with all albums or was this a single event? It's possible the GooglePhotos API was having issues. I've not been able to duplicate this issue.
What's the full name of the album? Any special characters in it?


Update video-calling-javascript.md

Small edit in the Run the code section (removed the ´ ) so that we can copy the code and run it directly.

If you work for microsoft, PR to azure-docs-pr and not here.


Add an API that can be compatible with cargo:error

Currently this crate recommends .unwrap():
pkg_config::Config::new().atleast_version("1.2.3").probe("foo").unwrap();

but the problem with this is that Rust and Cargo present unwrapped error in an ugly and noisy way. Users often struggle to find the relevant messge among all the cargo:rerun-if-env-changed= lines. If they have RUST_BACKTRACE=1 enabled, the error is also buried under an irrelevant backtrace.
I hoped that this could be fixed by emitting cargo:error=reason that would replace the noisy dump with a clear user-facing message. However, the current cargo:error implementation is incompatible with the .probe() call, because any printed error will fail the whole build script as a side effect, even if the caller handles the probe error gracefully and the script succeeds. It can't even be hacked by sneaking in a cargo:error= into the unwrapped pkg_config::Error's message, because unwrap prints to stderr, and cargo:error must be in stdout.
Therefore, I suggest adding a new API like .probe_or_exit(), that will print cargo:error=message on failure, and exit(101) the build script cleanly instead of panicking, so that Rust doesn't append unnecessary info about panicking and backtraces.
pkg_config users can do something like that themselves, here's an example, but 700+ crates are unlikely to make such improvements when .unwrap() is easier, and recommended by pkg_config.

Even without cargo:error support, such API could improve output significantly. That's because the noisy rerun-if-env-changed directives are not needed when the build fails (a failed build will always be retried, whether envs changed or not). This would let pkg-config remove the log noise that cargo won't.


spIsoNet can't run as relion_external_reconstruct

Hi Yun-Tao,
I got the following errors when I was using spIsoNet as external_reconstruct in Relion4 or Relion5-beta. Any suggestions? Thanks!
The following warnings were encountered upon command-line parsing:
WARNING: Option --keep_lowres	is not a valid RELION argument
04-15 19:26:44, INFO     voxel_size 1.399999976158142
04-15 19:26:45, INFO     voxel_size 1.399999976158142
04-15 19:39:35, INFO     Limit resolution to 10.0 for spIsoNet 3D FSC calculation. You can also tune this paramerter with --limit_res .
04-15 19:39:35, INFO     calculating fast 3DFSC, this will take few minutes
04-15 19:43:02, INFO     voxel_size 1.399999976158142
04-15 19:51:23, INFO     The Refine3D/job025 folder already exists, outputs will write into this folder
04-15 19:51:27, INFO     voxel_size 1.399999976158142
04-15 19:51:32, INFO     spIsoNet correction until resolution 10.0A!
Information beyond 10.0A remains unchanged
04-15 19:54:34, INFO     Start preparing subvolumes!
04-15 19:54:59, INFO     Done preparing subvolumes!
04-15 19:54:59, INFO     Start training!
04-15 19:55:02, INFO     Port number: 45495
[W socket.cpp:464] [c10d] The server socket cannot be initialized on [::]:45495 (errno: 97 - Address family not supported by protocol).
[W socket.cpp:697] [c10d] The client socket cannot be initialized to connect to [localhost]:45495 (errno: 97 - Address family not supported by protocol).
[W socket.cpp:697] [c10d] The client socket cannot be initialized to connect to [localhost]:45495 (errno: 97 - Address family not supported by protocol).
[W socket.cpp:697] [c10d] The client socket cannot be initialized to connect to [localhost]:45495 (errno: 97 - Address family not supported by protocol).
[W socket.cpp:697] [c10d] The client socket cannot be initialized to connect to [localhost]:45495 (errno: 97 - Address family not supported by protocol).
[W socket.cpp:697] [c10d] The client socket cannot be initialized to connect to [localhost]:45495 (errno: 97 - Address family not supported by protocol).
[W socket.cpp:697] [c10d] The client socket cannot be initialized to connect to [localhost]:45495 (errno: 97 - Address family not supported by protocol).
[W socket.cpp:697] [c10d] The client socket cannot be initialized to connect to [localhost]:45495 (errno: 97 - Address family not supported by protocol).
[W socket.cpp:697] [c10d] The client socket cannot be initialized to connect to [localhost]:45495 (errno: 97 - Address family not supported by protocol).
Traceback (most recent call last):
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/bin/spisonet.py", line 8, in
sys.exit(main())
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/spIsoNet/bin/spisonet.py", line 549, in main
fire.Fire(ISONET)
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/fire/core.py", line 143, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/fire/core.py", line 477, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/spIsoNet/bin/spisonet.py", line 182, in reconstruct
map_refine_n2n(halfmap1,halfmap2, mask_vol, fsc3d, alpha = alpha,beta=beta,  voxel_size=voxel_size, output_dir=output_dir,
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/spIsoNet/bin/map_refine.py", line 145, in map_refine_n2n
network.train([data_dir_1,data_dir_2], output_dir, alpha=alpha,beta=beta, output_base=output_base0, batch_size=batch_size, epochs = epochs, steps_per_epoch = 1000,
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/spIsoNet/models/network_n2n.py", line 265, in train
mp.spawn(ddp_train, args=(self.world_size, self.port_number, self.model,alpha,beta,
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 241, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 197, in start_processes
while not context.join():
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 158, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:
-- Process 1 terminated with the following error:
Traceback (most recent call last):
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 68, in _wrap
fn(i, *args)
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/spIsoNet/models/network_n2n.py", line 50, in ddp_train
model = model.cuda()
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/nn/modules/module.py", line 911, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/nn/modules/module.py", line 802, in _apply
module._apply(fn)
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/nn/modules/module.py", line 802, in _apply
module._apply(fn)
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/nn/modules/module.py", line 802, in _apply
module._apply(fn)
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/nn/modules/module.py", line 825, in _apply
param_applied = fn(param)
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/nn/modules/module.py", line 911, in
return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error: CUDA-capable device(s) is/are busy or unavailable
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
Traceback (most recent call last):
File "/home/groups/kornberg/donghuac/spIsoNet/spIsoNet/bin/relion_wrapper.py", line 517, in
with mrcfile.open(mrc1_cor) as d1:
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/mrcfile/load_functions.py", line 139, in open
return NewMrc(name, mode=mode, permissive=permissive,
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/mrcfile/mrcfile.py", line 109, in init
self._open_file(name)
File "/home/groups/kornberg/donghuac/anaconda3/envs/spisonet/lib/python3.10/site-packages/mrcfile/mrcfile.py", line 126, in _open_file
self._iostream = open(name, self._mode + 'b')
FileNotFoundError: [Errno 2] No such file or directory: 'Refine3D/job025/corrected_run_it001_half1_class001_unfil.mrc'
in: /home/groups/kornberg/donghuac/relion/src/backprojector.cpp, line 1323
ERROR:
ERROR: there was something wrong with system call: python /home/groups/kornberg/donghuac/spIsoNet/spIsoNet/bin/relion_wrapper.py Refine3D/job025/run_it001_half1_class001_external_reconstruct.star
=== Backtrace  ===
/home/groups/kornberg/donghuac/relion/bin/relion_refine_mpi(_ZN11RelionErrorC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_l+0x69) [0x4e6a59]
/home/groups/kornberg/donghuac/relion/bin/relion_refine_mpi() [0x46434e]
/home/groups/kornberg/donghuac/relion/bin/relion_refine_mpi(_ZN14MlOptimiserMpi12maximizationEv+0x1b02) [0x5222b2]
/home/groups/kornberg/donghuac/relion/bin/relion_refine_mpi(_ZN14MlOptimiserMpi7iterateEv+0x3e9) [0x523279]
/home/groups/kornberg/donghuac/relion/bin/relion_refine_mpi(main+0x55) [0x4d47b5]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7ff6f5fbb555]
/home/groups/kornberg/donghuac/relion/bin/relion_refine_mpi() [0x4d805e]
ERROR:
ERROR: there was something wrong with system call: python /home/groups/kornberg/donghuac/spIsoNet/spIsoNet/bin/relion_wrapper.py Refine3D/job025/run_it001_half1_class001_external_reconstruct.star
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 1.

You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

My command runs properly in RELION 3.1.4 but also met this warning:
The following warnings were encountered upon command-line parsing:
WARNING: Option --keep_lowres   is not a valid RELION argument

The command is:
mpirun -np 5 relion_refine_mpi --o Refine3D/extra/run --auto_refine --split_random_halves --i Extract/job017/particles.star --ref Ref/emd_0495.mrc --firstiter_cc --ini_high 10 --dont_combine_weights_via_disc --preread_images --pool 30 --pad 2 --ctf --particle_diameter 120 --flatten_solvent --zero_mask --solvent_mask MaskCreate/job019/mask.mrc --solvent_correct_fsc --oversampling 1 --healpix_order 2 --auto_local_healpix_order 5 --offset_range 5 --offset_step 2 --sym O --low_resol_join_halves 40 --norm --scale --j 4 --gpu 0:1 --external_reconstruct --keep_lowres

Hi,
I tested relion4 but not relion5, but we think the relion5 blush regularization shares similarities with spIsoNet denoising.
This problem is probably related to failing to open a port, which is 45495 in your case. spIsoNet will automatically detect a port that is not been used for communication. If Anisotropy correction for half maps can be executed correctly, this RELION embedded spIsoNet should also work.
What I have in mind is to check what differs between the environment when you are running "spisonet.py reconstruct" and the relion wrapper. Such as whether the correct conda is used, or whether there are firewall problems.

Hi,
Thanks for the reply. I also tested spIsoNet in Relion4 on my workstation. Here is the error. Please take a look if this is also related to the port or firewall issues.
The following warnings were encountered upon command-line parsing:
WARNING: Option --keep_lowres	is not a valid RELION argument
04-16 14:59:29, INFO     voxel_size 1.399999976158142
04-16 14:59:29, INFO     voxel_size 1.399999976158142
04-16 15:04:45, INFO     Limit resolution to 10.0 for spIsoNet 3D FSC calculation. You can also tune this paramerter with --limit_res .
04-16 15:04:45, INFO     calculating fast 3DFSC, this will take few minutes
04-16 15:06:31, INFO     voxel_size 1.399999976158142
04-16 15:09:55, INFO     The Refine3D/job025 folder already exists, outputs will write into this folder
04-16 15:09:57, INFO     voxel_size 1.399999976158142
04-16 15:10:00, INFO     spIsoNet correction until resolution 10.0A!
Information beyond 10.0A remains unchanged
04-16 15:11:06, INFO     Start preparing subvolumes!
04-16 15:11:24, INFO     Done preparing subvolumes!
04-16 15:11:24, INFO     Start training!
04-16 15:11:24, INFO     Port number: 44689
[rank3]:[E ProcessGroupNCCL.cpp:523] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=4, Timeout(ms)=600000) ran for 600701 milliseconds before timing out.
[rank3]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank3]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down.
[rank3]:[E ProcessGroupNCCL.cpp:1182] [Rank 3] NCCL watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=4, Timeout(ms)=600000) ran for 600701 milliseconds before timing out.
Exception raised from checkTimeout at /opt/conda/conda-bld/pytorch_1711403388920/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:525 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7efd44c15d87 in /data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1e6 (0x7efd45daed26 in /data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x19d (0x7efd45db227d in /data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7efd45db2e79 in /data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4:  + 0xdbbf4 (0x7efda2973bf4 in /data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/lib/../../../.././libstdc++.so.6)
frame #5:  + 0x9f802 (0x7efdb898d802 in /lib64/libc.so.6)
frame #6:  + 0x3f450 (0x7efdb892d450 in /lib64/libc.so.6)
[rank0]:[E ProcessGroupNCCL.cpp:523] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=4, Timeout(ms)=600000) ran for 600714 milliseconds before timing out.
[rank0]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank0]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down.
[rank0]:[E ProcessGroupNCCL.cpp:1182] [Rank 0] NCCL watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=4, Timeout(ms)=600000) ran for 600714 milliseconds before timing out.
Exception raised from checkTimeout at /opt/conda/conda-bld/pytorch_1711403388920/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:525 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f55840bcd87 in /data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1e6 (0x7f5585255d26 in /data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x19d (0x7f558525927d in /data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f5585259e79 in /data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4:  + 0xdbbf4 (0x7f55e1e1abf4 in /data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/lib/../../../.././libstdc++.so.6)
frame #5:  + 0x9f802 (0x7f55f7e34802 in /lib64/libc.so.6)
frame #6:  + 0x3f450 (0x7f55f7dd4450 in /lib64/libc.so.6)
[rank2]:[E ProcessGroupNCCL.cpp:523] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=4, Timeout(ms)=600000) ran for 600837 milliseconds before timing out.
[rank2]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank2]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down.
[rank2]:[E ProcessGroupNCCL.cpp:1182] [Rank 2] NCCL watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLGATHER, NumelIn=1, NumelOut=4, Timeout(ms)=600000) ran for 600837 milliseconds before timing out.
Exception raised from checkTimeout at /opt/conda/conda-bld/pytorch_1711403388920/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:525 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f8aca5aed87 in /data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1e6 (0x7f8acb747d26 in /data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x19d (0x7f8acb74b27d in /data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f8acb74be79 in /data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4:  + 0xdbbf4 (0x7f8b2830cbf4 in /data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/lib/../../../.././libstdc++.so.6)
frame #5:  + 0x9f802 (0x7f8b3e326802 in /lib64/libc.so.6)
frame #6:  + 0x3f450 (0x7f8b3e2c6450 in /lib64/libc.so.6)
Traceback (most recent call last):
File "/data/donghua/anaconda3/envs/spisonet/bin/spisonet.py", line 8, in
sys.exit(main())
File "/data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/spIsoNet/bin/spisonet.py", line 549, in main
fire.Fire(ISONET)
File "/data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/fire/core.py", line 143, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/fire/core.py", line 477, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/spIsoNet/bin/spisonet.py", line 182, in reconstruct
map_refine_n2n(halfmap1,halfmap2, mask_vol, fsc3d, alpha = alpha,beta=beta,  voxel_size=voxel_size, output_dir=output_dir,
File "/data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/spIsoNet/bin/map_refine.py", line 145, in map_refine_n2n
network.train([data_dir_1,data_dir_2], output_dir, alpha=alpha,beta=beta, output_base=output_base0, batch_size=batch_size, epochs = epochs, steps_per_epoch = 1000,
File "/data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/spIsoNet/models/network_n2n.py", line 265, in train
mp.spawn(ddp_train, args=(self.world_size, self.port_number, self.model,alpha,beta,
File "/data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 241, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
File "/data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 197, in start_processes
while not context.join():
File "/data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 140, in join
raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 3 terminated with signal SIGABRT
Traceback (most recent call last):
File "/data/donghua/spIsoNet/spIsoNet/bin/relion_wrapper.py", line 517, in
with mrcfile.open(mrc1_cor) as d1:
File "/data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/mrcfile/load_functions.py", line 139, in open
return NewMrc(name, mode=mode, permissive=permissive,
File "/data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/mrcfile/mrcfile.py", line 109, in init
self._open_file(name)
File "/data/donghua/anaconda3/envs/spisonet/lib/python3.10/site-packages/mrcfile/mrcfile.py", line 126, in _open_file
self._iostream = open(name, self._mode + 'b')
FileNotFoundError: [Errno 2] No such file or directory: 'Refine3D/job025/corrected_run_it001_half1_class001_unfil.mrc'
in: /data/donghua/relion/src/backprojector.cpp, line 1294
ERROR:
ERROR: there was something wrong with system call: python /data/donghua/spIsoNet/spIsoNet/bin/relion_wrapper.py Refine3D/job025/run_it001_half1_class001_external_reconstruct.star
=== Backtrace  ===
/data/donghua/relion/bin/relion_refine_mpi(_ZN11RelionErrorC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_l+0x69) [0x4be749]
/data/donghua/relion/bin/relion_refine_mpi() [0x44d378]
/data/donghua/relion/bin/relion_refine_mpi(_ZN14MlOptimiserMpi12maximizationEv+0x17a4) [0x4f4b14]
/data/donghua/relion/bin/relion_refine_mpi(_ZN14MlOptimiserMpi7iterateEv+0x4c1) [0x4f5b01]
/data/donghua/relion/bin/relion_refine_mpi(main+0x58) [0x4ad658]
/lib64/libc.so.6(+0x3feb0) [0x7f8469dbfeb0]
/lib64/libc.so.6(__libc_start_main+0x80) [0x7f8469dbff60]
/data/donghua/relion/bin/relion_refine_mpi(_start+0x25) [0x4b0815]
ERROR:
ERROR: there was something wrong with system call: python /data/donghua/spIsoNet/spIsoNet/bin/relion_wrapper.py Refine3D/job025/run_it001_half1_class001_external_reconstruct.star
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 1.

You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

Hi, are the above two errors the same? I got one from a cluster and another one from the workstation. Any suggestions? Thanks!

Hi,
I now know that the "MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD with errorcode 1...." problem is RELION reconstruction not properly performed. This will happen as spIsoNet does not works for RELION5.
I still does not have any understanding how the NCCL related error happens.

Again I want to confirm whether the Anisotropy Correction ("spisonet.py reconstruct")  does not give you these errors within the same environment.

Please also see whether this "https://github.com/IsoNet-cryoET/spIsoNet/issues/2" is related

I can confirm that the Anisotropy Correction worked without any error on my 2-GPU workstation which has the same type of GPU as my 4-GPU workstation where I got the second error above.

I now know that the "MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD with errorcode 1...." problem is RELION reconstruction not properly performed. This will happen as spIsoNet does not works for RELION5.

Is relion 5 compatibility on the roadmap? Or for now, would you recommend to set up a separate installation of relion 4 for misalignment correction?

I now know that the "MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD with errorcode 1...." problem is RELION reconstruction not properly performed. This will happen as spIsoNet does not works for RELION5.

Hi, are you sure that it does not work with RELION-5? Is this a matter of spIsoNet not running at all in RELION-5, or not giving the intended output? Because I have been running it through RELION-5 for initial testing before seeing this comment and it doesn't appear to have any issues, but I can't speak for whether it is producing "correct" results.

It runs with relion 5 (from within spisonet conda env), but at least in my hands the resulting maps are blank, and it seems to ignore the "IOSONET_START_RESOLUTION" environment variable and start external reconstruction from the first iteration (perhaps the cause of the blank maps?)

In our hands with relion 5 it seems to run, but the unfil.mrc and corrected.mrc maps are blank, leading to a crash after one iteration - this does not happen without --external_reconstruct. I haven't tried relion 4 yet

Hi All,
I used the recommended option 1 (as the following) in the tutorial to do the installation of spIsoNet, why I could not see the bin directory (which should contain the program spisonet.py)  created just under the directory of spIsoNet after the installation?
git clone https://github.com/IsoNet-cryoET/spIsoNet.git
conda env create -f setup.yml
conda activate spisonet
Any suggestions? Thanks!

Thank you for your clarification!

I found that the first error above {RuntimeError: CUDA error: CUDA-capable device(s) is/are busy or unavailable} was related to the GPU node's  setting for Compute Mode of Default (e.g. on our cluster: #SBATCH --gpu_cmode=shared), and the second error (Some NCCL operations have failed or timed out) was corrected by this setting (export NCCL_P2P_DISABLE=1).

Thank you for trouble shooting and report back

I wonder in this file spIsoNet_v1.0_Tutorial.pdf, whether one more step (pip install .) in Option 3 for the Installation should be added as the last step? Please confirm.


[SMALLFIX] Improve shell code style in bin.

Prefer ${var} over $var.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/9039/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/9040/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/9041/
Test PASSed.

Merged build finished. Test PASSed.

thanks @ifcharming left some comments

Thanks @jsimsa for the review! Comments are addressed. PTAL

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/9099/
Test PASSed.

a couple of small comments, thanks

Updated, thanks! PTAL

LGTM

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/9101/
Test PASSed.

Merged build finished. Test PASSed.

LGTM


Fade animation support on iOS (untested)

Please test it, if it’s working for you.

Tested and for me it works fine. Thanks @daniel-luberda :+1:
I will close #16 then


Board Review: Azure Communication Services (SPOOL) [Phone Numbers] Operator Connect (JS)

Responsible team: Phone Numbers team
Contacts:
<EMAIL_ADDRESS><EMAIL_ADDRESS>Basic info:
With the upcoming feature of Operator Connect, it will be possible for a user to purchase numbers not only from Microsoft, but also directly from other operators, but through Azure Portal. For that, UI needs to be able to retrieve information about which operator handles the purchased number.
Champion scenario:
-As Portal UI, I need to display information about operator, from whom the number was purchased.
Presentation
Introduction to Operator Connect for Azure Communication Services (OC4ACS) v1.docx
Funk Spec
Private Preview
Design
Api Spec: azure-rest-api-specs/operatorconnect.json at slpavkov/operatorconnect-review · Azure/azure-rest-api-specs (github.com)
Sdk view: @azure/communication-phone-numbers - apiview.dev

Recording (MS INTERNAL ONLY)


Kotlin 1.8.21 release

Blocked on:

https://github.com/JetBrains/compose-multiplatform/issues/3214
~https://github.com/cashapp/zipline/issues/1027~

#1299 releasing next week


Add analysis test case with Daytrader app

We need to add a test case for https://github.com/konveyor/daytrader

Done, actually part of TIER2 tests.


added bears.finance (fbear.eth)

Hi! What is your PR about?

[x] Add images for your space
[x] Add custom domain for your space
[ ] Add alias for migrate your space
[ ] Add or edit a skin
~Edit your space with GitHub?~ (deprecated) Please migrate your space to ENS https://docs.snapshot.page/spaces/migrate
~Create your space with GitHub~ (deprecated) Create your space with ENS instead https://docs.snapshot.page/spaces/create

@gokaiorg review please

DNS record for custom domain is already added and our space is listed on snapshot.page as Bears as well

Anything else to update?


remove type check from server app

This goes in with the caveat that everything in common gets type checked.

Closing as it's actually saved me a few times...


Resubmission: Save output dataframe as HDF5, not Pickle

I was having issues with reading Pickles from a buffer — as updated in ee641937, in order to accommodate multiple backends, the output Pickle is now read via a transport, rather than with a simple pd.read_pickle("filename") call.
HDF5 is more amenable to being read/written via buffers, so I've switched over. Also, HDF5 ought to be a more stable serialization format — people have reported that the Pickle spec often introduces breaking changes.

Coverage increased (+0.002%) to 59.545% when pulling 6dadba265ce10f8b2c7286d82e057d7188b8162a on julianhess:hdf5 into 6a2e28916ceda6b08d3b22a885425cb2878138c9 on broadinstitute:master.


Buttons with Payloads / Titles containing "Yes" or "No" get transformed incorrectly

Rasa Core version: 0.10.1
Python version: 3.6
Operating system (windows, osx, ...): Ubuntu
Issue:
After upgrading from 0.9 to 0.10.1 I noticed the following phenomenon when running my unit tests: a button utterance defined like this:
  utter_buttons:
  - text: "Buttons, there will be buttons:"
    buttons:
      - title: "Yes"
        payload: "Yes"
      - title: "No"
        payload: "No"

Will be serialized like this when using a CollectingOutputChannel:
{
    "messages": [
        {
            "data": [
                {
                    "payload": true,
                    "title": true
                },
                {
                    "payload": false,
                    "title": false
                }
            ],
            "recipient_id": "xxx",
            "text": "Buttons, there will be buttons:"
        }
    ]
}

when using 0.9 would do the expected:
{
    "messages": [
        {
            "data": [
                {
                    "payload": "Yes",
                    "title": "Yes"
                },
                {
                    "payload": "No",
                    "title": "No"
                }
            ],
            "recipient_id": "xxx",
            "text": "Buttons, there will be buttons:"
        }
    ]
}

Any string that is not "Yes" or "No" will still work as expected in 0.10.1

So I looked into this further and found out the following:

While training, my domain file containing the quoted "Yes" and "No" words will be converted and dumped into the model folder, which will unquote the strings into Yes and No.
When reading the compiled domain file in my models folder, they are unquoted which in turn interprets Yes and No as True and False resulting in the behavior explained above.
When I shortcircuit the if six.PY2 checks and make rasa always use pyyaml instead of ruamel.yaml it works as expected, delivering the strings (as the compiled domain file will contain quoted strings).
Using !!str Yes notation doesn't work either with ruamel.yaml.

I'm still pretty confused about what exactly is happening, especially after reading this: https://yaml.readthedocs.io/en/latest/pyyaml.html#defaulting-to-yaml-1-2-support, but maybe someone who really knows what changed from 0.9 to 0.10 can make more sense of it than I can.


load minimum amount of data for json releases/files

For projects with long release histories we were pulling altogether too much data to construct the releases key which just a dict of files for all previous releases.

/me sighs at codecov....

Report on codecov.io shows 100% I promise.


Copy Existing DMP selection needs a search function

Currently you have to page through the DMPs in a predefined order to select the one you want to copy. Can we add the same search function that is available in the Public DMPs lookup?

DataONE Usability testing (May 2014) noted same issue:
Provide a way to sort or search existing DMPs, either by selecting table column headings (name, owner) or inserting a sort/search box. Users also requested to see funding agency in the existing DMP table.

Kris Bolt from Harvard: It is tedious to scroll through long lists of templates, existing DMPs, etc. within the tool. A desired enhancement for these lists is the ability to search and/or sort.

I think this is fixed in the latest version.

yes it is. and we'll be improving the search and sort functionality for all tables soon. I also wanted to note that this is now a legacy/inactive repo. you can report any new issues here https://github.com/CDLUC3/dmptool/issues


New Feature?

I know that this is the issue section but I am new to Github. Your Kahoot Hack works great but I would like if you could add a feature that makes say 1 or multiple clients search for Kahoot pins to login to. Then it would spit out any working pins that you can then use the hack on. Thanks

that sounds very evil...I can't think of any good that could come of such a tool. You wouldn't even know who you're messing with.

oh :(... thanks for the reply though :/

Im just looking up pins on twitter or live streams currently :D

@NZHacker You could go on and use a brute force to just run through every possible combination of numbers. Ex: THC-Hydra or Burp suite brute-force.
You would just have to customize it to your own needs.


Server Response undefined

Uncaught TypeError: http.ServerResponse is undefined
    [31]</</< https://timbuktu-42c57.web.app/assets/bundle.js:4975
    [31]</< https://timbuktu-42c57.web.app/assets/bundle.js:6077
    [31]< https://timbuktu-42c57.web.app/assets/bundle.js:6077
    o https://timbuktu-42c57.web.app/assets/bundle.js:1
    o https://timbuktu-42c57.web.app/assets/bundle.js:1
    [27]< https://timbuktu-42c57.web.app/assets/bundle.js:4215
    o https://timbuktu-42c57.web.app/assets/bundle.js:1
    o https://timbuktu-42c57.web.app/assets/bundle.js:1
    [25]< https://timbuktu-42c57.web.app/assets/bundle.js:3543
    o https://timbuktu-42c57.web.app/assets/bundle.js:1
    o https://timbuktu-42c57.web.app/assets/bundle.js:1
    [1]</</< https://timbuktu-42c57.web.app/assets/bundle.js:5
    [1]</< https://timbuktu-42c57.web.app/assets/bundle.js:49
    [1]< https://timbuktu-42c57.web.app/assets/bundle.js:49
    o https://timbuktu-42c57.web.app/assets/bundle.js:1
    r https://timbuktu-42c57.web.app/assets/bundle.js:1
    <anonymous> https://timbuktu-42c57.web.app/assets/bundle.js:1
bundle.js:4975:11
    [31]</</< https://timbuktu-42c57.web.app/assets/bundle.js:4975
    [31]</< https://timbuktu-42c57.web.app/assets/bundle.js:6077
    [31]< https://timbuktu-42c57.web.app/assets/bundle.js:6077
    o https://timbuktu-42c57.web.app/assets/bundle.js:1
    o https://timbuktu-42c57.web.app/assets/bundle.js:1
    [27]< https://timbuktu-42c57.web.app/assets/bundle.js:4215
    o https://timbuktu-42c57.web.app/assets/bundle.js:1
    o https://timbuktu-42c57.web.app/assets/bundle.js:1
    [25]< https://timbuktu-42c57.web.app/assets/bundle.js:3543
    o https://timbuktu-42c57.web.app/assets/bundle.js:1
    o https://timbuktu-42c57.web.app/assets/bundle.js:1
    [1]</</< https://timbuktu-42c57.web.app/assets/bundle.js:5
    [1]</< https://timbuktu-42c57.web.app/assets/bundle.js:49
    [1]< https://timbuktu-42c57.web.app/assets/bundle.js:49
    o https://timbuktu-42c57.web.app/assets/bundle.js:1
    r https://timbuktu-42c57.web.app/assets/bundle.js:1
    <anonymous> https://timbuktu-42c57.web.app/assets/bundle.js:1

Leads to :
var res = Object.create(http.ServerResponse.prototype)

Typically express is placed outside of the src folder to serve your compiled client-side code.
It cannot be executed with the front-end code or in the client, it needs to operate from the back end from a separate location outside the src timbuktu.
It would look more like compiling timbuktu with npm and calling that from the back end using Express. Focusing on figuring out Firebase for now.
https://stackoverflow.com/questions/58263678/why-do-i-get-typeerror-when-i-try-adding-express-to-my-first-react-app


Sepolia

Provide data related to the network

Name
Sepolia
Chain id
11155111
URL
sepolia.org

Reasons why it is important for Semaphore to be deployed on this network.
Currently testnet-wise semaphore is only deployed on goerli.
Sepolia is intended to replace goerli for application testing.
See e.g. @q9f 's talk from @efdevcon

also see: https://blog.ethereum.org/2022/06/21/testnet-deprecation

ref https://github.com/eth-clients/sepolia


Port NP Atobarai gateway to 3.1

I want to merge this change because I want to port changes from #8640 to 3.1.
Impact

[ ] New migrations
[ ] New/Updated API fields or mutations
[ ] Deprecated API fields or mutations
[ ] Removed API types, fields, or mutations
[ ] Documentation needs to be updated

Pull Request Checklist

[ ] Privileged queries and mutations are guarded by proper permission checks
[ ] Database queries are optimized and the number of queries is constant
[ ] Database migration files are up to date
[ ] The changes are tested
[ ] GraphQL schema and type definitions are up to date
[ ] Changes are mentioned in the changelog

Task linked: SALEOR-5872 Port Net Protection from 3.0 to 3.1


Use tailwindcss prettier plugin and recommend extensions to contributors

Run Prettier as well after updating the config and I can merge this.

Superseded by https://github.com/sourcejump/website/pull/5

#5 merged, closing


Add reference architecture for using Kanister with Argo Workflows

Change Overview

This PR adds a reference architecture for using Kanister with Argo Workflows to automate creation of backup ActionSets.
Pull request type
Please check the type of change your PR introduces:

[ ] :construction: Work in Progress
[ ] :rainbow: Refactoring (no functional changes, no api changes)
[ ] :hamster: Trivial/Minor
[ ] :bug: Bugfix
[x] :sunflower: Feature
[x] :world_map: Documentation
[ ] :robot: Test

Issues

fixes #1514

Test Plan

[ ] :muscle: Manual
[ ] :zap: Unit test
[ ] :green_heart: E2E

In most of  the scenarios when we ask users to create an actionset using kanctl, we (kanctl) query a lot of things (k8s resources) that are passed to the kanctl command line tool.
So its, I think, I bit hard to figure out exact roles that a person need to be able to create an actionset. And that is the reason its a bit challenging to figure out the permissions that argo workflow needs to be able to create the actionset.
Also, I am not sure if giving cluster-admin role to the ServiceAccount is good idea.
Let's see what others say @pavannd1 @PrasadG193

@viveksinghggits The main goal of this document is to provide a reference to scheduling actions using Argo Workflows for interested users. Consider it as a tutorial similar to the blueprint examples and the Grafana/Loki example we have in the documentation. As with those documents, we are only showcasing how we integrate with ArgoWorkflows and not providing a standard way to schedule actions in Kanister. Given the purpose of this tutorial, I think it's acceptable to use the cluster-admin role. However, I agree with your concerns and we could iterate and find a more fine-grained Role that works for creating ActionSets as a follow-up.

I experimented with a lower admin role instead of cluster-admin. The pod to perform the workflow was not created.
The Argo documentation also allows you to create custom service accounts and then grant them the necessary permissions.
A note can be added saying that using the default service account and the mentioned privileges is not recommended in production and that the user can customize this based on their use case

@viveksinghggits Please take a look


Add  the  cpumask options to QEMU.

We must add the corresponding features to the QEMU code to implement the CPU pinning feature.
Add the 'cpumask' option to set the CPU mask to the threads created with QEMU. The mask affects all the threads belonging to a VM: both VCPU threads and non-VCPU threads. If the option is not provided, it's considered to be '-1', which corresponds to all the available CPUs.
Add the 'cpu-pin' option to pin any VCPU thread to a specific CPU. If the option is set, any VCPU thread will be assigned to a CPU provided with the 'cpumask'. If it's not provided, the VPCU threads can migrate from a CPU to a CPU within the set of CPUs provided by the 'cpumask' option.

@OhmSpectator I'll review this once Rene's comments have been addressed.

LGTM

Do we think these patches will make it to the upstream? Or are we signing up to maintaining these patches going forward?

At the moment, they are unfortunately not suitable for the upstream.
For example, the new option will work only for the POSIX threads.
Later it would be nice to rework them for the upstream, but it requires some extra work.

The Eden check fails with some strange error
  FAIL: ../update_eve_image/testdata/update_eve_image_oci.txt:62: command failure


Its not saving the IP address to the file

I inputted an IP address in Client mode and switched to Server mode. When I switched back to Client mode, the IP Address field is blank. Also, it doesn't get saved when I input one and close the program.

fixed


upgrade dependencies for react, vue and lit

Describe your changes

Fixes Issue

Check List (Check all the applicable boxes)

[x] My code follows the code style of this project.
[ ] My change requires changes to the documentation.
[ ] I have updated the documentation accordingly.
[x] All new and existing tests passed.

@melloware


own request Bytes & to_json

I'm looking to integrate the services with the http_client interface, but want to make a couple of adjustments before I do. Here is the first one.  Instead of serde_json::to_string, serde_json::to_vec is used. This avoids going from string to bytes later.

I really like this idea. bytes can help us avoid unnecessary allocations while keeping the code clean. 🥇 🥇

I really like this idea. bytes can help us avoid unnecessary allocations while keeping the code clean. 🥇 🥇

The storage rewrite in #119 was merged in, so I had to update  storage as well. @MindFlavor, can you do another review please with the updates from c99bbc0b9efad5b2e0468caa82ccbe85cf257b7f?

The storage rewrite in #119 was merged in, so I had to update  storage as well. @MindFlavor, can you do another review please with the updates from c99bbc0b9efad5b2e0468caa82ccbe85cf257b7f?

I like it, the examples showcase clearly how good this decision is! 👍

I like it, the examples showcase clearly how good this decision is! 👍


Add to Kanboard Plugins Directory

Yet to do

Fixed by https://github.com/kanboard/website/pull/338


Question: Vulkan raytracing support planned?

Hi,
just read about Nsight graphics update at Siggraph with Vulkan raytracing support (altough no drivers yet) and seeing you support already DXR wanted to ask if you plan to support Vulkan raytrace in the Forge in addition to DXR..
better yet (and seeing http://confettishadertranslator.azurewebsites.net) would be if you can support translation from DXR HLSL shaders to Vulkan GLSL raytrace extensions (yet to be published) so from one shader we are able to target Vulkan too..
Have a Volta card to test once Vulkan drivers are out..
thanks..

Hey Oscar,
I think we have that on our list. Thanks for the offer to beta test :-) This is very much appreciated. Does the current Ray Tracing unit tests all work on your card?

Wolfgang

Hi,
tested, but it doen't count as I'm currently running an insider build and seems all DXR content crashes..
gonna clean an SSD to install supported RS4 and report results..

Yep it is in there since two releases :-)