perf: remove need for channels in the sender recovery stage

This PR proposes a slightly different approach at the parallelization of the reading of the transactions: instead of using a single cursor for a big range, it slices the range and opens a cursor for each sub-range. This alternative approach makes the use of channels unnecessary, thus removing some overhead.
Unfortunately due to a separate error that seems to happen on my machine I wasn't able to run tests to validate nothing is broken nor the benchmarks in order to compare performance.

what's the performance impact of this

what's the performance impact of this

Unfortunately I couldn't measure it as I haven't been able to run benchmarks in my machine as running cargo bench only shows that all tests were ignored. Any pointers on how to run them locally would be much appreciated.
My expectation would be to reduce some overhead that's incurred from the use of the channels, probably some memory moves/clones would be avoided as well. Unsure if the use of multiple read cursors has a meaningful positive impact.

You can try running hyperfine over just the stage, if you're having issues with cargo bench.
eg.
hyperfine reth stage tx-lookup --from 1 --to 10000

@joshieDo didn't work:
$ ./target/release/reth stage tx-lookup --from 1 --to 10000
error: unrecognized subcommand 'tx-lookup'

Usage: reth stage [OPTIONS] <COMMAND>

For more information, try '--help'.

The error I get when I try to run the benches (same as when trying to run a test)
$ cargo bench --package reth-stages --bench criterion --features test-utils
    Finished bench [optimized] target(s) in 0.84s
     Running benches/criterion.rs (/home/user/reth/target/release/deps/criterion-bd88f246948d03c9)
thread 'main' panicked at 'Not able to create the database file.: Failed to open database: 37

Location:
    /home/user/reth/crates/storage/db/src/lib.rs:125:18', /home/user/reth/crates/storage/db/src/lib.rs:187:47
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: bench failed, to rerun pass `-p reth-stages --bench criterion`

I'm running this in a WSL2

@auterium The command is reth stage run tx-lookup --from 1 --to 10000. Note that it requires downloaded headers and bodies though (as this builds an index using another tx table)
In terms of the other error, is there a chance you could reach out to me on Telegram at @onbjerg? We have a similar open issue for WSL, but I am not entirely sure how to debug it. Would be great if we could jam a bit together on it? I assume it's either a permission error or a pathing error :)

Ran the benchmarks on my machine.
Take away: The performance is slightly worse, or around the same. It's somewhat hard to tell because the variance on these benchmarks is pretty high for whatever reason. Running the exact same branch (even on main) sometimes yields +-30% difference.
It might be because we collect the results in a Vec just to unfurl it after. Afaik rayon has a method to parallelize the work + send the results over a channel, so this might be an idea. We would not allocate a lot of channels, just one. This is just conjecture though
main
Run 1
Stages/SendersRecovery-batch-1000
                        time:   [37.534 ms 38.799 ms 40.285 ms]
Found 2 outliers among 10 measurements (20.00%)
  1 (10.00%) high mild
  1 (10.00%) high severe
Stages/SendersRecovery-batch-10000
                        time:   [49.665 ms 50.527 ms 51.285 ms]
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) low severe
Stages/SendersRecovery-batch-100000
                        time:   [51.090 ms 52.780 ms 54.757 ms]
Stages/SendersRecovery-batch-250000
                        time:   [50.149 ms 51.855 ms 53.591 ms]
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high severe

Run 2
Stages/SendersRecovery-batch-1000
                        time:   [30.965 ms 31.282 ms 31.616 ms]
                        change: [-61.393% -51.759% -34.872%] (p = 0.00 < 0.05)
                        Performance has improved.
Stages/SendersRecovery-batch-10000
                        time:   [30.643 ms 30.845 ms 31.141 ms]
                        change: [-9.1515% -6.9501% -4.1583%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high severe
Stages/SendersRecovery-batch-100000
                        time:   [30.559 ms 31.218 ms 31.588 ms]
                        change: [-11.122% -9.1172% -7.1083%] (p = 0.00 < 0.05)
                        Performance has improved.
Stages/SendersRecovery-batch-250000
                        time:   [30.627 ms 31.084 ms 31.472 ms]
                        change: [-11.601% -9.8231% -7.9093%] (p = 0.00 < 0.05)
                        Performance has improved.

Run 3
Stages/SendersRecovery-batch-1000
                        time:   [29.957 ms 30.412 ms 30.839 ms]
                        change: [-4.6193% -3.1516% -1.7266%] (p = 0.00 < 0.05)
                        Performance has improved.
Stages/SendersRecovery-batch-10000
                        time:   [30.556 ms 30.745 ms 30.890 ms]
                        change: [-5.7978% -2.7678% -0.4773%] (p = 0.06 > 0.05)
                        No change in performance detected.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) low mild
Stages/SendersRecovery-batch-100000
                        time:   [31.117 ms 31.448 ms 31.991 ms]
                        change: [+0.3564% +2.3855% +4.6315%] (p = 0.05 < 0.05)
                        Change within noise threshold.
Stages/SendersRecovery-batch-250000
                        time:   [30.769 ms 30.923 ms 31.226 ms]
                        change: [-0.4514% +1.6974% +4.5647%] (p = 0.21 > 0.05)
                        No change in performance detected.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high severe

Run 4
Stages/SendersRecovery-batch-1000
                        time:   [32.257 ms 32.526 ms 32.982 ms]
                        change: [+5.7752% +8.1309% +10.667%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild
Stages/SendersRecovery-batch-10000
                        time:   [30.997 ms 31.663 ms 32.697 ms]
                        change: [+1.1045% +3.4276% +5.7494%] (p = 0.01 < 0.05)
                        Performance has regressed.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild
Stages/SendersRecovery-batch-100000
                        time:   [31.044 ms 31.431 ms 31.989 ms]
                        change: [-1.4225% +0.5492% +2.8081%] (p = 0.64 > 0.05)
                        No change in performance detected.
Stages/SendersRecovery-batch-250000
                        time:   [29.411 ms 29.767 ms 30.630 ms]
                        change: [-4.9525% -1.8322% +1.2555%] (p = 0.31 > 0.05)
                        No change in performance detected.

Run 5
Stages/SendersRecovery-batch-1000
                        time:   [30.379 ms 30.816 ms 31.320 ms]
                        change: [-7.8451% -5.6067% -3.5526%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 10 measurements (30.00%)
  1 (10.00%) low mild
  2 (20.00%) high mild
Stages/SendersRecovery-batch-10000
                        time:   [29.539 ms 29.731 ms 30.081 ms]
                        change: [-7.9615% -5.6632% -3.4618%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild
Stages/SendersRecovery-batch-100000
                        time:   [30.088 ms 30.334 ms 30.573 ms]
                        change: [-5.4673% -1.7100% +3.5125%] (p = 0.57 > 0.05)
                        No change in performance detected.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high severe
Stages/SendersRecovery-batch-250000
                        time:   [30.830 ms 31.210 ms 31.737 ms]
                        change: [-1.2487% +1.8134% +4.7693%] (p = 0.28 > 0.05)
                        No change in performance detected.

This PR
Run 1
Stages/SendersRecovery-batch-1000
                        time:   [32.929 ms 33.245 ms 33.499 ms]
                        change: [+5.3412% +6.9373% +8.5605%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild
Stages/SendersRecovery-batch-10000
                        time:   [33.202 ms 33.344 ms 33.470 ms]
                        change: [+10.089% +11.852% +13.448%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) low mild
Stages/SendersRecovery-batch-100000
                        time:   [32.945 ms 33.218 ms 33.459 ms]
                        change: [+1.1740% +6.5664% +10.160%] (p = 0.02 < 0.05)
                        Performance has regressed.
Stages/SendersRecovery-batch-250000
                        time:   [29.735 ms 30.955 ms 33.245 ms]
                        change: [+0.1833% +4.2606% +8.8142%] (p = 0.09 > 0.05)
                        No change in performance detected.

Run 2
Stages/SendersRecovery-batch-1000
                        time:   [32.677 ms 33.110 ms 33.455 ms]
                        change: [-1.4213% -0.1893% +0.9812%] (p = 0.77 > 0.05)
                        No change in performance detected.
Found 2 outliers among 10 measurements (20.00%)
  1 (10.00%) low mild
  1 (10.00%) high mild
Stages/SendersRecovery-batch-10000
                        time:   [28.451 ms 28.606 ms 28.867 ms]
                        change: [-14.830% -13.826% -12.850%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) low mild
Stages/SendersRecovery-batch-100000
                        time:   [32.971 ms 33.296 ms 33.733 ms]
                        change: [-4.2021% -1.4313% +1.2871%] (p = 0.40 > 0.05)
                        No change in performance detected.
Found 3 outliers among 10 measurements (30.00%)
  2 (20.00%) low severe
  1 (10.00%) high mild
Stages/SendersRecovery-batch-250000
                        time:   [32.990 ms 33.124 ms 33.280 ms]
                        change: [-2.3208% +1.6661% +6.1809%] (p = 0.49 > 0.05)
                        No change in performance detected.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) low mild

Run 3
Stages/SendersRecovery-batch-1000
                        time:   [36.402 ms 39.034 ms 41.081 ms]
                        change: [+16.208% +20.750% +23.735%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) low severe
Stages/SendersRecovery-batch-10000
                        time:   [32.923 ms 33.191 ms 33.450 ms]
                        change: [+14.784% +15.903% +17.166%] (p = 0.00 < 0.05)
                        Performance has regressed.
Stages/SendersRecovery-batch-100000
                        time:   [33.214 ms 33.655 ms 33.916 ms]
                        change: [-0.0134% +2.6907% +6.1052%] (p = 0.12 > 0.05)
                        No change in performance detected.
Stages/SendersRecovery-batch-250000
                        time:   [33.107 ms 33.627 ms 34.119 ms]
                        change: [+0.1104% +1.9470% +3.8806%] (p = 0.07 > 0.05)
                        No change in performance detected.

Run 4
Stages/SendersRecovery-batch-1000
                        time:   [35.329 ms 35.804 ms 36.328 ms]
                        change: [-13.053% -10.487% -6.9558%] (p = 0.00 < 0.05)
                        Performance has improved.
Stages/SendersRecovery-batch-10000
                        time:   [33.909 ms 34.098 ms 34.507 ms]
                        change: [+2.6052% +3.9043% +5.2993%] (p = 0.00 < 0.05)
                        Performance has regressed.
Stages/SendersRecovery-batch-100000
                        time:   [33.276 ms 33.888 ms 34.816 ms]
                        change: [-0.4712% +1.5942% +3.8985%] (p = 0.17 > 0.05)
                        No change in performance detected.
Stages/SendersRecovery-batch-250000
                        time:   [33.917 ms 34.194 ms 34.494 ms]
                        change: [+0.1753% +2.2784% +4.5691%] (p = 0.07 > 0.05)
                        No change in performance detected.

Run 5
Stages/SendersRecovery-batch-1000
                        time:   [33.418 ms 34.083 ms 34.534 ms]
                        change: [-8.6696% -6.5266% -4.5708%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) low mild
Stages/SendersRecovery-batch-10000
                        time:   [34.921 ms 35.748 ms 36.291 ms]
                        change: [-1.0110% +1.2769% +3.8423%] (p = 0.35 > 0.05)
                        No change in performance detected.
Stages/SendersRecovery-batch-100000
                        time:   [33.248 ms 33.641 ms 34.322 ms]
                        change: [-2.0492% +0.4312% +2.9552%] (p = 0.75 > 0.05)
                        No change in performance detected.
Stages/SendersRecovery-batch-250000
                        time:   [33.160 ms 33.553 ms 33.874 ms]
                        change: [-4.9839% -3.2617% -1.6826%] (p = 0.00 < 0.05)
                        Performance has improved.

Thanks a lot @onbjerg, too bad the results were not what I was expecting once run on better hardware. I was trying to avoid the vector allocation on my first approach, but it failed when trying to create multiple cursors. I've got another approach I want to try that should avoid that extra vector allocation, so I'll close this for the time being and if the other idea works on beefy hardware I'll open another PR. Again, thanks for looking into this! 🙏

Few thoughts here on assumptions, channels in general are very lights it is just a vec and a mutex there isn't a lot of overhead there.
It is made to not write anything until all data is read as to not have a conflict on disk io. And it is read from one place for same reasob
I didnt read PR code, iirc present code is well optimized and the thing that could be do done better is maybe sending tx differently to threads so it is parallized from start.


[QUERY] Access token to connect to Cosmos Tables

Query/Question
Can you connect to Cosmos DB Tables using an access token? Is it possible to create a client that takes an access token to initialize and can interact with the tables in Cosmos? Are there any libraries that support this?
Environment:

Azure.Data.Tables v12 or Microsoft.Azure.Cosmos.Tables v 1.8

Follow up: I see an option for using SAS tokens to create a client for CosmosDb but there doesn't seem to be a way to get a SAS token for Cosmos account. I don't see one in the portal either. So how is this option supposed to be used?

Got confirmation from Cosmos team that MSI authentication for Cosmos Tables is not supported at this time.


Cekees/explicit hashdist dep

This looks fine to me.  We've had this conversation before but I wanted to note that this is 90% morally equivalent to using gitmodules to track a specific commit of hashdist/hashstack.

Yes, I considered that but decided it wasn't worth creating tracking branches. With this approach we just upgrade proteus to a new hashdist or hashstack commit via a proteus  PR's and it's explicit what we're  using.


Expand Parent by expanded  node

Is this option exist ?

You could use getParent and expandNode.  Although it's not clear what you mean.
All available options are documented.


Webclient.login() 4th argument?

Hi, i updated to develop branch today to solve authentication problem. However i now notice when i call ```
webclient.login("email", "password")

TypeError: login() takes exactly 4 arguments (3 given)

If i add a bogus argument to:

webclient.login("email", "password", "randomcharacters")

I successfully authenticate. Before updating to develop and before the auth broke i could call webclient.login with just email and password.

It's an android id, as mentioned in the docs: https://unofficial-google-music-api.readthedocs.org/en/develop/reference/mobileclient.html#gmusicapi.clients.Mobileclient.login. I wouldn't recommend using a random one.

Sorry - my bad, many thanks

Sure, no worries.


Mobile Nav issue with navbar-fixed?

Seems like there is a conflict between mobile side-nav and navbar-fixed.
See this image, where the menu is incomplete. Seems to be rendering only within the nav div.

FYI: I've simply used the template html,  in a Meteor project. Mobile nav works fine without the navbar-fixed class.

This has been fixed in the dev builds, but right now you will just have to remove theoverflow:hidden

@Dogfalo We have to remove overflow:hidden from where?


call module with click decorator fails

fails to me to run imported function with click decorator
run and error code
python example2.py
Traceback (most recent call last):
  File "example2.py", line 3, in <module>
    hello(3,'morgan')
  File "/usr/lib/python2.6/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/click/core.py", line 682, in main
    args = list(args)
TypeError: 'int' object is not iterable

examlpe1.py code  from http://click.pocoo.org/5/
import click

@click.command()
@click.option('--count', default=1, help='Number of greetings.')
@click.option('--name', prompt='Your name',
              help='The person to greet.')
def hello(count, name):
    """Simple program that greets NAME for a total of COUNT times."""
    for x in range(count):
        click.echo('Hello %s!' % name)

if __name__ == '__main__':
    hello()

example2.py
from example1 import hello
hello(3,'morgan')

python and click version
python
Python 2.6.6 (r266:84292, Nov 22 2013, 12:16:22)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import click
>>> click.__version__
'6.3'
>>>

same error with python3
python3
Python 3.5.0 (default, Oct 30 2015, 16:59:56)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import click
>>> click.__version__
'6.3'
>>>

click.command wraps the function with an object that has a different call signature. hello(3, 'morgan') is not valid.


versions: kernel: update to 4.19.26

Update kernel to latest version.
Fixes: #1288
Signed-off-by: Graham Whaley<EMAIL_ADDRESS>

/test
minor version update, let's see if it flies
/cc @jcvenegas @sboeuf

ARM CI fail looks on the surface like a network download error - I'll nudge it - it is in the kernel version area though...
12:44:48 INFO: Current Kernel version <IP_ADDRESS>
12:44:48 INFO: Get packaged kernel version
12:44:48 INFO: Packaged Kernel version
12:44:48 INFO: Install kernel from sources
12:44:48 INFO: Clone config and patches
12:44:48 INFO: Config version: 25
12:44:48 INFO: Kernel version: 4.19.26
12:44:48 INFO: kernel path does not exist, will download kernel
12:44:48   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
12:44:48                                  Dload  Upload   Total   Spent    Left  Speed
12:44:48
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  290k  100  290k    0     0  3267k      0 --:--:-- --:--:-- --:--:-- 3267k
12:44:49 Build step 'Execute shell' marked build as failure

ARM CI is now failing on what look like some CPU/arch type items. Here is a quick summary from the log:
16:42:21 --- FAIL: TestEnvGetEnvInfoSetsCPUType (0.02s)
16:42:21 --- FAIL: TestEnvGetHostInfo (0.00s)
16:42:21 --- FAIL: TestEnvGetEnvInfo (0.04s)
16:42:21 --- FAIL: TestEnvGetEnvInfoNoHypervisorVersion (0.02s)

@Pennyzct - something you can look into?
The 16.04-initrd fail I can't chase now, as that has moved to 18.04 in the CI. The only way to re-trigger that is to do a full test retrigger.... we'll have to decide if we need to (I suspect we do...)
Let's give @Pennyzct a chance to look at the ARM CI first..

Hi~ @grahamwhaley sorry for the delay. got sidetracked. I'm already working on this, related PR online asap. ;)

/test
as the two failing CIs links are now stale.
The only CI I think might currently fail (not due to this PR) is the vsocks ...

Hmm, CI errors still... 3 fails for what look like 3 different reasons:
18.04-initrd
Summarizing 1 Failure:

[Fail] Update CPU constraints Update number of CPUs to check period and quota [It] quota/period should be equal to 3.0
/tmp/jenkins/workspace/kata-containers-runtime-ubuntu-18-04-PR-initrd/go/src/github.com/kata-containers/tests/integration/docker/cpu_test.go:364

Ran 229 of 237 Specs in 1793.968 seconds
FAIL! -- 228 Passed | 1 Failed | 0 Pending | 8 Skipped --- FAIL: TestIntegration (1815.42s)

fedora-vsocks
72002e1944d775736cee99f64ea42e26e3b06829bef87ab32e415bf10e88411e
docker: Error response from daemon: OCI runtime create failed: Failed to check if grpc server is working: context deadline exceeded: unknown.
Checking 17 containers have all relevant components
Wrong number of shims running (17 != 16) - stopping
Wrong number of netmons running (17 != 16) - stopping
Wrong number of 'runtime list' containers running (17 != 16) - stopping
Wrong number of pods in /var/lib/vc/sbs (17 != 16) - stopping)

7.4-fc
Starting stability test: bind_mount_linux.sh
d0cfa96c1495eaa8db0dfbf3769b746f810ea806606f5d43fa4de35281a987f3
docker: Error response from daemon: OCI runtime create failed: Failed to check if grpc server is working: context deadline exceeded: unknown.
make: *** [docker-stability] Error 125
Build step 'Execute shell' marked build as failure

So, either the new kernel/minor update is causing instabilities, or we have some CI instabilities - but, this feels worse than 'the average' for CI right now.
/cc @chavafg for any thoughts.

We know that the fedora vsocks job is unstable right now.
About the firecracker failure, I hadn't seen it before. /cc @GabyCT may know more about this.
The hotplug cpu test failure on the initrd job seems like a real failure:

Stderr:
Running command '/usr/bin/docker [docker exec OpaPO8gnU3T5kYEu9WE9lWVFX0ij7D bash -c perl -e "printf ('%.1f', $(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us)/$(cat /sys/fs/cgroup/cpu/cpu.cfs_period_us))"]'
command failed error 'exit status 126'
[docker exec OpaPO8gnU3T5kYEu9WE9lWVFX0ij7D bash -c perl -e "printf ('%.1f', $(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us)/$(cat /sys/fs/cgroup/cpu/cpu.cfs_period_us))"]
Timeout: 60 seconds
Exit Code: 126
Stdout: OCI runtime exec failed: rpc error: code = Internal desc = Could not run process: container_linux.go:348: starting container process caused "process_linux.go:90: adding pid 105 to cgroups caused \"failed to write 105 to cgroup.procs: write /sys/fs/cgroup/cpu/docker/554b982daa0451b5edfafce6c6c74fd97b4bd49a89443f4ca41a9b45a978ff23/cgroup.procs: invalid argument\"": unknown

/cc @devimc

@grahamwhaley and @chavafg , I have not seen that failure before on firecracker

I'm going to re-spin some CIs here to see if we really do have breakage...

/test
had a few 404's on the CIs, so let's run them all...

I was about to retest all CIs as 3 of the failing jobs have expired anyway. However, @grahamwhaley maybe you could update the branch to remove the conflict and re-push to save me the effort? :)

boo for timing clash ;)

afaict, the only CI we should be ignoring here is that 16.04initrd - as that job no longer exists (that is a stale fail we cannot clear from the list easily....)

And we are now up to 4.19.28, so updated that whilst rebasing....

Hmm, no CI's pending. Let's try...
/retest

:boom:

@grahamwhaley @jodh-intel what's the blocker here? We should be able to merge right?

@sboeuf - only if CIs are happy, and we still have a vsock failure:
Summarizing 1 Failure:

[Fail] [k8s.io] Container runtime should support basic operations on container [BeforeEach] runtime should support removing container [Conformance]
/tmp/jenkins/workspace/kata-containers-runtime-fedora-vsocks-PR/go/src/github.com/kubernetes-sigs/cri-tools/pkg/framework/util.go:152

Ran 5 of 73 Specs in 22.155 seconds
FAIL! -- 4 Passed | 1 Failed | 0 Pending | 68 Skipped

metrics I could believe, as we merged a fix earlier today... but, we probably want to wait until I have tweaked the metrics CI and tried again - so we don't miss any regressions.
Sure, a relatively minor version bump 'should' be safe... but, that's what the CIs are here for...

Here is the fuller error from the vsocks CI log:
• Failure in Spec Setup (BeforeEach) [18.132 seconds]
[k8s.io] Container
/tmp/jenkins/workspace/kata-containers-runtime-fedora-vsocks-PR/go/src/github.com/kubernetes-sigs/cri-tools/pkg/framework/framework.go:72
  runtime should support basic operations on container
  /tmp/jenkins/workspace/kata-containers-runtime-fedora-vsocks-PR/go/src/github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:68
    runtime should support removing container [Conformance] [BeforeEach]
    /tmp/jenkins/workspace/kata-containers-runtime-fedora-vsocks-PR/go/src/github.com/kubernetes-sigs/cri-tools/pkg/validate/container.go:111

    failed to create PodSandbox: rpc error: code = Unknown desc = failed to start sandbox container: failed to create containerd task: Failed to check if grpc server is working: context deadline exceeded: unknown
    Expected error:
        <*status.statusError | 0xc0003abe30>: {
            Code: 2,
            Message: "failed to start sandbox container: failed to create containerd task: Failed to check if grpc server is working: context deadline exceeded: unknown",
            Details: nil,
        }
        rpc error: code = Unknown desc = failed to start sandbox container: failed to create containerd task: Failed to check if grpc server is working: context deadline exceeded: unknown
    not to have occurred

@grahamwhaley oh of course, I just thought those CI tests were flaky. But if this is a true bug introduced by this bump, then yes I agree it needs to be solved before we move :)

CIs have passed. Merging this..


Error When Compiling

Like the title. I got these error when trying to compile the latest version

how did you fix this?


[20.09] python{2,3}Package.pytest-bdd: upgrade and fix test

backport https://github.com/NixOS/nixpkgs/pull/100522
ZHF: #97479
cc @NixOS/nixos-release-managers
fix https://hydra.nixos.org/build/127939944
Things done

[X] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[X] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[X] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[X] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 100754 1

3 packages built:

python27Packages.pytest-bdd
python37Packages.pytest-bdd
python38Packages.pytest-bdd


"Can't read item at address: 0x43CA2B08   en BotwTrainer.MainWindow.LoadDataAsync()"

Everytime i try to use this , when i click "Load" it brings up this error on the error tab.
Any idea why? My game is USA region and i also got 1.1 version

Disable your anti virus?

My anti virus doesnt brings anything :o

Also , more info:
OS: Windows 7 SP1
Wii U Firmware: 5.5.1 U
Game Region: US

Make sure your .net is latest version. Check readme

Check your dotnet version

I downloaded the .net latest version , however , now im getting this:
El valor no puede ser nulo. / Value cannot be null.
Nombre del parámetro / Name of Parameter: text   en System.Windows.Documents.InlineCollection.AddText(String text, Boolean returnIndex)
en BotwTrainer.MainWindow.LogError(Exception ex, String more)
en BotwTrainer.MainWindow.ConnectClick(Object sender, RoutedEventArgs e)

Just got this trainer and getting this same error:
Can't read item at address: 0x43CA2B08   at BotwTrainer.MainWindow.LoadData()
OS: WIn10
Wii U: 5.5.1U
Game Region: US
Notes: No anti-virus. Also when I try to install the suggested .net, it says I already have it or a later version installed.
Not sure what to try next?

Yes you need Bullys TCPGecko as it uses specific server commands that my trainer talks to like JGecko U


Build failure: calibre

Steps To Reproduce
Steps to reproduce the behavior:

build calibre

Build log
Calibre Build Log.txt
Notify maintainers
@AndersonTorres

Metadata
 - system: `"x86_64-linux"`
 - host os: `Linux 6.10.5, NixOS, 24.11 (Vicuna), 24.11pre690827.5633bcff0c61`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.8`
 - channels(root): `"nixos, nixos-hardware"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Add a :+1: reaction to issues you find important.

This appears to be a duplicate of issue #349755, which was resolved in #349884.


nixos/mysql: add test mariadb with tokudb plugin

Motivation for this change
Add test mariadb with tokudb plugin
cc @flokli @aanderse
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Tested locally, works.


Add lookahead for fenced code blocks

Livebook can store outputs in markdown, which can lead
to pretty long code fence tags.
Earmark currently parses all lines, even if they belong to
code fences or html comments. This pull request adds
a small amount of lookahead for those two constructs.
With this, the problematic example report given to us
parses in 10ms, with no timeouts: https://raw.githubusercontent.com/mathiasose/advent-2021/main/aoc.livemd
The lookahead also removes the parallel mapping,
which I have verified to not be efficient in Elixir code
bases, as most lines tend to be short.
I also sneaked in some small optimizations, such as
length(rest) == 0 which is linear to the size of
rest being replaced by rest == [].

Oh, I also killed inside_code, which seems to be unused both in here and Earmark.

Pull Request Test Coverage Report for Build 4a667e7490810d167fc9884c98d0790dab58ab18-PR-70

15 of 15 (100.0%)  changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.7%) to 92.228%

Totals

Change from base Build 2993d0116e0e210cfb0402d97caa4843690b3406:
0.7%

Covered Lines:
712

Relevant Lines:
772

💛  - Coveralls

Just great, thank you so much

I'll release this as 1.5.0 (because of API changes) ASAP

@RobertDober hey! The PR has one more thing to address, see the comments. Please wait for those and we would appreciate a release then :D

hmm @jonatanklosko we should also deprecate the options we have removed as having no effect and release as 1.4.18 I guess

Sounds good to me!

@RobertDober the only option that was removed was timeout and honestly, I wouldn't deprecate it. Users can continue to pass it, it simply won't have an effect.


Driverv don't work after update js-controller: 4.0.23->4.0.24

Hello.
Driverv don't work after update js-controller: 4.0.23->4.0.24
`

telegram.0
2023-01-16 19:16:39.298
error
getMe (reconnect #10) Error:Error: EFATAL: Error: getaddrinfo EAI_AGAIN api.telegram.org

telegram.0
2023-01-16 19:14:49.106
error
getMe (reconnect #0) Error:Error: EFATAL: Error: getaddrinfo EAI_AGAIN api.telegram.org

telegram.0
2023-01-16 19:14:39.085
error
getMe Error:Error: EFATAL: Error: getaddrinfo EAI_AGAIN api.telegram.org

telegram.0
2023-01-16 19:14:29.086
error
setWebHook Error:Error: EFATAL: Error: getaddrinfo EAI_AGAIN api.telegram.org

telegram.0
2023-01-16 19:14:19.028
debug
Start polling with: 300(number) ms interval

telegram.0
2023-01-16 19:14:18.807
debug
Schedule restart: * */12 * * *

telegram.0
2023-01-16 19:14:18.759
info
starting. Version 1.15.2 in /opt/iobroker/node_modules/iobroker.telegram, node: v14.21.2, js-controller: 4.0.24

telegram.0
2023-01-16 19:14:18.473
debug
Plugin sentry Initialize Plugin (enabled=true)

telegram.0
2023-01-16 19:14:11.775
info
Terminated (ADAPTER_REQUESTED_TERMINATION): Without reason

telegram.0
2023-01-16 19:14:11.774
debug
Plugin sentry destroyed

telegram.0
2023-01-16 19:14:11.773
info
terminating

telegram.0
2023-01-16 19:14:11.271
info
Got terminate signal TERMINATE_YOURSELF

telegram.0
2023-01-16 19:14:10.769
error
getMe (reconnect #0) Error:Error: EFATAL: Error: getaddrinfo EAI_AGAIN api.telegram.org

`

Thiserror has nothing to do with the js-controller upgrade normally ... The error means that there is a DNS issue with resolving the given domainnname!
Error: getaddrinfo EAI_AGAIN api.telegram.org
Please check your DNS

I execte ping on a iobroker server
ping api.telegram.org
PING api.telegram.org (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS> (<IP_ADDRESS>): icmp_seq=1 ttl=55 time=67.6 ms
64 bytes from <IP_ADDRESS> (<IP_ADDRESS>): icmp_seq=2 ttl=55 time=73.3 ms

I can only state what the error message is about ... it means that the nodejs process was not able to lookup the IP for the domainname


View 2+ metrics on the tree

Option to view two metrics at a time when visualizing the tree output.


Upgraded dependencies including React 18

Brought dependencies up to date. Tested on local project that uses React 18. Since there are backwards compatibility issues between React 17 and 18, I suggest to release this as a new major version (3.0.0)

It's published now, hope it works :)


required field

I couldn't get required working is there something I am missing?

<mdl-textfield floating-label="First Name:" :required="required" :value.sync="user.name" id="name" name="name"></mdl-textfield>

                                    {{error}}

Also I coudln't get vee-validate working with any mdl-textfields.  It will work on normal input fields.

I couldn't find anything missing.
You should post on the forums or SO to get some help


Emmet in CSS / SCSS

With the Emmet extension installed Emmet snippets are working fine in the HTML part of the code but it doesn't appear to be working for CSS or SCSS within the style tag

@saminton could you elaborate?

Sorry upon rereading I realise I wasn't very clear.
Within a .svelte file the Emmet HTML "snippets" are working as expected as shown here.

But within the <style> tag they are not

The expected behaviour : it should be showing the same "snippets" as within a CSS file like so

I see

It currently works in .html and .css files (and obviously partially in .svelte files) I haven't tried anything else so far


Work around macro expansion bug in 2024-4-1 toolchain.

The latest main toolchain has introduced a regression in macro expansion (tracked by rdar://120559184.) This PR works around the regression, which is causing a build failure in one test.
Checklist:

[x] Code and documentation should follow the style of the Style Guide.
[x] If public symbols are renamed or modified, DocC references should be updated.

@swift-ci please test

@swift-ci please test macOS

@swift-ci please clean test macOS

Remaining failure is a SwiftPM regression tracked as rdar://120560817. All tests are passing in CI and it would be green except for a false EXIT_FAILURE from SwiftPM. Merging with failed CI.


AST: Assert we were given a type parameter in GenericSignatureImpl::requiresClass

Doing the absolute minimum here to ease progression.

Let us see if anything hits the assert
@swift-ci please smoke test

@swift-ci test

@swift-ci test source compatibility

@CodaFi is the source compat job expected not to show any details on failures?

I think you can access the logs from Status -> Build Artifacts

Neat, thank you!

Oh, the release and debug builders both hit a weird internal failure. I suspect there's either a file system race somewhere deep in the build, or they've run out of space. Let's try again.
@swift-ci test source compatibility

@swift-ci test Windows

Oh, the release and debug builders both hit a weird internal failure

Did the release builder not crash on generating SIL for an opaque type property in SwiftMovie?

@swift-ci please test source compatibility debug

@swift-ci please test source compatibility debug

@swift-ci please test source compatibility debug

Yay!

⛵


Made the archived scope definition work with paranoia v2 so that the …

…default scope is first unscoped and then the rest of the logic is still applied. Please accept and merge asap - since I need this for my current codebase

@vraravam merged and thanks for your contribution :)

Is it possible for you to generate a new version of the gem and publish on rubygems.org? That way, i can go back to getting the publicly available gem instead of the GH repo :)

@vraravam done https://rubygems.org/gems/active_admin_paranoia/versions/1.0.5


Add vout to listtransactions/gettransaction

Thanks for adding this update,
Has your testing been successful? Result as intended
Will be pulling this down for testing locally


Robot refactor

several extract methods and turn it to sugar syntax

This branch this ready to merge, the errors are rubocop, which will be arranged as they go confirming the PRs.


phpunit group/exclude-group options not working when using phpunit-slicer

When I use phpunit-slicer to slice my tests, these options are not working.
--group foo
--exclude-group bar

It would be great to slice tests of a given group an allow to slice tests that are not inside a given group
This is funny because all the other phpunit options I use are working.

The prepend option seem not working too
--prepend

@Jerome1337 Is this issue still here with the last version of the library?

I think I've a similar issue, in that running tests in a random order (as configured in the XML in my case: executionOrder="random"), is not honored.


Example integration with TorchElastic

Motivation
Optuna is an invaluable tool for model hyperparameter optimization, and many members of the community will likely be interested in using Optuna in conjunction with frameworks for fault-tolerant distributed training such as TorchElastic.
Currently Optuna supports and provides examples of distributed optimisation where n independent jobs optimize n independent trials. However, Optuna does not have a template (example optimization script) for usage with a Framework like TorchElastic, where Optuna is used in conjunction with distributed training, that is, a group of n dependent elastic workers optimise a single model that corresponds to a single Optuna trial.
Distributed training is common in deep learning use cases, and I think it would be very helpful to have an example of Optuna in the context of distributed training. If such an example already exists and I have missed it, my apologies, but I would be glad to be directed towards it!
Description
The following is a rough schema for a possible approach to using Optuna together with TorchElastic.
Assumptions
The main design assumption for this template is that we are proposing a schema for an Optuna optimization script that can be executed by the TorchElastic launch script, and that will execute all Optuna optimization and training within this script.

Assume a world size of n elastic workers
All workers should have access to the same trial
The trial should propose identical hyperparameters per worker
Only one worker should return optimization results for the trial

Logic of Example Optimization script

Initialize a GPU process group across all workers
One worker (random local rank) executes a study.optimize call to obtain hyperparameters before executing training logic
All workers initialize a temporary process group where the Optuna worker broadcasts its hyperparameters to all other workers
All workers execute identical training logic
The Optuna worker returns the optimization metric value on completion
If any of the workers are interrupted, the process group is restarted and the trial is marked as failed

See this example for a reference regarding the usage of temporary process groups to broadcast data within a worker group.
Complications
We may need to store additional state across surviving workers about the number of successfully completed trials to date. If the worker group restarts because of e.g. node failure, simply using an optimise call with a fixed n_trials can result in more trials than anticipated being run.
Alternatives
This is a topic that I haven't yet found clear design patterns for. There may be better options than my proposed schema, which it would be great to discuss!
In general the goal is to come up with a clear and functional example of integration between Optuna and TorchElastic, which can possibly also give inspiration for integrating Optuna with other distributed training frameworks.

Thank you for your quick response! The issue has been transferred, so let me re-open the issue on this repo.
It sounds great! We are looking forward to your proposal!


Allow contributors to have multiple roles

Attempts to fix #114.
I'm not sure if this is the correct way to do everything as I've not really worked on any Java projects previously, but I thought I'd give it a shot anyways.
Since I've not been able to properly test it locally I'll keep it as a draft until I can do so (thanks to @EnnuiL for the help with that!).

Using an unmodifiable data structure does seem better, thanks for the review! I've added all the suggestions to the branch.

I'm sorry for taking so long - this looks good, thanks for contributing!


Update impersonation_facebook.yml

Description
Changing display name match to regex w/ boundary characters to avoid FPs.
Associated samples

https://platform.sublimesecurity.com/messages/e88063ab9a0eade0e836b270cd5d636e37f995907e17946efa77278c10d16822

/mql-mimic-exempt 102241,227377,329725,271503,340087,338258,341918,335885,341471,345266,356734,357917,136256,343138,384349,384411,384424,384443,397795,391655,399766,405040,405928,414180,416148,416630,418663,427747,429085,435121,448150,458718,458720,458719,530698,539113
All MetaMask impersonation, will write a rule for this specifically.


Initialize DataFrame without specifying order

I was working with DataFrame initialization and think following would be convenient to have. What are your thougts?
# Initialization with array of vectors or arrays without specifying order.
>>> a = b = (1..3).dv
>>> Daru::DataFrame.new [a, b]
#<Daru::DataFrame:27152780 @name = 9e6a7851-96f6-43cb-93af-3ef59081e9a2 @size = 3>
                    0          1
         0          1          1
         1          2          2
         2          3          3

# Initialization with array of Enumerable like ranges without specifying order.
>>> Daru::DataFrame.new [1..3, 1..3]
#<Daru::DataFrame:26079900 @name = b0fb36c0-fd41-4a7e-9a02-36823cc9e1a5 @size = 3>
                    0          1
         0          1          1
         1          2          2
         2          3          3

The first approach is alright (one with Vectors).

Fixed with PR #329 - I think this issue can be closed now. :smile:


Magentic doesn't recognize function when using mistral/mistral-large-latest via litellm

This is on magentic==0.18 and litellm==1.33.4
Reproducible example:
from magentic import prompt_chain
from magentic.chat_model.litellm_chat_model import LitellmChatModel

def get_menu():
    return "On the menu today we have pizza, chips and burgers."

@prompt_chain(
    "<instructions>You are a helpful model that precisely follows instructions. What is on the menu? You can use the get_menu function. Return your answer as a list of strings.</instructions>",
    functions=[get_menu],
    model=LitellmChatModel(model="mistral/mistral-large-latest")
)
def on_the_menu() -> list[str]: ...

on_the_menu()

This leads to magentic complaining that the get_menu function is not known:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[12], line 15
      7 @prompt_chain(
      8     "<instructions>You are a helpful model that precisely follows instructions. What is on the menu? You can use the get_menu function. Return your answer as a list of strings.</instructions>",
      9     functions=[get_menu],
     10     model=LitellmChatModel(model="mistral/mistral-large-latest")
     11     #model=LitellmChatModel(model="anthropic/claude-3-sonnet-20240229")
     12 )
     13 def on_the_menu() -> list[str]: ...
---> 15 on_the_menu()

File ~/miniforge3/envs/obb-ai/lib/python3.11/site-packages/magentic/prompt_chain.py:74, in prompt_chain.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
     72 @wraps(func)
     73 def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
---> 74     chat = Chat.from_prompt(prompt_function, *args, **kwargs).submit()
     75     num_calls = 0
     76     while isinstance(chat.last_message.content, FunctionCall):

File ~/miniforge3/envs/obb-ai/lib/python3.11/site-packages/magentic/chat.py:95, in Chat.submit(self)
     93 def submit(self: Self) -> Self:
     94     """Request an LLM message to be added to the chat."""
---> 95     output_message: AssistantMessage[Any] = self.model.complete(
     96         messages=self._messages,
     97         functions=self._functions,
     98         output_types=self._output_types,
     99     )
    100     return self.add_message(output_message)

File ~/miniforge3/envs/obb-ai/lib/python3.11/site-packages/magentic/chat_model/litellm_chat_model.py:267, in LitellmChatModel.complete(self, messages, functions, output_types, stop)
    259     content = ParallelFunctionCall(
    260         self._select_tool_schema(
    261             next(tool_call_chunks), tool_schemas
    262         ).parse_tool_call(tool_call_chunks)
    263         for tool_call_chunks in _iter_streamed_tool_calls(response)
    264     )
    265     return AssistantMessage(content)  # type: ignore[return-value]
--> 267 tool_schema = self._select_tool_schema(
    268     first_chunk.choices[0].delta.tool_calls[0], tool_schemas
    269 )
    270 try:
    271     # Take only the first tool_call, silently ignore extra chunks
    272     # TODO: Create generator here that raises error or warns if multiple tool_calls
    273     content = tool_schema.parse_tool_call(
    274         next(_iter_streamed_tool_calls(response))
    275     )

File ~/miniforge3/envs/obb-ai/lib/python3.11/site-packages/magentic/chat_model/litellm_chat_model.py:168, in LitellmChatModel._select_tool_schema(tool_call, tools_schemas)
    165         return tool_schema
    167 msg = f"Unknown tool call: {tool_call.model_dump_json()}"
--> 168 raise ValueError(msg)

ValueError: Unknown tool call: {"id":null,"function":{"arguments":"{}","name":"get_menu"},"type":null,"index":0}

Thanks!

Currently expecting the type field of the tool_call to be "function" for function calls rather than null. Seems like a litellm bug but could work around this in magentic by instead checking if the "function" key is present and has a value.

Currently "function" is the only supported type at the moment, but that being said -- it is part of the official OpenAI API spec, which means it should be returned by litellm (since they are intended to emulate the OpenAI API).
I'll open an issue there.

@jackmpcollins I've been able to reproduce -- it occurs when stream=True is set on litellm.complete. I've opened an issue here: https://github.com/BerriAI/litellm/issues/2645

I removed the check for tool_call.type == "function" in https://github.com/jackmpcollins/magentic/releases/tag/v0.18.1  so hopefully this allows you to test out Mistral. I have only tested that this does not break existing functionality.

Apparently the Mistral API matches openai, so it should also be possible to use OpenaiChatModel(..., base_url="").

Apparently the Mistral API matches openai, so it should also be possible to use OpenaiChatModel(..., base_url="") if you encounter any more issues going via litellm.

Tested this and there are some differences in the API. Opening a new issue to track https://github.com/jackmpcollins/magentic/issues/163


Add copyAsync for asynchronous copies between host and device.

As discussed in #216.
Only works for contiguous Float- and CudaTensors.

@soumith @wickedfoo Any thoughts on this?

Looks fine to me, only comment is that this won't really do much (the copy is still effectively synchronous; it copies to a small pinned staging buffer managed by the driver which is then async copied to the GPU; if the buffer is too large, it loops until the last chunk is copied over) until we have a story for CPU pinned memory allocations, and some way to prevent the GC cleanup of that memory while any copies are completing.
I think the story in the longer term to prevent GC issues would be to addref the storage while performing an async copy, then use cudaStreamAddCallback in the current stream for the device for which we're copying memory to/from that will add the storage to an allowed cleanup list, which we look at the next time we try to malloc or free CUDA memory, at which point we release the reference, potentially freeing the storage. The reason why we can't release the memory in the callback is because CUDA APIs apparently can't be called in the callback. I don't know in what CPU thread the callback is made either yet.

We do have cutorch.createCudaHostTensor which allocates pinned memory.
You're right about the GC problem, but isn't that the case with regular CudaTensors and async operations already?

For regular CudaTensors it isn't an issue, because cudaFree is synchronizing and will wait for the completion of all operations on that memory (across all streams). I don't know if it is globally synchronizing (across all devices) or just the current device that holds the memory that is being reclaimed.
I guess the question is if freeing pinned CPU memory is synchronizing. If not, then this isn't a problem; if it is, then we need some kind of dependency wait.

(sorry edited the above comment; if it is synchronizing then it isn't an issue; if it isn't then we have to do something)

Yes, I forgot about that. Not sure if cudaFreeHost also implicitly synchronizes. I'll try to find out.

Thanks @wickedfoo. I've updated the patch.

looks good to me!

I guess we should figure out the story with pinned memory frees though.

thanks @dominikgrewe

Just to update: My contact at NVIDIA told me that cudaFreeHost waits for operations on all devices before the memory is freed. So everything is safe.
cudaFree waits for operations on the device that the memory was allocated on to finish.

Ah, cool.


Use proper FS abstraction when doing bramble.addNewFile() for Blob URL caching

This switches the bramble.addNewFile() API to use the right fs abstraction, such that files get automatically cached in the Blob URL cache.  I've also changed things to only support writing text files this way, which makes things a lot easier, in terms of re-using internal API calls.
NOTE: if you test this locally, you'll need to run grunt build-browser to get the client API changes rebuilt.

r+


Checkpoints + usage of different base BERT model

Hello, I would love to use ColBERT in my project as a retriever component in a retriever-reader architecture, hence I would prefer to use the huggingface API to create the ColBERT model in the code and load the checkpoints :)  I have three small questions:

About the checkpoints of the pretrained on MsMARCO ColBERT. Is it possible to download the pretrained model's weights so I could use them in my implementation? Or the only solution is to use the training script and slightly modify it so I can save the weights for later use.
The used BERT model was the bert-base-uncased. My project's domain is the medical area, hence I was wondering whether using the training script with a small change - replacing the bert-base-uncased with the biobert would result in a properly pretrained model.
Is there any plan on creating the implementation of the the ColBERT's workflow available via the huggingface library?

Thank you very much for your work 💪🏻

You can find a similar version of the model trained by other people here. tbh I am not sure how you can this huggingface pretrained model in a downstream task (ie. index marco or doing retrieval) - in other words whether this is compatible with the existing code repo. It is also a question for me whether we can directly use huggingface models instead of the expected colbert checkpoints that this repo expects during inference (either for retrieval or indexing) @okhat could you help us on that? :)
In this case, you need to train from scratch and make the necessary changes in the code. Briefly, you'd need to go through all of those code matches and change the input of from_pretrained to the model string you want. Surely in the training.py script but also possibly in evaluation

@littlewine thanks for the reply
@1 I found this model, however it has slightly different parameters (linear layer output size and from what I remember the lenght of the embeddings being 512 instead of 768). Still, for now I indeed plan to use that one :) And about the usage in the downstream task - I found it slightly difficult to follow the retrieve script's logic and I guess the smartest choice would be to follow the description of the workflow in the paper.

Hey folks! Happy to send you a checkpoint if you send me an email. A public/official release will take some time.
There are public checkpoints by other people that are compatible with (and produced from) this code, including that by PyTerrier-ColBERT: https://github.com/terrierteam/pyterrier_colbert --- see the Demo section.

We over at vespa.ai have released a ColBERT weights which is based on a 6 layer MiniLM model with just 22.7M parameters which uses 32 dimensions instead of 128.  So for question 2 you can replace bert-base-uncased with a different model which also uses BERT tokenization.
You can use https://huggingface.co/vespa-engine/col-minilm and then use the save_weights on it to get a .dnn checkpoint which can be used with more routines (but with similarity=cosine and dim=32).
(https://github.com/stanford-futuredata/ColBERT/blob/abb5b684e4cda297f3ff58b52e70d5b90270e900/colbert/utils/utils.py#L37)

Thanks all for your activity in the discussion :)

Hi @zyngielg,
Now do you have any idea about how to use huggingface API to create the ColBERT model in the code and load the checkpoints?


Mac OS 11 : Cannot find pf import from Github

I tried this form the tutorial:
app "hello"
    packages { pf: "https://github.com/roc-lang/basic-cli/releases/download/0.1.1/zAoiC9xtQPHywYk350_b7ust04BmWLW00sjb9ZPtSQk.tar.br" }
    imports [pf.Stdout]
    provides [main] to pf

Works fine in LInux.
But in Mac OS 11 I get
── FILE NOT FOUND ──────────────────────────────────────────────── UNKNOWN.roc ─

I am looking for this file, but it's not there:

    https://github.com/roc-lang/basic-cli/releases/download/0.1.1/zAoiC9xtQPHywYk350_b7ust04BmWLW00sjb9ZPtSQk.tar.br

Is the file supposed to be there? Maybe there is a typo in the file
name?⏎

I downloaded the latest nighlty release I found for Mac OS 11
Thanks

Hi @sporto, we encountered a test failure for macos x86_64 that we have not yet been able to resolve. So macos x86_64 releases have been lagging behind. I should have a 2018 mac mini in my possession soon to resolve this.
As a workaround; you can clone the basic-cli repo locally and use a path instead of a URL at pf:.

We can do builds on the github macos 86_64 server but I prefer not to do a build if there are test failures.

@Anton-4 workaround worked.
For the newbies like me, this is how we can use source code as package:
app "cli-tutorial"
    packages { pf: "basic-cli/src/main.roc" }
    imports [pf.Stdout]
    provides [main] to pf

main =
    Stdout.line "Hello, World!"

basic-cli didn't seem to work for me, I think because I annotated my functions maybe? Got errors about not being able to find Program and ExitCode which indeed don't seem to appear in basic-cli.
But providing a path to the examples/cli/cli-program/main.roc in the nightly release examples worked:
app "day1"
    packages { pf: "../roc_nightly-macos_11_x86_64-2022-11-23-0ac6fe7/examples/cli/cli-platform/main.roc" }
    imports [
        pf.File,
        pf.Path,
        pf.Program.{ ExitCode, Program },
        pf.Stderr,
        pf.Stdout,
        pf.Task.{ Task },
    ]
    provides [main] to pf

main : Program
main = Program.noArgs mainTask

mainTask : Task ExitCode [] [Read [File], Write [Stderr, Stdout]]
mainTask =

A new macos x86_64 release is available here. I have not been able to test it on macos 12 but I do expect it to work.
For the packages { pf: you'll now have to use this URL: "https://github.com/roc-lang/basic-cli/releases/download/0.1.2/3bKbbmgtIfOyC6FviJ9o8F8xqKutmXgjCJx3bMfVTSo.tar.br". I'll update the tutorial to use this URL on Friday.
If someone can confirm the new release works on macos 12, we can close this issue :)

Works for me. I'm on macOS 12.6 with M2 silicon
Lukes-MacBook-Air:euler luke$ ./roc p0.roc
Downloading https://github.com/roc-lang/basic-cli/releases/download/0.1.2/3bKbbmgtIfOyC6FviJ9o8F8xqKutmXgjCJx3bMfVTSo.tar.br
    into /Users/luke/.cache/roc/packages

Hello World!

Oh sorry, I meant macos 12 on a x86_64 device. I've edited my earlier message to clarify.

Hi @Anton-4 same error tested on macOS Monterey
$ roc version
roc nightly pre-release, built from commit 0ac6fe7 on Wed Nov 23 10:05:18 UTC 2022

$ uname
Darwin

$ sw_vers
ProductName:	macOS
ProductVersion:	12.6.1
BuildVersion:	21G217

$ cat main.roc
app "cli-tutorial"
    packages { pf: "https://github.com/roc-lang/basic-cli/releases/download/0.1.2/3bKbbmgtIfOyC6FviJ9o8F8xqKutmXgjCJx3bMfVTSo.tar.br" }
    imports [pf.Stdout]
    provides [main] to pf

main =
    Stdout.line "Hello, World!"

$ roc dev
── FILE NOT FOUND ──────────────────────────────────────────────── UNKNOWN.roc ─

I am looking for this file, but it's not there:

    https://github.com/roc-lang/basic-cli/releases/download/0.1.2/3bKbbmgtIfOyC6FviJ9o8F8xqKutmXgjCJx3bMfVTSo.tar.br

Is the file supposed to be there? Maybe there is a typo in the file
name?%

I just go the latest build for MacOS 11
It was working for a while, but odly I started seeing:
I was expecting this file to exist:

    /Users/Sebastian/.cache/roc/packages/github.com/roc-lang/basic-cli/releases/download/0.1.1/zAoiC9xtQPHywYk350_b7ust04BmWLW00sjb9ZPtSQk/macos-x86_64.o

However, it was not there!

I tried deleting the .cache/roc dir. After that it downloads the package again and shows the same error.

I just downloaded the latest MacOS 11 build. This seems to be working fine.

@oliverhr you're still using the old roc release. Can you try the new one?
I should have grouped my "things to change" better :)

I did not test with the version you pointed https://github.com/roc-lang/roc/releases/download/nightly/roc_nightly-macos_11_x86_64-2022-12-07-224dffec0.tar.gz sinces I'm running a v12 of macOS.
But I just tested with roc_nightly-macos_x86_64-2022-12-09-7d8b9b232 and worked as expected
# OS info
Darwin Kernel Version 21.6.0: Thu Sep 29 20:12:57 PDT 2022; root:xnu-8020.240.7~1/RELEASE_X86_64
ProductName:	macOS
ProductVersion:	12.6.1
BuildVersion:	21G217

# roc version
roc nightly pre-release, built from commit 7d8b9b232 on Fri Dec  9 13:37:55 UTC 2022

Awesome :)


Prep: Update CMake files to use include(GoogleCloudCppCommon)

In -cpp, most common CMake settings are configured in one place. So before the code is moved into -cpp, update the google/cloud/$project/CMakeLIsts.txt file to source this one file, and it probably don't need to source any others. Again, this is attempting to make it so the code and build work unchanged when moved into -cpp.
Example: https://github.com/googleapis/google-cloud-cpp-bigquery/pull/48

fixed in https://github.com/googleapis/google-cloud-cpp-pubsub/pull/115


google

Link to project (GitHub): TODO
Describe the project in 2-3 sentences: TODO
Why would Firebase developers be interested in this project?: TODO

Please fill out the issue template.


[PATCH] Operator: HowTo provide Node/Typha TLS certs

Description
Add new doc that was missing on how to configure node / typha TLS certs

Related issues/PRs

Todos

[ ] Tests
[ ] Documentation
[ ] Release note

Release Note

None required

Deploy preview for calico ready!
Built with commit c90fb23511231ed7baeb9582812f5dd25419ed4f
https://deploy-preview-3354--calico.netlify.com


Implementing VD4RL

Implement VD4RL datasets into GATE, and write tests.
Refer to

Dataset template: https://github.com/BayesWatch/RL-GATE/blob/main/gate/data/image/classification/happywhale.py
Test template: https://github.com/BayesWatch/RL-GATE/blob/main/tests/data/image/classification/test_happy_whale_dolphin.py

For examples.

@trevormcinroe Have a chat with Adam, and I am sure you can quickly adapt to this and make faster progress.


⚠ Bofrost.be has degraded performance

In 3166476, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1090 ms

Resolved: Bofrost.be performance has improved in 8538ae8.


Compaction

This PR adds a size-tiered compaction implementation to Segments. Some other refactoring is also included in this PR, and is documented below.
Changes to SSTable include:

index now holds the index
size now holds the size in bytes
SSTable.merge contains a k-way merge implmentation for SSTables
SSTable.Entries is a helper class for the merge implementation

Changes to Segments include:

segment_lock tries to solve the second readers–writers problem, which seemed preferrable since we rarely update the segments list, and we want to delete the files on disk as fast as possible
compaction_lock helps ensure only one thread can run compaction at a time
compaction_thread holds an instance of CompactionThread if a background thread is running
flush now acquires a write lock using segment_lock, and notifies the background thread when a flush has completed
search now acquires a read lock using segment_lock
compact now acquires a compaction_lock around the whole method, and grabs a read lock with segment_lock when computing buckets, then a write lock on segment_lock when updating the segments after a compaction (if any)
start_compaction_thread() creates and starts the background compaction thread
stop_compaction_thread() terminates the background thread
_compaction_loop the entry point for the background thread
_notify_compaction_thread helper that notifies the background thread a new segment has been flushed, but does a null check first
Segments.CompactionThread a helper class that provides an api for controlling the background thread

notify() notifies the background thread that a new segment has been created
start() starts the thread if not already started
stop() safely terminates the background thread
get_task() gets the next event from the work queue
task_done() acknowledges the event has been handled

The new compaction module encapsulates the logic for size-tiered compaction, and provides the following interface:

Bucket represents a bucket of segments that fit a certain size range

min contains the min size that fits in this bucket
max contains the max size that fits in this bucket
avg contains the avg size of files in the bucket
segments contains the actual segments (SSTable's)
compute_bounds() updates the min, max, avg based on the current state of segments
add(segment) adds a segment and updates the bounds
fits(segment) checks if the segment fits in this bucket
size() returns the number of files in the bucket
oldest(n) returns the n oldest segments in the bucket

compute_buckets(segments) generates a list of Buckets from a list of segments
compaction_pass(buckets) tries to perform a single compaction if any of the buckets require it
describe_buckets(buckets) helper that pretty prints a list of Buckets

[Closes #19 ]

Other than that commented out file LGTM.


MQTT

A question.
I've successfully set up ESPHome with this Panasonic code, which is brilliant by the way. But I can't wrap my head around how to get it talking via MQTT, though. (I've got HomeAssistant integrated with CBus via MQTT, and also want to trigger A/C messages to this code from a CBus automation controller via MQTT.)
Got any pointers / tips / examples / doco to read?

hello.
i use mqtt with node-red flow. and domoticz
for the topic's you need.
ac/climate/panasonic_sala/mode/command
ac/climate/panasonic_sala/mode/state
ac/climate/panasonic_sala/target_temperature/command
ac/climate/panasonic_sala/fan_mode/command
everything works only presets-mode doesn't work.
i don't now why.

Looks like I'm on the right path now.
Added this to my yaml and esphome began populating status topics via mqtt. (Or it was, before I blew up the board... LOLs.) Presumably the commands work, based on your hint @redbullterror, although I've not tried those yet.
`mqtt:
broker: 192.168.xxx.xxx
username: mqtt
password: password
climate:

platform: panasonic_ac
type: cnt
name: Store AC
mode_command_topic: storeac/climate/store_ac/mode/command
swing_mode_command_topic: storeac/climate/store_ac/swing_mode/command
fan_mode_command_topic: storeac/climate/store_ac/fan_mode/command
target_temperature_command_topic: storeac/climate/store_ac/target_temperature/command
`

As mentioned, the AC unit and all sensors/switches will automatically be exposed over MQTT as soon as you enable MQTT, you shouldn't have to define any topics manually unless you need to somehow redirect them somewhere else.

I can see all topics through mqtt explorer.
everything works with mqtt too.
only preset doesn't work,
only with home assistant it works to change preset.
looks like he doesn't make topic for preset

Looking at the ESPHome source it seems like the MQTT climate has no support for custom presets, so there's nothing I can do for now.

@DomiStyle it strikes me as inefficent that every climate status update received from the Panasonic results in an MQTT publish. This appears to occur every five seconds, regardless of previous state, and only for the 'climate' topics ('select' topics are not). Is it possible to prevent climate status publish where the value is already known to the ESPHome code?
My CBus SHAC will be responding to every status update, and running code consuming CPU but doing nothing as there is almost always no status change for the A/C (except if something is genuinely changed). Persistent messages means I get current status on topic subscribe, so frequent updates are irrelevant.

@ss4344 The integration currently does not save the previous state or compare to it when a new one arrives.
The polling interval is 5s for the CZ-TACG1 and 30s for the DNSK-P11. This mimics the behavior of the original wifi units.

Is it possible to prevent climate status publish where the value is already known to the ESPHome code?

Sure it is, but is has to be implemented here before parsing the packet.
I'm not planning to do that right now but feel free to open a separate issue for it so it can be tracked.

Looking at the ESPHome source it seems like the MQTT climate has no support for custom presets, so there's nothing I can do for now.

I'm working on a pull request for ESPHome for exactly that. Seems to work
You can use this as external component in the meantime:
external_components:
  - source: github://jmichiel/esphome@mqtt_climate_presets
    components: [ climate, mqtt ]

(if you don't also pull in the climate component, the preset won't get posted for this integration, as it only has custom presets)


Fail to run app leading to Exception invoking readAsArrayBuffer

The application works perfectly on Chrome, sadly the stack-trace tells very little:
com.gargoylesoftware.htmlunit.ScriptException: Exception invoking readAsArrayBuffer
	at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:1001)
	at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:590)
	at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:484)

Any hints would greatly appreciated. I can also share the app and one class demonstrating the problem.

Yes please share the demo class and i will try to fix it.

Please delete this ticket, opened by mistake.

sorry but there is no chance to delete a ticket - hope closing is ok.


The warnings were accurate - there was an extra column in that phaf (oops, my bad). I've fixed the file. Reopen if that doesn't make the warnings go away!

Thanks!  I'll check the next load.

Reopen if that doesn't make the warnings go away!

It's fixed.  Thanks.


💩  NICE-63 notion api killing me [b]

since no new episodes are
being recorded - just hard-code the XML
turn off the dynamic generation on the API RSS for now
and do pass the staticPaths for generation

This pull request has been deployed to Vercel.

Latest commit:
009c77e

✅ Preview:
https://jeromefitzgerald-lq5vvyehz-jerome5.vercel.app

🔍 Inspect:
https://vercel.com/jerome/jeromefitzgerald/FJ3G3fHoGyP1JYCJZqD5cznaZPM7

View Workflow Logs


Error with reply URLs

Both AAD and Facebook redirections will fail, unless respective apps have https://login.microsoftonline.com/te/********.onmicrosoft.com/oauth2/authresp added as authorized reply URLs (B2C domain hidden in the link).

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 906907ec-cbe2-59b1-a84e-ce1b1343756f
Version Independent ID: f498962c-f738-ccd8-a619-de854fac37da
Content: Tutorial - Enable authentication in a web application - Azure Active Directory B2C
Content Source: articles/active-directory-b2c/active-directory-b2c-tutorials-web-app.md
Service: active-directory
Sub-service: b2c
GitHub Login: @davidmu1
Microsoft Alias: davidmu

@mcurko Thanks for your feedback! We will investigate and update as appropriate.

@MarileeTurscak-MSFT: Please assign this item to @mmacy who is now supporting Azure AD B2C.

Any OAuth2 or OpenId Connect identity provider, such as Facebook and AAD, must return the token back to Azure AD B2C in the following URL. Replace your-tenant with the name of your tenant.
https://your-tenant-name.b2clogin.com/your-tenant-name.onmicrosoft.com/oauth2/authresp
If your application use https://login.microsoftonline.com domain for the authorization URL (instead of b2clogin.com), the URL will be:
https://login.microsoftonline.com/your-tenant-name.onmicrosoft.com/oauth2/authresp
We recommend  you to set your application to b2clogin.com. For more information, see: https://docs.microsoft.com/en-us/azure/active-directory-b2c/b2clogin

Hi @mcurko, as there has been no further activity on this issue for some time, I'm requesting the issue be closed. If you have further questions, please don't hesitate to ask.
#please-close