raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)

I was running the !accelerate launch train_dreambooth.py
and come up with this error
The following values were not passed to `accelerate launch` and had defaults used instead:
	`--num_processes` was set to a value of `1`
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--num_cpu_threads_per_process` was set to `1` to improve out-of-box performance
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
For effortless bug reporting copy-paste your error into this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link
================================================================================
/usr/local/lib/python3.7/dist-packages/bitsandbytes/cuda_setup/paths.py:99: UserWarning: /usr/lib64-nvidia did not contain libcudart.so as expected! Searching further paths...
  f'{candidate_env_vars["LD_LIBRARY_PATH"]} did not contain '
/usr/local/lib/python3.7/dist-packages/bitsandbytes/cuda_setup/paths.py:21: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('{"kernelManagerProxyPort"'), PosixPath('"/usr/local/bin/dap_multiplexer","enableLsp"'), PosixPath('6000,"kernelManagerProxyHost"'), PosixPath('true}'), PosixPath('["--ip=<IP_ADDRESS>"],"debugAdapterMultiplexerPath"'), PosixPath('"<IP_ADDRESS>","jupyterArgs"')}
  "WARNING: The following directories listed in your path were found to "
/usr/local/lib/python3.7/dist-packages/bitsandbytes/cuda_setup/paths.py:21: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('module'), PosixPath('//ipykernel.pylab.backend_inline')}
  "WARNING: The following directories listed in your path were found to "
/usr/local/lib/python3.7/dist-packages/bitsandbytes/cuda_setup/paths.py:21: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/env/python')}
  "WARNING: The following directories listed in your path were found to "
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 111
CUDA SETUP: Loading binary /usr/local/lib/python3.7/dist-packages/bitsandbytes/libbitsandbytes_cuda111.so...
Caching latents: 100% 12/12 [00:11<00:00,  1.04it/s]
Steps:   0% 0/900 [00:00<?, ?it/s]Traceback (most recent call last):
  File "train_dreambooth.py", line 657, in <module>
    main()
  File "train_dreambooth.py", line 600, in main
    noise_pred = unet(noisy_latents, timesteps, encoder_hidden_states).sample
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/utils/operations.py", line 507, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/usr/local/lib/python3.7/dist-packages/torch/amp/autocast_mode.py", line 12, in decorate_autocast
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/diffusers/models/unet_2d_condition.py", line 309, in forward
    upsample_size=upsample_size,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/diffusers/models/unet_blocks.py", line 1151, in forward
    hidden_states = attn(hidden_states, context=encoder_hidden_states)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/diffusers/models/attention.py", line 154, in forward
    hidden_states = block(hidden_states, context=context)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/diffusers/models/attention.py", line 203, in forward
    hidden_states = self.attn1(self.norm1(hidden_states)) + hidden_states
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/diffusers/models/attention.py", line 278, in forward
    hidden_states = self._attention(query, key, value)
  File "/usr/local/lib/python3.7/dist-packages/diffusers/models/attention.py", line 287, in _attention
    attention_probs = attention_scores.softmax(dim=-1)
RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 14.76 GiB total capacity; 12.29 GiB already allocated; 977.75 MiB free; 12.56 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Steps:   0% 0/900 [00:02<?, ?it/s]
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/accelerate_cli.py", line 43, in main
    args.func(args)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 837, in launch_command
    simple_launcher(args)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 354, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', 'train_dreambooth.py', '--pretrained_model_name_or_path=CompVis/stable-diffusion-v1-4', '--use_auth_token', '--instance_data_dir=/content/data/yasirInput', '--class_data_dir=/content/data/person', '--output_dir=/content/drive/MyDrive/stable_diffusion_weights/yasirOutput', '--with_prior_preservation', '--prior_loss_weight=1.0', '--instance_prompt=photosofyasir', '--class_prompt=person', '--seed=1337', '--resolution=512', '--center_crop', '--train_batch_size=1', '--mixed_precision=fp16', '--use_8bit_adam', '--gradient_accumulation_steps=1', '--learning_rate=5e-6', '--lr_scheduler=constant', '--lr_warmup_steps=0', '--num_class_images=12', '--sample_batch_size=4', '--max_train_steps=900']' returned non-zero exit status 1.

please give me some solutions

Same error, why no one help here?

Not much I can do here, your GPU does seem to run out of memory. It seems your attention is too large to fit into memory. I recommend using something to reduce the memory footprint further such as FlashAttention or gradient checkpointing.


Upgrade To ZIO 2.0.0-RC4

@sviezypan That's great feedback! I think the issue is most likely to be in the test framework rather than the resource management itself. We moved to a new streaming format for test output in RC4 which required a significant redesign of the internals of ZIO Test. We've made some changes that are relevant to this so I would try with the latest snapshot first and then if you are still having issues we can spend more time on it.
Copying @swoogles.

@sviezypan It looks like all tests are passing now for real this time. Would you please take a look when you have the chance?


fix(deps): update react

This PR contains the following updates:

Package
Type
Update
Change

@lingui/react
dependencies
minor
2.8.3 -> 2.9.0

@types/react
devDependencies
patch
16.9.13 -> 16.9.16

react-jsonschema-form
dependencies
patch
1.8.0 -> 1.8.1

Release Notes

lingui/js-lingui
v2.9.0
Compare Source
New Features

Add optional sort by origin/filename (#563).
Thanks to Kenneth Skovhus
Lazily split messages by value tags (#593).
Thanks to Danny Sellers
Optional line numbers for lingui format (#587).
Thanks to Martti Roitto
Moved typescript dependency to peer (#589).
Thanks to Daniel K.

Bug Fixes

Compile strict skips pseudo locale (#584).
Thanks to Daniel Chabr

mozilla-services/react-jsonschema-form
v1.8.1
Compare Source
Fixes

Use setTimeout instead of setImmediate (#1482)
Fix some vulnerabilities

Renovate configuration
:date: Schedule: At any time (no schedule defined).
:vertical_traffic_light: Automerge: Enabled.
:recycle: Rebasing: Whenever PR becomes conflicted, or if you modify the PR title to begin with "rebase!".
:ghost: Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

Available now for Enterprise: Renovate Pro with real-time webhook handling and priority job queue.

:tada: This PR is included in version 2.152.23 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


kui search-bar doesn't work in webpack mode

Only browser's built-in search bar popped out when cmd+f

Prolly disable the following browser support of Kui search bar. We could just use browser's search bar.
https://github.com/IBM/kui/blob/fe063deaeae704b0ec410bffe52cb4bda1ebe219/plugins/plugin-core-support/src/lib/text-search.ts#L47-L50


Initial commit

Python Scripts used to retrieve, from OpenHub Data set,  open source projects located in github that implement Java and C/C++ in their code .
outputs from all stages of filtering.

Okay, but I suggested to move data sets to another folder.


Online Demo Link Broken?

We've been watching this repo so we can start reusing as it approaches later previews.
I was going to show this to some other team members but looks like the repo demo link is not working?
https://aka.ms/blazorweather
Is there another online URL where this demo can be hit rather than running the solution locally?
Thanks in advance.

Hi @bigredky. I haven't deployed the new .NET 6 version of the app anywhere public yet unfortunately. The web app should be pretty easy to run locally though. For the .NET MAUI app, a bit more setup is required: https://docs.microsoft.com/dotnet/maui/get-started/installation.

Gotcha. No worries. I'll do that. Have a good one.


Support Python 3.12

Pull Request Test Coverage Report for Build<PHONE_NUMBER>8
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 61.519%

Totals

Change from base Build<PHONE_NUMBER>1:
0.0%

Covered Lines:
494

Relevant Lines:
803

💛  - Coveralls


Bdc dashboard tweaks

Tweaks for ALS-7478 and ALS-7477 to remove trailing periods after public study consents and add a check for "show_dashboard"=false

Ignore -have no idea how this pushed to this repo instead of dictionary, fixing it now


multicore projections tracing runs hang at startup on 129 pes

Original author: Jim Phillips (@jcphill)
Original issue: https://charm.cs.illinois.edu/redmine/issues/1080

jim`triton$./namd2.prj +p129 $APOA1 --outputenergies 100 +logsize 1000000 +traceroot /Scr/jim/triton +traceoff
Charm++: standalone mode (not using charmrun)
Charm++> Running in Multicore mode:  129 threads
Charm++> Using recursive bisection (scheme 3) for topology aware partitions
Converse/Charm++ Commit ID: v6.7.1-rc1-0-gbdf6a1b-namd-charm-6.7.1-proj-build-2016-Apr-16-217305
Trace: logsize: 1000000
Charm++: Tracemode Projections enabled.
Trace: traceroot: /Scr/jim/triton/namd2.prj
Warning> Randomization of stack pointer is turned on in kernel, thread migration may not work! Run 'echo 0 > /proc/sys/kernel/randomize_va_space' as root to disable it, or try run with '+isomalloc_sync'.
CharmLB> Load balancer assumes all CPUs are same.
Charm++> Running on 1 unique compute nodes (272-way SMP).
Charm++> cpu topology info is gathered in 0.047 seconds.
[hangs until killed]

jim`triton$./namd2.prj +p128 $APOA1 --outputenergies 100 +logsize 1000000 +traceroot /Scr/jim/triton +traceoff
Charm++: standalone mode (not using charmrun)
Charm++> Running in Multicore mode:  128 threads
Charm++> Using recursive bisection (scheme 3) for topology aware partitions
Converse/Charm++ Commit ID: v6.7.1-rc1-0-gbdf6a1b-namd-charm-6.7.1-proj-build-2016-Apr-16-217305
Trace: logsize: 1000000
Charm++: Tracemode Projections enabled.
Trace: traceroot: /Scr/jim/triton/namd2.prj
Warning> Randomization of stack pointer is turned on in kernel, thread migration may not work! Run 'echo 0 > /proc/sys/kernel/randomize_va_space' as root to disable it, or try run with '+isomalloc_sync'.
CharmLB> Load balancer assumes all CPUs are same.
Charm++> Running on 1 unique compute nodes (272-way SMP).
Charm++> cpu topology info is gathered in 0.054 seconds.
Info: NAMD 2.11 for Linux-x86_64-multicore
Info:

Original date: 2016-05-20 23:15:49

I was able to replicate this on triton, but I was also able (somewhat oddly) to successfully run a 129 PE job with Projections.  I'm not sure why it sometimes works, I'll keep looking into it.

Original date: 2016-05-23 00:36:13

I'm not sure how the status and target version were changed in my earlier update, I'm setting them back here.

Original date: 2016-12-07 20:34:19

Status? This could be release critical.

Original date: 2016-12-07 20:47:52

I'll try to dig into this with thread sanitized or valgrind (use --fair-sched=yes).  We suspect that this is a race condition that arises at high PE counts since this works sometimes.

Original date: 2016-12-14 21:14:05

Seonmyeong reported a segfault with Projections on KNL using >128 PEs. Adding him as a 'watcher' here.

Original date: 2016-12-20 16:24:47

EDIT: nevermind.

Original date: 2016-12-29 19:09:53

With error checking enabled, the stack trace looks like:
#0  0x00007ffff6c31845 in raise () from /lib/libc.so.6
#1  0x00007ffff6c35390 in abort () from /lib/libc.so.6
#2  0x0000000000612e7e in charmrun_abort (
    s=0x66a5e0 "Can only do registrations from rank 0 processors") at machine.c:1029
#3  0x00000000006128d2 in LrtsAbort (
    message=0x66a5e0 "Can only do registrations from rank 0 processors")
    at machine.c:551
#4  0x000000000061221a in CmiAbortHelper (source=0x681d71 "Called CmiAbort",
    message=0x66a5e0 "Can only do registrations from rank 0 processors",
    suggestion=0x0, tellDebugger=1, framesToSkip=0) at machine-common-core.c:1451
#5  0x0000000000612249 in CmiAbort (
    message=0x66a5e0 "Can only do registrations from rank 0 processors")
    at machine-common-core.c:1455
#6  0x000000000053c970 in CkRegisteredInfo<EntryInfo>::add (
    this=0x90f820 <_entryTable>, t=0x7ffa040e3610) at register.h:275
#7  0x000000000053b635 in CkRegisterEp (
    name=0x675bc8 "AddToInactiveList(CkReductionInactiveMsg* impl_msg)",
    call=0x5bbb42 <CkIndex_CkReductionMgr::_call_AddToInactiveList_CkReductionInactiveMsg(void*, void*)>, msgIdx=0, chareIdx=0, ck_ep_flags=8) at register.C:63
#8  0x00000000005bbb2b in CkIndex_CkReductionMgr::reg_AddToInactiveList_CkReductionInactiveMsg () at CkReduction.def.h:1256
#9  0x00000000005c2506 in CkIndex_CkReductionMgr::idx_AddToInactiveList_CkReductionInactiveMsg () at CkReduction.decl.h:1011
#10 0x00000000005ba4c1 in CProxyElement_CkReductionMgr::AddToInactiveList (
    this=0x7fffb37a54e0, impl_msg=0x7ffa04000910) at CkReduction.def.h:857
#11 0x00000000005ade47 in CkReductionMgr::informParentInactive (this=0x7ffa040e2ee0)
    at ckreduction.C:553
#12 0x00000000005adb6a in CkReductionMgr::checkIsActive (this=0x7ffa040e2ee0)
    at ckreduction.C:499
#13 0x00000000005ad65d in CkReductionMgr::doneCreatingContributors (
    this=0x7ffa040e2ee0) at ckreduction.C:312
#14 0x00000000005a9b3c in CkArrayReducer::ckEndInserting (this=0x7ffa040e3340)
    at ckarray.C:147
#15 0x00000000005a0153 in CkArray::remoteDoneInserting (this=0x7ffa040e2ee0)
    at ckarray.C:1232
#16 0x00000000005a4c24 in CkIndex_CkArray::_call_remoteDoneInserting_void (
    impl_msg=0x7fff6c1179f0, impl_obj_void=0x7ffa040e2ee0) at CkArray.def.h:974
#17 0x0000000000541e2b in CkDeliverMessageFree (epIdx=71, msg=0x7fff6c1179f0,
    obj=0x7ffa040e2ee0) at ck.C:593
#18 0x0000000000541f7f in _invokeEntryNoTrace (epIdx=71, env=0x7fff6c1179a0,
    obj=0x7ffa040e2ee0) at ck.C:637
#19 0x000000000054208e in _invokeEntry (epIdx=71, env=0x7fff6c1179a0,
    obj=0x7ffa040e2ee0) at ck.C:648
#20 0x000000000054369f in _deliverForBocMsg (ck=0x7ffcbc04b110, epIdx=71,
    env=0x7fff6c1179a0, obj=0x7ffa040e2ee0) at ck.C:1095
#21 0x0000000000543752 in _processForBocMsg (ck=0x7ffcbc04b110, env=0x7fff6c1179a0)
    at ck.C:1107
#22 0x0000000000543bc7 in _processHandler (converseMsg=0x7fff6c1179a0,
    ck=0x7ffcbc04b110) at ck.C:1238
#23 0x000000000061944b in CmiHandleMessage (msg=0x7fff6c1179a0) at convcore.c:1814
#24 0x00000000006197d8 in CsdScheduleForever () at convcore.c:2051
#25 0x000000000061971e in CsdScheduler (maxmsgs=-1) at convcore.c:1987
#26 0x0000000000611f6f in ConverseRunPE (everReturn=0) at machine-common-core.c:1297
#27 0x000000000060f6ea in call_startfn (vindex=0x7f) at machine-smp.c:406
#28 0x00007ffff79af9ca in start_thread () from /lib/libpthread.so.0
#29 0x00007ffff6ce945d in clone () from /lib/libc.so.6
#30 0x0000000000000000 in ?? ()

Without error checking, it segfaults with the following stack trace:
#0  0x00007ffff79b1d80 in pthread_mutex_destroy () from /lib/libpthread.so.0
#1  0x000000000060ee42 in LrtsDestroyLock (lock=0x0) at machine-common-core.c:1656
#2  0x000000000060c2f6 in CmiDestroyLocks () at machine-smp.c:519
#3  0x000000000060ee6f in machine_exit (status=1) at machine.c:331
#4  0x000000000060f12b in LrtsAbort (
    message=0x665d10 "Charm++: Invalid entry method executed.  Perhaps there is an unregistered module?") at machine.c:540
#5  0x000000000060eacc in CmiAbortHelper (source=0x677fe9 "Called CmiAbort",
    message=0x665d10 "Charm++: Invalid entry method executed.  Perhaps there is an unregistered module?", suggestion=0x0, tellDebugger=1, framesToSkip=0)
    at machine-common-core.c:1451
#6  0x000000000060eafb in CmiAbort (
    message=0x665d10 "Charm++: Invalid entry method executed.  Perhaps there is an unregistered module?") at machine-common-core.c:1455
#7  0x000000000053a5f2 in ckInvalidCallFn (msg=0x7fff540e5080, obj=0x7ffe38342ca0)
    at register.C:42
#8  0x0000000000540c05 in CkDeliverMessageFree (epIdx=0, msg=0x7fff540e5080,
    obj=0x7ffe38342ca0) at ck.C:593
#9  0x0000000000540d34 in _invokeEntryNoTrace (epIdx=0, env=0x7fff540e5030,
    obj=0x7ffe38342ca0) at ck.C:637
#10 0x0000000000540e43 in _invokeEntry (epIdx=0, env=0x7fff540e5030,
    obj=0x7ffe38342ca0) at ck.C:648
#11 0x0000000000542323 in _deliverForBocMsg (ck=0x7ffdd0080270, epIdx=0,
    env=0x7fff540e5030, obj=0x7ffe38342ca0) at ck.C:1095
#12 0x00000000005423d6 in _processForBocMsg (ck=0x7ffdd0080270, env=0x7fff540e5030)
    at ck.C:1107
#13 0x000000000054283a in _processHandler (converseMsg=0x7fff540e5030,
    ck=0x7ffdd0080270) at ck.C:1238
#14 0x0000000000615bec in CmiHandleMessage (msg=0x7fff540e5030) at convcore.c:1814
#15 0x0000000000615f79 in CsdScheduleForever () at convcore.c:2051
#16 0x0000000000615ebf in CsdScheduler (maxmsgs=-1) at convcore.c:1987
#17 0x000000000060e821 in ConverseRunPE (everReturn=0) at machine-common-core.c:1297
#18 0x000000000060c134 in call_startfn (vindex=0x3a) at machine-smp.c:406
#19 0x00007ffff79af9ca in start_thread () from /lib/libpthread.so.0
#20 0x00007ffff6ce945d in clone () from /lib/libc.so.6
#21 0x0000000000000000 in ?? ()

Original date: 2017-01-13 01:07:50

This ended up being caused by an array overflow overwriting a static guard, which then allowed a previously called function that had already been assigned to a static variable to be called again, causing erroneous behavior.
To fix this, the array, which is responsible for storing reductions, was changed to a vector.  The fix is at ~~https://charm.cs.illinois.edu/gerrit/#/c/2117/~~ https://github.com/UIUC-PPL/charm/commit/8ade8e5bdca44e87a55145d3a292438446e3242c


Where I can find the resize functions of jquery-gridly?

All I see on jquery-gridly is the drag&drop feature but nothing on re-seizing? How does the re-sizing work and where I can find the implementation?

This lib sucks, try to find another lib with responsive feature

I think you can simply init the gridly with the finction gridly() in your resize hndler. Anyway you can have a try.


Only re-create tags when required

issue seems to stem from re-creation of tags during a refresh
need to check before re-creating, adds equals and hashcode
not sure why this behaviour differs between the two databases

Note for future: the issue also had to do with a conflict between the default values via postgres and the @SequenceGenerator annotation and related via JPA. Our migrated tables retained default values which had an issue whereas newly created tables (whether via migration or hibernate create) did not.


Manually downloading updates fails silently

Microsoft PowerToys version
0.47.0
Running as admin

[X] Yes

Area(s) with issue?
Installer
Steps to reproduce
This is hard to reproduce as it involved the manual download button, which is only displayed when an update is available.
I had the settings app open and noticed that there was an update available, so I clicked the download button and the settings app closed without installing the update and without any warning.
The reason in my case was a corporate firewall that required authentication, but I'd guess that the problem would be similar with any other connectivity issues.
✔ Expected Behavior
The update should download and install, or the user should be alerted if there's a problem.
❌ Actual Behavior
The update could not be downloaded, but the app closed without alerting the user.
Other Software
No response

Hi, just had this same experience again upgrading to 51.1.

Is this still an issue with the latest version?


Made cardano-node compatible with changes in ShelleyGenesis

The sgsPools and sgsStake fields in ShelleyGenesisStaking were changed to association lists to make them streamable.
This PR fixes the errors caused by the change.
I also updated the cabal files to point to the commit of cardano-ledger that introduces these breaking changes.
https://github.com/input-output-hk/cardano-ledger/pull/2871

I'll close this PR since #4118 has more discussion around it.


CollectionView scrolling disabled.

Hi.
I apologize if this question has already been asked.
Inheritance from SwipeCollectionViewCell disables default collectionView scrolling, even if you return nil in editActionsForItemAt indexPath:  function.
Is there a simple way, if a cell doesn't have any actions, don't block default scrolling for CollectionView?
Thanks!

I can't reproduce your issue with the example Mail app provided. I attempted to reproduce it by returning an empty SwipeActions array, and I can scroll the collection view just fine.

@kurabi
What about horizontal scrolling. When you have a lane of items and you want to scroll them horizontally, if you have no actions for cell.

Any news on this problem ?
I have some cells that in witch I want to disable the "swipe" ability and others I want to be able to swipe in...

@AndreiRybak In that case, you should provide a different type of cell to the CollectionView.
The editActionsForItemAt:indexpath: is to provide temporary way to disable swipe actions. For example, the table/collection view is in an "edit" state where cells should not swipe. This way, if the user try to swipe while the table/collection view, the library continuously monitors if the edit state is no longer in effect.


"Install requirements and download repositories" error

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [1], in <cell line: 8>()
      4 # Import launch.py which will automatically run the install script but not launch the WebUI.
      5 # They require a few specific external git repo commits so we have to do it their way.
      6 # !python3 launch.py --exit
      7 import launch
----> 8 launch.prepare_enviroment()
     10 # Install requirements for this notebook
     11 get_ipython().system('pip install requests validators')

File /notebooks/stable-diffusion-webui/launch.py:142, in prepare_enviroment()
    140 git_clone("https://github.com/CompVis/taming-transformers.git", repo_dir('taming-transformers'), "Taming Transformers", taming_transformers_commit_hash)
    141 git_clone("https://github.com/crowsonkb/k-diffusion.git", repo_dir('k-diffusion'), "K-diffusion", k_diffusion_commit_hash)
--> 142 git_clone("https://github.com/sczhou/CodeFormer.git", repo_dir('CodeFormer'), "CodeFormer", codeformer_commit_hash)
    143 git_clone("https://github.com/salesforce/BLIP.git", repo_dir('BLIP'), "BLIP", blip_commit_hash)
    145 if not is_installed("lpips"):

File /notebooks/stable-diffusion-webui/launch.py:74, in git_clone(url, dir, name, commithash)
     71 if commithash is None:
     72     return
---> 74 current_hash = run(f'"{git}" -C {dir} rev-parse HEAD', None, f"Couldn't determine {name}'s hash: {commithash}").strip()
     75 if current_hash == commithash:
     76     return

File /notebooks/stable-diffusion-webui/launch.py:32, in run(command, desc, errdesc)
     24     if result.returncode != 0:
     26         message = f"""{errdesc or 'Error running command'}.
     27 Command: {command}
     28 Error code: {result.returncode}
     29 stdout: {result.stdout.decode(encoding="utf8", errors="ignore") if len(result.stdout)>0 else '<empty>'}
     30 stderr: {result.stderr.decode(encoding="utf8", errors="ignore") if len(result.stderr)>0 else '<empty>'}
     31 """
---> 32         raise RuntimeError(message)
     34     return result.stdout.decode(encoding="utf8", errors="ignore")

RuntimeError: Couldn't determine CodeFormer's hash: c5b4593074ba6214284d6acd5f1719b6c5d739af.
Command: "git" -C repositories/CodeFormer rev-parse HEAD
Error code: 128
stdout: HEAD

stderr: fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

They probably broke something again. Try updating the repository (start from the beginning of the notebook and run all the blocks again).

oops didn't mean to close

They probably broke something again. Try updating the repository (start from the beginning of the notebook and run all the blocks again).

I did this and it was resolved. I was having other issues which I think relate to transformers using up all of the available space, causing it to no longer work on the free tier. If this continues, I'll make a separate issue about it. Thank you!


Unrecognized command during establishing TLS connection

I've setup a FTP Server with explicit TLS settings. Sometimes (randomly) there are unrecognized command errors with the attached debug output. This error appears most frequently on the first connection after server startup. After connection timeout the next connection is established sucessfully.
Implementation of Certificate:
var cert = new X509Certificate2("localhost.pfx", "password");
            services.Configure<AuthTlsOptions>(cfg => {
                cfg.ServerCertificate = cert;
                cfg.ImplicitFtps = false;
            });

Debug Log:

NGettext: Creating a built-in plural rule for langcode "en" for locale "en".
NGettext: Translation not found for message id "FTP Server Ready".
info: FubarDev.FtpServer.FtpConnection[0]
Connected from <IP_ADDRESS>:65498
FubarDev.FtpServer.FtpConnection: Information: Connected from <IP_ADDRESS>:65498
dbug: FubarDev.FtpServer.ServerCommandHandlers.SendResponseServerCommandHandler[0]
220 FTP Server Ready
FubarDev.FtpServer.ServerCommandHandlers.SendResponseServerCommandHandler: Debug: 220 FTP Server Ready
dbug: FubarDev.FtpServer.FtpConnection[0]
AUTH TLS
FubarDev.FtpServer.FtpConnection: Debug: AUTH TLS
NGettext: Translation not found for message id "Enabling TLS Connection".
dbug: FubarDev.FtpServer.ServerCommandHandlers.PauseConnectionServerCommandHandler[0]
Receiver paused
FubarDev.FtpServer.ServerCommandHandlers.PauseConnectionServerCommandHandler: Debug: Receiver paused
dbug: FubarDev.FtpServer.ServerCommandHandlers.SendResponseServerCommandHandler[0]
234 Enabling TLS Connection
FubarDev.FtpServer.ServerCommandHandlers.SendResponseServerCommandHandler: Debug: 234 Enabling TLS Connection
dbug: FubarDev.FtpServer.FtpConnection[0]
????<4v;F??K???Pmk?'rH?M?p4?S????0?,?(?$??
?????????k?j?i?h?9?8?7?6?2?.?*?&?????=?5?/?+?'?#?? ?????????g?@???>?3?2?1?0?1?-?)?%?????<?/?????????????
FubarDev.FtpServer.FtpConnection: Debug: NGettext: Translation not found for message id "Syntax error, command unrecognized.".
dbug: FubarDev.FtpServer.FtpConnection[0]
?
FubarDev.FtpServer.FtpConnection: Debug: ?
NGettext: Translation not found for message id "Syntax error, command unrecognized.".
dbug: FubarDev.FtpServer.FtpConnection[0]
??
??E????
???????????
FubarDev.FtpServer.FtpConnection: Debug: ?NGettext: Translation not found for message id "Syntax error, command unrecognized.".

Which version do you use? It looks like the server received the TLS client authentication before it was able to stop the receiving process and thus tried to interpret the data as FTP command.

I'm using Version 3.1.1 with .NET core 3.0.1
My Startup is basically like the example QuickStart.AspNetCoreHost but with custom implementations for Authentication (Database) and FileSystem (Mixed InMemory/Database).
Are there any further code examples or information i could provide to help?

Hi,
anything new on this bug?

No, not yet, as I currently don't have much time and I'm not sure why this still happens (sometimes).


introduce retry for fetching data source configurations

Changes proposed

Introduce retry functionality for fetching data source configurations
configurable through environment variable DATA_STORE_CONN_RETRY

Check List (Check all the applicable boxes)

[x] I sign off on contributing this submission to open-source
[x] My code follows the code style of this project.
[x] My change requires changes to the documentation.
[x] I have updated the documentation accordingly.
[x] All new and existing tests passed.
[x] This PR does not contain plagiarized content.
[x] The title of my pull request is a short description of the requested changes.

Screenshots

Note to reviewers

Hi @thilak009,
Hmm... the FetchingEngine being used by the data fetcher already has a retry mechanism - look for BaseFetchProvider.DEFAULT_RETRY_CONFIG.
Does it not actually take effect?
It's also doesn't seem to be configurable... So your contribution is a good idea - But I would much rather it make use of the existing mechanism.
See FetchingEngine.queue_url - when an FetchEvent is initiated, you can provide retry configuration.
It would also be preferable because I plan on merging a change where the actual data fetches would occur asynchronously.

Hi @thilak009,
Hmm... the FetchingEngine being used by the data fetcher already has a retry mechanism - look for BaseFetchProvider.DEFAULT_RETRY_CONFIG. Does it not actually take effect? It's also doesn't seem to be configurable... So your contribution is a good idea - But I would much rather it make use of the existing mechanism. See FetchingEngine.queue_url - when an FetchEvent is initiated, you can provide retry configuration. It would also be preferable because I plan on merging a change where the actual data fetches would occur asynchronously.

Hm, i see that a FetchEvent has an option to pass in retry configuration
but the queue_fetch_event method only raises either a asyncio.QueueFull or a asyncio.TimeoutError exception and hence retry wouldn't happen for a bad status code
I see that the underlying class which makes the HTTP requests in fetcher is HttpFetchProvider
does editing the _fetch_ to raise a HTTP bad status code exception make sense if we want to use the existing retry mechanism of FetchEvent ?
this also makes me ask the question of why would we want to retry if we get a bad status code, not sure if this is a common use case, but the reason i ended up at a retry was because my data source server and OPAL client reside in same pod.
The data is fetched through a public URL for the data source server, but by the time pod starts and ingress becomes active, OPAL client would have already tried fetching data and since ingress isn't active, it gets a 502, so my OPAL client ends up with no data and health checks failing
retries solves this as ingress becomes active by 2nd or 3rd retry and fetch is successful
wondering if retry on bad status code is actually useful, would like to know your views @roekatz on this, if it makes sense we can see how to do it

Hi @thilak009,
Hmm... the FetchingEngine being used by the data fetcher already has a retry mechanism - look for BaseFetchProvider.DEFAULT_RETRY_CONFIG. Does it not actually take effect? It's also doesn't seem to be configurable... So your contribution is a good idea - But I would much rather it make use of the existing mechanism. See FetchingEngine.queue_url - when an FetchEvent is initiated, you can provide retry configuration. It would also be preferable because I plan on merging a change where the actual data fetches would occur asynchronously.

Hm, i see that a FetchEvent has an option to pass in retry configuration but the queue_fetch_event method only raises either a asyncio.QueueFull or a asyncio.TimeoutError exception and hence retry wouldn't happen for a bad status code
I see that the underlying class which makes the HTTP requests in fetcher is HttpFetchProvider does editing the _fetch_ to raise a HTTP bad status code exception make sense if we want to use the existing retry mechanism of FetchEvent ?

The queue_fetch_event only puts the fetch request in queue.
The queue is handled by fetch_worker which reads from that queue, then calls get_fetcher_for_event which creates a new fetch provider and sets the retry configuration from the FetchEvent object itself (by using BaseFetchProvider.set_retry_config).
The retry then, AFAIU, would occur for any kind of Exception.

The retry then, AFAIU, would occur for any kind of Exception.

i don't think so, for a bad status code let's say for a 5XX response, the queue_fetch_event does not raise any exception for this scenario, which means if i understand correctly fetch_worker or any other part which is handling the HTTP call does not raise an exception for 5XX status code
tenacity retry only does a retry if an exception is raised and hence in this case, retry does not happen
am i missing something here?

am i missing something here?

The queue_fetch_event isn't the method doing the actual fetch, but rather just queues it to be later done asynchronously by the fetch_worker - as I've detailed above.
Therefore, as you said, no exception raised during the fetch can or should be caught there (in queue_fetch_event). Instead, the already implemented retry wraps the actual fetch method in BaseFetchProvider. It simply uses the retry configuration from FetchEvent, that configuration is set in queue_fetch_event.

Closing for now

Got some time this weekend to take a stab at this again.
So i tried setting retry configuration to the FetchEvent inside the queue_url where we are creating the fetch event. Like so
event = FetchEvent(url=url, fetcher=fetcher, config=config, retry=self._retry_config)

which creates a new fetch provider and sets the retry configuration from the FetchEvent object itself (by using BaseFetchProvider.set_retry_config).
The retry then, AFAIU, would occur for any kind of Exception.

based on what u mentioned, doing this should have triggered a retry when the actual fetch call fails with a status code of 5XX, but retry is not happening.
pardon my knowledge on the codebase but i am having a hard time trying to figure this out

@roekatz i have made the changes as you suggested by setting the raise_for_status for the ClientSession and now the retry works as expected with just passing a retry config to the FetchEvent.
thank you for taking out time reviewing and guiding around the codebase


Change the default subscription using AZ cmdlets

Select-AzureSubscription is using a different Module than the default AZ Module, which could cause confusion and issues for users.
Select-AzSubscription is integrated in the AZ Modules

@michawets : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Hi @eva-seydl , got it to work 😊 Could you review the change? 😁
Will send you an email to explain

Note that publishing in this repository is suspended until Tuesday, May 19. Merging will resume 7:00 AM PT Tuesday. Authors can still review and approve PRs during this time. Thanks.

Hi @Heidilohr , could you review this change please? Or should I ask Eva or Stefan to review?

#sign-off

@ktoliver : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@ktoliver : Thanks for your contribution! The author(s) have been notified to review your proposed change.


Use Buffer.byteLength instead of length

Use Buffer.byteLength instead of just getting the length of a base64 encoded string. Buffer.byteLength is more accurate since base64 strings need to add padding in order to arrive at the appropriate length.
Issue #, if available:
#282
Checklist

[x] :wave: I have run the unit tests, and all unit tests have passed.
[ ] :warning: This pull request might incur a breaking change.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@jwerre, binary data need to be base64 encoded to pass to the API Gateway proxy. The if statement here checks whether base64 encoded image fits in 6M limit, see https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html.


No such file or directory: 'gta/cyclegta5/split.mat'

I I download the gta images, put it under the cycada folder, and set data_dir to be 'gta/'.
I tried to run bash scripts/train_fcn_adda.sh and got this error:
No such file or directory: 'gta/cyclegta5/split.mat'
Could you please let me know where I can find the split.mat? Thanks!

This will be the same split.mat file which is released with GTA5.

@jzhuospark  I met same problem. How do you solve it?

Same problem!

+1

Please read the website carefully.  https://download.visinf.tu-darmstadt.de/data/from_games/

Download Link for split info is:
https://download.visinf.tu-darmstadt.de/data/from_games/code/read_mapping.zip

@Bowenwu1 Thank you!


Gradle 7.4 fails on multi release jar's with JDK 19 code

When com.fasterxml.jackson.core:jackson-core:2.15.0-rc1 is in the buildscript dependencies it fails to write locks.
Expected Behavior
Build should succeed
Current Behavior
Build fails with the error
./gradlew build

 FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring root project 'project'.
> java.util.concurrent.ExecutionException: org.gradle.api.GradleException: Failed to create Jar file /Users/<redacted>/.gradle/caches/jars-9/c94f549fdaad0dab4fb786ae7d97e150/jackson-core-2.15.0-rc1.jar.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 569ms

Steps to Reproduce
Clone this repo and run ./gradlew build
Your Environment

Build scan URL:
cannot provide build scan
Additional notes:

Version 2.15.0-rc1 of jackson-core introduces a multi-release jar [link to release notes], which is likely the reason why this error occurs on this particular version and not earlier versions.
This bug does not reproduce in Gradle version 7.6.1

@yanivkr could you rename the issue that issue is associated with Gradle's instrumenting of multi-release jars? The current title makes it more likely that noone will look at the issue - that they might think it is user error as opposed to an underlying issue that might still affect gradle. A multi-release jar with support for Java 20 classes may still be a problem in latest Gradle releases - even if they appear to have fixed the issue with Java 19 classes.

@yanivkr Also... I know this is asking for a lot but just in case it was possible, it'd be great to know if possible earlier Gradle versions might not have the issue (or, conversely, do have). Any additional datapoints pre-7.4 would be useful.
Or if anyone else could help here it'd be much appreciated of course :)

@pjfanning Renamed the issue.
@cowtowncoder According to gradle doc Java 19 support was only added at Gradle 7.6.
Can the change be backported to Gradle 7.4.X?

I created a forum topic to see if I could interest any Gradle folk in this issue. https://discuss.gradle.org/t/is-there-any-way-to-skip-gradle-instrumenting-classpath-file-transformer/45213

@yanivkr Right, I get that support for JDK 19 bytecode was only added then. What I was/am wondering is whether earlier Gradle versions might have avoided the issue by not trying to process bytecode in question (sometimes earlier tool versions happen to work, not by design but by lucky omission).

The title/description here sounds like Gradle can't build anything that depends on multi release jar's with JDK 19 code
(and this seems to cause some 'concerns').
This is not true (!)
The error appears when you use such a Jar in Gradle Plugins (or custom build logic).
For example:
buildscript { // <- Here we talk about extending Gradle itself!
    dependencies {
        classpath 'com.fasterxml.jackson.core:jackson-core:2.15.0-rc1'
    }
}

It seems that this is already fixed in 7.6. This is kind of a normal situation for code that extends Gradle itself. I don't think there is anything left to do.
It does not mean that there is a problem with building Java/Kotlin/Android/etc Applications or Libraries with such Jars. I assume that this works fine (and these are probably like 99% of the cases where e.g. Jackson is used in Gradle builds).
For example:
plugins {
    id 'application'
}

dependencies {
   implementation 'com.fasterxml.jackson.core:jackson-core:2.15.0-rc1'
}

Thanks @jjohannes - that lowers the importance of the issue for Jackson users

@jjohannes this bug critically impacted my organization.
Each of our Java projects use our internally developed Gradle plugin, which somehow got com.fasterxml.jackson.core:jackson-core:2.15.0-rc1  transitively leaked into its' dependencies, and since our projects use Gradle version less than 7.6.1, then it caused all builds to fail.
For now as a workaround we added the following code section to the init.gradle in all developer machines and build services:
initscript {
    // Temporary workaround
    gradle.allprojects {
        buildscript {
            configurations.all {
                resolutionStrategy.force 'com.fasterxml.jackson.core:jackson-core:2.14.2'
                resolutionStrategy.force 'com.fasterxml.jackson.core:jackson-annotations:2.14.2'
                resolutionStrategy.force 'com.fasterxml.jackson.core:jackson-databind:2.14.2'
                resolutionStrategy.force 'com.fasterxml.jackson:jackson-bom:2.14.2'
                resolutionStrategy.force 'com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.14.2'
                resolutionStrategy.force 'com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.14.2'
                resolutionStrategy.force 'com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.14.2'
                resolutionStrategy.force 'com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.14.2'
                resolutionStrategy.force 'com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.14.2'
                resolutionStrategy.force 'com.fasterxml.jackson.module:jackson-module-jaxb-annotations:2.14.2'
            }
        }
    }}

I'm working on upgrading all our projects to Gradle 7.6.1, including our internally developed Gradle plugin. This is not a trivial upgrade because several of our plugin's dependencies themselves are incompatible with Gradle 7.6.1

@Toldry sorry to hear about all this trouble, and thank you for sharing it. This is very unfortunate.

My understanding is that this can be an issue even on Gradle 7.6.1.  Although that version supports running Gradle on JDK19, it still uses ASM 9.2, which only supports bytecode up to JDK18.  So instrumentation of JDK19 classes can still fail on 7.6.1.  Gradle 8.1 has updated ASM to 9.4 as part of the JDK20 support, but we should consider updating ASM for Gradle 7.x as well.

Done through #25050

This issue now happens with the new version of jackson-core:2.16.1 which uses JDK 21

This issue now happens with the new version of jackson-core:2.16.1 which uses JDK 21

The issue is resolved as of Gradle 8.4. Please follow #27156 for a potential backport to 7.6.x line.

I have 2 projects (gradle 7.5), the dependency tree uses jackson-core-2.15.3. But one of them works, another one has the issue as above. why? Thank you in advance!


[Bug] Data security - level-aware logging of user data

Description
A user of the Rocket.Chat-App recently published a system-log in order to analyze defects in another app. Doing this, he published also the Rocket.Chat.Android-App which contained usernames and presence information.
Devices and Versions
I can't seriously tell which version he was using back then, it was on June 22, our server version back then was 0.65.
Steps to reproduce
Start the app, log in, check other user's presence. Now check Logcat.
Expectation
Unless the log-level is not increased to debug, there should not be any data of the app logged.
Logs
Some (now anonymized) samples (replaced the relevant content with <..>)
06-22 14:39:24.600 14187 14286 D null    : SDK(https://<our URL>): Received text message: {"msg":"added","collection":"users","id":"<GUID>","fields":{"emails":[{"address":"<here's an email-address>","verified":true}],"username":"<username>"}}, channel: RendezvousChannel@379a391{ReceiveQueued}

06-22 14:39:24.816 14187 14286 D null    : SDK(https://<our URL>): Received text message: {"msg":"added","collection":"users","id":"<GUID>","fields":{"name":"<name removed>","status":"online","username":"<username removed>","utcOffset":2}}, channel: RendezvousChannel@379a391{SendQueued,queueSize=18}
06-22 14:39:24.817 14187 24449 D null    : Got update of active user stream for the user: User(id=<GUID>, username=<username removed>, name=<name removed>, status=online, utcOffset=2.0, emails=null, roles=null)
06-22 14:39:24.818 14187 24449 D null    : active User: User(id=<GUID>, username=<username removed>, name=<name removed>, status=online, utcOffset=2.0, emails=null, roles=null)

@RocketChat/android I don't know about the log level on Android or how it was set. If you confirm that userdata is only logged on debug, I'm more than fine ;)

@mrsimpson Thanks for the report! We'll be investigating it ASAP!


CI/DevOps Best practices for Meltano projects

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3421
Originally created by @aaronsteers on 2022-04-19 22:39:51

On the way to a "diff-only" CI cycle (#3410+), I thought it would be helpful to show what a standard CI flow might look like.
Unlike that mentioned in #3410, everything below is possible using only capabilities that exist in Meltano today. There's a lot of "optional" functionality and some project-specific preferences involved here, which I think should be expected. I think it is nevertheless helpful to come up with an opinionated approach and strategy for the CI/DevOps lifecycle - while also keeping the understanding that each user and company will adapt this guidance according to their own preferences.
DataOps Principles

All build artifacts should be disposable.
Breaking changes must be detected during CI.
Prioritize completeness of tests over speed of execution.
Prioritize fast failure of CI when feasible.
Allow EL to be included or excluded from tests, per the user and project preferences.
Zero manual deployment steps.
Prioritize testing the data movement, transformation, and application.
Service fabric testing would be addressed separately, as per #3422. Testing of service is ignored here because of the requirement for always-on compute and some manual user interaction.

On each CI build (non-main branch)
Step 1 - EL Builds run with Mappers applied

CI pipeline can run all EL pipelines, using a cicd Meltano environment.
Skipped/disabled if creds are not available or if project settings have disabled this feature.
Optionally, stream maps are applied on the cicd Meltano environment to (1) reduce the volume of data and/or (2) obfuscate sensitive data.

CI pipeline uses prod source data.
If build artifacts from main branch or a recent prod execution are available, dbt uses the defer feature. Otherwise a full run is performed.
dbt is configured to either run with limit 0 or with materialize=view on all tables. This makes the step fail much faster in cases where there is an error in column name mismatches, SQL syntax errors, or any other compile-type failure.

Step 2b - Transform Builds run (with data)

Same as 2a except runs with actual data.
Can run in parallel with 2a, but presumably can also abort early if 2a fails. (2a's role is 'fast failure', likely will complete in 2 minutes while this 'with data' version may take 20-30 minutes to complete.)
After executing dbt:run, Meltano also invokes dbt:test
dbt docs generate is executed and the results are made available either via static web host (prefered) or via CI artifacts.

Step 3 - Profiling and additional validation (optional)

Optionally, additional validation and comparisons can be run against the transform results. These have to be defined and created by the Meltano user, optionally using an external tool like Great Expectations.

Step 4 - BI and Reporting Builds run

If the BI tool supports it, generate the BI reports and other artifacts using the results of the transform builds.
This can run in parallel to Step 3, since neither depends on the other and both types of failures are valuable.
Optionally, the BI artifacts can be pushed to one or more BI servers, if applicable and if possible without affecting prod resources.

On main branch
IF enabled by the user or project, the CI pipeline auto-deploys commits to main as updates to production. This uses the same process as above, except targeting the prod Meltano environment instead of the cicd environment.
Note: Any existing production jobs may optionally reference the main branch directly. For instance, if the user is using dbt Cloud as a complement to Meltano, they likely would point dbt Cloud "production" jobs and schedules to the main branch of their project.
On development branch
Projects which want to have a slower deployment cycle in production (once per week, for instance, instead of multiple times per day) may use a development or develop branch as their default branch for PRs. Using the same method as the main->prod deployment, this branch deploys to a staging or beta environment on each new merged commit.
If using a development branch, then main updates can often be a simple fast-forward to development on whatever deployment cadence the team prefers. The exception would be in cases of hot-fixes, which would target main directly and get automatically deployed as soon as the PR is merged to main.
Future Topics Currently Out of Scope
"Slim" CI Builds (Aka Diff-Only Builds)
This approach doesn't try to be "slim" as of yet, which is the diff-only approach. That approach is documented here for any readers interested: #3410.
Reusable CI yaml assets
Presumably this CI process could be created once and then copy-pasted across disparate Meltano projects, only adapting for the environment name and certain optional implementation choices as noted.
In future iterations, the CI design could further be refined into reusable GitHub Actions.
Service Fabric Testing
Not that this proposal focuses on the core data artifacts which can be represented in file and table artifacts.
Service fabric testing is a separate matter and is discussed here: #3422

View 5 previous comments from the original issue on GitLab


Set up doc string tests

Create a GitHub actions that tests that all classes and functions are registered with sphinx.
Feel free to have a look at how pints solved this:
Workflow: https://github.com/pints-team/pints/blob/master/.github/workflows/docs-test.yml
Run-tests.py: https://github.com/pints-team/pints/blob/master/run-tests.py

[x] The run_doctests can be kept, but the internal doctest_examples_readme function can be removed for now.
[x] The doctest_sphinx can be copied as it is.
[x] The doctest_rst_and_public_interface is important but needs to be adapted to match our repository structure, i.e. only import seimo for now (instead of the pints modules).
[x] copy the get_all_documented_symbols as is.
[x] copy the check_exposed_symbols as is.

Checking that all classes and methods are documented in an RST file and that public interfaces are clean.
The following classes do not appear in any RST file:
undocumented class: seirmo.models.ForwardModel
undocumented class: seirmo.models.SEIRModel
FAILED
I had this error when running the doc string test. I updated the modules in the index.rst. I don't know how to resolve this.

Close with #31


Icon request: Betfair

URL of the site or app
https://www.betfair.com
Logo resources for the icon
No response
Anything else?
No response

Here you go!
https://github.com/aegis-icons/aegis-icons/blob/master/SVG/Betfair.svg
https://github.com/aegis-icons/aegis-icons/blob/master/SVG/.Variations/Betfair bg.var.svg

:question: How to add icon(s) to Aegis?
:star: Support this project by giving a star! Just adding like to this comment is also appreciated.


[ServiceBus] Implement retry policy according to the Python design guideline

Implement and expose retry options to sdk users in Track 2 like what has been to Event Hub.
Documentations should be updated.

Did this happen?  What is the retry policy by default?
I am looking for something similar to this, https://docs.microsoft.com/en-us/azure/architecture/best-practices/retry-service-specific#service-bus.

Hello @andrewkittredge , thanks for reaching out.
We're in the middle of plan for next ServiceBus preview.
Retry policy is definitely something we'll add support to, but we can't guarantee a specific time.
Thanks for your patience in advance and will update you if there's any progress.

Hey Andrew, just following up on this:
Our Preview release of the Track 2 Service Bus SDK was rolled out yesterday. (full disclosure, not at feature parity, currently supports queue functionality, will be reaching that in incremental releases over the next few months)
This release allows you to specify “retry_total", "retry_backoff_refactor", "retry_backoff_max" as part of the get_* functions (e.g. get_queue_receiver).  Currently only exponential backoff is supported.
Closing this for now, but don't hesitate to respond/reopen if there are issues or other questions.

@KieranBrantnerMagee
I am testing the retry parameters on ServiceBusClient, it is not clear if/how they work.  In the below, I expect the receiver to yield the message once a second for 15 seconds then push the message to the dead letter queue.  Instead, the message is re-delivered about once a minute.
Am I doing something wrong, do I not understand how retry works?
client = ServiceBusClient.from_connection_string(
    "my_connection_string",
    retry_backoff_max=1,
    retry_total=15,
    retry_mode="fixed"
)
receiver = client.get_subscription_receiver(
    topic_name="my_topic_name", subscription_name="my_subscription"
)

last_retry = datetime.now()

for message in receiver:
    print(f"time since last retry is {datetime.now() - last_retry}")
    last_retry = datetime.now()


refactor: use package json auto generated export also for npm package name

This PR is a suggestion to replace this instrumentation base constructor call:
  constructor(config: CucumberInstrumentationConfig = {}) {
    super("@opentelemetry/instrumentation-foo", VERSION, config);
  }

With this one:
  constructor(config: CucumberInstrumentationConfig = {}) {
    super(PACKAGE_NAME, PACKAGE_VERSION, config);
  }

Where the auto generated file version.ts will also include the package name from package.json.
This PR also rename the cryptic VERSION symbol to NPM_PACKAGE_VERSION which I think is more readable.
I think the new format is better since:

It makes it clear that the value we use is the package name and not something else and carries semantic meaning.
The value is guaranteed to match the content of the package.json, just like we currently use the VERSION from package.json in instrumentation base constructor. no room for mistakes or misunderstandings.
The code is better formatted and more consistent

@JamieDanielson Thanks for another thorough review.
You justifiably find many opportunities for further improvement, which I choose to keep out of the scope of this PR so it is very specific on one single change.


implement a bit different architecture

remove interface
everything would be a component.
If you have another adapter to send a SMS, you should create another component in config:

'anotherSms' => [
    'class' => '\codeonyii\tww\TWWSms'
     'username' => 'test',
     'username' => 'test',
]

I like you PR @ericmaicon , but I decided to develop this extension using another concept.
Thanks anyway!


Added support for JWT generated by IDPs like Keycloak

RSA Support
I was unable to use RSA256 to validate signed requests because the only supported encryption method was HMAC which is a symmetrical and I'm unable to exchange private key securely. In theory, it should also support the other asymmetrical key types, but I have not tested them.
To support RSA (and IDPs like Keycloak), I've added two new configuration parameters : PREST_JWT_WELLKNOWN and PREST_JWT_JWKS.
PREST_JWT_WELLKNOWN
Takes an URL to the well-known configuration of your IDP (ex: https://accounts.google.com/.well-known/openid-configuration) and is able to fetch the JWKS from your IDP. It will then verify if the key used to sign the received token is in the JWKS and verify the signature with said key.
PREST_JWT_JWKS
Instead of providing a well-known URL, you can set the JWKS yourself. It will have the same subsequent behavior as for the PREST_JWT_WELLKNOWN parameter.
Ex :
{
  "keys": [
    {
      "n": "vcP1njqmR82oUndYDYxPF2HO-foOvDT8W8swcBYcgVQ0L1dPOlXz1lgQipJ0G7w9fXqilxsFHTs_MmXuGWQ2m3Wr16dpfPRFAN0ixj0ozzN_ZsF7OSdF6ZMZIJb3ObPt70i_emqEcd5ApITQ4PFvAhpZrBZCrW1z4aceoE1xC3rDV6qjA6pQr9u5A9J8Qj2a4LBG9mAPZCHQt4z50nhAVieKXhBoqKBoDplxj6yqzTTEUi1j-R0Z0NEJUIgt1_ErVsVMOWRKE6-BG39Zu9mvg2UAzxcitjOwa4n_spSgRleIliUJDNu-YpCEwj-5S_cX8DyHxCpvGp4ZBxPq1DUgqQ",
      "alg": "RS256",
      "use": "sig",
      "e": "AQAB",
      "kty": "RSA",
      "kid": "08bf5c3772dd4e7a727a101bf520f6575cac326f"
    }
  ]
}

Unit tests fix
I changed the version of Go used in the docker image for the tests. It needed to be updated to at least 1.20 for the golang.org/x/exp module.
I also included tweaks to the docker-compose to be able to run it using podman instead of Docker.

Summary by CodeRabbit

New Features

Enhanced JWT handling by introducing functionality to fetch and set JWKS from a well-known URL, improving the app's security.

Tests

Increased testing coverage for JWT configurations and key management, including test cases for parsing JWT configurations and handling RSA encryption.

Chores

Updated Dockerfile to use golang:1.20 and added postgresql-client installation, ensuring the development environment is up-to-date.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

I received a comment by @arxdsilva about middlewares/config.go where I should have kept
JwtMiddleware(config.PrestConf.JWTKey, config.PrestConf.JWTAlgo) but the modified function in middlewares.go manages the two situations. The JWTAlgo has disappeared because it was not used in the JwtMiddleware function anyway.

Just a check for the integration of this pull request. Are there any more work to do?

@mcharest-mcn thanks for the contribution, I feel that once these points are addressed we can merge it.

c801f5508a38e766cd97dcd0df488dfed7f6b86b


Push login links inside the page

Resolves #263
Moves login links within the page so we will be able to see them when we view the page on a mobile.

@mark100net this fixes the problem  I think but is there a preferred way to do this?

@pollygee You don't really need to add any CSS to make the login available in narrow. If you remove the conditional (user.signed_in?) toward the top of the header layout around the thing that prints the hamburger menu, it will work in both narrow and wide.
You may however want to add some styling to the login/sign up links because (especially in the narrow context) they don't look great.

Also, do we know if any other routes are going to be active for non-logged in? If not we should consider removing the Carrier link in that context because it doesn't really add any value.

@mark100net I removed the conditional as you suggested.  Please take another look when you get a chance.  It now looks like this:


Boards are not recognized properly as MobiFlight boards

Describe the bug
Users report that during startup their boards are not detected properly and show as Arduino boards without MobiFlight firmware. After a couple of restarts the boards might all show ok.
Steps To Reproduce
Steps to reproduce the behavior:

Start MobiFlight

Expected behavior
All MobiFlight boards are detected properly
Actual behavior
A firmware upload popup appears and proposes to upload the MobiFlight firmware
Details

MobiFlight Version: 9.6+

Clone Nano boards are consistently not being recognized by MF <IP_ADDRESS>. The firmware is uploaded OK and the boards work normally with their existing device config in eeprom until you restart Mobiflight and the pattern repeats.
This does not happen in MF 9.7.1 or 9.7.0.

Mobiflight connection delay for the mega was increased to 1750 in this PR: https://github.com/MobiFlight/MobiFlight-Connector/issues/1225
Timeout value for the Nano was increased to 20000 in this PR: https://github.com/MobiFlight/MobiFlight-Connector/pull/1211
I believe that was everything required to address this issue with Nanos using genuine CH340s.

I don't remember if this was only in regard to the Nano or also Mega. In general, we had been seeing more user reports about unstable board detection regardless of the CH340 issue.
I will nevertheless close this now based on the comment by @neilenns

FYI: Same problem on some UNOs, 9.7.1 ok.
https://discord.com/channels/608690978081210392/1124830581008633969/1127649222523617380


Could you please provide some more detailed data of Fig.1 in your paper?

Thanks for your outstanding work and rich experimental verification!
For Figure. 1 in your paper, there are intuitive comparison with SOTA methods, in terms of overall error, runtime and memory consumption. But would mind please providing some more detailed data, such as the specific value of overall error of your method and other SOTA methods with image size 1152x864.
Thanks a lot and look forward to your kind reply!

Hi,
For our method, the performance is Acc./Comp/Overall: 0.458, 0.291, 0.374

Ok. Thanks a lot!


merge from template-esm-only

package.json

chore(scripts): add c8 -x 'tests/**/' --temp-directory build/tmp ava --timeout 2m tests/-ava.mjs tests/*-ava-node.mjs && c8 report -r lcov -o build/coverage --temp-directory build/tmp (scripts.cover)

:tada: This PR is included in version 1.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


open failed sequences correctly

To reproduce, run the standalone launch sequence in the ONC module. The failed tests should be open and not just the first sequence.

Confirmed that it works now, good catch.


Fix stage out deleting unwanted files

Attempt to fix (or mitigate) #6273
I could not find any flaw in the code, then as discussed with Seangchan, I'm removing as much as possible file deletion commands. In summary:

files cleanup (when failures happen) will be done mainly at top level (cleanSuccessfulStageOuts call)
since other stage out implementation don't care for file size and checksum, I also removed it from LCGImp such that we rely solely on the exit status of lcg-cp command.
removed deferDirectoryCreation since it was always returning False
trying to place all wmagent exit codes in a single place

I discussed this with @hufnagel today and he does not like the removal of additional checks. However there is not much we can do here since logs are not available after the workflow is archived. Moreover, I don't expect lcg-cp returning exit status 0 when transfer faces problems (!= file size or checksum).
Pylint cleanup (and my ALAN logs) were left for a bit later. For now only sharing this code for comments/review.

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/674/ for details

Summary of pylint changes for pull request 6291:

Warnings for src/python/WMCore/Storage/Backends/LCGImpl.py increased from 2 to 4
Score for src/python/WMCore/Storage/StageOutMgr.py decreased from 9.55 to 9.54

src/python/WMCore/Storage/Backends/LCGImpl.py got worse in pylint. Report follows:

Line 133 in LCGImpl.createStageOutCommand W0612 Unused variable 'localPFN'
Line 133 in LCGImpl.createStageOutCommand W0612 Unused variable 'remotePFN'
Line 8 W0611 Unused import re
Line 11 W0611 Unused import StageOutError
plus 4 comments on code style

src/python/WMCore/Storage/StageOutMgr.py got worse in pylint. Report follows:

Line 23 in StageOutMgr R0902 Too many instance attributes (9/7)
Line 200 in StageOutMgr.call W0703 Catching too general exception Exception
Line 222 in StageOutMgr.call W0703 Catching too general exception Exception
Line 226 in StageOutMgr.call E0702 Raising NoneType while only classes, instances or string are allowed
Line 274 in StageOutMgr.localStageOut W0612 Unused variable 'seName'
Line 20 W0611 Unused import WMCore
Line 11 W0611 Unused import os
plus 25 comments on code style

Unit test changes for pull request 6291:

WMCore_t.Storage_t.FileManager_t.FileManagerTest:testStageOutMgrWrapperRealCopyFallback changed from success to error
WMCore_t.WMSpec_t.Steps_t.Executors_t.LogArch_t.LogArchiveTest:testExecutorDoesntDetonate changed from success to error
WMCore_t.WMSpec_t.Steps_t.Executors_t.LogArch_t.LogArchiveTest:testUnitTestBackend changed from success to error
WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testQueueChainingStatusUpdates changed from error to failure

Ok, I went through the original discussion of the problem and looked at your changes. I don't think this has anything to do with the WMAgent code, otherwise we would have seen this with other sites. We use immediate removal of files after stageout failures for CERN for instance as xrdcp absolutely needs it. Tier0 never has seen a problem with a file being staged out and then deleted and we create and stageout an absolute ENORMOUS number of files.
I also have nothing against removing some protections if we can show that they indeed are the problem here. Making changes without understanding what is going on most likely means we don't fix the problem and remove protections that were there for a reason.
Thing is, as lcg-cp happily overwrites existing files, removing the removeCommand calls on size and checksum comparison failures makes sense, PROVIDED the last stageout retry that fails cleans up after itself somehow. If it doesn't we leave dark data at the site.
Also, why would you remove the size and checksum comparison ? We don't need to delete on them, but we should fail on them. There are reasons we in general don't trust exit codes. Also has nothing to do with this issue really.

I have one question, is lcg-del (which seems to be the delete command used here), run synchronously or asynchronously (ie. does it delete right away or does it queue the file for deletion somewhere ) ?
If the latter it could explain the problem we are seeing...

These commands seem to be synchronous, since we have the srm-timeout option. BTW, I'll add a 10min timeout for lcg-del commands. I couldn't find anything more direct than that.

PROVIDED the last stageout retry that fails cleans up after itself somehow. If it doesn't we leave dark data at the site.

if the last retry fails, then a cleanup is in place (was not changed)
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WMSpec/Steps/Executors/StageOut.py#L217

Also, why would you remove the size and checksum comparison ? We don't need to delete on them, but we should fail on them. There are reasons we in general don't trust exit codes. Also has nothing to do with this issue really.

I could add them back and only exit in case of failures, no deletion. I simply removed it because others implementation don't really care.

I think if it is working we should leave it in. Someone spend some time making it work and we should not just remove it without a good reason. Some implementation use them and in general these checks are useful.

Ok, I'll investigate it further this week then. But this "Someone spend some time making it work and we should not just remove it without a good reason." is not really a good argument to keep code around as well :). We know there are code in WMCore that is no longer used (or that was never used), or that has not been used during the last year or so. Thus removing them seems the most reasonable thing to do IMO, otherwise it only overcomplicates the code, makes it harder to maintain and etc (v2 of stage out plugins are a good example).

Size and checksum comparison for stageout is in general a really good idea. We had cases where the exit code of the copy program could simply not be trusted. In addition, comparing the checksum can catch errors that the client exit code cannot tell you. Corruption of data in-flight should be rare, but cannot be detected unless you compare what went in (checksum you know the file is supposed to have) to what came out (checksum of the data as written on the other end).

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/679/ for details

Updated the code with the file size and checksum checks again, but this time not deleting anything in place in case it fails.
Also added srm-timeout to ls and del commands, such that we remove them from srm queue in case it takes too long.
Moreover, this Alarm
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WMSpec/Steps/Executors/StageOut.py#L214
can brings us problems in case it is triggered and we don't cancel the srm requests (also leaving garbage  at the site). We could keep only the srm timeouts here, but this class is used by all plugins and then probably not a good idea.

@amaltaro Just quick test with 2 sites xmls, and both failed to do fallback stageout. If 'srmv2' is used as fallback, it requires to create directories before executing cp and now this code is not calling directory creation.

Summary of pylint changes for pull request 6291:

Score for src/python/WMCore/Storage/Backends/LCGImpl.py decreased from 9.08 to 9.07
Score for src/python/WMCore/Storage/StageOutMgr.py decreased from 9.55 to 9.54

src/python/WMCore/Storage/Backends/LCGImpl.py got worse in pylint. Report follows:

Line 8 W0611 Unused import re
Line 11 W0611 Unused import StageOutError
plus 4 comments on code style

src/python/WMCore/Storage/StageOutMgr.py got worse in pylint. Report follows:

Line 23 in StageOutMgr R0902 Too many instance attributes (9/7)
Line 200 in StageOutMgr.call W0703 Catching too general exception Exception
Line 222 in StageOutMgr.call W0703 Catching too general exception Exception
Line 226 in StageOutMgr.call E0702 Raising NoneType while only classes, instances or string are allowed
Line 274 in StageOutMgr.localStageOut W0612 Unused variable 'seName'
Line 20 W0611 Unused import WMCore
Line 11 W0611 Unused import os
plus 25 comments on code style

Unit test changes for pull request 6291:

WMCore_t.Storage_t.FileManager_t.FileManagerTest:testStageOutMgrWrapperRealCopyFallback changed from success to error
WMCore_t.WMSpec_t.Steps_t.Executors_t.LogArch_t.LogArchiveTest:testExecutorDoesntDetonate changed from success to error
WMCore_t.WMSpec_t.Steps_t.Executors_t.LogArch_t.LogArchiveTest:testUnitTestBackend changed from success to error
WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testQueueChainingStatusUpdates changed from error to failure

Thanks for checking this out Justas. I'd probably miss the fallback test. I'll check the rest tomorrow

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/682/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/686/ for details

Summary of pylint changes for pull request 6291:

Score for src/python/WMCore/Storage/Backends/LCGImpl.py decreased from 9.08 to 9.07
Score for src/python/WMCore/Storage/StageOutMgr.py decreased from 9.55 to 9.54

src/python/WMCore/Storage/Backends/LCGImpl.py got worse in pylint. Report follows:

Line 8 W0611 Unused import re
Line 11 W0611 Unused import StageOutError
plus 4 comments on code style

src/python/WMCore/Storage/StageOutMgr.py got worse in pylint. Report follows:

Line 23 in StageOutMgr R0902 Too many instance attributes (9/7)
Line 200 in StageOutMgr.call W0703 Catching too general exception Exception
Line 222 in StageOutMgr.call W0703 Catching too general exception Exception
Line 226 in StageOutMgr.call E0702 Raising NoneType while only classes, instances or string are allowed
Line 274 in StageOutMgr.localStageOut W0612 Unused variable 'seName'
Line 20 W0611 Unused import WMCore
Line 11 W0611 Unused import os
plus 25 comments on code style

Unit test changes for pull request 6291:

WMCore_t.Storage_t.FileManager_t.FileManagerTest:testStageInMgrWrapperRealCopyFallback changed from success to error
WMCore_t.Storage_t.FileManager_t.FileManagerTest:testStageOutMgrWrapperRealCopyFallback changed from success to error
WMCore_t.WMSpec_t.Steps_t.Executors_t.LogArch_t.LogArchiveTest:testExecutorDoesntDetonate changed from success to error
WMCore_t.WMSpec_t.Steps_t.Executors_t.LogArch_t.LogArchiveTest:testUnitTestBackend changed from success to error
WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testQueueChainingStatusUpdates changed from error to failure
WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testSiteStatus changed from failure to error

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/691/ for details

Summary of pylint changes for pull request 6291:

Score for src/python/WMCore/Storage/StageOutImpl.py decreased from 10.00 to 9.84
Warnings for src/python/WMCore/Storage/StageOutImpl.py increased from 0 to 1
Score for src/python/WMCore/Storage/Backends/LCGImpl.py decreased from 9.08 to 9.07
Score for src/python/WMCore/Storage/StageOutMgr.py decreased from 9.55 to 9.54

src/python/WMCore/Storage/StageOutImpl.py got worse in pylint. Report follows:

Line 34 in StageOutImpl.executeCommand R0201 Method could be a function
Line 119 in StageOutImpl.createRemoveFileCommand R0201 Method could be a function
Line 12 W0611 Unused import StageOutInvalidPath
Line 14 in StageOutImpl R0921 Abstract class not referenced
plus 9 comments on code style

src/python/WMCore/Storage/Backends/LCGImpl.py got worse in pylint. Report follows:

Line 8 W0611 Unused import re
Line 11 W0611 Unused import StageOutError
plus 4 comments on code style

src/python/WMCore/Storage/StageOutMgr.py got worse in pylint. Report follows:

Line 23 in StageOutMgr R0902 Too many instance attributes (9/7)
Line 200 in StageOutMgr.call W0703 Catching too general exception Exception
Line 222 in StageOutMgr.call W0703 Catching too general exception Exception
Line 226 in StageOutMgr.call E0702 Raising NoneType while only classes, instances or string are allowed
Line 274 in StageOutMgr.localStageOut W0612 Unused variable 'seName'
Line 20 W0611 Unused import WMCore
Line 11 W0611 Unused import os
plus 25 comments on code style

No changes to unit tests for pull request 6291

test this please

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/697/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/699/ for details

Summary of pylint changes for pull request 6291:

Score for src/python/WMCore/Storage/StageOutImpl.py decreased from 10.00 to 9.84
Warnings for src/python/WMCore/Storage/StageOutImpl.py increased from 0 to 1
Score for src/python/WMCore/Storage/Backends/LCGImpl.py decreased from 9.08 to 9.07
Score for src/python/WMCore/Storage/StageOutMgr.py decreased from 9.55 to 9.54

src/python/WMCore/Storage/StageOutImpl.py got worse in pylint. Report follows:

Line 34 in StageOutImpl.executeCommand R0201 Method could be a function
Line 119 in StageOutImpl.createRemoveFileCommand R0201 Method could be a function
Line 12 W0611 Unused import StageOutInvalidPath
Line 14 in StageOutImpl R0921 Abstract class not referenced
plus 9 comments on code style

src/python/WMCore/Storage/Backends/LCGImpl.py got worse in pylint. Report follows:

Line 8 W0611 Unused import re
Line 11 W0611 Unused import StageOutError
plus 4 comments on code style

src/python/WMCore/Storage/StageOutMgr.py got worse in pylint. Report follows:

Line 23 in StageOutMgr R0902 Too many instance attributes (9/7)
Line 200 in StageOutMgr.call W0703 Catching too general exception Exception
Line 222 in StageOutMgr.call W0703 Catching too general exception Exception
Line 226 in StageOutMgr.call E0702 Raising NoneType while only classes, instances or string are allowed
Line 274 in StageOutMgr.localStageOut W0612 Unused variable 'seName'
Line 20 W0611 Unused import WMCore
Line 11 W0611 Unused import os
plus 25 comments on code style

Unit test changes for pull request 6291:

WMCore_t.WorkQueue_t.Policy_t.Start_t.Block_t.BlockTestCase:testWhiteBlackLists changed from success to error

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/705/ for details

@ticoann do not merge please. It still has my loggings and I'd prefer discussing it in tomorrow's meeting.

No pylint warnings for pull request 6291:

No changes to unit tests for pull request 6291

Needs to be rebased.

@ticoann let's hold on this one for a bit more please. Anyways it's on the agent side only.

Closing it in favor of #6695


Allow replacements to include new lines

This feature would rely on tui-rs's new Spans API, which other PRs such as #18 have incorporated.
Until those new changes are released on crates.io this feature will have to wait.


Support separate cmake-only stage for build scripts

As part of an experimental project to wrap, isolate and adjust the UI for these scripts, it's helpful to be able to separate out the running of the 'cmake' (or 'configure') step from the build and install steps.
This is mostly quite straightforward: we just add a command 'cmake' (or 'configure') for each script to fit alongside the existing 'nocmake' and 'install' commands.  One slight difficulty is handling the flang-legacy script, because it actually builds two components in succession (llvm-legacy then flang-legacy).  Incremental rebuilds are more easily handled (in the above-mentioned project) by splitting the component into two scripts, so that's what this patch does.
A normal build still completes successfully with the patch.
OK?

Thank you! I have no write access to the repo though, so could you merge it for me please?


Feat: added cname options for CNAME file

Related to issue #1803

Looks good! Needs a changelog entry before merging, but otherwise :+1:

Looks good! Needs a changelog entry before merging, but otherwise 👍

Added, hope it's fine


Not compatible with current version of nu

I hope it's only me, but when I try plugin add ... nu_plugin_units, I get this error:
Error:   × Failed to send plugin call to `units`
  help: try loading the plugin again with `plugin use 'units'`

Error: nu::shell::plugin_failed_to_load

  × Plugin failed to load: Plugin `units` is compiled for nushell version 0.97.1, which is not compatible with version 0.99.1

It seems to me that some dependency might need to be updated and the plugin released again, but I have never done anything in the realm of plugin development, so don't let me lead you astray if this is implausible.
Love the plugin though, and I dearly want to use it again! :)

I'll hop on fixing this this weekend. I love Nu as well, but there sure are a lot of breaking changes to the protocols! I hope these stabilize one a bit one day. Thanks for bringing this to my attention.

This should now be fixed in v0.1.3!

Verified, it works. Thank you! 👍

It happened again 🙈

2 polar opposite things:

Nushell breaking plugins multiple times a year is killing me.
You coming here to report it makes me happy, as I know at least someone is using this thing. Would love to hear about where you are finding this thing useful - what sort of work you do.

Anyway, I'm on xmas break, so I'll get this, and my other nushell plugins, fixed.

@Aarkon - give v0.1.4 a shot.


Enable random seed option to make latency benchmarking more configurable

Enable random seed option to make latency benchmarking more configurable.
The reason of this pr come from some findings and questions:
(1) Fact 1: It seems that fixing the seed also fix the randomness even when temperature is 1.0 (see the old analysis section below)
(2) Q1: Should we use temperature = 0.0 even without using beam search to fix them and make things more consistent? (if yes, can modify this pr to add the changes)
(3) Q2: Maybe we still need to enable this seed fix to make things more configurable as this pr shows?
==== old analysis ====
Current benchmarking script set temperature as 1.0 when not using beam search and does not fix random seed and produces inconsistent output length results if set output_len to be larger than 1 even with other hparams fixed (top_p and temperature, etc) causing latency benchmarking inaccurate. This pr added the seed to fix it as 42 by default.
Testing done with tinyllama model with following scripts and print out the output after the calling of llm.generate :
python benchmark_latency.py \
--model tinyllama_test/ \
--input-len 20 \
--output-len 5 \
--batch-size 1 \
--num-iters 3

Before fixing the seed with temperature 1.0:
[RequestOutput(request_id=7, prompt=None, prompt_token_ids=[2732, 9845, 3264, 4859, 9225, 7891, 4373, 5874, 6744, 3468, 705, 2599, 2222, 7768, 2897, 9893, 537, 6216, 6921, 6036], prompt_logprobs=None, outputs=[CompletionOutput(index=0, text='e employment encryption law', token_ids=[29872, 5703, 358, 20956, 4307], cumulative_logprob=-33.64984667301178, logprobs=None, finish_reason=length, stop_reason=None)], finished=True, metrics=RequestMetrics(arrival_time=1718261321.015641, last_token_time=1718261321.015641, first_scheduled_time=1718261321.0158198, first_token_time=1718261321.0314093, time_in_queue=0.00017881393432617188, finished_time=1718261321.0474448), lora_request=None)]
[RequestOutput(request_id=8, prompt=None, prompt_token_ids=[2732, 9845, 3264, 4859, 9225, 7891, 4373, 5874, 6744, 3468, 705, 2599, 2222, 7768, 2897, 9893, 537, 6216, 6921, 6036], prompt_logprobs=None, outputs=[CompletionOutput(index=0, text=' automoblestart', token_ids=[3345, 711, 29880, 342, 442], cumulative_logprob=-19.766031503677368, logprobs=None, finish_reason=length, stop_reason=None)], finished=True, metrics=RequestMetrics(arrival_time=1718261321.047535, last_token_time=1718261321.047535, first_scheduled_time=1718261321.0477033, first_token_time=1718261321.0625982, time_in_queue=0.00016832351684570312, finished_time=1718261321.0775552), lora_request=None)]
[RequestOutput(request_id=9, prompt=None, prompt_token_ids=[2732, 9845, 3264, 4859, 9225, 7891, 4373, 5874, 6744, 3468, 705, 2599, 2222, 7768, 2897, 9893, 537, 6216, 6921, 6036], prompt_logprobs=None, outputs=[CompletionOutput(index=0, text="v5 ' - ", token_ids=[29894, 29945, 525, 448, 29871], cumulative_logprob=-32.859960079193115, logprobs=None, finish_reason=length, stop_reason=None)], finished=True, metrics=RequestMetrics(arrival_time=1718261321.0776284, last_token_time=1718261321.0776284, first_scheduled_time=1718261321.077778, first_token_time=1718261321.0920112, time_in_queue=0.00014972686767578125, finished_time=1718261321.1073508), lora_request=None)]

After fixing the seed=42 with still temperature 1.0:
[RequestOutput(request_id=4, prompt=None, prompt_token_ids=[2732, 9845, 3264, 4859, 9225, 7891, 4373, 5874, 6744, 3468, 705, 2599, 2222, 7768, 2897, 9893, 537, 6216, 6921, 6036], prompt_logprobs=None, outputs=[CompletionOutput(index=0, text='. Nhts n', token_ids=[29889, 405, 400, 29879, 302], cumulative_logprob=-23.869795322418213, logprobs=None, finish_reason=length, stop_reason=None)], finished=True, metrics=RequestMetrics(arrival_time=1718261464.786836, last_token_time=1718261464.786836, first_scheduled_time=1718261464.7870114, first_token_time=1718261464.7999206, time_in_queue=0.00017547607421875, finished_time=1718261464.8143234), lora_request=None)]
[RequestOutput(request_id=5, prompt=None, prompt_token_ids=[2732, 9845, 3264, 4859, 9225, 7891, 4373, 5874, 6744, 3468, 705, 2599, 2222, 7768, 2897, 9893, 537, 6216, 6921, 6036], prompt_logprobs=None, outputs=[CompletionOutput(index=0, text='. Nhts n', token_ids=[29889, 405, 400, 29879, 302], cumulative_logprob=-23.869795322418213, logprobs=None, finish_reason=length, stop_reason=None)], finished=True, metrics=RequestMetrics(arrival_time=1718261464.8144147, last_token_time=1718261464.8144147, first_scheduled_time=1718261464.814583, first_token_time=1718261464.8276753, time_in_queue=0.00016832351684570312, finished_time=1718261464.8424041), lora_request=None)]
[RequestOutput(request_id=6, prompt=None, prompt_token_ids=[2732, 9845, 3264, 4859, 9225, 7891, 4373, 5874, 6744, 3468, 705, 2599, 2222, 7768, 2897, 9893, 537, 6216, 6921, 6036], prompt_logprobs=None, outputs=[CompletionOutput(index=0, text='. Nhts n', token_ids=[29889, 405, 400, 29879, 302], cumulative_logprob=-23.869795322418213, logprobs=None, finish_reason=length, stop_reason=None)], finished=True, metrics=RequestMetrics(arrival_time=1718261464.8425188, last_token_time=1718261464.8425188, first_scheduled_time=1718261464.8426938, first_token_time=1718261464.8557236, time_in_queue=0.00017499923706054688, finished_time=1718261464.8700457), lora_request=None)]
[RequestOutput(request_id=7, prompt=None, prompt_token_ids=[2732, 9845, 3264, 4859, 9225, 7891, 4373, 5874, 6744, 3468, 705, 2599, 2222, 7768, 2897, 9893, 537, 6216, 6921, 6036], prompt_logprobs=None, outputs=[CompletionOutput(index=0, text='. Nhts n', token_ids=[29889, 405, 400, 29879, 302], cumulative_logprob=-23.869795322418213, logprobs=None, finish_reason=length, stop_reason=None)], finished=True, metrics=RequestMetrics(arrival_time=1718261464.8701239, last_token_time=1718261464.8701239, first_scheduled_time=1718261464.870278, first_token_time=1718261464.8830173, time_in_queue=0.00015401840209960938, finished_time=1718261464.8975763), lora_request=None)]

Without fixing random seed, if we change temperature to 0.0, it also produced same results (though different responses as expected).
BEFORE SUBMITTING, PLEASE READ THE CHECKLIST BELOW AND FILL IN THE DESCRIPTION ABOVE

 PR Checklist (Click to Expand)
Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.
PR Title and Classification
Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

[Bugfix] for bug fixes.
[CI/Build] for build or continuous integration improvements.
[Doc] for documentation fixes and improvements.
[Model] for adding a new model or improving an existing model. Model name should appear in the title.
[Frontend] For changes on the vLLM frontend (e.g., OpenAI API server, LLM class, etc.)
[Kernel] for changes affecting CUDA kernels or other compute kernels.
[Core] for changes in the core vLLM logic (e.g., LLMEngine, AsyncLLMEngine, Scheduler, etc.)
[Hardware][Vendor] for hardware-specific changes. Vendor name should appear in the prefix (e.g., [Hardware][AMD]).
[Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.
Code Quality
The PR need to meet the following code quality standards:

We adhere to Google Python style guide and Google C++ style guide.
Pass all linter checks. Please use format.sh to format your code.
The code need to be well-documented to ensure future contributors can easily understand the code.
Include sufficient tests to ensure the project to stay correct and robust. This includes both unit tests and integration tests.
Please add documentation to docs/source/ if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.

Notes for Large Changes
Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with rfc-required and might not go through the PR.
What to Expect for the Reviews
The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:

 After the PR is submitted, the PR will be assigned to a reviewer. Every reviewer will pick up the PRs based on their expertise and availability.
 After the PR is assigned, the reviewer will provide status update every 2-3 days. If the PR is not reviewed within 7 days, please feel free to ping the reviewer or the vLLM team.
 After the review, the reviewer will put an  action-required label on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.
 Please respond to all comments within a reasonable time frame. If a comment isn't clear or you disagree with a suggestion, feel free to ask for clarification or discuss the suggestion.

Thank You
 Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!

Hmm this is weird because we have the following set which should produce exactly the number of tokens requested
        ignore_eos=True,
        max_tokens=args.output_len,

Hmm this is weird because we have the following set which should produce exactly the number of tokens requested
        ignore_eos=True,
        max_tokens=args.output_len,

Thanks for the response! yeah, that's my understanding as well. I'm using 0.4.1 version to test, do you think 0.4.3 or 0.5 have any fix related? Thank you!

ignore_eos might be broken for llama3 in certain versions I remembered... but it should be fixed 0.4.3 in the latest

ignore_eos might be broken for llama3 in certain versions I remembered... but it should be fixed 0.4.3 in the latest

Gotcha. Let me try 0.4.3. Something weird I found is that dummy random ids are good sometimes, but if I'm using some real prompts, it becomes like this and stop early usually.


import vite rather than requiring it

PR Checklist
Please check if your PR fulfills the following requirements:

[ ] Tests for the changes have been added (for bug fixes / features)

PR Type
What kind of change does this PR introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[X] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Documentation content changes
[ ] infrastructure changes
[ ] Other... Please describe:

What is the current behavior?
require the CJS version of Vite
What is the new behavior?
import the ESM version of Vite. The next major version of Vite may start producing warnings when Vite is required to encourage people to move off of the legacy CJS syntax and onto modern ESM. We should make a change now to get ahead of it and ensure users don't see this warning
Other information

Thanks Ben.


Triton: use CUDA 12.3 tools from the base image

Previously, Triton would download its own copies of ptxas, cuobjdump and nvdisasm:
https://github.com/openxla/triton/blob/cl617459344/python/setup.py#L373-L393
This began to cause problems when those versions were bumped to CUDA 12.4, meaning that Triton started to generate PTX with version number 8.3. When this was compiled, using the ptxas from the base container, inside XLA, then there were errors:
CustomCall failed: ptxas exited with non-zero error code 65280, output: ptxas /tmp/tempfile-aac66f5d464c-1e8add55-32-61414d9c202e5, line 5;
fatal   : Unsupported .version 8.4; current version is '8.3'
ptxas fatal   : Ptx assembly aborted due to errors

in the nightly tests, which are taken from JAX-Triton.
Setting environment variables like TRITON_PTXAS_PATH has two effects:

it blocks downloading other versions during setup.py
at runtime, it is the highest precedence search location

If Triton starts depending on new fixes before the base container is updated to CUDA 12.4, problems may resurface.
Thanks to @andportnoy for help debugging.

Should we fix this upstream?

From: Andrey Portnoy @.>
Sent: Friday, March 22, 2024 9:09:38 AM
To: NVIDIA/JAX-Toolbox @.>
Cc: Frederic Bastien @.>; Review requested @.>
Subject: Re: [NVIDIA/JAX-Toolbox] Triton: use CUDA 12.3 tools from the base image (PR #656)
@andportnoy approved this pull request.
Thank you for the fix! Left a minor suggestion adding some context.

In .github/container/Dockerfile.tritonhttps://github.com/NVIDIA/JAX-Toolbox/pull/656#discussion_r1535831398:

@@ -2,10 +2,18 @@
ARG BASE_IMAGE=ghcr.io/nvidia/jax-mealkit:jax
ARG SRC_PATH_TRITON=/opt/openxla-triton

+FROM ${BASE_IMAGE} as base
+# Tell Triton to use CUDA binaries from the host container. These should be set
⬇ Suggested change
-# Tell Triton to use CUDA binaries from the host container. These should be set
+# Triton setup.py downloads and installs CUDA binaries at specific versions
+# hardcoded in the script itself:
+# https://github.com/openxla/triton/blob/84f9d9de158fb866fac67970f0f5d323999d9db1/python/setup.py#L373-L393
+# Tell Triton to use CUDA binaries from the host container instead. These should be set
—
Reply to this email directly, view it on GitHubhttps://github.com/NVIDIA/JAX-Toolbox/pull/656#pullrequestreview-1955208951, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AABMF627MQCSEU6JBTO5DBDYZRJUFAVCNFSM6AAAAABFDGANMOVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTSNJVGIYDQOJVGE.
You are receiving this because your review was requested.Message ID: @.***>

@nouiz

Should we fix this upstream?

I'm not sure what that would look like? What did you have in mind?

@nouiz

Should we fix this upstream?

I'm not sure what that would look like? What did you have in mind?

I taught it was for Triton via Pallas. All is good.


Add gcc-5.1 Dockerfile to create Docker image to test ITK

This is a follow up of the discussion here.

Before merging this, I want to make sure that this is the correct path. I just discovered module-ci which should also probably moved here. What do you think?

Superseded by #4

+1 for moving module-ci..


Uncaught Error: No such template: dropzone

clone this project and run in meteor but this error in chrome console
Uncaught Error: No such template: dropzone

Please try with the new version and report back if problem reoccurs. Yet, this should not even happen with the previous version.

Same thing for me just now.

Nevermind, I think it's because I didn't install the jquery package.