[Question] OOM on finetuning vicuna-7b llava model on 4*A800 80G, anything wrong with my cfg?

 Please try this script with flash attention and deepspeed.

It seems that A800 is supported in flash-attn. Please try this script with flash attention and deepspeed.

Yeah, I tried. Error mesage is the same~

Can you provide the version of your flash attention, transformers, accelerate, and pytorch? Also, is flash attention compiled with the same CUDA version as PyTorch?

flash attention

I followed the instruction from the project page.
Sorry that I cannot access the machine for now. some of the version I checked before was:

transformers: 4.31.0
flash-attn was installed by 'pip install flash-attn --no-build-isolation'. flash_attn-2.0.9+cu117torch2.0cxx11abiFALSE-cp310-cp310-linux_x86_64.whl.
pytorch: 2.0.1-cu117

Can you please tell if flash-attn and deepspeed is a MUST to do finetune on 4A800 80G?
What if I upgrade the machien to use 8A800, will it be able to run w/o N*A800?

Can you try pip install flash-attn==2.0.4 --no-build-isolation?
I find that all my current versions are 2.0.4.
Also, flash attention is necessary. You may do gradient accumutation with bs4xaccu4, which can make 7B fit in 8x A100s (maybe 4x as well). But flash-attention brings at least 2x speedup in my experiments.

Get~ Thanks.
I don't have the box now. Will report the result back.

Can you try pip install flash-attn==2.0.4 --no-build-isolation?
I find that all my current versions are 2.0.4.
Also, flash attention is necessary. You may do gradient accumutation with bs4xaccu4, which can make 7B fit in 8x A100s (maybe 4x as well). But flash-attention brings at least 2x speedup in my experiments. So spending some time to make flash-attn work should worth it.

BTW, even I make bs1xaccu16, it was still OOM... so that it looks like flash attn is a MUST.

Can you try pip install flash-attn==2.0.4 --no-build-isolation?
I find that all my current versions are 2.0.4.
Also, flash attention is necessary. You may do gradient accumutation with bs4xaccu4, which can make 7B fit in 8x A100s (maybe 4x as well). But flash-attention brings at least 2x speedup in my experiments. So spending some time to make flash-attn work should worth it.

I tried... no luck. still the same error:
 1  Traceback (most recent call last):
 2   File "/root/devroot/src/LLaVA/llava/train/train_mem.py", line 13, in <module>
 3     train()
 4   File "/root/devroot/src/LLaVA/llava/train/train.py", line 909, in train
 5     trainer.train()
 6   File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/transformers/trainer.py", line 1539, in train
 7     return inner_training_loop(
 8   File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/transformers/trainer.py", line 1809, in _inner_training_loop
 9     tr_loss_step = self.training_step(model, inputs)
10   File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/transformers/trainer.py", line 2654, in training_step
11     loss = self.compute_loss(model, inputs)
12   File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/transformers/trainer.py", line 2679, in compute_loss
13     outputs = model(**inputs)
14   File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
15     return forward_call(*args, **kwargs)
16   File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 171, in forward
17     outputs = self.parallel_apply(replicas, inputs, kwargs)
18   File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 181, in parallel_apply
19     return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
20   File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/parallel/parallel_apply.py", line 89, in parallel_apply
21     output.reraise()
22   File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/_utils.py", line 644, in reraise
23     raise exception
24 RuntimeError: Caught RuntimeError in replica 0 on device 0.
25 Original Traceback (most recent call last):
26   File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/parallel/parallel_apply.py", line 64, in _worker
27     output = module(*input, **kwargs)
28   File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
29     return forward_call(*args, **kwargs)
30   File "/root/devroot/src/LLaVA/llava/model/language_model/llava_llama.py", line 78, in forward
31     outputs = self.model(
32   File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
33     return forward_call(*args, **kwargs)
34   File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 685, in forward
35     layer_outputs = torch.utils.checkpoint.checkpoint(
36   File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 249, in checkpoint
37     return CheckpointFunction.apply(function, preserve, *args)
38   File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
39     return super().apply(*args, **kwargs)  # type: ignore[misc]
40   File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 107, in forward
41     outputs = run_function(*args)
42   File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 681, in custom_forward
43     return module(*inputs, output_attentions, None)
44   File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
45     return forward_call(*args, **kwargs)
46   File "/root/miniconda3/envs/llava/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 408, in forward
47     hidden_states, self_attn_weights, present_key_value = self.self_attn(

name: llava
channels:
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=5.1=1_gnu
  - bzip2=1.0.8=h7b6447c_0
  - ca-certificates=2023.05.30=h06a4308_0
  - ld_impl_linux-64=2.38=h1181459_1
  - libffi=3.4.4=h6a678d5_0
  - libgcc-ng=11.2.0=h1234567_1
  - libgomp=11.2.0=h1234567_1
  - libstdcxx-ng=11.2.0=h1234567_1
  - libuuid=1.41.5=h5eee18b_0
  - ncurses=6.4=h6a678d5_0
  - openssl=3.0.10=h7f8727e_2
  - python=3.10.12=h955ad1f_0
  - readline=8.2=h5eee18b_0
  - sqlite=3.41.2=h5eee18b_0
  - tk=8.6.12=h1ccaba5_0
  - xz=5.4.2=h5eee18b_0
  - zlib=1.2.13=h5eee18b_0
  - pip:
    - accelerate==0.21.0
    - aiofiles==23.2.1
    - aiohttp==3.8.5
    - aiosignal==1.3.1
    - altair==5.0.1
    - anyio==3.7.1
    - appdirs==1.4.4
    - async-timeout==4.0.3
    - attrs==23.1.0
    - bitsandbytes==0.41.0
    - certifi==2023.7.22
    - charset-normalizer==3.2.0
    - click==8.1.7
    - cmake==3.27.2
    - contourpy==1.1.0
    - cycler==0.11.0
    - deepspeed==0.9.5
    - docker-pycreds==0.4.0
    - einops==0.6.1
    - einops-exts==0.0.4
    - exceptiongroup==1.1.3
    - fastapi==0.102.0
    - ffmpy==0.3.1
    - filelock==3.12.2
    - flash-attn==2.0.4
    - fonttools==4.42.1
    - frozenlist==1.4.0
    - fsspec==2023.6.0
    - gitdb==4.0.10
    - gitpython==3.1.32
    - gradio==3.35.2
    - gradio-client==0.2.9
    - h11==0.14.0
    - hjson==3.1.0
    - httpcore==0.17.3
    - httpx==0.24.0
    - huggingface-hub==0.16.4
    - idna==3.4
    - jinja2==3.1.2
    - joblib==1.3.2
    - jsonschema==4.19.0
    - jsonschema-specifications==2023.7.1
    - kiwisolver==1.4.5
    - linkify-it-py==2.0.2
    - lit==16.0.6
    - llava==1.0.1
    - markdown-it-py==2.2.0
    - markdown2==2.4.10
    - markupsafe==2.1.3
    - matplotlib==3.7.2
    - mdit-py-plugins==0.3.3
    - mdurl==0.1.2
    - mpmath==1.3.0
    - multidict==6.0.4
    - networkx==3.1
    - ninja==1.11.1
    - numpy==1.25.2
    - nvidia-cublas-cu11==<IP_ADDRESS>
    - nvidia-cuda-cupti-cu11==11.7.101
    - nvidia-cuda-nvrtc-cu11==11.7.99
    - nvidia-cuda-runtime-cu11==11.7.99
    - nvidia-cudnn-cu11==<IP_ADDRESS>
    - nvidia-cufft-cu11==<IP_ADDRESS>
    - nvidia-curand-cu11==<IP_ADDRESS>
    - nvidia-cusolver-cu11==<IP_ADDRESS>
    - nvidia-cusparse-cu11==<IP_ADDRESS>
    - nvidia-nccl-cu11==2.14.3
    - nvidia-nvtx-cu11==11.7.91
    - orjson==3.9.5
    - packaging==23.1
    - pandas==2.0.3
    - pathtools==0.1.2
    - peft==0.4.0
    - pillow==10.0.0
    - pip==23.2.1
    - protobuf==4.24.2
    - psutil==5.9.5
    - py-cpuinfo==9.0.0
    - pydantic==1.10.12
    - pydub==0.25.1
    - pygments==2.16.1
    - pyparsing==3.0.9
    - python-dateutil==2.8.2
    - python-multipart==0.0.6
    - pytz==2023.3
    - pyyaml==6.0.1
    - referencing==0.30.2
    - regex==2023.8.8
    - requests==2.31.0
    - rpds-py==0.9.2
    - safetensors==0.3.3
    - scikit-learn==1.2.2
    - scipy==1.11.2
    - semantic-version==2.10.0
    - sentencepiece==0.1.99
    - sentry-sdk==1.29.2
    - setproctitle==1.3.2
    - setuptools==68.0.0
    - shortuuid==1.0.11
    - six==1.16.0
    - smmap==5.0.0
    - sniffio==1.3.0
    - starlette==0.27.0
    - svgwrite==1.4.3
    - sympy==1.12
    - threadpoolctl==3.2.0
    - timm==0.6.13
    - tokenizers==0.13.3
    - toolz==0.12.0
    - torch==2.0.1
    - torchvision==0.15.2
    - tqdm==4.66.1
    - transformers==4.31.0
    - triton==2.0.0
    - typing-extensions==4.7.1
    - tzdata==2023.3
    - uc-micro-py==1.0.2
    - urllib3==2.0.4
    - uvicorn==0.23.2
    - wandb==0.15.8
    - wavedrom==2.0.3.post3
    - websockets==11.0.3
    - wheel==0.38.4
    - yarl==1.9.2
prefix: /root/miniconda3/envs/llava

also, i don't provide a " --deepspeed /path/to/deepspeed.json " in the run script. that should be ok, right?

i don't provide a " --deepspeed /path/to/deepspeed.json " in the run script. that should be ok, right?

That is not okay. Please use zero3.json or zero2.

and, the pretrain ckpt comes from a non-flash-attn train.py instead of a flash-attn train_mem.py. should I re-pretrain everything w/ flash-attn enabled from a scratch?

That is not needed. it is a linear layer, so it will be fine.

i don't provide a " --deepspeed /path/to/deepspeed.json " in the run script. that should be ok, right?

That is not okay. Please use zero3.json or zero2.

and, the pretrain ckpt comes from a non-flash-attn train.py instead of a flash-attn train_mem.py. should I re-pretrain everything w/ flash-attn enabled from a scratch?

That is not needed. it is a linear layer, so it will be fine.

WoW! deepspeed w/ zero3.json works great~
Thanks for all the quick response and your amazing work.

By using deepspeed w/ zero3.json, on my 8 * A100 80G, it takes ~15 hours for a full funetine (3 epochs).
And, it takes only <35GB GPU memory for each A800 with a full (~90% GPU-util).
Is the speed expected?

3699 iters is 3 epochs already. So you do not need to multiply by 3. ~6 hours is expected.

3699 iters is 3 epochs already. So you do not need to multiply by 3. ~6 hours is expected.

BTW, I find there is a considerable loss drop at the begining of each epoch. So, the reason we limit to 3 epoches is to prevent from too much overfit, right?

3699 iters is 3 epochs already. So you do not need to multiply by 3. ~6 hours is expected.

BTW, I find there is a considerable loss drop at the begining of each epoch. So, the reason we limit to 3 epoches is to prevent from too much overfit, right?

@haotian-liu
Also, I tried LLaVA, blip2-flant5-xl/xxl, instructblip-vicuna7b and found that LLaVA works best for the photos taken by my iphone. May I take this conclusion away? :

the quality of train data >>> 2. complexity of the architecture (Linear Layer vs QFormer) >>> 3. count of vit/llm model parameters.
(coz LLaVa clearly has considerably less trainable parameters to connect vit and llm, but performs very well).

And, how do you think of ResNet-50/101 as an image encoder? Will it perform similar like ViT?

@ldfandian hi, I met the loss not converging well issue, can you post your full train log for me?

@ldfandian Btw, are you using gradient checkpointing under deepspeed zero3? In my env here, it seems zero3 conflicts with checkpointing, but zero2 does not.


Why does the slider slightly move down on load?

Issue description:  After loading my page, the slider smoothly but slightly moves down a small amount of pixels. Sorry im a noob but i can not figure out why its doing this.
Demo link/slider setting:    slideBy: '1',
autoplay: true,
autoWidth: true,
items: 2,
controls: true,
controlsContainer: "#custom_controlsContainer",
controlsPosition: 'bottom',
nav: false,
gutter: 10,
speed: 2500,
prevButton: '#prev',
nextButton: '#next',
autoplayTimeout: 4000,
rewind: false
Tiny-slider version:   2.9.2
Browser name && version:  Firefox 72.0.2
OS name && version:  MacOS Catalina

I have no idea why this was happening in the first place, but if this happens to some poor soul in the future, the fix is simple. add this css to your slideshow container
position: relative;
display: block;
width: 100%;


Cannot load prefixed nested model from dotenv.

How to reproduce:
Given .env:
NESTED_FOO=foo
BAR=bar

from pydantic_settings import BaseSettings, SettingsConfigDict

class NestedConfig(BaseSettings):
    foo: str = ""
    model_config = SettingsConfigDict(env_prefix="NESTED_")

class Config(BaseSettings):
    nested: NestedConfig = NestedConfig()
    bar: str = ""
    model_config = SettingsConfigDict(
        env_file=".env", env_file_encoding="utf-8", extra="ignore", case_sensitive=False
    )

print(Config())

results in:
nested=NestedConfig(foo='') bar='bar'

Setting env variable by hand e.g. NESTED_FOO=foo python test.py works fine.

@hramezani Thanks, that make sense. Is it possible then to override model config for submodels somehow?

pydantic-settigns only loads the config for the models you are initializing(Config in your example) and collect values for the model fields. so it only consider the parent model configs


[Question]: when task: AzureCLI@2 has version 2.60.0 by default?

Task name
No response
Task version
No response
Environment type (Please select at least one enviroment where you face this issue)

[ ] Self-Hosted
[X] Microsoft Hosted
[ ] VMSS Pool
[ ] Container

Azure DevOps Server type
dev.azure.com (formerly visualstudio.com)
Azure DevOps Server Version (if applicable)
No response
Operation system
windows
Question
when task: AzureCLI@2 has version 2.60.0 by default will be release? it contains an important fix for us

anyone?

@LiliDeng MS hosted agents always have the latest azureCLI. please share the task configuration.

now I see it uses the latest version 2.60.0, thanks.


Edit Organization Screen: Duplicate External Id Not Showing Validation Error

Very small UX issue.  If you enter a duplicate External Id on the form shown below, it (correctly) won't allow you to Save, but doesn't tell you why the Save isn't allowed.

Ah, thanks for letting us know, Phil!


Câmpuri De Sus, Hunedoara (Gurasada) - Is not written correctly

ENVIROMENT: ui.orase
URL: https://peviitor-ro.github.io/ui.orase/
Browser: Chrome
Device: Laptop
OS: Windows 7
STEPS TO REPRODUCE:

Open URL in browser
Type "Câmpuri De Sus" in the search bar
Select Câmpuri De Sus, Hunedoara (Gurasada) from the list

EXPECTED RESULTS:
The name of the locality written in the search bar is written correctly based off the law nr. 167/1929
https://legislatie.just.ro/Public/DetaliiDocument/189
ACTUAL RESULTS:
The name of the locality written in the search bar is not written correctly based off the law nr. 167/1929

Fixed! Poti retesta


Please fix bug Y

Please Fix Bug Y
This is how you can reproduce the bug.
ABCDEFG

Show your :heart:  for vegalite by adding a tip!

*Tips will only be transferred when the issue gets resolved

Show your :heart:  for vegalite by adding a tip!

*Tips will only be transferred when the issue gets resolved


Listing blocks and SSR

I noticed that listing blocks are not rendered in ssr.
We remember that this intentionally but i don't remember why.
Our problem is that if you have some listing blocks as top blocks in one page, you see the first load of the page without them, and after some time, they appear (beacuse get loaded on client side).
You can see an example here: https://www.comune.modena.it
The first 2 or 3 blocks are listing ones and on page load they are not in the html and get loaded later, moving all the contents down.
Are they omitted for some technical or performance reasons? The ssr can't perform additional api calls and update the store?

@cekk: I might be wrong, but I think your best bet is to write a block transformer to "prepopulate" the listing in a volatile value in the block data (I've proposed the volatile "smart field" concept for tasks like this).
The problem is that the network-fetching call is only triggered by the block itself, on discovery of its data. There is some code in Volto that uses asyncConnect for SSR, but the view components are dynamically computed, so they can't use asyncConnect.

Some pointers to the code, why exactly the asyncConnect misses all backend calls done by the blocks:
https://github.com/makeomatic/redux-connect/blob/6e4d865fa5fbb8db2a505cb177583f34fb9edd95/modules/helpers/utils.js#L105 You can see here that it uses the router to identify components that have reduxAsyncConnect set on them https://github.com/makeomatic/redux-connect/blob/6e4d865fa5fbb8db2a505cb177583f34fb9edd95/modules/helpers/utils.js#L83


Solved : NPE if the server doesn't provide any HTTP content for an error

Fixes #2532

@louismurerwa will fix style errors and re-push.

@louismurerwa there have been an external breaking change in our test environment recently (#2537). Your PR branch had been branched out before #2537 was fixed. Merge with the latest master HEAD to fix the test failure and push the up-to-date branch again.


Option to keep test report using runtime order

Possible to display the testng report using the same sequence of tests showed at testng.xml?. Assume no parallelism  here

@dantran in order to make things go faster, can you provide a sample xml results file and the order in which the tests ran ?
I’m pretty sure this used to work like this, but when I addes the filtering features this got broke

to be clearer, what I need is  'testByNameReport' and keep the order showing up in the testng-results.xml.  This way we can use the report to view  test steps
Also note, the report does not track TestNG test suite, if it does we should see multiple sections
do you still need sample testng-results.xml?

to be clearer, what I need is  'testByNameReport' and keep the order showing up in the testng-results.xml.  This way we can use the report to view  test steps ...

@dantran please attach it, I’ll try to wrapp it up under a new section which will have the reports as they are in the xml file
Sent with GitHawk

attached is sample testng-results.txt ( xml file) with multiple suites
testng-results.txt


feature request: multiple selector/command sets

forgive me if this is already possible, but I haven't been able to find out how to create multiple sets of commands that get show based on different selectors. Presumably it'd be first match, or maybe merge when multiple selectors match. Let me know if this would be a good feature, I'll try to make time to implement this if it is. Would just manage multiple instances.

sorry didn't mean to close it

commands can be function( node ){ return [ /* ... */ ]; } so the commands can be dynamic.  You don't have to use the selector to differentiate elements: selector could be node, edge and commands() could specify everything.

I definitely see the use in that and even how it can be used to accomplish this to some extent. But now if I want all the nodes with a certain class to have a cxtmenu but not the others, would I just return an empty array from the commands function for those without the class? What if I want a single node to have a menu and none others, does the command function need to run for each node? Feel free to close this, I understand how to solve my immediate situation given your advice. Thank you for this and this awesome library! Lots of fun to work with.

You can use both the commands function and a selector if you like.
You can alternatively use enabled per command.  This was missed in the readme.  This is usually the best solution, because you maintain the user's muscle memory.

Sorry I never got back to you on this thanks for the solution and great
library!
On Oct 23, 2017 11:06 AM, "Max Franz"<EMAIL_ADDRESS>wrote:
Closed #59 https://github.com/cytoscape/cytoscape.js-cxtmenu/issues/59.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/cytoscape/cytoscape.js-cxtmenu/issues/59#event-1305842090,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADBhdPXZsCp_MMu3FY3v-ud008GSDlWRks5svKuFgaJpZM4NNiW5
.

I just want to point out, that the following code works,
ctxMenuRef.current = cyRef.current.cxtmenu({
     selector: 'node, edge', // or '*'
     commands: (nodeOrEdge) => {
          if (nodeOrEdge.isNode()) {
               // do something with node
          } else if (nodeOrEdge.isEdge()) {
               // do something with edge
          }
     }
})

and this works,
ctxMenuRef.current = cyRef.current.cxtmenu({
     selector: 'core',
     commands: (nodeOrEdge) => {
          // do something with core
     }
})

but this does not work,
ctxMenuRef.current = cyRef.current.cxtmenu({
     selector: 'node, edge, core', // <-- core is never triggered in this case
     commands: (nodeOrEdge) => { . . . }
})


[FEA] Audit_3.0.1: Minor change in BatchScanExec

commit 30e3fcbb081b6b07a12316c9dd63c66f77b8fec7
Author: Ryan Blue<EMAIL_ADDRESS>Date:   Wed Jul 8 16:06:40 2020 -0700
[SPARK-32168][SQL] Fix hidden partitioning correctness bug in SQL overwrite

@razajafri  Could we elaborate on why we need to audit this further?

In BatchScanExec, this is the change.
-  override lazy val partitions: Seq[InputPartition] = batch.planInputPartitions()
+  @transient override lazy val partitions: Seq[InputPartition] = batch.planInputPartitions()

GpuBatchScan needs to be updated.

Fixed by #5897


Assertion Failed -> Miner Crash

Sometimes my miner crashes while mining C31 on CPU with the following message:
"...cuckoo/cuckatoo/mean.hpp:939: void edgetrimmer::trimrename1(u32, u32) [with bool TRIMONV = true; u32 = unsigned int]: Assertion `renames < buckets[NX][NY].renameu1' failed.
Aborted (core dumped)"

I have the exact same problem, any updates?


Unrecognized app - Windows protected your PC

After downloading the file PrISM-0.1.0.msi and clicking on it I received a warning:

Windows Defender Smartscreen prevented an unrecognized app from starting.
Running this app might put your PC at risk.
App: PrISM-0.1.0.msi
Publisher: Unknown

I can just bypass the warning with the Run Anyway button, as I trust the developers, however, for a public release the app should eventually be a known publisher.

The next warning is from Norton Download Insight

Our information on this file is inconclusive.
We recommend not using this file unless you know it is safe

Keywords and links for the issue.
Microsoft Authenticode code-signing certificate

doc.microsoft
MS Sign tool
SO

Symantec "Code Signing Certificate":link: asks for

Windows Live ID :
Corporation:
Country
Type of software
How many code signings

Symantec code signing marketing page

It looks like the concerns are:

Finding a trusted authentic organization(business or nonprofit) with register-able credentials like a business license, mailing address, phone number etc.

Convincing that organization that the PrISM app is trustworthy enough for the organization to warrant the trust of a code signing company.

Finding a payor to finance the payment of the code signing certificate.


Create initial data collector platform servivce

Changes

Comments

#335  👈
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @forstmeier and the rest of your teammates on  Graphite


README.md is wrong?

go func() {
	ctx := context.Background()
	progressChan := progress.NewTicker(ctx, r, size, 1*time.Second)
	for p := range progressChan {
		fmt.Printf("\r%v remaining...", p.Remaining().Round(time.Second))
	}
	fmt.Println("\rdownload is completed")
}()

I guess the line ctx := context.Background() should be deleted since ctx is already defined outer goroutine.

Also need fix string
progressChan := progress.NewTicker(ctx, r, size, 1*time.Second)
Must be
progressChan := progress.NewTicker(ctx, r, int64(size), 1*time.Second)


Add mask cleanup function, demo notebook, etc.

This set of code aims to avoid problems I've been having while cross-validating PPSeq using the existing masked sampling procedures. It also creates some things that make masked sampling as easy as normal sampling for people who don't know PPSeq so well.
The specific problems concern the assignment of spikes into masks, and errors I was getting while the masked sampler checks that all the masked spikes are in the masks and vice versa.

Thanks @WilburDoz this is looking good. If you are ready I'll ask Anthony to give it a quick OK and then we can merge it!

I think that's everything fixed, thanks for being patient and I think I might have learnt some things about writing more understandable code from this!

I think I'm signed off. @degleris1 -- do you want to take a look before we merge or should we just go for it?


add bulk endpoint for data api

Need to add the ability to send bulk documents to the data api, the bulk documents should support both inserts and updates.

It could support deletes too by adding a _delete: true flag

It could support deletes too by adding a _delete: true flag


transfer string to double got fatal error!

when i use this function strtod in my contract ,it will report a error like the following:

i tested many times,and you guys can test on your enviroment.
my compiler is eosio-cpp version 1.6.1
and nodeos version is:v1.7.3 master branch
And the strtod function is standard function,right??????

when you execture your action which contain strtod ,it will get the error like that.

I can reproduce this issue by calling "strtod" inside a smartcontract. looks like the implementation of "strtod" is buggy.
Error 3070002: Runtime Error Processing WASM
Error Details:
wabt execution failure (out of bounds memory access)

The reason is that the current implementation of strtod uses some local variables of size larger than 8KB which results in stack underflow.

The reason is that the current implementation of strtod uses some local variables of size larger than 8KB which results in stack underflow.
so ,how should we to solve this,honey?

you can implement your own strtod, or wait for the fix (in https://github.com/EOSIO/musl/pull/10) in the next version of cdt

you can implement your own strtod, or wait for the fix (in EOSIO/musl#10) in the next version of cdt

ok, i will wait that, thank you.


Question Regarding Volume Rendering

Hello,
I've enjoyed using the volume rendering function in neuroglancer and had a couple of questions. I am noticing that when I use the volume rendering, objects in background chunks can cause transparent "holes" to appear in the foreground objects, eg. here.
Similarly, if I just set everything to have alpha=1.0 with a high brightness, the same thing happens:

Rather than a solid block, it appears as patched.
Is this the expected behavior or is there a way to prevent this? And what values can alpha have in emitRGBA? In the documentation it says [0,1] for pixel values, but it seems that based on brightness and alpha here it can be higher? And what scale level is it retrieving the data from, does it vary based on depth/zoom level, and is there a way to control it manually? Also, is the scale level the reason that things that are clear when zoomed in disappear when zoomed out, like here?
Thanks!

Unfortunately the volume rendering is currently still rather experimental and has a number of issues as you have observed.  I don't know the cause of this issue --- but I suspect it may have something to do with the accumulation buffer becoming saturated.
For volume rendering there are no specific bounds on the range of alpha values.  In general I haven't figured out the best way to automatically normalize the overall brightness, which is why there is a need for a manual brightness control.
Currently volume rendering always uses 64 depth samples along the camera view direction, and chooses the data resolution so that the voxel size is approximately 1/64 the view frustrum depth.  (Currently only a single resolution is used at a time.)  The "Resolution (3d)" control does not currently have any effect, though it does show the current resolution of data in use.  You can affect which resolution is used by changing the depth of the view frustrum, using alt+mousewheel, but of course that will also have other effects.

Just in case it's helpful, it is now possible to change the resolution level for 3D data. E.g. here the resolution slider was changed to match from the level of the first data resolution from the original link in this issue.
The rendering algorithm was also changed, so the alpha = 1.0 case should show a solid block as here.

This looks great, thanks!


Use Char instead of String at escape sequences

And some other improvements like using String#delete instead of String#gsub when it's something like gsub(' ', "").

Can we please stop with the PRs trying to change such trivial stuff as this, it likely provides little measurable impact and isn't the most productive use of anyone's time.

I think this can be merged now. Or did anybody else noticed something that could maybe be improved?

Oh all the checks failed. But thats probably because CI isn't using 0.25.0 yet.

Please remove the unrelated commit and create a new PR for that.

@r00ster91 Why not simply reset the branch to 4fafe427437d3e8d57d0cc9e370a5b8acd0a3ff5?

@straight-shoota I'm not very experienced with git yet. I tried resetting to HEAD~{1} or something but it didn't really worked so I changed it back by hand. But next time I will remember to specify the commit ID. Thanks.

I personally don't think such cosmetic changes are needed or have any benefit. Maybe they improve the performance (or not) of a small part of the program that's not a bottleneck. But it does distract developers and contributors to comment and argue on these small things, instead of having them focus on more important things (be it Crystal-related or not).

Okay I will stop now with these commits and wait for this to get merged.

Closing since there are conflicts. And the last comment states a stop in this changes.
I notice they were some changes in doc also, if they are wanted due to readability feel free to open a separate PR.
@r00ster91 please note that your contributions are very welcome and I hope you will keep them coming :-)

Too bad, this PR was not that far from a merge.
More than cosmetic changes, keep also in mind that the code also acts as a reference for newcomers.
@asterite Everyone, including Crystal contributors, is free to spend their time on what (s)he deems worth to do :)

I'd suggest finishing and merging this PR. Cosmetic or not, all in all they improve code readability and performance so they worthwhile enough.

Cosmetic or not, all in all they improve code readability and performance so they worthwhile enough

While they do improve performance (imperceptibly) and readability (imperceptibly), these PRs use up review bandwidth as well. You're free to make them, but I don't think it's worth my time to review such PRs too, especially when they end up with 14 commits and about 10 reviews such as this one.


Update the-githubdev-web-based-editor.md

Why: External resources should not be surrounded in quotes, per this issue.
Closes: https://github.com/github/docs/issues/35284

What's being changed (if available, include any code snippets, screenshots, or gifs):

Check off the following:

[x] I have reviewed my changes in staging, available via the View deployment link in this PR's timeline (this link will be available after opening the PR).

For content changes, you will also see an automatically generated comment with links directly to pages you've modified. The comment won't appear if your PR only edits files in the data directory.

[x] For content changes, I have completed the self-review checklist.

@Dusch4593 Thanks so much for opening a PR with this fix! I'll update the branch and get this merged once tests are passing 🍏


Integrate the NVIDIA container toolkit

This is an issue that has been spun off from the Discord channel.
@Murazaki : It might be good to find a better workflow for providing drivers to Wolf.
On Debian, drivers are pretty old in the main stable repo, and updated ones can be found on CUDA drivers, but do not exactly match manual installation ones.
@ABeltramo : I guess I should go back to look into the Nvidia Docker Toolkit for people that would like to use that
I agree though, it's a bit of a pain point at the moment
@Murazaki : Cuda drivers repo :
https://developer.download.nvidia.com/compute/cuda/repos/
Linux manual installer :
https://download.nvidia.com/XFree86/Linux-x86_64/
right now, latest in cuda packaged installs is 545.23.08.
It doesn't exist as a manual installer.
that breaks the dockerfile and renders wolf unusable
I wanted to make a docker image for debian packages install, but it uses apt-add-repository which is installing a bunch of supplementary stuff
Here it is for Debian Bookworm :
https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Debian&target_version=12&target_type=deb_network
More thorough installation steps here :
https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html
@juliosueiras : There is one problem though, nvidia driver toolkit doesn’t inject driver in the container and still require a driver installed in the container image itself,

And here, I start with what interventions I made with NVIDIA for the last three years to not require the NVIDIA drivers to run Wayland inside the NVIDIA container toolkit.
What NVIDIA container toolkit does: it's pretty simple. It injects (1) kernel devices, and (2) userspace libraries, into a container. (1) and (2) compose a subset of the driver.
(1) kernel devices: /dev/nvidiaN, /dev/nvidiactl, /dev/nvidia-modeset, /dev/nvidia-uvm, and /dev/nvidia-uvm-tools. In addition, /dev/dri/cardX and /dev/dri/renderDY, where N, X, and Y depend on the GPU the container toolkit provisions. The /dev/dri devices were added with https://github.com/NVIDIA/libnvidia-container/issues/118.
(2) userspace libraries:
OpenGL libraries including EGL: '/usr/lib/libGL.so.1', '/usr/lib/libEGL.so.1', '/usr/lib/libGLESv1_CM.so.525.78.01', '/usr/lib/libGLESv2.so.525.78.01', '/usr/lib/libEGL_nvidia.so.0', '/usr/lib/libOpenGL.so.0', '/usr/lib/libGLX.so.0', and '/usr/lib/libGLdispatch.so.0', '/usr/lib/libnvidia-tls.so.525.78.01'
Vulkan libraries: '/usr/lib/libGLX_nvidia.so.0' and the configuration '/etc/vulkan/icd.d/nvidia_icd.json'
EGLStreams-Wayland and GBM-Wayland libraries: '/usr/lib/libnvidia-egl-wayland.so.1' and the config '/usr/share/egl/egl_external_platform.d/10_nvidia_wayland.json' '/usr/lib/libnvidia-egl-gbm.so.1' and the config '/usr/share/egl/egl_external_platform.d/15_nvidia_gbm.json'
NVENC libraries: /usr/lib/libnvidia-encode.so.525.78.01, which depends on /usr/lib/libnvcuvid.so.525.78.01, which depends on /usr/lib/x86_64-linux-gnu/libcuda.so.1
VDPAU libraries: /usr/lib/vdpau/libvdpau_nvidia.so.525.78.01
NVFBC libraries: /usr/lib/libnvidia-fbc.so.525.78.01
OPTIX libraries: /usr/lib/libnvoptix.so.1
Not very relevant but of note, perhaps for XWayland: NVIDIA X.Org driver: /usr/lib/xorg/modules/drivers/nvidia_drv.so, NVIDIA X.org GLX driver: /usr/lib/xorg/modules/extensions/libglxserver_nvidia.so.525.78.01
In many cases, things don't work because the below configuration files are absent inside the container. Without these, applications inside the container don't know which library to call (what each file does is self-explanatory):
The contents of /usr/share/glvnd/egl_vendor.d/10_nvidia.json:
{
    "file_format_version" : "1.0.0",
    "ICD" : {
        "library_path" : "libEGL_nvidia.so.0"
    }
}

The contents of /etc/vulkan/icd.d/nvidia_icd.json (note that api_version is variable based on the Driver version):
{
    "file_format_version" : "1.0.0",
    "ICD": {
        "library_path": "libGLX_nvidia.so.0",
        "api_version" : "1.3.205"
    }
}

The contents of /etc/OpenCL/vendors/nvidia.icd:
libnvidia-opencl.so.1

The contents of /usr/share/egl/egl_external_platform.d/15_nvidia_gbm.json:
{
        "file_format_version" : "1.0.0",
        "ICD" : {
                "library_path" : "libnvidia-egl-gbm.so.1"
        }
}

The contents of /usr/share/egl/egl_external_platform.d/10_nvidia_wayland.json:
{
    "file_format_version" : "1.0.0",
    "ICD" : {
        "library_path" : "libnvidia-egl-wayland.so.1"
    }
}

I'm pretty sure that now (was different a few months ago), the newest NVIDIA container toolkit provisions all of the required libraries plus the json configurations for Wayland (not for X11 but you don't have to care).
If only the json configurations are absent, it's trivial to manually add the above template.
About GStreamer:
https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/3108
Now, it is correct that NVENC does require CUDA. But that doesn't mean that it requires the whole CUDA Toolkit (separate from the CUDA drivers). The CUDA drivers are the four following libraries installed with the display drivers, independent of the CUDA Toolkit: libcuda.so, libnvidia-ptxjitcompiler.so, libnvidia-nvvm.so, libcudadebugger.so
These versions go with the display drivers, and are all injected into the container by the NVIDIA container toolkit.
GStreamer 1.22 and before in nvcodec requires just two files of the CUDA Toolkit: libnvrtc.so and libnvrtc-bulletins.so. This can be installed from the network repository like the current approach, or be extracted from a PyPi package:
# Extract NVRTC dependency, https://developer.download.nvidia.com/compute/cuda/redist/cuda_nvrtc/LICENSE.txt
cd /tmp && curl -fsSL -o nvidia_cuda_nvrtc_linux_x86_64.whl "https://developer.download.nvidia.com/compute/redist/nvidia-cuda-nvrtc/nvidia_cuda_nvrtc-11.0.221-cp36-cp36m-linux_x86_64.whl" && unzip -joq -d ./nvrtc nvidia_cuda_nvrtc_linux_x86_64.whl && cd nvrtc && chmod 755 libnvrtc* && find . -maxdepth 1 -type f -name "*libnvrtc.so.*" -exec sh -c 'ln -snf $(basename {}) libnvrtc.so' \; && mv -f libnvrtc* /opt/gstreamer/lib/x86_64-linux-gnu/ && cd /tmp && rm -rf /tmp/*

One thing to note here is that libnvrtc.so is not minor version compatible with CUDA. Thus, it will error on any display driver version older than its corresponding display driver version. However, backwards compatibility always works. Thus, it is a good idea to use the oldest possible libnvrtc.so version.
Display CUDA
545 - 12.3
535 - 12.2
530 - 12.1
525 - 12.0
520 - 11.8
515 - 11.7
(and so on...)

https://docs.nvidia.com/deploy/cuda-compatibility/
So, I have moderate to high confidence that if you guys try the newest NVIDIA container toolkit again, you won't need to install the drivers, assuming that you ensure the json files are present or written.
Environment variables that currently work for me:
RUN echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf && \
    echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf
# Expose NVIDIA libraries and paths
ENV PATH /usr/local/nvidia/bin:${PATH}
ENV LD_LIBRARY_PATH /usr/lib/x86_64-linux-gnu:/usr/lib/i386-linux-gnu${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
# Make all NVIDIA GPUs visible by default
ENV NVIDIA_VISIBLE_DEVICES all
# All NVIDIA driver capabilities should preferably be used, check `NVIDIA_DRIVER_CAPABILITIES` inside the container if things do not work
ENV NVIDIA_DRIVER_CAPABILITIES all
# Disable VSYNC for NVIDIA GPUs
ENV __GL_SYNC_TO_VBLANK 0

TLDR: as of the latest Nvidia Container Toolkit (1.14.3-1) unfortunately this is still not possible.
What's the issue?
With the latest versions I can run both Wolf and the Gstreamer pipeline just by running the container with --gpus=all unfortunately for some apps this is still missing some important library.
Gamescope to run seems to require the following additional libraries that aren't provided by the toolkit:
libnvidia-egl-gbm.so.1
libnvidia-egl-wayland.so.1

libnvidia-vulkan-producer.so

gbm/nvidia-drm_gbm.so

Last one seems to just be a simlink to libnvidia-allocator.so.1 which is already present so that might be fine.
Now this is running from an X11 host and I can see that those additional libraries aren't present in my host system:
ls -la /usr/lib/x86_64-linux-gnu/libnv
libnvcuvid.so@                         libnvidia-container.so.1@              libnvidia-glvkspirv.so.530.30.02       libnvidia-opticalflow.so.1@
libnvcuvid.so.1@                       libnvidia-container.so.1.14.3*         libnvidia-ml.so.1@                     libnvidia-ptxjitcompiler.so.1@
libnvidia-allocator.so.1@              libnvidia-eglcore.so.530.30.02         libnvidia-ngx.so.1@                    libnvidia-rtcore.so.530.30.02
libnvidia-cfg.so.1@                    libnvidia-encode.so.1@                 libnvidia-ngx.so.530.30.02             libnvidia-tls.so.530.30.02
libnvidia-compiler.so.530.30.02        libnvidia-fbc.so.1@                    libnvidia-nvvm.so@                     libnvidia-wayland-client.so.530.30.02
libnvidia-container-go.so.1@           libnvidia-glcore.so.530.30.02          libnvidia-nvvm.so.4@                   libnvoptix.so.1@
libnvidia-container-go.so.1.14.3       libnvidia-glsi.so.530.30.02            libnvidia-opencl.so.1@

Anyone that can confirm the output of
docker run --rm -it --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all  -e NVIDIA_DRIVER_CAPABILITIES=all ubuntu ls /usr/lib/x86_64-linux-gnu/libnvidia*
/usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.1	   /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.530.30.02     /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.4
/usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.1		   /usr/lib/x86_64-linux-gnu/libnvidia-glsi.so.530.30.02       /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.530.30.02  /usr/lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.530.30.02  /usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-eglcore.so.530.30.02   /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1		       /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-encode.so.1		   /usr/lib/x86_64-linux-gnu/libnvidia-ngx.so.1		       /usr/lib/x86_64-linux-gnu/libnvidia-rtcore.so.530.30.02
/usr/lib/x86_64-linux-gnu/libnvidia-fbc.so.1		   /usr/lib/x86_64-linux-gnu/libnvidia-ngx.so.530.30.02        /usr/lib/x86_64-linux-gnu/libnvidia-tls.so.530.30.02

on a Nvidia Wayland host?
What can we do better?
I think we should keep manually downloading and linking the drivers like we are doing at the moment. We should probably add a proper check for mismatch between the downloaded drivers and the host installed drivers either on startup of the containers (somewhere in the base-app) or on startup of Wolf.

@ABeltramo One comment I have here is that there isn't really a concept called an "Xorg host" and a "Wayland host". It depends on what the desktop environment and login manager uses, and by default, all drivers bundle both libraries.

Hi,
I've been working on a project using Unreal Engine Pixel Streaming to stream games at scale with kubernetes. Packaging docker images with the right nvidia docker drivers is a pain. I invite you to take a look at the work done by Adam   He used nvidia/cuda image as base to create a set of images that with various configurations:
22.04-vulkan: Ubuntu 22.04 + OpenGL + Vulkan + PulseAudio Client + PulseAudio Server
22.04-cudagl11: Ubuntu 22.04 + OpenGL + Vulkan + CUDA 11.8.0 + PulseAudio Client + PulseAudio Server
22.04-cudagl12: Ubuntu 22.04 + OpenGL + Vulkan + CUDA 12.2.0 + PulseAudio Client + PulseAudio Server
22.04-vulkan-noaudio: Ubuntu 22.04 + OpenGL + Vulkan (no audio support)
22.04-cudagl11-noaudio: Ubuntu 22.04 + OpenGL + Vulkan + CUDA 11.8.0 (no audio support)
22.04-cudagl12-noaudio: Ubuntu 22.04 + OpenGL + Vulkan + CUDA 12.2.0 (no audio support)
22.04-vulkan-hostaudio: Ubuntu 22.04 + OpenGL + Vulkan + PulseAudio Client (uses host PulseAudio Server)
22.04-cudagl11-hostaudio: Ubuntu 22.04 + OpenGL + Vulkan + CUDA 11.8.0 + PulseAudio Client (uses host PulseAudio Server)
22.04-cudagl12-hostaudio: Ubuntu 22.04 + OpenGL + Vulkan + CUDA 12.2.0 + PulseAudio Client (uses host PulseAudio Server)
22.04-vulkan-x11: Ubuntu 22.04 + OpenGL + Vulkan + PulseAudio Client (uses host PulseAudio Server) + X11
22.04-cudagl11-x11: Ubuntu 22.04 + OpenGL + Vulkan + CUDA 11.8.0 + PulseAudio Client (uses host PulseAudio Server) + X11
22.04-cudagl12-x11: Ubuntu 22.04 + OpenGL + Vulkan + CUDA 12.2.0 + PulseAudio Client (uses host PulseAudio Server) + X11
I think you should check the docker files

Hi, I've been working on a project using Unreal Engine Pixel Streaming to stream games at scale with kubernetes. Packaging docker images with the right nvidia docker drivers is a pain. I invite you to take a look at the work done by Adam He used nvidia/cuda image as base to create a set of images that with various configurations
[...]
I think you should check the docker files

Could you provide a link to that/those Dockerfiles maybe ?

Oh sorry I actually know what they talk about it's adamrehn/ue4-runtime. I think I wrote an answer and forgot to send ^^

If those are the images that @ohayak was talking about, unfortunately, there's nothing there that can help us.
As I explained in a comment above, the nvidia toolkit is not mounting all libraries that are needed in order to spawn our custom Wayland compositor. There are only a few possible solutions that I can think of:

Ask users to install the dependencies in a volume that will be mounted and shared with the apps (the current solution in Wolf)
Ask users to manually install the libraries if missing from the host and then fill out all the correct mounts
Try to fill a patch upstream with the nvidia driver toolkit project and hope that it gets merged

I'm very open to suggestions or alternative solutions!

modprobe -r nvidia_drm ; modprobe nvidia_drm modeset=1
Our experience was related to the above kernel module. Also, we have reinstalled the OS and installed everything from scratch, then things started to work again.

The core issue is whether the EGL Wayland library is installed or not, not the container toolkit
This is available if you use the .run file.
https://download.nvidia.com/XFree86/Linux-x86_64/550.67/README/installedcomponents.html
I don't think the Debian/Ubuntu PPA repositories install the Wayland components automatically.
If so, the following components need to be installed. This is sufficient and contains all missing files.
libnvidia-egl-gbm1
libnvidia-egl-wayland1

Hi,
I am currently trying to get this flying with CRI-o and the nvidia-ctk by using the runtime and CDI config. Afair this can be used in docker as well.
So far all configs and libs are injected/mounted into the container as far as I can see.
I can double check for the reported libs here later.
Currently I'm facing the vblank resource unavailable issue. (Driver version 550.something - cannot look it up atm)

Here are some short steps for what I've done so far:
nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
nvidia-ctk runtime configure --runtime=crio

For docker I guess it's enough to use --runtime=docker
I used the nvidia runtime when starting the container(s) and setting the following Env-Vars:
NVIDIA_DRIVER_CAPABILITIES=all
NVIDIA_VISIBLE_DEVICES="nvidia.com/gpu=all"

As documented, this enables the CDI integration which will mount the host libs and binaries.
What is working so far:

vkgears (directly)
vkgears (gamescope)

What is not working:

glxgears (gamescope): Failed to load driver zink, Blank surface, success render output on STDOUT
steam (gamescope): Failed to load driver zink, vblank sync failures, shm failures seems to be webkit sandbox related

TLDR: as of the latest Nvidia Container Toolkit (1.14.3-1) unfortunately this is still not possible.
What's the issue?
With the latest versions I can run both Wolf and the Gstreamer pipeline just by running the container with --gpus=all unfortunately for some apps this is still missing some important library. Gamescope to run seems to require the following additional libraries that aren't provided by the toolkit:
libnvidia-egl-gbm.so.1
libnvidia-egl-wayland.so.1

libnvidia-vulkan-producer.so

gbm/nvidia-drm_gbm.so

Last one seems to just be a simlink to libnvidia-allocator.so.1 which is already present so that might be fine.
Now this is running from an X11 host and I can see that those additional libraries aren't present in my host system:
ls -la /usr/lib/x86_64-linux-gnu/libnv
libnvcuvid.so@                         libnvidia-container.so.1@              libnvidia-glvkspirv.so.530.30.02       libnvidia-opticalflow.so.1@
libnvcuvid.so.1@                       libnvidia-container.so.1.14.3*         libnvidia-ml.so.1@                     libnvidia-ptxjitcompiler.so.1@
libnvidia-allocator.so.1@              libnvidia-eglcore.so.530.30.02         libnvidia-ngx.so.1@                    libnvidia-rtcore.so.530.30.02
libnvidia-cfg.so.1@                    libnvidia-encode.so.1@                 libnvidia-ngx.so.530.30.02             libnvidia-tls.so.530.30.02
libnvidia-compiler.so.530.30.02        libnvidia-fbc.so.1@                    libnvidia-nvvm.so@                     libnvidia-wayland-client.so.530.30.02
libnvidia-container-go.so.1@           libnvidia-glcore.so.530.30.02          libnvidia-nvvm.so.4@                   libnvoptix.so.1@
libnvidia-container-go.so.1.14.3       libnvidia-glsi.so.530.30.02            libnvidia-opencl.so.1@

Anyone that can confirm the output of
docker run --rm -it --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all  -e NVIDIA_DRIVER_CAPABILITIES=all ubuntu ls /usr/lib/x86_64-linux-gnu/libnvidia*
/usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.1	   /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.530.30.02     /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.4
/usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.1		   /usr/lib/x86_64-linux-gnu/libnvidia-glsi.so.530.30.02       /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.530.30.02  /usr/lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.530.30.02  /usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-eglcore.so.530.30.02   /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1		       /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-encode.so.1		   /usr/lib/x86_64-linux-gnu/libnvidia-ngx.so.1		       /usr/lib/x86_64-linux-gnu/libnvidia-rtcore.so.530.30.02
/usr/lib/x86_64-linux-gnu/libnvidia-fbc.so.1		   /usr/lib/x86_64-linux-gnu/libnvidia-ngx.so.530.30.02        /usr/lib/x86_64-linux-gnu/libnvidia-tls.so.530.30.02

on a Nvidia Wayland host?
What can we do better?
I think we should keep manually downloading and linking the drivers like we are doing at the moment. We should probably add a proper check for mismatch between the downloaded drivers and the host installed drivers either on startup of the containers (somewhere in the base-app) or on startup of Wolf.

ls -1 /usr/lib/x86_64-linux-gnu/libnvidia*
/usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.550.54.14
/usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.550.54.14
/usr/lib/x86_64-linux-gnu/libnvidia-egl-gbm.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-egl-gbm.so.1.1.1
/usr/lib/x86_64-linux-gnu/libnvidia-eglcore.so.550.54.14
/usr/lib/x86_64-linux-gnu/libnvidia-encode.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-encode.so.550.54.14
/usr/lib/x86_64-linux-gnu/libnvidia-fbc.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-fbc.so.550.54.14
/usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.550.54.14
/usr/lib/x86_64-linux-gnu/libnvidia-glsi.so.550.54.14
/usr/lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.550.54.14
/usr/lib/x86_64-linux-gnu/libnvidia-gpucomp.so.550.54.14
/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.550.54.14
/usr/lib/x86_64-linux-gnu/libnvidia-ngx.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-ngx.so.550.54.14
/usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.4
/usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.550.54.14
/usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.550.54.14
/usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.550.54.14
/usr/lib/x86_64-linux-gnu/libnvidia-pkcs11-openssl3.so.550.54.14
/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.1
/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.550.54.14
/usr/lib/x86_64-linux-gnu/libnvidia-rtcore.so.550.54.14
/usr/lib/x86_64-linux-gnu/libnvidia-tls.so.550.54.14

It seems that libnvidia-egl-wayland.so.1 and  libnvidia-vulkan-producer.so is missing. driver version is 550.54.14

libnvidia-egl-wayland seems to be present in the package libnvidia-egl-wayland1.
libnvidia-vulkan-producer.so seems to be dropped from the driver. Need to confirm by building the driver volume for the current release.

Those libraries are for the EGLStreams backend. I believe compositors have now stopped supporting them.
https://github.com/NVIDIA/egl-wayland

Maybe. The latest Nvidia Driver comes with a gbm backend. Maybe that's something useful. I'll give GBM_BACKEND=nvidia-gbm a try. Maybe this will help: https://download.nvidia.com/XFree86/Linux-x86_64/510.39.01/README/gbm.html
From my approach yesterday evening, glxgears is running without any error messages so far. But the Output in moonlight stays black.

I think NVIDIA Container Toolkit 1.15.0 (released not long ago) fixes most of the problems.
Please check it out.

I've finally been successful to run steam and cyberpunk with wolf using the nvidia container toolkit instead of the drivers image.
As @ehfd metioned, the Symlink nvidia-drm_gbm.so is missing, so I had to create it manually before running gamescope/steam:
mkdir -p /usr/lib/x86_64-linux-gnu/gbm;
ln -sv ../libnvidia-allocator.so.1 /usr/lib/x86_64-linux-gnu/gbm/nvidia-drm_gbm.so;

After that launching steam and the game was successful. This all works without the use of libnvidia-egl-wayland.

That sounds really good, thanks for reporting back!
We could easily add that to the images, my only concern is that this only works with a specific version of the toolkit. We should probably add a check on startup for the required libraries and print a proper error message if missing..

If someone has knowledge of Go, could they do the PR for https://github.com/NVIDIA/nvidia-container-toolkit/pull/490?
I will give access to https://github.com/ehfd/nvidia-container-toolkit/tree/main if they ask.

If someone has knowledge of Go, could they contribute fixes for the unsolved aspects of the PR for NVIDIA/nvidia-container-toolkit#490?
I will give write access to https://github.com/ehfd/nvidia-container-toolkit/tree/main if they ask in order to keep it into one PR.

I can take a look at it later

@ehfd Thanks for trusting with the access to your fork. I have addressed the pending issues on the code side and requested some feedback.

Thanks for trusting with the access to your fork. I have addressed the pending issues on the code side and requested some feedback.

No sweat. You know Go better than me and seems like you did a great job at it.

Congrats on getting this merged! This is going to substantially simplify getting the drivers up and running

I've tried upgrading Gstreamer to 1.24.5 unfortunately it's now failing to use Cuda with:
0:00:00.102290171   172 0x560d70ab1190 ERROR              cudanvrtc gstcudanvrtc.cpp:165:gst_cuda_nvrtc_load_library_once: Failed to load 'nvrtcGetCUBINSize', 'nvrtcGetCUBINSize': /usr/local/nvidia/lib/libnvrtc.so: undefined symbol: nvrtcGetCUBINSize

It looks like nvrtcGetCUBINSize was recently added, so I guess there's a mismatch between the libnvrtc.so and what Gstreamer expects. Seems that this was added in this commit which reference @ehfd issue.
I could successfully run this Gstreamer version on my host which has Cuda12 installed, I guess this will break compatibility with older cards so I'm going to revert back Gstreamer in Wolf for now..

@ABeltramo This should not be an issue as long as NVRTC CUDA version is kept at around 11.3.

Thanks for the very quick reply! What's the compatibility matrix for NVRTC? Would upgrading to 11.3 still work for older Cuda installations?

Mostly the internal ABI of NVIDIA.
CUBIN was designed so that there can be forward compatibility for the NVRTC files, but since CUBIN itself didn't exist, there's a problem here.
You can probably fix the issue yourself on GStreamer and then backport it to GStreamer 1.24.6 by a simple error handling in the C code.

Thanks, I've got enough on my plate already.. This is definitely lower priority compared to the rest, looks like we are going to stay on 1.22.7 for a bit longer..

The pull request was technically merged today because it was reverted.
https://github.com/NVIDIA/nvidia-container-toolkit/pull/548

I've tried upgrading Gstreamer to 1.24.5 unfortunately it's now failing to use Cuda with nvrtcGetCUBINSize

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7223
This issue has been resolved for 1.24.6.

I've tried upgrading Gstreamer to 1.24.5 unfortunately it's now failing to use Cuda with nvrtcGetCUBINSize

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7223
This issue has been resolved for 1.24.6.

Thanks for the heads-up, I'm going to test it out again in the next few days!

@tux-rampage did you manage to make it work with CDI? I was trying myself (cause in NixOS the old nvidia wrapper method is now deprecated) but didn't manage to make wolf to use my gpu.

I discovered that CDI overwrites /usr/local/nvidia/lib/ which makes libnvrtc* inaccessible. Would it make sense to copy those libraries somewhere else?

Copying the libraries might break stuff when there's an update in the Nvidia libraries, and you inevitably upgrade the installation.
I've never used CDI myself, but normally with the nvidia runtime the amount and type of libraries that gets mounted from the host is controlled with the NVIDIA_DRIVER_CAPABILITIES env variable. Have you set that or the equivalent of that in order to pull all libraries? Are you running the latest container toolkit version?
sudo nvidia-container-cli -V

Must be >= 1.16.0

@ABeltramo The thing is that with CDI (which is the adviced way of using nvidia with docker now) docker mounts everything defined in a json. One of the entries has  "containerPath": "/usr/local/nvidia/lib" which makes libnvrtc* inaccessible as there is where those libraries are copied in docker image  https://github.com/games-on-whales/wolf/blob/36f407bac40278a20dcdc2b4b4b80c4484bba9ff/docker/gpu-drivers.Dockerfile#L36

This is the docker-compose I'm trying to use and I get the the error of not finding libnvrtc.so
services:
  wolf:
    image: ghcr.io/games-on-whales/wolf:stable
    environment:
      - XDG_RUNTIME_DIR=/tmp/sockets
      - HOST_APPS_STATE_FOLDER=/etc/wolf
      - NVIDIA_DRIVER_CAPABILITIES=all
      - NVIDIA_VISIBLE_DEVICES=nvidia.com/gpu=all
    volumes:
      - /etc/wolf/:/etc/wolf/
      - /tmp/sockets:/tmp/sockets:rw
      - /var/run/docker.sock:/var/run/docker.sock:rw
      - /dev/:/dev/:rw
      - /run/udev:/run/udev:rw
    device_cgroup_rules:
      - 'c 13:* rmw'
    devices:
      - /dev/dri
      - /dev/uinput
      - /dev/uhid
    deploy:
      resources:
        reservations:
          devices:
           - driver: cdi
             device_ids:
                - nvidia.com/gpu=all
    network_mode: host
    restart: unless-stopped

@ABeltramo The thing is that with CDI (which is the adviced way of using nvidia with docker now) docker mounts everything defined in a json. One of the entries has  "containerPath": "/usr/local/nvidia/lib" which makes libnvrtc* inaccessible as there is where those libraries are copied in docker image  https://github.com/games-on-whales/wolf/blob/36f407bac40278a20dcdc2b4b4b80c4484bba9ff/docker/gpu-drivers.Dockerfile#L36

Sorry, my bad. I've misunderstood what you meant, and I forgot about that extra library that we added.
Sure, we can easily move that in a different place and just update the ldconfig; that stuff predates the container toolkit solution, and it probably is safer to move into a non-standard location.
Have you tried re-building the image with that change and see if it fixes it? I'm going to try it out moving that lib and using Wolf with the toolkit. If it doesn't break it, I'm going to push it later so that a new image will be built automatically.

I tried with moving libnvrtc* to /usr/lib/ that way it was found, but when I started an app (firefox) in wolf, it crashed.
| libEGL warning: egl: failed to create dri2 screen
wolf-1  | 2024-09-09T18:31:57.158112Z ERROR smithay::backend::egl::ffi: [EGL] 0x3001 (NOT_INITIALIZED) eglInitialize: DRI2: failed to create screen

this is the stacktrace (sorry to add this here, no idea if you wanna make it a different issue):
wolf-1  |    0:     0x7f573666c575 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hd736fd5964392270
wolf-1  |    1:     0x7f57366900fb - core::fmt::write::hc6043626647b98ea
wolf-1  |    2:     0x7f57366697df - std::io::Write::write_fmt::h0d24b3e0473045db
wolf-1  |    3:     0x7f573666c34e - std::sys_common::backtrace::print::h45eb8174d25a1e76
wolf-1  |    4:     0x7f573666d899 - std::panicking::default_hook::{{closure}}::haf3f0170eb4f3b53
wolf-1  |    5:     0x7f573666d63a - std::panicking::default_hook::hb5d3b27aa9f6dcda
wolf-1  |    6:     0x7f573666dd33 - std::panicking::rust_panic_with_hook::h6b49d59f86ee588c
wolf-1  |    7:     0x7f573666dc14 - std::panicking::begin_panic_handler::{{closure}}::hd4c2f7ed79b82b70
wolf-1  |    8:     0x7f573666ca39 - std::sys_common::backtrace::__rust_end_short_backtrace::h2946d6d32d7ea1ad
wolf-1  |    9:     0x7f573666d947 - rust_begin_unwind
wolf-1  |   10:     0x7f5736456f93 - core::panicking::panic_fmt::ha02418e5cd774672
wolf-1  |   11:     0x7f5736457426 - core::result::unwrap_failed::h55f86ada3ace5ed2
wolf-1  |   12:     0x7f57364ba2f5 - waylanddisplaycore::comp::init::h0646e1a9d0f59f64
wolf-1  |   13:     0x7f5736474cd0 - std::sys_common::backtrace::__rust_begin_short_backtrace::h3e12ecad4fbffac1
wolf-1  |   14:     0x7f5736478a7e - core::ops::function::FnOnce::call_once{{vtable.shim}}::h66c8be57d5da151d
wolf-1  |   15:     0x7f573667074b - std::sys::pal::unix::thread::Thread::new::thread_start::hb85dbfa54ba503d6
wolf-1  |   16:     0x7f573329ca94 - <unknown>
wolf-1  |   17:     0x7f5733329a34 - __clone
wolf-1  | 2024-09-09T18:31:57.170706Z ERROR waylanddisplaycore: Compositor thread panic'ed! err=Any { .. }
wolf-1  | thread '<unnamed>' panicked at /tmp/gst-wayland-display/wayland-display-core/src/lib.rs:87:41:
wolf-1  | called `Result::unwrap()` on an `Err` value: RecvError
wolf-1  | stack backtrace:
wolf-1  |    0:     0x7f573666c575 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hd736fd5964392270
wolf-1  |    1:     0x7f57366900fb - core::fmt::write::hc6043626647b98ea
wolf-1  |    2:     0x7f57366697df - std::io::Write::write_fmt::h0d24b3e0473045db
wolf-1  |    3:     0x7f573666c34e - std::sys_common::backtrace::print::h45eb8174d25a1e76
wolf-1  |    4:     0x7f573666d899 - std::panicking::default_hook::{{closure}}::haf3f0170eb4f3b53
wolf-1  |    5:     0x7f573666d63a - std::panicking::default_hook::hb5d3b27aa9f6dcda
wolf-1  |    6:     0x7f573666dd33 - std::panicking::rust_panic_with_hook::h6b49d59f86ee588c
wolf-1  |    7:     0x7f573666dc14 - std::panicking::begin_panic_handler::{{closure}}::hd4c2f7ed79b82b70
wolf-1  |    8:     0x7f573666ca39 - std::sys_common::backtrace::__rust_end_short_backtrace::h2946d6d32d7ea1ad
wolf-1  |    9:     0x7f573666d947 - rust_begin_unwind
wolf-1  |   10:     0x7f5736456f93 - core::panicking::panic_fmt::ha02418e5cd774672
wolf-1  |   11:     0x7f5736457426 - core::result::unwrap_failed::h55f86ada3ace5ed2
wolf-1  |   12:     0x7f573645e0a3 - waylanddisplaycore::MaybeRecv<T>::get::hfdc57fb8b9f1c5c8
wolf-1  |   13:     0x7f573645f74a - display_get_devices_len
wolf-1  |   14:     0x5602cd93be74 - _ZN4wolf4core15virtual_display22create_wayland_displayERKN5immer5arrayINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS2_13memory_policyINS2_21free_list_heap_policyINS2_8cpp_heapELm1024EEENS2_15refcount_policyENS2_15spinlock_policyENS2_20no_transience_policyELb0ELb1EEEEERKS9_
wolf-1  |                                at /wolf/src/core/src/platforms/linux/virtual-display/wayland-display.cpp:30:36
wolf-1  |   15:     0x5602cd735971 - _ZZZ23setup_sessions_handlersRKN5immer3boxIN5state8AppStateENS_13memory_policyINS_21free_list_heap_policyINS_8cpp_heapELm1024EEENS_15refcount_policyENS_15spinlock_policyENS_20no_transience_policyELb0ELb1EEEEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt8optionalI11AudioServerEENK3$_2clERKNS0_INS1_13StreamSessionESA_EEENKUlvE_clEv
wolf-1  |                                at /wolf/src/moonlight-server/wolf.cpp:225:29
wolf-1  |   16:     0x7f57336eabb4 - <unknown>
wolf-1  |   17:     0x7f573329ca94 - <unknown>
wolf-1  |   18:     0x7f5733329a34 - __clone
wolf-1  | fatal runtime error: failed to initiate panic, error 5

Sounds like CDI isn't mounting the right libraries from the host, definitely worth opening up a separate issue for this.
It would be helpful to report:

General system info, nvidia driver version (nvidia-smi), nvidia toolkit version nvidia-container-cli -V, GPU and if it's running with the latest wolf:stable
Does the same problem occur even when running wolf without CDI and with the standard Container Toolkit (copy paste the command in the quickstart guide

@tux-rampage did you manage to make it work with CDI? I was trying myself (cause in NixOS the old nvidia wrapper method is now deprecated) but didn't manage to make wolf to use my gpu.

Yes, but I use CRIO. Does your Docker instance use the nvidia runtime? If not, the drivers and devices will not be populated.


php-worker can't run artisan commands

Description:
php-worker container does not have 'zip' installed which is causing supervisor to fail.
Expected Behavior:
Supervisor should be able to run artisan commands.
Context information:
Output of git rev-parse HEAD
fc4baf1fc7f2ca150acb76db88c60d83bb48c910

Output of docker version
Client: Docker Engine - Community
 Cloud integration: 1.0.1
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 16 17:00:27 2020
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:07:04 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker-compose version
docker-compose version 1.27.4, build 40524192
docker-py version: 4.3.1
CPython version: 3.7.4
OpenSSL version: OpenSSL 1.1.1c  28 May 2019

System info: Mac, Windows or Linux. Include which disto/version
Windows 10 Pro 64-bit

Stacktrace & Additional info:
docker exec -it {php-worker-container-id} php /var/www/artisan horizon

Composer detected issues in your platform:

Your Composer dependencies require the following PHP extensions to be installed: zip

php-worker logs:
![image](https://user-images.githubusercontent.com/52448924/98358501-1a2e7e00-2027-11eb-89e3-45f4679afa2c.png)

@vdbMark you need to add zip package to PHP-worker container to fix it. It's not a bug.https://github.com/laradock/laradock/blob/89c9cfe09941bead05d63ec3c0fe9b9173ce0526/php-worker/Dockerfile#L62
see, you can add custom packages to Dockerfile to fix. I had also fixed for me for other packages.
I think you can mark this as close.


Handle bad URI by normalizing URLs with invalid characters

As mentioned in #160, I made an approach in order to try to handle this URL with bad/invalid characters. Perhaps it's not yet the best solution so I'm looking for some feedbacks! :)

:+1:

Coverage decreased (-12.96%) to 81.9% when pulling 8eb5af8f633e1bc5eb6ab7fe2d798beedfcf64a8 on fpgentil:handle-bad-url into a728e4c872edbe84c8eb790668662693d44aff30 on peter-murach:master.

Coverage increased (+2.73%) to 97.59% when pulling 401b60e72c60a166a77da72f7a2b32058f989586 on fpgentil:handle-bad-url into a728e4c872edbe84c8eb790668662693d44aff30 on peter-murach:master.

Coverage increased (+2.73%) to 97.59% when pulling 401b60e72c60a166a77da72f7a2b32058f989586 on fpgentil:handle-bad-url into a728e4c872edbe84c8eb790668662693d44aff30 on peter-murach:master.

Hi Felipe, as a way of feedback, the solution is pragmatic, clean and in keeping with current codebase! Thanks!

Thank you, Piotr! :)


Content-Transfer-Encoding for POST request on multipart message not working

Hola everyone,
there seems to be an issue with busboy reading the Content-Transfer-Encoding.
I cannot access this information per field although it is present in the raw data.
Please note that i truncated the actual field values.
I verified its presence by outputting the raw received data on nginx level:
Transmitted data on nginx level
<IP_ADDRESS> - - [07/Apr/2023:13:15:27 +0200] "POST /truncated HTTP/2.0" 200 27 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36 Edg/111.0.1661.62" "----------040723131528994\x0D\x0AContent-Disposition: form-data; name=\x22kndnr\x22\x0D\x0AContent-Type: text/plain\x0D\x0AContent-Transfer-Encoding: quoted-printable\x0D\x0A\x0D\x0Atruncated\x0D\x0AContent-Disposition: form-data; name=\x22name_kunde\x22\x0D\x0AContent-Type: text/plain\x0D\x0AContent-Transfer-Encoding: quoted-printable\x0D\x0A\x0D\x0Atruncated\x0D\x0A----------040723131528994\x0D\x0AContent-Disposition: form-data; name=\x22pw_kunde\x22\x0D\x0AContent-Type: text/plain\x0D\x0AContent-Transfer-Encoding: quoted-printable\x0D\x0A\x0D\x0Atruncated\x0A----------040723131528994\x0D\x0AContent-Disposition: form-data; name=\x22version\x22\x0D\x0AContent-Type: text/plain\x0D\x0AContent-Transfer-Encoding: quoted-printable\x0D\x0A\x0D\x0Atruncated—————040723131528994\x0D\x0AContent-Disposition: form-data; name=\x22action\x22\x0D\x0AContent-Type: text/plain\x0D\x0AContent-Transfer-Encoding: quoted-printable\x0D\x0A\x0D\x0Atruncated\x0D\x0A----------040723131528994\x0D\x0AContent-Disposition: form-data; name=\x22hookurl\x22\x0D\x0AContent-Type: text/plain\x0D\x0AContent-Transfer-Encoding: quoted-printable\x0D\x0A\x0D\x0Ahttp://<IP_ADDRESS>\x0D\x0A----------040723131528994\x0D\x0AContent-Disposition: form-data; name=\x22warenkorb\x22\x0D\x0AContent-Type: text/plain\x0D\x0AContent-Transfer-Encoding: quoted-printable\x0D\x0A\x0D\x0Atruncated\x0D\x0A----------040723131528994--\x0D\x0A"0.014 0.014 . 297
Output of test script
Field [kndnr]:
  Content-Type: undefined
  Content-Transfer-Encoding: undefined
  value: trunc
Field [name_kunde]:
  Content-Type: undefined
  Content-Transfer-Encoding: undefined
  value: trunc
Field [pw_kunde]:
  Content-Type: undefined
  Content-Transfer-Encoding: undefined
  value: trunc
Field [version]:
  Content-Type: undefined
  Content-Transfer-Encoding: undefined
  value: trunc
Field [action]:
  Content-Type: undefined
  Content-Transfer-Encoding: undefined
  value: trunc
Field [hookurl]:
  Content-Type: undefined
  Content-Transfer-Encoding: undefined
  value: trunc
Field [warenkorb]:
  Content-Type: undefined
  Content-Transfer-Encoding: undefined
  value: trunc

Test script
app.post('/ids', (req, res) => {
  const busboy = Busboy({ headers: req.headers });

  busboy.on('headers', (headers, fieldName, encoding, mimetype) => {
    console.log(`Headers for part [${fieldName}] with mimetype [${mimetype}]`);
    console.log(headers);
  });
    busboy.on('field', (fieldname, val, fieldnameTruncated, valTruncated, encoding, mimetype) => {
      console.log('Field [' + fieldname + ']:');
      console.log('  Content-Type: ' + mimetype);
      console.log('  Content-Transfer-Encoding: ' + encoding);
      console.log('  value: ' + val);
    });
  busboy.on('finish', () => {
    res.send('File uploaded successfully!');
  });
  req.pipe(busboy);
});

This is running on an express server. We were actually using multer to access the multipart data and knowing that busboy is the underlying lib - we tested it using the example code (with the same result).
Versions
node -v
v14.18.0

cat package.json | grep busboy
        "busboy": "^1.6.0",

I see two issues here:

I'm not sure where you're getting this 'headers' event, there is no such thing in busboy.

You haven't migrated your code to be compatible with busboy v1.x. Please take a look at this to learn about important changes since v1.0.

You are right, the encoding is available with:
  const busboy = Busboy({ headers: req.headers });

    busboy.on('field', (name, value, info) => {
      console.log('Field [' + name + ']:');
      console.log('  value: ' + value);
      console.log('  Encoding: ' + info.encoding);
    });
  busboy.on('finish', () => {
    res.send('File uploaded successfully!');
  });
  req.pipe(busboy);
});```

Thanks for the super fast response.
Then this must be an issue with multer or a version conflict there. I will further investigate.


Multiple croppers, one image object

Hi, I was wondering whether it's possible to have multiple croppers but with one image object?
What I mean by this is I have a photo uploader where a user can upload up to 10 images. Each image can be cropper. They select an image by clicking on it's thumbnail and on click it updates the image src with originals image src that they clicked. This leads to cropper not recognise which image it's editing and  therefore it doesn't work.
The easiest way is probably to load all images and just hide/show the correct one, but I was wondering if there are any other ways?
Please let me know if you need any more information in order to understand what I'm trying to do.

It is possible to recognize the images by their indexes. This is not a problem of Cropper exactly.

Ok, thanks. Happy to close this


Crash if an input event is sent from a deferred call or notification

Operating system or device - Godot version:
Godot 2.1 and 3.0
Issue description:
MessageQueue::flush() is not designed to be reentrant. But it can contain a message that indirectly makes it called again while the first call has not exited. The new call will reprocess every message already processed by the first call. This produces wrong behaviors and crashes.
This only happens on SceneTree::input_event() because it calls MessageQueue::flush() (at https://github.com/godotengine/godot/blob/de0045cf1b0a5e20fbf74da192039d344ee8d0c7/scene/main/scene_main_loop.cpp#L493 and https://github.com/godotengine/godot/blob/de0045cf1b0a5e20fbf74da192039d344ee8d0c7/scene/main/scene_main_loop.cpp#L514) so if an input event is sent while flush() is already running, that is, during a deferred call, a notification, etc., the bug appears.
I've tried to fix it by making MessageQueue::flush() reentrant (https://github.com/RandomShaper/godot/commit/53f0a2c18287da19d1342206dbeb3cdd22a4212a) but that doesn't sort out every possible situation.
Steps to reproduce:
Create a test scene with this script:
func _ready():
    queue_free()

func _notification(what):
    if what == NOTIFICATION_EXIT_TREE:
        var ie = InputEvent()
        ie.type = 1
        get_tree().input_event(ie)

This example is particularly sadistic because free() will be called twice. (I hope this is correct since I've typed it from the top of my head.)

First of all thank you for your report and sorry for the delay.
We released Godot 3.0 in January 2018 after 18 months of work, fixing many old issues either directly, or by obsoleting/replacing the features they were referring to.
We still have hundreds of issues whose relevance/reproducibility needs to be checked against the current stable version, and that's where you can help us.
Could you check if the issue that you described initially is still relevant/reproducible in Godot 3.0 or any newer version, and comment about its current status here?
For bug reports, please also make sure that the issue contains detailed steps to reproduce the bug and, if possible, a zipped project that can be used to reproduce it right away. This greatly speeds up debugging and bugfixing tasks for our contributors.
Our Bugsquad will review this issue more in-depth in 15 days, and potentially close it if its relevance could not be confirmed.
Thanks in advance.
Note: This message is being copy-pasted to many "stale" issues (90+ days without activity). It might happen that it is not meaningful for this specific issue or appears oblivious of the issue's context, if so please comment to notify the Bugsquad about it.

Tested in master, the crash doesn't seem to happen anymore with the above script.


Fix links to changelog in changelog

Hey someone is reading this.
And someone else messed the links up... :)
TY!


Image Editor: make the size adjustable

Adjust size of image editor / empty image editor for more space etc.

re "zoom" in the title: you can zoom, just scroll wheel.
What I assume you probably want, is to be able to adjust the center bar -- currently it does a 50/50 split between the editor canvas and the output image, that should be adjustable.
Note that the left bar, right bar, and bottom bar are all collapsible, it's only the center bar that's not currently easily adjustable.

There's now a central size bar for the image editor

Thank you legend


On treenode hover show the node information

When the mouse do an hover on a treenode I want to see the node information

Enhancement added


specifying URL query parameters in the jasmine_url option

I use jasminerice with guard-jasmine and the jasmine docs suggest that my specs will load much faster if I set debug=true in a query parameter:
http://localhost:3000/jasmine?debug=false

Wanting to do this when running the specs with guard-jasmine I tried adding the following options:
port: 55998, jasmine_url: 'https://localhost:55998/jasmine?debug=false'

However, this didn't work because when running individual specs it ended up like this:
https://localhost:55998/jasmine?debug=false?spec=SomeSpecs

And the whole set run.
It would be helpful if Guard::Jasmine::Runner.query_string_for_suite checked for whether the url already contains any query parameters and begins the encoded_url with either ? or & accordingly.
I resorted to the following hack, that I placed in the end of my Guardfile:
module ::Guard::Jasmine::Runner
  class << self
    unless method_defined? :query_string_for_suite
      alias_method :orig_query_string_for_suite, :query_string_for_suite
    end
    def query_string_for_suite(*args)
      encoded_url = orig_query_string_for_suite *args
      if encoded_url =~ /^\?/
        encoded_url = encoded_url.sub /^\?/, "?debug=false&"
      else
        encoded_url = "?debug=false"
      end
      encoded_url
    end
  end
end

Is there a less hacky way to accomplish this? Or does it require a change to guard-jasmine? I'm not familiar with the spectrum of typical configurations and runner combinations to know whether this is the right place for such a change.
Thanks!

I'm pruning old issues I hope the new version worked for you, please open a new issue if you're still having problems

Ah. No worries. I switched away from using Jasmine because QUnit was better supported for Ember at the time. Appologies for the dangling issue.


Revert "Switch IBM Cloud for multi-platform to new account (#2996)"

This reverts commit 7ecde5da4b026d5556f368e108c04ab922aa3540.

/lgtm


Possible deadlock while running tests

Describe the bug
I had to cancel this test run (Python 3.12) after 5 minutes:

The second attempt succeeded just fine, but still shows a deprecation warning, which is likely related to the issue:
   /opt/hostedtoolcache/Python/3.12.0/x64/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=1909) is multi-threaded, use of fork() may lead to deadlocks in the child.
    self.pid = os.fork()

To Reproduce

Run test_should_execute_pipeline_return_valid_placeholder repeatedly
Eventually, the process gets stuck in wait_for_messages.

Expected behavior
Tests should always run through. As an extra safeguard, we might want to limit the runtime of some "dangerous" tests and fail them after x seconds. This can be done using pytest-timeout.
Screenshots (optional)
No response
Additional Context (optional)
This issue also occurred in this run.

:tada: This issue has been resolved in version 0.4.0 :tada:
The release is available on:

v0.4.0
GitHub release

Your semantic-release bot :package::rocket:


Make poutine aliases appear in sphinx

Resolves #1069
Part of #826

Consider combining docs/source/advanced.rst + docs/source/pyro.poutine.txt into docs/source/poutine.rst ?


HTMLInput object has no attribute intervideosink

When attempting to add any html as input via the Web GUI, the following message is logged
Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/brave-cXY7_SU8/lib/python3.6/site-packages/sanic/app.py", line 917, in handle_request
    response = await response
  File "/brave/brave/api/route_handler.py", line 13, in all
    'inputs': request['session'].inputs.summarise(),
  File "/brave/brave/abstract_collection.py", line 40, in summarise
    s.append(obj.summarise())
  File "/brave/brave/inputs/input.py", line 58, in summarise
    cap_props = self.get_input_cap_props()
  File "/brave/brave/inputs/html.py", line 48, in get_input_cap_props
    element = self.intervideosink
AttributeError: 'HTMLInput' object has no attribute 'intervideosink'

This is a docker instance build from master on Augh 29 2019

Sorry for the deplaying in getting back to you
If you do gst-inspect-1.0 intervideosink is anything printed out ?

Hello @mhzbg,
The GstWebRender was more of demonstration of what was possible. It was also made in a bit of a rush and as such there were some bugs with it. Since the GStreamer conference in 2018, there have been two other HTML renders created by other teams which have been very kindly open sourced.
Please have a look at:
https://github.com/centricular/gstcefsrc
and
https://base-art.net/Articles/web-overlay-in-gstreamer-with-wpewebkit/
As much more stable alternatives.

Hello @moschopsuk ,
thank you for the response.
I have already looked at the links you provided perviously.
The real question is how to get the html overlays to work with the GUI that brave provides.
What module does it relay on, and how to resolve the error reported earlier.
Thanks

Apologies @mhzbg I miss understood the question.
The file https://github.com/bbc/brave/blob/master/brave/inputs/html.py is what will try to load the element.
Currently it loads thecef element but this could be changed to use the two other web renders o list above.
Thanks.

Closing as a Stale issue


Rename to stolostron and open-cluster-management.io

Signed-off-by: Dominique Vernier<EMAIL_ADDRESS>

Seems this is already in process


Bringing FAB back

For mobile: FAB quick-submit is back and we're hiding the Create button from the navbar.

Travis tests are not passing, I already updated snaps on my side.

I don't understand why it's changing the escaping of spaces.

@andreadelrio seems like you've submitted new snapshots which don't match what's happening in CI (and what was happening before).  So, something about your snapshot generation process is different.  Can you make sure you have the right version of Jest installed?

Also covers #338 and #339.

Also covers #291


1024 Playwright protocol implementation, fixes #1024

Thank you for contributing to Apache StormCrawler.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ x] Is there a issue associated with this PR? Is it referenced in the commit message?

[x ] Does your PR title start with #XXXX where XXXX is the issue number you are trying to resolve?

[x ] Has your PR been rebased against the latest commit within the target branch (typically main)?

[ x] Is your initial contribution a single, squashed commit?

[ x] Is the code properly formatted with mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false?

For code changes:

[ x] Have you ensured that the full suite of tests is executed via mvn clean verify?
[ x] Have you written or updated unit tests to verify your changes?
[ x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[x ] If applicable, have you updated the LICENSE file, including the main LICENSE file?
[ x] If applicable, have you updated the NOTICE file, including the main NOTICE file?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions for build issues and submit an update to your PR as soon as possible.

A great addition to SC!


Added mkfs command

I think there is a step missing. Otherwise there will be a filesystem error:
mount: /storage1: wrong fs type, bad option, bad superblock on /dev/mapper/storage1, missing codepage or helper program, or other error.

Actually, I have done a reboot and I realised that maybe my suggested change is not what it was intended in the documentation, but then perhaps this bit can be removed:
mkdir /storage1
And leave only:
$ cryptsetup luksOpen /dev/vdb storage1
$ mount -t ext4 /dev/mapper/storage1 /storage1

Actually, I have done a reboot and I realised that maybe my suggested change is not what it was intended in the documentation, but then perhaps this bit can be removed: mkdir /storage1
And leave only:
$ cryptsetup luksOpen /dev/vdb storage1
$ mount -t ext4 /dev/mapper/storage1 /storage1

ok but if you don't have the /storage1 folder created already the mount command will fail no?

Apparently the filesystem is already created in step 2: https://docs.egi.eu/users/data/storage/block-storage/#storage-encryption, so I don't think mkfs should be used here.
I agree that the mkdir may be useless as it got done in step 3, unless those commands are to be run from a different VM mounting the already encrypted and formatted disk. Maybe exact use case could be clarified, or mkdir -p could be used, as it wouldn't complain if the folder already exist.

@andrea-manzi Yes, it will fail, but these steps are assuming that mkfs.ext4 /dev/mapper/storage1 has been already done too, so it is reasonable to assume that other "permanent" operations, such as mkdir /storage1 have been done too.   That is why I got confused, I initially thought that these were the steps for a "permanent" use, so it should include only permanent steps. On my reboot, I realised that these were the "routine" steps, so then the creation of the folder made no sense (unless used in a different VM, as pointed by @gwarf )
Anyway, it is a minor detail. I found this documentation very clear and straightforward, very nicely written.

For the automatic mounting, one could use /etc/crypttab and /etc/fstab, but the main problem is probably the unlock of the encrypted FS, and the need to provide the passphrase.
With luks you can use a key file to unlock a partition and have the key optionally without a passphrase, but it's also making this way less secure/useful, not that recommended if the data really needs to be protected...

aaah ok got it now, you are referring to the last part of the doc. Yes there we could remove the mkdir or add the -p option as reported by @gwarf

I have made a change based on the comments here. I also changed the word "following" to be more explicit, as it confused me.