"error: cuda_runtime_api.h: No such file or directory"

Hello, I'm trying to run the basic example. I have several LLMs working and have used Huggingface Hub to download them, for reference. However, I get this error in the title. Indeed this file is not found in:
/home/user/.local/lib/python3.10/site-packages/torch/include/c10/I did find it here:
/usr/local/cuda-11.7/targets/x86_64-linux/include/cuda_runtime_api.h
I had a challenging time getting my nvidia driver to work with the right cuda version during torch install. Current PyTorch version is: Version: 1.12.1+cu116. You can see the version 11.7 in the above path. I'm not sure how relevant that is, but this is the only combination of cuda and torch versions I could get working. I think c10 denotes the default version of torch installed with python 3.10 on Ubuntu 22.04. Which is supported by this quote from SE:

"PyTorch doesn't use the system's CUDA library. When you install PyTorch using the precompiled binaries using either pip or conda it is shipped with a copy of the specified version of the CUDA library which is installed locally."

The output does say:
Installed CUDA version 11.7 does not match the version torch was compiled with 11.6 but since the APIs are compatible, accepting this combination Using /home/user/.cache/torch_extensions/py310_cu116 as PyTorch extensions root...
Do I need to set some environment vars and/or install another version of PyTorch in a virtualenv? I'm a little short on space, so hopping not. It seems there is some conflict between the default PyTorch c10 locations and the discovered 11.6/11.7 version of Cuda.
Quick side note: the models downloaded to /tmp/mii_models. Is it possible to use the standard Huggingface model locations?

Hi @auwsom sorry to see that you're having trouble running MII. The output about mismatched CUDA versions shouldn't be an issue. It's only a warning and I'm currently running with the same setup (CUDA11.7 on the machine, pytorch compiled with 11.6). Could you please verify that your torch setup is working with CUDA?
import torch
print(torch.__version__)
print(torch.cuda.is_available())

Also, if you can include the entire output, so we can see when this error happens that would help us debug the issue.
As for the model download location, you can specify where you want to download the models with the model_path parameter in mii.deploy(). See the source code here. If you want to use the HF cache (assuming you have the latest version of transformers): model_path="~/.cache/huggingface/hub"

Hi @mrwyattii, thanks for the reply.
Here is the python output.
>>> print(torch.__version__)
1.12.1+cu116
>>> print(torch.cuda.is_available())
True

Here is the entire error message. Some is output through Jupyter.
Error - click
```
[2022-10-24 15:07:31,460] [INFO] [deployment.py:85:deploy] ************* MII is using DeepSpeed Optimizations to accelerate your model *************
[2022-10-24 15:07:31,481] [INFO] [server_client.py:217:_initialize_service] MII using multi-gpu deepspeed launcher:
 ------------------------------------------------------------
 task-name .................... text-generation
 model ........................ bigscience/bloom-560m
 model-path ................... /tmp/mii_models
 port ......................... 50050
 provider ..................... hugging-face
 ------------------------------------------------------------
[2022-10-24 15:07:34,203] [WARNING] [runner.py:179:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2022-10-24 15:07:34,221] [INFO] [runner.py:507:main] cmd = /usr/bin/python3 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMF19 --master_addr=<IP_ADDRESS> --master_port=29500 --no_python --no_local_rank /usr/bin/python3 -m mii.launch.multi_gpu_server --task-name text-generation --model bigscience/bloom-560m --model-path /tmp/mii_models --port 50050 --ds-optimize --provider hugging-face --config eyJ0ZW5zb3JfcGFyYWxsZWwiOiAxLCAicG9ydF9udW1iZXIiOiA1MDA1MCwgImR0eXBlIjogImZwMTYiLCAiZW5hYmxlX2N1ZGFfZ3JhcGgiOiBmYWxzZSwgImNoZWNrcG9pbnRfZGljdCI6IG51bGwsICJkZXBsb3lfcmFuayI6IFswXSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiaGZfYXV0aF90b2tlbiI6IG51bGwsICJyZXBsYWNlX3dpdGhfa2VybmVsX2luamVjdCI6IHRydWUsICJwcm9maWxlX21vZGVsX3RpbWUiOiBmYWxzZX0=
[2022-10-24 15:07:36,491] [INFO] [server_client.py:115:_wait_until_server_is_live] waiting for server to start...
[2022-10-24 15:07:36,592] [INFO] [launch.py:136:main] WORLD INFO DICT: {'localhost': [0]}
[2022-10-24 15:07:36,592] [INFO] [launch.py:142:main] nnodes=1, num_local_procs=1, node_rank=0
[2022-10-24 15:07:36,592] [INFO] [launch.py:155:main] global_rank_mapping=defaultdict(, {'localhost': [0]})
[2022-10-24 15:07:36,593] [INFO] [launch.py:156:main] dist_world_size=1
[2022-10-24 15:07:36,593] [INFO] [launch.py:158:main] Setting CUDA_VISIBLE_DEVICES=0
[2022-10-24 15:07:41,498] [INFO] [server_client.py:115:_wait_until_server_is_live] waiting for server to start...
[2022-10-24 15:07:46,504] [INFO] [server_client.py:115:_wait_until_server_is_live] waiting for server to start...
[2022-10-24 15:07:51,508] [INFO] [server_client.py:115:_wait_until_server_is_live] waiting for server to start...
[2022-10-24 15:07:56,515] [INFO] [server_client.py:115:_wait_until_server_is_live] waiting for server to start...
[2022-10-24 15:08:01,521] [INFO] [server_client.py:115:_wait_until_server_is_live] waiting for server to start...
[2022-10-24 15:08:06,529] [INFO] [server_client.py:115:_wait_until_server_is_live] waiting for server to start...
> --------- MII Settings: ds_optimize=True, replace_with_kernel_inject=True, enable_cuda_graph=False
[2022-10-24 15:08:10,989] [INFO] [logging.py:68:log_dist] [Rank -1] DeepSpeed info: version=0.7.4, git-hash=unknown, git-branch=unknown
[2022-10-24 15:08:10,989] [INFO] [logging.py:68:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
[2022-10-24 15:08:11,535] [INFO] [server_client.py:115:_wait_until_server_is_live] waiting for server to start...
Installed CUDA version 11.7 does not match the version torch was compiled with 11.6 but since the APIs are compatible, accepting this combination
Using /home/user/.cache/torch_extensions/py310_cu116 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/user/.cache/torch_extensions/py310_cu116/transformer_inference/build.ninja...
Building extension module transformer_inference...

Traceback (most recent call last):
  File "/home/user/.local/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1808, in _run_ninja_build
    subprocess.run(
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/user/dev/DeepSpeed-MII/mii/launch/multi_gpu_server.py", line 70, in
main()
File "/home/user/dev/DeepSpeed-MII/mii/launch/multi_gpu_server.py", line 56, in main
inference_pipeline = load_models(task_name=args.task_name,
File "/home/user/dev/DeepSpeed-MII/mii/models/load_models.py", line 73, in load_models
engine = deepspeed.init_inference(getattr(inference_pipeline,
File "/home/user/.local/lib/python3.10/site-packages/deepspeed/init.py", line 305, in init_inference
engine = InferenceEngine(model,
File "/home/user/.local/lib/python3.10/site-packages/deepspeed/inference/engine.py", line 149, in init
self._apply_injection_policy(
File "/home/user/.local/lib/python3.10/site-packages/deepspeed/inference/engine.py", line 367, in _apply_injection_policy
replace_transformer_layer(
File "/home/user/.local/lib/python3.10/site-packages/deepspeed/module_inject/replace_module.py", line 930, in replace_transformer_layer
replaced_module = replace_module(model=model,
File "/home/user/.local/lib/python3.10/site-packages/deepspeed/module_inject/replace_module.py", line 1187, in replace_module
replaced_module, _ = _replace_module(model, policy)
File "/home/user/.local/lib/python3.10/site-packages/deepspeed/module_inject/replace_module.py", line 1214, in _replace_module
_, layer_id = _replace_module(child, policies, layer_id=layer_id)
File "/home/user/.local/lib/python3.10/site-packages/deepspeed/module_inject/replace_module.py", line 1214, in _replace_module
_, layer_id = _replace_module(child, policies, layer_id=layer_id)
File "/home/user/.local/lib/python3.10/site-packages/deepspeed/module_inject/replace_module.py", line 1204, in _replace_module
replaced_module = policies[child.class][0](child,
File "/home/user/.local/lib/python3.10/site-packages/deepspeed/module_inject/replace_module.py", line 920, in replace_fn
new_module = replace_with_policy(child,
File "/home/user/.local/lib/python3.10/site-packages/deepspeed/module_inject/replace_module.py", line 505, in replace_with_policy
new_module = transformer_inference.DeepSpeedTransformerInference(
File "/home/user/.local/lib/python3.10/site-packages/deepspeed/ops/transformer/inference/transformer_inference.py", line 777, in init
inference_cuda_module = builder.load()
File "/home/user/.local/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 459, in load
return self.jit_load(verbose)
File "/home/user/.local/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 494, in jit_load
op_module = load(
File "/home/user/.local/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1202, in load
return _jit_compile(
File "/home/user/.local/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1425, in _jit_compile
_write_ninja_file_and_build_library(
File "/home/user/.local/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1537, in _write_ninja_file_and_build_library
_run_ninja_build(
File "/home/user/.local/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1824, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'transformer_inference'
[2022-10-24 15:08:15,639] [INFO] [launch.py:286:sigkill_handler] Killing subprocess 2649619
[2022-10-24 15:08:15,640] [ERROR] [launch.py:292:sigkill_handler] ['/usr/bin/python3', '-m', 'mii.launch.multi_gpu_server', '--task-name', 'text-generation', '--model', 'bigscience/bloom-560m', '--model-path', '/tmp/mii_models', '--port', '50050', '--ds-optimize', '--provider', 'hugging-face', '--config', 'eyJ0ZW5zb3JfcGFyYWxsZWwiOiAxLCAicG9ydF9udW1iZXIiOiA1MDA1MCwgImR0eXBlIjogImZwMTYiLCAiZW5hYmxlX2N1ZGFfZ3JhcGgiOiBmYWxzZSwgImNoZWNrcG9pbnRfZGljdCI6IG51bGwsICJkZXBsb3lfcmFuayI6IFswXSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiaGZfYXV0aF90b2tlbiI6IG51bGwsICJyZXBsYWNlX3dpdGhfa2VybmVsX2luamVjdCI6IHRydWUsICJwcm9maWxlX21vZGVsX3RpbWUiOiBmYWxzZX0='] exits with return code = 1
[2022-10-24 15:08:16,542] [INFO] [server_client.py:115:_wait_until_server_is_live] waiting for server to start...
RuntimeError                              Traceback (most recent call last)
Input In [8], in <cell line: 3>()
1 import mii
2 mii_configs = {"tensor_parallel": 1, "dtype": "fp16"}
----> 3 mii.deploy(task="text-generation",
4            model="bigscience/bloom-560m",
5            deployment_name="bloom560m_deployment",
6            mii_config=mii_configs)
File ~/dev/DeepSpeed-MII/mii/deployment.py:112, in deploy(task, model, deployment_name, deployment_type, model_path, enable_deepspeed, enable_zero, ds_config, mii_config, version)
110     _deploy_aml(deployment_name=deployment_name, model_name=model, version=version)
111 elif deployment_type == DeploymentType.LOCAL:
--> 112     return _deploy_local(deployment_name, model_path=model_path)
113 else:
114     raise Exception(f"Unknown deployment type: {deployment_type}")
File ~/dev/DeepSpeed-MII/mii/deployment.py:118, in _deploy_local(deployment_name, model_path)
117 def _deploy_local(deployment_name, model_path):
--> 118     mii.utils.import_score_file(deployment_name).init()
File /tmp/mii_cache/bloom560m_deployment/score.py:29, in init()
26 assert task is not None, "The task name should be set before calling init"
28 global model
---> 29 model = mii.MIIServerClient(task,
30                             model_name,
31                             model_path,
32                             ds_optimize=configs[mii.constants.ENABLE_DEEPSPEED_KEY],
33                             ds_zero=configs[mii.constants.ENABLE_DEEPSPEED_ZERO_KEY],
34                             ds_config=configs[mii.constants.DEEPSPEED_CONFIG_KEY],
35                             mii_configs=configs[mii.constants.MII_CONFIGS_KEY],
36                             use_grpc_server=use_grpc_server,
37                             initialize_grpc_client=initialize_grpc_client)
File ~/dev/DeepSpeed-MII/mii/server_client.py:90, in MIIServerClient.init(self, task_name, model_name, model_path, ds_optimize, ds_zero, ds_config, mii_configs, initialize_service, initialize_grpc_client, use_grpc_server)
83     self.process = self._initialize_service(model_name,
84                                             model_path,
85                                             ds_optimize,
86                                             ds_zero,
87                                             ds_config,
88                                             mii_configs)
89     if self.use_grpc_server:
---> 90         self._wait_until_server_is_live()
92 if self.initialize_grpc_client and self.use_grpc_server:
93     self.stubs = []
File ~/dev/DeepSpeed-MII/mii/server_client.py:113, in MIIServerClient._wait_until_server_is_live(self)
111 process_alive = self._is_server_process_alive()
112 if not process_alive:
--> 113     raise RuntimeError("server crashed for some reason, unable to proceed")
114 time.sleep(4)
115 logger.info("waiting for server to start...")
RuntimeError: server crashed for some reason, unable to proceed
</details>

Thanks for the tip on the Hub usage.

@auwsom The error is originating from DeepSpeed when trying to JIT compile the inference kernels. Can you share the output of ds_report?
Also, let's try pre-compiling the kernels. You can do that by running: DS_BUILD_TRANSFORMER_INFERENCE=1 pip install deepspeed

@mrwyattii
find -iname report didnt find ds_report.py. is it somewhere not in the repo?
rerunning pip with that pre-compile var gives the same error FAILED: relu.cuda.o
(Also, it may be handy to put the model_path huggingface hub line in the example, even if commented out. And also the import mii so people could just run the example.)

the only mention of ds_report is in this file: https://github.com/microsoft/DeepSpeed-MII/blob/16ea809285d60e6e28ca8ab773cd63b2183565fd/.github/workflows/nv-torch-latest-v100.yaml
i see its in: https://github.com/microsoft/DeepSpeed/tree/master/bin

--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
 [WARNING]  please install triton==1.0.0 if you want to use sparse attention
sparse_attn ............ [NO] ....... [NO]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
utils .................. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/user/.local/lib/python3.10/site-packages/torch']
torch version .................... 1.12.1+cu116
torch cuda version ............... 11.6
torch hip version ................ None
nvcc version ..................... 11.7
deepspeed install path ........... ['/home/user/.local/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.7.4, unknown, unknown
deepspeed wheel compiled w. ...... torch 1.12, cuda 11.6

The ds_report script is installed when DeepSpeed is installed. It looks like your DeepSpeed install is ok and your system appears compatible with the inference kernels used with the bloom model. Can you please try pre-compiling those kernels with DS_BUILD_TRANSFORMER_INFERENCE=1 pip install deepspeed?

@mrwyattii interesting, I didn't originally install DeepSpeed, only DeepSpeed-MII. I cloned the repo yesteday just to run ../bin/ds_report. But I also tried your precompile var DS_BUILD_TRANSFORMER_INFERENCE=1 pip install deepspeed. And the same error reply was after that.

DeepSpeed is a dependency for DeepSpeed-MII and pip will install it when you do pip install . in the MII repo or pip install deepspeed-mii
I'm a bit confused about your setup as you were able to pre-compile the kernels when you installed deepspeed, so you shouldn't be running into the error you saw previously. I think there may be an issue with your environment. You mention you are using a venv - where is that venv? I see multiple paths: /home/user/.local/lib/python3.10/ and /usr/bin/python3 in the error message, which make me believe MII is picking up the wrong python and/or python packages. If you can, start with a fresh environment:
python3 -m virtualenv test_venv
source ./test_venv/bin/activate
pip3 install --pre torch torchvision --extra-index-url https://download.pytorch.org/whl/nightly/cu117
pip3 install deepspeed
pip3 install deepspeed-mii

Let me know if you get the same error doing this. Thanks!

@mrwyattii  thanks. I dont have enough space currently to install another torch (already have 2 installed, one from python default and the 11.6 for my nvidia drivers).
I the meanwhile here is my pip list inside my venv. It is basically the same as system with --system-packages, then with DeepSpeed-MII installed for testing.
 It doesnt do that on GH Gist. So here it is again if it helps, but it is just the same error. I'll try to clone this VM and run it on the HDD after expanding.
https://gist.github.com/auwsom/2faf04fc8280685a3342e87a32402113

@mrwyattii also, sorry, I didnt realize that collapsing of the error above broke the formatting. It doesnt do that on GH Gist. So here it is again if it helps, but it is just the same error. I'll try to clone this VM and run another torch install on the HDD after expanding.
https://gist.github.com/auwsom/2faf04fc8280685a3342e87a32402113

This error message is much easier to parse and I found the real error here:
/home/user/.local/lib/python3.10/site-packages/torch/include/c10/cuda/CUDAStream.h:6:10: fatal error: cuda_runtime_api.h: No such file or directory
    6 | #include <cuda_runtime_api.h>

This indicates it's a problem with your environment, but it's hard to say exactly what needs to change in your environment to fix this.  That file is definitely not there.
I have a VM image of Ubuntu 22.04 with the Kubuntu added on top of the cloud-init server image. Wasn't installed from ISO, but should be pretty vanilla to the same images used in GCP, AWS, etc.
The Host is the same LTS Ubuntu image before it was upgraded at 20.04. GPU is passed through with virtio and Virt-manager pretty standardly. Added about 50GB ram and 12 CPU cores to it. Storage is about 40GBs.
I have added other python packages, and installed the Nvidia driver with Apt and compatible Cuda by a direct download. I'll edit in a link about the process, from askubuntu.com.
Then created a venv after cloning the MII repo and ran the pip install .
Somewhere this .h api file wasn't built. That's why my initial theory was the C complier wasn't getting info about the correct Cuda version. But that is a guess. Is there a verbose mode for the compile process or a log file maybe?

You're right, this is the first error you pointed out. Sorry this thread is getting quite long and I missed that.
The way you're setting up your VM and venv seems fine. The error is really coming from torch and not DeepSpeed / MII. I see that you have found the header file that the compiler cannot find here /usr/local/cuda-11.7/targets/x86_64-linux/include/cuda_runtime_api.h. You may try including this in CUDA_HOME or LD_LIBRARY_PATH in your environment. See the suggestion here for a similar error: https://github.com/HawkAaron/warp-transducer/issues/15#issuecomment-467668750

closing this due to inactivity, please reopen if there are updates.


Refactor updateGame to reduce complexity

Summary
Refactor updateGame in two two more more discrete functions
Detailed Description
updateGame in client/js/store/modules/game.js currently fires twice when joining a lobby. Once with the full game state which works as expected and updates all the values.
The second call to updateGame comes from updating the players and only has the player values. This was noticed when working on the isRanked logic. For instance, we currently update values in this way:
updateGame(state, newGame) {
  ...
  if (Object.hasOwnProperty.call(newGame, 'isRanked')) state.isRanked = newGame.isRanked;
},

We're locking the logic behind the existence of the keys on the object we're passing which means we expect one or more schemas for that newGame object. Ideally, this would be refactored to something like this:

updateGame - always takes the full game object so we don't need the hasOwnProperty calls and can directly update the state with a set of keys
updateGamePlayers - always takes the players and adds them to the game data
Other functions may be needed depending on how many ways this function is called and with which payloads

@itsalaidbacklife may have more information here, but this should be enough to do some digging, come up with a plan and then implement an initial pass.

See #189 for reworking this to support localization by generating the card text client side


fix https://opentable.atlassian.net/browse/SOUS-874

add back the + check in front of file, not checking comment, will add that later if we really need it.

  Generated by :no_entry_sign: Danger

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@4cb0e07). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #588   +/-   ##
=========================================
  Coverage          ?   70.52%
=========================================
  Files             ?      224
  Lines             ?    12697
  Branches          ?        0
=========================================
  Hits              ?     8955
  Misses            ?     3124
  Partials          ?      618

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4cb0e07...b782f78. Read the comment docs.


How to create rules and permissions?

It's returns an exeption "Class {a} is undefined" when I'm writing for example "site/contact", what's the meter? What's wrong?

Rule is a class which you should create before using. You can read more about rules here


In train model session have error

2024-12-26 08:29:24 | INFO | main | Use gpus: 0
2024-12-26 08:29:24 | INFO | main | Execute: "/usr/local/envs/rvc/bin/python" infer/modules/train/train.py -e "Necco" -sr 40k -f0 1 -bs 19 -g 0 -te 64 -se 5 -pg assets/pretrained_v2/f0G40k.pth -pd assets/pretrained_v2/f0D40k.pth -l 0 -c 0 -sw 0 -v v2
INFO:Necco:{'data': {'filter_length': 2048, 'hop_length': 400, 'max_wav_value': 32768.0, 'mel_fmax': None, 'mel_fmin': 0.0, 'n_mel_channels': 125, 'sampling_rate': 40000, 'win_length': 2048, 'training_files': './logs/Necco/filelist.txt'}, 'model': {'filter_channels': 768, 'gin_channels': 256, 'hidden_channels': 192, 'inter_channels': 192, 'kernel_size': 3, 'n_heads': 2, 'n_layers': 6, 'p_dropout': 0, 'resblock': '1', 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'resblock_kernel_sizes': [3, 7, 11], 'spk_embed_dim': 109, 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4], 'upsample_rates': [10, 10, 2, 2], 'use_spectral_norm': False}, 'train': {'batch_size': 19, 'betas': [0.8, 0.99], 'c_kl': 1.0, 'c_mel': 45, 'epochs': 20000, 'eps': 1e-09, 'fp16_run': True, 'init_lr_ratio': 1, 'learning_rate': 0.0001, 'log_interval': 200, 'lr_decay': 0.999875, 'seed': 1234, 'segment_size': 12800, 'warmup_epochs': 0}, 'model_dir': './logs/Necco', 'experiment_dir': './logs/Necco', 'save_every_epoch': 5, 'name': 'Necco', 'total_epoch': 64, 'pretrainG': 'assets/pretrained_v2/f0G40k.pth', 'pretrainD': 'assets/pretrained_v2/f0D40k.pth', 'version': 'v2', 'gpus': '0', 'sample_rate': '40k', 'if_f0': 1, 'if_latest': 0, 'save_every_weights': '0', 'if_cache_data_in_gpu': 0}
/usr/local/envs/rvc/lib/python3.10/site-packages/torch/nn/utils/weight_norm.py:143: FutureWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
WeightNorm.apply(module, name, dim)
DEBUG:infer.lib.infer_pack.models:gin_channels: 256, self.spk_embed_dim: 109
INFO:Necco:loaded pretrained assets/pretrained_v2/f0G40k.pth
/content/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py:231: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
torch.load(hps.pretrainG, map_location="cpu")["model"]
INFO:Necco:
INFO:Necco:loaded pretrained assets/pretrained_v2/f0D40k.pth
/content/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py:246: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
torch.load(hps.pretrainD, map_location="cpu")["model"]
INFO:Necco:
/content/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py:263: FutureWarning: torch.cuda.amp.GradScaler(args...) is deprecated. Please use torch.amp.GradScaler('cuda', args...) instead.
scaler = GradScaler(enabled=hps.train.fp16_run)
/content/Retrieval-based-Voice-Conversion-WebUI/infer/lib/train/data_utils.py:114: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
spec = torch.load(spec_filename)
/content/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py:429: FutureWarning: torch.cuda.amp.autocast(args...) is deprecated. Please use torch.amp.autocast('cuda', args...) instead.
with autocast(enabled=hps.train.fp16_run):
/content/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py:457: FutureWarning: torch.cuda.amp.autocast(args...) is deprecated. Please use torch.amp.autocast('cuda', args...) instead.
with autocast(enabled=False):
/content/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py:476: FutureWarning: torch.cuda.amp.autocast(args...) is deprecated. Please use torch.amp.autocast('cuda', args...) instead.
with autocast(enabled=False):
/content/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py:486: FutureWarning: torch.cuda.amp.autocast(args...) is deprecated. Please use torch.amp.autocast('cuda', args...) instead.
with autocast(enabled=hps.train.fp16_run):
/content/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py:489: FutureWarning: torch.cuda.amp.autocast(args...) is deprecated. Please use torch.amp.autocast('cuda', args...) instead.
with autocast(enabled=False):
/usr/local/envs/rvc/lib/python3.10/site-packages/torch/autograd/graph.py:825: UserWarning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed.  This is not an error, but may impair performance.
grad.sizes() = [64, 1, 4], strides() = [4, 1, 1]
bucket_view.sizes() = [64, 1, 4], strides() = [4, 4, 1] (Triggered internally at ../torch/csrc/distributed/c10d/reducer.cpp:327.)
return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
INFO:Necco:Train Epoch: 1 [0%]
INFO:Necco:[0, 0.0001]
INFO:Necco:loss_disc=4.441, loss_gen=2.692, loss_fm=0.571,loss_mel=27.202, loss_kl=9.000
DEBUG:matplotlib:matplotlib data path: /usr/local/envs/rvc/lib/python3.10/site-packages/matplotlib/mpl-data
DEBUG:matplotlib:CONFIGDIR=/root/.config/matplotlib
DEBUG:matplotlib:interactive is False
DEBUG:matplotlib:platform is linux
Process Process-1:
Traceback (most recent call last):
File "/usr/local/envs/rvc/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/local/envs/rvc/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/content/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 268, in run
train_and_evaluate(
File "/content/Retrieval-based-Voice-Conversion-WebUI/infer/modules/train/train.py", line 545, in train_and_evaluate
"slice/mel_org": utils.plot_spectrogram_to_numpy(
File "/content/Retrieval-based-Voice-Conversion-WebUI/infer/lib/train/utils.py", line 239, in plot_spectrogram_to_numpy
data = data.reshape(fig.canvas.get_width_height()[::-1] + (3,))
ValueError: cannot reshape array of size 800000 into shape (200,1000,3)

fixed:
/content/Retrieval-based-Voice-Conversion-WebUI/infer/lib/train/utils.py
plot_spectrogram_to_numpy
revise to:
def plot_spectrogram_to_numpy(spectrogram):
global MATPLOTLIB_FLAG
if not MATPLOTLIB_FLAG:
import matplotlib
matplotlib.use('Agg')
MATPLOTLIB_FLAG = True
import matplotlib.pyplot as plt
else:
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(10, 2))
im = ax.imshow(spectrogram, aspect="auto", origin="lower", interpolation="none")
plt.colorbar(im, ax=ax)
plt.xlabel("Frames")
plt.ylabel("Channels")
plt.tight_layout()

fig.canvas.draw()
data = np.frombuffer(fig.canvas.buffer_rgba(), dtype=np.uint8)
width, height = fig.canvas.get_width_height()
data = data.reshape(height, width, 4)
data = data[:,:,:3]
plt.close()
return data


T1.1 About subpage
 template design

Design for this page is here: http://iati.surge.sh/page-templates/01-1-about-subpage
The model for the page would be something like:

title
content_editor

The members assembly block in the sidebar is an optional item, the model for which should be something like:

block_title
block_content
block_button_text
block_button_url

From Trello card: https://trello.com/c/9lCcuVpq/82-t11-about-subpage-template-design

@dalepotter @allthatilk @akmiller01 we now have guidance on the submenu design when it's at a couple of layers deep: http://iati.surge.sh/page-templates/01-1b-about-subpage-3-levels

Now implemented. Closing issue.


Fix categorical partitioning

Ensure categories are properly re-encoded
Fix sharding issue with categoricals all mapping to same partition

Fixes #1200.

I'm going to merge for now. If/when we bump the dependency on partd to the latest release, we can simplify the categorical handling here.


Fixed upload structure. Source and test folder handling now works

I have made modifications to the uploading logic. Source folders are now treated properly as before, test folders are skipped unless they are explicitly selected for the upload, only their contents get uploaded though, when they are selected - this might need to be fine-tuned according to your vision.
I have tested this and it works for me, I haven't been able to find a bug in this.

Unfortunately, does not work normally with a project with several source roots, ex layout-b

Okay, I can look into that, look at the included test layouts and see if I can get it working changing some code

Please try the latest commit.

I am working now on adding some tests for this functionality. When it's done, I'll have a look

Closing this PR as the changes have been consolidated into #347 due to being too intertwined with ones from other PRs.


Don't Round Shipping If It's BTC

When editing a listing, and your currency is in BTC, the price for the shipping is always rounded up to the second decimal when it is saved.
If the currency is BTC, the shipping should be able to be set to the satoshi.

This is just a confusing UX, not a bug. Changing your currency to BTC doesn't change the currency of the listing to BTC, so if it's in fiat it remains in fiat, but the user may not notice if they are only editing the shipping price (which doesn't show the currency).


Unable to load addon: Blender v 2.68a Win 7 sp1 x64

First some background: Originally, PRMan failed to load b/c it couldn't find an RMSTREE.  I did not originally install a PRMan20 Studio client since I don't have Maya.  I re-installed and told it to install the Maya 2016 PRManStudio and that resolved the first error I was getting.
Now I'm stuck; it fails to import the prman_for_python_3 .pyd.  It's unclear to me why..????   Here is the stack I'm seeing:

I fixed the bug where it was failing to load for not having RMSTREE.  In fact you shouldn't have to set that.
For the above issue.  Do you have RMANTREE variable set and RenderMan Pro Server installed?

Also do you have your pythonpath variable set to something?

Yeah...I saw your recent change to fix the RMSTREE issue :-)
RMANTREE is set to C:\Program Files\Pixar\RenderManProServer-20.0\  (which is correct install location)
I didn't have an environment variable for pythonpath.  So I created one and set it to %RMANTREE%bin\

Just to be clear you don't HAVE to set either of those variables, It should find them on their own as long as they're installed in the normal place.  Are you still getting the error?
I think this might be a python version issue.  What python version does 2.68 have? I haven't tested with that, only 2.74+.  You can find that by running the help->system info menu

Blender 2.68 used Python 3.3, if I remember correctly.  It switched to Python 3.4 around 2.72 or so.

Yes.  That's what I'm seeing.  2.68 is using Python 3.3

Well then... I'd guess the prman_for_python module would not load then.  It's compiled against 3.4 python.  The blender version in the bl_info of the addon doesn't limit where you can load the addon?

Just downloading latest 2.75 release and will retry.

Nope unfortunately, it's basically a reference for the user to tell them which Blender version it was developed for and tested against.

Working in 2.75 as expected.


MLDB-1721 pathelements quoted

This patch clarifies what is allowed inside a path element (all characters apart from nulls) and ensures that paths that are written can always be read.  This should fix problems with MLDB accepting control characters in paths on entry, but then failing when processing or printing them.

+1


About conformer cache mechanism

Hi,
I find that the conformer encoder does not support cache mechanism for inference.
I think it's helpful, so I just want to give a suggestion here. : )
Regards

@JianyuanZhong, any insight on that?

Hi @Speakin-sy, thank you for raising this issue!
As far as I understand, the cache mechanism for the transformer/conformer is to cache to input feature for t -1 steps and only compute feature for the t step when being used for auto-regressive inference.
TBH, I don't know how much time it can save, but it will take some space to store caches for each layer. For example, the most time-consuming parts of ASR beam-search IMO are the CTC-prefix-scorer and index-slicing. But we basically have to store features from 18 transformer layers (AM + LM) with caches.
I think we hope to decode our ASR models with a single GPU to speed up the decoding at the current stage. However, I am not sure if we can do that with the cache mechanism.
We also have predefined interfaces for decoding with transformers if your question is related to inference without such a cache mechanism.
Let me know if you still would like to see this feature!

Hi @JianyuanZhong , thank you for your reply~
I agree that CTC and auto-regression decoding calculation are more time consuming. And cache mechanism can only reduce the time that the data pass through transformer/conformer structures. Maybe it can be added in the future~
Regards


TLS Handshake error in kube-rbac-proxy-main container

What did you do?
I installed the the kube-prometheus bundle on my k8s cluster
What did you expect to see?
Metrics being pulled from kube-state-metrics
What did you see instead? Under which circumstances?
Metrics are not provided by kube-state-metrics pod
Environment

Prometheus Operator version:

3267a403726436175becc6d3adc41611dbad0e55

Kubernetes version information:

v1.10.11

Kubernetes cluster kind:

kops

Manifests:
https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus/manifests

Prometheus Operator Logs:

level=info ts=2019-03-20T13:23:28.372176118Z caller=operator.go:970 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2019-03-20T13:23:38.377126618Z caller=operator.go:970 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2019-03-20T13:23:38.640354225Z caller=operator.go:970 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2019-03-20T13:23:39.375069141Z caller=operator.go:970 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2019-03-20T13:23:48.645172118Z caller=operator.go:970 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2019-03-20T13:23:48.982030889Z caller=operator.go:970 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
level=info ts=2019-03-20T13:23:54.379331505Z caller=operator.go:970 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s

Kube State Metrics Logs:
kube-rbac-proxy-mainn

I0320 13:21:20.965129       1 main.go:213] Generating self signed cert as no cert is provided
I0320 13:22:06.364455       1 main.go:243] Starting TCP socket on :8443
I0320 13:22:06.364624       1 main.go:250] Listening securely on :8443
2019/03/20 13:22:20 http: TLS handshake error from <IP_ADDRESS>:33994: tls: first record does not look like a TLS handshake
2019/03/20 13:22:35 http: TLS handshake error from <IP_ADDRESS>:34152: tls: first record does not look like a TLS handshake
2019/03/20 13:22:50 http: TLS handshake error from <IP_ADDRESS>:34332: tls: first record does not look like a TLS handshake

Am I correct assuming this keeps on logging?
2019/03/20 13:22:35 http: TLS handshake error from <IP_ADDRESS>:34152: tls: first record does not look like a TLS handshake

@metalmatze Absolutely correct.

this looks very iffy indeed. @slashr Do you know where did the IP <IP_ADDRESS> come from and from which client? This rather looks like a bin address rather than a valid client IP. This message usually indicates, that someone initiated a non-tls communication, i.e. a http client.

@s-urbaniak I'm not sure. I do have Datadog running on my cluster and I'm trying to make it talk to the kube-state-metrics pod inside the monitoring namespace.
In fact, the troubleshooting of Datadog lead me to inspect the logs of kube-state-metrics and then I found this.

Seems like the incoming connections are indeed from the Datadog pods. I'm in touch with Datadog customer support to fix this. But in case this is a problem on the kube-rbac-proxy-main end please do hint me :)

Closing due to inactivity.

Having the exact same thing with kube-prometheus trying to reach kube-rbac-proxy
TLS handshake error from <IP_ADDRESS>:52888: tls: first record does not look like a TLS handshake


parallelize zar zq

Most of the zar operations can be parallelized to concurrently perform against the data or index files in the archive. Let's start with zar zq; this work can be the basis for parallelizing the rest of the operations.

@alfred-landrum: I know we'd done some parallelizing work lately, so with zar commit 56ed80d I just ingested the wrccdc "year1" data set to make a 4.5 GB archive, then did:
$ zar zq -t 'count()'
#0:record[count:uint64]
0:[96217189;]

My per-second CPU chart showed zar utilizing about 8 of the 12 cores on my laptop.

Can we say we've addressed this issue and can close it?

@philrz : yes, the final enabling of zar zq parallel operations landed in #1138 , closing.


drivers: Add cellular API for network configuration

Adds two APIs which allow for configuring the cellular network configuration of a cellular network device. like a cellular modem. The first allows for configuring which access technology to use, and optionally, which bands to use. The second allows for getting all supported access technologies are supported, and which bands for each tech are supported.
The specified radio access technologies are incomplete, and I think adding a "MAX" at the end to allow for vendor specific technologies may be appropriate, but that can be added any time in the future as well

Hello, I'd like to work on this PR, I need to be able to poll information from the cellular modem like rssi, imei and imsi.
Do you mind if I use your changes ?

Was added in PR #65685 :)


Spire Agent Pods Fail to Deploy

When trying to deploy the service mesh, my deployment hangs at initializing the spire agent pods.
Finally gives up and NGINX rolls back
K8s version v1.20.1
running k8s on Ubuntu 18.04.5 LTS EC2 AWS instances
1 control
2 workers
Nodes are in a healthy state
Installed nginx-meshctl via the download via git
ubuntu@k8s-control:~$ kubectl get pods -n nginx-mesh
NAME                            READY   STATUS                  RESTARTS   AGE
spire-agent-kxdnr               0/1     Init:CrashLoopBackOff   6          12m
spire-agent-pgxz8               0/1     Init:CrashLoopBackOff   6          12m
spire-server-6cfc7df7d7-ftkjx   2/2     Running                 0          12m
ubuntu@k8s-control:~$ kubectl logs spire-agent-7w9c9 -n nginx-mesh
Error from server (BadRequest): container "spire-agent" in pod "spire-agent-7w9c9" is waiting to start: PodInitializing
ubuntu@k8s-control:~$ kubectl describe pod spire-agent-7w9c9 -n nginx-mesh
Name:         spire-agent-7w9c9
Namespace:    nginx-mesh
Priority:     0
Node:         k8s-worker2/<IP_ADDRESS>
Start Time:   Wed, 02 Jun 2021 23:47:48 +0000
Labels:       app.kubernetes.io/name=spire-agent
app.kubernetes.io/part-of=nginx-service-mesh
controller-revision-hash=5c6d478476
pod-template-generation=1
Annotations:
Status:       Pending
IP:           <IP_ADDRESS>
IPs:
IP:           <IP_ADDRESS>
Controlled By:  DaemonSet/spire-agent
Init Containers:
init:
Container ID:  containerd://4e8c126bab2bb00647679094b6e9083aed177509a20039b972a2e3d6b7e40ba9
Image:         gcr.io/spiffe-io/wait-for-it
Image ID:      gcr.io/spiffe-io/wait-for-it@sha256:d9bdc931e4404237d2fb0ba84db5ece88b236c40eeca570d786ee54fd243f4ae
Port:
Host Port:
Args:
-t
30
spire-server:8081
State:          Waiting
Reason:       CrashLoopBackOff
Last State:     Terminated
Reason:       Error
Exit Code:    143
Started:      Wed, 02 Jun 2021 23:56:28 +0000
Finished:     Wed, 02 Jun 2021 23:56:58 +0000
Ready:          False
Restart Count:  6
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from spire-agent-token-94prq (ro)
Containers:
spire-agent:
Container ID:
Image:         gcr.io/spiffe-io/spire-agent:0.12.1
Image ID:
Port:
Host Port:
Args:
-config
/run/spire/config/agent.conf
State:          Waiting
Reason:       PodInitializing
Ready:          False
Restart Count:  0
Liveness:       exec [/opt/spire/bin/spire-agent healthcheck -shallow -socketPath /run/spire/sockets/agent.sock] delay=15s timeout=3s period=60s #success=1 #failure=2
Readiness:      exec [/opt/spire/bin/spire-agent healthcheck -socketPath /run/spire/sockets/agent.sock] delay=5s timeout=1s period=5s #success=1 #failure=3
Environment:
MY_NODE_NAME:   (v1:spec.nodeName)
Mounts:
/run/spire/bundle from spire-bundle (rw)
/run/spire/config from spire-config (ro)
/run/spire/sockets from spire-agent-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from spire-agent-token-94prq (ro)
/var/run/secrets/tokens from spire-token (rw)
Conditions:
Type              Status
Initialized       False
Ready             False
ContainersReady   False
PodScheduled      True
Volumes:
spire-config:
Type:      ConfigMap (a volume populated by a ConfigMap)
Name:      spire-agent
Optional:  false
spire-bundle:
Type:      ConfigMap (a volume populated by a ConfigMap)
Name:      spire-bundle
Optional:  false
spire-agent-socket:
Type:          HostPath (bare host directory volume)
Path:          /run/spire/sockets
HostPathType:  DirectoryOrCreate
spire-token:
Type:                    Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds:  7200
spire-agent-token-94prq:
Type:        Secret (a volume populated by a Secret)
SecretName:  spire-agent-token-94prq
Optional:    false
QoS Class:       BestEffort
Node-Selectors:
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type     Reason     Age                     From               Message

Normal   Scheduled  9m50s                   default-scheduler  Successfully assigned nginx-mesh/spire-agent-7w9c9 to k8s-worker2
Normal   Pulled     9m50s                   kubelet            Successfully pulled image "gcr.io/spiffe-io/wait-for-it" in 232.655347ms
Normal   Pulled     9m19s                   kubelet            Successfully pulled image "gcr.io/spiffe-io/wait-for-it" in 235.545751ms
Normal   Pulled     8m36s                   kubelet            Successfully pulled image "gcr.io/spiffe-io/wait-for-it" in 228.685843ms
Normal   Started    7m39s (x4 over 9m50s)   kubelet            Started container init
Normal   Pulled     7m39s                   kubelet            Successfully pulled image "gcr.io/spiffe-io/wait-for-it" in 245.295299ms
Normal   Created    6m28s (x5 over 9m50s)   kubelet            Created container init
Normal   Pulling    6m28s (x5 over 9m51s)   kubelet            Pulling image "gcr.io/spiffe-io/wait-for-it"
Normal   Pulled     6m28s                   kubelet            Successfully pulled image "gcr.io/spiffe-io/wait-for-it" in 234.332423ms
Warning  BackOff    4m45s (x13 over 8m48s)  kubelet            Back-off restarting failed container

Did you enable Service Account Token Volume Projection? https://docs.nginx.com/nginx-service-mesh/get-started/platform/kubeadm/

I already had these configured in my  /etc/kubernetes/manifests/kube-apiserver.yaml
Am I suppose to have these configured elsewhere?
- --service-account-key-file=/etc/kubernetes/pki/sa.pub
- --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
- --service-account-api-audiences=api
- --service-account-issuer=api

Thanks for checking.  Did your k8s API server restart?

I agree, ive been investigating the network errors but no luck yet.
The API server restarted when I added the additional commands for token volume projection.
ubuntu@k8s-control:~$ kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-7f4f5bf95d-t5ccp   1/1     Running   7          3d3h
calico-node-sfvlf                          0/1     Running   1          3d3h
calico-node-trkz4                          0/1     Running   1          3d2h
calico-node-vwwgb                          0/1     Running   1          3d2h
coredns-74ff55c5b-sgvhx                    1/1     Running   1          3d3h
coredns-74ff55c5b-zsr5h                    1/1     Running   1          3d3h
etcd-k8s-control                           1/1     Running   1          3d3h
kube-apiserver-k8s-control                 1/1     Running   0          12h
kube-controller-manager-k8s-control        1/1     Running   6          3d3h
kube-proxy-hszlt                           1/1     Running   1          3d3h
kube-proxy-qcbbr                           1/1     Running   1          3d2h
kube-proxy-vk4hs                           1/1     Running   1          3d2h
kube-scheduler-k8s-control                 1/1     Running   6          3d3h

Is it possible there is a security groups or acl setting that is block communication?

This is a fairly new cluster, i checked for any network policies I may have deployed and there are none.
ubuntu@k8s-control:~$ kubectl get networkpolicy --all-namespaces
No resources found
I will continue to investigate the network error.  If I cant find the cause, then ill probably tear the cluster down and rebuild, its possible something got misconfigured.

ill close this out if I end up rebuilding

Ok, found the networking issue. I was missing the 443 port in my AWS security group.  In the Kubeadm guide it mentions 6443 but no mention of 443 , only that this port can be overridden.  After changing my SG to allow all, NGINX service mesh installed.  I should have noticed the 443 earlier in the logs but got tunnel vision looking at the k8s deployment.
Ill close this one out......sigh
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

closing this issue out as it was related to a networking issue in AWS

Thanks for following up @therealnoof


RetryWithError is not integrated with LCEL.

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code
This is the way it should work natually but as RetryOutputParser is outdated we can not use this code.
parser = PydanticOutputParser()
fix_parser = RetryOutputParser.from_llm(parser=parser, llm=ChatOpenAI())
structured_llm = ai_model | fix_parser
work_chain = template_obj  | structured_llm
 work_chain = work_chain.with_retry(stop_after_attempt=3) # type: ignore
 # invoke_result: P = await work_chain.ainvoke(input_dict) # type: ignore
  invoke_result: P = await  work_chain.ainvoke(input_dict) # type: ignore

Error Message and Stack Trace (if applicable)
No response
Description
The RetryWithError output parser is not integrated  with LCEL. I would like to use it with the with_retry method but it is clearly outdated.
In my opinion it should have the parse_result method.
System Info
langchain in the latest version

Can you explain what doesn't work? What do you expect to see vs. what do you actually see?

@math-sasso you can try with_retry and RetryOutputParser as follow:
parser = PydanticOutputParser()
retry_parser = RetryOutputParser.from_llm(parser=parser, llm=llm)
work_chain = template_obj  | structured_llm
final_chain =  RunnableParallel(
    completion=work_chain, prompt_value=template_obj
) | RunnableLambda(lambda x :{'completion':x['completion'].content,
                              'prompt_value':x['prompt_value']}) | RunnableLambda(lambda x: retry_parser.parse_with_prompt(**x))
 work_chain = work_chain.with_retry(stop_after_attempt=3,retry_if_exception_type=(OutputParserException,)) # type: ignore
 # invoke_result: P = await work_chain.ainvoke(input_dict) # type: ignore
  invoke_result: P = await  work_chain.ainvoke(input_dict) # type: ignore


Run unittests in parallel

Closes: #DNE
Reintroduction of #4112, with following issues fixed:

invoke unittest-parallel returned exit status 0 even when tests fails.
There was a clash between workers for fixture file, each worker now uses separate fixture file.
Better tests distribution to run each worker single group only.
Relativise tests times to avoid unnecessary changes to repo.

What's Changed
Added invoke unittest-parallel, which runs unit tests in parallel with results written to .test-results directory.
Added invoke sum-tests-times to update file for distributing parallel tests.
Added invoke ps to display docker compose services.
Added invoke export-docker-compose to export docker compose configuration to compose.yaml file.
Added invoke logs to display docker compose logs.
Added default_exec group_index fixture_file arguments to invoke unittest.
Added --fixture-file argument to Nautobot test runner.
Changed --label argument for invoke unittest commands to be iterable.
Changed class NautobotTestRunner to inherit from XMLTestRunner, exporting tests results to NAUTOBOT_TEST_OUTPUT_DIR environment variable.
Considerations

Non-durable Django Postgres Testing.

@bryanculver Could you pls. test this. After success I'll make this PR ready for review.

This is believed redundant after #5130 and #5163.


feat!(updates.jenkins.io) use new statically provisioned PVC and stop using mirrorbits-parent chart

Related to https://github.com/jenkins-infra/helpdesk/issues/2649#issuecomment-2364054995
This PR introduces the following breaking changes in publick8s's updates-jenkins-io namespace:

Use the statically provisioned PVCs for mirrorbits (PVCs introduced in https://github.com/jenkins-infra/azure/pull/845)
Stop using the chart mirrorbits-parent for updates.jenkins.io (the "secured" original component) and use mirrorbits instead

Since we introduces the "redirects" standalone httpd , the parent chart does not need httpd anymore
Rsync is still defined, but should disappear once mirrorbits will support scanning through S3 protocol (this rsync is only a hack)
The recent support of "addition ingresses" remove the need to have an "umbrella" ingress as each ingress will serve the children's chart backend
Using the new static PVCs  removes the need to have an "umbrella storage"

Initial deployment failed. Fix opened in https://github.com/jenkins-infra/azure/pull/849 .
Note: this error could only be seen in production, hence the self merge

Second deployment failed: the unsecured release had to be uninstalled with
$ helm -n updates-jenkins-io uninstall updates-jenkins-io-unsecured
release "updates-jenkins-io-unsecured" uninstalled

This PR leds the mirrors.updates.jenkins.io service to answer HTTP/404 to all requests.
Using the mirrorbits SVC (with a kubectl port-forward) immediatly pointed fingers at the ingresses.
Fix in https://github.com/jenkins-infra/kubernetes-management/pull/5752


[Compile] fix compile fail on clang

Proposed changes
These compile error come from pr
#10843
#10355
#10937
Problem Summary:
Describe the overview of changes.
Checklist(Required)

Does it affect the original behavior: (No)
Has unit tests been added: (No Need)
Has document been added or modified: (No Need)
Does it need to update dependencies: (No)
Are there any changes that cannot be rolled back: (No)

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

Thanks, LGTM.


Converting to a GitHub organisation

Hey,
I've decided to consider moving all Pronto related repositories to an organisation. Before I decide to pull the trigger, I'd like to ask for some input. I personally think that it would help organise, maintain and contribute in a better way ✊.
First thing that I've checked:

[x] Github will automatically redirect the old personal repo to the new org repo.

@mknapik, @doomspork, @jeroenj, @aergonaut, @nicka, @andrey-skat, @nbekirov, @gabealmer: all of you have contributed significant chunks of code to Pronto, so your input would be especially valuable 🙇.

Howdy @mmozuras!  I see no problems with switching to an organization so long as you're happy to do so 😀
Let me know if there's anything I can do to help.

Yep I see no problems doing this!

Same here, please go ahead 👍

Sounds great to me. As long as you're okay with it. :)

That sounds like a great idea. 👍

Thanks for support! 🙂
And here are some initial ideas for the logo: 1, 2, 3, 4. Drawn up by a good friend. Which do you like? Which conveys Pronto's idea in the best way? Any other ideas on what could be Pronto's logo?

@mmozuras I like #2.   I'll ponder logo ideas over the next few days.  Thanks again for including all of us in the decision making process 😁

Yeah, I like 2 the most. Conveys the theme of speed the best in my opinion.

2nd

I like the first one the most. But I really like the second one too. :)

I really like #2

I'll use that drawing of 2 as a temporary avatar until we'll get a proper version of it 😄.
Next problem: pronto is taken. pronto-org? Not a fan. Anyone has better ideas? 🙂

Ideas:

prontissimo
pronto-rb
pronto-labs

I wasn't invited and I don't have any contribution yet (I would like to ☺ ), but I would vote for the 2nd logo.
pronto-rb would be nice organization name.

I prefer getpronto, pronto-labs, or pronto-org.  Albeit a Ruby gem, given that a lot of the runners are not Ruby, I would shy away from pronto-rb or pronto-gem.

@doomspork good point about a lot of the runners not being Ruby related 👍.
@aergonaut @doomspork where does pronto-labs idea come from? Is Redis Labs the inspiration? Or are there more labs out there? 😄

@mmozuras my inspiration came from @aergonaut's suggestion 😁

Just thinking of other cool-sounding suffixes 😃  I think I like getpronto more than pronto-labs, personally.

I like pronto-org and pronto-labs so far the most. It would also be nice to have the domain available 😄.

@mmozuras these are available:

pronto-labs.com
pronto-labs.io
prontolabs.io
pronto-org.com
pronto-org.io

👍  for pronto-labs and pronto-labs.io

@doomspork @aergonaut what are your thoughts and arguments on prontolabs & prontolabs.io vs. pronto-labs & pronto-labs.io?

@mmozuras no strong feelings, which do you prefer?

@mmozuras I'd go with pronto-labs & pronto-labs.io.
Have you tried contacting the user who has the pronto username?

@jeroenj I haven't, but I'll try 👍

In case he's willing to give it up I'd go with getpronto.io. If not I'd stick with pronto-labs. :)

I went with http://prontolabs.io (set up as redirect) and https://github.com/prontolabs. There's no wrong answer 😄. I think that no dash makes it easier to describe in real life. I also like the consistency with other Labs, like https://github.com/postmanlabs/, https://github.com/awslabs, https://github.com/RedisLabs, which all chose to not have a dash.
I added the second sketch as a temporary logo and moved all my personal repositories to the org.
@aergonaut thanks for the labs idea! And thanks everyone for your thoughts and feedback 👍.
Next two steps:

[ ] figure out the principles for inviting people to the org.
[ ] figure out the principles for adding pronto related libs/gems/runners (and their owners) to the org.

Thoughts/feedback on those two? Any examples of how other orgs do it? 🙂

Should you change the attribution in the copyright and LICENSE to something like "Pronto Contributors"?

figure out the principles for inviting people to the org.

I don't know how Rails/Ember/etc. decide who is added to the core teams, but I think in general members of the org should have a commitment to triaging incoming issues and reviewing Pull Requests.

figure out the principles for adding pronto related libs/gems/runners (and their owners) to the org.

I think 3rd party runners can stay separate from the org, but you could link them from the list of runners.

I tend to agree with @aergonaut 👍

figure out the principles for inviting people to the org.

This is always tough and I've tried different things with the couple of the orgs I'm in.  I personally feel a person needs to demonstrate a commitment to continued participation and support of Pronto; there's a lot of folks out there looking to "collect them all" with regards to org badges on their profile.
As people are added to the organization it'll be important to agree on a process of reviewing, merging, and releasing new features that keeps the overall vision intact; its easy to merge everything that comes through without giving consideration to it's impact on the long term project health and growth.
@mmozuras I believe you should always retained the right to veto features and be the deciding vote in the event of a disagreement amongst contributors.

figure out the principles for adding pronto related libs/gems/runners (and their owners) to the org.

With Ueberauth we only keep packages maintained by the core team in the organization and rely on a Wiki page for third parties.  If someone no longer intends to maintain their project you could certainly take over ownership of it under the umbrella of the organization but I would be hesitant to take on the maintenance burden of them all.

@aergonaut @doomspork thanks, very valuable thoughts. I'll make a suggestion on principles this week 🙂
I've worked with a designer on a logo. Here's the result:

Any thoughts on which one (or ones?) to use?

Oh wow, all very nice. For the organization's avatar image, I like the orange and white "P" on the black background.

Really cool! I like 2nd line from the bottom (with orange bars). They can be used as long and short version.

I agree with @aergonaut & @ivanovaleksey.  The singular white P, orange lines, and blackground really pops.
The rest are awesome too, kudos to the designer!

I like those ones the most too. I guess a white background with a black P with orange lines would be an option too for the avatar?

@aergonaut @ivanovaleksey @doomspork @jeroenj thanks for your valuable opinions! I've now changed the avatar to a new beautiful one. I'll give the compliments to the designer 🙇.

@mmozuras you can probably close this now 😀

@doomspork soon 😄
I've added welcome repo to serve as the central hub for org related things: https://github.com/prontolabs/welcome.

Looking good @mmozuras 👍


Intermittent Connection Issues with NetBird 0.28.9

Describe the problem
NetBird intermittently stops working for some users. When clicking the "Connect" button, nothing happens. This issue affects around 10 out of 40 users. Running the netbird down command results in an error (screenshot attached). Restarting the laptop temporarily resolves the issue.
To Reproduce
Launch NetBird on the client machine.
Click the "Connect" button.
Notice that the connection does not occur.
Run the netbird down command and observe the error.
Expected behavior
The connection should be established successfully after clicking "Connect" without the need to restart the device.
Are you using NetBird Cloud?
No, we are using a self-hosted version of NetBird.
NetBird version
NetBird version: 0.28.9
Do you face any (non-mobile) client issues?
The issue is occurring only on laptops; other devices are unaffected.
Additional context
The problem is only resolved by rebooting the affected laptops.

Here are some additional NetBird logs, though they are from a different user. The issue seems to be similar, but the logs are not directly related to the screenshot provided earlier.
netbird.err.log
netbird.out.log

same issue https://github.com/netbirdio/netbird/issues/2613
closing this one


what about pointerlock control when running on desktop

currently when running on desktop, the controls is using VRControls.js
it is fine but require click to move orientation
got the usual limit at the border

Why not using pointerlock ? it seems more natural desktop

We had this at one point. I'm open to adding it.

excelent, i will see how to include it and do a pull request

duplicate from #123


Merge assignment 1 into assignment 2

To fulfill this requirement for assignment 2: (b) In this part, you will enhance Question Repository created in Assignment 1 by designing APIs for the CRUD operations and maintaining question records in a NoSQL database such as MongoDB., which is currently done in assignment 1 repo.

Should merge assignment-1 to dev branch first, then from dev to assignment-2

Can merge once PR #53 is completed


Feature request - make Attack and Damage buttons appear side-by-side

Instead of the default layout...

Make them look like this instead...

This is easily done by adding flexrow to the card-buttons class...

Though this doesn't look great when you add a saving throw and an "other" roll into the mix...

Pretty sure that's why I didn't go for this. Especially when couples with MRE and items with multiple damage groups.
Could try something like flex wrap, with a width on the buttons of either 50% or max-content.


Allow clients to provide fine-grained diagnostics

This adds new delegate methods for "missing inputs" and "multiple producers for a single node" errors. These provide context to the client now instead of constructing strings.
See rdar://problem/34333034

@swift-ci please smoke test

@swift-ci please smoke test

@swift-ci please smoke test


New version: Microsoft.Teams version <IP_ADDRESS>78

Pull request has been created with Komac v1.11.0 :rocket:
Microsoft Reviewers: Open in CodeFlow

@stephengillie this seems to flagging duplicate but because of the preview version. Does that mean it's ok?

It appears that the Preview version has become the Release version.

It appears that the Preview version has become the Release version.

Thanks mate!


North Pole International not selectable

In the all stations announcement preset, for phil at least, the destination plays as North Pole International, however it is not visible or selectable.

Related to #184


AppEngine flex samples run locally on windows

The samples for netcore on app engine flex all show running the apps in visual studio on a local windows machine. When the apps are deployed they are run on a debian8 linux container.
Although it is improving, the current state of cross platform support in netcore is not 100%. I've run into many issues moving code from windows to linux and had to implement workarounds. It would be helpful to show a sample of doing local development on a windows pc using the same linux containers that the app will be deployed under in the flex app engine.
This is relatively easy now with Docker for Windows and Docker tools for VS. If desired I can update a repo and submit a pull request.

I've hit issues moving code from windows to linux too.  Such a demo would definitely be valuable to users.
A pull request would be awesome!
Please note, all contributors to this repo must accept the Google Contributer License Agreement here:
https://cla.developers.google.com/clas
It's only one page and not too intimidating.  So please have a look at it before beginning a pull request.

Hi @SurferJeffAtGoogle
Thanks for sharing this .NET Vision API samples. I have issues setting up when I got to the ADC. Kindly help because I couldn't run my code.
Thanks


Some cleaning up of the swagger doc and corresponding code

General cleaning up and error fixing in swagger doc
Remove some not likely used endpoints.
For stage model add notified_at and name attributes.
Add workflows/workflow_id/groups endpoint

@gmcculloug can we merge this?


Operator Filter Registry Initializer Not Working When Upgrading

How can I upgrade my NFT smart contract to implement Operator Filter Registry? I know there's they release an upgradeable OFR, but it requires to be initialized. I'm just wondering how are you going to initialize it when you already deployed the smart contract and you're just upgrading it?

The initialize methods are meant to configure the contract to use the registry, but they don't set any contract state that is not configurable outside of initialization.
If you are upgrading your contract to use the registry, the owner() of the contract can register on behalf of the upgradeable contract by interacting with the Operator Filter Registry directly, or the contract itself can include a one-time method to register itself as it would in the constructor: https://github.com/ProjectOpenSea/operator-filter-registry/blob/main/src/upgradeable/OperatorFiltererUpgradeable.sol#L23


Update .NET PowerShell Kernel to PowerShell 7 RC2

Is your feature request related to a problem? Please describe.
No
Describe the solution you'd like
Just a question...
I notice that the .NET PowerShell kernel is PowerShell 7 RC 1. Is there a way to update to RC 2.
Or, I'm guessing that you all waiting for the next release to GA.
But I would love to know is there's a way to get RC 2 going while waiting for the GA version.
Thanks!

Duplicate of #168


WSA crashed at startup in windows 10 v19045.2364

Hello
basically I dont understand step one :
Get WSA appx zip. You can do this by following instructions in https://github.com/LSPosed/MagiskOnWSALocal (You need to "build" this yourself with your local WSL2).
and because of that I got WSA MSIX from https://store.rg-adguard.net  and then extract it with 7zip and install it
I copied your provided patched dlls to WSAClient dir but WSA does not work and crashed at startup
can you upload and share your own built WSA appx zip file ? I can not "build" it myself with my local WSL2
thanks for your effort

2210.40000.7.0

Try run Add-AppxPackage -ForceApplicationShutdown -ForceUpdateFromAnyVersion -Register .\AppxManifest.xml in PowerShell (run as admin) in your WSA installation directory and check whether there is any error message. If any error exists, follow the instruction (run Get-AppPackageLog -ActivityID <UUID> as prompted) to view the detail error logs.

still does not work the setting window is crashing please share your WSA dir as .zip

Thanks issue solved


Remove non-default locales from the repo

We should remove all the locale files from the repo and and them to the .gitignore, and update the docs and run scripts to fetch them on postinstall.
Having generated artefacts in a repo always leads to trouble eventually, and in this case it sets us up to waste the time of people who volunteer to fix translation issues.
See  #950

@olizilla  removing locales from repo will introduce various issues. I am against mainly for two reasons:

Builds are no longer reproducible in offline mode  + require access to the internet and Transifex API

Third party web service can break our builds
Also, I am not sure if you can fetch locales without account on Transifex and API key in ~/.transifexrc, which complicates CI and local setups for drive-by contributors

We will be getting an amazing improvement for translations soon, but it needs locales to be present in repos (added you to email thread with Transifex, tl;dr is in https://github.com/ipfs/i18n/issues/5)

To avoid people submitting PRs with translations, we could:

put copy from ipfs/i18n#how-can-i-contribute-translation-for-my-language  in public/locales/README.md  or public/locales/READ_THIS_BEFORE_TRANSLATING.md
create PR template with checklist and add an item "I am aware that translations should be submited via https://www.transifex.com/ipfs/public/ instead of creating a PR"

Also, I suggest we always close PRs that  bypass Transifex (close them without merging).
Gently thank for translation and ask them to apply changes via Transifex or just link to  ipfs/i18n#how-can-i-contribute-translation-for-my-language

Example: https://github.com/ipfs-shipyard/ipfs-companion/pull/658
This creates a paper trail that future translators can discover while doing research before submiting PR. So not only informs the author of the current PR, but also people that could do similar thing in the future, but look for examples of previous PRs before creating their own.


LLM engine: add timeout

If a job has been waiting for too long, just remove it from the queue

Done by https://github.com/Endle/fireSeqSearch/pull/154


fixfix merge project with window function bug

Proposed changes
Issue Number: close #xxx
Problem summary

don't merge projects if any window function exists
bypass SimplifyArithmeticRule for decimalV3 type

Checklist(Required)

[ ] Does it affect the original behavior
[ ] Has unit tests been added
[ ] Has document been added or modified
[ ] Does it need to update dependencies
[ ] Is this PR support rollback (If NO, please explain WHY)

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall


feat(details-section):allow collapsibility

add initialCollapsed, collapsible, id, hasLoaded props

LGTM

you should add a section that is loading in the demo too. also make sure you squash everything into a single commit (everything looks good)

:+1:


High memory consumption during pdfSimpleExport to file, possible memory leak

Describe the bug
I am using FastReport.OpenSource 2023.2.3 in my C# .NET project. This project is supposed to load template .frx file from disk and export report to pdf and save it on the disk.
`                report.Prepare();
            PDFSimpleExport pdf = new PDFSimpleExport();
            Guid g = Guid.NewGuid();
            var filepath = Path.Combine(this._pathToTmp, g.ToString() + ".pdf");
            report.Export(pdf, filepath);`

When I want to generate pdf document with size around 300MB (In fact I need to generate a lot bigger reports, like 3 GB) my application is consuming around 12,5GB of memory and after saving report to the disk memory is not lowering.
To Reproduce
Just prepare template, load a lof data i.e. from the database and use PDFSimpleExport to export it to the file.
Expected behavior
Free used memory and ofc do not use so much memory during the pdf export.
Stacktrace
Screenshots
Device (please complete the following information):

OS: Linux in Docker, dotnet image https://hub.docker.com/_/microsoft-dotnet-aspnet/
Browser none
Version 2023.2.3

Additional context
Is there anything I forgot to implement after exporting PDF file? Should I somehow clear the report object?
Is commercial .NET version of fast-report handling such issues better? I would buy .NET Fast-report If I would have guarantee there will be no such errors.

Hi!
I have the same problem, except with a small report (~2MB).
My application is also running on Docker, using the same image and FastReport 2023.2.17.
After generating 20 reports, the application consumes 1.5GB of memory and does not release it. I noticed that this only occurs on Linux. When running locally on Windows, memory consumption doesn't go over 600MB.
To support FastReport on this docker image we added this libs:
libgif-dev autoconf libtool automake build-essential gettext libglib2.0-dev libcairo2-dev libtiff-dev libexif-dev libpango1.0-dev
and instaled libgdiplus from source

Hi! I have the same problem, except with a small report (~2MB). My application is also running on Docker, using the same image and FastReport 2023.2.17. After generating 20 reports, the application consumes 1.5GB of memory and does not release it. I noticed that this only occurs on Linux. When running locally on Windows, memory consumption doesn't go over 600MB.
To support FastReport on this docker image we added this libs: libgif-dev autoconf libtool automake build-essential gettext libglib2.0-dev libcairo2-dev libtiff-dev libexif-dev libpango1.0-dev and instaled libgdiplus from source

Hello,
recently I have found that there might a memory leak or something and I have to download libgdi source and compile it by myself:
https://github.com/FastReports/FastReport/issues/316#issuecomment-974017793
Now my container consume around 500MB of RAM and the memory is being freed after the job.


Fixed note count on first click

1. Why is this change necessary?
First click on mark product in note returns count 0, because response sUniqueID cookie cannot be found since 5.3.0 RC1.
SW-18393 added support for Cookies for paths and this broke that functionality (https://github.com/shopware/shopware/commit/e01d43cfc84b66f59c28540b3de3157f745996a2#diff-8ee74466675cdfac64b39296cf3020ccR127),
https://github.com/shopware/shopware/blob/5.3/engine/Shopware/Core/sBasket.php#L1330 does not contain "-/" at end of the key for the path.
2. What does this change do, exactly?
Loop throught all cookies to find the right one without using the key.
3. Describe each step to reproduce the issue or behaviour.
Clear all Cookies, mark a product as note and you get 0 as count
4. Please link to the relevant issues (if any).
5. Which documentation changes (if any) need to be made because of this PR?
6. Checklist

[ ] I have written tests and verified that they fail without my change
[X] I have squashed any insignificant commits
[] This change has comments for package types, values, functions, and non-obvious lines of code
[X] I have read the contribution requirements and fulfil them.

Fixed with https://github.com/shopware/shopware/commit/40ec126c20d2b2e7177acfc3569aa1e62ffa9efd


Part 1 migration of standard bundle

Part milestone for #699
:point_up: diff excluding whitespace to avoid noise.
Noteworthy changes to keep in mind for doing the code review:

Migrated the core files used used in standard bundle (+related vendor files)
Wrapped classes/objects to make implicit and varying dependencies explicit, to be wired up in the bundle entry file. To give example of how it actually works see this small fictional example:

// my-model.js
var Backbone = require('Backbone'); // doesn't vary so require here
module.exports = function(opts) {
  var ajax = opts.$ajax || opts.reqwest;
  var MyModel = Backbone.Model.extend(…); // uses ajax for something in the model
  return MyModel;
};

// standard.js
var setupMyModel = require('./my-model')
var MyModel = setupMyModel(window.$.ajax); // assumes jquery being loaded already

// core.js
var reqwest = require('reqwest');
var setupMyModel = require('./my-model');
var MyModel = setupMyModel(reqwest);

This has several benefits which are important for upcoming changes:

Gives an easy overview of what dependencies there are
Will make it easier to refactor/untangle the code since the wiring of dependencies are done in a single place
standard bundle is essentially the same as for the no-jquery and no-leaflet too (with the obvious differences the bundle names implies), so this approach will make it trivial to implement those bundles later

Also:

pay attention to the // TODOs, added them for things I wanted to highlight in particular

@xavijam pretty please review :kiss:

@alonsogarciapablo, take a look too because it touches layer definition.

For me looks ok!


Update DeveloperGuide.md

Remove [Proposed] from implementation of undo/redo
Rename fields/methods to current implementation
Credit original proposed implementation used in AB-3
Update UC for redo

Honestly no clue why codecov is failing when no changes were made to functional code


React Native requires Babel devDependencies

Hey, thanks for this lib! I'm excited to start building with it — on first glance it looks like exactly what I'm after 😀.
I'm using React Native and noticed that I needed to install the following devDependencies in order to get react-redux-form to compile:
    "babel-core": "^6.3.17",
    "babel-preset-es2015": "^6.3.13",
    "babel-preset-react": "^6.3.13",
    "babel-preset-stage-0": "^6.3.13",

This is my first foray into RN, so I'm not sure how its Babel compiler works in cases like these. Should this be a necessary step for using react-redux-form with RN, or is there a better way I'm overlooking?

Interesting, that might be a React Native issue rather than a React-Redux-Form issue, since RRF exposes /lib/index.js, which is already pretranspiled into ES5 code, so Babel isn't even necessary.
Let me know if you have any more issues with React Native and I'll respond as soon as I can!

@davidkpiano Got the same issue, don't know why. I removed the .babelrc from the installed module and it works now. It's not really a solution, but maybe you could exclude it with your npm-ignore from your package like this: https://github.com/kadirahq/npm-base/blob/master/.npmignore?

Sure thing, @MiMaMuh. Doing that now.

@MiMaMuh @davidkpiano thanks for revisiting this! I did a little more research, and the root of it seems to be RN's move to Babel 6. Here's a thread with more info:  https://github.com/facebook/react-native/issues/4062
The team's official fix is just as @MiMaMuh suggested.

Should be available in 0.9.11 now! @jonlong @MiMaMuh


Resolves issue with special characters in dir

Proposing this as a fix for Issue 317 where spaces, ( and ) are not escaped appropriately. In addition, move from a regex expression to a normalize() command to get path structures functional cross platform. Requesting windows user support in confirming normalize() still allows file paths to properly escape

-Does this work for Windows users?
-Are there any other special characters that should be considered for escaping? Currently supported: spaces, (, )

I think you're a little confused about that normalize() method.
Looks like you think you're using path.normalize(), but you're actually using String.prototype.normalize().

You're dang straight I'm a little confused! 😆
This is my first node contribution, so I was not aware of the chaining abilities and yes it was meant to be called on the path, not the string.
According to commit notes, we don't need this due to the last commit covering Windows users.
Normalize was being called on the string instead of the path. It was introduced to remove replace(/\\/g, ‘\\\\’), which we have determined is still required.

Hey @JeffreyWay - What's the next steps for getting this merged?

I'm hesitant to merge this because I don't have any tests to confirm that it does in fact work.

The support thread includes a few different platforms confirming fix, but I'm more hesitant to merge because I feel this might be more of a webpack level issue and shouldn't be fixed at each plugin level. Shouldn't their file management portion be able to escape characters? They already do for some, just doesn't seem like it does a perfect job yet.


Bug: READONLY not being applied through Designer

Describe the bug
When working on a schemaful table structure (i have yet to test schemaless as i dont use it in my database currently), i create a new field, and hit read only toggle, which according to the documentation should apply a READONLY tag on the field.

I hit apply, save etc. But when i go back into the designer view, and it's unticked again.

Looking up the INFO FOR TABLE data, this also shows no current definition of READONLY on the field.

This appears to happen regardless of the type field.
Steps to reproduce
Create table, or edit table.
Create field, or edit field.
Set the read only toggle.
Save and apply.
Open the field again. Check INFO FOR TABLE query.
Expected behaviour
READONLY to be applied when read only is ticked. If additional fields need to be altered for this to function, such was read / write perms, this should correct, highlight or warn if these fields are not in the right state for read only to be applied.
Surrealist Environment
surreallist 3.0.3
Contact Details
<EMAIL_ADDRESS>Is there an existing issue for this?

[X] I have searched the existing issues

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Found this to be a general functionality of the tooling in general.
When the fields don't exist in the database, but have READONLY ahead of, or behind the PERMISSIONS FULL definiton, it also fails to apply it.

But manually running

the query, in either position, applies it, and then designer picks up that it exists.

So this will be affecting importing data too, to correct assign READONLY on fields on fresh databases @macjuul

This also happens to ALL fields when editing the table in designer.
If i edit one field to remove READONLY, all fields no longer have READONLY.


Analyzer: high fraction of time in 	PackageUriResolver.resolveAbsolute

When running analyzer on a large app with 3330 packages, I see that
13% of the time is spent in PackageUriResolver.resolveAbsolute.
Half of that time is in JavaFile.getAbsolutePath which seems to be inefficient at string level, but I can't help wondering if the real problem is that PackageUriResolver.resolveAbsolute is called too many times.
I believe @munificent noticed the same.

https://codereview.chromium.org/1817423002 avoids some of the computations.
There are still many more calls than expected, but this helps.

We've profiled several times since this issue was filed, so I assume we now longer see these hot spots.


Trainer instantiation leads to hanging Work with fork warnings

Bug description
Running a work with a Trainer inside hangs
How to reproduce the bug
import lightning as L

class Finetune(L.LightningWork):
    def run(self):
        L.Trainer()
        print("finished")

app = L.LightningApp(Finetune())

Error messages and logs
adrian@quantumgrid ~/r/Finetune-miniLM (main)> lightning run app app.py                                                                                (mini-lm)
Your Lightning App is starting. This won't take long.
INFO: Your app has started. View it in your browser: http://<IP_ADDRESS>:7501/view
/Users/adrian/miniconda3/envs/mini-lm/lib/python3.10/site-packages/lightning/app/core/app.py:490: UserWarning: One of your Flows returned `.ready` as `False`. This feature is not yet enabled so this will be ignored.
  warnings.warn(
objc[11096]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[11096]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

Environment

Current environment
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow): Work
#- PyTorch Lightning Version (e.g., 1.5.0): x
#- Lightning App Version (e.g., 0.5.2): 1.8.5 and master
#- PyTorch Version (e.g., 1.10): 1.13
#- Python version (e.g., 3.9): 3.10
#- OS (e.g., Linux): MacOS
#- CUDA/cuDNN version: x
#- GPU models and configuration: M1
#- How you installed Lightning(`conda`, `pip`, source): pip install -e .
#- Running environment of LightningApp (e.g. local, cloud): local

More info
Possibly related to #15699
cc @tchaton @justusschock @awaelchli @borda

This demonstrates how processes are created under the hood of Lightning App, shows the same error:
import multiprocessing

import torch.backends.mps

def main():
    print("forked")
    torch.backends.mps.is_available()

if __name__ == "__main__":
    start_method = "fork"
    context = multiprocessing.get_context(start_method)
    process = context.Process(target=main)
    process.start()

Note that the issue disappears with start_method="spawn". However, the reason we are using fork (I'm not saying this is how it should be) in LightningApp is because spawn is much slower.
In order to override this behavior in an app, you can set
class YourWork(LightningWork):
    _start_method = "spawn"  # set this attribute

    def run(self):
        ...

Then process for this work will spawn instead of fork.

I would not be opposed to that. I opened a PR to see what the impact is.

After switching, we get one failing test on ubuntu.
Test runtime on master:
Ubuntu ~6 min, Windows 15 min, MacOS ~13 min
Test runtime after switching to spawn:
All are >= 15 mins

Yes, that s great 😃