关于用自己的数据集训练

你们好，首先非常感谢你们的工作！我想请问如果要尝试在自己的数据集上进行训练的话，需要改那些文件？我发现你们还没有提供对自己数据集的支持，但是非常想尝试用你们的模型试着跑跑实验看看分割效果
感谢：）

请问相关环境需要装的库/包可以分享一下吗 感谢！

Hi @xqzeng13 ,
You do not need to change it. DDP is also supported in single card scenario. You can run the training script as follows:
CUDA_VISIBLE_DEVICES='0' torchrun --nproc_per_node=1 train.py --mixed --benchmark --task $TASK --exp_name $EXP_NAME --wandb --entity $USER_NAME

Hi @youngprogrammerBee , Thank you for your interest in our work. Actually, training on customized dataset will NOT include additional Python libraries. A brief introduction of customized data training is listed below:

Prepare your data into NIFTI format (or other format that MONAI library could support, please refer to their official documentation for details);
Split your training and validation data under the root path of your data storage. The expected format of data storage is listed below:

- data_root
    - train_images
    - train_labels
    - val_images
    - val_labels

It should be noted that training images should have labels (which means that some labeled images should be regarded as unlabeled, in order to simulate the semi-supervised setting on LA or Pancreas dataset)

If you are training on a real-world semi-supervised setting (some images are labeled while others are not), you need to prepare your labeled data and unlabeled data separately:

- labeled_root
    - train_images
    - train_labels
    - val_images
    - val_labels
- unlabeled_root
    - train_images
    - train_labels (an empty folder)

Then you need to modify the train.py file as below:
data_pipeline = TrainValDataPipeline(image_root, 'labeled', label_ratio=1.0, random_seed=seed)
unlabeled_pipeline = TrainValDataPipeline(unlabeled_root, 'unlabeled', label_ratio=1.0, random_seed=seed)
trainset, _, valset = data_pipeline.get_dataset(train_aug, val_aug, cache_dataset=False)
unlabeled_set, _, _ = unlabeled_pipeline.get_dataset(train_aug, val_aug, cache_dataset=False)
train_sampler = DistributedSampler(trainset, shuffle=True)
unlabeled_sampler = DistributedSampler(unlabeled_set, shuffle=True)
val_sampler = DistributedSampler(valset)

By defining two data_pipeline instances, you could generate the corresponding training dataset, unlabeled dataset and test dataset, respectively. Then the training will be identical to LA or Pancreas dataset.

Support for 2D training is under development.

Please note that the training hyperparameter may need sophisticated finetuning on customized dataset, and the performance of our method can NOT be guaranteed. Thanks again for your interest. Please feel free to ask if you have further problems.

您好，感谢回复！我按照您说的准备了数据集（选择了第三条，即有unlabeled_root），在训练时遇到以下问题想请问 ：
1.我的数据集都为nii.gz格式 这个应该是可以的吧
2.请问使用自己的数据集进行训练，观察您所给出的需要改的代码，其中的image_root来源于 prepare_experiment函数，但是该函数并没有返回unlabeled_root的路径，我应该按照该函数已有任务（la 或者Pancreas）中的来吗，还是要自己定义新的任务，如果需要自己定义任务，请问又该如何定义？

现在我的文件路径图如下

其中semi_label对应 labeled_root
即- labeled_root
- train_images
- train_labels
- val_images
- val_labels
semi_unlabel对应unlabeled_root
即- unlabeled_root
- train_images
- train_labels (an empty folder)

Hi @youngprogrammerBee ,

1.我的数据集都为nii.gz格式 这个应该是可以的吧

NIFTI is supported, that's fine.

请问使用自己的数据集进行训练，观察您所给出的需要改的代码，其中的image_root来源于 prepare_experiment函数，但是该函数并没有返回unlabeled_root的路径，我应该按照该函数已有任务（la 或者Pancreas）中的来吗，还是要自己定义新的任务，如果需要自己定义任务，请问又该如何定义？

You can simply pass the labeled_root and unlabeled_root, without calling prepare_experiment function.
We introduce prepare_experiment function just to simplify the train.py, as we do not want to set a fixed argument in the main training file.
You could either define a new prepare_experiment function which suits your case, or you could simply hardcode all the arguments needed in the main training file.

@youngprogrammerBee
你好你好，请问你最后在自己的数据集上运行成功了吗？我也在做这个方面的研究，方便互相交流一下吗


python3Packages.fiona: skip fewer tests

Motivation for this change
https://github.com/Toblerity/Fiona/issues/1012 was resolved
Things done

Built on platform(s)

[x] x86_64-linux
[x] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 161601 run on x86_64-linux 1

8 packages built:

python310Packages.cartopy
python310Packages.fiona
python310Packages.geopandas
python310Packages.osmnx
python39Packages.cartopy
python39Packages.fiona
python39Packages.geopandas
python39Packages.osmnx


Allow for specifying --project-cache-dir in gradle.properties file

Expected Behavior
I don't want to have duplicated .gradle directories, so I usually run:
gradle --project-cache-dir ./android/.gradle :tasks --all

but I'd prefer to specify it in some gradle.properties file.
Current Behavior (optional)
No response
Context
This is a very small feature that would make my work a tiny bit easier.
See also:

How to specify project cache dir from a gradle config

Thank you for your interest in Gradle!
Have you tried using org.gradle.projectcachedir in the properties file?
Even if it works, this should be documented, will requalify the issue as a documentation one.

Thanks for response @jbartok. Unfortunately that didn't work.
$ cat gradle.properties
org.gradle.projectcachedir=./android/.gradle

But the .gradle directory is still created in my $PWD instead of $PWD/android.

Hmm... just tried it (with Gradle 8.3-rc-4) and works as you would expect for me... What Gradle version are you using?

7.5

You are correct; it indeed doesn't work correctly in Gradle 7.5. The smallest Gradle version I could find, where it does work correctly is 8.2.1. Is upgrading to that version an option for you?

Yeah, it's not a problem. For no apparent reason, I'm getting OutOfMemory with 8.2.1:
$ ./gradlew :android:app:assembleDebug
> Task :android:app:packageDebug FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':android:app:packageDebug'.
> A failure occurred while executing com.android.build.gradle.tasks.PackageAndroidArtifact$IncrementalSplitterRunnable
   > java.lang.OutOfMemoryError (no error message)

Thank you for support, I'll debug it and get back once I manage to fix it.

Okay, it works. OOM disappeared.
Thank you!

I'll leave it up to your whether to keep it open, because docs could surely be better :)

Fixed in https://github.com/gradle/gradle/pull/25724.


Change default cython flags to have smaller compiled files

Pip install results in a much larger lib artefact compared to conda install (both in a conda environment):

pip install pydantic: 80M
conda install pydantic -c conda-forge: 6.6M

I don't know if this is expected or not. I encountered this while trying to minimize the size of a Docker container I'm building, and was surprised that pydantic took up 80M when installed with pip.
I wasn't sure to file this as bug or not, but given the extreme difference in size I thought there might be something going wrong with the pip install.
I've added a full list of the files in site-packages:
Pip install file sizes
> du -sh /opt/conda/lib/python3.8/site-packages/pydantic/* | sort -h
0	/opt/conda/lib/python3.8/site-packages/pydantic/py.typed
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/__init__.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/json.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/parse.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/tools.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/version.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/datetime_parse.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/env_settings.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/error_wrappers.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/generics.py
12K	/opt/conda/lib/python3.8/site-packages/pydantic/dataclasses.py
12K	/opt/conda/lib/python3.8/site-packages/pydantic/decorator.py
12K	/opt/conda/lib/python3.8/site-packages/pydantic/typing.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/class_validators.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/errors.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/networks.py
20K	/opt/conda/lib/python3.8/site-packages/pydantic/color.py
20K	/opt/conda/lib/python3.8/site-packages/pydantic/validators.py
24K	/opt/conda/lib/python3.8/site-packages/pydantic/utils.py
28K	/opt/conda/lib/python3.8/site-packages/pydantic/mypy.py
28K	/opt/conda/lib/python3.8/site-packages/pydantic/types.py
32K	/opt/conda/lib/python3.8/site-packages/pydantic/fields.py
40K	/opt/conda/lib/python3.8/site-packages/pydantic/main.py
40K	/opt/conda/lib/python3.8/site-packages/pydantic/schema.py
196K	/opt/conda/lib/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so
332K	/opt/conda/lib/python3.8/site-packages/pydantic/__pycache__
516K	/opt/conda/lib/python3.8/site-packages/pydantic/parse.cpython-38-x86_64-linux-gnu.so
552K	/opt/conda/lib/python3.8/site-packages/pydantic/version.cpython-38-x86_64-linux-gnu.so
684K	/opt/conda/lib/python3.8/site-packages/pydantic/tools.cpython-38-x86_64-linux-gnu.so
948K	/opt/conda/lib/python3.8/site-packages/pydantic/json.cpython-38-x86_64-linux-gnu.so
1.4M	/opt/conda/lib/python3.8/site-packages/pydantic/datetime_parse.cpython-38-x86_64-linux-gnu.so
1.5M	/opt/conda/lib/python3.8/site-packages/pydantic/error_wrappers.cpython-38-x86_64-linux-gnu.so
1.6M	/opt/conda/lib/python3.8/site-packages/pydantic/env_settings.cpython-38-x86_64-linux-gnu.so
1.7M	/opt/conda/lib/python3.8/site-packages/pydantic/dataclasses.cpython-38-x86_64-linux-gnu.so
2.0M	/opt/conda/lib/python3.8/site-packages/pydantic/typing.cpython-38-x86_64-linux-gnu.so
2.1M	/opt/conda/lib/python3.8/site-packages/pydantic/decorator.cpython-38-x86_64-linux-gnu.so
2.7M	/opt/conda/lib/python3.8/site-packages/pydantic/class_validators.cpython-38-x86_64-linux-gnu.so
3.4M	/opt/conda/lib/python3.8/site-packages/pydantic/color.cpython-38-x86_64-linux-gnu.so
4.0M	/opt/conda/lib/python3.8/site-packages/pydantic/networks.cpython-38-x86_64-linux-gnu.so
5.1M	/opt/conda/lib/python3.8/site-packages/pydantic/validators.cpython-38-x86_64-linux-gnu.so
5.2M	/opt/conda/lib/python3.8/site-packages/pydantic/utils.cpython-38-x86_64-linux-gnu.so
6.2M	/opt/conda/lib/python3.8/site-packages/pydantic/mypy.cpython-38-x86_64-linux-gnu.so
6.4M	/opt/conda/lib/python3.8/site-packages/pydantic/errors.cpython-38-x86_64-linux-gnu.so
6.4M	/opt/conda/lib/python3.8/site-packages/pydantic/fields.cpython-38-x86_64-linux-gnu.so
8.0M	/opt/conda/lib/python3.8/site-packages/pydantic/schema.cpython-38-x86_64-linux-gnu.so
8.5M	/opt/conda/lib/python3.8/site-packages/pydantic/main.cpython-38-x86_64-linux-gnu.so
11M	/opt/conda/lib/python3.8/site-packages/pydantic/types.cpython-38-x86_64-linux-gnu.so

Conda install file sizes:
> du -sh /opt/conda/lib/python3.8/site-packages/pydantic/* | sort -h
0	/opt/conda/lib/python3.8/site-packages/pydantic/py.typed
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/__init__.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/json.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/parse.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/tools.py
4.0K	/opt/conda/lib/python3.8/site-packages/pydantic/version.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/dataclasses.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/datetime_parse.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/decorator.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/env_settings.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/error_wrappers.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/generics.py
8.0K	/opt/conda/lib/python3.8/site-packages/pydantic/typing.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/class_validators.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/errors.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/networks.py
16K	/opt/conda/lib/python3.8/site-packages/pydantic/validators.py
20K	/opt/conda/lib/python3.8/site-packages/pydantic/color.py
20K	/opt/conda/lib/python3.8/site-packages/pydantic/utils.py
28K	/opt/conda/lib/python3.8/site-packages/pydantic/mypy.py
28K	/opt/conda/lib/python3.8/site-packages/pydantic/types.py
32K	/opt/conda/lib/python3.8/site-packages/pydantic/fields.py
36K	/opt/conda/lib/python3.8/site-packages/pydantic/main.py
36K	/opt/conda/lib/python3.8/site-packages/pydantic/schema.py
48K	/opt/conda/lib/python3.8/site-packages/pydantic/__init__.cpython-38-x86_64-linux-gnu.so
80K	/opt/conda/lib/python3.8/site-packages/pydantic/tools.cpython-38-x86_64-linux-gnu.so
84K	/opt/conda/lib/python3.8/site-packages/pydantic/parse.cpython-38-x86_64-linux-gnu.so
88K	/opt/conda/lib/python3.8/site-packages/pydantic/version.cpython-38-x86_64-linux-gnu.so
116K	/opt/conda/lib/python3.8/site-packages/pydantic/json.cpython-38-x86_64-linux-gnu.so
144K	/opt/conda/lib/python3.8/site-packages/pydantic/datetime_parse.cpython-38-x86_64-linux-gnu.so
156K	/opt/conda/lib/python3.8/site-packages/pydantic/env_settings.cpython-38-x86_64-linux-gnu.so
164K	/opt/conda/lib/python3.8/site-packages/pydantic/dataclasses.cpython-38-x86_64-linux-gnu.so
188K	/opt/conda/lib/python3.8/site-packages/pydantic/error_wrappers.cpython-38-x86_64-linux-gnu.so
188K	/opt/conda/lib/python3.8/site-packages/pydantic/typing.cpython-38-x86_64-linux-gnu.so
192K	/opt/conda/lib/python3.8/site-packages/pydantic/decorator.cpython-38-x86_64-linux-gnu.so
280K	/opt/conda/lib/python3.8/site-packages/pydantic/errors.cpython-38-x86_64-linux-gnu.so
296K	/opt/conda/lib/python3.8/site-packages/pydantic/class_validators.cpython-38-x86_64-linux-gnu.so
308K	/opt/conda/lib/python3.8/site-packages/pydantic/networks.cpython-38-x86_64-linux-gnu.so
324K	/opt/conda/lib/python3.8/site-packages/pydantic/__pycache__
356K	/opt/conda/lib/python3.8/site-packages/pydantic/color.cpython-38-x86_64-linux-gnu.so
360K	/opt/conda/lib/python3.8/site-packages/pydantic/utils.cpython-38-x86_64-linux-gnu.so
452K	/opt/conda/lib/python3.8/site-packages/pydantic/fields.cpython-38-x86_64-linux-gnu.so
456K	/opt/conda/lib/python3.8/site-packages/pydantic/validators.cpython-38-x86_64-linux-gnu.so
472K	/opt/conda/lib/python3.8/site-packages/pydantic/mypy.cpython-38-x86_64-linux-gnu.so
480K	/opt/conda/lib/python3.8/site-packages/pydantic/types.cpython-38-x86_64-linux-gnu.so
572K	/opt/conda/lib/python3.8/site-packages/pydantic/main.cpython-38-x86_64-linux-gnu.so
572K	/opt/conda/lib/python3.8/site-packages/pydantic/schema.cpython-38-x86_64-linux-gnu.so

Bug
Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())" from conda install:
root@c6b3b43340a6:/# python -c "import pydantic.utils; print(pydantic.utils.version_info())"
             pydantic version: 1.6.1
            pydantic compiled: True
                 install path: /opt/conda/lib/python3.8/site-packages/pydantic
               python version: 3.8.5 (default, Sep  4 2020, 07:30:14)  [GCC 7.3.0]
                     platform: Linux-5.8.0-38-generic-x86_64-with-glibc2.10
     optional deps. installed: []

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())" from pip install:
root@c6b3b43340a6:/# python -c "import pydantic.utils; print(pydantic.utils.version_info())"
             pydantic version: 1.7.3
            pydantic compiled: True
                 install path: /opt/conda/lib/python3.8/site-packages/pydantic
               python version: 3.8.5 (default, Sep  4 2020, 07:30:14)  [GCC 7.3.0]
                     platform: Linux-5.8.0-38-generic-x86_64-with-glibc2.10
     optional deps. installed: []

conda info:
     active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /root/.condarc
 populated config files : /root/.condarc
          conda version : 4.9.2
    conda-build version : not installed
         python version : 3.8.5.final.0
       virtual packages : __glibc=2.28=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /opt/conda  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/conda/pkgs
                          /root/.conda/pkgs
       envs directories : /opt/conda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.9.2 requests/2.25.1 CPython/3.8.5 Linux/5.8.0-38-generic debian/10 glibc/2.28
                UID:GID : 0:0
             netrc file : None
           offline mode : False

I noticed there is a version difference, however, even with conda install pydantic=1.7.3 -c conda-forge I only get a 7.0M artefact.

I noticed there is a version difference, however, even with conda install pydantic=1.7.3 -c conda-forge I only get a 7.0M artefact.

Hello @peterroelants and thanks for reporting.
Your issue comes from compiled files with cython (the .so files).
Can you remove everything and try again with SKIP_CYTHON=1 pip install pydantic?

Hello @peterroelants and thanks for reporting.
Your issue comes from compiled files with cython (the .so files).
Can you remove everything and try again with SKIP_CYTHON=1 pip install pydantic?

I uninstalled the conda pydantic and installed it again with SKIP_CYTHON=1 pip install pydantic. After pip install I get a 80M large /opt/conda/lib/python3.8/site-packages/pydantic/ folder. The files seem to be the same as the one I posted in "Pip install file sizes" above.
Could it be that cython is not skipped?

I uninstalled the conda pydantic and installed it again with SKIP_CYTHON=1 pip install pydantic. After pip install I get a 80M large /opt/conda/lib/python3.8/site-packages/pydantic/ folder. The files seem to be the same as the one I posted in "Pip install file sizes" above.
Could it be that cython is not skipped?

Note that the files in the conda build are the same, it also contains the .so files, they are just of smaller size.
What is the sequence of commands I need to run to build the repo?

Note that the files in the conda build are the same, it also contains the .so files, they are just of smaller size.
What is the sequence of commands I need to run to build the repo?

You might be right that it's related to the cython flags, see this comment: https://github.com/cython/cython/issues/2102#issuecomment-397755796

You might be right that it's related to the cython flags, see this comment: https://github.com/cython/cython/issues/2102#issuecomment-397755796

I finally got my hands on a ubuntu and tested with default CFLAGS and -Os -g0.
(env) ~/pydantic-test/pydantic$ du -sh build
13M	build
(env) ~/pydantic-test/pydantic$ du -sh build-original
74M	build-original

You can try it out yourself by just cloning it, change in setup.py -> os.environ['CFLAGS'] = '-Os -g0'
and run python setup.py build_ext

I finally got my hands on a ubuntu and tested with default CFLAGS and -Os -g0.
(env) ~/pydantic-test/pydantic$ du -sh build
13M	build
(env) ~/pydantic-test/pydantic$ du -sh build-original
74M	build-original

You can try it out yourself by just cloning it, change in setup.py -> os.environ['CFLAGS'] = '-Os -g0'
and run python setup.py build_ext

Thanks for verifying.
I see that -Os optimizes for size. From the gcc docs: "-Os enables all -O2 optimizations except those that often increase code size. -g0 just seems to turn of debugging.
However, I haven't been able to reproduce your build results. python setup.py build_ext doesn't produce a build folder.
When using python setup.py build install && pip install . I'm only gettiung a 712K folder for pydantic in site-packages.

Thanks for verifying.
I see that -Os optimizes for size. From the gcc docs: "-Os enables all -O2 optimizations except those that often increase code size. -g0 just seems to turn of debugging.
However, I haven't been able to reproduce your build results. python setup.py build_ext doesn't produce a build folder.
When using python setup.py build install && pip install . I'm only gettiung a 712K folder for pydantic in site-packages.

This is currently in "feedback wanted" state. What kind of feedback is wanted?
The big binaries are problematic enough in that they warranted a special warning here: https://awslabs.github.io/aws-lambda-powertools-python/utilities/parser/
Even if the optimized binaries were 2x faster, they wouldn't be worth the size increase in modern workloads (lambdas, containers etc) - and people needing to extract the last bit of performance should be able to compile their own version of pydantic

This has already been discussed here: https://github.com/samuelcolvin/pydantic/issues/1325.  The .so files were an issue in AWS Lambda already back then but now the situation is even worse as the files have grown a lot. (I actually already hit the limits in one of my AWS Lambda deployments and after investigating why the package was so big I found this discussion.) In order FastAPI to become the next recommended framework for serverless APIs this major dependency should be more lightweight (i.e. smaller). Installing packages straight from Git repository doesn't sound right.

@samuelcolvin I've run some tests and these are the results:

size
benchmark avg time

default pip install
45M
70.7μs

pip install --no-binary
796K
101.4μs

pip install --no-binary with cython
6.4M
67.8μs

These are run on the validation benchmark task.
What is strange to me is that "pip install --no-binary with cython" is much than "default pip install ". I would have expected  them to have the same size.

"default pip install " is the regular pip install.
pip install --no-binary  is the no-binary option you suggested without Cython
The installation for "pip install --no-binary with cython" is:

# Setup build and other requirements
pip install --no-cache-dir \
    Cython==0.29.22 \
    devtools==0.6.1 \
    pip \
    python-dateutil \
    setuptools
# Install PyDantic
pip install \
    --no-cache-dir \
    --no-binary pydantic \
    --global-option=build_ext \
    --global-option=-j8 \
    pydantic==1.8.1

Additionally I also updated the documentation and suggested a small change to setup.py to allow for custom CFLAGS passed by the user: https://github.com/samuelcolvin/pydantic/pull/2517

@peterroelants how did you manage to have a 6.4M installation package with cython? I did exactly the same and got 40M.

@alexanderluiscampino That's weird, I hope I didn't make a mistake.
I ran my tests on a python:3.9.2-slim-buster Docker image with a Python env in /venv. The size should be the output from du -sh /venv/lib/python3.9/site-packages/pydantic

Hi @peterroelants  here are my size results, I tested every possible form in this thread, always get 40M
there results below are for the command:
CFLAGS="-Os -g0 -s" pip install --no-binary pydantic --global-option=build_ext --global-option=-j8 pydantic

suggested by you. But whichever way I go about it, always get 40M. I'm running on Docker as well,
FROM mcr.microsoft.com/vscode/devcontainers/python:3.8

$ du -sh ./src/python/pydantic/* | sort -h
0       ./src/python/pydantic/py.typed
1.0K    ./src/python/pydantic/version.py
2.0K    ./src/python/pydantic/parse.py
2.5K    ./src/python/pydantic/__init__.py
2.5K    ./src/python/pydantic/annotated_types.py
2.5K    ./src/python/pydantic/tools.py
3.5K    ./src/python/pydantic/json.py
5.0K    ./src/python/pydantic/error_wrappers.py
7.5K    ./src/python/pydantic/datetime_parse.py
9.0K    ./src/python/pydantic/env_settings.py
9.5K    ./src/python/pydantic/dataclasses.py
10K     ./src/python/pydantic/decorator.py
12K     ./src/python/pydantic/generics.py
13K     ./src/python/pydantic/typing.py
14K     ./src/python/pydantic/_hypothesis_plugin.py
14K     ./src/python/pydantic/class_validators.py
15K     ./src/python/pydantic/networks.py
16K     ./src/python/pydantic/errors.py
17K     ./src/python/pydantic/color.py
20K     ./src/python/pydantic/validators.py
22K     ./src/python/pydantic/utils.py
28K     ./src/python/pydantic/mypy.py
29K     ./src/python/pydantic/types.py
38K     ./src/python/pydantic/fields.py
41K     ./src/python/pydantic/main.py
42K     ./src/python/pydantic/schema.py
142K    ./src/python/pydantic/__init__.cpython-39-x86_64-linux-gnu.so
328K    ./src/python/pydantic/__pycache__
388K    ./src/python/pydantic/parse.cpython-39-x86_64-linux-gnu.so
415K    ./src/python/pydantic/tools.cpython-39-x86_64-linux-gnu.so
438K    ./src/python/pydantic/annotated_types.cpython-39-x86_64-linux-gnu.so
446K    ./src/python/pydantic/version.cpython-39-x86_64-linux-gnu.so
604K    ./src/python/pydantic/json.cpython-39-x86_64-linux-gnu.so
835K    ./src/python/pydantic/datetime_parse.cpython-39-x86_64-linux-gnu.so
954K    ./src/python/pydantic/error_wrappers.cpython-39-x86_64-linux-gnu.so
1.1M    ./src/python/pydantic/dataclasses.cpython-39-x86_64-linux-gnu.so
1.1M    ./src/python/pydantic/env_settings.cpython-39-x86_64-linux-gnu.so
1.2M    ./src/python/pydantic/decorator.cpython-39-x86_64-linux-gnu.so
1.3M    ./src/python/pydantic/_hypothesis_plugin.cpython-39-x86_64-linux-gnu.so
1.3M    ./src/python/pydantic/errors.cpython-39-x86_64-linux-gnu.so
1.3M    ./src/python/pydantic/typing.cpython-39-x86_64-linux-gnu.so
1.7M    ./src/python/pydantic/class_validators.cpython-39-x86_64-linux-gnu.so
1.8M    ./src/python/pydantic/color.cpython-39-x86_64-linux-gnu.so
1.8M    ./src/python/pydantic/networks.cpython-39-x86_64-linux-gnu.so
2.5M    ./src/python/pydantic/utils.cpython-39-x86_64-linux-gnu.so
2.9M    ./src/python/pydantic/mypy.cpython-39-x86_64-linux-gnu.so
3.0M    ./src/python/pydantic/validators.cpython-39-x86_64-linux-gnu.so
3.4M    ./src/python/pydantic/types.cpython-39-x86_64-linux-gnu.so
3.6M    ./src/python/pydantic/fields.cpython-39-x86_64-linux-gnu.so
3.9M    ./src/python/pydantic/main.cpython-39-x86_64-linux-gnu.so
4.0M    ./src/python/pydantic/schema.cpython-39-x86_64-linux-gnu.so

Weird, I remember checking the .so files, they were there, only much smaller than the ones you are getting.
Btw, the CFLAGS="-Os -g0 -s" options won't actually work until https://github.com/samuelcolvin/pydantic/pull/2517 is merged since Pydantic currently overwrites CFLAGS.
Let me try to reproduce my results, and maybe clean up the testing scripts for sharing. It will probably take me a few days to find the time.

@alexanderluiscampino I tried reproducing my results and I have a Dockerfile for you to try at https://gist.github.com/peterroelants/e344ac416948296f7fcdc84a20ce6eb5
For me this results in a working Python environment with Pydantic env of 6.4M:
docker run pydantic-test-env /bin/bash -c 'du -sh /usr/local/lib/python3.9/site-packages/pydantic'
6.4M	/usr/local/lib/python3.9/site-packages/pydantic

docker run pydantic-test-env /bin/bash -c 'du -sh /usr/local/lib/python3.9/site-packages/pydantic/* | sort -h'
0	/usr/local/lib/python3.9/site-packages/pydantic/py.typed
4.0K	/usr/local/lib/python3.9/site-packages/pydantic/__init__.py
4.0K	/usr/local/lib/python3.9/site-packages/pydantic/annotated_types.py
4.0K	/usr/local/lib/python3.9/site-packages/pydantic/json.py
4.0K	/usr/local/lib/python3.9/site-packages/pydantic/parse.py
4.0K	/usr/local/lib/python3.9/site-packages/pydantic/tools.py
4.0K	/usr/local/lib/python3.9/site-packages/pydantic/version.py
8.0K	/usr/local/lib/python3.9/site-packages/pydantic/datetime_parse.py
8.0K	/usr/local/lib/python3.9/site-packages/pydantic/error_wrappers.py
12K	/usr/local/lib/python3.9/site-packages/pydantic/dataclasses.py
12K	/usr/local/lib/python3.9/site-packages/pydantic/decorator.py
12K	/usr/local/lib/python3.9/site-packages/pydantic/env_settings.py
12K	/usr/local/lib/python3.9/site-packages/pydantic/generics.py
16K	/usr/local/lib/python3.9/site-packages/pydantic/_hypothesis_plugin.py
16K	/usr/local/lib/python3.9/site-packages/pydantic/class_validators.py
16K	/usr/local/lib/python3.9/site-packages/pydantic/errors.py
16K	/usr/local/lib/python3.9/site-packages/pydantic/networks.py
16K	/usr/local/lib/python3.9/site-packages/pydantic/typing.py
20K	/usr/local/lib/python3.9/site-packages/pydantic/color.py
20K	/usr/local/lib/python3.9/site-packages/pydantic/validators.py
24K	/usr/local/lib/python3.9/site-packages/pydantic/utils.py
28K	/usr/local/lib/python3.9/site-packages/pydantic/mypy.py
32K	/usr/local/lib/python3.9/site-packages/pydantic/types.py
36K	/usr/local/lib/python3.9/site-packages/pydantic/__init__.cpython-39-x86_64-linux-gnu.so
40K	/usr/local/lib/python3.9/site-packages/pydantic/fields.py
44K	/usr/local/lib/python3.9/site-packages/pydantic/main.py
44K	/usr/local/lib/python3.9/site-packages/pydantic/schema.py
68K	/usr/local/lib/python3.9/site-packages/pydantic/annotated_types.cpython-39-x86_64-linux-gnu.so
68K	/usr/local/lib/python3.9/site-packages/pydantic/parse.cpython-39-x86_64-linux-gnu.so
72K	/usr/local/lib/python3.9/site-packages/pydantic/tools.cpython-39-x86_64-linux-gnu.so
72K	/usr/local/lib/python3.9/site-packages/pydantic/version.cpython-39-x86_64-linux-gnu.so
96K	/usr/local/lib/python3.9/site-packages/pydantic/json.cpython-39-x86_64-linux-gnu.so
128K	/usr/local/lib/python3.9/site-packages/pydantic/datetime_parse.cpython-39-x86_64-linux-gnu.so
148K	/usr/local/lib/python3.9/site-packages/pydantic/error_wrappers.cpython-39-x86_64-linux-gnu.so
168K	/usr/local/lib/python3.9/site-packages/pydantic/env_settings.cpython-39-x86_64-linux-gnu.so
172K	/usr/local/lib/python3.9/site-packages/pydantic/dataclasses.cpython-39-x86_64-linux-gnu.so
180K	/usr/local/lib/python3.9/site-packages/pydantic/decorator.cpython-39-x86_64-linux-gnu.so
188K	/usr/local/lib/python3.9/site-packages/pydantic/typing.cpython-39-x86_64-linux-gnu.so
196K	/usr/local/lib/python3.9/site-packages/pydantic/_hypothesis_plugin.cpython-39-x86_64-linux-gnu.so
212K	/usr/local/lib/python3.9/site-packages/pydantic/errors.cpython-39-x86_64-linux-gnu.so
236K	/usr/local/lib/python3.9/site-packages/pydantic/class_validators.cpython-39-x86_64-linux-gnu.so
248K	/usr/local/lib/python3.9/site-packages/pydantic/networks.cpython-39-x86_64-linux-gnu.so
276K	/usr/local/lib/python3.9/site-packages/pydantic/color.cpython-39-x86_64-linux-gnu.so
320K	/usr/local/lib/python3.9/site-packages/pydantic/utils.cpython-39-x86_64-linux-gnu.so
372K	/usr/local/lib/python3.9/site-packages/pydantic/__pycache__
384K	/usr/local/lib/python3.9/site-packages/pydantic/mypy.cpython-39-x86_64-linux-gnu.so
388K	/usr/local/lib/python3.9/site-packages/pydantic/types.cpython-39-x86_64-linux-gnu.so
444K	/usr/local/lib/python3.9/site-packages/pydantic/validators.cpython-39-x86_64-linux-gnu.so
472K	/usr/local/lib/python3.9/site-packages/pydantic/fields.cpython-39-x86_64-linux-gnu.so
544K	/usr/local/lib/python3.9/site-packages/pydantic/schema.cpython-39-x86_64-linux-gnu.so
572K	/usr/local/lib/python3.9/site-packages/pydantic/main.cpython-39-x86_64-linux-gnu.so

Thank you for your dedication @peterroelants !
I managed to replicate your results using your dockerfile. I'm assuming the difference between your method and the mine, is that you do a clean installation of pip and cython from the beginning, whereas I already started with a pip installation. Somehow, when I ran the pydantic installation script (w/ cython) it wasn't using it, and defaulted to the normal way of installing it.
I'm still trying to get to the bottom of this, and will update here.

@dotorg-richard you mention you are running on a Macbook with M1 chip, this means that the base image pulled will be of a different architecture. The python:3.9.2-slim-buster base image should support arm64.
Can you try the following?:
Make sure you have the latest Docker version and fulfil the system requirements as documented in "Docker Desktop for Apple silicon".
And then build the exact same image again using docker buildx:
docker buildx build --progress=plain --target pydantic_env --tag pydantic-test-env --file ./Dockerfile .

I'm curious if this would solve your problem. If not, could you share the results of running the following on your M1 machine:
docker run python:3.9.2-slim-buster /bin/bash -c "dpkg --print-architecture && uname -m"

@peterroelants I was feeling confident realizing I was still running the older Docker Preview and not the latest release but after installing the latest release and following the directions to install I still get the exact same issue as above, failing in the exact same place.
The output you requested is:
arm64
aarch64

That's weird, I can build the same exact same Docker container I posted without any issues. And apparently so can @alexanderluiscampino?
Note from the output error: command '/usr/bin/gcc' failed with exit code 1 that gcc seems to be failing. I'm running Ubuntu 20.04 on an Intel chipset myself. I tried running the build against an emulated arm64 but I didn't run into any issues.
I'm wondering if the M1 chipset is the cause of your issue.

@peterroelants I can confirm, able to build it with the dockerfile provided, running on windows.

@dotorg-richard I'm curious if a different base image would resolve your issue.
Could you try building the following Alpine-based-image: https://gist.github.com/peterroelants/e344ac416948296f7fcdc84a20ce6eb5#file-python-alpine-dockerfile with:
docker buildx build --progress=plain --load --target pydantic_env --tag pydantic-test-env-alpine --file ./python.alpine.Dockerfile .

@peterroelants thank you so much I'm happy to report that worked! Hopefully others with the M1 chip can use this alternate base image. Package size report is 6.2MB!

That's great to hear!
Interesting that the Debian based image resulted in issues on the M1, while the Alpine based images work.
I also experimented with plain Ubuntu images and there I got a Pydantic install that was 8 times as large as the ones in the Debian or Alpine Python images.

To clarify a bit, the excessive size is almost entirely due to the debug info, which is usually included by default (see the output of e.g. python3.9-config --cflags). So it's the -g0 that does most of the "work" of the size reduction.
Perhaps given the large size overhead, pydantic could disable debug info by default? The debug info is useful for gdb debugging, crashes, benchmarking etc., but most library users probably won't have a use for it.
If disabling debuginfo entirely is not desirable, a middle ground can be to reduce the debug level from -g2 (the default) to -g1. Citing the gcc manpage:

Level 1 produces minimal information, enough for making backtraces in parts of the program that you don't plan to debug.  This includes descriptions of functions and external variables, and line number tables, but no information about local variables.

The size comparison is:
2.6M	g0/pydantic-1.8.1-cp39-cp39-linux_x86_64.whl
7.8M	g0/pydantic-1.8.1-cp39-cp39-linux_x86_64/

4.7M	g1/pydantic-1.8.1-cp39-cp39-linux_x86_64.whl
14M	g1/pydantic-1.8.1-cp39-cp39-linux_x86_64/

11M	g2/pydantic-1.8.1-cp39-cp39-linux_x86_64.whl
39M	g2/pydantic-1.8.1-cp39-cp39-linux_x86_64/

@bluetech How did you test this (since pydantic overwrites cflags in setup.py afaik)?
Btw, depending on the system I'm running on I get different results when building Pydantic:

Python Buster-Slim:

6.4M	/usr/local/lib/python3.9/site-packages/pydantic
python3.9-config --cflags: -I/usr/local/include/python3.9 -I/usr/local/include/python3.9  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall

Python Alpine:

6.7M	/usr/local/lib/python3.9/site-packages/pydantic
python3.9-config --cflags: -I/usr/local/include/python3.9 -I/usr/local/include/python3.9  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall

Ubuntu hirsute Python from apt:

39M	/usr/local/lib/python3.9/dist-packages/pydantic
python3.9-config --cflags: -I/usr/include/python3.9 -I/usr/include/python3.9  -Wno-unused-result -Wsign-compare -g -ffile-prefix-map=/build/python3.9-y4oXHL/python3.9-3.9.4=. -flto=auto -ffat-lto-objects -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -O3 -Wall

@peterroelants I cloned pydantic, edited the line you linked to, ran pip wheel ., extracted the wheel, ran du -s. That's as if pydantic itself makes the change.

Btw, depending on the system I'm running on I get different results when building Pydantic:

That is to be expected, but the differences do not seem too big, the major thing is the debug info.

+1 for adding -g0 to CFLAGS

Bumping this issue - any plans on this?

This will be solved in V2, we're stopping using cython completely.
Instead the core validation logic is written in rust, meaning:

Runtime performance has increased by roughly 15x
the pydantic-core wheel is now ~1.5mb on Linux, <1mb on macos and windows
The pydantic wheel should be a few hundred kb

Although looking at this again, I don't think I had realised that this could be solved by setting a debug flag, sorry.
Maybe we can fix this in v1.10.3 as I know that will be used for a while.

For anyone else looking for a post-install solution (i.e., you are fine with downloading the large binaries and you just want to save space), you can simply run strip on the shared libs to remove the debug info. Got me from ~50MB down to ~12MB.

Although looking at this again, I don't think I had realised that this could be solved by setting a debug flag, sorry.
Maybe we can fix this in v1.10.3 as I know that will be used for a while.

Just to bump the discussion - we're also waiting (albeit passively) for some news on this issue.
Should we expect a fix in v1.10.3, and if so, what is the ETA for this release?
Thank you!

happy to accept a fix for this for v1.10.3, otherwise I'll try to get to it.
In terms of ETA for V1.10.3, I had hoped to get it done over the last few weeks but COVID followed by bad cold combined with sick toddler has pretty much wiped me out.
I'll devote what time I have to it over the Christmas "break", but giving a firm deadline seems foolish at this point. Sorry for the delay.

happy to accept a fix for this for v1.10.3, otherwise I'll try to get to it.
In terms of ETA for V1.10.3, I had hoped to get it done over the last few weeks but COVID followed by bad cold combined with sick toddler has pretty much wiped me out.
I'll devote what time I have to it over the Christmas "break", but giving a firm deadline seems foolish at this point. Sorry for the delay.

Sorry to hear that, get well soon!
No worries about the timeline, we'll be happy to just get a notification whenever this issue is referenced in the eventual patch release.
I'm afraid at this point we're too much of surface level users of Pydantic to offer a well thought out fix, so we'll wait for someone more qualified to chime in.
Thank you and take care.

This is fixed in #4862.
Thanks so much everyone for your inputs, and particularly @bluetech for pointing out that this could be fixed by just -g0. Sorry it took so long to get to a resolution.


Create a test procedure for User Testing

Dewberry should start developing an internal process to test software before release. This is user testing in addition to automated tests.

ERT can provide production scripts and workflow to mimic production.
Flows2FIM should also be executed to make sure, the data contract is not being violated by ripple1d changes. If there is a breaking change then that must go in the changelog.


Support pip 7

Sooo pip7 refactored the way it reads requirements and the line reported ignore any comment lines. For example for the following requirements.txt file
# sha: foo
useless==1.0

pip reports (from -r requirements.txt (line 1)) instead of (from -r requirements.txt (line 2)). This practically means that we cannot rely on pip for the line number. I refactored _path_and_line to fix this but now if multiple lines in requirements file list same package, always the first will be returned. I'm not sure if that's a problem anyway, so posting here for feedback.

Informed by your diagnosis I also implemented a fix for this problem, but in a different way, here: https://github.com/kouk/peep/commit/be6437ec01cfd7ef92ef2e5d1eb11403bf7fac53
I'm fine with either fix really.

FWIW, I just filed a fix for that problem in pip: https://github.com/pypa/pip/pull/3030


SetDownloadBehavior no longer working since update

What versions are you running?

chromedp (master):
1b49b2238bb04ffd9b3bc2a6f24290b2a4deec5d
$ google-chrome --version
Google Chrome 81.0.4044.138
$ go version
go version go1.13.7 linux/amd64

What did you do? Include clear steps.
removed local packges .../src/github.com/chromedp
go get -u github.com/chromedp/chromedp
then tried to run code that was working fine:
if err := chromedp.Run(mctx, chromedp.Tasks{ page.SetDownloadBehavior(page.SetDownloadBehaviorBehaviorAllow).WithDownloadPath(fullPathBoleta("/mydir")), }); err != nil { log.Fatal(err) }
and I get:
undefined: page.SetDownloadBehavior undefined: page.SetDownloadBehaviorBehaviorAllow
notice SetDownloadBehavior was moved from page to browser package in cdproto repo so I update the code:
if err := chromedp.Run(mctx, chromedp.Tasks{ browser.SetDownloadBehavior(browser.SetDownloadBehaviorBehaviorAllow).WithDownloadPath(fullPathBoleta("./mydir")), }); err != nil { log.Fatal(err) }
now it compiles but throws error:
'Browser.setDownloadBehavior' wasn't found (-32601)
What did you expect to see?
no errors
What did you see instead?
'Browser.setDownloadBehavior' wasn't found (-32601)

Upstream moved the protocol. I assume that a newer Chrome version will implement Browser.setDownloadBehavior. Since Target.setDownloadBehavior was deprecated and thus removed from the latest cdproto, if you want to stick to the current version of Chrome, you can use an older version of cdproto.

thanks !, will use cdproto before chromedp/cdproto@0de008e
Still its a little bit strange since I'm using the latest stable version for linux https://chromereleases.googleblog.com/2020/05/stable-channel-update-for-desktop.html
cdproto works with dev ?


allow creating libraries artifacts

Is there already some discussion regarding supporting library building?
I am talking about RPMs, DEBs, etc.
I do have some opinions and might contribute.

Do you mean building packages for use with a package manager? We support Homebrew for Mac right now, but we'd love to support more.

The most important part (IMO) is not building a RPM or DEB out of the artifacts (I assume that's not even that hard), but package library artifacts.  My use case is PostgreSQL extensions.

C extensions, or PGRX?

Sorry, for not getting back.  I was pretty busy.
I am talking about pgrx extensions.  But I'd also like to package so libraries (generated by rust and built by cargo).  Is that going to be much different?
Any ideas on how it might look like?


Use correct features padding for encoder input

This adheres faster whisper implementation to the original OpenAI implementation as discussed in #1084
These are WER comparisons before and after
These figures can be reproduced by running benchmarks/yt_commons.py and switching the batched inference to sequential
word_timestamps=False,
without_timestamps=True,
vad_filter=True,

Model
Before WER
After WER

distil-large-v3
26.277
14.762

distil-large-v2
71.848
81.456

distil-medium.en
68.044
66.565

distil-small.en
68.719
67.220

the performance regression of distil-large-v2 can be ignored because distil-large-v3 should be used as a drop-in replacement
This should also affect all use cases where chunk length is less than 30 for all models
There is also an average WER improvement by around 2% (relative) for batched inference across all models

Model
Before WER
After WER

tiny.en
15.437
15.063

tiny
21.765
21.390

base.en
14.300
13.816

base
17.709
17.251

small.en
13.054
12.617

small
16.413
16.088

medium.en
13.299
12.894

medium
15.991
15.593

large-v1
19.458
19.590

large-v2
15.237
15.148

large-v3
16.514
15.997

large-v3-turbo
14.576
14.044

distil-small.en
14.004
13.918

distil-medium.en
14.074
13.972

distil-large-v2
13.419
13.574

distil-large-v3
13.688
13.533

This PR disabled the ability to change the encoder input and output dimensions (audio_ctx), any chunk_length passed to the transcribe will be respected, but the encoder input will be padded to 30s equivalent regardless of the chunk length
https://github.com/SYSTRAN/faster-whisper/issues/171#issuecomment-1701177072


Joins only support up to 3 composite keys

Description
I have a project which requires a join on 4 composite keys, but only up to 3 will compile. Furthermore, there's no way to specify a composite key with the belongs_to macro. The same issue occurs when implementing RelationTrait.
The error I get is:
error[E0271]: type mismatch resolving <packages::Entity as sea_orm::EntityTrait>::Column == (packages::Column, packages::Column, packages::Column, packages::Column)
Steps to Reproduce
The query looks like this:
let query = A::find()
    .join(
        JoinType::InnerJoin,
        TableA::Entity::belongs_to(TableB::Entity)
            .from((
                TableA::Column::Col1,
                TableA::Column::Col2,
                TableA::Column::Col3,
                TableA::Column::Col4,
            ))
            .to((
                TableB::Column::Col1,
                TableB::Column::Col2,
                TableB::Column::Col3,
                TableB::Column::Col4,
            ))
            .into(),
    )

Additional Information
It looks like this could be fixed by extending identity.rs to support more operations. Is there any way around this problem?

Hey @ctrlaltdel85, welcome!! Correct, under the current architecture we need to add Quaternary, Quinary and perhaps Senary. Another way could be rewrite it and take a dynamic Vec<DynIden>.

merge into 1294


Update benchmark

Add some benchmark. It also compare CPU vs GPU in Brighton beach dataset (https://github.com/pierotofy/drone_dataset_brighton_beach/tree/master)

Thank you Muhammad! Could you update the datafile slightly and remove the leading spaces (or maybe they are tabs) from your new data rows? I'm afraid the parser may not be smart enough to remove those.

alright, done

Looks great, thank you @ichsan2895


Unable to upload dashes

The script fails to upload dashes previously downloaded to another instance. The directories are created, but it fails with:

[16:32:51] INFO     Writing Router uptime dashboard..                                              dashboard_upload.py:106
INFO     Writing SNMP Interface Detail dashboard..                                      dashboard_upload.py:106
INFO     Setting home dashboard..                                                       dashboard_upload.py:117
Request failed - 'HTTP error 404: Dashboard not found'

Today I encountered the same issue. Have you resolved it somehow @alopes-dn ?

This error should be fixed now with the most recent release


Should changes countries list with state from another component and config one country is leading of list

Problem:

I have a list of 25 countries (define) & a button show all.
When I clicked to show all,  list countries will show all countries.

I can change the leading of the list.

Hi @dereknguyen269 ! I'm not quite sure I understand your question / suggestion, would you mind elaborating on it? Thanks!


colab

great work, please add a google colab demo for inference

Thanks for your suggestion, it is certainly a good idea. I will work on preparing a colab notebook soon.


Bug Report: when I play thorns spirit doesnt load

Please check for duplicates or similar issues, as well performing simple troubleshooting steps (such as deleting the YoshiCrafter29 folder from AppData) before submitting an issue.
If you're using an older version of the engine, check if the bug haven't been already fixed
PS : the "everything goes black while changing state in developer mode bug" is not a bug, it's been designed this way to refresh all bitmaps between states to allow easier modding.
Which target are you using?

[ x] Windows
[ ] Mac
[ ] Linux

What version of the engine are you using (Bottom left of the screen in main menu) ?
v<IP_ADDRESS>
Have you identified any steps to reproduce the bug? If so, please describe them below in as much detail as possible. Use images if possible.
Please describe your issue. Provide extensive detail and images if possible.

When I play thorns on freeplay or story mode spirit doesnt load

Fixed in 1.6.1


Ionic Dependencies

Ionic dependencies should be included and loaded directly.

#37

Done in 0.3.0


ci: prod tests fail-fast

- What I did

prod tests has strategy - fail-fast: true

- How I did it
- How to verify it
- Description for the changelog

@JeremyTubongbanua it's obvious what you're doing here, and I can guess why, but maybe you can give your version?

@JeremyTubongbanua it's obvious what you're doing here, and I can guess why, but maybe you can give your version?

I wish that other jobs do not get cancelled when one of the jobs fail. For example, if @rv_am fails, I would still want the @rv_eu and @rv_ap tests to run.


Trust store reload endpoint and node ops method

Fixes #483

As noted in the Slack, I'll add it here also: Add a new item to the FeatureSet to indicate this API endpoint is available.

@Miles-Garnsey I double checked and there are classes in dse-commons that are not in dse-core, though both artifacts provide classes in some of the same packages. That is something that should probably be addressed in the DSE code base, but that outside the scope here.

@emerkle826 is there some reason why we need both core and commons? It doesn't seem that anything from core is used which isn't in commons, and I don't know how they'll play together given both artifacts contain the same class names....

@Miles-Garnsey please add a Changelog entry as well when you get this all sorted.


hacktuberfest

hacktuberfest

hacktuberfest

This does not meet the requirements:

These URLs do not appear to be referral/advertisment/ip-logger sites
The URLs are not categorized correctly
The URLs are not alphabetized
One of the URLs is a subdomain


✨  add support for configuring android sdk via Ansible variables

To verify,

change the default android sdk configurations
Run

ansible-playbook -i <inventory file> sample-build-playbook.yml --tags=login,android-sdk -e "project_name=digger"

Wait for it to finish. rsh into the android sdk pod and you should see the android sdk configuration is updated to match the configuration. And it should be installed

ping @laurafitzgerald @PhilipGough @aidenkeating  for review.

Review looks good @wei-lee. Going to verify ASAP

Verified on test cluster using steps shown above


update to latest knative serving release

this was a security fix release for golang.org/x/net, which we have already patched.

Codecov Report
All modified lines are covered by tests :white_check_mark:

Comparison is base (190a7b9) 70.76% compared to head (55e3ee7) 70.76%.
Report is 1 commits behind head on main.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #178   +/-   ##
=======================================
  Coverage   70.76%   70.76%
=======================================
  Files           4        4
  Lines         195      195
=======================================
  Hits          138      138
  Misses         40       40
  Partials       17       17

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Reduce jitter

Greatly reduce jitteriness of movement over a real world connection by only overwriting positions when the server sends its  game state snapshot to the client when the server's idea of a position is very different from what the client thinks it is. This way, so long as latency is low enough, movement is smooth, with no abrupt changes in position.
For perf, and to keep tests passing, also includes a change updates players to be a BTreeMap rather than a Vec. A really cool data structure that combines most of the advantages of a map and an array.

There's two commits here, they're easiest reviewed separately


feat(besu): refactor genesis creation for ibft consensus

Description
As a ... I want to ... so that ....
(add more description and relevant links)
Acceptance Criteria

First acceptance criteria
...
...

I'take this issue


[BUG] Inability to Upload an AASX File Created with Package Explorer

Describe the bug
Inability to Upload an AASX File Created with Package Explorer version 2024-06-10.
It seems to be related to this answer, https://github.com/eclipse-basyx/basyx-java-server-sdk/issues/500#issuecomment-2437001447
On second thought, based on your comment, I guess this issue will be fixed in the pull request https://github.com/eclipse-basyx/basyx-java-server-sdk/pull/503 ?
To Reproduce
Steps to reproduce the behavior:

Create a simple Asset Administration Shell (AAS) using the Package Explorer.
Attempt to upload the AASX file using the aas-ui.
Encounter an error with status code 400.

Observations

The AAS is present in the repository.
The AAS is absent in the registry.
The submodel is missing in both the submodel repository and the submodel registry.

These observations seem similar to the issue https://github.com/eclipse-basyx/basyx-java-server-sdk/issues/500
Additional Information

The AASX file can be read using the basyx-python-sdk.
An AASX file created with the basyx-python-sdk can be successfully uploaded.

Expected behavior
Uploaded with no errors.
BaSyx (please complete the following information):
BaSyx Version [ v2.0]
BaSyx SDK [Python]
AAS Version [DotAAS V3]
BaSyx configuration
Default from the Quick Start Guide.

Hi @JS-Aibel,
Can you share your AAS that you created in AASPE? That way we can have a look at what might be wrong (on our end or in the AAS that you created in the AASPE).

Thank you for the reply, @aaronzi
It is any AASX I create with Package Explorer version 2024-06-10. I will e-mail you an example.

Thank you, I will have a look.

Found the problem. It seems that the asset type causes the problem and as discussed in #500 the registry integration creates the entry in the repo even if the entry in the registry could not be created. Oddly enough, this should have been fixed in the current version of BaSyx 2.0.0_SNAPSHOT.
This only happens if the asset type is set to NotApplicable. Attached are two shells to reproduce this. One uses the Instance type which works and one uses NotApplicable.
test-instance-asset-kind.zip
test-not-applicable-asset-kind.zip
To try the demo shells, change the file extension to .aasx.
We will have a look into this and come up with a fix.

The newest images did not include the pull request that fixes #500.
I pushed a new version of the AAS Environment to Dockerhub. I will now also update the 2.0.0_SNAPSHOT tag for all other components as well. This fixes the problem.


How to define a Custom Validation Error in Autoform >= 6.0

Hi there,
I don't know whether I just didn't find this passage in either the autoform, simpl-schema or messageBox readme/help but I could not happen to find a working solution of the new Syntax.
When using
myValidationContext.addValidationErrors([{ name: 'password', type: 'wrongPassword' }]);
should I then take
AutoForm.getValidationContext(...);  as Context?
Regards

The Meteor SimpleSchema package is no longer maintained other than critical fixes to keep it running with each latest Meteor release. Closing non-critical issues in this repo. Anyone who cares about this may do one or more of the following:

Switch to the NPM package. Be sure to adjust for the breaking changes. There can be a lot of work involved in switching, but it is worth the effort.
If this issue still occurs in the NPM package, submit an issue there. Be sure to include a link to a reproduction repo or a PR that adds a failing test for the issue.
If you have more time than money, submit a pull request to resolve any issue labeled "help wanted" in the NPM package repo.
If you have more money than time, donate to help support development if you are able


[poppler] Add glib support

The poppler port lacks the glib bindings. Adding poppler[glib] would help.
Proposed solution
Adding poppler[glib]
Describe alternatives you've considered
Self compiling and adding it to the path.

Another thing: why is poppler in the namespace "unofficial::". When searching package manager independent, this will lead to a conflict. find_package(poppler) vs find_package(unofficial-poppler)

why is poppler in the namespace "unofficial::".

Only upstream can provide official config. The unofficial config is a vcpkg polyfill. Vcpkg maintainers cannot know how upstream would design its package (package name, variable names, target names).

When searching package manager independent, this will lead to a conflict. find_package(poppler) vs find_package(unofficial-poppler)

No, the unofficial:: namespace avoids a conflict. It requires a concious decision by the consumer whether to do tight coupling with vcpkg, or to use another mechanism.
(Tight coupling is no problem within vcpkg, or in transitive dependencies of ports. Such as port gdal forwarding a depenency on poppler.)


Occasional neuron failure

Describe the bug
Sometimes neurons don't seem to work properly. I've seen LIF neurons which never leak and always fire. Reloading the network usually cures the problem.
Application: Brain Simulator
To Reproduce
Steps to reproduce the behavior:

I noted this in BasicNeurons AND Hebbian synapses.
Make many edits to the network like moving neurons around
Notice that some LIF neurons get "stuck" firing all the time
Save and reload, the problem goes away

Suspect some uninitialized variable in engine?
Desktop (please complete the following information):

OS: Win10
Version [v0.9d]

Add any other context about the problem here.

This one seems to be fixed...


Quick fixes in https://contribute.cncf.io/contributors/

https://contribute.cncf.io/contributors/
IMO the structure here is not the best. It starts with a definition of TOC, TAG etc.
There's a parallel ticket, https://github.com/cncf/tag-contributor-strategy/issues/331, which is broader and there's mentions of persona based responses, best practices, videos and such.
While I like all of those ideas, I would prefer getting this page fixed quickly with less effort.
I think something like this structure would work:

Welcome

Welcoming message
Contribution is not only code-contribution: Docs, design, TAGs, TOC, meetups, etc.

How to contribute

New to open source: The getting started article is there, linked on the navigation panel on the left, but not in an in-your-face way.
Other personas:

Some advice to check CNCF landscape for projects
Contributing to TAGs: define what TAGs and WGs are, and how they work
Community Engagement: organize meetups, record videos, become a mentor, apply for being an ambassador,

Some more changes:

Change the title from "Contribution Guides to the CNCF Ecosystem" to "Contribute to CNCF Ecosystem"
List of projects: I would remove this, if we can filter projects by "Focus" and "Primary Language" information in CNCF Landscape.

/assign


suggestion to fix openframe unused power domains

Trial to fix  https://github.com/efabless/caravel/issues/485

no need anymore


Latitude error at position 0

Hello, sorry for the short issue. I tried using these coordinates:
from pypolyline.cutil import encode_coordinates

coords = [
    [-6.944527, 107.580655],
    [-6.944806, 107.581525],
]

print(encode_coordinates(coords, 5))

it return
b'Latitude error at position 0: 107.580655'
is there something wrong with negative coordinates? or am i missing something?
Thanks!

You are missing something. Please read the first line of the "Usage" section, and consider the minimum and maximum possible values of latitude.

sorry, my bad. it supposed to be Longitude and Latitude not the other way around 😅 im so used to latitude, longitude!


change NT geojson location

The name of the NT geojson file has changed and this will also need to be changed in the OIMOM's map page for NT. In the front matter: canada_northwestTerritory_50k_nts.geojson --> canada_northwestTerritories_50k_nts.geojson

Same for YT: canada_yukonTerritory_50k_nts.geojson --> canada_yukon_50k_nts.geojson


What is the purpose of WinkPage?

I don't see any information about the WinkPage Model.
What's the use-case if there's any at all? At first glance it looks like a simplified version of WinkPost?

It's just a static page that you wouldn't want it to be displayed in the posts list.
For example an About Us Page.


Images are reloaded and not cached. Creates Flickering.

Recreate the bug:

Use the master branch or develop branch
Start CVAT and annotate a video (in debug mode or in docker
Open the network tab in the Chrome debugger.
Move the slider to change the frames.
For each frame a new HTTP-Request is send to the server to get the image for the frame (see the waiting time in the for the Request). While loading the player is empty and blue. This creates a flickering and the user cannot quickly judge if everything is annotated correctly in the video.

I think the images were cached in the past and were only loaded at the beginning. The code for the FrameProvider seems to implement some caching, but it is not working.

when I restarted my Mac, and reload this repo and tool, I got ERR_EMPTY_RESPONSE, anyway, I can't restart the tool again

@Ai-is-light , run the following commands docker-compose down and docker-compose up -d. It should help.

We will investigate the issue with caching images. Thanks for the report.

@Ai-is-light , a couple of questions:

Could you please check that when you open "Chrome debugger" caching actually is enabled?

@azhavoro
I suppose we can close the issue for now with introducing data streaming patch


Update dependency to address Vulnerability CVE-2017 18640 that remains since the last update

Is your feature request related to a problem? Please describe.
Yes. One of the dependencies (snakeyaml 1.25) has a known vulnerability:
      ** This is reportedly fixed in 1.26 **

CVE-2017-18640
CVSS v2 Base Score: 5 | Impact Score: 2.9 | Exploitability Score: 10
Vulnerability Summary:
The Alias feature in SnakeYAML allows entity expansion during a load operation, a related issue to CVE-2003-1564.
Describe the solution you'd like
Please update to the latest snakeyaml
Describe alternatives you've considered

Done! Thanks for suggestion

Thanks @mrniko


Edit Non-ITA Exam Details Exam Types

During testing, it was found that the exam type drop down items for non-ita exams were showing up as the ITA exam types. This was fixed by placing a conditional filter on the examType objects returned based on whether or not the ita_ind field is set to 0 or 1.

Needs a new target branch I'm assuming.


[DDW-685] Stake Pools - Adjustments

This PR adds fine adjustments to the Stake Pools screen.
Todos

[X] https://docs.google.com/document/d/10UwMUoDBalrRS4QvLZq2dtXJ42ICdpfgDPIUo3Fcxeg/edit#

Tooltip positioning and Search/Filtering will be handled on different PR's

Temporary Storybook URL, while Netifly is failing.

Screenshots

Testing Checklist

Slack QA thread
[ ] Test

Review Checklist
Basics

[ ] PR has been assigned and has appropriate labels (feature/bug/chore, release-x.x.x)
[ ] PR is updated to the most recent version of the target branch (and there are no conflicts)
[ ] PR has a good description that summarizes all changes and shows some screenshots or animated GIFs of important UI changes
[ ] CHANGELOG entry has been added to the top of the appropriate section (Features, Fixes, Chores) and is linked to the correct PR on GitHub
[ ] Automated tests: All acceptance and unit tests are passing (yarn test)
[ ] Manual tests (minimum tests should cover newly added feature/fix): App works correctly in development build (yarn dev)
[ ] Manual tests (minimum tests should cover newly added feature/fix): App works correctly in production build (yarn package / CI builds)
[ ] There are no flow errors or warnings (yarn flow:test)
[ ] There are no lint errors or warnings (yarn lint)
[ ] There are no prettier errors or warnings (yarn prettier:check)
[ ] There are no missing translations (running yarn manage:translations produces no changes)
[ ] Text changes are proofread and approved (Jane Wild / Amy Reeve)
[ ] Japanese text changes are proofread and approved (Junko Oda)
[ ] UI changes look good in all themes (Alexander Rukin)
[ ] Storybook works and no stories are broken (yarn storybook)
[ ] In case of dependency changes yarn.lock file is updated

Code Quality

[ ] Important parts of the code are properly commented and documented
[ ] Code is properly typed with flow
[ ] React components are split-up enough to avoid unnecessary re-renderings
[ ] Any code that only works in main process is neatly separated from components

Testing

[ ] New feature/change is covered by acceptance tests
[ ] New feature/change is manually tested and approved by QA team
[ ] All existing acceptance tests are still up-to-date
[ ] New feature/change is covered by Daedalus Testing scenario
[ ] All existing Daedalus Testing scenarios are still up-to-date

After Review

[ ] Merge the PR
[ ] Delete the source branch
[ ] Move the ticket to done column on the YouTrack board
[ ] Update Slack QA thread by marking it with a green checkmark

@daniloprates Netlify build is fixed 👍

@daniloprates Netlify build is fixed 👍

@DominikGuzei 💛

left distance != right distance (due to the scroll) is this an issue for this PR ?

this bug relates to position, right?

close cross is bigger than 8px
should we decrease size?

for controlled stake should be different "color-logic", but this is not issue of this PR, just mentioning

Rest is very good! 👍

for controlled stake should be different "color-logic", but this is not issue of this PR, just mentioning

@a-rukin this issue is for this PR. What should be the logic in this case?

@a-rukin this issue is for this PR. What should be the logic in this case?
I'm not completely sure, but we are aiming for all the stake be separated equally between pools
so I would assume that the perfect (green) percentage of stake is (1 / number_of_stakes)%
something like 90% of stake should be red -> monopoly

@a-rukin this issue is for this PR. What should be the logic in this case?

I'm not completely sure, but we are aiming for all the stake be separated equally between pools
so I would assume that the perfect (green) percentage of stake is (1 / number_of_stakes)%
something like 90% of stake should be red -> monopoly

Cool, I'll create a separated method for this algorithm and we can improve it once we get new information.

this bug relates to position, right?

I'll remove the fixed Search component until we find a better solution. ATM it's taking too much space.

left distance != right distance (due to the scroll) is this an issue for this PR ?

@a-rukin in this case, I can't just ignore the right margin, as it would make it even when there's the scrollbar, but uneven when there is not.
Let's leave this one like this and I'll open a card for us to study handling the scrollbar in a better way. Maybe we can apply one of those Javascript solutions globally in the app.

Let's leave this one like this and I'll open a card for us to study handling the scrollbar in a better way. Maybe we can apply one of those Javascript solutions globally in the app.

Sure 👌

Hey reviewers, all the requests were made and are available here: https://daniloprates.github.io/daedalus/?selectedKind=Staking&selectedStory=Stake Pools&full=0&addons=1&stories=1&panelRight=0&addonPanel=storybooks%2Fstorybook-addon-knobs

@MarcusHurney please review this one too 🙏

@daniloprates there are some issues with AppVeyor. @cleverca22 will help us resolve them in the scope of this PR.


fixing python parquet example

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Can one of the admins verify this patch?


Getting Error while loading png Image in lua

Hi,
I am loading an image using image package in lua.
It is working fine for many png, jpeg images. But for some images it is getting crashed while loading image.
A=image.load(filename )
libpng error: Read Error
/usr/local/torch/install/share/lua/5.1/image/init.lua:136: [read_png_file] Error during read_image
stack traceback:
[C]: in function 'load'
/usr/local/torch/install/share/lua/5.1/image/init.lua:136: in function 'loader'
/usr/local/torch/install/share/lua/5.1/image/init.lua:337: in function 'load'
[string "A=image.load(filename )"]:1: in main chunk
[C]: in function 'xpcall'
/usr/local/torch/install/share/lua/5.1/trepl/init.lua:648: in function 'repl'
...ocal/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:185: in main chunk
[C]: at 0x00406670
Please let me know how to fix this issue as it is crashing in libpng library
Please find the attached images for which it is crashing.

This image sounds corrupted, e.g. with ImageMagick (image renamed foo.png here):
$ identify -verbose foo.png
identify: Expected 8192 bytes; found 5714 bytes `foo.png' @ warning/png.c/MagickPNGWarningHandler/1656.
identify: Read Exception `foo.png' @ error/png.c/MagickPNGErrorHandler/1630.
identify: corrupt image `foo.png' @ error/png.c/ReadPNGImage/3959.

Hi deltheil,
Thanks for quick feedback. If image is corrupted, then still it should throw some error message or some thing like that.
But here it is just aborted and then we can not do anything.
I have thousands images and in that many images like that.
How to handle such cases in torch.
Regards,
Jitendra Bansal

If image is corrupted, then still it should throw some error message or some thing like that

The error is reported as a Lua error so you can use pcall to avoid aborting your program (see #58 for more details), e.g.:
local ok, img = pcall(image.load, 'foo.png')
if not ok then
  print('error: ' .. img)
-- else use `img`
end

Hi deltheil,
Thank you for solving the issue.
Regards,
Jitendra Bansal


Add network analysis

This will include a consensus_clustering util to determine a stable clustering of data by using a 3-step process.

Shall we break this into several issues? Maybe one issue for each of the features we have discussed?

Generate co-occurence network based on lists of entities
Generate knn network
Run community detection on a network

Yes, sorry. This was supposed to be the issue for the community detection util, but I realised it's not very clear. I will rename this and open two new issues for the other two features.


hide accept decline buttons when request is in the past

Subject: hide accept decline buttons when request is in the past
Description:
self-explanatory
Results:
aapeli made a mistake here
Backend version: develop-65563756
Frontend version: develop-65563756
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36
Page: https://couchers.org/messages/request/81115
User: slabbster (4334)

I think this is a dupe of #5010


How to get sessionID of my own connection, I need to judge whether my connection is the owner of a node

How to get sessionID of my own connection, I need to judge whether my connection is the owner of a node.
I see that sessionID is a unexported field of conn

@ayrb13 i needed this as well, and opened https://github.com/samuel/go-zookeeper/pull/81

Use conn. SessionID()
This can be closed.


linking error for the demo "ShowTexture"

Linking the "ShowTexture" fails because the libglfw3.a is missing

The library path is not general in CMakeList.txt
set(GLFW_LIBRARY    /Users/Chihiro/Desktop/OpenGL/glfw/src/libglfw3.a)

The glfw/src directory was missing "libglfw3.a" file, so I added it.
And I changed to use a relative path.


FATAL ERROR

Wassup guys!
How can i fix that error?
That happens when i open the ''MOD'' paste.
My ARK is from EPIC GAMES, im using the steamcmd.

Answered in Discord


Unable to run rules with settings file that run fine with hashtable

I've got a script with some compatibility bugs in it. I can run the compatibility rules on it like this:
$settings = @{
    Rules = @{
        PSUseCompatibleCommands = @{
            Enable = $true
            TargetProfiles = @(
                'win-8_x64_10.0.17763.0_5.1.17763.316_x64_4.0.30319.42000_framework' # Server 2019 - PS 5.1 (the platform it already runs on)
                'win-8_x64_6.2.9200.0_3.0_x64_4.0.30319.42000_framework' # Server 2012 - PS 3
                'ubuntu_x64_18.04_6.1.3_x64_4.0.30319.42000_core' # Ubuntu 18.04 - PS 6.1
            )
        }
        PSUseCompatibleTypes = @{
            Enable = $true
            TargetProfiles = @(
                'win-8_x64_10.0.17763.0_5.1.17763.316_x64_4.0.30319.42000_framework'
                'win-8_x64_6.2.9200.0_3.0_x64_4.0.30319.42000_framework'
                'ubuntu_x64_18.04_6.1.3_x64_4.0.30319.42000_core'
            )
        }
        PSUseCompatibleSyntax = @{
            Enable = $true
            TargetVersions = @('3.0', '5.1', '6.1')
        }
    }
}

Invoke-ScriptAnalyzer -Settings $settings -Path ./script.ps1

And get:

RuleName                            Severity     ScriptName Line  Message
--------                            --------     ---------- ----  -------
PSUseCompatibleCommands             Warning      blogBad.ps 9     The parameter 'FullyQualifiedName' is not available for
                                                 1                command 'Import-Module' by default in PowerShell version
                                                                  '3.0' on platform 'Microsoft Windows Server 2012 Datacenter'
PSUseCompatibleCommands             Warning      blogBad.ps 13    The parameter 'Depth' is not available for command
                                                 1                'Get-ChildItem' by default in PowerShell version '3.0' on
                                                                  platform 'Microsoft Windows Server 2012 Datacenter'
PSUseCompatibleCommands             Warning      blogBad.ps 22    The command 'Get-AuthenticodeSignature' is not available by
                                                 1                default in PowerShell version '6.1.3' on platform 'Ubuntu
                                                                  18.04.2 LTS'
PSUseCompatibleCommands             Warning      blogBad.ps 30    The command 'Compress-Archive' is not available by default
                                                 1                in PowerShell version '3.0' on platform 'Microsoft Windows
                                                                  Server 2012 Datacenter'
PSUseCompatibleTypes                Warning      blogBad.ps 12    The method 'Get' is not available on type
                                                 1                'System.Management.Automation.WildcardPattern' by default
                                                                  in PowerShell version '3.0' on platform 'Microsoft Windows
                                                                  Server 2012 Datacenter'
PSUseCompatibleSyntax               Warning      blogBad.ps 11    The constructor syntax '[System.Collections.Generic.List[Sys
                                                 1                tem.Management.Automation.Signature]]::new()' is not
                                                                  available by default in PowerShell versions 3,4

But when I put the settings in a psd1 file and run the same with:
Invoke-ScriptAnalyzer -Settings ./mysettings.psd1 -Path ./script.ps1

I get no output.
Is this expected behaviour -- is there something I need to add to a settings file to make this work? Or is this something I should be fixing in the engine?

Hmm, maybe there is a small problem in the file that makes problems when parsing of the settings file. PSSA does not have a good error behaviour on that. Run it with -Verbose to see more details of where it goes wrong.

I've figured it out!


MultiBLAST: Sequence with spaces at the start of lines results in an unrelated/invalid error message.

tested in qa.toxo

Fix on QA

just tested in qa.plasmo and i got the error:

here is a sequence
sequence-with-spaces.txt


IPV6 APIVIP Handling

Thanks for working on this @dbw7 - I recall @mchiappero was working on some similar changes in https://github.com/suse-edge/edge-image-builder/pull/513 so perhaps Marco can review to ensure those changes are also captured here
As a next step beyond this, I think we'll need to add dual-stack support too (particularly for the management cluster use-case because I think Rancher only supports dual-stack, not yet single-stack ipv6)
I guess that will require a couple of configuration changes:

Configure the k8s cluster networks for dual stack e.g https://docs.rke2.io/networking/basic_network_options#dual-stack-configuration
Configure the APIVIP to support two addresses (perhaps comma delimited like in the RKE2 config above), such that the k8s API is accessible via both address families - I think this will require adding another CIDR to the addresses in the MetalLB IPAddressPool

@hardys We'd indeed want to add dual-stack support - in fact, the first iteration of @dbw7 work was on it. I pushed back on it for now since it's safer (especially with changes to definition file) to do this in multiple steps where the first one is to enable the IPv6 support and have it thoroughly tested (in SV too) before proceeding with dual-stack.


feat: introduce v2.x

Since we have two major versions of cdk8s, we need the cli to support initializing templates that use either one.
We could have done this by adding a --cdk8s-version flag to the cli and thats it. However:

We want the default to be cdk8s v2, and if we change the default, thats a breaking change.
Keeping cdk8s-cli at v1 is going to cause some confusion with users since other cdk8s versions will be v2.

Granted, introducing a new release branch creates a maintenance burden, but I feel we can handle it with automatic backports for a while, like we do with cdk8s-core and cdk8s-plus.

Note that this PR is going to fail building (due to version conflicts in constructs) until https://github.com/cdk8s-team/cdk8s-plus/pull/696 is merged and we can bump cdk8s-plus version.


How to exit the GitBash text editor

When some users go through GitBash for the first time and type in their login credentials (usually when typing a commit message) they might be taken to a terminal that looks like this.

This can be exited by using the following steps
Step 1: press "i" on the keyboard. At the bottom of the screen there should be a line that says INSERT

Step 2: press the keys in the following order "Esc" then ":" (you can type this by using Shift+;) then press "w" , and finally "q"
Step3: hit "Enter" and you will be back to the original editor. Keep in mind you can just press the up arrow key to find your original commit message and publish your message.

Thanks for posting this, @JeremyF5737 ! Here's a quick shorthand:
Press I to insert.  (make edits if you need to)
Exit with:
[Esc]
:
w
q


Build binary for Linux/Windows with specific python/pyinstaller

Need to investigate how to build binary for Linux/Windows with the following required versions:
python >= 3.9 or pyinstaller >= 4.0
Currently action-python/pyinstaller-py3.8 is used with python == 3.8 and pyinstaller == 3.6

Binary build lacks of PYTHONIOENCODING=utf-8 env variable and it is impossible to set it even manually.
https://github.com/pyinstaller/pyinstaller/issues/2032


Retrieving the magic key

Hi,
I want to use this integration with my EasyControl CT200. It seems some kind of magic key is missing to get going with this. I just tried decompiling the Android version of the Bosch EasyControl app. I found some strings and auth_config json data that might be the key to getting this working.
I have no idea if I can simply share this without legal consequences?
Kind regards,
Ton

you can use lib and try it out here: https://github.com/bosch-thermostat/bosch-thermostat-client-python/blob/dev/bosch_thermostat_client/encryption/nefit_encryption.py
you can come to discord and PM me later, we can figure out something.
you can also try to check if it's not same useless data as here: https://github.com/robertklep/bosch-xmpp/issues/4

I think he found the magic key check https://github.com/robertklep/bosch-xmpp/blob/6368b4ac3dbe563da70391c94945884fe0ea611f/lib/ivt-client.js#L4

I think he found the magic key check https://github.com/robertklep/bosch-xmpp/blob/6368b4ac3dbe563da70391c94945884fe0ea611f/lib/ivt-client.js#L4


show first-to-ascend for spl

Ok, so, unfurtunately that won't quite work.
NHS currently defines the abbreviation of SpliceHack to be "sph", but that's not the only problem.
There is information missing from cfg/nethack_def.json, that needs to be fixed before I can merge this.
I have added the missing roles and races for SpliceHack, but other data also need to be there to define
what race/role/align combinations are possible in each variant - this is not yet there for SpliceHack.
It doesn't actually make much sense to merge this PR tbh.
However, you have successfully spurred me into making related changes.
I'm also wondering how much of a pain it would be to change the scoreboard so it abbreviates SpliceHack as spl instead of sph.
Seems like everyone else normally says spl (myself included), including the Beholder bot reporting on hardfought games.

See commit 9391bbe.
For the combinations, we'll need that information in there too. Either in the form of exclusion rules (as is done for vanilla and a few variants), or a full, explicit list of allowed combinations (EvilHack, dNethack).
SLASH'EM, slex and dnhslex are permanently barred from consideration for first-to-ascend.
Mandevil made this decision in the past with the older codebase, and I have no intention of changing that.

well now I added all the stuff I shall pull your change and try to test it out. woop


[WIP] Update SIEM-at-Home examples

Changes (still WIP):

beats-general-config.yml - Add comment regarding netinfo.enabled setting and added a section to disable beats setup within the configuration file
README.md - Provide additional information regarding the beats-configs in this example

Thanks for this Rob.  I was thinking that the docs  / default config file for Filebeat and Metricbeat should be updated to set:
 - add_host_metadata:
      netinfo.enabled: true

because it seems like sometimes the fields that get populated by that setting are needed for the Infrastructure UI.  What do you think about that?  I don't have a ton of experience with that setting, other than seeing an unpopulated field being used as the filter for the Infrastructure UI.

Hi @DanRoscigno, thanks for the feedback!  I'm not sure if there is a plan to update that as a default, that might be something to bring up with the beats team.
I'm adding that configuration to these examples for use with Elastic SIEM.  Please ping me if you want to discuss more, thanks!


After quantifying the model, the speed of model inference slows down

Today, I learned how to use tensorflow Lite to quantify the model. I ran the code in the example. I found that the model inference speed after quantification is 0.3 seconds slower than the original model. Is this normal? I firstly do it and who can answer, thank you~~~

i use tensorflow-gpu-1.14 and quantify to int8 and float16

I just got promoted. I found that their speed was almost the same. Is this because of the version? But there was no significant increase in speed.

I just got promoted to tensorflow-gpu-1.15. I found that their speed was almost the same. Is this because of the version? But there was no significant increase in speed.

Quantization procedures require TensorFlow 1.15 and above. Quantized models provide 2-3x speed up. Thanks!

@ymodak，but，I used ‘from tensorflow.tools.graph_transforms import TransformGraph’  to it , and lite may be.not 1.15  is 1.14

I found the reason because I quantified the model after pruning, the model itself was small, and the effect of reduction and acceleration was not obvious.


[camel-main] Restore langchain4j version synchronization with Quarkus

Bug description
Due to recent upgrade of langchain4jin Camel, the version is noc synchronized with Quarkus anymore on camel-main branch.
Return the synchronization of versions once the Quarkus is upgraded to the same version.

The correct solution is to always synchronise with Quarkus andignore the version from Camel.


docs(developers):commit message format typo

Any line of the commit message cannot be longer than 100 characters!
AngularJS is in LTS mode
We are no longer accepting changes that are not critical bug fixes into this project.
See https://blog.angular.io/stable-angularjs-and-long-term-support-7e077635ee9c for more detail.

Does this PR fix a regression since 1.7.0, a security flaw, or a problem caused by a new browser version?
No

What is the current behavior? (You can also link to an open issue here)
Typographical error
What is the new behavior (if this is a feature change)?
None
Does this PR introduce a breaking change?
No
Please check if the PR fulfills these requirements

[x] The commit message follows our guidelines
[x] Fix/Feature: Docs have been added/updated
[ ] Fix/Feature: Tests have been added; existing tests pass

Other information:
Correct typographical error

@googlebot I signed it!


Thank you very much

Thank you very much for publishing your work, I tried to run it in docker as well, but I messed up with dependencies and burned out, abandoned it for a while, returned, and here is your option, if you allow it, I will fork to play with dependencies and pull the project during build not through compose?
In any case, thanks again, I mentioned your repository in my article: https://habr.com/en/articles/743110/
From me, if not difficult, I would ask, I think for many people, to change your default ip to <IP_ADDRESS>
app.py
parser = argparse.ArgumentParser()
parser.add_argument(
'--listen',
type=str,
default='<IP_ADDRESS>',
help='IP to listen on for connections to Gradio',
)
or using --listen, just the endpoint is not available out of the box. Thank you very much again.
Come to my repository:
https://github.com/j0schihatake

Still, it did not work out to run according to your sources, first I changed ip, connected but did not see the video card. I did it my way, here are the sources: https://github.com/j0schihatake/NN_audiocraft_docker and video: https://www.youtube.com/watch?v=SnGZDKT9ifg&t=222s
Thank you very much again, in any case.


Add option to allow DIC to retry C2 error reading while still allowing a dump to finish if the retries aren't successful (feature request)

Describe the bug
Currently, DIC is only able to dump a disc with either no C2 error rereadings, or to have it attempt to reread the errors but not complete the dump if the rereads are unsuccessful. This is an issue for multiple situations, including development CD-Rs that have 2-3 errored sectors at the end of the disc due to the mastering method used. The /c2 option has to be disabled for dumping these discs, but without proper retrying of errors, consistent dumps of the same disc can be impossible. Another situation includes if a copy protected disc that uses bad sectors (but isn't supported by DIC's protection scanning) has bad sectors not related to the copy protection. Discs like these couldn't be accurately dumped by DIC.

It must be possible to do something better than the current behaviour. For example, changing /c2 to work something like this:

If a sector has a C2 error, AND
The sector has been retried for the full reread count without success, THEN

If the disc identifies as a CD-R, AND
Only the last 2 sectors are bad, THEN

Log that we are assuming a badly mastered CD-R
Read the sector for the full reread count and count how many times each result is obtained
Accept whichever result is obtained a plurality of the time and write that to the image
Increase the error count by 1
Move on to the next sector

Otherwise (not a CD-R or error is elsewhere on the disc), abort with an error as currently

That would provide a much better out-of-the-box experience for users trying to dump CD-Rs with this mastering issue that seems to be pretty common.
There could also (or instead) be a "C2 optional" parameter for handling other types of situations which I think is what Silas had in mind:

If a sector has a C2 error, AND
The sector has been retried for the full reread count without success, THEN

If "C2 optional" is enabled, THEN

Read the sector for the full reread count and count how many times each result is obtained
Accept whichever result is obtained a plurality of the time and write that to the image
Increase the error count by 1
Move on to the next sector

That would allow for power users to handle situations like damaged one-of-a-kind discs where the goal is to recover as much data as possible even if some sectors are hopeless. You don't necessarily want to abort after the first unrecoverable sector (like /c2) but you also don't want to just take the first bad read for every sector (like DIC without /c2) when there are potentially some sectors that would have gotten a good read if they had been retried.


Add CI for building and testing

Currently we have no way to ensure that the building works and the test cases are passing. The following CI should be provided at least:

Format
Lint
Build
Unit Test
Integration Test with TiDB or other components
Coverage Report

Related: https://github.com/pingcap/ng-monitoring/issues/4