any complete installation instructions?

Hi.
I've spent several days trying to install scGPT. I may be mistaken, but the instructions appear to have significant gaps. Is it possible for you to post a single, complete set of instructions that you have followed from beginning to end, by starting a new instance and documenting every step, including the specific versions of the instance or container and of every tool? I've tried to guess, but after I solve one problem, I bump into another (R, ld, etc.).

Whether I use pip install or poetry install, that fails because rpy2 expects R to be installed in advance (see example below). If that's true, can you make sure the instructions indicate how R is supposed to be installed?
documents/environment.yml suggests that you're using conda, possibly to install R. If you are using conda, can you update the instructions, and does "pip install" mean the pip installed by conda or another pip? If you aren't using conda, what do those requirements mean?
Here you recommend CUDA 11.7 and flash-attn < 1.05, and you refer users to the installation instructions for flash-attn. Those instructions are a mess. There are endless variants, and they are all incomplete. I've tried <1.05, 1.05, 1.06, compiling from source, installing gcc with conda, etc.
Here, someone recommends the Nvidia Pytorch container. There are many versions of this container. Are they all equivalent? If not, could you specify the version(s) that you have tried and that you know will work? I tried the most recent one (23.07-py3), then assumed that I should be using a container CUDA 11.7, so 22.05 through 22.08. I took the latest of these (22.08) and got a bit further in the installation, but got another error.

It would not be useful to post answers to individual questions here. Again, the request is for a single complete set of instructions.
Just as an example, here's the error message I get when I use the poetry install option due to apparently undeclared expectation that R is installed in advance (similar errors occur with pip install, and whether using Nvidia Pytorch container 23.07-py3 or 22.08-py3):

Installing rpy2 (3.4.2): Failed
ChefBuildError
Backend subprocess exited when trying to invoke build_wheel
['cffi>=1.10.0', 'jinja2', 'pytz', 'tzlocal']
Error: rpy2 in API mode cannot be built without R in the PATH or R_HOME defined. Correct this or force ABI mode-only by defining the environment variable RPY2_CFFI_MODE=ABI
at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/chef.py:147 in _prepare
144│                 error = ChefBuildError("\n\n".join(message_parts))
146│             if error is not None:
→ 147│                 raise error from None
149│             return path
151│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
Note: This error originates from the build backend, and is likely not a problem with poetry but with rpy2 (3.4.2) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "rpy2 (==3.4.2)"'.

Hi, to summarize, the installation instruction itself is complete. However, it is true that flash-attn has been reported to be difficult to install.
To answer your questions:

It is true that rpy2 would require R installed. I previously assumed many users may have R installed at system level already, but thanks for the reminder, I added a new note for this to avoid future confusion. For your case, if you haven't have R installed, you may follow the official instruction here https://cran.r-project.org/ . There are also a lot of more user-friendly guides for installing R online, for example, https://www.digitalocean.com/community/tutorials/how-to-install-r-on-ubuntu-22-04 . If your case is that you already have R installed somewhere, but rpy2 can't find it, please see the note here https://github.com/rpy2/rpy2#issues-loading-shared-c-libraries

If you mean the docs/environment.yml file, that is for building up the documentation website. You don't need to follow that one for your working env. Please see the instruction in the Readme.md. On the other hand, it is a good idea to use conda to manage your virtual envs. Although recommended, this is not required. pip install should work both with or without conda.

3 and 4. Usually, I found the issues encountered when installing flash-attn are "stand-alone" issues not related to other dependencies. Again, the package is strongly dependent with the local GPU versions, cuda, and pytorch versions. I personally have tested cuda 11.7, pytorch 1.13.0 with flash-attn 1.0.1 or 1.0.2 . Those have worked for my scenario with A40 and A100 GPUs. So, I will suggest you share the exact error messages you had when pip install flash-attn==x.x.x and let us know your GPU, cuda, and pytorch versions as well.

Hi. Thanks for your reply.
I'm not able to launch A40 or A100 GPUs. My options are M60, A10G and T4. I previously used an M60, but I have now done everything described here on both an M60 and an A10G, with identical results.

Preliminary step: on the Docker server, upgrade the Nvidia drivers and CUDA (previously 11.4) so they match the Nvidia PyTorch container (11.7):

$ nvidia-smi
| NVIDIA-SMI 515.65.01    Driver Version: 515.65.01    CUDA Version: 11.7     |
|   0  Tesla M60           Off  | 00000000:00:1E.0 Off |                    0 |

$ nvcc -V
Built on Wed_Jun__8_16:49:14_PDT_2022
Cuda compilation tools, release 11.7, V11.7.99
Build cuda_11.7.r11.7/compiler.31442593_0

I didn't do this previously, and it seems to make no difference.

Start the most recent Nvidia PyTorch container running CUDA 11.7:

docker run -it --privileged -w /workspace --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --rm --runtime=nvidia --gpus all nvcr.io/nvidia/pytorch:**22.08-py3** /bin/bash

Check versions:

$ nvidia-smi
| NVIDIA-SMI 515.65.01    Driver Version: 515.65.01    CUDA Version: 11.7     |
|   0  Tesla M60           Off  | 00000000:00:1E.0 Off |<PHONE_NUMBER> |

$ nvcc -V
Built on Wed_Jun__8_16:49:14_PDT_2022
Cuda compilation tools, release 11.7, V11.7.99
Build cuda_11.7.r11.7/compiler.31442593_0

$ ninja --version
1.11.0

$ python
Python 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:18)
[GCC 10.3.0] on linux
>>> import torch
>>> torch.__version__
'1.13.0+cu117'
>>> torch.cuda.is_available()
True

In the container Python was installed with conda and pip means /opt/conda/bin/pip. Many of the packages needed by scGPT were installed with pip, except that packaging was installed with conda.
Python package versions found in the container:
torch                     1.13.0a0+d321be6          pypi_0    pypi
torch-tensorrt            1.2.0a0                  pypi_0    pypi
torchtext                 0.11.0a0                 pypi_0    pypi
torchvision               0.14.0a0                 pypi_0    pypi
mdit-py-plugins           0.3.0                    pypi_0    pypi
markdown-it-py            2.1.0                    pypi_0    pypi
protobuf                  3.20.1                   pypi_0    pypi
cuda-python               11.6.0                   pypi_0    pypi
packaging                 21.3               pyhd8ed1ab_0    **conda-forge**

Install R (briefly, apt install -y r-base r-base-core r-recommended r-base-dev).
Install flash-attn:

# on M80
pip install flash-attn==1.0.2
# on A10G
pip install flash-attn==1.0.1 --no-build-isolation

Both installations succeeded and what follows was identical.

install scGPT with pip install scgpt. Pip uninstalls packages, installs different versions, and reports an error because of the missing versions (which it just uninstalled).

  Attempting uninstall: torchtext
    Found existing installation: torchtext 0.11.0a0
    Uninstalling torchtext-0.11.0a0:
      Successfully uninstalled torchtext-0.11.0a0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.14.0a0 requires torch==1.13.0a0+d321be6, but you have torch 1.13.0 which is incompatible.
mdit-py-plugins 0.3.0 requires markdown-it-py<3.0.0,>=1.0.0, but you have markdown-it-py 3.0.0 which is incompatible.
jupytext 1.14.1 requires markdown-it-py<3.0.0,>=1.0.0, but you have markdown-it-py 3.0.0 which is incompatible.
cudf 22.6.0a0+319.g97422602b8 requires protobuf<3.21.0a0,>=3.20.1, but you have protobuf 3.19.6 which is incompatible.
Successfully installed aiohttp-3.8.5 aiosignal-1.3.1 anndata-0.9.2 anndata2ri-1.2 async-timeout-4.0.2 backports.zoneinfo-0.2.1 cached-property-1.5.2 chex-0.1.7 comm-0.1.4 datasets-2.14.3 deprecated-1.2.14 dill-0.3.7 dm-tree-0.1.8 docrep-0.3.2 et-xmlfile-1.1.0 etils-1.3.0 flax-0.7.1 frozenlist-1.4.0 h5py-3.9.0 huggingface-hub-0.16.4 igraph-0.9.11 ipywidgets-8.1.0 jax-0.4.13 jaxlib-0.4.13 jupyterlab-widgets-3.0.8 leidenalg-0.8.10 lightning-utilities-0.9.0 llvmlite-0.38.1 louvain-0.7.2 markdown-it-py-3.0.0 ml-dtypes-0.2.0 multidict-6.0.4 multipledispatch-1.0.0 multiprocess-0.70.15 natsort-8.4.0 numba-0.55.2 numpyro-0.12.1 nvidia-cublas-cu11-<IP_ADDRESS> nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-<IP_ADDRESS> openpyxl-3.1.2 opt-einsum-3.3.0 optax-0.1.7 orbax-checkpoint-0.2.3 pandas-1.3.5 patsy-0.5.3 protobuf-3.19.6 pyDeprecate-0.3.1 pyarrow-12.0.1 pygments-2.16.1 pynndescent-0.5.10 pyro-api-0.1.2 pyro-ppl-1.8.6 pytorch-lightning-1.5.10 rich-13.5.2 rpy2-3.5.13 safetensors-0.3.1 scanpy-1.9.3 scgpt-0.1.2.post1 scib-1.0.4 scikit-misc-0.1.4 scipy-1.10.1 scvi-tools-0.16.4 seaborn-0.12.2 session-info-1.0.0 statsmodels-0.14.0 stdlib-list-0.9.0 tensorstore-0.1.40 texttable-1.6.7 tokenizers-0.13.3 torch-1.13.0 torchmetrics-1.0.2 torchtext-0.14.0 transformers-4.31.0 tzlocal-5.0.1 umap-learn-0.5.3 widgetsnbextension-4.0.8 wrapt-1.15.0 xxhash-3.3.0 yarl-1.9.2

start Python and import scgpt:

> $ python
> Python 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10)
> [GCC 10.3.0] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import scgpt
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/opt/conda/lib/python3.8/site-packages/scgpt/__init__.py", line 18, in <module>
>     from . import model, tokenizer, scbank, utils
>   File "/opt/conda/lib/python3.8/site-packages/scgpt/model/__init__.py", line 1, in <module>
>     from .model import (
>   File "/opt/conda/lib/python3.8/site-packages/scgpt/model/model.py", line 12, in <module>
>     from flash_attn.flash_attention import FlashMHA
>   File "/opt/conda/lib/python3.8/site-packages/flash_attn/flash_attention.py", line 7, in <module>
>     from flash_attn.flash_attn_interface import flash_attn_unpadded_qkvpacked_func
>   File "/opt/conda/lib/python3.8/site-packages/flash_attn/flash_attn_interface.py", line 5, in <module>
>     import flash_attn_cuda
> ImportError: /opt/conda/lib/python3.8/site-packages/flash_attn_cuda.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN3c105ErrorC2ENS_14SourceLocationENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

Install poetry, add to path, clone the scGPT repo, and run poetry install:

$ poetry install
Creating virtualenv scgpt-R2X-0p9C-py3.8 in /root/.cache/pypoetry/virtualenvs
Installing dependencies from lock file
Package operations: 214 installs, 2 updates, 0 removals
  • Installing zipp (3.8.0)
  •
  • (installs many packages)
  •
  • Installing flash-attn (1.0.1): Failed
  ChefBuildError
  Backend subprocess exited when trying to invoke get_requires_for_build_wheel
  Traceback (most recent call last):
    File "/root/.local/share/pypoetry/venv/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/root/.local/share/pypoetry/venv/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/root/.local/share/pypoetry/venv/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/tmpjhe11x3w/.venv/lib/python3.8/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/tmp/tmpjhe11x3w/.venv/lib/python3.8/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
      self.run_setup()
    File "/tmp/tmpjhe11x3w/.venv/lib/python3.8/site-packages/setuptools/build_meta.py", line 487, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/tmp/tmpjhe11x3w/.venv/lib/python3.8/site-packages/setuptools/build_meta.py", line 338, in run_setup
      exec(code, locals())
    File "<string>", line 6, in <module>
  ModuleNotFoundError: No module named 'packaging'
  at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/chef.py:147 in _prepare
      144│                 error = ChefBuildError("\n\n".join(message_parts))
      146│             if error is not None:
    → 147│                 raise error from None
      149│             return path
      151│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with flash-attn (1.0.1) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "flash-attn (==1.0.1)"'.

Just to confirm, packaging is installed, so it seems that it must be installed in a specific way:
$ pip list |grep packaging
packaging                     23.1
$ pip install packaging
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: packaging in /opt/conda/lib/python3.8/site-packages (23.1)

Thanks.

Hi, thank you for the detailed messages! It looks like the rpy2 installation is OK now after installing R. Now the rest of the issue seems related to flash-attn.
After this step:

Install flash-attn:
# on M80
pip install flash-attn==1.0.2
# on A10G
pip install flash-attn==1.0.1 --no-build-isolation

Both installations succeeded and what follows was identical.

can you try run any tests for flash-attn, for example, the official ones here https://github.com/Dao-AILab/flash-attention/blob/main/benchmarks/benchmark_flash_attention.py and https://github.com/Dao-AILab/flash-attention/blob/main/tests/test_flash_attn.py
This is to verify the installation of flash-attn alone. Optionally, you may not need to do the installation again. Just run these tests in your current env to see whether the flash-attn has been properly installed.
BTW, there are several related issues in the flash-attn repo as well https://github.com/Dao-AILab/flash-attention/issues?q=flash_attn_cuda.cpython-38-x86_64-linux-gnu.so%3A+undefined+symbol . Please have a look and hope some walkaround there can help your case. I also think the official repo hasn't found a universal solution yet. The answers are case-by-case so far.

Hi @bzip2 , do you have any updates?

Hi, @subercui
Thank you for asking. I'm able to install flash-attn and run the tests, but scGPT is another matter. With the setup described above, I am only able to install scGPT like this:

install flash-attn 1.0.1
pip install scgpt
install flash-attn 1.0.4
git clone & poetry install
pip uninstall mdit-py-plugins jupytext cudf

This also works if I install flash 1.0.4 the first time then 1.0.1 the second time.

Does scGPT expect R to be installed in a specific way? In this case, I've installed scGPT in a conda environment as a way to cope with all the dependencies and the limitations of pip, but R is definitely on my path. I can import all the Python packages just fine except for scGPT:
# python
Python 3.8.17
[GCC 11.2.0] :: Anaconda, Inc. on linux
>>> import rpy2, torch, flash_attn
>>> from flash_attn.flash_attn_interface import flash_attn_unpadded_qkvpacked_func
>>> from flash_attn.flash_attention import FlashMHA
>>> torch.cuda.is_available()
True
>>> import scgpt
During startup - Warning message:
package ‘stats’ in options("defaultPackages") was not found
Global seed set to 0
>>>

# R
R version 4.3.1 (2023-06-16) -- "Beagle Scouts"
(...)
> library(stats)
>

I installed scGPT with pip install. It imports a package it did not install:
# python scGPT/tests/test_scbank.py
Traceback (most recent call last):
  File "scGPT/tests/test_scbank.py", line 5, in <module>
    import pytest
ModuleNotFoundError: No module named 'pytest'

Successfully installed MarkupSafe-2.1.3 absl-py-1.4.0 aiohttp-3.8.5 aiosignal-1.3.1 anndata-0.9.2 anndata2ri-1.2 async-timeout-4.0.3 attrs-23.1.0 backports.zoneinfo-0.2.1 cached_property-1.5.2 cachetools-5.3.1 certifi-2023.7.22 cffi-1.15.1 charset-normalizer-3.2.0 chex-0.1.7 contourpy-1.1.0 cycler-0.11.0 datasets-2.14.4 deprecated-1.2.14 dill-0.3.7 dm-tree-0.1.8 docrep-0.3.2 et-xmlfile-1.1.0 etils-1.3.0 filelock-3.12.2 flax-0.7.2 fonttools-4.42.0 frozenlist-1.4.0 fsspec-2023.6.0 future-0.18.3 google-auth-2.22.0 google-auth-oauthlib-1.0.0 grpcio-1.57.0 h5py-3.9.0 huggingface-hub-0.16.4 idna-3.4 igraph-0.9.11 importlib-resources-6.0.1 ipywidgets-8.1.0 jax-0.4.13 jaxlib-0.4.13 jinja2-3.1.2 joblib-1.3.2 jupyterlab-widgets-3.0.8 kiwisolver-1.4.4 leidenalg-0.8.10 lightning-utilities-0.9.0 llvmlite-0.38.1 louvain-0.7.2 markdown-3.4.4 markdown-it-py-3.0.0 matplotlib-3.7.2 mdurl-0.1.2 ml-dtypes-0.2.0 msgpack-1.0.5 multidict-6.0.4 multipledispatch-1.0.0 multiprocess-0.70.15 natsort-8.4.0 networkx-3.1 numba-0.55.2 numpy-1.22.4 numpyro-0.12.1 nvidia-cublas-cu11-<IP_ADDRESS> nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-<IP_ADDRESS> oauthlib-3.2.2 openpyxl-3.1.2 opt-einsum-3.3.0 optax-0.1.7 orbax-checkpoint-0.2.3 pandas-1.3.5 patsy-0.5.3 pillow-10.0.0 protobuf-3.20.1 pyDeprecate-0.3.1 pyarrow-12.0.1 pyasn1-0.5.0 pyasn1-modules-0.3.0 pycparser-2.21 pydot-1.4.2 pynndescent-0.5.10 pyparsing-3.0.9 pyro-api-0.1.2 pyro-ppl-1.8.6 pytorch-lightning-1.5.10 pytz-2023.3 pyyaml-6.0.1 regex-2023.8.8 requests-2.31.0 requests-oauthlib-1.3.1 rich-13.5.2 rpy2-3.5.13 rsa-4.9 safetensors-0.3.2 scanpy-1.9.3 scgpt-0.1.2.post1 scib-1.0.4 scikit-learn-1.3.0 scikit-misc-0.1.4 scipy-1.10.1 scvi-tools-0.16.4 seaborn-0.12.2 session-info-1.0.0 setuptools-59.5.0 statsmodels-0.14.0 stdlib_list-0.9.0 tensorboard-2.14.0 tensorboard-data-server-0.7.1 tensorstore-0.1.41 texttable-1.6.7 threadpoolctl-3.2.0 tokenizers-0.13.3 toolz-0.12.0 torch-1.13.0 torchmetrics-1.0.3 torchtext-0.14.0 tqdm-4.66.1 transformers-4.31.0 tzlocal-5.0.1 umap-learn-0.5.3 urllib3-1.26.16 werkzeug-2.3.7 widgetsnbextension-4.0.8 wrapt-1.15.0 xxhash-3.3.0 yarl-1.9.2

Does scGPT expect R to be installed in a specific way? In this case, I've installed scGPT in a conda environment as a way to cope with all the dependencies and the limitations of pip, but R is definitely on my path. I can import all the Python packages just fine except for scGPT:
# python
Python 3.8.17
[GCC 11.2.0] :: Anaconda, Inc. on linux
>>> import rpy2, torch, flash_attn
>>> from flash_attn.flash_attn_interface import flash_attn_unpadded_qkvpacked_func
>>> from flash_attn.flash_attention import FlashMHA
>>> torch.cuda.is_available()
True
>>> import scgpt
During startup - Warning message:
package ‘stats’ in options("defaultPackages") was not found
Global seed set to 0
>>>

# R
R version 4.3.1 (2023-06-16) -- "Beagle Scouts"
(...)
> library(stats)
>

A generic R installation is fine, for example, conda install would be fine. The only reason we need R is that the dependency scib used rpy2 and then rpy2 needs R.
It looks like the package is already imported? Most likely, you can ignore the warning message.

I installed scGPT with pip install. It imports a package it did not install:
# python scGPT/tests/test_scbank.py
Traceback (most recent call last):
  File "scGPT/tests/test_scbank.py", line 5, in <module>
    import pytest
ModuleNotFoundError: No module named 'pytest'

Successfully installed MarkupSafe-2.1.3 absl-py-1.4.0 aiohttp-3.8.5 aiosignal-1.3.1 anndata-0.9.2 anndata2ri-1.2 async-timeout-4.0.3 attrs-23.1.0 backports.zoneinfo-0.2.1 cached_property-1.5.2 cachetools-5.3.1 certifi-2023.7.22 cffi-1.15.1 charset-normalizer-3.2.0 chex-0.1.7 contourpy-1.1.0 cycler-0.11.0 datasets-2.14.4 deprecated-1.2.14 dill-0.3.7 dm-tree-0.1.8 docrep-0.3.2 et-xmlfile-1.1.0 etils-1.3.0 filelock-3.12.2 flax-0.7.2 fonttools-4.42.0 frozenlist-1.4.0 fsspec-2023.6.0 future-0.18.3 google-auth-2.22.0 google-auth-oauthlib-1.0.0 grpcio-1.57.0 h5py-3.9.0 huggingface-hub-0.16.4 idna-3.4 igraph-0.9.11 importlib-resources-6.0.1 ipywidgets-8.1.0 jax-0.4.13 jaxlib-0.4.13 jinja2-3.1.2 joblib-1.3.2 jupyterlab-widgets-3.0.8 kiwisolver-1.4.4 leidenalg-0.8.10 lightning-utilities-0.9.0 llvmlite-0.38.1 louvain-0.7.2 markdown-3.4.4 markdown-it-py-3.0.0 matplotlib-3.7.2 mdurl-0.1.2 ml-dtypes-0.2.0 msgpack-1.0.5 multidict-6.0.4 multipledispatch-1.0.0 multiprocess-0.70.15 natsort-8.4.0 networkx-3.1 numba-0.55.2 numpy-1.22.4 numpyro-0.12.1 nvidia-cublas-cu11-<IP_ADDRESS> nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-<IP_ADDRESS> oauthlib-3.2.2 openpyxl-3.1.2 opt-einsum-3.3.0 optax-0.1.7 orbax-checkpoint-0.2.3 pandas-1.3.5 patsy-0.5.3 pillow-10.0.0 protobuf-3.20.1 pyDeprecate-0.3.1 pyarrow-12.0.1 pyasn1-0.5.0 pyasn1-modules-0.3.0 pycparser-2.21 pydot-1.4.2 pynndescent-0.5.10 pyparsing-3.0.9 pyro-api-0.1.2 pyro-ppl-1.8.6 pytorch-lightning-1.5.10 pytz-2023.3 pyyaml-6.0.1 regex-2023.8.8 requests-2.31.0 requests-oauthlib-1.3.1 rich-13.5.2 rpy2-3.5.13 rsa-4.9 safetensors-0.3.2 scanpy-1.9.3 scgpt-0.1.2.post1 scib-1.0.4 scikit-learn-1.3.0 scikit-misc-0.1.4 scipy-1.10.1 scvi-tools-0.16.4 seaborn-0.12.2 session-info-1.0.0 setuptools-59.5.0 statsmodels-0.14.0 stdlib_list-0.9.0 tensorboard-2.14.0 tensorboard-data-server-0.7.1 tensorstore-0.1.41 texttable-1.6.7 threadpoolctl-3.2.0 tokenizers-0.13.3 toolz-0.12.0 torch-1.13.0 torchmetrics-1.0.3 torchtext-0.14.0 tqdm-4.66.1 transformers-4.31.0 tzlocal-5.0.1 umap-learn-0.5.3 urllib3-1.26.16 werkzeug-2.3.7 widgetsnbextension-4.0.8 wrapt-1.15.0 xxhash-3.3.0 yarl-1.9.2

We used pytest as developing dependencies. Did not pack it into the pip release since it is only used in the tests. So are you trying to run these unit test scripts? If you wish to do so, just run pip install pytest and then it should work.

@bzip2: I try to contact you in private, what is your email address?
Can you comment here and after remove the comment?
Thanks in advance.

I think I have a similar problem I jsut go in circles about which packages are causing which problems:
piroot@74c6468c7528:/usr/src/app# pip show flash_attn Name: flash-attn Version: 1.0.4 Summary: Flash Attention: Fast and Memory-Efficient Exact Attention Home-page: https://github.com/HazyResearch/flash-attention Author: Tri Dao Author-email<EMAIL_ADDRESS>License:  Location: /opt/conda/lib/python3.10/site-packages Requires: einops, packaging, torch Required-by: scgpt
but then
`>>> import flash_attn

import scgpt
/opt/conda/lib/python3.10/site-packages/scgpt/model/model.py:19: UserWarning: flash_attn is not installed
warnings.warn("flash_attn is not installed")
Traceback (most recent call last):
File "", line 1, in
File "/opt/conda/lib/python3.10/site-packages/scgpt/init.py", line 18, in
from . import model, tokenizer, scbank, utils, tasks
File "/opt/conda/lib/python3.10/site-packages/scgpt/tokenizer/init.py", line 1, in
from .gene_tokenizer import *
File "/opt/conda/lib/python3.10/site-packages/scgpt/tokenizer/gene_tokenizer.py", line 11, in
import torchtext.vocab as torch_vocab
File "/opt/conda/lib/python3.10/site-packages/torchtext/init.py", line 12, in
from . import data, datasets, prototype, functional, models, nn, transforms, utils, vocab, experimental
File "/opt/conda/lib/python3.10/site-packages/torchtext/datasets/init.py", line 3, in
from .ag_news import AG_NEWS
File "/opt/conda/lib/python3.10/site-packages/torchtext/datasets/ag_news.py", line 12, in
from torchdata.datapipes.iter import FileOpener, IterableWrapper
File "/opt/conda/lib/python3.10/site-packages/torchdata/init.py", line 9, in
from . import datapipes
File "/opt/conda/lib/python3.10/site-packages/torchdata/datapipes/init.py", line 9, in
from . import iter, map, utils
File "/opt/conda/lib/python3.10/site-packages/torchdata/datapipes/iter/init.py", line 121, in
from torchdata.datapipes.iter.util.sharding import (
File "/opt/conda/lib/python3.10/site-packages/torchdata/datapipes/iter/util/sharding.py", line 9, in
from torch.utils.data.datapipes.iter.sharding import SHARDING_PRIORITIES
ModuleNotFoundError: No module named 'torch.utils.data.datapipes.iter.sharding'

`
although:
root@74c6468c7528:/usr/src/app# pip show torch Name: torch Version: 1.13.0 Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration Home-page: https://pytorch.org/ Author: PyTorch Team Author-email<EMAIL_ADDRESS>License: BSD-3 Location: /opt/conda/lib/python3.10/site-packages Requires: nvidia-cublas-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-runtime-cu11, nvidia-cudnn-cu11, typing-extensions Required-by: flash-attn, pyro-ppl, pytorch-lightning, scgpt, scvi-tools, torchaudio, torchdata, torchelastic, torchmetrics, torchtext, torchvision, triton root@74c6468c7528:/usr/src/app# pip show torchtext Name: torchtext Version: 0.14.0 Summary: Text utilities and datasets for PyTorch Home-page: https://github.com/pytorch/text Author: PyTorch core devs and James Bradbury Author-email<EMAIL_ADDRESS>License: BSD Location: /opt/conda/lib/python3.10/site-packages Requires: numpy, requests, torch, tqdm Required-by: scgpt

Hi @mjstrumillo , it seems to be an issue with torchtext, but the versions you showed should be compatible, https://github.com/pytorch/text#installation . what happens if you just try this in python?
import torch
import torchtext
print(torch.__version__, torchtext.__version__)
import torchtext.vocab as torch_vocab

@mjstrumillo I just had your torchtext / torchdata error issue too. I believe your problem is with the "torchdata" package (which was likely already in your package environment before trying to install scGPT) and not torchtext itself. The error No module named 'torch.utils.data.datapipes.iter.sharding' exists because torchdata expects torch>=2.0, while scgpt expects torch=1.3. You should uninstall torchdata (e.g. pip uninstall torchdata). If you try importing scGPT again after, your error about torchdata should be gone.

ok, I trouble shot this with pip uninstall torchvision torchaudio torchdata torch
(I tried one by one, but all of this needed to go)
then pip installing scgpt (that covered the correct torch versions)
p.s. I never did install anything torch related manually, all of this was in the env
now the only remaining issue is that I do in fact have a correct version of flash_attn
pip show flash_attn
Name: flash-attn
Version: 1.0.4

then I can import it to python, but scgpt claims its not there:
>>> import flash_attn
>>> import scgpt
/opt/conda/lib/python3.10/site-packages/scgpt/model/model.py:19: UserWarning: flash_attn is not installed
  warnings.warn("flash_attn is not installed")
/opt/conda/lib/python3.10/site-packages/torchmetrics/utilities/imports.py:18: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  from distutils.version import LooseVersion
Global seed set to 0

both of the paths seems to match
>>> print(flash_attn.__file__)
/opt/conda/lib/python3.10/site-packages/flash_attn/__init__.py
>>> print(scgpt.__file__)
/opt/conda/lib/python3.10/site-packages/scgpt/__init__.py

so I honestly dont know what to do now...

I checked what actually gets imported in model.py:19 and its from flash_attn.flash_attention import FlashMHA
when running jsut that line, I get:
>>> from flash_attn.flash_attention import FlashMHA
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.10/site-packages/flash_attn/flash_attention.py", line 7, in <module>
    from flash_attn.flash_attn_interface import flash_attn_unpadded_qkvpacked_func
  File "/opt/conda/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 5, in <module>
    import flash_attn_cuda
ImportError: /opt/conda/lib/python3.10/site-packages/flash_attn_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda20CUDACachingAllocator9allocatorE

so Im reinstalling the flash_attn, probably wrong compiled somehow?...

I realise last 3 comments are all mine, but that worked.
so installing flash_attn, then scgpt, then uninstalling all torch dependencies, then installing scgpt, then uninstalling flash_attn, then recompiling flash_attn from scratch.
I guess that kind of makes sense, but oh man....

Hi @mjstrumillo glad to know it finally works. Sorry about the mess in your installation process, and admittedly, it was supposed to be easier. Reading from the above comments, it looks like there were two issues during your process, just to summarize them here and wish to provide some insights:

The torchtext is indeed picky about the version of torch.  We thus have set the specific versions in the pyproject.toml file, so in theory pip should handle this very well. It is still unclear to me why your early installation encountered conflict. https://github.com/bowang-lab/scGPT/issues/58#issuecomment-1710854116 Thankfully, after reinstalling this seems worked out fine.

The "undefined symbol: xxx" you mentioned here https://github.com/bowang-lab/scGPT/issues/58#issuecomment-1719872047 indicates incompatibility between flash-attn and the environment. You can find similar issues in the flash-attn repo. Glad to know it worked after reinstallation.

Reading your last comment

I realise last 3 comments are all mine, but that worked. so installing flash_attn, then scgpt, then uninstalling all torch dependencies, then installing scgpt, then uninstalling flash_attn, then recompiling flash_attn from scratch. I guess that kind of makes sense, but oh man....

I just realized some issue may be related to the order of the commands. For example, if you start with pip install flash_attn, it probably will select very new versions of flash_attn and pytorch>2.1, and then the later command of pip install scgpt will again change the versions and cause the incompatibility.

In summary, I think the whole installation could be optimized and completed just with the following two lines:
pip install torch==1.13.0
pip install scgpt "flash-attn<1.0.5"
# As of 2023.09, pip install may not run with new versions of the google orbax package, if you encounter related issues, please use the following command instead:
# pip install scgpt "flash-attn<1.0.5" "orbax<0.1.8"

I've personally tested it and updated the README with this. So thank you very much for sharing your case, and it greatly helps optimize this. Feel free to try the above commands if you need a fresh installation at any time, and please let me know your feedback.
We are also working on updating to pytorch>2.0.1 recently, where flash-attn has been better integrated in pytorch itself. So we expect the installation will be even more simplified and hope to provide the new updates soon

also, once I installed faiss, then entire installation is broken (Im trying to follow the https://github.com/bowang-lab/scGPT/blob/main/tutorials/Tutorial_Reference_Mapping.ipynb).
once I installed faiss, and I can import it, the numba versioning just gives up

Hi, do you mean you can still import the packages, and there is only some warning about the numba version? I would think the tutorial may still work if that's the case.
The numba is not used in tutorial applications.
Just to share the experience of installing faiss, I personally tried their conda-forge installation option and it worked fine.

yes, I installed via conda install -c pytorch -c nvidia faiss-gpu=1.7.4 mkl=2021 blas=1.0=mkl
as that was the only option it worked (the conda forge did not compile a workign version). so I have a wokring faiss, I tried pip uninstalling scgpt, then pip install, but nothing changed. its the same numba version as before, which makes me question all the depended packages - technically only these are changing:
The following NEW packages will be INSTALLED:
  cudatoolkit        nvidia/linux-64::cudatoolkit-11.4.1-h8ab8bb3_9
  faiss-gpu          pytorch/linux-64::faiss-gpu-1.7.4-py3.10_hc0239a3_0_cuda11.4
  libfaiss           pytorch/linux-64::libfaiss-1.7.4-h13c3c6d_0_cuda11.4

The following packages will be UPDATED:

  ca-certificates                     2023.01.10-h06a4308_0 --> 2023.08.22-h06a4308_0
  certifi                          2023.5.7-py310h06a4308_0 --> 2023.7.22-py310h06a4308_0
  conda                              23.3.1-py310h06a4308_0 --> 23.7.4-py310h06a4308_0
  openssl                                 1.1.1t-h7f8727e_0 --> 1.1.1w-h7f8727e_0

The following packages will be DOWNGRADED:

  intel-openmp                      2023.1.0-hdb19cb5_46305 --> 2021.4.0-h06a4308_3561
  mkl                               2023.1.0-h6d00ec8_46342 --> 2021.4.0-h06a4308_640
  mkl-service                         2.4.0-py310h5eee18b_1 --> 2.4.0-py310h7f8727e_0
  mkl_fft                             1.3.6-py310h1128e8f_1 --> 1.3.1-py310hd6ae3a3_0
  mkl_random                          1.2.2-py310h1128e8f_1 --> 1.2.2-py310h00e6091_0
  numpy                              1.24.3-py310h5f9d8c6_1 --> 1.24.3-py310hd5efca6_0
  numpy-base                         1.24.3-py310hb5e798b_1 --> 1.24.3-py310h8e6c178_0

Based on your updated comment, https://github.com/bowang-lab/scGPT/issues/58#issuecomment-1720267290, it looks like that is related to scanpy instead of faiss.
You can try in your python env to check if this is the cause for the issue
import scanpy

Which version of scanpy do you use? Can you try pip install scanpy==1.9.1, this is the version we tested with as indicated in https://github.com/bowang-lab/scGPT/blob/main/pyproject.toml . Usually, the pip should handle the dependencies and find compatible versions of scanpy with numba, but it looks like not the case in your installation.

so I can use scgpt when using scanpy 1.9.4 (that was somehow installed automatically in the process, I didnt manually install it). If with that I isntall faiss, the above issue raises.
I downgraded to scanpy 1.9.1 and that fails importing both, scanpy and scgpt, with the following errors:

import scanpy
Traceback (most recent call last):
File "", line 1, in
File "/opt/conda/lib/python3.10/site-packages/scanpy/init.py", line 16, in
from . import plotting as pl
File "/opt/conda/lib/python3.10/site-packages/scanpy/plotting/init.py", line 1, in
from ._anndata import (
File "/opt/conda/lib/python3.10/site-packages/scanpy/plotting/_anndata.py", line 28, in
from . import _utils
File "/opt/conda/lib/python3.10/site-packages/scanpy/plotting/_utils.py", line 35, in
class _AxesSubplot(Axes, axes.SubplotBase, ABC):
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
import scgpt
Traceback (most recent call last):
File "", line 1, in
File "/opt/conda/lib/python3.10/site-packages/scgpt/init.py", line 18, in
from . import model, tokenizer, scbank, utils, tasks
File "/opt/conda/lib/python3.10/site-packages/scgpt/model/init.py", line 8, in
from .generation_model import *
File "/opt/conda/lib/python3.10/site-packages/scgpt/model/generation_model.py", line 21, in
from ..utils import map_raw_id_to_vocab_id
File "/opt/conda/lib/python3.10/site-packages/scgpt/utils/init.py", line 1, in
from .util import *
File "/opt/conda/lib/python3.10/site-packages/scgpt/utils/util.py", line 14, in
import scib
File "/opt/conda/lib/python3.10/site-packages/scib/init.py", line 8, in
from . import integration, metrics, preprocessing, utils
File "/opt/conda/lib/python3.10/site-packages/scib/integration.py", line 7, in
import scanpy as sc
File "/opt/conda/lib/python3.10/site-packages/scanpy/init.py", line 16, in
from . import plotting as pl
File "/opt/conda/lib/python3.10/site-packages/scanpy/plotting/init.py", line 1, in
from ._anndata import (
File "/opt/conda/lib/python3.10/site-packages/scanpy/plotting/_anndata.py", line 28, in
from . import _utils
File "/opt/conda/lib/python3.10/site-packages/scanpy/plotting/_utils.py", line 35, in
class _AxesSubplot(Axes, axes.SubplotBase, ABC):
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

I checked all, scanpy 1.9.0, 1.9.1, 1.9.2, 1.9.3 and 1.9.4 - after installation of faiss, I cannot import any of these due to the above error, so neither can I use scgpt. But I can use faiss, so thats something ;)

yes, I installed via conda install -c pytorch -c nvidia faiss-gpu=1.7.4 mkl=2021 blas=1.0=mkl as that was the only option it worked (the conda forge did not compile a workign version). so I have a wokring faiss, I tried pip uninstalling scgpt, then pip install, but nothing changed. its the same numba version as before, which makes me question all the depended packages - technically only these are changing: The following NEW packages will be INSTALLED:
  cudatoolkit        nvidia/linux-64::cudatoolkit-11.4.1-h8ab8bb3_9
  faiss-gpu          pytorch/linux-64::faiss-gpu-1.7.4-py3.10_hc0239a3_0_cuda11.4
  libfaiss           pytorch/linux-64::libfaiss-1.7.4-h13c3c6d_0_cuda11.4

The following packages will be UPDATED:

  ca-certificates                     2023.01.10-h06a4308_0 --> 2023.08.22-h06a4308_0
  certifi                          2023.5.7-py310h06a4308_0 --> 2023.7.22-py310h06a4308_0
  conda                              23.3.1-py310h06a4308_0 --> 23.7.4-py310h06a4308_0
  openssl                                 1.1.1t-h7f8727e_0 --> 1.1.1w-h7f8727e_0

The following packages will be DOWNGRADED:

  intel-openmp                      2023.1.0-hdb19cb5_46305 --> 2021.4.0-h06a4308_3561
  mkl                               2023.1.0-h6d00ec8_46342 --> 2021.4.0-h06a4308_640
  mkl-service                         2.4.0-py310h5eee18b_1 --> 2.4.0-py310h7f8727e_0
  mkl_fft                             1.3.6-py310h1128e8f_1 --> 1.3.1-py310hd6ae3a3_0
  mkl_random                          1.2.2-py310h1128e8f_1 --> 1.2.2-py310h00e6091_0
  numpy                              1.24.3-py310h5f9d8c6_1 --> 1.24.3-py310hd5efca6_0
  numpy-base                         1.24.3-py310hb5e798b_1 --> 1.24.3-py310h8e6c178_0

If you can import successfully previously, but not after these changes. I think the changes to mkl and numpy caused errors with numba or scanpy.
So I have two options in mind,

can you try:

pip install scanpy==1.9.1 anndata

you probably want to relax the requirement of faiss installation, I think even if the conda forge didn't work, the faiss-gpu can be installed via

conda install -c pytorch faiss-gpu

This way you give conda more freedom to choose the versions.
Also, I noticed the cudatookit changed after the installation? It may influence the running flash-attn. Actually, if you only need to run the reference mapping for cells less than tens of thousands, like in the tutorial notebook, then the faiss-cpu can be fast enough

Also, let me see if I can remove the dependency of scanpy at least from the core package

the conda install -c pytorch faiss-gpu faills due to old python :
root@fdc8cbb72558:/usr/src/app# conda install -c pytorch faiss-gpu
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: /
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

  - faiss-gpu -> python[version='>=3.11,<3.12.0a0']

Your python: python=3.10

If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.

The following specifications were found to be incompatible with your system:

  - feature:/linux-64::__glibc==2.31=0
  - feature:|@/linux-64::__glibc==2.31=0
  - faiss-gpu -> __glibc[version='>=2.17,<3.0.a0']
  - faiss-gpu -> libgcc-ng[version='>=11.2.0'] -> __glibc[version='>=2.17']

Your installed version is: 2.31

Im so sorry about this :D

so I guess it comes down to finding a version of faiss, that will be not new too much for python 3.10 and then not too old for the scanpy?

I see. I will try to test further with python 3.10 and let you know. I think there should be a solution probably with a fresh installation

I installed faiss-cpu both form conda-forge and then pip install just to try. None of these solved the aforementioned issue and come back to that numba blabber. I will try to build from source manually tomorrow?

so I finally got to the bottom, that faiss has the python requirement   - faiss-gpu=1.7.4 -> python[version='>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0']
so since Im using python 3.10... I tried 5 older faiss verisons, and they all have the 3.10 contraint...Im starting a new docker with python 3.8

here's the only combination that worked for me:
ubuntu 20
python 3.8.18
cuda 11.4 [devel: nvidia/cuda:11.4.3-devel-ubuntu20.04]
faiss 1.7.4. [conda install -c conda-forge faiss-gpu]
requirements.txt:
packaging
wandb
gseapy
maybe that will save someone days of playing with the combinations - but the bottleneck was faiss.


feat: DBEST stack, Floating UI

Added my own Solidjs project, and a great project I am using :)

Thank you @itsyoboieltr


Rpi4 cfg file

Could someone have a look at a raspi4 cfg file? Tried the script using a pi4 but without succes. Would like to root my Toon, but only have a raspi4 :)!
Thanks!
Br,
Bart

I think that would be possible, but I don't have a Toon anymore and I don't have a Pi 4 either. It's probably best to try and find an OpenOCD config somewhere on the internet and use that (the config is not all that specific to this script). If you find one please let me know and I'll add it

Hi Marten,
Made one based on the rpi3 cfg file. Everything worked fine for me ;)!
Thanks,
rpi4.cfg.txt

@Bart-1992 Thanks for the file, I've included it


Fixed Cilium CLI fatal error: concurrent map read and map write

Fixes #35222
Fixed Cilium CLI fatal error: concurrent map read and map write

/test

@tommyp1ckles ping! 🏓
i asked @chaunceyjiang to add a bit more description in the commit message about how this fixes the issue 🚀🙏

/test

@chaunceyjiang We had to revert this pull request in https://github.com/cilium/cilium/pull/35391 as the end-to-end upgrade test was failing in several configurations (cf. https://github.com/cilium/cilium/actions/runs/11328217910).

as the end-to-end upgrade test was failing in several configurations (cf.

@pchaigno 🙀，let me take a look at this issue. The end-to-end upgrade test passed successfully when I merged this PR.


improve vcf parsing speed

This PR greatly improves the speed at which VCF file are parsed. Appending rows to a Pandas DataFrame is very slow, appending to a list of rows and then loading them into a DataFrame is much faster.

Codecov Report

Merging #32 into develop will increase coverage by 1.37%.
The diff coverage is 85.71%.

@@             Coverage Diff             @@
##           develop      #32      +/-   ##
===========================================
+ Coverage    88.68%   90.06%   +1.37%
===========================================
  Files            5        5
  Lines         1052     1097      +45
  Branches       196      196
===========================================
+ Hits           933      988      +55
+ Misses          73       62      -11
- Partials        46       47       +1

Impacted Files
Coverage Δ

src/snps/io.py
85.62% <85.71%> (+0.39%)
:arrow_up:

src/snps/__init__.py
94.04% <0%> (+0.25%)
:arrow_up:

src/snps/utils.py
87.32% <0%> (+0.36%)
:arrow_up:

src/snps/resources.py
94.33% <0%> (+4.46%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ead7c4b...4e128e8. Read the comment docs.


Remove null?

null is redundant with undefined. The only reason it is currently kept is compatibility with platform APIs the user may want to send a null value to. A compatibility solution for this use case should be studied instead, and null should be removed from the value domain.

Now that null has become the subtype of all objects, it is no longer redundant. null is different from both undefined and unknown, and all three must be kept.

See 047e2d4.


Removed store apps still showing in Search command +  error code could not be found

Brief description of your issue

even after winget source update This particular unofficial Github Desktop app was removed from store like 7 months ago, still showing in Search command, installing it shows a vague error code
An unexpected error occurred while executing the command:  the text associated with this error code could not be found
Expected behavior
I expect removed apps to be not shown up in Search command, even if they show up , installing them should say it was removed from store(in this case)/not available in my country or something like this.
Environment
Windows Package Manager v0.2.10771 Preview
Copyright (c) Microsoft Corporation. All rights reserved.

Windows: Windows.Desktop v10.0.19042.870
Package: Microsoft.DesktopAppInstaller v1.11.10771.0

Thanks for reporting this @ecovio1. It's likely due to our manually curated list of packages for the experimental store source.

This app has been removed from our experimental store source. Thanks for the feedback. Please feel free to reopen if this app is still in the list after updating the msstore source. Thanks.


Navbar text cannot center

after bootstrap v4.0 beta navbar text cannot center. how to fix this issue. I tired so many times but i cannot do that.

I found solution


Running on macOS 10.13.2

Hi! First of all, I would like to congratulate you for this amazing work! Despite the fact your software is the only one I can use to run this cheap engraver on Mac, it certainly is much better than the original windows only.
After digging with the drivers and protocol versions, I could engrave with my new DK-8-KZ.
Only things I miss so far is the engrave progress indication (absent from version 4.0.0 Beta, as indicated in release notes) and the directly text insert tool.
So, I would like to suggest you to include the option, on EzGraver (if not very difficult), to insert text on the image area, so for the times when all is needed is to engrave text, I can do it directly from EzGraver, instead of create the image of the text on another program and then import it to be burn.
Another suggestion is to include rulers on image area, to it's possible to get an approximate idea of the final size of the engrave job.
Thank you for your great work so far in this amazing piece of software!
Best regards,
Harlley

@harlleysathler I just brought NEJE DK-BL1500mw It work via their Android and Windows app but I'm on macOS so EzGraver should be great choice but I just can't make it work.
The question is how you make it work? Any step or screenshot would be nice, Thanks


[eas-cli] check if package.json is inside git repo

Checklist

[ ] I've added an entry to CHANGELOG.md if necessary. You can comment this pull request with /changelog-entry [breaking-change|new-feature|bug-fix|chore] [message] and CHANGELOG.md will be updated automatically.

Why
if package.json is outside git repo, eas.json is also created there and git add is failing
How
check projectDir is outside of vcs dir
Test Plan
reproduced a case described in why section and tun local build

/changelog-entry chore Add error message when package.json is outside git repoistory


Adding metrics exporter

Adding typing for most methods
Adding param docs for protocol
Fixing issue with error being triggered when options have unknown property
Fixing issue when no spans are send to exporter, it was storing empty list in localStorage and returning wrong result
Fixing issue with condition to release events from localStorage, missing cast was trowing exception
Fixes #24

Codecov Report

Merging #42 into master will increase coverage by 0.24%.
The diff coverage is 86.95%.

@@            Coverage Diff             @@
##           master      #42      +/-   ##
==========================================
+ Coverage   93.54%   93.78%   +0.24%
==========================================
  Files           7        8       +1
  Lines         511      563      +52
  Branches       72       80       +8
==========================================
+ Hits          478      528      +50
  Misses         26       26
- Partials        7        9       +2

Impacted Files
Coverage Δ

azure_monitor/src/azure_monitor/protocol.py
89.16% <100%> (+1.98%)
:arrow_up:

azure_monitor/src/azure_monitor/utils.py
89.47% <100%> (+0.9%)
:arrow_up:

azure_monitor/src/azure_monitor/trace.py
98.83% <75%> (+3.54%)
:arrow_up:

azure_monitor/src/azure_monitor/metrics.py
81.08% <81.08%> (ø)

azure_monitor/src/azure_monitor/exporter.py
97.36% <87.5%> (+0.07%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4613919...82d7a39. Read the comment docs.


Add dump_format=json option

This adds the option to dump as JSON.
The bindings for python, R, and java have hardcoded the format as "text" to retain the earlier usage.
Related issue is at https://github.com/dmlc/xgboost/issues/1722

Um, what's up with Travis ? The R build's bootstrap seems to be failing when downloading R

Here's an example of the JSON it produces with the mushroom example:
[
  { "nodeid": 0, "split": "odor=pungent", "yes": 2, "no": 1, "children": [
    { "nodeid": 1, "split": "stalk-root=cup", "yes": 4, "no": 3, "children": [
      { "nodeid": 3, "split": "stalk-root=missing", "yes": 8, "no": 7, "children": [
        { "nodeid": 7, "leaf": 1.899 },
        { "nodeid": 8, "leaf": -1.94737 }
      ]},
      { "nodeid": 4, "split": "bruises?=no", "yes": 10, "no": 9, "children": [
        { "nodeid": 9, "leaf": 1.78378 },
        { "nodeid": 10, "leaf": -1.98135 }
      ]}
    ]},
    { "nodeid": 2, "split": "spore-print-color=orange", "yes": 6, "no": 5, "children": [
      { "nodeid": 5, "split": "stalk-surface-below-ring=silky", "yes": 12, "no": 11, "children": [
        { "nodeid": 11, "leaf": -1.98546 },
        { "nodeid": 12, "leaf": 0.938776 }
      ]},
      { "nodeid": 6, "leaf": 1.87097 }
    ]}
  ]},
  { "nodeid": 0, "split": "odor=pungent", "yes": 2, "no": 1, "children": [
    { "nodeid": 1, "split": "bruises?=no", "yes": 4, "no": 3, "children": [
      { "nodeid": 3, "leaf": 1.14608 },
      { "nodeid": 4, "split": "gill-spacing=crowded", "yes": 8, "no": 7, "children": [
        { "nodeid": 7, "leaf": -6.87995 },
        { "nodeid": 8, "leaf": -0.106592 }
      ]}
    ]},
    { "nodeid": 2, "split": "spore-print-color=orange", "yes": 6, "no": 5, "children": [
      { "nodeid": 5, "split": "gill-size=narrow", "yes": 10, "no": 9, "children": [
        { "nodeid": 9, "leaf": -0.0930658 },
        { "nodeid": 10, "leaf": -1.15261 }
      ]},
      { "nodeid": 6, "leaf": 1.00423 }
    ]}
  ]}
]

Thanks for the contribution. For the sake of backward compatiblity. Let us not change the DumpModel function, but instead create a new funciton DumpModelEx, with the additional option. and redirect the legacy dump function to it.

That makes sense, will do

Done. I've added a new DumpModelEx function in the c_api and the bindings continue to use that.

Please update a python wrapper, and add a testcase, then we can merge it in

Could you elaborate on what you mean by updating the python wrapper? Do you
mean add a new function to the python wrapper to be able to dump JSON? Or
use JSON instead of the parser currently being used in get_scores()?
If you'd like to have a real-time conversation for clarity, maybe we can
discuss on Gitter.
On 04-Nov-2016 12:35 am, "Tianqi Chen"<EMAIL_ADDRESS>wrote:

Please update a python wrapper, and add a testcase, then we can merge it in
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dmlc/xgboost/pull/1726#issuecomment-258243042, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACGUp9cF7Tj4MgFxMw1OD6mPoi9EPoQbks5q6jCLgaJpZM4Kl-IC
.

enhance getdump https://github.com/dmlc/xgboost/blob/master/python-package/xgboost/core.py#L1041 to enable a format option

@tqchen Done

Thanks this is merged!

this is a great addition @AbdealiJK!  are you able to update the R wrapper as well?

@tonydifranco I've never used the R wrapper, so if you could add that in a PR - it would be good. Else I'll look into it when I'm more free and have time to experiment ?

Hi all,
Thanks a lot for this feature (dump a model as JSON). Very useful.
Could you please tell if it is possible to increase the precision of the floats that are dumped (getting a maximal one is awesome ;) ?
Thanks in advance
Antoine

Currently it uses the same formatting that the text format uses - 6 digits. As the data type used is float for most stats in XGBoost 6 is already the max precision I think

@AbdealiJK thanks for your feedback.
I am trying to translate the prediction of xgboost models into a SQL code, like this
https://github.com/antoinecarme/sklearn2sql-demo/blob/master/sample_outputs_round_8/XGBClassifier/iris/oracle/demo2_XGBClassifier_oracle.sql
and the loss of precision makes that I have small differences between python version and SQL version.
I really need to get something close to a serialized form (the maximum precision). This is just an extra parameter to be added (to be confirmed).
Should I add an issue for this task ? I know of a developer who may be interested in implementing it (myself ;)
Cheers
Antoine

@antoinecarme you may use a quick hack I've suggested in #1204 as a temporary workaround.
@AbdealiJK the default iostream precision (regardless of whether it's text or json format) is often not sufficient for reasonable conversions b/w decimal and binary.
This issue was brought up a few times, e.g., in addition to #1204 see also #1750, #2077. In #2268 I've proposed a mechanism allowing to easily parameterize DumpModel (including the precision), but I haven't heard any feedback from @tqchen .

@khotilov
This hack is what I did locally. I am developping a tool tha translates xgboost models in SQL code by working on the dump result. The hack cannot however be distrivbuted without a merge ;). I can not ask all users to change xgboost this way !!!
Thanks for pointing all these issues.

Score (prediction) calculated by xgboost does not match what is expected from tree model generated by xgboost ==> #1204
xgb.dump needs argument for number of digits ===> #1750
Dumpmodel has insufficient precision for dumping floats ==> #2077
Refactor passing of parameters to DumpModel ==> #2268

The most generic and useful ones to implement are IMHO, the #2268 and #1750
There is a need for a higher-level decision (@tqchen ?) to define the priorities here.


Hash files using regexp

Hi,
I see that using regexp in hash files like virtual, transport and canonical is not suported due the augeas len. I don't know enough about augeas to implement it. Is there any plan to implement that in the augeas or any alternative to achive this?
Thanks.

ping @raphink

+1 ping @raphink ?

I had a look at trying to get regexes to work via augeas and gave up in the end, instead I ended up doing the following:
profile::postfix::virtual:
    - "/rob.*/ root"

$virtual = lookup('profile::postfix::virtual',
{
  value_type    => Array,
  default_value => [],
  merge         => 'unique',
})

# A bit of a hack to work around the fact augeaus really doesn't want to
# use regexes in key names
#
# Note the double quote within double quote (which works here)
postfix::hash { '/etc/postfix/virtual':
  content => "${ $virtual.join("\n") }\n",
}

@jordiclariana 	@OlivierLM 	do you have a specific example of the syntax you'd like to pass?

As an example, this virtual map shall work :
<EMAIL_ADDRESS>sympabounce:
/^bounce\+.*/ sympabounce:
<EMAIL_ADDRESS>sympa:

@OlivierLM 	ok, and what would the Puppet code you'd expect to use for that?

  postfix::transport<EMAIL_ADDRESS>    ensure      => present,
    destination => 'sympabounce',
  }

Don't know hox to try this easily :(

Fixed in https://github.com/camptocamp/puppet-postfix/pull/225


Support arrays for LatLng objects

Hi,
It would be useful to support arrays whenever a LatLng object is expected, according to what Leaflet offers.
Typically, passing an array as the center prop of the map currently works, but it does not work for Marker position, which has to be an object with a lat and lng key.
Also, for bounds prop on Map, one has to pass a L.latLngBounds object whereas leaflet should support passing an Array directly.
Thanks!

Hi @Phyks,
v0.0.46 should fix this for marker, map center and circle center.

Thanks!


Add basic Unit Test for PutArtifactType logic

introduce basic go test Unit Test
for testing logic of gRPC PutArtifactType

Description

How Has This Been Tested?
$ go test -timeout 30s -run PutArtifactType github.com/opendatahub-io/model-registry/internal/server -v
=== RUN   TestPutArtifactType
    grpc_server_test.go:63: Should PutArtifactType: type_id:1
    grpc_server_test.go:66: Should PutArtifactTypeResponse.TypeId an ID > 0: type_id:1
--- PASS: TestPutArtifactType (0.01s)
=== RUN   TestPutArtifactTypeNoNameFailure
    grpc_server_test.go:95: Should error on PutArtifactType with Request missing Name: rpc error: code = Internal desc = server panic: runtime error: invalid memory address or nil pointer dereference
--- PASS: TestPutArtifactTypeNoNameFailure (0.00s)
PASS
ok  	github.com/opendatahub-io/model-registry/internal/server	0.608s

Merge criteria:

[x] The commits and have meaningful messages; the author will squash them after approval or will ask to merge with squash.
[x] Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
[x] The developer has manually tested the changes and verified that the changes work (see above)

I don't get why we're using "should ..." here every time. Some of them don't make any sense to me.

👉  https://github.com/opendatahub-io/model-registry/pull/18#discussion_r1334532493

Also: please use the imperative on the commit message.

I have used title within 50 chars, newline, description; I am afraid I did not understand what you meant here. Do you have an article to point me to, please.

Also: please use the imperative on the commit message.

👉  21743b5

The last commit could've been a fixup. If you do squash this, please remove that message.

Re-use this as a unit test for the DB service handler. That way it's a quick and easy unit test and we leave integration testing to python scripts.

I think we discussed about having small unit test for "surface contacts" or "core capabilities" and serving as smoke testing was helpful.
This could have served as a small smoke/unit test in that sense.
It seems however that there is no way to have a change-of-mind about this.
Noted. ✍
Rejected with https://github.com/opendatahub-io/model-registry/pull/18#pullrequestreview-1644747578 , closing PR.


where are the complex file instructions and filters

I need to work out why condensin complex (GO:0000796) is filtered. I think it is to do with the removal of the nuclear versions of the complex, I suspect this is an ID I used to filter.

Hi Val.
It's configured in the "file_exports" section of the config file: https://github.com/pombase/pombase-config/blob/master/website/pombase_v2_config.json#L9161
Have a look in "excluded_terms"
Sorry I haven't documentation this part of the config file.  I've added an issue about that:

https://github.com/pombase/website/issues/1961

Sorry you didi tell mer recently, and I forgot it moved into the config.

fixed


Fix the Nesterov bug

The nesterov code has a bad bug. It does an intermediate parameter update and then calculate the gradient based on these new weights. However, the real update is also adding the same parameter updates!??
There are several solutions to this.

Maybe this is simpler then I think.... If I read the code right is is simply just the order of the operations that is different
Without nesterov (but with momentum) we do:

Calculate the gradient.
Update the velocity: v_new = momentum * v - lr * g
Update network weights: w_new = w + v_new

With Nesterov we do:

Update (interim) network weights:  w_new = w + momentum * v
Calculate the gradient (g) based on these new weights.
Update the velocity: v_new = momentum * v - lr * g
Update network weights: w_new = w + v_new

however w_new is then expressed: w_new = w + momentum * v - lr * g, and momentum * v has already been applied in the interim update. That means, I can probably do this:

Update (interim) network weights:  w_new = w + momentum * v
Calculate the gradient (g) based on these new weights.
Update the velocity: v_new = momentum * v - lr * g
Update network weights: w_new = w - lr * g

(I think so.... however I have to think to make this general. Maybe I should split this code into several files?)

Yes! It is probably fu*king easy. I just need to keep track of two parameter arrays.

momentum * velocity
-learning_rate * g

Wish me good luck!

24a2fd2 seems to fix the issues, but make a good test case and compare to a known implementation ie. Keras.


About a Third-Party (external) authorization server interface

Dear @aliev and cons
Your library is amazing but, as you indeed mentioned is still missing some extra-docs for guidance on implementation.
I just would like to know If you have any example of an Third party implementation interface like typical Google, Microsoft auth ... and so on. The level of class abstractions make me go ahead so slow.
I suppose that the key is in creating an interface of Storage class and methods, to create the correct instance for each desired third party chosen.
Please any help would be amazing man.

Hi @TatanParker ! aioauth is only server implementation of oauth2.0 protocol. I think authlib is exactly what you need: https://docs.authlib.org/en/latest/client/starlette.html

Yes man. You are right. However your framework is fantastic to implement oauth2 protocol for clients. I get the point. Thanks


Consider renaming Regions2D.mask and Regions2D.transect_mask?

From a comment to #162:

This is a somewhat philosophical comment (and we should probably not touch anything right now):
We have Regions2D.mask and Regions2D.transect_mask. Conceptually they do the same thing, but Regions2D.mask uses data stored with the Regions2D object (the region annotation) and da_Sv to generate masks, and Regions2D.transect_mask uses transect sequence-related dicts and da_Sv that are both external of the Regions2D object.
My other comment is about potential method name changess -- seems that if we were to make the names clear and specific wrt the actual functionality, we would change Regions2D.mask to Regions2D.region_mask.

@leewujung I agree with this. In addition, should we also change lines.mask to lines.bottom_mask?


Fix yarn install

Description
In the integration of Foundry and Governance contracts branch the command yarn install in the root folder is not working (branch: core/foundry)
Motivation
Integrate Governance and Foundry
Implementation details
Not needed

We do not use foundry


Is the plugin currently working?

Hi I was wondering is there any way to 'reset' Meta Data menu? it used to work before,  I don't know if its installing other plugins or I opened a couple of other vaults with the same plugin, but right now it doesn't work at all.
I have tried moving the template folder, adding new file classes from scratch, deleting and reinstalling the plugin and it doesn't work, everytime I restart it gives me an error
Plugin no longer active
The plugin that created this view (FileClassView__All) has gone away
Any idea how I can do this?

Hi, you can try this:

open the console
go to the Application tab
deploy the IndexedDB section
look for every metadata_menustorage item
delete them by clicking on it and click on the "Delete database button" in the main section


[BUG] Update Link in README

🙁 Actual behavior
https://eddiehubcommunity.github.io/api/ -> 404
🙂 Expected behaviour
https://api.eddiehub.io/
🔢 Steps to Reproduce the Problem
No response

hii , i want to work on this ;;

Seems it not fixed yet. So, I want to fix this @eddiejaoude if you allow.

https://github.com/EddieHubCommunity/api/pull/294


Updated links for failing browser tests

Added "www" to Loan Estimates link
Updated "closing costs" link to reflect the page redirect

@cfarm please review

Works on my local, merging


Vim-jsx-pretty thinks "<=" is start of jsxRegion.

I have this React code in a .jsx file where an if-statement breaks the highlighting.
class Foo extends React.Component {
   constructor(props) {
      super(props);

      if( 0 <= 1) { // <-- Thinks "<=" is start of jsxRegion. Highlighting is wrong after this line.
         console.log("bar");
      }
   }

   render() {
      return (

Hello World

      )
   }
}

Thank you reporting!!

Sorry, for bad format on the code. I made it better now.

Fixed!

That was fast. Thanks!


Inline html does not work

The jade documentation indicates you can include inline html
https://github.com/visionmedia/jade#a6-10
Doing same with pyjade results in html being escaped and appearing as html code
Django==1.5.1
pyjade==2.0.2

this was really bugging me, see, I am trying to write a website, and I wanted to read a post from my database, of course I wanted that post to have links in arbitrary places, so I thought I'd just store the post's text as pure html and put that on the website! In regular jade I would've done something like
section.content!= post.text

however, unfortunately pyjade doesn't allow you to do that (yet).
And this was frustrating me a lot, I even tried to do some sort of html parser, so I would store the post in some weird format, to be read by python and made into some object that jade would read and then output, like, if ["p"] output p= something like that. That is easier said than done, and I later found a slightly better solution, simply have a small javascript function that "unescapes" exactly what I need (and only there of course).
Then of course, just now, I figured I could do this a lot easier and clean, by using django's templating syntax... so now I tried
section.content {{ post.text|safe }}

and that seems to work fine!
Hope this saves someone else in my shoes some trouble and hair pulling.

You can use in your pug/jade file the filter | n
| n  filter ensures they are not escaped.
so if you store your html and you want to render it then you can do {post.text | n} in your pug/jade file.
http://stackoverflow.com/questions/12137421/mako-escaping-issue-within-pyramid


Captcha and input be reset after submission

Updated the code according to the recent change in the main.
Closes #27

Kudos great work!!


Merge in utils and sdk

No submodules anymore!
Utils were copied as-is into the backend directory.
SDK was copied into the backend directory, until it can be installed from PyPI or at least from its git repo.
The liquidity was copied as is, with utils and SDK as subdirectories, similarly to backend.

This is a workaround to make the wallet functional again asap. Once the dust settles, the external dependencies should be fixed.

What’s the plan for updating the sdks and publishing them? Can we get that done asap? @ericnakagawa

Fixing the permissions in client-sdk-python would be good enough for the transition period.


Adjust testing code to take proper URL of the files located at the HTTP import server.

Introduce a new test utility function called GetUrl which will be used in order to take proper URL of testing files located at the HTTP import server. That is because the HTTP server can be deployed to a configurable namespace and it's FQDN will be affected by that.
Signed-off-by: Igor Bezukh<EMAIL_ADDRESS>What this PR does / why we need it:
Functional tests are using testing infrastructure pods. These pods can be deployed to a namespace other then the traditional "kubevirt" namespace. Therefore The FQDN of the HTTP server cannot be determined in compile time using constants. The suggested PR introduces the changes required to invoke the correct URL according to run-time parameters.
Fixes #
Test automation
Special notes for your reviewer:
Release note:
Improvement of testing infrastructure code. Calculate FQDN of HTTP import server in run-time.

/assign @omeryahud @stu-gott

/lgtm

/approve
/override continuous-integration/travis-ci/pr

/retest pull-kubevirt-e2e-k8s-1.16

/cherrypick release-0.26

/retest


[AMBARI-24947] Support for complex Add Service request in secure cluster

What changes were proposed in this pull request?
Allow using Add Service request in clusters with Kerberos.

Add security and credentials to the request syntax
Store credentials from the request, but allow the case when credential is already available (eg. from initial cluster install)
security/type is optional, but validated if specified

Kerberos descriptor is not stored yet.
How was this patch tested?
Deployed secure cluster with ZooKeeper and Infra Solr via blueprint.  Added Kafka and Ambari Metrics via Add Service request.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/4630/
Test PASSed.


live from start

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

[X] I'm asking a question and not reporting a bug or requesting a feature
[X] I've looked through the README
[X] I've verified that I'm running yt-dlp version 2023.01.06 (update instructions) or later (specify commit)
[X] I've searched the bugtracker for similar questions including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Please make sure the question is worded well enough to be understood
Hi, is "live from start" intented for youtube only or for other sites too? thanks. (It failed on live conference on ORF TV Thek.
Provide verbose output that clearly demonstrates the problem

[ ] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
No response

live from start should work for other websites besides youtube
streamlink might be a better solution for live tv.

--live-from-start               Download livestreams from the start.
                                Currently only supported for YouTube
                                (Experimental)

Is it safe to use --live-from-start on all sites, assuming that yt-dlp falls back to --no-live-from-start on unsupported sites?
Or could this create problems ?

No, it does not "fall back". When the requested type of stream is unavailable, it throws error

No, it does not "fall back". When the requested type of stream is unavailable, it throws error

From this log it seems the generic extractor doesn't.
What is the reason for aborting with an error instead of just issuing a warning and trying to fall back to record from the current position (rather than recording nothing) ?

From this log it seems the generic extractor doesn't.

because it was not being detected as live, obviously. That was the whole issue that was being reported...


sycl::stream gives no output

I've installed hipSycl on an OpenMP platform with the Intel compiler. Using sycl::stream inside parallel_for gives no output. The same program works correctly on Intel devcloud.
#include <CL/sycl.hpp>
#include <vector>

namespace sycl = cl::sycl;

int main(int argc, char **argv) {

  const auto global_range = 4;

  sycl::queue myQueue;
  std::cout << "Hello example running on "
            << myQueue.get_device().get_info<sycl::info::device::name>()
            << std::endl;
  // Create a command_group to issue command to the group
  myQueue.submit([&](sycl::handler &cgh) {
      sycl::stream sout(1024, 256, cgh);
      cgh.parallel_for<class hello_world>
        (
         sycl::range<1>(global_range), [=](sycl::id<1> idx) {
           sout << "Hello, World: World rank " << idx << sycl::endl;
         }); // End of the kernel function
    });       // End of the queue commands.
  return 0;
}

[staff cxx:150] make hello && ./hello
syclcc -c hello.cxx  -std=c++17 -I/opt/apps/intel19/python3_7/boost/1.69/include -O2 -g
syclcc -o hello hello.o -L/opt/apps/intel19/python3_7/boost/1.69/lib
Hello example running on hipSYCL OpenMP host device

sycl::stream is a no-magic implementation in hipSYCL that just calls printf(), so it's unlikely to be the cause of the problem.
The issue is probably that the SYCL specification does not have implicit synchronization for queue in the destructor. Because of this, your program probably just exits before all kernels have completed.
When there's no other synchronization point (e.g. host accessor, or a buffer destructor) you need to call queue::wait() in SYCL to make sure that all work is done before the program exits.

Thanks, that solved. it. Still a little surprised that even running on the host device the IO is not flushed, but I can't pretend to understand those layers, so: so be it.

Probably you are not even getting to the actual IO inside the kernel. A SYCL implementation will construct a DAG in the background and submit operations asynchronously. This means it can happen that your kernel is still enqueued and has not yet run when the application terminates. Whether you are running on host or on some device makes no difference for these mechanisms.


Missing dependency for org.apache.http

It looks like Apache HttpClient 4.5 is missing in the Maven dependencies.
I had the following error when compiling on a fresh Ubuntu machine:
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /Truck-Factor/gittruckfactor/src/aserg/gtf/model/LogCommitInfo.java:[17,35] package org.apache.http.impl.client does not exist
[ERROR] /Truck-Factor/gittruckfactor/src/aserg/gtf/util/AliasesIdentifier.java:[12,28] package org.apache.http.auth does not exist
[INFO] 2 errors

Version 4.5 seems required to get AIMDBackoffManager.
(with version 4.0:)
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project gitdownloader: Compilation failure
[ERROR] /Truck-Factor/gittruckfactor/src/aserg/gtf/model/LogCommitInfo.java:[17,35] cannot find symbol
[ERROR] symbol:   class AIMDBackoffManager
[ERROR] location: package org.apache.http.impl.client

Hi @pchaigno, this dependence is not more necessary. I updated the code removing some unused imports (fe781ff50cc85f7da22503c9393a4a350b6d023c). So, I believe that the problem was resolved.

Yep, it works. Thanks!


Unit test improvements

Implemented a set of behaviour tests that assert the correctness of an implementation by treating it as a black box.

LGTM

LGTM


version field of HadoopTuningConfig is always set to default value

http://druid.io/docs/latest/ingestion/batch-ingestion.html says that user can specify version of segments.
But, HadoopIndexTask always replaces version with DateTime of indexing start time (which is default when no version is given).
I'm not sure the behavior of HadoopIndexTask is intended or not
but I think it needs consistency between docs and implementation.

Hi Keuntae, this is likely a bug as if you provide a version explicitly in
the config, Druid should respect that version.
On Wed, Jun 15, 2016 at 7:57 PM, Keuntae Park<EMAIL_ADDRESS>wrote:

http://druid.io/docs/latest/ingestion/batch-ingestion.html says that user
can specify version of segments.
But, HadoopIndexTask always replaces version with DateTime of indexing
start time (which is default when no version is given).
I'm not sure the behavior of HadoopIndexTask is intended or not
but I think it needs consistency between docs and implementation.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/druid-io/druid/issues/3155, or mute the thread
https://github.com/notifications/unsubscribe/AAaJJQKHoMbD_PwHmR3zGs0h2xiYvjmSks5qMLuMgaJpZM4I2-x8
.

Actually this is by design when using the indexing service, the user-provided version is always ignored in favor of the internally generated version (to assure data consistency). So we should update the docs. While we're at it we might as well also document the other properties that are ignored for the indexing service:

metadataUpdateSpec
workingPath
segmentOutputPath
version

thanks for pointing this out @sirpkt.

fixed by #3557


sauce-connect: 4.3.14 -> 4.3.16

Motivation for this change

Release notes 4.3.15
Release notes 4.3.16

The latest version of sauceconnect contains several bug fixes and improvements regarding HTTP proxy.
Things done

Built on platform(s)

[x] OS X

[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

So sc is broken on NixOS (and, one would presume, on more pure Darwin systems if those still exist):
$ .../bin/sc
bash: /home/nckx/.nix-profile/bin/sc: No such file or directory
$ patchelf --print-interpreter .../bin/sc
/lib64/ld-linux-x86-64.so.2

This PR doesn't seem to break anything that wasn't already broken, so merging. Thanks!


x/build: Android emulator builders sometimes fail with: go_android_exec: no exit code: ""

For example:
https://build.golang.org/log/f8a11e4d226addf61b7bb461663e63e34b699783 (android/amd64)
https://build.golang.org/log/a30de87209a4b63ed9fc511199bfcbe2aade16e9 (android/386)
I seems to be a bug in adb. I reported it here:
https://issuetracker.google.com/issues/73230216

It seems like the upstream issue hasn't seen any activity, unfortunately. I count 6 occurrences just in the first page of build.golang.org, which makes it the most common flake on the Android builders by far.

Yes, and the issue also blocks my hope of having android emulator builders in the trybot set. If it's not too inappropriate to ask, could someone at Google prod the Android team for progress on the upstream issue?

/cc @bradfitz @ianlancetaylor @andybons


Cleaned up the views

Replaced all the try/except block with the helper function get_object_or_404, as well as changing the way we pass the event's absolute url to the template in the event_atom view. This closes #1 and closes #5.

When you try to access one of the views for a specific event and the UUID is not the right UUID format, it throws a ValueError.  We should 404, not 500.
Example: http://<IP_ADDRESS>:8000/major_event_log/event/bb08td808-6d6c-4c43-a6a8-4bc42f5d89c9.premis.xml/.

Yeah, I see what you mean. The solutions I can think of for dealing with this would be to either wrap the get_object_or_404 in a try/except block (which is what that helper function was used to replace), or doing some regex verification in the view before calling that helper function (not pretty). Is there a way I'm missing?

The helper function was not put in to replace the use of just any try/except. It was put in to replace a try/except that was being executed to do basically the exact same thing as the helper function. You could search for if there is a preferred Django way of handling things like ValueErrors that are thrown in cases like this.

Didn't find anything definitive. Saw one example of someone using the try/except block. Also saw a mention of doing the input validation in the url regex. Interestingly, someone proposed having get_object_or_404 give a 404 on a ValueError too, not just on a DoesNotExist exception. However, that proposed change was rejected. So I suppose I shall go with the try/except block.

Can you provide a link to the rejected proposa, @somexpert?

Proposed change

I would recommend not wrapping the get_object_or_404 with a try/except. IMO, we should go the URL route. This is also what is recommended in the linked ticket.
I believe we advised not to use such stringent regexes in the URL patterns previously, but I think we made that comment thinking that Django would look for the UUID in the same fashion as it does the PKs, which it does not.

@ldko, does the url regex approach sound good to you?

yes, putting it in the url regex sounds good to me.

Alright, I'll make that change in another PR dealing with the urls. Can I go ahead and merge this PR?

I think we should complete #16 as part of this PR.

If I'm going to touch the urls, then can I also complete #15?

fine with me.

Those two issues have been fixed. Look good to merge?

Actually, gonna change the way we verify the uuid. Moving that to the views. More transparent and possibly less coupled.

Should be ready for review now. There is a simple function in views.py that checks to make sure the passed event id is a valid uuid.

Ah, right, yeah that makes sense.

get_event_or_404 is something that will also deserve adequate tests :wink:.

19bbf0c eliminates the need for #16.

Fixed the docstring.

Gonna add the test/s for the new function before merging this PR.

@ldko, @damonkelley, PR is ready for review. This should close issues #1, #5, #15, and #18.

Think I got in all the changes that were suggested.

To be nitpicky, since we have been moving toward more standardized capitalization and grammar in docstrings and comments, proper capitalization of acronyms would be good (like HTTP and UUID) to use in such places--unless you are specifically referring to a variable or something that is not capitalized.

Easy enough to do, I'll make the change.

Ready for review!

This should now fix issues #8 and #4.

:+1:

:+1:

/play trololo

This PR fixes #1, fixes #4, fixes #5, fixes #8, fixes #15, and fixes #18.


BUG Fix FulltextsearchEnable

Fixes #5126
It looks like there were a few get_class($this) left over in the old ORM connectors that needed refactoring out.

Note to self: Close https://silverstripe.atlassian.net/browse/CWP-536 once merged

Ah, thanks!


Fix McapProtobufMessage docstring

Public-Facing Changes
None
Description
Fix docstring parameter name for McapProtobufMessage.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Owners to merge.


Minimal font-size in Chrome is 12px by default

Not sure how this is supposed to work with font-size scaling. Chrome for example has minimal 12px font-size by default so the rendered icons look cropped (right and bottom edges are cut off): https://dl.dropboxusercontent.com/spa/1dqminp255vakv6/cgu20ie6.png
When I go to (Chrome) settings and set the minimal font-size to 10px it works as intended, but I leave it on 12px default purposely always, including when developing since this will be the default behaviour for most users.
I saw your plan to add the option to change the base font-size, but I'm not sure if this really is the way to go. Different clients / devices / user's preference (visually impaired) can influence this issue regardless of our "safe size asumptions" and then the sprite breaks again without any way for us to know or affect this.

just set the baseSize option and if needed load your own css template.


Fix issue 98

Fix for issue #98 by whitelisting files we upload

Do we want to do the same in create_mod_data?
Also I notice that pywren will almost always be included as a module dependency even the user function does not use it.  This might be something we want to look into.


First of several md-buttons immediately showing up styled as if in hover-state

My dev environment:
@angular/cli: 1.0.1
node: 7.4.0
os: darwin x64
OS X El Capitan 10.11.6
MacBook Air
I'm using Material in my Angular 2 app, and am running into a situation where, after triggering the opening of a dialog box, which itself contains several vertically stacked md-buttons a user can click on, the first of those buttons immediately shows up with a darker color than the others, which is usually reserved for hover state. Not sure why this is happening, and what I can do to prevent or disable this behavior. The cursor is not located near the button, and yet it's showing behavior as if it was being hovered over or active.
The HTML for this dialog box containing buttons looks like this:

Initiate Discharge

            <button md-button class="discharge-steps-button" (click)="sendFamilyDischargeEmail()">Create Family Discharge Email</button>

            <button md-button class="discharge-steps-button" (click)="sendFamilyDischargeRestaffEmail()">Create Family Restaff/Discharge Email</button>

            <button md-button class="discharge-steps-button" (click)="sendTherapistDischargeSummary()">Create Discharge Summary for Therapist</button>

            <button md-button class="discharge-steps-button" (click)="moveToDischarge()">Complete Move to Discharge</button>

        <button md-button class="discharge-close-button" (click)="close()">CLOSE</button>

The CSS for the four main buttons looks like this:
.discharge-steps-button {
      display: flex;
      flex-direction: column;
      padding: 5px 20px;
      margin-bottom: 10px;
      font-size: 18px;
      background-color: #f4f4f4;
      justify-content: center;
      align-items: center;
}

To clarify, I'n not defining the hover/active styling, this is all handled by Angular Material. If I click within the opened dialog box, the darker styling of the button returns to normal. But upon re-triggering the opening of the dialog box, the first button is again styled as if hover-over or active.
Any ideas as to why this is happening? Is this a bug? Is there a way I can prevent or disable the behavior? By the way, the same thing happened when I made the items straight hrefs, instead of buttons. The first of the links showed us as if in hover or active state.

Would you mind reproducing it? It doesn't seem to be happening with beta 6
https://plnkr.co/edit/PfeoBtqwLuyRuA6ZzbEI?p=preview
So technically that button should be focused because focus changes to the first tabbable element in the dialog (once the dialog is closed, it'll dump back onto the document where it was before). You can verify this in the above plunker by opening the dialog and immediately hitting space and looking at the console.
That said, the focus styling should only be applied when you have focused it with the keyboard (will have the class cdk-focused). It should not be showing up when the dialog forces the focus to change onto itself.

Thanks for the reply. When I tried what you suggested with the plunkr you set up, I didn't see the first item selected when hitting the 'space' key. I had to hover over an item to see focused styling applied. I'm using Chrome Version 58.0.3029.110 (64-bit). But knowing this is the default behavior, is there a way to disable it?

This is working as intended; the first item in the dialog is focused automatically for accessibility.

Is there still no way to disable this behaviour? It doesn't look very good:


Rescue from bundler require error

Resolves an issue where spring tries to File.expand_path and the
$HOME variable is not set.
This was identified as an issue with AWS Elastic Beanstalk deployments

Thank you for reopening this PR so we could get this cleared up!


The Golang version of version v0.30.0 specifced  in the mod file is a precise verson number

Hi Team,
Let's see https://github.com/kubernetes/api/blob/v0.30.0/go.mod
// This is a generated file. Do not edit directly.

module k8s.io/api

go 1.22.0

Please take into account that the Go version specified in the v0.30.0 mod file is 1.22.0. Consequently, it necessitates that any Golang modules relying on v0.30.0 must also adhere to the precise Golang version number
is it possible to use the Golang version without the patch number in later versions? For example https://github.com/kubernetes/api/blob/v0.29.4/go.mod

maybe apply a little go mod edit -go=1.22 -toolchain=go1.22.0?

Would it be possible to get this fixed sooner than later? My enterprise Snyk scan is flagging this and hitting us with a number of vulnerabilities because of it. We're building our application currently with golang 1.22.5+ however because of this item, the Snyk scan is flagging the entire thing as being vulnerable.

let's use https://github.com/kubernetes/kubernetes/issues/126636
/close


Integrate with Markdown Linters

I use a Markdown linter with pre-commit and GitLab CI pipelines. It's useful but I'd like to use a formatter before running the linter: one of the biggest problems I have is that, as well as raising rule violations that require manual correction, the linter frequently raises large numbers of trivial formatting rule violations that could very easily be corrected automatically by a formatter. If they are going to be coupled, it's really important that the formatter and linter work together and don't conflict. I would like to suggest that mdformat be designed from the beginning to integrate with Markdown linters, particularly markdownlint [1].
I expect that this means mdformat will have to observe the markdownlint rules and the rule structure, to avoid creating linting conflicts. The advantage of this is that there's an existing structure and code (largely regexps) which can inform mdformat development.
What do you think?
[1] There are two Markdown linters I'm aware of: a Ruby version https://github.com/markdownlint/markdownlint and a node.js version: https://github.com/DavidAnson/markdownlint, both named "markdownlint" and the respective authors and maintainers work together to ensure that there is minimal divergence of rules between the two versions.

Hey @jamesquilty , thanks for the issue!
I have personally never used either of the markdownlint implementations so don't know much about them. Have you perhaps tried running both markdownlint and mdformat against some lengthy and/or feature-rich piece of Markdown code? Are you aware of what the possible style clashes are?
It seems like at least the Ruby implementation of markdownlint is very configurable (https://github.com/markdownlint/markdownlint/blob/master/docs/configuration.md). If there are lots of style clashes, I'm thinking the easiest way to achieve support between markdownlint and mdformat might be simply providing a compatible markdownlint configuration in mdformat documentation. If the styles are already very similar but not exactly, we could consider changing mdformat style accordingly.

@hukkinj1 Thanks for the reply. I looked at your code and saw that you're taking a very different approach to formatting than I anticipated, accomplishing the formatting by processing the text to be formatted with markdown-it (IIUC). It appears that that method won't be compatible with the way markdownlint operates, due to the configurability of markdownlint.
While I'll be happy to be told that's not the case, I think it would be too much work to implement an arbitrary markdown formatter even with the example provided by markdown-it.

I looked at your code and saw that you're taking a very different approach to formatting than I anticipated, accomplishing the formatting by processing the text to be formatted with markdown-it (IIUC)

Yes, what mdformat does is it parses the Markdown using markdown-it-py Markdown parser. markdown-it-py outputs a token stream that is for the most parts abstract syntax (most style choices are not present in this data anymore). Using this token stream as input, mdformat then re-renders the Markdown document, applying its own style choices.

I think it would be too much work to implement an arbitrary markdown formatter even with the example provided by markdown-it.

I'm not sure I understand what you mean by "arbitrary formatter" here.

It appears that that method won't be compatible with the way markdownlint operates, due to the configurability of markdownlint.

I am also not sure why this would be the case?
I see this as an equivalent compatibility as Black Python formatting with flake8/pylint which work fine together.

Closing this as stale. I believe both markdownlint tools mentioned here are configurable enough that they can coexist with mdformat so it's not clear to me what the issue is.

@hukkin Thanks for following-up, it has been a while! I did some tests some time ago and then had to set it aside due to time pressures. To be absolutely clear, the issue is the other way around: how to configure mdformat so that it will coexist with an existing set of tools and style.
I installed with pip install mdformat-gfm and ran on a couple of sample Markdown files which passed markdownlint. I noted that mdformat:

changed the unordered list marker* to -
escaped all occurrences of [ and ] which aren't part of a link, e.g. my reference format [1]  becomes  \[1\] (raised here as #112).

From the mdformat style guide consecutive ordered and unordered lists will alternate between two different markers, which violates markdownlint rule MD004 - Unordered list style default consistent (but may pass sublist if mdformat's definition of "consecutive list" is the same as markdownlint's definition of "nested list) and rule MD029 - Ordered list item prefix.
I will want to configure mdformat to eliminate all of these problems because, it's not reasonable or feasible for me to reformat my Markdown files, configure markdownlint, my editors, and other tools, and generate a large number of commits just to conform to mdformat.
I did notice that mdformat handled raw HTML in Markdown files gracefully, which was definitely a plus!
I'd really like it if mdformat could be configured so it will integrate with an existing environment. Would that be possible?

Generally speaking, it is not a design goal for mdformat to be configurable. Quite the opposite actually.

Regarding the problems you had:

From the mdformat style guide consecutive ordered and unordered lists will alternate between two different markers (edit to add: this was introduced by #83), which violates markdownlint rule MD004 - Unordered list style default consistent

mdformat only uses - as list marker under normal operation. It only alternates if there are two consecutive lists in the markdown with no other content in between. Note that it is not possible to express this Markdown in any other way (besides resorting to raw HTML). Were mdformat to convert the list item markers, it would turn two separate lists into one, and change semantics of the docuemnt. I'd argue that if markdownlint's consistent setting clashes with this, then that is an issue with markdownlint and should be fixed there.
Regarding escaping of [ and ] you'll find my thoughts in https://github.com/executablebooks/mdformat/issues/112

Thanks for the extraordinarily prompt reply, it's very much appreciated! I don't feel that there's an issue with markdownlint, we're talking about style and it's not something that should be argued over. I appreciate the motivation behind your design goal!
Unfortunately, this isn't a situation analagous to black & pylint where there's a single mandated style guide which allows black to be "opinionated" and "stubborn" (i.e. it defines a style which can not [easily] be configured) and not conflict with pylint. [Most of the time. I've had the not-very-fun situation where black and pylint conflict in pre-commit]. There's no single "correct" Markdown style, which is clearly the reason you've ended up  writing multiple versions to "support" different styles: CommonMark, GFM and MyST.
In this context, I think it's reasonable to be "opinionated but not stubborn" and to allow some configuration. Which brings me to my original request for integration with Markdown linters: "If they are going to be coupled, it's really important that the formatter and linter work together and don't conflict."

I don't feel that there's an issue with markdownlint, we're talking about style and it's not something that should be argued over.

We are not talking about style however.
Consider the following markdown
- item1
+ item2
- item3

If you render this as HTML you'll notice that there are three lists with one item in each. Now if I decide that for "consistency" I format it in a way that only - is used as list item marker, then suddenly we have only one list with three items in it. Semantics have changed, not only style. Mdformat will not change semantics, you'll have to do that manually.

There's no single "correct" Markdown style, which is clearly the reason you've ended up writing multiple versions to "support" different styles: CommonMark, GFM and MyST.

The ones you listed have nothing to do with formatting styles. They are three separate markup language specifications, and as such need to be parsed in a different manner.

Which brings me to my original request for integration with Markdown linters: "If they are going to be coupled, it's really important that the formatter and linter work together and don't conflict."

Then I'll need a concrete where the two tools do conflict and what needs to be changed for them not to.

I'd like to concentrate, please, on the most significant issue with mdformat which troubles me:  it's not reasonable or feasible for me to reformat my Markdown files, configure markdownlint, my editors, and other tools, and generate a large number of commits just to conform to mdformat's style configuration.
If mdformat is "stubborn" about its style configuration, for example only using - as list marker, then that imposes a significant workload burden on me. I'm not sure that the benefits of adopting mdformat outweigh the costs!
I'd really like it if mdformat could be configured so it will integrate with an existing environment.

Sure thing, but mdformat is an opinionated formatter and as a maintainer I'm not willing to test and maintain conf options based on opinions other than what the tool has. We have some configuration (ordered list numbering style, word wrap options) but I try to avoid adding more. New options will only be added for very good reasons. Simply "I prefer asterisk to dash" isn't a good reason here unfortunately. If you have a concrete proposal I'll be happy to discuss it in a separate issue.
In case you need a more configurable tool, the license is MIT, so you can fork and do whatever changes you wish, as long as you keep the license.


Add topic dotenv

I'd suggest specifying that .env is a file extension rather than the name of the file.

@KiiOther .env is the name of the file.