No Transformer-based API LLM Examples Working

The Transformer Python API section is not working. I've tried Python 3.7, 3.8, 3.10 and 3.11.
I am running this with Ubuntu on Intel Sapphire Rapid CPUs.
The code:
from transformers import AutoTokenizer, TextStreamer
from intel_extension_for_transformers.transformers import AutoModelForCausalLM
model_name = "Intel/neural-chat-7b-v1-1"     # Hugging Face model_id or local model
prompt = "Once upon a time, there existed a little girl,"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
inputs = tokenizer(prompt, return_tensors="pt").input_ids
streamer = TextStreamer(tokenizer)

model = AutoModelForCausalLM.from_pretrained(model_name, load_in_4bit=True)
outputs = model.generate(inputs, streamer=streamer, max_new_tokens=300)

results in:
2023-11-30 20:50:43 [INFO] CPU device is used.
2023-11-30 20:51:33 [INFO] Applying Weight Only Quantization.
2023-11-30 20:51:33 [INFO] Using LLM runtime.
Traceback (most recent call last):
  File "scratch/step2.py", line 10, in <module>
    model = AutoModelForCausalLM.from_pretrained(model_name, load_in_4bit=True)
  File "/home/caseyhoward/intel-extension-for-transformers/intel_extension_for_transformers/llm/runtime/graph/llm-examples/lib/python3.8/site-packages/intel_extension_for_transformers/transformers/modeling/modeling_auto.py", line 136, in from_pretrained
    quantization_config.post_init_runtime()
  File "/home/caseyhoward/intel-extension-for-transformers/intel_extension_for_transformers/llm/runtime/graph/llm-examples/lib/python3.8/site-packages/intel_extension_for_transformers/transformers/utils/quantization_config.py", line 127, in post_init_runtime
    raise ValueError(f"weight_dtype must be 'int4', 'int8'.")
ValueError: weight_dtype must be 'int4', 'int8'.

I have been unable to get any Transformer-based API examples working.
I have been able to make the scripts work as per description. So I know my system hardware and kernel configuration do work.
I had originally been trying to get python_api_example.py working, but this too is broken:
Traceback (most recent call last):
  File "scripts/python_api_example.py", line 29, in <module>
    model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=woq_config, trust_remote_code=True)
  File "/home/caseyhoward/intel-extension-for-transformers/intel_extension_for_transformers/llm/runtime/graph/try2/lib/python3.8/site-packages/intel_extension_for_transformers/transformers/modeling/modeling_auto.py", line 128, in from_pretrained
    model = cls.ORIG_MODEL.from_pretrained(
  File "/home/caseyhoward/intel-extension-for-transformers/intel_extension_for_transformers/llm/runtime/graph/try2/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 553, in from_pretrained
    model_class = get_class_from_dynamic_module(
  File "/home/caseyhoward/intel-extension-for-transformers/intel_extension_for_transformers/llm/runtime/graph/try2/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 499, in get_class_from_dynamic_module
    return get_class_in_module(class_name, final_module.replace(".py", ""))
  File "/home/caseyhoward/intel-extension-for-transformers/intel_extension_for_transformers/llm/runtime/graph/try2/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 199, in get_class_in_module
    module = importlib.import_module(module_path)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/caseyhoward/.cache/huggingface/modules/transformers_modules/Intel/neural-chat-7b-v1-1/b80b062deb3220b71dc9dfd4e093b298541961f1/modeling_mpt.py", line 18, in <module>
    from .hf_prefixlm_converter import add_bidirectional_mask_if_missing, convert_hf_causal_lm_to_prefix_lm
  File "/home/caseyhoward/.cache/huggingface/modules/transformers_modules/Intel/neural-chat-7b-v1-1/b80b062deb3220b71dc9dfd4e093b298541961f1/hf_prefixlm_converter.py", line 15, in <module>
    from transformers.models.bloom.modeling_bloom import _expand_mask as _expand_mask_bloom
ImportError: cannot import name '_expand_mask' from 'transformers.models.bloom.modeling_bloom' (/home/caseyhoward/intel-extension-for-transformers/intel_extension_for_transformers/llm/runtime/graph/try2/lib/python3.8/site-packages/transformers/models/bloom/modeling_bloom.py)

Some other models work, but it's hit and miss. For example model meta-llama/Llama-2-7b-chat-hf also fails:
python scripts/python_api_example.py
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00,  1.39it/s]
2023-11-30 22:36:41 [INFO] Applying Weight Only Quantization.
2023-11-30 22:36:41 [INFO] Using LLM runtime.
cmd: ['python', PosixPath('/home/caseyhoward/intel-extension-for-transformers/intel_extension_for_transformers/llm/runtime/graph/venv/lib/python3.8/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_llama.py'), '--outfile', 'ne_llama_f32.bin', '--outtype', 'f32', 'meta-llama/Llama-2-7b-chat-hf']
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
Loading model file meta-llama/Llama-2-7b-chat-hf
Traceback (most recent call last):
  File "/home/caseyhoward/intel-extension-for-transformers/intel_extension_for_transformers/llm/runtime/graph/venv/lib/python3.8/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_llama.py", line 1271, in <module>
    main()
  File "/home/caseyhoward/intel-extension-for-transformers/intel_extension_for_transformers/llm/runtime/graph/venv/lib/python3.8/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_llama.py", line 1251, in main
    model_plus = load_some_model(args.model)
  File "/home/caseyhoward/intel-extension-for-transformers/intel_extension_for_transformers/llm/runtime/graph/venv/lib/python3.8/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_llama.py", line 1177, in load_some_model
    models_plus.append(lazy_load_file(path))
  File "/home/caseyhoward/intel-extension-for-transformers/intel_extension_for_transformers/llm/runtime/graph/venv/lib/python3.8/site-packages/intel_extension_for_transformers/llm/runtime/graph/scripts/convert_llama.py", line 945, in lazy_load_file
    fp = open(path, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'meta-llama/Llama-2-7b-chat-hf'
Traceback (most recent call last):
  File "scripts/python_api_example.py", line 29, in <module>
    model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=woq_config, trust_remote_code=True)
  File "/home/caseyhoward/intel-extension-for-transformers/intel_extension_for_transformers/llm/runtime/graph/venv/lib/python3.8/site-packages/intel_extension_for_transformers/transformers/modeling/modeling_auto.py", line 140, in from_pretrained
    model.init(
  File "/home/caseyhoward/intel-extension-for-transformers/intel_extension_for_transformers/llm/runtime/graph/venv/lib/python3.8/site-packages/intel_extension_for_transformers/llm/runtime/graph/__init__.py", line 74, in init
    assert os.path.exists(fp32_bin), "Fail to convert pytorch model"
AssertionError: Fail to convert pytorch model

I get
python3.8 -m venv venv
. venv/bin/activate
git clone https://github.com/intel/intel-extension-for-transformers
cd intel-extension-for-transformers
pip install -r requirements.txt
pip install transformers==4.33.1
python setup.py install
error: huggingface-hub 0.19.4 is installed but huggingface_hub<0.18,>=0.16.4 is required by {'tokenizers'}

Attempting to resolve that error with
pip install huggingface-hub==0.17.3

#Output
Collecting huggingface-hub==0.17.3
  Downloading huggingface_hub-0.17.3-py3-none-any.whl (295 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 295.0/295.0 kB 6.1 MB/s eta 0:00:00
Requirement already satisfied: packaging>=20.9 in ./venv/lib/python3.8/site-packages (from huggingface-hub==0.17.3) (23.2)
Requirement already satisfied: requests in ./venv/lib/python3.8/site-packages (from huggingface-hub==0.17.3) (2.31.0)
Requirement already satisfied: fsspec in ./venv/lib/python3.8/site-packages (from huggingface-hub==0.17.3) (2023.10.0)
Requirement already satisfied: filelock in ./venv/lib/python3.8/site-packages (from huggingface-hub==0.17.3) (3.13.1)
Requirement already satisfied: tqdm>=4.42.1 in ./venv/lib/python3.8/site-packages (from huggingface-hub==0.17.3) (4.66.1)
Requirement already satisfied: typing-extensions>=<IP_ADDRESS> in ./venv/lib/python3.8/site-packages (from huggingface-hub==0.17.3) (4.8.0)
Requirement already satisfied: pyyaml>=5.1 in ./venv/lib/python3.8/site-packages (from huggingface-hub==0.17.3) (6.0.1)
Requirement already satisfied: certifi>=2017.4.17 in ./venv/lib/python3.8/site-packages (from requests->huggingface-hub==0.17.3) (2023.11.17)
Requirement already satisfied: idna<4,>=2.5 in ./venv/lib/python3.8/site-packages (from requests->huggingface-hub==0.17.3) (3.6)
Requirement already satisfied: charset-normalizer<4,>=2 in ./venv/lib/python3.8/site-packages (from requests->huggingface-hub==0.17.3) (3.3.2)
Requirement already satisfied: urllib3<3,>=1.21.1 in ./venv/lib/python3.8/site-packages (from requests->huggingface-hub==0.17.3) (2.1.0)
Installing collected packages: huggingface-hub
  Attempting uninstall: huggingface-hub
    Found existing installation: huggingface-hub 0.19.4
    Uninstalling huggingface-hub-0.19.4:
      Successfully uninstalled huggingface-hub-0.19.4
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 2.15.0 requires huggingface-hub>=0.18.0, but you have huggingface-hub 0.17.3 which is incompatible.
intel-extension-for-transformers 1.3rc2.dev30+g318e5cbf22 requires transformers==4.34.1, but you have transformers 4.33.1 which is incompatible.
Successfully installed huggingface-hub-0.17.3

Rerunning:
python setup.py install

results in:
Beginning with Matplotlib 3.8, Python 3.9 or above is required.
You are using Python 3.8.18.

This may be due to an out of date pip.

Make sure you have pip >= 9.0.1.

So retrying with Python 3.9:
sudo apt install python3.9 python3.9-venv python3.9-dev python3-pip;
python3.9 -m venv python3.9-venv
. python3.9-venv/bin/activate
pip install -U pip
pip install -r requirements.txt
pip install transformers==4.33.1
python setup.py install

Complains about:
error: huggingface-hub 0.19.4 is installed but huggingface_hub<0.18,>=0.16.4 is required by {'tokenizers'}

So I install the preferred huggingface-hub version and setup.py install:
pip install huggingface-hub==0.17.3
python setup.py install

# output
Collecting huggingface-hub==0.17.3
  Downloading huggingface_hub-0.17.3-py3-none-any.whl.metadata (13 kB)
Requirement already satisfied: filelock in ./python3.9-venv/lib/python3.9/site-packages (from huggingface-hub==0.17.3) (3.13.1)
Requirement already satisfied: fsspec in ./python3.9-venv/lib/python3.9/site-packages (from huggingface-hub==0.17.3) (2023.10.0)
Requirement already satisfied: requests in ./python3.9-venv/lib/python3.9/site-packages (from huggingface-hub==0.17.3) (2.31.0)
Requirement already satisfied: tqdm>=4.42.1 in ./python3.9-venv/lib/python3.9/site-packages (from huggingface-hub==0.17.3) (4.66.1)
Requirement already satisfied: pyyaml>=5.1 in ./python3.9-venv/lib/python3.9/site-packages (from huggingface-hub==0.17.3) (6.0.1)
Requirement already satisfied: typing-extensions>=<IP_ADDRESS> in ./python3.9-venv/lib/python3.9/site-packages (from huggingface-hub==0.17.3) (4.8.0)
Requirement already satisfied: packaging>=20.9 in ./python3.9-venv/lib/python3.9/site-packages (from huggingface-hub==0.17.3) (23.2)
Requirement already satisfied: charset-normalizer<4,>=2 in ./python3.9-venv/lib/python3.9/site-packages (from requests->huggingface-hub==0.17.3) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in ./python3.9-venv/lib/python3.9/site-packages (from requests->huggingface-hub==0.17.3) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in ./python3.9-venv/lib/python3.9/site-packages (from requests->huggingface-hub==0.17.3) (2.1.0)
Requirement already satisfied: certifi>=2017.4.17 in ./python3.9-venv/lib/python3.9/site-packages (from requests->huggingface-hub==0.17.3) (2023.11.17)
Downloading huggingface_hub-0.17.3-py3-none-any.whl (295 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 295.0/295.0 kB 7.0 MB/s eta 0:00:00
Installing collected packages: huggingface-hub
  Attempting uninstall: huggingface-hub
    Found existing installation: huggingface-hub 0.19.4
    Uninstalling huggingface-hub-0.19.4:
      Successfully uninstalled huggingface-hub-0.19.4
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 2.15.0 requires huggingface-hub>=0.18.0, but you have huggingface-hub 0.17.3 which is incompatible.
intel-extension-for-transformers 1.3rc2.dev30+g318e5cbf22 requires transformers==4.34.1, but you have transformers 4.33.1 which is incompatible.
Successfully installed huggingface-hub-0.17.3

# Outputs redacted of copious quantities of log lines
Using /home/caseyhoward/intel-extension-for-transformers/python3.9-venv/lib/python3.9/site-packages
Finished processing dependencies for intel-extension-for-transformers==1.3rc2.dev30+g318e5cbf22

Running:
# example.py
from transformers import AutoTokenizer, TextStreamer
from intel_extension_for_transformers.transformers import AutoModelForCausalLM
model_name = "Intel/neural-chat-7b-v1-1"     # Hugging Face model_id or local model
prompt = "Once upon a time, there existed a little girl,"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
inputs = tokenizer(prompt, return_tensors="pt").input_ids
streamer = TextStreamer(tokenizer)

model = AutoModelForCausalLM.from_pretrained(model_name, load_in_4bit=True)
outputs = model.generate(inputs, streamer=streamer, max_new_tokens=300)

Results in:
Traceback (most recent call last):
# ...
# Stacktrace removed
# ...
  File "/home/caseyhoward/intel-extension-for-transformers/python3.9-venv/lib/python3.9/site-packages/transformers/dynamic_module_utils.py", line 179, in check_imports
    raise ImportError(
ImportError: This modeling file requires the following packages that were not found in your environment: einops. Run `pip install einops`
Traceback (most recent call last):
  File "/home/caseyhoward/ex.py", line 10, in <module>
    model = AutoModelForCausalLM.from_pretrained(model_name, load_in_4bit=True)
  File "/home/caseyhoward/intel-extension-for-transformers/python3.9-venv/lib/python3.9/site-packages/intel_extension_for_transformers-1.3rc2.dev30+g318e5cbf22-py3.9-linux-x86_64.egg/intel_extension_for_transformers/transformers/modeling/modeling_auto.py", line 179, in from_pretrained
    model.init(
  File "/home/caseyhoward/intel-extension-for-transformers/python3.9-venv/lib/python3.9/site-packages/intel_extension_for_transformers-1.3rc2.dev30+g318e5cbf22-py3.9-linux-x86_64.egg/intel_extension_for_transformers/llm/runtime/graph/__init__.py", line 122, in init
    assert os.path.exists(fp32_bin), "Fail to convert pytorch model"
AssertionError: Fail to convert pytorch model

Running
pip install einops
python example.py

Results in:
model_quantize_internal: model size  = 25362.62 MB
model_quantize_internal: quant size  =  4737.50 MB
ARCH_REQ_XCOMP_PERM XTILE_DATA successful.
AVX:1 AVX2:1 AVX512F:1 AVX_VNNI:1 AVX512_VNNI:1 AMX_INT8:1 AMX_BF16:1 AVX512_BF16:1 AVX512_FP16:1
beam_size: 1, do_sample: 0, top_k: 40, top_p: 0.950000
model.cpp: loading model from runtime_outs/ne_mpt_q_int4_jblas_cint8_g32.bin
init: n_vocab    = 50279
init: n_embd     = 4096
init: n_mult     = 4096
init: n_head     = 32
init: n_layer    = 32
init: n_rot      = 32
init: n_ff       = 16384
init: n_parts    = 1
load: ne ctx size = 4737.55 MB
load: mem required  = 12929.55 MB (+ memory per state)
..................................................................................................
model_init_from_file: support_jblas_kv = 1
model_init_from_file: kv self size =  276.00 MB
Once upon a time, there existed a little girl, who was born in the year of the dragon. She was born in the year of the dragon because her mother was born in the year of the dragon. Her mother was born in the year of the dragon because her grandmother was born in the year of the dragon. Her grandmother was born in the year of the dragon because her grandfather was born in the year of the dragon. Her grandfather was born in the year of the dragon because his father was born in the year of the dragon. His father was born in the year of the dragon because his mother was born in the year of the dragon. Her mother was born in the year of the dragon because her father was born in the year of the dragon. Her father was born in the year of the dragon because his mother was born in the year of the dragon. Her mother was born in the year of the dragon because her father was born in the year of the dragon. Her father was born in the year of the dragon because his mother was born in the year of the dragon. Her mother was born in the year of the dragon because her father was born in the year of the dragon. Her father was born in the year of the dragon because his mother was born in the year of the dragon. Her mother was born in the year of the dragon because her father was born in the year of the dragon. Her father was born in the year of the dragon because his mother was born in the year of the dragon. Her mother was born in

It works.
Based on my experience, I have some thoughts:

It would be helpful to update intel-extension-for-transformers in PyPI and other repos (or the docs Install docs to reflect the current build process)
This project seems to have a python3.9 version floor. I don't know if that's intended (I think there are some docs that say it's 3.7+ up to 3.10)
There are some dependency issues that with einops, huggingface-hub, transformers  and datasets (due to downgrading huggingface-hub)

Might there a way to update the package in PyPI and Conda repos so pip install intel-extension-for-transformers works? I think it would significantly help with adoption of this software and Intel's AMX capabilities.

Thanks very much for you valuable suggestion!
We will update the README, and consider to fix following dependency issue.
Regards
Bo

I am encountering an error similar to @cphoward . I am uncertain if my server hardware is not supported. Here is the hardware information:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
Address sizes:                      39 bits physical, 48 bits virtual
CPU(s):                             20
On-line CPU(s) list:                0-19
Thread(s) per core:                 2
Core(s) per socket:                 10
Socket(s):                          1
NUMA node(s):                       1
Vendor ID:                          GenuineIntel
CPU family:                         6
Model:                              165
Model name:                         Intel(R) Core(TM) i9-10900K CPU @ 3.70GHz
Stepping:                           5
CPU MHz:                            4900.000
CPU max MHz:                        5300.0000
CPU min MHz:                        800.0000
BogoMIPS:                           7399.70
Virtualization:                     VT-x
L1d cache:                          320 KiB
L1i cache:                          320 KiB
L2 cache:                           2.5 MiB
L3 cache:                           20 MiB
NUMA node0 CPU(s):                  0-19
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit:        KVM: Mitigation: VMX disabled
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Retbleed:             Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:                Mitigation; Microcode
Vulnerability Tsx async abort:      Not affected
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfm
                                    perf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ib
                                    pb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_a
                                    ct_window hwp_epp pku ospke md_clear flush_l1d arch_capabilities

Thank you so much!

@CaoHaiNam Hi, the current problem may not be your hardware. Please check the above my comment in this issue to fix this error.
If your hardward is not supported, the error will be reported. No worries. Feel free to omment this issue if you meet any problems.

Hi, sorry to reply late because I didn't receive any email notification    : (
I can't see the screenshot you shared. Please upload again.
Please provide more details so that I can reproduce your error. The script, the model name / card_id of HF and commands.

@rajivmehtaflex
hi, the error you met might be the transformer version. Please try to update the latest transformer version. That's why KeyError: 'mistral'".

@rajivmehtaflex
Hi, This issue has been fixed. I'll close this issue for now. If you have more questions, please feel free to ask and @ me.


fix(parser): fix lexer error while parsing parenthesized arrow expressions

closes #3399

#3400  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @Boshen and the rest of your teammates on  Graphite


WSL2 CUDA Does Not Respect CUDA - Sysmem Fallback Policy

Windows Version
Microsoft Windows [Version 10.0.22635.3061]
WSL Version
<IP_ADDRESS>
Are you using WSL 1 or WSL 2?

[X] WSL 2
[ ] WSL 1

Kernel Version
<IP_ADDRESS>-1
Distro Version
Ubuntu 22.04
Other Software
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.36                 Driver Version: 546.33       CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090        On  | 00000000:01:00.0  On |                  Off |
|  0%   38C    P8              16W / 450W |   1528MiB / 24564MiB |      8%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A        38      G   /Xwayland                                 N/A      |
+---------------------------------------------------------------------------------------+

torch                     2.3.0.dev20231227+cu121

Repro Steps
With a recent version of NVIDIA GPU Driver, in Windows NVIDIA Driver Settings, disable CUDA - Sysmem Fallback Policy globally and restart the computer.

In WSL terminal, execute the following command to allocate a tensor that should exceed the overall GPU memory available (30 GiB required VS 24 GiB available).
python3 -c 'import torch; x = torch.ones((30, 1024, 1024, 1024), dtype=torch.uint8, device="cuda"); print(x.device)'

Instead of throwing an OOM exception, the command executes successfully.

However, the expected behaviour is to throw an OOM exception so that some applications like StableDiffusion WebUI can detect that the inadequacy of the available GPU memory and choose a memory efficient way to do inference. The current behaviour just uses the slow fallbacked CPU memory, which causes the inference to be really slow.
The total available GPU memory showed by torch also looks weired.
python3 -c 'import torch; print(torch.cuda.get_device_properties(0).total_memory)'

Expected Behavior
Executing the following command throws an OOM exception
python3 -c 'import torch; x = torch.ones((30, 1024, 1024, 1024), dtype=torch.uint8, device="cuda"); print(x.device)'

Actual Behavior
The command executes peacefully, which is not expected.
python3 -c 'import torch; x = torch.ones((30, 1024, 1024, 1024), dtype=torch.uint8, device="cuda"); print(x.device)'

Diagnostic Logs
No response

@chengzeyi try running your command multiple times. In my case the OOM triggered on the 2nd run:
(phi-2-env) root@texas:/mnt/e/ai/phi-2# python
Python 3.9.18 (main, Sep 11 2023, 13:41:44)
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> x = torch.ones((30, 1024, 1024, 1024), dtype=torch.uint8, device="cuda"); print(x.device)
cuda:0
>>> x.device
device(type='cuda', index=0)
>>> x = torch.ones((30, 1024, 1024, 1024), dtype=torch.uint8, device="cuda"); print(x.device)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 30.00 GiB. GPU 0 has a total capacty of 15.99 GiB of which 0 bytes is free. Including non-PyTorch memory, this process has<PHONE_NUMBER>4.00 GiB memory in use. Of the allocated memory 30.00 GiB is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

nvidia-smi output:
Sun Jan 21 10:04:33 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.36                 Driver Version: 546.33       CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA RTX A4000               On  | 00000000:61:00.0  On |                  Off |
| 41%   33C    P8              11W / 140W |    515MiB / 16376MiB |      3%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A        35      G   /Xwayland                                 N/A      |
+---------------------------------------------------------------------------------------+

@elsaco Yeah, that's the bug, the first run should trigger an OOM instead of the following runs. In your case 30GB x 2 is too large even for fallbacked sysmem, so it fails in the second run.

Just ran into this issue today and was briefly confused why my processing speed dropped like a stone when it looked like things were fitting into VRAM and I knew I had sysmem fallback off.
Thanks for the workaround, but I too hope to see action on this.


[stable/rabbitmq] allow extra ports to be exposed with extraPorts

took reference from bitnami/wordpress
Signed-off-by: Michael Lee<EMAIL_ADDRESS>
Is this a new chart

NOTE: We're experiencing a high volume of PRs to this repo and reviews will be delayed. Please host your own chart repository and submit your repository to the Helm Hub instead of this repo to make them discoverable to the community. Here is how to submit new chart repositories to the Helm Hub.

What this PR does / why we need it:
Which issue this PR fixes
(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

fixes #17924

Special notes for your reviewer:
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[ x ] DCO signed
[ x ] Chart Version bumped
[ x ] Variables are documented in the README.md
[ x ] Title of the PR starts with chart name (e.g. [stable/chart])

Thanks for the PR, @michaellee8, looks very good. Can you please bump the chart version in a minor instead, since you are adding new functionality?
Also, I have a question: would it be necessary to expose the ports in the RabbitMQ container in the Stateful Set? I can see the ports exposed in the service but the container is not making use of them. Does it need them? What's the use case?

@alemorcuq I have bumped a the version instead. Although the containerPort in StatefulSet is primarily informational, I have added a new config called extraContainerPorts to allow users adding ports config in it.
One maybe able to test the new configs with microk8s here, just in case you need it:
## Global Docker image parameters
## Please, note that this will override the image parameters, including dependencies, configured to use the global value
## Current available global Docker image parameters: imageRegistry and imagePullSecrets
##
# global:
#   imageRegistry: myRegistryName
#   imagePullSecrets:
#     - myRegistryKeySecretName
#   storageClass: myStorageClass

## Bitnami RabbitMQ image version
## ref: https://hub.docker.com/r/bitnami/rabbitmq/tags/
##
image:
  registry: docker.io
  repository: bitnami/rabbitmq
  tag: 3.8.0-debian-9-r0

  ## set to true if you would like to see extra information on logs
  ## it turns BASH and NAMI debugging in minideb
  ## ref:  https://github.com/bitnami/minideb-extras/#turn-on-bash-debugging
  debug: false

  ## Specify a imagePullPolicy
  ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
  ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
  ##
  pullPolicy: IfNotPresent
  ## Optionally specify an array of imagePullSecrets.
  ## Secrets must be manually created in the namespace.
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  ##
  # pullSecrets:
  #   - myRegistryKeySecretName

## String to partially override rabbitmq.fullname template (will maintain the release name)
##
# nameOverride:

## String to fully override rabbitmq.fullname template
##
# fullnameOverride:

## Use an alternate scheduler, e.g. "stork".
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
# schedulerName:

## does your cluster have rbac enabled? assume yes by default
rbacEnabled: true

## RabbitMQ should be initialized one by one when building cluster for the first time.
## Therefore, the default value of podManagementPolicy is 'OrderedReady'
## Once the RabbitMQ participates in the cluster, it waits for a response from another
## RabbitMQ in the same cluster at reboot, except the last RabbitMQ of the same cluster.
## If the cluster exits gracefully, you do not need to change the podManagementPolicy
## because the first RabbitMQ of the statefulset always will be last of the cluster.
## However if the last RabbitMQ of the cluster is not the first RabbitMQ due to a failure,
## you must change podManagementPolicy to 'Parallel'.
## ref : https://www.rabbitmq.com/clustering.html#restarting
##
podManagementPolicy: OrderedReady

## section of specific values for rabbitmq
rabbitmq:
  ## RabbitMQ application username
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  username: user

  ## RabbitMQ application password
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  # password:
  # existingPasswordSecret: name-of-existing-secret

  ## Erlang cookie to determine whether different nodes are allowed to communicate with each other
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  # erlangCookie:
  # existingErlangSecret: name-of-existing-secret

  ## Node name to cluster with. e.g.: `clusternode@hostname`
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  # rabbitmqClusterNodeName:

  ## Value for the RABBITMQ_LOGS environment variable
  ## ref: https://www.rabbitmq.com/logging.html#log-file-location
  ##
  logs: "-"

  ## RabbitMQ Max File Descriptors
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ## ref: https://www.rabbitmq.com/install-debian.html#kernel-resource-limits
  ##
  setUlimitNofiles: true
  ulimitNofiles: "65536"

  ## RabbitMQ maximum available scheduler threads and online scheduler threads
  ## ref: https://hamidreza-s.github.io/erlang/scheduling/real-time/preemptive/migration/2016/02/09/erlang-scheduler-details.html#scheduler-threads
  ##
  maxAvailableSchedulers: 2
  onlineSchedulers: 1

  ## Plugins to enable
  plugins: "rabbitmq_management rabbitmq_peer_discovery_k8s"

  ## Extra plugins to enable
  ## Use this instead of `plugins` to add new plugins
  extraPlugins: "rabbitmq_mqtt"

  ## Clustering settings
  clustering:
    address_type: hostname
    k8s_domain: cluster.local

  loadDefinition:
    enabled: false
    secretName: load-definition

  ## environment variables to configure rabbitmq
  ## ref: https://www.rabbitmq.com/configure.html#customise-environment
  env: {}

  ## Configuration file content: required cluster configuration
  ## Do not override unless you know what you are doing. To add more configuration, use `extraConfiguration` of `advancedConfiguration` instead
  configuration: |-
    ## Clustering
    cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
    cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
    cluster_formation.node_cleanup.interval = 10
    cluster_formation.node_cleanup.only_log_warning = true
    cluster_partition_handling = autoheal
    # queue master locator
    queue_master_locator=min-masters
    # enable guest user
    loopback_users.guest = false

  ## Configuration file content: extra configuration
  ## Use this instead of `configuration` to add more configuration
  extraConfiguration: |-
    #disk_free_limit.absolute = 50MB
    #management.load_definitions = /app/load_definition.json

  ## Configuration file content: advanced configuration
  ## Use this as additional configuraton in classic config format (Erlang term configuration format)
  advancedConfiguration: |-

  ## Enable encryption to rabbitmq
  ## ref: https://www.rabbitmq.com/ssl.html
  ##
  tls:
    enabled: false
    failIfNoPeerCert: true
    sslOptionsVerify: verify_peer
    caCertificate: |-
    serverCertificate: |-
    serverKey: |-
    # existingSecret: name-of-existing-secret-to-rabbitmq

## Kubernetes service type
service:
  type: NodePort
  ## Node port
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  # nodePort: 30672

  ## Node port Tls
  ##
  # nodeTlsPort: 30671

  ## Amqp port
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  port: 5672

  ## Amqp Tls port
  ##
  tlsPort: 5671

  ## Dist port
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  distPort: 25672

  ## RabbitMQ Manager port
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  managerPort: 15672

  ## Service annotations
  annotations:
    # service.beta.kubernetes.io/aws-load-balancer-internal: <IP_ADDRESS>/0

  ## Load Balancer sources
  ## https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service
  ##
  # loadBalancerSourceRanges:
  # - <IP_ADDRESS>/24

  ## Extra ports to expose
  extraPorts:
    - name: mgmt
      port: 30001
      targetPort: stats
      nodePort: null
    - name: mqtt
      port: 1883
      targetPort: 1883
      nodePort: null

  extraContainerPorts:
    - name: mqtt
      containerPort: 1883

# Additional pod labels to apply
podLabels: {}

## Pod Security Context
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
##
securityContext:
  enabled: true
  fsGroup: 1001
  runAsUser: 1001
  extra: {}

persistence:
  ## this enables PVC templates that will create one per pod
  enabled: true

  ## rabbitmq data Persistent Volume Storage Class
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  storageClass: "microk8s-hostpath"
  accessMode: ReadWriteOnce

  ## Existing PersistentVolumeClaims
  ## The value is evaluated as a template
  ## So, for example, the name can depend on .Release or .Chart
  # existingClaim: ""

  # If you change this value, you might have to adjust `rabbitmq.diskFreeLimit` as well.
  size: 3Gi

  # persistence directory, maps to the rabbitmq data directory
  path: /opt/bitnami/rabbitmq/var/lib/rabbitmq

## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
  requests:
    memory: 256Mi
    cpu: 100m

## Replica count, set to 3 to provide a default available cluster
replicas: 3

## Pod priority
## https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/
# priorityClassName: ""

## updateStrategy for RabbitMQ statefulset
## ref: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies
updateStrategy:
  type: RollingUpdate

## Node labels and tolerations for pod assignment
## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature
nodeSelector:
  beta.kubernetes.io/arch: amd64
tolerations: []
affinity: {}

## annotations for rabbitmq pods
podAnnotations: {}

## Configure the ingress resource that allows you to access the
## Wordpress installation. Set up the URL
## ref: http://kubernetes.io/docs/user-guide/ingress/
##
ingress:
  ## Set to true to enable ingress record generation
  enabled: false

  ## The list of hostnames to be covered with this ingress record.
  ## Most likely this will be just one host, but in the event more hosts are needed, this is an array
  ## hostName: foo.bar.com
  path: /

  ## Set this to true in order to enable TLS on the ingress record
  ## A side effect of this will be that the backend wordpress service will be connected at port 443
  tls: true

  ## If TLS is set to true, you must declare what secret will store the key/certificate for TLS
  tlsSecret: myTlsSecret

  ## Ingress annotations done as key:value pairs
  ## If you're using kube-lego, you will want to add:
  ## kubernetes.io/tls-acme: true
  ##
  ## For a full list of possible ingress annotations, please see
  ## ref: https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md
  ##
  ## If tls is set to true, annotation ingress.kubernetes.io/secure-backends: "true" will automatically be set
  annotations:
  #  kubernetes.io/ingress.class: nginx
  #  kubernetes.io/tls-acme: true

## The following settings are to configure the frequency of the lifeness and readiness probes
livenessProbe:
  enabled: true
  initialDelaySeconds: 120
  timeoutSeconds: 20
  periodSeconds: 30
  failureThreshold: 6
  successThreshold: 1

readinessProbe:
  enabled: true
  initialDelaySeconds: 10
  timeoutSeconds: 20
  periodSeconds: 30
  failureThreshold: 3
  successThreshold: 1

metrics:
  enabled: true
  image:
    registry: docker.io
    repository: bitnami/rabbitmq-exporter
    tag: 0.29.0-debian-9-r61
    pullPolicy: IfNotPresent
    ## Optionally specify an array of imagePullSecrets.
    ## Secrets must be manually created in the namespace.
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    ##
    # pullSecrets:
    #   - myRegistryKeySecretName
  ## environment variables to configure rabbitmq_exporter
  ## ref: https://github.com/kbudde/rabbitmq_exporter#configuration
  env: {}
  ## Metrics exporter port
  port: 9419
  ## Comma-separated list of extended scraping capabilities supported by the target RabbitMQ server
  ## ref: https://github.com/kbudde/rabbitmq_exporter#extended-rabbitmq-capabilities
  capabilities: "bert,no_sort"
  resources: {}
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9090"

  ## Prometheus Service Monitor
  ## ref: https://github.com/coreos/prometheus-operator
  ##      https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#endpoint
  serviceMonitor:
    ## If the operator is installed in your cluster, set to true to create a Service Monitor Entry
    enabled: false
    ## Specify the namespace in which the serviceMonitor resource will be created
    # namespace: ""
    ## Specify the interval at which metrics should be scraped
    interval: 30s
    ## Specify the timeout after which the scrape is ended
    # scrapeTimeout: 30s
    ## Specify Metric Relabellings to add to the scrape endpoint
    # relabellings:
    ## Specify honorLabels parameter to add the scrape endpoint
    honorLabels: false
    ## Used to pass Labels that are used by the Prometheus installed in your cluster to select Service Monitors to work with
    ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#prometheusspec
    additionalLabels: {}

##
## Init containers parameters:
## volumePermissions: Change the owner of the persist volume mountpoint to RunAsUser:fsGroup
##
volumePermissions:
  enabled: true
  image:
    registry: docker.io
    repository: bitnami/minideb
    tag: stretch
    pullPolicy: Always
    ## Optionally specify an array of imagePullSecrets.
    ## Secrets must be manually created in the namespace.
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    ##
    # pullSecrets:
    #   - myRegistryKeySecretName
  resources: {}

## forceBoot: executes 'rabbitmqctl force_boot' to force boot cluster shut down unexpectedly in an
## unknown order.
## ref: https://www.rabbitmq.com/rabbitmqctl.8.html#force_boot
##
forceBoot:
  enabled: false

## Optionally specify extra secrets to be created by the chart.
## This can be useful when combined with load_definitions to automatically create the secret containing the definitions to be loaded.
##
extraSecrets:
  {}
  # load_definition:
  #   load_definition.json: |
  #     {
  #       ...
  #     }

/verify-DCO

/check-dco

/ok-to-test

Looks good to me, @michaellee8. However, you have to sign your commits to pass the DCO check. Thank you very much.

@alemorcuq I signed all the commits with the line Signed-off-by: Michael Lee<EMAIL_ADDRESS>since the contribution guideline requires use of real names. However, I made my commit in my github username michaellee8.

According to the CI they are incorrectly signed, @michaellee8. See: https://github.com/helm/charts/pull/17946/checks?check_run_id=259446863

@alemorcuq I have edited the commit messages, can you give it a test again?

/retest

Thank you, @michaellee8!
@carrodher, can you proceed?

/ok-to-test
/lgtm


Update slack plugin

The old slack plugin was using the deprecated API. It's no longer possible to generate incoming webhook tokens, the API has moved to just use webhook URLs.
For the most part, this uses the exact same logic and layout, just updates for the new API format.

Closing this in favor of #79. I pulled your commit in then removed the vendored dependencies and changed the formatting a bit. Thanks for this!


Problem using dat-fetch in Firefox with Browserify

Trying to use dat-fetch in client-side JS
This is my hyper.js file:
async function fetchFunction(){
    const fetch = require('dat-fetch')()
    const someURL = `hyper://blog.mauve.moe`
    const response = await fetch(`${someURL}/index.json`)
    const json = await response.json()
    console.log(json)
}

fetchFunction()

Executing dat-fetch
I execute these commands on the shell to install the required dependencies and check that the script works:
npm install --save dat-fetch
npm install --save<EMAIL_ADDRESS>node hyper.js

And the script properly runs:
{
  title: 'RangerMauve - Blog',
  description: 'My Blog - Talking about Dat stuff',
  type: [ 'website' ],
  fallback_page: '/index.html'
}

Creating Browserify bundle
If I try to create the bundle with:
 browserify hyper.js -o bundle.js

I get an error:
browserify hyper.js -o bundle.js
Error: Cannot find module 'babelify' from '<my_path>/BrowserifyTest/node_modules/dat-sdk'
    at /usr/local/lib/node_modules/browserify/node_modules/resolve/lib/async.js:115:35
    at processDirs (/usr/local/lib/node_modules/browserify/node_modules/resolve/lib/async.js:268:39)
    at isdir (/usr/local/lib/node_modules/browserify/node_modules/resolve/lib/async.js:275:32)
    at /usr/local/lib/node_modules/browserify/node_modules/resolve/lib/async.js:25:69
    at FSReqCallback.oncomplete (fs.js:176:21)

So I proceed installing the specified "babilify" module:
npm install --save babelify

I get a warning: npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none is installed. You must install peer dependencies yourself. so I proceed:
npm install --save<EMAIL_ADDRESS>
I finally get the bundle and include it as a source in an index.html file:
<script src="bundle.js"></script>

When I open the file in a browser and I check the console, rather than getting the JSON from the hyperdrive I get:
Uncaught (in promise) Error: No native build was found for platform=browser arch=javascript runtime=node abi=undefined uv= libc=glibc node=undefined
    loaded from: /node_modules/sodium-native

Can somebody reproduce the error and give me some tips about what should be done in order to execute the dat-fetch API as part of a browser-first javascript proyect?

Hmm, something is up.
Try adding this babel config to your project: https://github.com/datproject/sdk/blob/master/babel.config.json
See if that helps make it compile properly. 😁

Actually, that might not be good either...
Try cloning dat-sdk, building a bundle there, and importing that bundle for now.
I'll need to work on making the workflow better there once I have spare cycles. (probably not for a few weeks at least)

Yeap, you are right. The babel.config did not work either :D
No rush. Just let me know whenever you have time to check it ;)
In those weeks I can just play with the dat-sdk. I had fun with your workshop at the 2020 Dat Conf :)

Gonna close this for now and track the issue in the dat-sdk repo. 😁


Errors running the example

Hi, After running any of both examples. I have the next error:
File "logistic_regression.py", line 4, in
import tf_seal as tfs
File "/home/ubuntusystem/Descargas/tf-seal/tf_seal/init.py", line 1, in
from tf_seal.python.tensor import constant
File "/home/ubuntusystem/Descargas/tf-seal/tf_seal/python/tensor.py", line 4, in
import tf_seal.python.ops.seal_ops as ops
File "/home/ubuntusystem/Descargas/tf-seal/tf_seal/python/ops/seal_ops.py", line 5, in
seal_ops = load_library.load_op_library(op_lib_file)
File "/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/load_library.py", line 61, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /home/ubuntusystem/Descargas/tf-seal/tf_seal/python/ops/_seal_ops.so: cannot open shared object file: No such file or directory
Why?, I have follow every step you have on readme section. Thanks in advance

Hi, After running any of both examples. I have the next error:
File "logistic_regression.py", line 4, in
import tf_seal as tfs
File "/home/ubuntusystem/Descargas/tf-seal/tf_seal/init.py", line 1, in
from tf_seal.python.tensor import constant
File "/home/ubuntusystem/Descargas/tf-seal/tf_seal/python/tensor.py", line 4, in
import tf_seal.python.ops.seal_ops as ops
File "/home/ubuntusystem/Descargas/tf-seal/tf_seal/python/ops/seal_ops.py", line 5, in
seal_ops = load_library.load_op_library(op_lib_file)
File "/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/load_library.py", line 61, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /home/ubuntusystem/Descargas/tf-seal/tf_seal/python/ops/_seal_ops.so: cannot open shared object file: No such file or directory
Why?, I have follow every step you have on readme section. Thanks in advance

Hi @CarlosMoralesSal, how did you install TF SEAL? Using the pip package or from source? In the latter case, have you first compiled the binary file?

If you mean, run the make in the latter case, yes and I have these errors as well
`bazel test ... --test_output=all
 Starting local Bazel server and connecting to it...
 ... still trying to connect to local Bazel server after 10 seconds ...
 ... still trying to connect to local Bazel server after 20 seconds ...
 INFO: Call stack for the definition of repository 'local_config_tf' which is a tf_configure (rule
 definition at /home/ubuntusystem/Descargas/tf-seal/external/tf/tf_configure.bzl:203:16):
  - /home/ubuntusystem/Descargas/tf-seal/WORKSPACE:32:1
 ERROR: An error occurred during the fetch of repository 'local_config_tf':
 Traceback (most recent call last):
File "/home/ubuntusystem/Descargas/tf-seal/external/tf/tf_configure.bzl", line 178
	_symlink_genrule_for_dir(repository_ctx, tf_header_dir, "incl...", ...")
File "/home/ubuntusystem/Descargas/tf-seal/external/tf/tf_configure.bzl", line 150, in _symlink_genrule_for_dir
	_norm_path(src_dir)
File "/home/ubuntusystem/Descargas/tf-seal/external/tf/tf_configure.bzl", line 122, in _norm_path
	path[-(1)]
    index out of range (index is -1, but sequence has 0 elements)
    INFO: Call stack for the definition of repository 'remote_java_tools_linux' which is a http_archive (rule definition at  /home/ubuntusystem/.cache/bazel/_bazel_ubuntusystem/27983eb97ba749e7bd2e448e234dfcaa/external/bazel_tools/tools/build_defs/repo/http.bzl:229:16):
     - /DEFAULT.WORKSPACE.SUFFIX:229:1
   ERROR: /home/ubuntusystem/Descargas/tf-seal/tf_seal/BUILD:5:1: no such package '@local_config_tf//': Traceback (most recent call last):
File "/home/ubuntusystem/Descargas/tf-seal/external/tf/tf_configure.bzl", line 178
	_symlink_genrule_for_dir(repository_ctx, tf_header_dir, "incl...", ...")
File "/home/ubuntusystem/Descargas/tf-seal/external/tf/tf_configure.bzl", line 150, in _symlink_genrule_for_dir
	_norm_path(src_dir)
File "/home/ubuntusystem/Descargas/tf-seal/external/tf/tf_configure.bzl", line 122, in _norm_path
	path[-(1)]

index out of range (index is -1, but sequence has 0 elements) and referenced by '//tf_seal:python/ops/_seal_ops.so'
ERROR: /home/ubuntusystem/Descargas/tf-seal/tf_seal/BUILD:5:1: no such package '@local_config_tf//': Traceback (most recent call last):
File "/home/ubuntusystem/Descargas/tf-seal/external/tf/tf_configure.bzl", line 178
_symlink_genrule_for_dir(repository_ctx, tf_header_dir, "incl...", ...")
File "/home/ubuntusystem/Descargas/tf-seal/external/tf/tf_configure.bzl", line 150, in _symlink_genrule_for_dir
_norm_path(src_dir)
File "/home/ubuntusystem/Descargas/tf-seal/external/tf/tf_configure.bzl", line 122, in _norm_path
path[-(1)]
index out of range (index is -1, but sequence has 0 elements) and referenced by '//tf_seal:python/ops/_seal_ops.so'
ERROR: Analysis of target '//tf_seal:seal_ops_py' failed; build aborted: no such package
'@local_config_tf//': Traceback (most recent call last):
File "/home/ubuntusystem/Descargas/tf-seal/external/tf/tf_configure.bzl", line 178
_symlink_genrule_for_dir(repository_ctx, tf_header_dir, "incl...", ...")
File "/home/ubuntusystem/Descargas/tf-seal/external/tf/tf_configure.bzl", line 150, in _symlink_genrule_for_dir
_norm_path(src_dir)
File "/home/ubuntusystem/Descargas/tf-seal/external/tf/tf_configure.bzl", line 122, in _norm_path
path[-(1)]
index out of range (index is -1, but sequence has 0 elements)
INFO: Elapsed time: 33.745s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (21 packages loaded, 150 targets c
FAILED: Build did NOT complete successfully (21 packages loaded, 150 targets c
`
Do you mean this?. Thanks

@CarlosMoralesSal can you confirm exactly what your environment looks like, things like: operating system, building tf seal vs. install with pip, docker etc.
I'd recommend also checking the following:

bazel version 0.26.1 (the docs say >= 0.26.1 but it should probably be changed to ==)
custom tensorflow build: https://github.com/tf-encrypted/tf-seal#custom-tensorflow install the one depending on your OS

Once that also seems good run:
make clean
make build
make test

For this case, I have downloaded the git repository and I have done the previous steps, I have created the conda environment, I have downloaded the whl file I have compiled it and then I have done the rest of things. I have the bazel version 0.26,1 and I have done the make things and I have this output
(tfseal) ubuntusystem@ubuntu:~/Descargas/tf-seal$ make build
TF_NEED_CUDA=0 ./configure.sh
WARNING: Package(s) not found: tensorflow
Using installed tensorflow
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorflow_core/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
bazel build build_pip_pkg
WARNING: /home/ubuntusystem/.cache/bazel/_bazel_ubuntusystem/27983eb97ba749e7bd2e448e234dfcaa/external/local_config_tf/BUILD:1625:1: target 'libtensorflow_framework.so' is both a rule and a file; please choose another name for the rule
INFO: Analyzed target //:build_pip_pkg (22 packages loaded, 170 targets configured).
INFO: Found 1 target...
ERROR: /home/ubuntusystem/.cache/bazel/_bazel_ubuntusystem/27983eb97ba749e7bd2e448e234dfcaa/external/seal/BUILD.bazel:7:1: error executing shell command: '/bin/bash -c function cleanup_function() {
local ecode=$?
if [ $ecode -eq 0 ]; then
cleanup_on_success
else
cleanup_on_failure
fi
}
set -e
function cleanup_on_success() {
printf "rules_foreign_cc: ...' failed (Exit 127)
rules_foreign_cc: Build failed!
rules_foreign_cc: Keeping temp build directory  and dependencies directory  for debug.
rules_foreign_cc: Please note that the directories inside a sandbox are still cleaned unless you specify '--sandbox_debug' Bazel command line flag.
rules_foreign_cc: Printing build logs:
_____ BEGIN BUILD LOGS _____
Bazel external C/C++ Rules #0.0.8. Building library 'lib'
Environment:______________BUILD_SCRIPT=bazel-out/k8-opt/bin/external/seal/lib/logs/CMake_script.sh
EXT_BUILD_ROOT=/home/ubuntusystem/.cache/bazel/_bazel_ubuntusystem/27983eb97ba749e7bd2e448e234dfcaa/execroot/main
BUILD_LOG=bazel-out/k8-opt/bin/external/seal/lib/logs/CMake.log
PWD=/home/ubuntusystem/.cache/bazel/_bazel_ubuntusystem/27983eb97ba749e7bd2e448e234dfcaa/execroot/main
TF_HEADER_DIR=/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorflow_core/include
TF_SHARED_LIBRARY_NAME=libtensorflow_framework.so.1
TMPDIR=/tmp
EXT_BUILD_DEPS=/tmp/tmp.AwuOdIWjl9
BUILD_TMPDIR=/tmp/tmp.ZQlAnE3jqn
SHLVL=2
TF_NEED_CUDA=0
INSTALLDIR=/home/ubuntusystem/.cache/bazel/_bazel_ubuntusystem/27983eb97ba749e7bd2e448e234dfcaa/execroot/main/bazel-out/k8-opt/bin/external/seal/lib
TF_SHARED_LIBRARY_DIR=/home/ubuntusystem/.local/lib/python3.7/site-packages/tensorflow_core
PATH=/home/ubuntusystem/.cache/bazel/_bazel_ubuntusystem/27983eb97ba749e7bd2e448e234dfcaa/execroot/main:/home/ubuntusystem/anaconda3/envs/tfseal/bin:/home/ubuntusystem/anaconda3/condabin:/home/ubuntusystem/.local/bin:/home/ubuntusystem/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
_=/usr/bin/env
__________________________bazel-out/k8-opt/bin/external/seal/lib/logs/CMake_script.sh: line 35: cmake: command not found
_____ END BUILD LOGS _____
rules_foreign_cc: Build script location: bazel-out/k8-opt/bin/external/seal/lib/logs/CMake_script.sh
rules_foreign_cc: Build log location: bazel-out/k8-opt/bin/external/seal/lib/logs/CMake.log
Target //:build_pip_pkg failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 3.533s, Critical Path: 1.37s
INFO: 1 process: 1 local.
FAILED: Build did NOT complete successfully
make: *** [Makefile:37: build] Error 1
Thanks

Problem is close, I had mixed python versions and everything went bad... I unistall everything and just installed python 3.7. I modified the configure file and everything ran.


Generate tray icons on a schedule

Signed-off-by: Sebastian Malton<EMAIL_ADDRESS>

Why do we need something like this?

This is not needed due to https://github.com/lensapp/lens/pull/7101


VS2012 don't undestand DllExport attribute

VS2012, C#.
Create ClassLibrary project, install DllExport 1.7.3 package, add function from example. Compiler can't find DllExport and DllExportAttribute.
What's wrong?

Namespace in DllExport.dll looks like this:
D3F00FF1770DED978EC774BA389F2DC901F4.B00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.C00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.D00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.E00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.F00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
DllExport.zip

Such situation shown in video.
In my case it's slightly wrong: DllExport.bat lays not in project path (as in video), but deeper in \packages\DllExport.1.7.3.
Running from there it breaks the project, so VS can`t find DllExport.bat.
To solve the problem: copy DllExport.bat into project path (near sln file), run it, click Apply.

@ak964
Not sure what you are trying to find in \packages folder but you need either getting DllExport.bat from here or activation from here
Your mentioned way is incorrect or just for not regular use.
Please simply follow the official steps - https://github.com/3F/DllExport/wiki/Quick-start
Thanks for using!
Please also note, this repo only for examples and for related things/problems.
You need to open an issue in the original project if you have something related to it - https://github.com/3F/DllExport/issues
Thanks,


chore: add potential protocols + btc currency

Summary
Add useful enumerations.

:tada: This PR is included in version 1.8.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.8.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Raw data map returns error when indexing without being attached

The appropriate behavior is to prompt an error that says the map is not attached, and offer the user instructions on how to attach the map.

Now queries isattached(obj.Parent) and outputs an error with instructions to attach.


Publish release

This looks like a handy tool. I'd love to create and AUR package for eva. It's best to build these against publish releases instead of straight from git master though. I would be great if you could tag a release.

TIL about drafting releases!
ill get to work on this as soon as possible!

done! i have released v0.2.1. let me know if everything is alright.
thanks a lot for showing interest in eva and maintaining an AUR package <3

Brilliant. Here's the package: https://aur.archlinux.org/packages/eva/


got view working for activities and chat view call identitiy will fil…

…l auto as group id and name

 Create activity view for group.


pip3 pkg provider on Python 3.11.2 fails without break-system-packages

Describe the Bug
on Python 3.11.2 (bookworm default) pip3 pkgs can't be installed without a specific flag --break-system-packages
Expected Behavior
pkgs to be installed without adding flags everywhere
Steps to Reproduce
Steps to reproduce the behavior:
install python 3.11 on bookworm
install a pkg with pip3 provider
Environment

Version [puppet 7.29.1]
Platform [Python 3.11.2, debian 12 bookworm]

Additional Context
Add any other context about the problem here.
i have to add this all over my codebase:
    provider        => 'pip3',
    install_options => '--break-system-packages',

Convinience over safety?

System-wide apps on a stable Debian installation are mainly managed by apt.
Puppet provides the ability to manage system-wide packages via apt (why pip at all?)
pip is just a random app, that should not break a system installation (that's what a lot of server admins expect) -- as an admin I don't care what anybody does with "--user", go and play there
Sytem-wide pip on Debian is a minefield

To me the flag is anything but a bug. I consider it a careful decision that the choice made by Debian regarding pip is mirrored in puppet.
Implicit handling of the flag would assume everbody is aware about the implications for the whole OS.
IMO should be closed as "invalid".

When we use puppet to configure a server the assumption is that I, as the one configuring puppet, know the ramifications of installing packages, from whatever source. The other assumption is that I’m competent enough to recover when things go sideways.
Maybe just a flag we can set in hiera would solve for both of us: I don’t have to whitelist every app individually, and you can have plausible deniability when somebody’s systems blow up?
It occurs to me that maybe this isn’t even the right forum for this discussion- is the pip package provider from puppet or is it part of the python module?
On Jul 4, 2024 at 4:48 PM -0500, Benjamin @.***>, wrote:

Convinience over safety?

System-wide apps on a stable Debian installation are mainly managed by apt.
Puppet provides the ability to manage system-wide packages via apt (why pip at all?)
pip is just a random app, that should not break a system installation (that's what a lot of server admins expect) -- as an admin I don't care what anybody does with "--user", go and play there
Sytem-wide pip on Debian is a minefield

To me the flag is anything but a bug. I consider it a careful decision that the choice made by Debian regarding pip is mirrored in puppet.
Implicit handling of the flag would assume everbody is aware about the implications for the whole OS.
IMO should be closed as "invalid".
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

It occurs to me that maybe this isn’t even the right forum for this discussion- is the pip package provider from puppet or is it part of the python module?

The pip3 (and related) providers are in puppet https://github.com/puppetlabs/puppet/blob/main/lib/puppet/provider/package/pip3.rb There are separate modules for managing pip and virtual environments like https://github.com/voxpupuli/puppet-python/blob/master/examples/pip.pp

expected behavior: pkgs to be installed without adding flags everywhere

Are you suggesting we add --break-system-packages by default? That seems like not a good idea in general. Can you provide more details about what package you're installing and the error you get?

@damonbreeden it's possible you were running into new behavior in pip3 that broke how puppet detects installed packages. We updated puppet's pip3 provider in https://github.com/puppetlabs/puppet/pull/9481 Can you test a puppet7 nightly build from https://nightlies.puppet.com/apt to verify --break-system-packages is no longer needed.

hi, i'm sorry, maybe i didn't explain the behavior correctly
this is new behavior in pip3 (externally-managed-environment):
$ pip3 install nano
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.

    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

i'm suggesting that puppet adds a flag that i can use to always add --break-system-packages to our pip3 pkgs

Hi @damonbreeden, thank you for reporting this issue. We do not anticipate addressing this as it represents a technical direction that we have decided not to follow at Puppet. As such, this issue will be closed as “Won’t Do”. If any watcher believes this is an error, please add a comment explaining.


BUG: Metasploit Fails to Upgrade/Install - metasploit.flare

Describe the bug and expected behavior
Metasploit fails on "cup all" or "cinst -y metasploit.flare --force"
To Reproduce
Trying to upgrade from CommandoVM Version 2020.1
"cup all" or "cinst -y metasploit.flare --force"
Output:
metasploit.flare package files install completed. Performing other installation steps.
Attempt to get headers for http://windows.metasploit.com/metasploit-framework-5.0.90%2B20200518152542.git.2.33708c2-1rapid7-1-x64.msi failed.
The remote file either doesn't exist, is unauthorized, or is forbidden for url 'http://windows.metasploit.com/metasploit-framework-5.0.90%2B20200518152542.git.2.33708c2-1rapid7-1-x64.msi'. Exception calling "GetResponse" with "0" argument(s): "The remote server returned an error: (404) Not Found."
File appears to be downloaded already. Verifying with package checksum to determine if it needs to be redownloaded.
Error - hashes do not match. Actual value was '03A22A4570FCC1A9F28C3360F4879DABBABB1BBD5F481B76D968566E4E435C15'.
Downloading metasploit.flare 64 bit
from 'http://windows.metasploit.com/metasploit-framework-5.0.90%2B20200518152542.git.2.33708c2-1rapid7-1-x64.msi'
ERROR: The remote file either doesn't exist, is unauthorized, or is forbidden for url 'http://windows.metasploit.com/metasploit-framework-5.0.90%2B20200518152542.git.2.33708c2-1rapid7-1-x64.msi'. Exception calling "GetResponse" with "0" argument(s): "The remote server returned an error: (404) Not Found."
The install of metasploit.flare was NOT successful.
Error while running 'C:\ProgramData\chocolatey\lib\metasploit.flare\tools\chocolateyInstall.ps1'.
See log for details.
Screenshots

Version

OS: Windows 10 Pro (1903)

Additional context
n/a

Got a little farther today running "cup all" but still not completely....

I ended up manually installing the latest Metasploit package directly from Rapid7 and it fixed.


feat(integrations): add ability to log learning rate using WandbMetricsLogger

Adds GROWTH2-77
Description
Add support for tracking learning rate with WandbMetricsLogger
Testing
How was this PR tested?
Checklist

[x] Include reference to internal ticket "Fixes WB-NNNN" and/or GitHub issue "Fixes #NNNN" (if applicable)
[x] Ensure PR title compliance with the conventional commits standards

cc: @ayulockin @dmitryduev
Re-raising https://github.com/wandb/wandb/pull/4378

@ayulockin I don't think that the WandbMetricsLogger callback from this branch is logging system metrics. Here's a sample run  for your reference.

Hey @dmitryduev should we have separate tests to cover the missing logics. Since we are only using yea test, should we have multiple yea tests to cover the different functionalities. What should be recommended here?

@ayulockin yea, adding more yea tests is probably the simplest thing to do here.


Double application

After adding the project as a dependency, when I run my application in an Android device it installs two applications, one of them is the actual application and the other one is the sample app from Progress Wheel. I cant figure out how to solve this.

add the project as module ,next clear the intent-filter in AndroidManifest in module ProgressWheel . I hope help you . regards

That was the way that I was doing it before but now i have it as a dependency in the build.gradle
repositories {
maven { url "https://jitpack.io" }
}
dependencies {
    compile 'com.github.Todd-Davies:ProgressWheel:1.0'
}

is there a way to solve this?

@mateobosco unfortunately until @Todd-Davies removes the launcher filter from the manifest you will have this issue since an app can only have one launcher filter (at least if you only want one icon).
However there's something you can do about it without having to add the project as a module.
Basically gradle merges all the manifests into one that you can find in build/intermediates/manifests/full and you can "override" the activity causing this issue. You only need to add the activity to your app manifest and remove the filters.
<activity
       android:name="com.todddavies.components.progressbar.Main"
      android:label="@string/app_name" >
     <intent-filter tools:node="removeAll" />
 </activity>
Add that to your Manifest and will be ok.

If it's urgent, you can submit a pull request. Otherwise, I'll get around to doing this at some point. Thanks!

Hey @mateobosco, @Orbycius, this should be fixed now :)

Is it fixed? I still can see two apps..

Are you using the latest version @sangrime? This should be fixed in e6241f97a.

@Todd-Davies Hi, By declaring the library version 1.0 in Gradle I still see two applications. How to use the latest version in gradle method?

Hello
How can I reslove this issue:
Failed to resolve: com.github.Todd-Davies:ProgressWheel:1.2
Thanks

I also can not resolve com.github.Todd-Davies:ProgressWheel:1.2

this theme is from 2015 four years a go :(
where is
todd davies

Hey, sorry for my inactivity; I'm rather busy and this library is a legacy from another part of my life. Finding the time to maintain it properly isn't really feasible right now.
My best advice would be to avoid using the library in the first place; it was created specifically for old versions of Android (~version 2), which according to [1] has ~0.2% market share right now. Simply using the built-in Android progress spinner is likely to be far more stable and compatible, and I'm sure there must be more up-to-date libraries if you want something more fancy.
I'll do my best to fix the dependencies/build in future, but can't promise when.
Have a lovely day!
[1] https://developer.android.com/about/dashboards

thanks for coming :) im a fan of you :)

Hey, sorry for my inactivity; I'm rather busy and this library is a legacy from another part of my life. Finding the time to maintain it properly isn't really feasible right now.
My best advice would be to avoid using the library in the first place; it was created specifically for old versions of Android (~version 2), which according to [1] has ~0.2% market share right now. Simply using the built-in Android progress spinner is likely to be far more stable and compatible, and I'm sure there must be more up-to-date libraries if you want something more fancy.
I'll do my best to fix the dependencies/build in future, but can't promise when.
Have a lovely day!
[1] https://developer.android.com/about/dashboards

hey todd the fix of two apk's work ?
if not work the wheel i can remove from my apk how ////thanks

I resolved this issue by donwngrading the version to
implementation 'com.github.Todd-Davies:ProgressWheel:1.0'
Cheers


[WIP] Feature/tiling

🚀 🚀 Pull Request
Checklist:

[ ]  My code follows the style guidelines of this project and the Contributing document
[ ]  I have commented my code, particularly in hard-to-understand areas
[ ]  I have kept the coverage-rate up
[ ]  I have performed a self-review of my own code and resolved any problems
[ ]  I have checked to ensure there aren't any other open Pull Requests for the same change
[ ]  I have described and made corresponding changes to the relevant documentation
[ ]  New and existing unit tests pass locally with my changes

Changes
meat:
ds.create_tensor("image", dtype="int")  # must define `dtype` (also works with compressors)

# create samples that exceed memory
ds.image.append_empty((10000, 10000))
ds.image.extend_empty((100, 10000, 10000))

# can also use for samples that fit in memory
ds.image.append_empty((10, 10))

# fill in data randomly
ds.image[0, 100:200, 300:400] = np.ones((100, 100))

# read subset of data
ds.image[0, 150:300, 350:500].numpy()

bonus:
ds.image.num_chunks  # returns the number of chunks + tiles
ds.image.set_dtype("float")  # allows you to set the dtype for a tensor if it doesn't have any samples yet

don't kill the branch