Adding ctranslate2 quantization and inference: moving the contribution

Thanks to https://github.com/huggingface/text-generation-inference/issues/726, I withdrew any support on the existing PR https://github.com/huggingface/text-generation-inference/pull/684, at least partially reviewed by @Yard1.
What does this PR do?
This PR extends introduces a new quantization method ctranslate2 (MIT-Licence), which is one of the remaining inference engines that are faster than plain PyTorch.
For comparison, ctranslate2 does everything in plain C++ and cuBLAS (chaining multiple forward passes in C++, next_token), making it up to 1.6x faster than TGI with Custom Kernels in Llama-2-13b-chat, for static (non-continuous) batch sizes, and also faster than vllm
The following features are used:

ctranslate2's decoder-only forward  method working for all supported (decoder-only architectures)[https://opennmt.net/CTranslate2/guides/transformers.html]
ctranslate2's quantization and native loading int8 / int8_float16 / float16 / bfloat16
devices: cpu and cuda

The following features are not available in the forward call:

despite no KV caching running quite fast (no cache supported in Python API of CT2
sharding. Implementing tensor parallel / model shared in CTranslate2 is still a open enchancement. This means that model needs to fit in memory once.

My measured improvements:

Plain GPU speedup (Codegen-2B) runs around 325% faster (measured time per token) while taking only 50% of the cuda memory. This is valid for models without special kernel / best effort /  AutoModel with  fp16 .
lower latency:
- First token latency: Due to the fact, that are no KV caches, especially short generations (max_new_token) are fast, e.g.
first token.
- Startup latency: Converted model weights are limited by SSD speed. My case: GPU 10x and CPU 4x faster startup times.
CPU-speedup:
-  support of native int8 e.g. with intel MKL -> memory savings
-  should be a faster (20-80%) compared to current torch implementations.
Wide compatibility: ctranslate2 has cross platform tests, and works across multiple Nvidia architectures (almost everything that supports cuda 11 or mkl)

How is this done:
I duplicated the causal_lm, and overwrote some parts of forward and init.
Summary:
This quantization and inference method is if you have one of the following:

inference on one GPU + no custom kernel available (not yet added or kernel compatibility with gpu)
inference on CPU
balance between low-memory usage and fast inference
low latency: server startup or first token latency

Fixes #496
Fixes  #680
Disclaimers

installs a new dependency with ctranslate2 (MIT) - itself it only bundles pybind11
I don't have a option to build the docker image on my laptop

Before submitting

[x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[x] Did you read the contributor guideline,
Pull Request section?
[x] Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
[x] Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
[x] Did you write any new necessary tests?

Who can review?
Anyone.
Benchmarking
Here are the numbers from some raw benchmarking done right on this branch, versus before the PR.

cuda: 11.872665ms (ct2) vs 39ms per token -> 325% of the original throughput
cpu: 105 vs 189ms per token -> 180% of the original throughput
while reducing memory foodprint by 45%
4772MiB (ct2_causal_lm.py) vs 6234MiB (causal_lm.py)
startup times from 44 to 4 seconds and 16 to 4 seconds.

setup:
new environment:
git checkout thisPR
make install
model=Salesforce/codegen-2B-mono

Note: without the special gpu kernels or quantisation, but for codegen or gpt-j they also do not exist.

Validation:
curl -X 'POST' \
  'http://localhost:3001/generate' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "inputs": "def",
  "parameters": {
    "best_of": 1,
    "decoder_input_details": false,
    "details": true,
    "do_sample": true,
    "max_new_tokens": 120,
    "repetition_penalty": 1.03,
    "return_full_text": false,
    "seed": null,
    "stop": [
    ],
    "temperature": 0.5,
    "top_k": 10,
    "top_p": 0.95,
    "truncate": null,
    "typical_p": 0.95,
    "watermark": false
  }
}'

CUDA
Option 1: pytorch + transformers AutoModel/causal_lm  + cuda
text-generation-launcher --model-id $model --port 3001

Internal logs:
Files are already present on the host. Skipping download. ..
 Shard ready in 44.764038287s rank=0
...
 INFO HTTP request{otel.name=POST /generate http.client_ip= http.flavor=1.1 http.host=localhost:3001 http.method=POST http.route=/generate http.scheme=HTTP http.target=/generate http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36 Edg/115.0.1901.183 otel.kind=server trace_id=9bee1198cc85e22be2ca0d6c106d766d}:generate{parameters=GenerateParameters { best_of: Some(1), temperature: Some(0.5), repetition_penalty: Some(1.03), top_k: Some(10), top_p: Some(0.95), typical_p: Some(0.95), do_sample: true, max_new_tokens: 120, return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: true, decoder_input_details: false, seed: None } total_time="4.761710836s" validation_time="366.275µs" queue_time="114.322µs" inference_time="4.761230741s" time_per_token="39.676922ms" seed="Some(17530934110773679293)"}: text_generation_router::server: router/src/server.rs:289: Success
 INFO HTTP request{otel.name=POST /generate http.client_ip= http.flavor=1.1 http.host=localhost:3001 http.method=POST http.route=/generate http.scheme=HTTP http.target=/generate http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36 Edg/115.0.1901.183 otel.kind=server trace_id=56e0978f15cd6d2bca86aba09df816a9}:generate{parameters=GenerateParameters { best_of: Some(1), temperature: Some(0.5), repetition_penalty: Some(1.03), top_k: Some(10), top_p: Some(0.95), typical_p: Some(0.95), do_sample: true, max_new_tokens: 120, return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: true, decoder_input_details: false, seed: None } total_time="4.798228559s" validation_time="394.134µs" queue_time="106.167µs" inference_time="4.79772878s" time_per_token="39.981072ms" seed="Some(8798374265046340001)"}: text_generation_router::server: router/src/server.rs:289: Success

Option 2: ctranslate2 + cuda
# shutdown server on 3001
text-generation-launcher --model-id $model --quantize ct2 --port 3000

Internal logs:
Files are already present on the host. Skipping download. ..
Shard ready in 4.407001409s rank=0
...
INFO HTTP request{otel.name=POST /generate http.client_ip= http.flavor=1.1 http.host=localhost:3000 http.method=POST http.route=/generate http.scheme=HTTP http.target=/generate http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36 Edg/115.0.1901.183 otel.kind=server trace_id=ff6117d217cba63b3cc524ef3221163c}:generate{parameters=GenerateParameters { best_of: Some(1), temperature: Some(0.5), repetition_penalty: Some(1.03), top_k: Some(10), top_p: Some(0.95), typical_p: Some(0.95), do_sample: true, max_new_tokens: 120, return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: true, decoder_input_details: false, seed: None } total_time="1.425193083s" validation_time="372.209µs" queue_time="101.416µs" inference_time="1.424719954s" time_per_token="11.872665ms" seed="Some(10419386700903774357)"}: text_generation_router::server: router/src/server.rs:289: Success
INFO HTTP request{otel.name=POST /generate http.client_ip= http.flavor=1.1 http.host=localhost:3000 http.method=POST http.route=/generate http.scheme=HTTP http.target=/generate http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36 Edg/115.0.1901.183 otel.kind=server trace_id=313b8e5334e61c50acf03df1f529dbcd}:generate{parameters=GenerateParameters { best_of: Some(1), temperature: Some(0.5), repetition_penalty: Some(1.03), top_k: Some(10), top_p: Some(0.95), typical_p: Some(0.95), do_sample: true, max_new_tokens: 120, return_full_text: Some(false), stop: [], truncate: None, watermark: false, details: true, decoder_input_details: false, seed: None } total_time="1.527237676s" validation_time="361.927µs" queue_time="101.318µs" inference_time="1.526774581s" time_per_token="12.723121ms" seed="Some(6326530321642227570)"}: text_generation_router::server: router/src/server.rs:289: Success

CPU
same as above, set line of code where model is moved to cuda, i.e. torch.cuda.is_available()
in the init. Use the default request params.
Option 1: pytorch + transformers AutoModel/causal_lm  + cpu
text-generation-launcher --model-id $model --port 3001

Internal logs:
Files are already present on the host. Skipping download. ..
 Shard ready in 16.422224256s rank=0
...
 Skipping download. ..
Shard ready in 4.105731024s rank=0
...
 INFO HTTP request{otel.name=POST /generate http.client_ip= http.flavor=1.1 http.host=localhost:3000 http.method=POST http.route=/generate http.scheme=HTTP http.target=/generate http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36 Edg/115.0.1901.183 otel.kind=server trace_id=2012707db82277e5a20583888f20c83b}:generate{parameters=GenerateParameters { best_of: Some(1), temperature: Some(0.5), repetition_penalty: Some(1.03), top_k: Some(10), top_p: Some(0.95), typical_p: Some(0.95), do_sample: true, max_new_tokens: 20, return_full_text: Some(false), stop: ["photographer"], truncate: None, watermark: true, details: true, decoder_input_details: true, seed: None } total_time="2.111624125s" validation_time="480.523µs" queue_time="104.529µs" inference_time="2.111039367s" time_per_token="105.551968ms" seed="Some(14277779136852605145)"}: text_generation_router::server: router/src/server.rs:289: Success
 INFO HTTP request{otel.name=POST /generate http.client_ip= http.flavor=1.1 http.host=localhost:3000 http.method=POST http.route=/generate http.scheme=HTTP http.target=/generate http.user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36 Edg/115.0.1901.183 otel.kind=server trace_id=637dd3c93de6df617a1f500db0bb0a83}:generate{parameters=GenerateParameters { best_of: Some(1), temperature: Some(0.5), repetition_penalty: Some(1.03), top_k: Some(10), top_p: Some(0.95), typical_p: Some(0.95), do_sample: true, max_new_tokens: 20, return_full_text: Some(false), stop: ["photographer"], truncate: None, watermark: true, details: true, decoder_input_details: true, seed: None } total_time="2.058076897s" validation_time="514.542µs" queue_time="142.167µs" inference_time="2.057420513s" time_per_token="102.871025ms" seed="Some(4392844796956118990)"}: text_generation_router::server: router/src/server.rs:289: Success

Note, tested with this on my 4090:
./target/release/text-generation-launcher --model-id TheBloke/Llama-2-13B-chat-fp16 --quantize ct2 --port 8080

Previously this model would not boot at all (trying to boot with quantize ct2 leads to out of memory)
Getting about 38ms per token.

Previously this model would not boot at all (trying to boot with quantize ct2 leads to out of memory)
Without ct2 leads to OOM?

This is very interesting. Since we have not yet currently set up a public CI, we will pull this PR to Preemo's internal repository and run CI.

Finally, all changes performed.

I think this PR looks good to me. I will test it and merge it if no error is found in tests.

I think it's due to ImportError on the line import ctranslate2. Maybe there should be some extra C/C++ dependencies for Dockerfile to successfully install ct2?

Can't reproduce. The issue you have should be resolved by pip install ctranslate2. The dependencies for C++ are all shipped with the pip installation of ctranslate2 / wheel.
I built a docker image for this, and pushed it to the public side of dockerhub.
docker build --build-arg="DOCKER_LABEL=0.9.4-ct2support" --build-arg="GIT_SHA=nosha" . -t michaelf34/tgi:0.9.4-ct2support

Locally, also
make install
cd server && pytest tests/models/test_ct2.py

(tgi) michi@michi:~/premotgi/server$ pytest tests/models/test_ct2.py
========================================================================================= test session starts
...

==========================================================================================
platform linux -- Python 3.9.17, pytest-7.4.0, pluggy-1.2.0
==================================================================================== 2 passed, 8 warnings in 8.19s =

Thanks for developing this. A lot of LLM frameworks such as LMQL, Outlines and Guidance could benefit from the HF wrapper of ctranslate2. What are your thoughts about having this as a PR to ctranslate2 directly? I made an issue about this last week on the ctranslate2 repo.

Hi @michaelfeil  i tried to use llama-2-7b-hf-chat model with ct2.
when i try to use your repo
run: |   docker run --gpus all --shm-size 1g -p 8082:80 -v ~/data:/data michaelf34/tgi:0.9.4-ct2support --model-id $MODEL_NAME --quantize ct2
envs:
MODEL_NAME: michaelfeil/ct2fast-Llama-2-7b-hf
i get the error
(task, pid=12662) 2023-09-30T02:50:13.304334Z ERROR download: text_generation_launcher: Download encountered an error: Traceback (most recent call last): (task, pid=12662)  (task, pid=12662)   File "/opt/conda/bin/text-generation-server", line 8, in <module> (task, pid=12662)     sys.exit(app()) (task, pid=12662)  (task, pid=12662)   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/cli.py", line 185, in download_weights (task, pid=12662)     utils.convert_files(local_pt_files, local_st_files, discard_names) (task, pid=12662)  (task, pid=12662)   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/convert.py", line 106, in convert_files (task, pid=12662)     convert_file(pt_file, sf_file, discard_names) (task, pid=12662)  (task, pid=12662)   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/convert.py", line 65, in convert_file (task, pid=12662)     loaded = torch.load(pt_file, map_location="cpu") (task, pid=12662)  (task, pid=12662)   File "/opt/conda/lib/python3.9/site-packages/torch/serialization.py", line 815, in load (task, pid=12662)     return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args) (task, pid=12662)  (task, pid=12662)   File "/opt/conda/lib/python3.9/site-packages/torch/serialization.py", line 1033, in _legacy_load (task, pid=12662)     magic_number = pickle_module.load(f, **pickle_load_args) (task, pid=12662)  (task, pid=12662) _pickle.UnpicklingError: invalid load key, '\x06'. (task, pid=12662)
it doesn't find the safe tensor files so its trying to convert them.
if try to use
`
run: |
docker run --gpus all --shm-size 1g -p 8082:80 -v ~/data:/data michaelf34/tgi:0.9.4-ct2support --model-id $MODEL_NAME --quantize ct2
envs:
MODEL_NAME: NousResearch/Llama-2-7b-chat-hf
`
i get an error
(task, pid=12634)     self._run_once() (task, pid=12634)   File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once (task, pid=12634)     handle._run() (task, pid=12634)   File "/opt/conda/lib/python3.9/asyncio/events.py", line 80, in _run (task, pid=12634)     self._context.run(self._callback, *self._args) (task, pid=12634) > File "/opt/conda/lib/python3.9/site-packages/text_generation_server/server.py", line 136, in serve_inner (task, pid=12634)     model = get_model( (task, pid=12634)   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/__init__.py", line 89, in get_model (task, pid=12634)     return CT2CausalLM( (task, pid=12634)   File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/ct2_causal_lm.py", line 135, in __init__ (task, pid=12634)     raise ValueError( (task, pid=12634) ValueError: conversion with ctranslate2 for NousResearch/Llama-2-7b-chat-hf failed : Error Vocabulary has size 32001 but the model expected a vocabulary of size 32000
does this support llama2 models?

how can i pass my HF TOKEN to access private and gated models? on tgi main docker build, setting HF_TOKEN as env variable works but it seems same doesn't work in michaelf34/tgi:0.9.4-ct2support build

I can merge it any way, but I cannot test this PR in our internal CI because our internal implementation is already partially refactored for modularization as we promised in README.
Once the internal version is ready to publish to this open-source repository, I expect that changes like this PR will need to migrate to new architecture.

how would it be possible to pass int8_float16 as dtype of ct2 conversion?
when i use  --dtype int8_float16 as cli arg get the below error
--dtype int8_float16

error: invalid value 'int8_float16' for '--dtype <DTYPE>'
  [possible values: float16, bfloat16]

  tip: a similar value exists: 'float16'


Error on 'go get' - impossible to build the suite

Hello,
I'm not knowledgeable on Go, but running the go get command gives multiple errors on uuid :
$ go version
go version go1.10.1 linux/amd64
$ go get -u github.com/ops-class/test161/test161
# github.com/ops-class/test161
go/src/github.com/ops-class/test161/build.go:137:30: multiple-value uuid.NewV4() in single-value context
go/src/github.com/ops-class/test161/build.go:624:23: multiple-value uuid.NewV4() in single-value context
go/src/github.com/ops-class/test161/conf.go:166:19: multiple-value uuid.NewV4() in single-value context
go/src/github.com/ops-class/test161/conf.go:395:28: multiple-value uuid.NewV4() in single-value context
go/src/github.com/ops-class/test161/conf.go:494:30: multiple-value uuid.NewV4() in single-value context
go/src/github.com/ops-class/test161/mongopersist.go:214:28: multiple-value uuid.NewV4() in single-value context
go/src/github.com/ops-class/test161/submission.go:313:18: multiple-value uuid.NewV4() in single-value context
go/src/github.com/ops-class/test161/submission.go:620:22: multiple-value uuid.NewV4() in single-value context
go/src/github.com/ops-class/test161/usage.go:56:29: multiple-value uuid.NewV4() in single-value context

A little of Googling led me to satori/go.uuid#66 that seems to be the origin of the bug, so I think the only way to fix it is to either change the included dependency or see how the API has changed, but I'm not sure.

I see following lines in generator.go.
func NewV4() (UUID, error) {
return global.NewV4()
}
The function is changed and it breaking the earlier references.

@shaseley: is there a good fix for this?


Added support for any() function as described. Fully tested and ...

updated the readme.

Thanks for your pull request :heart_eyes:
I'll check it out as soon as possible

Can you please fix issues returned by Codacy  ?

Sure. Will do.

There we are! All issues fixed

Closed by #13
Thank you very much for your contribution


All key bindings and contextual menu options stopped working

Hello. Seems something has changed and now Origami is not working. Contextual menu options are greyed out and keybindings don't trigger the commands.
In the console, I see this error:
reloading python 3.3 plugin Origami.origami Traceback (most recent call last):     File "/Applications/Sublime Text.app/Contents/MacOS/Lib/python33/sublime_plugin.py", line 308, in reload_plugin     m = importlib.import_module(modulename)     File "./python3.3/importlib/__init__.py", line 90, in import_module     File "<frozen importlib._bootstrap>", line 1584, in _gcd_import     File "<frozen importlib._bootstrap>", line 1565, in _find_and_load     File "<frozen importlib._bootstrap>", line 1532, in _find_and_load_unlocked    File "/Applications/Sublime Text.app/Contents/MacOS/Lib/python33/sublime_plugin.py", line 1692, in load_module     exec(compile(source, source_path, 'exec'), mod.__dict__)     File "/Users/figueroa/Library/Application Support/Sublime Text 3/Installed Packages/Origami.sublime-package/origami.py", line 83, in <module>     File "/Users/figueroa/Library/Application Support/Sublime Text 3/Installed Packages/Origami.sublime-package/origami.py", line 266, in PaneCommand AttributeError: 'module' object has no attribute 'Layout'
This may be related to another recent issue with the functionality.

Sorry about that. Released https://github.com/SublimeText/Origami/releases/tag/2.2.2 that should be available soon.

No need to apologize. I appreciate your work on this. In the meantime, I disabled auto updates in Sublime and reverted to an older copy of Origami.
Take care!


[RFC] Add Lua bindings

This work is based on top of ynezz/various-fixes and ynezz/custom-clientid-example branches. It's and RFC as I don't know if you'll accept it. It contains several fixes as well, I've found during debugging & testing sessions of the Lua bindings. Building of the bindings is not enabled by default.

Thanks.


Collections issue

Is the collections code referring correctly? Needs further investigation.

Might be an issue with this user, but the categories don't appear to be searched or filtered in the collections.


gap option

Is it possible to adjust the "gap" intensity?
Right now, each column feels to complete, I think there should be more gaps in between glyphs.
It should be random though, not every column will have the same amount of gaps and for a particular column maybe there are no gaps at all, maybe a gap of some length or even more than one gap at different locations

Your current available options are to modify raindropLength and fallSpeed, but what you really want is for the wobble function in the rain pass to accept another parameter.
I'll rename this ticket and keep it in mind as I spin up work on this project in the coming weeks/months.


do not use hardcoded /tmp, respect TMPDIR

Hello.
you may want to respect TMPDIR environnement varaible instead of hrd-coding /tmp paths for temporary files.
on some clusters TMPDIR points to some scratch location and /tmp is too small to hold temporary files
regards
Eric

@googlebot I signed it!

@googlebot I fixed it.

@googlebot I fixed it.

typo. will resubmit


Updating debugprotocol for simulator

This PR updates the debug protocol that the simulator uses to something a bit more usable. This is still very much a work in progress.

This PR looks good to me.


Add TP-Link support for device_tracker

Added support for TP-Link access points. Unfortunatly, the device doesn't support names of the associated devices, so it only returns the MAC-addresses.
I'm not totally sure in what format the MAC addresses should be replied. It currently is in the xx-xx-xx-xx-xx-xx format, since that's what is returned from the access point.
I'm not a Python-developer, code is based on the other device_trackers and some googling. It's working fine in my branch.

Sorry for the code style commits, I wasn't familiar with the guidelines. I hope the test run fine now.

Cool!
One little PyLint warning left and I suggested some code to return the MAC addresses in the correct format. When both fixed it should be ready to be merged!

Awesome, merging it in, thanks!


[<IP_ADDRESS>] Fast downward movement broken

From the <IP_ADDRESS> changelog:

Disabled anti-cheat temporarily, until the movement bugs are fixed

I guess this change was made to fix the random damage ticks? But it seems to have broken Elytras, falling from great heights, etc. (basically: everything where you move downwards fast). Just do a simple

/tp @p ~ 300 ~

and while fallig down you'll experience really bad stuttering as if the fall-speed is reset to 0. The same happens if you fly downwards using an elytra - its even worse and movement becomes were choppy.

Fixed for next release.


Apis and Microservices Projects - Timestamp Microservice - requires both glitch and github url

Describe your problem and - if possible - how to reproduce it
This issue exists for all the Apis and Microservices Projects
The instructions state to use glitch OR github. But, both glitch AND github fields are required.
Add a Link to the page with the problem
https://learn.freecodecamp.org/apis-and-microservices/apis-and-microservices-projects/timestamp-microservice
https://learn.freecodecamp.org/apis-and-microservices/apis-and-microservices-projects/request-header-parser-microservice
https://learn.freecodecamp.org/apis-and-microservices/apis-and-microservices-projects/url-shortener-microservice
https://learn.freecodecamp.org/apis-and-microservices/apis-and-microservices-projects/exercise-tracker
https://learn.freecodecamp.org/apis-and-microservices/apis-and-microservices-projects/file-metadata-microservice
Tell us about your browser and operating system

Browser Name: Chrome
Browser Version: Version 66.0.3359.181 (Official Build) (64-bit)
Operating System: Manjaro 17.1.10

If possible, add a screenshot here

I think this isn't bug but misunderstanding. You need give link for solution (that can be outside of Glitch) and link for GitHub repository that have your solutions source code.

@P1xt I agree with @JKorhonen. It asks for a link of solution (where you project is hosted) and Github link (for the source code). It encourages campers to put their project on Github which is nice in my opinion.

Glitch is not required, just a link to a working solution of your project - github, however, is required i think - so perhaps a little bit of rewording could make it more clear - I think its fine - @P1xt any suggestions?

Last time I checked it was fixed. There is only one field now - Solution.

still shows two fields for me when I click any of the links above

You're right @moT01. Still two input fields for the final projects. It was fixed for the challenges in the module though.

Glitch is not required, just a link to a working solution of your project - github, however, is required I think - so perhaps a little bit of rewording could make it more clear - I think its fine - @P1xt any suggestions?

@moT01 The challenge text does say that

Optionally you may choose to write your project on another platform but it must be publicly visible for our testing.

I think it's already clear that Glitch is not required as long as you host it on a platform which is publicly visible for the testing.

@anku255 - yes, I saw that - it's clear to me glitch is not required - what I was getting at is that it's not necessarily clear that github is required, as it says you can use glitch or github

@moT01 Ah, I see. I am not sure why Github is required though. It doesn't even save the Github link in the settings page.

The way I read the challenge, you can either use glitch via the provided link, OR you can use the github link, clone the repo, and host the project wherever you want. It says nothing about "if you do this on glitch you need to download your project from glitch, create a github repo, push your glitch code into that repo, then submit both the glitch and github links.
I'm perfectly fine with "clone this on github, write your code, deploy your code wherever you want, then link both your github and live version". My point is that that is not what the challenge says.

Perhaps the place to put the github URL should be removed - Because the challenge page asks for two URL's (live demo + github), but in the settings area you can turn in any of the projects with just the live demo link

Same here, it's not clear and demands the link to github.

It's not clear that both fields are required!
Also, following the other challenges, it is easily assumed that the same thing about (glitch or github) will apply.
imo, glitch or github should only be required since the code in glitch can be viewed using the /edit/!# route.

I am closing this issue.
Why? Because it is an issue, the instructions are wrong - and, yet, no one has any intention of ever fixing it. Any contributor who's posted in this thread just says 'well if you do it this way, it works' when the entire point of the issue is that 'this way' is not what the instructions say to do.
And, since it's never going to get fixed - I'm sick of getting notifications every time someone posts that it's still an issue. If someone else wants to report it, create your own issue, it's sad to watch a legitimate issue just stagnate for over a year. I'm done watching it.


Support MPI job by kube-batch

For now, we support several feature for batch job. And MPI is a typical batch job we need to support. In 0.3, we'd like to get MPI job run by kube-batch.

We should test it with https://www.kubeflow.org/docs/guides/components/mpi/

What are your thoughts on creating another CRD like "mpiJob.scheduling.incubator.k8s.io" which then creates a:

k8s job with the kube-batch scheduler
podGroup for the job
Headless Service that targets all pods for the job
Network Policy that locks down communication between job-pods (optional/configurable)

The pods themselves have an init container that will dig the headless service, grab all the pods, and generate the mpi "hostfile"
From there we should be able to run mpiexec successfully.

My original thoughts of this issue is to build such kind of mpijob; I'm considering mpi-operator because I may not have enough effort on it :(
If you can contribute, that'll be great :)

I'd love to contribute to this. :)
Were you looking to re-implement the kube-flow MPI operator or were you looking to start from scratch?
If from scratch, what are your thoughts on my above idea?

I prefer to start it from scratch; similar to your above idea; in additional, we need to provide index to each 'pod/task', and prefer not to use StatefulSet which is used for long-running service, IMO.

I intend on cleaning the code up, then creating a PR into kube-batch, but it works.

That's great !
Just go through the code, it seems we use configmap + /etc/hosts; is it possible to use headless service?

I ran into a problem with that: The service endpoints don't populate till the pods are actually in a "running" state.
With what I've learned so far, it's likely I could probably make headless services work, it'd just be another object that we'd manage.
Thoughts?

Heya @k82cn -- Any further thoughts on this?
Thanks!

@jeefy , sorry for the delay, I'm testing that. keep you posted later.

/close
kube-batch only focus on scheduling part, so it's out of scope to this repo :)


better UI for title + bug fix for auto link

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s).
To prevent this PR from going to the changelog marked it with the "skip changelog" label.

Snapshot stored with reference name:
refs/pull/13511/merge
Test environment:
https://babylonsnapshots.z22.web.core.windows.net/refs/pull/13511/merge/index.html
To test a playground add it to the URL, for example:
https://babylonsnapshots.z22.web.core.windows.net/refs/pull/13511/merge/index.html#WGZLGJ#4600
Links to test babylon tools with this snapshot:
https://playground.babylonjs.com/?snapshot=refs/pull/13511/merge
https://sandbox.babylonjs.com/?snapshot=refs/pull/13511/merge
https://gui.babylonjs.com/?snapshot=refs/pull/13511/merge
https://nme.babylonjs.com/?snapshot=refs/pull/13511/merge
To test the snapshot in the playground with a playground ID add it after the snapshot query string:
https://playground.babylonjs.com/?snapshot=refs/pull/13511/merge#BCU1XR#0


Add different exit codes in rm, rmi and inspect commands

Currently, the inspect command does not return any specific exit code when something goes wrong and rm/rmi commands only return 1 when specified container/image does not exist. In order for toolbox to be able to give more specific error messages when image/container couldn't be removed, we need separate exit codes.
In this PR, I modified the inspect command so that it returns 1 when the specified image/container was not found (consistent with exit code 1 in rm/rmi). The rm command returns 2 when the specified container is paused or running and the rmi command returns 2 when the specified image has child images or is being used by a container.
Feel free to suggest any other exit codes if the ones I chose are unsuitable. Also, this is my first contribution here, so please excuse any silly mistakes.
Signed-off-by: Ondrej Zoder<EMAIL_ADDRESS>

/approve

I am fine with the podman rm/rmi changes, I am not crazy about the inspect changes.

LGTM except the inspect error message.  Would like to see something better there.

Needs a rebase to get the SELinux fixes.

Where are we on this one? @rhatdan Are you satisfied on the changes?

I am fine with it.

Stopping and removing paused containers is broken in CI - potentially exit code changes?

Restarted the tests to check what's going on as they pass locally on my machine.

@ozoder The e2e tests are failing since you changed the return code from 125 to 2.  Do you need assistance in fixing those tests to pass with your code changes?

@ozoder The e2e tests are failing since you changed the return code from
125 to 2. Do you need assistance in fixing those tests to pass with your code
changes?

@ozoder has gotten caught up with university, so I took the liberty to try and fix up the tests.
It looks like I can't force-push to his branch, so I put the fixes in PR #3077

Closing in favor of #3113


[Feature Request] Clarification of the release cycle

Is your feature request related to a problem? Please describe.
Hi all
Guys, thanks a lot for the .net sdk
Unfortunately, it's been about 3 months since the first version of the sdk was released in nuget,. Nevertheless, it can be seen that some useful fixes have been made (and work on .net sdk continues). Based on this, I have a quick question
Have you ever thought about describing the release cycle of .net sdk in some way ? This would help a lot in waiting for certain features or bug fixes. For example, this could be done with projects or milestones on github.
What do you think about this ?
Thank you

Have you ever thought about describing the release cycle of .net sdk in some way ? This would help a lot in waiting for certain features or bug fixes.

We don't have a fixed release cycle since there are not enough changes to justify one (as there were for past SDKs where we were adding a lot of features). Therefore we try not to have public milestones to build expectations. Basically it'd just be a single "next" milestone with no specific time set.
We tend to release as issues warrant. We will happily field requests to release sooner (here, or maybe even better, on Slack in the #dotnet-sdk channel).

(closing as question, but feel free to continue to discuss here or on Slack or wherever)


qemu: Show full path to qemu binary at launch time

Rather than show the generic "qemu", log the full path to the
particular qemu binary being used.
Signed-off-by: James O. D. Hunt<EMAIL_ADDRESS>

Coverage remained the same at 79.855% when pulling 46970781fae7623434406bab5cb9ebf1c387f48d on jodh-intel:show-qemu-path-on-launch into f03df80fc3dc52f65ed6c7d12806279b44185d32 on intel:master.


Parallel transaction verification

While investigating block import time spikes in snarkOS, profiling showed that DPCScheme::verify_transactions accounted for a significant part of that process; I was wondering if it could be improved and noticed that it was run sequentially, while it could be parallelized - it just required some tweaks to traits. I was being conservative and only added Sync (and sometimes Send) where the compiler required it.

This looks to be desirable changes. Could you rebase this for testnet2 branch?

Done; it was much simpler with just testnet2, too :ok_hand:.

Beautiful!


Trezor Integration

Funds were raised for Trezor integration and I need to work on getting it functioning again.

I have some XCP stucked in my Trezor, tried to sign a tx or watch wallet but Trezor pop-up dont works.

I don't want to bother you, but do you think this feature is feasible? Regards.

Hi @jdogresorg
I just discovered Freewallet and I am saving up for the 'full access' version of the app. Please let me know if there's a better way to support your work. Thank you for the ongoing maintenance of this project!
I would greatly appreciate a Trezor integration. I have a 'rare pepe' that is currently inaccessible as it's in my Trezor Model T hardware wallet. It was my fault for sending it there without checking first! There is a final word, a.k.a. 'passphrase' at the end of the seed, if that matters for the integration.
Thanks again!

Hi @jdogresorg
I appreciate all of your work on this and I am saving up for the 'full access' version of the app. Please let me know if there's a better way to support your work. Thank you for the ongoing maintenance of this project!
I would greatly appreciate a Trezor integration. I have a 'rare pepe' that is currently inaccessible as it's in my Trezor Model T hardware wallet. It was my fault for sending it there without checking first! There is a final word, a.k.a. 'passphrase' at the end of the seed, if that matters for the integration.
Thanks again!

Trezor integration seems to work now. I have been able to send a TOREROPEPE from a Trezor T native Segwit address to a standard legacy address.

trezor integration should be working now


Deleting individual key, or whole group of keys

Once something is added to the config. how can we programmatically remove it?

There's no way to effectively remove a record from the config. It would mean to implement a remove or discard method like in sets, but no, it's not implemented. As a workaround you can set a record to None and act accordingly.


Creating multiple runtimes with the same seed yields different random values

Steps to reproduce:
Sequentially create multiple runtimes from madsim::runtime::Builder with the same seed
Print getrandom::getrandom in each simulation
The first value is different, then all other values are the the same, e.g.
[114]
[139]
[139]
[139]
...

Reproducer

Hi, thanks for your feedback! We have fixed this bug and released a new version 0.2.27.

Great, thanks. I found this issue while I was verifying determinism manually, but I will create a follow-up issue for it.


Notebook files larger than usual due to saving SVG's for plotviewer

Bug: Notebook Editor, Interactive Window, Editor cells

Steps to cause the bug to occur

Working with Notebook on Visual Studio Code

Actual behavior
Asides from the fact that my VSCode has began to slow down drastically, predictions are really slow.
My notebook file of 696KB gets as large as 53mb when I re-run it on my VSCode. This makes loading it up take longer. How can I resolve this?

Your Jupyter and/or Python environment
Please provide as much info as you readily know

Jupyter server running: Local
Extension version: 2020.8.101144
VS Code version: 1.48.0
Setting python.jediEnabled: false
Setting python.languageServer: Jedi | Microsoft | None
Python and/or Anaconda version: 4.8.3
OS: Windows
Virtual environment: N/A

Python Output

Microsoft Data Science for VS Code Engineering Team: @rchiodo, @IanMatthewHuff, @DavidKutu, @DonJayamanne, @greazer, @joyceerhl

I ran into the same issue and I tried setting that flag in VSCode's version 1.53.2 and it says it is an unknown configuration setting. Does anyone have a work around or is this flag supposed to work in the most up-to-date version of VSCode?

The new setting is
"jupyter.enablePlotViewer": "false"

We used to have our code in the python extension when this bug was originally created.

The setting:
"jupyter.enablePlotViewer": false
worked like a charm for me. Just for context, the issue I was running into was not having GitHub render my notebook created from VSCode because of the massive file size multiple SVG plots were creating. That setting reduced my file size by about 80%. Thank you so much for the quick response @rchiodo.

The fix works great, but I would never have guessed that this option is going to bloat my (plot-heavy) notebooks by a factor of 30 in size. If the option were called storePlotsAsSVG that would have caught my attention. Or maybe it should default to false, I have not found the plot viewer to be all that useful and as mentioned it can break viewing on GitHub.

Now we just generate PNGs by default and show PNGs in the plot viewer. You need to opt into the SVGs explicitly.


"Defaultify"

In #33, @mpvl  does IMHO a very good job at highlighting the core difference between cue and jsonnet.
I like cue very much; it really has the potential to tame the complexity of configs.
Its generative aspect works well for templating when the templates have been explicitly designed for extension (and arguably forcing the author to be explicit is instrumental for the success of taming the complexity in the first place)
One of my main annoyances with Helm, is that chart authors are effectively forced to expose every possible field of every k8s object as values in the values.yaml template, because sooner or later some consumer of that chart (usually from outside the administrative domain of the chart author) will have a reasonable use case for the override.
In jsonnet, you can just take an upstream yaml or json object, parse it and apply overrides with jsonnet.
For cue to be useful in this scenario, would we need a function that converts a cue object tree into an object tree that replaces every field value x with *x | _.
Then we could do something like (I know that "Defaultify" is a horrible name, cannot currently think of a better name right now):
import (
  "encoding/yaml"
  "override"
)

res: override.Defaultify(yaml.Unmarshal("""
     foo:
       bar:
         a: 1
         b: 2
  """
)) & {
  foo bar b: 10
}

once an external value is lifted into cue value model, it can be further refined applying type and value limitations that make sense to the user.
Perhaps I'm missing something. Is there already a way to achieve this today?
(I saw there is an interesting approach for importing external legacy configs based on source level imports, but I think that solves a different use case)

@mkmik Is this similar to what you're looking for? https://github.com/cuelang/cue/issues/15

Yes thanks, closing as duplicate of #15

This issue has been migrated to https://github.com/cue-lang/cue/issues/59.
For more details about CUE's migration to a new home, please see https://github.com/cue-lang/cue/issues/1078.


Separate function of left and right panels

As proposed in the mailing list we implemented some modifications to the three panels view.
We separated the functionalities : We add every kind of items with the left panel, and all the configuration is done with the right panel.
We also add a way to reduce/close the right panel. It will stay closed until voluntarily reopened.
You can see the result in the following video : https://webmshare.com/ayKMQ

@sferot Could you rebase this against master please, it would be nice to get this merged


author normalize problem

currently, some of the authors can not be normalized , e.g.
[[Workshop] FAIR data handling for microscopy: Structured metadata annotation in OMERO]
Authors: Fuchs, Vanessa Aphaia Fiona, Schmidt, Christian, Boissonnet, Tom

Correct, done in #170


salt state installs failing on linux-6|9|10|11

check on salt state install on linux-6|9|10|11

Ran state.apply from linux-6 and it applied correctly.

linux-10, accepted salt key, and states applied successfully.
linux-11 applied states correctly.


Support GCC 9.1.0

Rebased the patches for GCC 8.3.0 onto 9.1.0. Not all applied cleanly, particularly the PIE patch so I had to get a little creative. Could definitely use an extra pair of eyes on these patches.
Independent verification of the SHA1 is always appreciated.
Verified compile with the following config.mak:
BINUTILS_VER = 2.32
GCC_VER = 9.1.0
GMP_VER = 6.1.2
MPC_VER = 1.1.0
MPFR_VER = 4.0.2
ISL_VER = 0.21
LINUX_VER = 3.0.35

COMMON_CONFIG += --disable-nls
GCC_CONFIG += --enable-languages=c,c++
GCC_CONFIG += --disable-libquadmath --disable-decimal-float
GCC_CONFIG += --disable-multilib

I should add that this is clearly a cocktail of the other PRs I submitted earlier once you look at the config.mak I used and I haven’t established the true minimum versions of say, MPFR, for building GCC 9.1.0. Because of this I wanted to keep each of the version update PRs separate since it’s completely reasonable one could be rejected and another accepted. It also doesn’t make sense to hold up accepting a PR to update a support library like MPFR while we determine if the patches are good for this GCC version. Distinct PRs for distinct concerns, as it were.

Thanks for splitting them up that way. For future reference I think it's fine to do all the libs as a single PR, since there shouldn't be anything controversial or needing review/revision on them. I've asked others to take a look at this PR, and I'll look at it soon myself too, probably after the musl release.

Sorry this got left so long. Others have been asking about it recently, so I came back and looked and found gcc at 9.2 now. nsz's patches from http://port70.net/~nsz/musl/gcc-9.1.0/ applied cleanly still, so I've added this as gcc 9.2.0 based on them and skipped 9.1.0.


[Bug Report] Still hitting "too many values to unpack (expected 2)" for env.reset() with several recent versions (0.26.x)

Describe the bug
I am hitting this error in all my attempts, after upgrading from gym 0.21 (I had a hard dependency on stables-baselines3) to any of the following versions (installing through Conda):

gym 0.26.1
gymnasium 0.26.3
gymnasium 0.27.1
I have removed all pycache files, removed all references to gym and deleted all old versions of gym from both Conda and Pip, but still this error pops up.
I have build a custom environment and implemented a custom DQN (this one).

Code example
Not sure what code to share here and I feel code won't be very useful. Let me know if there's any code that could help you debugging this. Attaching error stack trace:

  ~/Desktop/UNI/Q2/IntroToResearch/RL-FloorPlan/v11   master *2 !5 ?4 ❯ make run-train                                                                                  intro2research
python src/agent.py --data-file INC_253_1.png --num-rooms 4 \
	--reward-fn new_heuristic --test false --timesteps 100000  --step 10 --steps-per-episode 256
Traceback (most recent call last):
  File "/Users/guillem.rossello/Desktop/UNI/Q2/IntroToResearch/RL-FloorPlan/V11/src/agent.py", line 52, in <module>
    run()
  File "/Users/guillem.rossello/opt/anaconda3/envs/intro2research/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/guillem.rossello/opt/anaconda3/envs/intro2research/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/guillem.rossello/opt/anaconda3/envs/intro2research/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/guillem.rossello/opt/anaconda3/envs/intro2research/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/guillem.rossello/Desktop/UNI/Q2/IntroToResearch/RL-FloorPlan/V11/src/agent.py", line 48, in run
    new_training(config=config, timesteps=timesteps, fname_model=fname_model)
  File "/Users/guillem.rossello/Desktop/UNI/Q2/IntroToResearch/RL-FloorPlan/V11/src/training.py", line 16, in training
    trainer = TrainerRL(config)
  File "/Users/guillem.rossello/Desktop/UNI/Q2/IntroToResearch/RL-FloorPlan/V11/src/trainer_rl.py", line 50, in __init__
    state = self.env.reset()
  File "/Users/guillem.rossello/opt/anaconda3/envs/intro2research/lib/python3.9/site-packages/gymnasium/core.py", line 457, in reset
    obs, info = self.env.reset(**kwargs)
ValueError: too many values to unpack (expected 2)
make: *** [run-train] Error 1

System info
Gymnasium installed with Conda, project is executed within this Conda environment. I'm on MacOS Monterey, using Python 3.9.13. Showing logs for my last attempt, but as mentioned earlier I've hit this same error with either gym 0.26.1, gymnasium 0.26.3 and gymnasium 0.27.1
  ~/Desktop/UNI/Q2/IntroToResearch/RL-FloorPlan/v11   master *2 !5 ?4 ❯ conda list gym                                                                                  intro2research
packages in environment at /Users/guillem.rossello/opt/anaconda3/envs/intro2research:

Name                    Version                   Build  Channel
gymnasium                 0.26.3           py39hecd8cb5_0
gymnasium-notices         0.0.1            py39hecd8cb5_0
  ~/Desktop/UNI/Q2/IntroToResearch/RL-FloorPlan/v11   master *2 !5 ?4 ❯ pip show gym                                                                                  intro2research
WARNING: Package(s) not found: gym
Additional context
Is this a Conda-related issue? I was using Conda because the project I joined was originally using stable-baselines3, but I really want to migrate to Poetry because honestly I see no advantages using Conda (and a lot of engineering pains). Is Poetry a good fit for Gymnasium?
Checklist

[X] I have checked that there is no similar issue in the repo

It looks like
  File "/Users/guillem.rossello/Desktop/UNI/Q2/IntroToResearch/RL-FloorPlan/V11/src/trainer_rl.py", line 50, in __init__
    state = self.env.reset()

which I suspect will be the return type, not state, info.
As for Poetry, this has been investigated here https://github.com/Farama-Foundation/Gymnasium/pull/86

Hi, thanks for the quick reply! Not sure I understand what you mean, I did try both
state = self.env.reset() and state, info = self.env.reset(), although I feel this is not what you mean.

I migrated the project to Poetry, this is the dependency tree of the project:
click 8.1.3 Composable command line interface toolkit
└── colorama *
glob2 0.7 Version of the glob module that can capture patterns and supports recursive wildcards
gymnasium 0.28.1 A standard API for reinforcement learning and a diverse set of reference environments (formerly Gym).
├── cloudpickle >=1.2.0
├── farama-notifications >=0.0.1
├── importlib-metadata >=4.8.0
│   └── zipp >=0.5
├── jax-jumpy >=1.0.0
│   └── numpy >=1.18.0
├── numpy >=1.21.0
└── typing-extensions >=4.3.0
matplotlib 3.7.1 Python plotting package
├── contourpy >=1.0.1
│   └── numpy >=1.16
├── cycler >=0.10
├── fonttools >=4.22.0
├── importlib-resources >=3.2.0
│   └── zipp >=3.1.0
├── kiwisolver >=1.0.1
├── numpy >=1.20
├── packaging >=20.0
├── pillow >=6.2.0
├── pyparsing >=2.3.1
└── python-dateutil >=2.7
    └── six >=1.5
pygame 2.3.0 Python Game Development
torch 2.0.0 Tensors and Dynamic neural networks in Python with strong GPU acceleration
├── filelock *
├── jinja2 *
│   └── markupsafe >=2.0
├── networkx *
├── nvidia-cublas-cu11 <IP_ADDRESS>
│   ├── setuptools *
│   └── wheel *
├── nvidia-cuda-cupti-cu11 11.7.101
│   ├── setuptools *
│   └── wheel *
├── nvidia-cuda-nvrtc-cu11 11.7.99
│   ├── setuptools *
│   └── wheel *
├── nvidia-cuda-runtime-cu11 11.7.99
│   ├── setuptools *
│   └── wheel *
├── nvidia-cudnn-cu11 <IP_ADDRESS>
│   ├── setuptools *
│   └── wheel *
├── nvidia-cufft-cu11 <IP_ADDRESS>
├── nvidia-curand-cu11 <IP_ADDRESS>
│   ├── setuptools *
│   └── wheel *
├── nvidia-cusolver-cu11 <IP_ADDRESS>
│   ├── setuptools *
│   └── wheel *
├── nvidia-cusparse-cu11 <IP_ADDRESS>
│   ├── setuptools *
│   └── wheel *
├── nvidia-nccl-cu11 2.14.3
├── nvidia-nvtx-cu11 11.7.91
│   ├── setuptools *
│   └── wheel *
├── sympy *
│   └── mpmath >=0.19
├── triton 2.0.0
│   ├── cmake *
│   ├── filelock *
│   ├── lit *
│   └── torch * (circular dependency aborted here)
└── typing-extensions *

And I'm now hitting the following seed parameter error (that if I'm not mistaken was addressed in previous versions, not even gymnasium versions but gym versions!).
Traceback (most recent call last):
  File "/Users/guillem.rossello/Desktop/UNI/Q2/IntroToResearch/RL-FloorPlan/V11/src/agent.py", line 62, in <module>
    run()
  File "/Users/guillem.rossello/Library/Caches/pypoetry/virtualenvs/rl-floorplan-VxTjqtjI-py3.9/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/guillem.rossello/Library/Caches/pypoetry/virtualenvs/rl-floorplan-VxTjqtjI-py3.9/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/guillem.rossello/Library/Caches/pypoetry/virtualenvs/rl-floorplan-VxTjqtjI-py3.9/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/guillem.rossello/Library/Caches/pypoetry/virtualenvs/rl-floorplan-VxTjqtjI-py3.9/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/guillem.rossello/Desktop/UNI/Q2/IntroToResearch/RL-FloorPlan/V11/src/agent.py", line 58, in run
    new_training(config=config, timesteps=timesteps, fname_model=fname_model)
  File "/Users/guillem.rossello/Desktop/UNI/Q2/IntroToResearch/RL-FloorPlan/V11/src/training.py", line 16, in training
    trainer = TrainerRL(config)
  File "/Users/guillem.rossello/Desktop/UNI/Q2/IntroToResearch/RL-FloorPlan/V11/src/trainer_rl.py", line 50, in __init__
    state = self.env.reset()
  File "/Users/guillem.rossello/Library/Caches/pypoetry/virtualenvs/rl-floorplan-VxTjqtjI-py3.9/lib/python3.9/site-packages/gymnasium/core.py", line 462, in reset
    obs, info = self.env.reset(seed=seed, options=options)
TypeError: reset() got an unexpected keyword argument 'seed'

How can this even happen? I'm printing the gymnasium version right before the reset that triggers the error (the same line 50 in trainer_rl.py of the original error in this issue) and it is 0.28.1.

Actually I made an error when I reported this "bug": with the originally explained Conda setup and Gymnasium 0.27.1 I hit this same seed error, not the one I shared at the beginning. Just sharing more data to see if we can find the cause of these issues. Thanks beforehand for the support.

I suspect that the environment was originally written in gym v0.21, I would use the gym compatibility wrapper in gymnasium https://gymnasium.farama.org/content/gym_compatibility/
Otherwise, I would read the migration guide https://gymnasium.farama.org/content/migration-guide/


EdgeEdgeIntersection Bug

Hello ! :)
I discovered an edge case bug inside the EdgeEdgeIntersection function. In the case where two edges are not intersecting but are both on the same line, it cause the EdgeEdgeIntersection function to return true.
What happens ?
The cross product should be returning zero, but instead returns 1E-13 (for example) due to floating point error (also happening for double) when using utils.neq and utils.sign (might also be an issue for other functions).
Here is the data i had:
double2 A0 = new double2(-703d, 290.42d);
double2 A1 = new double2(-701.15d, 288.57d);
double2 B0 = new double2(-193.43d, -219.15d);
double2 B1 = new double2(-186.58d, -226d);

With A0 and A1 corresponding to an edge, and B0 and B1 to another edge.
I tried some stuff to fix this like:
public readonly bool neq(double v, double w) => math.abs(v - w) > 1E-6d;
public readonly double sign(double a) => math.sign(neq(a, 0) ? a : 0);

The issue was gone but some other deeper issues appeared, so i'm sure there is a better solution.
Thank you :)

Hi @DiiiiiZ,
Thank you for reporting this issue! I'm surprised this bug came out just now! :)
The edge vs. edge intersection check is very fast but does not account for collinear edges. I believe this is not a problem for most uses of this utility in the package. However, your reported bug proves that this routine fails for extreme cases of non-intersecting collinear lines.
I first drew your input to show the extremity of your case. Here are the lines at true scale:

To solve the issue, I changed the CCW definition in my tests:
+ int CCW(double2 a, double2 b, double2 c) => (c.y - a.y) * (b.x - a.x) > (b.y - a.y) * (c.x - a.x) ? 1 : 0;
- int CCW(double2 a, double2 b, double2 c) => math.sign(Cross(b - a, b - c));

This elliminates the problem with float/double precision. I wrote a draft test to verify if this resolves the issue:
private static bool EdgeEdgeIntersection(double2 a0, double2 a1, double2 b0, double2 b1) =>
    CCW(a0, a1, b0) != CCW(a0, a1, b1) && CCW(b0, b1, a0) != CCW(b0, b1, a1);
private static int CCW(double2 a, double2 b, double2 c) => (c.y - a.y) * (b.x - a.x) > (b.y - a.y) * (c.x - a.x) ? 1 : 0;

private static readonly TestCaseData[] edegeVsEdgeTestData = new TestCaseData[]
{
    new(
        math.double2(0, 0),
        math.double2(1, 0),
        math.double2(2, 0),
        math.double2(2, 1)
    ) { TestName = "Case 1 (ortogonal without intersection)", ExpectedResult = false},
    new(
        math.double2(-1, 0),
        math.double2(1, 0),
        math.double2(0, -1),
        math.double2(0, 1)
    ) { TestName = "Case 2 (ortogonal with intersection)", ExpectedResult = true},
    new(
        math.double2(0, 0),
        math.double2(1, 0),
        math.double2(2, 0),
        math.double2(3, 0)
    ) { TestName = "Case 3 (collinear without intersection)", ExpectedResult = false},
    new(
        math.double2(0, 0),
        math.double2(2, 0),
        math.double2(1, 0),
        math.double2(3, 0)
    ) { TestName = "Case 4 (collinear with intersection)", ExpectedResult = true},
    new(
        math.double2(-703d, 290.42d),
        math.double2(-701.15d, 288.57d),
        math.double2(-193.43d, -219.15d),
        math.double2(-186.58d, -226d)
    ) { TestName = "Case 5 (gh issue #173)", ExpectedResult = false},
}.SelectMany(i =>
{
    var args = i.Arguments;
    return new[]{
        new TestCaseData(args[0], args[1], args[2], args[3]) { TestName = i.TestName + ": abcd", ExpectedResult = i.ExpectedResult},
        new TestCaseData(args[0], args[1], args[3], args[2]) { TestName = i.TestName + ": abdc", ExpectedResult = i.ExpectedResult},
        new TestCaseData(args[1], args[0], args[2], args[3]) { TestName = i.TestName + ": bacd", ExpectedResult = i.ExpectedResult},
        new TestCaseData(args[1], args[0], args[3], args[2]) { TestName = i.TestName + ": badc", ExpectedResult = i.ExpectedResult},
    };
}
).ToArray();

[Test, TestCaseSource(nameof(edegeVsEdgeTestData))]
public bool EdegeVsEdgeTest(double2 a0, double2 a1, double2 b0, double2 b1) => EdgeEdgeIntersection(a0, a1, b0, b1);

In the previous implementation, your test data did not pass, but now it's green :)

Intersecting collinear test cases still fail, but this should not be a problem, since we check for collinearity at input validation step. I know how to fix this as well, but due to performance reasons, I will first investigate if this is even necessary.

I am currently preparing the new release v3.1 and your issue will definitely be included (with tests)! I just need a little time to prepare this properly (with some CCW math cleanup), but I think the release will be scheduled for next week.
Please let me know if this resolves the issue.
Best,
Andrzej

PS I did not check if this CCW implementation breaks something else, so during test please add additional CCW without replacing the previous one.

Hey Andrzej ! Thank you for your quick answer :)
First i tried to replace the CCW implementation with your new one, and it did break at other places.
In all places we check CCW against utils.Zero(), like in "PointLineSegmentIntersection" or "IsConvexQuadrilateral", it breaks with the new implementation.
So i did as you suggested and used the new "EdgeEdgeIntersectionCCW" implementation only for the "EdgeEdgeIntersection" function, and it worked !
In conclusion, there is definitely something to fix here, but its in the right path :)
That said, I think it could be a lot easier by avoiding float point errors by using int. I believe i saw that it was work in progress on your side ? If so, do you have any updates about it ?
(Also, it might be really nice to directly log positions and edges positions in addition to indexes inside the EdgeValidations functions to facilitate debugging)
Thank you !

So i did as you suggested and used the new "EdgeEdgeIntersectionCCW" implementation only for the "EdgeEdgeIntersection" function, and it worked !

Great, I'm glad it helped!

That said, I think it could be a lot easier by avoiding float point errors by using int. I believe i saw that it was work in progress on your side ? If so, do you have any updates about it ?

Yes, I see that the CCW part can be simplified and refactored a bit. This is probably a good time to add tests for internal utilities. The current development status is as follows:

I want to release a new version next week (v3.1). This will include:

[X] Managed input (merged)
[X] Unsafe triangulator triangulate api (merged)
[X] Constrained halfedges in output data (merged)
[ ] unsafe triangulator refine and plant seeds api (draft ready)
[ ] Fix edge-edge issue + CCW refactor (draft ready)

In the next release (v3.2), I plan to add very exciting features, I hope to release this next month (most of the features are already draft, last step is to implement the tests)!

[ ] Support for int2 (draft ready)
[ ] Support for fp64 (float fixed point in Q31.32 format), which should be interesting for you if you are looking for an integer like implementation. (draft ready)
[ ] Suport for Vector2 (via reinterpreted, draft ready)
[ ] Dynamic point insertion/deletion via Unsafe triangulator api (initially, support will be added just for bulk operations in the mesh, however, this feature might be published in v3.3 as well)

(Also, it might be really nice to directly log positions and edges positions in addition to indexes inside the EdgeValidations functions to facilitate debugging)

Good idea! I've just reported this on the project board: https://github.com/users/andywiecko/projects/1/views/1?pane=issue&itemId=72267533

I will let you know when the fix for your issue is released.
If you have any additional questions or suggestions, let me know!
Best,
Andrzej

Alright, that's good news !
Yes let me know when thoses fixes are released :)
Thank you for your hard work and your availability !

Hi @DiiiiiZ
I have great news!
The new release, v3.1, is already published. You can find the full release notes here). It is already available on OpenUPM.
This should resolve your issue. Thank you for your contribution to the project!
If you have any additional questions or suggestions, let me know.
Best,
Andrzej


[Driver] TSan runtime now has a link-time dependency on libdispatch

Previously, the TSan runtime only required libdispatch on Darwin, which
required no explicit linker flags, because libdispatch is always
provided by the system (via libSystem).
Now, the TSan runtime also has a link-time dependency on libdispatch on
Linux, where libdispatch (and the blocks runtime) is just another
library. We therefore need to specify them as additional linker options.

@swift-ci please test

Build failed
Swift Test Linux Platform
Git Sha - 1cac0bedf125b53f4337d5ff463fde4569d55759

@swift-ci please test Linux


feat(backend): added hono-rate-limiter

This PR introduces the following changes -

[x] Added hono-rate-limiter
[x] Removed rate-limiter-flexible
[x] Update the code to use hono-rate-limiter

Hey @flipvh, how can I run the test? I need to check the implementation.

@MathurAditya724 Sorry I missed the notifcation for this one (Github notifications are pretty crappy or not? :D)
This is great news. We do not have any tests actually, if that is what you mean. Let us review the changes now and merge it into development if its good!

Hey 👋🏼 , I just wanted to confirm that we will now merge this branch, https://github.com/cellajs/cella/tree/hono-rate-limiter, instead of this PR.


CI hidden error in build

There seems a hidden error in the CI server build job.
Summary
The following error occurs in build for pull-requests (and seems to hidden).
...
[1BDigest: sha256:3c2b0ab0c77ae10ff8fb77c58a803f601c255c8045c18094137bba4af98c730c
Status: Downloaded newer image for cucumber/cucumber-build:39e69ae56ad2861d86a8f26d3cec0285
source scripts/functions.sh && rsync_files
XXX-ERROR: /bin/sh: scripts/functions.sh: line 13: syntax error: bad substitution
XXX-ERROR: fatal: Invalid path '/**': No such file or directory
touch .rsynced
cd event-protocol && make default
make[1]: Entering directory '/app/event-protocol'
yarn install --network-concurrency 1
...

Expected Behavior
The lines marked with prefix "XXX-ERROR: " should not fail.
Possible Solution
The root_dir line in "scripts/function.sh" seems to be causing the problem.
# -- FILE: scripts/function.sh
...
# The root dir of the monorepo
root_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && cd .. && pwd )"
...

Steps to Reproduce (for bugs)
See pull-request #342 (continuous-integration/travis-ci/pr)

Still needs to be fixed


A problem reading the DIELECTRIC TENSOR

Hi
I came a cross a problem with vasp and the dielectric tensor, sometimes when the valuesof the tensor are very big the numbers in the OUTCAR are mixedup and so the python script doesn't read them correctly. For example see the following sinnpet
MACROSCOPIC STATIC DIELECTRIC TENSOR (including local field effects in DFT)
  ------------------------------------------------------
       13738.431079-16959.202530     2.426989
        -7663.060793 23943.531628    -3.049640
            2.471794    -5.009690  1175.736030
  ------------------------------------------------------

In vasprun.xml the data is printed correctly, so I suggest to try and read the OUTCAR, and if it doesn't succeed then read the data from the xml with the following commands:
from lxml import etree
doc = etree.parse('vasprun.xml')
values = [c.text for c in doc.xpath("/modeling/calculation/varray")[3].getchildren()]

Or even try and use this code:
        if "MACROSCOPIC STATIC DIELECTRIC TENSOR" in line:
            outcar_fh.readline()
            try:
                epsilon.append([float(x) for x in outcar_fh.readline().split()])
                epsilon.append([float(x) for x in outcar_fh.readline().split()])
                epsilon.append([float(x) for x in outcar_fh.readline().split()])
            except:
                doc = etree.parse('vasprun.xml')
                epsilon = [[float(x) for x in c.text.split()] for c in doc.xpath("/modeling/calculation/varray")[3].getchildren()]
            return epsilon


DB fields suggestion

Hi, thanks very much for your project.
Here I would like to add two small suggestions about the sql:

I see that some fields are using mysql keywords/reserved words, such as status, owner, name, context.
And also DECIMAL that does not lose precision is preferred over FLOAT, DOUBLE  precision-loss types, current usage of these fields seem all fine though.
id fields maybe can consider bigint unsigned

And also DECIMAL that does not lose precision is preferred over FLOAT, DOUBLE precision-loss types, current usage of these fields seem all fine though.
id fields maybe can consider bigint unsigned

PR is welcom :)


Show dependencies on package page

It would be very helpful if the main page (eg http://package.elm-lang.org/packages/agrafix/elm-bootforms/4.0.0/ ) of each package would show (and link) its dependencies.

Agreed. The information is there, though. It is as easy as adding elm-package.json to the link. For example, http://package.elm-lang.org/packages/agrafix/elm-bootforms/4.0.0/elm-package.json

With the current updates to 0.17 going on, it would be especially useful to see the supported elm Version (probably with color coding) right on top of the package.

I agree. Especially, evancz/elm-html is a trap for beginner (I also catch this page by googling "elm html").

For now, you can use the unofficial elm-package-info browser extension which aims at testing some improvements (including the package dependencies) without requiring changes to the server.
Firefox extension Chrome extension

Each package now has an "About" page that includes the package dependencies.

Thank you to @rlefevre for thinking of putting this info on the About page, so it is available without making things to crowded otherwise!


Implement configuration specific imports in the analyzer

This is the analyzer specific issue for #24581. That issue has the details.

DEP committee notes on configuration specific imports

Available now through the options file.
See e17f2b98cf7fb4f9679ed0a2e65c288b9344e8f6


FileNotFoundError: [WinError 2] The system cannot find the file specified

I keep getting this error and cannot figure out how to resolve it.
View In Browser plugin v2.0.0, Python 3
C:\Program Files (x86)\Mozilla Firefox\firefox.exe -new-tab file:///c:/users/r_bal/appdata/local/temp/tmp2x386a.htm
Traceback (most recent call last):
File "./python3.3/subprocess.py", line 1104, in _execute_child
FileNotFoundError: [WinError 2] The system cannot find the file specified
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "F:\Code Editors\Sublime Text 3\sublime_plugin.py", line 818, in run_
return self.run(edit)
File "ViewInBrowserCommand in C:\Users\R_Bal\AppData\Roaming\Sublime Text 3\Installed Packages\View In Browser.sublime-package", line 238, in run
File "ViewInBrowserCommand in C:\Users\R_Bal\AppData\Roaming\Sublime Text 3\Installed Packages\View In Browser.sublime-package", line 191, in openBrowser
File "./python3.3/subprocess.py", line 819, in init
File "./python3.3/subprocess.py", line 1110, in _execute_child
FileNotFoundError: [WinError 2] The system cannot find the file specified
If it helps, I have followed the installation guide on your wiki page and also searched google for resolutions. It appears that it is not using my user settings and only the default settings. which as a note the only line I changed was for the firefox install directory on line 25 of the default sheet to this:
"firefox": "C:\Program Files\Mozilla Firefox\firefox.exe -new-tab",
I took out the (x86) by program files to find my default install directory - yet it seems to revert back to the default and not attempt to use the "user" defined settings.

Given the nt/win32 key it appears the plugin makes no attempt to understand where 64-bit browser installations are on a Windows system. The workaround is to update the win32 path for your browser to actually point to the correct directory
Preferences -> Package Settings -> View In Browser -> Settings - User
{
	"nt": {
		"win32": {
			"firefox": "C:\\Program Files\\Mozilla Firefox\\firefox.exe -new-tab"
		}
	},
	"browser": "firefox"
}

Update as appropriate by removing the  (x86) from the path of the browser of your choice. This plugin appears abandoned so I wouldn't expect a fix from the author anytime soon.


Exercise 4.8

https://github.com/Mooophy/Cpp-Primer/tree/master/ch04#exercise-48
You wrote:
logical AND : true only if both its operands evaluated to true; logical OR : true if either of its operands evaluated to true; equality operators : true only if its operands are equal.
nope, the question is about rules of operands evaluation. I suggest to rewrite this way:
logical OR : evaluate the second operand only if the first operand is false;
equality operators : cause of compiler optmization there aren't rule about which operand is evaluated first, it can be the first operand then the second operand or viceversa.```
Best regards

@frank67  Agree with you.


ignore istio.test.revision set to default

In response to a cherrypick label: #35215 failed to apply on top of branch "release-1.10":
Applying: ignore istio.test.revision set to default
Using index info to reconstruct a base tree...
M	pkg/test/framework/resource/version.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/test/framework/resource/version.go
CONFLICT (content): Merge conflict in pkg/test/framework/resource/version.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 ignore istio.test.revision set to default
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to a cherrypick label: new issue created for failed cherrypick: #35221

In response to a cherrypick label: new pull request created: #35222

In response to a cherrypick label: #35215 failed to apply on top of branch "release-1.10":
Applying: ignore istio.test.revision set to default
Using index info to reconstruct a base tree...
M	pkg/test/framework/resource/version.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/test/framework/resource/version.go
CONFLICT (content): Merge conflict in pkg/test/framework/resource/version.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 ignore istio.test.revision set to default
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to a cherrypick label: new issue created for failed cherrypick: #35223

In response to a cherrypick label: new pull request could not be created: failed to create pull request against istio/istio#release-1.11 from head istio-testing:cherry-pick-35215-to-release-1.11: status code 422 not one of [201], body: {"message":"Validation Failed","errors":[{"resource":"PullRequest","code":"custom","message":"A pull request already exists for istio-testing:cherry-pick-35215-to-release-1.11."}],"documentation_url":"https://docs.github.com/rest/reference/pulls#create-a-pull-request"}


[BUG] When clicking on close button, should close Alby instead of go to previous window

Is there an existing issue for this?

[X] I have searched the existing issues

Describe the bug
Hi,
When clicking on close button, should close Alby instead of go to previous window.
https://github.com/getAlby/lightning-browser-extension/assets/1178856/7071343d-bfe6-4dc9-a115-e89ea1988495
Screenshots [optional]
No response
Steps To Reproduce
1- Visit https://geyser.fund/
2- Send Sats
3- Click on Close button
Expected behavior
When clicking on close button, should close Alby instead of go to previous window
Alby information
Alby version: 3.0
Device information
No response
Additional context
No response
Are you working on this?
No

When popup is triggered by webln it autocloses as it should be, but If users triggers manually the popup I don't think it should autoclose the popup. To do this, you can click Alby icon.


⚠ ClamAV has degraded performance

In eb861fc, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 491 ms

Resolved: ClamAV performance has improved in 4d65b1d.


Optimize Redix.Connection.parse_client_reply/3

Fixes #177

Great catch @zoldar! I completely missed this. Thanks for the great find and quick fix 👍


Added support for gradio

This PR adds support for Gradio. #362
Reference: https://gradio.app/plot-component-for-maps/

A plotly map example:
import gradio as gr
import leafmap.plotlymap as leafmap

def create_map(url):
    m = leafmap.Map()
    m.add_cog_layer(url)
    return m

url = 'https://opendata.digitalglobe.com/events/california-fire-2020/pre-event/2018-02-16/pine-gulch-fire20/1030010076004E00.tif'

demo = gr.Interface(
    fn=create_map,
    inputs=gr.Text(value=url, label='Enter a COG URL'),
    outputs=gr.Plot().style(),
)
demo.launch()

https://user-images.githubusercontent.com/5016453/218625821-a4ef18b1-3a47-476f-b95d-d566e2492044.mp4

Is it possible to use it to provide input instead of using it to visualize output? It would be great if a user for example could select a point in the map, see https://github.com/gradio-app/gradio/discussions/3492

@ysig I am looking for the same functionality: to use the coordinates from a map as input. In our case, for a classification model that uses a geo-prior. Your link to the discussions is no longer available. Was there any helpful guidance there? There is a similar, unanswered question in the huggingface forums: https://discuss.huggingface.co/t/creating-a-leaflet-map-as-input/38315

@mihow I managed to do it using streamlit with folium - not sure gradio supports input functionality yet and it's not a problem of leafmap. I suggest that you open an issue at their github so that the demand is more expressed.


PopupMenu loops over items when the ui_down/ui_up button is pressed

Tested versions
4.2.2
System information
win 10
Issue description
after being visible in multi window mode (including editor), pressing the ui_down/ui_up loops over many items, because internal process is active when it becomes visible in multi window mode.
Steps to reproduce
Press any main menu button in the editor and then press the ui_down/ui_up button.
Minimal reproduction project (MRP)
N/A

Why is it wrong though?

because the internal process is used to simulate gamepad echo event when we hold up/down buttons, but it can't check if it's a gamepad or keyboard keys, so it simulates the echo event rapidly when a key is just pressed at first time, resulting in looping over 3 or 4 menu items so fast when we just press the down or up down keys. you can just reveal any editor top menu with mouse and you press up or down keys once, you will notice it looping so fast over so many items.


typescript word hightLight issue

VSCode Version:1.71
OS Version: mac os

Word hight light not work  for these code which below Map<any>.
If Map<(any)> or Map<T,P> ,it works.

Try installing this extension:
https://marketplace.visualstudio.com/items?itemName=ms-vscode.typescript-javascript-grammar
It is a preview of what is to come in TypeScript syntax highlighting

Thank you ,but it's not work for me

It's likely caused by #14921 as well.

@aeschli can you look into this? If there is something I can do to help please let me know.

@TheColorRed The https://marketplace.visualstudio.com/items?itemName=ms-vscode.typescript-javascript-grammar is no longer needed as we have now adopted the latest version of TS in VSCode 1.7.1.

Reproduced with this minimal sample in a js or tsx file:
this._blockRenderMap = Map<any>({});
this._eventsMap = {};

This issue was moved to Microsoft/TypeScript-TmLanguage#351


Add termination and parachute examples

This adds two examples:

Termination: sends termination to autopilot (use with caution!)
Parachute component: implements a separate mavlink parachute component (depends on https://github.com/PX4/PX4-Autopilot/pull/17564)

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

I'm just gonna assume that's unrelated :joy:.


toRGBBytes returns 4 numbers instead of 3

This function is called RGB not RGBA bytes. We used to have distinction https://github.com/pex-gl/pex-color/issues/8. It currently breaks code like this by returning alpha as 255.
ctx2d.fillStyle = `rgba(${toRGBBytes(bgColor).join(",")}, 1)`;

I think we should add toRGBABytes for clarity?

Assuming bgColor is vec4 in your case.
All toColorSpace methods work by transforming the input array.
Here you should really use ctx2d.fillStyle = toCSSRGB(bgColor) (or bgColor.slice(0, 3) but what you want is a string for fillStyle).

But the i propose to remove it and rename to RGBABytes or toBytes. pex-color colors should always be RGBA.
Another inconsistency is toHex([1,0,0,1]) returns #FF0000 not #FF0000FF.

Then i propose to remove it and rename to RGBABytes or toBytes. pex-color colors should always be RGBA.
Another inconsistency is toHex([1,0,0,1]) returns #FF0000 not #FF0000FF.

hex space is special yes. Last time we discussed alpha for it, we settled on the fact it shouldn't be the default to return alpha but should still be possible. So currently you have to pass the second parameter toHex([1,0,0,1], true) to get all an #RRGGBBAA string. (and another improvement we have in v2 is support for RGB[A] or RRGGBB[AA] input for fromHex keeping the current alpha).
toBytes makes sense to me.

toBytes makes sense to me.

Could we remove toRGBBytes from api/docs and throw or log deprecated without doing major version bump?

toBytes makes sense to me.

Could we remove toRGBBytes from api/docs and throw or log deprecated without doing major version bump?

yes

console.error then


package url

https://github.com/shadowsocks/shadowsocks-go/blob/master/shadowsocks/encrypt.go#L16
golang.org  has been blocked by GFW, please change them to https://github.com/golang/crypto/

Using golang package from github repo will causing ploblems.
If you have issues compiling binaries, please use the precompiled release.


Time-varying seasonality in prophet

I was wondering: is there a way of taking into account a time-varying seasonality?
For instance, an increase in level is often accompanied by an increase in the magnitude of the seasonal oscillations.
The following example in R:
library(prophet)
library(forecast)
data(gas)
df <- data.frame(ds = as.Date.ts(gas), y = as.numeric(gas))
m <- prophet(df)
future <- make_future_dataframe(m, periods = 365)
forecast <- predict(m, future)
plot(m, forecast)

In this case, it looks like prophet overestimates the oscillations at the beginning and underestimates them at the end.
How should I deal with such a feature?

Good question!  I suggest log-transforming the data:
df <- data.frame(ds = as.Date.ts(gas), y = log(as.numeric(gas)))

By default, the seasonality is additive, while in this case it's likely to be multiplicative.  By taking the log you get a constant seasonality:

However as of right now there's now way for Prophet to learn time-varying seasonal parameters.  It's certainly an extension to the model we've thought about but adding the complexity is a tradeoff.

I notice this applies to holidays as well. I input a bunch of holidays into the model and saw that they were all assigned the same coefficient. Am I right in assuming you don't model the variation from year to year for a given holiday? Would it help to pass every holiday type as a separate holiday itself?


update?

Does currently not work anymore

@Cmachtel could you please provide more details?

Hy,
The API Webpage does not exist anymore or has changed place. Additionaly it seams not working on MM2 but this can also be done due to API changes. Would be happy to use it and maybe help in putting it again up to date

@Cmachtel looks like the API is not available. It returns 503 error. I could find any information about moving it somehere alse. Let's hope this is temporarry issue on their side.

No, it is like this since a few months :-(
The talk about a new interface over https://www.mobiliteit.lu/fr/services/mconcept/ but i could not get more information for now. maybe if you contact them and tell them what exactly is needed

I have done some search and the new api is explained here: https://data.public.lu/fr/datasets/api-mobiliteit-lu/


AttributeError: 'numpy.ndarray' object has no attribute 'unique'

Hi, while I'm using apply_and_summarize_labels,
it's causing an issue as below. Please help.
df_summary, labeled_docs = model.apply_and_summarize_labels(data_sample.sentence)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_21756/2555802596.py in <module>
----> 1 df_summary, labeled_docs = model.apply_and_summarize_labels(data_sample.sentence)

/opt/conda/lib/python3.9/site-packages/chatintents/ChatIntents.py in apply_and_summarize_labels(self, df_data)
    418         df_clustered[category_col] = self.best_clusters.labels_
    419
--> 420         numerical_labels = df_clustered[category_col].unique()
    421
    422         # create dictionary mapping the numerical category to the generated

AttributeError: 'numpy.ndarray' object has no attribute 'unique'

Hi,
It might be due to the datatype of the input you're using. I think what you are using (data_sample.sentence) is a pandas Series. Try using data_sample[['sentence']] instead, which is a pandas DataFrame.


Add Travis CI badge

#7

By the way, you can go to the code section on this repo and move branch, to check how it will look:

@nirgn975  Thanks for the comments, Markdown language is used in other places than readme.me in github?

@daniellevi1
Yes, in gaining popularity especially in website who're appeal to developers, geeks, etc (see Markdown).

@nirgn975 Usually I leave 2 white spaces between 'paragraphs', it is common syntax to leave 1 in .md?

@daniellevi1 Yes, but not just in md files, in every file. Leave one space in the end of each and every file.


Can use the proposal Id 0 and update the totalProposals after assigning the catalog proposals

Here https://github.com/dapperlabs/nft-catalog/blob/8cb26110e41190316e718bb0797da338f0713662/cadence/contracts/NFTCatalog.cdc#L194
We are not storing anything for proposal id 0

Any reason we should start at 0 instead of 1?
IMO it looks nicer and is more readable to start from 1. Also, it might be too late to make this change now that there have been some proposals made in mainnet


On iOS, what's the difference between Virtualize and Emulate, for running arm64 VMs? And why does it say "virtualization is not supported on your system" on my iPad? (it's not jailbroken btw)

More specifically, what are differences in the ways VMs run between option?

On iOS, Apple does not allow UTM to use virtualization. It is not known for sure whether iDevice CPUs (including iPad “M1”) have the necessary hardware to theoretically support it, either. So we need to use emulation instead, and you get the “not supported” message when attempting to use virtualization.
Check out this technical overview of UTM software architecture for related information.


Not Able to Get Processes in Node Dashboard

Hi We are Using pm2 and Pm2-gui.
We have started pm2-gui in node and from Browser we are able to Login and See the CPU and Memory usage. We are not able to see the Processes.
We have put conf.js  which was generated by command pm2 ecosystem
Below is the Execution status we get

node pm2-gui.js start ./pm2-gui.ini
[WARN] 2018-2-27 18:54:39 Can not load PM2 config, the file "/TMS/.pm2/conf.js" does not exist or empty, fallback to auto-load by pm2 home.
[INFO] 2018-2-27 18:54:39 Listening connection event on sys by func:_connectSysSocket
[INFO] 2018-2-27 18:54:39 Listening connection event on log by func:_connectLogSocket
[INFO] 2018-2-27 18:54:39 Listening connection event on process by func:_connectProcessSocket
[INFO] 2018-2-27 18:54:39 Connecting to pm2 daemon: /TMS/.pm2/pub.sock
[INFO] 2018-2-27 18:54:39 [pm2:log] Activate
[WARN] 2018-2-27 18:54:39 Avoid duplicated listening!
[INFO] 2018-2-27 18:54:39 Web server is listening on <IP_ADDRESS>:12345
[INFO] 2018-2-27 18:54:43 Connected to /system!
[INFO] 2018-2-27 18:54:43 Fetching pm2 version: /TMS/.pm2/rpc.sock
[INFO] 2018-2-27 18:54:43 awake:1st Tracking-11
[INFO] 2018-2-27 18:54:43 [pm2:log] deactivate 'cause no subscriber

please paste your ./pm2-gui.ini...

;
; Home directory of pm2.
;
pm2 = ~/.pm2
;
; The monitor will fetch system CPU/Memory amount after this certain time interval.
; it could be milliseconds or various time formats(https://github.com/zeit/ms)
;
refresh = 5s
;
; Port of Web server and socket agent.
;
port = 8088
;
; A value indicates whether or not run the pm2-gui as damonization.
;
daemonize = false
;
; A value indicates whether or not the action buttons (i.e. restart, stop all...) should be displayed on web page.
;
readonly = false
;
; The monitor will fetch process CPU/Memory amount after this certain time interval.
; it could be milliseconds or various time formats(https://github.com/zeit/ms)
;
process_refresh = 3s
[log]
;
; Log directory.
;
dir = ./logs
;
; A value indicates whether or not display the [INFO], [ERROR].. prefixes before log message.
;
prefix = true
;
; A value indicates whether or not display the local date string before log message.
;
date = true
;
; Log level, one of debug, log, info, warn, error.
;
level = info
;
; Socket.io origins check, e.g.:
;   origins = 'example.com:* http://example.com:* http://www.example.com:8088'
; By default:
;   origins = :
[agent]
;
; This authorization will be used to authorize socket / web connections if it's set.
;
;authorization = AuTh
;
; A value indicates whether agent offline or not.
;
; offline = true
[remotes]
;
; the dashboard and web server will use this section to connect remoting socket server
;   server_name = [authorization@]host:port
;
; pm2@171 = AuTh@https://<IP_ADDRESS>:9002/sockserv
; pm2@172 = <IP_ADDRESS>:9001
; pm2@173 = <IP_ADDRESS>:9000
;

just changing daemonize = true, and you can access the processes by pm2-gui mon after pm2-gui start /your/path/of/pm2-gui.ini

Dear Tjatse
Thanks for the Quick revert. We have done the changes but the results remain the same.
below is the Console when we run on node....

$ node pm2-gui.js start /var/opt/node-apps/pm2-gui/pm2-gui.ini
[WARN] 2018-3-1 08:43:55   CPCPCP /TMS/.pm2/conf.jsCan not load PM2 config, the file "/TMS/.pm2/conf.js" does not exist or empty, fallback to auto-load by pm2 home.
[INFO] 2018-3-1 08:43:55 Listening connection event on sys by func:_connectSysSocket
[INFO] 2018-3-1 08:43:55 Listening connection event on log by func:_connectLogSocket
[INFO] 2018-3-1 08:43:55 Listening connection event on process by func:_connectProcessSocket
[INFO] 2018-3-1 08:43:55 Connecting to pm2 daemon: /TMS/.pm2/pub.sock
[INFO] 2018-3-1 08:43:55 [pm2:log] Activate
[WARN] 2018-3-1 08:43:55 Avoid duplicated listening!
[INFO] 2018-3-1 08:43:55 Web server is listening on <IP_ADDRESS>:8088
[INFO] 2018-3-1 08:43:57 awake:1st Deviceview-3
[INFO] 2018-3-1 08:43:57 [pm2:log] deactivate 'cause no subscriber
[INFO] 2018-3-1 08:44:18 Connected to /system!
[INFO] 2018-3-1 08:44:18 Fetching pm2 version: /TMS/.pm2/rpc.sock
[INFO] 2018-3-1 08:44:37 Connected to /system!
[INFO] 2018-3-1 08:44:37 Fetching pm2 version: /TMS/.pm2/rpc.sock
[INFO] 2018-3-1 08:44:42 Connected to /system!
[INFO] 2018-3-1 08:44:42 Fetching pm2 version: /TMS/.pm2/rpc.sock
[INFO] 2018-3-1 08:44:45 Connected to /system!
[INFO] 2018-3-1 08:44:45 Fetching pm2 version: /TMS/.pm2/rpc.sock

can you open the dev-tools of your browser and inspect the console logs:

Dear
The console log is clear and has no errors
[image: Inline image 1]
Best Regards
CP SinghFounder & CEO
[image: Mobile]    : +91<PHONE_NUMBER>
[image: Mail]<EMAIL_ADDRESS>[image: Phone]  : +91<PHONE_NUMBER>
[image: Globe]   www.smart.org.in/
Smart  Address - Smile Campus,
Plot 241, Udyog Vihar, Phase 1,
Gurgaon, Haryana-122015
[image: Inline image 1]
"Save Trees,please consider the environment before printing this e-mail"
On Thu, Mar 1, 2018 at 9:17 AM, Jun<EMAIL_ADDRESS>wrote:

can you open the dev-tools of your browser and inspect the console logs:
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Tjatse/pm2-gui/issues/67#issuecomment-369465597, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AH3a8yUnD0YYywBWmjsJyWowA4Epbo4uks5tZ29vgaJpZM4SU7-7
.

so, you're visiting the dashboard (web ver.) on your linux server? (and whats the address?)

Dear Jun
Yes we are trying to  monitor the Pm2 via pm2-gui on Web Dashboard.
The URL cannot be shared as it is hosted behind Firewall.
We have Ubuntu Server 16 with PM2 and Niginx.
We are accessing the Webbrowser on Windows Machine put next to pm2 server. So they can access from local ip address.
L

gotcha!
whats your pm2-gui version?
Please make sure your web/socket address && port of pm2-gui could be resolved on your PC
it seems a network issue, i guess :)

From Package.json
"name": "pm2-gui",
"version": "0.1.4",
"lockfileVersion": 1,
"requires": true,
"dependencies":
Will check the web/socket Address
Can you let us know what this setting relate to
;
; Socket.io origins check, e.g.:
;   origins = 'example.com:* http://example.com:* http://www.example.com:8088'
; By default:
;   origins = :
We have all the lines commented in our ini.

Dear Jun
Can you help on how to troubleshoot
Please make sure your web/socket address && port of pm2-gui could be resolved on your PC
And What settings to be changed in which file.

Are'y visiting dashboard via ip address? or domain?

We get Blank Screen when we click om processes.

All this leads up to the privilege issues, you're running pm2 with root, but pm2-gui by TMZ, actually TMZ can not access the processes of root

Dear Jun
root      1442  1.8  0.2<PHONE_NUMBER>0 ?        Ssl  18:35   0:14 PM2 v2.8.0: God Daemon (/TMS/.pm2)
root      1559  0.3  0.3 1210376 50684 ?       Ssl  18:35   0:03 PM2: KM Agent (/TMS/.pm2)
root      2237  0.1  0.3 1135200 51032 pts/0   Sl+  18:37   0:00 pm2-gui slave
Both are running as root user.
It looks like it has something to do with Socket.io now. I have Changed Settings and now started getting error on console and web dashboard

the nginx configuration is the main issue now, congrats.
Are'y serving pm2-gui with a sub domain (e.g: http://my.site.com/pm2-gui/)?

Yes , We are serving pm2-gui with http://my.site.com/pm2-gui/

this is a nginx proxy (sub domain) issue, it's always very easy to get stuck.
I'll test this on my centOS ASAP.

Dear Jun
Were you able to implement the solution?

I've just tried sub domain and it's working as scheduled:
#
# The default server
#
server {
    listen       80;
    server_name  pm2.mydomain.com;

    charset utf-8;

    location / {
        root   /var/www/node-sites;
        proxy_redirect off ;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header REMOTE-HOST $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        client_max_body_size 50m;
        client_body_buffer_size 256k;
        proxy_connect_timeout 30;
        proxy_send_timeout 30;
        proxy_read_timeout 60;
        proxy_buffer_size 256k;
        proxy_buffers 4 256k;
        proxy_busy_buffers_size 256k;
        proxy_temp_file_write_size 256k;
        proxy_next_upstream error timeout invalid_header http_500 http_503 http_404;
        proxy_max_temp_file_size 128m;
        proxy_pass http://<IP_ADDRESS>:8088;
    }
    location /socket.io/ {
      proxy_pass http://<IP_ADDRESS>:8088;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      proxy_http_version 1.1;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $host;
    }

    error_page  404              /404.html;
    location = /404.html {
        root   /usr/share/nginx/html;
    }

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    # proxy the PHP scripts to Apache listening on <IP_ADDRESS>:80
    #
    #location ~ \.php$ {
    #    proxy_pass   http://<IP_ADDRESS>;
    #}

    # pass the PHP scripts to FastCGI server listening on <IP_ADDRESS>:9000
    #
    #location ~ \.php$ {
    #    root           html;
    #    fastcgi_pass   <IP_ADDRESS>:9000;
    #    fastcgi_index  index.php;
    #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
    #    include        fastcgi_params;
    #}

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    #location ~ /\.ht {
    #    deny  all;
    #}
}

I'll test the sub folder such as http://mydomain.com/pm2-gui

Hi Jun, i got the same problem, is there any solution at this moment? Regards.

Getting same issue with pm2 version 4.4.0

Ref #84


can't retrieve non-text files using preston get

I haven't tested it, but I believe the recent changes to preston get (allowing it to accept content-based locations instead of just content hashes as input) mean that it can only retrieve text data, and will fail to retrieve things like archives, compressed files, media files, etc.. I'll verify and fix this when I get a chance. fyi @jhpoelen

I can confirm that:
$ java -jar preston.jar get 'hash://sha256/94a4fc4824d951c55860e3e5c4c662afcfae55d52a25105bca1cb6a7b3a062' | sha256sum
94a4fc4824d951c0155860e3e5c4c662afcfae55d52a25105bca1cb6a7b3a062  -

with preston v0.0.10 (the one that comes with https://github.com/bio-guoda/preston-amazon)
however, when running v0.2.3 :
 preston get 'hash://sha256/94a4fc4824d951c0155860e3e5c4c662afcfae55d52a25105bca1cb6a7b3a062' | sha256sum
java.io.IOException: problem retrieving [hash://sha256/94a4fc4824d951c0155860e3e5c4c662afcfae55d52a25105bca1cb6a7b3a062]
	at bio.guoda.preston.cmd.CmdGet.handleContentIri(CmdGet.java:73)
	at bio.guoda.preston.cmd.CmdGet.run(CmdGet.java:51)
	at bio.guoda.preston.cmd.CmdGet.run(CmdGet.java:38)
	at bio.guoda.preston.cmd.CmdLine.run(CmdLine.java:18)
	at bio.guoda.preston.cmd.CmdLine.run(CmdLine.java:26)
	at bio.guoda.preston.Preston.main(Preston.java:19)
Caused by: java.io.IOException: failed to resolve to content
	at bio.guoda.preston.cmd.CmdGet$ContentQuery.dereference(CmdGet.java:111)
	at bio.guoda.preston.cmd.CmdGet.processQuery(CmdGet.java:78)
	at bio.guoda.preston.cmd.CmdGet.handleContentIri(CmdGet.java:70)
	... 5 more
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  -

with
$ preston version
0.2.3

However, when running:
$ preston get 'zip:hash://sha256/94a4fc4824d951c0155860e3e5c4c662afcfae55d52a25105bca1cb6a7b3a062!/eml.xml'
<?xml version='1.0' encoding='utf-8'?><eml:eml xmlns:eml="eml://ecoinformatics.org/eml-2.0.1" xmlns:md="eml://ecoinformatics.org/methods-2.0.1" xmlns:proj="eml://ecoinformatics.org/project-2.0.1" xmlns:d="eml://ecoinformatics.org/dataset-2.0.1" xmlns:res="eml://ecoinformatics.org/resource-2.0.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:dc="http://purl.org/dc/terms/" system="Plazi" scope="system" packageId="FFF76241FFB3B773A711FF98DF6BFFF3/eml-1519775222262.xml"><dataset><alternateIdentifier>FFF76241FFB3B773A711FF98DF6BFFF3</alternateIdentifier><alternateIdentifier>http://dx.doi.org/10.3897/zookeys.740.14640</alternateIdentifier><a

content is returned as expected. . .

It's fixed now:
$ preston get 'hash://sha256/94a4fc4824d951c0155860e3e5c4c662afcfae55d52a25105bca1cb6a7b3a062' | sha256sum
94a4fc4824d951c0155860e3e5c4c662afcfae55d52a25105bca1cb6a7b3a062  -

@mielliott Thanks for fixing and for adding the test cases!
The ContentQuery implementation of the CmdGet class looks pretty neat.
In a way it is a content resolver (dereferencer) that uses a content-based universe (content hash, blobstore) instead of the location-based universe (url and http client).
Did you consider to make the ContentQuery implement the Dereferencer interface?

Did you consider to make the ContentQuery implement the Dereferencer interface?

Ooh, good idea. The suggested change is implemented in bf3465cca131c5781dcce3cec2aaa880859a7941.
I've also put the "get bytes identified by a content query" logic into it's own ContentDerefencer class (see bf4d2aa6fa70b4c066079e387949cf6a483659d4). I feel like this could be useful for all sorts of applications.