Other pipx list output formats

How would this feature be useful?

Hopefully this will synthesize (and possibly supersede) Issues #390 and #109.  It addresses the reason for PRs #572 and #392.
This Issue is to track exactly what users are looking for with different pipx list output formats.
"Import / Export" set of pipx packages case
In #109, it appears that one primary motivator is the ability to export a file that specifies a set of pipx-installable packages that pipx can then install (possibly on another machine) in the future.
How many users desire this or would find it useful?
It would probably best be done by making a file in json format.
Within this there are some further points for those desiring this functionality:
Do people consider it desired (or necessary?) to freeze all versions of all dependencies for this to be useful?  If so then this might be very difficult, based on my investigation creating test code for this.
Simply re-creating a pipx environment, and pinning the version of the main package and any injected packages, and any pipx options is very doable.  I have a branch of doing this basically working.  In this case, the output would be equivalent to what you would end up with on one machine executing pipx reinstall-all.  You can be sure the main package and any injected packages will stay with their specified versions, but you cannot be sure that the versions of any dependencies that still satisfy all package requirements will be the same.
Archive / Documentation case
In #390 it appears that one motivation is simply a concise version of pipx list for "keeping track" or for archival or documentation purposes.
How many users desire this or would find it useful?
Is this type of list output (that is merely more concise) still desired if the "Import / Export" case is available?
What are the goals of this format?  What are the use cases specifically?  How would one know if the output of this was "successful"?
I must confess that I need this one explained to me, it doesn't naturally occur to me what is desired.  If all that's needed is documentation of what is currently installed, then the current pipx list seems to cover that.  If what is desired is a structured output that would allow one to recreate a pipx environment, then the "Import / Export" json scenario seems to satisfy the need.
@comkieffer @gvoysey @sanketdg @zachvalenta
Describe the solution you'd like

Describe alternatives you've considered

For #390, my idea is:
# freeze list of pkg
pipx list > $DOTFILES_DIR/pkg-pipx.txt

# parses list of pkg and installs each
cat pkg-pipx.txt | pipx_install.sh

Right now this second step (piping pkg list file to install script) is fine but would just be easier with more structured output format. If output format that could be directly be read by pipx itself to install, all the better.

For #390, my idea is:
# freeze list of pkg
pipx list > $DOTFILES_DIR/pkg-pipx.txt

# parses list of pkg and installs each
cat pkg-pipx.txt | pipx_install.sh

Right now this second step (piping pkg list file to install script) is fine but would just be easier with more structured output format. If output format that could be directly be read by pipx itself to install, all the better.

@zachvalenta do I understand you correctly that if you had a structured json output, you wouldn't need another kind of pipx list output format?  (Your personal opinion)

@zachvalenta do I understand you correctly that if you had a structured json output, you wouldn't need another kind of pipx list output format? (Your personal opinion)

Yes indeed. I'm looking for something more structured than the status quo but agnostic about format.

Thanks for pulling together a bunch of threads!

Simply re-creating a pipx environment, and pinning the version of the main package and any injected packages, and any pipx options is very doable.

This is all i want, anyway (and i filed #109 ).
An ancillary use case that I am now looking forward to is bumping python interpreter versions for my pipx-managed venvs on the same machine.  I have many 3.6.8 venvs, and wouldn't mind kicking them all up to 3.8.x or higher without manually redoing all of it.

@itsayellow It occurs to me that if you do not already know about it, https://pypi.org/project/pipdeptree/  might be extremely useful for inspecting pipx-created environments and generating a list of top-level packages that were injected into them.
For a busy example: i have a very full ipython environment managed by pipx right now.  If i pipx inject ipython pipdeptree and then manually run .local/pipx/venvs/ipython/bin/pipdeptree, i get:
cattrs==1.2.0
  - attrs [required: >=20.1.0, installed: 20.3.0]
dataset==1.4.5
  - alembic [required: >=0.6.2, installed: 1.5.5]
    - Mako [required: Any, installed: 1.1.4]
      - MarkupSafe [required: >=0.9.2, installed: 1.1.1]
    - python-dateutil [required: Any, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.15.0]
    - python-editor [required: >=0.3, installed: 1.0.4]
    - SQLAlchemy [required: >=1.3.0, installed: 1.3.23]
  - banal [required: >=1.0.1, installed: 1.0.6]
  - sqlalchemy [required: >=1.3.2, installed: 1.3.23]
desert==2020.11.18
  - attrs [required: Any, installed: 20.3.0]
  - marshmallow [required: >=3.0, installed: 3.10.0]
  - typing-inspect [required: Any, installed: 0.6.0]
    - mypy-extensions [required: >=0.3.0, installed: 0.4.3]
    - typing-extensions [required: >=3.7.4, installed: <IP_ADDRESS>]
glom==20.11.0
  - attrs [required: Any, installed: 20.3.0]
  - boltons [required: >=19.3.0, installed: 20.2.1]
  - face [required: >=20.1.0, installed: 20.1.1]
    - boltons [required: >=20.0.0, installed: 20.2.1]
httpx==0.16.1
  - certifi [required: Any, installed: 2020.12.5]
  - httpcore [required: ==0.12.*, installed: 0.12.3]
    - h11 [required: ==0.*, installed: 0.12.0]
    - sniffio [required: ==1.*, installed: 1.2.0]
  - rfc3986 [required: >=1.3,<2, installed: 1.4.0]
  - sniffio [required: Any, installed: 1.2.0]
hyperlink==21.0.0
  - idna [required: >=2.5, installed: 3.1]
imageio==2.9.0
  - numpy [required: Any, installed: 1.20.1]
  - pillow [required: Any, installed: 8.1.0]
ipython==7.20.0
  - backcall [required: Any, installed: 0.2.0]
  - decorator [required: Any, installed: 4.4.2]
  - jedi [required: >=0.16, installed: 0.18.0]
    - parso [required: >=0.8.0,<0.9.0, installed: 0.8.1]
  - pexpect [required: >4.3, installed: 4.8.0]
    - ptyprocess [required: >=0.5, installed: 0.7.0]
  - pickleshare [required: Any, installed: 0.7.5]
  - prompt-toolkit [required: >=2.0.0,<3.1.0,!=3.0.1,!=3.0.0, installed: 3.0.16]
    - wcwidth [required: Any, installed: 0.2.5]
  - pygments [required: Any, installed: 2.8.0]
  - setuptools [required: >=18.5, installed: 53.0.0]
  - traitlets [required: >=4.2, installed: 5.0.5]
    - ipython-genutils [required: Any, installed: 0.2.0]
lxml==4.6.2
Pint==0.16.1
  - packaging [required: Any, installed: 20.9]
    - pyparsing [required: >=2.0.2, installed: 2.4.7]
pipdeptree==2.0.0
  - pip [required: >=6.0.0, installed: 21.0.1]
plotnine==0.7.1
  - descartes [required: >=1.1.0, installed: 1.1.0]
    - matplotlib [required: Any, installed: 3.3.4]
      - cycler [required: >=0.10, installed: 0.10.0]
        - six [required: Any, installed: 1.15.0]
      - kiwisolver [required: >=1.0.1, installed: 1.3.1]
      - numpy [required: >=1.15, installed: 1.20.1]
      - pillow [required: >=6.2.0, installed: 8.1.0]
      - pyparsing [required: >=2.0.3,!=2.1.6,!=2.1.2,!=2.0.4, installed: 2.4.7]
      - python-dateutil [required: >=2.1, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.15.0]
  - matplotlib [required: >=3.1.1, installed: 3.3.4]
    - cycler [required: >=0.10, installed: 0.10.0]
      - six [required: Any, installed: 1.15.0]
    - kiwisolver [required: >=1.0.1, installed: 1.3.1]
    - numpy [required: >=1.15, installed: 1.20.1]
    - pillow [required: >=6.2.0, installed: 8.1.0]
    - pyparsing [required: >=2.0.3,!=2.1.6,!=2.1.2,!=2.0.4, installed: 2.4.7]
    - python-dateutil [required: >=2.1, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.15.0]
  - mizani [required: >=0.7.1, installed: 0.7.2]
    - matplotlib [required: >=3.1.1, installed: 3.3.4]
      - cycler [required: >=0.10, installed: 0.10.0]
        - six [required: Any, installed: 1.15.0]
      - kiwisolver [required: >=1.0.1, installed: 1.3.1]
      - numpy [required: >=1.15, installed: 1.20.1]
      - pillow [required: >=6.2.0, installed: 8.1.0]
      - pyparsing [required: >=2.0.3,!=2.1.6,!=2.1.2,!=2.0.4, installed: 2.4.7]
      - python-dateutil [required: >=2.1, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.15.0]
    - numpy [required: Any, installed: 1.20.1]
    - palettable [required: Any, installed: 3.3.0]
    - pandas [required: >=1.1.0, installed: 1.2.2]
      - numpy [required: >=1.16.5, installed: 1.20.1]
      - python-dateutil [required: >=2.7.3, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.15.0]
      - pytz [required: >=2017.3, installed: 2021.1]
  - numpy [required: >=1.16.0, installed: 1.20.1]
  - pandas [required: >=1.1.0, installed: 1.2.2]
    - numpy [required: >=1.16.5, installed: 1.20.1]
    - python-dateutil [required: >=2.7.3, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.15.0]
    - pytz [required: >=2017.3, installed: 2021.1]
  - patsy [required: >=0.5.1, installed: 0.5.1]
    - numpy [required: >=1.4, installed: 1.20.1]
    - six [required: Any, installed: 1.15.0]
  - scipy [required: >=1.2.0, installed: 1.6.1]
    - numpy [required: >=1.16.5, installed: 1.20.1]
  - statsmodels [required: >=0.11.1, installed: 0.12.2]
    - numpy [required: >=1.15, installed: 1.20.1]
    - pandas [required: >=0.21, installed: 1.2.2]
      - numpy [required: >=1.16.5, installed: 1.20.1]
      - python-dateutil [required: >=2.7.3, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.15.0]
      - pytz [required: >=2017.3, installed: 2021.1]
    - patsy [required: >=0.5, installed: 0.5.1]
      - numpy [required: >=1.4, installed: 1.20.1]
      - six [required: Any, installed: 1.15.0]
    - scipy [required: >=1.1, installed: 1.6.1]
      - numpy [required: >=1.16.5, installed: 1.20.1]
pugsql==0.2.3
  - sqlalchemy [required: >=1.3,<2.0, installed: 1.3.23]
pydantic==1.7.3
PyYAML==5.4.1
scikit-learn==0.24.1
  - joblib [required: >=0.11, installed: 1.0.1]
  - numpy [required: >=1.13.3, installed: 1.20.1]
  - scipy [required: >=0.19.1, installed: 1.6.1]
    - numpy [required: >=1.16.5, installed: 1.20.1]
  - threadpoolctl [required: >=2.0.0, installed: 2.1.0]
seaborn==0.11.1
  - matplotlib [required: >=2.2, installed: 3.3.4]
    - cycler [required: >=0.10, installed: 0.10.0]
      - six [required: Any, installed: 1.15.0]
    - kiwisolver [required: >=1.0.1, installed: 1.3.1]
    - numpy [required: >=1.15, installed: 1.20.1]
    - pillow [required: >=6.2.0, installed: 8.1.0]
    - pyparsing [required: >=2.0.3,!=2.1.6,!=2.1.2,!=2.0.4, installed: 2.4.7]
    - python-dateutil [required: >=2.1, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.15.0]
  - numpy [required: >=1.15, installed: 1.20.1]
  - pandas [required: >=0.23, installed: 1.2.2]
    - numpy [required: >=1.16.5, installed: 1.20.1]
    - python-dateutil [required: >=2.7.3, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.15.0]
    - pytz [required: >=2017.3, installed: 2021.1]
  - scipy [required: >=1.0, installed: 1.6.1]
    - numpy [required: >=1.16.5, installed: 1.20.1]
tabulate==0.8.9
typer==0.3.2
  - click [required: >=7.1.1,<7.2.0, installed: 7.1.2]
wheel==0.36.2
xlrd==2.0.1

and running pipdeptree -j spits out json that's more programmatically handy to attack with something like glom.

Thanks @gvoysey , I am aware of pipdeptree (and use it myself if I accidentally install something into my system python packages!)  Luckily in pipx we have a bit of an easier time of it because of our isolated venvs, and our internal metadata.  Our internal metadata keeps track of what we intentionally installed, and each venv can be pip listed to find any dependencies added.

@itsayellow oh.  well, that's just /cheating/! :grin:

The problem seems to be in trying to implement something to install from such a json list.  Doing a basic job is fairly straightforward, but covering all the edge cases gets a little hairy.
For example, how do you handle things in your json file that do not exist on the target install system?  Like paths to specific local packages, or paths to specific python interpreters.
This might just mean that implementing a json format for list output is doable, but installing from such an output might be DIY.

This might just mean that implementing a json format for list output is doable, but installing from such an output might be DIY.

This makes sense to me as a fair middle ground btw added convenience for users and not forcing pipx to deal with a hairy problem.

i noticed when using pipx reinstall-all  that packages that i installed "stringly" to include extras failed and were tidly logged at the end.  (e.g., "reinstalled ipython, black, failed to reinstall python-language-server[all]", because you have to pipx install 'python-language-server[all]').
Flow like that for injected libraries in a pipx install seems like a good compromise to me.

i noticed when using pipx reinstall-all that packages that i installed "stringly" to include extras failed and were tidly logged at the end. (e.g., "reinstalled ipython, black, failed to reinstall python-language-server[all]", because you have to pipx install 'python-language-server[all]').
Flow like that for injected libraries in a pipx install seems like a good compromise to me.

I don't quite parse what you mean by '"stringly" including extras', but I think I get the basic point, that we could just go through the list and try to install everything as described, and just fail on each particular package that doesn't work out, but otherwise keep installing.
I suppose specifying an invalid package or python path would just fail to install that one as if one were trying to install something improperly from the command line.

Here's a UX question.  How would a command that outputs json be structured?
Possibilities that come to mind:

The user must pipe the output from pipx list to a file, e.g. pipx list --json > my_packages.pipx
Uses pipx list but have an option to specify an output file, e.g. pipx list --json-file=my_packages.pipx
A separate command, e.g. pipx export-spec my_packages.pipx

I'm leaning toward something like 2 or 3, or in general specifying an output file.  It seems to me that you'd really never want to stream the json text to the console anyway.  And for a command that always wants an output file, it seems weird to me to make the user have to redirect by hand.
I suppose the reason to output the json in a stdout stream would be to pipe it to some other command.  But would that be useful?  Would that be useful if pipx had its own native command to import such json?

The best reasons I can think of for streaming to stdout would be if list:

took any alternative format (list --txt, list --yaml
if list was able to show top-level keys only (maybe you have a lot installed and just want to get a high-level view; the Ansible CLI is a good example that adds a lot of extra output you might want to trim).

But specifying a file output makes sense to me as well.

I am much more in favor of streaming to stdout. The user can always redirect to a file if they want to. And you could also pipe the output to jq or grep or whatever else you might need.
pipx list --json | jq '[].env_name' | xargs pipx reinstall

I would pretty naturally spell that as "a subcommand with an optional argument".  so pipx list [-j,--json] does as @gwerbin proposes, and pipx list [-j,json] [outfile] stuffs it into a file of your choice.

I'm starting to think the clearest next step for a PR is to implement (as suggested above)
pipx list --json

Which just sends the json output to stdout (and only the json output to stdout).
I have more experience with macOS and linux--I assume redirecting to a file is just as easy on the various Windows shells?

I have more experience with macOS and linux--I assume redirecting to a file is just as easy on the various Windows shells?

The Windows command prompt and Powershell both support > and >> syntax for output redirection.

JSON is fine, but another modern alternative could be TOML which would be a good balance between machine readable and human readable, without pitfalls of YAML.
[metadata]
version = 0.2

[package.xonsh]
spec = "xonsh"
version = "0.9.27"
apps = ["xon.sh", "xonsh", "xonsh-cat"]
apps_of_dependencies = []
include_apps = true
include_dependencies = false
package = "xonsh"
pip_args = []
suffix = ""

[[app_paths]]
"xon.sh" = "/home/user/.local/pipx/venvs/xonsh/bin/xon.sh"
"xonsh-cat" = "/home/avmo/.local/pipx/venvs/xonsh/bin/xonsh- "Path"

[main_package.app_paths_of_dependencies]

There's no TOML equivalent of JQ yet, and JSON generally has more support across programming languages. Therefore TOML would singificantly limit the usefulness of this option as something you can pipe into other scripts/tools for downstream processing.
Also JSON comes with the Python standard library, which keeps the dependency count down (although I expect that TOML will end up in the stdlib eventually because it's part of PEP 517/518).
That said, maybe Pipx could support multiple output formats?
pipx list --format=list  # default if --format is omitted
pipx list --format=json
pipx list --format=toml
pipx list --format=yaml

There are well-supported TOML and YAML libraries on PyPI to do the output formatting.

I think it is best if this would be combined with an option like pipx reinstall-all --file pipx_packages.toml so that we do not rely on platform dependent shell commands to reinstall pipx packages..
Some pros and cons of different options

pros
cons

list
human friendly
non-standardised, can break with whitespace changes, cannot represent hierarchial data

json
portable, machine-readable, in standard library
verbose, not human friendly

toml
machine-readable, support for basic types, allows comments, human friendly
new

yaml
machine-readable, support for basic types, allows comments, human friendly
can break with whitespace changes, has known security vulnerablities

What is the advantage of having a format that is both human friendly and machine readable?
I'm aware we could get toml to work, but I'm just not sure why we need it.

The argument for TOML usually arises in the context of config files, which this isn't.
I'm fine with just spitting out JSON, fwiw.

config files, which this isn't

It is very similar to a requirements.txt, environment.yml, Config.toml, pyproject.toml etc - call it what you will :)

What is the advantage of having a format that is both human friendly and machine readable?

I understand the arguments made for JSON. Here are some reasons for the file to be human readable. In the event when:

the package does not install (for. eg. change in git URL, due to upgrade of user's Python and wheels being unavailable etc)
tracking the file in a dotfiles repository and handling merge conflicts

it would make it easier for the user to go in and edit the file. That's my take on why TOML would be a better fit.

The json I'm producing right now is pretty-printed.  So editing a particular path by hand is actually quite reasonable.
Also, I don't think I'm eager to have this format be created by hand from scratch, like a config file.  I'd like to keep it pipx-generated so that we don't have to support human quirks that might pop up when we parse it.

Also, I don't think I'm eager to have this format be created by hand from scratch, like a config file.

For the folks who want a language associated w/ config files like TOML/YAML, you can generate those e.g. Poetry generates a pyproject.toml. IMO JSON is the best option out of the gate, just assuming that's what they're thinking of when suggesting TOML/YAML.

Not sure if the linked PR will trigger anyone's notifications so I'll post this:  Check out PR #660 and tell me what you think.  I made a short blurb describing the format.

:+1: from me; i think this is the output side of #109 and it looks pretty tidy!

I'm going to close this now because I believe #660 has implemented json list output format, and that's probably the only other format besides text that we want to implement right now.
Thanks everybody for the good discussion.


修复paddlenlp develop版本适配错误

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.luxiaobin seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


[Core] Optimize SPMD architecture with delta + serialization optimization

This PR is a part of performance roadmap https://github.com/vllm-project/vllm/issues/6801 and based on https://github.com/vllm-project/vllm/issues/6556.
Also it exactly implements https://github.com/vllm-project/vllm/issues/6241
I verified the PR works with

chunked prefill
preemption
PP + TP + ray aDAG
seed (we moved this to worker already thanksfully!)

It doesn't currently work with mp yet.
More details about the PR;

Send delta input for SPMD. Only SequenceData is a state, so we find delta from SequenceData.
Optimize serialization performance using msgspec. I found pickle is at least 2X faster than ray's default serialization, and msgspec is at least 2X faster than pickle.
Integration to PP. We also send delta to next stage workers other than the entire SequenceGroupMetadata.

The following items will go to follow-up PRs

[WIP] Spec decoding + SPMD
Skip prepare_inputs for PP stage > 0 (blocked due to aDAG issue)
Lora tests
guided decoding

Performance benchmark

For realistic online serving workload, I found performance match for high TP (e.g., llama 70B + tp 4). I will post the result soon.
For PP, we found it has much higher throughput

We are still running more benchmark.

[WIP] We are benchmark PP8 + a10g + throughput benchmark.
[WIP] We are going to benchmark 405B + PP 2 + TP 8 on 2 A100 nodes.

TP + small model + throughput benchmark
max_num_seqs=256, opt 125m, tp 2
This is the setup control plane overhead is most amplified. We found there's only 5% regression in this setup.
# after
Throughput: 20.12 requests/s, 23181.18 tokens/s
# before
Throughput: 21.43 requests/s, 24684.19 tokens/s

max_num_seqs=64, opt 125m, tp 2
We found at smaller batch size (which is more common for "real workloads"), SPMD is faster
# after
Throughput: 14.68 requests/s, 28297.32 tokens/s
# before
Throughput: 13.87 requests/s, 26739.96 tokens/s

PP throughput benchmark
We also ran PP benchmark with QPS 10, 1000 requests, PP4 + a10g + llama 3 8B + shareGPT dataset. We enabled chunked prefill to reduce bubbles (it uses OpenAI server). This was before including zeromq + decoupling OpenAI server from engine, so with that, the perf impact may be not as good as this result.
Default
┏━━━━━┳━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓
┃ Run ┃ qps ┃ server_name ┃ e2e_time_s ┃ error_rate ┃ e2e_throughput_request_per_s ┃ itl_ms_mean ┃ ttft_ms_mean ┃ request_e2e_s_mean ┃
┡━━━━━╇━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩
│   0 │ 10  │ local_vllm  │ 38.7       │ 0          │ 2.6                          │ 207.3       │ 422.3        │ 26.8               │
└─────┴─────┴─────────────┴────────────┴────────────┴──────────────────────────────┴─────────────┴──────────────┴────────────────────┘

SPMD
┏━━━━━┳━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓
┃ Run ┃ qps ┃ server_name ┃ e2e_time_s ┃ error_rate ┃ e2e_throughput_request_per_s ┃ itl_ms_mean ┃ ttft_ms_mean ┃ request_e2e_s_mean ┃
┡━━━━━╇━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩
│   0 │ 10  │ local_vllm  │ 30.2       │ 0          │ 3.3                          │ 155.4       │ 312.5        │ 19.1               │
└─────┴─────┴─────────────┴────────────┴────────────┴──────────────────────────────┴─────────────┴──────────────┴────────────────────┘

So we have about 30% throughput improvement (when chunked prefill is disabled, the diff was bigger)
PP Batch inference benchmark
Generally 10~15% throughput improvement for batch inference workloads (this doesn't use OpenAI server) with PP8 + L4 GPUs. It is the batch inference workloads we care at Anyscale.
L4 + PP8 + chunked
Throughput:<PHONE_NUMBER>078258 tokens/s
L4 + PP8 + not chunked
Throughput: 77.0155615308245 tokens/s
L4 + PP8 + chunked + SPMD + no nccl
Throughput:<PHONE_NUMBER>03865 tokens/s
L4 + PP8 + chunked + SPMD + nccl
Throughput:<PHONE_NUMBER>018308 tokens/s

TP latency benchmark
WIP... (make sure it matches)
TP throughput benchmark
WIP... (make sure it matches)

 PR Checklist (Click to Expand)
Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.
PR Title and Classification
Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

[Bugfix] for bug fixes.
[CI/Build] for build or continuous integration improvements.
[Doc] for documentation fixes and improvements.
[Model] for adding a new model or improving an existing model. Model name should appear in the title.
[Frontend] For changes on the vLLM frontend (e.g., OpenAI API server, LLM class, etc.)
[Kernel] for changes affecting CUDA kernels or other compute kernels.
[Core] for changes in the core vLLM logic (e.g., LLMEngine, AsyncLLMEngine, Scheduler, etc.)
[Hardware][Vendor] for hardware-specific changes. Vendor name should appear in the prefix (e.g., [Hardware][AMD]).
[Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.
Code Quality
The PR need to meet the following code quality standards:

We adhere to Google Python style guide and Google C++ style guide.
Pass all linter checks. Please use format.sh to format your code.
The code need to be well-documented to ensure future contributors can easily understand the code.
Include sufficient tests to ensure the project to stay correct and robust. This includes both unit tests and integration tests.
Please add documentation to docs/source/ if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.

Notes for Large Changes
Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with rfc-required and might not go through the PR.
What to Expect for the Reviews
The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:

 After the PR is submitted, the PR will be assigned to a reviewer. Every reviewer will pick up the PRs based on their expertise and availability.
 After the PR is assigned, the reviewer will provide status update every 2-3 days. If the PR is not reviewed within 7 days, please feel free to ping the reviewer or the vLLM team.
 After the review, the reviewer will put an  action-required label on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.
 Please respond to all comments within a reasonable time frame. If a comment isn't clear or you disagree with a suggestion, feel free to ask for clarification or discuss the suggestion.

Thank You
 Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!

cc @ruisearch42

I am fixing tests and resolving merge conflict now. Please provide a high level feedback!

I will try to finish my review next week.

hmm mypy failures are unexpected.. investigating

LGTM, mostly minor issues. Since this is a significant change, just wondering if this can be config guarded.

LGTM, mostly minor issues. Since this is a significant change, just wondering if this can be config guarded.
Entire delta feature is guarded. I can also add additional env var if you think that's better. For msgspec.STruct, it is pretty difficult to guard unfortunately..

All tests are passing now

I will merge the PR tomorrow unless there are more comments!

@rkooo567  can you wait for me to take a look this week? sorry for the delay 🙏

addressing comments soon...

comment addressed and test fixed. @youkaichao lmk if you want me to do implicit conversion of list -> array (without having correct input type)

lm-eval-small-models  seems to fail in the master.

@youkaichao lmk if you have additional feedback!

will take another pass today.

working on fixing CI failures. It'd be great if we can merge this by this weekends, so I can merge the spec decoding changes! We are seeing pretty good result from spec decoding actually (I will share the result soon )

My concern is, what do you deal with the data msgpack cannot send, e.g. logits processor?

we can implement the custom hook actually. Actually array is not supported by msgspec, and I manually implemented the ser/deser hook. I think things like logit processor, we can probably just use pickle.dumps and pickle.loads inside a hook. (also, imo we should remove logit_processor APIs, and just make it use static string, but it is part of other proposal... https://github.com/vllm-project/vllm/issues/5423)

My suggestion is to have a ARRAY_INT32 constant somewhere, and switches between I or L depending on the actual size.

This is a great point, and I just reverted it to 'l'. I think if majority platform just uses 4 bytes I, it is probably simpler to just keep it to 'L' for now.

Since this feature is mainly used for ray spmd path, I'm okay with it if it cannot deal with logit processors currently, and we will refactor logits processor anyway in the future.

My suggestion is to have a ARRAY_INT32 constant somewhere, and switches between I or L depending on the actual size.

I think this part should be addressed. We should not have a machine-dependent code here. It would be difficult to modify in the future.
We can use some constant like VLLM_TOKEN_ID_ARRAY_TYPE , in case we need to change it to some other types in the future.

We can use some constant like VLLM_TOKEN_ID_ARRAY_TYPE , in case we need to change it to some other types in the future.

yeah actually a good point. working on it now

I'd like to see more perf number comparison here, what's the benefit with this pr in spmd case, and what's the gap (if any) with the main branch (non-spmd case).

I'd like to see more perf number comparison here, what's the benefit with this pr in spmd case, and what's the gap (if any) with the main branch (non-spmd case).

Yep! I am preparing a doc actually.
https://docs.google.com/document/d/1XSKPna9-seHrYf1Vwaw-puKWW15avk9Myn8yzWbSQ9Q/edit#heading=h.jtbcvct9biud
It is not fully done yet (still need result from spec decoding,  latency benchmark, and 2 nodes PP perf). But there are some numbers here already

VLLM_TOKEN_ID_ARRAY_TYPE

addressed

I will merge it tonight unless there are more comments!

@rkooo567 maybe you could wait until tomorrow, I will try to review tonight (sorry it has been on my todo list!)

@njhill is it okay if I follow up after merging the PR? It's been almost 3 weeks since I made a PR... and it affects our internal timeline

@rkooo567 was hoping it should be ok to wait until Sunday rather than Saturday night :)
I was now planning to look early in the morning..
My main question is whether it’s still optional/opt-in and whether it has any negative perf impact at all on any cases where spmd is not enabled?
Where we are making multiple different kinds of optimizations I wonder if it’s better for them to be in separate PRs so that the benefits can be assessed individually. Like the changes here for msgspec, deltas, use of python arrays, etc. It would probably then also be easier to get them merged quicker and have bit less merge conflict pain etc.

My main question is whether it’s still optional/opt-in and whether it has any negative perf impact at all on any cases where spmd is not enabled?

All the delta input & spmd related code is opt-in, and completely separated from original code path.
There's no other change other than delta + serialization. array related changes are same as before (just a bit of refactoring for msgspec). Serialization optimization doesn't affect the main code path because it is not serialized for regular path.
Only con I can think of is that now all sequence group metadata inputs are using msgspec.Struct instead of dataclass. Both of them has almost the same semantics, but msgspec.Struct is a little more restrictive (e.g., you cannot have union of the same type). I think performance-wise, I actually msgspec.Struct may be faster because it is a c object technically. For example, I found removing this reallocation optimization has no performance impact after this PR. https://github.com/vllm-project/vllm/pull/7109#discussion_r1713178576

also benchmark result from master; https://buildkite.com/vllm/ci-aws/builds/7074#01916209-fef7-45e9-9b63-34641474c4ea
this PR: https://buildkite.com/vllm/ci-aws/builds/7072
so I think there's almost no change

Some more higher level thoughts (not suggesting to change in this PR):

Yeah I generally agreed with the idea. I think once the basic spmd version is working, it should be straightforward to make it fulll spmd. I think make batching stateful within workers are the right idea.
One diff in my mind is that imo it is still more beneficial to decouple scheduler from workers for more complicated scheduling requirement in the future (starting from pipeline parallelism). And I think the data that needs to be exchanged from scheduler <> worker is pretty small if the batches become stateful. And also it should be totally possible to have low overhead for scheduler <> worker (I think it is already less than 300us for most of cases).

comments all addressed @njhill !

okay, I am merging the PR. I will follow up new comments in a follow up PR


docs: publish listening ports explicitly

Instead of using --network=host so that testing on macOS (and probably Windows) works. (the host option doesn't work on some systems--https://docs.docker.com/network/host/)
Final deploy target will probably be Linux, but this allows the command to work on more OSes I believe?

Actually, never mind. This doesn't make sense.


Proxy to PouchDB instance

This PR implements the proxy to PouchDB functionality.
Closes #12
Closes #14

Hey Chris, can you tell when you’ll be able to continue working on this?

this should also fix https://github.com/hoodiehq/hoodie-server-store/issues/14

Hey @christophwitzko is this ready for review, or are you still working on it?

@gr2m Yes, it is ready for review. :smile:

oops, documentation is missing. :speak_no_evil:

awesome, I’ll review this right away :) If you could add documentation that’d be great :)

@gr2m on it :relaxed:

@gr2m updated the PR :v:

I tried to get our Tracker App running with PouchDB only, but not quite there yet. You can try it out with this branch: https://github.com/hoodiehq/hoodie-app-tracker/tree/tent-pouchdb
I’ve created tent-pouchdb branches for hoodie and hoodie-server too. So far I only made adjustments to hoodie-server: https://github.com/hoodiehq/hoodie-server/commit/42fc1b40e1c1f0630f28d6be825230a99a57058f
We want to store all database related files in .hoodie/data. I found the prefix option for PouchDB, but express-pouchdb still creates a log.text in the root folder of the app. Somehow we must be able to pass options for the log file & log level, but I’m not sure how: https://github.com/pouchdb/express-pouchdb/blob/master/lib/logging-infrastructure.js#L85-L86
Any idea?

Okay, that is quite tricky. One option is to use the minimumForPouchDB mode (that would remove all the authentication, authorization and session features). Another thing we can do is to add a config option to express-pouchdb.

@christophwitzko could you have a quick look at https://github.com/pouchdb/pouchdb-express-router? I think it’s a minimal subset of what PouchDB needs for its sync. And as we only use the sync APIs, maybe it will be a better solution with less overhead?

pouchdb-express-router does not include the changes feed. the minimumForPouchDB mode looks perfect for our usecase :+1:

ok, pouchdb-express-router does have a changes feed, I overlooked it. I would still stuck with express-pouchdb it looks quite more maintained.

okay thanks for looking into it :+1:

@gr2m I have added the minimumForPouchDB mode config.

HOLY SHID IT WORKS \o/ I just used the new Hoodie with PouchDB, so frigging cool

@gr2m awesome :sparkling_heart:

@gr2m rebased everything :ok_hand:

LGTM :+1:

LGTM – amazing work :clap:

LGTM

LGTM


Fix issues with rotator transforms

Addresses: #724

Thanks for the pull request @nithinp7!

:heavy_check_mark: Signed CLA found.
:grey_question: CHANGES.md was not updated.

If this change updates the public API in any way, please add a bullet point to CHANGES.md.

Reviewers, don't forget to make sure that:

[ ] Cesium for Unreal Samples works.

Thanks again for your contribution @nithinp7!
No one has commented on this pull request in 30 days. Maintainers, can you review, merge or close to keep things tidy?
I'm going to re-bump this in 30 days. If you'd like me to stop, just comment with @cesium-concierge stop. If you want me to start again, just delete the comment.

@kring What should we do with this PR? This fixes at least one thing that's definitely wrong, but we should probably throw out these functions in general and encourage people to only use the matrix version. I'll take this PR out of draft, but feel free to close it without merging if you would prefer to deprecate these functions instead.


[FLINK-28417][table] Add interfaces for LookupCache and default implementation

What is the purpose of the change
This pull request introduces interfaces for LookupCache and CacheMetricGroup, also a default implementation of LookupCache.
Brief change log

Add new interfaces LookupCache and CacheMetricGroup
Add default implementation for LookupCache as DefaultLookupCache
Add an internal implementation for CacheMetricGroup as InternalCacheMetricGroup

Verifying this change
This change added unit tests for the newly added DefaultLookupCache and InternalCacheMetricGroup
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@wuchong Thanks for the review! I addresses your comments in the latest commit. Please take a look when you are available.

@wuchong @SmirAlex @lincoln-lil Thank you all for the review! I updated the PR to address your comments. Please take another look at your convenience.


[Bug]Hide Gift Feature not working

Hi! I just installed betterdiscord to remove the minimum window size limit, however I found that the hide gift feature is not working. Keep up the good work, devs!
Stable 84785 (fe5e237)
BetterDiscord 1.1.0
Host 1.0.9001
Windows 10 64-Bit (10.0.19042)

kinda sucks the official settings are broken, for now you can just use css snippets, this one works really well that i found
/* 🠗 Hide GIF button 🠗*/ .buttonContainer-28fw2U:not(.send-button) .button-3AYNKb, /* 🠗 Hide GIft button 🠗*/ .buttons-3JBrkn > .button-38aScr > :not(.translateButton-DhP9x8) {display: none;}

This has been fixed at latest level


Clean up table generation

Two issues:

Code has been copy-pasted and can be turned into a for loop
There are two separate functions that return table items, and they behave slightly differently.

https://github.com/ianklatzco/twitter-to-bsky/blob/892a27dc3af8ac79f311320cf0c287f4f0bdd261/webapp/main.py#L241

The User class is used in two ways - once to simple hold username strings, and the other time to hold profiles.
Instead, we should not consume remote profiles and just store the username contained in the twitter bio into User.
We could add back in bsky profile fetching asynchronously, but currently it makes things pretty slow 😔

@Heartade cleaning up that  codegen would be Awesome..... 🥺🙏🙇♂

done in #10!


Quart will not auto-reload on code changes when run via docker container

Hello! I am using latest quart==0.18.0 When running quart in debug mode via QUART_DEBUG=true and QUART_ENV=development, quart will not auto-reload when running in a docker container. The source directory is volume-mounted properly into the container. A similar issue affecting hypercorn also been reported here: https://stackoverflow.com/questions/67506424/hypercorn-with-reload-and-docker-volumes, albeit with no solution yet.
docker-compose.yaml
services:
  app:
    build: .
    command: quart run --host=<IP_ADDRESS> --port=8080
    environment:
      QUART_APP: src:app
      QUART_DEBUG: true
      QUART_ENV: development
    ports:
      - 8080:8080
    volumes:
      - ./src:/app/src

Dockerfile
FROM python:3.10-slim-bullseye
WORKDIR /app
RUN pip install hypercorn==0.14.3 quart==0.18.0
COPY src ./src
EXPOSE 8080
# This is the production command; docker-compose.yaml overwrites it for local development
CMD hypercorn --bind <IP_ADDRESS>:8080 src:app

src/init.py
from quart import Quart

app=Quart(__name__)

@app.route('/', methods=['GET'])
def get_root():
    return "Hello world!"

Now run the stack via docker-compose up and change src/__init__.py. quart will not reload.
If however the command in docker-compose.yaml is replaced to use hypercorn --reload, the server will reload on code changes!
services:
  app:
    build: .
    command: hypercorn --bind <IP_ADDRESS>:8080 --reload src:app
    # Rest as above
    # ...

This should be fixed by 7634a5a56b757eb6c5602407114d587ace8f5590 (not yet released)

Now released in 0.18.1


[WIP] Reprojection quest

Handles reprojection errors for vector data.

Oops, noticed that the error message in<EMAIL_ADDRESS>versus<EMAIL_ADDRESS>is different. I've updated the error message check to be more general and just look for vector_tile_processor:. I would add reproject to the check, but it's misspelled in mapnik-vector-tile so we'll stick with this for now.


added build_version label to gossip handshake

It will allow us to monitor which nodes are using which version of the binary.
I've added a metric to count peers by their version (it is unbounded though).
I've added it to the http page display as well.
unrelated: I've removed Default impl for committees - it was creating invalid object.
unrelated: Exposed the validator::ProofOfPossession type.

The string is constrained by the Handshake message size. We will restrict it further if the need arises.


Azure Sphere is not able to connect to Edge device -- TLS handshake failed

Hello, I am trying to configure and setup Azure Sphere device as child device to a IoT Edge device. Following is the procedure

Followed this link https://docs.microsoft.com/en-us/azure/iot-edge/tutorial-nested-iot-edge?view=iotedge-2020-11 to forma a hierarchy of Azure IoT Edge devices (Top layer + low layer 1 + low layer 2), total 3 layers.
Followed https://docs.microsoft.com/en-us/azure-sphere/app-development/setup-iot-edge?tabs=cliv1 to configure SPhere as child device of Top layer.
I am using Azure Samples, Azure IoT Example to connect to edge device. Made necessary changes in CMakeLists.txt and app_manifest.json file (Hostname as IP address of Edge Device, certificate as the root certificate (iotedge_config_cli_root.pem) generated at step 1 above)
Via Azure portal set the Azure Sphere device as child of edge device (top-layer).
Azure Sphere device is manually provisioned by following instructions in link https://docs.microsoft.com/en-us/azure-sphere/app-development/setup-iot-hub?tabs=cliv2beta
Azure Edge device, as per Step 6. Provide the tenant CA certificate of the Azure Sphere device to the IoT Edge device, appended tenant CA certificate in PEM format to iotedge_config_cli_root.pem.
Restarted the edge device and tried to sphere device tried to connect to edge device. But it was unsuccessful. Following were errors at edgeHub module

<4> 2021-08-18 13:09:00.901 +00:00 [WRN] - TLS handshake failed., System.AggregateException: One or more errors occurred. (Authentication failed, see inner exception.)
---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.
---> Interop+Crypto+OpenSslCryptographicException: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
--- End of inner exception stack trace ---
at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, Byte[] recvBuf, Int32 recvOffset, Int32 recvCount, Byte[]& sendBuf, Int32& sendCount)
at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteContext& context, ArraySegment1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)    --- End of inner exception stack trace ---    at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)    at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)    at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)    at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)    at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)    at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest) --- End of stack trace from previous location where exception was thrown ---    at System.Net.Security.SslStream.ThrowIfExceptional()    at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)    at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)    at System.Net.Security.SslStream.EndAuthenticateAsServer(IAsyncResult asyncResult)    at System.Net.Security.SslStream.<>c.<AuthenticateAsServerAsync>b__68_2(IAsyncResult iar)    at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of inner exception stack trace ---, 734368d8
<4> 2021-08-18 13:10:01.660 +00:00 [WRN] - TLS handshake failed., System.IO.IOException: Channel is closed, 5cc9b5ea

At Sphere side below was error

IoT connection status: IOTHUB_CLIENT_CONNECTION_NO_NETWORK
INFO: Azure IoT Hub connection started.
INFO: Azure IoT Hub connection complete.
Azure IoT connection status: IOTHUB_CLIENT_CONNECTION_NO_NETWORK
INFO: Azure IoT Hub connection started.
INFO: Azure IoT Hub connection complete.
Azure IoT connection status: IOTHUB_CLIENT_CONNECTION_NO_NETWORK
INFO: Azure IoT Hub connection started.
/run/daa/41d781ff-519a-406d-b1dd-c4078f372dff
[Azure IoT] Using HSM cert at /run/daa/41d781ff-519a-406d-b1dd-c4078f372dff
[Azure IoT] Using HSM cert at /run/daa/41d781ff-519a-406d-b1dd-c4078f372dff
[Azure IoT] Using HSM cert at /run/daa/41d781ff-519a-406d-b1dd-c4078f372dffINFO: Azure IoT Hub connection complete.
Azure IoT connection status: IOTHUB_CLIENT_CONNECTION_NO_NETWORK
INFO: Azure IoT Hub connection started.
INFO: Azure IoT Hub connection complete.
Azure IoT connection status: IOTHUB_CLIENT_CONNECTION_NO_NETWORK
INFO: Azure IoT Hub connection started.
INFO: Azure IoT Hub connection complete.
Azure IoT connection status: IOTHUB_CLIENT_CONNECTION_NO_NETWORK
INFO: Azure IoT Hub connection started.
INFO: Azure IoT Hub connection complete.
Azure IoT connection status: IOTHUB_CLIENT_CONNECTION_NO_NETWORK
INFO: Azure IoT Hub connection started.**

As per https://github.com/Azure/azure-sphere-samples/blob/master/Samples/AzureIoT/Troubleshooting.md the error points to These errors may occur if:

Inbound port 8883 has not been opened on the IoT Edge device. For more information, see Step 3 in Open IoT Edge gateway device ports for communication.

The sample is using an incorrect or invalid IoT Edge device root CA certificate.

The edgeAgent and edgeHub modules are not running on the IoT Edge device.

But reason 1 and 3 can be ruled out. Inbound port 8883 is open at edge device and all edge modules are running.
Need your help in resolving this issue. Note that the sphere device was able to connect to IoT hub directly.
Thanks,
Jeevan
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 430b0292-ece1-8efc-9526-cb8835840984
Version Independent ID: 83ce7260-dbba-9585-775e-103a7946b330
Content: Set up Azure IoT Edge for Azure Sphere
Content Source: AzureSphere/articles/app-development/setup-iot-edge.md
Product: azure-sphere
GitHub Login: @LouisBerner
Microsoft Alias: azurespheredocs

@jeevannagvekar  Thanks for the question! We are investigating and will update you shortly.

@jeevannagvekar Sorry for the delay in response. Thanks for leveraging Azure docs feedback channel to raise this question. This channel is reserved for ‘Azure docs’ feedback- for sharing doc feedback and suggesting content improvements. After reviewing the ask, it seems like generic query and not a doc feedback.
Hence, we request you to please reach out in forums like Microsoft Q&A or Stackoverflow. We have our engineers and other community members who will be happy to assist. Forums have a better suited audience for such queries. Once your issue is posted in forum, please do share the forum thread link here for tracking.
If you have a document which you are following and if you think it needs an update, please let us know about that here.
We will now proceed to close this thread. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.
Thank you


Remove reference to SpecFlow feature parser

We should rely on the generic Gherkin parser in the x97mdr/gherkin repository.  Any references to the SpecFlow parser should be removed

This should also allow us to capture Scenario Outlines which are impossible to do with the current SpecFlow parser


Parameterizing blur size

Cool app! Can't wait to see where it goes :)
I attempted to parameterize the blur depth in the blur shader you posted to the shared library, however it seems 'variable length arrays are not supported in Metal':

I know this kind of thing is possible in general, but I've never implemented it from scratch. Would parameterizing the blur amount require a more substantial refactoring of the blur implementation to avoid the 'no variable length arrays' constraint?
Thanks!

Hi,
I guess the thing you could do is to fix the array on the array size for the largest parameter size you want to support and calculate kernel size for the parameter independently.
Does this help ? We need to work around the limitations. Other way would be to pass the array to the shader but this kind of custom data flows to the shader are not yet supported in ShaderMania.

Hi,
I guess the thing you could do is to fix the array on the array size for the largest parameter size you want to support and calculate kernel size for the parameter independently.
Does this help ? We need to work around the limitations. Other way would be to pass the array to the shader but this kind of custom data flows to the shader are not yet supported in ShaderMania.

I think it does help, yes - thanks :) If I come up with anything I'll post it the cloud library.


[5.1] Added command to make tests

So many great artisan commands to make Models, Controllers etc. I would like to make one to make a Test File.
For example
php artisan make:test FooTest

Would make a default file to get going
<?php

class FooTest extends \TestCase {

}

Or
php artisan make:test FooTest --database_migration

<?php

class FooTest extends \TestCase {
    use \Illuminate\Foundation\Testing\DatabaseMigrations;

}

Please rebase.

Need to fix the one line too where I did not remove the line use Illuminate\Support\Str; which was caught in the StyleCI

+1
-----Original Message-----
From: "Alfred Nutile"<EMAIL_ADDRESS>Sent: 9/6/2015 6:46 PM
To: "laravel/framework"<EMAIL_ADDRESS>Subject: [framework] [Proposal] PHP Artisan Command to Make Test (#10170)
So many great artisan commands to make Models, Controllers etc. I would like to make one to make a Test File.
For example
php artisan make:test FooTest
Would make a default file to get going
<?php

class FooTest extends \TestCase {

}Or
php artisan make:test FooTest --database_migration
<?php

class FooTest extends \TestCase {
    use \Illuminate\Foundation\Testing\DatabaseMigrations;

}

You can view, comment on, or merge this pull request online at:
  https://github.com/laravel/framework/pull/10170
Commit Summary
Adding files to make php artisan make:test FooTest work
File Changes
A src/Illuminate/Foundation/Console/TestMakeCommand.php (80)
A src/Illuminate/Foundation/Console/stubs/test.database_migration.stub (8)
A src/Illuminate/Foundation/Console/stubs/test.stub (8)
M src/Illuminate/Foundation/Providers/ArtisanServiceProvider.php (14)
Patch Links:
https://github.com/laravel/framework/pull/10170.patch
https://github.com/laravel/framework/pull/10170.diff
—
Reply to this email directly or view it on GitHub.

Please squash to one commit.

@GrahamCampbell okay working on that now. Sorry still getting the hang of the rebase work flow.

I will learn from this pain how to rebase or die trying!

oops did not mean this to start a new cycle still need to squash the three fixes noted above

okay I think this now includes the rebase of removing the back slash and the comments

Please don't remove the comments if Taylor has them in the other stubs.

Please don't remove the comments if Taylor has them in the other stubs.

Nah I would never do that, I was just removing the one I had there. Just noting I got the file from model stubs.
Anyway these files I made no longer have that \\ you commented on.

What I'm saying is if the other ones have them, then you need to add them to your new ones.

:( hold on going back to update and rebase

k now added slashes back to match the model and also made my trait just like the other traits defining the use statement above the class.
<?php

use \Illuminate\Foundation\Testing\DatabaseMigrations

class DummyClass extends TestCase
{
    use DatabaseMigrations;
    //
}

Just like here src/Illuminate/Foundation/Console/stubs/event.stub
<?php

namespace DummyNamespace;

use DummyRootNamespaceEvents\Event;
use Illuminate\Queue\SerializesModels;
use Illuminate\Contracts\Broadcasting\ShouldBroadcast;

class DummyClass extends Event
{
    use SerializesModels;

    /**
     * Create a new event instance.
     *
     * @return void
     */
    public function __construct()
    {
        //
    }

    /**
     * Get the channels the event should be broadcast on.
     *
     * @return array
     */
    public function broadcastOn()
    {
        return [];
    }
}

Those are fundamentally different though, because the comments are inside methods.

Also, you don't want a leading slash there.

You need:
class DummyClass extends TestCase
{
    use DatabaseMigrations;

    //
}

Because the // will be where you actually start writing code.

right bad example if you look at my files the // are as you note. Add extra line brake though after the Trait.

now with line break between the // and the use trait statement.

LGTM :)

great!


Creating property on non-config slot should throw type error.

Based on the spec, when we try to set property on the non-configurable property slot should throw type error. Fixed that multiple places.

@suwc - please have a look.

fixes #757.

:shipit:

LGTM.


Added VTA for VK_EXT_tooling_info

This VTA displays a cube and prints out tooling info on the console.
Tested on Windows and GGP.
Example output:
Tooling Info returned 1 active tools.
Tool #0:
Name: RenderDoc
Version: v1.18 (0f645b557d84eb42473399492691a087d5624f7a)
Purpose: TRACKING
Purpose: MODIFYING_FEATURES
Purpose: DEBUG_MARKERS
Description: Debugging capture layer for RenderDoc
Layer:
(Baldur specifically left out the layer string for some reason, but if
it was there it would have been printed.)

I got a crash on my Linux machine, which is really weird, that part of code is the same as the cube sample, I'm not sure why.  Probably due to my env is kind of different.
#0  vkAllocateCommandBuffers (device=0x0, pAllocateInfo=0x7ffff79fa1b0, pCommandBuffers=0x7ffff79fa198)
    at /root/sdk-build/<IP_ADDRESS>/source/Vulkan-Loader/loader/trampoline.c:1530
#1  0x0000555555670101 in std::result_of<VkResult (*(vulkan::VkDevice, VkCommandBufferAllocateInfo*, VkCommandBuffer_T**))(VkDevice_T*, VkCommandBufferAllocateInfo const*, VkCommandBuffer_T**)>::type LazyFunction<VkResult (*)(VkDevice_T*, VkCommandBufferAllocateInfo const*, VkCommandBuffer_T**), VkDevice_T*, vulkan::DeviceFunctions>::operator()<vulkan::VkDevice, VkCommandBufferAllocateInfo*, VkCommandBuffer_T**>(vulkan::VkDevice const&, VkCommandBufferAllocateInfo* const&, VkCommandBuffer_T** const&) ()
#2  0x000055555566129c in vulkan::CreateCommandBuffer(vulkan::VkSubObject<vulkan::CommandPoolTraits, vulkan::DeviceTraits>*, VkCommandBufferLevel, vulkan::VkDevice*) ()
#3  0x000055555562eeca in vulkan::VulkanApplication::GetCommandBuffer(unsigned int) ()
#4  0x0000555555636b44 in sample_application::Sample<CubeFrameData>::Sample(containers::Allocator*, entry::EntryData const*, unsigned int, unsigned int, unsigned int, unsigned int, sample_application::SampleOptions const&, VkPhysicalDeviceFeatures const&, std::initializer_list<char const*>, std::initializer_list<char const*>) ()
#5  0x0000555555630d05 in CubeSample::CubeSample(entry::EntryData const*) ()
#6  0x000055555562d10f in main_entry(entry::EntryData const*) ()
#7  0x00005555556a27de in main::{lambda()#1}::operator()() const ()
#8  0x00005555556a2f7a in void std::__invoke_impl<void, main::{lambda()#1}>(std::__invoke_other, main::{lambda()#1}&&)
    ()
#9  0x00005555556a2f3d in std::__invoke_result<main::{lambda()#1}>::type std::__invoke<main::{lambda()#1}>(main::{lambda()#1}&&) ()
#10 0x00005555556a2eea in void std::thread::_Invoker<std::tuple<main::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#11 0x00005555556a2ebe in std::thread::_Invoker<std::tuple<main::{lambda()#1}> >::operator()() ()
--Type <RET> for more, q to quit, c to continue without paging--
#12 0x00005555556a2ea2 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main::{lambda()#1}> > >::_M_run() ()
#13 0x00007ffff7e3b694 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007ffff7ae9947 in start_thread (arg=<optimized out>) at pthread_create.c:435
#15 0x00007ffff7b79a44 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

I think as long as it works on GGP, it should not be a problem

I got a crash on my Linux machine, which is really weird, that part of code is the same as the cube sample, I'm not sure why. Probably due to my env is kind of different.
#0  vkAllocateCommandBuffers (device=0x0, pAllocateInfo=0x7ffff79fa150, pCommandBuffers=0x7ffff79fa148) at /root/sdk-build/<IP_ADDRESS>/source/Vulkan-Loader/loader/trampoline.c:1530
#1  0x00000000004e41c3 in LazyFunction<VkResult (*)(VkDevice_T*, VkCommandBufferAllocateInfo const*, VkCommandBuffer_T**), VkDevice_T*, vulkan::DeviceFunctions>::operator()<vulkan::VkDevice, VkCommandBufferAllocateInfo*, VkCommandBuffer_T**> (this=0x7ffff03e5210, args=@0x7ffff79fa130: 0x7ffff79fa148, args=@0x7ffff79fa130: 0x7ffff79fa148, args=@0x7ffff79fa130: 0x7ffff79fa148) at ../vulkan_wrapper/lazy_function.h:56
#2  0x00000000004dcb4c in vulkan::CreateCommandBuffer (pool=0x7ffff01fe330, level=VK_COMMAND_BUFFER_LEVEL_PRIMARY, device=0x7ffff79fa738) at ../vulkan_helpers/helper_functions.cpp:1012
#3  0x00000000004afd77 in vulkan::VulkanApplication::GetCommandBuffer (this=0x7ffff79fa548, queueFamilyIndex=0) at ../vulkan_helpers/vulkan_application.h:736
#4  0x00000000004ae07e in sample_application::Sample<CubeFrameData>::Sample (this=0x7ffff79fa508, allocator=0x7fffffffdb20, entry_data=0x7fffffffda78, host_buffer_size_in_MB=1, image_memory_size_in_MB=512,
    device_buffer_size_in_MB=1, coherent_buffer_size_in_MB=1, options=..., physical_device_features=..., instance_extensions=..., device_extensions=...)
    at ../application_sandbox/sample_application_framework/sample_application.h:249
#5  0x00000000004ac3df in CubeSample::CubeSample (this=0x7ffff79fa508, data=0x7fffffffda78) at ../application_sandbox/tooling_info/main.cpp:54
#6  0x00000000004aba7a in main_entry (data=0x7fffffffda78) at ../application_sandbox/tooling_info/main.cpp:351
#7  0x0000000000519e1c in main::$_0::operator() (this=0x667ed8) at ../support/entry/entry.cpp:489
#8  0x0000000000519ded in std::__invoke_impl<void, main::$_0> (__f=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
#9  0x0000000000519d7d in std::__invoke<main::$_0> (__fn=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96
#10 0x0000000000519d55 in std::thread::_Invoker<std::tuple<main::$_0> >::_M_invoke<0ul> (this=0x667ed8) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:253
#11 0x0000000000519d25 in std::thread::_Invoker<std::tuple<main::$_0> >::operator() (this=0x667ed8) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:260
#12 0x0000000000519c89 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main::$_0> > >::_M_run (this=0x667ed0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211
#13 0x00007ffff7e3b694 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007ffff7acf947 in start_thread (arg=<optimized out>) at pthread_create.c:435
#15 0x00007ffff7b5fa44 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

I think as long as it works on GGP, it should not be a problem

there's  a log printout "Could not find physical device or queue that can present"
I guess becuase of my ICD doesn't support this yet and the device creation failed.

Yes, unfortunately the common failure more for VTAs whose extensions are not supported seems to be to print that error message and then crash.  Ideally the message would be more helpful and it would return with an error code instead, but I guess these are pretty rough test apps and not meant to be that polished...?  I do feel kind of dirty leaving it like this, but I want to be consistent with the other existing tests.

application_sandbox/README.md seemed to be way out of date to me.  I'll check tomorrow and see... maybe we can add all of the missing ones.


[cosmosdb] Updating Cosmos DB SDK and changing update api

This PR contains two changes -

Moved to latest cosmos sdk 4.28.0
Changed the cosmos ycsb update operation to use patch api, earlier it was inefficiently calling two apis for single operation i.e.  read and then replace.

Any update on this please

Update please.

@joshelser can you please suggest who is current maintainer of this repo , i see your review in couple of other PRs. This PR has been active since April. Is this something  you can help ?


Java language bindings

Developers using Java would like to use OpenTimelineIO. We could provide language bindings to make this easy to integrate with Java.

A starting point: https://github.com/reinecke/java-opentimelineio
Noting down this for my convenience(about c/cpp linking): https://gist.github.com/gubatron/32f82053596c24b6bec6

Could someone please create a feature branch so that I can open a PR for this?

Here you go https://github.com/PixarAnimationStudios/OpenTimelineIO/tree/java-bindings


Use of socioeconomic metic weights in the synthea.properties file.

Hi all,
I wanted to clarify use of socioeconomic metic weights in the synthea.properties file.
a) Do each of these need to add up to 1, or is that just for the income/education/occupation grouping? I note that that others don't add upto 1.0.
b) What does it mean if I change the income weight from 0.2 to 0.7? does it mean that income currently has 20% impact on determine health status, and that increasing it to 0.7 will make low income people significantly sicker?
generate.demographics.socioeconomic.weights.income = 0.2
generate.demographics.socioeconomic.weights.education = 0.7
generate.demographics.socioeconomic.weights.occupation = 0.1
generate.demographics.socioeconomic.score.low = 0.0
generate.demographics.socioeconomic.score.middle = 0.25
generate.demographics.socioeconomic.score.high = 0.66
generate.demographics.socioeconomic.education.less_than_hs.min = 0.0
generate.demographics.socioeconomic.education.less_than_hs.max = 0.5
generate.demographics.socioeconomic.education.hs_degree.min = 0.1
generate.demographics.socioeconomic.education.hs_degree.max = 0.75
generate.demographics.socioeconomic.education.some_college.min = 0.3
generate.demographics.socioeconomic.education.some_college.max = 0.85
generate.demographics.socioeconomic.education.bs_degree.min = 0.5
generate.demographics.socioeconomic.education.bs_degree.max = 1.0
generate.demographics.socioeconomic.income.poverty = 11000
generate.demographics.socioeconomic.income.high = 75000

Do each of these need to add up to 1, or is that just for the income/education/occupation grouping? I note that that others don't add upto 1.0.

Only the generate.demographics.socioeconomic.weights.* properties need to add up to 1.0. The weighting determines the relative weight of income, education, and occupation in determining a normalized socioeconomic category. See the below code (in fact, the entire class is useful to look at overall for this question): https://github.com/synthetichealth/synthea/blob/a3482c856d30410e438047845796095a827cca41/src/main/java/org/mitre/synthea/world/geography/Demographics.java#L349-L362

b) What does it mean if I change the income weight from 0.2 to 0.7? does it mean that income currently has 20% impact on determine health status, and that increasing it to 0.7 will make low income people significantly sicker?

No. It means the normalized income values from the census would be considered 70% of the socioeconomic calculation. While the socioeconomic status should be tied into many modules, it is currently only tied into a few places: COPD, Dementia, Homelessness, Opioid Addiction. You can find them by searching src/main/resources/modules for the string "socioeconomic".


[e2e]: nodejs push branch1 default slsa3

Repo: https://github.com/slsa-framework/example-package/tree/branch1
Run: https://github.com/slsa-framework/example-package/actions/runs/9691550743
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.nodejs.push.branch1.default.slsa3.yml
Workflow runs: https://github.com/slsa-framework/example-package/actions/workflows/e2e.nodejs.push.branch1.default.slsa3.yml
Trigger: push
Branch: branch1
Date: Thu Jun 27 06:14:32 UTC 2024

Repo: https://github.com/slsa-framework/example-package/tree/branch1
Run: https://github.com/slsa-framework/example-package/actions/runs/9708265976
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.nodejs.push.branch1.default.slsa3.yml
Workflow runs: https://github.com/slsa-framework/example-package/actions/workflows/e2e.nodejs.push.branch1.default.slsa3.yml
Trigger: push
Branch: branch1
Date: Fri Jun 28 06:13:11 UTC 2024
Tests are passing now. Closing this issue.


Feature Request: Hide Accessory Component in Homebridge/Home.app

I have a hubspace ceiling fan that is exposed as two separate accessories, a fan and a ceiling light. I never want to turn on the ceiling light, but I want to control the fan in Home.app. It would be ideal if there was a way to selectively hide specific accessories.

I'd love this functionality too. I have like 30+ Hubspace landscape lights that I don't want exposed to homekit at all. I also have several Hubspace smart plugs and fans that I DO want exposed. Is there a way to selectively include/exclude devices in the JSON config?


Security vulnerability in one of the dependencies

Hello,
Our vulnerability detection tool reports a security vulnerability in one of your dependencies:
<EMAIL_ADDRESS>-><EMAIL_ADDRESS>-><EMAIL_ADDRESS>The vulnerability is in<EMAIL_ADDRESS>More information regarding the vulnerability itself - here.
Could you update the version of your System.Text.Json dependency to depend on a secure version of System.Text.Encodings.Web? The secure versions, as stated in the issue, are 4.5.1, 4.7.2 and >=5.0.1.
Thanks.

Hello @tiliev, thanks for reporting this issue. You're right, System.Text.Json v4.6.0 depends on System.Text.Encodings.Web v4.6.0 which has a security vulnerability. However, this vulnerability does not exist in Microsoft.Extensions.Configuration.AzureAppConfiguration package because we have another dependency that overrides the System.Text.Encodings.Web package to v4.7.2, which is a safe version. Here's the dependency chain:
Microsoft.Extensions.Configuration.AzureAppConfiguration (5.0.0)  -> Azure.Data.AppConfiguration (1.2.0) -> Azure.Core (1.20.0) -> System.Text.Encodings.Web (4.7.2)
So technically, even though we reference System.Text.Json 4.6.0, the actual version of System.Text.Encodings.Web that gets installed is 4.7.2 because Azure.Core mandates that version.
That being said, we are also looking into addressing the issue more explicitly.

@avanigupta thanks for your input. You are correct. That's good news.
At the end, the workaround we used is to explicitly reference<EMAIL_ADDRESS>in our project. That made our vulnerability detection tool happy.
Thanks for you time.

Closing as per https://github.com/dotnet/runtime/issues/68931.


Try cibuildwheels on Windows

Attempt to use cibuildwheels on windows to find out - As usual, I don't know what I am doing

My apologies for this nonsense, it must have been quite obnoxious. I did not realize I could PR against my fork's master and run this ugly fight there. Let me close for now

No worries!


Pubsub: TLS, TCP options and more

This commit makes eredis_sub more like eredis, but reusing much of the
internal code of eredis_client in eredis_sub_client. Some refactoring
of eredis_client is done to allow this.
New options accepted by eredis_sub:start_link/1:

connect_timeout
socket_options
tls
username
database
name (for registered process)

Fixes #47.

Pubsub tests are not added, but much of the connect and reconnect logic is shared with eredis_client so some of it should be covered already.
TODO: Handle send errors and setopts errors are not handled. (Currently, they are matched against ok as in ok = Transport:send(Socket, Data).)

@e-kmisk

One question - if the eredis_sub_client disconnects and "seamlessly" reconnects, from the perspective of the redis server, it's a new client right? So I assume it would need to re-subscribe to the channels it was subscribed to before the disconnection, no?
Is this performed in current implementation? If not, then I don't think the seamless reconnection will work as expected - since the caller would need to re-subscribe explicitly, which it will not do unless it's notified somehow that it should.

Ok, nevermind, I see that this is invoked:
send_to_controller({eredis_disconnected, Self}, State),

So I guess the controller process needs to handle this, and redo the subscriptions? I will inspect our current implementation.

So far it's looking okay, no crashes with the replaced .beam files, and the TCP connections get dropped quickly if a redis server is disconnected forcefully, thanks to the custom TCP options we set now.

Actually, there is another bug: after reconnection, the channels field of the #state{} record inside eredis_sub_client is not purged (remains what it was before disconnect), even though from Redis Server's view the subscriptions are tied to the specific TCP connection.

Re-subscribing seems like the appropriate behaviour. Why would the process keep track of channels otherwise?
Can it be dangerous to fix it? I don't think so. It would affect users who handle resubscriptions manually, but I don't think anything will break if you subscribe to a channel already subscribed.
Let's consider this a separate bug. We can have an issue and tests for it. This PR is already too large enough.

Let's consider this a separate bug. We can have an issue and tests for it. This PR is already too large enough.

Fine by me. One aspect I'm not sure on is whether the process would need to store more info than just the channels' names. I see there are two  [*]SUBCSCRIBE calls, <<"SUBSCRIBE">> but also <<"PSUBSCRIBE">>. So will the process know which one to "redo" upon reconnection? In our case I think we only use regular <<"SUBSCRIBE">> so this is not an issue, but maybe for others it could be?

I don't think anything will break if you subscribe to a channel already subscribed.

Imo. if we are already subscribed to a channel, then the call to subscribe should not be sent to redis but instead OK'd locally. But maybe that's not a big issue, if we know how redis behaves in that case (does it treat a client doing that as a no-op?).

the process would need to store more info than just the channels' names. I see there are two [*]SUBCSCRIBE calls, <<"SUBSCRIBE">> but also <<"PSUBSCRIBE">>. So will the process know which one to "redo" upon reconnection?

Good point. PSUBSCRIBE takes a pattern with wildcards, so it's different to SUBSCRIBE. We need to store that too.

Imo. if we are already subscribed to a channel, then the call to subscribe should not be sent to redis but instead OK'd locally.

Good idea. This should be easy to achieve.

For me, this PR seems okay by the way.
I tried the patched BEAM files and everything worked as expected - the TCP connection gets terminated from "ESTABLISHED" state in 1-2s after severing the connection without hanging up, and eredis reconnects in 2-5s (after DNS record updates).
In my test I re-subscribed explicitly in the controller process by catching the {eredis_connected, Pid} message, but it would be much simpler if we didn't have to do that. Then we would only need to set the new socket options and not change other code.

For me, this PR seems okay by the way.

That's good news. I'm working on some tests, mainly just reusing the same tests as for main eredis module for eredis_sub. Then we can merge this.
Side track issues #50 and #51 will be handled in separate PRs.


Error in hmmSegmentation

I want to segment sound by HMM,  the trainHMM_fromDir dose work, but hmmSegmentation get error followed. I've tried everything but it does not work.
ValueError                                Traceback (most recent call last)
<ipython-input-16-d4ede9b89a3c> in <module>()
      1 from pyAudioAnalysis import audioSegmentation as aS
      2 aS.trainHMM_fromDir('D:/SanSys/Desktop/test1/', 'hmmTemp1', 0.1, 0.1)
----> 3 aS.hmmSegmentation('D:/SanSys/Desktop/test2/NJW1.wav','hmmTemp1',True,'NJW1.segments')

C:\ProgramData\Anaconda2\lib\site-packages\pyAudioAnalysis\audioSegmentation.pyc in hmmSegmentation(wavFileName, hmmModelName, PLOT, gtFileName)
    438     #Features = audioFeatureExtraction.stFeatureExtraction(x, Fs, 0.050*Fs, 0.050*Fs);    # feature extraction
    439     [Features, _] = aF.mtFeatureExtraction(x, Fs, mtWin * Fs, mtStep * Fs, round(Fs * 0.050), round(Fs * 0.050))
--> 440     flagsInd = hmm.predict(Features.T)                            # apply model
    441     #for i in range(len(flagsInd)):
    442     #    if classesAll[flagsInd[i]]=="silence":

C:\ProgramData\Anaconda2\lib\site-packages\hmmlearn\base.pyc in predict(self, X, lengths)
    332             Labels for each sample from ``X``.
    333         """
--> 334         _, state_sequence = self.decode(X, lengths)
    335         return state_sequence
    336

C:\ProgramData\Anaconda2\lib\site-packages\hmmlearn\base.pyc in decode(self, X, lengths, algorithm)
    292         """
    293         check_is_fitted(self, "startprob_")
--> 294         self._check()
    295
    296         algorithm = algorithm or self.algorithm

C:\ProgramData\Anaconda2\lib\site-packages\hmmlearn\hmm.pyc in _check(self)
    173
    174         _validate_covars(self._covars_, self.covariance_type,
--> 175                          self.n_components)
    176
    177     def _init(self, X, lengths=None):

C:\ProgramData\Anaconda2\lib\site-packages\sklearn\mixture\gmm.pyc in _validate_covars(covars, covariance_type, n_components)
    764                              "(n_components, n_dim)")
    765         elif np.any(covars <= 0):
--> 766             raise ValueError("'diag' covars must be non-negative")
    767     elif covariance_type == 'full':
    768         if len(covars.shape) != 3:

ValueError: 'diag' covars must be non-negative

Same problem here, have you got any solution?

I also got the same error. Had anyone resolved it @phliulei @pvanickova  ?

I gave up... @ervaibhavkumar

Ohk. So are you able to find some alternate libraries to it ?


feat: ✨ syn-dropdown opening Chevron

Component
I suggest adding the feature to the following component: syn-dropdown
In the Syn-dropdown we are using a chevron that only shows in one direction. When the Menu is opening the chevron should change its direction, to identify that it is opening.
Acceptance criteria

[ ] All DoD items successfully completed

Is this really necessary? Implementation is quite complicated...

In the HTML default element the arrow doesn't change.
I also looked at other Design Systems, and most also don't change the icon.
-> Won't Do


Update Terminology in "Add Outbound Provisioner" Wizard from "Identity Provider" to "Connection"

Describe the issue:
The "Add Outbound Provisioner" wizard in the application still uses the term "Identity Provider" despite the shift in terminology to "Connections.

How to reproduce:

Create Application OIDC
Click on Provisioning Tab > Outbound Provisioning
Navigate to the "Add Outbound Provisioner" wizard.
Observe that the term "Identity Provider" is used in the wizard.

Expected behavior:
The terminology in the "Add Outbound Provisioner" wizard should be updated to use "Connection" instead of "Identity Provider" to align with the current terminology being used in the application.
Environment information (Please complete the following information; remove  any unnecessary fields) :

Product Version: wso2is-7.0.0-beta4-SNAPSHOT
OS: [e.g., Windows, Linux, Mac]
Database: [e.g., MySQL, H2]
Userstore: [e.g., LDAP, JDBC]

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!


Integrate with iframe

Can you please give a full example of the syntax to integrate with irame?
<%=pdfjs_viewer pdf_url: "/sample.pdf", style: :full %> is too simple for me.

Hi, I'm not sure if it is what are you looking for, but you may try to use the following code on your page:
<iframe src="<%= pdfjs.full_path(file: "/here/is/you/pdf/url.pdf") %>" style="width: 100%; height: 100%">
</iframe>

I reckon that resolved it.
Recommend Close.


AR, MAR, RAR, ERAR Contractions

Can we add the following contractions?
ar, contraction of a re-
mar, contraction of me a re-
rar, contraction of er a re-
erar, contraction of er a re-
These seem to be very common in my Biblia, and I suspect they are common in older writings.

I don't know about this one. We definitely added some contractions already. If they are only common in older writings, then I don't think we should add them because we don't want to promote incorrect writing. Maybe we try to use tekinged.com to help us write Palauan more consistently.
Maybe we can add as 'slang contraction' or something and put a note in the definition that says, "This is a commonly seen but grammatically incorrect construction." That way we help people who will be looking for these words but we attempt to put a top to incorrect usage.

That makes sense. I remember seeing these contractions as a beginner, and when I could not find any definition for them, I was very lost. Agreed, promotion of proper Palauan is a good priority.

I propose to show this contractions in a special section of DOSUUB. we can call it OLD SPELLING or MOST COMMON SPELLING MISTAKES.

I propose to show this contractions in a special section of DOSUUB. we can call it OLD SPELLING or MOST COMMON SPELLING MISTAKES.

That's a good idea. I also think we want them to show up in a search so when someone is reading something and they find one of these, then they can quickly and easily figure out what the writer was trying to say. So let's add them as contractions but have the definition say, "common spelling mistake; misspelling of xxxx".


fix: corrected colorion link in docs

Fixes Issue

Changes proposed

Screenshots

Note to reviewers

@afrinashar do you have any closing issue?

@rupali-codes no, issue resolved

@afrinashar We do not accept PRs without an closing issue (as per our Contributing Guidelines), kindly create an issue and then raise the PR.
Thank You.


ARROW-13332: [C++] TSAN failure in TestAsyncUtil.ReadaheadFailed

Minor cleanup fix from my earlier fix of this test in #10602

@github-actions crossbow submit test-ubuntu-20.04-cpp-thread-sanitizer

Is this ok to merge/@westonpace are you planning on making further changes here?

I'll implement @cyb70289 's suggestion real quick and then merge after CI

@github-actions crossbow submit test-ubuntu-20.04-cpp-thread-sanitizer

CI failures unrelated, merging.


Create DateTimeGenerator and add it to data generator

This PR adds 2 new fields to DataGeneratorSpec

_dateTimeFormatMap
_dateTimeGranularityMap

which allows us to create the DateTimeGenerator  for  DATE_TIME field type columns.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (a200315) 61.58% compared to head (bfbc072) 46.42%.

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #12206       +/-   ##
=============================================
- Coverage     61.58%   46.42%   -15.17%
+ Complexity     1152      942      -210
=============================================
  Files          2414     1809      -605
  Lines        131068    95158    -35910
  Branches      20242    15352     -4890
=============================================
- Hits          80722    44179    -36543
- Misses        44444    47828     +3384
+ Partials       5902     3151     -2751

Flag
Coverage Δ

custom-integration1
?

integration
?

integration1
?

integration2
?

java-11
?

java-21
46.42% <ø> (-15.01%)
:arrow_down:

skip-bytebuffers-false
46.42% <ø> (-15.15%)
:arrow_down:

skip-bytebuffers-true
?

temurin
46.42% <ø> (-15.17%)
:arrow_down:

unittests
46.42% <ø> (-15.16%)
:arrow_down:

unittests1
46.42% <ø> (-0.20%)
:arrow_down:

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


no mo' spaces

We allow spaces in names. Maybe we shouldn't do that, it makes things look super weird.

Comment during planning meeting: we maybe shouldn't allow dots either, since they break lookups. I'm thinking the pattern of acceptable characters is: [\w\d\-_]+

See #535


Observe failed when upgrading to v2.0.0

What happened?
I'm creating kubernetes AuthBackendRole and vault Policy resources with crossplane.
This worked fine with version v1.0.0.
After Upgrading to v2.0.0 both resources can not by synced any more with the following error:
observe failed: failed to observe the resource: [{0 no vault token set on Client  []}]
How can we reproduce it?
Provider Config:
apiVersion: vault.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: vault-provider-config
spec:
  address: https://vault.foo.ch
  credentials:
    secretRef:
      key: credentials
      name: vault-auth
      namespace: crossplane-system
    source: Secret

Policy :
apiVersion: vault.vault.upbound.io/v1alpha1
kind: Policy
metadata:
  name: vault-secret-store-config-cgmjg
spec:
  deletionPolicy: Delete
  forProvider:
    name: team-ns-secret-store
    policy: |
      path "My/Path/*" {
        capabilities = ["read"]
      }
  initProvider: {}
  managementPolicies:
    - '*'
  providerConfigRef:
    name: vault-provider-config

AuthBackendRole
apiVersion: kubernetes.vault.upbound.io/v1alpha1
kind: AuthBackendRole
metadata:
  name: vault-secret-store-config-cgmjg
spec:
  deletionPolicy: Delete
  forProvider:
    aliasNameSource: serviceaccount_uid
    backend: k8s
    boundServiceAccountNames:
      - vault-secret-store
    boundServiceAccountNamespaces:
      - my-ns
    roleName: team-ns-secret-store
    tokenPolicies:
      - team-ns-secret-store
    tokenTtl: 3600
    tokenType: default
  initProvider: {}
  managementPolicies:
    - '*'
  providerConfigRef:
    name: vault-provider-config

What environment did it happen in?
crossplane: v1.18.1
provider vault: v2.0.0

Hi @bakito, could you please provide anonymized content of the credentials or at least the top level keys to determine which authentication method is used?
I am running v2 with authentication secret generated and refresh by External Secrets (ESO uses SA to auth against Vault and periodically update token in the Kubernetes Secret for provider to use)

@m1so I'm using approle login:
{
  "auth_login": {
      "path": "auth/approle/login",
      "parameters": {
          "role_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxx",
          "secret_id": "yyyyyyyyyyyyyyyyyyy"
      }
  }
}

@bakito @Mareo fixed in https://github.com/upbound/provider-vault/pull/50


Test methods for single files

Codecov Report

Merging #235 (fd62b44) into master (4a7205b) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #235   +/-   ##
=======================================
  Coverage   93.31%   93.31%
=======================================
  Files          80       80
  Lines        1959     1959
=======================================
  Hits         1828     1828
  Misses        131      131

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Youtube test

When I click on a video in youtube with the cli, then skip ad does not skip the ad.

Lets make a test that goes to youtube.com and clicks on a video. We don't need to test "skip ad"

Mistakenly tagged this in #64 so reopening

This ticket has been moved to RUN-162


Code block breaks formatting in Markdown preview in Insider build

Issue Type: Bug
This issue occurs only on the insiders build. Version 1.25.0 works as expected.
Steps to reproduce: Create a new file, set the language to Markdown and insert the following snippet:
Normal text

```text
Code block
```

Everything is broken

Now preview the file with Cmd-K V or Ctrl-K V. The text after the code block is not styled correctly. On the dark theme it's displayed in a yellow monospace font as seen in screenshot.
Insider version:

Normal version:

VS Code version: Code - Insiders 1.25.0-insider (28d5806f24f99c2c6ad633e4e08ed5ea00169c83, 2018-06-11T05:11:22.150Z)
OS version: Darwin x64 17.5.0

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz (8 x 2200)

GPU Status
2d_canvas: enabledchecker_imaging: disabled_offflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onnative_gpu_memory_buffers: enabledrasterization: enabledvideo_decode: enabledvideo_encode: enabledwebgl: enabledwebgl2: enabled

Load (avg)
2, 2, 2

Memory (System)
16.00GB (0.11GB free)

Process Argv
/private/var/folders/9_/jkrpz3px2658swfrw0qg9zwr0000gp/T/AppTranslocation/4D8D7D57-5580-426F-ADA7-140AEEB99958/d/Visual Studio Code - Insiders.app/Contents/MacOS/Electron

Screen Reader
no

VM
0%

Extensions: none

I realized that I don't have the newest version of insider, it seems to work properly on the 5a4eab812bd485942061f71af39f3eabc52a1035 commit on Windows. I'll test tomorrow on OS X too.

Works fine on the newest insider build on OS X too.


Re-design Logger hierarchy

The Logger interface is a bit of a mess, doing a bunch of different things some Mill-internal and some user-facing. https://github.com/com-lihaoyi/mill/pull/3577 extends it a bit but makes it even more messy.
Probably we should have more narrow specialized Logger interfaces for different parts of the codebase, e.g. the logger inside a Task would have a ticker, whereas the logger outside would not, and a top-level logger would have more power to do things like configuring the prompt that task-specific loggers would not have. close() probably needs to be re-thought since the lifecycle of the various loggers is all different, with some short-lived and some long-live
The whole color scheme and debug/info/warning/error conventions also need to be redone. Currently infoColor is blue and normally used for "less important" things, but then logger.info uses it for normal-important things. And how should a user decide between info and just println?
Arguably, all the concrete FooLogger classes should be private[mill], since there's no reason that a Mill user would need to instantiate their own PrintLogger or whatever.
This would break bincompat and would have to go into 0.13.0

At time of writing this is the current logging hierarchy that is typically available inside any Mill task:
MultiLogger(
    PrefixLogger(
        PrefixLogger(
            PromptLogger("foo.test"),
            "",
            ""
        ),
        "[106] ",
        ""
    ),
    FileLogger(/Users/lihaoyi/Github/mill/example/scalalib/testing/4-test-sharding/out/foo/test/test.log)
)


This site needs to be supported

https://www.tipeeestream.com/
It's a donation site most of the german minecraft streamers use when streaming. I use this site as well and I'd like to include Pay2Spawn in my modpack so my viewers can contribute. Would love to see this site being supported in this mod.

API documentation: https://api.tipeeestream.com/api-doc/
This will be needed if its posible to add. It will be looked into!

It should be noted, as an aside to the above, that there are even streaming services listed in the cfg now that are defunct or nonexistent such as supportthestream. The entire list needs a once-over.

If / when we update, new sites will be added and old ones will be removed.
This site may still be added if I need to work on p2s in another capacity.

http://www.donationalerts.ru
Can you add this one? Most popular donation system in Russian Federation. Thx

I can't speak russian, and google translate is not nearly enough to develop something like this. If they have English support or information maybe.


Please invite me to the GitHub Community Organization

Name
valloon91234
Discord Username (if applicable)
No response
Additional Context
I hope to contribute to great projects.

I believe this issue is already addressed in #Issue-254: link to original issue. So this Issue close  as a duplicate .


Docsy template: poor appearance with narrow viewports

This is a Bug Report against the draft Docsy style

Problem:
With the draft Docsy template and a narrow-enough viewport, /docs/home/ looks wrong:

The appearance is as if the main page header has too little margin-bottom or padding-bottom.
Proposed Solution:
Revise Docsy template styling.
For a narrow viewport, and specifically if the page heading is the same text as the section heading, I'd consider relying on the section headings (top nav) only.
Here's a mockup of how this might look:

Page to Update:
https://kubernetes-hugo-staging.netlify.com/docs/home/ (preview URL)

/remove-lifecycle stale

Fixed, enough to close this.
/close


Install gcc for dlrn reporting

As a pull request owner and reviewers, I checked that:

[x] Appropriate testing is done and actually running

/approve


Hallowed Sepulchre statue fires and ground teleporters not glowing

I dont know if intentional, but you cant really see the ground teleporters when they light up

This can now be closed as it was fixed


too slow

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Hi Steven,
Which Host OS are you using, and what are your CPU and RAM specs? Which operation is slow specifically?