PluginSearch Error

+1

The same issue while trying to search. Can't even catch text log - it disappears and empty search window showed:

I came to the solution, simply remove the .vundle folder from your .vim directory and use :PluginInstall! to update/reinstall the vundle plugin. Then use :VundleSearch or :PluginSearch again and you shall see the result come out smoothly.
thks for this helpful issue posted by @phillipwei , related issues are #535 and #602

update@20150924:
I dived into the vim script and found that the reason is simple, because the command :PluginSearch did just about one thing, use curl to download the json file representing the vim plugins in vim-scripts.org, and so the solution is very simple. Just use curl to download the file from vim-scripts.org to the folder ~/.vim/bunlde/.vundle/ and save as script-names.vim-scripts.org.json.
The main logic is as follows(located in ~/.vim/bundle/Vundle.vim/autoload/vundle/scripts.vim:

So the solution is simple:

just remove the file ~/.vim/bundle/.vundle/script-names.vim-scripts.org.json and try :PluginSearch again.

or use curl or wget to get the file(either is ok, pick one as you like), like:

curl -v -o ~/.vim/bundle/.vundle/script-names.vim-scripts.org.json http://vim-scripts.org/api/scripts.json

or

wget http://vim-scripts.org/api/scripts.json -v -O ~/.vim/bundle/.vundle/script-names.vim-scripts.org.json

I followed these steps to fix the scripts not found error...and found that the domain vim-scripts.org has expired. Is the file available anywhere else?

it works for me: i can access the website and the json file.
What am I doing wrong?

The domain is back up now. I haven't tested :PluginSearch though

Just tried it. I was able to download the json file using wget as described above. :PluginSearch! works for me now.

@nfischer @fourlincoln10 thanks for the feedback!

Strangely this curl command that was listed above:
curl -v -o ~/.vim/bundle/.vundle/script-names.vim-scripts.org.json http://vim-scripts.org/api/scripts.json

downloaded a 0-byte file, which caused PluginSearch to continue to fail. I had to manually open a browser to http://vim-scripts.org/api/scripts.json and save the content to ~/.vim/bundle/.vundle/script-names.vim-scripts.org.json
But now it seems to be working.

I tried every solution that is given above but I am still facing the same issue any help?

I tried every solution that is given above but I am still facing the same issue any help?

I'm also facing this issue, I wonder if you have any update??

@yungHa It looks like the site that used to serve that JSON file is no longer active. However, it looks like the JSON is simply a list of package names. I am looking into this :coffee:

The root of this problem is that PluginSearch queries http://vim-scripts.org, and this site no longer exists. This is because most vim plugin development and distribution occurs via github/gitlab now.
I recommend searching for plugins directly, or searching for plugins on github. This functionality should be considered dead now.
or use curl or wget to get the file(either is ok, pick one as you like), like:
curl -v -o ~/.vim/bundle/.vundle/script-names.vim-scripts.org.json https://raw.githubusercontent.com/i-cooltea/resource/master/vim-api_scripts.json
or
wget https://raw.githubusercontent.com/i-cooltea/resource/master/vim-api_scripts.json -v -O ~/.vim/bundle/.vundle/script-names.vim-scripts.org.json

there is no .vundle in ~/.vim/bundle instead of Vundle.vim.I change dir to ~/.vim/bundle/Vundle.vim/autoload,there is there files in it.then use "wget http://vim-scripts.org/api/scripts.json" to download this json file .it works.no errors report anymore.

changing http://vim-scripts.org/api/scripts.json to https://raw.githubusercontent.com/i-cooltea/resource/master/vim-api_scripts.json In .vim/bundle/Vundle.vim/autoload/vundle/scripts.vim fixed problem for me.

@i-cooltea ..all evening trying the different fixes for this issue and none worked but i-cooltea has it spot on. The vim-scripts site no longer exists.
I deleted my current "script-names.vim-scripts.org.json" from the .vundle folder and ran:
curl -v -o ~/.vim/bundle/.vundle/script-names.vim-scripts.org.json https://raw.githubusercontent.com/i-cooltea/resource/master/vim-api_scripts.json
PluginSearch now works a charm.
Thank you I-cooltea.

worked for me. thanks

@i-cooltea ..all evening trying the different fixes for this issue and none worked but i-cooltea has it spot on. The vim-scripts site no longer exists.
I deleted my current "script-names.vim-scripts.org.json" from the .vundle folder and ran:
curl -v -o ~/.vim/bundle/.vundle/script-names.vim-scripts.org.json https://raw.githubusercontent.com/i-cooltea/resource/master/vim-api_scripts.json
PluginSearch now works a charm.
Thank you I-cooltea.

worked for me. thanks

Gracie!

The root of this problem is that PluginSearch queries http://vim-scripts.org, and this site no longer exists. This is because most vim plugin development and distribution occurs via github/gitlab now.
I recommend searching for plugins directly, or searching for plugins on github. This functionality should be considered dead now.
or use curl or wget to get the file(either is ok, pick one as you like), like:
curl -v -o ~/.vim/bundle/.vundle/script-names.vim-scripts.org.json https://raw.githubusercontent.com/i-cooltea/resource/master/vim-api_scripts.json
or
wget https://raw.githubusercontent.com/i-cooltea/resource/master/vim-api_scripts.json -v -O ~/.vim/bundle/.vundle/script-names.vim-scripts.org.json

after a lot of google search , this particular worked. I'm happy it worked but can anybody explain me how?

Probably the URL in the code here needs to be replaced with another proper one: https://github.com/VundleVim/Vundle.vim/blob/master/autoload/vundle/scripts.vim#L237

The root of this problem is that PluginSearch queries http://vim-scripts.org, and this site no longer exists. This is because most vim plugin development and distribution occurs via github/gitlab now.
I recommend searching for plugins directly, or searching for plugins on github. This functionality should be considered dead now.
or use curl or wget to get the file(either is ok, pick one as you like), like:
curl -v -o ~/.vim/bundle/.vundle/script-names.vim-scripts.org.json https://raw.githubusercontent.com/i-cooltea/resource/master/vim-api_scripts.json
or
wget https://raw.githubusercontent.com/i-cooltea/resource/master/vim-api_scripts.json -v -O ~/.vim/bundle/.vundle/script-names.vim-scripts.org.json

after a lot of google search , this particular worked. I'm happy it worked but can anybody explain me how?

Well the original URL is not available anymore. The code here needs to be fixed to point to a new properly downloadable resource: https://github.com/VundleVim/Vundle.vim/blob/master/autoload/vundle/scripts.vim#L237

The root of this problem is that PluginSearch queries http://vim-scripts.org, and this site no longer exists. This is because most vim plugin development and distribution occurs via github/gitlab now.
I recommend searching for plugins directly, or searching for plugins on github. This functionality should be considered dead now.
or use curl or wget to get the file(either is ok, pick one as you like), like:
curl -v -o ~/.vim/bundle/.vundle/script-names.vim-scripts.org.json https://raw.githubusercontent.com/i-cooltea/resource/master/vim-api_scripts.json
or
wget https://raw.githubusercontent.com/i-cooltea/resource/master/vim-api_scripts.json -v -O ~/.vim/bundle/.vundle/script-names.vim-scripts.org.json

after a lot of google search , this particular worked. I'm happy it worked but can anybody explain me how?

Well the original URL is not available anymore. The code here needs to be fixed to point to a new properly downloadable resource: https://github.com/VundleVim/Vundle.vim/blob/master/autoload/vundle/scripts.vim#L237

Such behaviour creates a lot of confusion for such new users like me. Why this is not resolved yet with so many ( working ) proposed solutions/workarounds?

Such behaviour creates a lot of confusion for such new users like me. Why this is not resolved yet with so many ( working ) proposed solutions/workarounds?

I think the VundleVIM is not actively maintained any more.  and the 'http://vim-scripts.org/api/scripts.json' in the search script doesn't exist any more. maybe someone can pick it up and start to maintain it.


Win 10 requirements

hi there
i have a windows 10 running and your handy app it just don't wanna start on it
This app can't run on your PC
what are the requirements for it to work pls?

It needs a 64-bit computer and Windows 1809 or higher. Your error is probably because your computer is 32-bit.

Closed this issue due to this more than likely being an incompatibility with an EOL version of Windows or a 32-bit version of Windows which this program is not designed to run with.


labels module improvements

Describe your changes
Extension of work in https://github.com/TomographicImaging/CIL/pull/1692

Rewrite of labels.py to use enums.
Broken down in to smaller groups of labels
Updated code to use new labels - however still stored the enum value not the enum member for many cases to keep this update manageable
Updated tests
Also removed the new BaseAcquisitionGeometry Class by adding child processes for reorder to ImageData and AcquisitionData

ToDo:

Decide on labels names
Add unit tests for labels.py methods and functionality
Add specific tests for enums vs. strings in user API
Replace more of the strings throughout the code (in a follow up PR)

Describe any testing you have performed
Please add any demo scripts to CIL-Demos/misc/
Link relevant issues
Checklist when you are ready to request a review

[x] I have performed a self-review of my code
[x] I have added docstrings in line with the guidance in the developer guide
[x] I have implemented unit tests that cover any new or modified functionality
[x] CHANGELOG.md has been updated with any functionality change
[x] Request review from all relevant developers
[x] Change pull request label to 'Waiting for review'

Contribution Notes
Please read and adhere to the developer guide and local patterns and conventions.

[x] The content of this Pull Request (the Contribution) is intentionally submitted for inclusion in CIL (the Work) under the terms and conditions of the Apache-2.0 License.
[x] I confirm that the contribution does not violate any intellectual property rights of third parties

If you think it's very way, then let's do it! Happy for merge.
J
Sent from Outlook for Androidhttps://aka.ms/AAb9ysg

From: Gemma Fardell @.>
Sent: Thursday, August 15, 2024 6:12:50 pm
To: manchester-jhellier/CIL @.>
Cc: Joshua Hellier @.>; Mention @.>
Subject: Re: [manchester-jhellier/CIL] labels module improvements (PR #1)
@manchester-jhellier [github.com]https://urldefense.com/v3/__https://github.com/manchester-jhellier__;!!PDiH4ENfjr2_Jw!CzAkj2fGeF30aQuLrufh9irTvJcbHTh2v4GIIRaLgCtjCKi7SpYAjAhI4O63ZOxlmzxokrMVwc5E5aIrbRTWKcuU9_WZolJFzJJFb94$ if you're happy with this could you merge it in to your PR. It passes all our CI tests.
—
Reply to this email directly, view it on GitHub [github.com]https://urldefense.com/v3/__https://github.com/manchester-jhellier/CIL/pull/1*issuecomment-2291744748__;Iw!!PDiH4ENfjr2_Jw!CzAkj2fGeF30aQuLrufh9irTvJcbHTh2v4GIIRaLgCtjCKi7SpYAjAhI4O63ZOxlmzxokrMVwc5E5aIrbRTWKcuU9_WZolJF6-oTYy0$, or unsubscribe [github.com]https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/BFGMOYX4MPEUB3G2S3EZ5X3ZRTOQRAVCNFSM6AAAAABMQ3OGB6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJRG42DINZUHA__;!!PDiH4ENfjr2_Jw!CzAkj2fGeF30aQuLrufh9irTvJcbHTh2v4GIIRaLgCtjCKi7SpYAjAhI4O63ZOxlmzxokrMVwc5E5aIrbRTWKcuU9_WZolJFFz8pOzU$.
You are receiving this because you were mentioned.Message ID: @.***>

Hi I am currently reviewing so please do not merge yet

There are a lot of suggestions covering the code in acquisition geometry. I think as this was supposed to be refactoring as priority rather than style/linting it's better leaving these for a follow-up PR on the CIL side.


Add browser e2e test for client capabilities

Adds sample and e2e to acquire a token with client capabilities and pop.

Note, the cae pop token acquired doesnt appear to work for the /me endpoint, which is fine. The point of this sample is to show how to acquire it, and test MSAL can get both bearer and pop cae tokens.


Add support to MPS

I worked on adding support to MPS. During this work, I also did a bit of refactoring.
Namely the changes are:

Adding more modularity by adding new types of models
Adding LightningWhisperMLX since it works considerably better on MPS
Adding MeloTTS to have a more lightweight solution for TTS, more adequate for macbooks
Add a "Local audio streamer", to avoid messing with connections to a localhost, which proved difficult on my macbook.
Bypass bugs that I found while testing with mac (mainly having to generate all the text from the LLM and then synchronize the memory before generating with the TTS)

I also added support to pass mps as a device, see https://github.com/huggingface/speech-to-speech/issues/18

Also mentioned here: https://github.com/huggingface/speech-to-speech/issues/9

Just tested and having the same issues as with here
Are there any additional installations required?
What command do you recommend for running? I'm using:
git clone https://github.com/eustlb/speech-to-speech.git
cd speech-to-speech
pip install -r requirements.txt
pip install<EMAIL_ADDRESS>python s2s_pipeline.py --recv_host localhost --send_host localhost --lm_model_name HuggingFaceTB/SmolLM-360M-Instruct --lm_device mps --stt_device mps --tts_device mps

Many thanks.

First, run it with:
python s2s_pipeline.py --mode local --device mps
Second, you need to install MeloTTS. For that you can add this to your script.
git clone https://github.com/myshell-ai/MeloTTS.git
cd MeloTTS
pip install -e .
python -m unidic download

Let me know if that works!

Looking again at your code, to test this code you need to checkout the branch as well!
git checkout add-mps-support

Many thanks @andimarafioti ,making good progress now with this:
git clone https://github.com/eustlb/speech-to-speech.git
cd speech-to-speech
git fetch --all
git checkout add-mps-support
python -m venv mpsEnv
source mpsEnv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
pip install --upgrade transformers
pip install<EMAIL_ADDRESS>git clone https://github.com/myshell-ai/MeloTTS.git
cd MeloTTS
pip install -e .
python -m unidic download
cd ../
python s2s_pipeline.py --mode local --device mps --lm_model_name HuggingFaceTB/SmolLM-360M-Instruct

takes quite a while at this point:
python s2s_pipeline.py --mode local --device mps
/Users/ronanmcgovern/TR/mpsEnv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
  warnings.warn(
2024-08-19 17:48:47,969 - parler_tts.modeling_parler_tts - WARNING - Flash attention 2 is not installed
Using cache found in /Users/ronanmcgovern/.cache/torch/hub/snakers4_silero-vad_master
python(10073) MallocStackLogging: can't turn off malloc stack logging because it was not enabled.
Fetching 13 files: 100%|███████████████| 13/13 [00:00<00:00, 35614.60it/s]

What are the default models btw, and what are your recommendations for reducing RAM requirements.
I see distil-whisper is there in large v3 form, which is 1.5 GB - not too bad. I reckon whisper small is good enough (I use it for youtube transcriptions). And I see you are defaulting to smollm and parler jenny so those are small already.
Nice work, thanks

What is the recommended python version?

I tried this but so far wasnt successful:
> git clone https://github.com/eustlb/speech-to-speech.git
> cd speech-to-speech
> git fetch --all
> git checkout add-mps-support
> python -m venv mpsEnv
> source mpsEnv/bin/activate
> pip install --upgrade pip
> pip install -r requirements.txt
> pip install --upgrade transformers
> pip install<EMAIL_ADDRESS>> git clone https://github.com/myshell-ai/MeloTTS.git
> cd MeloTTS
> pip install -e .
> python -m unidic download
> cd ../
> python s2s_pipeline.py --mode local --device mps --lm_model_name HuggingFaceTB/SmolLM-360M-Instruct
>

Initially i missed both a Rust compiler and mecap. I installed mecab:
brew install mecab
brew install mecab-ipadic

and Rust compiler using:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
. "$HOME/.cargo/env"

But tiktoken still wont compile. Rust compiler is giving out an error about wrong reference to mutable variable (see full ouput below):
error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`

I guess I could run the compiler with allowing invalid_reference_casting but TBH don't know enough about Rust to feel comfortable doing that. Are there any precompiled packages for tiktoken or does someone else know a better solution.
Full error message:
      warning: variable does not need to be mutable
         --> tokenizers-lib/src/models/unigram/model.rs:265:21
          |
      265 |                 let mut target_node = &mut best_path_ends_at[key_pos];
          |                     ----^^^^^^^^^^^
          |                     |
          |                     help: remove this `mut`
          |
          = note: `#[warn(unused_mut)]` on by default

      warning: variable does not need to be mutable
         --> tokenizers-lib/src/models/unigram/model.rs:282:21
          |
      282 |                 let mut target_node = &mut best_path_ends_at[starts_at + mblen];
          |                     ----^^^^^^^^^^^
          |                     |
          |                     help: remove this `mut`

      warning: variable does not need to be mutable
         --> tokenizers-lib/src/pre_tokenizers/byte_level.rs:200:59
          |
      200 |     encoding.process_tokens_with_offsets_mut(|(i, (token, mut offsets))| {
          |                                                           ----^^^^^^^
          |                                                           |
          |                                                           help: remove this `mut`

      error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`
         --> tokenizers-lib/src/models/bpe/trainer.rs:526:47
          |
      522 |                     let w = &words[*i] as *const _ as *mut _;
          |                             -------------------------------- casting happend here
      ...
      526 |                         let word: &mut Word = &mut (*w);
          |                                               ^^^^^^^^^
          |
          = note: for more information, visit <https://doc.rust-lang.org/book/ch15-05-interior-mutability.html>
          = note: `#[deny(invalid_reference_casting)]` on by default

      warning: `tokenizers` (lib) generated 3 warnings
      error: could not compile `tokenizers` (lib) due to 1 previous error; 3 warnings emitted

Yeah, there have been updates it seems since yesterday and issues now with
tokenizers. I'm seeing the same.
On Tue, Aug 20, 2024 at 10:59 AM David Finsterwalder <
@.***> wrote:

I tried this but so far wasnt successful:

git clone https://github.com/eustlb/speech-to-speech.git
cd speech-to-speech
git fetch --all
git checkout add-mps-support
python -m venv mpsEnv
source mpsEnv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
pip install --upgrade transformers
pip install @.***
git clone https://github.com/myshell-ai/MeloTTS.git
cd MeloTTS
pip install -e .
python -m unidic download
cd ../
python s2s_pipeline.py --mode local --device mps --lm_model_name HuggingFaceTB/SmolLM-360M-Instruct

Initially i missed both a Rust compiler and mecap. I installed mecab:
brew install mecab
brew install mecab-ipadic
and Rust compiler using:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
. "$HOME/.cargo/env"
But tiktoken still wont compile. Rust compiler is giving out an error
about wrong reference to mutable variable (see full ouput below):
error: casting &T to &mut T is undefined behavior, even if the reference is unused, consider instead using an UnsafeCell
I guess I could run the compiler with allowing invalid_reference_casting
but TBH don't know enough about Rust to feel comfortable doing that. Are
there any precompiled packages for tiktoken or does someone else know a
better solution.
Full error message:
  warning: variable does not need to be mutable
     --> tokenizers-lib/src/models/unigram/model.rs:265:21
      |
  265 |                 let mut target_node = &mut best_path_ends_at[key_pos];
      |                     ----^^^^^^^^^^^
      |                     |
      |                     help: remove this `mut`
      |
      = note: `#[warn(unused_mut)]` on by default

  warning: variable does not need to be mutable
     --> tokenizers-lib/src/models/unigram/model.rs:282:21
      |
  282 |                 let mut target_node = &mut best_path_ends_at[starts_at + mblen];
      |                     ----^^^^^^^^^^^
      |                     |
      |                     help: remove this `mut`

  warning: variable does not need to be mutable
     --> tokenizers-lib/src/pre_tokenizers/byte_level.rs:200:59
      |
  200 |     encoding.process_tokens_with_offsets_mut(|(i, (token, mut offsets))| {
      |                                                           ----^^^^^^^
      |                                                           |
      |                                                           help: remove this `mut`

  error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`
     --> tokenizers-lib/src/models/bpe/trainer.rs:526:47
      |
  522 |                     let w = &words[*i] as *const _ as *mut _;
      |                             -------------------------------- casting happend here
  ...
  526 |                         let word: &mut Word = &mut (*w);
      |                                               ^^^^^^^^^
      |
      = note: for more information, visit <https://doc.rust-lang.org/book/ch15-05-interior-mutability.html>
      = note: `#[deny(invalid_reference_casting)]` on by default

  warning: `tokenizers` (lib) generated 3 warnings
  error: could not compile `tokenizers` (lib) due to 1 previous error; 3 warnings emitted

—
Reply to this email directly, view it on GitHub
https://github.com/huggingface/speech-to-speech/pull/20#issuecomment-2298465491,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ASVG6CXO26X4DIEVR5OFFC3ZSMHR7AVCNFSM6AAAAABMXZZL26VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJYGQ3DKNBZGE
.
You are receiving this because you commented.Message ID:
@.***>

Let me check this quickly

I tried the bash script you uploaded and it also doesn't work for me but with another error (pip can't install MuCab, no clue why). I'm using uv (https://github.com/astral-sh/uv/) mainly because it's super fast, but in this case it does solve the dependency better. Another thing I noticed in your bash script that's different from mine is that I reinstall the latest transformers and tokenizers at the end, because melotts downgrades them. There is a more elegant way to do this and I'll work on it to close this PR, but I'm currently finishing the tech report for Idefics3 😅 The final thing that I broke with my last commit is that I  made parlerTTS the default tts and it uses a global variable that's declared on the whisper model. That's pretty hacky, I'll fix it as well.
The bash script that worked for me is:
git clone https://github.com/eustlb/speech-to-speech.git
cd speech-to-speech
git fetch --all
git checkout add-mps-support
uv venv --python 3.11
source .venv/bin/activate
uv pip install -r requirements.txt
cd MeloTTS
uv pip install -e .
python -m unidic download
cd ..
uv pip install -r requirements.txt  # melotts downgrades transformers...
python s2s_pipeline.py --mode local --device mps --lm_model_name HuggingFaceTB/SmolLM-360M-Instruct  --tts melo

Please let me know if this works for you as well!

I just noticed this might still break for you if you are missing some nltk files on your machine. That one is hard to test, but if you run this in docker it will probably happen.

I'm also working on it on #26
My current error after after some changes (some were mentioned here) is:
ImportError: cannot import name 'TextIteratorStreamer' from 'transformers' (/Users/eden.yavin/Projects/speech-to-speech/venv/lib/python3.9/site-packages/transformers/__init__.py)

Yes, you need t update transformers again 😅  because melotts downgrades it. You can re-do uv pip install -r requirements.txt after installing melotts

I tried to follow the examples in this thread and have gotten s2s to work.  I'm getting a lot of slow downs and stuttering when I use Parler.  I'm on an M1 Max 64Gb.  Is this to be expected or should it perform better than this?

@scalar27
Try using Melo : --tts melo. It made the audio much better when I switched to it

@scalar27
Try using Melo : --tts melo. It made the audio much better when I switched to it

Yes, that is much faster but I specifically would like to try to use Parler if it's reasonably responsive.

Hi! To make Parler work, you can increase the "play_steps_s" parameter we pass to it. Then, your chunks are larger, so naturally, the audio is less chunky. With my setup (M3 Pro, 36 GB), I couldn't get Parler to be above real-time, so it always chops up since it needs time for the next chunk to be ready after playing the last one. I'm trying to convince the Parler team to train a smaller model for this precise use case. Fingers crossed!

Here are install commands that work for me:

When I run those, the TTS seems to default to parler. I changed the default from parler to melo in s2s_pipeline.py and then get:
python s2s_pipeline.py --mode local --device mps --lm_model_name HuggingFaceTB/SmolLM-360M-Instruct  --tts melo
[nltk_data] Downloading package punkt_tab to
[nltk_data]     /Users/ronanmcgovern/nltk_data...
[nltk_data]   Package punkt_tab is already up-to-date!
Using cache found in /Users/ronanmcgovern/.cache/torch/hub/snakers4_silero-vad_master
Fetching 13 files: 100%|█████████████████████| 13/13 [00:00<00:00, 25491.33it/s]

at which point I make no further progress.

@andimarafioti thanks for your work on this. When you've a chance, could you ensure that passing --tts melo overrides the default?
Any ideas on why things are freezing above? Do you know the total VRAM required? I'm guessing it's about 1.5 for whisper + 0.700 for Smol + 0.400 for Melo. So maybe 3.6 GB? Or more if any of this is in 32 bits... probably does get challenging for a Mac M1 with 8 GB RAM.

@RonanKMcGovern
When I passed the --melo TTS I noticed a big difference in the speech performance so I assume it works and override the default.
The thing I did find is that passing the -lm_mode_name to SMOLM-360M did not override the Phi3 default which is drastically bigger.

I pushed a change to let you use the SMOLLM, I had MLX-LM by default which is optimized for mac, but doesn't implement the SMOLLM. For the melo tts, it should work if you pass it :S The hangup you are experiencing sounds to me like it's taking too long to load, it might be because of some off loading of models on initialization. I can recommend you to install asitop to monitor what's going on under the hood: https://github.com/tlkh/asitop

I fixed the install 🥳 Now it should work with uv pip install -r requirements.txt. That will install Melo from my fork at: https://github.com/andimarafioti/MeloTTS
There is after the step to do: python -m unidic download if you want to use Melo, I didn't add that to the install because it's a lot of overhead for people that don't want melo, but I added a warning if you get the error you should get without it. I also fixed a bunch of things that were open. I'm going to close this PR with this to focus on other improvements I want to do to the codebase, please do share if any issues arise and I'll address them promptly.

Hi! To make Parler work, you can increase the "play_steps_s" parameter we pass to it. Then, your chunks are larger, so naturally, the audio is less chunky. With my setup (M3 Pro, 36 GB), I couldn't get Parler to be above real-time, so it always chops up since it needs time for the next chunk to be ready after playing the last one. I'm trying to convince the Parler team to train a smaller model for this precise use case. Fingers crossed!

Are we already using Parler mini?  https://huggingface.co/parler-tts/parler-tts-mini-v1

Yes, the default is "ylacombe/parler-tts-mini-jenny-30H"

@andimarafioti
The installation with uv seems to work well.
But, after typing the command to run it I'm getting an error with melo
Traceback (most recent call last):
  File "/Users/eden.yavin/Projects/speech-to-speech/s2s_pipeline.py", line 1143, in <module>
    main()
  File "/Users/eden.yavin/Projects/speech-to-speech/s2s_pipeline.py", line 1119, in main
    from TTS.melotts import MeloTTSHandler
  File "/Users/eden.yavin/Projects/speech-to-speech/TTS/melotts.py", line 1, in <module>
    from melo.api import TTS
  File "/Users/eden.yavin/Projects/speech-to-speech/venv/lib/python3.9/site-packages/melo/api.py", line 18, in <module>
    from .download_utils import load_or_download_config, load_or_download_model
  File "/Users/eden.yavin/Projects/speech-to-speech/venv/lib/python3.9/site-packages/melo/download_utils.py", line 4, in <module>
    from cached_path import cached_path
  File "/Users/eden.yavin/Projects/speech-to-speech/venv/lib/python3.9/site-packages/cached_path/__init__.py", line 13, in <module>
    from ._cached_path import cached_path
  File "/Users/eden.yavin/Projects/speech-to-speech/venv/lib/python3.9/site-packages/cached_path/_cached_path.py", line 15, in <module>
    from .schemes import (
  File "/Users/eden.yavin/Projects/speech-to-speech/venv/lib/python3.9/site-packages/cached_path/schemes/__init__.py", line 6, in <module>
    from .r2 import R2Client
  File "/Users/eden.yavin/Projects/speech-to-speech/venv/lib/python3.9/site-packages/cached_path/schemes/r2.py", line 18, in <module>
    class R2Client(SchemeClient):
  File "/Users/eden.yavin/Projects/speech-to-speech/venv/lib/python3.9/site-packages/cached_path/schemes/r2.py", line 20, in R2Client
    botocore.exceptions.HTTPClientError,
AttributeError: module 'botocore.exceptions' has no attribute 'HTTPClientError'


SSE 4.2 implementation of v_masked_store() contains AVX code

Hi Daniel,
recently an issue was opened in nmoehrle/mvs-texturing/issues/100 where a compile error was reported in mapmap. It turned out that mapmap's SSE 4.2 implementation of v_masked_store() includes a call to _mm_maskstore_ps() which is an AVX instruction.
Regards,
Andre

Kudos to @magcks for his PR!


Blank Screenshot !!!

Whenever I do a screenshot, it saves a black image with the cursor only. I am currently running KShare on  Ubuntu 17.10

This is most likely caused by Wayland. There is (and will be) no Wayland support in the near future. These are some ways to confirm Wayland is running
echo $XDG_SESSION_TYPE
ps aux | grep wayland # some stuff like GNOME have --wayland flags
loginctl show-session $(loginctl | grep $(whoami) | awk '{print $1}') -p Type # shows session type, possibly most reliable for systemd systems


update-spec broken

CloudFormation Lint Version
0.72.8
What operating system are you using?
Macos 13.1
Describe the bug
When I tried to run cfn-lint --update-specs I got the error KeyError: 'AWS::WAFv2::RuleGroup.Allow' continuously.  Now when ever I run cfn-lint anything I get this:
❰lbohm❙~/Downloads❱✘≻ cfn-lint -v
Traceback (most recent call last):
File "/usr/local/bin/cfn-lint", line 5, in
from cfnlint.main import main
File "/usr/local/lib/python3.10/site-packages/cfnlint/init.py", line 7, in
import cfnlint.conditions
File "/usr/local/lib/python3.10/site-packages/cfnlint/conditions.py", line 10, in
import cfnlint.helpers
File "/usr/local/lib/python3.10/site-packages/cfnlint/helpers.py", line 464, in
initialize_specs()
File "/usr/local/lib/python3.10/site-packages/cfnlint/helpers.py", line 461, in initialize_specs
RESOURCE_SPECS[region] = load_region(region)
File "/usr/local/lib/python3.10/site-packages/cfnlint/helpers.py", line 455, in load_region
spec[section][key] = RESOURCE_SPECS["us-east-1"][section][key]
KeyError: 'AWS::WAFv2::RuleGroup.Allow'
Expected behavior
show me the version
Reproduction template
Do not have a template to give you.

I'll be taking a look into this shortly.

I ended up removing via brew remove and then installing via pip install and was able to install 0.73.1 just fine.  I have not tried to update the specs and will avoid that if I can.
Louis

On Feb 14, 2023, at 12:04, Kevin DeJong @.***> wrote:
I'll be taking a look into this shortly.
—
Reply to this email directly, view it on GitHub https://github.com/aws-cloudformation/cfn-lint/issues/2588#issuecomment-1430083220, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGOYCQT6K3WU4JS4SLYXCDWXO3K5ANCNFSM6AAAAAAU3ZW3JA.
You are receiving this because you authored the thread.

I'll still look into it. Sometimes the specs we get aren't valid and need to be fixed so I want to track down the issue to make sure everything is okay.

Going to close this one.  I'm guessing there are chances where something could get in a bad state.  We have some improvements in v1 that will fix some of this.


Bug fix when saving collection tree for first time in split repos

Fixes https://github.com/statamic/eloquent-driver/issues/198
When saving a collection for the first time in split config mode (file for collection config, eloquent for data), it will return a stache tree as the collection structure is designed to do this.
The work around is to not assume we get passed an eloquent collection tree.

Closing in favour of https://github.com/statamic/eloquent-driver/pull/203


[storage][datalake] uploadFile does not abort when timeout

https://github.com/Azure/azure-sdk-for-js/blob/752628be7932cd03833c66b9697d12e5430fa3ec/sdk/storage/storage-file-datalake/test/node/highlevel.node.spec.ts#L488
I expected the timeout aborter could abort the uploadFile operation. But it doesn't. The 4GB upload hasn't done yet after 6 minutes. We passed the abortSignal to underlying requests so what's going wrong? @XiaoningLiu @jeremymeng
https://github.com/Azure/azure-sdk-for-js/blob/752628be7932cd03833c66b9697d12e5430fa3ec/sdk/storage/storage-file-datalake/src/clients.ts#L1483
  it("uploadFile with chunkSize = FILE_UPLOAD_MAX_CHUNK_SIZE should succeed", async () => {
    recorder.skip("node", "Temp file - recorder doesn't support saving the file");
    const fileSize = FILE_UPLOAD_MAX_CHUNK_SIZE * 2 + MB;
    const tempFile = await createRandomLocalFile(tempFolderPath, fileSize / MB, MB);
    try {
      await fileClient.uploadFile(tempFile, {
        chunkSize: FILE_UPLOAD_MAX_CHUNK_SIZE,
        abortSignal: AbortController.timeout(20 * 1000) // takes too long to upload the file
      });
    } catch (err) {
      assert.equal(err.name, 'AbortError');
    }

    fs.unlinkSync(tempFile);
  }).timeout(timeoutForLargeFileUploadingTest);

azure:storage-file-datalake:info Request: {
  "streamResponseBody": false,
  "url": "https://jsv12.dfs.core.windows.net/filesystem159593998218302360/file159593998381509579?position=REDACTED&action=REDACTED",
  "method": "PATCH",
  "headers": {
    "_headersMap": {
      "content-length": "4194304000",
      "x-ms-version": "2019-12-12",
      "user-agent": "azsdk-js-storagedatalake/12.1.0 (NODE-VERSION v12.13.0; Windows_NT 10.0.19041)",
      "x-ms-client-request-id": "f3ad7dd1-2f1a-4a60-a064-06d0d0a6821d"
    }
  },
  "withCredentials": false,
  "timeout": 0,
  "keepAlive": true,
  "requestId": "f3ad7dd1-2f1a-4a60-a064-06d0d0a6821d"
}
azure:storage-file-datalake:info RetryPolicy: =====> Try=1 Primary
azure:storage-file-datalake:info Request: {
  "streamResponseBody": false,
  "url": "https://jsv12.dfs.core.windows.net/filesystem159593998218302360/file159593998381509579?position=REDACTED&action=REDACTED",
  "method": "PATCH",
  "headers": {
    "_headersMap": {
      "content-length": "1048576",
      "x-ms-version": "2019-12-12",
      "user-agent": "azsdk-js-storagedatalake/12.1.0 (NODE-VERSION v12.13.0; Windows_NT 10.0.19041)",
      "x-ms-client-request-id": "dac08a48-bb53-46f3-bb0a-e06cd04764b2"
    }
  },
  "withCredentials": false,
  "timeout": 0,
  "keepAlive": true,
  "requestId": "dac08a48-bb53-46f3-bb0a-e06cd04764b2"
}
azure:storage-file-datalake:info Response status code: 202
azure:storage-file-datalake:info Headers: {
  "_headersMap": {
    "content-length": "0",
    "date": "Tue, 28 Jul 2020 12:39:49 GMT",
    "server": "Windows-Azure-HDFS/1.0 Microsoft-HTTPAPI/2.0",
    "x-ms-client-request-id": "dac08a48-bb53-46f3-bb0a-e06cd04764b2",
    "x-ms-request-id": "0cb03dee-501f-0071-14dc-6437a7000000",
    "x-ms-request-server-encrypted": "true",
    "x-ms-version": "2019-12-12"
  }
}

I think this is the same issue in #12029. We remove the abort event handler after receiving response, instead of waiting for streaming to complete.

I think this is the same issue in #12029. We remove the abort event handler after receiving response, instead of waiting for streaming to complete.

timeout should trigger abortSignal now.


KiriKiri .soc scripts

I found a game with this odd .soc extension for the script it's not encrypted
I tried working around it by just replacing the extension into .ks and it did sort of work
only it copied the first line 5 times and it didn't detect the names
here's the script hopefully it's not a hassle to solve this
scenario.zip

I didn't think this warranted a new issue since the solution for it is mostly the same I found another KiriKiri game with an odd way to call the names in the script it has an "m" next to every name so VNTextPatch is ignoring the names.
scenario.zip
I hope this doesn't come off as me complaining I really like this tool and just want it to improve.

For the .soc scripts, which file has the duplicate lines? I don't see any at first sight. Also, are you sure the duplication is caused by VNTextPatch? (Maybe the lines are in fact duplicated in the .soc files?)
For the second game, the "m" prefix is a bit too short to assume it'll always indicate a name. In other games, this same prefix might be used for something completely different, which would cause VNTextPatch to import garbage into the translation file. Because this convention is so specific to this particular game, I won't be adding it into the "official" source code - please add it yourself. (VNTextPatch is open source specifically to allow for this kind of customization)

So yeah for the first thing about the .soc my bad I didn't check more than one file but yeah this one is definitely getting duplicated by VNTextPatch I've included both the original and the JSON for you my guess is it's getting confused because there's code after that line before the script resumes
__testbu.zip
as for the second issue, it's actually present in all of https://vndb.org/p467 cataloge well almost all the last 2 games didn't have the m prefix not sure if that affects your decision as for me personally I can deal with it on my own no problem :)

this one is definitely getting duplicated by VNTextPatch

I opened the .ks file to see this:
[voice file="A0001_001" ]
[talk name="緋沙音"]
「おはよう、アナタ。今日も一日頑張って欲しいから／
頑張ってみたの」
[hitret ]

[FcActMove layer=1 pos=1438 y=&-1840 time=10000   ]

[voice file="A0001_001" ]
[talk name="緋沙音"]
「おはよう、アナタ。今日も一日頑張って欲しいから／
頑張ってみたの」
[hitret ]

[voice file="A0001_001" ]
[talk name="緋沙音"]
「おはよう、アナタ。今日も一日頑張って欲しいから／
頑張ってみたの」
[hitret ]

[voice file="A0001_001" ]
[talk name="緋沙音"]
「おはよう、アナタ。今日も一日頑張って欲しいから／
頑張ってみたの」
[hitret ]

[voice file="A0001_001" ]
[talk name="緋沙音"]
「おはよう、アナタ。今日も一日頑張って欲しいから／
頑張ってみたの」
[hitret ]

as for the second issue, it's actually present in all of https://vndb.org/p467 cataloge

It's not so much that it's not worth adding the prefix for these games. As mentioned, the main issue I have is that it could cause problems for other games.
I made VNTextPatch recognize the .soc extension and the "talk" command, but I'll leave it at that for this ticket.


More variable types that don't match the docs

After #1198 I took a look at some nearby fields and noticed a few more that don't match the docs here (I think the states CVR_2S0 is the same table as Cvr2SoCd) . Specifically, enty_naml which is 194 in the django model and 200 in the docs, and enty_namf, which is 34 in the django model and 45 in the docs. It's pretty unlikely that any one of these is gonna be a problem immediately, but it does seem like the sorta thing that may occasionally be a pain.
I'm guessing the original models were written before the docs were available by just looking at the maximum size of a current dump. And not all the fields listed in the docs actually have max sizes (that I can figure out, anyways). But it would probably be possible to parse out lotsa them. One question, though--is 'cd' always appended, or does it signify something I might be missing?

I think that your theory on the field lengths is probably right, though the memory of how it was done might rest with @armendariz. I suppose we probably need to do a run through all the fields in the database and increase the size of any where the documentation is higher than what we have to avoid the error you patched earlier in the day for #1198.
As far the Cd suffix on tables, that is there because the same string is appended to every text file in the ZIP file source data. I could see a case for removing something to redundant. When it was drafted I was simply trying to hew as closely as possible to the source data.

Ah, I get it about the 'cd'. I just wanted to make sure I knew how to translate table names...
FWIW ind_group (length 87) also doesn't match.
I could probably hack something up to do the comparison between the docs and the models, though if there's institutional memory about what's already been cleaned up (and maybe this one table was just missed) it might not be worth doing.

It's probably worth hacking something up to compare the docs to the models.
MySQL throws a ton of truncation errors that I intentionally ignored as I
tried to figure out how it all fit together and what I actually cared about
in there.
On Fri, Feb 5, 2016 at 12:29 AM, Jacob Fenton<EMAIL_ADDRESS>wrote:

Ah, I get it about the 'cd'. I just wanted to make sure I knew how to
translate table names...
FWIW ind_group
https://github.com/california-civic-data-coalition/django-calaccess-raw-data/blob/master/calaccess_raw/models/campaign.py#L547
(length 87) also doesn't match.
I could probably hack something up to do the comparison between the docs
and the models, though if there's institutional memory about what's already
been cleaned up (and maybe this one table was just missed) it might not be
worth doing.
—
Reply to this email directly or view it on GitHub
https://github.com/california-civic-data-coalition/django-calaccess-raw-data/issues/1199#issuecomment-180205691
.

That sounds awesome @jsfenfen. Should we also stop ignoring these warnings here?
Alternatively (or perhaps as well?), I've been wondering if we should add some checks to cleancalaccessrawfile. When we iterate over each line in the .CSV, we could go read these into model instances and then call .clean_fields() to see if we get any validation errors. I assume this would be less overhead than actual trying to save the records because it would avoid hitting the db.

Is there a performance concern? I wonder if saving individual models, and making individual error reports, would be fine for this application?
In that case, you can also shove the csv line into an "errors" table, that can be re-run once the issue has been debugged & fixed. We did this for https://github.com/learningequality/ka-lite , and it seemed to work well.
Could be too much complexity here, just an idea..

It's definetely a performance concern, but I can't say we've tested what exactly the hit would be. My gut says it would slow things down considerably, but I could very well be wrong.
If it could become part of a general "resilience" framework of change that included things like the error handling you describe, I could see some value.
However, due to the very very low percentage of errors in the overall database, I'm torn and undecided about whether a better investment of time on the "clean" step might be to move the other direction. By that I mean pursuing a "speed" agenda that would try to replace Python-level operations that require looping through the records with attempt lower-level shell operations that handle the same tasks.

I'd heed Ben's speed concerns. I had lots of checking and really slow code
to start with that Ben sped up exponentially and it was a very good thing
he did.
It's been a while but I believe the error rate was so low that it didn't
matter to me. And every time I dug into the database I never noticed any
blatant ones.
On Fri, Feb 5, 2016 at 12:06 PM, Ben Welsh<EMAIL_ADDRESS>wrote:

It's definetely a performance concern, but I can't say we've tested what
exactly the hit would be. My gut says it would slow things down
considerably, but I could very well be wrong.
If it could become part of a general "resilience" framework of change that
included things like the error handling you describe, I could see some
value.
However, due to the very very low percentage of errors in the overall
database, I'm torn and undecided about whether a better investment of time
on the "clean" step might be to move the other direction. By that I mean
pursuing a "speed" agenda that would try to replace Python-level operations
that require looping through the records with attempt lower-level shell
operations that handle the same tasks.
—
Reply to this email directly or view it on GitHub
https://github.com/california-civic-data-coalition/django-calaccess-raw-data/issues/1199#issuecomment-180443515
.

My gut says it would slow things down considerably, but I could very well be wrong.

It will slow things down. My point is only that, in production, performance is not a concern, is it? As long as the process completes daily, does it matter if it takes 5 minutes or 500 minutes?

I'd heed Ben's speed concerns.

I'm also Ben ;) But I'm all for speed, if it's a need. :)

Ha! So much for my ability to read. Hi Ben I don't know! Sorry about that.
When a filing deadline hits processing speed will matter a lot
On Fri, Feb 5, 2016 at 12:14 PM, Ben Cipollini<EMAIL_ADDRESS>wrote:

My gut says it would slow things down considerably, but I could very well
be wrong.
It will slow things down. My point is only that, in production,
performance is not a concern, is it? As long as the process completes
daily, does it matter if it takes 5 minutes or 500 minutes?
I'd heed Ben's speed concerns.
I'm also Ben ;) But I'm all for speed, if it's a need. :)
—
Reply to this email directly or view it on GitHub
https://github.com/california-civic-data-coalition/django-calaccess-raw-data/issues/1199#issuecomment-180447619
.

@bcipolli: Our newsroom use case prioritizes speed very highly. We want to be able to review and analyze the freshest data as soon as possible. However, I recognize that some other use cases, like slow-speed research or website development might not rank it has highly.

I tried removing these the MySQL warnings filters from the load command, but was still not getting any errors. Seems like MySQL will just always quietly truncate the data.

Interesting. This is speculative, but it's also possible there is some lower level MySQL option that controls that sort of thing.

Ok, I ran the dox vs whats in the django models for fields that I could match and that had maxlengths. These are the places where the django maxlength was * less * than the cal access dox maxlengths (and the reference page in the documentation). I may have messed this up, please lemme know. Fwiw here are the calaccess_raw field lengths and those I parsed from the docs (by looking at p. 14-98ish).
CVR2_SO.ENTY_NAML django:194 - dox: 200 see p. 46
CVR2_SO.ENTY_NAMF django:34 - dox: 45 see p. 46
CVR2_SO.ENTY_NAMT django:9 - dox: 10 see p. 46
CVR2_SO.MAIL_CITY django:25 - dox: 30 see p. 47
CVR2_SO.EMAIL_ADR django:40 - dox: 60 see p. 47
CVR2_SO.IND_GROUP django:87 - dox: 90 see p. 47
CVR2_SO.PARTY_NAME django:63 - dox: 200 see p. 48
CVR2_SO.POF_TITLE django:44 - dox: 45 see p. 47
CVR3_VERIFICATION_INFO.SIG_LOC django:39 - dox: 45 see p. 49
CVR3_VERIFICATION_INFO.SIG_NAML django:56 - dox: 200 see p. 49
CVR3_VERIFICATION_INFO.SIG_NAMS django:8 - dox: 10 see p. 50
CVR_REGISTRATION.FILER_ID django:8 - dox: 9 see p. 30
CVR_LOBBY_DISCLOSURE.FILER_ID django:8 - dox: 9 see p. 25
LOBBY_AMENDMENTS.D_L_NAML django:56 - dox: 200 see p. 73
LOBBY_AMENDMENTS.D_L_NAMF django:35 - dox: 45 see p. 73
LOBBY_AMENDMENTS.D_L_NAMS django:8 - dox: 10 see p. 73
LOBBY_AMENDMENTS.D_LE_NAML django:160 - dox: 200 see p. 73
LOBBY_AMENDMENTS.D_LE_NAMS django:9 - dox: 10 see p. 74
LCCM.CTRIB_NAML django:120 - dox: 200 see p. 63
I think there were 184 fields I couldn't match (stuff that's in the django but not in the dox), I haven't really looked at that. The only text munging I did was to remove the _CD from the db_table names.

Also, fwiw, the keys I couldn't match are here. What's listed are [db table name].[field name] that I couldn't find in the documentation, either because they weren't listed in the part of the dox I was parsing or because there's some name munging that I did wrong or something else entirely. Suffice it to say I couldn't check them.

Ahh, I see, most if not all of the ones I couldn't find are in fact documented in a later section of the documentation document i.e. around page, around page 155 for table F501_502. Sigh.

Hey so I double checked both parts of the cal access documentation, and all the fields that were too short in my parse of the first part of the dox are also too long in the second part of the dox. Comfortingly all the fields I looked at were identical, btw. Unless there's a good reason not to I'm gonna submit a PR with the model field changes...

@jsfenfen Thanks so much for looking into this. Go ahead with the PR.
Were you parsing the docs in Python? If you're willing to share that code, maybe we could find a home for it in this repo (or a related one). Might be helpful as we get deeper into documentation.

@gordonje yeah I am parsing using python, but am doing so using a combination of pdfplumber and a util I have to turn chars into words. It's repeatable (and I can dump it in a gist or something) but you wouldn't want the dependencies.

Got it. Yeah, I gist would be fine. Then I could refer to it as a resource for some of the other tasks we have in the works.

You think you got them all? Or are there more?

So I think I got the ones that are mentioned in the first half of the documentation--roughly page 14-98. I didn't realize this initially, but some tables only appear in the second part of the documentation (p. 99-234 I think).
To be honest, I didn't realize the two parts include different tables. One's dated april of 2002 and the other is feb. 2002--have no idea what that signifies though.
There were about 175 I couldn't check outta 980-ish total fields. I'll likely hack something up to check the second part of the documentation as well. It seems like most of the errors were clustered in a few tables so that might be all of them.

I just happened to be in the docs near this table, so I took care of this one.

So have all these been corrected?


feat: semistandard linting

Work done to address issue #98
Description of work

Add semistandard package for linting
Use semistandard's semistandard --fix feature to fix some of the existing linting errors

To test
Pull down the branch and run npm run lint to see semistandard in action

Hey @mdt2,
Your changes look good to me!
View build log
TravisBuddy Request Identifier: c908e960-f05d-11e9-983c-41f05746e0df

Good call, @liyasthomas. I'm also getting linting failures in /assets/js/pwa.js for localStorage being undefined, as well as /functions/index.js for never using functions. Should those be added to the ignore list too?

Hey @mdt2,
Your changes look good to me!
View build log
TravisBuddy Request Identifier: 90089460-f0e1-11e9-aa1b-53c400f64b87

@mdt2 well, ignoring all will make linting irrelevant isn't it? I should've consider linting at the beginning of the project. My bad.
Let's do one thing now, ignore all files causing build failure and pull again.

a09d7d76d3a41fc9504e8768f224423ae9360276 resolves this, hence closing.

a09d7d76d3a41fc9504e8768f224423ae9360276 resolves this, hence closing.


[BUG] - Issue with Rancher Monitoring Prometheus Node Exporter Pod

Rancher Server Setup

Rancher version: rancher version v2.9.0-alpha7 (c5733e1ae)
Installation option (Docker install/Helm Chart): Helm

If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): RKE2

Proxy/Cert Details: Airgap Installation

Information about the Cluster

Kubernetes version: v1.29.6+rke2r1
Cluster Type (Local/Downstream): Local

If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider):

User Information

What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom)

If custom, define the set of permissions: Admin

Describe the bug

Issue with Rancher Monitoring Prometheus Node Exporter Pod on v2.9.0-alpha7 (airgap)
In the airgap installation, I noticed an issue with the Rancher Monitoring Prometheus Node Exporter pod in the cattle-monitoring-system namespace, it's trying to pull the wrong image:
To Reproduce

Install rancher-monitoring-104.0.0-rc1+up57.0.3 chart on an airgap rancher server
Result
root@ip-10-0-13-92:~/rancher# kubectl get pods -n cattle-monitoring-system rancher-monitoring-prometheus-node-exporter-tzstb -o jsonpath='{.status.containerStatuses}' | jq
[
  {
    "image": "xxxxxxxx.us-east-2.compute.amazonaws.com/xxxxxxxx.us-east-2.compute.amazonaws.com/rancher/mirrored-prometheus-node-exporter:v1.3.1",
    "imageID": "",
    "lastState": {},
    "name": "node-exporter",
    "ready": false,
    "restartCount": 0,
    "started": false,
    "state": {
      "waiting": {
        "message": "Back-off pulling image \"xxxxxxxx.us-east-2.compute.amazonaws.com/xxxxxxxx.us-east-2.compute.amazonaws.com/rancher/mirrored-prometheus-node-exporter:v1.3.1\"",
        "reason": "ImagePullBackOff"
      }
    }
  }
]
root@ip-10-0-13-92:~/rancher#
root@ip-10-0-13-92:~/rancher# kubectl -n cattle-system exec -it rancher-7d8bfb5976-btqq5 -- rancher -v
rancher version v2.9.0-alpha7 (c5733e1ae)
root@ip-10-0-13-92:~/rancher#

root@ip-10-0-13-92:~/rancher# helm ls -n cattle-monitoring-system
NAME                  	NAMESPACE               	REVISION	UPDATED                                	STATUS  	CHART                                      	APP VERSION
rancher-monitoring    	cattle-monitoring-system	1       	2024-07-16 12:25:11.614883033 +0000 UTC	deployed	rancher-monitoring-104.0.0-rc1+up57.0.3    	v0.72.0
rancher-monitoring-crd	cattle-monitoring-system	1       	2024-07-16 12:25:05.402031705 +0000 UTC	deployed	rancher-monitoring-crd-104.0.0-rc1+up57.0.3
root@ip-10-0-13-92:~/rancher#

Expected Result

should install successfully and fetch the right image
Screenshots

Additional context

There's also a missing version bump for prometheus-node-exporter that should be included (1.3.1 -> 1.7.0)

Issue has been fixed and verified on Air-gap setup. Rancher Monitoring Prometheus Node Exporter pod in the cattle-monitoring-system namespace pulling correct image.


keepalived-operator new version

New Submissions

[ ] Has you operator nested directory structure?
[ ] Have you selected the Project Community Operator Submissions in your PR on the right-hand menu bar?
[ ] Are you familiar with our contribution guidelines?
[ ] Have you packaged and deployed your Operator for Operator Framework?

[ ] Have you tested your Operator in all supported installation modes?
[ ] Is your submission signed?

Updates to existing Operators

[x] Is your new CSV pointing to the previous version with the replaces property?
[x] Is your new CSV referenced in the appropriate channel defined in the package.yaml ?
[ ] Have you tested an update to your Operator when deployed via OLM?
[x] Is your submission signed?

Your submission should not

[x] Modify more than one operator
[x] Modify an Operator you don't own
[x] Rename an operator - please remove and add with a different name instead
[x] Submit operators to both upstream-community-operators and community-operators at once
[x] Modify any files outside the above mentioned folders
[x] Contain more than one commit. Please squash your commits.

Operator Description must contain (in order)

[x] Description about the managed Application and where to find more information
[x] Features and capabilities of your Operator and how to use it
[x] Any manual steps about potential pre-requisites for using your Operator

Operator Metadata should contain

[x] Human readable name and 1-liner description about your Operator
[x] Valid category name1
[x] One of the pre-defined capability levels2
[x] Links to the maintainer, source code and documentation
[x] Example templates for all Custom Resource Definitions intended to be used
[x] A quadratic logo

Remember that you can preview your CSV here.
--
1 If you feel your Operator does not fit any of the pre-defined categories, file a PR against this repo and explain your need
2 For more information see here

Can you please bump patch version? For example to 0.1.1? Thanks

@mvalahtv no, I cannot bump: this is a patch to the current version.
@dmesser we said it was OK to patch current versions.

@raffaelespazzoli It's ok to patch cosmetic issues. Your's is a breaking change and conflicts with what your Operator claims it supports regarding install modes. If you are not able to inject the watch namespace configuration you cannot at the same time claim support single namespace install mode (where OLM will provide the namespace selected by the user via an annotation on your deployment).

/hold


Add history PE, PB data of us stocks. 添加美股个股历史PE、PB数据.

Is your feature request related to a problem? Please describe.
No. Please add history PE, PB data of us stocks, if possible. 添加美股个股历史PE、PB数据.
Describe the solution you'd like
None.
Describe alternatives you've considered
None.
Additional context
Please add history PE, PB data of us stocks, if possible. 添加美股个股历史PE、PB数据.

有无合适的数据源？


Incorrect Function Error when using get_feature_report

Code:

Error:

There is not any space in your buffer to actually write the received report into.
Use a larger buf.
Also as per documentation:

Set the first byte of buf to the ‘Report ID’ of the report to be read. Upon return, the first byte will still contain the Report ID, and the report data will start in buf[1].

The first byte in the given buf is input, and contains the report id you wish to receive.

Thank you for your answer. It's working now.


Add support for Stripe Connect

I've implemented managed accounts. This code is stable and in-use, now I'd like to get it back into this project.
It adds the basic ability to create managed accounts, transact on their behalf and transfer funds to external bank accounts.
@paltman I've obviously got some work to do on tests, but before finishing those up I'd like the greenlight from you on:

general approach
addition of stripe_account to various models
improvements to the robustness of the webhook handling code
the initial and dynamic forms for collecting Bank Account data
changes to sync_customers (I think this really needs to be built upon as it's still only customers, sources, charges and invoices)

Let me know what you think!
Thanks,
Luke

Hola @paltman - I've updated as requested. Glad it's mostly good! Let me know if there are any other adjustments you'd like me to make.
Is your preference for everything to be rebased into a single commit for merge?

I'm glad I found this. I was about to embark on the same work. A few questions:

I've implemented managed accounts

Per Stripe's documentation, there are three Connect Account Types:

Standard
Express
Custom.

Which of these types are supported by this PR?

@iMerica My use case required full customisation of the Connect experience - so the pieces are added with the ability to add, update, transfer to and payout to connected accounts via the API.
The other account types won't allow you to manipulate connected account data via the API. But I suppose some subset of the Account info is available and sync'd via webhooks?
You'll need to experiment!

Okay cool thanks for the update. I'm not an expert or maintainer in this codebase, but your changes LGTM. Maybe just add a new "Connect" section to the docs describing a hypothetical use case that can be supported by these changes?

This needs to be rebased / updated.
What is the status of this?
I'd be glad to help out getting this finished.

Hello - I'll rebase my changes in the next couple of days. Keen to get this pushed through, too.
@blueyed - one way you could help would be playing around a bit with the DynamicManagedAccountForm and InitialManagedAccountForm classes. I wrote them and have not had a chance to give them much of a try, as the project I'm working on is lower-volume for account owners and we won't be hitting the later stages of account verification in the near-term.

@lukeburden

Keen to get this pushed through, too.

Awesome.

I think there are things that could be moved out of this PR into separate ones, starting with "Improving speed of syncing large data sets from stripe" already (and the followups to it).
I could move them out into separate PRs, if you don't have the time yourself at the moment.

We're not making use of the forms ourselves, so they are likely the last for me to look at anyway.

Just for reference: some discussion on Stripe Connect integration for dj-stripe: https://github.com/dj-stripe/dj-stripe/issues/97#issuecomment-125778458

I'm very excited for this feature. Thanks for hammering on it.

On Sep 4, 2017, at 5:27 AM, Daniel Hahler<EMAIL_ADDRESS>wrote:
@blueyed commented on this pull request.
"Managed accounts" has been renamed to "Custom accounts": https://stripe.com/docs/connect/custom-accounts#name-change
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

I starting to catch up on all the details (and trying to do so in the abstract as I've never dealt with connected accounts before).  At one point in this thread, it's mentioned that this is for custom type accounts.  Would that mean that express is inherently supported as a subset of what custom would require?

Hey crew,
An update on this. I've just rebased off master and will do the following as soon as I can:

add basic support for direct, destination and separate charges. Nothing too fancy though.
look at making django-ipware an optional dependency, although my sense is this just creates complication at the deployment phase (and thoughts there welcome)
split out improvements to data syncing into a separate PR if it's not too much of a PITA
basic tests for the dynamic account form, that self-configures on the basis of the specific verification required for a particular account.

@paltman - to answer your question, standard and express accounts will be support in so far as you can create a charge and your data will be sync'd properly - however, those both require integration with an oauth-like-flow that takes users to a Stripe hosted setup page for account creation. I have not added support for this flow in the form of a redirect view and callback view as I simply do not need this functionality for my use case. It shouldn't be too hard for someone else to add when they need it. :)
Two other issues:

Stripe have split payouts and transfers. I will not address this in this PR, and will leave adding support for this split for another PR.
Stripe have renamed "standalone" and "managed" accounts to "standard", "express" and "custom" accounts. I will also likely leave this for an extra PR. I'm worried adding support for this now will, by virtue of requiring us to default to a very recent version of Stripe's API, also force us to add support for the payout/transfer split.

Your thoughts are welcome - keen to get this code merged!
Luke

Awesome work!  Excited about this!

@paltman
I would prefer to not have gifs in here.. :/

@lukeburden
Could you make use of my branch with fixups?

@lukeburden

Stripe have renamed "standalone" and "managed" accounts to "standard", "express" and "custom" accounts. I will also likely leave this for an extra PR. I'm worried adding support for this now will, by virtue of requiring us to default to a very recent version of Stripe's API, also force us to add support for the payout/transfer split.

It seems more that this renaming should get reflected in pinax-stripe after all.
There does not seem to be an API breakage involved here, but it seems to be rather about how things are called, and therefore it makes sense to include it here.

split out improvements to data syncing into a separate PR if it's not too much of a PITA

I can help with that (do not remember if I've done it already).

Let's push this, but alsoo have some still:

I've spent some time on this today:

removed changes to sync_customers, reverting to that in master (I have some neat code that I'd like to put into an improved version still, at some stage! Syncing using the current command on a modest project would take many days of runtime)
updated code and migrations to reflect the current naming conventions for Connect (standard, express and custom accounts). I updated the names of the forms too.
tested the InitialCustomAccountForm with the prompting of @sreyemnayr
fix a subscription retrieval test
changed all single quotes to the project standard, double quotes
add support for destination, direct and flex Charge creation

Any other blockers before we merge this beast?

@lukeburden this is a tremendous amount of quality work.  I really appreciate it.  I'll dig in next week and start reviewing things closer and figuring out how to test things out locally.  Any guidance you could provide would be helpful.  In fact, a short bit in the docs about this feature would be greatly appreciated, while it's all still fresh in your mind.  If you are not up for it, I completely understand and I'm happy to write the docs if you can provide me some bullet points / key points / outline and use that combined with some testing direction, I can write out the docs.
The only other thing I can think of prior to doing a more intensive review is just considering more tests.  We have maintained 100% coverage and I'd like to continue that.  I can help with tests as I have time but you probably have much more context to write better tests, than I.  But I'll want to get this branch up to 100% before merging.
I feel really bad asking even more of you (docs and tests) after all this work because I really do appreciate all the work you've done so far and am excited to get this merged in.  I'll do what I can to help with these requests.

Most of this is not too dissimilar to the rest of the project, I would say. In order to properly use custom accounts, people will need to write their own views and subclass the forms provided. They'll also need to hook up their own logic to webhooks for when custom accounts require further verification. I can definitely write a little in the docs about how this works.
If you have time @paltman it'd be great if you helped with some tests. I can write some too. Maybe @blueyed, if you have time, could pitch in a little? Your feedback so far has been great.
Also @blueyed - I saw you had some minor edits on a branch in your own fork. Would you be able to squash them and submit a PR against the connect branch on my fork? (my fork is still called django-stripe-payment).

I'll try put some time into this some evening this coming week. Looking forward to your review, @paltman!

@lukeburden

I saw you had some minor edits on a branch in your own fork. Would you be able to squash them and submit a PR against the connect branch on my fork?

I've notified you about them 2 weeks ago (https://github.com/pinax/pinax-stripe/pull/336#pullrequestreview-60393191), and later asked about it again (https://github.com/pinax/pinax-stripe/pull/336#issuecomment-329295252).
They were meant to be squashed into your existing commits - and I've reordered/applied them accordingly already.
Since you have force-pushed your branch (eaaac17...a3f7f87) it causes conflicts now.
And I've noticed that there are still commits that have conflict markers in your branch anyway.
So I will rather wait for feedback from you on how to proceed here.  I think the best would be to apply the patches from my branch still, which could be done during an interactive rebase on your side.
You can see them here: https://github.com/blueyed/pinax-stripe/compare/eaaac17...connect-fixes

@blueyed - I'd rather you stuck to the regular workflow on Github and submitted a PR. If you had provided a PR in the first instance, I'd have reviewed and merged. The force update was an oversight on my side; not the best move for collaboration - noted and I'll avoid in the future.
For your PR, I'd recommend cherry-picking your specific fix commits onto a branch based off of the HEAD of the branch behind this PR.

I'd rather you stuck to the regular workflow on Github and submitted a PR

It is not common to have PRs for PRs, is it for you?

I'd have reviewed and merged

And after all you cannot do that on GitHub with fixup commits.

I'd recommend cherry-picking your specific fix commits onto a branch based off of the HEAD of the branch behind this PR.

My commits are not on top of your (original) branch/PR but meant to be squashed/fixed up in there - I've gone through it with interactive rebase and amended stuff.

@blueyed - your help is truly appreciated - I don't mean to be a pain, I just don't have the time to play around with your branch and commits. Thanks! Got to switch focus back to work now. I'll await that PR.

All good.

I'll await that PR.

Don't wait for it though - please just go ahead.

I think that @paltman has no problems with merging commits with conflict makers

I'm not sure what is meant by conflict makers.  If that was a typo and you meant conflict markers then I think you previously misunderstood me.  I'd prefer that those be resolved before pushing.  I don't think having:
+++
===
---

lines in the middle of a Python file helps anyoone.

@paltman
Yeah.. markers of course..
but they are in here (in the commit history) and my impressions was that you would not care as long as the final diff does not contain them.

(as for having them in the Git history this is bad with using git-bisect, and even empty or non-squashed commits are bad then (more overhead))

Ah, then it was my misunderstanding.  I merged based on the fact that it was a PR without a WIP message or title and the resulting diff looked great.  It's there now.  Nothing we can really do about it.

I've added some tests for the Account forms - a little work there to do yet, namely testing an ID document upload. Coverage should almost be there.
A little splash of documentation and we should be close to sweet, sweet merge time!

@paltman - I've pushed coverage about as far as I have time for. Going to turn to the documentation next!

Can we get https://github.com/pinax/pinax-stripe/pull/373 merged first, to have a better overview about coverage (changes)?

Fantastic. I’ll check it out.
Sent from my iPhone

On Sep 26, 2017, at 2:10 PM, Luke Burden<EMAIL_ADDRESS>wrote:
@paltman - I've pushed coverage about as far as I have time for. Going to turn to the documentation next!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@lukeburden I'm sending you another PR shortly for the master failures.
How are the docs coming?

@paltman the docs are on my mind, just a bit slammed with my work so struggling to get to it. How would you recommend I add to them? What editor, or workflow do you use? How can I test what they'll look like on RTD, etc? A little info would be great to speed me up when I get to it.

Just markdown, don't stress about formatting.  Maybe just add a new markdown file in the docs/ folder.  I can tidy up formatting, linking, etc.  I'll test the docs myself doing a trial run setup and polish off the rough corners as well.  Just need the broad strokes.  I appreciate the patience on the back and forth of this really long PR.

I’ll merge this when I some free time this evening.
Thanks for all this great work.


Corrected INSTANCE_PREFIX usage

INSTANCE_PREFIX environment variable is being overwritten by KUBE_AWS_INSTANCE_PREFIX environment variable or defaulting to kubernetes, see https://github.com/kubernetes/kubernetes/blob/59628744141bb6debd4ac042882ffa6855d846ce/cluster/aws/config-default.sh#L66
INSTANCE_PREFIX="${KUBE_AWS_INSTANCE_PREFIX:-kubernetes}"

This change is

I signed it!

@mwl Would you merge upstream master into your branch to get rid of the warning message?

Thanks for finding @mwl.  LGTM, though needs a rebase.

@pwittrock @justinsb Thanks. I rebased the branch.

Thanks everyone!


feat: add option for instance name

Summary
Add options for instance name to be used for Analytics connector
amplitude.init(API_KEY, {
  instanceName: 'env',
});

Checklist

[x] Does your PR title have the correct title format?
Does your PR have a breaking change?:  No

all instances with the same name also share the same storage.

Are they not using api key?

@kevinpagtakhan what exactly is this for? are there docs explaining this attribute?


Disable the monitoring of certain attributes for all models

I want to ignore changes to the updated_atattribute in all my models.
Is there a way to do this in the initializer? If not, will there be any effort to implement a possibility to do so?

I want to ignore changes to the updated_at attribute in all my models.

AFAICT, the timestamp attributes (created_at, updated_at) cannot be ignored, but they can be skipped.
Please see https://github.com/airblade/paper_trail/blob/v8.1.1/README.md#skip

Is there a way to do this in the initializer?

No.

If not, will there be any effort to implement a possibility to do so?

I'm open to suggestions.

@jaredbeck: For sure they can be ignored. See https://github.com/airblade/paper_trail/blob/v8.1.1/README.md#2c-choosing-attributes-to-monitor

I don't plan to look into this any further, but I welcome suggestions.
You're the second person to ask about configuration applicable to all models, so it's safe to say other people would appreciate such a feature.
To keep this issue open please describe in detail how you would implement such a feature, and I'll be happy to give you some feedback before you start code.

Please describe in detail how you would implement such a feature, and I'll be happy to give you some feedback before you start coding.

Closing due to inactivity. I'm still open to this idea, and happy to help as described above.


Building for iphonesimulator

when installing via Carthage building for iphonesimulator fails if signature is not present (no iPhone developer account)
Base ObjectMapper library builds fine for iphonesimulator target

This PR for Carthage actually says that having iPhone Developer account is no longer a requirement to build for iPhone Simulator and debugging
https://github.com/Carthage/Carthage/pull/583

The bug is in the iOS target build settings - it requires signature for iOS Simulator, while the correct value should be 'Don't sign'. Changing it resolves the problem

Looks like manual change in the project settings and building in Carthage results in Xcode crashes on breakpoints that are set inside closer of responseObject() method.
Any way this could be fixed in repository?

Crash when breakpoint is inside responseObject()
https://gist.github.com/AAverin/e3d293ca168b53443773

Hi @AAverin,
I'm not sure exactly what is causing the problem you are experiencing... I just created a new project using Carthage and I was able to put a break point within the responseObject() function without a problem.

The case might be that I don't have iOS developer license and build for simulator-only.

I think you are probably right about this...


Add documentation for cross-version image builds and UEFI support

Adding docs for cross-version image build support and UEFI support in image-builder.
Documentation added/planned (if applicable):
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/approve
/cherrypick

/cherrypick release-0.17


Update documentation, add examples for asyncio based Python gRPC

A simple Google search for "from grpc import aio" yields 9(!) results (as of 2020-10-30) of which 0 are relevant to start with the new asyncio based gRPC Python API.
The documentation at https://grpc.github.io/grpc/python/grpc_asyncio.html# does not cover a single example, it is basically just a list of method signatures. There is no information given about tying together all the necessary parts to build an actual client/server architecture based upon the new asyncio API.
This not very user friendly...

I will add asyncio version of existing examples soon. Currently, we are short on people. If you find any example you may able to contribute, feel free to help out. (The only example up there is contributed...)
For immediate usage lookup, we have extensive test cases at https://github.com/grpc/grpc/tree/master/src/python/grpcio_tests/tests_aio.

@lidizheng thanks for the fast reply! I will check the test cases, this might be a good starting point.

@lidizheng I'd totally be willing to contribute to the examples.
Unfortunately, I was unable to find some background information about end-to-end examples.
So far, it seems to me that
python -m grpc_tools.protoc --proto_path=. --python_out=. --grpc_python_out=. async.proto
...
from grpc import aio

is pretty much the common way to implement a server based upon the well-known API with...
MyServicer(ServiceServicer):
    async def Service(self, request_iterator, context):
        ...

...while the new API directly accessing the stream via...
import grpclib
...

MyServicer(ServiceBase):
    async def Service(self, stream) -> None:
        request = await stream.recv_message()
        ...
        await stream.send_message(...)

...needs to be build with...
python -m grpc_tools.protoc -I. --python_out=. --grpclib_python_out=. async.proto
                                               ^--------------------^

Could the former be considered the high-level API where one simply gets the well-known request[_iterator], context methods while the latter is the new "low-level" API?
One thing, for example, that needed custom request iterators in the past are bi-directional streams where input needs to be cached and single responses sent for multiple iterations of the request stream. This seems to be significantly easier and more sophisticated with .recv_message() and .send_message(). Is this intended to be used for such cases? Are there any - especially negative - performance implications?

@lidizheng additional question (if I may): how would the client-side look like with read and write calls?

Could the former be considered the high-level API where one simply gets the well-known request[_iterator], context methods while the latter is the new "low-level" API?

The official gRPC Python Protobuf generation flag is "--grpc_python_out". It works with the asyncio API, thanks for Python's tolerance of function signature change, we can assign an "async def" implementation and override parent's "def".
It's hacky, but allows the same generated code to work with py2/py3 and sync/asyncio.

Is this intended to be used for such cases? Are there any - especially negative - performance implications?

New API supports both "async generator" and "send/receive API". There isn't any observed performance implication. After all, they are implemented in coroutines and shares most underlying logic, just have different shapes of interface.

how would the client-side look like with read and write calls?

We designed two type of API because they works best under different scenarios. For the "async generator" API, it meant for RPC logic that the sending and receiving of messages are pretty much decoupled. But for the "send/receive" API, it works for RPCs that each request maps to one or several responses, like database queries.
If you find the application needs to write "while True", then it is a good indicator that it should change to "async generator" pattern. In your snippet, it will look like:
MyServicer(ServiceBase):
    async def BidirectService(self, stream) -> None:
        async for request in stream:
            ...
            await stream.send_message(...)

@lidizheng thanks for the update, really appreciate the time taken! I will definitely push some examples soon.
However, I'd have some follow-up questions.

But for the "send/receive" API, it works for RPCs that each request maps to one or several responses, like database queries.

This is, indeed, a very interesting use-case. However, with the "standard" --grpc_python_out flag I was unable to find the ServiceBase API in the generated "grpc_pb2.py" file. There was only ServiceServicer with the regular request_iterator.
How would we get the async def BidirectService(self, stream) -> None interface then instead of async def Service(self, request_iterator, context)?
Additionally, wouldn't async for request in stream: just allow for 1:n (request:responses)? What if we'd like to have n:1 or n:m? Shouldn't it be comparably easy to read but more modular to do something like while stream: and buffer incoming messages, so that the method could handle n:m cases?

Bidi-streaming is not simple. For cases like n:m, usually the requests and responses are decoupled, or they can be grouped together into an "1:1" mapping. The async generator and read/write API can work together, like using generator to send requests, but using read to fetch responses.

There was only ServiceServicer with the regular request_iterator.

You should be able to plug ServiceServicer into your service implementation. If you are expecting async def BidirectService(self, stream) -> None, it means you might confused grpcioandgrpclib`. Both projects implement gRPC protocol, but have different interface.
For an example service method implementation for streaming RPC, see https://github.com/grpc/grpc/blob/master/src/python/grpcio_tests/tests_aio/unit/server_test.py#L163.

@lidizheng thanks for the reply!
Yes, I am aware that grpclib is another framework implementing gRPC with the AsyncIO API. I've used this so far for the server because:

I've only managed to achieve [stream interface with recv/send logic] with the use of grpclib for the server, which is another AsyncIO-native implementation

Thus, I used grpcio's aio for the client and grpclib for the server.
I've also looked up...
    async def _stream_stream_evilly_mixed(self, request_iterator, context):
        assert _REQUEST == await context.read()
        request_count = 0
        async for request in request_iterator:
            assert _REQUEST == request
            request_count += 1
        assert _NUM_STREAM_REQUESTS - 1 == request_count

        yield _RESPONSE
        for _ in range(_NUM_STREAM_RESPONSES - 1):
            await context.write(_RESPONSE)

...but does this really allow for n:m mappings?
With grpclib I could, just for the sake of an example, loop 20x over await stream.recv_message() and then execute stream.send_message() 2x and put all of this in a while stream loop to let the server chunk all requests in 20:2 mappings.
In async def _stream_stream_evilly_mixed(self, request_iterator, context) we would loop through async for request in request_iterator: as long as results are being sent, right? We would need to break out of the request_iterator loop after 20 iterations to realize the example of 20:2 mappings.
I've struggled to see how we would realize the same thing with the request_iterator solution of grpcio.

Couple lines above, grpcio also offers read and write methods.
    async def _stream_stream_reader_writer(self, unused_request, context):
        for _ in range(_NUM_STREAM_REQUESTS):
            assert _REQUEST == await context.read()
        for _ in range(_NUM_STREAM_RESPONSES):
            await context.write(_RESPONSE)


[Workspace Feeds] [$50] Change Cards & Domains tab to Domains in settings

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Reproducible in staging?: Y
Reproducible in production?: Y
Slack conversation: https://expensify.slack.com/archives/C06ML6X0W9L/p1720038761870689
Action Performed:

Sign up on NewDot
Create a workspace
Navigate to settings

Expected Result:
Cards & Domains tab should just say Domains, since cards will  be supported in NewDot within the workspace.
Actual Result:
The tab says Cards & Domains.
Screenshots/Videos

View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01b56ebd1169d98c19
Upwork Job ID:<PHONE_NUMBER>720638908
Last Price Increase: 2024-08-08

Issue OwnerCurrent Issue Owner: @DylanDylann

also discussing here

No UI changes need so I will take @dubielzyk-expensify off of this.

all good Melvin

Moving forward, since we are weeks out and we're seeing a bit of confusion among new signups trying to import cards.

Proposal
Please re-state the problem that we are trying to solve in this issue.
The tab says Cards & Domains.
What is the root cause of that problem?
New feature
What changes do you think we should make in order to solve the problem?

Just change the translation and key here https://github.com/Expensify/App/blob/0559307e0b0fb0d08d454b68a21adf8b984a9553/src/languages/en.ts#L618, and whereever we use it
What alternative solutions did you explore? (Optional)

Proposal
Please re-state the problem that we are trying to solve in this issue.
The tab says Cards & Domains.
What is the root cause of that problem?
New feature
What changes do you think we should make in order to solve the problem?

We can add a new translation key just for Domains, then use it in here. Meanwhile in InitialSettingsPage here, it can stay as Cards & Domains with existing translation key

Once workspace feeds is live

If this means it should show Domains only if there's active workspace, otherwise still shows Cards & Domains, then we can make the translation key here conditional based on if there's any active workspace or not, the check for this can be found in here (Policy.hasActiveChatEnabledPolicies(allPolicies))
What alternative solutions did you explore? (Optional)
Update the translation key and value here to Domains https://github.com/Expensify/App/blob/0559307e0b0fb0d08d454b68a21adf8b984a9553/src/languages/en.ts#L618, and update the key whereever we use it

Once workspace feeds is live
If this means it should show Domains only if there's active workspace, otherwise still shows Cards & Domains

@kevinksullivan Could you clarify on this point? Thanks

Proposal updated to clarify some points

Could you clarify on this point, is the understanding correct? Thanks

@nkdengineer updated OP, since we are doing this now and not waiting for the feature I mentioned.

If this means it should show Domains only if there's active workspace

Proposal updated to remove above optional part because it was confirmed here that we do not need it.

The solution is straightforward. Let's go with @nkdengineer's proposal because they go first. I am curious about AllSettingsScreen because I don't see we use this component in anywhere. @nkdengineer Could you help to verify that?
🎀 👀 🎀 C+ reviewed

@chiragsalian it seems @mountiny will take care of this issue as mentioned here

@ShridharGoel Thanks for your reminder. But I think it is so minor, we can handle detail in the PR phase

@ShridharGoel To be fair, I clearly noted that

If this means it should show Domains only if there's active workspace

That means it's an optional part of my proposal and will only be applied if the understanding is correct. I also acknowledged that my understanding might not be correct, that's why I put it as an optional part, the main solution still stands. This is a very straight forward issue so I think we don't need to discuss further here, my initial proposal already has all the details needed to get it done.
cc @DylanDylann

@chiragsalian Bump on your assignment

@nkdengineer

I am curious about AllSettingsScreen because I don't see we use this component in anywhere. @nkdengineer Could you help to verify that?

Wdyt about this point?

@DylanDylann Yes we indeed don't seem to use that component. It looks like a legacy component, I think we should update it too for consistency, but that's optional.

Kind bump @chiragsalian for assignment (C+ review comment is here). Thx!

oh sorry i got a bit confused. I thought the reason @mountiny assigned to himself was to tackle it internally since it was a small copy change. I'll ask in slack here if he's going to work on it or not.

Yeah sorry for the confusion, I have asked @kevinksullivan if this is ready to be worked on but I dont think we are ready to remove the Cards part yet.
I wanted to tackle this on my own as its not really a change worth external job and a payout due to its triviality

@mountiny The issue was made External and Help Wanted so contributors spent time working on it (and there's C+ approved proposal). So we'd appreciate if it remains as External 🙏

Not overdue.

Not overdue.

@mountiny @kevinksullivan Any update here?

I dont think we want to make this change yet

@nkdengineer I can keep it external but for lower bounty as its trivial

I agree this should hold on the workspace feeds rollout.

Wait what? Can we please prioritize this? We're confusing lots of users who are looking for personal card import. I don't think this has anything to do with whether workspace feeds are ready or not.

Said differently, I think we should make this change now. Cards and Domains is inaccurate so we should fix this ASAP. The feature someone is accessing is domains/domain control. Let's make that clear and change this to Domains now, not later.

We're confusing lots of users who are looking for personal card import.

Are they? I just know this convo came up on the topic of (coming soon) workspace feed card features in the workspace. At which point, it's confusing to point to card management elsewhere with "Cards & Domains". Until then, we don't have any visibility for card settings.
Not passionate, we're talking a week or so until we take it out of beta.

Right, but I saying that we don't have card functionality on NewDot either way. Again, looking at Fullstory sessions, this is consistently a point of confusion. People click this button and immediately come back and start clicking somewhere else.

Created a draft for this, just simple copy changes https://github.com/Expensify/App/pull/47119

So I don't think this is a solution to anything in particular, so I think we should remove it. Whether or not we have workspace feeds doesn't change anything for me. You still need to create a workspace and then go to More Features to find company feeds regardless.

They are looking for something that we don't have in my opinion, which is personal card import.

Alright, or they're just early exploring the features and not ready to connect cards yet.

Created a draft for this, just simple copy changes https://github.com/Expensify/App/pull/47119

Ship it 👍

Shipped
@nkdengineer sorry this was just quick fix

@shawnborton wanted to ask if we should also update the icon for the Domains now that we only link to domains and no cards

I like the idea of using the globe icon only here, but I'll let Mr Borton have the call.

Agree with the global icon only, though I don't have approval power over Mr. Borton.

I can use the Globe icon, I am not Borton though so I am worried to see what might happen

Maybe we push the button and if Mr Borton is mad tomorrow we can re-assess.

See ya'll in detention!

Hi there, Mr. Borton here, and I like where you all landed. Class dismissed.

All good to be closed here

@anmurali @mountiny It seems we missed to process payment on this issue

@DylanDylann, can you please remind for what the payment is? I only see internal PR with internal review

@mountiny Ahh, I noticed these comments. Please help to review again 😄 Sorry If I misunderstand your means

https://github.com/Expensify/App/issues/45021#issuecomment-2268911688
https://github.com/Expensify/App/issues/45021#issuecomment-2275508478

I think this was one line copy change and there was no pr either. sorry that there was miscommunication about if this should be external or not, but I think we should just move ahead for this one as it was not complex task


421 Misdirected Request with FQDN (domains with trailing dot) with SAN

This is tightly related to https://bz.apache.org/bugzilla/show_bug.cgi?id=58007 which is still reproducible on latest httpd 2.4.x series. Possibly interesting as well: https://bz.apache.org/bugzilla/show_bug.cgi?id=60969
When setting mod_md in a .conf file, even when adding all md_* settings before virtual hosts, it's triggering this issue when trying to open e.g. a 2nd virtualhost fqdn from browser, e.g. https://foo.example.org. after a https://bar.example.net. is set, like it was setting per-vhost instead of a common setting for all of them. This could even like a mod_ssl limitation. I can confirm all FQDNs work properly when using mod_ssl without per-vhost ssl settings. Anyways, when testing TLS with external clients, no SNI looks involved.
Maybe it's better to fix this behavior in mod_ssl or httpd instead of working around mod_md, however it was not possible to find a workaround to make mod_md work with FQDNs.

All virtual hosts were configured without trailing dot, so is not related to configuration that requires documentation updates. The trailing dot triggers the issue when added client side (e.g. browsers).
Problem with mod_md is, if you configure certificates with two or more subdomains, the subsequent virtualhost won't load if you add a trailing dot when accessed from the browser. E.g. this pair works as expected, but in affected subdomains the fqdn version would return HTTP 421 if used mod_md:
https://httpd.apache.org/
https://httpd.apache.org./ (fqdn version)
For example:
MDomain example.org www.example.org

<VirtualHost *:443>
    ServerName example.org
    DocumentRoot /var/www/example.org
</VirtualHost>

<VirtualHost *:443>
    ServerName www.example.org
    DocumentRoot /var/www/example.org
</VirtualHost>

Browsing to https://www.example.org./ would fail. Similar configuration with mod_ssl will work unless you add SSL settings inside virtual hosts, e.g. downgrading to TLS 1.2 for a specific virtual host. This seem to happen with mod_ssl when setting SSL* configuration parameters on virtual hosts that potentially imply SNI, even when apparently there is no SNI involved if unused (client certificate authentication with TLS reauthentication, for example).
The trailing dot on clients is mostly unknown nowadays (and most sites have broken support on them), however it is a protection for the search path spoofing attack: on a malicious network with bad DHCP, the assigned DNS resolver could fail to resolve e.g. https://example.com/ and have set a search domain, e.g. badsearchdomain.example, making the request forwarding to https://example.com.badsearchdomain.example/, while https://example.com./ would prevent that redirection and just failing to resolve. The user might not realize of the domain change and get scammed. DNSSEC does not help here if the DNS server has it disabled and the local resolver does not enforce it (usual default on most OSes).
I did not test this with MDBaseServer on, could potentially workaround this issue. However, I'm not using mod_md currently due to this issue for further retries.

Sorry, but I still fail to see how SNI and virtual host selection has anything to do with mod_md. The module "just" provides certificates to mod_ssl and  that module does select the server config to use based on the SNI and its configuration.
If you see something wrong with that behaviour, you should open an issue against httpd itself and mod_ssl in particular. The 421 response, internally the define HTTP_MISDIRECTED_REQUEST, is used in http_protocol.c and ssl_engine_kernel.c only. You might have observed 421 in your tests with a config involving mod_md, but that does not make the module do anything wrong.
If I am wrong on this, please point me to the code that you regards as faulty.

Thanks, then I'll add some comments on https://bz.apache.org/bugzilla/show_bug.cgi?id=58007 on how to reproduce this issue in detail in latest version yet, and how mod_md gets affected, besides how to reproduce it with plain mod_ssl and how to work around it.


Ensure we never property check lists longer than 10.

Previously, we could generate intermediate computations using lists of
length longer than 10, since we only checked if the final result was
longer than 10. Example of a test that actually failed:
(enforce
  (=
    (make-list 10 0)
    (drop 1 (reverse (make-list 11 0)))
    ""))

The new method of generating lists only generates valid lists of length
<= 10. We do this by only allowing integer literals as the length
argumenent to make-list. The downside is that we rule out a bunch of
valid programs.
Ways lists are generated:

make-list: we only make-list lists of a valid length
reverse / sort: these preserve length
concat: for each sublist we halve the allowed length
drop / take: these always return a list no longer than their
argument.

Good call -- this solution seems reasonable to me.


Fix compatibility with Swift 5.6

extension Foo<Bar> was not a thing in Swift 5.6.
Checklist

[X] I've read the Contribution Guidelines
[X] My contributions are licensed under the Swift license.
[X] I've followed the coding style of the rest of the project.
[ ] I've added tests covering all new code paths my change adds to the project (if appropriate).
[X] I've verified that my change does not break any existing tests.
[ ] I've updated the documentation if necessary.

@swift-ci test

@swift-ci test macOS platform


mcomix: Update to version v2.0.2

[x] I have read the Contributing Guide.

/verify

I think we can add autoupdate for this app.
    "checkver": {
        "url": "https://sourceforge.net/projects/mcomix/files/",
        "regex": "files/MComix-([\\d.]+)"
    },
    "autoupdate": {
        "url": "https://downloads.sourceforge.net/project/mcomix/MComix-$version/mcomix-$version.win64.all-in-one.zip"
    }

/verify