Create wiki pages for CobiGen debugging methods#1605

Adresses #1605
Implements:

Integration test debugging procedures
CLI attach to process method
Eclipse Runtime debugging

@devonfw/cobigen

Closed because the PR is missing the points from the issue, it was supposed to be about a guide specific to CobiGen debugging.


edit internet code for new data pipeline

Why this PR
Why we need this PR?
Changes

Write some changes here

Related Issues
Close #
Checklist

[ ] PR should be in the Naming convention
[ ] Assign yourself in to Assigneees
[ ] Tag related issues
[ ] Constants name should be ALL_CAPITAL, function name should be snake_case, and class name should be CamelCase
[ ] complex function/algorithm should have Docstring
[ ] 1 PR should not have more than 200 lines changes (Exception for test files). If more than that please open multiple PRs
[ ] At least PR reviewer must come from the task's team (model, eval, data)

I will edit readme.md soon krub

The processed datasets are in /project/lt200056-opgpth/mond_tmp_datasets krub with format according to new data pipeline.
I have to put them there cause I have no permission to write in /project/lt200056-opgpth/datasets krub. But we can move them soon.


[Dart][example] tutorial/toh-5: hosted sample is always "Loading..."

When I go to http://angular-examples.github.io/toh-5/, it just says "Loading..." Looking at the console, I see these messages:
http://angular-examples.github.io/styles.css Failed to load resource: the server responded with a status of 404 (Not Found)
http://angular-examples.github.io/main.dart.js Failed to load resource: the server responded with a status of 404 (Not Found)
http://angular-examples.github.io/styles.css Failed to load resource: the server responded with a status of 404 (Not Found)

Note the lack of toh-5/ in those URLs. Those files are in https://github.com/angular-examples/toh-5/tree/gh-pages. For example, http://angular-examples.github.io/toh-5/styles.css works. Ah, routing. Perhaps we can change the example to use a base that works for the hosted version? We should probably put a visible note about this in the index.html, since this is a common stumbling block.
cc @chalin

I've identified the problem. Will fix shortly.

Oups, sorry, I had not read the full text of the issue before I signed off yesterday (and seen that you outlined the problem :). Will look into this shortly.

Temporarily fixed by manually changing the base href to
<base href="/toh-5/">

Let's leave this open until we find a permanent solution (where the dart-doc-syncer can generate the proper index.html).

Fixed by https://github.com/angular/dart-doc-syncer/pull/14.


fix type errors in setup.py

Pull Request Test Coverage Report for Build 304

0 of 0 (NaN%) changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 94.858%

Totals

Change from base Build 297:
0.0%

Covered Lines:
1771

Relevant Lines:
1867

💛  - Coveralls


Unable to create account

Steps to recreate:

Go to UAT site and attempt to create account:
http://elyu1.undergroundtheater.org:8080/login
Fill out form (all fields)
Submit form

Get 403

note, fully aware this is likely a configuration issue, but if so, it should be documented as a gotcha at the very least. :)

Note: account IS created, but it appears it's not counting me as "authenticated" after the post processes.

Fixed.


Simulation Relative import errors

Environment

OS: [Windows 10 Home ]
Python version: 3.10.0

To reproduce
Steps to reproduce the behavior:

pip install --upgrade "mosaicml-streaming[simulator]"
simulator

Expected behavior
After initial installation during main command execution experienced multiple issues with relative import statements not working. After editing code from things like from core.sim_dataset import SimulationDataset to from simulation.core.sim_dataset import SimulationDataset I was able to get the simulator to run

I've a PR open to address this in https://github.com/mosaicml/streaming/pull/838

PR https://github.com/mosaicml/streaming/pull/838 merged, closing issue


validation loops run the partial dataset with horovod

Hello,
It seems to be the same issue as https://github.com/PyTorchLightning/pytorch-lightning/issues/1161.
When I use horovod, validation_step and validation_epoch_end are called multiple times).
Thank you.

@tgaddair pls ^^

I'll take a look.

Hey @thnkim, can you provide a minimum reproducible example that demonstrates the behavior you're describing?
I just ran quick test with an MNIST dataset.  With 1 GPU, it ran 3750 training steps and 1875 validation steps per epoch.  With 2 GPUs, it ran 1876 training steps and 938 validation steps per worker, which is consistent with the expected behavior.

Hi @tgaddair!
Thank you. It looks I'm missing something.
As you mentioned, with 2 GPUs and horovod, my 1901 validation samples are splitted to 951 for one GPU and 951 (not 950 here) for the other.
Then validation_epoch_end() is called two times; their outputs are as follows:
Validation Accuracy: 95.0578% (904/951)
Validation Accuracy: 94.9527% (903/951)

I have two questions:

How can I merge these two outputs into one?
Since both processes handled 951 samples, I guess there will be one duplicate. Isn't it problematic?

Thank you!

Hey @thnkim, to answer your questions:

With Horovod, every worker process is going to call validation_epoch_end() separately, which is why you're seeing it called twice (for -np 2).  If you want to only do something on one of the workers, you can write some Horovod-specific code like this:

import horovod.torch as hvd

...

    def on_validation_end(self, outputs):
        if hvd.rank() == 0:
            # do something only on the first process
            ...

That's the behavior of PyTorch's DistributedSampler, which is what PyTorch Lightning will use to distribute the dataset if you don't provide a sampler yourself.  So one option would be to create your own sampler if it becomes an issue, but in practice it shouldn't be a problem (the oversampled elements will change each epoch anyway).  This would be a good change to PyTorch itself, though, to allow DistributedSampler to not pad the lists to be the same length for every worker.

Thank you, @tgaddair!
I guessed merging the validation results from multiple processes are internally merged.
I did it using hvd.allreduce(). :)
And for DistributedSampler, yes it would not be problematic in my case.
Thank you again!


[Bug]: Incorrect Output Strings Camel Case

Contact Details
<EMAIL_ADDRESS>What happened?
A bug happened!

The objective of the code is to convert a given string to CamelCase format, which is not resolved in this code, as you can see in the output.
Input: hii this is camel case
Output: HiiThisIsCamelCase
Expected Output: hiiThisIsCamelCase
Please send me this issue so I can work on it as soon as possible.
Version
1.0.2 (Default)
What browsers are you seeing the problem on?
No response
Relevant log output
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

So is this error in all languages?

In C++, the output matches with the expected output.
in Python, C and Java the output doesn't match the expected output.
Java
[cid:8b7fe4fe-1a4f-4698-ad7f-59707385288a]
C
[cid:886d2235-06ee-41f7-92d9-7d0af482f2e7]
AND the Python I have already given that on the issue.
I would be installing JDK and MSYS2 for Java and C/C++ for doing my work completely on VS Code.
I have intermediate level experience in all 4 languages so I can do this.

From: Kumar Laxmikant @.>
Sent: 02 July 2023 16:42
To: Kumar-laxmi/Algorithms @.>
Cc: Pavan Sriram Kannuri @.>; Author @.>
Subject: Re: [Kumar-laxmi/Algorithms] [Bug]: Incorrect Output Strings Camel Case (Issue #1656)
So is this error in all languages?
—
Reply to this email directly, view it on GitHubhttps://github.com/Kumar-laxmi/Algorithms/issues/1656#issuecomment-1616603282, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUVXCAMOO2AE4DTDGKZ46YDXOFJQ5ANCNFSM6AAAAAAZ3LJKYA.
You are receiving this because you authored the thread.Message ID: @.***>

Assigned!
@SportingDot1024 : Bug


Updated iOS deployment target to 8.0, fixed ambiguous responseString err...

...or

Thanks for your PR @kiancheong.
These changes have made their way into the master branch as part of the 1.2.0 release. The responseString change was actually fixed inside Alamofire itself. Thanks again and feel free to close this out whenever you get a chance.

Thank you for your PR, @kiancheong. This should now be fixed in master, for the 1.2.0 release.


Suppress -std=c++0x option

Followup-to: https://github.com/sass/node-sass/pull/2553

As it turns out, we need it for g++47 for node v0.10 and v0.12.

@saper can you please rebase this on the v5 branch?

@saper can you please rebase this on the ~v5~ next-major branch?

I guess it did merge without issues?

Yep all good.
On Sat, 9 Nov 2019, 10:26 am Marcin Cieślak<EMAIL_ADDRESS>wrote:

@saper https://github.com/saper can you please rebase this on the v5
next-major branch?
I guess it did merge without issues?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/sass/node-sass/pull/2761?email_source=notifications&email_token=AAENSWBP6WD7MUTQQDAWR4TQSXYRVA5CNFSM4JC3NPCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDTVAAI#issuecomment-552030209,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAENSWGT6JYA4NLRCFVT6BDQSXYRVANCNFSM4JC3NPCA
.


Implement v1/artists/{id}/albums in ArtistsApi

Info
Implement a call for the v1/artists/{id}/albums endpoint in the ArtistsApi class. I believe this is the last remaining endpoint that needs to be implemented for artists, so that would wrap up coverage there.
Follow existing patterns to have an overload returning T to let the consumer handle conversion, and one returning Album[]. Add an integration test for each method.
Reference
https://developer.spotify.com/documentation/web-api/reference/artists/get-artists-albums/

Any updates? I'm kinda in need of this for something I'm working on so it would be quite helpful

Hey @Starman0620  - I will have this done by Wednesday.

@Starman0620 there is a preview release for this: https://www.nuget.org/packages/SpotifyApi.NetCore/3.1.0-preview
I just have tests to complete before 3.1.0 release.

Completed in v3.1.0 https://github.com/Ringobot/SpotifyApi.NetCore/releases/tag/3.1.0

Thanks for taking care of this @DanielLarsenNZ!


How to implement Logout functionality

As there is not inbuilt authentication flow, i added login page with passport.js local strategy based auth flow.
but how to achieve logout functionality with this now?

don't know how bull board UI is coded, but if possible we can let user provide their own logout function.
so when user setup bull-board server, we can take a flag from user which signifies if user need logout or not.
if that flag is passed as true, Logout button will be shown in dashboard header somewhere.
also take callback function from user, which will be called as action of logout button click.

It is not so simple to pass a function from backend (the only place that you have access to) to the client...
If you will be able to pass a custom links, you will be able to link to your logout logic api...

I'm also coming across this issue. I think that as you've suggested @felixmosh where if you can just define in uiConfig an array of custom links that would be mapped as a set of buttons in the header or in a dropdown in the header. Then we just define a logout route that clears the auth cookie and redirects to the login page.

Yeah this is the path that I think to make...

hey, any update on this?


Create index.md

Thank you for contributing! Please confirm this pull request meets the following requirements:

[X] I followed the contributing guidelines: https://github.com/github/explore/blob/master/CONTRIBUTING.md
[X] I have no affiliation with the project I am suggesting (as a maintainer, creator, contractor, or employee).

Which change are you proposing?

[ ] Suggesting edits to an existing topic or collection
[X] Curating a new topic or collection
[ ] Something that does not neatly fit into the binary options above

Editing an existing topic or collection
I'm suggesting these edits to an existing topic or collection:

[ ] Image (and my file is *.png, square, dimensions 288x288)
[ ] Content (and my changes are in index.md)

Please replace this line with an explanation of why you think these changes should be made.

Curating a new topic or collection

[X] I've formatted my changes as a new folder directory, named for the topic or collection as it appears in the URL on GitHub (e.g. https://github.com/topics/[NAME] or https://github.com/collections/[NAME])
[X] My folder contains a *.png image (if applicable) and index.md
[X] All required fields in my index.md conform to the Style Guide and API docs: https://github.com/github/explore/tree/master/docs

Golfing languages are getting more popular on GitHub, and people need to know what a golfing language is

Something that does not neatly fit into the binary options above

[ ] My suggested edits are not about an existing topic or collection, or at least not a single one
[ ] My suggested edits are not about curating a new topic or collection, or at least not a single one
[ ] My suggested edits conform to the Style Guide and API docs: https://github.com/github/explore/tree/master/docs

Please replace this line with an explanation of your proposed changes.

Please note: we will close your PR without comment if you do not check the boxes above and provide ALL requested information.

@RocketScienc can you update your fork with the latest main? There is a fix there for the failing CI test.


credits.txt

Describe your problem here.
what folder do i put the credits.txt in?
Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows x64
Did you edit anything in this build? If so, mention or summarize your changes.
nope

mods/data


Missing character on results screen

On a Pixel 3 / Android 10 with Chrome, the results screen at http://emojicompass.com/ displays a missing character square instead of the tildes between lines

Confirmed it's not happening on desktop Chrome w/ Windows 10

That character is not actually a tilde but the `U+2053 SWUNG DASH character. It appears that it's not supported by Gentium Book Basic, which is a very limited subset of the most common characters. Neither the swung dash nor the more common tilde is in their chart of glyphs.
I'm not certain why this character displayed properly on every other system but this one. Perhaps other browsers or devices use the fallback serif font for missing characters automatically (this is not behavior I'm aware of, but it's certainly possible).
However, Gentium Plus is still open source and licensed the same as the Basic version. It's likely we can swap out the original font for the plus version of the files for the desired look. The plus version of this font is much larger file, but the additional loading time might not be a problem if we're bundling for apps or PWAs. Alternatively, with a little more effort, we can decide to subset the Plus font so that it's basically just the Basic set plus the swung dash.


References from previous report included in current report?

I have just done two GPT-researcher reports in succession using the web interface, with the master branch (hash: 557207f424ccc92b84eca3ec9a15182f66e5ed43)
Setup:

AzureOpenAI
Bing search

There are many many references cited in the list of reference in Report 2 that were originally found for Report 1.
Seems to be a bug?

Two important questions please:

First important question: Will GPT Researcher deployed on the RepoCloud also be able to synchronize updates once these improvements are completed?
The second important question: How does GPT Researcher deployed on the RepoCloud use local documents for knowledge QA? After making sure to add the DOC_PATH environment variable pointing to the documents folder.

I have just done two GPT-researcher reports in succession using the web interface, with the master branch (hash: 557207f)
Setup:

AzureOpenAI
Bing search

There are many many references cited in the list of reference in Report 2 that were originally found for Report 1.
Seems to be a bug?

I had a similar issue; using Tavily with OpenAI. The references at the end seemed mixed with two earlier searches.
I wonder if possibly I had multiple sessions (chrome tabs) open? Or is it potentially something not being cleared between runs.

I asked CVhatGPT to review; here is the shared chat: https://chatgpt.com/share/09b69dec-842e-4601-afe6-fa1029621ceb
Findings from ChatGPT:
"The conduct_research method initializes the research process and sets up the context for the research task. However, it does not explicitly reset or clear the visited_urls set or the source_urls list before starting new research. This could lead to references from previous runs being carried over.
To address this issue, we should ensure that visited_urls and source_urls are cleared at the start of each research task. This can be done by modifying the conduct_research method to reset these attributes at the beginning.
Here is a proposed modification to the conduct_research method to include resetting visited_urls and source_urls:
`async def conduct_research(self):
"""
Runs the GPT Researcher to conduct research
"""
# Reset visited_urls and source_urls at the start of each research task
self.visited_urls.clear()
self.source_urls = []
if self.verbose:
    await stream_output("logs", f"🔎 Starting the research task for '{self.query}'...", self.websocket)

# Generate Agent
if not (self.agent and self.role):
    self.agent, self.role = await choose_agent(query=self.query, cfg=self.cfg,
                                               parent_query=self.parent_query, cost_callback=self.add_costs)

if self.verbose:
    await stream_output("logs", self.agent, self.websocket)

# If specified, the researcher will use the given urls as the context for the research.
if self.source_urls:
    self.context = await self.__get_context_by_urls(self.source_urls)
elif self.report_source == ReportSource.Local.value:
    document_data = await DocumentLoader(self.cfg.doc_path).load()
    self.context = await self.__get_context_by_search(self.query, document_data)
# Default web-based research
else:
    self.context = await self.__get_context_by_search(self.query)

time.sleep(2)
if self.verbose:
    await stream_output("logs", f"Finalized research step.\n💸 Total Research Costs: ${self.get_costs()}", self.websocket)

return self.context

`

Quick test seems to validate this addresses the error; in agent.py add the lines to reset visited_urls and source_urls.
I'll create a pull request.
async def conduct_research(self):
"""
Runs the GPT Researcher to conduct research
"""
# Reset visited_urls and source_urls at the start of each research task
self.visited_urls.clear()
self.source_urls = []
    if self.verbose:
        await stream_output("logs", f"🔎 Starting the research task for '{self.query}'...", self.websocket)

Thanks for contributing a fix for this!


[DNS Resolver] Bump from 2020-04-01-preview to 2022-07-01 via CodeGen v2

Remove SDK

This checklist is used to make sure that common guidelines for a pull request are followed.

For new extensions:

[ ] My extension description/summary conforms to the Extension Summary Guidelines.

About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update src/index.json automatically.
The precondition is to put your code inside this repository and upgrade the version in the pull request but do not modify src/index.json.

/azp run

DNS Resolver


Can you help add feature support type UUID, DateTime, Decimal and Interval

Because Tarantool has format MP_EXT abit of type UUID, DateTime, Decimal and Interval can you help add those features
Thanks

Can you elaborate where exactly that should be supported?
Currently all de/serialization is done via rmpv::Value. You can serialize any data into Ext. I can probably add some utility functions to do just that. But I don't know whether they support deserialization from such format or not, and I don't know would it be possible to add that support without PR into rmpv.
In future probably it would be a good idea to move from rmpv to something custom, but I am not sure when that would happen.

Types UUID, DateTime, Decimal and Interval good in some use case like API enpoint expose URL end point to end user for security, Decimal for financial system..,
There are two challenges:

Serialize pack send request to Tarantool any space field type: UUI, DATETIME, DECIMAL need to follow officail document Tarantool MessagePack extensions
Deserialize when get response from Tarantool need implement follow requirement too

Thank you


When you remap {backward} and {forward}, portal labels keys don't work

For example, I have this setup. When I remap backward/forward, the labels keymap don't work
query = { "modified", "different", "valid", "grapple" },
labels = { "j", "k", "h", "l" },
backward = "",
forward = "",
integrations = {
grapple = true,
},

https://github.com/cbochs/portal.nvim#configuration

   ---Keycodes used for jumping forward and backward. These are not overrides
   ---of the current keymaps, but instead will be used internally when a jump
   ---is selected.
   backward = "<c-o>",
   forward = "<c-i>",

These are not meant to be remapped unless you have a good reason to. There are no default keymaps for Portal. For keymap suggestions, please see the suggested keymaps section.


llama_self_extend_patch_4_36 is not work

when I use 4.36.2, it's not work. But if I use 4.32.0, it's work.
I only changed "import llama_self_extend_patch as LlamaSE" in "llama_example.py" to "import llama_self_extend_patch_4_36 as LlamaSE"

We found that after 4.36, the default attention of llama changed from "LlamaAttention" to "LlamaSdpaAttention". Hence the replacement function will not work. Instead, you may try:
modify_method_of_instance(base_model, "LlamaAttention", "forward", self_extend_forward)
--> modify_method_of_instance(base_model, "LlamaSdpaAttention", "forward", self_extend_forward)
This might be the reason for the failure.

We found that after 4.36, the default attention of llama is changed from "LlamaAttention" to "LlamaSdpaAttention". Hence the replacement function will not work. Instead, you may try:
modify_method_of_instance(base_model, "LlamaAttention", "forward", self_extend_forward) --> modify_method_of_instance(base_model, "LlamaSdpaAttention", "forward", self_extend_forward)
This might be the reason for the failure.

it work, thank you.
I have another question. I want to add it here. but it can still only run normally on 4.32, and the running result of 4.36 is still wrong.
I just added the following three pieces of code and used this command to run them: CUDA_VISIBLE_DEVICES=0,1 python eval/passkey.py --model /data/supry/models/llama-2/llama2-7b-hf --min-tokens 4096 --max-tokens 8192 --tokens-step 4096 --length-step 1024 --iterations 20 --serope

Hi YL-9! Could you please test whether self-extend can work by instance wise modification, like the example we provide? Sometimes, direct modification to the transformers' class does not take effect, while the cause of failure is case by case. That's the reason why we choose to modify the forward function of a model instance rather than its class. (Of course,, this can avoid any unexpected behavior for the modification only happens to the specific model instance)

ok, thank you!

Hi, thanks for the nice work! I see the current implementation in llama_self_extend_patch_4_36.py is regular pytorch. I wonder if you plan to implement Flash attention for transformers==4.36?

Hi, thanks for the nice work! I see the current implementation in llama_self_extend_patch_4_36.py is regular pytorch. I wonder if you plan to implement Flash attention for transformers==4.36?

Hi, thank you for your interests. The main different between transformers==4.36 and transformers==4.38.2 is how the RoPE is applied to KV. You may have a check. The computation of self attention is nearly the same. This means you can follow our 4.38.2 implementation to have a flash attention implementation for 4.36 with minor modification.
One of the possible issues is the flash_attn version used by 4.36. In that case, you may use our triton flash attention implementation instead of flash_attn. It's 10~20% slower than flash_attn.


Make \Tuf\KeyDB::computeKeyIds() private

I am not sure why this method is public. Change to private and changed the test to test \Tuf\KeyDB::createKeyDBFromRootMetadata()

Can we freshen this code from main to get the tests passing? I suspect main already has the fix.

This was fixed in #95


Memory leak (IDFGH-4300)

https://github.com/espressif/esp-idf/blob/178b122c145c19e94ac896197a3a4a9d379cd618/components/esp_https_server/src/https_server.c#L162
Memory is allocated, but never freed when error occurs.

@cas1no Thanks for reporting. Would you please help provide more details as suggested in the issue template? Information like elf, sdk configuration, backtrace, log outputs, commit ID, hardware and etc. would help us debug further. Thanks.

@Alvin1Zhang that is not necessary since the problematic code is linked

@negativekelvin Thanks for the notes. @cas1no Thanks for reporting, we have a fix under internal reviewing, and the issue will be closed automatically once the issue is available on GitHub. Thanks.


Fix kdl joint weights and update our branch from main

Description
Trivial change in reading joint weights in KDL kinematics plugin.
Checklist

[x] Required by CI: Code is auto formatted using clang-format
[ ] Extend the tutorials / documentation reference
[ ] Document API changes relevant to the user in the MIGRATION.md notes
[ ] Create tests, which fail without this PR reference
[ ] Include a screenshot if changing a GUI
[ ] While waiting for someone to review your request, please help review another open pull request to support the maintainers

They removed moveit_msgs and ompl from moveit2.repos, so we have to add them to our robocon.repos file...


Getting Error: Can't connect to local MySQL server through socket '/tmp/mysql.sock'

Hi,
We have deployed shield v8.0.11 .
Able to run import errand, test-store task also done, but now facing issue with backup , backup task is in failed status.
Error is as follow-
**mysqldump: Got error: 2002: "Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2 "No such file or directory")"
when trying to connect
Unable to exec '/var/vcap/packages/mariadb/bin/mysqldump': exit status 2
    EXITING 3
    shield worker 4 unable to run command against   ##IP##**

Sounds like you are using default socket paths, which won't work in a BOSH environment.
You probably want to check the mysql_options field for the mysql plugin:
http://shieldproject.io/docs/plugins/#mysql-mariadb

Hi,
We have successfully resolve the issue. The backup stores without prefix. is there any configuration where i can provides the prefix of all databases?


requirements contains pinned versions that are incompatible with current env

trying to install into our badger-ape repo, but getting
There are incompatible versions in the resolved dependencies:
  attrs==20.3.0 (from badger-utils==0.0.9->-r /var/folders/q0/wj9dn5_91y37j42gk1kls8_r0000gn/T/pipenvuh8hbawurequirements/pipenv-6meo6px5-constraints.txt (line 8))
  attrs==21.2.0 (from eth-brownie==1.16.4->-r /var/folders/q0/wj9dn5_91y37j42gk1kls8_r0000gn/T/pipenvuh8hbawurequirements/pipenv-6meo6px5-constraints.txt (line 5))

is it possible to unpin (some of) the requirements, eg using attrs>=20.3.0 or even attrs instead?
if we want to add badger-utils to our prod branch, the less (pinned) requirements the better. i have no experience publishing to pypi, but i think you just did a freeze? are all those reqs really needed, and are they needed at exactly that version?
maybe you could only include the core dependencies in requirements.txt, and let the sub dependencies follow from those?
here is our requirements-core.txt for badger-ape:
ape-safe==0.2.2
pandas
pycoingecko
rich
tabulate

all other sub dependencies follow from those five reqs.
some background: https://towardsdatascience.com/why-you-should-consider-adding-requirements-core-txt-to-your-next-python-project-2c36c1381b12

@jorijnsmit thanks for reporting the issue. Will have a look at it soon

@jorijnsmit by saying badger-api are you referring to this repo: https://github.com/Badger-Finance/badger-system ?

i never said badger-api; im talking about badger-ape. i realise now it might not be visible for you. it is a fork of badger-system we use to run multisig scripts (built around https://github.com/banteg/ape-safe).

@jorijnsmit please check new version: https://pypi.org/project/badger-utils/0.1.0/

✔ Installation Succeeded


(High) Mosaiq.Measurement.UNIT_SOURCE_VALUE - Need Code Descriptions Too

Mosaiq.Measurement.UNIT_SOURCE_VALUE appears to us a non-standard vocabulary, so I need a valid list of codes and descriptions so I can look up the associated OMOP concept code.

Fixed by sending custom mapping by spreadsheet


dep init cannot support multi GOPATH on Windows 10

my go version: go1.8.1 windows/amd64
go env:
set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=E:\projects\go\imports;E:\projects\go\study;E:\projects\go\apps
set GORACE=
set GOROOT=C:\Go
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\geo\AppData\Local\Temp\go-build117483407=/tmp/go-build -gno-record-gcc-switches
set CXX=g++
set CGO_ENABLED=1
set PKG_CONFIG=pkg-config
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2

I had multi GOPATH on windows 10，the command 'dep init'  was correctly in first GOPATH, but failed in other GOPATH:
E:\projects\go\apps\src\dps>dep init
determineProjectRoot: E:\projects\go\apps\src\dps not in any GOPATH

Thanks for reporting this @ringrid. dep should support multiple GOPATH. This seems to be a bug on windows.
Could you try running this gist and report what it outputs?

@ibrasho I had tried it, the outputs were:
absRoot: E:\projects\go\apps\src\gist
resolvedAbsRoot: E:\projects\go\apps\src\gist
absRoot == resolvedAbsRoot: true
GOPATHs: [E:\projects\go\imports E:\projects\go\study E:\projects\go\apps]
trying to detect GOPATH for E:\projects\go\apps\src\gist
        E:\projects\go\imports: false
        E:\projects\go\study: false
        E:\projects\go\apps: true
GOPATH: E:\projects\go\apps

by the way, I have already upgrade to r1.8.3 yesterday, this problem still exists.

@ibrasho I think I have found the reason of this problem. In [https://github.com/golang/dep/blob/master/internal/fs/fs.go#L69](this code). because of c.PATHs
value of c.PATHs was:
[E:/projects/go/imports E:/projects/go/study E:/projects/go/apps]
not be
[E:\projects\go\imports E:\projects\go\study E:\projects\go\apps]

in go windows version, the definition of os.PathSeparator is
const (
	PathSeparator     = '\\' // OS-specific path separator
	PathListSeparator = ';'  // OS-specific path list separator
)

not '/', so in previous code,
prefix = strings.TrimSuffix(prefix, string(os.PathSeparator))
=> []

You are quite welcome. I submitted that for another issue and happy to hear it solves yours too. 😄
Closing this since it's fixed by #778.


Normalize normal before interpolating in the mobile renderer to avoid precision errors on heavily scaled meshes

Fixes: https://github.com/godotengine/godot/issues/89101
Fixes: https://github.com/godotengine/godot/issues/86275
Fixes: https://github.com/godotengine/godot/issues/86707
We interpolate normals using half precision in the Mobile renderer. As a result, very small normals can become corrupted. We can end up with very small normals when using a small scale in a MeshInstance's transform. In all three bug reports large meshes are used with a tiny scale (0.01 or smaller), the tiny scale transforms the normals to also be tiny, then the conversion to half float breaks them.
There are two options to solve that problem:

Interpolate with full precision (this is what we do in the Forward+ renderer which is why it doesn't have issues)
Normalize the normal before interpolating so it is in a good range

I chose the second option since a single normalize in the vertex shader will be relatively cheap and we need to minimize the amount of varying data that we use as it can easily be a bottleneck on Mobile devices.

@clayjohn This fixes everything for us except 1 shader! :)  So with that news we can celebrate!  I have attached an mrp of our one shader that still has this issue:
Desktop

Android

This shader was created using Godot's shader graph.
mrp_custom_shader2.zip
And here is the shader for quick reference:

We are so close though!  The fix is almost there and this is most definitely a HUGE improvement! :)

And you may be wondering why we would create such a simple shader in the shader graph.  That actually isn't the whole shader.  That is just a stripped down version of the shader where this issue occurs.  Aka, the mrp shader 😎.  The real shader we use is animated so that the Blocky Ball logo moves across the arch in the background.  But I figured an mrp shader would be easier to debug :)

@TCROC Thank you for the MRP! Looks like we need to do the same fix for tangents and binormals (which are needed when using anisotropy). Just testing now, but I should be able to push a fix in a few minutes

@TCROC I just pushed a change that fixes this for your new MRP too! Should be good to go now

@TCROC I just pushed a change that fixes this for your new MRP too! Should be good to go now

Brilliant!  I'll go try it out now! :)

@clayjohn I can confirm that it works!  Thank you so much for the quick fix on this issue! :)  Just in case you weren't aware, I just want you to know that you are indeed the man! 😎

And the Godot community in general!  You all never fail to impress me! :)

Does it mean that after this PR is merged I should roll back the change in my fork's scene_forward_mobile.glsl precision for normal and binormal back to mediump? https://github.com/godotengine/godot/commit/cb99e40ffcf34a2c248633d4cb173000c81d9aa4#diff-94158bc3d510819bb8d7ebd62a8890eeebad49608d9e934d148b73b6b65aeff9

Thanks!


Feature that affects dev dependencies can't be enabled

Problem
I have crates A, B, C with the following manifests:
[package]
name = "a"
version = "0.0.0"

[features]
f = ["b/f"]

[dependencies]
b = { path = "../b" }

[package]
name = "b"
version = "0.0.0"

[features]
f = ["c/f"]

[dev-dependencies]
c = { path = "../c" }

[package]
name = "c"
version = "0.0.0"

[features]
f = []

Expected behavior: I believe it should be possible to build crate A. Running cargo check in crate A should build crate A without feature f and crate B without feature f. Running cargo check --features f in crate A should build crate A with feature f and crate B with feature f, but not C because that is a dev dependency of a dependency.
Actual behavior: crate A cannot be built, with or without f.
$ cargo check

error: failed to select a version for `b`.
    ... required by package `a v0.0.0 (/path/to/a)`
versions that meet the requirements `= 0.0.0` are: 0.0.0

the package `a` depends on `b`, with features: `c` but `b` does not have these features.

failed to select a version for `b` which could resolve this conflict

Possible objection: "but B/f requires C/f, we can't enable B/f without building C!" I don't think this is valid because cargo check --features f in crate B will already build crate B with feature f enabled without building the dev dependency C.

Notes
cargo 1.36.0-nightly (beb8fcb52 2019-04-30)

Is there a known workaround for this?

I ran
$ cargo new --lib a
$ cargo new --lib b
$ cargo new --lib c

And replaced the Cargo.tomls with what was in the Issee.
I then ran
$ cargo check
$ cargo check -F f

And it worked.
Seems like this was fixed (maybe when weak/namespaced features were added?).  Closing.  If there is something I missed, let us know!

Thank you!
The fix bisects to nightly-2021-11-24. The commit range is https://github.com/rust-lang/rust/compare/936f2600b6c903b04387f74ed5cbce88bb06d243...65c55bf931a55e6b1e5ed14ad8623814a7386424, in which the only Cargo update is https://github.com/rust-lang/rust/pull/91144. In there, I bet it was https://github.com/rust-lang/cargo/pull/10103, although the PR description does not directly describe the scenario in this issue.

Confirmed it was #10103.
Glad this works! Hopefully it stays that way — would it be worth reopening this issue to track adding a dedicated test, or are the tests added by #10103 focused on feature cycles good enough coverage for this case?
I'll take a look through the tests myself and send a PR.


Performance issues in your project (by P3)

Hello! I've found a performance issue in /tf_encrypted and /examples: batch() should be called before map(), which could make your program more efficient. Here is the tensorflow document to support it.
Detailed description is listed below:

tf_encrypted/protocol/aby3/aby3.py: .batch(batch_size=batch_size)(here) should be called before .map(decode)(here).
examples/securenn/conv_convert.py: .batch(batch_size)(here) should be called before .map(decoder)(here) and .map(normalize)(here).
examples/federated-learning/run.py: dataset = dataset.batch(50)(here) should be called before dataset = dataset.map(decode)(here) and dataset = dataset.map(normalize)(here).
examples/federated-learning/run.py: dataset = dataset.batch(self.BATCH_SIZE)(here) should be called before dataset = dataset.map(decode)(here) and dataset = dataset.map(normalize)(here).
examples/federated-learning/convert.py: .batch(batch_size)(here) should be called before .map(decode)(here) and .map(normalize)(here).
examples/mnist/run.py: dataset = dataset.batch(self.BATCH_SIZE)(here) should be called before dataset = dataset.map(decode)(here), dataset = dataset.map(normalize)(here) and dataset = dataset.map(flatten)(here).
examples/mnist/run.py: dataset = dataset.batch(self.BATCH_SIZE)(here) should be called before dataset = dataset.map(decode)(here) and dataset = dataset.map(normalize)(here).
examples/mnist/convert.py: .batch(batch_size)(here) should be called before .map(decode)(here) and .map(normalize)(here).
examples/notebooks/keras-training/common.py: .batch(self.batch_size)(here) should be called before .map(norm)(here).
examples/notebooks/keras-training/common.py: .batch(self.test_set_size)(here) should be called before .map(norm)(here).
examples/logistic/common.py: .batch(self.batch_size)(here) should be called before .map(norm)(here).
examples/logistic/common.py: .batch(self.test_set_size)(here) should be called before .map(norm)(here).
examples/logistic/data.py: .batch(batch_size)(here) should be called before .map(norm)(here).
examples/logistic/data.py: .batch(batch_size)(here) should be called before .map(norm)(here).

Besides, you need to check the function called in map()(e.g., norm called in map()) whether to be affected or not to make the changed code work properly. For example, if norm needs data with shape (x, y, z) as its input before fix, it would require data with shape (batch_size, x, y, z).
Looking forward to your reply. Btw, I am very glad to create a PR to fix it if you are too busy.

I will close this issue for now.


zchunk: 1.1.8 -> 1.1.9

Motivation for this change
Update to the newest release 1.1.9 and remove the patch that was previously needed to get it to compile on darwin.
Things done

Switch tag to 1.1.9 from 1.1.8
Remove applied patch

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[X] macOS https://gist.github.com/nima2007/856e32b2c6bb2c2d9f715f6243567cc8
[X] other Linux distributions https://gist.github.com/nima2007/2e8c8e565c16c42f78f72641c9f29fe8

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[X] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[X] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 108441 run on x86_64-linux 1

3 packages failed to build:

libdnf
microdnf
rpm-ostree

5 packages built:

librepo
python37Packages.librepo
python38Packages.librepo
python39Packages.librepo
zchunk

Result of nixpkgs-review pr 108441 run on x86_64-linux 1

3 packages failed to build:

libdnf
microdnf
rpm-ostree

5 packages built:

librepo
python37Packages.librepo
python38Packages.librepo
python39Packages.librepo
zchunk

Result of nixpkgs-review pr 108441 run on x86_64-linux 1

3 packages failed to build:

libdnf
microdnf
rpm-ostree

5 packages built:

librepo
python37Packages.librepo
python38Packages.librepo
python39Packages.librepo
zchunk

Result of nixpkgs-review pr 108441 run on x86_64-linux 1

3 packages failed to build:

libdnf
microdnf
rpm-ostree

5 packages built:

librepo
python37Packages.librepo
python38Packages.librepo
python39Packages.librepo
zchunk


Automated Package Upgrade

:tada: This PR is included in version 7.7.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


fix email link

Fixed email with subject hyperlink, the spaces did not render the link correctly.

@flyingzumwalt do we have a better email for this now?

We should probably remove that section entirely. For now, the more accurate info is
Protocol Labs occasionally is able to hire developers for part time or full time positions, to work on IPFS. If you are interested, check out [the job listings](http://protocol.ai/join). If you'd like to help in other ways, propose your ideas on the IPFS forums at https://discuss.ipfs.io.

@flyingzumwalt done!

Thank you all!


[Feature-request] Being able to serialize and deserialize a request.

Hello,
I found your Crate very useful, tho, in my opinion, there is a problem with it, we can't serialize and deserialize the http::Request and http::Response objects, but it would be very handy !
I thought about something like this:
let request = Request::get("https://google.fr")
    .body(());

let rawrequest = request.as_raw();
/* rawrequest content:
 * GET https://google.fr HTTP/1.1\r\n
 * \r\n
 */

and
let rawresponse = r#"
HTTP/1.1 200 OK\r\n
Connection: Closed\r\n
...
\r\n
<Google Content>
"#
let response = Response::from_raw(rawresponse);

This would permit very low-performance-cost http requests over UnixStream and TcpStream !

Thanks for the request! However, this is a duplicate of one that already explains why it feels out of scope for this crate: https://github.com/hyperium/http/issues/81


Add three examples

Description of the problem

This PR adds the Rayleigh-Benard convection example. This PR is the first step of a larger work (adding three examples).

Future changes

Add Rayleigh-Taylor instability example
Add Marangoni effect example

If @CaroleAnneDaunais is cool with it, we can merge Shahab :)


Add rerank

Codecov Report
Attention: 18 lines in your changes are missing coverage. Please review.

Comparison is base (7e6f6c3) 89.13% compared to head (fce5b1f) 87.71%.
 Consider uploading reports for the commit 404fdce to get more accurate results

Files
Patch %
Lines

libs/infinity_emb/infinity_emb/infinity_server.py
46.42%
15 Missing :warning:

...finity_emb/infinity_emb/fastapi_schemas/convert.py
40.00%
3 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #50      +/-   ##
==========================================
- Coverage   89.13%   87.71%   -1.42%
==========================================
  Files          24       24
  Lines        1012     1050      +38
==========================================
+ Hits          902      921      +19
- Misses        110      129      +19

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Freeze when opening external terminal

If I try to open an external terminal, spyder freezes and no terminal opens.
I can launch a script in an external terminal without any problem.

What do you mean then by

If I try to open an external terminal

And are you on master or on 2.3.4?

Sorry, are you on Windows or Linux?

Oh sorry, I'm on master branch on Linux.
The command is in the menu Tools > Open a terminal

Ok, I'll take a look at it for beta1

I can't reproduce this problem.
Could you test again? I merged PR #2247 some days ago that could help to solve this problem.

It works on another machine, I'll have to retry on the machine which had a problem.

Maybe it's a bug in the script externalconsole.py
@ccordoba12 My fix is for programs.py

This is working fine for me :-) Please reopen if you find the cause.


Documenting 'end' and 'destroy' events for Streaming API

Documenting the 'destroy' event for Streaming API, which is emitted
when Twitter silently disconnects.

Thanks!


Road speed unit field matches invalid value to km/h

If I enter an unrecognized value into the Speed Limit field’s unit textbox, then move the focus elsewhere, the value changes to “km/h”, regardless of the original value.
Matching the first value is a problem because, like a lot of users, I’m in the habit of hitting the Tab key to commit the value of a textbox. For example, if a road has a 55 mph speed limit, I’ll enter 55 into the main numeric textbox then hit Tab. If I’m unaware that the focus is in the unit textbox, I’ll hit 2 to start drawing something else, but all I did was enter that number into the unit textbox. Then I’ll click on the map, see that nothing is being drawn, and press 2 again, unaware that I’ve actually tagged the roadway with maxspeed=55, meaning 55 km/h (34 mph). I’m pretty sure this is what happened to this way in this changeset and this way in this changeset, despite tagging speed limits correctly on all the surrounding roads.
This is unlikely to be the main source of incorrect speed limits in the United States and United Kingdom.[^josm] However, I think iD’s behavior is easier to fix. The following code only checks whether the value is literally mph and defaults to km/h otherwise. Instead, the function should bail if the value isn’t one of km/h or mph. This would be consistent with other combo boxes that don’t accept freeform input, such as the “Add field” combo box and Wikipedia field’s language combo box.
https://github.com/openstreetmap/iD/blob/ce136d0144487bc3c8fb569910958865860559a3/modules/ui/fields/roadspeed.js#L73-L74
[^josm]: That distinction probably belongs to JOSM, which doesn’t have a unit option to begin with. Go Map!! similarly lacks this option despite being based on id-tagging-schema: bryceco/GoMap#600.

The Max Height field has the same issue. Entering a value other than “m” or “ft” results in “m”:
https://github.com/openstreetmap/iD/blob/ce136d0144487bc3c8fb569910958865860559a3/modules/ui/fields/roadheight.js#L104-L105


armv7k architecture is not supported

We want to use this framework with WatchOS2 but get an error for the framework file: file is universal (5 slices) but does not contain a(n) armv7k slice

yes, me too .....


can't terminate the worker process if there is a UDF in the workflow

scan -> udf -> html_viz -> result
One worker has an error (e.g html_viz) in normal situations, we can terminate the worker process after the exception, if there is a UDF in the workflow, the worker can't be terminated as usual only force-kill

Hi @sadeemsaleh how do I reproduce the issue?

Hi @sadeemsaleh how do I reproduce the issue?

Shared a video to reproduce it in Slack

closing the issue as it is not reproducible in the current master. The referred html_viz is not in the master branch. Will continue the discussion from slack.

I don't think this is an issue restricted to html-viz. I got this issue while working on my branch too. I thought it was due to some issue that I had locally. But if @sadeemsaleh is also getting it, then it doesn't seem to be a problem local to me.

I really don't get the issue that you are describing. I have just tested with @sadeemsaleh on the latest master, and I cannot reproduce it. Could you please share the exact steps to reproduce the issue with the master branch?
Also I think the reason @sadeemsaleh is also getting it is that she used your branch. We can fix your branch instead.

The issue is - When a UDF is being used, the worker process in intellij doesn't shut down. I do not know the root cause. It may be that my branch changes brought out some hidden bugs in the UDF or it is actually a problem from my end, not related to UDF at all. Still it is better to keep the issue here, till we get to the root cause, so that we don't forget.

OK. then we should do the investigation on your local branch first.
When an issue is raised and assigned, I can only assume it is related to the current master branch, yet I cannot reproduce it so far. It's better if we do not track local issues from branches that are WIP here. But if you insist, we can keep the issue open here to continue the discussion later.
I will remove my assignment for now, you can re-assign it later so that I know I need to work on it. Thanks.

Yes. You should remove your assignment since we do not know the root-cause. I don't agree with the practice that an issue should only be reported if it is on master branch. Asking a dev, who is most likely on a different branch, to cross-check with master and find the root cause themselves is enough for them to ignore even important issues. An issue should be reported (with no assignment maybe).

Then, please mention the ways to reproduce the issue, including which branch, commit, or even test files.

scan -> udf -> html_viz -> result
One worker has an error (e.g html_viz) in normal situations, we can terminate the worker process after the exception, if there is a UDF in the workflow, the worker can't be terminated as usual only force-kill

Since I was assigned, I started to fix it. From the description, I really do not know which code to work on to fix it? I do not want to waste more time fixing false alarms.

I assume a video was shared showing the scenario and a meeting was done. On a high level, you can go to avinash-topic-html-vis branch. I have a simple topic-modeling workflow using mallet shown in the screenshot. The workflow is successful with no errors. But if we try to stop the worker process, it does not terminate.

The changes in the branch can be seen in the PR I have raised #1144 . We can have a 1:1 meeting to discuss further if needed.

Just discussed on Zoom with @avinash0161 and we suspect the issue is actually related to pyLDAvis library. There are 6 unknown python processes started if include pyLDAvis code, which are blocking the termination. So the solution right now is try to override the close() method in mallet trainer operator to try to kill the processes spawned by pyLDAvis library in a clean way.

I found this issue https://github.com/bmabey/pyLDAvis/issues/102#issuecomment-397857297 where they mentioned about subprocess being spawned during pyLDAvis.prepare. Maybe a direction to look into, especially the n_job parameter.
I found in the library's import, they have a server code. so lets try to minimize the import. try to do
from pyLDAvis import prepare
instead of import pyLDAvis to completely remove dependency on the server code and IPython (interactive python code)
If still fail, the last solution is to manually terminate the subprocesses in our code. Since the subprocesses are spwaned by the library, I consider this solution very hacky, but potentially possible workaround. In the udf.close() or def __del__(self) (the python's destructor), we have to find the subprocesses and kill them. https://stackoverflow.com/questions/26688936/how-to-get-pid-by-process-name this might be a way...

https://stackoverflow.com/questions/67495271/joblib-parallel-doesnt-terminate-processes this might be related.

this might be related to joblib issue. see https://github.com/joblib/joblib/issues/1121


SimpleImputer strange behaviour

Hi,
I am currently experiencing some weird errors while trying to port a pipeline from sklearn to dask. With respect to the SimpleImputer preprocessing class, I have noticed that the most_frequent strategy is currently implemented in such a way that the result is not what I think it should be. It reads:
def _fit_frame(self, X):
    if self.strategy == "mean":
        avg = X.mean(axis=0).values
    elif self.strategy == "median":
        avg = X.quantile().values
    elif self.strategy == "constant":
        avg = np.full(len(X.columns), self.fill_value)
    else:
        avg = [X[col].value_counts().nlargest(1).values for col in X.columns]
        avg = np.concatenate(*dask.compute(avg))

    self.statistics_ = pd.Series(dask.compute(avg)[0], index=X.columns)

My problem is with the following line:
avg = [X[col].value_counts().nlargest(1).values for col in X.columns]

I have been debugging this part and the previous line returns the counts for the most frequent item, not the item itself, which can introduce some weird errors when dealing with categorical data.
Does anybody have an explanation for this? Or did I miss something? Any hint will be much appreciated.
Regards.
Gus.

I think you’re correct, sorry about that. We should get the .index to get the item, not the count.


CLI tools are still missing

"Although Bistro has been in production at Facebook for over 3 years, the present public release is partial, including just the server components. The CLI tools and web UI will be shipping shortly."
It would be great to see these.

The CLI has been ready for open sourcing for a few months - I'll ping @lesha. The Web UI is Phabricator-based IIRC, I'm not sure what @lesha's plan was there for how to distribute that.

@snarkmaster on GitHub is @lesha at Facebook I guess :)

@brett-miller -- thanks for the interest! The OSS releases have hit a lull due to some personal challenges I've had. However, I'm preparing a major update, which will include the CLI for sure. The Web UI uses the Phabricator PHP stack (is this familiar to you?), but it has drifted a bit from trunk, and it may now take some effort to get it to run on upstream. Can you share a bit about the application you had in mind?

Hi, any news on that topic ? thx

@knx007, well, yes, and no. I haven't forgotten about it, no. I have made progress on this, yes. Why so slow? I've wanted to get some more features into the release, but in retrospect, it's a bad idea. I'll focus on the new release after I finish the feature I'm currently working on.

Thx :)

bistro can support docker？and Waiting for a new release. thanks

@neven7 -- which one do you mean?

Run within a Docker container? Sure.
Start Docker containers -- yes, if your provide a script, which boots your container, and cleans it up upon receiving SIGTERM. If you want nicer support, it is possible to add that too. But Docker integration is not currently a first-class feature.

Hi, for last few days i am playing with Bistro and so far i like it! There were some struggles with building and using with docker, but im fine now :-).
I would love to share that i am as well interested in Bistro CLI.
Thank you for your work!

@JanMikes, so the Bistro CLI has been abandoned internally for the past ~4 years -- neither used nor maintained.  As a concrete blocker, the CLI is in Python2, which is no longer supported. Another issue (IMO) is that the CLI relies on pandas, which is a heavy and slow dependency that most people would not benefit from. A third issue is that its handling of multi-shard jobs is poor, because it was quickly hacked together primarily for a single-shard-job setting.
If a customer with resources were willing to take on the maintenance of the CLI, I would be glad to help get this off the ground, either by consulting on a from-scratch project, or by sharing the existing code.
The good news is that the existing CLI is 1600 lines of code, so it's not hard to replace it with something modern that works.
Internally, people tend to interact with Bistro via the REST (for data / UI-type queries) or Thrift APIs (for command-and-control), or via the web UI.

Internally, people tend to interact with Bistro via the REST

Thank you, looks like this is the way to go a was anyways my favorite solution. I was just thinking about that calling curl from crontab seems to be way less elegant than using a cli, but both would do the job done anyways :-).


Converting batch_meta_collate to a class to make it pickle-able for '…

…spawn' method multiprocessing.

For dealing with issue https://github.com/tristandeleu/pytorch-meta/issues/38. This is just a really quick fix that's been loosely tested. Perhaps there's a better way of doing this?


Converting IsOnlineLog.xml misses entry for node 1.1.2

Most likely this is due to the fact that the node is initially offline.

This bug is due to an inherent limitation, which is now documented.


Add NotoSans font to Qt makefile includes

Follow-up from PR #2676, attempts to address error in #3415.

I am testing this with gitian straight, to save having to write custom Dockerfiles that test i686 Qt.
Marked it urgent as this is sitting on the critical path.


The "Always" trigger doesn't respect site selection

Describe the bug
问题描述
A clear and concise description of what the bug is.
When I disable sites, I would expect the dialog to not appear on the disabled site, even when "Always" is selected.
To Reproduce
如何复现
Steps to reproduce the behavior:

Disable a site

Change the trigger to "Always"

Go to the disabled site, we would expect the dialog to not appear, but it still appears.

Expected behavior
期望行为
A clear and concise description of what you expected to happen.
The dialog should not appear.
Screenshots
截图说明
If applicable, add screenshots to help explain your problem.

Please complete the following information):
请补全以下内容

OS: macOS
Browser: Chrome
Extension Version: v2.2.0

Additional context
其他
Add any other context about the problem here.
N/A

that's a bug

fixed in v2.2.1


[mesheryctl] fixed meshery start command bug with adapter components

Notes for Reviewers
This PR fixes #9149
Signed commits

[x] Yes, I signed my commits.

There are a couple of these that you might be able to tackle at the same time, @zakisk - SA1019: cliflags.NewCommonOptions is deprecated

What other commands other than start will now need to be updated to use this new Compose manifest reference?


ProjectionMappingSample error

When I open (and run as admin) the ProjectionMappingSample, I get an error.
Unhandled Exception: System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at RoomAliveToolkit.ProjectionMappingSample..ctor(String[] args) in C:\Users\Ayan\Desktop\ProCamCalibration\ProjectionMappingSample\ProjectionMappingSample.cs:line 25
   at RoomAliveToolkit.ProjectionMappingSample.Main(String[] args) in C:\Users\Ayan\Desktop\ProCamCalibration\ProjectionMappingSample\ProjectionMappingSample.cs:line 19

Then it says ProjectionMappingSample has stopped working. It gave me the option to debug, so I tried debugging and it said this was the problem:
string path = args[0];
I tried to rebuild. What should I do next?
Thanks, Ayan.

As mentioned in the README you need to pass the path to the calibration .xml file on the command line either in Visual Studio's debugging options or on the command line directly if you are launching from a command prompt. You should not have to run the sample as an admin.

I'm new to this. How exactly do you you do that?

In VS Solution Explorer right click ProjectionMappingSample project and select Properties. In the Debug tab there is an edit box for "Command line arguments". Put the path to your calibration file there, e.g., "C:\users\Ayan\Desktop\calibration\calibration.xml".


HUD broken with Firefox 81

Looks like Firefox is not setting the document.referrer again :/
https://bugzilla.mozilla.org/show_bug.cgi?id=1626616

Have confirmed that its just the missing document.referrer thats causing the problems.


Ceedling chokes on bad stdout data:  "ArgumentError: invalid byte sequence in UTF-8"

If the tests output bad utf8 data, the shell_result[:output] =~ TEST_STDOUT_STATISTICS_PATTERN line will choke.
Not sure what the proper fix is, but I think either scrubbing stdout to remove bad byte sequences, or at least a more indicative error would be helpful.
I have a patch, but for I can't seem  to make a change to the repo without git trying to replace the line endings of every file single file.  Very confused.
Anyway:
In generator_helper.rb
sanitized_output = shell_result[:output].dup
sanitized_output.scrub!

and a similar change to unity_shell_result[:output] in generator_test_results.rb
Original error:
rake aborted!
ArgumentError: invalid byte sequence in UTF-8
/src/vendor/ceedling/lib/ceedling/generator_helper.rb:21:in `=~'
/src/vendor/ceedling/lib/ceedling/generator_helper.rb:21:in `test_results_error_handler'
/src/vendor/ceedling/lib/ceedling/generator.rb:153:in `generate_test_results'
/src/vendor/ceedling/lib/ceedling/rules_tests.rake:42:in `block in <top (required)>'
/src/vendor/ceedling/lib/ceedling/task_invoker.rb:74:in `invoke_test_results'
/src/vendor/ceedling/lib/ceedling/test_invoker.rb:68:in `block in setup_and_invoke'
/src/vendor/ceedling/lib/ceedling/test_invoker.rb:32:in `each'
/src/vendor/ceedling/lib/ceedling/test_invoker.rb:32:in `setup_and_invoke'
/src/vendor/ceedling/lib/ceedling/rules_tests.rake:57:in `block (2 levels) in <top (required)>'

However - it's probably best to sanitize once, earlier in the pipeline.  Not sure where that place would be, though.

Thanks! I have this fixed, I believe. It's in the main branch on github. It'll make it into the next gem release.


Testing automatic image building

Note: dockerfile changed to improve debug time. Do not merge.
Will fail now without correct secrets.
Internal reference: https://github.com/allenai/docker-images

I ran some training on the new auto image, seemed fine.
Need to lower the LR on the default SFT config, but that's unrelated.


🐛 BUG: Error after the build step. Wrangler fails due to missing files.

Which Cloudflare product(s) does this pertain to?
Workers Runtime
What version(s) of the tool(s) are you using?
Build system version 1
What version of Node are you using?
16.16.0
What operating system and version are you using?
Build system v1
Describe the Bug
Observed behavior
Please describe.
Expected behavior
After the build step, I get this error:
ENOENT: no such file or directory, open '/pages/build_tool/js/.wrangler/tmp/pages-aGLTJl/functionsWorker-0.3268775573332672.js'

It started out of nowhere. No changes where made to the code to have this kind of problem.
Please provide a link to a minimal reproduction
No response
Please provide any relevant error logs
No response

I also have this issue. Both build system v1 and v2, both through automated deployments, and when manually deploying locally.

btw, this is also being discussed here: https://community.cloudflare.com/t/unable-to-build-functions-no-such-file-or-directory-open-pages-build-tool-js-wr/635355/10
For now, I am just using wrangler version 2.21.1 since it seems to not have this issue.

This should have ben fixed (internal ref: CUSTESC-40345)

Thank you, @penalosa


How to install with Oh My Zsh

I have tried the manual step but that didn't work for me. I am using oh-my-zsh and cloned it under ~/.oh-my-zsh/custom/plugins/zsh-vimode-visual
Here is my .zshrc configuration
plugins=(
    vi-mode
    zsh-vimode-visual
)

I have also tried to use source command on terminal but it still didn't work.

Unfortunately I have no idea because i'm not oh-my-zsh user. Maybe this plugin will work fine because zsh-vimode-visual.plugin.zsh is provided. Please ask to oh-my-zsh repo.


SMS OTP: Does user provising and authentication can be done with only mobile number and otp

Description:
When i configure SMS OTP of WSO2 3.1.0. I'm able to see two steps of authentication like first step with username and password and second step  with OTP.

But I need authentication in one step with otp:
List Of APIS required  for Provining
1) API to Register with mobile number - It will generate OTP.
2) API to Verify OTP - It will verify the otp,  provide token as response and save the user to user store
  List of APIS required for authentication
        1) API to generate OTP - It will take mobile number as input and generates an OTP
        2) API to verify OTP - It will verify otp and provide token as response.

Steps to reproduce:
    Followed WSO2 3.1.0 Configuring SMS OTP documentation (https://is.docs.wso2.com/en/latest/learn/configuring-sms-otp/#configuring-backup-codes-for-smsotp)

Affected Product Version: 3.1.0 APIM

Environment details (with versions):

OS: Ubuntu 16
Client:
Env (Docker/K8s):

Optional Fields
Related Issues:

Suggested Labels:

Suggested Assignees:

Hi @adityamajeti ,
are you trying this only using an APIM Pack? to enable SMS OTP features you will need to using Wso2IS as well. You can setup WSO2IS 5.10 as a KM for APIM 3.1

I used WSO2IS 5.10 and WSO2APIM 3.1.0. But I need apis to register user with mobile number and api to verify OTP and generate token as response


GTS: imports to GTS files are reporting as unresolved

Description
When importing from a GTS file, IntelliJ reports the import as unresolvable.
This also causes the imported values to not have their types known.
Steps to Reproduce

Clone https://github.com/universal-ember/ember-primitives
Open ./ember-primitives/src/components/popover.gts

Expected behavior:
The imports import { Portal } from './portal'; and import { TARGETS } from './portal-targets'; should resolve.
The Portal and TARGETS should have intellisense.
Actual behavior:
IntelliJ shows the import as unresolved.
The values act as if they have a type of unknown
Reproduces how often:
This can always be reproduced
Versions
IntelliJ IDEA 2023.3.2 (Ultimate Edition)
Build #IU-233.13135.103, built on December 20, 2023
Plugin version: 2023.3.1

Glint References are fixed. But they are still marked red.
I know why, but needs more work


Implement build pipeline for web components

Fixes GovAlta/ui-components#260
add alpha workflow to deploy to dev
adjust beta workflow to only deploy to test and not to build storybook website because it's already built
add prod workflow to deploy to prod

:tada: This PR is included in version 1.0.0-alpha.8 :tada:
The release is available on:

npm package (@alpha dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 3.4.0-alpha.11 :tada:
The release is available on:

npm package (@alpha dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.0.0-alpha.2 :tada:
The release is available on:

npm package (@alpha dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 2.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 4.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Failed to release SDKs for v3.3.0-rc1 release

The SDK release workflow failed because Python SDK release build failed and then Java SDK release build got canceled.
Run make --directory sdks/python publish -B
make: Entering directory '/home/runner/work/argo-workflows/argo-workflows/sdks/python'
Traceback (most recent call last):
  File "./sdk_version.py", line 35, in <module>
    raise e
  File "./sdk_version.py", line 28, in <module>
    version_digits = [int(i) for i in git_tag.replace(VERSION_PREFIX, '').split('.')]
  File "./sdk_version.py", line 28, in <listcomp>
    version_digits = [int(i) for i in git_tag.replace(VERSION_PREFIX, '').split('.')]
ValueError: invalid literal for int() with base 10: '0-rc1'
Makefile:3: *** "Failed to obtain a valid version for the SDK".  Stop.

I’ll look into the errors soon.
Build: https://github.com/argoproj/argo-workflows/runs/4987739996?check_suite_focus=true

This will be fixed in https://github.com/argoproj/argo-workflows/pull/7693.
@sarabala1979 @alexec Could we re-trigger the SDK release workflow once it's merged?

sure

Just checked that both SDKs have been released successfully.


Build Failure

Using VS 2019, SDK 10.0.19041.0
Attempted to build several samples all having the same result as building D3D12Multithreading:
NuGet Package restore failed for project D3D12Multithreading: Unable to find version '1.0.161208001' of package 'WinPixEventRuntime'.
  C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\: Package 'WinPixEventRuntime.1.0.161208001' is not found on source 'C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\'.
. Please see Error List window for detailed warnings and errors.
1>------ Build started: Project: D3D12Multithreading, Configuration: Debug x64 ------
1>D:\Personal\Projects\MyProjects\ProjDirectXGraphicsSamples\Samples\Desktop\D3D12Multithreading\src\D3D12Multithreading.vcxproj(157,5): error : This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them.  For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is D:\Personal\Projects\MyProjects\ProjDirectXGraphicsSamples\Samples\Desktop\D3D12Multithreading\src\packages\WinPixEventRuntime.1.0.161208001\build\WinPixEventRuntime.targets.
1>Done building project "D3D12Multithreading.vcxproj" -- FAILED.
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========

Already applied the suggestions for restoring packages and got
NuGet Package restore failed for project D3D12Multithreading: Unable to find version '1.0.161208001' of package 'WinPixEventRuntime'.
  C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\: Package 'WinPixEventRuntime.1.0.161208001' is not found on source 'C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\'.
. Please see Error List window for detailed warnings and errors.
NuGet package restore finished.
Time Elapsed: 00:00:00.0389909
========== Finished ==========

Add https://api.nuget.org/v3/index.json as the source of nuget.org in Vsual Studio


Sound display shows up, but only for microphone audio

So this plugin just started working again (it didn't display in the past), but with one big issue, the audio visualization isn't of spotify, but of my microphone. Just for context, I'm on linux.

now after #4 was reverted in #6 no audio is visualized at all.

#4 offered a fix and after quickly testing it seemed to work, but it was reverted shortly after due to it only recording microphone audio rather than desktop audio

but now no audio is displayed at all.


Update README.md

I have changed the name to a cool name,Hope you'll like it

I have changed the name to a cool name,Hope you'll like it

-_-

I've closed this pull request.Kindly look at it

@param-code you please close this pull request..


Fix casts between m256 and m256i

make casts work

The reinterpet_casts it was doing do not compile on MSVC at least, you must use the actual intrinsic casts


Send logs to an OTPL endpoint

Looking at the following configuration: https://github.com/grafana/k8s-monitoring-helm/tree/main/examples/otel-metrics-service
We can see that this generated an agent river configuration that sends Prometheus metrics to an OTPL endpoint. However, it does not deliver the logs via OTPL, but still using Loki endpoint:
  loki:
    host: https://loki.example.com
    basicAuth:
      username: 12345
      password: "It's a secret to everyone"

Generated .river config
loki.write "grafana_cloud_loki" {
  endpoint {
    url = nonsensitive(remote.kubernetes.secret.logs_service.data["host"]) + "/loki/api/v1/push"
    tenant_id = nonsensitive(remote.kubernetes.secret.logs_service.data["tenantId"])

    basic_auth {
      username = nonsensitive(remote.kubernetes.secret.logs_service.data["username"])
      password = remote.kubernetes.secret.logs_service.data["password"]
    }
  }
  external_labels = {
    cluster = "otel-metrics-service-test",
  }

We required a configuration that similarly can configure an otpl endpoint as follows:
otelcol.processor*...

otelcol.exporter.otlphttp "default" {
    client {
        endpoint = <OTEL endpoint>
        auth     = otelcol.auth.basic.credentials.handler
[...] // Other configuration
    }
}
otelcol.auth.basic "credentials" {
    // Retrieve credentials using environment variables.
    username = "xxx"
    password = "xxx"
}

Hello @petewall just to double-check, with this change we can still use the values:
  loki:
    host: <OTPL endpoint>
    basicAuth:
      username: 12345
      password: "It's a secret to everyone"

But defining the OTPL endpoint directly in the host, correct?

Can we also add processors here, similar to the Prometheus block?
    processors:
      memoryLimiter:
        enabled: true
        limit: 100MiB

Yeah, let me add those before I make the release.

FYI @carlmagnusholmstrom-tomtom , @jasuade : https://github.com/grafana/k8s-monitoring-helm/pull/414

@petewall I'd propose to create a new folder here https://github.com/grafana/k8s-monitoring-helm/tree/main/examples with an example of how to use OTLP for all signals. I can create a PR if that's OK with you?

actually, it was on my list to update this example to include logs delivered via otlp: https://github.com/grafana/k8s-monitoring-helm/tree/main/examples/otel-metrics-service
If you're up for modifying that one, be my guest!
The name of the folder should be updated. When you've updated the values.yaml file, make sure to run make regenerate-example-outputs to generate the other files.

Perfect! Let me know if you need anything along the way.


Additional properties should add to a basic type as opposed to replac…

…ing it with a Record type
This is about a flag called additionalProperties. At the moment when it is set to true the output type is always Record<string, any> I think it should be Record<string, any> & {foo: string}
Or something of that sort

TODO: Auto-detect Prettier


Input.svelte should handle type="number" correctly

Current Behavior
 should have value of type number - not string. it appears that it's a string.
Expected Behavior
we want the value to be of type number
Steps To Reproduce

npx shadcn-svelte add input

Link to Reproduction / Stackblitz
No response
More Information
one solution that worked for me was to add the following to Input.svelte but perhaps you can think of a less hacky approach:
export let type: HTMLInputAttributes['type'];
export let value: HTMLInputAttributes['value'] = undefined;

{#if type === 'number'}
    <input
        bind:value
        type="number"
        ....
    />
{:else}
    <input
        bind:value
        ....
    />
{/if}

Does this issue still occur?


Juliainstaller Crash Report on x86_64-apple-darwin: Failed to open juliaup config file

Log message:
Now installing Juliaup
Well, this is embarrassing.

Juliainstaller had a problem and crashed. To help us diagnose the problem you can send us a crash report.

We have generated a report file at "/var/folders/dg/7mtlrlqd6fz01jdb9djb33bm0000gp/T/report-7fe674d3-a33e-46db-8626-81b807edeedb.toml". Submit an issue or email with the subject of "Juliainstaller Crash Report" and include the report as an attachment.

- Homepage: https://github.com/JuliaLang/juliaup

We take privacy seriously, and do not perform any automated error collection. In order to improve the software, we rely on people to submit reports.

Thank you kindly!

Log file:
name = 'Juliainstaller'
operating_system = 'unix:OSX'
crate_version = '1.6.12'
explanation = '''
Panic occurred in file 'src/operations.rs' at line 559
'''
cause = '''
called `Result::unwrap()` on an `Err` value: Failed to open juliaup config file.

Caused by:
    No such file or directory (os error 2)'''
method = 'Panic'
backtrace = '''

   0: 0x105226cac - juliaup::command_config_modifypath::run_command_config_modifypath::h3eb12daf0cd2c1d2
   1: 0x105136a76 - juliainstaller::main::h7abd7c0c48f4b2f4
   2: 0x105120a36 - std::sys_common::backtrace::__rust_begin_short_backtrace::h5636139a57dca0d6
   3: 0x10513aa51 - _main'''

Steps to reproduce

rm -rf ~/.julia
curl -fsSL https://install.julialang.org | sh
Select "Customize installation".
Answer the questions as follows:

"Enter the folder where you want to install Juliaup ›": press Enter to select the default value.
"Do you want to add the Julia binaries to your PATH by manipulating various shell startup scripts? (y/n) ›": press n.
"Do you want to add channel specific symlinks? (y/n) ›": press n.
"Enter minutes between check for new version at julia startup, use 0 to disable (1440) ›": type 0 and press Enter.
Enter minutes between check for new version by a background task, use 0 to disable (0) ›": type 0 and press Enter.

Select "Proceed with installation".

I suspect that what's happening here is that juliaup is trying to open the ~/.zshrc file because I am on macOS. However, I do not use zsh, and in fact the ~/.zshrc file does not exist in my home directory. So I think that juliaup is erroring here because the ~/.zshrc file does not exist.

I can confirm this hypothesis. If I do touch ~/.zshrc before I run the "steps to reproduce" from my original post, then everything works fine.


Could not retrieve a transcript for the video

Could not retrieve a transcript for the video https://www.youtube.com/watch?v=98TQv5IAtY8! This is most likely caused by: The video is no longer available
It works on my local computer (Windows 10) but when I try to use it on Ubuntu 20.04(DigitalOcean Droplet) I get this error!
I assume the error is caused by sender I.P. address.
I got a similar problem using youtube-dl on my droplet and when I tried using "--force-ipv4" with youtube-dl It worked. Is there a similar solution to this?
Code
YouTubeTranscriptApi.get_transcript("98TQv5IAtY8", languages=['en'])

Hi @iercetin
my first guess would've been, that you are running into some kind of rate limit because the IP address of your cloud machine is also used by others (see #60 for more on rate limits). However, the IPv6 thing is a really interesting clue!
As of right now, there's no functionality in this module to force IPv4 connections, but I could implement something like this if this turns out to be a common problem with YouTube. Unfortunately I don't have a machine or setup were I can replicate this error. So I'll need your help debugging this.
Could you do a IPv4 and IPv6 ping to YouTube from your local machine and from your cloud machine and see how that goes on either machine?
ping youtube.com
ping6 youtube.com

Sure,
-- Windows 10 --
ping youtube.com
Pinging youtube.com [<IP_ADDRESS>] with 32 bytes of data:
Reply from <IP_ADDRESS>: bytes=32 time=44ms TTL=115
Reply from <IP_ADDRESS>: bytes=32 time=68ms TTL=115
Reply from <IP_ADDRESS>: bytes=32 time=49ms TTL=115
Reply from <IP_ADDRESS>: bytes=32 time=55ms TTL=115
Ping statistics for <IP_ADDRESS>:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 44ms, Maximum = 68ms, Average = 54ms
-- Ubuntu (Digital Ocean Droplet) --
ping youtube.com
PING youtube.com(ams16s32-in-x0e.1e100.net (2a00:1450:400e:80c::200e)) 56 data bytes
64 bytes from ams16s32-in-x0e.1e100.net (2a00:1450:400e:80c::200e): icmp_seq=1 ttl=117 time=7.78 ms
64 bytes from ams16s32-in-x0e.1e100.net (2a00:1450:400e:80c::200e): icmp_seq=2 ttl=117 time=7.01 ms
64 bytes from ams16s32-in-x0e.1e100.net (2a00:1450:400e:80c::200e): icmp_seq=3 ttl=117 time=6.94 ms
64 bytes from ams16s32-in-x0e.1e100.net (2a00:1450:400e:80c::200e): icmp_seq=4 ttl=117 time=6.98 ms
64 bytes from ams16s32-in-x0e.1e100.net (2a00:1450:400e:80c::200e): icmp_seq=5 ttl=117 time=7.07 ms
64 bytes from ams16s32-in-x0e.1e100.net (2a00:1450:400e:80c::200e): icmp_seq=6 ttl=117 time=6.98 ms
--- youtube.com ping statistics ---
9 packets transmitted, 9 received, 0% packet loss, time 8012ms
rtt min/avg/max/mdev = 6.935/7.100/7.779/0.243 ms
ping6 youtube.com
PING youtube.com(ams16s32-in-x0e.1e100.net (2a00:1450:400e:80c::200e)) 56 data bytes
64 bytes from ams16s32-in-x0e.1e100.net (2a00:1450:400e:80c::200e): icmp_seq=1 ttl=117 time=7.83 ms
64 bytes from ams16s32-in-x0e.1e100.net (2a00:1450:400e:80c::200e): icmp_seq=2 ttl=117 time=7.07 ms
64 bytes from ams16s32-in-x0e.1e100.net (2a00:1450:400e:80c::200e): icmp_seq=3 ttl=117 time=7.07 ms
64 bytes from ams16s32-in-x0e.1e100.net (2a00:1450:400e:80c::200e): icmp_seq=4 ttl=117 time=7.24 ms
64 bytes from ams16s32-in-x0e.1e100.net (2a00:1450:400e:80c::200e): icmp_seq=5 ttl=117 time=6.99 ms
--- youtube.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4007ms
rtt min/avg/max/mdev = 6.990/7.240/7.831/0.305 ms

Thanks @iercetin
It definitely seems that your Ubuntu machine defaults to IPv6 when connecting to YouTube, which proves your point.
Could you try and get the video using curl on your Ubuntu machine, forcing it to use IPv4/6?
 curl -L -4 "http://youtube.com/watch?v=98TQv5IAtY8"
 curl -L -6 "http://youtube.com/watch?v=98TQv5IAtY8"

Sure,
 curl -L -4 "http://youtube.com/watch?v=98TQv5IAtY8"
https://github.com/iercetin/testtest/blob/master/4.html
 curl -L -6 "http://youtube.com/watch?v=98TQv5IAtY8"
https://github.com/iercetin/testtest/blob/master/6.html

@iercetin mh, that's interesting. It in fact seems that the IPv6 requests returns a different response than the IPv4 response, which proves your point even further.
However, could it be that you maybe mixed up the 6.html and 4.html file names? Because from what I can tell 6.html actually is the file which contains the information needed to extract the subtitles, while 4.html doesn't. If that would be true IPv6 should the working request and not the other way around. Could you please double check that?

I'm having the same problem - works fine locally (OS X) but on my server (Ubuntu 18.04.4 LTS (Bionic Beaver)) I get the same error as @iercetin

@sdtblck could you execute those commands and upload the results on your Ubuntu machine:
 curl -L -4 "http://youtube.com/watch?v=98TQv5IAtY8"
 curl -L -6 "http://youtube.com/watch?v=98TQv5IAtY8"

@iercetin @sdtblck any news on this?

Hi @jdepoix, I'm using this on GCP cloud functions and I think I'm facing similar issue.  Would I have to route all the egress traffic through a IPV4 VPC network with static IP to test if would help with the rate limiting on the shared machine?

Hi @jdepoix, I'm using this on GCP cloud functions and I think I'm facing similar issue. Would I have to route all the egress traffic through a IPV4 VPC network with static IP to test if IPV4 connections would help with the rate limiting on the shared machine?

I was able to get the API to work again on GCP Cloud functions now with a static IP following a GCP guide! https://dev.to/alvardev/gcp-cloud-functions-with-a-static-ip-3fe9

Hi @adongu,
that's great news, thanks for sharing! So the static IP is IPv4 I guess? I am not sure whether the solution is the IP being static or it being IPv4 instead of IPv6. Is there any way for you to find out whether your requests will now default to IPv4 instead of IPv6?

Hey @jdepoix I did some digging and found this tidbit the documentations for Google VPC networks.  https://cloud.google.com/vpc/docs/vpc#specifications
"VPC networks only support IPv4 unicast traffic. They do not support broadcast, multicast, or IPv6 traffic within the network; VMs in the VPC network can only send to IPv4 destinations and only receive traffic from IPv4 sources. However, it is possible to create an IPv6 address for a global load balancer."
It looks like all VPC traffic is IPV4, unless I create a IPV6 address on global LB, and route service all traffic first to the LB.  The guide I followed didn't create any LB as far as I know and the VPC network routing mode is regional.

So if I am understanding this correctly, you were probably doing IPv6 requests before setting up the VPC, while now you're doing IPv4 requests. Which would further support the assumption that this module can fail when sending IPv6 requests to YouTube. Thank you for sharing!
I guess my best bet would be to implement something which forces this module to use IPv4. I'll look into that when I have some time at hand.

Hey @jdepoix , apologies if I was being vague.  I'm not actually sure if it was serving via IPV6 before since I didn't have any global LB set up as it was a light project.  I think my issue might be related to https://github.com/jdepoix/youtube-transcript-api/issues/60 instead of this.  Apologies for spinning your wheel.

Thanks for clarifying @adongu.
I guess, we will need further proof before forcing this module to use IPv4...

@sdtblck could you please execute those commands on your Ubuntu machine and upload the results:
 curl -L -4 "http://youtube.com/watch?v=98TQv5IAtY8"
 curl -L -6 "http://youtube.com/watch?v=98TQv5IAtY8"

I also have the same problem on AWS EC2.
To add my context, the requests initially worked fined on AWS. After a few thousands requests (at about 1 / sec), I started getting VideoUnavailable error and now I'm still getting those error 3 days after, even for a single request.
curl -L -4 is working fine though (returns the html), and curl -L -6 returns curl: (7) Couldn't connect to server
Thanks @jdepoix for you diligence in fixing it - I can try to take over making tests from the server!

Hi @cramdoulfa,
Thanks for the information. Could you upload the HTML which is returned by calling curl -L -4 from the EC2 instance, so I can have a look at what is being returned?

Hi @cramdoulfa,
Thanks for the information. Could you upload the HTML which is returned by calling curl -L -4 from the EC2 instance, so I can have a look at what is being returned?

curl_L4_return.txt
Here it is @jdepoix for curl -L -4 "http://youtube.com/watch?v=98TQv5IAtY8"

Thanks for the additional information @cramdoulfa!
This seems a bit odd though, as the information which is required for this module is actually being returned by your request. Are you sure that the module was still failing, while trying to retrieve this video, as you did the requests? Maybe there were some rate limits which did reset. Did you check this, before executing the curl request?

Thanks for the additional information @cramdoulfa!
This seems a bit odd though, as the information which is required for this module is actually being returned by your request. Are you sure that the module was still failing, while trying to retrieve this video, as you did the requests? Maybe there were some rate limits which did reset. Did you check this, before executing the curl request?

Hum very good point, the package is actually working again now! I will start a batch of query and update if it starts blocking again.
It's probably a matter of quota or rate limit.

I had this issue also myself. It's due to youtube blocking your IP. I switched on a VPN and everything worked as expected.

@jacksonw765 yeah, that's what I was guessing. It would be great though, if I could se what HTML YouTube returns after they blocked you, so that I can add a proper error message to this module.

@jdepoix  here is a sample HTML page for a video with available transcripts when the API seems to be blocked:
curl_result_blocked_transcript_API.txt

I had this issue also myself. It's due to youtube blocking your IP. I switched on a VPN and everything worked as expected.

Slight sidetrack but I'm curious @jacksonw765 do you use a commercial VPN or did you configure one yourself with openVPN?
I could not find a nice VPN client for Linux AMI

@cramdoulfa huh, that seems really odd. Once again the HTML seems to contain all the information needed by this module to retrieve the transcripts. The exception you got was a VideoUnavailable?

Ups sorry for the false call - it seems that the API had indeed been de-blocked in the meantime!
Thanks for the code snippet, I will verify next time it happens and notify you if I find an HTML for which _extract_captions_json fails

@cramdoulfa no worries, thanks for putting in the time trying to resolve this! 😊👍

I had this issue also myself. It's due to youtube blocking your IP. I switched on a VPN and everything worked as expected.

Slight sidetrack but I'm curious @jacksonw765 do you use a commercial VPN or did you configure one yourself with openVPN?
I could not find a nice VPN client for Linux AMI

I use PIA

@cramdoulfa no worries, thanks for putting in the time trying to resolve this! 😊👍

Ok I think this is the right one this time. The page actually says 'We have been receiving large amounts of requests from your nework.'
curl_result_blocked_transcript_API.txt

@cramdoulfa no worries, thanks for putting in the time trying to resolve this! 😊👍

Ok I think this is the right one this time. The page actually says 'We have been receiving large amounts of requests from your nework.'
curl_result_video_unavailable.txt

Perfect, that's exactly what I was looking for @cramdoulfa! Thank you very much! 👍
I will add a custom error for this suggesting the user to wait for the rate limit to reset, or use a VPN/change IP. I am quite busy right now, as I am in the last weeks of writing my master thesis, so I probably won't be doing any coding on this module for a few weeks, but I'll try to get that done as soon as I can.
The other thing which still remains interesting is the IPv4 vs IPv6 thing suggested above. I would be great if you could try executing a IPv4 and IPv6 request next time you run into the rate limit and upload the results here. The responses which have been uploaded so far have been contradicting each other a bit and the people have unfortunately stopped replying.

Also, could you make any guesses on how long the rate limit persists until it is reseted, or was it inconsistent for you @cramdoulfa ?

Also, could you make any guesses on how long the rate limit persists until it is reseted, or was it inconsistent for you @cramdoulfa ?

Great!
I have been making requests every 1.5 second. Not sure how consistent it is, but it started blocking after about 15,000 videos this time. I will try to monitor how long it takes before it resets.

Perfect, that's exactly what I was looking for @cramdoulfa! Thank you very much! 👍
I will add a custom error for this suggesting the user to wait for the rate limit to reset, or use a VPN/change IP. I am quite busy right now, as I am in the last weeks of writing my master thesis, so I probably won't be doing any coding on this module for a few weeks, but I'll try to get that done as soon as I can.
The other thing which still remains interesting is the IPv4 vs IPv6 thing suggested above. I would be great if you could try executing a IPv4 and IPv6 request next time you run into the rate limit and upload the results here. The responses which have been uploaded so far have been contradicting each other a bit and the people have unfortunately stopped replying.

curl -L -6 "http://youtube.com/watch?v=0vAfIcmpqzQ" returns curl: (7) Couldn't connect to server for me
Good luck with the thesis!

In v0.4.0 the Exception TooManyRequests has been added which is raised when running into rate limits. This could be used to further investigate the issue. My guess with the IPv4 vs. IPv6 thing @iercetin mentioned is that the IPv6 has been blocked due to extensive usage, while IPv4 still works as it hasn't been blocked yet.
As this issue is kinda all over the place now with different things being reported (most of them most likely due to rate limits, which now have a more speaking error message) I will close this for now. If individual issues arise again feel free to open a new issue with a title more specific to that issue.

I will add a custom error for this suggesting the user to wait for the rate limit to reset, or use a VPN/change IP.

@jdepoix Is it possible to use an IP rotation service like https://scrapingant.com with this module?

@jdepoix Given that the official YouTube API does not allow one to obtain captions of YouTube videos one does not own, have you heard of any option, including paid ones, that one can build upon with reasonable certainty?

I have been making requests every 1.5 second. Not sure how consistent it is, but it started blocking after about 15,000 videos this time.

@cramdoulfa Have you been able to determine what frequency of requests does not result in being blocked.


prompb: Generate client proto using Opaque API, for parsing only.

Fixes https://github.com/prometheus/prometheus/issues/14668
Depends on https://github.com/prometheus/prometheus/pull/15720
Second attempt after https://github.com/prometheus/prometheus/pull/15726

prompb: Generate client proto using Opaque API, for parsing only.. This also creates a io/prometheus/client/v1 proto that supports Opaque API that will be by default at some point.

Just OpaqueAPI, before tricks (:
benchstat -filter="/fmt:PromProto" gogo=scrape-loop-v1.txt opaqueAPI=scrape-loop-v4nolazy.txt opaqueAPILazy=scrape-loop-v4.txt
goos: darwin
goarch: arm64
pkg: github.com/prometheus/prometheus/scrape
cpu: Apple M1 Pro
                                 │    gogo     │             opaqueAPI              │            opaqueAPILazy             │
                                 │   sec/op    │   sec/op     vs base               │    sec/op     vs base                │
ScrapeLoopAppend/fmt=PromProto-2   54.48µ ± 8%   74.42µ ± 5%  +36.61% (p=0.002 n=6)   119.36µ ± 3%  +119.11% (p=0.002 n=6)

                                 │     gogo     │              opaqueAPI              │           opaqueAPILazy            │
                                 │     B/op     │     B/op      vs base               │     B/op      vs base              │
ScrapeLoopAppend/fmt=PromProto-2   75.58Ki ± 1%   64.09Ki ± 0%  -15.21% (p=0.002 n=6)   81.55Ki ± 1%  +7.90% (p=0.002 n=6)

                                 │    gogo    │             opaqueAPI              │            opaqueAPILazy            │
                                 │ allocs/op  │  allocs/op   vs base               │  allocs/op   vs base                │
ScrapeLoopAppend/fmt=PromProto-2   810.0 ± 0%   1613.0 ± 0%  +99.14% (p=0.002 n=6)   2218.0 ± 0%  +173.83% (p=0.002 n=6)