CLOUD-467 check slack

PXC operator docker - perconalab/percona-xtradb-cluster-operator:PR-299-36e0c2a

Test name
Status

scaling
passed

scaling-proxysql
passed

upgrade
passed

upgrade-consistency
passed

self-healing
passed

self-healing-advanced
passed

operator-self-healing
passed

init-deploy
passed

one-pod
passed

storage
passed

auto-tuning
passed

limits
passed

monitoring
passed

monitoring-2-0
passed

affinity
passed

proxysql-sidecar-res-limits
passed

big-data
passed

recreate
passed

demand-backup
passed

scheduled-backup
passed

PXC operator docker - perconalab/percona-xtradb-cluster-operator:PR-299-0a27af6

PXC operator docker - perconalab/percona-xtradb-cluster-operator:PR-299-f49fe36

Test name
Status

init-deploy
passed

big-data
passed

scaling
passed

self-healing
passed

recreate
passed

storage
passed

scaling-proxysql
passed

self-healing-advanced
passed

demand-backup
passed

limits
passed

upgrade
passed

operator-self-healing
passed

scheduled-backup
passed

monitoring
passed

upgrade-consistency
passed

one-pod
passed

monitoring-2-0
passed

auto-tuning
passed

affinity
passed

proxysql-sidecar-res-limits
passed

PXC operator docker - perconalab/percona-xtradb-cluster-operator:PR-299-78ac69f

Test name
Status

big-data
passed

init-deploy
passed

scaling
passed

self-healing
passed

recreate
passed

storage
passed

scaling-proxysql
passed

self-healing-advanced
passed

demand-backup
passed

limits
passed

upgrade
passed

operator-self-healing
passed

scheduled-backup
passed

monitoring
passed

upgrade-consistency
passed

one-pod
passed

monitoring-2-0
passed

auto-tuning
passed

affinity
passed

proxysql-sidecar-res-limits
passed

PXC operator docker - perconalab/percona-xtradb-cluster-operator:PR-299-8c21bbb

Test name
Status

init-deploy
passed

scaling
passed

self-healing
passed

recreate
passed

big-data
passed

storage
passed

scaling-proxysql
passed

self-healing-advanced
passed

demand-backup
passed

limits
passed

upgrade
passed

operator-self-healing
passed

scheduled-backup
passed

monitoring
passed

upgrade-consistency
passed

one-pod
passed

monitoring-2-0
passed

auto-tuning
passed

affinity
passed

proxysql-sidecar-res-limits
passed

PXC operator docker - perconalab/percona-xtradb-cluster-operator:PR-299-9e93fa8

Test name
Status

big-data
passed

init-deploy
passed

self-healing
passed

scaling
passed

recreate
passed

storage
passed

scaling-proxysql
passed

self-healing-advanced
passed

demand-backup
passed

limits
passed

upgrade
passed

operator-self-healing
passed

scheduled-backup
passed

monitoring
passed

upgrade-consistency
passed

one-pod
passed

monitoring-2-0
passed

auto-tuning
passed

affinity
passed

proxysql-sidecar-res-limits
passed

Test name
Status

PXC operator docker - perconalab/percona-xtradb-cluster-operator:PR-299-9950a73

PXC operator docker - perconalab/percona-xtradb-cluster-operator:PR-299-a0b71f6

Test name
Status

init-deploy
passed

big-data
passed

scaling
passed

self-healing
passed

recreate
passed

scaling-proxysql
passed

storage
passed

self-healing-advanced
passed

demand-backup
passed

upgrade
passed

limits
passed

operator-self-healing
passed

scheduled-backup
passed

upgrade-consistency
passed

monitoring
passed

one-pod
passed

auto-tuning
passed

monitoring-2-0
passed

affinity
passed

proxysql-sidecar-res-limits
passed

PXC operator docker - perconalab/percona-xtradb-cluster-operator:PR-299-3778527

Test name
Status

init-deploy
passed

scaling
passed

self-healing
passed

recreate
passed

big-data
passed

storage
passed

scaling-proxysql
passed

self-healing-advanced
passed

demand-backup
passed

limits
passed

upgrade
passed

operator-self-healing
passed

scheduled-backup
passed

monitoring
passed

upgrade-consistency
passed

one-pod
passed

monitoring-2-0
passed

auto-tuning
passed

affinity
passed

proxysql-sidecar-res-limits
passed

PXC operator docker - perconalab/percona-xtradb-cluster-operator:PR-299-d74f30d

Test name
Status

scaling
passed

init-deploy
passed

self-healing
passed

recreate
passed

big-data
passed

self-healing-advanced
passed

demand-backup
passed

scaling-proxysql
passed

storage
passed

operator-self-healing
passed

scheduled-backup
passed

upgrade
passed

limits
passed

one-pod
passed

upgrade-consistency
passed

monitoring
passed

auto-tuning
passed

monitoring-2-0
passed

affinity
passed

proxysql-sidecar-res-limits
passed

Test name
Status

big-data
passed

self-healing
passed

scaling
passed

recreate
passed

init-deploy
failed

storage
passed

scaling-proxysql
passed

demand-backup
passed

self-healing-advanced
failed

limits
passed

operator-self-healing
passed

monitoring
passed

upgrade
passed

upgrade-consistency
passed

one-pod
passed

monitoring-2-0
passed

auto-tuning
passed

affinity
passed

scheduled-backup
failed

proxysql-sidecar-res-limits
passed


youtube_transcript_api._errors.TranscriptsDisabled But it's not disabled

To Reproduce
using youtube-transcript-api-0.6.2:
cat test.py
from youtube_transcript_api import YouTubeTranscriptApi

print(YouTubeTranscriptApi.get_transcript('w8rYQ40C9xo'))

outputs:
python3 ./test.py
Traceback (most recent call last):
  File "/root/border0-plugin/./test.py", line 3, in <module>
    print(YouTubeTranscriptApi.get_transcript('w8rYQ40C9xo'))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/border0-plugin/myenv2/lib/python3.11/site-packages/youtube_transcript_api/_api.py", line 137, in get_transcript
    return cls.list_transcripts(video_id, proxies, cookies).find_transcript(languages).fetch(preserve_formatting=preserve_formatting)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/border0-plugin/myenv2/lib/python3.11/site-packages/youtube_transcript_api/_api.py", line 71, in list_transcripts
    return TranscriptListFetcher(http_client).fetch(video_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/border0-plugin/myenv2/lib/python3.11/site-packages/youtube_transcript_api/_transcripts.py", line 48, in fetch
    self._extract_captions_json(self._fetch_video_html(video_id), video_id),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/border0-plugin/myenv2/lib/python3.11/site-packages/youtube_transcript_api/_transcripts.py", line 62, in _extract_captions_json
    raise TranscriptsDisabled(video_id)
youtube_transcript_api._errors.TranscriptsDisabled:
Could not retrieve a transcript for the video https://www.youtube.com/watch?v=w8rYQ40C9xo! This is most likely caused by:

Subtitles are disabled for this video

If you are sure that the described cause is not responsible for this error and that a transcript should be retrievable, please create an issue at https://github.com/jdepoix/youtube-transcript-api/issues. Please add which version of youtube_transcript_api you are using and provide the information needed to replicate the error. Also make sure that there are no open issues which already describe your problem!

What code / cli command are you executing?
I am running
from youtube_transcript_api import YouTubeTranscriptApi
print(YouTubeTranscriptApi.get_transcript('w8rYQ40C9xo'))

Which Python version are you using?
Python 3.11.6
Which version of youtube-transcript-api are you using?
youtube-transcript-api-0.6.2
Expected behavior
Describe what you expected to happen.
I expected to receive the english transcript
I can see it in browser, see screenshot:

Actual behaviour
Traceback (most recent call last):
  File "/root/border0-plugin/./test.py", line 3, in <module>
    print(YouTubeTranscriptApi.get_transcript('w8rYQ40C9xo'))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/border0-plugin/myenv2/lib/python3.11/site-packages/youtube_transcript_api/_api.py", line 137, in get_transcript
    return cls.list_transcripts(video_id, proxies, cookies).find_transcript(languages).fetch(preserve_formatting=preserve_formatting)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/border0-plugin/myenv2/lib/python3.11/site-packages/youtube_transcript_api/_api.py", line 71, in list_transcripts
    return TranscriptListFetcher(http_client).fetch(video_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/border0-plugin/myenv2/lib/python3.11/site-packages/youtube_transcript_api/_transcripts.py", line 48, in fetch
    self._extract_captions_json(self._fetch_video_html(video_id), video_id),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/border0-plugin/myenv2/lib/python3.11/site-packages/youtube_transcript_api/_transcripts.py", line 62, in _extract_captions_json
    raise TranscriptsDisabled(video_id)
youtube_transcript_api._errors.TranscriptsDisabled:
Could not retrieve a transcript for the video https://www.youtube.com/watch?v=w8rYQ40C9xo! This is most likely caused by:

Subtitles are disabled for this video

If you are sure that the described cause is not responsible for this error and that a transcript should be retrievable, please create an issue at https://github.com/jdepoix/youtube-transcript-api/issues. Please add which version of youtube_transcript_api you are using and provide the information needed to replicate the error. Also make sure that there are no open issues which already describe your problem!

Same Issue :(

Hi @atoonk, do you only have this issue with this specific video, or all videos you are trying to retrieve? I can retrieve the subtitles for that video without any issues, which usually means that you are being rate-limited by YouTube (which would also mean that this should happen for all videos).

Hi @jdepoix, I encountered the same problem yesterday with every video I tried. Although I don't use the API frequently, I do access it a few times per day. I hope it's not some new restriction from YouTube. I experienced the same problem as @atoonk, and the issue is still present today.

Hi @SKVNDR, then you're most definitely being blocked by YouTube. The only way to work around this is to change your IP address in any way (VPN, proxy, or assign a new IP if possible).

I can confirm that YouTube is most likely blocking =/
It works from my local dev env but it doesn't work in production all things equal.

If you're running your code on a cloud machine it could be that (depending on your setup) you're getting assigned an IP from a poll that is shared with other machines. So the IP you're using could potentially be blocked without you doing anything. YouTube could also generally black list certain IPs that are known to belong to cloud providers (just a guess, I don't know if they actually do that!).

Ah yes, i tried it from my laptop at home and it works fine now. And indeed, it affected all videos, which I why I thought it was a bug or new behaviour in YT api.
So, I guess YouTube blocked me (this was on Digital ocean machine). Bummer, gotta find a way around that. Any docs on the ratelimit numbers or when folks get added? I only run this once every few weeks and only for a dozen videos or so. So bit surprised I was blocked. Unless it's all of digital ocean.

Since this is not an official API, there unfortunately is no information on rate limits and when or for how long you will get blocked. People have been reporting different things, so I don't feel like it is consistent either.

I will pin this issue and leave it open, since there are issues being opened due to this all the time.
Feel free to discuss workarounds and share your experience on YouTubes blocking heuristics, but be aware that there is no proper fix here and probably never will be. That's the nature of using an unofficial API unfortunately.

Same for me. I use a droplet on DigitalOcean, and YouTube probably blocked the IP from there, but using a proxy fixed the issue...

Same for me. I use a droplet on DigitalOcean, and YouTube probably blocked the IP from there, but using a proxy fixed the issue...
how did you create a proxy can you share the code. did you use a free proxy or paid? how did you obtain that proxy?

Hi @auspy,
from youtube_transcript_api import YouTubeTranscriptApi
YouTubeTranscriptApi.get_transcript(video_id, proxies={"https": "https://user:pass@domain:port"})

I'm using a paid proxy from smartproxy.com with the "Residential" offer.
There are probably other better proxies available; I chose this one randomly.

confirmed, using a proxy from my droplet worked. I used this to proxy traffic from my digital ocean droplet to my local laptop. https://docs.border0.com/docs/expose-a-http-proxy
can make a more details quick video if folks are interested in how to use that.

Having the exact same issue, & also using DigitalOcean droplet

Same here. Subscribed this issue.

confirmed, using a proxy from my droplet worked. I used this to proxy traffic from my digital ocean droplet to my local laptop. https://docs.border0.com/docs/expose-a-http-proxy which will allow you to expose a proxy on localhost and have it egress on a separate machine (in my case my laptop)
transcript = YouTubeTranscriptApi.get_transcript(video_id, proxies={"https": "http://localhost:8080"})

can make a more details quick video if folks are interested in how to use that.

sure would love a video on it. drop the link here

Hi @auspy,
from youtube_transcript_api import YouTubeTranscriptApi
YouTubeTranscriptApi.get_transcript(video_id, proxies={"https": "https://user:pass@domain:port"})

I'm using a paid proxy from smartproxy.com with the "Residential" offer. There are probably other better proxies available; I chose this one randomly.

thank you for sharing. this surely looks like a cheap option but I was looking for something free. don't want to pay in initial stages of my project.

confirmed, using a proxy from my droplet worked. I used this to proxy traffic from my digital ocean droplet to my local laptop. https://docs.border0.com/docs/expose-a-http-proxy which will allow you to expose a proxy on localhost and have it egress on a separate machine (in my case my laptop)
transcript = YouTubeTranscriptApi.get_transcript(video_id, proxies={"https": "http://localhost:8080"})

can make a more details quick video if folks are interested in how to use that.

sure would love a video on it. drop the link here

Would Appreciate a vid. or just more info. ::: I'm all new to proxies etc. seems most info. online is kinda for the more experienced :::

Just ran across this issue today, glad I found this thread. I too am on Digital Ocean, running my code in a Docker container. Getting transcripts runs fine locally, but not on DO.
I would appreciate the video mentioned above, as proxies are new to me. If I use my localhost as a proxy, it means I need to leave the machine running 24/7 right?

Yep, same with me -- looks like youtube blocked many DO servers at once -- i didn't spent so much requests and I'm banned.

I also use Digital Ocean droplet, i think they block IPs from DO servers. now I using google cloud functions.

I can confirm that it is a problem with digital ocean servers being blocked.
Using a proxy is the solutiion.

Now this error also in  google cloud functions.

Has anyone faced same issue on pythonanywhere?

faced the same issue today in aws ec2

same issue today on aws lambda

Hetzner VPS are blocked too.

Same issue, but on Deepnote environment. Does anyone know how to change Deepnote's IP address or something about changing proxies in Deepnote? When I run the same code on my local environment, the transcription works fine. However, it seems like YouTube is blocking Deepnote's IP.

@danielsanmartin I see that you have forked a repository to avoid IP Ban. Can you write instructions how to download CA_BUNDLE ?

I also have the same issue from yesterday. Does anyone know if youtube_transcript_api use any intermediate servers if proxy are not set explicitly?
Because in my case issue is very strange.
youtube-transcript-api works well locally without proxies and with proxies, but when I setup it on Pythonanywhere, it stopped working without proxies and even with proxies.
At the same time when I make a direct request to YouTube public API endpoint https://www.youtube.com/api/timedtext with parameters that I extract via Chrome Web Console -> Network tab and use requests library, in such case it works both locally and at Pythonanywhere, with and without proxies.
What can be the issue?
Might be it relates to headers that youtube_transcript_api generates?

Ho @meera. I used a proxy call from zyte.com to make calls to the YouTubeTranscriptApi class. Since it works via https, I needed to use the certificate. So far this is working. Using my fork, the code is:
from youtube_transcript_api import YouTubeTranscriptApi
YouTubeTranscriptApi.get_transcript('video id', ['language'], proxies={"http": "http://zyte-api-key:@api.zyte.com:8011/","https": "http://zyte-api-key:@api.zyte.com:8011/",}, verify='/path/to/zyte-ca.crt')

Same issue, but on Deepnote environment. Does anyone know how to change Deepnote's IP address or something about changing proxies in Deepnote? When I run the same code on my local environment, the transcription works fine. However, it seems like YouTube is blocking Deepnote's IP.

This may help you.

Same problem. Not sure how it is possible but I can also confirm that the proxy that worked locally isn't working on the AWS Lambda function. I used brightdata isp one.

I deployed on Render. Same problem here. Proxying through smart proxy fixed it. Although smart proxy residential plan starts at $7 per GB.

Ho @meera. I used a proxy call from zyte.com to make calls to the YouTubeTranscriptApi class. Since it works via https, I needed to use the certificate. So far this is working. Using my fork, the code is:
from youtube_transcript_api import YouTubeTranscriptApi
YouTubeTranscriptApi.get_transcript('video id', ['language'], proxies={"http": "http://zyte-api-key:@api.zyte.com:8011/","https": "http://zyte-api-key:@api.zyte.com:8011/",}, verify='/path/to/zyte-ca.crt')

bro.. is functional at now?...

Yh. Just used proxy guys. I solved the problem using NodeMaven proxy @ https://nodemaven.com/.
just like this:
proxy_url = f'http://{username}:{password}@{proxy_host}:{proxy_port}'
transcript = YouTubeTranscriptApi.get_transcript(video_id, languages=lang, proxies={'http': proxy_url,
'https': proxy_url})

Confirm, "Subtitles are disabled for this video" as response using Render.

The best and fully functional option is this one: https://rapidapi.com/DataFanatic/api/youtube-media-downloader, it's also highly scalable and very fast. The free version only allows about 150 calls per month to their API, but it's useful. Additionally, it has been tested on two different hosting providers, Heroku and PythonAnywhere.
Next, I need to test the proxy setup, but I haven't gotten that far yet.

Has any one tried Youtube Data API to retrieve YouTube subtitles? What are pros and cons ?
There is daily quota of 1000. How many requests can you fit in daily quota?

I experienced the same issue, looks like YouTube is blocking IPs. Mine is in AWS EC2. I have a Cloudflare Worker that does the job for now: https://github.com/jamesflores/youtube-subtitles-worker

Hi,
We have also started facing issue in our company where instance is running on amazon eks. Can there be a way to reach out to YT to get ip unblocked ?

Hi, We have also started facing issue in our company where instance is running on aws eks. Can there be a way to reach out to YT to get ip unblocked ?

@AniketModi that's funny!
so essentially you'd be telling Youtube - "hey i was scraping data which is against your policy, you blocked the ip and i want you to unblock it so i can do some more" - it's not possible, they wouldn't even see it. The library works by scraping the youtube video for the transcript which is against their policy. They don't even want you interacting with their video expect through YT.

same issue, youtube block the IP

Hi all,
I am trying to improve the error message on this a bit, but to be able to do so I need to know the exact response that YouTube sends in such cases. Since I currently don't have a machine with a blocked IP available to try this on, it would be great if someone could run the following (on a machine with a blocked IP) and upload the resulting dump.html somewhere for me?
import requests
from youtube_transcript_api._transcripts import TranscriptListFetcher
html = TranscriptListFetcher(requests.Session())._fetch_html("vJEbP2Vdq2U")
with open("dump.html", "w+") as fp:
    fp.write(html)

Thank you! 🙏

Ran your code on my AWS server. I'm having the same issue. https://file.io/Xy37LeKU4TSP

Thanks a lot @iamscottweber!
Interestingly, when rendering the HTML it includes the error message "Sign in to confirm you're not a bot". So this means that you might actually be able to continue scraping if you're signed in! You can do authenticated requests using Cookies, as explained in the README.
Could maybe someone who's currently blocked give this a try and see whether this allows them to continue scraping?
(Please note that I don't know if YouTube will ban your account at some point if you scrape too much, so it might be better to do this with an account you don't care about, just to be on the safe side)

@jdepoix
Just curious - do you know what's the age limit for cookies?

Thanks a lot @iamscottweber!
Interestingly, when rendering the HTML dump it includes the error message "Sign in to confirm you're not a bot". So this means that you might actually be able to continue scraping if you're signed in! You can do authenticated requests using Cookies, as explained in the README.
Could maybe someone who's currently blocked give this a try and see whether this allows them to continue scraping?
(Please note that I don't know if YouTube will ban your account at some point if you scrape too much, so it might be better to do this with an account you don't care about, just to be on the safe side)

Okay! this is pretty interesting.
Since the library is basically just a scraper they put a sign in block. However using cookies puts you at a risk of losing your account, if you wanted to do that you can build your own scraper or better still create a scraper that uses these free transcription websites.
I also have a automation that I am selling for a small fee, you can make a simple request to it and it would return the transcript. If anyone is interested let me know.

@jdepoix
Just curious - do you know what's the age limit for cookies?

there is no right answer to be honest - depends on Youtube or the site, could last anywhere from a day to whatever. Just watch your program closely or build a failsafe around such failures.

I have it working with a secure proxy without cookies and the fork that @danielsanmartin provided.  I'll watch it and possibly incorporate cookies.

My code using YouTubeTranscriptApi works locally but fails on the server with this error:
Failed to retrieve transcript: Subtitles are disabled for this video
I’ve confirmed the subtitles are available and have the same library version on both environments. I also had trouble with proxy settings.
Any suggestions or solutions would be appreciated!

Thanks for creating this thread, I was tearing my hair out to figure out what the hell happened to my AWS lambda function!
So following the suggestions from before, In prod: I tried using a proxy when fetching the transcript (caveat: http proxy), but I still get the same "TranscriptsDisabled" error.
Locally, it works fine. Any idea what this could be about? How is it technically possible that Youtube bans an ip address proxy that works locally but not on AWS servers?
Appreciate any insight you guys might have

I had exactly the same issue on prod with lambda functions. It could be
something that lambda is putting on the header that gives away it’s an AWS
lambda and they might have banned all Lambdas using that. But this is
completely a speculation, haven’t tested it yet.
I used the rapid api solution presented above. Works good for now 👍
On Sun, Aug 11, 2024 at 2:07 PM Satya @.***> wrote:

Thanks for creating this thread, I was tearing my hair out to figure out
what the hell happened to my AWS lambda function!
So following the suggestions from before, In prod: I tried using a proxy
when fetching the transcript (caveat: http proxy), but I still get the same
"TranscriptsDisabled" error.
Locally, it works fine. Any idea what this could be about? How is it
technically possible that Youtube bans an ip address proxy that works
locally but not on AWS servers?
Appreciate any insight you guys might have
—
Reply to this email directly, view it on GitHub
https://github.com/jdepoix/youtube-transcript-api/issues/303#issuecomment-2282720324,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A62HCCVWR7O6PFIV5RPIHZTZQ5AWLAVCNFSM6AAAAABLBOWBE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBSG4ZDAMZSGQ
.
You are receiving this because you commented.Message ID:
@.***>

Thanks a lot @iamscottweber!
Interestingly, when rendering the HTML dump it includes the error message "Sign in to confirm you're not a bot". So this means that you might actually be able to continue scraping if you're signed in! You can do authenticated requests using Cookies, as explained in the README.
Could maybe someone who's currently blocked give this a try and see whether this allows them to continue scraping?
(Please note that I don't know if YouTube will ban your account at some point if you scrape too much, so it might be better to do this with an account you don't care about, just to be on the safe side)

@jdepoix
I ran your code on my aws ec2 linux instance and generated a dump.html
I then opened the html on my local workstation and got the "sign in".  Before I signed in, the transcript was not accessible. After the sign-in the transcript was.  So in essence that part of the test worked.  I wanted to then try using your cookie methodology, however the extension you have listed in your readme is no longer available on chrome.  Do you have another extension you would recommend, or perhaps another way to get to the cookie info so I can continue testing the methodology?

I'd like to understand how the YouTubeTranscript website obtains transcript support, as it must be deployed somewhere to provide this functionality.

I had exactly the same issue on prod with lambda functions. It could be something that lambda is putting on the header that gives away it’s an AWS lambda and they might have banned all Lambdas using that. But this is completely a speculation, haven’t tested it yet. I used the rapid api solution presented above. Works good for now 👍
…
On Sun, Aug 11, 2024 at 2:07 PM Satya @.> wrote: Thanks for creating this thread, I was tearing my hair out to figure out what the hell happened to my AWS lambda function! So following the suggestions from before, In prod: I tried using a proxy when fetching the transcript (caveat: http proxy), but I still get the same "TranscriptsDisabled" error. Locally, it works fine. Any idea what this could be about? How is it technically possible that Youtube bans an ip address proxy that works locally but not on AWS servers? Appreciate any insight you guys might have — Reply to this email directly, view it on GitHub <#303 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/A62HCCVWR7O6PFIV5RPIHZTZQ5AWLAVCNFSM6AAAAABLBOWBE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBSG4ZDAMZSGQ . You are receiving this because you commented.Message ID: @.>
The youtube rapidapi website only supports up to 150 requests per month, so I need a permanent solution for obtaining transcripts consistently.

Thanks a lot @iamscottweber!
Interestingly, when rendering the HTML dump it includes the error message "Sign in to confirm you're not a bot". So this means that you might actually be able to continue scraping if you're signed in! You can do authenticated requests using Cookies, as explained in the README.
Could maybe someone who's currently blocked give this a try and see whether this allows them to continue scraping?
(Please note that I don't know if YouTube will ban your account at some point if you scrape too much, so it might be better to do this with an account you don't care about, just to be on the safe side)

@jdepoix
I ran your code on my aws ec2 linux instance and generated a dump.html
I then opened the html on my local workstation and got the "sign in". Before I signed in, the transcript was not accessible. After the sign-in the transcript was. So in essence that part of the test worked. I wanted to then try using your cookie methodology, however the extension you have listed in your readme is no longer available on chrome. Do you have another extension you would recommend, or perhaps another way to get to the cookie info so I can continue testing the methodology?

UPDATE: The cookie method seems to work.  My purpose is not for usage at scale so I will give this a try until it does not work. I had to install "get cookies clean" https://chromewebstore.google.com/detail/get-cookiestxt-clean/ahmnmhfbokciafffnknlekllgcnafnie?hl=en-US&utm_source=ext_sidebar and export to Netscape cookie format.
WARNING: If used at scale, I can imagine a scenario where the associated account could be impacted.

I have been running the youtube transcript api for my startup over months. We solved the transcript disabled problem long time ago and we have wrote some script in house that make sure that it never break on us. If you are interested in this solution you can reach out to me<EMAIL_ADDRESS>

I have been running the youtube transcript api for my startup over months. We solved the transcript disabled problem long time ago and we have wrote some script in house that make sure that it never break on us. If you are interested in this solution you can reach out to me<EMAIL_ADDRESS>
Send you an email!

Hi @Joe-hitthecode,
Just a quick note—I’ve emailed you at<EMAIL_ADDRESS>regarding the YouTube Transcript API solution. Looking forward to your response!

Hi @Joe-hitthecode,
Just a quick note—I’ve emailed you at<EMAIL_ADDRESS>regarding the YouTube Transcript API solution. Looking forward to your response!

Can you email to me as well. I am  also currently using same API and facing same issue while running the code on AWS EKS but working well when tried to run on my local.

Hi @Joe-hitthecode,
Just a quick note—I’ve emailed you at<EMAIL_ADDRESS>regarding the YouTube Transcript API solution. Looking forward to your response!

Can you pls email to me as well. I am also currently using same API and facing same issue while running the code on AWS EKS but working well when tried to run on my local.

It is a scam. Why such secrecy that one must email to get answer?

lol Meera. It is not a Scam. This is my LinkedInd: https://www.linkedin.com/in/joe-georgeo-slamie-413b7a170/. I am also busy so can't reply very fast as you will need

check my startup here: https://tubesynopsis.vercel.app/about

can you reply me @Joe-hitthecode

I am have a lot of emails. I am preparing a general response for everyone. Like that I don't reply people individually. Part of the solution include firstly trying to use proxy, which I mentioned here before.

@Joe-hitthecode You are going to post the solution here?

@satyajit-bagchi you'll have to proxy your https requests, since the YouTube requests are done using https. Setting up a proxy for http won't do anything.

@Joe-hitthecode I'll try to add a more explicit exception for this type of error when I find time to do so. This should allow for catching them and falling back to a proxy or rotating through a pool of IPs as they get banned.

@jdepoix Ah, thanks for pointing that out!
I'll try out a https proxy. Any chance of YouTube-transcript-api supporting socks5 proxy in the future?
Also, has anyone tried the official YT api and asked for a quota increase? I think the default quota allows only 50 captions a day

For those in the starting stages of their projects, webshare offers a free 10 proxies with the SOCKS5 protocol. You can use a socks5 proxy out of the box with youtube-transcript-api. Just pass it to the proxies dict:
https://stackoverflow.com/questions/12601316/how-to-make-python-requests-work-via-socks-proxy
https://github.com/jdepoix/youtube-transcript-api?tab=readme-ov-file#proxy

I am facing the same issue. It is not working in the cloud server.

I am facing the same issue. It is not working in the cloud server.

You need to signup with a proxy to fix this.

If you are using the SOCKS5 protocol, make sure to install pysocks. pip install pysocks and you are good to go

Thanks for pointing that out @jdepoix :). I now have it working for me on the cloud
To everyone else: For those in the starting stages of their projects, webshare offers a free 10 proxies with the SOCKS5 protocol. You can use a socks5 proxy out of the box with youtube-transcript-api. Just pass it to the proxies dict: https://stackoverflow.com/questions/12601316/how-to-make-python-requests-work-via-socks-proxy https://github.com/jdepoix/youtube-transcript-api?tab=readme-ov-file#proxy

Excellent, worked.
Though still evaluating using the captions api with the Oauth directly, https://developers.google.com/resources/api-libraries/documentation/youtube/v3/python/latest/youtube_v3.captions.html

Thanks for pointing that out @jdepoix :). I now have it working for me on the cloud
To everyone else: For those in the starting stages of their projects, webshare offers a free 10 proxies with the SOCKS5 protocol. You can use a socks5 proxy out of the box with youtube-transcript-api. Just pass it to the proxies dict: https://stackoverflow.com/questions/12601316/how-to-make-python-requests-work-via-socks-proxy https://github.com/jdepoix/youtube-transcript-api?tab=readme-ov-file#proxy

@satyajit-bagchi its not working on Azure.

I ran into this on google cloud run. I tried dataimpulse with a $1/GB pay-as-you-go plan and it worked for me:
transcript = YouTubeTranscriptApi.get_transcript(video_id, proxies={"https": f"https://{dataimpulse_login}:{dataimpulse_password}@gw.dataimpulse.com:823"})

We've noticed this issue cropping up more in the past week, but interestingly it is not happening for all videos.  Sometimes the same video will fail, and then succeed.  Are others seeing this behavior?  Does that still indicate that Youtube is blocking/rate limiting?

Yes, I've noticed a similar behavior where the same video is blocked even with a proxy.  It will sometimes fail and other times work as expected.  But adding the proxy has helped immensely and I'm assuming the fails happen to be on an IP address that is already rate limited.

I am going off my system for a while so let me just go over how we are managing this expected problem. Before I go into it I just want to put it out there that the api we are using is basically employing web scrapping and btw these solutions are bound to be troublesome, however the biggest way we handled it was to use proxy but in a more skillful way. Before during anything else try a proxy that is easy to configure like https://nodemaven.com/ after you get a practical understanding of how to configure proxy then you can move on to using free proxy like https://www.croxyproxy.com/ which is a little complex. If you are successful with these steps, use the domain you get from your proxy provider like this:
proxy_url = f'http://{username}:{password}@{proxy_host}:{proxy_port}' transcript = YouTubeTranscriptApi.get_transcript(video_id, languages=lang, proxies={'http': proxy_url, 'https': proxy_url})
If you cloned the repository and looked at the source code you will see that the place where the error is raised is in the _errors.py file: specifically this error message: class TranscriptsDisabled(CouldNotRetrieveTranscript): CAUSE_MESSAGE = 'Subtitles are disabled for this video' - When you called the .get_transcript method it creates a TranscriptListFetcher class and calls the .fetch method, below in the source code
class TranscriptListFetcher(object): def init(self, http_client): self._http_client = http_client
def fetch(self, video_id):
    return TranscriptList.build(
        self._http_client,
        video_id,
        self._extract_captions_json(self._fetch_video_html(video_id), video_id),
    )

def _extract_captions_json(self, html, video_id):
    splitted_html = html.split('"captions":')

    if len(splitted_html) <= 1:
        if video_id.startswith('http://') or video_id.startswith('https://'):
            raise InvalidVideoId(video_id)
        if 'class="g-recaptcha"' in html:
            raise TooManyRequests(video_id)
        if '"playabilityStatus":' not in html:
            raise VideoUnavailable(video_id)

        raise TranscriptsDisabled(video_id)

    captions_json = json.loads(
        splitted_html[1].split(',"videoDetails')[0].replace('\n', '')
    ).get('playerCaptionsTracklistRenderer')
    if captions_json is None:
        raise TranscriptsDisabled(video_id)

    if 'captionTracks' not in captions_json:
        raise NoTranscriptAvailable(video_id)

    return captions_json

The .fetch method calls the self._extract_caption_json method and there is where the error is. The error is raised when these three conditions are checked and programmatically when these conditions doesn't failed the code illusively raise the TranscriptDisabled method which in some cases could be because of some other reason other than transcript disabled. However the reason this is happening is because your IP is blocked or something. So there is no magic bullet or secret sauce, you have to use a proxy or some type of vpn service that hide away your ip before fetching the transcript data.
ps: My Backend is running on pythonanywhere.com

Jup thats exactly the problem I noticed. Having also problems on AWS. I mean youtube is returning something even with the videoDetails prop, but it is not complete as the crucial part is missing: playerCaptionsTracklistRenderer
So I guess the only solution is right now using proxy/VPN/dynamic IP or the official YT-api

I use yt-dlp as a failover to download the audio then send to my whisper server for transcription. It was failing as well until I added a proxy to it. Appears YouTube just blacklisted all known commercial IPs

For those using the official YT api, it looks as though only the creator of the video can authorize download of captions based on the scopes.  Am I understanding that correctly?

Thanks guys the proxy works. Should i implement proxy rotating logic?

I made a webshare account to get access to Socks5 proxies. I have tried multiple of their proxies and pip installed & imported pysocks as @Joe-hitthecode suggested and keep getting the following exception:
Exception message: SOCKSHTTPSConnectionPool(host='www.youtube.com', port=443): Max retries exceeded with url: /watch?v=kvlWtA136FM (Caused by NewConnectionError('<urllib3.contrib.socks.SOCKSHTTPSConnection object at 0x000001F1FFCEABA0>: Failed to establish a new connection: 0x04: Host unreachable'))
is anybody else experiencing this or does anyone know a solution?

@aj-bei
I think Your not providing webshare credentials proxies
proxies = {
'http': 'socks5://user:pass@proxy_ip:proxy_port',
'https': 'socks5://user:pass@proxy_ip:proxy_port'
}
Define a function to fetch transcripts using proxies
def generate_video_transcript(video_id):
try:
# Fetch transcript with the YouTubeTranscriptApi
response = YouTubeTranscriptApi.get_transcript(video_id, proxies=proxies)
if your try to use different  proxy ip list which has status working

What is proxy? how to use this in ubuntu machine

Just ran across this issue today, glad I found this thread. I too am on Digital Ocean, running my code in a Docker container. Getting transcripts runs fine locally, but not on DO.
I would appreciate the video mentioned above, as proxies are new to me. If I use my localhost as a proxy, it means I need to leave the machine running 24/7 right?

Did you create the Docker container yourself? I am interested in running this from my home server.

@aj-bei
I think Your not providing webshare credentials proxies
proxies = {
'http': 'socks5://user:pass@proxy_ip:proxy_port',

'https': 'socks5://user:pass@proxy_ip:proxy_port'

}
Define a function to fetch transcripts using proxies
def generate_video_transcript(video_id):
try:

    # Fetch transcript with the YouTubeTranscriptApi

    response = YouTubeTranscriptApi.get_transcript(video_id, proxies=proxies)

if your alread doing it,try to use different proxy ip from proxy ip list which has status working

I am not even running on the cloud and all my proxies still don't work

@aj-bei
I think Your not providing webshare credentials proxies
proxies = {
'http': 'socks5://user:pass@proxy_ip:proxy_port',

'https': 'socks5://user:pass@proxy_ip:proxy_port'

}
Define a function to fetch transcripts using proxies
def generate_video_transcript(video_id):
try:

    # Fetch transcript with the YouTubeTranscriptApi

    response = YouTubeTranscriptApi.get_transcript(video_id, proxies=proxies)

if your alread doing it,try to use different proxy ip from proxy ip list which has status working

I am not even running on the cloud and all my proxies still don't work

If your not using in cloud,you will get transcript directly using it in local without any problem.

@aj-bei

I think Your not providing webshare credentials proxies

proxies = {

'http': 'socks5://user:pass@proxy_ip:proxy_port',

'https': 'socks5://user:pass@proxy_ip:proxy_port'

}

Define a function to fetch transcripts using proxies

def generate_video_transcript(video_id):

try:

# Fetch transcript with the YouTubeTranscriptApi

response = YouTubeTranscriptApi.get_transcript(video_id, proxies=proxies)

if your alread doing it,try to use different proxy ip from proxy ip list which has status working

I am not even running on the cloud and all my proxies still don't work

If your not using in cloud,you will get transcript directly using it in local without any problem.

I would think so, but my attempts to run my program using the API locally are met with the same errors as everyone else on the thread, maybe could be a restriction on the WiFi network I am using.

Hello, i am using webshare socks5 proxies
proxies = {
'http': f'socks5://{username}:{password}@{host}:{port}',
'https': f'socks5://{username}:{password}@{host}:{port}'
}
transcript_list = YouTubeTranscriptApi.list_transcripts(video_id, proxies=proxies)
i am still getting this error on aws lambda
ERROR- SOCKSHTTPSConnectionPool(host='www.youtube.com', port=443): Max retries exceeded with url: /watch?v=WTOm65IZneg (Caused by NewConnectionError('<urllib3.contrib.socks.SOCKSHTTPSConnection object at 0x7fe5786f6510>: Failed to establish a new connection: All offered SOCKS5 authentication methods were rejected')) - WTOm65IZneg

I use a tool that can download transcripts one by one from any channel with this script built in. It works for some time and then it stops, but right after I switch a country on my paid VPN it works again, then it stops loading again until I switch to another country and so on.  After some time, it starts working with the same IPs for the limited amount of videos again. So if it does not work for you, make sure your proxy/VPN is actually working well

If you encounter SOCKSHTTPSConnectionPool error, try socks5h not socks5. I have succeed.
https://stackoverflow.com/questions/12601316/how-to-make-python-requests-work-via-socks-proxy

I use VPN, it works for a while, but then got the same "subtitles are disabled for this video" again. I need transcripts for hundreds of videos.

hundreds

@corngk is it a one time thing or is continuous. I have an automation that runs all you have to do is make a request to the link. Send me an email at<EMAIL_ADDRESS>

To get this working reliably in production, adding a proxy layer is essential. If you need help, feel free to reach out: https://linktr.ee/clearcode

Thanks for pointing that out @jdepoix :). I now have it working for me on the cloud
To everyone else: For those in the starting stages of their projects, webshare offers a free 10 proxies with the SOCKS5 protocol. You can use a socks5 proxy out of the box with youtube-transcript-api. Just pass it to the proxies dict: https://stackoverflow.com/questions/12601316/how-to-make-python-requests-work-via-socks-proxy https://github.com/jdepoix/youtube-transcript-api?tab=readme-ov-file#proxy

@satyajit-bagchi its not working on Azure.

I'm currently using Azure with proxies, but it's not working for me either. Is it working for you right now?
Do you know why it might work on AWS and not on Azure? Is there any specific reason for this? Also, what could be the possible solution to resolve this issue in azure?

I am hoping this won't get spammed now. But this is a working solution for free. https://gist.github.com/Ashes47/f03d8f8dfd024783a8a34ba34141d6ec

It worked like a charm!

Can this solution be built in the tool itself as default? @jdepoix

I am using Digital Ocean and the problem still exists.

Problems with using TOR

No IP Rotation, YT can (and will) ban this tommorrow
Latency is gonna be bad at scale for sure with TOR

I know I am late to the party but one of my customers of my solution reported the problem to me. Downloading YouTube transcript using either youtube_transcript_api or YoutubeLoader.from_youtube_url stopped working on Cloud based VM (I am using Amazon Ec2) and only works locally. Pls let me know if there is any workaround.

Is there a free work around...Cant pay for proxies..

https://github.com/jdepoix/youtube-transcript-api/issues/303#issuecomment-2331633359

What are the best, cheap proxy providers people are using?
I have tried free proxies but these don't seem to work so I think I need to go for a paid solution.

I am using Webshare for my YouTube transcript generation website, and it works pretty well. Users generated more than 100K transcripts after adding this proxy. Even the 10 out of 20 free proxies it gives you work pretty well, but there is a bandwidth limit of 1 GB per month, which my user exhausted in 12 hours.
I have also tried Nodemaven (only residential, expensive) and Speedproxies (good, cheap, but Webshare is cheaper and also has a better interface).

I am using Webshare for my YouTube transcript generation website, and it works pretty well. Users generated more than 100K transcripts after adding this proxy. Even the 10 out of 20 free proxies it gives you work pretty well, but there is a bandwidth limit of 1 GB per month, which my user exhausted in 12 hours.
I have also tried Nodemaven (only residential, expensive) and Speedproxies (good, cheap, but Webshare is cheaper and also has a better interface).

are you using residential or static residential and why? Sorry for noob questions, I am new to proxies!

Residential proxies only. I am using residential proxy. Static residential has higher failure rate for me.

I am using Webshare for my YouTube transcript generation website, and it works pretty well. Users generated more than 100K transcripts after adding this proxy. Even the 10 out of 20 free proxies it gives you work pretty well, but there is a bandwidth limit of 1 GB per month, which my user exhausted in 12 hours.
I have also tried Nodemaven (only residential, expensive) and Speedproxies (good, cheap, but Webshare is cheaper and also has a better interface).

Thank you! Are you using residential or static residential and why? Sorry for noob questions, I am new to proxies!

so i tried the sample solution and created a http proxy using border0 but its only works on my local machine and on deployed aks on azure it gives following error:

so i tried the sample solution and created a http proxy using border0 but its only works on my local machine and on deployed aks on azure it gives following error:

additional info: i also tried both ports 8080 and 443 but no luck

My solution for this problem on my VPS was to activate a VPN.
Seams that the IP-Adresses of the VPS Provider (Hetzner) where blocked.
After activating VPN (NordVPN) i was able to get the transcript as on my local pc.
I haven't tried it with a proxy, because i have no plan how to set it up :)

Also can confirm using proxy works. I used smartproxy.com, with $7 per GB.
Want to surface someone in the thread recommended this free tool (limit to 150 calls a month), which does not use this python package, and may require some fiddling (get subtitle URL from video detail call and then use a second call to get the subtitle text from subtitle URL): https://rapidapi.com/DataFanatic/api/youtube-media-downloader

I've had the same problem, but using SmartProxy made it work, which is not free though.
I use "Heroku" as cloud server and it doesn't work on Heroku. ( but it works locally )
Here's what I did. This is just for your reference
go to SmartProxy（https://dashboard.smartproxy.com/residential-proxies/proxy-setup）
→ Residential Proxy（plan: Residential proxies 2GB, Price per GB$6.00 with 3days free trial）
→ Residential Proxy Setup Tab
→ you can see the sample codes for each languages by scrolling to the bottom
my codes are like this ↓
Python

# import requests ← unnecessary code
# url = 'https://ip.smartproxy.com/json' ← unnecessary code
username = 'xxxxxxxxxxx'
password = 'yyyyyyyyyyy'
proxy = f"http://{username}:{password}@[proxy server url]:[port]"
# result = requests.get(url, proxies = { ← unecessary code
#    'http': proxy,
#    'https': proxy
# })

transcript_list = YouTubeTranscriptApi.list_transcripts(video_id, proxies = {
        'http': proxy,
        'https': proxy
    })

Issue with YouTube-transcript-api After Ubuntu UpgradeProblem Statement:
My YouTube-transcript-api has stopped working after upgrading to Ubuntu 24.04.1 LTS (GNU/Linux 6.8.0-45-generic x86_64) on two different Hostinger VPS servers. However, it works perfectly fine on a local, older distro: Ubuntu 20.04.6 LTS (GNU/Linux <IP_ADDRESS>-microsoft-standard-WSL2 x86_64).Environment Details:
Server Type | OS Version | Python Version | YouTube-transcript-api Version
-- | -- | -- | --
Cloud (VPS) | Ubuntu 24.04.1 LTS (GNU/Linux 6.8.0-45) | Python 3.10.x | 0.6.2
Local (WSL) | Ubuntu 20.04.6 LTS (GNU/Linux <IP_ADDRESS>) | Python 3.8.x | 0.6.2
Details:On the cloud servers (Hostinger VPS), the API does not function after the Ubuntu upgrade.On the local machine (running Ubuntu 20.04.6 LTS with WSL), the Yo

I've had the same problem, but using SmartProxy made it work, which is not free though. I use "Heroku" as cloud server and it doesn't work on Heroku. ( but it works locally )
Here's what I did. This is just for your reference
go to SmartProxy（https://dashboard.smartproxy.com/residential-proxies/proxy-setup） → Residential Proxy（plan: Residential proxies 2GB, Price per GB$6.00 with 3days free trial） → Residential Proxy Setup Tab → you can see the sample codes for each languages by scrolling to the bottom
my codes are like this ↓
Python

username = 'xxxxxxxxxxx'
password = 'yyyyyyyyyyy'
proxy = f"http://{username}:{password}@[proxy server url]:[port]"

transcript_list = YouTubeTranscriptApi.list_transcripts(video_id, proxies = {
        'http': proxy,
        'https': proxy
    })

Do you have to use the latest version with a browser cookie? Or does it work for you without providing one?

I've had the same problem, but using SmartProxy made it work, which is not free though. I use "Heroku" as cloud server and it doesn't work on Heroku. ( but it works locally )
Here's what I did. This is just for your reference
go to SmartProxy（https://dashboard.smartproxy.com/residential-proxies/proxy-setup） → Residential Proxy（plan: Residential proxies 2GB, Price per GB$6.00 with 3days free trial） → Residential Proxy Setup Tab → you can see the sample codes for each languages by scrolling to the bottom
my codes are like this ↓
Python

username = 'xxxxxxxxxxx'
password = 'yyyyyyyyyyy'
proxy = f"http://{username}:{password}@[proxy server url]:[port]"

transcript_list = YouTubeTranscriptApi.list_transcripts(video_id, proxies = {
        'http': proxy,
        'https': proxy
    })

Hi, even this is not working in my case. I did the same

Hi @SK3620 , please let me know that you're able to make it work after the deployment to cloud.
Thanks!

This should be added as a limitation to the library, I wouldn't have started my project if I knew this

@OmarGitHubDev there is a warning section in the README that informs you that this uses an unofficial API and therefore I cannot guarantee that YouTube won't stop things from working.

@jdepoix  The warning doesn't document the actual issue, which is that the API doesn't work on non residential IPs. The difference is that the issue in the readme is detectable in a POC, while this issue here is not unless it's deployed, which is the latest step

@OmarGitHubDev I understand your frustration but it is completely your fault for starting a project without going through the Issues. Use this as a learning moment
For your current project, just buy some cheap residential proxy from Webshare; it will cost you less than $5-6 but will give you 250GB of bandwidth.
Out of the 20 you buy, at least 12 will work with YT. And it will come out to be very cheap. I run a transcript site, and people generate thousands of transcripts; no problem while using this proxy. If a proxy starts failing, replace it with a new one.

I'm not frustrated just suggesting so that people are aware. Sorry for giving my feedback!
Btw I'm not a fan of using residential proxies because they're not scalable

Hi all,
So, what is the solution for now?
Hi @therohitdas , how did you implement the proxy. Please write code and show like what exactly should be passed in the proxy dict.
Thanks!

I'm not frustrated just suggesting so that people are aware. Sorry for giving my feedback! Btw I'm not a fan of using residential proxies because they're not scalable

So, what is the solution for now @OmarGitHubDev

Hi @iqbalfarz
This project accepts the proxy in the same way the requests library does. On Webshare, choose the Connection method: Direct connection and download the list of proxies.
You can have it like this:
import requests
from youtube_transcript_api import YouTubeTranscriptApi
from youtube_transcript_api._errors import TranscriptsDisabled

proxy = {
"http": "http://username:password@ip:port",
"https": "http://username:password@ip:port" # Notice that the protocol is http and not https; keep it that way.
} # You need to add both, if you add only one, it won't work.

# Info: You can create an array of proxies. and pick one randomly.

try:
    default_transcript = YouTubeTranscriptApi.get_transcript(video_id, proxies=proxy)
except TranscriptsDisabled:
    print(f'This proxy is blocked; replace it: {proxy["https"]} ')
    # if using an array, remove this proxy from the array
    # todo implement a logic to use a diff proxy
except requests.exceptions.ProxyError:
    print(f'Proxy unavailable')
    # if using an array, remove this proxy from the array
    # todo implement a logic to use a diff proxy
except Exception as e:
    print(e)

A pro tip is to rotate the proxies yourself. Do not use the rotating proxy endpoint. This is true for any proxy service you may choose to use, not just Webshare

Has anyone used the official YouTube Transcript API for a project?
Can you use the transcripts from the official YouTube API for a revenue-generating website?
How easy is it to increase the quota limit?

I believe the daily limit is 10,000 and each caption is 50 quota points.
Any help addressing any of these queries would be much appreciated.
I am looking for an alternative that does not require a proxy due to the cost and slight delay involved (I am currently using smartproxy at $7/GB).
Thanks

Has anyone used the official YouTube Transcript API for a project?

Can you use the transcripts from the official YouTube API for a revenue-generating website?

How easy is it to increase the quota limit?

I believe the daily limit is 10,000 and each caption is 50 quota points.

Any help addressing any of these queries would be much appreciated.

I am looking for an alternative that does not require a proxy due to the cost and slight delay involved (I am currently using smartproxy at $7/GB).

Thanks

Where is there an official YouTube Transcript API? I didn't think one existed.

google cloud console, you should create a project first

I believe the daily limit is 10,000 and each caption is 50 quota points.

The quota cost is 200 points, So you can probably download 50 transcripts per day.

I am looking for an alternative that does not require a proxy due to the cost and slight delay involved (I am currently using smartproxy at $7/GB).

I’ve got a solution that can help reduce your proxy costs. I can code a script that will:

Use static residential proxies from webshare, For around 8$, I get 20 proxies + 50 replacement proxies + 250GB bandwidth .
Pre-test your proxy list: This will ensure blocked proxies are automatically removed before you even use them.
Monitor proxies returning errors: I’ll track proxies that throw the TranscriptDisabled error. If a proxy gives that error 5 times in a row, it will be removed from rotation to avoid unnecessary usage.
retry logic for proxy failing with ProxyError.

I believe you can code it yourself, but if you lack the bandwidth, I can do it for you for $50.
Please note that this setup does not address the latency issue. One method to reduce the latency would be to add a cache.

If anyone is still struggling with this. I have made a working solution using oauth2 that doesn't uses proxies and avoids Youtube ban. You can use it for free on Apify: https://apify.com/invideoiq/video-transcript-scraper. You only pay for Apify usage, however Apify gives you free credit of 5$ which will give you around 5000 transcripts

Hi @hatemmezlini, this is the issues section of an open-source project. This is meant to be a place where we discuss solutions together, such that we can all benefit from it. So if you found a way to avoid being banned, without the use of proxies, it would be great if you could share it here. Of course, you are absolutely free to keep that knowledge to yourself and make a profit out of it, but please don't promote your product in this issue, without contributing anything to the discussion.

@jdepoix apologies! I would love to share how I did it.
There is an oauth plugin that was created by yt-dlp developers. It uses the Youtube on TV client because the token is never refreshed on TVs. All I had to do is a create a dummy gmail account, it asked me for password once and than I saved the token and kept passing it in every request. I believe something similar can be developed here

Thanks for sharing @hatemmezlini!
We had already discovered that another way to circumvent the block is to be authenticated (although we did use cookies thus far, so it's good to know that the oauth token could provide a never-expiring alternative in the future). However, I think most people shied away from using this out of fear of YouTube banning their account after some time (although nobody actually reported that this has happened to them). Do you have a rough estimate of the amount of transcripts you've been able to scrape thus far without your account getting banned?

@jdepoix I am up to around 60 transcripts for the last couple days. The other users are using the same token. We are probably over 100. But can't know for sure. I know it's not too many so I made a fallback to use residential proxies just in case I get banned or token expires. But yes, no one reported being banned from Youtube using oauth

Thank you for the insights @hatemmezlini. That really isn't that much, so I am a bit sceptical to be honest whether you will be able to get away without needing proxies, once you retrieve transcripts at a high volume. But I have no experience with this, so it would be great if you could keep us updated, as I would love to be wrong about this 😄

Sure thing @jdepoix, I will  stress test the API the next few days to see if I get banned from Youtube. I will keep you posted

@jdepoix Yup I got banned after a while 😄 ! I rolled back to using residential proxies

Thanks for the update @hatemmezlini! Did they block the IP or did they ban the account you were using?

You are welcome @jdepoix . They banned the account, I couldn't view youtube videos even from the browser

My teammate, @wnd180, and I discovered that AWS IP blocking caused an issue.
We tested running the server on Google Cloud Platform (GCP), and it was successful! I recommend deploying the server on GCP. 🙌

@hatemmezlini, I am about to feature your fantastic API on my channel but I just need to make sure that using this does not put one's channel or IP address at risk. I ask because I see that your account was banned in testing, as you mentioned, and that you are not using proxies to save money, so what method are you using to keep the user of the API safe? My use case is as part of an automation that grabs YT transcripts and metadata so my use case is 5-10 calls a day or so and the viewers would, I imagine be similar - in other words, we are not building apps on this - but still I want to make sure the API is safe before recommending it.
Also, THANK YOU for creating it! I was bummed when yt-transcript-api stopped working.

@IversusAI Thank you for your kind words. As @jdepoix noted out, promoting the actor is outside the scope of this discussion, so please feel free to reach out<EMAIL_ADDRESS>But in short, no your users have nothing to worry as we don't use their IPs, they only send us the video url and their Apify token for authentication to the Apify platform.

Hi @IversusAI,
just to clarify on this:

I was bummed when yt-transcript-api stopped working.

This module did not stop working! You'll just have to use it with proxies if you're using it with a blocked IP (as is the case with many cloud providers IPs) 🙂

Hello, I've got the same problem here. It works perfectly on my local machine with and without proxies. But on my server (OVH French provider), with and without proxies, this not work at all. I'm usign proxies from Apify.

It seems to work for me when I use the residential proxies.

Possible solution - I set up fastapi on an old android (termux) and then ssh proxy free serveo.net url.  My backend just calls the serveo.net url which triggers the request on the android phone. Very fast and easy, no blocking, no authentication.

somewhat unrelated but i was playing around with this repo on a oracle cloud instance today and ran into this same issue. I was able to use cloudflare warp to proxy. I used this repo to create a warp container and just set the proxy to http://warp:1080.

FWIW, I'm having the exact same issue. I can retrieve the captions/transcripts of any given video locally, but not when I deploy the same code to a server. Super strange issue.

@jdepoix is there a proxy service available that i can use to run it, brightdata doesn't seem to work. Sorry for being out of the topic here

@jdepoix Yup I got banned after a while 😄 ! I rolled back to using residential proxies

could i possibly get the service you are using?

@jackye315

somewhat unrelated but i was playing around with this repo on a oracle cloud instance today and ran into this same issue. I was able to use cloudflare warp to proxy. I used this repo to create a warp container and just set the proxy to http://warp:1080.

How many transcripts were you able to generate?
Did you get banned so far?
This can be a great free alternative to residential proxies.

I havent tested the limits but using it sparingly to get a few video transcripts here and there (~20-30 a day) has been completely fine. Unless you are trying to scrape massive amounts of youtube videos, using the warp proxy has been fine for me.

I deployed on Render. Same problem here. Proxying through smart proxy fixed it. Although smart proxy residential plan starts at $7 per GB.

Can you please tell me how you did this on Render?

the Proxying through smart proxy fixed it , works for me

Hi @FVTAL, that is not quite accurate. When you receive this TranscritpDisabled error, it is because YouTube returns something like "Sign in to confirm you're not a bot". By using a cookie you're basically signing in, which allows you to continue doing requests. However, they will ban your YouTube account eventually if you keep doing a lot of requests, as reported by @hatemmezlini in this thread:

You are welcome @jdepoix . They banned the account, I couldn't view youtube videos even from the browser

So yes, you can use cookies to temporarily work around this issue, but your account will be banned eventually. Therefore, I wouldn't really consider this a proper solution, while using proxies is much more scalable!

Hi @auspy,
from youtube_transcript_api import YouTubeTranscriptApi
YouTubeTranscriptApi.get_transcript(video_id, proxies={"https": "https://user:pass@domain:port"})

I'm using a paid proxy from smartproxy.com with the "Residential" offer. There are probably other better proxies available; I chose this one randomly.

I can confirm that this works. I also got a $7 package at Smart Proxy, and it worked just fine.

Came across the same issue, while running on aws ec2. A bummer for sure. Wondering if anyone knows if this workaround good for production applications, where users will be making 1000s of requests? Does anyone use this for production applications?

I got it working using https://smartproxy.com/.  I'm planning on having a relatively large scale so going to incorporate a proxy pool down the line.
proxy_enabled = os.getenv("USE_PROXY", "false").lower() == "true"
logger.debug(f"Proxy enabled: {proxy_enabled}")

if proxy_enabled:
    transcript = YouTubeTranscriptApi.get_transcript(video_id, proxies=get_proxy_dict())
else:
    transcript = YouTubeTranscriptApi.get_transcript(video_id)

I've been trying with proxymesh and it seems it doesn't works.
Could you guys using smartproxy tell me about your usage per request? 1gb/7$ will be enough up to 1k/month?
Regards

I used tor proxy. it seems working fine on ec2 server so far.
it's free and no need to sign up or configure it.
just install it and use it.
you can check the implementation here.
it's AI slack bot providing function to fetch youtube transcript.
calling youtube-transacript-api
use tor on docker compose

@hatemmezlini  I’ve been using your service on apify for some time now and I really appreciate it, thanks a lot for it!
When YouTube blocked cloud provider IPs, it was really frustrating, but your service has been a lifesaver. That said, I’ve noticed it takes around 10 seconds to get a transcript on apify, which can be a bit slow since our users often send thousands of requests daily. Do you know of any other services that might be able to handle this faster?

Hi @Amiraz64 ! Thank you for your kind words. However promoting a product is outside the scope of this disucussion. If you have any questions please reach out to my email<EMAIL_ADDRESS>But to answer your question, inshort, yes, we developed a large scale API just for that matter:
https://rapidapi.com/invideoiq-invideoiq-default/api/video-transcript-scraper


Key services don't work correctly

As far as I can tell, it looks like systemd path units are always active, so binding to them causes bound units to be activated, even when the paths aren't mounted.  In particular, this line causes ${name}-key.service to activate as soon as the machine is up.  The service immediately fails due to this block, but all the same it being activated is enough to trigger all dependent units.  What this means is that units that depend on keys and are configured to wait for key units before activating are activating as soon as the machine boots, even though the keys aren't present.
Note that the binding line isn't necessary anyways as path units will automatically activate their related services once the path conditions are met.
I think we need to remove the bindsTo line.  I'll put in a PR.
EDIT: My understanding of systemd here is limited and I don't really understand what's going on.  In any case, the path unit does not appear to work as expected and seems to be getting triggered unintentionally, and sometimes failing with unit-start-limit-hit.  I'm struggling to find any docs that can help me understand, but at this point I'll probably put together a fork of colmena that doesn't use systemd path units here.

Hmm, is it possible to come up with a minimal example where this happens? I put my secrets in a separate persistent filesystem that has neededForBoot = true;, and so far it seems to work fine.

I'm now less convinced there's an issue with colmena here and I'm thinking there's a more general issue with how my systemd is configured.  I haven't yet been able to figure out exactly what's going on or get a minimal repro, so I'll close this PR until I have a better understanding of the underlying problem and can propose a fix.  Sorry to waste your time @zhaofengli .


Added string representation for model resources.

Custom resources are printable for easier debugging.
Addressed issues
Proposal for #77
String formats are very much up to discussion - to capture the most important info in each entity.
Screenshots
# Changes this:
Pandora error: value <Resource#-9223368908105259524> is incompatible with type reference
# into this:
Pandora error: value <PandoraReference<PandoraEntity 'Epic'>> is incompatible with type reference

Thank you for this!


As a developer I would like to interactively explore the ID(s) which implement a requirement ID.

Prerequisites

[x] I have written a descriptive issue title
[x] I have verified that I am running the latest version of reqifviewer
[x] I have searched open and closed issues to ensure it has not already been reported

Description

Feature relates to traceability and change impact analysis functionality category.
Steps to Reproduce

System Configuration

ReqIFSharp version:

[ ] reqifviewer:
[ ] Other:

Environment (Operating system, version and so on):
.NET Framework version:
Additional information:

@fkromer can you be a bit more explicit on what you expect from the feature request. From the description and i cannot easily make out what you would like to see in the app.

@fkromer any update on this?


How to enable debugging of the collector in the layer

I'm having a problem where it seems that traces that are 'too big' don't send at all. I'm trying to see if there's any additional information I can get out of the collector, since no errors are generated that I can see. According to some other threads I've found, you can enable collector debugging with loggingLevel in extracfg.txt, but I can't figure out how to do that with the lambda layer, unless I need to build my own?

sethAmazon commented on Jun 28, 2021
From the docs
enable debugging log
add a key value pair into /opt/aws/aws-otel-collector/etc/extracfg.txt and restart collector
echo "loggingLevel=DEBUG" | sudo tee -a /opt/aws/aws-otel-collector/etc/extracfg.txt
sudo /opt/aws/aws-otel-collector/bin/aws-otel-collector-ctl -a stop
sudo /opt/aws/aws-otel-collector/bin/aws-otel-collector-ctl -a start

https://github.com/aws-observability/aws-otel-collector/issues/547#issuecomment-869701111
My underlying problem is that it seems if I send either too many spans with my trace, or a small number of spans with even moderately sized attributes (just a few hundred characters), none of my spans are sent to Honeycomb. I must be hitting some limit, but there's no errors printed out to the lambda logs.
My collector.yaml file looks like this currently. The spans I'm trying to send do output to the lambda logs in a debug message, but there's no errors displayed that look anything like a size limit, or a send failure.
receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  otlp:
    endpoint: api.honeycomb.io:443
    headers:
      "x-honeycomb-team": ${HONEYCOMB_KEY}
  logging:
    loglevel: debug

service:
  telemetry:
    logs:
      level: debug
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [otlp, logging]

Hey @MarkFischer thanks a lot for raising this issue! Please note that on GitHub we have a best effort policy in resolving queries. If you have Enterprise support, please create a support ticket and we will get back to you, internally.

Changing the loggingLevel in extracfg.txt is used in the ADOT collector distribution to set the service::telemetry::logs::level value in the configuration.  You have already done that in your configuration, so you're not missing anything there.  You can set the OPENTELEMETRY_EXTENSION_LOG_LEVEL environment variable to set the logging level for the lambda extension part of the embedded collector.

Hey @MarkFischer thanks a lot for raising this issue! Please note that on GitHub we have a best effort policy in resolving queries. If you have Enterprise support, please create a support ticket and we will get back to you, internally.

Thanks, I do have an issue in for the underlying cause of this, not specifically relating to the configuring logging issue. Not sure if giving you the case ID here is a good idea or would help, or if y'all will just find it eventually internally. :)

Changing the loggingLevel in extracfg.txt is used in the ADOT collector distribution to set the service::telemetry::logs::level value in the configuration. You have already done that in your configuration, so you're not missing anything there. You can set the OPENTELEMETRY_EXTENSION_LOG_LEVEL environment variable to set the logging level for the lambda extension part of the embedded collector.

Thanks, that's what I was thinking would be the case, although I'm still not really seeing any useful errors or debug messages that would give me a hint as to what's going on when things don't work. Hopefully support comes back with something.

Could you please share the support ticket with me via<EMAIL_ADDRESS>and I'll look into it?

@Aneurysm9 basically answered the original question, so I will close this one.
For the underlying problem with running things locally with SAM seems to be a problem on the SAM side. I heard back from our AWS support rep:

If the telemetry API is not implemented in SAM, the OTEL extension will not be able to subscribe to the events that it needs in order to collect all the required data and is expected to fail as a result. I'm not sure why other versions of the layer were behaving differently, it's possible that this was just a difference in how they were handling the error or logging as it seems from the customers research none of these were fully functional in SAM.

This looks like the related bug on the SAM CLI side of things: https://github.com/aws/aws-sam-cli/issues/4570
Thank you everyone!


Creating job with an undefined priority causes redis error

Dsing the priority method with an undefined value causes when creating a job causes node_redis to complain with:
node_redis: Deprecated: The HSET command contains a "undefined" argument.
This is converted to a "undefined" string now and will return an error from v.3.0 on.
Please handle this in your code to make sure everything works as you intended it to.

Would it not be preferable to have Kue throw an exception  if the priority is undefined?
It took me some time to find the cause of this problem in my code :p

I'd rather like kue to set a default priority, and I expected so!
Can you confirm and create a PR if not?

It does not set the default priority when job.priority is used with either undefined as argument or anything that is not in the priority map or a function.
I can fix it.
To what branch do you want the pull?

@behrad
Why is there a 'critical' priority in:
var priorities = exports.priorities = {
  low: 10, normal: 0, medium: -5, high: -10, critical: -15
};

whilst the docs for the Job.prototype.priority does not mention it
and states that the priority should be between -10 and 10?
Is 'critical' priority reserved for the internal implementation?
Should the user be able to set it?

Is 'critical' priority reserved for the internal implementation?

No

whilst the docs for the Job.prototype.priority does not mention it
and states that the priority should be between -10 and 10?

Documentation Issue, user can use and set to it :)

I'd expect all jobs priority to be normal if not set by user :)

@behrad
I agree that the default should normal what i meant to say was that if the argument to priority is malformed an error/warning should be given and not silently set to default.


What is the IDE you are using?

i have found your blog.thanks
https://medium.com/@cesarmcferreira/what-terminal-am-i-using-cesar-ferreira-2e19e5f58fc5


Schema Registry Config SslCaLocation doesn't work

Description
Nuget Version: Confluent.SchemaRegistry 2.3.0.
Apache Kafka version: Doesn't have influence
Operating system: Windows
SchemaRegistry Client is adding the CA certificates from the configuration SslCaLocation and SslKeystoreLocation to the HttpClientHandler.ClientCertificates Property:

https://github.com/confluentinc/confluent-kafka-dotnet/blob/007a8ba83ce1c769b294e690309a48414a111944/src/Confluent.SchemaRegistry/CachedSchemaRegistryClient.cs#L298
https://github.com/confluentinc/confluent-kafka-dotnet/blob/007a8ba83ce1c769b294e690309a48414a111944/src/Confluent.SchemaRegistry/Rest/RestService.cs#L89

But from the documentation, only the SSlKeystoreLocation should be placed on this HttpHandlers property for MTLS. The CA certificates should be handled differently: https://learn.microsoft.com/en-us/dotnet/api/system.net.http.httpclienthandler.clientcertificates?view=netcore-2.0
So, the current implementation doesn't allow you to connect to a schema registry server if it uses a certificate signed by a local CA. It throws the following exception:
Local: Key serialization error
 ---> System.Net.Http.HttpRequestException: [SERVER_URL] HttpRequestException: The SSL connection could not be established, see inner exception.
   at Confluent.SchemaRegistry.RestService.ExecuteOnOneInstanceAsync(Func`1 createRequest) in...

How to reproduce
Execute this code using a SchemaRegistry server using a locally signed SSL certificate:
var schemaConfig = new SchemaRegistryConfig
{
    Url = SCHEMA_REGISTRY_URL,
    SslKeystoreLocation = CERTIFICATE_PATH,
    SslKeystorePassword = CERTIFICATE_PASSWORD,
    SslCaLocation = TRUSTED_CA_FILEPATH,
};

var schemaregistry = new CachedSchemaRegistryClient(schemaConfig);
var keySerializer = new AvroSerializer<Application>(schemaregistry).AsSyncOverAsync();
var valueSerializer = new AvroSerializer<Event>(schemaregistry).AsSyncOverAsync();

var privKeyBytes = Convert.FromBase64String(PRIVATE_KEY_BASE64);
var privKeyPem = PemEncoding.Write("RSA PRIVATE KEY", privKeyBytes);

var producerConfig = new ProducerConfig
{
    BootstrapServers = BOOTSTRAP_SERVERS,

    SocketTimeoutMs = 30_000,
    ReconnectBackoffMaxMs = 10_000,

    SecurityProtocol = SecurityProtocol.Ssl,
    SslEndpointIdentificationAlgorithm = SslEndpointIdentificationAlgorithm.None,
    EnableSslCertificateVerification = true,
    SslCertificateLocation = PUBLIC_KEY_FILE_PATH,
    SslCaCertificateStores = "My",
    SslKeyPem = new string(privKeyPem),
};

var producer = new ProducerBuilder<Application, Event>(producerConfig)
    .SetKeySerializer(keySerializer)
    .SetValueSerializer(valueSerializer)
.Build();

var message = new Message<Application, Event>
{
    Key = EventHelper.CreateApplication(),
    Value = EventHelper.CreateEvent()
};

var result = await producer.ProduceAsync(TOPIC, message);

Console.WriteLine($"Written to offset {result.Offset.Value}");

The SchemaRegistry is hiding the InnerException, but with some debugging, you can check the real exception on
https://github.com/confluentinc/confluent-kafka-dotnet/blob/007a8ba83ce1c769b294e690309a48414a111944/src/Confluent.SchemaRegistry/Rest/RestService.cs#L233
The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: UntrustedRoot
   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Boolean async, CancellationToken requestCancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Confluent.SchemaRegistry.RestService.ExecuteOnOneInstanceAsync(Func`1 createRequest) in

Workarounds

Disable the SSL Certificate Checking using the EnableSslCertificateVerification=false, which is not recommended at all.

If you add the same file used in TRUSTED_CA_FILEPATH to the Trust Root Certificate Authorities in Windows Certificate Stores, you can execute the code with success. The problem is that in some cases (Azure Function Apps), you can't do this operation

Sugestion for solution
We could use the same approach as the Confluent.Kafka.ProducerConfig: being able to specify which certificate store you want to use:
https://github.com/confluentinc/confluent-kafka-dotnet/blob/007a8ba83ce1c769b294e690309a48414a111944/src/Confluent.Kafka/Config_gen.cs#L759
From .NET 5 and on, it is possible to add new Certificate Stores to the HttpClientHandler
https://github.com/dotnet/runtime/issues/39835#issuecomment-663020581
I would be happy to send a PR implementing the suggestion or another approach.
Checklist
Please provide the following information:

[x] A complete (i.e., we can run it) minimal program demonstrating the problem. No need to supply a project file.
[x] Confluent.Kafka nuget version.
[x] Apache Kafka version.
[x] Client configuration.
[x] Operating system.
[x] Provide logs (with "debug" : "..." as necessary in configuration).
[x] Provide broker log excerpts.
[x] Critical issue.

I too face the same !

Hi Team, As the PR is merged when is this planned to release. Please let us know as we have the same issue and wanted to check if this already in the pipeline of release with in a few weeks

+1, we are also facing issues with this, especially when dotnet apps run in function apps in azure where we cannot import certificates into the Systems Truststore. Please fix this asap, really bad that it has still not been picked up because now the clients are lacking security and are vulnerable when setting up the connection to schema registry.


skip validation when configmap is empty

This change is

Coverage decreased (-0.02%) to 43.114% when pulling 1e1e4dca9418354a935a2b8f0aeb3d9d0429271f on gianrubio:skip-validation into 153fdf516e781ddf16befeb1e3794806bb979644 on kubernetes:master.

/lgtm


Fix parsing of cell path

Fix parsing when paths contain hyphens or underscores (e.g. $nu.home-path, $env.LAST_EXIT_CODE).
This fixes highlighting issues when paths contain hyphens or underscores.
Fix #33

maybe this is some windows line ending prettier problem in the ci? not sure.

i wonder if prettier could tell what the format error is exactly, just like rustfmt does :thinking:
that'd be more helpful maybe

Same question here, will this fix the CI? https://github.com/prettier/prettier/issues/7853#issuecomment-603843422


AnimationPlayer Call Method Track not being able to select C# custom function from editor

Godot version:
Godot Engine v3.2.1.stable.mono.official
OS/device including version:
Ubuntu 18.04.4 LTS
Issue description:
I added a Call Method Track to an AnimationPlayer, which is a child node of another node that has a C# script attached to it.
Then I tried to add a key to said track to call a custom function on the node's script, I am able to see the function I want to call as well as the other public functions on the script, but when selecting it I get the following error message: Method not found in object: ChangeToNextScene.
A workaround for this, which is pretty much the same used on this issue https://github.com/godotengine/godot/issues/19869, was to just assign any arbitrary non-custom function to the key, and on AnimationTrackKeyEdit on the inspector change the function name to what it actually should be, which in this case was ChangeToNextScene and it actually works.
At the end of the day, doing this workaround works and is not a lot of a hassle, but we probably shouldn't be relying on it.

Yes, I can confirm that, and thanks for the workaround method.

I had this problem too, what I did is to recompile the project (by starting the game) and it found my method right after.

Fixed by #52607.


Can't dynamically set disabledDates via AJAX call

Hi,
I am trying to set the disabledDates via an AJAX call to a PHP script which reads a database and returns an array of dates to disable when the user clicks to the next calendar month. I know the data returned from the AJAX call is ok as I do this when initializing the calendar for the current month and this works ok (initGetDisabledDays). But when I use the same code for a subsequent month and use the 'configure' method the disabled days are not set (getDisabledDays).
My jQuery code is:
  // Calendar
  $('.calendar').pignoseCalendar({
    init: initGetDisabledDays,
    select: onSelectHandler,
    page: getDisabledDays,
    format: 'D MMMM YYYY',
    minDate: new Date(),
    theme: 'blue',
    week: 1
  });

  function initGetDisabledDays(context) {
    /**
     * @context is an object which stored calendar internal data.
     * @context.calendar is a root element reference.
     * @context.calendar is a calendar element reference.
     * @context.storage.activeDates is all toggled data, If you use toggle type calendar.
     * @context.storage.events is all events associated to this date
     */

    var $element = context.element;
    var $calendar = context.calendar;
    var d = new Date();
    m = d.getMonth();
    $.ajax({
       url: "getDisabledDays.php?month="+m+"&standardBikes="+standardBikes+"&eBikes="+eBikes+"&bikesReqd="+numberOfBikes,
       type: "GET",
       success: function(data){
           alert (JSON.parse(data));
           $('.calendar').pignoseCalendar({
            select: onSelectHandler,
            page: getDisabledDays,
            format: 'D MMMM YYYY',
            minDate: new Date(),
            theme: 'blue',
            week: 1,
            disabledDates: JSON.parse(data)
           })
       }
    })
  }

  function getDisabledDays(info, context) {
    /**
     * @context is an object which stored calendar internal data.
     * @context.calendar is a root element reference.
     * @context.calendar is a calendar element reference.
     * @context.storage.activeDates is all toggled data, If you use toggle type calendar.
     * @context.storage.events is all events associated to this date
     */

    var $element = context.element;
    var $calendar = context.calendar;
    m = info.month-1;
    $.ajax({
      url: "getDisabledDays.php?month="+m+"&standardBikes="+standardBikes+"&eBikes="+eBikes+"&bikesReqd="+numberOfBikes,
      type: "GET",
      success: function(data){
        if (data.indexOf("No rows") >= 0){
          alert("No data returned");
        }
        else{
          alert (JSON.parse(data));
          $('.calendar').pignoseCalendar('configure', {
            disabledDates: JSON.parse(data)
          })
        }
       }
    })
  }

Am I using the wrong method to dynamically set the disabled days?

I have the same issue. Can anyone help ?

i have a same problem please help @KennethanCeyer


Making style changes

Hey I have reviewed the code
& I can make some awesome design changes to the project
Do assign me this issue under hacktoberfest

Hey I have made some design and aesthetic changes
confirm so I start a PR 👇

Hey I have made some design and aesthetic changes
confirm so I start a PR 👇


Colored comment unreadable for certain block comments

Since we can't guarantee a good color contrast, we might want to go back to a single color theme for warnings.

Fixed


chore: add devcontainer

:tada: This PR is included in version 1.12.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


add hive dialect to sql provider

@matthewwardrop
Also a bugfix for the sql replace.
Do you think that we should eventually fully templatize the the selected dialects quote's ?

Thanks Dan!
Regarding full templatisation... not sure yet. It makes the code so much harder to read; but mayhap it is worth it. When it comes time to do code cleanup and documentation, I'll take a closer look.


Add support for profiles in the Maven deployment scripts

The deployment scripts for Maven should be able to include profiles

Done


These dependencies were not found:

~/config.json in ./utils/lock.js
~components/ForkThis in ./node_modules/babel-loader/lib?{"babelrc":false,"cacheDirectory":true,"presets":["E://private_projects//example-auth0//node_modules//babel-preset-vue-app//dist//index.common.js"]}!./node_modules/vue-loader/lib/selector.js?type=script&index=0&bustCache!./layouts/default.vue
~components/Navbar in ./node_modules/babel-loader/lib?{"babelrc":false,"cacheDirectory":true,"presets":["E://private_projects//example-auth0//node_modules//babel-preset-vue-app//dist//index.common.js"]}!./node_modules/vue-loader/lib/selector.js?type=script&index=0&bustCache!./layouts/default.vue
~components/SuperSecretDiv in ./node_modules/babel-loader/lib?{"babelrc":false,"cacheDirectory":true,"presets":["E://private_projects//example-auth0//node_modules//babel-preset-vue-app//dist//index.common.js"]}!./node_modules/vue-loader/lib/selector.js?type=script&index=0&bustCache!./pages/index.vue

To install them, you can run: npm install --save ~/config.json ~components/ForkThis ~components/Navbar ~components/SuperSecretDiv
When I execute the command“npm install”"npm run dev",What happened？

you 把  ~components 改成 ../components 试试

I like this:

npm install --save ~/config.json ~components/rkThis ~components/Navbar ~components/SuperSecretDiv
npm ERR! code ENOLOCAL
npm ERR! Could not install from "../../../config.json" as it does not contain a package.json file.

I have the same error. Please, fix this!

I solved this by adding a forward slash on those three dependencies.
So index.vue, line 21 went from:
import SuperSecretDiv from '~components/SuperSecretDiv'
to:
import SuperSecretDiv from '~/components/SuperSecretDiv'
And default.vue lines 12 & 13, from:
import Navbar from '~components/Navbar'
import ForkThis from '~components/ForkThis'
to:
import Navbar from '~/components/Navbar'
import ForkThis from '~/components/ForkThis'

Warpedpuppy has it right.
Adding forward slashes helps. You will need to also edit the store/index.js and replace the above three lines with this:
export const state = () => ({
  user: null
})

As was said in
Probably has to do with nuxt development improving and this was an old example. Needs a tiny update this git to get it to work


[BEAM-1584] Add clean up for bigquery integration test in Python

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Run mvn clean verify to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Add clean up util for bigquery integration test and enable it for BigqueryTornadoesIT. Clean up will be executed no matter whether pipeline job is successful or verification is passed.

+R: @aaltay

retest this please

Coverage decreased (-0.002%) to 70.168% when pulling 1b12297ca3cbbaf340afe2d2c8eb281596a87ed5 on markflyhigh:cleanup into ba5e314664563f23ac8c5125b89a12ddbbd4e433 on apache:master.

Run Python Postcommit

LGTM generally. What do you think about my other comment about renaming to error to something more test specific like GcpTestIOError?

Will do. just want to make sure everything else is good to you.

Run Python Postcommit

LGTM, I can merge after tests.

Done. Also move cleanup function registration before pipeline execution and verification, in order to make sure cleanup will be called even when pipeline failed or verification failed.

Coverage decreased (-0.007%) to 70.163% when pulling 864bab97d53b6add9b80a74189b38a9cca5042e8 on markflyhigh:cleanup into ba5e314664563f23ac8c5125b89a12ddbbd4e433 on apache:master.