[SPARK-21869][SS] A cached Kafka producer should not be closed if any task is using it.

What changes were proposed in this pull request?
By updating the access time for the producer on each iteration, we can ensure that during long running task, we don't close an in-use producer.

Test build #81281 has started for PR 19096 at commit f21bf16.

Test build #81282 has started for PR 19096 at commit 0afb48c.

Test build #81281 has finished for PR 19096 at commit f21bf16.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81281/
Test PASSed.

Test build #81282 has finished for PR 19096 at commit 0afb48c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81282/
Test PASSed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81557/
Test FAILed.

Test build #81558 has started for PR 19096 at commit 7013e8a.

Test build #81558 has finished for PR 19096 at commit 7013e8a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81558/
Test PASSed.

Hi @zsxwing, are you okay with the changes?

Test build #85022 has started for PR 19096 at commit ab4c1dd.

Test build #85022 has finished for PR 19096 at commit ab4c1dd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85022/
Test PASSed.

Test build #85169 has started for PR 19096 at commit 74bcdbf.

Test build #85170 has started for PR 19096 at commit f609d7e.

Test build #85174 has started for PR 19096 at commit 488c70f.

Test build #85169 has finished for PR 19096 at commit 74bcdbf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85169/
Test PASSed.

Test build #85170 has finished for PR 19096 at commit f609d7e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85170/
Test PASSed.

Test build #85176 has started for PR 19096 at commit dbabbf9.

Test build #85174 has finished for PR 19096 at commit 488c70f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85174/
Test PASSed.

Test build #85180 has started for PR 19096 at commit edd5bc3.

Test build #85176 has finished for PR 19096 at commit dbabbf9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85176/
Test PASSed.

Test build #85180 has finished for PR 19096 at commit edd5bc3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85180/
Test PASSed.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2133/
Test PASSed.

@HeartSaVioR Thanks for your interest, let me reopen and take some time to update it.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8097/
Test PASSed.

Test build #102535 has started for PR 19096 at commit 024a407.

Test build #102535 has finished for PR 19096 at commit 024a407.

This patch fails due to an unknown error code, -9.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/102535/
Test FAILed.

@ScrapCodes just wondering which scenario ends-up in multi-threaded access to the same KafkaConsumer where the key is a TopicPartition? If there is no such case I would say no benefit and just increase the complexity without value.

As a general comment +1 on solving this issue. If a task uses a producer more than the configured eviction time it will be closed.

This is a useful fix, a customer recently hit this while running a task over 10 minutes. The producer being closed causes an odd message in the logs.

@skonto maybe it's better to review when the conflicts resolved since the code may change significantly.

@gaborgsomogyi I will, just didnt know what was planned here.

Yeah, I am working on it. The code will change significantly.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8654/
Test PASSed.

Test build #103210 has started for PR 19096 at commit 4d57a82.

Hello All,
This is initial preview, I have pushed it. So that I can get some initial feedback on the current approach.

Test build #103210 has finished for PR 19096 at commit 4d57a82.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103210/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8847/
Test PASSed.

Test build #103427 has started for PR 19096 at commit 0655e6b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8848/
Test PASSed.

Test build #103428 has started for PR 19096 at commit 7a02de9.

Test build #103427 has finished for PR 19096 at commit 0655e6b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103427/
Test PASSed.

Test build #103428 has finished for PR 19096 at commit 7a02de9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103428/
Test PASSed.

@gaborgsomogyi Did you get a chance to look at the updated code? with above changes, it works okay in the local cluster. Do you have further comments?

@gaborgsomogyi and @skonto Thank you for taking a look, several improvements to the patch could be made, due to that. If possible, please take a look at the updated patch, and let me know if you have any further comments.

@ScrapCodes please see my suggestion added in the review main comment:

Would be good to add some stress test because it involves race possibility.

Since threading is essential here...

The more I'm considering cases and checking the code the more I have a general fear. Namely if for some reason the inUseCount gets out of sync there will be no way back and the query may stop produce anything. A good example when delegation token expires and producer not re-created.

Test build #103712 has started for PR 19096 at commit f6ad7ec.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9099/
Test PASSed.

Test build #103712 has finished for PR 19096 at commit f6ad7ec.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103712/
Test PASSed.

@zsxwing Thanks for taking a look again.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9114/
Test PASSed.

Test build #103733 has started for PR 19096 at commit 7174217.

Test build #103733 has finished for PR 19096 at commit 7174217.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103733/
Test PASSed.

Test build #103971 has started for PR 19096 at commit f287d79.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9328/
Test PASSed.

Test build #103971 has finished for PR 19096 at commit f287d79.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103971/
Test PASSed.

Test build #103974 has started for PR 19096 at commit f5406bb.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9329/
Test PASSed.

Test build #103975 has started for PR 19096 at commit a1b4311.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9330/
Test PASSed.

Test build #103974 has finished for PR 19096 at commit f5406bb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103974/
Test PASSed.

Test build #103975 has finished for PR 19096 at commit a1b4311.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/103975/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9353/
Test PASSed.

Test build #104003 has started for PR 19096 at commit ac3af2f.

Test build #104003 has finished for PR 19096 at commit ac3af2f.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104003/
Test FAILed.

Not sure its relevant:
19/03/27 00:05:14.049 Thread-3 ERROR KafkaTestUtils: Found a leak KafkaTestUtils.
org.apache.spark.SparkException: It was created at:
	at org.apache.spark.sql.kafka010.KafkaTestUtils.setup(KafkaTestUtils.scala:134)
	at org.apache.spark.sql.kafka010.KafkaDataConsumerSuite.beforeAll(KafkaDataConsumerSuite.scala:43)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:54)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:507)
	at sbt.ForkMain$Run$2.call(ForkMain.java:296)
	at sbt.ForkMain$Run$2.call(ForkMain.java:286)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

retest this please

Test build #104007 has started for PR 19096 at commit ac3af2f.

Test build #104007 has finished for PR 19096 at commit ac3af2f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104007/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9393/
Test PASSed.

Test build #104044 has started for PR 19096 at commit 83dd0a4.

Test build #104044 has finished for PR 19096 at commit 83dd0a4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104044/
Test PASSed.

As there are several discussions are there could you please resolve the solved ones? It would help the review side not to parse it over and over again.

I'm really sorry to say on the PR which already goes through pretty long discussions and actually I asked to resurrect, so blame me please.
First of all, I guess the patch is addressing the issue properly in current caching approach, and we should adopt this patch if we stick with current approach. (That's why I said the patch is still needed and asked to resurrect this.)
Revisiting the current caching approach based on the fact, the caching approach now requires concerning about thread-safety, and to handle it we are also adding flags and counters to handle, making cached producer instances being stateful (except the things Kafka producer itself may have), which would make us not easy to understand and maintain the code.
If I remember it correctly, in SPARK-25151 (#22138 - the issue is still open and valid I think) someone (I guess Gabor) asked me "why don't you apply commons-pool to producer as well", I answered producer is thread-safe so can apply current cache approach whereas consumer is not thread-safe so it should have to be isolated (good to apply general connection pool implementation).
Now I feel it's worth to consider applying it to producer as well (isolating per thread) which keeps producer to be stateless (same here, except the things Kafka producer itself may have) - it should require more connections to Kafka if there're concurrent usages, but once the number of necessary producers are pooled it would not hurt, which we expect for long running query like streaming one.
Would like to hear everyone's voices. Thanks in advance!

Test build #104082 has started for PR 19096 at commit 114e8b4.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9430/
Test PASSed.

Test build #104082 has finished for PR 19096 at commit 114e8b4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104082/
Test PASSed.

From direction perspective I think commons pool is a good choice in general.
When I apply this to this specific case I'm not really sure about this for the following reasons:

The use-case is much more simple than the consumer side
Guava interface what we're using is thread-safe (the not thread-safe part in Spark code has no brain surgery complexity and don't foresee it will expand heavily)
Not sure if this sudden technology change would provide enough confidence for committers to merge it

I would personally round-up this approach and change it only if committer explicitly request it.

I had some kind of wondering that how much beneficial the complexity we are adding is, because the current code is a kind of no-brainer so it has benefits with no specific concern, but now it concerns about concurrency - actually not feeling much from the code diff (maybe because they were addressed), but from review comments concerning scope of synchronization, race condition, deadlock, etc.
While I mentioned general connection pool, I basically meant thread-isolated usage. I wouldn't concern producer takes same approach as consumer with current custom cache code. (Though I think getting rid of custom cache code would be ideal.)
But you're right I agree someone can manage the complexity (given there're reviewers and at least one committers will review and merge) when this patch is merged, then my wondering is not a big deal. And while we don't know clearly about benefits on current approach against thread-isolated usage  (like perf. benefit via co-use of producer), we also don't know clearly about opposite way. So let's leave it as a second-thought.

@HeartSaVioR could you re-check the answers?

@gaborgsomogyi Looks like @ScrapCodes and I don't reach consensus on my comments. You can review my comments and leave your comments in case-by-case manner.
I would want to make it clear: I'm not a blocker for this PR. If majority including commiter(s) think my comment is not valid, the comment can be ignored.

@HeartSaVioR because 2 comments effect threading consistency I would personally sort them out.

Test build #104470 has started for PR 19096 at commit da71589.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9800/
Test PASSed.

Test build #104470 has finished for PR 19096 at commit da71589.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104470/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9805/
Test PASSed.

Test build #104475 has started for PR 19096 at commit da71589.

retest this please.

Test build #104475 has finished for PR 19096 at commit da71589.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104475/
Test PASSed.

da71589 works for me. Thanks @ScrapCodes to update the code.

Test build #104539 has started for PR 19096 at commit e3890a5.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9865/
Test PASSed.

Test build #104540 has started for PR 19096 at commit a3443ea.

Test build #104539 has finished for PR 19096 at commit e3890a5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104539/
Test PASSed.

Test build #104540 has finished for PR 19096 at commit a3443ea.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104540/
Test PASSed.

Hi @zsxwing, can you please take a look.

I am not sure why "Changes requested" status did not clear up, inspite of making changes on the PR. Do I need to force push?

@zsxwing requested changes, I think he has to change this.

@zsxwing Ping !

@ScrapCodes could you resolve the conflict?
@srowen could you help us out with a review. I would like to round-up the producer side as well, just like in https://github.com/apache/spark/pull/24590. This PR would be an initial step.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10622/
Test PASSed.

Test build #105355 has started for PR 19096 at commit 5c9e9ef.

Test build #105355 has finished for PR 19096 at commit 5c9e9ef.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105355/
Test PASSed.

@ScrapCodes may I ask to resolve conflicts?
@vanzin not much movement here but this blocks to finish delegation token (additionally several users suffering with long running batch queries). https://github.com/apache/spark/pull/23956 is required to pick up the latest token but this PR is needed to finish it. I would like to ask you to have a look please.

@gaborgsomogyi Yes will do it soon.

Test build #107440 has started for PR 19096 at commit 972164d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12575/
Test PASSed.

Test build #107440 has finished for PR 19096 at commit 972164d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107440/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12657/
Test PASSed.

Test build #107531 has started for PR 19096 at commit 8a0906e.

Test build #107531 has finished for PR 19096 at commit 8a0906e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107531/
Test PASSed.

To say that we will never release a resource until a task says it's okay is inherently dangerous in a distributed system

That's also the way how consumer instances are cached (missing close brought file descriptor leak - #21997), though the logic is pretty much simpler since we don't allow co-use of instance (Kafka consumer is not thread-safe, whereas Kafka producer is thread-safe). Same behavior applies to Commons Pool as well, once you borrow the instance, pool never forces to destroy the instance until you return it.

If we think it should be longer than 10 minutes, let's increase it

I don't think it's realistic to expect the user to measure the average time of a batch query and set the timeout accordingly.

I don't know where the 10 minutes comes from

There is a config named spark.kafka.producer.cache.timeout on master, previously it was with different name. As a workaround users are setting this value to a high number.

To say that we will never release a resource until a task says it's okay is inherently dangerous in a distributed system

True! I'm not telling it's the most safe solution but on the consumer side we've similar approach already. Catching committer attention is super hard and thought it would be less risky from committer perspective to add ref counting than introducing a complete new lib (like Commons Pool). Thinking about alternatives I see mainly 2 (pretty sure there are others):

Apache Commons Pool would be a good alternative here as well which is proposed on the consumer side in https://github.com/apache/spark/pull/22138. (my personal preference)
We can switch to similar solution just like on the consumer side, see an extract from the Kafka integration guide:

The size of the cache is limited by <code>spark.kafka.consumer.cache.capacity</code> (default: 64).
If this threshold is reached, it tries to remove the least-used entry that is currently not in use.
If it cannot be removed, then the cache will keep growing. In the worst case, the cache will grow to
the max number of concurrent tasks that can run in the executor (that is, number of tasks slots),
after which it will never reduce.

https://github.com/apache/spark/pull/22138 has been merged which changed my view on how to solve this issue here (until know I was not sure committers have enough confidence to merge that new technology).
Proposal: use Apache Commons Pool
My considerations:

Such way we don't have to do reference counting manually (reflecting to @jose-torres concerns)
Monitoring Kafka consumer/producer cache is on my table for long time. Apache Commons Pool by default provides metrics.
TD suggested this when the consumer side PR was filed/merged. Of course he made his suggestion on the consumer side but his reasoning still applies here (no manual reference counting).

If you agree happy to give helping hand during review. @ScrapCodes if you don't have time to invest then I'm happy to do the coding part. My PR https://github.com/apache/spark/pull/23956 is depending on this for long time and would like to push this forward (not all the cases will delegation token work).
Guys, please share your thoughts.

I second on this, as that's in line what I proposed before. Either I can help reviewing or even work on this.
We still have some consideration regarding performance and number of connections (as we allow multiple-threads access whereas Apache Commons Pool will prevent this), but even in javadoc of Kafka producer it just says The producer is thread safe and sharing a single producer instance across threads will generally be faster than having multiple instances. which doesn't provide any perf. consideration, so I'd rather say let's try this out and measure it.

@gaborgsomogyi Interestingly, I also wanted to make a fresh start on this. This is important for me as well, and thanks for pointing in new direction, I will start working on it.
@HeartSaVioR Thanks for your continued interest and encouragement.


match_time_inds fails

    def match_time_inds(self, inds=None):
         """ Find the nearest matching sim time indices for the current module """
         if inds is None: inds = Ellipsis
         self_tvec = self.t.abstvec
         sim_tvec = self.sim.t.abstvec
         if len(self_tvec) == len(sim_tvec): # Shortcut to avoid doing matching
             return inds
         else:
             out = sc.findnearest(sim_tvec, [inds])
             return out

The function is called with inds=None, so inds will be set to Ellipsis. But sc.findnearest doesn't like Ellipsis, resulting in an exception.

Closed by #749


Could not pass event PlayerQuitEvent / PlayerJoinEvent to DiscordWhitelister

Hello, This is a very nice plugin.  Absolutely love being able to whitelist a player from discord.
However, I do get this error and a stack dump every time a user joins or leaves the game.   Could not pass event PlayerQuitEvent to DiscordWhitelister.   (It says JoinEvent for joins)

Attaching a dump from the log:
19.11 10:11:42 [Server] ERROR Could not pass event PlayerJoinEvent to DiscordWhitelister v1.4.8
19.11 10:11:42 [Server] INFO java.lang.NoClassDefFoundError: de/myzelyam/api/vanish/VanishAPI
19.11 10:11:42 [Server] INFO at uk.co.angrybee.joe.Utils.isVanished(Utils.java:22) ~[discord-whitelister-1.4.9.jar:?]
19.11 10:11:42 [Server] INFO at uk.co.angrybee.joe.events.JoinLeaveEvents.onPlayerJoin(JoinLeaveEvents.java:23) ~[discord-whitelister-1.4.9.jar:?]
19.11 10:11:42 [Server] INFO at com.destroystokyo.paper.event.executor.asm.generated.GeneratedEventExecutor346.execute(Unknown Source) ~[?:?]
19.11 10:11:42 [Server] INFO at org.bukkit.plugin.EventExecutor.lambda$create$1(EventExecutor.java:69) ~[patched_1.17.1.jar:git-Paper-386]
19.11 10:11:42 [Server] INFO at co.aikar.timings.TimedEventExecutor.execute(TimedEventExecutor.java:80) ~[patched_1.17.1.jar:git-Paper-386]
19.11 10:11:42 [Server] INFO at org.bukkit.plugin.RegisteredListener.callEvent(RegisteredListener.java:70) ~[patched_1.17.1.jar:git-Paper-386]
19.11 10:11:42 [Server] INFO at org.bukkit.plugin.SimplePluginManager.callEvent(SimplePluginManager.java:628) ~[patched_1.17.1.jar:git-Paper-386]
19.11 10:11:42 [Server] INFO at net.minecraft.server.players.PlayerList.postChunkLoadJoin(PlayerList.java:360) ~[patched_1.17.1.jar:git-Paper-386]
19.11 10:11:42 [Server] INFO at net.minecraft.server.players.PlayerList.lambda$placeNewPlayer$1(PlayerList.java:302) ~[patched_1.17.1.jar:git-Paper-386]
19.11 10:11:42 [Server] INFO at net.minecraft.server.network.ServerGamePacketListenerImpl.tick(ServerGamePacketListenerImpl.java:307) ~[app:?]
19.11 10:11:42 [Server] INFO at net.minecraft.network.Connection.tick(Connection.java:555) ~[app:?]
19.11 10:11:42 [Server] INFO at net.minecraft.server.network.ServerConnectionListener.tick(ServerConnectionListener.java:201) ~[app:?]
19.11 10:11:42 [Server] INFO at net.minecraft.server.MinecraftServer.tickChildren(MinecraftServer.java:1656) ~[patched_1.17.1.jar:git-Paper-386]
19.11 10:11:42 [Server] INFO at net.minecraft.server.dedicated.DedicatedServer.tickChildren(DedicatedServer.java:490) ~[patched_1.17.1.jar:git-Paper-386]
19.11 10:11:42 [Server] INFO at net.minecraft.server.MinecraftServer.tickServer(MinecraftServer.java:1483) ~[patched_1.17.1.jar:git-Paper-386]
19.11 10:11:42 [Server] INFO at net.minecraft.server.MinecraftServer.runServer(MinecraftServer.java:1282) ~[patched_1.17.1.jar:git-Paper-386]
19.11 10:11:42 [Server] INFO at net.minecraft.server.MinecraftServer.lambda$spin$0(MinecraftServer.java:319) ~[patched_1.17.1.jar:git-Paper-386]
19.11 10:11:42 [Server] INFO at java.lang.Thread.run(Thread.java:831) ~[?:?]
19.11 10:11:42 [Server] INFO Caused by: java.lang.ClassNotFoundException: de.myzelyam.api.vanish.VanishAPI
19.11 10:11:42 [Server] INFO at org.bukkit.plugin.java.PluginClassLoader.loadClass0(PluginClassLoader.java:151) ~[patched_1.17.1.jar:git-Paper-386]
19.11 10:11:42 [Server] INFO at org.bukkit.plugin.java.PluginClassLoader.loadClass(PluginClassLoader.java:103) ~[patched_1.17.1.jar:git-Paper-386]
19.11 10:11:42 [Server] INFO at java.lang.ClassLoader.loadClass(ClassLoader.java:519) ~[?:?]

I may have fixed the issue causing the exception but I am unable to test it myself at the moment. I have uploaded the latest version in releases if you would like to test it out for yourself.

I really appreciate you taking a look at this; however, I am still getting the same error.   I have pulled down the code from here to try to look into it.   Is the latest code posted here on GitHub?

I really appreciate you taking a look at this; however, I am still getting the same error. I have pulled down the code from here to try to look into it. Is the latest code posted here on GitHub?     Can you also tell me what you are using for the development IDE and if it is InteliJ or Something else, are you using Maven or what exactly.   So I can try to duplicate the setup.

It fixed it up for me :) thanks

I would like to correct my previous answer.   This does fix my issue.    I did not see releases here and went back to the releases on the spigot site.     This absolutely does correct my issue.   Thank you.


[core] Added graph workload and classes for recording and recreating …

…benchmark runs.
The graph workload should be used with coming graph databases, but can also be used with traditional relational databases or anything else in YCSB.
Since there was no README.md in the core module I added information about the additional parameters to the CHANGES.md file.
The data (values for the databases, operations and the randomly generated numbers) are stored by the *Recorder classes and can be reproduced with the corresponding *Recretor classes.
I think the recording and recreating part of my implementation could be extended to be used with all workloads if that's a desire, maybe in a future issue/pull request.
Looking forward to your constructive critics and suggestions.
PS: I ditched the filestore binding (#1053), since I decided to "record" the data this way.

I added that plugin to enable junit tests, because they were skipped (as
far as I can tell). But after adding the plugin I was not able to built the
project by mvn clean package, some bindings threw an exception.
Now that it is commented out testng tests should all run, so testing is not
completely disabled.
What do you mean by necessary formatting changes?
On Sat, 17 Mar 2018, 23:01 Kevin Risden<EMAIL_ADDRESS>wrote:

@risdenk requested changes on this pull request.
Why remove testing from pom.xml? Also there are lot of formatting changes
that are necessary for this.
In pom.xml
https://github.com/brianfrankcooper/YCSB/pull/1111#discussion_r175271442
:

@@ -165,6 +166,34 @@ LICENSE file.

 <!-- Used to execute junit tests but crashes on some bindings.

This removed testing?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/brianfrankcooper/YCSB/pull/1111#pullrequestreview-104779569,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALZ4ttNx7s4PXKNWQJEr5OdgeAqVcmNeks5tfYfVgaJpZM4Su92l
.

Whoops sorry I meant unnecessary. There is reordering and changes all over that I don't understand. It makes the change a lot bigger. If there is commented out code it should be removed.

The reformatting is mainly sorting the entries lexicographically, which was mentioned in a comment.
Sorry, I didn't ment to make it harder to review my changes.
The comment is useless of course, I just thought maybe someone has a good solution to run both, junit and testng tests without crashing.

@ChristianNavolskyi are you still working on this?

I did some more changes in my fork which have to be merged into this branch.
I will try to clean up the code and merge my changes in the next days.
After finishing my Bachelor thesis I forgot to do so. 😅

sounds good. Please ping with a comment when you're all set, I don't think github notifies me when new commits are pushed.

Hey @busbey, I just cleaned up my commits and rebased the branch.

Oh man, the notification for your comment must have been filtered by my mail client. I'll add this to my review queue. Since it's a bit large I might try to get the next release branched first.

Okay no problem. I am not in a hurry. 😄
Before you review the code, maybe take a look at #1124 and tell me what you think about that.


build(npm): update npm deploy token

IgniteUI/ignite-ui#1764

Coverage remained the same at 96.257% when pulling 73754ea438afda15178ce79db11ccc160a9949d8 on dpetev/npm-deploy-token into c5657a5234ebc79ec80265173a1169fdd852f6b1 on master.


[website] Add Charts to the pricing table

This is the first pricing for charts, to answer questions such as https://github.com/mui/mui-x/issues/10137.
I only put charts that might be developed this year.
Preview: https://deploy-preview-38680--material-ui.netlify.app/pricing/

From a meeting with @joserodolfofreitas, I'm not sold that we should attempt at "Advanced Gantt" or "Candlestick". These are niches, maybe a rich text editor would be a much better vertical to go into once we cover the most common Chart Pro features. So, the thought was we could either display these as greyed with a request to upvote on GitHub or not display them at all.

I only see one issue with not having "Advanced Gantt" in the pricing table: not informing user that some feature could be restricted to premium. But if we never develop them before year, it should be ok to add premium plan latter for them

@joserodolfofreitas I think we are good with design now


I've tested my code in JSBin for the following error: "After updateRecords(2548, "artist", ""), artist should not be set"  and it worked in JSBIN. The property 'artist' has been deleted and for the hasOwnProperty("artist") it gives false in JSBIN, but it still gives me the error message in the excercise.

Challenge Name

Issue Description

Browser Information

Browser Name, Version:
Operating System:
Mobile, Desktop, or Tablet:

Your Code

Screenshot

@rkornel75 please post your code so we can take a look. Also bear in mind that different editors have varying levels of stringency and linting.

@rkornel75 This is one long title, consider changing it. Let the description be in the body instead.

Closing for now. Please reopen if you are still having an issue with the site.


Code coverage is showing as 0% in SonarQube when using the Jacoco option in the MavenV3 task.

Note
Issues in this repo are for tracking bugs, feature requests and questions for the tasks in this repo
For a list:
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks
If you have an issue or request for the Azure Pipelines service, use developer community instead:
https://developercommunity.visualstudio.com/spaces/21/index.html )
Required Information
Entering this information will route you directly to the right team and expedite traction.
Bug
Type: Bug
Enter Task Name: MavenV3
MavenV3:
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks
Environment

Server - Azure Pipelines or TFS on-premises? Azure Pipelines

If using Azure Pipelines, provide the account name, team project name, build definition name/build number:
Org:  MetLife-US
Project Name:  Dental
Build Definition Name:  DMA-PFR3-Release
Build Number:  1.37441

Agent - Hosted or Private:

Private, RedHat 7 OS, Agent version - 2.165.0

Issue Description
It seems that the MavenV3 task was recently updated from version 3.171.0 to version 3.173.0.  After the update we can see that the code coverage is showing as 0% in SonarQube when using the Jacoco code coverage option in the task within Azure DevOps.  We can see in the commit history for the MavenV3 task that there were some updates for the code coverage report path and we are wondering if that is the source of the issue.  If we switch to the Cobertura option the code coverage appears properly in Sonar.
Task logs
[Enable debug logging and please provide the zip file containing all the logs for a speedy resolution]
Troubleshooting
To Troubleshoot the issue we tried installing the newest SonarQube plugins, specifically SonarJava and Jacoco.
Error logs
There is no error in the logs, but the code coverage appears as 0%, where as previously it was approximately 50%.

@tlbonds2 Do you still run into this issue?

Hello,
Yes we are still seeing the issue and we are wondering if the following commit caused the issue:
https://github.com/microsoft/azure-pipelines-tasks/commit/d176141b06c23aac345ff21c4d2fa74e3ef95956#diff-99ba5d31f4a9541e7aff1ae8d5e37cdf
When the MavenV3 task went from version v3.171.0 to 3.173.0, the Jacoco code coverage stopped being passed to Sonar and Sonar then reports 0% code coverage.  If we switch to the Cobertura option there is no issue.
Thanks,
Tom

@tlbonds2 Could you please provide your logs?

@tlbonds2 As a workaround, could you please try to upload the patch from attachments using tfx-cli. To do that you may use these instructions. This patch reverts changes made in the "Maven and Gradle tasks fixes" commit.
Upload command example:
tfx build tasks upload -u https://<YOUR_ORGANISATION>.visualstudio.com/DefaultCollection -t <YOUR_PERSONAL_ACCESS_TOKEN> --task-path <PATH_TO_TASK>
Please, let us know if it works for you or if you have any questions.

Hi Ekaterina,
Sure I will try it and let you know.  Thanks!
-Tom

@tlbonds2 This issue is not reproducible for me. It works correctly. Could you please provide your logs? Does the patch help to resolve the issue?

@ekaterina-tatanova
Sorry for the wait.  Reverting the changes to the task did, in fact, resolve the issue.  Let me know your thoughts on next steps.
Thanks,
Tom

Hi @tlbonds2 we are going to add ability to control this new behavior by input, and specify it as disabled by default - created PR with related fix

Thanks, @anatolybolshakov!  I will look out for the release.  Much appreciated.

Hi @tlbonds2 the fix for this issue has been rolled out - I'm closing it at the moment, please feel free to reopen it if this is still actual for you


Temp file created but not passed in as parameter

Is there any reason not to pass the temp disk you just created as a parameter to the Shrink function?
https://github.com/mbdavid/LiteDB/blob/23b326ab99bac4b7e9ce447de6fd63c1621a199f/LiteDB/Database/LiteDatabase.cs#L235-L237

Hi @vjacquet, I forgot to pass... but it's already fixed in dev branch... I will release a beta version this week

👍


执行到pip install .这一步时报错了

报错信息如下：
subprocess.CalledProcessError: Command '['cmake', '--build', '.']' returned non-zero exit status 1.
详细报错信息：
Processing /opt/dev/python_rtmpstream/python
Preparing metadata (setup.py) ... done
Building wheels for collected packages: rtmp_streaming
Building wheel for rtmp_streaming (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [181 lines of output]
running bdist_wheel
running build
running build_ext
Building with cfg =  Release
-- The C compiler identification is GNU 10.2.1
-- The CXX compiler identification is GNU 10.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- pybind11 v2.12.0 dev1
-- Found Python: /opt/conda/envs/nerfstream/bin/python3.10 (found suitable version "3.10.13", minimum required is "3.6") found components: Interpreter Development.Module Development.Embed
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Configuring done (2.4s)
-- Generating done (0.0s)
CMake Warning:
Manually-specified variables were not used by the project:
      EXAMPLE_VERSION_INFO

  -- Build files have been written to: /opt/dev/python_rtmpstream/python/build/temp.linux-x86_64-cpython-310
  Change Dir: '/opt/dev/python_rtmpstream/python/build/temp.linux-x86_64-cpython-310'

  Run Build Command(s): /opt/conda/envs/nerfstream/bin/ninja -v
  [1/3] /usr/bin/c++ -Drtmp_streaming_EXPORTS -I/opt/dev/python_rtmpstream/python/../streamer -I/opt/conda/envs/nerfstream/include -isystem /opt/conda/envs/nerfstream/include/python3.10 -isystem /opt/dev/python_rtmpstream/python/pybind11/include -O3 -DNDEBUG -O3 -std=c++14 -Wall -std=gnu++14 -fPIC -fvisibility=hidden -flto -fno-fat-lto-objects -MD -MT CMakeFiles/rtmp_streaming.dir/opt/dev/python_rtmpstream/streamer/streamer.cpp.o -MF CMakeFiles/rtmp_streaming.dir/opt/dev/python_rtmpstream/streamer/streamer.cpp.o.d -o CMakeFiles/rtmp_streaming.dir/opt/dev/python_rtmpstream/streamer/streamer.cpp.o -c /opt/dev/python_rtmpstream/streamer/streamer.cpp
  FAILED: CMakeFiles/rtmp_streaming.dir/opt/dev/python_rtmpstream/streamer/streamer.cpp.o
  /usr/bin/c++ -Drtmp_streaming_EXPORTS -I/opt/dev/python_rtmpstream/python/../streamer -I/opt/conda/envs/nerfstream/include -isystem /opt/conda/envs/nerfstream/include/python3.10 -isystem /opt/dev/python_rtmpstream/python/pybind11/include -O3 -DNDEBUG -O3 -std=c++14 -Wall -std=gnu++14 -fPIC -fvisibility=hidden -flto -fno-fat-lto-objects -MD -MT CMakeFiles/rtmp_streaming.dir/opt/dev/python_rtmpstream/streamer/streamer.cpp.o -MF CMakeFiles/rtmp_streaming.dir/opt/dev/python_rtmpstream/streamer/streamer.cpp.o.d -o CMakeFiles/rtmp_streaming.dir/opt/dev/python_rtmpstream/streamer/streamer.cpp.o -c /opt/dev/python_rtmpstream/streamer/streamer.cpp
  In file included from /opt/dev/python_rtmpstream/streamer/streamer.cpp:1:
  /opt/dev/python_rtmpstream/streamer/streamer.hpp: In member function ‘int streamer::AudioFrame::init(AVSampleFormat, int, uint64_t)’:
  /opt/dev/python_rtmpstream/streamer/streamer.hpp:132:16: error: ‘AVFrame’ {aka ‘struct AVFrame’} has no member named ‘channel_layout’; did you mean ‘ch_layout’?
    132 |         frame->channel_layout = channel_layout;
        |                ^~~~~~~~~~~~~~
        |                ch_layout
  /opt/dev/python_rtmpstream/streamer/streamer.cpp: In function ‘void streamer::log_packet(const AVFormatContext*, const AVPacket*)’:
  /opt/dev/python_rtmpstream/streamer/streamer.cpp:38:18: warning: format ‘%d’ expects argument of type ‘int’, but argument 2 has type ‘long int’ [-Wformat=]
     38 |     printf("pts:%d dts:%d duration:%d stream_index:%d timebase:%d/%d\n",
        |                 ~^
        |                  |
        |                  int
        |                 %ld
     39 |            pkt->pts,
        |            ~~~~~~~~
        |                 |
        |                 long int
  /opt/dev/python_rtmpstream/streamer/streamer.cpp:38:25: warning: format ‘%d’ expects argument of type ‘int’, but argument 3 has type ‘long int’ [-Wformat=]
     38 |     printf("pts:%d dts:%d duration:%d stream_index:%d timebase:%d/%d\n",
        |                        ~^
        |                         |
        |                         int
        |                        %ld
     39 |            pkt->pts,
     40 |            pkt->dts,
        |            ~~~~~~~~
        |                 |
        |                 long int
  /opt/dev/python_rtmpstream/streamer/streamer.cpp:38:37: warning: format ‘%d’ expects argument of type ‘int’, but argument 4 has type ‘long int’ [-Wformat=]
     38 |     printf("pts:%d dts:%d duration:%d stream_index:%d timebase:%d/%d\n",
        |                                    ~^
        |                                     |
        |                                     int
        |                                    %ld
  ......
     41 |            pkt->duration,
        |            ~~~~~~~~~~~~~
        |                 |
        |                 long int
  /opt/dev/python_rtmpstream/streamer/streamer.cpp: In function ‘int streamer::encode_and_write_frame(AVCodecContext*, AVFormatContext*, AVStream*, AVFrame*, int64_t)’:
  /opt/dev/python_rtmpstream/streamer/streamer.cpp:48:24: warning: ‘void av_init_packet(AVPacket*)’ is deprecated [-Wdeprecated-declarations]
     48 |     av_init_packet(&pkt);
        |                        ^
  In file included from /opt/conda/envs/nerfstream/include/libavcodec/avcodec.h:44,
                   from /opt/dev/python_rtmpstream/streamer/streamer.hpp:6,
                   from /opt/dev/python_rtmpstream/streamer/streamer.cpp:1:
  /opt/conda/envs/nerfstream/include/libavcodec/packet.h:653:6: note: declared here
    653 | void av_init_packet(AVPacket *pkt);
        |      ^~~~~~~~~~~~~~
  /opt/dev/python_rtmpstream/streamer/streamer.cpp: In function ‘int streamer::set_options_and_open_encoder_audio(AVFormatContext*, AVStream*, AVCodecContext*, AVCodec*, int, int, AVCodecID)’:
  /opt/dev/python_rtmpstream/streamer/streamer.cpp:138:16: error: ‘AVCodecContext’ {aka ‘struct AVCodecContext’} has no member named ‘channels’
    138 |     codec_ctx->channels = channel;
        |                ^~~~~~~~
  /opt/dev/python_rtmpstream/streamer/streamer.cpp:139:16: error: ‘AVCodecContext’ {aka ‘struct AVCodecContext’} has no member named ‘channel_layout’; did you mean ‘ch_layout’?
    139 |     codec_ctx->channel_layout = av_get_default_channel_layout(channel);
        |                ^~~~~~~~~~~~~~
        |                ch_layout
  /opt/dev/python_rtmpstream/streamer/streamer.cpp:139:33: error: ‘av_get_default_channel_layout’ was not declared in this scope
    139 |     codec_ctx->channel_layout = av_get_default_channel_layout(channel);
        |                                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  /opt/dev/python_rtmpstream/streamer/streamer.cpp: In member function ‘void streamer::Streamer::cleanup()’:
  /opt/dev/python_rtmpstream/streamer/streamer.cpp:192:36: warning: ‘int avcodec_close(AVCodecContext*)’ is deprecated [-Wdeprecated-declarations]
    192 |         avcodec_close(out_codec_ctx);
        |                                    ^
  In file included from /opt/dev/python_rtmpstream/streamer/streamer.hpp:6,
                   from /opt/dev/python_rtmpstream/streamer/streamer.cpp:1:
  /opt/conda/envs/nerfstream/include/libavcodec/avcodec.h:2387:5: note: declared here
   2387 | int avcodec_close(AVCodecContext *avctx);
        |     ^~~~~~~~~~~~~
  /opt/dev/python_rtmpstream/streamer/streamer.cpp:196:42: warning: ‘int avcodec_close(AVCodecContext*)’ is deprecated [-Wdeprecated-declarations]
    196 |         avcodec_close(out_codec_audio_ctx);
        |                                          ^
  In file included from /opt/dev/python_rtmpstream/streamer/streamer.hpp:6,
                   from /opt/dev/python_rtmpstream/streamer/streamer.cpp:1:
  /opt/conda/envs/nerfstream/include/libavcodec/avcodec.h:2387:5: note: declared here
   2387 | int avcodec_close(AVCodecContext *avctx);
        |     ^~~~~~~~~~~~~
  /opt/dev/python_rtmpstream/streamer/streamer.cpp: In member function ‘int streamer::Streamer::init(const streamer::StreamerConfig&)’:
  /opt/dev/python_rtmpstream/streamer/streamer.cpp:379:108: error: ‘AVCodecContext’ {aka ‘struct AVCodecContext’} has no member named ‘channel_layout’; did you mean ‘ch_layout’?
    379 |     audioframe.init(out_codec_audio_ctx->sample_fmt, out_codec_audio_ctx->frame_size, out_codec_audio_ctx->channel_layout);
        |                                                                                ^~~~~~~~~~~~~~
        |                                                                                ch_layout
  /opt/dev/python_rtmpstream/streamer/streamer.cpp:381:60: error: ‘AVCodecContext’ {aka ‘struct AVCodecContext’} has no member named ‘channels’
    381 |                                       out_codec_audio_ctx->channels, 1))) {
        |                                                            ^~~~~~~~
  /opt/dev/python_rtmpstream/streamer/streamer.cpp: At global scope:
  /opt/dev/python_rtmpstream/streamer/streamer.cpp:35:13: warning: ‘void streamer::log_packet(const AVFormatContext*, const AVPacket*)’ defined but not used [-Wunused-function]
     35 | static void log_packet(const AVFormatContext *fmt_ctx, const AVPacket *pkt)
        |             ^~~~~~~~~~
  [2/3] /usr/bin/c++ -Drtmp_streaming_EXPORTS -I/opt/dev/python_rtmpstream/python/../streamer -I/opt/conda/envs/nerfstream/include -isystem /opt/conda/envs/nerfstream/include/python3.10 -isystem /opt/dev/python_rtmpstream/python/pybind11/include -O3 -DNDEBUG -O3 -std=c++14 -Wall -std=gnu++14 -fPIC -fvisibility=hidden -flto -fno-fat-lto-objects -MD -MT CMakeFiles/rtmp_streaming.dir/rtmp_streaming.cpp.o -MF CMakeFiles/rtmp_streaming.dir/rtmp_streaming.cpp.o.d -o CMakeFiles/rtmp_streaming.dir/rtmp_streaming.cpp.o -c /opt/dev/python_rtmpstream/python/rtmp_streaming.cpp
  FAILED: CMakeFiles/rtmp_streaming.dir/rtmp_streaming.cpp.o
  /usr/bin/c++ -Drtmp_streaming_EXPORTS -I/opt/dev/python_rtmpstream/python/../streamer -I/opt/conda/envs/nerfstream/include -isystem /opt/conda/envs/nerfstream/include/python3.10 -isystem /opt/dev/python_rtmpstream/python/pybind11/include -O3 -DNDEBUG -O3 -std=c++14 -Wall -std=gnu++14 -fPIC -fvisibility=hidden -flto -fno-fat-lto-objects -MD -MT CMakeFiles/rtmp_streaming.dir/rtmp_streaming.cpp.o -MF CMakeFiles/rtmp_streaming.dir/rtmp_streaming.cpp.o.d -o CMakeFiles/rtmp_streaming.dir/rtmp_streaming.cpp.o -c /opt/dev/python_rtmpstream/python/rtmp_streaming.cpp
  In file included from /opt/dev/python_rtmpstream/python/rtmp_streaming.cpp:12:
  /opt/dev/python_rtmpstream/python/../streamer/streamer.hpp: In member function ‘int streamer::AudioFrame::init(AVSampleFormat, int, uint64_t)’:
  /opt/dev/python_rtmpstream/python/../streamer/streamer.hpp:132:16: error: ‘AVFrame’ {aka ‘struct AVFrame’} has no member named ‘channel_layout’; did you mean ‘ch_layout’?
    132 |         frame->channel_layout = channel_layout;
        |                ^~~~~~~~~~~~~~
        |                ch_layout
  ninja: build stopped: subcommand failed.

  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "<pip-setuptools-caller>", line 34, in <module>
    File "/opt/dev/python_rtmpstream/python/setup.py", line 125, in <module>
      setup(
    File "/opt/conda/envs/nerfstream/lib/python3.10/site-packages/setuptools/__init__.py", line 104, in setup
      return distutils.core.setup(**attrs)
    File "/opt/conda/envs/nerfstream/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 184, in setup
      return run_commands(dist)
    File "/opt/conda/envs/nerfstream/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
      dist.run_commands()
    File "/opt/conda/envs/nerfstream/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/opt/conda/envs/nerfstream/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
      super().run_command(command)
    File "/opt/conda/envs/nerfstream/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/opt/conda/envs/nerfstream/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 368, in run
      self.run_command("build")
    File "/opt/conda/envs/nerfstream/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
      self.distribution.run_command(command)
    File "/opt/conda/envs/nerfstream/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
      super().run_command(command)
    File "/opt/conda/envs/nerfstream/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/opt/conda/envs/nerfstream/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 132, in run
      self.run_command(cmd_name)
    File "/opt/conda/envs/nerfstream/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
      self.distribution.run_command(command)
    File "/opt/conda/envs/nerfstream/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
      super().run_command(command)
    File "/opt/conda/envs/nerfstream/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/opt/conda/envs/nerfstream/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 91, in run
      _build_ext.run(self)
    File "/opt/conda/envs/nerfstream/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
      self.build_extensions()
    File "/opt/conda/envs/nerfstream/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 479, in build_extensions
      self._build_extensions_serial()
    File "/opt/conda/envs/nerfstream/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 505, in _build_extensions_serial
      self.build_extension(ext)
    File "/opt/dev/python_rtmpstream/python/setup.py", line 116, in build_extension
      subprocess.check_call(
    File "/opt/conda/envs/nerfstream/lib/python3.10/subprocess.py", line 369, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['cmake', '--build', '.']' returned non-zero exit status 1.
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for rtmp_streaming
Running setup.py clean for rtmp_streaming
Failed to build rtmp_streaming
ERROR: Could not build wheels for rtmp_streaming, which is required to install pyproject.toml-based projects

In my case, I encountered this using ffmpeg version>=7. please downgrade your ffmpeg to 6.1.1, use conda install ffmpeg==6.1.1. And problem solved.

In my case, I encountered this using ffmpeg version>=7. please downgrade your ffmpeg to 6.1.1, use conda install ffmpeg==6.1.1. And problem solved.

Thank you very much.I have downgrade the ffmpeg to 6.1.1, but still error.

我也是这个错误，你尝试按照它给出的报错信息，/opt/dev/python_rtmpstream/streamer/streamer.hpp:132:16: error: ‘AVFrame’ {aka ‘struct AVFrame’} has no member named ‘channel_layout’; did you mean ‘ch_layout’?
132 |         frame->channel_layout = channel_layout;
|                ^~~~~~~~~~~~~~
|                ch_layout
将这里的channel_layout改为ch_layout，我目前是这么解决的

我也是这个错误，你尝试按照它给出的报错信息，/opt/dev/python_rtmpstream/streamer/streamer.hpp:132:16: error: ‘AVFrame’ {aka ‘struct AVFrame’} has no member named ‘channel_layout’; did you mean ‘ch_layout’? 132 | frame->channel_layout = channel_layout; | ^~~~~~~~~~~~~~ | ch_layout 将这里的channel_layout改为ch_layout，我目前是这么解决的

您方便贴一下修改的代码不？我尝试frame->ch_layout = channel_layout; 还是报错：
132 |         frame->ch_layout = channel_layout;
|                                          ^~~~~~~~~~~~~~
感谢您！

问题解决了，把ffmpeg的版本降成6.0.1，conda install ffmpeg==6.0.1，再执行就不报错了。


Fix odig odoc --force

See https://github.com/ocaml-doc/odoc/issues/20#issuecomment-267001510

Closed by 2946f950cffc8d98d2


docsrs: fix build

This should fix the docsrs build issues for 6.2.0
The following command works thanks to this fix:
cargo +nightly doc --features docsrs,default

For reference the build logs can be found there https://docs.rs/crate/nom/6.2.0/builds/401812

great, thanks!


Decide what to do with the Git abstraction and deprecated it  from the backend-common package

Context
Ref: https://github.com/backstage/backstage/issues/24493
Before the stable release, we will be deprecating the backend common package, and the Git abstraction will be moved elsewhere. This task is for us to decide what to do with the Git abstraction, and then make the changes according to our decision.
Open questions
Some of it can be moved into scaffolder-node implementations, but more discussion is needed.
Suggested approach

[x] Determine whether more than one plugin uses it
[x] Examine whether it fits with a new backend concept
[x] Extract the files and update the references import statements
[x] Deprecate the export from the backend common package (re-export from the new package with "@deprecated" as a annotation)
[x] Verify everything is working as before and update any documentation

Hello, how can I use the Git abstraction after deprecation?

Hi @javiermartingonzalez 👋🏻 , as we mentioned in the changelog, the abstraction will be removed from the backstage backend packages because it is not used in the framework itself, the implementation was created by the community and was only used by plugins. We recommend you copying the implementation to you plugin in case you are using it or as it’s a fairly thin wrapper on top of the underlying library you may enjoy using that directly instead of the wrapper 🙂 .


prepare changelog for 0.3.3

Description
Fixup changelog

Reverting #121 due to test failures, will re-apply in a new branch.


Fog of War with Hidden Units not hiding built units

Cynic — Today at 01:33
yo, I have a bug to report
https://www.youtube.com/watch?v=uH4kfRgkoOQ&lc=UgxVjneRoZcdVETuJTh4AaABAg
20:05 minute
here is the replay
738876.zip

It seems units aren't being hidden after being built. There should probably be a test for this so it doesn't break again.

This was caused by replays not containing actions for units attacking pipes. The steps to replicate this was:

Have a unit move towards a pipe and attack it.
Have a unit move/be built on the position the previous unit was.
It would set only 1 of 2 units hidden, typically the first one.

This has been fixed in 61901698e33cc765e0f166e9933bcbfdd6413312


Fix issues with private streams containing only guests

From the original report in https://github.com/zulip/zulip/pull/8361#issuecomment-475059223:

Hi! Reading upper messages I'm still not sure this is the right thread. I'm in a bizare situation related to admins' permissions:

I'm an admin and I've created a private stream
I've invited guests user to the stream
I've accidentaly unsubscribed myself from that stream
I can't subriscribe anymore untill I'm invited
All the users are guests and have not the permission to invite me

The right fix for this, given our present model, is probably that we shouldn't allow the last non-guest user to unsubscribe from a private stream.  Longer-term, when we have a "stream owner" feature, this will become "the stream owner cannot unsubscribe without first making someone else the stream owner".

@zulipbot claim

@zulipbot claim

@clarammdantas I have already an open PR on this issue which is left to be reviewed.

@sahil839 aaah I'm sorry, I didn't saw it. I'm abandoning this issue then.

@zulipbot abandon


Created tests for mongodb schema

Removed some unused secondary indexes and created tests for mongodb schema

Current coverage is 96.19% (diff: 100%)

Merging #971 into master will not change coverage

@@             master       #971   diff @@
==========================================
  Files            35         35
  Lines          1842       1842
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits           1772       1772
  Misses           70         70
  Partials          0          0

Powered by Codecov. Last update 2b9a351...050990a


Added Euler 21 and 31 tests

Current coverage is 97.08% (diff: 100%)

Merging #1464 into master will not change coverage

@@             master      #1464   diff @@
==========================================
  Files            86         86
  Lines         10412      10412
  Methods           0          0
  Messages          0          0
  Branches       1823       1823
==========================================
  Hits          10108      10108
  Misses          177        177
  Partials        127        127

Powered by Codecov. Last update 80cdd21...731075d

Great, thank you!


Create RPM scriptlets for the rcm subpackage

https://github.com/osbuild/osbuild-composer/blob/54e27e607ca865d1a7793640ca5f134daea13d45/golang-github-osbuild-composer.spec#L124

[ ] Investigate what the RPM scriptlets do and if it makes sense, create them for the rcm subpackage as well.

This makes sense. I believe the policy is that we should do the systemd scriptlets for all unit files we ship, but we should verify that.

Fixed in #344


Make library work with python 3

When installing for python 3 I get this error:
ImportError: /usr/local/lib/python3.5/dist-packages/libuuid/_uuid.cpython-35m-x86_64-linux-gnu.so: undefin
ed symbol: PyString_FromStringAndSize

This is because PyString_FromStringAndSize doesn't exist for python 3. Switch to using PyBytes_FromStringAndSize.

@dln are you still maintaining this? If not, would you mind handing it off?


#6955 make JupyterLab JS renderer separately installable

beakerx install --lab should also run
jupyter labextension install @jupyter-widgets/jupyterlab-manager
jupyter labextension install beakerx-jupyterlab
jupyter labextension install beakerx-jupyterlab-js

needs to be updated for current jupyter lab:
  "dependencies": {
    "@jupyterlab/rendermime": "^0.16.3",
    "@jupyterlab/rendermime-interfaces": "^1.0.10"
  },

will be obviated by https://github.com/jupyterlab/jupyterlab/pull/4515


TypeError: can't concat str to bytes

Tried to run ultra_ping using Python 3.10.
$ ./echo.py --client <IP_ADDRESS>
Sending 100 256-byte packets at about 400 kB/s to <IP_ADDRESS>:8888... Process Process-1: Traceback (most recent call last):   File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap     self.run()   File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/home/snigel/src/ultra_ping/measurement.py", line 46, in send_packets     payload = bytes(payload + n_fill_bytes * fill_char) TypeError: can't concat str to bytes Note: timed out waiting to receive packets So far, had received 0 packets Received 0/100 packets back from server
If I understand correctly, the above can be solved by changing from
payload = bytes(payload + n_fill_bytes * fill_char)
to
payload = bytes(payload + str.encode(n_fill_bytes * fill_char))
After that has been done, a new error appears.
Sending 100 256-byte packets at about 400 kB/s to <IP_ADDRESS>:8888... Process Process-2: Traceback (most recent call last):   File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap     self.run()   File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/home/snigel/src/ultra_ping/roundtripmeasurement.py", line 103, in recv_packets     payload = str.encode(packet).rstrip("a") TypeError: descriptor 'encode' for 'str' objects doesn't apply to a 'bytes' object Finished sending packets! (Actually sent packets at 326 kB/s)
The problem comes from this line:
payload = packet.rstrip("a")
It's not possible to rstrip on a bytes object and the object can't be decoded to a string since the first bytes won't decode as ascii or UTF-8.

Thanks for reporting! I probably won't have time to fix this any time soon, but I'd gladly accept a pull request.

See my pull request for python3.

Fixed with hydroid7's PR.


building with dart2js is causing strange error with package logging

I've been using the logging package for months but suddenly I'm running in to issues with it when building with dart2js:
  js_helper.dart:1772 Uncaught Error: NoSuchMethodError: method not found: 'call$0' (message.call$0 is not a function)
    at Logger.log$5 (logging.dart:170)
    at Logger.log$4 (main.dart.js:172577)
    at logging.dart:218
    at _wrapJsFunctionForAsync_closure.$protected (async_patch.dart:213)
    at _wrapJsFunctionForAsync_closure.call$2 (async_patch.dart:237)
    at _awaitOnObject_closure.call$1 (async_patch.dart:187)
    at _RootZone.runUnary$2 (zone.dart:1371)
    at _Future__propagateToListeners_handleValueCallback.call$0 (future_impl.dart:129)
    at Object._Future__propagateToListeners (future_impl.dart:665)
    at _Future__addListener_closure.call$0 (future_impl.dart:343)

This is logging package code that the exception is pointing to:
 void log(Level logLevel, message,
      [Object error, StackTrace stackTrace, Zone zone]) {
    Object object;
    if (isLoggable(logLevel)) {
      if (message is Function) message = message();

I added some debug:
  void log(Level logLevel, message,
      [Object error, StackTrace stackTrace, Zone zone]) {
    Object object;
    print ('level: $logLevel message: ${message.toString()} is function: ${(message is Function).toString()}');
    print (message);
    if (isLoggable(logLevel)) {
      if (message is Function) message = message();

It seems that this is failing on the first time it's called. It prints the following to the console before it fails:
13:48:51.772 level: INFO message: endpoint:api.mydomain.com/v3.1/ hash:no is function: true
13:48:51.802 endpoint:api.mydomain.com/v3.1/ hash:no

It's very strange that is thinks this String is a Function - In Dartium there is no issue and it prints false when checking if the message is a function.
This is the code I use to make that first log call (this has worked for months up until last week):
  _log.info('endpoint:${config.api} hash:${config.hashRouting?'yes':'no'}');

@jimmyff Consider trying the latest Dart2JS (with --preview-dart-2) and open if persists.


Add the option to view patch log for individual files

Very usefull to see all the commits that changed a specific file and what changes were made (diffs)
cli: git log -p /path/file

http://stackoverflow.com/questions/2952507/can-github-show-the-history-of-changes-made-to-one-file-in-patch-form


[WIP] Improve headers in EC-Lab output

Addressing Issue #86

Current coverage is 85.66%

Merging #87 into master will increase coverage by +0.07% as of a3e8396

@@            master     #87   diff @@
======================================
  Files           78      80     +2
  Stmts         5137    5150    +13
  Branches         0       0
  Methods        307     311     +4
======================================
+ Hit           4397    4412    +15
  Partial          0       0
+ Missed         740     738     -2

Review entire Coverage Diff as of a3e8396

Powered by Codecov. Updated on successful CI builds.

Note: that we will also need to provide access to the post processor in the new supercap

I deferred the responsibility to gather all the info displayed in the EC-Lab headers to the inspector.
Surprisingly, the input parameters used to build the supercap seem to be the more delicate to get.

Now all the hooks are in place. We need to compute correctly the mass of the electrode (computing total mass right now) and decide whether we print info for each electrode or not.


vals version does not work

Hi,
I'm using vals 0.18.0 from github releases. Running vals version does not report a version.
% vals version
Version: dev
Git Commit:

Same here


Extend buffer size for name of loaded PPDs up to 256 characters

We've recently received the report of an user of Endless mentioning that he could not install his Epson Stylus Photo T60 printer and, after some investigation, I found the 128 characters limit was the reason for the failure, so I wonder if that could be raised and such a patch accepted upstream, which would be ideal.
For extra context, we have a system in place that automatically downloads and installs drivers from Openprinting.org, making them discoverable to CUPS via paths under /usr/share/ppd/eos-config-printerwhich, in the case of the printer reported by our user would look something really long:
/usr/share/ppd/eos-config-printer/epson-inkjet-printer-stylus-photo-t50-series_ppds_Epson/Epson-Stylus_Photo_T60_Series-epson-driver.ppd.gz.
This, when translated into a virtual path, would look something like this:
lsb/usr/eos-config-printer/epson-inkjet-printer-stylus-photo-t50-series_ppds_Epson/Epson-Stylus_Photo_T60_Series-epson-driver.ppd.gz
...which would be 132 characters long and therefore would get truncated, making CUPS fail with a Unable to open "/usr/[...]T60_Series-epson-driver.p" - No such file or directory error.
I understand that the fact we have such a looong pathname to point to there the PPD files will end up living is unfortunate and probably suggests that a patch like that should live downstream only, but before assuming so or looking into other options I thought I would ask, as perhaps it's not a terrible idea either to raise this buffer size a bit (e.g. to 256 characters, to keep powers of 2 sizes) upstream?

Just send a Pull Request for this: https://github.com/mariospr/cups/commit/759c9b49003bb2a07f2b4db38a1ae4ac58d2617b
Please let me know what you think, and feel free to merge it right away if it's good enough. Thanks!

The maximum allowed size of a "name" value in IPP is 255 octets (bytes), so that's as far as we can go... I'll need to review things to make sure it doesn't cause any regressions.

Thanks @michaelrsweet, for looking into this so quickly. 255 bytes would be certainly more than enough even for our particular case so that should be fine. Please let me know what you think once you review it, and thanks again


when running example code to receive message via listen funcion, dartzmq throws exception

I don't know if the reason to this issue happens is the project is now still under development stage. but through observing the source code, I found that maybe one of the reason is the dartzmq doesn't provide the receive function, and the inner strategy is using loop to handle the receive data without sending data for reply. but whether I use reply mode or request mode, zmq doesn't allow no sending action within two receiving actions. so, it seemed the exception occurs. i want to know why the dartzmq has not provide receive funciton directly? or the dartzmq developer has had a solution to solove the problem? I am looking forward to the reply very eagerly. Thanks in advance.

@bigchen2008 Can you provide the error that you are getting? The stacktrace would also be helpful.


Upgrade from upstream and password management

Opendistro

Upgrade to OpenDistro 1.6.0
Add pod affinity
Allow subpath for Persistent Volumes
Allow use of unmanaged Persistent Volumes
Configurable init images

FluentD

Upgrade fluentD version
Get auth from secret

ElasticSearch Exporter

Get auth info from secret
prometheusRule.rules are now processed as Helm template, allowing to set variables in them. This means that if a rule contains a {{ $value }}, Helm will try replacing it and probably fail.

ElasticSearch Curator

Get auth info from secret

Still no answer from customer :(


Test if a field is enabled

Hi,
I'm currently playing with the API and it's very nice, but I haven't found how to test if a field is enabled.
May be I missed something, but if not, it would be nice to have this possibility :-)

This is something we don't have a specific API for currently but can access the enabled attribute on the element.    You can use the xrmBrowser.Document.getElementById("firstname").GetAttribute("disabled");  try get the details.

Sorry but could you provide some more help with this. When I tried grabbing a field with
var FName = xrmBrowser.Document.getElementById("firstname")
I can see the full element in the watcher. However if I try
var FNameDisplay = xrmBrowser.Document.getElementById("firstname").GetAttribute("disabled");
or
var FNameDisplay = xrmBrowser.Document.getElementById("firstname").GetAttribute("Displayed");
it comes up as null in the watcher. What am I doing wrong?

Ok, it looks like there will be a span that has a lock image. The disabled setting is never set.   Try something like this.
if (xrmBrowser.Document.getElementById("firstname_lock") != null)
{
//do something
}

You can also execute client script that will get the control disabled status.
xrmBrowser.Driver.ExecuteScript("Xrm.Page.ui.controls.get('firstname').getDisabled();").ToString();
That should return True or False.

So I was still getting an error with
xrmBrowser.Driver.ExecuteScript("Xrm.Page.ui.controls.get('firstname').getDisabled();").ToString();
I was getting a NullReferenceException
We ended up realising it was already present in the IWebElement properties.....
var FNameDisplay = xrmBrowser.Document.getElementById("firstname").Displayed

this actually works
xrmBrowser.Driver.ExecuteScript($"return Xrm.Page.getControl('firstname').getDisabled();");
so the previous
xrmBrowser.Driver.ExecuteScript("Xrm.Page.ui.controls.get('firstname').getDisabled();").ToString();
was missing the "return"


[Proposal] Include MailHog in Homestead

I've recently switched away from MailCatcher (due to multiple issues with Ruby + Gems) - and found MailHog - https://github.com/mailhog/MailHog
Mailhog is an alternative for Mailcatcher that doesnt require any special dependencies like Ruby, so installation dead simple. Here is a quick article I found on it: http://iankent.uk/2014/04/20/introducing-go-mailhog/
Here is an example of bash installation script that would cover everything needed to include in Homestead: https://gist.github.com/varghesejacob/68caf7aeee53305a1ffa#file-mailhog-bash-script
MailHog includes hooks (like Mailcatcher) to use in PHPUnit Tests so you can 'test' your emails are been sent etc etc: https://gist.github.com/varghesejacob/6113ac971a00611015c5#file-mailhog-test-helper.
Obviously people could install this themselves - but it would seem to be a natural synergy with Homestead - providing everything 'out of the box'. Taylor has previously tweeted that perhaps MailCatcher should be in Homestead by default: https://twitter.com/taylorotwell/status/536923096213516288 - and it was highly "+1" from many replied tweets (including @JeffreyWay) - so this would potentially be a better 'drop in' replacement.
Obviously there is also MailTraip.io - but this is a local service - so is much faster/reliable etc etc.

Ping @taylorotwell.

Ping @taylorotwell.

ping @taylorotwell
Perhaps this could be considered for the next Homestead release (which I am assuming there will be a new one when PHP7 is released)?

ping @taylorotwell +1

Sorry, but we don't process proposal issues. Feel free to discuss on slack or whatever though.


Support for zeit now v1

Test Photon workflows on the lines of https://github.com/prisma/prisma-examples/tree/prisma2/deployment-platforms with now v1.


Fix unsafe cast in SqlException for SerializationEntry.Value

Fixes #524
It looks like the driver is performing unsafe cast of value to Guid without considering any other type:
https://github.com/dotnet/SqlClient/blob/8884f6308e119915c15e50619b6c0e45eaa5dc7f/src/Microsoft.Data.SqlClient/netcore/src/Microsoft/Data/SqlClient/SqlException.cs#L41
In normal scenarios this probably works fine, but if one passes a JSON string, the deserializer fails (see reported issue for repro code). Since the type of Value in SerializationEntry is object, casting to Guid is unsafe.
This PR fixes the issue by generating new Guid for SqlException.ClientConnectionId property from object type.
@saurabh500
I also noticed differences in SqlException serialization implementation in .NET Core v/s .NET Framework, e.g. null value serialization in .NET Core v/s implementation in .NET Framework
Would like to know more about it and if it should also be updated for .NET Framework.
Edit: Change in NetFx class was needed for serialization to work.

@cheenamalhotra, does it make sense I create a PR for the test in my previous comment or not (considering the small change)?

Hi @amartini-n
Thanks for validating! I'm going to add tests here so that won't be necessary.

Hi,
This will not work with custom IFormatterConverter for SerializationInfo.
See my comment : https://github.com/dotnet/SqlClient/issues/524#issuecomment-617396325


Update java function dependency

Fix: #170
Three changes here:

com.microsoft.azure.serverless.functions to com.microsoft.azure.functions
azure-functions-java-core to azure-functions-java-library
Remove APIHub bindings since it's not supported in V2 runtime

Codecov Report

Merging #171 into develop will decrease coverage by 0.35%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           develop    #171      +/-   ##
==========================================
- Coverage    60.56%   60.2%   -0.36%
==========================================
  Files           50      49       -1
  Lines         1567    1548      -19
  Branches       171     171
==========================================
- Hits           949     932      -17
+ Misses         558     556       -2
  Partials        60      60

Impacted Files
Coverage Δ

.../azure/maven/function/bindings/BindingFactory.java
91.66% <ø> (-0.65%)
:arrow_down:

...oft/azure/maven/function/bindings/BlobBinding.java
92.85% <ø> (ø)
:arrow_up:

...zure/maven/function/bindings/EventGridBinding.java
100% <ø> (ø)
:arrow_up:

...ft/azure/maven/function/bindings/QueueBinding.java
90% <ø> (ø)
:arrow_up:

...aven/function/bindings/NotificationHubBinding.java
71.42% <ø> (ø)
:arrow_up:

...ft/azure/maven/function/bindings/TimerBinding.java
75% <ø> (ø)
:arrow_up:

...ft/azure/maven/function/bindings/TableBinding.java
79.16% <ø> (ø)
:arrow_up:

...maven/function/handlers/AnnotationHandlerImpl.java
91.04% <ø> (ø)
:arrow_up:

...azure/maven/function/bindings/CosmosDBBinding.java
58.33% <ø> (ø)
:arrow_up:

...re/maven/function/bindings/MobileTableBinding.java
78.94% <ø> (ø)
:arrow_up:

... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ea369c9...b94aa45. Read the comment docs.

@pragnagopa The PR for templates repo has been sent out: https://github.com/Azure/azure-functions-templates/pull/695


Extract preview-middleware client code into separate module

This PR extracts the sandboxInit.js from preview-middleware/templates into a separate private module allowing to test this file and to write it in typescript.
Details

new module preview-middleware-client
converting preview-middleware/templates/sandboxInit.js to preview-middleware-client/src/flp/init.ts
adding jest tests
moving the types/sapui5 also into this new module since they will only be required here (and having them centrally caused lots of instability with the ts language server)

Getting a warning about peer dependencies when doing pnpm install in this branch.

Could you please take care of the code smells?

I cannot fix the chaining 'smells' because that is not supported in UI5 :(. I marked them as won't fix.
The cognitive complexity one is bothering me but I would prefer not fixing it in this PR because right now it is just a transformation from js to ts, I didn't change the functionality. To fix this one, I would need to restructure the functionality a bit. @zdravko-georgiev, what do you think? Leave it or fix it? If you prefer, I will fix it.

Could you please take care of the code smells?

I cannot fix the chaining 'smells' because that is not supported in UI5 :(. I marked them as won't fix. The cognitive complexity one is bothering me but I would prefer not fixing it in this PR because right now it is just a transformation from js to ts, I didn't change the functionality. To fix this one, I would need to restructure the functionality a bit. @zdravko-georgiev, what do you think? Leave it or fix it? If you prefer, I will fix it.

@tobiasqueck OK, we can fix it later.

Getting a warning about peer dependencies when doing pnpm install in this branch.

That is not caused by my changes, you also see it if you checkout main and modify any dependency e.g. typescript from 4.9.4 to 4.9.5 and then run pnpm i.
@devinea any comment here?

Getting a warning about peer dependencies when doing pnpm install in this branch.

That is not caused by my changes, you also see it if you checkout main and modify any dependency e.g. typescript from 4.9.4 to 4.9.5 and then run pnpm i.
@devinea any comment here?

Those dependencies don't have newer versions so the peer  dependencies are older than we use. But that seems to work fine for now.

Not so obvious why sandboxInit.js was renamed to init.ts ? I preferred the sandboxInit.xx FWIW.

The sandbox name made sense when it was a standalone file next to the html file, now it is moved into a namespace clearly identifying it as a preview coding. Additionally, that namespace will get a lot of additional files for CPE and ADP, so that it would start looking odd if all of them get a sandbox prefix.
PS: if you insist I can change the namespace from open.ux.preview.client to open.ux.preview.sandbox or so but to be honest I think that would be less descriptive.


debug_assert is disabled in the standard library due to poor assertion messages

The custom sysroot build for the MIR Linker introduced by #1717 builds the standard library in debug mode. This enables a bunch of safety checks and debug assertions from the standard library. Some of these checks are for intrinsics safety checks. The issues here are:

A lot of these checks make our intrinsics checks redundant.
The std checks are added before actually calling the intrinsic and they are not super user friendly.

For example, the following testcase from the expected suite: https://github.com/model-checking/kani/blob/78761bdac057a910a8f23f7622c0a3a9df6953fb/tests/expected/intrinsics/copy-nonoverlapping/copy-overflow/main.rs#L6-L18
Without the MIR Linker, it fails with the following error:
Failed Checks: copy_nonoverlapping: attempt to compute number in bytes which would overflow

With the MIR Linker, the error is:
Failed Checks: called `Option::unwrap()` on a `None` value

This happens because of the std check is_nonoverlapping which has a call to checked_mul().unwrap(). The checked_mul() is None in the overflow scenario.

Another interesting one is that the checks are not encoded as normal debug_assert!() which panic!(), they call abort instead to optimize the compilation size. So other failures will look like:
Failed Checks: reached intrinsic::abort

The current behavior implemented for MIR Linker keeps the same behavior as before the MIR Linker. Enabling these checks are desirable but they are orthogonal to the MIR Linker work. Hence, I'm removing the link with the MIR Linker milestone.

Hi, I've just run into this issue trying to verify a program that depends on unicode-bidi, which has this snippet:
#[cfg(feature = "std")]
std::debug_assert!(
    false,
    "Found broken indices in level run: found indices {}..{} for string of length {}",
    level_run.start,
    level_run.end,
    text.len()
);

This fails with an error:
error: cannot find macro `__kani__workaround_core_assert` in this scope

I'm very new to kani, so I'm not exactly sure if this is actually the same issue, or just user error. I just stumbled upon this issue when googling.
Are there steps a user can take to work around this error? (Apologies if this is the wrong place to ask these sorts of questions :sweat_smile: )

Thanks for the bug report @cameron1024. Can you please file a separate issue and include information on how to reproduce the error, and the Kani version? Thanks!

Opened https://github.com/model-checking/kani/issues/2187


Pusher :  : [{"type":"WebSocketError","error":{"type":"PusherError","data":{"code":4001,"message":"Could not find app key xxxxxxxx."}}}]

Bootstrap.js:
import Echo from 'laravel-echo'

window.Pusher = require('pusher-js');
window.Pusher.logToConsole = true;

window.Echo = new Echo({
    broadcaster: 'pusher',
    key: process.env.MIX_PUSHER_APP_KEY,
    cluster: process.env.MIX_PUSHER_APP_CLUSTER, //eu
    wsHost: window.location.hostname,
    wsPort: 6001,
    wssPort: 6001,
    disableStats: true,
    encrypted: true,
    enabledTransports: ['ws', 'wss'],
});

Broadcasting:
'connections' => [

        'pusher' => [
            'driver' => 'pusher',
            'key' => env('PUSHER_APP_KEY'),
            'secret' => env('PUSHER_APP_SECRET'),
            'app_id' => env('PUSHER_APP_ID'),
            'options' => [
                'cluster' => env('PUSHER_APP_CLUSTER'),
                'host' => env('LARAVEL_WEBSOCKETS_HOST', '<IP_ADDRESS>'),
                'port' => 6001,
                'scheme' => 'https',
                'encrypted' => true,
                'curl_options' => [
                    CURLOPT_SSL_VERIFYHOST => 0,
                    CURLOPT_SSL_VERIFYPEER => 0,
                ]
            ],
        ],

Websockets:
'apps' => [
        [
            'id' => env('PUSHER_APP_ID'),
            'name' => env('APP_NAME'),
            'key' => env('PUSHER_APP_KEY'),
            'secret' => env('PUSHER_APP_SECRET'),
            'path' => env('PUSHER_APP_PATH'),
            'capacity' => null,
            'enable_client_messages' => false,
            'enable_statistics' => false,
        ],
    ],
'ssl' => [
        'local_cert' => env('LARAVEL_WEBSOCKETS_SSL_LOCAL_CERT', null),

        'local_pk' => env('LARAVEL_WEBSOCKETS_SSL_LOCAL_PK', null),

        'passphrase' => env('LARAVEL_WEBSOCKETS_SSL_PASSPHRASE', null), //null

        'verify_peer' => false
    ],

artisan config:cache and npm run dev did not solve the problem
What causes this error? After all, the key is not needed for laravel-websockets to work

The server cached the module js and issued it constantly, not the newly created

@Sans84 can you share how you solved this issue? It happens to me, restart valet, clear all cache but the error on app id still uses the old one.

I have the same error. The connection is perfectly functional, but once on a while it retrieves that error. I have to change the clouster and compile again. I am not using Valet, but a reverse proxy on nginx instead


drivers: rtc: sam0: Add RTC calendar driver and clock fixes

This introduces the RTC clock calendar mode on a new RTC driver. At moment the MODE2 is fully implemented and tested with SAMD20, SAMR21 and SAME54 variants to cover all possible situations.
This driver imposes challenges in clock selection. The SAM family requires, in some situations, generic clock matrix configuration to use peripherals and or different oscillators. These generic clocks should be defined by product depending of developer application and it applies to the 1 KHz clock source. Currently in the SoC common initialization are reserved a few generic channels for specific use. The remaining channels after this change can be defined by developer. In the future when a proper clock driver/manager is in place this could be 100% defined at devicetree. To allow such configuration this introduces the Linux assigned-clock property.

Rebase
Tried to fix all building issues

Hi @henrikbrixandersen ,

Given the size of this PR, it might make sense to split into one PR reworking the clocking, and follow-up with a PR for adding the RTC.

Yes, I already considered this and will do it.

By the way, did you see #72102 ?

I know Daniel is working on the management but I'm not following at moment.
The main issue with sam0 it that it needs that developer should define clock generators and then connect the peripherals to some of that lines. The assigned-clock property makes live easy to the developer assign the lines in the board file. What I made was basic clock configuration which can be improved later. Current these devices don't have clock drivers, only some helpers to enable/disable clock sources. In this case, I consider the step a preparation to clock driver/management and can be reworked. At least, this gives freedom to developers to define their own clock lines and associate until a better solution will be in place.

Can someone have an idea why I can not reproduce the issue with the ./scripts/ci/check_compliance.py locally ?

This is what CI runs:
  ./scripts/ci/check_compliance.py --annotate -e KconfigBasic -e ClangFormat \
  -c origin/${BASE_REF}..

probably one or more of those flags are needed :)

Can someone have an idea why I can not reproduce the issue with the ./scripts/ci/check_compliance.py locally ?

This is what CI runs:
  ./scripts/ci/check_compliance.py --annotate -e KconfigBasic -e ClangFormat \
  -c origin/${BASE_REF}..

probably one or more of those flags are needed :)

Ok, I could reproduce CI. I recommend an update on docs. The below line shows the details.
./scripts/ci/check_compliance.py --annotate -c HEAD..main
In my case, I want to exclude ClangFormat
./scripts/ci/check_compliance.py --annotate -e ClangFormat  -c HEAD..main
Problem is, when you run without excluding ClangFormat it return an error and, as far I know, ClangFormat still optional.

Fixed compliance checks.
Split PR in two. Review should be made on the last 4 commits only. Kept first 2 commits until #82183 got merged.


community[integrations]: add persistable implementation of bm25 retriever

Description: This PR adds an alternative (additional, i.e. not a replacement of the existing one) implementation of the BM25 based on bm25s retriever which addresses the following two problems:

rank_bm25 - the package the currently implementation is based on - is undermantained, as noted in #18086
The current version of that retriever is not persistable, which makes it practically unusable as noted in #26112

Issue:  #18086 #26112
Dependencies: bm25s

@ccurme @efriis @baskaryan

@ccurme @baskaryan @efriis

Hi, can you have a look at this PR @efriis @ccurme @baskaryan?

@ccurme sorry for pinging you again. Just wondering if this PR was of interest, otherwise please feel free to close to :)

is this implementable as an improvement to the existing instead of a replacement?

@efriis yes. As I said, this version is faster (see  the paper for benchmarks),  maintained and persistable (most important feature to notice imho).
The existing version based on rank_bm25 was unusable for me: I had to recreate the index everytime I had to run an experiment.

@efriis I'm not sure. As mentioned above, the original is based on rank_bm25 which is undermaintained. To supporto this feature one needs to first implement it in rank_bm25 or to have this functionality in the langchain integration.
On the other hand, bm25s natively supports persistance. It also comes with additional improvements in execution time (see benchmarks in their paper or project readme).

@efriis I just wanted to follow up on the above.

got it. Rationale for having a separate one makes sense to me. In general we're moving towards dedicated externally-maintained integration packages instead of community ones for things like this, so would recommend launching a langchain-bm25s type integration package. I'll close this PR, and would recommend reopening with just docs updates, as well as registering your package in libs/packages.yml! We'll be able to review simple PRs that only modify these two things much faster :)
Here's the guide, and if you have questions, feel free to leave them in the comments on those pages so others can see them! https://python.langchain.com/docs/contributing/how_to/integrations/


Prompt not updating

Issue Description:
When utilizing the zi (zoxide interactive) command as a keybinding, the prompt does not update or repaint to reflect the changes in the current directory.
Steps to Reproduce:

Install zoxide and set it up with fish.
Bind zi to a key similar to: bind \e\cz "__zoxide_zi; commandline -f repaint"
Navigate to a directory using the keybinding.
Observe that the prompt does not update to show the new directory.

Expected Behavior:
The prompt should automatically update to reflect the current working directory after using the keybinding.

Did you find a fix? I have a similar problem with #77


cannot download data from public session if not your session

If you are not logged in, you cannot download data from a session, even if public. Not sure what we want there, do we want to allow or not? We should either remove the button (if we do not allow) or make possible (if we keep the button). Thoughts @suhlrich ?

I think we do want to be able to download data from public sessions if not logged in.
On Jun 3, 2023, at 5:25 AM, Antoine Falisse @.***> wrote:

If you are not logged in, you cannot download data from a session, even if public. Not sure what we want there, do we want to allow or not? Thoughts @suhlrichhttps://github.com/suhlrich ?
—
Reply to this email directly, view it on GitHubhttps://github.com/stanfordnmbl/opencap-api/issues/79, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKOYGJ5RKMTEKNYLOPLYHSDXJMUJTANCNFSM6AAAAAAYZIJ2RI.
You are receiving this because you were mentioned.Message ID: @.***>

@anastasiiadidan could we make possible downloading public sessions by anyone? Currently, you need to be logged in. Here is an example: https://dev.app.opencap.ai/session/00e58b94-c70c-4835-81a0-e45abc3efcfc

@antoinefalisse sure, will work on it


Finetune Ovis 1.6 with LoRA

Thank for your great work! @runninglsy
How can I finetune Ovis 1.6 with LoRA?

Ovis1.6-Gemma2-9B is now supported by ms-swift.

@runninglsy  Do we have any documentation? to finetune it and any expected documents required like documents split and what is the dataset format is expected and what machine is required to finetune the model.