[SPARK-14649][CORE] DagScheduler should not run duplicate tasks on fe…

What changes were proposed in this pull request?
When a fetch failure occurs, the DAGScheduler re-launches the previous stage (to re-generate output that was missing), and then re-launches all tasks in the stage that haven't completed by the time the stage gets resubmitted (the DAGScheduler re-lanches all of the tasks whose output data is not available -- which is equivalent to the set of tasks that hadn't yet completed). This some times leads to wasteful duplicate task run for the jobs with long running task.
To address the issue following changes have been made.
Dag scheduler maintains a pending task list, which is a list of tasks that  have been submitted to the lower-level scheduler and they should not be resubmitted when rerun of the stage.

When a fetch failure happens, the task set manager informs the dag scheduler to mark all the non-running tasks to the pending task list. However, the running tasks in the task set are not killed.
In case of resubmission of the stage, the dag scheduler only resubmits the tasks which are in pending stage.

How was this patch tested?
Added new tests.

Test build #74558 has started for PR 17297 at commit e5429d3.

Test build #74558 has finished for PR 17297 at commit e5429d3.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class TasksAborted(stageId: Int, tasks: Seq[Task[_]]) extends DAGSchedulerEvent

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74558/
Test FAILed.

cc - @kayousterhout - Addressed your earlier comment about https://github.com/apache/spark/pull/12436 ignoring fetch failure from stale map output. I have addressed this issue by adding epoch for each map output registered, that way if the task's epoch is smaller than the epoch of the map output, we can ignore the fetch failure. This also takes care of  epoch changes which will be triggered due to executor loss for a shuffle task when its shuffle map task executor is gone as pointed out by @mridulm.
Let me know what you think of the approach.

Test build #74560 has started for PR 17297 at commit 279b09a.

Test build #74560 has finished for PR 17297 at commit 279b09a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class TasksAborted(stageId: Int, tasks: Seq[Task[_]]) extends DAGSchedulerEvent

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74560/
Test FAILed.

Test build #74562 has started for PR 17297 at commit f127150.

Test build #74566 has started for PR 17297 at commit 0bcc69a.

Test build #74562 has finished for PR 17297 at commit f127150.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class TasksAborted(stageId: Int, tasks: Seq[Task[_]]) extends DAGSchedulerEvent

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74562/
Test FAILed.

Merged build finished. Test FAILed.

Test build #74566 has finished for PR 17297 at commit 0bcc69a.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class TasksAborted(stageId: Int, tasks: Seq[Task[_]]) extends DAGSchedulerEvent

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74566/
Test FAILed.

Merged build finished. Test FAILed.

@sitalkedia I won't have time to review this in detail for at least a few weeks, just so you know (although others may have time to review / merge it).
At a very high level, I'm concerned  about the amount of complexity that this adds to the scheduler code.  We've recently had to deal with a number of subtle bugs with jobs hanging or Spark crashing as a result of trying to handle map output from old tasks.  As a result, I'm hesitant to add more complexity -- and the associated risk of bugs that cause job failures + expense of maintaining the code -- to improve performance.
At the point I'd lean towards cancelling outstanding map tasks when a fetch failure occurs (there's currently a TODO in the code to do this) to simplify these issues.  This would improve performance in some ways, by freeing up slots that could be used for something else, at the expense of wasted work if the tasks have already made significant progress.  But it would significantly simplify the scheduler code, which given the debugging + reviewer time that has gone into fixing subtle issues with this code path, I think is worthwhile.
Curious what other folks think here.

@kayousterhout - I understand your concern and I agree that canceling the running tasks is definitely a simpler approach, but this is very inefficient for large jobs where tasks can run for hours.  In our environment where fetch failures are common, this change will not only improve the performance of the jobs in case of fetch failure, this also helps reliability. If we cancel all running reducers, we might end of in a state where jobs will not make any progress at all in case of frequent fetch failure, because they will just flip-flop between two stage.
Comparing this approach to how Hadoop handles fetch failure, it does not fail any reducer in case it detects any map output missing. The reducers just continue processing output from other mappers while the missing output is being recomputed concurrently. This approach give Hadoop a big edge over Spark for long running jobs with multiple fetch failure. This change is one step towards making Spark robust against fetch failure, we would eventually want to have the hadoop model, where we would not fail any task in case of map output missing.
Regarding the approach,  please let me know if you can think of some way to reduce the complexity of this change.
cc -@markhamstra, @rxin, @sameeragarwal

Test build #74631 has started for PR 17297 at commit 901c9bf.

Test build #74631 has finished for PR 17297 at commit 901c9bf.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74631/
Test FAILed.

I'm a bit confused by the description:

When a fetch failure happens, the task set manager ask the dag scheduler to abort all the non-running tasks. However, the running tasks in the task set are not killed.

this is already true.  when there is a fetch failure, the TaskSetManager is marked as zombie, and the DAGScheduler resubmits stages, but nothing actively kills running tasks.

re-launches all tasks in the stage with the fetch failure that hadn't completed when the fetch failure occurred (the DAGScheduler re-lanches all of the tasks whose output data is not available -- which is equivalent to the set of tasks that hadn't yet completed).

I don't think its true that it relaunches all tasks that hadn't completed when the fetch failure occurred.  it relaunches all the tasks haven't completed, by the time the stage gets resubmitted.  More tasks can complete in between the time of the first failure, and the time the stage is resubmitted.
But there are several other potential issues you may be trying to address.
Say there is stage 0 and stage 1, each one has 10 tasks.  Stage 0 completes fine on the first attempt, then stage 1 starts.  Tasks 0 & 1 in stage 1 complete, but then there is a fetch failure in task 2.  Lets also say we have an abundance of cluster resources so tasks 3 - 9 from stage 1, attempt 0 are still running.
Stage 0 get resubmitted as attempt 1, just to regenerate the map output for whatever executor had the data for the fetch failure -- perhaps its just one task from stage 0 that needs to resubmitted.  Now, lots of different scenarios are possible:
(a) Tasks 3 - 9 from stage 1 attempt 0 all finish successfully while stage 0 attempt 1 is running.  So when stage 0 attempt 1 finishes, then stage 1 attempt 1 is submitted, just with Task 2.  If it completely succesfully, we're done (no wasted work).
(b) stage 0 attempt 1 finishes, before tasks 3 - 9 from stage 1 attempt 0 have finished.  So stage 1 gets submitted again as stage 1 attempt 1, with tasks 2 - 9.  So there are now two copies running for tasks 3 - 9. Maybe all the tasks from attempt 0 actually finish shortly after attempt 1 starts.  In this case, the stage is complete as soon as there is one complete attempt for each task.  But even after the stage completes successfully, all the other tasks keep running anyway.  (plenty of wasted work)
(c) like (b), but shortly after stage 1 attempt 1 is submitted, we get another fetch failure in one of the old "zombie" tasks from stage 1 attempt 0.  But the DAGScheduler realizes it already has a more recent attempt for this stage, so it ignores the fetch failure.  All the other tasks keep running as usual.  If there aren't any other issues, the stage completes when there is one completed attempt for each task.  (same amount of wasted work as (b)).
(d) While stage 0 attempt 1 is running, we get another fetch failure from stage 1 attempt 0, say in Task 3, which has a failure from a different executor.  Maybe its from a completely different host (just by chance, or there may be cluster maintenance where multiple hosts are serviced at once); or maybe its from another executor on the same host (at least, until we do something about your other pr on unregistering all shuffle files on a host).  To be honest, I don't understand how things work in this scenario.  We mark stage 0 as failed, we unregister some shuffle output, and we resubmit stage 0.  But stage 0 attempt 1 is still running, so I would have expected us to end up with conflicting task sets.  Whatever the real behavior is here, it seems we're at risk of having even more duplicated work for yet another attempt for stage 1.
etc.
So I think in (b) and (c), you are trying to avoid resubmitting tasks 3-9 on stage 1 attempt 1.  the thing is, there is a strong reason to believe that the original version of those tasks will fail.  Most likely, those tasks needs map output from the same executor that caused the first fetch failure.  So Kay is suggesting that we take the opposite approach, and instead actively kill the tasks from stage 1 attempt 0.  OTOH, its possible that  (i) the issue may have been transient or (ii) the tasks already finished fetching that data before the error occurred.  We really have no idea.

Thanks a lot @squito for taking a look at it and for your feedback.

this is already true. when there is a fetch failure, the TaskSetManager is marked as zombie, and the DAGScheduler resubmits stages, but nothing actively kills running tasks.

That is true but currently the DAG scheduler has no idea about which tasks are running and which are being aborted. With this change, the task set manager informs the dag scheduler about currently running/aborted tasks so that the DAG scheduler can avoid resubmitting duplicates.

I don't think its true that it relaunches all tasks that hadn't completed when the fetch failure occurred. it relaunches all the tasks haven't completed, by the time the stage gets resubmitted. More tasks can complete in between the time of the first failure, and the time the stage is resubmitted.

Yes that's true. I will update the PR description.

So I think in (b) and (c), you are trying to avoid resubmitting tasks 3-9 on stage 1 attempt 1. the thing is, there is a strong reason to believe that the original version of those tasks will fail. Most likely, those tasks needs map output from the same executor that caused the first fetch failure. So Kay is suggesting that we take the opposite approach, and instead actively kill the tasks from stage 1 attempt 0. OTOH, its possible that (i) the issue may have been transient or (ii) the tasks already finished fetching that data before the error occurred. We really have no idea.

In our case, we are observing that any transient issue on the shuffle service might cause few tasks to fail. While other reducers might not see the fetch failure because either they already fetched the data from that shuffle service or they are yet to fetch it.  Killing all the reducers in those cases is waste of a lot of work and also as I mentioned above, we might end of in a state where jobs will not make any progress at all in case of frequent fetch failure, because they will just flip-flop between two stage.

I don't think its true that it relaunches all tasks that hadn't completed when the fetch failure occurred. it relaunches all the tasks haven't completed, by the time the stage gets resubmitted. More tasks can complete in between the time of the first failure, and the time the stage is resubmitted.

Actually, I realized that it's not true. If you looked at the code (https://github.com/sitalkedia/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1419), when the stage fails because of fetch failure, we remove the stage from the output commiter. So if any task completes between the time of first fetch failure and the time stage is resubmitted, will be denied to commit the output and so the scheduler re-launches all tasks in the stage with the fetch failure that hadn't completed when the fetch failure occurred.

when the stage fails because of fetch failure, we remove the stage from the output commiter. So if any task completes between the time of first fetch failure and the time stage is resubmitted, will be denied to commit the output

oh, that is a great point.  I was mostly thinking of another shufflemapstage, where that wouldn't matter, but if its a result stage which needs to commit its output, you are right.

Test build #75029 has started for PR 17297 at commit 99b4069.

Test build #75029 has finished for PR 17297 at commit 99b4069.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75029/
Test FAILed.

Thanks @markhamstra for review comments, addressed. I also found an issue with my previous implementation that we do not allow task commits from old stage attempts, I fixed that issue as well.

Test build #75030 has started for PR 17297 at commit 40a3742.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75030/
Test FAILed.

Test build #75124 has started for PR 17297 at commit c0bdca6.

Test build #75126 has started for PR 17297 at commit 05770b9.

Test build #75126 has finished for PR 17297 at commit 05770b9.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75126/
Test FAILed.

Test build #75127 has started for PR 17297 at commit 1aab715.

Jenkins retest this please.

Test build #75151 has started for PR 17297 at commit 1aab715.

@sitalkedia can you file a JIRA in the future when you see flaky test failures?  In this case I updated an existing JIRA (https://issues.apache.org/jira/browse/SPARK-19612) but please do this next time -- otherwise these issues never get fixed.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75151/
Test FAILed.

Test build #75176 has started for PR 17297 at commit b179439.

@kayousterhout - Sure will file a JIRA in future. Latest test failed and I am not sure if this is the same issue - https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75151/

Test build #75176 has finished for PR 17297 at commit b179439.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75176/
Test FAILed.

Merged build finished. Test FAILed.

@kayousterhout - It seems like the test timeout might be related to the change. But I am not able to find the culprit test case from the build log. Any idea what is wrong?

@sitalkedia I took a closer look -- I think this is from "o.a.s.InternalAccumulatorSuite: 'internal accumulators in resubmitted stages'".  From the console output on jenkins, that was the last test run, and also I can log into the box and see the log4j output from the tests in 'core/target/unit-test.log', and it shows that is the last test and it ends with:
17/03/24 13:44:19.537 dag-scheduler-event-loop ERROR DAGSchedulerEventProcessLoop: DAGSchedulerEventProcessLoop failed; shutting down SparkContext
java.lang.ArrayIndexOutOfBoundsException: -1
        at org.apache.spark.MapOutputTrackerMaster$$anonfun$getEpochForMapOutput$1.apply(MapOutputTracker.scala:431)
        at org.apache.spark.MapOutputTrackerMaster$$anonfun$getEpochForMapOutput$1.apply(MapOutputTracker.scala:430)
        at scala.Option.flatMap(Option.scala:171)
        at org.apache.spark.MapOutputTrackerMaster.getEpochForMapOutput(MapOutputTracker.scala:430)
        at org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1298)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1731)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1689)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1678)
        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
17/03/24 13:44:19.540 dispatcher-event-loop-11 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/03/24 13:44:19.546 stop-spark-context INFO MemoryStore: MemoryStore cleared
17/03/24 13:44:19.546 stop-spark-context INFO BlockManager: BlockManager stopped
17/03/24 13:44:19.546 stop-spark-context INFO BlockManagerMaster: BlockManagerMaster stopped
17/03/24 13:44:19.546 dispatcher-event-loop-16 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/03/24 13:44:19.547 stop-spark-context INFO SparkContext: Successfully stopped SparkContext
17/03/24 14:02:19.934 metrics-console-reporter-1-thread-1 ERROR ScheduledReporter: RuntimeException thrown from ConsoleReporter#report. Exception was suppressed.
java.lang.NullPointerException
        at org.apache.spark.deploy.master.MasterSource$$anon$2.getValue(MasterSource.scala:35)
        at org.apache.spark.deploy.master.MasterSource$$anon$2.getValue(MasterSource.scala:34)
        at com.codahale.metrics.ConsoleReporter.printGauge(ConsoleReporter.java:239)
...

with those NPE's repeatedly thrown every 15 minutes.
Normally that 'unit-tests.log' file is available as an archived artifact from jenkins, but I guess it doesn't show up if the test times out, so I don't think there is anyway for you to get at this info yourself.  Hopefully you can reproduce and debug locally?

@squito - Thanks, that helps a lot. I will fix the issue and submit a patch soon.

Test build #75287 has started for PR 17297 at commit 1e6e88a.

To recap the issue that Imran and I discussed here, I think it can be summarized as follows:

A Fetch Failure happens at some time t and indicates that the map output on machine M has been lost
Consider some running task that's read x map outputs and still needs to process y map outputs
Scenario A: (PRO of this PR) If the output from M was in the x outputs that are already read, we should keep running the task (as this PR does), because the task already successfully fetched the output from the failed machine. We don't do this currently, meaning we're throwing away the wasted work.
Scenario B: (CON of this PR) If the output from M was in the y outputs that have not yet been read, then we should cancel the task, because the task won't learn about the new location for the re-generated output of M (IIUC, there's no functionality to do this now) so is going to fail later on.  The current code will re-run the task, which is what we should do.  This code will try to re-use the old task, which means the job will take longer to run because the task will fail later on and need to be re-started.

If my description above is correct, then this PR is assuming that scenario A is more likely than scenario B, but it seems to me that these two scenarios are equally likely (in which case this PR provides no net benefit).  @sitalkedia what are your thoughts here / did I miss something in my description above?

@squito - I am not able to reproduce this issue locally.
The tests fails with some other issue -
``None.get
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:347)
at scala.None$.get(Option.scala:345)
at org.apache.spark.InternalAccumulatorSuite$$anonfun$1.apply$mcV$sp(InternalAccumulatorSuite.scala:43)
at org.apache.spark.InternalAccumulatorSuite$$anonfun$1.apply(InternalAccumulatorSuite.scala:39)
at org.apache.spark.InternalAccumulatorSuite$$anonfun$1.apply(InternalAccumulatorSuite.scala:39)
at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:68)
at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
at org.apache.spark.InternalAccumulatorSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(InternalAccumulatorSuite.scala:28)
at org.scalatest.BeforeAndAfterEach$class.runTest(BeforeAndAfterEach.scala:255)
at org.apache.spark.InternalAccumulatorSuite.runTest(InternalAccumulatorSuite.scala:28)
at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
at org.scalatest.Suite$class.run(Suite.scala:1424)
at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:31)
at org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)
at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)
at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:31)
at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:55)
at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2563)
at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2557)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2557)
at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044)
at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043)
at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722)
at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043)
at org.scalatest.tools.Runner$.run(Runner.scala:883)
at org.scalatest.tools.Runner.run(Runner.scala)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:138)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) ```
Please note that all InternalAccumulatorSuite tests fail on my laptop.
In the Jenkins log, do you see any other test cases having the java.lang.ArrayIndexOutOfBoundsException from MapOutputTrackerMaster ?

@sitalkedia how are you trying to run the test?  Works fine for me on my laptop on master.  Note that the test is referencing a var which is only defined if "spark.testing" is a system property: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala#L199
which it is in the sbt and maven build.  (maybe doens't work inside an IDE?  I'd strongly just using ~testOnly with sbt for faster dev iterations if you're not already)

Test build #75287 has finished for PR 17297 at commit 1e6e88a.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75287/
Test FAILed.

Merged build finished. Test FAILed.

@squito - I am able to reproduce the issue by running ./build/sbt  "test-only org.apache.spark.InternalAccumulatorSuite, however test case logs are not being printed on the console, do you know where can I find the test case logs on my laptop?
Also, one weird thing is that after adding system.testing property to my Intellij, all test cases succeeds without being stuck :/ .

@sitalkedia they're in core/target/unit-tests.log
Sometimes it's easier to move the logs to the tests (so they show up in-line), which you can do by changing core/src/test/resources/log4j.properties to log to the console instead of to a file.

@kayousterhout - Both the scenario A and B you described above are likely (it totally depend on the nature of the job and available cluster resources) and you are right that in case of scenario B, this PR will not provide any benefit.
I am planning to have a follow up PR to make the fetch failure handling logic better by not failing a task at all. In that case, the reducers can just inform the scheduler of lost map output and can still continue processing other available map outputs while the scheduler concurrently recomputes the lost map output. But that will be a bigger change in the scheduler.

btw I filed https://issues.apache.org/jira/browse/SPARK-20128 for the test timeout -- fwiw I don't think its a problem w/ the test but a potential real issue with the metrics system, though I don't really understand how it can happen.

@sitalkedia This change is pretty contentious, there are lot of questions about whether or not this is a good change.  I don't think discussing this here in github comments on a PR is the best form.  I think of PR comments as being more about code details -- clarity, tests, whether the implementation is correct, etc.  But here we're discussing whether the behavior is even desirable, as well as trying to discuss this in relation to other changes.  I think a better format would be for you to open a jira and submit a design document (maybe a shared google doc at first), where we can focus more on the desired behavior and consider all the changes, even if the PRs are smaller to make them easier to review.
I'm explicitly not making a judgement on whether or not this is a good change.  Also I do appreciate you having the code changes ready, as a POC, as that can help folks consider the complexity of the change.  But it seems clear to me that first we need to come to a decision about the end goal.
Also, assuming we do decide this is desirable behavior, there is also a question about how we can get changes like this in without risking breaking things -- I have started a thread on dev@ related to that topic in general, but we should figure that for these changes in particular as well.
@kayousterhout @tgravescs @markhamstra makes sense?

Sounds good to me.

Agree sounds good!

Test build #75332 has started for PR 17297 at commit bdaff12.

@squito - Sounds good to me, let me compile the list of pain points related to fetch failure we are seeing and also a design doc to have better handling of the issues.

Agreed. Let's establish what we want to do before trying to discuss the
details of how we are going to do it.
On Tue, Mar 28, 2017 at 8:17 AM, Imran Rashid<EMAIL_ADDRESS>wrote:

@sitalkedia https://github.com/sitalkedia This change is pretty
contentious, there are lot of questions about whether or not this is a good
change. I don't think discussing this here in github comments on a PR is
the best form. I think of PR comments as being more about code details --
clarity, tests, whether the implementation is correct, etc. But here we're
discussing whether the behavior is even desirable, as well as trying to
discuss this in relation to other changes. I think a better format would be
for you to open a jira and submit a design document (maybe a shared google
doc at first), where we can focus more on the desired behavior and consider
all the changes, even if the PRs are smaller to make them easier to review.
I'm explicitly not making a judgement on whether or not this is a good
change. Also I do appreciate you having the code changes ready, as a POC,
as that can help folks consider the complexity of the change. But it seems
clear to me that first we need to come to a decision about the end goal.
Also, assuming we do decide this is desirable behavior, there is also a
question about how we can get changes like this in without risking breaking
things -- I have started a thread on dev@ related to that topic in
general, but we should figure that for these changes in particular as well.
@kayousterhout https://github.com/kayousterhout @tgravescs
https://github.com/tgravescs @markhamstra
https://github.com/markhamstra makes sense?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/17297#issuecomment-289803690, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZ4-pbaJWHOMCLOB2JZFReBYx0E1xOHks5rqSSTgaJpZM4MdN08
.

Test build #75332 has finished for PR 17297 at commit bdaff12.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75332/
Test FAILed.

Merged build finished. Test FAILed.

Test build #75339 has started for PR 17297 at commit ace8464.

Test build #75339 has finished for PR 17297 at commit ace8464.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75339/
Test PASSed.

Merged build finished. Test PASSed.

@kayousterhout, @squito - Since we need more discussion on this change over a design doc, I have put out a temporary change (https://github.com/apache/spark/pull/17485) to kill the running tasks in case of fetch failure. Although this is not ideal but that would be better than current situation.

Should we temporarily close the PR and wait for the design doc to be finalized? @sitalkedia

okay, closing the PR.


There seems to be another (scamming?) website that is advertising this app...

So, I'm not really sure where to check about the legitimacy of this... i hope is not disrespectful to post it here.
I found this website: https://drawio-app.com which seems to copy the logo and advertise the same app, but with a twist: it asks for money. So, i'm kinda confused of what is going on here...

There are many implementations of draw.io to other products. This one is a paid integration to Confluence and Jira, this is completely allowed under the license.


Catching CacheManager exceptions globally

Hi,
We are using Redis as our backplane and would like to prevent the CacheManager exceptions from throwing when there is a connection exception as it short circuits a lot of code that should continue on (i.e. keep working even if the cache is down).  How do we do this globally?   There's no feasible way to wrap this around every call.
Specifically I'd like to prevent this from collapsing on us when we get in within our production Azure environment:
Message: An unknown error occurred when writing the message
StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:2622
StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/RedisBase.cs:54
StackExchange.Redis.RedisDatabase.ScriptEvaluate(Byte[] hash, RedisKey[] keys, RedisValue[] values, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:1182
CacheManager.Redis.RedisCacheHandle`1.Eval(ScriptType scriptType, RedisKey redisKey, RedisValue[] values, CommandFlags flags):138
CacheManager.Redis.RedisCacheHandle`1+<>c__DisplayClass48_0.<GetCacheItemAndVersion>b__0()
CacheManager.Redis.RetryHelper.Retry[T](Func`1 retryme, Int32 timeOut, Int32 retries, ILogger logger):50
CacheManager.Redis.RedisCacheHandle`1.Retry[T](Func`1 retryme)
CacheManager.Redis.RedisCacheHandle`1.GetCacheItemAndVersion(String key, String region, Int32& version)
CacheManager.Redis.RedisCacheHandle`1.GetCacheItemInternal(String key, String region)
CacheManager.Redis.RedisCacheHandle`1.GetCacheItemInternal(String key)
CacheManager.Core.Internal.BaseCache`1.GetCacheItem(String key):12
CacheManager.Core.BaseCacheManager`1.GetCacheItemInternal(String key, String region):70
CacheManager.Core.BaseCacheManager`1.GetCacheItemInternal(String key)
CacheManager.Core.Internal.BaseCache`1.GetCacheItem(String key):12
CacheManager.Core.Internal.BaseCache`1.Get(String key):40
CacheManager.Core.Internal.BaseCache`1.Get[TOut](String key)
EFCoreSecondLevelCacheInterceptor.EFCacheManagerCoreProvider+<>c__DisplayClass6_0.<GetValue>b__0()
EFCoreSecondLevelCacheInterceptor.ReaderWriterLockProvider.TryReadLocked[T](Func`1 action, Int32 timeout):19
EFCoreSecondLevelCacheInterceptor.EFCacheManagerCoreProvider.GetValue(EFCacheKey cacheKey, EFCachePolicy cachePolicy):38
EFCoreSecondLevelCacheInterceptor.DbCommandInterceptorProcessor.ProcessExecutingCommands[T](DbCommand command, DbContext context, T result, String methodName):506
EFCoreSecondLevelCacheInterceptor.SecondLevelCacheInterceptor.ReaderExecutingAsync(DbCommand command, CommandEventData eventData, InterceptionResult`1 result, CancellationToken cancellationToken):8

thanks,
jasen

I'm not really sure what you mean by that exactly and what EFCacheManager is.
This library can only do so much to ensure the code is working, which is, trying to make sure the call succeeds but eventually will fail.
Consumers have to react to that as needed. In most cases, errors are fine and can happen.
If in your case errors have to be handled, that has to be done in a layer on top of CacheManager I think.
But if you can explain the concrete use case with a good example, maybe I can help you out ;)

I'm pretty sure I understand what all these components are :-)
What I'm saying is that, in most scenarios, caching is not an choke point on failing out an entire transaction.  The design decision to throw an exception on anything that happens within the cache, in my case Redis connection errors, means that entire transaction has to deal with those exceptions.
Since most caching exceptions can probably be ignored, it would be nice if CacheManager supported either a configuration property to not throw any exceptions, or provided a global handler that would let us create a single point of managing them.
The problem is compounded even further when other third parties implement your library.  For example, our primary caching implementation is done using EFCoreSecondLevelCache, which perfectly integrates your library as a provider.  The end result is we have thousands of entity framework calls ultimately implementing your library with simple EF queries.  These thousands of calls look something as basic as:
var x = await _context.MyTable.Where(....).Cacheable().ToListAsync()
We don't want to, nor do I think it would be a proper implementation to, to wrap all of our calls to deal with caching exceptions on the off chance that a cache provider is not available.  With database transactions that's obviously important, but with caching, it simply is not critical path and we want the "normal" operations to continue, even if cache storage fails.  This is especially important with simple cache GETS.  I understand we can cause stale caches if exceptions are ignored on PUTS, but if the consumer of the cache doesn't care about potentially stale data, we should have that option.
I was just wondering if there was a way to handle this scenario globally without having to account for it on every, in our case, query.
Thanks again for this great library btw, I think that often times doesn't get said enough.

any chance of a "silent" option to not throw exceptions and interrupt the pipeline if caching fails?

It is very unlikely I'd add a single option to disable any exceptions. That would be extremely bad for most use cases.
I'd probably go for a more extendable error/retry handling.
Not sure yet what the best way would be or when that would be released though, sorry.

That would be my preferred suggestion as well, thank you!  We have a high volume web environment and we've found 99% of the exceptions that get thrown are Redis connections.  Perhaps there's an easy way to manage that one globally?


How to integrate with KeyCloak version 18 or 19?

The instruction mentioned seems not applicable for KeyCloak version 18 or 19, could you pl. help here and guide how to enable this feature.
Thanks.

+1

+1


Group icon color broken?

Describe the bug

Thx for this great card. I love it!
I configured the icon_color option for battery groups.
The icons remain white (in dark mode).
Option broken? Or did I not configure it correctly?
How to reproduce

Expected behavior

YAML configuration

# please paste here your card config

collapse:

group_id: group.batteries
name: 'Above 80% (count: {count})'
min: 81
icon: mdi:battery
icon_color: first
group_id: group.batteries_mobiles
name: 'Mobile sensors ({range}%, count: {count})'
icon: mdi:cellphone-basic
icon_color: first

Screenshots

Version

V2.1.0

I moved from manual install, to a HACS managed install and it's working as expected now. Don't understand, but issue resolved


Update git clone url.

Thanks for an awesome product. I noticed when cloning that this was incorrect in the docs.
Cheers!

Thanks!

Thanks!


Use base database

As part of #120 with this PR we are replacing internal/database with base/database package. Major change is that now we have /migations where all migrations are stored.
Some issues/concerns:

base/config can't be used out of the box right now as a replacement for Customers' configuration as it does not support environment variables. It reads config from the file. After we update base/config to work with env variables then we can replace Config in customers. Temporary Config is created.
we should backup production DB as I have no idea how this new migrations may affect production DB (as all migrations will be run on deploy) 🤷♂ -> to avoid this we can manually create (schema_migrations table and insert value of last migration (15))...
I don't know how OSS users will be affected by this migration (they may be on different versions, etc.). I don't see the easy way to migrate. (here is an raw idea) -> One way would be to ask them to upgrade to 0.4 version if they still are not, in 0.5 we can create schema_migrations table and insert number so next time migrations from /migrations will be skipped, in 0.6 we can migrate, ...

Tasks:

[x] replace internal/database with base/database

[x] create separate migration files in /migrations

[x] update makefile to generate file with embedded migrations (they are embedded into go file using pkger)

[x] fix all loggers to follow new base/log interface

[x] squash migrations

[ ] create instructions on how to update to v0.5 (dump; drop; create;load)

[ ] update docs with instructions on how to embed migrations (install pkger; make embed-migrations)

re: deployment in prod - I'm going to take a backup and move the tables to backfill with the new schemas. Will take a few minutes to do, so I'm not worried.
For OSS folks we will announce that v0.5 is a breaking change with this and they'll need to follow steps similar to what I end up doing.

@adamdecaf

re: deployment in prod - I'm going to take a backup and move the tables to backfill with the new schemas. Will take a few minutes to do, so I'm not worried.

then we have to manually create migration table (schema_migrations) and insert number of the latest migration there... or our application during the next run will run all migrations again.
is it ok?

We don't need to manually create any tables. The migrator library creates them for us.

Oh. Rigth. So I'll add it to /migrations

Codecov Report

Merging #253 into master will decrease coverage by 0.33%.
The diff coverage is 53.53%.

@@            Coverage Diff             @@
##           master     #253      +/-   ##
==========================================
- Coverage   61.81%   61.48%   -0.34%
==========================================
  Files          52       50       -2
  Lines        2988     2905      -83
==========================================
- Hits         1847     1786      -61
+ Misses        819      806      -13
+ Partials      322      313       -9

I see CI failing and I think we should run the same steps CI is doing locally to figure out what's going wrong.

Great work Pavel!


Invalidate gateway cache when blocking subscriptions while using Opaque tokens

Describe your problem(s)
When Opaque tokens are used to invoke APIs, they are cached in the Gateway. Therefore if a subscription is blocked from the Publisher portal, the blocking doesn't take effect immediately until the cache is expired.
Describe your solution
When JWT tokens are used and a subscription is blocked from the Publisher portal, the blocking takes effect immediately. It is suggested to implement this behavior for the Opaque tokens as well in away that gateway cache is invalidated when a subscription is blocked.
Suggested Labels:
3.0.0
3.1.0

This is currently on hold and will be reconsidered if any requirements arise.


Some helpers don't work on LegacyDatatableView

Because default_value don't go into the kwargs. Example code:
def get_bool_check_mark(object, **kwargs):
    if kwargs.get('default_value', None):
        return '✔'
    else:
        return '✘'

class LegacyPersonDatatableView(LegacyDatatableView):
    model = Person
    template_name = "school/persons_grid_dt.html"
    datatable_options = {
        'columns': ['id',
                    ('username', modal_link_to_model),
                    'first_name',
                    'last_name',
                    'email',
                    'school',
                    ('is_active', get_bool_check_mark),
                    ('is_pending', get_bool_check_mark),
                    ('is_staff', get_bool_check_mark),
                    ('is_school_contact', get_bool_check_mark),
                    'actions']
    }
    structure_template = "datatableview/bootstrap_structure.html"

Here modal_link_to_model works but not the get_bool_check_mark.

I think what's going on in the code you've posted is that the helper function is actually being provided as if it was the data source, which doesn't receive any special arguments.  Calling a callable in that second position is something that was added in 35d21d9 during our recent merges.  I think it risks confusion like this, but I see the value of providing callbacks (not helpers).
To correctly specify a processor callback, it must be the third argument in the tuple, which means that you are compelled to provide a verbose name in the first position:
("Active", 'is_active', get_bool_check_mark)

It's one of the several reasons why the legacy syntax was a pain to work with :(

I see. If you plan on supporting old syntax longer, and if it is worth the trouble, it can be required that the processor be provided only as a callable, so you know when it is string, it is the field name.

You can close this if you like..


add fastifySeverOptions

Add an optional second parameter to the createServer function to allow passing in extra Fastify server factory options.
My use case is to set higher bodylimit, the default 1Mib is too low for larger operations.
Passing bodylimit in the server.listen() options doesn't work in this case.

@mdluo
Thanks for the PR.
Fixed so that bodyLimit can be used.


Exception when using .vox file from MagicaVoxel

Hi! Great tool, but when I use a standard MagicaVoxel file like teapot.vox I get the following Exception:
Exception in thread "main" java.lang.NoClassDefFoundError: Ljava/lang/IndexOutOfBoundsException;
at com.oracle.svm.jni.functions.JNIFunctions.FindClass(JNIFunctions.java:348)
at java.io.FileInputStream.readBytes(FileInputStream.java)
at java.io.FileInputStream.read(FileInputStream.java:276)
at com.github.tommyettinger.LittleEndianDataInputStream.read(LittleEndianDataInputStream.java:89)
at com.github.tommyettinger.io.VoxIOExtended.readVox(VoxIOExtended.java:230)
at com.github.tommyettinger.headless.HeadlessLauncher.call(HeadlessLauncher.java:58)
at com.github.tommyettinger.headless.HeadlessLauncher.call(HeadlessLauncher.java:19)
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
at picocli.CommandLine.access$1300(CommandLine.java:145)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2358)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2352)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2314)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
at picocli.CommandLine$RunLast.execute(CommandLine.java:2316)
at picocli.CommandLine.execute(CommandLine.java:2078)
at com.github.tommyettinger.headless.HeadlessLauncher.main(HeadlessLauncher.java:46)
Any idea what is causing this?

Same error for me, i am using the latest Release tag of Spotvox and the version available on the MagicaVoxel website.
I can use the script on the provided .vox files but with whatever version of MagicaVoxel i want, if i export the .vox myself ( even if i open the .vox files included with Spotvox and save them myself as .vox) i get the same error.
Exception in thread "HeadlessApplication" java.lang.ArrayIndexOutOfBoundsException: Index -2 out of bounds for length 2
        at com.github.tommyettinger.Renderer.splatOnly(Renderer.java:472)
        at com.github.tommyettinger.Renderer.drawSplats(Renderer.java:519)
        at com.github.tommyettinger.SpotVox.create(SpotVox.java:78)
        at com.badlogic.gdx.backends.headless.HeadlessApplication.mainLoop(HeadlessApplication.java:108)
        at com.badlogic.gdx.backends.headless.HeadlessApplication$1.run(HeadlessApplication.java:93)
        at com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:704)
        at com.oracle.svm.core.windows.WindowsPlatformThreads.osThreadStartRoutine(WindowsPlatformThreads.java:143)

Nvm, now i regret sending the message.
So the issue only appears if you export to .vox
As I am new to this and didn't even saved a project was only testing the isomorphic part i didn't realise .vox is actually the extension of the project and that you just need to save the project normally and it will work

Well, I just downloaded the latest stable release of SpotVox (0.0.6) and the latest beta of MagicaVoxel (<IP_ADDRESS> released in May), unzipped both, copied Teapot.vox from MagicaVoxel's vox/ folder to SpotVox's folder, and dragged+dropped Teapot.vox onto the .exe and then the turntable .bat . It certainly works here:

I know using .vox files that were most recently saved by older versions of MagicaVoxel will not work, due to breaking changes ephtracy made in the .vox format. SpotVox tries to be compatible with the current MagicaVoxel release version. I've just saved older .vox files with the current MagicaVoxel when I've encountered issues with older files; typically making any change and undoing it is enough to allow it to save over the old-format file.


Paypal error: "based in a country that isn't enabled to receive payments"

Describe the bug
When I try to pay this expense, I get this Paypal error. The user says his account should not be restricted. Do we know what causes this? The error implies it's his country but he says it should work.

PayPal error: The email account Receiver not in receive money enabled country is based in a country that isn't enabled to receive payments (error id: 539041)

https://opencollective.com/phalcon/expenses/9255

As per https://www.paypal-community.com/t5/About-Payments/Can-t-Recieve-money-Error-539041/td-p/1145656, the user should try to see if USD is properly listed on https://www.paypal.com/myaccount/money/balances/summary. He can add it if it's not the case:

If it doesn't solve the problem, we'll need to have his PayPal country (from the address) and the nationality to investigate deeper. He can find them on https://www.paypal.com/myaccount/settings/:

Thanks! I'll tell the user.


fix: better output when importing

to use catalog folder names rather than the confusing 'build pack' text.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign jstrachan
You can assign the PR to them by writing /assign @jstrachan in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/approve

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: jstrachan
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [jstrachan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


As an educator, I want to assess the students knowledge with grading coding exercises so that I can confirm they are retaining the material at a deeper level.

Acceptance criteria:

[ ] Explore Lab 1 Quantum Circuits example of an integrated grader experience in a code block

[x] Qiskit/platypus#489
[x] Qiskit/platypus#823
[x] Qiskit/platypus#822
[x] Qiskit/platypus#259
[x] Qiskit/platypus#700
[x] Qiskit/qiskit.org#359
[x] Qiskit/qiskit.org#360

Note: The second acceptance criteria may be moved to its own user story

The auto-grader work is complete despite it not currently in Lab 1. I will close out this user story and open a new one around implementation of auto-grader in problem sets.


Minimum dataset in regulation

According to the latest adopted text in the regulation the minimum dataset shall include
date of vaccination, indicating the date of each dose received and of the latest dose received;
https://www.europarl.europa.eu/doceo/document/TA-9-2021-0145_EN.html

Introduce extra field in vaccination_entry that is an array of dates?

How do we fulfill the regulation regarding the other information about the vaccine if they are of different kinds the two doses? It states that the information about the vaccine manufacturer and type should be part of the datasets. So the schema works and fulfills the regulation if you add both doses without schema update.

Or we can just keep the current schema and allow multiple vaccinations in a single array.

Or we can just keep the current schema and allow multiple vaccinations in a single array.

Yes, that would be the best solution.
The only remaining topic is the ID ... (unique, see previous discussions ...)

How do we fulfill the regulation regarding the other information about the vaccine if they are of different kinds the two doses? It states that the information about the vaccine manufacturer and type should be part of the datasets. So the schema works and fulfills the regulation if you add both doses without schema update.

Yes, that would be the best solution.
But the only remaining topic is the ID ... (unique, see previous discussions ...)

In BE, we are going for a solution in which we will create 2 separate certificate for a person who received two doses.
Each certificate will house ONLY the data of the concerned dose.
However, the second certificate will mention the fact that it is the second dose that was administered.
Seems compliant to both Json and https://www.europarl.europa.eu/doceo/document/TA-9-2021-0145_EN.html
That would make the management of corrections a lot easier. What do you guys think?

@PeterEbraert This would be most unfortunate - how would that work at in inspection point? And how does the person verifying know that he/she must scan multiple HCERTs?
The schema was design explicitly to allow for multiple vaccinations etc., just for this use case. E.g. someone would be vaccinated with vaccin A and for some reason followed by B, both can go into the same DGC.

@PeterEbraert nothing preventing you doing that, LU are doing the same. Is a valid legal option that is supported by the DGC Schema. As @jschlyter points out, this may not be the most user-friendly approach, which is why the DGC Schema is designed to also accomodate those Member States who wish to incorporate more than one certificate per QR code. So long as you are in the end compliant with EU regulation, the choice is yours. The DGC Schema supports all these use cases.

Actually, in DE we are going for a similar approach. In fact, the doctor administering the second dose should check that there has been a dose administered before and thus certifies the full vaccination. I am not a doctor, nor regulator, but it would be my interpretation that a 2/2 certificate actually means exactly that: fully vaccinated. It only gets complicated if a combination would be forbidden later on.

but it would be my interpretation that a 2/2 certificate actually means exactly that: fully vaccinated

Not necessarily - quite a few edge cases here. In any case, two different perspectives here: (i) medical, and (ii) regulatory - as per EU regs: the legal people can set that 2/2 "is equivalent to" fully vaccinated for the purposes of cross border travel. From a medical perspective, the only thing I can conclude from 2/2 is that 2 doses have been administered with a nominal total series dosage of 2, nothing more than that. I would need to do a lot more looking at other medical to make any medical claim about whether this means - from a medical perspective - person is fully vaccinated. This is why e.g. the HL7 model has a separate field called "status" which can be set to e.g. completed. Anyway, out-of-scope here: for our purposes for cross-border travel - 2/2 (or 1/1 etc) is deemed to be equivalent to "fully vaccinated".

I fully agree. However, from a practical point of view, we can assume lazy people.


useBackgroundSession not working

I tried to enable background model download logic in WhipserAX app by modified the following code.
However, progressCallback return no updates somehow. How can I make it works?
From
    folder = try await WhisperKit.download(variant: model, from: repoName, progressCallback: { progress in
        DispatchQueue.main.async {
            loadingProgressValue = Float(progress.fractionCompleted) * specializationProgressRatio
            modelState = .downloading
        }
    })

To
    folder = try await WhisperKit.download(variant: model, useBackgroundSession: true, from: repoName, progressCallback: { progress in
        DispatchQueue.main.async {
            loadingProgressValue = Float(progress.fractionCompleted) * specializationProgressRatio
            modelState = .downloading
        }
    })

@raida42 Did this get resolved?

@atiorh No, I thought I should create the issue in https://github.com/huggingface/swift-transformers instead.
If it's fine to keep this issue in this repo. will reopen.


slugbuilder: Bump buildpacks

Many very minor changes and a few version bumps.
https://github.com/heroku/heroku-buildpack-ruby/compare/526dfadc...154fab36
https://github.com/heroku/heroku-buildpack-nodejs/compare/e14c33b6...b4921004
https://github.com/heroku/heroku-buildpack-clojure/compare/16d2e8fe...ddb79281
https://github.com/heroku/heroku-buildpack-python/compare/129ce5f0...ce3bdb37
https://github.com/heroku/heroku-buildpack-java/compare/13dee501...f4286d20
https://github.com/heroku/heroku-buildpack-scala/compare/91d0cfc4...5cd8c9b2
https://github.com/heroku/heroku-buildpack-php/compare/044a2899...1f671f8d
https://github.com/heroku/heroku-buildpack-go/compare/70f622b4...e25246f3

LGTM


Begin deploy fails with dep using relative file:../../../ reference

If a dependency uses a relative reference like:
 "dependencies": {
    "@enhance/arc-plugin-enhance": "file:../../../src/plugin/"
  },

It will run in begin dev or arc sandbox but it will fail to deploy with begin deploy with the following error:
Error: Begin exception: Plugin error:
- Cannot find plugin 'enhance/arc-plugin-enhance'! Are you sure you have installed or created it correctly?

I usually only need to do this if I am testing a plugin from a directory inside that repository.
Here is a repository that reporduces the error. I install the starter app with begin new project, then move the plugin to a directory outside that project directory, and reference it relatively.
repro-repo

I can use a different approach for testing. I understand that this is an unsupported use case. A project should have all dependencies inside the project folder.

OK, I see what's going on here now. Begin deploys from the project directory; it takes all of the contents of that project directory and sends it to the deployment API to publish. It is not inspecting the paths of any of the project's dependencies or the require or import statements in its business logic, it expects that if something is specified in the project, it would be found somewhere within the project folder. So this is not something that is going to work.


remove unnecessary require

require 'rubygems' is already required in Ruby 1.9 or later.

Stable still supports old Ruby, but can you target this PR against the master branch which is Ruby 2.0+

Thanks for reviewing. I will create another PR to master branch.
By the way, as long as  see the gemspec, stable branch appears to have only supports 1.9 or later.
https://github.com/middleman/middleman/blob/v3-stable/middleman-core/middleman-core.gemspec#L19
https://github.com/middleman/middleman/blob/v3-stable/middleman/middleman.gemspec#L19
do I misunderstand something?

You may be right, I'd been a long time since we branched from stable.
That said, this build is failing on 1.9.3: https://travis-ci.org/middleman/middleman/builds/83407233

Hmm, that actually looks unrelated. Merging.

Thanks!


dockerize everything

[x] murmur lib/cli
[x] murmur dapp
[x] murmur api
[x] docker compose

@driemworks here's the gist for the docker compose https://gist.github.com/juangirini/a58126b8bf9d6a32d4c6bdfed782b832
Please note that it doesn't run on Apple M, so I can't test it, could you please try it out?

amd64 version https://gist.github.com/juangirini/a58126b8bf9d6a32d4c6bdfed782b832
arm64 version https://gist.github.com/juangirini/5c3283c3d4ae4b8720b52e400b6303ac


authentication required

On a windows machine (server 2012 R2) the plugin trigger an authentication pop-up when dragging issues around

Hello, @brillox !
Are you sure that this alert is related to Taskodrome itself?
When you drag'n'drop an issue, the plugin sends XMLHttpRequest to the host. Maybe such requests are forbidden without appropriate authentication?

it only happen with that plugin
--- Original message ---
From:   Maxim Kuzmin<EMAIL_ADDRESS>Sent:   27 December 2017 08:31:40
To:     AuthenticEshkinKot/Taskodrome<EMAIL_ADDRESS>CC:     BRILLANTE, Massimo (NHS TAYSIDE<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject:        Re: [AuthenticEshkinKot/Taskodrome] authentication required (#11)
Hello, @brilloxhttps://github.com/brillox !
Are you sure that this alert is related to Taskodrome itself?
When you drag'n'drop an issue, the plugin sends XMLHttpRequest to the host. Maybe such requests are forbidden without appropriate authentication?

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/AuthenticEshkinKot/Taskodrome/issues/11#issuecomment-354075428, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAp2OlExrtQCqqaFrjo_l-xgclVAEPleks5tEgBhgaJpZM4RKlx9.

This message may contain confidential information. If you are not the intended recipient please inform the
sender that you have received the message in error before deleting it.
Please do not disclose, copy or distribute information in this e-mail or take any action in relation to its contents. To do so is strictly prohibited and may be unlawful. Thank you for your co-operation.
NHSmail is the secure email and directory service available for all NHS staff in England and Scotland. NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and other accredited email services.
For more information and to find out how you can switch, https://portal.nhs.net/help/joiningnhsmail


[Feature Request] Add Folia from PaperMC

Will it possible to add Folia support?
Folia url: https://github.com/PaperMC/Folia

Hey buddy,
Thanks for the suggestion regarding Folia - it’s certainly on our radar to look into once Folia is more stable. We have added your suggestion to our internal tracking system.
Our team is currently working on a number of important projects, including a recode of the Buycraft plugin shortly. We will keep your suggestion in mind as we work on this update and will note it in our development roadmap.
Since we have now noted your suggestion in our internal tracking system, we will be closing this GitHub ticket. However, if you have any further suggestions or feedback, please don't hesitate to open a new ticket and we'll be happy to take a look.
Thanks again for your valuable suggestion and for your continued support of Tebex.
Best regards,
The Integrations Team.


OptionButton Dropdown doesn't inherit transform

Tested versions
v4.2.1.stable.official [b09f793f5]
System information
Godot v4.2.1.stable - Windows 10.0.22631 - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 3060 Ti (NVIDIA; <IP_ADDRESS>65) - 11th Gen Intel(R) Core(TM) i5-11400F @ 2.60GHz (12 Threads)
Issue description
An OptionButton node's dropdown selection menu does not move with the rest of the node.

Steps to reproduce
Create an OptionButton, move it.
Minimal reproduction project (MRP)
optionbutton_bug.zip

The dropdown selection menu is a PopupMenu. PopupMenu inherits Popup, which inherits Window. It doesn't inherit the transform because Window does not have one, unlike OptionButton which inherits Control.
But it's a bit interesting, in my opinion. It could be an undesirable change in behavior from 3.x, because Popups used to be Controls.
You can work around this though, by moving the PopupMenu manually. You can get it from OptionButton.get_popup.


RFE request for SAP FAST Template creation

We have a customer who is requesting a FAST template for SAP.  Not sure if this is on the list of templates yet, but please add this request to the list if so.  Thank you.

We currently do not have an SAP template planned in the backlog. Can you provide some details around your use case and some details on what you are looking for?

Hi Shyawn,
I have a customer who is wanting to move to the new FAST templates.  They are currently using Sharepoint and SAP iApps.  It is still a use-case for them so I told them I would ask for an RFE for them.
Kind Regards,
[https://cdn.f5.com/webcommon/email-signature/images/f5-logo-rgb-30x30.jpg]https://f5.com/
Jenny Kendrick | Solutions Engineer
<EMAIL_ADDRESS>From: Shyawn Karim @.>
Sent: Tuesday, April 19, 2022 3:24 PM
To: F5Networks/f5-appsvcs-templates @.>
Cc: Jenny Kendrick @.>; Author @.>
Subject: Re: [F5Networks/f5-appsvcs-templates] RFE request for SAP FAST Template creation (Issue #102)
EXTERNAL MAIL: @.***
We currently do not have an SAP template planned in the backlog. Can you provide some details around your use case and some details on what you are looking for?

Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FF5Networks%2Ff5-appsvcs-templates%2Fissues%2F102%23issuecomment-1103225660&data=05|01|J.Kendrick%40f5.com|0d2986be0928431aad4308da22535a78|dd3dfd2f6a3b40d19be0bf8327d81c50|0|0|637860038659832344|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=G4jtWuj8escGsWArB8IYmrewudT1RDL4zWqhdzplESs%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJZYKRBPQNPL45UWJPBWPDTVF4W77ANCNFSM5TWNLKYQ&data=05|01|J.Kendrick%40f5.com|0d2986be0928431aad4308da22535a78|dd3dfd2f6a3b40d19be0bf8327d81c50|0|0|637860038659832344|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=zu47Hnlv304koEGD8O51ijyDqE9OvSWd4c%2FjZrs8Zt0%3D&reserved=0.
You are receiving this because you authored the thread.Message ID: @.***>


Don't mess with parent window scrollbars

Currently when the modal popup appears, the dApps' scrollbars disappear.  The space gained by hiding the scrollbar(s) causes all the dApp's content to shift around.  This is messy UX.
Would be cleaner if the popup did not mess with parent window scrollbars.

Can you pinpoint a necessary CSS change that could fix this?
Can you also show an example where this happens?
Happy to look for a fix.

Thanks @miohtama!
Examples, from https://alchemy.daostack.io/:
Before modal pops up, see scrollbar:

With modal popped up, no scroll bar:

Everything on the page moves when the scrollbar disappears and reappears.  I guess I would play with the styling of the background that appears under the modal.

Is this still necessary?

No, is not fixed.

Alright, I will get it on next release

@pedrouid @miohtama @dkent600  any news on this? the layout shift is unfortunate:/ thanks for the info

With stable version 2.0.0 of Web3Modal now released, we are officially dropping support for version 1.x
Due to this this issue/pr was marked for closing. It is highly recommended to upgrade as 2.x will be receiving further updates that will enable functionality for some of our newer sdks like auth and push as well as support for WalletConnect v2 (See this post about WalletConnect v1 being deprecated https://medium.com/walletconnect/walletconnect-v1-0-sunset-notice-and-migration-schedule-8af9d3720d2e)
If you need to continue using Web3Modal 1.x and require this feature/fix implemented, we suggest adding it via forking V1 branch.

I met same question. Is there any solution?

I met the same question and make body not scroll but root div scoll it can solve


Add New Orleans, LA a letter to the mayor and city council

To<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Subject: [*** INSERT UNIQUE SUBJECT LINE***]
Message (Don’t forget to replace [x]’s with your information!)
Dear Mayor Cantrell and City Council Members,
My name is [YOUR NAME] and I am a New Orleans resident writing to urge you to defund the New Orleans Police Department.
The NOPD takes an enormous share of the city’s budget, and that percentage has risen exponentially over the last two decades, taking away desperately needed resources from essential community programs and services. As a member of the community, I am an opposer to the high incarceration rates we have here in New Orleans. The investment in policing has not made us in the Black and Brown communities any safer and the system in which it operates is inarguably anti black, brown, and lgbtq+ – NOPD remains an embarrassment to this city and a lethal threat to what makes us New Orleans. Meanwhile you increase police spending. With New Orleans’s current precarious finances in jeopardy, it is clear that we must defund the police.
As New Orleans residents, we therefore demand that you take immediate action to ensure the following:

Reduce NOPD’s allocation from the General Fund by 50% (roughly $100 Million)
Discontinue use of fund dollars to pay for settlements due to police murder, misconduct, and negligence
Invest in housing, jobs, youth programs, restorative justice, and mental health workers to keep the community safe.

New Orleans can not wait any longer for a budget that meets the needs of its residents. The only way to achieve this is to take immediate steps to Defund NOPD.
Thank you for your time,
[YOUR NAME] [YOUR EMAIL] [YOUR PHONE NUMBER]

Let's go with:
Dear Mayor Cantrell and City Council Members,
My name is [YOUR NAME] and I am a New Orleans resident writing to urge you to defund the New Orleans Police Department.
The NOPD takes an enormous share of the city’s budget, and that percentage has risen exponentially over the last two decades, taking away desperately needed resources from essential community programs and services. As a member of the community, I am opposed to the high incarceration rates seen in New Orleans. The investment in policing has not made our Black and Brown communities any safer. NOPD remains an embarrassment to this city and a lethal threat to what makes us New Orleans. Meanwhile, police spending has increased. With New Orleans’ budget already under stress, it is clear that we must defund the police.
As New Orleans residents, we therefore demand that you take immediate action towards the following ends:

Reduce NOPD’s allocation from the General Fund by 50% (roughly $100 Million)
Discontinue use of General Fund dollars to pay for settlements due to police murder, misconduct, and negligence
Invest in housing, jobs, youth programs, restorative justice, and mental health workers to keep the community safe.

New Orleans can not wait any longer for a budget that meets the needs of its residents. The only way to achieve this is to take immediate steps to Defund NOPD.
Thank you for your time,
[YOUR NAME]
[YOUR EMAIL]
[YOUR ADDRESS]
[YOUR PHONE NUMBER]

Is it possible to add an email address? And add a this paragraph in the beginning? It looks great regardless! @minicreative @bingbongle @defund12neworleans
To:
<EMAIL_ADDRESS>My name is [NAME]. I am a resident of New Orleans, Louisiana. I am contacting you to demand a significant reduction to the $194 million that is currently allocated to public safety in the upcoming fiscal budget of 2021 and for a formal review of the New Orleans Police Department Union Contract from an independent party that has no affiliation with NOPD. The police funds make up roughly 20% of the budget, while other important public services do not receive even half as much funding. By reducing police funding and reducing incarceration rates, those funds can be reallocated to housing, education or sanitation (etc.). That way we can focus on building our community efforts and local organizations, while preventing future police brutality and violence.

@hp1872 Yep, will update


Add test case test_migration_with_rebuilding_replica

Signed-off-by: Chris Chien<EMAIL_ADDRESS>Add test case test_migration_with_rebuilding_replica

@chriscchien Please resolve the conflicts from the branch first.

@chriscchien Please backport to 1.3 and 1.2. Thanks.


How does it work? FAQs request

Hey really cool project thank you for your contributions to open source! My question is a direct one: on a high level, how does harper work? Instead of making my assumptions based on the code I'd love to read straight from the source if able, thank you. PS: once you get enough of these types of questions you may be able to put an FAQ section together.

You should go through this one, it does give some insight:
https://github.com/elijah-potter/harper/blob/master/harper-ls/README.md
Further, this reddit post is somewhat close to a FAQ section. He has answered a lot of questions there.
https://www.reddit.com/r/neovim/comments/1elhsxb/been_writing_a_better_grammar_checker_for_neovim/
But yes, an official FAQ section could be put up when the project reaches a certain stage.

@elijah-potter Thanks for creating this repo!
After reading the architecture doc, my main question is: what's the long-term plan for Harper? Will it always stick to patterns, or are you planning on expanding from there? Will it eventually be expanded to include the full capabilities of checkers like Grammarly or LanguageTool?

Sorry it took so long for me to get to this. I've created a draft of an architecture document. Give it a read and let me know if you have any questions it doesn't answer.

Thank you @elijah-potter I think this along with the linked docs in there helps a ton. Closing this issue since my question is answered but I'm curious about LandonSchropp's question as well. Thanks.


--include-distribution-metadata doesn't work with main package: Error, including metadata for distribution without including related package ''

I'm trying to access my project's full metadata at runtime.
Running poetry run python -m nuitka example --run (the other options are in the code file) outputs:
Nuitka-Options: Used command line options: --standalone --python-flag=-m --include-distribution-metadata=example example --run
Nuitka: Starting Python compilation with Nuitka '2.3.10' on Python '3.11' commercial grade 'not installed'.
FATAL: Error, including metadata for distribution 'example' without including related package ''.

When not using --include-distribution-metadata, the output is:
Nuitka-Options: Used command line options: --standalone --python-flag=-m example --run
Nuitka: Starting Python compilation with Nuitka '2.3.10' on Python '3.11' commercial grade 'not installed'.
Nuitka: Completed Python level compilation and optimization.
Nuitka: Generating source code for C backend compiler.
Nuitka: Running data composer tool for optimal constant value handling.
Nuitka: Running C compilation via Scons.
Nuitka-Scons: Backend C compiler: gcc (gcc 9).
Nuitka-Scons: Backend linking program with 8 files (no progress information available for this stage).
Nuitka-Scons: Compiled 7 C files using ccache.
Nuitka-Scons: Cached C files (using ccache) with result 'cache hit': 7
Nuitka: Keeping build directory 'example.build'.
Nuitka: Successfully created 'example.dist/example.bin'.
Nuitka: Launching 'example.dist/example.bin'.
Traceback (most recent call last):
  File "/media/sf_VM_Stuff/nuitka_test/example.dist/example/__main__.py", line 7, in <module>
RuntimeError: cannot locate package '' associated with metadata

The behavior is the same when not using package mode. I've tired quite a few other variations without any success thus far. Is this how the main package's metadata is supposed to be accessed with Nuitka?
Versions
❯ poetry run python -m nuitka --version
2.3.10
Commercial: None
Python: 3.11.9 (main, Apr  6 2024, 17:59:24) [GCC 9.4.0]
Flavor: Debian Python
Executable: /home/vboxuser/.cache/pypoetry/virtualenvs/example-r5nuuTwf-py3.11/bin/python
OS: Linux
Arch: x86_64
Distribution: Ubuntu (based on Debian) 20.04.6
Version C compiler: /usr/bin/gcc (gcc 9).

Python is installed from this PPA on Ubuntu 20.04.
Nuitka is installed with the latest Poetry using their official installer in a virtualenv generated by Poetry. I tried both Nuitka 2.3.10 and develop.
❯ poetry run python -m pip list -v
Package     Version Editable project location      Location                                                                                Installer
----------- ------- ------------------------------ ----------------------------------------------------------------------------------------------- ------------
example     0.1.0   /media/sf_VM_Stuff/nuitka_test /home/vboxuser/.cache/pypoetry/virtualenvs/example-r5nuuTwf-py3.11/lib/python3.11/site-packages poetry
Nuitka      2.3.10                                 /home/vboxuser/.cache/pypoetry/virtualenvs/example-r5nuuTwf-py3.11/lib/python3.11/site-packages Poetry 1.8.3
ordered-set 4.1.0                                  /home/vboxuser/.cache/pypoetry/virtualenvs/example-r5nuuTwf-py3.11/lib/python3.11/site-packages Poetry 1.8.3
zstandard   0.22.0                                 /home/vboxuser/.cache/pypoetry/virtualenvs/example-r5nuuTwf-py3.11/lib/python3.11/site-packages Poetry 1.8.3
...

Minimal Example
full_example.zip
example/__main__.py:
# The PySide6 plugin covers qt-plugins
# nuitka-project: --standalone
# nuitka-project: --python-flag=-m
# nuitka-project: --include-distribution-metadata=example

import importlib.metadata

print(importlib.metadata.metadata("example"))

pyproject.toml:
[tool.poetry]
name = "example"
version = "0.1.0"
description = "My favorite description"
authors = ["Your Name<EMAIL_ADDRESS>
[tool.poetry.dependencies]
python = "^3.11"
Nuitka = "^2.3.10"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Compilation vs. source cannot find the metadata as it's only created when you install it. I don't know how poetry works, but I assume you cannot access it either when you run your code via poetry run python -m example  either, so how would Nuitka be able to do it.
That said, something about FATAL: Error, including metadata for distribution 'example' without including related package '' seems wrong and you may in fact be able to access it, but "" seems strange for the package name, might be a misdetection of some sorts.

This does work at runtime, because the package is installed in editable mode. This happens automatically with poetry. The pip equivalent is pip install --editable. See https://setuptools.pypa.io/en/latest/userguide/development_mode.html. The files that end up created within site-packages in this case are example.pth and example-0.1.0.dist-info.

Ok, Nuitka finds them at compile time, the editable ones, but during relaunch the .pth effects that give metadata, that is lost, which is sort of the bug there then. I will look into that, could be easy.

Not going to manage this one in time, although it sort of could be relevant.


Adds more logs to try to track down account creation error

What is being added?
Adds more logs to try to better track where the account creation error is happening.
Checklist before requesting review

[ ] I have tested this locally
[ ] I have included unit tests
[ ] I have updated any corresponding documentation

Steps To Manually Test
Please provide us steps to reproduce the scenarios needed to test this.  If an integration test is provided, let us know how to run it. If not, enumerate the steps to validate this does what it's supposed to.

Start the operator
Run the thing
Observe X in the spec for the thing
Clean up the thing

Ref OSD-16603

/lgtm

Codecov Report

Merging #784 (4496671) into master (95adb97) will decrease coverage by 0.06%.
The diff coverage is 0.00%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #784      +/-   ##
==========================================
- Coverage   43.01%   42.95%   -0.06%
==========================================
  Files          43       43
  Lines        5933     5941       +8
==========================================
  Hits         2552     2552
- Misses       3140     3148       +8
  Partials      241      241

Files
Coverage Δ

controllers/account/account_controller.go
25.77% <0.00%> (-0.26%)
:arrow_down:

/lgtm


How to set partition ?

in a Kafka Producer (for example nodejs - kafka-node), in addition the topic, We can set the partition number as well for the message.
Like stated in the docs, we can set the topic: scheduler-target-topic: online-videos
Lets say we want to send the message to "online videos" and partition : 4
Can we set that in the header also ?

hello @yusuf5033 ,
Thank you for your message.
Today it is not possible to set the target-partition, but i think it can be done easily in the scheduler-target-topic like so :
scheduler-target-topic: online-videos:4 where 4 is the target partition.
WDYT ?

Hello,
just to be clear, when you say ".. i think it can be done easily ..." means after you implemented it right ? :)
I think that would be a great feature.

;) yes
For my information, how to plan to use the scheduler ? user use case are always an interesting feedback :)

Thank you,
We are switching to Kafka to deliver our bulk push notifications to user's mobile apps. In our admin panel, customers can send bulk notifications to their end users. We developed the the part where they can send messages right away, but  we also want to give them ability to schedule their messages to a future time. For that feature kafka-message-scheduler seems really a good solution.
But we use different topic partitions for each "notification type" so we are kind of stuck in that part. I can use yet another topic for just deliver "scheduled" messages to  actual topic / partitions but it is going to make things a bit longer.

Hello @yusuf5033,
To be sure to correctly undestand. Why did you use partition to identify each notification type ?
It seems more simple to use different topic for each notification type. What do you think ?


Cannot run play.spotify.com

Spotify complains about not supporting QBRT. I haven't looked into why this is the case, but maybe a lack of DRM + some user-agent/js detection?
The issue is light for now as I don't have a lot of details, but I might look at it more if I have more time.

Can confirm that they're using UA detection as setting application.ini to that of a regular Firefox browser will allow me to login. However, DRM doesn't seem to exist with QBRT and thus cannot playback Spotify yet.

In theory, qbrt should support the same web platform features as Firefox. Do you know the particular "DRM" feature that qbrt would need to support in order for Spotify to play music?

https://support.mozilla.org/en-US/kb/enable-drm
I believe this is downloadable module which needs to be enabled. I'm not sure how this would be downloaded in qbrt, tho.

Hmm, qbrt would need to at least set a couple of runtime preferences. qbrt doesn't currently support an app setting runtime preferences (#73), but you could manually edit qbrt's defaults/preferences/prefs.js file to set those preferences.
However, I'm unsure if that's sufficient to enable/download the WideVine CDM that Spotify presumably needs to play music. It might be necessary to do something else, such as loading browser/base/content/browser-media.js into the shell. More research is needed.


chatops.post_message seems to be ignoring Slack pretext

SUMMARY
chatops.post_message seems to be ignoring Slack pretext.
ISSUE TYPE
Pick one below and delete the rest:

Bug Report

STACKSTORM VERSION
st2 2.10.1, on Python 2.7.6
OS / ENVIRONMENT / INSTALL METHOD
Standard docker install.
STEPS TO REPRODUCE

st2 run chatops.post_message channel=testing message='To get a list of commands type: ```!help```' extra='{"slack": {"color":"#f48527","pretext":"Hey <!channel>, Ready for ChatOps?","title": "Ansible and ChatOps. Get started :rocket:","title_link":"https://stackstorm.com/2015/06/24/ansible-chatops-get-started-%f0%9f%9a%80/","author_name":"by StackStorm - IFTTT for Ops","author_link":"https://stackstorm.com/","author_icon":"https://stackstorm.com/wp/wp-content/uploads/2015/01/favicon.png","image_url":"https://i.imgur.com/HWN8T78.png","fields":[{"title":"Documentation","value":"https://docs.stackstorm.com/chatops/","short":true}]}}' > /dev/null

EXPECTED RESULTS
A pretext line in the Slack channel.
ACTUAL RESULTS
Pretext line missing.

I tested this using err-stackstorm and the problem isn't situated with the chatops.post_message action.  It handles and extra.slack options correctly and passes them on to the bot.
$ st2 run chatops.post_message channel='#ops' message='To get a list of commands type: `!st2help`' extra='{"slack": {"color": "#f48527","pretext": "Hey <!channel>, Ready for ChatOps?","title": "Ansible and ChatOps. Get started :rocket:","title_link": "https://stackstorm.com/2015/06/24/ansible-chatops-get-started-%f0%9f%9a%80/","author_name": "by StackStorm - IFTTT for Ops","author_link": "https://stackstorm.com/","author_icon": "https://stackstorm.com/wp/wp-content/uploads/2015/01/favicon.png","image_url": "https://i.imgur.com/HWN8T78.png","fields": [{"title": "Documentation","value": "https://docs.stackstorm.com/chatops/","short": true}]}}'

.
id: 5c6e6b47d506c3bda13ab35c
status: succeeded
parameters:
  channel: '#ops'
  extra:
    slack:
      author_icon: https://stackstorm.com/wp/wp-content/uploads/2015/01/favicon.png
      author_link: https://stackstorm.com/
      author_name: by StackStorm - IFTTT for Ops
      color: '#f48527'
      fields:
      - short: true
        title: Documentation
        value: https://docs.stackstorm.com/chatops/
      image_url: https://i.imgur.com/HWN8T78.png
      pretext: Hey <!channel>, Ready for ChatOps?
      title: 'Ansible and ChatOps. Get started :rocket:'
      title_link: https://stackstorm.com/2015/06/24/ansible-chatops-get-started-%f0%9f%9a%80/
  message: 'To get a list of commands type: ```!st2help```'
result:
  channel: '#ops'
  extra:
    slack:
      author_icon: https://stackstorm.com/wp/wp-content/uploads/2015/01/favicon.png
      author_link: https://stackstorm.com/
      author_name: by StackStorm - IFTTT for Ops
      color: '#f48527'
      fields:
      - short: true
        title: Documentation
        value: https://docs.stackstorm.com/chatops/
      image_url: https://i.imgur.com/HWN8T78.png
      pretext: Hey <!channel>, Ready for ChatOps?
      title: 'Ansible and ChatOps. Get started :rocket:'
      title_link: https://stackstorm.com/2015/06/24/ansible-chatops-get-started-%f0%9f%9a%80/
  message: 'To get a list of commands type: `!st2help`'
  output:
    channel: '#ops'
    extra:
      slack:
        author_icon: https://stackstorm.com/wp/wp-content/uploads/2015/01/favicon.png
        author_link: https://stackstorm.com/
        author_name: by StackStorm - IFTTT for Ops
        color: '#f48527'
        fields:
        - short: true
          title: Documentation
          value: https://docs.stackstorm.com/chatops/
        image_url: https://i.imgur.com/HWN8T78.png
        pretext: Hey <!channel>, Ready for ChatOps?
        title: 'Ansible and ChatOps. Get started :rocket:'
        title_link: https://stackstorm.com/2015/06/24/ansible-chatops-get-started-%f0%9f%9a%80/
    message: 'To get a list of commands type: `!st2help`'
    user: null
    whisper: false
  user: null
  whisper: false

The output contains the pretext.

This leads me to think the problem is with how the hubot-stackstorm adapter is handling the extra parameters.

Agreed. How do we proceed?

It would probably be best to open an issue under the hubot-stackstorm repo https://github.com/StackStorm/hubot-stackstorm/issues and reference this issue in it.

Not sure if this helps but using mattermost I have no problem with the pretext in the following format
{
  "color": "#474C5D",
  "mattermost": {
    "attachments": [
      {
        "pretext": "---",
        "actions": [
          {
            "integration": {
              "url": "https://example.com",
              "context": {
                "some": "value"
              }
            },
            "name": "button name"
          }
        ],
        "text": "example"
      }
    ]
  }
}

Thanks @nzlosh, done here: https://github.com/StackStorm/hubot-stackstorm/issues/162

Hi, so after much work with @blag over in the other issue, his conclusion is I need to come back here.
Can someone kindly review what we've done there, and give me some thoughts?

@lingfish To help folks who haven't kept up on StackStorm/hubot-stackstorm#162, please copy and paste the following files here:

Your sensor YAML file
Your sensor Python code (if you don't want to post this publicly please DM it to me on Slack)
Rule YAML file
The exact data that is being passed into chatops.post_message (please redact any private data)
The full trigger instance data (please redact any private date)


cupx files (ie 2 zip concatenated files) are not extracted correctly

Hello,
not sure if that's a misunderstanding of how to use binwalk (I'm quite beginer in that kind of tools) but it doesn't extract cupx files correctly.
Steps to reproduce:

Download Big_sample_file.cupx or Small_sample_file.cupx
from https://tophatsoaring.atlassian.net/browse/TH-760

Run binwalk --dd='.*' small_sample_file.cupx -M

binwalk creates 4 files
$ tree _small_sample_file.cupx.extracted/
_small_sample_file.cupx.extracted/
├── 0
├── 1ABB2
├── 1ABC8
└── 1AD2D

It should in fact simply find 2 zip files (which have been simply concatenated into one).
Reference bug https://github.com/XCSoar/XCSoar/issues/795
Cupx file format description can be found at https://downloads.naviter.com/docs/SeeYou_CUPX_file_format.pdf
Foremost extracts file correctly https://github.com/XCSoar/XCSoar/issues/795#issuecomment-1008918231 but I'd prefer using binwalk as it can easily be used as Python module.
Kind regards

This is likely due to binwalk identifying each ZIP entry as a separate ZIP file. Binwalk v3 is much better at preventing this type of thing, and successfully reports and extracts only the two ZIP files in the cupx files you linked to:


Creating a "build packet" and release pipeline

Per the conversation over in the forums, let's consider this feature request to help reduce obstacles and bring down the barrier to entry for the project.

We want to make it as easy as possible for people to get involved. The structure was designed with modularity in mind, and being able to have many different builds of subsystems in manageable ways. But it might be smart to consider implementing a more "build packet" version for those looking for instruction kit with less navigation. It is definitely worth thinking about.

emphasis mine

We love this idea. If anyone has any idea how to get this done, we'd love to hear it!

Years later, we're finally making progress on this with the release of v2 and Gobilda's OSR order page. While it's not a complete build packet, the number of manufacturers to source parts from has drastically gone down to the point that this no longer presents as large of an issue. While the build packet isn't there yet, since it's on its way and to clean up some old issues, I'll close this.

Hi Achlle, great news... BUT, I think the intent of this issue may have been lost over the years. This was originally created to address the lack of a "build packet", that is, provide students and makers a packet of information that contained everything they needed to get going (e.g. not have to build LaTex, etc. but rather have a simple book-like document ready to go). It's been years since I have been following the project, so maybe this is no longer an issue... but the ticket wasn't related to sourcing parts etc. Thanks for working on the project though, I had a blast participating way back in 2019. I hope all is still going well and there are hundreds, no, thousands, of Rovers driving around high school gymnasiums.

Gotcha - we did switch to readme files instead of latex a few years back and recently we created a website that has all the documentation autogenerated from the readme.md files, so I think that intent is accomplished. Most of the work going forward would be improving and extending the documentation for this new build version.
Maybe the time has come for you to start building again? We could use some experienced people like yourself 🙂

That would do it, so it looks like this issue is def. closed! Nice work! The LaTex was super interesting to use and learn, but def. a barrier to entry for most.
If you seriously need some help, I would be interested in learning the state of the project to see if my expertise could be utilized. Maybe I can lend a hand or two... shoot me a message, email, etc. I always wanted to add a LIDAR to the Rover, maybe you guys already did...