[SPARK-31801][API][SHUFFLE] Register map output metadata

What changes were proposed in this pull request?
Adds a ShuffleOutputTracker API that can be used for managing shuffle metadata on the driver. Accepts map output metadata returned by the map output writers.
Requires https://github.com/apache/spark/pull/28616.
Why are the changes needed?
Part of the design as discussed in this document, and part of the wider effort of SPARK-25299.
Does this PR introduce any user-facing change?
Enables additional APIs for the shuffle storage plugin tree. Usage will become more apparent when the read side of the shuffle plugin tree is introduced.
How was this patch tested?
We've added a mock implementation of the shuffle plugin tree here, to prove that a Spark job using a different implementation of the plugin can use all of the plugin points for an alternative shuffle data storage solution. But we don't include it here, in order to minimize the diff and the code to review in this specific patch. See https://github.com/apache/spark/pull/28902.

Test build #123022 has started for PR 28618 at commit 98821b9.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27665/
Test PASSed.

Test build #123022 has finished for PR 28618 at commit 98821b9.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123022/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28095/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28125/
Test PASSed.

Test build #123502 has started for PR 28618 at commit 25e98e7.

Test build #123504 has started for PR 28618 at commit 51df151.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28127/
Test PASSed.

Test build #123507 has started for PR 28618 at commit e7c9988.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28131/
Test PASSed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123502/
Test FAILed.

Test build #123504 has finished for PR 28618 at commit 51df151.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123504/
Test FAILed.

Test build #123507 has finished for PR 28618 at commit e7c9988.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123507/
Test FAILed.

I was thinking a lot on ShuffleDriverComponents and I have an idea how to improve it.
The problem I believe this class tries to fulfil two very separate roles: be a builder and the result of the building in the same time.
This is why we need this kind of check:
https://github.com/apache/spark/blob/289050e4309b024146867b7b32974ab4746eb570/core/src/main/java/org/apache/spark/shuffle/sort/io/LocalDiskShuffleDriverComponents.java#L47-L49
If the building is cleanly separated from the result of the building then we can be sure the prerequisites are fulfilled before.
I would change it by transforming it to be the result of the building in the following way:

the initializeApplication (I mean the process of the initialisation and not the returned Map) should be part of the building. The documentation of the ShuffleDataIO#driver method can be extended by mentioning this is the right place to initialize.
the ShuffleDriverComponents could have a new method which gives back the "additional SparkConf settings necessary for initializing the executor components" we can call it like additonalExecutorConfigs. This new method would replace the old initializeApplication

One more idea / question:

I do not see why the ShuffleOutputTracker is optional. Either we or the API user can provide an implementation where the methods are empty this way the API a bit simpler.

@mccheah what do you think?

Test build #124261 has started for PR 28618 at commit e7c9988.

Test build #124261 has finished for PR 28618 at commit e7c9988.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124261/
Test FAILed.

Retest this please.

Test build #124307 has started for PR 28618 at commit e7c9988.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28926/
Test PASSed.

Test build #124307 has finished for PR 28618 at commit e7c9988.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124307/
Test FAILed.

I was thinking a lot on ShuffleDriverComponents and I have an idea how to improve it.
The problem I believe this class tries to fulfil two very separate roles: be a builder and the result of the building in the same time.
This is why we need this kind of check:
https://github.com/apache/spark/blob/289050e4309b024146867b7b32974ab4746eb570/core/src/main/java/org/apache/spark/shuffle/sort/io/LocalDiskShuffleDriverComponents.java#L47-L49
If the building is cleanly separated from the result of the building then we can be sure the prerequisites are fulfilled before.
I would change it by transforming it to be the result of the building in the following way:

the initializeApplication (I mean the process of the initialisation and not the returned Map) should be part of the building. The documentation of the ShuffleDataIO#driver method can be extended by mentioning this is the right place to initialize.
the ShuffleDriverComponents could have a new method which gives back the "additional SparkConf settings necessary for initializing the executor components" we can call it like additonalExecutorConfigs. This new method would replace the old initializeApplication

One more idea / question:

I do not see why the ShuffleOutputTracker is optional. Either we or the API user can provide an implementation where the methods are empty this way the API a bit simpler.

@mccheah what do you think?

You bring up a good point. I can adjust the PR accordingly. It does seem like the components does both an initialization and a runtime mode, and it would be more ideal to separate the two. Thanks for critically thinking about this!

Also I think it makes sense for the executor side and the driver side to be mirrored.

I rebased on master in my latest patch. I also addressed your comments @attilapiros. Thanks for the feedback!
The diff was growing extremely large (> 1000 lines), so I removed all the tests for now. I'm going to open a separate patch with tests for this.

Test build #124385 has started for PR 28618 at commit dc8d15c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29004/
Test PASSed.

Test build #124385 has finished for PR 28618 at commit dc8d15c.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124385/
Test FAILed.

In BaseReceivedBlockHandlerSuite one parameter (a ShuffleOutputTracker) is missing but as the test does not do any shuffle we can pass a null as value (or a NoOpShuffleOutputTracker). With null:
index 0976494b6d..5f6a0f164f 100644
--- a/streaming/src/test/scala/org/apache/spark/streaming/ReceivedBlockHandlerSuite.scala
+++ b/streaming/src/test/scala/org/apache/spark/streaming/ReceivedBlockHandlerSuite.scala
@@ -70,7 +70,7 @@ abstract class BaseReceivedBlockHandlerSuite(enableEncryption: Boolean)
   val streamId = 1
   val securityMgr = new SecurityManager(conf, encryptionKey)
   val broadcastManager = new BroadcastManager(true, conf, securityMgr)
-  val mapOutputTracker = new MapOutputTrackerMaster(conf, broadcastManager, true)
+  val mapOutputTracker = new MapOutputTrackerMaster(conf, null, broadcastManager, true)
   val shuffleManager = new SortShuffleManager(conf)
   val serializer = new KryoSerializer(conf)
   var serializerManager = new SerializerManager(serializer, conf, encryptionKey)

 [info] ReceivedBlockHandlerSuite:
  [info] - BlockManagerBasedBlockHandler - store blocks (315 milliseconds)
  [info] - BlockManagerBasedBlockHandler - handle errors in storing block (5 milliseconds)
  [info] - WriteAheadLogBasedBlockHandler - store blocks (216 milliseconds)
  [info] - WriteAheadLogBasedBlockHandler - handle errors in storing block (13 milliseconds)
  [info] - WriteAheadLogBasedBlockHandler - clean old blocks (53 milliseconds)
  [info] - Test Block - count messages (121 milliseconds)
  [info] - Test Block - isFullyConsumed (24 milliseconds)
  [info] ReceivedBlockHandlerWithEncryptionSuite:
  [info] - BlockManagerBasedBlockHandler - store blocks (27 milliseconds)
  [info] - BlockManagerBasedBlockHandler - handle errors in storing block (2 milliseconds)
  [info] - WriteAheadLogBasedBlockHandler - store blocks (89 milliseconds)
  [info] - WriteAheadLogBasedBlockHandler - handle errors in storing block (8 milliseconds)
  [info] - WriteAheadLogBasedBlockHandler - clean old blocks (16 milliseconds)
  [info] - Test Block - count messages (71 milliseconds)
  [info] - Test Block - isFullyConsumed (14 milliseconds)
  [info] ScalaTest
  [info] Run completed in 4 seconds, 30 milliseconds.
  [info] Total number of tests run: 14
  [info] Suites: completed 2, aborted 0
  [info] Tests: succeeded 14, failed 0, canceled 0, ignored 0, pending 0
  [info] All tests passed.

Took a pass through this to address @attilapiros's latest comments and also to merge with the upstream master branch. Is there any more feedback for this PR, or this feature as a whole, the API design, etc?

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31454/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31459/
Test PASSed.

Test build #126831 has started for PR 28618 at commit edd5c05.

Test build #126836 has started for PR 28618 at commit 7baa3d2.

Test build #126831 has finished for PR 28618 at commit edd5c05.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126831/
Test FAILed.

Test build #126836 has finished for PR 28618 at commit 7baa3d2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126836/
Test FAILed.

Test build #126894 has started for PR 28618 at commit cce67ab.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31519/
Test PASSed.

Test build #126896 has started for PR 28618 at commit 409bebb.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31521/
Test PASSed.

Test build #126894 has finished for PR 28618 at commit cce67ab.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126894/
Test FAILed.

Test build #126896 has finished for PR 28618 at commit 409bebb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126896/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31527/
Test PASSed.

Test build #126904 has started for PR 28618 at commit a6fabd2.

Test build #126904 has finished for PR 28618 at commit a6fabd2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126904/
Test FAILed.

Test build #126998 has started for PR 28618 at commit 3c66353.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31622/
Test PASSed.

Test build #126998 has finished for PR 28618 at commit 3c66353.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126998/
Test FAILed.

Test build #127000 has started for PR 28618 at commit b88d724.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31624/
Test PASSed.

Test build #127000 has finished for PR 28618 at commit b88d724.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127000/
Test FAILed.

Test build #127012 has started for PR 28618 at commit 3505af8.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31636/
Test PASSed.

Test build #127012 has finished for PR 28618 at commit 3505af8.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class RegExpExtractAll(subject: Expression, regexp: Expression, idx: Expression)
    .doc(\"The name of a class that implements \" +
trait CachedBatch
trait CachedBatchSerializer extends Serializable
trait SimpleMetricsCachedBatch extends CachedBatch
abstract class SimpleMetricsCachedBatchSerializer extends CachedBatchSerializer with Logging
case class ColumnarToRowExec(child: SparkPlan) extends ColumnarToRowTransition with CodegenSupport
case class ApplyColumnarRulesAndInsertTransitions(
class ColumnStatisticsSchema(a: Attribute) extends Serializable
class PartitionStatistics(tableSchema: Seq[Attribute]) extends Serializable
case class DefaultCachedBatch(numRows: Int, buffers: Array[Array[Byte]], stats: InternalRow)
class DefaultCachedBatchSerializer extends SimpleMetricsCachedBatchSerializer
trait HashJoin extends BaseJoinExec with CodegenSupport

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127012/
Test PASSed.

Ok, I think the build passes according to Jenkins, though the tests run by Github actions appears to have been flaky. This is ready for review or for sign off.

Test build #127295 has started for PR 28618 at commit 89bb528.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31914/
Test PASSed.

Test build #127295 has finished for PR 28618 at commit 89bb528.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127295/
Test PASSed.

+CC @otterc

Ping here - what would be the best way to get this merged?

@mccheah I will take a look at this later this week/early next week.
+CC @squito, @holdenk who reviewed the design doc.

+CC @tgravescs
This is part of the changes currently being reviewed, which could impact push based shuffle.

thanks @mridulm I need to catch up on the docs and I'll take a look.

+CC @jiangxb1987

Test build #128474 has started for PR 28618 at commit 2b5108f.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33098/
Test PASSed.

@tgravescs sorry for the delay - addressed all the comments above.

Test build #128474 has finished for PR 28618 at commit 2b5108f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128474/
Test PASSed.

Btw, noticed that updateMapOutput was added as part of executor decommission support. We will need to hook that into ShuffleOutputTracker as well.
+CC @holdenk

I looked at the changes proposed here so that we can use the interfaces here for Push-based shuffle (SPIP and code).
In Push-based shuffle, we have introduced merge-statuses which represent all the map outputs that were merged into a larger block. These statuses are collected by the driver from the Shuffle Services.
I think we will be able to use the current ShuffleOutputTracker API. The implementation of this API could have the triggers for finalizing the shuffle merge.
I still have to wrap my head around how we can model mergeStatus as part of MapOutputMetadata. MultiplemapStatus's would point to a single mergeStatus, so this would introduce some complexity.
We may need to evolve them to fit the push-based shuffle use case. As long as we are open to potentially making some backward incompatible changes, these APIs look good to me for now.

Btw, noticed that updateMapOutput was added as part of executor decommission support. We will need to hook that into ShuffleOutputTracker as well.

@mridulm Can we also do this as follow-up? The main thing is, this patch is already at ~900 lines changed total (+s and -s combined) and I really don't want to increase the scope of this. This patch has already stalled from merging for awhile and I'd rather get something completed and have follow-up tasks than try to make the entire feature perfect in a single patch.
Can we add follow-up JIRA tasks that followed from the comments for purely additive changes to the API, and move forward with the scope of this patch as-is?

Sure @mccheah we can do that in follow up work to keep things more bite sized.
I was making sure we dont miss out of it, and tagging holden on this.

Thanks for working on this :)

sorry for my delay on getting back to this, could you up merge to latest?

Test build #129906 has started for PR 28618 at commit a6d974c.

Test build #129906 has finished for PR 28618 at commit a6d974c.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129906/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34512/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34512/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34512/
Test PASSed.

Test build #129950 has started for PR 28618 at commit f69cba7.

Test build #129950 has finished for PR 28618 at commit f69cba7.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129950/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34555/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34555/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34555/
Test PASSed.

Jenkins retest this please.

Test build #130236 has started for PR 28618 at commit f69cba7.

Test build #130236 has finished for PR 28618 at commit f69cba7.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130236/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34836/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34836/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34836/
Test PASSed.

For helping @mccheah who is busy with other projects I cloned this PR as #30763.
I will keep that up-to-date with master and react to review comments.
So if this fine for you please continue the review discussions there.


JDMTool error when accessing the card reader in MacOS

Hi,
I have tried jdmtool in MacOS Ventura 13.6.3 (22G436)
Python version 3.11.
App works and I can download my subscription but when I try to get information from the card reader as a previous step to writing (card reader works with original JDM application from Jeppesen) I get:
(jdmtool) ~ ➤ jdmtool detect
Found device: Bus 020 Device 016: ID 0e39:1250
Traceback (most recent call last):
  File "/Users/icordoba/anaconda3/envs/jdmtool/bin/jdmtool", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/icordoba/anaconda3/envs/jdmtool/lib/python3.11/site-packages/jdmtool/main.py", line 442, in main
    func(**kwargs)
  File "/Users/icordoba/anaconda3/envs/jdmtool/lib/python3.11/site-packages/jdmtool/main.py", line 66, in wrapper
    with handle.claimInterface(0):
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/icordoba/anaconda3/envs/jdmtool/lib/python3.11/site-packages/usb1/__init__.py", line 1146, in claimInterface
    mayRaiseUSBError(
  File "/Users/icordoba/anaconda3/envs/jdmtool/lib/python3.11/site-packages/usb1/__init__.py", line 127, in mayRaiseUSBError
    __raiseUSBError(value)
  File "/Users/icordoba/anaconda3/envs/jdmtool/lib/python3.11/site-packages/usb1/__init__.py", line 119, in raiseUSBError
    raise __STATUS_TO_EXCEPTION_DICT.get(value, __USBError)(value)
usb1.USBErrorAccess: LIBUSB_ERROR_ACCESS [-3]
(jdmtool) ~ ➤ jdmtool read-metadata
Found device: Bus 020 Device 016: ID 0e39:1250
Traceback (most recent call last):
  File "/Users/icordoba/anaconda3/envs/jdmtool/bin/jdmtool", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/icordoba/anaconda3/envs/jdmtool/lib/python3.11/site-packages/jdmtool/main.py", line 442, in main
    func(**kwargs)
  File "/Users/icordoba/anaconda3/envs/jdmtool/lib/python3.11/site-packages/jdmtool/main.py", line 66, in wrapper
    with handle.claimInterface(0):
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/icordoba/anaconda3/envs/jdmtool/lib/python3.11/site-packages/usb1/__init__.py", line 1146, in claimInterface
    mayRaiseUSBError(
  File "/Users/icordoba/anaconda3/envs/jdmtool/lib/python3.11/site-packages/usb1/__init__.py", line 127, in mayRaiseUSBError
    __raiseUSBError(value)
  File "/Users/icordoba/anaconda3/envs/jdmtool/lib/python3.11/site-packages/usb1/__init__.py", line 119, in raiseUSBError
    raise __STATUS_TO_EXCEPTION_DICT.get(value, __USBError)(value)
usb1.USBErrorAccess: LIBUSB_ERROR_ACCESS [-3]

Hi! Sorry for the slow response. I don't currently have a Mac to test it - but will in a few weeks.
I don't suppose you've found some workaround for this? I see a bug report about HID devices (https://github.com/libusb/libusb/issues/859) that basically says that libusb shouldn't be used for HID devices - but, I don't think a card reader counts as HID. In any case, I'll have a better idea once I can try it myself.

Can't help for this case but only confirm it works on macOS Sonoma 14.4.1
[~/dev/jdmtool/.venv/bin]$ ./jdmtool detect
Found device: Bus 002 Device 001: ID 0e39:1250
Firmware version: 20071203
Card inserted:
  IID: 0x1004100
  Unknown identifier: 0x10000000

this is "silver" card

Going to close this; feel free to reopen if it still happens.


Setup Kanban Board

Click here to visit our Kanban Board.
We are 5 in our team:

@MAleemH
@zewdu444
@fitz95
@EmekaEmmanuel
@techEmprez

Hi @zewdu444
Good job so far!
There are some issues that you still need to work on to go to the next project but you are almost there!
To highlight:

All cards have been assigned✔

Required Changes ♻

[ ] According to the project requirements:

At the time of this review, these are the pointings:
@zewdu444 18
@techEmprez 23.5
@MAleemH 18
@fitz95 18
@EmekaEmmanuel 23.5
According to the requirement, the difference among members cannot be more than 5.  However, 23.5 - 18 = 5.5.
5.5 > 5.
Kindly, correct this issue so we can meet this particular requirement.
Tip: Just reassign tasks, or play with the numbers.
Optional suggestions
Every comment with the [OPTIONAL] prefix is not crucial enough to stop the approval of this PR. However, I strongly recommend you to take them into account as they can make your code better.
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me in your question so I can receive the notification.
Please, do not open a new Pull Request for re-reviews. You should use the same Pull Request submitted for the first review, either valid or invalid unless it is requested otherwise.

As described in the Code reviews limits policy you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.

Hi @MAleemH ,
Wow, you did it 🎉

Thank you for implementing the changes requested by the previous reviewer 💪 🥇 ㊗
Well done!
Your project is complete!  There is nothing else to say other than... it's time to merge it :shipit:
Congratulations! 🎉
To highlight

Pointings are now equal✔

Optional suggestions
Every comment with the [OPTIONAL] prefix won't stop the approval of this PR. However, I strongly recommend you to take them into account as they can make your code better. Some of them were simply missed by the previous reviewer and addressing them will really improve your application.
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me in your question so I can receive the notification.

As described in the Code reviews limits policy  you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.


Add Spring app sample

We are using Spring Webflux and it will be great to also add some Spring app sample.

I think you can find some blogs with examples of how to use HR with Spring.
But since we want to minimize dependencies, and since we're not Spring experts, I don't want to have such an example as part of this repo.


ItemsControl is not arranged properly under some circumstances

Describe the bug
ItemsControl / ListBox etc. might miss an arrange in certain scenarios, leading to invalid Bounds (e.g. with a Height of 0) and therefore not showing up on the UI.
Issue #2462 might be related.
The same code works as expected in WPF.
To Reproduce
Check out the following simple project:
https://github.com/tvass83/AvaloniaIssues/tree/main/InvalidBoundsForItemsControlInAutoGrid

Click on 'SubView2' button => note the ItemsControl is displayed properly.
Click on 'SubView1' button.
Click on 'SubView2' button => ItemsControl is not shown at all.
Check Bounds property for ItemsControl => note Height=0.
Resize Window => ItemsControl is displayed properly.

Expected behavior
Just like in WPF, ItemsControl shall be properly arranged all the time.
Desktop (please complete the following information):

OS: Windows 10 version 20H2 (OS build 10.0.19042.804)
Avalonia version: observed in 0.10.6, but still reproducible in 0.10.13.

Additional context
This is kind of an evil bug. In complex UIs, it can be really difficult to track down, let alone provide a decent workaround.

@Gillibald @maxkatz6 Any thoughts on this guys?

I was hoping that this would be fixed by #9677 but investigating further the problem seems to be at the layout level. When the ItemsControl is hidden and re-shown, its MeasureOverride is called but then ArrangeOverride isn't called to actually arrange it. Needs more investigation.

Fixed by #10864.


VQE for molecules giving OSerror

Information

Qiskit Aqua version:0.19.6
Python version: 3.8
Operating system:Windows: WSL based venv

What is the current behavior?
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python3.8/concurrent/futures/process.py", line 102, in _python_exit
thread_wakeup.wakeup()
File "/usr/lib/python3.8/concurrent/futures/process.py", line 90, in wakeup
self._writer.send_bytes(b"")
File "/usr/lib/python3.8/multiprocessing/connection.py", line 183, in send_bytes
self._check_closed()
File "/usr/lib/python3.8/multiprocessing/connection.py", line 136, in _check_closed
raise OSError("handle is closed")
OSError: handle is closed
Steps to reproduce the problem
I have attached the file I am using. I am simply running the program with WSL Ubuntu 20.04 and python 3.8.2 in vs code
What is the expected behavior?
It should be returning graphs of the VQE energy and Exact Energy.
Suggested solutions
I think maybe this has to do with the optimizer? or maybe the program is running out of memory?
import numpy as np
import matplotlib.pyplot as plt
import copy
from qiskit import BasicAer
from qiskit.aqua import aqua_globals, QuantumInstance
from qiskit.aqua.algorithms import VQE, NumPyEigensolver
from qiskit.aqua.components.optimizers import SLSQP
from qiskit.chemistry.components.initial_states import HartreeFock
from qiskit.chemistry.components.variational_forms import UCCSD
from qiskit.chemistry.drivers import PySCFDriver
from qiskit.chemistry.core import Hamiltonian, QubitMappingType

molecule = 'H .0 .0 .0; Li .0 .0 {0}'
distances = np.arange(0.5,4.0,0.1)
vqe_energies = []
exact_energies = []

for d in distances:
    driver = PySCFDriver(molecule.format(d), basis='sto3g')
    qmolecule = driver.run()
    operator =  Hamiltonian(qubit_mapping=QubitMappingType.PARITY,
                            two_qubit_reduction=True, freeze_core=True,
                            orbital_reduction=[-3, -2])
    qubit_op, aux_ops = operator.run(qmolecule)

    result = NumPyEigensolver(qubit_op, aux_operators=aux_ops).run()
    exact_energies.append(np.real(result.eigenvalues))
    optimizer = SLSQP(maxiter=5)

    initial_state = HartreeFock(
                                operator.molecule_info['num_orbitals'],
                                operator.molecule_info['num_particles'],
                                qubit_mapping=operator._qubit_mapping,
                                two_qubit_reduction=operator._two_qubit_reduction)

    var_form = UCCSD(
                    num_orbitals=operator.molecule_info['num_orbitals'],
                    num_particles=operator.molecule_info['num_particles'],
                    initial_state=initial_state,
                    qubit_mapping=operator._qubit_mapping,
                    two_qubit_reduction=operator._two_qubit_reduction)

    vqe = VQE(qubit_op, var_form, optimizer)
    backend = BasicAer.get_backend("statevector_simulator")
    vqe_result = np.real(vqe.run(backend)['eigenvalue'])
    vqe_energies.append(vqe_result)

plt.plot(distances, vqe_energies, label='VQE')
plt.plot(distances,exact_energies, label='Exact')
plt.xlabel('Interatomic distance')
plt.ylabel('Energy')
plt.legend(loc='upper right')

The Qiskit code tries to take advantage of multiprocessing, where its going though parts that can run in parallel, with a goal of speeding up things overall. This seems to be failing under WSL in the area of multiprocessing. Now we do test under Windows directly and support that as a platform, same for Linux directly. One thing you might like to try is to set num_processes to 1 in aqua_globals and see if restricting computation to a single process will allow it to run for you.

I did so but none of my graphs are coming up.

You need to add a plt.show() at the end to make sure the graph appears when you run this outside of a notebook as python script.

I was able to run it and show the graph under WSL-2 Ubuntu 20.04 and python 3.8.3 (Anaconda env.) in vs code.
Very important: You need a XServer running, please look at: https://stackoverflow.com/questions/43397162/show-matplotlib-plots-in-ubuntu-windows-subsystem-for-linux-wsl1-wsl2
First I added to my .bashrc: export DISPLAY=grep -oP "(?<=nameserver ).+" /etc/resolv.conf:0.0
Once it ran I got in my case DISPLAY=<IP_ADDRESS>:0.0
Then I installed VcXsrv on Windows, enabled private/public firewall access and started it with "Disable access control" ticked.
I was able to show the graph with  plt.show() at the end. I never got the multiprocess error you mentioned. My Windows is Windows Version 2004 (OS Build 19041.388) running on a VMWare Fusion VM.

thank you @manoelmarques @woodsp-ibm ... I added plt.show() but forgot to include the updated script in this issue. I will take both suggestions into consideration and will update this issue tomorrow with my results. again, thank you for the help so far!

@mertall Did the information help you? If the issue is solved for you then please feel free to close this issue off.


Question about .gif files

Hello, when I upload .gif file the gif itself is not working. Is the problem just for me or it's not supposed to work?

Please follow the upgrade document and upgrade to v1.7.1.

I did the upgrade. I can upload the .gif but the image is not moving.

.gif files won't animate after being cropped.

Please upgrade to v1.7.2.


Error with Wkhtmltopdf 0.12.1

Hello
Today I tried the script and I got this error:
---- Install Wkhtmltopdf 0.12.1 ----
--2015-08-10 16:07:59-- http://downloads.sourceforge.net/project/wkhtmltopdf/ar chive/0.12.1/wkhtmltox-0.12.1_linux-trusty-amd64.deb
Resolving downloads.sourceforge.net (downloads.sourceforge.net)... <IP_ADDRESS>
Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|<IP_ADDRESS> 9|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2015-08-10 16:08:00 ERROR 404: Not Found.

Actually I have two errors, here is the second one:
---- Install Other Dependencies ----
Downloading/unpacking graphviz
Downloading graphviz-0.4.6-py2.py3-none-any.whl
Downloading/unpacking ghostscript
Downloading ghostscript-0.4.1.tar.bz2
Running setup.py (path:/tmp/pip_build_root/ghostscript/setup.py) egg_info for package ghostscript
warning: no previously-included files found matching '.hgignore'
warning: no previously-included files found matching '.hgtags'
warning: no files found matching '*.xml' under directory 'tests'
warning: no files found matching '*.py' under directory 'tests'
warning: no files found matching '*.html' under directory 'doc'
warning: no files found matching '*.1' under directory 'doc'

Downloading/unpacking gcc
Could not find any downloads that satisfy the requirement gcc
Cleaning up...
No distributions at all found for gcc
Storing debug log for failure in /home/fabrice/.pip/pip.log
Requirement already satisfied (use --upgrade to upgrade): gevent in /usr/lib/python2.7/dist-packages

I found the problem for the first error. The files are not hosted at sourceforge anymore.
All downloads are now hosted at http://download.gna.org/wkhtmltopdf/
The migration was announced on 2-July-2015, see the post on mailing list
For the second error I still don't know what is the problem.

Hello
It seems some modifications have been done but still I have errors
---- Install Other Dependencies ----
Downloading/unpacking graphviz
Downloading graphviz-0.4.6-py2.py3-none-any.whl
Downloading/unpacking mc
Could not find any downloads that satisfy the requirement mc
Some externally hosted files were ignored (use --allow-external mc to allow).
Cleaning up...
No distributions at all found for mc
Storing debug log for failure in /home/fabrice/.pip/pip.log
Requirement already satisfied (use --upgrade to upgrade): gevent in /usr/lib/pyt                              hon2.7/dist-packages
Downloading/unpacking gevent-psycopg2
Downloading gevent-psycopg2-0.0.3.tar.gz
Running setup.py (path:/tmp/pip_build_root/gevent-psycopg2/setup.py) egg_info                               for package gevent-psycopg2
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0                              .6.24.tar.gz
Extracting in /tmp/tmpvZQpVq
Now working in /tmp/tmpvZQpVq/distribute-0.6.24
Building a Distribute egg in /tmp/pip_build_root/gevent-psycopg2
warning: no files found matching 'Makefile' under directory 'docs'
warning: no files found matching 'indexsidebar.html' under directory 'docs'
And
---- Install Wkhtmltopdf 0.12.1 ----
--2015-08-23 15:25:26--  http://download.gna.org/wkhtmltopdf/0.12/0.12.1/wkhtmlt                              ox-0.12.1_linux-trusty-amd64.deb
Resolving download.gna.org (download.gna.org)... <IP_ADDRESS>
Connecting to download.gna.org (download.gna.org)|<IP_ADDRESS>|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14166104 (14M) [application/x-debian-package]
Saving to: ‘Downloads/wkhtmltox-0.12.1_linux-trusty-amd64.deb’
100%[======================================>] 14,166,104  32.4KB/s   in 8m 24s
2015-08-23 15:33:56 (27.4 KB/s) - ‘Downloads/wkhtmltox-0.12.1_linux-trusty-amd64                              .deb’ saved [14166104/14166104]
dpkg: error processing archive wkhtmltox-0.12.1_linux-trusty-amd64.deb (--instal                              l):
cannot access archive: No such file or directory
Errors were encountered while processing:
wkhtmltox-0.12.1_linux-trusty-amd64.deb
cp: cannot stat ‘/usr/local/bin/wkhtmltopdf’: No such file or directory
cp: cannot stat ‘/usr/local/bin/wkhtmltoimage’: No such file or directory


pandas.api.types.infer_dtype is not specified in stubs

This import is valid due to the documentation
from pandas.api.types import infer_dtype
But infer_dtype is not listed in typed/__init__.pyi file

Thanks @ivan-piskunov-cndt. Will have a fix soon

No solution was delivered but pandas-stubs has moved to another repository and will now be managed alongside pandas itself: https://github.com/pandas-dev/pandas-stubs
Not sure if that's fixed there. You can try it out by
pip install pandas-stubs==<IP_ADDRESS>626

Closing. Feel free to open it up in the new repository if newest version hasn't fixed that.


Improve support for doctests

STATUS:

#40 fixed all of the errors and many of the warnings

The remaining warnings are probably the column offset errors mentioned in rust-lang/rust#79417 (comment), and it's not clear whether we can fix them on our side at this time.

This is currently available via --doctests flag as an unstable feature.
cargo llvm-cov --doctests

Refs:

https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/instrument-coverage.html#including-doc-tests
https://github.com/rust-lang/rust/issues/79417

STATUS: I think it's better than before, but there are still a few problems.

warning: 4 functions have mismatched data

It would be great to have this working better. This tool is super handy.
Currently, running with --doctests I get some error output at the end, and it seems like the files that contain doctests get included twice in the coverage output:
error: src/block.rs: No such file or directory
warning: The file 'src/block.rs' isn't covered.
error: src/block/filter.rs: No such file or directory
warning: The file 'src/block/filter.rs' isn't covered.
error: src/block/iter.rs: No such file or directory
warning: The file 'src/block/iter.rs' isn't covered.
error: src/network.rs: No such file or directory
warning: The file 'src/network.rs' isn't covered.
error: src/protocol/addrmgr.rs: No such file or directory
warning: The file 'src/protocol/addrmgr.rs' isn't covered.
error: src/time.rs: No such file or directory
warning: The file 'src/time.rs' isn't covered.

@cloudhead Is there an example that can reproduce the error you encountered?

Yeah, though after reproducing this I realize some of these errors come from the --html flag, and on its own, --doctests doesn't output these errors, only in combination with --html.
The repo is here:
https://github.com/cloudhead/nakamoto
I ran: cargo llvm-cov --all --doctests --html
The issue with coverage output for files included twice appears with only --doctests though. Basically the paths are not matched, eg.:
net/poll/src/time.rs                     45                22    51.11%          14                 5    64.29%          78                30    61.54%           0                 0         -
src/time.rs                              22                 2    90.91%           4                 0   100.00%          89                 0   100.00%           0                 0         -

These are actually the same file, but I'm guessing the second one is from the doctest run, while the first is from the unit test run, I'm not sure.

Thanks. I'll take a look.

Ok, I think #40 fixed all of the errors and many of the warnings that happen in cloudhead/nakamoto.
Before #40:
warning: 5 functions have mismatched data
error: src/block.rs: No such file or directory
warning: The file 'src/block.rs' isn't covered.
error: src/block/filter.rs: No such file or directory
warning: The file 'src/block/filter.rs' isn't covered.
error: src/block/iter.rs: No such file or directory
warning: The file 'src/block/iter.rs' isn't covered.
error: src/network.rs: No such file or directory
warning: The file 'src/network.rs' isn't covered.
error: src/protocol/addrmgr.rs: No such file or directory
warning: The file 'src/protocol/addrmgr.rs' isn't covered.
error: src/time.rs: No such file or directory
warning: The file 'src/time.rs' isn't covered.
warning: 5 functions have mismatched data

After #40:
warning: 5 functions have mismatched data
warning: 5 functions have mismatched data

The remaining warnings are probably the column offset errors mentioned in https://github.com/rust-lang/rust/issues/79417#issuecomment-756418399, and it's not clear whether we can fix them on our side at this time.

Awesome, thanks a bunch!

I am not sure whether I have missed any of the prerequisites, but I have tried to use the --doctests option in the rust-numpy crate, c.f. https://github.com/PyO3/rust-numpy/pull/286, but it appears that while the doctests are executed, they are not included in the coverage computation (whereas both unit tests and integration tests work). There are no error messages and no warnings besides the initial warning that the feature is unstable. I can also reproduce this locally, e.g. if I run cargo llvm-cov --doc --html I get an empty report.
Can anybody advise if am missing some dependency or step? Thank you!

My understanding is that the coverage is broken due to a bug of rustc's --remap-path-prefix (#140, #122).
Could you try #141?

Could you try https://github.com/taiki-e/cargo-llvm-cov/pull/141?

I installed cargo-llvm-cov using
> cargo install --force --git https://github.com/taiki-e/cargo-llvm-cov.git --branch disable-remap-path-prefix cargo-llvm-cov

but did not observe any changes in the output. I guess I need to wait for #122 and therefore https://github.com/rust-lang/rust/pull/92648 to land?

I don't know if this is related to this issue here but adding the below lines to a test will give me warning: 4 functions have mismatched data.
let candy_color = [
	String::from("#ea3223"),
	String::from("#377d22"),
	String::from("#fffd54"),
	String::from("#ea3df7"),
	String::from("#74fbfd"),
	String::from("#ee776d"),
	String::from("#8cf57b"),
	String::from("#fffb7f"),
	String::from("#6974f6"),
	String::from("#ee82f8"),
	String::from("#8dfafd"),
];
assert!(candy_color.contains(&color2hex(&Colors::Candy, &options)));

These lines are within a *_test.rs file inside the test folder and inside a #[cfg(test)] and #[test] block.
They test the output of a function that randomizes it's output.

@dominikwilkowski I believe that's also a rustc bug.

fwiw, now that instrument-coverage is stabilized, https://doc.rust-lang.org/rustc/instrument-coverage.html#including-doc-tests is the link to the relevant paragraph in the upstream docs

@jonhoo Thanks. I updated the link.

@dominikwilkowski If an issue you have encountered has not yet been fixed, would you mind submitting an issue about it at https://github.com/rust-lang/rust?

@taiki-e what's a good way to re-produce this bug without llvm-cov? Once I know how to re-produce this I'll happily submit a ticket to rust

I think we can get the commands that cargo-llvm-cov internally calls and adjust them. For example, the output of the following command should be able to be run as-is:
cargo llvm-cov --html -vv 2>&1 | grep CARGO_INCREMENTAL | sed -E -e 's/^ *Running `//g; s/`$//g; s|-sparse -f .*-o|-sparse target/llvm-cov-target/*.profraw -o|; s|.*/bin/llvm-|"$(rustc --print target-libdir)"/../bin/llvm-|g' -e "s|$(pwd)/||g"

Here's the output from mobilecoinfoundation/from-random
CARGO_INCREMENTAL=0 LLVM_PROFILE_FILE='target/llvm-cov-target/from-random-%p-%m.profraw' RUSTFLAGS='-C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /home/user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo test --tests --manifest-path Cargo.toml --target-dir target/llvm-cov-target -v

"$(rustc --print target-libdir)"/../bin/llvm-profdata merge -sparse target/llvm-cov-target/*.profraw -o target/llvm-cov-target/from-random.profdata

"$(rustc --print target-libdir)"/../bin/llvm-cov show -format=html -instr-profile=target/llvm-cov-target/from-random.profdata -object target/llvm-cov-target/debug/deps/mc_from_random-0b047475ff4e3b36 -ignore-filename-regex '/rustc/[0-9a-f]+/|^/home/user/src/mobilecoinfoundation/from\-random(/.*)?/(tests|examples|benches)/|^/home/user/src/mobilecoinfoundation/from\-random/target/llvm\-cov\-target($|/)|^/home/user/\.cargo/(registry|git)/|^/home/user/\.rustup/toolchains($|/)' -show-instantiations=true -show-line-counts-or-regions -show-expansions -Xdemangler=/home/user/.cargo/bin/cargo-llvm-cov -Xdemangler=llvm-cov -Xdemangler=demangle -output-dir=target/llvm-cov/html


Performance/Memory issues with rdl.Validate when FullValidation enabled

The current go implementation of rdl.Validate calls NewTypeRegistry(schema) on every invocation.  For a complex schema, this is a costly operation.
https://github.com/ardielle/ardielle-go/blob/0e40fcc616a01e6cd32f526a866ef18c762df270/rdl/validator.go#L46
This becomes apparent in go when unmarshalling large objects with nested structs containing a lot of "derived-from-String" types.
In tests, I've found that code which would just cache the TypeRegistry per schema when calling schema.Build() and then use that in a rdl.Validate2() style call which can accept a preconstructed TypeRegistry gives a close to 10x improvement on JSON unmarshalling speed.

Thanks, good find. Will fix that.

I have a fix which involves exposing a Validator object -- and then having the code generator generate an instance of that at schema build time.   I'll submit a PR, but I don't know enough details about the implementations in other languages to know whether it's the best way to solve this issue.

Well, Fixing it behind the scenes would be best, it is just a performance optimization for the Go library, other libraries have different ways to deal with it, so don't worry about those just yet. If you want to just send me a sketch (or pointer to your changes) I can integrate it, I just thought you were closer to it, having done the performance comparison already.

I changed it to use a transparent cache - no change to the exposed API.
Take a look at #29

Fixed in Release 1.4.13, PR #29


Bump dependencies

Was just trying this repo out and getting this error:
$ docker-compose run app py.test
Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.11/site-packages/py/_apipkg.py", line 118, in __makeattr
    modpath, attrname = self.__map__[name]
                        ~~~~~~~~~~~~^^^^^^
KeyError: '__spec__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/bin/py.test", line 5, in <module>
    from pytest import main
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.11/site-packages/pytest.py", line 9, in <module>
    from _pytest.config import (
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.11/site-packages/_pytest/config.py", line 13, in <module>
    import _pytest._code
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.11/site-packages/_pytest/_code/__init__.py", line 3, in <module>
    from .code import Code  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.11/site-packages/_pytest/_code/code.py", line 11, in <module>
    reprlib = py.builtin._tryimport('repr', 'reprlib')
              ^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.11/site-packages/py/_apipkg.py", line 125, in __makeattr
    result = importobj(modpath, attrname)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.11/site-packages/py/_apipkg.py", line 48, in importobj
    module = __import__(modpath, None, None, ['__doc__'])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1138, in _find_and_load_unlocked
  File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.11/site-packages/py/_apipkg.py", line 123, in __makeattr
    raise AttributeError(name)
AttributeError: __spec__

It seems to be this issue, I ran pipenv update and it now passes:
$ docker-compose run app py.test
=========================================================== test session starts ===========================================================
platform linux -- Python 3.11.1, pytest-7.2.0, pluggy-1.0.0
rootdir: /app
collected 1 item

test_example.py .                                                                                [100%]

============================================================ 1 passed in 0.00s ============================================================

This was fixed in #18


Automated Package Upgrade

:tada: This PR is included in version 5.0.472 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Browser back button malfunctioning.

I could be missing something, but I have the latest update and now the back button is no longer responsive on my site. With the 'debugMode' set to true, I can see that clicking the back button fires the events:
swup:popState
swup:animationSkipped
swup:pageRetrievedFromCache
swup:willReplaceContent
swup:contentReplaced
Removed 0 / added 1 tags in head
swup:pageView
swup:animationInDone
However there is no response from the browser. My guess is that I am running into some sort of a cache issue, where it is returning the current page as a cache object instead of the previous page. Any insight would be much appreciated.

NEVERMIND. I forgot I had done some previous debugging with the history pushstate and left some breaking code in. Disregard. My issue had nothing to do with Swup


fix add table sink and some fix in pipelineX

Proposed changes
do pipelineX = true in this pr
https://github.com/apache/doris/pull/25303
and
https://github.com/apache/doris/pull/25315

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall

run p0

run p0


double-beep: Watch nelaanna

double-beep requests the watch of the watch_keyword nelaanna. See the MS search here and the Stack Exchange search here.

nelaanna has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by tripleee in Charcoal HQ


Tnb Sample App

Basic structure and boilerplate to build over the app on with MVVM
Profile Overview
Profile Transactions
Validator Overview
Validator Accounts
Bank Overview
Bank Accounts

Hey @amarjeet987 can you show me screenshots of what the app looks like? It would might easier for me to review this large PR.

Hey @amarjeet987 can you show me screenshots of what the app looks like? It would might easier for me to review this large PR.

Sending you a screen recording on Slack.

Hey @amarjeet987 do you need help with this?

Hey @amarjeet987 do you need help with this?

Working on the DI rn and the viewmodel scope update, will ping for sure if I get stuck somewhere.

Hey @amarjeet987 can we fix the conflicts? other than that it looks good.

Hey @amarjeet987 can we fix the conflicts? other than that it looks good.

Yeah sure, let me know when we can do it.

Hey @amarjeet987 can we fix the conflicts? other than that it looks good.

Yeah sure, let me know when we can do it.

Could you not resolve the merge conflicts from your branch? Just merge master into your feature and you should be able to use the Android Studio UI to resolve them.

Hey @amarjeet987 can we fix the conflicts? other than that it looks good.

Yeah sure, let me know when we can do it.

Could you not resolve the merge conflicts from your branch? Just merge master into your feature and you should be able to use the Android Studio UI to resolve them.

Oh okay that, will do.


External PD Libraries

Hi,
I'm just wondering is there away to use an external PD library through heavy to unity, or would they considered incompatible.
I'm planning on working on an ambisonics patch and it seems I'll have to use various external to achieve success.

Unfortunately there is no way to run an existing Pd external through heavy. This feature is unlikely to be supported soon.
But in your case, there are existing ambisonics and spatial audio plugins for unity that you are probably better off using (from an integration perspective).

What about the rest of the objects from the vanilla distribution like fft~ and ifft~, that we can both find in the object list from the help menu?

fft~ and ifft~ have been on our list for a long time. I can't say when we'll have them, but we are absolutely planning on supporting them.


Black Screen and loading 2 -3 minutes

I am using you tube video, when i use 2-3 minutes loading and show black screen, i use 20 minutes big video how can i solve this problem please help me i want to loading less time

Duplicate #9


avisynthplus: Add version 3.7.2

Closes #10718

[x] I have read the Contributing Guide.

/verify


What's new in .NET 6 Preview 3 [WIP]

What's new in .NET 6 Preview 3 [WIP]
This issue is for teams to highlight work for the community that will release .NET 6 Preview 3.
To add content, use a new conversation entry. The entry should include the team name and feature title as the first line as shown in the template below.
## Team Name: Feature title

[link to the tracking issue or epic item for the work]

Tell the story of the feature and anything the community should pay particular attention
to be successful using the feature.

Preview 1: https://github.com/dotnet/core/issues/5853
Preview 2: https://github.com/dotnet/core/issues/5889
Preview 3: https://github.com/dotnet/core/issues/5890
Preview 4: https://github.com/dotnet/core/issues/6098
Preview 5: https://github.com/dotnet/core/issues/6099

.NET Libraries: System.Runtime.InteropServices - CollectionsMarshal.GetValueRef
dotnet/runtime#49388
Updating struct values in dictionaries currently can be expensive as it requires a dictionary lookup and a copy to stack of the struct; then after changing the struct it needs to be assigned to the dictionary key again resulting in another look up and copy operation.
A new unsafe api has been added; for high preformance scenarios, that returns a ref to the struct value which can then be updated in place. This reduces the key hashing to 1 (from 2) and removes all the struct copy operations.
Example:
ref MyStruct value = CollectionsMarshal.GetValueRef(dictionary, key);
// Returns Unsafe.NullRef<TValue>() if it doesn't exist; check using Unsafe.IsNullRef(ref value)
if (!Unsafe.IsNullRef(ref value))
{
    // Mutate in-place
    value.MyInt++;
}

Faster interface checking and casting
dotnet/runtime#49257
One of the advantages of more parts of the .NET VM moving from C++ to managed C# is its easier to contribute to,
this includes interface casting.
.NET 6 Preview 3 includes a 16% - 38% performance boost for interface casting; this is particularly useful for C#'s pattern matching to and between interfaces.

Method
Implementation
Mean
Version
Ratio
Improvement

IsInterface1
Interfaces1
2.166 ns
.NET 6 Preview 2
1.000

IsInterface1
Interfaces1
1.629 ns
.NET 6 Preview 3
0.752
x1.3

IsNotImplemented
Interfaces1
2.166 ns
.NET 6 Preview 2
1.000

IsNotImplemented
Interfaces1
1.950 ns
.NET 6 Preview 3
0.900
x1.1

IsInterface1
Interfaces6
2.155 ns
.NET 6 Preview 2
1.000

IsInterface1
Interfaces6
1.936 ns
.NET 6 Preview 3
0.898
x1.1

IsInterface2
Interfaces6
2.161 ns
.NET 6 Preview 2
1.000

IsInterface2
Interfaces6
1.908 ns
.NET 6 Preview 3
0.883
x1.1

IsInterface3
Interfaces6
2.188 ns
.NET 6 Preview 2
1.000

IsInterface3
Interfaces6
1.672 ns
.NET 6 Preview 3
0.764
x1.3

IsInterface4
Interfaces6
2.346 ns
.NET 6 Preview 2
1.000

IsInterface4
Interfaces6
1.965 ns
.NET 6 Preview 3
0.838
x1.2

IsInterface5
Interfaces6
3.122 ns
.NET 6 Preview 2
1.000

IsInterface5
Interfaces6
2.173 ns
.NET 6 Preview 3
0.696
x1.4

IsInterface6
Interfaces6
3.472 ns
.NET 6 Preview 2
1.000

IsInterface6
Interfaces6
2.687 ns
.NET 6 Preview 3
0.774
x1.3

IsNotImplemented
Interfaces6
3.644 ns
.NET 6 Preview 2
1.000

IsNotImplemented
Interfaces6
3.071 ns
.NET 6 Preview 3
0.843
x1.2

Thanks for adding these @benaadams !

CodeGen
Community contributions

@nathan-moore Remove bounds checks with tests against length https://github.com/dotnet/runtime/pull/40180
@SingleAccretion Span bounds check elision with top-level range check nodes removal https://github.com/dotnet/runtime/pull/49271

Dynamic PGO https://github.com/dotnet/runtime/issues/43618

JIT: profile updates for return merges and tail calls  https://github.com/dotnet/runtime/pull/48773
Class profile: Use unknown placeholder for collectible class typehandle https://github.com/dotnet/runtime/pull/48707

Stabilize performance measurements https://github.com/dotnet/runtime/issues/43227

Blog post on Loop Alignment https://github.com/microsoft/dotnet-blog/pull/572

Keep structs in register https://github.com/dotnet/runtime/issues/43867

Completed struct improvement, part1: create more LCL_FLD https://github.com/dotnet/runtime/pull/48377
Improve liveness for 'STORE_BLK(lcl_var)' https://github.com/dotnet/runtime/pull/48797

Completed .NET 6 EH Write Thru https://github.com/dotnet/runtime/issues/35923

Enregister EH var that are single def  https://github.com/dotnet/runtime/pull/47307
7 ~ 18% perf improvement https://github.com/DrewScoggins/performance-2/issues/4275#issuecomment-797001008

Optimizations:

Support loop cloning of byte array accesses https://github.com/dotnet/runtime/pull/48894
JIT: Non-void ThrowHelpers https://github.com/dotnet/runtime/pull/48589
CSE for floating-point constants https://github.com/dotnet/runtime/pull/44419
Invariant static readonly fields to enable CSE/Hoist-from-loops optimizations https://github.com/dotnet/runtime/pull/44562
Fold more with null checks for const string https://github.com/dotnet/runtime/pull/49930
Zero init elimination: Don’t init tracked temps without GC fields https://github.com/dotnet/runtime/pull/49879

.NET MAUI
Community Contributions

Switch Entry and Editor from EditText to AppCompatEditText
maui-check dotnet CLI tool

Port font to Editor
Implement IsTextPredictionEnabled property on Editor
Implemented LineHeight on Label
Entry ClearButtonVisibility handler
Implement CharacterSpacing property in Entry handler
Port Editor Placeholder text and color properties

Windows Platform Added
https://github.com/dotnet/maui/pull/548
// images coming
With the release of Project Reunion 0.5, .NET MAUI now supports Windows development on .NET 6 Preview 3. Using Visual Studio 16.10 latest public preview, clone the .NET 6 sample here for your first look at .NET MAUI with WinUI 3. When .NET build tooling support is enabled for Project Reunion, we will add Windows to the project template and enable single project features.
Incremental Updates
// control gallery image coming

Continued porting of UI controls and properties from Xamarin.Forms renderers to .NET MAUI handlers:
o	ActivityIndicator
o	Button
o	CheckBox
o	DatePicker
o	Editor
o	Entry
o	Label
o	Picker
o	ProgressBar
o	SearchBar
o	Slider
o	Stepper
o	Switch
o	TimePicker
Expanded font support across handlers
Grid added support for absolute and auto-sizing
Implemented LayoutAlignment for all layouts
Application updates for

simplified HostBuilder integration
new RegisterFormsCompatibility() method enables using App.xaml and auto-registers compatibility renderers
added loading global ResourceDictionary

Changes from Preview 2

MauiApp is replaced with Application
SharedImage and SharedFont are changed to MauiImage and MauiFont

@leecow Since Dotnet 6 preview 3 is already released this issue can be closed.

Preview 3 was released: https://devblogs.microsoft.com/dotnet/announcing-net-6-preview-3/


Fancy CSS Class Names

What is the reason for fancy class names like "_c-notification__countdown_of6y9q"? For me this indicates that it is "private" and shouldn't be used. Is there a better way to alter the appearance?

I agree this makes it impossible to attach CSS and style the components.

Howdy folks 👋
Unfortunately there is no simple answer for this situation. These "fancy class names" are both intentional and unintentional at the same time. Let me explain...
Those class suffixes are as a result of us using ember-css-modules which you can find more information about here. It is both an implementation detail and how the system is explicitly designed in that it is theoretically impossible for ember-cli-notificaitons to break any CSS in your application because each class that we use and declare is entirely unique.
It is potentially overkill since we are already using BEM quite extensively but it's a precaution that we decided on very early in the design of this addon.
There are some specific issues already recorded that are essentially requests to be able to have more control over the style of the notifications (see https://github.com/stonecircle/ember-cli-notifications/issues/76 https://github.com/stonecircle/ember-cli-notifications/issues/115 and https://github.com/stonecircle/ember-cli-notifications/issues/43), but I don't think we will be tackling any one of these issues on their own. There is likely going to be some work around restructuring this addon to allow for a specifically designed way to extend the design, but I'm not 100% sure if it's going to be giving you full control. An example of an addon that has these sorts of restrictions is ember-power-select which explicitely does not allow extending the styles.
I don't think we need to go as far as preventing all extension of the styles, but it's important to remember that we are unlikely to sacrifice ease of use for ease of extension. There are always more Ember ways of extending this addon like overriding the notification-container for example.

Thanks @mansona for the explanation! Nice that you thought about this and we can still use custom styles via a prefix match like div[class^="_c-notification__countdown_"] {}. Without the explanation I thought it was unnecessarily complicated.


Feature of Prompting user while using youtube-dl

It will be great to prompt the user about the commands of youtube-dl while using it.As it contains too much options it is very difficult for user to use this , so it will be great to have such a feature in it.

Out of scope of youtube-dl.


More CCM test fixes

A few more fixes for intermittently failing tests. Nothing extraordinary, most changes consist in raising timeouts or pausing before/after tests. Note that this includes fixes for JAVA-1039 from #575 .
This PR also makes a few changes to allow CCMTestsSupport to be easily subclassed.

Direct access to ccm, cluster and session is replaced with methods (also provides better protection against tests that would reassign these fields);
createTestFixtures has been refactored into onTestContextInitialized, to allow e.g. graph queries to be executed at this step as well as normal queries.

The PR looks big mostly because of 183795f and 2289c5b.

@tolbertam pushed 2 new commits with support for DSE workloads and also a way to fully customize your CCMBridge.Builder :
@CCMConfig(ccmProvider = "createCcmBuilder")
public class MyTest extends CCMTestsSupport {

    public CCMBridge.Builder createCcmBuilder() {
        return CCMBridge.builder().withDSE().withVersion("4.8.3").withWorkload(1, "spark");
    }

}

@adutra that looks perfect!  Both commits solve different issues I was having :+1:


-10.7 V dip before analog output starts

I am using your mcculw library with Python 3.6.6 on Windows 10 64 bit to connect to a USB-3101FS D/A converter.
I am able to generate the waveforms without issue but when looking at them on a scope, I am noticing there is a -10.7V glitch that occurs approximately 20 ms before waveform generation starts. My waveform starts and finishes at 0V corresponding to a DAC value of 32768.
Is there any way to prevent this? That sort of a spike could potentially be dangerous when amplified...

I have attached my code. I had to rename it to .txt for GitHub to allow it.
a_out_scan.txt

It appears the call to AnalogOutputProps(board_num) actually changes some of the settings. I have hard-coded the output of this function.


Sync elemental and ros-installer

Currently elemental and ros-installer are out of sync, specially regarding the environmental variables usage. So far I noted ros-installer aims to make use of:
form pkg/config/read.go

ELEMENTAL_CLOUD_INIT: to define an extra cloud-init config to include within the installation
ELEMENTAL_ISO: to define the URI of an ISO to use as a source for the installation
ELEMENTAL_POWEROFF: set poweroff after install operation
ELEMENTAL_TARGET: to set the target device for the installation
ELEMENTAL_DOCKER_IMAGE: to set a docker source as the OS image for the installation
ELEMENTAL_FORCE_EFI: to set EFI firmware during install
ELEMENTAL_NO_FORMAT: to prevent formatting during installation
ELEMENTAL_DEBUG: to set debug flag in elemental
ELEMENTAL_TTY: to set an extra console in kernel parameters
ELEMENTAL_REGISTRATION_URL: I don't think this relates to elemental in any way.

As a result of rancher-sandbox/elemental#189 these environment variables are not valid anymore. Assuming we need to keep all the variables from the above list they could easily be mapped to:

ELEMENTAL_CLOUD_IINIT -> ELEMENTAL_INSTALL_CLOUD_INIT
ELEMENTAL_TARGET -> ELEMENTAL_INSTALL_TARGET
ELEMENTAL_POWEROFF -> keep it as is
ELEMENTAL_ISO ->  ELEMENTAL_INSTALL_ISO
ELEMENTAL_DOCKER_IMAGE -> ELEMENTAL_INSTALL_SYSTEM (and the value expects a docker:<image-ref>)
ELEMENTAL_FORCE_EFI -> ELEMENTAL_INSTALL_FIRMWARE (and the value expects efi|bios)
ELEMENTAL_NO_FORMAT -> ELEMENTAL_INSTALL_NO_FORMAT
ELEMENTAL_DEBUG -> keep it as is
ELEMENTAL_TTY -> ELEMENTAL_INSTALL_TTY
ELEMENTAL_REGISTRATION_URL -> drop it, I don think it relates to elemental in any way

Note that installation specific data has a the ELEMENTAL_INSTALL_ prefix, I did it this way to prevent clashing variables across commands, but probably this is not a problem.
Just provide and example elemental install supports ELEMENTAL_INSALL_SYSTEM variable and elemental upgrade supports ELEMENTAL_UPGRADE_SYSTEM variable, this way we can define the installation and upgrade system independently.
From an environment variables perspective is is probably reasonable to accept that ELEMENTAL_SYSTEM could be applied for elemental upgrade and elemental install as it is unlikely that both commands are executed in a row within the same environment. If this is acceptable the above list of conversions is really simplified. I vote for this option, but realize this is going to be hard to change afterwards if we experience undesired variables collisions across commands (I don't think this is going to happen though).
Another option could be being slightly inconsistent and use ELEMENTAL_SYSTEM for the installation and ELEMENTAL_UPGRADE_SYSTEM for the upgrade.
Regarding the image reference syntax change we can try to be backward compatible in elemental (like assuming docker type if undefined or some similar criteria) or simply modify ros-installer to prepend the docker: scheme to the image reference uri.
Tasks:

[ ] agree on the supported variables set
[ ] add those variables into elemental maps pkg/constants/constants.go
[ ] code tests for all of them (in cmd/config/config_test.go)
[ ] adapt ros-installer (if needed)

Implementation wise this is a chore, probably testing requires some time, but coding it is should be really straight forward, specially if we agree to minimize the impact on ros-installer.

Blocked until ros-installer finds a final location

the ros-installer final location will be inside the ros-operator repository. Although, we are in the middle of migrating and the ros-installer code isn't there (yet)

We might have also to backport:

https://github.com/rancher-sandbox/ros-installer/pull/1  (already merged)
https://github.com/rancher-sandbox/ros-installer/pull/2 (to merge)

Blocked by https://github.com/rancher/elemental/issues/136

Since ros-installer is not yet merged to elementa-operator and the repository is in archived mode I created a fork with the expectations of the change set required for this card in here https://github.com/davidcassany/ros-installer/pull/1. Note this has not been validated yet.
Note in former ros-installer the config.Install.ContainerImage field expects an image reference, so it can't be mapped directly to ELEMENTAL_INSTALL_SYSTEM which expects a URI (e.g. docker:<imageref>).
@mudler correct me if I am wrong, but I assume the filed config.Install.ContainerImage should remaing being a simple container reference, I don't expect to use channel or dir source types within the context of the elemental-operator, so adding a docker scheme is just cumbersome. If this is the case then the syntax of config.Install.ContainerImage should remaing untouched meaning we need a way to convert the given image reference into something understandable to elemental-cli. I foresee a couple of options:

Turn image registry references into the default source type in elemental-cli, that is, from an elemental-cli PoV the following two configs mean the same:

install:
  system:
    uri: docker:registry.org/my/image:tag

install:
  system:
    uri: registry.org/my/image:tag

I think this is a reasonable option for elemental-cli, this is already the case for the build-iso command. We can extend it to all commands.

Make the unmarshall process of the installer here to prepend a docker: prefix in any value given for config.Install.ContainerImage. So assume there is a conversion to elemental-cli syntax somewhere in the code. I don't like the idea, I find it too obscure.

Now elemental when unmarshaling sources fallsback to treat it as a docker registry image reference if the given value does not match not match any of the known schemes (docker, file, channel or dir) or it fails to parse it as a URI. In addition it also defaults to latest tag if no tag is given.

Since ros-installer is not yet merged to elementa-operator and the repository is in archived mode I created a fork with the expectations of the change set required for this card in here davidcassany/ros-installer#1. Note this has not been validated yet.
Note in former ros-installer the config.Install.ContainerImage field expects an image reference, so it can't be mapped directly to ELEMENTAL_INSTALL_SYSTEM which expects a URI (e.g. docker:<imageref>).
@mudler correct me if I am wrong, but I assume the field config.Install.ContainerImage should remain being a simple container reference, I don't expect to use channel or dir source types within the context of the elemental-operator, so adding a docker scheme is just cumbersome. If this is the case then the syntax of config.Install.ContainerImage should remain untouched meaning we need a way to convert the given image reference into something understandable to elemental-cli. I foresee a couple of options:
1. Turn image registry references into the default source type in `elemental-cli`, that is, from an `elemental-cli` PoV the following two configs mean the same:

install:
  system:
    uri: docker:registry.org/my/image:tag

install:
  system:
    uri: registry.org/my/image:tag

I think this is a reasonable option for elemental-cli, this is already the case for the build-iso command. We can extend it to all commands. Did a PR for that here rancher/elemental-cli#264
2. Make the unmarshall process of the installer here to prepend a `docker:` prefix in any value given for `config.Install.ContainerImage`. So assume there is a conversion to elemental-cli syntax somewhere in the code. I don't like the idea, I find it too obscure.

It is totally safe to assume config.Install.ContainerImage has only container images as input. Alternatively we can also think about extending the install with another field to provide a source field which is mapped 1:1 with elemental, but the usage here is limited to containers, the boot live medium itself, and an ISO file as a source of installation, so it sounds ok to keep it as is too.

The code from https://github.com/davidcassany/ros-installer/pull/1 is already part of elemental-operator and a successful communication between the elemental-cli wrapper and elemental-cli is already functional (despite not all flags and variables have been completely validated, there could be some small mismatch on variables names).
Note is has been managed in the scope of rancher/elemental-operator#7. Closing since all the essential logic related to this specific task is already in place.


4.5.0-0.okd-2020-06-18-025306 bootstrap never completes

Describe the bug
Bootstrap coming up but master never join. bootstrap phase never completes.
Version
UPI / KVM 0.12.0
4.5.0-0.okd-2020-06-18-025306
32.20200615.2.0
How reproducible
100%
Log bundle
journal.txt

No log bundle provided

This is all I have and was requested by Josef...

@disposab1e
I will be building a cluster this weekend with a 4.5 nightly build and FCOS 32.  Since our setups are similar, I'll let you know if I have any issues.  There might be a nuance that is tripping you.

wasted time for @vrutkovs so better closed

@disposab1e
No offense, but he asked you for logfiles a few times and you didn't give them to him. Apparently, you guys talked at cross purposes or had a bad day :-)
Calm down and help each other, please.

[admin@okd4-services ~]$ export KUBECONFIG=install_dir/auth/kubeconfig
[admin@okd4-services ~]$ oc adm must-gather
[must-gather      ] OUT the server could not find the requested resource (get imagestreams.image.openshift.io must-gather)
[must-gather      ] OUT
[must-gather      ] OUT Using must-gather plugin-in image: quay.io/openshift/origin-must-gather:latest
[must-gather      ] OUT namespace/openshift-must-gather-82dtp created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-t2tlg created
[must-gather      ] OUT pod for plug-in image quay.io/openshift/origin-must-gather:latest created
[must-gather-67ttf] OUT gather did not start: timed out waiting for the condition
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-t2tlg deleted
[must-gather      ] OUT namespace/openshift-must-gather-82dtp deleted
error: gather did not start for pod must-gather-67ttf: timed out waiting for the condition
[admin@okd4-services ~]$

[admin@okd4-services ~]$ oc get machineconfig
NAME                            GENERATEDBYCONTROLLER   IGNITIONVERSION   AGE
99-master-disable-mitigations                           3.1.0             69m
99-master-ssh                                           3.1.0             69m
99-worker-disable-mitigations                           3.1.0             69m
99-worker-ssh                                           3.1.0             69m
[admin@okd4-services ~]$ oc get machineconfigpools
NAME     CONFIG   UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master                                                                                57m
worker                                                                                57m
[admin@okd4-services ~]$


Handle cancel button clicks in forms

Summary

Propagate cancel event from forms up to App
Add an "unsaved changes" warning component which keeps the form open if the user clicks "Stay"

I'm not sure if my approach is the best, especially in toggleSidebar() which is getting a bit messy. Any suggestions?
Test Plan
Run in browser
Related Issues
Which issues does this PR resolve/work on?
Closes #114

LGTM, I agree that there is probably a better design to toggleSidebar, but I cannot think of one... Only suggestion is to consider using an Enum as per my comment, and maybe make toggleSidebar take Enum || null

this is definitely an improvement
another option (which would take a major refactor) is to wrap the forms in a common Form component which provides save and cancel functionality
now that I'm writing this, I'm realizing that functionality would fit inside of ActionButtons quite nicely


[BUG] - Policy limit exceeded

Currently occurring in the nightly deploy env
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]     deploy_env(env_name=env_name, manifest_dir=manifest_dir)
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]   File "/root/.venv/lib/python3.7/site-packages/aws_codeseeder/codeseeder.py", line 199, in wrapper
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]     return func(*args, **kwargs)
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]   File "/root/.venv/lib/python3.7/site-packages/aws_orbit/remote_files/deploy.py", line 371, in deploy_env
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]     changeset=changeset,
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]   File "/root/.venv/lib/python3.7/site-packages/aws_orbit/remote_files/eksctl.py", line 494, in deploy_env
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]     f"{context.eks_oidc_provider}:sub": f"system:serviceaccount:orbit-system:orbit-{context.name}-admin"
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]   File "/root/.venv/lib/python3.7/site-packages/aws_orbit/services/iam.py", line 110, in add_assume_role_statement
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]     iam_client.update_assume_role_policy(RoleName=role_name, PolicyDocument=policy_body)
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]   File "/root/.venv/lib/python3.7/site-packages/botocore/client.py", line 388, in _api_call
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]     return self._make_api_call(operation_name, kwargs)
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]   File "/root/.venv/lib/python3.7/site-packages/botocore/client.py", line 708, in _make_api_call
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]     raise error_class(parsed_response, operation_name)
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD] botocore.errorfactory.LimitExceededException: An error occurred (LimitExceeded) when calling the UpdateAssumeRolePolicy operation: Cannot exceed quota for ACLSizePerRole: 2048```

When I look at this role (orbit-nightly-us-west-2-admin) I see that the trust policy has 4 entries for OIDC to have Assume RoleWithWebIdentity.  I believe there should only be one....for the active cluster.

Looking at the code, the command orbit toolkit creates the IAM admin role.  Once created, the orbit deploy env updates the role's trust policy with OIDC of the cluster.
When we delete the env (the cluster) and create a new one, the trust policy is updated with the new cluster OIDC, but the references to the deleted cluster are NOT removed from the policy.
: recycling the env multiple times fills up the space in the trust policy


Docs request: headline vs bio in languages.en.toml

The exampleSite as of v2.38.1 has an identical bio/headline field in languages.en.toml: https://github.com/nunocoracao/blowfish/blob/main/exampleSite/config/_default/languages.en.toml#L19
The Getting Started docs don't currently describe the difference.
What is the difference, and can we add this to the Getting Started docs?

I discovered https://blowfish.page/docs/configuration/#author and it answers my question.


fix(ui): avoid using prefix for replayed logs

Description
Currently on a cache hit the replayed logs in the UI contain the task prefix which is unnecessary in the UI. This brings the cache hit/miss output to be the same in the UI (modulo cache hit/miss lines).
Testing Instructions
Before:

After:

(Also making sure non-UI behavior is preserved)

Closes TURBO-2684

Nice! can the cache hit line also remove the prefix?

We can, but I do to some degree like keeping the prefix for lines that turbo is producing to differentiate between output from turbo and output from the actual task.


[release/1.2] Prepare v1.2.13 release

depends on:

[ ] https://github.com/containerd/containerd/pull/4025 [release/1.2 backport] Fix container pid race condition
[ ] https://github.com/containerd/containerd/pull/4001 [release/1.2] bump cgroups dependency to address blkio issue
[ ] https://github.com/containerd/containerd/pull/4028 [release/1.2] backport: Set octet-stream content-type on put request
[ ] https://github.com/containerd/containerd/pull/4015 [release/1.2 backport] Pin to libseccomp 2.3.3
[ ] https://github.com/containerd/containerd/pull/4031 [release/1.2 backport] Update Golang 1.12.17

The thirteenth patch release for containerd 1.2 fixes a regression introduced
in v1.2.12 that caused container/shim to hang on single core machines, fixes an
issue with blkio, and updates the Golang runtime to 1.12.17.
Notable Updates

Fix container pid race condition containerd#4025
Update containerd/cgroups dependency to address blkio issue containerd#4001
Set octet-stream content-type on PUT request containerd#4028
Pin to libseccomp 2.3.3 to preserve compatibility with hosts that do not have libseccomp 2.4 or up installed containerd#4015
Update Golang runtime to 1.12.17, which includes a fix to the runtime contrainerd#4031

Generated release notes (incomplete, because dependent PR's haven't been merged yet);
containerd 1.2.13
Welcome to the v1.2.13 release of containerd!
The thirteenth patch release for containerd 1.2 fixes a regression introduced
in v1.2.12 that caused container/shim to hang on single core machines, fixes an
issue with blkio, and updates the Golang runtime to 1.12.17.
Notable Updates

Fix container pid race condition containerd#4025
Update containerd/cgroups dependency to address blkio issue containerd#4001
Set octet-stream content-type on PUT request containerd#4028
Pin to libseccomp 2.3.3 to preserve compatibility with hosts that do not have libseccomp 2.4 or up installed containerd#4015
Update Golang runtime to 1.12.17, which includes a fix to the runtime contrainerd#4031

Please try out the release binaries and report any issues at
https://github.com/containerd/containerd/issues.
Contributors

Sebastiaan van Stijn

Changes

9a517f681 Prepare v1.2.13 release

Dependency Changes
Previous release can be found at v1.2.12

Codecov Report

Merging #4033 into release/1.2 will increase coverage by 3.67%.
The diff coverage is n/a.

@@               Coverage Diff               @@
##           release/1.2    #4033      +/-   ##
===============================================
+ Coverage        44.19%   47.87%   +3.67%
===============================================
  Files              100       91       -9
  Lines            10847     8483    -2364
===============================================
- Hits              4794     4061     -733
+ Misses            5313     3688    -1625
+ Partials           740      734       -6

Flag
Coverage Δ

#linux
47.87% <ø> (ø)
:arrow_up:

#windows
?

Impacted Files
Coverage Δ

remotes/docker/auth.go
63.82% <0%> (-3.97%)
:arrow_down:

remotes/docker/status.go
21.42% <0%> (-3.58%)
:arrow_down:

platforms/cpuinfo.go
3.77% <0%> (-0.85%)
:arrow_down:

log/context.go
36.84% <0%> (-0.66%)
:arrow_down:

namespaces/context.go
55% <0%> (-0.56%)
:arrow_down:

errdefs/grpc.go
74.6% <0%> (-0.4%)
:arrow_down:

remotes/docker/pusher.go
0% <0%> (ø)
:arrow_up:

content/local/locks.go
100% <0%> (ø)
:arrow_up:

labels/validate.go
100% <0%> (ø)
:arrow_up:

platforms/defaults.go
100% <0%> (ø)
:arrow_up:

... and 57 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 35bd7a5...9a517f6. Read the comment docs.

Rebased, updated mailmap, and updated the generated release notes above https://github.com/containerd/containerd/pull/4033#issuecomment-586925285
@dmcgowan @estesp @mikebrow @fuweid @Random-Liu this should be ready for review

Found a couple really minor typos, but we might as well fix

Thanks @estesp - fixed the nits 👍
I noticed that the last part of the generated Markdown is a bit confusing:

Dependency Changes
Previous release can be found at v1.2.12

github.com/containerd/cgroups<PHONE_NUMBER>f9 -> c4b9ac5c7601

Wondering if "previous release can be found at..." should be after the dependency change

Opened https://github.com/containerd/release-tool/pull/9 for the above


Research NPM package

Research process for Creating and  publishing NPM package

https://drive.google.com/file/d/1b65YDSmG1AnEiibdUrQ41x7aE6-cWh1a/view?usp=sharing


A "Use of Uninitialized Variable" bug occurs in file match.c

I hope this message finds you well. I have identified a potential "Use of Uninitialized Variable" bug in the project, and I would like to bring it to your attention in case it may be helpful. This bug might lead to unexpected behavior due to the usage of uninitialized variables, potentially affecting program stability. Can you please help me check it? Thank you for your effort and patience!
Bug Description:
The bug is located in the file src/match.c, involving the functions match_positions and match_row. The specific code flow is as follows:

In the match_positions function, on line 188, two pointers, last_D and last_M, are declared but not initialized.
The code on line 195, within match_positions, directly uses these uninitialized pointers as parameters and passes them into the match_row function.
Within the match_row function, on lines 90 and 92, data is accessed through last_D and last_M, leading to a "Use of Uninitialized Variable" bug.

Thanks! Fixed in b2389a67fcd4102370ec600418be0743944be3b2
I don't think this could ever be dereferenced in its uninitialized form, since it's only uninitialized for i=0, but this is good to fix.


[FEATURE] Durability

So basically durability is of course a thing with most items and for some, it is laughably low such as a staff breaking after 8 hits which doesn't help it's already terrible damage.
A simple request of maybe adding some way either to lower the amount of durability lost when using items or even prevent durability loss entirely
Anyway, thank you for your work on the mod it is greatly appreciated!

Actually coming in next update, and I've been making it for a while now
Hoping everything functions correctly and right now I'm making sure so that if invalid values are inserted that it still works :)

Ah alright, honestly good job for thinking ahead, you're doing gods work!

Added in 68dbb630cb0d56139a71d392d87929b244fc893b


[FEATURE] batch of images

Hello
Thanks for the 5 insert images tabs available,
What about being able to insert a path to a folder full of images, and being able to generate prompts for all of them?
Can this feature be implemented? thanks

Hi.
Multiple images are processing as one set to create combined single prompt. For example a image with a cat and an image a woman will create some mix in result. You can read details here.
In fact, as I see it works more or less normally for two-three images max. So no any sense to use folder with images as a source.
Batch processing will require a lot of redesign, probably later will implement this but not so soon.

Ah I was not aware, that is actually cool. Ok then put it in the "to do list later" list if you wish.
SInce I have you here, could you tell me how to know which "model" is best to use? What are the differences? What is the mindset /workglow when using pez here, what can we wish to achieve and how to achieve these different things, if there aremany methods/goals?

Please check neighbor thread about models. That is all what I know about them. Just playing with different models too and getting different results, but have no "right recipe".

Added folder batch processing in txt2img script section.


Display only number of anonymously connected persons, instead of their pseudo

Proposal

The first commit can be removed as it is already treat in PR #1601

The first commit can be removed as it is already treat in PR #1601

I think we should keep it since it does not fix the  same things even it is less likely to occured with #1601 is  merged but still is cleaner

You should also modify the changelog with this modification.


Restart a Javascript server.

Before there was an option to restart the Typescript server but this command also restarted the Javascript Server. I included this change as it was confusing before I released what the command did. This can help beginners whom do not realise that they can restart their Javascript server with the Restart TS Server Command.

Thanks but there is no JavaScript server. The same TS server handles both Javascript and TypeScript files
I also don't think discoverability is relevant here: this command should never need to be run by users and is primarily included for development. If you find yourself running it often for some reason, please file an issue