[SPARK-8425][CORE] New Blacklist Mechanism

What changes were proposed in this pull request?
This adds a blacklisting mechanism for Spark to handle bad executors and nodes, eg. for when a disk fails.  Previously there was an undocumented blacklisting mechanism, but it was very limited.  This supports blacklisting individual tasks on an executor, executors and nodes for an entire stage, and executors and nodes across an entire spark application.  Full details are in a design doc attached to the jira.
How was this patch tested?
Added unit tests, ran them via Jenkins, also ran a handful of them in a loop to check for flakiness.
The added tests include:

verifying BlacklistTracker works correctly
verifying TaskSchedulerImpl interacts with BlacklistTracker correctly (via a mock BlacklistTracker)
an integration test for the entire scheduler with blacklisting in a few different scenarios

TODO

[ ] More testing on a real cluster
[x] Fix & test handling of shuffle-fetch failures
[x] Fix & test single task failure on a bad node should not trigger job failure
[ ] Documentation

Test build #61873 has started for PR 14079 at commit 5bfe941.

Test build #61873 has finished for PR 14079 at commit 5bfe941.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61873/
Test PASSed.

I took another look at having BlacklistTracker just be an option, rather than having a NoopBlacklist.  After some other cleanup, I decided it made more sense to go back to the option, but its in one commit so easy to go either way https://github.com/apache/spark/pull/14079/commits/a34e9aeb695958c749d306595d1adebe0207fdf9

Test build #61931 has started for PR 14079 at commit cf58374.

Test build #61931 has finished for PR 14079 at commit cf58374.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61931/
Test FAILed.

Test build #62119 has started for PR 14079 at commit 487eb66.

Test build #62119 has finished for PR 14079 at commit 487eb66.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62119/
Test FAILed.

discussed this offline with @vanzin, realized that actually with the latest design, it doesn't make sense to have so many maps inside one BlacklistTracker -- a lot of info is now intended to only be tied to one TaskSet, so I should push that info back into the TaskSetManager (where the old blacklist lived).  I'll do that refactoring, which will hopefully clean things up.  No behavior changes planned, though.

Test build #62134 has started for PR 14079 at commit c22aaad.

Test build #62134 has finished for PR 14079 at commit c22aaad.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62134/
Test PASSed.

Test build #62245 has started for PR 14079 at commit 338db65.

Test build #62253 has started for PR 14079 at commit fa3e34a.

Test build #62245 has finished for PR 14079 at commit 338db65.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62245/
Test PASSed.

Merged build finished. Test PASSed.

Test build #62253 has finished for PR 14079 at commit fa3e34a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62253/
Test PASSed.

Just some minor stuff, I'll let people more familiar with the scheduler comment further.

Test build #62351 has started for PR 14079 at commit 7aff08a.

Test build #62352 has started for PR 14079 at commit 351a9a7.

Test build #62351 has finished for PR 14079 at commit 7aff08a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62351/
Test PASSed.

Test build #62352 has finished for PR 14079 at commit 351a9a7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62352/
Test PASSed.

this can probably be a separate jira but it would be really nice to show blacklisted on executors page when it happens.  i guess it can get a bit complicated with timeout from blacklist and between stages.  Without that all I see is an executor that doesn't have any tasks and I start to wonder why.

thanks for the feedback @tgravescs .  Thanks for pointing out the issue after executors have died, I updated the behavior and added a test case (I'll update the design doc shortly too).  I think I addressed the other comments, including finally adding the docs.

Test build #62621 has started for PR 14079 at commit dbf904e.

Test build #62621 has finished for PR 14079 at commit dbf904e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62621/
Test FAILed.

Test build #62630 has started for PR 14079 at commit f0de0db.

Test build #62630 has finished for PR 14079 at commit f0de0db.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62630/
Test PASSed.

Test build #62734 has started for PR 14079 at commit 8a12adf.

I updated it to take into account preemption, and also noticed that TaskKilled and TaskCommitDenied weren't handled either, so I added those and a test.  I also tried to clean up some of the handling using countTowardsTaskFailures -- that wasn't on TaskKilled earlier, the code just special-cased it elsewhere (by checking state = KILLED, which is the same thing), I thought it would be better to unify it.

Test build #62734 has finished for PR 14079 at commit 8a12adf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62734/
Test FAILed.

Jenkins, retest this please

Test build #62737 has started for PR 14079 at commit 8a12adf.

Test build #62737 has finished for PR 14079 at commit 8a12adf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62737/
Test PASSed.

Test build #62894 has started for PR 14079 at commit c9e3662.

Test build #62894 has finished for PR 14079 at commit c9e3662.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62894/
Test PASSed.

I left a bunch of cosmetic comments to try to make this code as easy as possible to read / reason about. Taking a break for a meeting but will finish reviewing this later this afternoon (letting you know so you can batch work on this until I'm done if you want). Sorry this took so long for me to get to!

OK done with a pass over this -- overall this looks good.  Sorry for the huge number of naming suggestions.  This behavior is subtle/complex and I'm in favor of spending some effort to make it as understandable as possible to future folks who modify this code.
A few more comments:
The design doc mentions some invariants that must be true of the parameters (e.g., that some values must be lower than others). Can you enforce those invariants in the creation of the BlacklistManager?
It would be helpful to add a comment somewhere describing the division of functionality.  Maybe a good place would be in the BlacklistTracker docstring? Something that explains that the TaskSetManager is responsible for handling blacklisting at the task set and task level, while the TaskSchedulerImpl is responsible for handling application-level blacklists?
I added a few high level comments / concerns to the design doc, also.

Test build #63989 has started for PR 14079 at commit f0428b4.

Test build #63989 has finished for PR 14079 at commit f0428b4.

This patch fails to build.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63989/
Test FAILed.

Test build #63990 has started for PR 14079 at commit a5fbce7.

Test build #63990 has finished for PR 14079 at commit a5fbce7.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63990/
Test FAILed.

Test build #63992 has started for PR 14079 at commit b582d8e.

@kayousterhout @tgravescs Sorry for the long-delay on my end, I have finally updated this.  I think I have addressed all the naming / commenting issues -- I'm going to do a bit more testing on it and then come back for a fresh pass to see if there are more things.
Some important changes in this revisions:

timeouts of individual task failures, in application blacklisting only.  This is to avoid blacklisting an executor in a long-lived app when you've got a few random failures that are very far apart in time.  But everything still has its own timeout -- here's the comment I've put in the code to explain that:

      // Note that we do *not* remove executors and nodes from
      // the blacklist as we expire individual task failures -- each have their own timeout.  Eg.,
      // suppose:
      // * timeout = 10, maxFailuresPerExec = 2
      // * Task 1 fails on exec 1 at time 0
      // * Task 2 fails on exec 1 at time 5
      // -->  exec 1 is blacklisted from time 5 - 15.
      // This is to simplify the implementation, as well as keep the behavior easier to understand
      // for the end user.

config name changes.  Also, docs mark everything as "Experimental" except for spark.blacklist.enabled.  The idea is to convey that blacklisting will continue to be exported, but the specific knobs may go through another revision.
bug fix -- in some more testing, I discovered that I was blacklisting nodes whenever the total number of blacklisted executors was over the threshold, rather than checking the number of blacklisted executors on that node.  Fixed, cleaned up code a bit, added a test.

Test build #63993 has started for PR 14079 at commit cec36c9.

Test build #63992 has finished for PR 14079 at commit b582d8e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63992/
Test FAILed.

Test build #63993 has finished for PR 14079 at commit cec36c9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63993/
Test FAILed.

Merged build finished. Test FAILed.

Test build #64008 has started for PR 14079 at commit 290b315.

Test build #64008 has finished for PR 14079 at commit 290b315.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64008/
Test FAILed.

Test build #64021 has started for PR 14079 at commit 8c58ad9.

Test build #64021 has finished for PR 14079 at commit 8c58ad9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64021/
Test PASSed.

Test build #64215 has started for PR 14079 at commit fc45f5b.

also just realized that I forgot about @kayousterhout 's comment to add in checks on the invariants for the confs -- I've added that now as well.

Test build #64220 has started for PR 14079 at commit f8b1bff.

Test build #64215 has finished for PR 14079 at commit fc45f5b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64215/
Test PASSed.

Test build #64220 has finished for PR 14079 at commit f8b1bff.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64220/
Test FAILed.

Test build #64242 has started for PR 14079 at commit e56bb90.

Test build #64242 has finished for PR 14079 at commit e56bb90.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64242/
Test PASSed.

one interesting thing I found during testing was with spark-shell and dynamic allocation, you can hit the blacklist and then job failure very quickly because dynamic allocation takes to long (cannot run anywhere due to node and executor blacklist).  Hopefully something we can improve upon later when we tie it into the backends more.

@tgravescs not totally sure I know what you mean -- are you saying that with blacklisting and dynamic allocation, you might have a situation with only one executor, which then gets blacklisted, and the app fails rather than creating another executor?  That sounds related to https://issues.apache.org/jira/browse/SPARK-15815.  I agree this isn't ideal, but I can live with it, as long as this only happens when there really is something getting blacklisted.  You haven't encountered scenarios where the blacklist causes an app fail with perfectly good executors, have you?

correct, just not ideal right now but can live with it. I haven't seen it cause failure without task failures.

Test build #64498 has started for PR 14079 at commit cc3b968.

Test build #64499 has started for PR 14079 at commit 5fdfe49.

Test build #64499 has finished for PR 14079 at commit 5fdfe49.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64499/
Test FAILed.

Test build #64498 has finished for PR 14079 at commit cc3b968.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64498/
Test PASSed.

Test build #64721 has started for PR 14079 at commit 1297788.

Test build #64723 has started for PR 14079 at commit c78964f.

Test build #64721 has finished for PR 14079 at commit 1297788.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64721/
Test PASSed.

Test build #64723 has finished for PR 14079 at commit c78964f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64723/
Test PASSed.

Merged build finished. Test PASSed.

I got through everything except the tests.  Can you fix the merge conflicts (which I suspect relate to Josh's change)?
One more meta comment: there's a ton of blacklisting logic, especially in the TSM.  Do you think it would make sense to have a second TSM-specific blacklist helper class that encapsulates all of this logic, and have the TSM call methods from that class? It seems like the blacklisting data structures separate nicely, and it might help the TSM to be more readable.

Test build #65783 has started for PR 14079 at commit 18ef5c6.

Test build #65783 has finished for PR 14079 at commit 18ef5c6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65783/
Test PASSed.

@kayousterhout @tgravescs sorry for the long delay from me.  I've addressed most the feedback.  But I haven't looked at separating out the blacklist logic into a separate class inside TaskSetManager -- I will look at that and ping you again when I make that update, just thought it might be worth looking at now.

Test build #65919 has started for PR 14079 at commit 0c3ceba.

Test build #65922 has started for PR 14079 at commit 2381b25.

Test build #65919 has finished for PR 14079 at commit 0c3ceba.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65919/
Test PASSed.

Merged build finished. Test PASSed.

Test build #65922 has finished for PR 14079 at commit 2381b25.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ExecutorFailuresInTaskSet(val node: String)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65922/
Test PASSed.

Test build #65939 has started for PR 14079 at commit 278fff3.

for anyone else who might be watching, I've carved out part of this change here: https://github.com/apache/spark/pull/15249

Test build #65939 has finished for PR 14079 at commit 278fff3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65939/
Test PASSed.

Test build #67205 has started for PR 14079 at commit 21907a5.

Test build #67207 has started for PR 14079 at commit 0c57d9d.

Test build #67205 has finished for PR 14079 at commit 21907a5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67205/
Test PASSed.

@kayousterhout I've updated this now to include changes made on SPARK-17675.  Though, with another pass, I found an issue w/ standalone mode -- I'll make an inline comment about it, not sure what the right solution is.

Test build #67207 has finished for PR 14079 at commit 0c57d9d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67207/
Test PASSed.

Merged build finished. Test PASSed.

Test build #67271 has started for PR 14079 at commit 162cb0d.

Test build #67271 has finished for PR 14079 at commit 162cb0d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67271/
Test FAILed.

Merged build finished. Test FAILed.

Jenkins, retest this please

Test build #67286 has started for PR 14079 at commit 162cb0d.

Test build #67286 has finished for PR 14079 at commit 162cb0d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67286/
Test FAILed.

Test build #67317 has started for PR 14079 at commit 6b3babc.

Test build #67317 has finished for PR 14079 at commit 6b3babc.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class YarnSchedulerBackendSuite extends SparkFunSuite with MockitoSugar with LocalSparkContext

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67317/
Test PASSed.

Can you update the PR description and title? (to eliminate the per-task stuff / make it obvious what's new here)

Test build #67620 has started for PR 14079 at commit cdd9f33.

Test build #67623 has started for PR 14079 at commit 255e9b6.

Test build #67623 has finished for PR 14079 at commit 255e9b6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #67620 has finished for PR 14079 at commit cdd9f33.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67623/
Test FAILed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67620/
Test PASSed.

Merged build finished. Test PASSed.

Test build #68694 has started for PR 14079 at commit cc3faaf.

Test build #68694 has finished for PR 14079 at commit cc3faaf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68694/
Test FAILed.

Test build #68790 has started for PR 14079 at commit 5d8500a.

the last test failure was from a rare and pre-existing race in the SchedulerIntegration asserts.  I couldn't recreate it on my laptop in 1000 runs, but in any case I handed it here: https://github.com/apache/spark/pull/14079/commits/5d8500a948ee7a2df5b59bef71bc0a52fc4f04de.
I actually wonder if this is desired behavior -- is it OK for the DAGScheduler to mark a stage as completed before the TaskScheduler does?  Though it is unusual and surprising, from a look through the code I couldn't see any potential harm, so I left the possibility there and just handled it in the test.

Test build #68790 has finished for PR 14079 at commit 5d8500a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68790/
Test PASSed.

Merged build finished. Test PASSed.

Test build #69092 has started for PR 14079 at commit a8d922f.

Test build #69092 has finished for PR 14079 at commit a8d922f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69092/
Test FAILed.

Merged build finished. Test FAILed.

Test build #69263 has started for PR 14079 at commit 72036f4.

Test build #69263 has finished for PR 14079 at commit 72036f4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69263/
Test PASSed.

Test build #69417 has started for PR 14079 at commit fd57d86.

Test build #69428 has started for PR 14079 at commit 35978e2.

Test build #69417 has finished for PR 14079 at commit fd57d86.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69417/
Test FAILed.

Jenkins, retest this please

Test build #69433 has started for PR 14079 at commit 35978e2.

Test build #69428 has finished for PR 14079 at commit 35978e2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69428/
Test PASSed.

Merged build finished. Test PASSed.

Test build #69433 has finished for PR 14079 at commit 35978e2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69433/
Test FAILed.

Jenkins, retest this please

Test build #69482 has started for PR 14079 at commit 35978e2.

Test build #69482 has finished for PR 14079 at commit 35978e2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69482/
Test PASSed.

ping @kayousterhout.  I've merged in the other changes so this is up to date now.
I also did another pass and did a bit of minor cleanup and commenting, I think in line with stuff we've discussed elsewhere.  I hope this is in pretty good shape now.

@squito finally looking at this now -- sorry for the delay.  Do you have time to fix the YARN conflicts?

thanks for the review @kayousterhout.  I also added a testcase to BlacklistTrackerSuite, "task failure timeout works as expected for long-running tasksets" to cover your point about the long running tasksets.

Test build #70102 has started for PR 14079 at commit c422dd4.

Test build #70102 has finished for PR 14079 at commit c422dd4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70102/
Test PASSed.

Test build #70120 has started for PR 14079 at commit c95462f.

Test build #70120 has finished for PR 14079 at commit c95462f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70120/
Test FAILed.

Jenkins, retest this please

Test build #70122 has started for PR 14079 at commit c95462f.

Test build #70122 has finished for PR 14079 at commit c95462f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70122/
Test PASSed.

LGTM!!!!! 🎉 🎉 🎉 🎉 🎉
Nice work on this -- this will be awesome to have in.

Great!
I've been working on some additional changes on top of this: UI representation for blacklisted executors (SPARK-16654), and implicit killing of blacklisted executors (SPARK-16554). I'll be sending pull requests for those soon after this is merged.

Test build #70194 has started for PR 14079 at commit f249b00.

Test build #70194 has finished for PR 14079 at commit f249b00.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class EventTimeStats(var max: Long, var min: Long, var sum: Long, var count: Long)
class EventTimeStatsAccum(protected var currentStats: EventTimeStats = EventTimeStats.zero)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70194/
Test PASSed.

thanks @kayousterhout !  appreciate all the time you've spent helping out on this issue.
merged to master

@squito Scala 2.10 is broken: https://amplab.cs.berkeley.edu/jenkins/view/Spark QA Compile/job/spark-master-compile-sbt-scala-2.10/3394/console Could you submit a PR to fix it? Thanks!

oops, thanks for letting me know @zsxwing , I just submitted https://github.com/apache/spark/pull/16298


Feature: Hide Shortcut

When I right click and dictate something, I don't always want to see the popup. It would be nice to either be able to disable this auto popup completely, or a quick shortcut to click the "done" button

I'm not completely sure I understand.  You want to hide the Dictater interface when dictating text?  In your mind, how would you open the interface once it was closed?

I'm not exactly sure I guess :) , I just find myself closing out the little window frequently when all I really want is to hear the spoken text and move on.

I'm still thinking about the best way to do this.
Dictater doesn't put an icon in your dock so hiding the window will just permanently keep Dictater in memory without any UI.
If I could lower the opacity of the window when it wasn't active would that work as a solution?

I don't think changing the opacity is any different than the current implementation. Not saying the current form is bad, just that this is how I use it. Maybe I'll just grow to ignore it and close it later :)

wouldn´t it just be esasier to use osx speech feature rather than Dictater in this case? After all, what you're proposing is just to disable the features Dictater offers...

You are correct. I'm on El Capitan now and simply use the context menu option now. Happy to close this issue (feature?).


Closing the browser after several hours of use causes the entire operating system to freeze

System Details

OS: Debian 12 6.1.0-13-amd64
DE: KDE Plasma 5.27.5 (X11)
Thorium: 117.0.5938.157
CPU: 11th Gen Intel Core i7-1165G7 (8) @ 4.700GHz
RAM: 2x16 GB DDR4 3200 MHz

Problem
Closing the entire browser window after several hours of use causes the entire operating system to freeze for a period of several seconds to several minutes, I can only move the mouse cursor. This does not happen if the browser is only open for a few minutes.
Additional Notes
This doesn't happen with Chromium & Google Chrome.

Unable to reproduce the issue you are experiencing, keep this issue open for more information

I've had this occur after running fragmented mp4 streams using MSE for a while.  I'm using Ubuntu 23.10 on an HP OMEN laptop with VAAPI  using the Intel GPU. This is not a problem with MS Edge, Chromium and Brave which all use the GPU.


Make night mode for the website

hi! could i take a stab at this?

yes sure..but make sure in night mode layout css should not get changes

please fetch development1.0 with it ,it will be easy to create night mode..plus please make sure for conflicts thank you

or please try to make new css file..

@ravisaxena23 if no one is this can I work on this?

Yes sure..please make sure to work on development1.0 branch

@amreshk005  i checked your pull request you have done a great work. but we are looking for css to one the person tap on button it should change the css to you have to write css to so i have to revert the pull request!! if it is a long task you can create css for basic element such as navbar and footer

@tttson you are doing it brother just need a confirmation!!

@tttson will you help us out ma'am


Unable to get serverless working on aks

By default you don't have permission for my config map even though my kubectl config is properly configured.
serverless deploy -v

/home/posix4e/src/serverlesstest/node_modules/serverless-kubeless/lib/config.js:56
      return JSON.parse(this.configMag.data[key]);
                                           ^
TypeError: Cannot read property 'runtime-images' of undefined
    at Config.get (/home/posix4e/src/serverlesstest/node_modules/serverless-kubeless/lib/config.js:56:44)
    at Object.getRuntimeDepfile (/home/posix4e/src/serverlesstest/node_modules/serverless-kubeless/lib/helpers.js:182:34)
    at ReadStream.s.on (/home/posix4e/src/serverlesstest/node_modules/serverless-kubeless/deploy/kubelessDeploy.js:103:39)
    at ReadStream.emit (events.js:187:15)
    at ReadStream.EventEmitter.emit (domain.js:442:20)
    at endReadableNT (_stream_readable.js:1092:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)

When I run kubectl proxy and hard code the CM to localhost I get
-----------------------------------

Error: Found errors while processing the given functions:
Error: Unable to update the function hello. Received:
  Code: 403
  Message: functions.kubeless.io "hello" is forbidden: User "clusterUser" cannot patch functions.kubeless.io in the namespace "default"
    at Object.checkFinished (/home/posix4e/src/serverlesstest/node_modules/serverless-kubeless/lib/helpers.js:209:14)
    at deployFunction.catch.then (/home/posix4e/src/serverlesstest/node_modules/serverless-kubeless/lib/deploy.js:479:21)
From previous event:
    at PluginManager.invoke (/home/posix4e/.nvm/versions/node/v10.10.0/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.run (/home/posix4e/.nvm/versions/node/v10.10.0/lib/node_modules/serverless/lib/classes/PluginManager.js:421:17)
    at variables.populateService.then.then (/home/posix4e/.nvm/versions/node/v10.10.0/lib/node_modules/serverless/lib/Serverless.js:157:33)
    at runCallback (timers.js:694:18)
    at tryOnImmediate (timers.js:665:5)
    at processImmediate (timers.js:647:5)
    at process.topLevelDomainCallback (domain.js:121:23)
From previous event:
    at Serverless.run (/home/posix4e/.nvm/versions/node/v10.10.0/lib/node_modules/serverless/lib/Serverless.js:144:8)
    at serverless.init.then (/home/posix4e/.nvm/versions/node/v10.10.0/lib/node_modules/serverless/bin/serverless:43:50)

This is the same error than https://github.com/serverless/serverless-kubeless/issues/149#issuecomment-416654147. It's not possible yet to use this plugin with EKS due to the authentication method for that platform. How are you authenticating against you AKS plugin? Do you have RBAC permissions to get the Kubeless configmap in the kubeless namespace?
If this is an authentication issue you can still try to deploy your functions with the kubeless binary: https://github.com/kubeless/kubeless/releases

@andresmgot I believe it's caused by a different issue
kubectl get cm -n kubeless kubeless-config -o yaml | wc -l
180
Seems to work just fine

Yes, the problem is that the javascript library cannot use the exec auth plugin, so that may be the reason.

this was kinda addressed in https://github.com/serverless/serverless-kubeless/pull/189
it works with aws-iam-authenticator, created a PR
https://github.com/serverless/serverless-kubeless/pull/192


Allow concatenating numbers to string

It is impossible right now to construct a string from various parts because the addition operator only works on strings and we have no way to convert numbers to strings. We should add a mechanism to do this, probably by overloading the addition operator.
This is necessary to, for example, easily center computed strings on the screen.

We now have the STR function, which allows doing this and matches the behavior of QuickBasic and GW-BASIC.


fix highlighting

sync the latest highlight.js from lean4.

There is a conflict. Could you try to produce cleaner commit histories, preferably without merge commits?

Can you just use this squash merge feature?


1.6.640 compatibility

Trying implement backward compatibility with 1.6.640
testing on 640 still in progress, may be I identify more problems

some random CTDs, looks is race condition, trying debug.
for now - https://github.com/AndbGame/Acheron/commit/78363b06eb5c6ca4e782af8796e6dc01920f15ad - looks pretty stable, not have CTD, but need more tests

this solution definitely works - for fix PC control after defeat on 1.6.640, (regarding CTD - it is another issue, in progress)


[Docs]: How to resolve postcss imports

hey guys, not seeing any docs or examples on how to resolve postcss imports (e.g. tailwind).
with webpack, i would add something like the below to my main.js module exports to accomplish this. what is the approach to use with this plugin?
  webpackFinal: async (config) => {
    config.module.rules.push({
      test: /\.css$/,
      use: [
        {
          loader: 'postcss-loader',
          options: {
            ident: 'postcss',
            plugins: [
              require('tailwindcss'),
              require('autoprefixer'),
            ],
          },
        },
      ],
      include: path.resolve(__dirname, '../'),
    })
    return config
  },

Instead of webpackFinal, use viteFinal, and configure the tailwind postcss config the way Vite wants it to be configured.


Connections from SSMS to Windows Container don't work properly over IPv6

Please see https://dba.stackexchange.com/q/197499/8841 for thorough documentation of the issue and triage.
I am running the microsoft/mssql-server-windows-developer container using Docker CE for Windows on Windows 10. When running SSMS on the Windows 10 host and connecting to the container as localhost or ., queries execute fine, but browsing the object explorer is slow, and various menu items (like server properties or attach database) time out. On my machine, localhost resolves to ::1. When I connect to <IP_ADDRESS> instead, SSMS functions as expected.

I think this issue is from SMO taking a long time to failover to the IPv4 loopback, or from Docker on Windows not supporting exposing ports to an IPv6 address. Neither of those can be fixed in this repo, but perhaps some documentation is warranted.

I'm running into this same issue on an identical setup. However, I'm not able to connect in SSMS 17.5 to <IP_ADDRESS> at all. Only . or localhost works, so I can't seem to work around it. Not sure what's going on there.

@mattnischan Do you see any entries for TCP 1433 if you do a netstat -a after starting the container?

I did check netstat -a, which doesn't show any binding to 1443 after container start. I can however, despite that, connect to the instance in SSMS using . or localhost. Additionally, sqlmd in an interactive terminal works just fine.

This is Docker For Windows version 17.12.0-ce-win47 (15139), channel: stable, 9c692cd


Migrate away from Cloudflare

Can you believe it? Cloudflare still uses an outdated build image, which limits Zola to v0.14.0. That means no Bugfixes and no new features! For example, there is no author field in the RSS right now.
The only reason I used Cloudflare was HTTP/3. Well, I don't think there is that much need for me to use it. My website is minimal and will be barely faster. Other than that, I don't like Cloudflare hosting my DNS entries. It also makes v* Subdomains hard to do. AFAIK Codeberg uses Cloudflare under the hood, but their Pages are way superior. I hope they've fixed the problems with the hosting by now.

[x] set up a GitHub Action to build the website
[x] set up a publishing job

let's use Codeberg Pages again

[x] unbind the Codeberg Proxy

there is a weird thing one should do when migrating from one CF user to another. If I don't do this, the domains won't work lol
to-do: make a post about it

[x] to self: move DNS to deSEC
[ ] configure building the subdomain pages too
[ ] add DNS entries for the subdomains

Cloudflare did require to 'liberate' domains before changing SaaS providers: https://community.cloudflare.com/t/archived-troubleshooting-custom-hostnames-cloudflare-for-saas-when-moving-provider/394279
There was a tool for this: https://liberate-the-hostname.pages.dev/
But it's not needed any more, so it should work just so

It seems like Codeberg still has problems: https://codeberg.org/Codeberg/Community/issues/873. Let's use GH Pages

Alright, let's hope that the transition runs smoothly


GUI: send-to-self transactions are incorrectly displayed

See the attached screencast.
Screencast from 2022-12-08 17-23-15.webm

It's not incorrect but at the very least extremely confusing. @kloaec any suggestion on improving this?

Yeah I do not know what to do for this kind of transaction

I'm still not a fan of this. Would appreciate a "Send to self" label instead of 0.0000

Status update: this will be addressed as part of a larger overhaul of the UI for 1.0.


Improve validation error message for 'type'.

Wanted to get the basic pattern down. This also has a few other changes:

fixes some typos
creates a pattern for testing the response body (currently we only test the status code which is partly useful)
dry'ed up some of the test code
sort frameworks/types to avoid flaky tests due to ordering

Closing in favor of a different approach (TBD)


Arbitrary instances with alternate, valid, serializations.

The ledger is very careful to never re-serialize any data structures that will be hashed. This is important since there are multiple valid ways to encode many data structures in CBOR. Developers who use reuse some of the ledger code, however, are not always as aware of how important this is. In particular, sometimes projects will mix and match the ledger code and other third party serialization libraries, leading to confusion about why hashes do not match. See  #2943 as an example.
Section 3.9 of the CBOR RFC describes the various choices allowed in CBOR (in the context of describing which choice to make if you want to be canonical).
The ledger itself, however, has to make specific choices about how to serialize, even though the deserializers are flexible.
In order to make it more transparent that some of the ledger serialization is arbitrary (using definite vs indefinite lists, etc), we can write our Arbitrary generators such that they vary these choices.
The structure that seems to lead to the most confusion seems to be Datum. We could address Datums directly, or perhaps write a general way of "twiddling" all of our CBOR encodings.

If it's helpful, here are the three big encoding variances we've encountered and had to work around when interacting with browser and hardware wallets:

definite / indefinite length arrays
the ordering of map keys
empty list vs omitted key in maps

The ledger itself, however, has to make specific choices about how to serialize, even though the deserializers are flexible.

Isn't it very important as well that we define the canonical deterministic serialization choices we made, to keep the hashes reproducible by third-party?

@yihuang no, it's important that people not be mislead into thinking that there is a canonical representation. There is not a canonical representation. When you sign or hash the data, you must only hash the original bytes, and not a re-serialisation.
It's a very bad security practice to check signatures or hashes on re-serialised data. It must only be done on the original bytes. Otherwise it leads to all sorts of nasty security problems (think txs with different bytes but that have the same hash).

@yihuang no, it's important that people not be mislead into thinking that there is a canonical representation. There is not a canonical representation. When you sign or hash the data, you must only hash the original bytes, and not a re-serialisation.
It's a very bad security practice to check signatures or hashes on re-serialised data. It must only be done on the original bytes. Otherwise it leads to all sorts of nasty security problems (think txs with different bytes but that have the same hash).

But shouldn't there be a specification or something, or do you mean the only specification is the Haskell code itself, what about the clients implemented in other languages?

@yihuang we have a wire specification (CDDL) for every ledger era. See the table at the top of the readme in this repository. For example, the latest one is here: https://github.com/input-output-hk/cardano-ledger/blob/11e4d4a8ac88adf33baf6b0602635bf37a53803e/eras/babbage/test-suite/cddl-files/babbage.cddl

@yihuang we have a wire specification (CDDL) for every ledger era. See the table at the top of the readme in this repository. For example, the latest one is here: https://github.com/input-output-hk/cardano-ledger/blob/11e4d4a8ac88adf33baf6b0602635bf37a53803e/eras/babbage/test-suite/cddl-files/babbage.cddl

I mean the spec of the serialization details, so third-parties can reproduce the same result? Is there a complete doc on it that I'm not aware of?

no, there is no such a document. there is also no good reason for anyone to try to reproduce the exact arbitrary choices that our code uses which is not captured by the CDDL spec. if they are, they are likely doing exactly what we are trying to prevent. See https://github.com/input-output-hk/cardano-ledger/issues/2943#issuecomment-1203989504

no, there is no such a document. there is also no good reason for anyone to try to reproduce the exact arbitrary choices that our code uses which is not captured by the CDDL spec. if they are, they are likely doing exactly what we are trying to prevent. See https://github.com/input-output-hk/cardano-ledger/issues/2943#issuecomment-1203989504

I can think of at least one case though, implement Cardano in different languages, maybe only part of it, for example build and sign tx in a wallet.

I can think of at least one case though, implement Cardano in different languages, or do you mean even if the alternative client don't serialize a block in exact way, it still works, because the other nodes won't try to re-serialize it?

exactly! if everyone conforms to the CDDL spec, and does not re-serialize, then everything works.

Exactly. The spec is the CDDL, and there is no need for other interoperable implementations to serialise in exactly the same way so long as they follow the CDDL specification.

@Quantumplation, what do you mean by

* empty list vs omitted key in maps

@Soupstraw Consider collateral, for example.  If no collateral is specified, the cardano-api code will serialize the transaction body as a map, with key 13 set to an empty list:
84            ; Array of 4 elements
  a8          ; Map with 8 keys
    ...
    0d 9f ff  ; key 13, array; end array

whereas it is also a valid encoding (and the one preferred by hardware wallets, it appears) to just leave the 13 key out of the map entirely
84            ; Array of 4 elements
  a7          ; Map with 8 keys
    ...

The same thing applies to a few other fields, IIRC

right, so what @Quantumplation is referring to is not an ambiguity in CBOR, but rather in our CDDL. We often have optional keys in maps, and there is no semantic difference between leaving that key out, or including that key with a mempty value.

Ahh, I see, thanks for clarifying!


Using struct names as enum variants when [de]serializing

I have an idea to use struct names as tags of enums when deserializing and serializing. I've noticed there is enum grammar in the docs: docs/grammar.md but I failed to understand how it can be used.
I've found #519, which probably has the same blockers as this feature would have. I don't know how possible this is to implement and whether it has a lot of usability for others. I plan to use this to load a list of objects from a file and using enum variants would probably be the best way to detect which concrete type to deserialize into.

Ah, so the default behavior with externally tagged enums is exactly what I'm asking about? Sweet!


Self deployed .WAR not working.

I made a simple change in the genres and when I generate the .WAR file the application does not work. It does not load the assets.
Even if I try to generate the .WAR file from the original source code (1.0.18) the same problem occurs.
I am generating the .WAR file through the "gredlew assemble" command.

This is my stacktrace in browser.
Error: Bootstrap's JavaScript requires jQuery version 1.9.1 or higher, but lower than version 3  vendor-b5740a2e6746524695cb8c236f764e46.js:52296:11
ReferenceError: _ is not defined[Learn More]  streama-99400d7ab2214c01da39a906cbf3db01.js:2:47
Error: [$injector:modulerr] Failed to instantiate module streama due to:
[$injector:nomod] Module 'streama' is not available! You either misspelled the module name or forgot to load it. If registering a module ensure that you specify the dependencies as the second argument.
http://errors.angularjs.org/1.4.14/$injector/nomod?p0=streama
minErr/<@http://localhost:8080/assets/vendor-b5740a2e6746524695cb8c236f764e46.js:13173:12
module/<@http://localhost:8080/assets/vendor-b5740a2e6746524695cb8c236f764e46.js:15127:17
ensure@http://localhost:8080/assets/vendor-b5740a2e6746524695cb8c236f764e46.js:15051:38
module@http://localhost:8080/assets/vendor-b5740a2e6746524695cb8c236f764e46.js:15125:14
loadModules/<@http://localhost:8080/assets/vendor-b5740a2e6746524695cb8c236f764e46.js:17587:22
forEach@http://localhost:8080/assets/vendor-b5740a2e6746524695cb8c236f764e46.js:13446:11
loadModules@http://localhost:8080/assets/vendor-b5740a2e6746524695cb8c236f764e46.js:17571:5
createInjector@http://localhost:8080/assets/vendor-b5740a2e6746524695cb8c236f764e46.js:17496:11
bootstrap/doBootstrap@http://localhost:8080/assets/vendor-b5740a2e6746524695cb8c236f764e46.js:14798:20
bootstrap@http://localhost:8080/assets/vendor-b5740a2e6746524695cb8c236f764e46.js:14819:12
angularInit@http://localhost:8080/assets/vendor-b5740a2e6746524695cb8c236f764e46.js:14713:5
@http://localhost:8080/assets/vendor-b5740a2e6746524695cb8c236f764e46.js:42952:5
resolve/</mightThrow@http://localhost:8080/assets/vendor-b5740a2e6746524695cb8c236f764e46.js:6454:21
resolve/</process<@http://localhost:8080/assets/vendor-b5740a2e6746524695cb8c236f764e46.js:6522:12

http://errors.angularjs.org/1.4.14/$injector/modulerr?p0=streama&p1=%5B%24injector%3Anomod%5D%20Module%20'streama'%20is%20not%20available!%20You%20either%20misspelled%20the%20module%20name%20or%20forgot%20to%20load%20it.%20If%20registering%20a%20module%20ensure%20that%20you%20specify%20the%20dependencies%20as%20the%20second%20argument.%0Ahttp%3A%2F%2Ferrors.angularjs.org%2F1.4.14%2F%24injector%2Fnomod%3Fp0%3Dstreama%0AminErr%2F%3C%40http%3A%2F%2Flocalhost%3A8080%2Fassets%2Fvendor-b5740a2e6746524695cb8c236f764e46.js%3A13173%3A12%0Amodule%2F%3C%40http%3A%2F%2Flocalhost%3A8080%2Fassets%2Fvendor-b5740a2e6746524695cb8c236f764e46.js%3A15127%3A17%0Aensure%40http%3A%2F%2Flocalhost%3A8080%2Fassets%2Fvendor-b5740a2e6746524695cb8c236f764e46.js%3A15051%3A38%0Amodule%40http%3A%2F%2Flocalhost%3A8080%2Fassets%2Fvendor-b5740a2e6746524695cb8c236f764e46.js%3A15125%3A14%0AloadModules%2F%3C%40http%3A%2F%2Flocalhost%3A8080%2Fassets%2Fvendor-b5740a2e6746524695cb8c236f764e46.js%3A17587%3A22%0AforEach%40http%3A%2F%2Flocalhost%3A8080%2Fassets%2Fvendor-b5740a2e6746524695cb8c236f764e46.js%3A13446%3A11%0AloadModules%40http%3A%2F%2Flocalhost%3A8080%2Fassets%2Fvendor-b5740a2e6746524695cb8c236f764e46.js%3A17571%3A5%0AcreateInjector%40http%3A%2F%2Flocalhost%3A8080%2Fassets%2Fvendor-b5740a2e6746524695cb8c236f764e46.js%3A17496%3A11%0Abootstrap%2FdoBootstrap%40http%3A%2F%2Flocalhost%3A8080%2Fassets%2Fvendor-b5740a2e6746524695cb8c236f764e46.js%3A14798%3A20%0Abootstrap%40http%3A%2F%2Flocalhost%3A8080%2Fassets%2Fvendor-b5740a2e6746524695cb8c236f764e46.js%3A14819%3A12%0AangularInit%40http%3A%2F%2Flocalhost%3A8080%2Fassets%2Fvendor-b5740a2e6746524695cb8c236f764e46.js%3A14713%3A5%0A%40http%3A%2F%2Flocalhost%3A8080%2Fassets%2Fvendor-b5740a2e6746524695cb8c236f764e46.js%3A42952%3A5%0Aresolve%2F%3C%2FmightThrow%40http%3A%2F%2Flocalhost%3A8080%2Fassets%2Fvendor-b5740a2e6746524695cb8c236f764e46.js%3A6454%3A21%0Aresolve%2F%3C%2Fprocess%3C%40http%3A%2F%2Flocalhost%3A8080%2Fassets%2Fvendor-b5740a2e6746524695cb8c236f764e46.js%3A6522%3A12%0A  vendor-b5740a2e6746524695cb8c236f764e46.js:13173:12

Can I get access to your system / Setup?

Yes i sent to you.

So .. with access to your system, I redeployed the app using this guide and had no issues.
For anyone who stumbles upon this, here is what I did:

ive initialized a directory under /data/streama where the application.yml resides and the executable .war

then downloaded the application.yml and the .war and placed them there, and changed the ownership of everything to streama (you shouldnt deploy web apps as sudo, too much could go wrong)

then I installed byobu, its nice to use for running the application and also doing other stuff, unless you want to install it as a service, in which case you can follow this guide: https://github.com/dularion/streama/wiki/Getting-started-with-v1#10-how-do-i-install-it-as-a-service

then in order to test locally, I used an ssh tunnel, like so: ssh streama@yourIP -L 8080:localhost:8080 -N which allowed me to just call http://localhost:8080/ in the browser to access
of course, it works via http://yourIP:8080/ too


304 has different response when useing different user-agent.

[-] I've read, understood, and done my best to follow the *CONTRIBUTING guidelines.

What did you do?
I hava two examples with different user-agent, and they have two different results like below.
Alamofire.request(urlString, headers: ["If-None-Match": "\"ee5-VhC3k32EP+WaDUtWC+bGYEFL+Fs\""]).responseJSON { resp in
    print(resp.response?.statusCode)
})

// The result is:
Optional(200)

Alamofire.request(urlString, headers: ["User-Agent": "iOS", "If-None-Match": "\"ee5-VhC3k32EP+WaDUtWC+bGYEFL+Fs\""]).responseJSON { resp in
    print(resp.response?.statusCode)
})

// The result is:
Optional(304)

I am sure I havan't change other things, like cache poliy. And I am sure the server had return the same response(304) when I test examples above.
And I hope both of them return the Optional(200).
Alamofire Environment
Alamofire version: Alamofire/4.5.0
Xcode version: Version 8.3.3 (8E3004b)
Swift version: Apple Swift version 3.1 (swiftlang-802.0.53 clang-802.0.42)
Platform(s) running Alamofire: iOS 10.3.1
macOS version running Xcode: macOS Sierra Version 10.12.5

I suggest you investigate how the server you're connecting to works, specifically how it deals with user agents. Also, in the future, please follow the issue template.
Also, we use our GitHub project for bug reports and feature requests. In the future, you should open questions like this on Stack Overflow and tag alamofire.
Cheers. 🍻

From our Contribution Guidelines
Asking Questions
We don't use GitHub as a support forum. For any usage questions that are not specific to the project itself, please ask on Stack Overflow instead. By doing so, you'll be more likely to quickly solve your problem, and you'll allow anyone else with the same question to find the answer. This also allows maintainers to focus on improving the project for others.


Google Maps API and JavaFX Package Incompatibility

The JavaFX GUI interface does not support Google Maps display.

JxBroswer solved the problem.


[SW-185803] Enable FusedSDPA fp8 in Llama FT

Same PR as in https://github.com/HabanaAI/optimum-habana-fork/pull/291, but I fixed the error occurring in the text-generation/run_lm_eval.py script by moving one import from top-level to function-level, i.e., see https://github.com/HabanaAI/optimum-habana-fork/commit/3af45d241a8cd7ada170b7eefad758de55b880bf

@pbielak Thanks for updates. Please check my latest responses.


[Feature Request]: Send App Service logs to Blob Storage with sasUrl

Description
There should be an option for enabling the App Service logs and sending to the Blob Storage with sasUrl. The related Bicep file is https://github.com/Azure/ResourceModules/blob/main/modules/web/sites/main.bicep

Need to investigate if the feature is part of the latest API version.


Implementation of various screens with transitions

Provide a new demo on how use and create a simple game menu.
This would be useful for many students to switch between games levels, maps, etc.

First implementation done in accc8896eef5eede5aaedf7d1dce1955c0480fbb, with a demo as well. Some polishing required but the principles are there. The whole library will be rewritten like this.


Inconsistent ODatabase API

Important Questions
There is the following description of ODatabase.command()
Execute a command against the database. A command can be a SQL statement or a Procedure. If the OStorage used is remote (OStorageRemote) then the command will be executed remotely and the result returned back to the calling client.

But in fact code of that method is the following:
public OCommandRequest command(final OCommandRequest iCommand) {
    checkSecurity(ORule.ResourceGeneric.COMMAND, ORole.PERMISSION_READ);
    checkIfActive();

    final OCommandRequestInternal command = (OCommandRequestInternal) iCommand;

    try {
      command.reset();
      return command;

    } catch (Exception e) {
      throw new ODatabaseException("Error on command execution", e);
    }
  }

As you can see there is NO actual execution of a command.
I do understand that it's possible to do like this: db.command(command).execute(args), but it looks inconsistent with query(). Btw, it will be good to have the same signature as query: I mean to have ability to specify arguments to be passed to command.
Imho it will be more safe to change just JavaDoc, because I guess there are tons of code which have execute() afterwards.
What do you think?
OrientDB Version
v2.1.19

This was already discussed internally and will be changed in v3.0 for sure.

hi @PhantomYdn
the discussion is happening here: https://github.com/orientechnologies/orientdb-labs/issues/5 feel free to join gracefully.
I'll close this, is kind of duplicate


Allow to use a different goss_test_directory

Replace goss_render.sh with template

I think we can replace the script. This role brings it's own test so the glob always matches at least one file and doesn't fail.
Having no tests is anyway not supported by goss:
TASK [dockpack.base_goss : Execute Goss tests] ******************************************************
fatal: [jabber.systemli.org]: FAILED! => {"changed": false, "cmd": ["/usr/local/bin/goss", "-g", "/root/goss.yaml", "validate", "--format", "tap"], "delta": "0:00:00.005130", "end": "2019-05-27 11:49:36.585700", "msg": "non-zero return code", "rc": 1, "start": "2019-05-27 11:49:36.580570", "stderr": "", "stderr_lines": [], "stdout": "Error: found 0 tests, source: /root/goss.yaml", "stdout_lines": ["Error: found 0 tests, source: /root/goss.yaml"]}

For example, I'm running your great role with the following parametes:
goss_test_directory: /etc/goss.d/
goss_test_directory_mode: 0755
goss_path: /usr/local/bin
goss_dst: /usr/local/bin/goss

I need to think about this. This role is used to validate health checks that are created as goss files by other roles. I opened issues in each of those.

I'm not sure, if this was on purpose, but you already merge this with #11. The default behavior is as before, all tests go to /root. I don't think you have to adjust anything in your other roles if you don't want to. Thanks a lot.

I merged your PRs in a different order than you created them and pushed the 'Updated branch' button on Github for a final Travis run. Now most of your stuff is merged, but I wonder if this one is in good shape for both off us now.
goss_render.sh is no longer installed, and 2 lines still have hard-coded /root:

https://github.com/dockpack/base_goss/blob/472ef44ad7968a47c0ee06eb1227ffea72533fec/tasks/main.yml#L74
https://github.com/dockpack/base_goss/blob/472ef44ad7968a47c0ee06eb1227ffea72533fec/tasks/main.yml#L88

The current state is fine for me and I don't need this PR anymore. Thank you for your careful watching.

2 lines still have hard-coded /root:

That's okay for me for the moment. If I have a better idea, I'll open another PR which handles /root/goss.yaml explicitly.


Remove AppState dependencies from BaseGameUtils java classes

As AppState is depricated, i think it dependencies can be removed from BaseGameUtils library!

Thanks for the comment. If you'd like, feel free to submit a pull request with the update.

Hello, not only the dependency should be removed, the project now does not compile since their is an import in the GamerHelper.Java class ( line 13) for the appstate (import com.google.android.gms.appstate.AppStateManager;) and it is used later on in the class ( 2 instances ), thus the class also needs to be updated in order for the project to compile.
I use android studio and I have set the playservices with a '+' instead of version to always use the latest, visiting an old project to update. It didn't compile with the latest play services version.

Hello, I'm having the same issue after updating the play-services to 8.1 version.


Textfield magnifier doesn't disappear sometimes

Steps to Reproduce

Create two pages (one with a button to open the second page and the other with a TextField)
Put your finger on top of the TextField and don't let it go.
Using your other finger, try to dispose the keyboard by starting to pop the second page
The magnifier doesn't dispose

Example: https://vimeo.com/800740464
Expected results:
The keyboard magnifier should disappear when keyboard disappears.
Actual results:

Code sample

Make two pages, one with a text field and the other with a button to open the first page.

Logs

flutter run --verbose

attach all the log output below between the lines with the backticks.

I hope I could. It's so long to attach.

GitHub says There was an error creating your Issue: body is too long, body is too long (maximum is 65536 characters).

I have no problem with running and launching the app.

flutter analyze
shawn@Shawns-MBP atrablechat % flutter analyze
Analyzing atrablechat...

warning • The include file 'package:flutter_lints/flutter.yaml' in
       '/Users/shawn/StudioProjects/atrablechat/analysis_options.yaml' can't be
       found when analyzing '/Users/shawn/StudioProjects/atrablechat' •
       analysis_options.yaml:10:10 • include_file_not_found

1 issue found. (ran in 5.5s)

flutter doctor -v
shawn@Shawns-MBP atrablechat % flutter doctor -v
[!] Flutter (Channel stable, 3.7.3, on macOS 13.2 22D49 darwin-arm64, locale en-US)
    • Flutter version 3.7.3 on channel stable at /Users/shawn/development/flutter
    ! The flutter binary is not on your path. Consider adding /Users/shawn/development/flutter/bin to your path.
    ! Warning: `dart` on your path resolves to /opt/homebrew/Cellar/dart/2.16.1/libexec/bin/dart, which is not inside your current Flutter SDK checkout at /Users/shawn/development/flutter. Consider adding
      /Users/shawn/development/flutter/bin to the front of your path.
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision<PHONE_NUMBER> (12 days ago), 2023-02-08 15:46:04 -0800
    • Engine revision 248290d6d5
    • Dart version 2.19.2
    • DevTools version 2.20.1
    • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades.

[✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at /Users/shawn/Library/Android/sdk
    • Platform android-33, build-tools 31.0.0
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.2)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14C18
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)

[✓] IntelliJ IDEA Community Edition (version 2022.3.2)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] VS Code (version 1.75.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.58.0

[✓] Connected device (4 available)
    • Shawn’s iPhone (mobile) • 00008110-000171423EB8801E            • ios            • iOS 16.3.1 20D67
    • iPhone 14 Pro (mobile)  • 4B5F585A-C9AE-4192-B2DF-D2724816438C • ios            • com.apple.CoreSimulator.SimRuntime.iOS-16-2 (simulator)
    • macOS (desktop)         • macos                                • darwin-arm64   • macOS 13.2 22D49 darwin-arm64
    • Chrome (web)            • chrome                               • web-javascript • Google Chrome 110.0.5481.100

[✓] HTTP Host Availability
    • All required HTTP hosts are available

! Doctor found issues in 1 category.

Hello @Shawn-sudo. While I can reproduce this issue on stable, it seems that it is already fixed in master. Therefore, I am closing this issue. Thank you


vfox install nodejs fails on Windows when using msys2 bash

I installed vfox via scoop: scoop install vfox. A prior issue on the vfox main repo discusses more about my vfox environment, which uses msy2 bash: https://github.com/version-fox/vfox/issues/172
As shown below, vfox install nodejs failed with this message:
Please use `vfox install nodejs@<version>` to install the version you need.
plugin [PreInstall] method error: C:\Users\ben\.version-fox\plugin\nodejs\hooks\pre_install.lua:23: cannot perform concat operation between string and nil
stack traceback:
        C:\Users\ben\.version-fox\plugin\nodejs\hooks\pre_install.lua:23: in main chunk
        [G]: ?

Maybe it was supposed to fail and  you have to specify a version? I think some spacing and tweak to the wording could help then.
ben@lenovo-legion dotfiles [master]$ vfox install nodejs
[nodejs] not added yet, confirm that you want to use [nodejs]?
Please confirm [y/N]: Yes
Fetching plugin manifest...
Downloading https://github.com/version-fox/vfox-nodejs/releases/download/v0.2.2/vfox-nodejs-0.2.2.zip...
Validating C:\Users\ben\.version-fox\plugin\vfox-nodejs-0.2.2.zip ...
Plugin info:
Name     -> nodejs
Version  -> 0.2.2
Homepage -> https://github.com/version-fox/vfox-nodejs
Desc     -> Node.js runtime environment.
Add nodejs plugin successfully!
Please use `vfox install nodejs@<version>` to install the version you need.
plugin [PreInstall] method error: C:\Users\ben\.version-fox\plugin\nodejs\hooks\pre_install.lua:23: cannot perform concat operation between string and nil
stack traceback:
        C:\Users\ben\.version-fox\plugin\nodejs\hooks\pre_install.lua:23: in main chunk
        [G]: ?
ben@lenovo-legion dotfiles [master]$ vfox add nodejs
plugin nodejs already exists
ben@lenovo-legion dotfiles [master]$ vfox install nodejs
plugin [PreInstall] method error: C:\Users\ben\.version-fox\plugin\nodejs\hooks\pre_install.lua:23: cannot perform concat operation between string and nil
stack traceback:
        C:\Users\ben\.version-fox\plugin\nodejs\hooks\pre_install.lua:23: in main chunk
        [G]: ?
ben@lenovo-legion dotfiles [master]$ vfox install nodejs@16
Preinstalling<EMAIL_ADDRESS>Downloading... 100% [=====================================================================================================================] (30 MB/s)
Verifying checksum f8bb35f6c08dc7bf14ac753509c06ed1a7ebf5b390cd3fbdc8f8c1aedd020ec3...
Unpacking C:\Users\ben\.version-fox\cache\nodejs\node-v16.20.2-win-x64.zip...
Install<EMAIL_ADDRESS>success!
Please use vfox use<EMAIL_ADDRESS>to use it.
ben@lenovo-legion dotfiles [master]$

Thanks for feedback this that is a known issue, see version-fox/vfox#205.
We plan to fix this issue on vfox, not plugins.


Coins confirmed in Siamining but not showing in the Sia UI wallet

Hi
The following transaction shows that they were completed in siamining but does not reflect in my Wallet, what should i do?

Date/Time
Amount
Transaction ID
Status

2018-04-02 22:55 UTC
635.11 SC
aa72d3ab8191c6f6f46146a792c5713e5cb91265e9f40ee43d8d652eba2ce147
confirmed

2018-04-01 22:55 UTC
506.02 SC
b5ab415b3f291d0edeba6cabf0baba82ebbb8dd41708a6110523be3f7e36ace1
confirmed

Try going through these troubleshooting steps: https://siawiki.tech/wallet/lost_transactions_troubleshooter


laravel-elixir ^4.0.0 compatability

For now npm drops error with laravel-elixir ^4.0.0:
npm ERR! peerinvalid The package<EMAIL_ADDRESS>does not satisfy its siblings' peerDependencies requirements!
npm ERR! peerinvalid Peer<EMAIL_ADDRESS>wants<EMAIL_ADDRESS>

Thanks, just published a new version with the dependency bumped. a8fffa3


Add download data option to collect info to add tbp to member's transcripts

Something like "Download member data for registrar/for updating transcripts"
Ideal if it could download names, uniqnames, umid's, and semester initiated.
Due date: end of next semester? or the next? depends when you want to not have to download stuff manually

Also add the ability to have the spreadsheet emailed to the president or
secretary when the semester is changed
On Friday, December 18, 2015, domansyl<EMAIL_ADDRESS>wrote:

Something like "Download member data for registrar/for updating
transcripts"
Ideal if it could download names, uniqnames, umid's, and semester initiated
Due date: end of next semester? or the next? depends when you want to not
have to download stuff manually
—
Reply to this email directly or view it on GitHub
https://github.com/tbpmig/mig-website/issues/199.

--
Mike Hand

I know you probably consider this low priority since it is a feature enhancement, but it would be much appreciated.
I'm currently working on putting a list together to send the university with this information dating back to F14 (the last time we sent them anything). Names and uniqnames are in the MemberData.csv file that I can download, but to get the UMIDs and semester initiated I'll have to scrape them from everyone's profiles. Then I need to use the HQ online system to determine if there is anyone who initiated as an undergrad but is now a grad since MemberData.csv only lists current standing, not standing when initiated.


high throughput inference

Was chatting with @Chillee about our plans in AO today and he mentioned we should be focusing on a few concrete problems like

Demonstrate compelling perf for fp8 gemm at a variety of batch sizes.
Demonstrate compelling perf for weight only int8 gemm at a variety of batch sizes.
Demonstrate compelling perf for weight only intX gemm at low batch sizes.
Demonstrate compelling perf for weight intX, activation fp8 at a variety of batch sizes.

We could as a baseline extend gpt-fast to work with bs=n w/o doing any kv cache management work and measure perf there. Copying feedback as is, open to discussing more and adding more details as time progresses

@HDCharles on the int8 work
@vkuzo on fp8
@vayuda and @jerryzh168 on intx

@msaroufim
Would be interesting to bench against something like QoQ, which implements  W4A8KV4 (int8 GEMM) using a nested quantization scheme and neat kernel-level optimizations.

Demonstrate compelling perf for fp8 gemm at a variety of batch sizes.

Note that I'm putting up a PR soon for a quick roofline estimator for float8 gemm + overhead specific to training to see for which M, K, N float8 is faster than bfloat16, it would be easiliy extendable to inference at a later time.

Demonstrate compelling perf for weight intX, activation fp8 at a variety of batch sizes.

While this is possible technically, I'm not sure I understand the value, would be interested to learn more.


Organise a working group review of Navigation mvp

What
Organise a working group review of mvp prototype, defined by the outcomes of #3678.
The working group review will run between 15-19 April (one week).
Why
For the working group members to assess whether the proposal is a useful and unique contribution to the Design System.
Who needs to work on this
Trang
Who needs to review this
Nav squad (Charlotte, Mia, Brett, beeps, Calvin, Kelly)
Dependencies
#3678
Prototype
Done when

[x] we have informed the working group of the upcoming review
[ ] we have identified a date for the feedback session (early cycle 3)
[ ] we have prepared supporting materials for the review
[ ] we have made the wider Design System team aware that a working group review is taking place
[ ] we start the review on 15 April

Messaged the working group on https://ukgovernmentdigital.slack.com/archives/C0216FD6YMP

Hello working group.
You may have heard, we’ve started working on the navigation component.
Just a heads up, we’re going to send the proposal your way for the first review. It’s planned for 15 April and will be open for a 1 week review period. We hope you can make time for it.
The first review asks, “Is the proposal useful and unique?” You can brush up on the full contribution criteria here: https://design-system.service.gov.uk/community/contribution-criteria/
We would also like to run a followup session to discuss the feedback you provided on Navigation the week after. Can you please select all the times you are available so we can pick a time that suits most people. https://rallly.co/invite/JVMcjw6cLubC
Thanks.

Emailed the Working Group for availability today

We have sent out the Navigation proposal to the Working Group on Monday 15 April 2024. We are planning the Working Group video review for Tuesday 23 April 2024, 2:00 - 3:00pm.


Add step to install sqlalchemy driver to Connect to Your CockroachCloud Cluster Doc

Re: Connect to Your CockroachCloud Cluster
Issue Description
sqlalchemy instructions on this page are missing the steps to install the driver. It only says to replace the postgresql url scheme with cockroachdb.
Suggested Resolution
Include the step to install sqlalchemy cockroachdb driver
pip install sqlalchemy sqlalchemy-cockroachdb psycopg2

I think that's by design, based on this sentene:

The following language-specific versions assume that you have installed the relevant client drivers.

But I could see the value in providing the install instructions so there's nowhere else to go to get through the tutorial.

Closing - as this is no longer relevant (sqlalchemy steps are no longer included on this doc)


OSX / Linux versions

Essentially needs "SpotifyEnabledBrowser" part of the client porting to OSX...

@earthling Spotify used to use QT but now it uses nodewebkit
https://nodesource.com/blog/node-desktop-applications


Add TOC to books landing page

Should be collapsed on mobile view.
Should be expanded on desktop view.

Experimented with it.
But this would steal space from content.
For now, the "TOC" is sufficient as-is.
In the future, I have to think about a general UI overhaul if this comes up again.


Support WN34 Soil Temp Sensor

Thanks for forking the repo and cleaning it up!
This is a newer for soil moisture which doesn't show up (and throws errors in the log if you have one). There is a longer discussion in the garbled1 repo but thought I would post the issue here since I've switched over to this version. https://github.com/garbled1/homeassistant_ecowitt/issues/80

Of course this was already added in fcecf78422c3f86eeb05d1389ba4240875590675 -- filed before I had switched over!


Performance of Resnet50 is much slower than Tensorflow on Intel i7 of Mac

I follow the https://github.com/dmlc/nnvm/blob/master/examples/benchmark/gpu_imagenet_bench.py, I modify the target to "llvm -mcpu=core-avx2".
The model of VGG16 / mobilenet is faster than tensorflow. VGG16 is 141ms, mobilenet is 12.5ms, tensorflow's VGG16 is 274.434 ms and mobilenet is 56.511. However for the model Resnet50, TVM is 339ms, but tensorflow is 138.267ms. Does the TVM not optimize the Resnet50 on the Intel i7 CPU? Thanks.

Recently we are doing some optimization work more focusing on Intel Xeon. See https://github.com/dmlc/tvm/pull/883 as an example. We believe most of our scheduling schemes could be applied to Intel Core. Stay tuned.
Also, you are welcome to redirect your questions to https://discuss.tvmlang.org/ in the future.

Right. I have set the mcpu to avx2, without it, TVM is slower. But from the view of mine, this key issue should be TVM has not done schedule for Resnet50, only for Resnet18. See https://github.com/dmlc/tvm/blob/8ef4fdef229fcce37f9c71bd4b916811471fea9d/topi/python/topi/x86/conv2d.py#L35
If it is not right, please correct me.

We also have Resnet50 along with a lot of other networks optimized for Intel CPUs. We will release the code and benchmarking results later. @yzhliu @kevinthesun can give you more details.
In addition, TF does not perform well on Intel CPUs in general. It is better to use MXNet with MKLDNN enabled as the baseline.

we just haven't contribute the schedules back to tvm. sorry for the inconvenience. will make a PR in a couple of days.

we just haven't contribute the schedules back to tvm. sorry for the inconvenience. will make a PR in a couple of days.

@yzhliu I am glad to hear that. I go through your code repository, I don't find related changeset of schedule code. Because I am investigating the performance of TVM, could I use the code of your PR to test? I am caring about Resnet50's performace. Currently, if I close loop-unroll, TVM's Resnet50 will be about 800~900ms, if turn on, will be about 339ms. I think if TVM has schedule of Resnet50, it will have better performance than tensorflow.

@kevinthesun Thanks. Glad to hear this news!

@FrozenGene well, if you want to quickly test resnet-50, here's the missing schedules and workloads (just grabbed them out from my shitty code, it should work) :
schedules:
# workloads of resnet50_v1 on imagenet
AVXConv1x1Fwd(16, fp32_vec_len, 2, 28),
AVXConv1x1Fwd(16, fp32_vec_len, 2, 28),
AVXConv1x1Fwd(16, fp32_vec_len, 2, 28),
AVXConv1x1Fwd(16, fp32_vec_len, 2, 28),
AVXConv1x1Fwd(16, fp32_vec_len, 2, 28),
AVXConv1x1Fwd(16, fp32_vec_len, 2, 28),
AVXConv1x1Fwd(16, fp32_vec_len, 2, 14),
AVXConv1x1Fwd(16, fp32_vec_len, 2, 14),
AVXConv1x1Fwd(16, fp32_vec_len, 2, 14),
AVXConv1x1Fwd(16, fp32_vec_len, 2, 14),
AVXConv1x1Fwd(16, fp32_vec_len, 2, 7),
AVXConv1x1Fwd(16, fp32_vec_len, 2, 7),
AVXConv1x1Fwd(16, fp32_vec_len, 2, 7),
AVXConv1x1Fwd(16, fp32_vec_len, 2, 7),
# workloads of resnet50_v2 on imagenet
AVXConvCommonFwd(16, fp32_vec_len, 28, True),
AVXConvCommonFwd(16, fp32_vec_len, 14, False),
AVXConvCommonFwd(16, fp32_vec_len, 7, True),

workloads:
# workloads of resnet50_v1 on imagenet
Workload('float32', 'float32', 56, 56, 64, 256, 1, 1, 0, 0, 1, 1),
Workload('float32', 'float32', 56, 56, 256, 64, 1, 1, 0, 0, 1, 1),
Workload('float32', 'float32', 56, 56, 256, 128, 1, 1, 0, 0, 2, 2),
Workload('float32', 'float32', 28, 28, 128, 512, 1, 1, 0, 0, 1, 1),
Workload('float32', 'float32', 56, 56, 256, 512, 1, 1, 0, 0, 2, 2),
Workload('float32', 'float32', 28, 28, 512, 128, 1, 1, 0, 0, 1, 1),
Workload('float32', 'float32', 28, 28, 512, 256, 1, 1, 0, 0, 2, 2),
Workload('float32', 'float32', 14, 14, 256, 1024, 1, 1, 0, 0, 1, 1),
Workload('float32', 'float32', 28, 28, 512, 1024, 1, 1, 0, 0, 2, 2),
Workload('float32', 'float32', 14, 14, 1024, 256, 1, 1, 0, 0, 1, 1),
Workload('float32', 'float32', 14, 14, 1024, 512, 1, 1, 0, 0, 2, 2),
Workload('float32', 'float32', 7, 7, 512, 2048, 1, 1, 0, 0, 1, 1),
Workload('float32', 'float32', 14, 14, 1024, 2048, 1, 1, 0, 0, 2, 2),
Workload('float32', 'float32', 7, 7, 2048, 512, 1, 1, 0, 0, 1, 1),
# workloads of resnet50_v2 on imagenet 3 26-28
Workload('float32', 'float32', 56, 56, 128, 128, 3, 3, 1, 1, 2, 2),
Workload('float32', 'float32', 28, 28, 256, 256, 3, 3, 1, 1, 2, 2),
Workload('float32', 'float32', 14, 14, 512, 512, 3, 3, 1, 1, 2, 2),

Put them after this line https://github.com/dmlc/tvm/blob/8ef4fdef229fcce37f9c71bd4b916811471fea9d/topi/python/topi/x86/conv2d.py#L59 and this line https://github.com/dmlc/tvm/blob/8ef4fdef229fcce37f9c71bd4b916811471fea9d/topi/python/topi/nn/conv2d.py#L47 respectively.
It should enable you to run ResNet-152 as well. Have fun.
As @kevinthesun described, don't be surprised when you see your network become faster in a week or two.

@yzhliu Yes, it works. TVM's resnet runtime is 118.12ms using this schedule (config is opt_level 3 / Loop unroll step is 128, llvm -mcpu=core-avx2), which is better than tensorflow now (Tensorflow is 138.627ms). I am investigating the TVM's performance and plan to introduce TVM into my company, I must compare with tensorflow (MXNet + MKLDNN is one plus, but not must), which is one key factor to help me push TVM successfully. If success, our modification of TVM will contribute back to the community. And I am also glad to hear that we have plan to make TVM faster.

Glad to see good discussion here, let us move more discussions to https://discuss.tvmlang.org/
@FrozenGene Your contribution for joining the discussion is more the welcomed, TVM stack itself adopts apache style governance which means its roadmap and future depends on the diverse expertise of its contributors.

@tqchen @yzhliu @yidawang @kevinthesun Thanks, everyone. I have pushed NNVM / TVM successfully into one project into my company Alibaba. I must say that our project's scenario is very amazing and will collect VERY MUCH MUCH data, I think it will help TVM better. However, because our trained framework is tensorflow and can not change, we need teach TVM recognize Tensorflow model. I have opened one question about this, if you can give some advice, which will be very nice. https://discuss.tvmlang.org/t/teach-nnvm-recognize-tensorflow-model/82


Update to Node 16

Node.js 12 actions are deprecated. More info here: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/.
Please update this action to use Node.js 16 to make these annoying annotations go away! 🤣

I've submitted a PR for resolving this issue: #33


Reenable tests and forward fixture logs to test logs

Assuming this is for debugging the failure?

Yep

C:\projects\iisintegration\test\Common.FunctionalTests\Inprocess\SynchronousReadAndWriteTests.cs(15,48): error CS0246: The type or namespace name 'FixtureLoggedTest' could not be found (are you missing a using directive or an assembly reference?) [C:\projects\iisintegration\test\IIS.FunctionalTests\IIS.FunctionalTests.csproj]

cc @JunTaoLuo parts of this PR (attaching current test logger to fixture that has it's own logging) would be nice to have in Testing.Logging eventually.


Automator: update istio/api@master dependency in istio/istio@master

Generated by Automator - 2021-01-15T00:54:34+00:00

@istio-testing: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-telemetry-mc-k8s-tests_istio
a6dcdd5048cf84d91161bbdd87695b929105ab24
link
/test integ-telemetry-mc-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-telemetry-mc-k8s-tests_istio
a6dcdd5048cf84d91161bbdd87695b929105ab24
link
/test integ-telemetry-mc-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


Fix - Pdf order special characters

This PR fix special characters in the pdf order. Core fonts only support characters represented in Windows ANSI.

@lguichard Not sure we should be specifying DejaVu Sans as it's not even a font I have installed so likely many others won't either.

I fixed, use * {} directive fix the issue

@alecritson DejaVu Sans is included with dompdf.

Also, @lguichard, the formatting seems wrong with double quotes wrapping most of the string:
font-family: "Helvetica Neue', 'Helvetica', Helvetica, Arial", sans-serif;
             ^                                              ^

/* SHOULD BE */
font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif;

(Not sure why Helvetica was included two times. And you only need quotes if font name includes space.)
Suggestion
We could prepend the fallback list with DejaVu Sans, and it'll be used if found (which it always should be).
font-family: 'DejaVu Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif;

@alecritson DejaVu Sans is included with dompdf.
Also, @lguichard, the formatting seems wrong with double quotes wrapping most of the string:
font-family: "Helvetica Neue', 'Helvetica', Helvetica, Arial", sans-serif;
             ^                                              ^

/* SHOULD BE */
font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif;

(Not sure why Helvetica was included two times. And you only need quotes if font name includes space.)
Suggestion
We could prepend the fallback list with DejaVu Sans, and it'll be used if found (which it always should be).
font-family: 'DejaVu Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif;

Ok, it seemed that the font type included in dompdf

Here's some more info about why DejaVu might be needed for international support:

PDF documents internally support the following fonts: Helvetica, Times-Roman, Courier, Zapf-Dingbats, & Symbol. These fonts only support Windows ANSI encoding. In order for a PDF to display characters that are not available in Windows ANSI, you must supply an external font. Dompdf will embed any referenced font in the PDF so long as it has been pre-loaded or is accessible to dompdf and reference in CSS @font-face rules. See the font overview for more information on how to use fonts.
The DejaVu TrueType fonts have been pre-installed to give dompdf decent Unicode character coverage by default. To use the DejaVu fonts reference the font in your stylesheet, e.g. body { font-family: DejaVu Sans; } (for DejaVu Sans). The following DejaVu 2.34 fonts are available: DejaVu Sans, DejaVu Serif, and DejaVu Sans Mono.

https://github.com/dompdf/dompdf?tab=readme-ov-file#about-fonts--character-encoding


[QUESTION] One DB or multiple?

If my application has a multiple pieces of data that are unrelated to each other, is it best practice to have

One LiteDB database with multiple unrelated collections or
Multiple LiteDB databases that are independent of each other?

A single litedb is used for related data.
Non-relevant data uses multiple litedb.
I think if you use non-related data, you use multiple litedb.
And, consider litedb performance, need maintain small file size.


Added pandoc-quotes.lua.

pandoc-quotes.lua replaces non-typographic quotation marks with typographic ones for languages other than US English.

Thanks, this looks great and quite useful.
CI checks fail because access to the repo via git fails. While I like the idea of having a separate repo, it's worth noting that submodules have a couple of disadvantages. I'd prefer using git subtree instead.
Usage of the filter might become even easier if the YAML file was converted to a Lua table and integrated into the main script. Installation would be as simple as dropping the script in $/HOME/.pandoc/filters. But thanks to the good README, this is only a minor concern.

Both submodule and subtree should be avoided, I think.

Thanks! Also for the info about submodules. I wasn't aware of their disadvantages. I’ve made the changes you suggested and simplified the script.

Thank you for this and for your patience! Merging now.


Fix/Change the structure for Wire/Codec

Per the Java ILP Call and PR #28, we agreed that the wire package will be renamed to codec, and the current codec package will be pulled up one level so there's only org.interledger.codec with everything currently underneath.
Basically, this is just a package-rename task.

This is complete via #44


Lower MAX_IDENTIFIER_LENGTH to 61 chars

This PR fixes #12732.
Quoting from https://www.postgresql.org/docs/current/static/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS

The system uses no more than NAMEDATALEN-1 bytes of an identifier; longer names can be written in commands, but they will be truncated. By default, NAMEDATALEN is 64 so the maximum identifier length is 63 bytes. If this limit is problematic, it can be raised by changing the NAMEDATALEN constant in src/include/pg_config_manual.h.

63 is OK for creating tables, but for some reason, during ALTER TABLE foo RENAME, Postgres appends an underscore in the autogenerated type name, making the crash when two long names are almost the same:
ERROR:  type "_carto_long_filename_that_almost_matches_another_one_63chars_aa" already exists in schema "xxxxx"

Lowering our MAX_IDENTIFIER_LENGHT to 61 we can be sure we are not running again in problems when RENAMEing tables.

63 is OK for creating tables, but for some reason, during ALTER TABLE foo RENAME, Postgres appends an underscore in the autogenerated type name

Do you have STR for this?

I dug deeper into this because reducing the maximum name lenght is a behaviour change that I don't think that we can afford. For example, we might break maps with sync tables of long filenames.
We have a problem. From the documentation.
CREATE TABLE also automatically creates a data type that represents the composite type corresponding to one row of the table. Therefore, tables cannot have the same name as any existing data type in the same schema.

So, your "for some reason, during ALTER TABLE foo RENAME, Postgres appends an underscore in the autogenerated type name" is actually "on table creation PG creates a type with the same name than the table; as the table name length is equal or bigger than the maximum one, the type name is truncated".
The previous PG documentation excerpt is both the cause and the hint for the solution: "Therefore, tables cannot have the same name as any existing data type in the same schema". If you search for propose_valid_table_name you get 4 usages, and all of them have the same issue: taken_names are the user tables names. As PG documentation say, it should also be the type names.
So, in order to fix this, in this order:
1.- Add an acceptance test that imports two files with the same name than the issue suggest. I think that the expected name for the second file should be carto_long_filename_that_almost_matches_another_one_63chars_a_1, because although there's no table with a matching name (carto_long_filename_that_almost_matches_another_one_63chars_aab), the type that gets generated (_ carto_long_filename_that_almost_matches_another_one_63chars_aa) does exist. cc @javitonino  . This test must fail with the known error.
2.- Fix it by adding existing data types to the list of taken names at the 4 usages. Please minimize code duplication.

👍
Should I close this PR and delete this branch? To start with your proposal in a new one.

So I revert my commits & continue the work in this branch, isn't it?

Yup.

Following your example, we run in the same problem with the _1 and _2 versions:
carto_db_development=# CREATE TABLE public.carto_long_filename_that_almost_matches_another_one_63chars_a_1 (id serial);
CREATE TABLE
carto_db_development=# CREATE TABLE cdb_importer.carto_long_filename_that_almost_matches_another_one_63chars_a_2 (id serial);
CREATE TABLE
carto_db_development=# ALTER TABLE cdb_importer.carto_long_filename_that_almost_matches_another_one_63chars_a_2 SET SCHEMA public;
ERROR:  type "_carto_long_filename_that_almost_matches_another_one_63chars_a_" already exists in schema "public"

PostgreSQL is smart enough to rename conflicting versions when creating new tables, like:
carto_db_development=# CREATE TABLE public.carto_long_filename_that_almost_matches_another_one_63chars_a_2 (id serial);
CREATE TABLE
carto_db_development=# CREATE TABLE public.carto_long_filename_that_almost_matches_another_one_63chars_a_3 (id serial);
CREATE TABLE

This creates the typnames __carto_long_filename_that_almost_matches_another_one_63chars_a and ___carto_long_filename_that_almost_matches_another_one_63chars_ in the pg_catalog.pg_type table.
I've tried renaming first the table, and I got another error with this approach:
carto_db_development=# ALTER TABLE cdb_importer.carto_long_filename_that_almost_matches_another_one_63chars_aab rename to foobar;
ALTER TABLE
carto_db_development=# ALTER TABLE cdb_importer.foobar SET SCHEMA public;
ERROR:  relation "carto_long_filename_that_almost_matches_another_one_63ch_id_seq" already exists in schema "public"

The table changes it's name, but the sequence maintains the name. If I create another table ending in aac, I'll end with a sequence named carto_long_filename_that_almost_matches_another_one_63c_id_seq1.

Following your example, we run in the same problem with the _1 and _2 versions:

Yep, maybe we need a more complex approach. From your failing example:
carto_db_development=# CREATE TABLE public.carto_long_filename_that_almost_matches_another_one_63chars_a_1 (id serial);
CREATE TABLE
carto_db_development=# CREATE TABLE cdb_importer.carto_long_filename_that_almost_matches_another_one_63chars_a_2 (id serial);
CREATE TABLE
carto_db_development=# ALTER TABLE cdb_importer.carto_long_filename_that_almost_matches_another_one_63chars_a_2 SET SCHEMA public;
ERROR:  type "_carto_long_filename_that_almost_matches_another_one_63chars_a_" already exists in schema "public"

As you see that the straightforward approach would be anticipating the type. We shouldn't propose carto_long_filename_that_almost_matches_another_one_63chars_a_2 because it leads to _carto_long_filename_that_almost_matches_another_one_63chars_a_ type, which is taken. Iterating with the current approach should lead to carto_long_filename_that_almost_matches_another_one_63chars__10, which leads to _carto_long_filename_that_almost_matches_another_one_63chars__1 type, which is available.
Did you consider this?
Your attempt of rename + change schema was not bad, but the sequence error is worse, because it's based in our own conventions. I'd prefer to struggle only against posgres', instead of against both of them.
Can you come up with another alternative that doesn't change the current behaviour for working imports?
cc @javitonino

My initial idea when I saw this was that instead of finding tables with matching names to see if we have to rename, we can look at matching only the first 62 characters instead (keeping the truncation at 63). Not 100% sure if this work, but it's what pops to mind.

Please, fill in NEWS.md and then move to acceptance.


upgrade to 2.0.0-beta.12 - angular material and ng - 4.4.x as well

2.0.0-beta.12. ng material deprecated md and replaces with mat.
Also upgrade to ng 4.4.x as well as angular material requires a minimum of ng 4.4.3 .

Issue #56 about upgrading ng to 4.4.x also handled as part of ng material upgradation.

Hi @akkumar
The app top navigation icon and carousel navigation buttons does not look right.

Cheers

work in progress. The buttons do not seem to appear yet.

Hi @bfwg - Please check now. MatButtonModule and MatButtonToggleModule imported as well. Should fix the view of the buttons as well.

Eventually - as compared to the custom css (especially the media selectors in css) all over the place - it would be useful to use angular/flex-layout lib . But that is for a different PR I would imagine.

LGTM 👍  nice job.