[SPARK-13432][SQL] add the source file name and line into a generated Java code

What changes were proposed in this pull request?
This PR adds the source file name and line into a comment of a Java code generated by Catalyst. It would be helpful to quickly identify the original source file from a position where an error occurs during a problem determination of a customer. It supports only DataFrame and SQL.
This PR consists of three parts:

add line information to Origin
pass Origin from a master to executors
add Origin into a comment

This PR adds the information to a comment for existing operations. Other PRs will address the followings:

Add a comment for Dataset
Insert a comment for other places

Here is an example. The original Java program.
object Test {
  ...
  df.filter("v <= 3")
    .filter("v % 2 == 0")
    .show()
  ...
}

Generated Java code
...
/* 031 */   protected void processNext() throws java.io.IOException {
/* 032 */     while (input.hasNext()) {
/* 033 */       InternalRow inputadapter_row = (InternalRow) input.next();
/* 034 */       /* input[0, string] @ filter at Test.scala:23 */
/* 035 */       boolean inputadapter_isNull = inputadapter_row.isNullAt(0);
/* 036 */       UTF8String inputadapter_value = inputadapter_isNull ? null : (inputadapter_row.getUTF8String(0));
/* 037 */       /* input[1, int] @ filter at Test.scala:23 */
/* 038 */       boolean inputadapter_isNull1 = inputadapter_row.isNullAt(1);
/* 039 */       int inputadapter_value1 = inputadapter_isNull1 ? -1 : (inputadapter_row.getInt(1));
/* 040 */       /* ((input[1, int] <= 3) && ((input[1, int] % 2) = 0)) @ filter at Test.scala:23 */
/* 041 */       /* (input[1, int] <= 3) @ filter at Dataset1.scala:22 */
...

How was the this patch tested?
Unit test  (add a test to keep Origin during SerDe)

This PR is derived from an offline discussion with @sarutak .

Jenkins, retest this please

Test build #51664 has started for PR 11301 at commit 0678a66.

Test build #51664 has finished for PR 11301 at commit 0678a66.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51664/
Test PASSed.

ping @sarutak

Thanks @kiszk ! I'll check this later.

Test build #51977 has started for PR 11301 at commit 7671742.

I also add one feature. This is to show a message points out the origin of a generated method when an exception occurs in the generated method at runtime.
An example of a message (the first line is newly added)
07:49:29.525 ERROR org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator: The method GeneratedIterator.processNext() is generated for filter at Test.scala:23
07:49:29.526 ERROR org.apache.spark.executor.Executor: Exception in task 1.0 in stage 2.0 (TID 4)
java.lang.NullPointerException:
        at ...

Here is a part of generated code.
...
/* 031 */   protected void processNext() throws java.io.IOException {
/* 032 */     try {
/* 033 */       while (input.hasNext()) {
/* 034 */         InternalRow inputadapter_row = (InternalRow) input.next();
/* 035 */         /* input[0, string] @ filter at Test.scala:23 */
/* 036 */         boolean inputadapter_isNull = inputadapter_row.isNullAt(0);
/* 037 */         UTF8String inputadapter_value = inputadapter_isNull ? null : (inputadapter_row.getUTF8String(0));
/* 038 */         /* input[1, int] @ filter at Test.scala:23 */
/* 039 */         boolean inputadapter_isNull1 = inputadapter_row.isNullAt(1);
/* 040 */         int inputadapter_value1 = inputadapter_isNull1 ? -1 : (inputadapter_row.getInt(1));
/* 041 */         /* ((input[1, int] <= 3) && ((input[1, int] % 2) = 0)) @ filter at Test.scala:23 */
/* 042 */         /* (input[1, int] <= 3) @ filter at Dataset1.scala:22 */
...
/* 068 */         if (shouldStop()) {
/* 069 */           return;
/* 070 */         }
/* 071 */       }
/* 072 */     } catch (final Throwable e) {
/* 073 */       org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(this.getClass());
/* 074 */       logger.error("The method processNext() is generated for " +
/* 075 */         "filter at Test.scala:22");
/* 076 */       throw e;
/* 077 */     }
...

Closed by my mistake. Reopen this.

Test build #51977 has finished for PR 11301 at commit 7671742.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51977/
Test FAILed.

Test build #52059 has started for PR 11301 at commit 8497208.

Test build #52059 has finished for PR 11301 at commit 8497208.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52059/
Test FAILed.

Test build #52108 has started for PR 11301 at commit f0d2b81.

Test build #52108 has finished for PR 11301 at commit f0d2b81.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52108/
Test FAILed.

Test build #52120 has started for PR 11301 at commit 1a1ff67.

Test build #52120 has finished for PR 11301 at commit 1a1ff67.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52120/
Test FAILed.

The failed test ParquetHadoopFsRelationSuite is due to the lack of short type support in UnsafeRowParquetRecordReader. When we merge PR #11412, no failure will occur.

Test build #52131 has started for PR 11301 at commit 02ebed2.

Test build #52131 has finished for PR 11301 at commit 02ebed2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52131/
Test PASSed.

@sarutak , this is ready for your review now.

O.K. I'll inspect this change.

@davies Can you also check this? Because you have modified code changed in this PR many times.

The information of callsite is helpful in general. The try-catch in generated code just added too much noise. You will see more information when the job fails, I'm not sure how helpful that logging is.
cc @rxin

Note that the try/catch also expands the size of the bytecode.

I agree with @rxin that it includes the size of the bytecode. I can alleviate an increase of bytecode size to create and call a utility routine instead of generating all in the processNext().
@davis and @sarutak, the original motivation of this is that it is not easy to identify the original method of a method that an exception throws. Its name is always the same as processNext(). Is there any good idea to identify the original method name?

I will create another PR to address a feature regarding traceback. This PR focuses on the information of callsite in generated code.

Test build #52388 has started for PR 11301 at commit c763975.

Test build #52388 has finished for PR 11301 at commit c763975.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52388/
Test PASSed.

Addressed all of CRs

Test build #52502 has started for PR 11301 at commit ba2ce60.

Test build #52502 has finished for PR 11301 at commit ba2ce60.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52502/
Test PASSed.

@sarutak and @davies , I resolved conflicts. Would it be possible to review this again?

Sorry for the late reply and I have some comments. I'll leave it soon.

@kiszk It seems this PR covers only Expression. Why don't you cover operators like sort and join too?

Test build #52805 has started for PR 11301 at commit a517e90.

Test build #52807 has started for PR 11301 at commit 0494299.

Test build #52807 has finished for PR 11301 at commit 0494299.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52807/
Test FAILed.

Test build #52809 has started for PR 11301 at commit e08fb66.

Test build #52809 has finished for PR 11301 at commit e08fb66.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52809/
Test PASSed.

Test build #52805 has finished for PR 11301 at commit a517e90.

This patch fails from timeout after a configured wait of `250m`.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52805/
Test FAILed.

@Sarutak, I think that join and sort may generate Java code by own routines. Can I address other operations in other future PRs incrementally?

Jenkins, retest this please

Test build #52823 has started for PR 11301 at commit e08fb66.

Test build #52823 has finished for PR 11301 at commit e08fb66.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52823/
Test PASSed.

Merged build finished. Test PASSed.

@kiszk O.K, let's focus on Expressions in this PR.

I'll leave the rest of feedback comments tomorrow or the day after tomorrow.

Test build #53321 has started for PR 11301 at commit 2965e93.

Test build #53321 has finished for PR 11301 at commit 2965e93.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53321/
Test PASSed.

Test build #53517 has started for PR 11301 at commit 62fa1de.

Test build #53517 has finished for PR 11301 at commit 62fa1de.

This patch fails R style tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53517/
Test FAILed.

Test build #53521 has started for PR 11301 at commit 7e53de4.

Test build #53521 has finished for PR 11301 at commit 7e53de4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53521/
Test PASSed.

Test build #53994 has started for PR 11301 at commit c0b1f47.

Test build #53996 has started for PR 11301 at commit 078b272.

Test build #53994 has finished for PR 11301 at commit c0b1f47.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53994/
Test FAILed.

Merged build finished. Test FAILed.

Test build #53996 has finished for PR 11301 at commit 078b272.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53996/
Test FAILed.

Merged build finished. Test FAILed.

Test build #54044 has started for PR 11301 at commit 3074e65.

Test build #54044 has finished for PR 11301 at commit 3074e65.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54044/
Test PASSed.

Test build #54075 has started for PR 11301 at commit 5f6cda7.

Test build #54075 has finished for PR 11301 at commit 5f6cda7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54075/
Test PASSed.

Merged build finished. Test PASSed.

@sarutak , I resolved conflicts. Could you please review this again?

Thanks. I'll check it soon.

@kiszk Now I'm inspecting this patch and I noticed some defects so far.

I ran a job like as follows.
sc.parallelize(1 to 1, 1).toDF.show

And then, I got generated code like as follows.
/* 006 */ class SpecificSafeProjection extends org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection {
/* 007 */
/* 008 */   private Object[] references;
/* 009 */   private MutableRow mutableRow;
/* 010 */   private org.apache.spark.sql.types.StructType schema;
/* 011 */
/* 012 */
/* 013 */   public SpecificSafeProjection(Object[] references) {
/* 014 */     this.references = references;
/* 015 */     mutableRow = (MutableRow) references[references.length - 1];
/* 016 */     this.schema = (org.apache.spark.sql.types.StructType) references[0];
/* 017 */   }
/* 018 */
/* 019 */   public java.lang.Object apply(java.lang.Object _i) {
/* 020 */     InternalRow i = (InternalRow) _i;
/* 021 */     /* createexternalrow(if (isnull(input[0, int])) null else input[0, int], StructField(value,IntegerType,false)) @ selectExpr at <console>:27 */
/* 022 */     boolean isNull = false;
/* 023 */     final Object[] values = new Object[1];

The code above is about SpecificSafeProjection generated by boundTEncoder.fromRow called in Dataset#collect.
I think the code above is not directly related to query or DSL written by application developers so It's funny that call sites are in the comments.

When columns are described in DSL style like $"column", call sites in comment can be $.
You will reproduce it by the code bellow.
sc.parallelize(1 to 1, 1).select($"value").show

Call sites of multiple jobs can be scrambled. For example, you can reproduce by following code.
val df = sc.parallelize(1 to 1, 1).toDF
df.select($"value").show
df.selectExpr("value").show

After you run the second job above, you can see generated code as follows.
/* 006 */ class SpecificSafeProjection extends org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection {
/* 007 */
/* 008 */   private Object[] references;
/* 009 */   private MutableRow mutableRow;
/* 010 */   private org.apache.spark.sql.types.StructType schema;
/* 011 */
/* 012 */
/* 013 */   public SpecificSafeProjection(Object[] references) {
/* 014 */     this.references = references;
/* 015 */     mutableRow = (MutableRow) references[references.length - 1];
/* 016 */     this.schema = (org.apache.spark.sql.types.StructType) references[0];
/* 017 */   }
/* 018 */
/* 019 */   public java.lang.Object apply(java.lang.Object _i) {
/* 020 */     InternalRow i = (InternalRow) _i;
/* 021 */     /* createexternalrow(if (isnull(input[0, int])) null else input[0, int], StructField(value,IntegerType,false)) @ selectExpr at <console>:27 */
/* 022 */     boolean isNull = false;
/* 023 */     final Object[] values = new Object[1];
/* 024 */     /* if (isnull(input[0, int])) null else input[0, int] @ selectExpr at <console>:27 */

As you can see, select and selectExpr appear in the comment above although select is not used in the second job.
Another case is here.
val df = sc.parallelize(1 to 1, 1).toDF
df.select($"value" + 1).show
df.selectExpr("value + 2").show

...

/* 009 */ final class GeneratedIterator extends org.apache.spark.sql.execution.BufferedRowIterator {

...

/* 034 */       /*** CONSUME: Project [(value#1 + 2) AS (value + 2)#7] */
/* 035 */
/* 036 */       /*** CONSUME: WholeStageCodegen */
/* 037 */
/* 038 */       /* (input[0, int] + 2) @ selectExpr at <console>:27 */
/* 039 */       /* input[0, int] @ select at <console>:27 */
/* 040 */       /* input[0, int] @ selectExpr at <console>:27 */
/* 041 */       int inputadapter_value = inputadapter_row.getInt(0);

...

/* 050 */ }

As I mentioned before, could you make TreeNode serializable?

Sorry, I said we don't need to make TreeNode serializable but it's needed otherwise origin is not serialized and origin information like callsite is not in the comment in the generated code when WholeStageCodegen is disabled.

@sarutak , thanks, I will address them. Since I am making a business trip next week, its progress may be slow.
I realized one big issue when I tried to resolve conflicts. In the latest code, ASTNode, whose constructor captures a stack trace, was dropped.

Test build #55486 has started for PR 11301 at commit e073762.

Test build #55486 has finished for PR 11301 at commit e073762.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55486/
Test FAILed.

Merged build finished. Test FAILed.

Test build #55497 has started for PR 11301 at commit 02d75c7.

Test build #55497 has finished for PR 11301 at commit 02d75c7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55497/
Test PASSed.

Test build #55576 has started for PR 11301 at commit d3c6ae8.

Test build #55576 has finished for PR 11301 at commit d3c6ae8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product with Serializable

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55576/
Test FAILed.

Merged build finished. Test FAILed.

@sarutak, for 4), I made TreeNode serializable. However, I found six errors in #55576. For example,  in ALSSuite.different block setting, I found an exception org.apache.spark.SparkException: Task not serializable.

Test build #55594 has started for PR 11301 at commit e2f8d98.

Test build #55594 has finished for PR 11301 at commit e2f8d98.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55594/
Test PASSed.

Test build #55635 has started for PR 11301 at commit 6e0b7c5.

Test build #55635 has finished for PR 11301 at commit 6e0b7c5.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55635/
Test FAILed.

Test build #55687 has started for PR 11301 at commit d22d610.

Test build #55687 has finished for PR 11301 at commit d22d610.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55687/
Test PASSed.

Merged build finished. Test PASSed.

@sarutak , could you please review this again? I addressed (1), (2), and (3), but cannot make it for (4).
I added a test suite.

@kiszk Thanks for the update. I'll check it next Monday or Tuesday.

Test build #56529 has started for PR 11301 at commit 634518c.

Test build #56534 has started for PR 11301 at commit 73caad5.

Test build #56529 has finished for PR 11301 at commit 634518c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56529/
Test PASSed.

Merged build finished. Test PASSed.

Test build #56534 has finished for PR 11301 at commit 73caad5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56534/
Test PASSed.

ping @sarutak

@kiszk  Sorry I was going to  review last week but I didn't have enough time. I might make time this weekend.

No problem, it seemed to be some conflicts in this PR last week. I will continue to resolve conflicts as soon as possible.

Test build #57153 has started for PR 11301 at commit 2ce743a.

Test build #57153 has finished for PR 11301 at commit 2ce743a.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57153/
Test FAILed.

Jenkins, retest this please

Test build #57156 has started for PR 11301 at commit 2ce743a.

Test build #57156 has finished for PR 11301 at commit 2ce743a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57156/
Test PASSed.

Test build #57730 has started for PR 11301 at commit 6ecc00e.

Test build #57730 has finished for PR 11301 at commit 6ecc00e.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57730/
Test FAILed.

Merged build finished. Test FAILed.

Test build #57735 has started for PR 11301 at commit 1149796.

Test build #57735 has finished for PR 11301 at commit 1149796.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57735/
Test PASSed.

@kiszk
Now I'm reviewing the latest patch.
I'll leave comments tomorrow or the day after tomorrow.

Test build #57932 has started for PR 11301 at commit aa64405.

Test build #57932 has finished for PR 11301 at commit aa64405.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57932/
Test PASSed.

Merged build finished. Test PASSed.

@kiszk I've left some comments.
Further more, I noticed a big problem of this change.
The call site to be embedded into generated source code includes line-number.
If we call a same queries multiple time, generated codes are essentially equal except for the line-numbers.
Because of the difference of the comments, those generated source codes are both compiled even though they are essentially equal.
May be this problem can be resolved by #12979 .

Test build #58107 has started for PR 11301 at commit 1b1d688.

Test build #58107 has finished for PR 11301 at commit 1b1d688.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58107/
Test FAILed.

Merged build finished. Test FAILed.

Test build #58109 has started for PR 11301 at commit df774eb.

Test build #58109 has finished for PR 11301 at commit df774eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58109/
Test PASSed.

Test build #59847 has started for PR 11301 at commit df5820b.

Test build #59848 has started for PR 11301 at commit df774eb.

Test build #59847 has finished for PR 11301 at commit df5820b.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59847/
Test FAILed.

Test build #59851 has started for PR 11301 at commit 597b732.

Test build #59848 has finished for PR 11301 at commit df774eb.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59848/
Test PASSed.

Test build #59851 has finished for PR 11301 at commit 597b732.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59851/
Test PASSed.

(Hi all, i am just wondering where we are on this)

I have less bandwidth for this since I am busy for others. Let me close for now.


changed error stylings,history and textbox

added sounds to keys and error stylings
also modified history table and textbox cursor

There is some functional logic as mentioned above, also merge conflicts to be fixed @kmahesh18

conflicts and function issues are solved .. recheck pr

When you click on a number, instead of displaying once it gets displayed twice. @kmahesh18
As per the below image number 8 was clicked once

Ohh...sorry...I'll update the functionality and pr


Remove explicit setting default connector.blockchain.rpc

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (f060841) 63.09% compared to head (8ef58ff) 63.09%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #31   +/-   ##
=======================================
  Coverage   63.09%   63.09%
=======================================
  Files          22       22
  Lines         897      897
=======================================
  Hits          566      566
  Misses        325      325
  Partials        6        6

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Automatically save some route points to the user's list?

Should we consider requesting the user location even when the app is inactive, and automatically save some locations when user is on a hitchhiking trip? This would cause the route to be drawn on the map a bit more realistic - but is it really worth it do work on?
Issue #42 should be kept in mind if we decide to automatically save spots.

No


Tweaks to NERSC CI needed for TXPipe

Hi Stuart, Heather,
Many thanks for the NERSC CI instructions, which look like they must have been a lot of work to figure out!
I had to make two small changes to get this to work for TXPipe:

I needed to add   GIT_SUBMODULE_STRATEGY: recursive to the variables in the target repo's .gitlab-ci.yml because we use submodules for some dependencies. Probably worth putting this in by default since it shouldn't hurt if unused.
I needed to change --form ref=main to master in the mirror repo's .gitlab-ci.yml - might be worth mentioning something in the docs.

Cheers,
Joe

both of these seem fine.
I'd like to test what happens when GIT_SUBMODULE_STRATEGY  is set to recursive for repos without submodules.  Including some short comment in the .gitlab-ci.yml might be helpful.
I have been wondering about promoting migration to main rather than master as the primary branch, though understandably this may be difficult for repos that have developers who may have the code locally.  This probably isn't the place to deal with that issue. Easy enough to just mention how to deal with primary branch names other than 'main

Thanks Joe for the comments.
I have tried to make it a bit clearer that main is the default branch in the CI files and how to change it (Also the GITHUB_SOURCE_BRANCH var in the yml file wasn't actually being called in the curl command so I've updated that).
I've also added a not in the docs about GIT_SUBMODULE_STRATEGY: recursive.
Thanks again :)


Import error for libscip.so.6.0

I followed the install instructions to build SCIPOptsuite as follows:
mkdir build && cd build
cmake3 ..
make READLINE=false
make check
make install INSTALLDIR=/usr/local

then I exported the install path as follows:
export SCIPOPTDIR=/usr/local

then I installed PyScipOpt using pip install pyscipopt
I do not get any errors, but when I do from pyscipopt.scip import Model I get the error:
ImportError: libscip.so.6.0: cannot open shared object file: No such file or directory

I can see libscip.so.6.0 in the /usr/local/lib folder and all other folders seem to be ok (bin, include). I can also launch the scip terminal via scip. Not sure why the import in my python script is failing.
I am using CentOS7 with Python 3.6.

OK, i see that I forgot to add /usr/local to my PATH, so
export PATH=$PATH:/usr/local

solved it for me.

I got this error today too for 8.0 and this fixed it:
ldconfig /opt/pyscipopt/lib

Where ldconfig /opt/pyscipopt is where I executed the install script https://www.scipopt.org/download.php?fname=SCIPOptSuite-8.0.3-Linux-ubuntu.sh


feat: Added CI job for client doc building & installation check

This PR introduces the following features:

CI job for package installation & flake8 verification
Sphinx autodoc building files for the API client
CI job to build the documentation, deploy it, verify that the doc has been deployed

Here is how the documentation renders:

Any feedback is welcome!

@florianriche not sure if you had the notification but I solved the EOF :ok_hand:


[enhancement] prevent working in production

It is possible (although, if I recall, advised against) to use DDEV as a production environment, but even a direct deployment from a git repository that holds the .ddev folder and with it, the profiler files might encounter a situation in which the profiler is accidentally running.
An example where this might happen, is e.g. on platform.sh, where I noticed that a standard composer project, installs with composer dev requirements.
It might be good to have a way to ensure the profiler isn't running in a production environment, even if the required xhgui php files exist in the expected location and/or the php-profiler package is installed.

I guess IMO people have to sort these things out if they use Casual hosting or use it for other similar purposes. (Lots of people use it for github tests, etc.)

PRs are welcome.
My feeling is DDEV is for local (or cloud) development and testing and therefore it's not the addons responsibility.
Developers are free to use ddev service disable or wrap the profiler commands with ENV checkers.
Let's get the addon to a solid place first and then see where the currents flow.
Closing but happy to open with demand.


chore: Devcontainer config and documentation

Asana task: Consider using a VSCode dev container for development
This adds config and documentation for running the app in a VS Code dev container, which confers the advantage of quick, repeatable builds and much easier version upgrades.
I manually tested the setup steps from both a freshly cloned repo and an existing project that's been built/run under local development previously.

Awesome work! Thanks a ton for tackling this. READMEs look great.


Simpler and more readable code for isSelected method

This is a thing of beauty?! Haha amazing thanks!


Review conflicts from 1.6-release/master merges

Restore any changes lost from master when pulled into release-1.6.

This change is

This should address any remaining concerns raised in PR #3250, see https://github.com/kubernetes/kubernetes.github.io/pull/3250#issuecomment-293098532


Added Lite option to EfficientNets.

Description
This PR modifies the existing code for EfficientNet inside tf.keras.applications so that it allows to use the Lite [1] variants of the model. The Lite variants were introduced in the original EfficientNet repository and had the following differences:

Removed squeeze-and-excite (SE).
Replaced swish with RELU6.
Fixed the stem and head while scaling models up.

I am also referencing my helper repository [2], that uses existing Tensorflow code, for direct conversion of .ckpt files to Keras .h5 variants.
Motivation
The existing Lite variants of EfficientNets are only available in original repository and Tensorflow Hub [3]. These are written in Tensorflow 1.x and hence do not allow for flexible use with Keras.
The option to add Lite variants has been requested few times in Tensorflow Ecosystem:
https://github.com/tensorflow/tensorflow/issues/45091
https://github.com/tensorflow/hub/issues/751
This PR addresses the first issue and partially solves the second one.
Usage / Changes
In the original version one could call EfficientNet models as such:
import tensorflow as tf

model = tf.keras.applications.efficientnet.EfficientNetB0()

With this PR it would be possible to do:
import tensorflow as tf

model_lite = tf.keras.applications.efficientnet.EfficientNetB0(lite=True)

This behaviour is not introducing any breaking changes to existing codebase and is very similar to minimalistic option in MobilenetV3Small and Large.
Todos:
This PR is sadly a not finished work and to make it fully compatible with existing codebase I would need help from someone on the Tensorflow team. Most notably there are two things to be discussed:

Add converted weights to Remote Storage + add lite variants weights hashes so they can be automatically loaded. I can provide the converted weights (top and notop variants) or they can be obtained by my helper repository [2].
Prevent the user from creating lite variants of the models B5 and higher. These are not supported in the original repository, yet still the user can pass the lite=True flag via kwargs. This will throw rather cryptic error that the number of layers does not match in architecture vs weight file. Perhaps a different error message would be better?

References
[1] https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet/lite
[2] https://github.com/sebastian-sz/efficientnet-lite-keras
[3] https://tfhub.dev/s?q=efficientnet lite

@googlebot I fixed it.

This is promising!

As @ameyparanjape noticed, there is a deserialisation error if the lite model gets saved in h5 format and loaded back. In more detail:
model = EfficientNetB0(lite=True)
tf.keras.models.save_model(model, "path_to_model.h5", save_format="h5")

# Error upon loading
loaded = tf.keras.models.load_model("path_to_model.h5").

The above will throw:
ValueError: Unknown activation function: relu6.
I tried replacing the activation in line 286 with layers.ReLu(6.) similar to MobilenetV2 and also with tf.nn.relu6 but both throw similar errors.
I am looking for possible fixes (without making too many changes), but I'm also open to suggestions.

As @ameyparanjape noticed. These models will throw errors when loading them from keras:
model = EfficientNetB0(lite=True)
tf.keras.models.save_model(model, "model.h5", save_format="h5")

loaded = tf.keras.models.load_model("model.h5")  # <--- this will throw error.

The problem is in the activation function, and more precisely, in Activation(relu6). Other variants like Activation(tf.nn.relu6) or Activation(ReLU(max_value=6.)) will also fail to load back.
The closest I came to solving this is via introduction of helper function:
def get_activation(activation, name, lite):
    """Returns activation layer for given model variant."""
    if lite:
        return layers.ReLU(max_value=6, name=name)
    else:
        return layers.Activation(activation, name=name)

And replacing every Activation with get_activation. I am not sure if this is clean enough though.
At this point, when the scope of changes got bigger, I'm wondering if it isn't better to just have a separate function for EfficientNetLite (which on the other hand would introduce some duplication). Perhaps maybe even separate models (which would fit with the existng testing code base).
Let me know how do you feel about this.

@sebastian-sz  Can you please resolve conflicts? Thanks!

@gbaned
Hi! Thank you for the notice.
I updated the repository, the conflicts should be resolved.

@gbaned
Hello, there has been almost 2 months with not much activity. I know that there are other, more important PRs, going through. I don't mind waiting for a review, but could you tell me, whether you would be interested in functionality presented here?
Thanks!

@sebastian-sz  Sorry for the delay, it is under review phase. Thank you!

@gbaned Understood, thanks for the info!

Sorry for the long wait. I was fully occupied by other issue for quite a while, and wasn't able to review this PR.
Also to verify the correctness of this PR, I was trying to setup some internal test to let the model run the imagenet dataset (mainly to verify the accuracy number). Do u have the converted weight file available so that we can use it for verification?

No problem, thanks for looking into this.
Yes I have the converted weights on Google Drive:
https://drive.google.com/file/d/1rcUoNjpUYhoOIgz2VFsfVGFFyNQK5PQ4/view?usp=sharing
One more thing:
These weights are not hosted anywhere, and are not referenced in the code. Calling
m = tf.keras.applications.efficientnet.EfficientNetB0(lite=True)
will raise error. However,
m = tf.keras.applications.efficientnet.EfficientNetB0(lite=True, weights=None)
m.load_weights(...)

will work.

@qlzh727 Any update on this PR? Please. Thanks!

@qlzh727 Given current separation of Keras and TF should I close this PR and resubmit it in the keras repository?

Sorry for the long wait. Yes, the keras code has been removed to keras-team/keras, and this PR will need to be ported there. Thanks.

Understood, I am closing this PR for now. Given more time I will reopen it in the Keras repo.


tectonic: 0.1.9 -> 0.1.11

Motivation for this change

handles unicode filenames correctly (since 0.1.10)
switch to hopefully more reliable permantent URL for the default bundle (in 0.1.11)

Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Assured whether relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

/cc @lluchs

@GrahamcOfBorg build tectonic

I notice this hasn't been updated since 0.1.8 on 18.09.

Yeah, a backport would be appreciated. I can make a PR for one, if wanted.

Shall I backport both?

What do you mean by "both"? 0.1.9 as well as 0.1.11? I doubt that's needed, let's just backport the latest version. Or did you mean something else?


Cannot scrape URI on a different port

Hi there, thanks for a very useful project. I noticed an issue running this alongside NGINX with VTS.
We expose the VTS status reports on a virtual server running on port 8080, with IP whitelisting to ensure they can only be accessed from localhost, for security. This is using the following NGINX config (vhost_traffic_status_zone is specified elsewhere):
    server {
        listen      8080;
        server_name localhost;

        # Allow localhost (for health checks)
        allow <IP_ADDRESS>;
        deny all;

        # Expose VTS metrics (only on localhost for security)

        location = /vts/format/html {
            # Exposes vhost traffic status as a HTML report
            vhost_traffic_status_display;
            vhost_traffic_status_display_format html;
        }
        location = /vts/format/json {
            # Exposes vhost traffic status as a JSON report (the HTML format
            # relies on this URL to retrieve data)
            # vts exporter also scrapes from this URL
            vhost_traffic_status_display;
            vhost_traffic_status_display_format json;
        }
    }

We then run the VTS exporter via the Docker Hub image (digest 8d14166508a5b19ccfd8d75b8933c52f327c7ba5fe728267cd6dc2eb869fd049), specifying NGINX_STATUS=http://localhost:8080/vts/format/json. This URL works fine with curl, but when trying to retrieve metrics, the exporter logs fetchHTTP failed parse "http://localhost:8080/vts/format/json": first path segment in URL cannot contain colon.
This is potentially to do with a change to URL parsing in Go 1.8, although I've not investigated fully: golang/go#19297
Our workaround for now is just to move the vts endpoints to a virtual server listening on port 80, but it'd be really useful to be able to use a different port. Thanks!

I am sorry but it's seems ok in my mac.

download and run

docker pull sophos/nginx-vts-exporter
docker run  -ti -p 9913:9913 --rm --env NGINX_STATUS=http://localhost:9090/vts/format/json sophos/nginx-vts-exporter

nginx configuration

server {
        listen      9090;
        server_name localhost;

        location = /vts/format/html {
            # Exposes vhost traffic status as a HTML report
            vhost_traffic_status_display;
            vhost_traffic_status_display_format html;
        }
        location = /vts/format/json {
            # Exposes vhost traffic status as a JSON report (the HTML format
            # relies on this URL to retrieve data)
            # vts exporter also scrapes from this URL
            vhost_traffic_status_display;
            vhost_traffic_status_display_format json;
        }
}

logs

2017/08/08 02:03:19 Starting nginx_vts_exporter (version=0.4, branch=fix-docker-error, revision=0f3dbb44a86340d65bf3d6abbcc0ee88663cb419)
2017/08/08 02:03:19 Build context (go=go1.8, user=Administrator@LS--20151110SAS, date=20170316-03:16:26)
2017/08/08 02:03:19 Starting Server at : :9913
2017/08/08 02:03:19 Metrics endpoint: /metrics
2017/08/08 02:03:19 Metrics namespace: nginx
2017/08/08 02:03:19 Scraping information from : http://<IP_ADDRESS>:9090/vts/format/json


Update compatible networks list for BullaBanker safe app

Update compatible networks list for BullaBanker safe app

Type

Update

Compatible Networks

- Rinkeby
- Gnosis Chain
- Polygon
- Avalanche (new)
- Aurora (new)
- Arbitrum (new)

Desired date
March 4th

Tested the app on Avalanche, Aurora and Arbitrum and should be all green from our end to add BullaBanker support for these chains on our next deployment.
Cheers! ;)

Thanks, Jago!

App should be shortly available in those new networks by default.
Thank you!

Morning guys! I can see that safe connection is not working on avalanche chain. Opening the app is prompting me to connect to a safe:

Could you please have a look?
Thank you :)

Hey thanks for bringing this bug to our attention @JagoFigueroa ! We just deployed a fix for this. Should be good to go

Working like a charm. Thank you for your quick action! ;)


Integration with 'collection-helpers'

Hello and thanks for nice lib!
Is any way to use 'collection-helpers' module
( https://atmospherejs.com/dburles/collection-helpers )
with meteor-rxjs?
Thanks!

Create all of you helpers, then use MongoObservable.fromExisting in order to wrap the instance of the collection.
Example:
let books = new Mongo.Collection('books');

books.helpers({
  author() {
    return Authors.findOne(this.authorId);
  }
});

export const Books = MongoObservable.fromExisting(books);

@dotansimha thanks!


DW-5393 policies for cross account OpsMI bucket access

this will be merged as part of DW-5446

this will be merged as part of DW-5446


TypeLoadException: Could not load type 'Firebase.MonoPInvokeCallbackAttribute'

I am receiving this error, when building to Android after adding Firebase.Messaging lib.
TypeLoadException: Could not load type 'Firebase.MonoPInvokeCallbackAttribute' from assembly 'Firebase.Messaging'.
System.MonoType.GetMethods (BindingFlags bindingAttr) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System/MonoType.cs:204)
UnityEditor.Build.BuildPipelineInterfaces.InitializeBuildCallbacks (BuildCallbacks findFlags) (at /Users/builduser/buildslave/unity/build/Editor/Mono/BuildPipeline/BuildPipelineInterfaces.cs:182)
UnityEditor.BuildPlayerWindow:BuildPlayerAndRun()

any clues ?
Firebase SDK version 4.2.1 ( I am using Analytics, Database, Auth ) & they were working fine.
Unity version 2017.1.1p4
Thanks

Not sure where is the error,
I changed some settings and now worked,
But now I have the
Too many field references to fit in one dex file: 81486; max is 65536.
I think this is easy to solve by googling it :)

Was able to get it to run ( by exporting Android studio project )
anything else,, fail :(

@MohHeader I assume the problem you've run into when trying to get your application to run is due to the dex limit for Android applications.  You need to proguard your project.  We have an experimental feature in development in the Play Services Resolver called "Gradle Prebuild" that should eventually help with this sort of thing.  TBH You'll run into exactly the same problem when including any non-trivial number of Android libraries in a Unity project.  I would be nice if Unity updated their build system to build with Gradle and enable proguard for internal (i.e non-export) builds.


[QUESTION] Reusable Components

Very interesting concept! How would one create reusable named components?

Nothing yet! I have some ideas on this, but it requires a bit of fiddling, truth be told.
I'm leaning towards Web Components + HTML string templates being the way to go.
Using Vite, it might look something like:
<!-- counter.component.html -->
<button data-on-click="updateCount()">Add one to {{count.value}}</button>

/* counter.component.ts */
import template from "./counter.component.html?raw";
function Counter() {
  let count = createState(
    0
  );

  function updateCount() {
    count.value++;
  }

  return {
    count,
    updateCount
  }
}

Counter.selector = "app-counter";
Counter.template = template;
registerComponent(Counter);

/* app.component.ts */
function App() {
}

// Register with the same name as `data-island-comp`
App.selector = "app-app";
App.template = `<app-counter></app-counter>`;
registerComponent(App);
render()


Everything breaks if IsLocal already exists in the animator as a bool (rework)

If IsLocal is already used as a boolean in the controller, OSCmooth should not add it as an additional float parameter as the duplicate parameter breaks the controller. If the type must be a float, and the controller already contains it as a bool, OSCsmooth MUST not only update the type of the existing parameter, it must also walk the entire controller (including sub-state machines) updating the type of every transition condition using IsLocal to swap it from a boolean condition to a float condition.

Issue reported by a user in the vrcfury discord.

Will be fixed for release

Fixed in c065d497baff43065916c890ca1e740549898802, will contextually use an existing IsLocal parameter to generate trees.


Create kubescape.yml

Close #36 Added kubescape actions file to the project

Thank you for your contribution. However, this project doesn't get deployed and has no kubernetes config for checking. So I will close this PR.


Industry Docs

Bootstrapped from existing design. To be used in Project Websites later on.

https://github.com/erpnext/foundation/pull/174


useAccountManager popup improvements

Few ideas I had for improving useAccountManager, mostly when using popup.

improved status checking

check if popup window is currently open
something like const {popupOpen: boolean} = useAccountManager()
for example: this would make it easy to display a loader on the base website while the authentication is happening in the popup window (i.e. {popupOpen && })

callback events

something like useAccountManager(options, onAccountAdded : () => {}, onAccountRemoved: () => {}, onAccountsUpdated:: () => {},  onAuthCancelled: () => {})
ex: in table editor, after authentication is finished, i want to use a callback to execute some cmds. here's how i do it now:

@samMatenaer These are great recommendations, @brandonlammey any thoughts?


False Positive

The product exposes sensitive information to an actor that is not explicitly authorized to have access to that information.
Mitigation Tip : The product exposes sensitive information to an actor that is not explicitly authorized to have access to that information.
Code : else -> applyThemeMode(
Processed Code : else -> applyThemeMode(
Code Vulnerability Probability : 0.7864081765216437
Probability Breakdown of Vulnerable Code Words : [('else', 0.538470957736102), ('applyThemeMode', -0.18929641468221153)]
Predicted CWE ID Probability : 0.56
Probability Breakdown of CWE Related Vulnerable Code Words :
More Details : https://cwe.mitre.org/data/definitions/200.html

Code:
when (mode) {
            AppCompatDelegate.MODE_NIGHT_NO -> applyThemeMode(
                AppCompatDelegate.MODE_NIGHT_YES,
                R.drawable.ic_mode_night_default_black
            )
            AppCompatDelegate.MODE_NIGHT_YES -> applyThemeMode(
                Settings.MODE_NIGHT_DEFAULT,
                R.drawable.ic_mode_night_no_black
            )
//SELECTED SNIPPET
            else -> applyThemeMode(
                AppCompatDelegate.MODE_NIGHT_NO,
                R.drawable.ic_mode_night_yes_black
            )
        }
//////////

Please try with Android apps written in Java. The model trained using the latest LVDAndro dataset contains training code snipptes mainly from Apps written in Java.


Codegen error: cannot find type Time in module super

I get the following error when building my project. I have a huge schema (84MB), which sadly I can't share. I'm not sure where the issue lies. How could I debug this further?
error[E0412]: cannot find type `Time` in module `super`
  --> src/main.rs:10:10
   |
10 | #[derive(GraphQLQuery)]
   |          ^^^^^^^^^^^^ not found in `super`

The message looks like Time is a custom scalar. The custom scalar has to be defined in the scope where the struct under derive is located. So if Time is serialized as a string for example, a simple type Time = String; next to the query struct should work.
We should document this better - I'll add it to the milestone for the coming release.

Yep, that fixed it!


perf: implement lazy load for eszip entities

What kind of change does this PR introduce?
Enhancement
Description
This PR changes the way opaque data is stored in eszip and makes allow lazy-load for individual data stored in eszip.
Sample
import { createClient } from 'npm<EMAIL_ADDRESS>console.log(typeof createClient);
Deno.serve((_req) => new Response("Hello, world"));

Metrics (Observed lowest value in local)

Previous
This PR
Change (%)

Bundle Size
22MiB
6.3MiB
-349.2%

Boot Time
1939ms
84ms
-2308.3%

Notes
It supports runtime migration for bundles created before this PR. However, performance will revert back to what it was before, so I recommend regenerating the bundle.
...WIP

Note:
Currently, a single entity in eszip contains a big opaque data for virtual fs.
I think we need to bring it up to the eszip entity level to improve performance.

Note:
Once this PR is merged, it is worth looking closely to ensure that it is completely compatible with the previous eszip implementation.
cc @laktek @andreespirela


Missing xmlversion.h ?

Can't build on a windows system.  mkmf.log shows a few lines of --
fatal error: libxml/xmlversion.h: No such file or directory
Thanks

You need to have libxml installed in order to build libxml-ruby.


Folder for thunderstore uploads

So you can simply paste in the DLL of the plugin and then just zip this folder. I'm assuming you already have a folder for this but I thought keeping it in the Git repository would be nice.
The new icon is from the original artist of Siren Head, and it's his first picture of it. Definitely copyrighted.

I usually install the mod, make changes, and upload it again because all the files are there already. This will be useful though, thank you!

Can probably also do something with GitHub actions to automatically build and package a zip to upload to thunderstore. I've got my mod pack doing that with a simple upload artifact


Pylint support. Replace pass with type for handling magics

This is required to support pylint. Using pass causes pylint to raise unnecessary pass warnings. Instead if we use builtin function type we are clean of any flake8 and pylint warnings.

Pending

Pylint tests
Confirm we don't break any of the already supported tools

Input
Though %%time is cell magic, having it closing the function/code seems to look normal than having two lines separating the magic which makes the cell look odd.
%%time
def func():
     pass

Current behavior
After black formats current code, we have two lines separating the cell magic and python function which looks odd.
pass  # nbqa <token>

def func():
     pass

New behavior
Cell magics will become comments. After black formats current code, comments are not moved two lines apart
# CELL_MAGIC %%time <token>
def func():
     pass

Nice! This would be a breaking change for nbqa black but I think it's OK, this is better, and also this library is very young.
Did __import__( not work?

Did __import__( not work?

I used __import__. __import__ template alone takes up around 60-65 characters. When I tried solving https://github.com/nbQA-dev/nbQA/issues/279,  I ran in to the risk of line length exceeding 90 chars sometimes.
Once this happens black splits the statement into multiple lines which makes replacing the original statement a bit harder.
In case of linters, we would get line length exceeded warnings.
So I resorted back to using builtin function type as it takes just 4 characters.

But we still have those issues of black splitting these replacement statements in to multiple lines and linter warnings. But the probability of running in to those is low compared to __import__.
The approach I used to solve https://github.com/nbQA-dev/nbQA/issues/279 first extracts only the python code from the statement. Then parse the python code snippet using ast.parse and check if the python code contains any Call node. I used Call node type since most of the time we either import a function or class. I know this is not the best of approaches. But atleast we can start with this. If the python code contains a  Call node, I would place the entire code inside the template string callable(lambda _: {python code}). Impl can be found here https://github.com/girip11/nbQA/blob/b83aaadb9b6f334a719d00ddc74abd18b396cb6d/nbqa/handle_magics.py#L15

I see, thanks for explaining.
Would it be possible to also put the magic in a comment, so that the only valid Python code here is type() and so there's no risk of black splitting it into multiple lines?

Would it be possible to also put the magic in a comment, so that the only valid Python code here is type() and so there's no risk of black splitting it into multiple lines?

Sorry I didn't get you. Could you elaborate on this?

You currently have
'type(""" {magic} """)  # {token}'

but I think black could potentially split it into
'type(
    """ {magic} """
)  # {token}'

How about something like
'type()  # {magic} {token}'

? I don't think there's any scenario under which black would split this

'type()  # {magic} {token}'

Good point. But we can't have just type() without any argument. mypy would raise error. Just checked. And the moment we add type(1) black is going to split the lines.
I am inclining towards handling multi line split in replace_magics with the help of token in the string. We can take those changes in a separate PR, since this kind of scenario is rare.

Good point. But we can't have just type() without any argument. mypy would raise error.

:rofl: it feels like flattening a carpet that's too big for the floor

@girip11 how about int()?

@girip11 how about 'int() # {magic} {token}'?

Thanks @MarcoGorelli. This would work with currently supported/tested tools.
Meanwhile I was able to come up with a solution using regex matches. Even if modifications done those temporary placeholder lines we would still be able to restore the original line magics as long as the token is untouched.
I used type() and I used the format suggested by you 'type({token}) # {magic} {token}' but truncated the {magic} to first 10 characters.
By using the token in the beginning and at the end of the code, we would be able to replace the entire code across multiple lines using regex match. I will raise next iteration of the PR with those changes incorporated. @MarcoGorelli  and @s-weigand feel free to take a look at those changes though I have kept the PR still in the draft mode.
With tests added for pylint, I will make the PR ready for review.

Codecov Report

Merging #302 into master will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##            master      #302    +/-   ##
==========================================
  Coverage   100.00%   100.00%
==========================================
  Files            9         6     -3
  Lines          418       215   -203
==========================================
- Hits           418       215   -203

Impacted Files
Coverage Δ

nbqa/handle_magics.py
100.00% <100.00%> (ø)

nbqa/notebook_info.py
100.00% <100.00%> (ø)

nbqa/replace_source.py
100.00% <100.00%> (ø)

nbqa/save_source.py
100.00% <100.00%> (ø)

nbqa/cmdline.py

nbqa/__main__.py

nbqa/config_parser.py

nbqa/config.py

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 448ef07...bf1c2ef. Read the comment docs.

By using the token in the beginning and at the end of the code, we would be able to replace the entire code across multiple lines using regex match.

Clever! I'm normally scared of regexes, but this seems like a safe/legitimate use of them!

Nice!
Generally, this looks really good
Regarding

I will raise next iteration of the PR with those changes incorporated.

do you mean this would be a separate PR? Because currently, if I run nbqa black tests/data/notebook_with_indented_magics.ipynb --line-length=15 --nbqa-mutate then the second cell becomes
from random import (
    randint,
)

if (
    randint(
        5, 10
    )
    > 8
):
    type(
        0xD0136BEE
    )  # %time prin 0xD0136BEE

TBH I'd rather not knowingly introduce a regression...I don't know how common it is to run black with a short line-length, but still, I guess there may be legitimate uses for it and I'd rather not break such notebooks with mysterious hashes :smile:

@girip11 if I apply the following diff
diff --git a/nbqa/handle_magics.py b/nbqa/handle_magics.py
index e0ad7c6..e5ee464 100644
--- a/nbqa/handle_magics.py
+++ b/nbqa/handle_magics.py
@@ -73,8 +73,8 @@ class MagicHandler(ABC):
     # `{token}` is not used as `String` because with different formatters (e.g: yapf)
     # we would run in to formatting issues like single quotes formatted
     # to double quotes or vice versa. `{token}` is used as hexadecimal number.
-    _MAGIC_TEMPLATE: str = "type({token})  # {magic:10.10} {token}"
-    _MAGIC_REGEX_TEMPLATE: str = r"type\({token}\).*{token}"
+    _MAGIC_TEMPLATE: str = "int()  # {token} {magic:10.10} {token}"
+    _MAGIC_REGEX_TEMPLATE: str = r"int\(\)  # {token}.*{token}"

     def replace_magic(self, ipython_magic: str) -> MagicSubstitution:
         """

then all the current tests still pass, and nbqa black tests/data/notebook_with_indented_magics.ipynb --line-length=15 --nbqa-mutate keeps the notebook in a good state.
Would this work for you? Is there an advantage of type (or disadvantage or int()) which I'm missing?

TBH I'd rather not knowingly introduce a regression...I don't know how common it is to run black with a short line-length, but still, I guess there may be legitimate uses for it and I'd rather not break such notebooks with mysterious hashes.

Good catch. There are two ways to go about the line length issue.

Fix existing regex to handle spaces r"type\s*\(\s*{token}\s*\).*{token}"
Use the int() solution suggested that  you suggested.

I was using type() just because i thought it was safer compared to int(). What if some tool throws warnings on its usage?
We could also use int({token}) regex with spaces handles. So the reported formatter issue would go away.
What are your thoughts on this?

I will raise next iteration of the PR with those changes incorporated.

I have made all the changes. This PR is ready of review. Only pylint usage in docs needs to be updated.

(we might still get a false positive from flake8 like this, but IMO a false positive from a static analysis tool is far less undesirable than modified output from a formatter, which I fear would lose people's confidence in our tool)

I agree with you on this. I just thought who would use line length to be as less as 50. So I just ignored space matches in the regex. Fixing the regex would solve the formatting issue.

Fixing the regex would solve the formatting issue.

Sure, I'm happy to go with that! :+1:

I will add this short line length as a test case for black in order to avoid regressions of this sort in the future.

Cool, this one works! Thanks, excellent work! I think we're ready for a 0.3.0 release (as this is a new feature), I'll just try to get #305 in as well

Awesome, look good!!


Cannot update momentjs:moment meteor package

When performing meteor update I get the following message:
=> Errors while upgrading packages:

While checking for<EMAIL_ADDRESS>error: No compatible binary build found for this package. Contact the package
author and ask them to publish it for your platform.

I am on OS X 10.11.4

Having the same problem! Just updated to Meteor <IP_ADDRESS>

1 !

+1

Closing this as a duplicate of #3130. We haven't forgotten you, I promise. We just don't need two issues.

+1


Explorer Zip uncompress very slow

Isssue description
The zip uncompression in Windows Explorer is not very performant. Depending on the zip archive it is so painfully slow that it is unusable. It does not utilize modern multi-core CPU cores and does IO in incredibly inefficient ways.
This is a very known issue (https://twitter.com/BruceDawson0xB/status/1399931424585650180, https://devblogs.microsoft.com/oldnewthing/20180515-00/?p=98755) and has been for years. There is no real diagnosis here need: The library doing the unzipping, is from 1998 and nobody at Microsoft knows how it works.
Developers work with zip files very often. Want to quickly look at an archive of zipped log files from something? Might as well make a coffee while you wait.
For me the process is often like this: Start unzip from Explorer, get frustrated that it takes so long, open 7-zip (or similar), uncompress there again, 3rd party tool finishes unzip, I check over to Explorer which is still at around 10% done.
Steps to reproduce
Uncompress a zip file, especially one containing many small files
Expected Behavior
Unzip should be very fast, to enable a developer to stay in the flow and not be slowed down by unnecessary wait times.
Actual Behavior
The zip compression tool in Explorer is so painfully slow that for most devs it is unusable.
Windows Build Number
10.0.19043.0
Processor Architecture
AMD64
Memory
8GB
Storage Type, free / capacity
512GB SSD
Relevant apps installed
None
Traces collected via Feedback Hub
I can if needed

Unfortunately, 3rd party tools for ZIP-archives are quite a mixed bag. A high quality tool that is fast, secure, user-friendly and free does not seem to exist.

see this thread, turns out the ancient code from 1998 isn't very efficient, it's reading one byte at a time
https://twitter.com/ericlaw/status/1399856554690715659

There are a number of core performance problems with the ZIP code. The one related to "Move" operations (mentioned just above), is more fully described here: https://textslashplain.com/2021/06/02/leaky-abstractions/

Unfortunately, 3rd party tools for ZIP-archives are quite a mixed bag. A high quality tool that is fast, secure, user-friendly and free does not seem to exist.

I would say libarchive fits the job. It has already part of Windows since https://techcommunity.microsoft.com/t5/containers/tar-and-curl-come-to-windows/ba-p/382409.

See also https://github.com/microsoft/Windows-Dev-Performance/issues/27
This ticket has good sample zip file to work with.
https://github.com/microsoft/Windows-Dev-Performance/issues/27#issuecomment-670613369
My personal experience is my work Dell Precision laptop with SSD, unzipping a 50MB file with 25k files inside:

Windows (with lots of IT installed security agents) installed takes 12m.
Linux VM on same laptop takes 5s.

The gap is ... yawning.
I disabled all the AV and Windows time came down to 6m. Might be bitlocker is affecting the speed (or one of the other security agents).

I'd say the zip tools need a proper overhaul, not only performance fixes, as the UX alone is bad enough to make me install 3p tools, absolute minimum would be 7-zip context menu equivalent, adding smart compression that skips compression of files that doesn't reduce size enough and double-click-to-extract (with option to enable removal of source file when done) would be another step
And I absolutely never wanted the wizard that adds few extra clicks to my flow

adding smart compression that skips compression of files that doesn't reduce size enough and double-click-to-extract

God no. The current implementation is already problematic in enough ways.  Do you want to be responsible for the hack that throws away and recalculates archive dictionaries over late and very much arbitrary deflate/store selection?

regarding data I performed quick and dirty experiment using Bandizip to compress my downloads folder (4 946 928KB) containing mostly poorly compressible files (installers, videos and so on) with as they call it High speed archiving active and not, using Deflate  setting maximum available compression (except for the HSA switch)

Deflate with HSA: 0:26, 4 577 983KB
Deflate w/o HSA:  0:21, 4 577 982KB

with this out of the way I decided to try some well compressible data like a source code with almost no binaries, 1 029 669KB

Deflate with HSA: 0:43, 310 554KB
Deflate w/o HSA: 0:26, 310 554KB

after that I added some video files and archives to the mix:

Deflate with HSA: 0:30, 2 946 500KB
Deflate w/o HSA: 0:36, 2 945 660KB

so in a mixed bag it definitely works and is worth it, for everything else it looks like the way they determine if they should use Deflate or Store is not efficient enough to be worth it, at least on my machine
to dive deeper I tried constraining the process to single thread while compressing my downloads:

Deflate with HSA: 1:24, 4 577 988KB
Deflate w/o HSA: 2:05, 4 576 999KB

so it looks like sometimes it's worth it and sometimes not, possibly with improved detection it could be made to be worth it almost every time, but I'm not intending to push this idea any further

so it looks like sometimes it's worth it and sometimes not, possibly with improved detection it could be made to be worth it almost every time, but I'm not intending to push this idea any further

The problem I see with this is that, the detection requires you to run the deflate algorithm anyway. One could probably buffer the raw streams to solve the double read issue; however, you're still stuck with this problem of "detection." The only non-hack way to do solve this would be to accept the CPU overhead of compressing once & buffer the output + changes to the compression dictionary before committing. Most compressors don't scale to multiple threads and the non-hack solution would toll the CPU just as much. Worse, I still suspect multi-threaded optimized compressors would be suffer the most - if their dictionaries have to be thrown away. From my tests with zstd, flushing at the end of each substream doesn't obliterate the compression ratio (+1GB, end: 18GB +- 1GB, total in: ~30GB raw assets), which might be indicative of the extent to which this could scale. My concern with these numbers is they all come from proprietary software whose baseline and implementation efficiencies are unknown -- we have no idea what they did to fudge those numbers. Needless to say, I would suspect bandizip of inefficiently compressing files, delegating throwaway compression operations to an otherwise wasted thread to solve the store/deflate question, and using preemptive hacks to throw away likely-uncompressible files (not so smart. you wouldn't want this advertised as the "smart" method uninformed users will treat like the defacto compress button).
I think we should be focused on the state of parallelization in the world of archives than cheap hacks to throw away probably-uncompressible files.

zip container, despite all shortcomings, is used for basically everything nowadays and is the go to container format for anyone intending to create their own container
but as there already exists a decent MT capable algorithm and container it would be nice to have support for it too

It is kind of hilarious when I have to wait ~10 mins. to extract a specific folder from a 1GB zip file. The explorer Window shows transfer rates of a few KB/s. On a 6 core laptop with NVMe.

It's also kind of ridiculous that numerous folks at Microsoft know about this, they also talk and complain about it on Twitter, and yet none of them has just gone in and fixed it.
Even changing the ReadFile calls to operate on more than 1 byte at a time or use any form of user-space caching (re: the comment above, https://github.com/microsoft/Windows-Dev-Performance/issues/91#issuecomment-853081988) would be a low-risk, high-performance gain.
(If nobody at Microsoft is going to take action on the issues filed in this repo, why does this repo even exist?)

Actually, this will probably be the reason to migrate to Linux.
This is not the first time Linux is faster, but in my new project, all data is in zipped csv updated daily). I can't believe unzipping is so slow.

7-zip is a lot faster at uncompressing zip files than Windows Explorer.
I have hoped a change with recent Windows 11 support for RAR/7Z/libarchive, but it is still quite slow.
Eclipse and other IDEs, Jenkins and Artifactory are good exemples of large zips.

@nmoreaud, Are you running a version of Windows 11 that supports decompressing .rar, .7z, etc.?

@Bosch-Eli-Black I thought so but I was wrong. I'll wait for the update.

@nmoreaud Sounds good :)  Would be awesome if this update also made ZIP files much faster! :D

Libarchive support was added to FileExplorer in the Sept 2023 & 23H2 update for extraction. This complements the support added at the commandline via tar.exe.

I would keep this issue open as apparently PowerShell's Expand-Archive is still faster than Explorer.exe's Expand Archive (even with the current Win11 build - 22631.3007):
https://www.reddit.com/r/PowerShell/comments/1972k40/why_is_expandarchive_significantly_faster_than/

With this zip : https://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/2023-12/R/eclipse-java-2023-12-R-win32-x86_64.zip
7-zip: 3.7s - 8s
Explorer: 21s
Powershell: 30s

Appreciate the metrics!

Extracting https://github.com/dotnet/msbuild/archive/refs/tags/v17.10.4.zip (10 MB, 2500 files)

7-Zip: 1 second
Expand-Archive: 4 seconds
Windows Explorer: 45 seconds

Windows 11 Pro 23H2 22631.3593

@AdamBrMSFT @AvriMSFT Please could you reopen the issue?

Tagging @DHowett for visibility.

Perhaps not directly related, but i tried using the tar.gz decompression support in the windows 11 file explorer today on https://github.com/Kitware/CMake/releases/download/v3.29.8/cmake-3.29.8-linux-x86_64.tar.gz today, Windows wanted to spend 1hr 45 minutes on it.  Extracting the equivalent windows build https://github.com/Kitware/CMake/releases/download/v3.29.8/cmake-3.29.8-windows-x86_64.zip took 5 seconds.
I'd have instead posted this comment on https://github.com/microsoft/Windows-Dev-Performance/issues/27, but that issue was incorrectly closed as fixed.

I'd have instead posted this comment on #27, but that issue was incorrectly closed as fixed, and then locked to additional comments.

I don't think #27 was related to your issue, because it was testing with tar.exe and PowerShell's Expand-Archive, not the Windows File Explorer. For "File Explorer extraction is slower than tar.exe or PowerShell Expand-Archive", this seems like the correct ticket.
Unless tar.exe or Expand-Archive show the same slowness, in which case it might be the same problem as #27, particularly if disabling Windows Defender Realtime Protection fixes the speed issue. Since #27 is closed, maybe a new ticket would be called-for in this case.

I believe the whole process needs an overhaul from UX perspective, in many cases going through the Explorer wizard takes long enough I wouldn't bother using it even if it was faster than 7-zip, and if that overhaul takes care of some GUI related slowdowns that'd be even better

Yeah, the 7-zip explorer integration (particularly the dialog-free options) is terrific. I recall that the right-click-and-drag actions needed to extract into a temporary directory first and then move into place due to some limitation of the Explorer integration, but I don't know if that's still the case (or maybe it never was, and I'm misremembering...)
That said, I know the Windows design direction has been to reduce the amount of right-click stuff, amongst other reasons because it has a performance cost and with sufficient extensions active the right-click menu takes time to visibly appear. I thought the abbreviated menu plus the "Show More Options" menu was intended to address that, but even that list is growing and at least on my machine has a visible gap between right-click and the abbreviated menu appearing.
Anyway, UX is wandering off from the repo topic, and is a very open-ended discussion in itself.


时钟及rst配置问题

看着波形里dump的是rst从1 开始的，时钟周期是2ps， 麻烦问下 这两个在哪里配置的

C++ code for dumping wavefrom is at https://github.com/OpenXiangShan/difftest/blob/master/src/test/csrc/verilator/emu.cpp#L610-L618

By default, --dump-wave will dump the waveform only at clock=1 after the reset is deasserted. This option greatly (50%) reduces the waveform size and the simulation time. Using --dump-wave-full will dump the full waveform at the cost of lower simulation speed.

We are using the default timescale option from Verilator (see https://veripool.org/guide/latest/exe_verilator.html?highlight=timescale#cmdoption-timescale), which is 1ps/1ps. It's worth noting that, for RTL simulation, timescale does not actually affect the simulation results. Commercial RTL simulators are using the timescale to match real time. However, this is meaningless for early-stage RTL simulation.

(Question issues are by default closed when answered. Please feel free to leave future comments in this issue for further discussions.)


[natives] New function exists_at

This adds a new native function globals::exists_at which allows to check existance of a resource outside of the module which defines this resource. This functionality is needed for Move Objects to be able to build typed object references.
This is an alternative to #852. Preview state, no tests added yet, and the gas formula is incomplete.

I see, I hadn't realized I could pop out a Type, thus my approach had to do some weird reading of storage. This looks good, only concern is that we cannot constrain the exists_at function to where we want it, we move quickly in opening it up to the world. My only thought was do we want to move to this so quickly? I don't really mind either way, but would like to land an approach to unblock the next phase of object code ;)


Implement PEP 639

What's the problem this feature will solve?
Standardized licensing metadata will in time unify the currently largely chaotic landscape.
Describe the solution you'd like
See: https://peps.python.org/pep-0639
PEP 639 has been provisionally accepted with one of the conditions being implementation in PyPI.

PyPI uses https://github.com/pypa/packaging to parse and validate metadata, so it will need an update to support the new metadata version, keys, and deprecations.
I've opened draft PRs for that and the packaging docs at:

https://github.com/pypa/packaging/pull/828
https://github.com/pypa/packaging.python.org/pull/1595

Once those PRs (or something that supersedes them) are merged, implementation in PyPI can begin.

Draft PR is now up at https://github.com/pypi/warehouse/pull/16949 awaiting merge/release of https://github.com/pypa/packaging/pull/828.

#16949 is now ready for review with the release of packaging 24.2

PEP 639 implementation is live and online for PyPI now. We'll keep an eye out for issues filed if folks run into any issues as adoption takes off.

Reopening this because part of implementing PEP 639 is also deprecating the Licence metadata field and the various License :: trove classifiers. We should define a deprecation period, during which we warn users, and a suitable threshold for dropping support for these fields.

By my read of 639, PyPI's responsibility is to not add any new license classifiers...

New license classifiers MUST NOT be added to PyPI; users needing them SHOULD use the License-Expression field instead. License classifiers may be removed from a new version of the specification in a future PEP.

I think deprecating them in the way proposed in https://github.com/pypa/trove-classifiers/pull/199 is left to a future pep.

I'm also pretty certain we are already meeting the spec for the License field:

For all newly-uploaded distribution archives that include a License-Expression field, the Python Package Index (PyPI) MUST reject any that specify both License and License-Expression fields.
The License field may be removed from a new version of the specification in a future PEP.

https://github.com/pypi/warehouse/blob/9a270a2d01cba250795daf1e05133fbbafeeb7b0/warehouse/forklift/metadata.py#L238-L249


History Previous keybind description typo

Issue Type: Bug
Noticed a small typo at debug.ts#37. There's an extra 'e' between first and debug. This also shows up in the Keyboard Shortcuts view under the "when" column for the History Previous keybind.

VS Code version: Code - Insiders 1.22.0-insider (5450fdc79778b5a76a27ea7e5fd68e9ebbe44107, 2018-03-12T05:15:51.483Z)
OS version: Windows_NT x64 10.0.16299
Reproduces without extensions

@mattacosta thanks a lot. Next time also feel free to submit a PR.


AD410 not recognized

AD410 appears as "Generic RTSP" device.
As such there is no extra select, sensors, or switches.  I.e. no button push.
Could Amcrest  have  changed or ?

I am also getting this now with a new fresh install of Home Assistant. It still works on my old HA box but not on the fresh install.
Both are running the same version 2023.9.2.
The fresh install just has generic RTSP where as the old install has the AD410, firmware, and the button press.

Which firmware version are you using? There is a new firmware for the AD410: https://github.com/rroller/dahua/issues/282#issuecomment-1732595571
There were firmware versions from 2022 that caused problems with newer Home Assistant versions.
I've been using an Imou DB61i for a few days now, which is basically identical to the Amcrest AD410.

Firmware from 2021 is working.
Firmware from 2022 is not working.
New Firmware from 2023 (1.000.0000002.1.R,build:2023-08-19) is working.

My active Home Assistant installation is version 2023.8.4 with Dahua Addon 0.9.44 = The Imou DB61i works without any problems. See https://github.com/rroller/dahua/issues/282#issuecomment-1732550022
I just did a new fresh Home Assistant installation (2023.9.2) as a virtual machine as a test (for you) and the Imou DB61i works with that too (Dahua Addon Version 0.9.44)

If you are using the current Amcrest firmware and the Dahua addon version 0.9.44 and it still doesn't work, there are probably differences between the Imou and the Amcrest API.

My Doorbell is running firmware: 1.000.00AC002.0.R,build:2023-08-10 I am running the latest version on my old HA box and now am running the 0.9.44 on the new box. still will not add with the button press sensor.
Old Box:

New Box:

Yes with my old Amcrest AD110 also have a Sensor with "Button Pressed".

But now, when the bell button is pressed on my Imou DB61i, it is signaled via "Alarm Local" Sensor. That's ok for me.
My Imou DB61i camera is also not recognized as a Generic RTSP Camera but as a DB61i from Dahua and I can see the full Firmware-Version and not only "Firmware: 1.0"

But there are probably differences in the API and perhaps Amcrest has made changes by/in/with the new firmware.

I am also getting this now with a new fresh install of Home Assistant. It still works on my old HA box but not on the fresh install. Both are running the same version 2023.9.2. The fresh install just has generic RTSP where as the old install has the AD410, firmware, and the button press.

I'm running HA 2023.9.3 and the 8-10 Amcrest firmware.  I'm not sure about the integration version, where are you checking that?  I've got the latest off of HACS.
There is much more missing than just the button push, that was just an example in the OP.  Things like IR control, White light select, siren, etc.

@GaryOkie
But there are probably differences in the API and perhaps Amcrest has made changes by/in/with the new firmware.

I have yet to install the newest firmware on my AD410. The main reason for holding off was specifically to be sure no new issues with event handling and Home Assistant were encountered.
Since the AD410 with new firmware is being identified as a generic RTSP camera this could be due a change in the firmware system identifier.
@Phrede &/or @starfleetk - what response do you get from this API command (after authenticating)?
<camera_ip>/cgi-bin/magicBox.cgi?action=getSystemInfo
With the original firmware, here is the expected response:
appAutoStart=true
deviceType=AD410
hardwareVersion=1.00
processor=SSC339
serialNumber=Z17D3xxxxx
updateSerial=DB6I
updateSerialCloudUpgrade=DB6I-xxxxxxxxxxxxxxxxx
I'm not sure what API command the custom Dahua code is using to determine device type, but I do know the python-amcrest library (which the core Amcrest integration uses) relies on getSystemInfo for deviceType.
@Phrede / @startfleetk - run this getSystemInfo command to confirm if there might have been a change to deviceType in the new firmware.

There is much more missing than just the button push, that was just an example in the OP. Things like IR control, White light select, siren, etc.

The fact that some sensors are no longer displayed under Home Assistant could be due to an API change.
Unfortunately, today I have to realize that my Imou DB61i isn't working properly for me. When the Imou DB61i is powered by the bell transformer, it makes “cracking noises” after a while and a whistling noise after another while.:-( So far I had only connected and tested the camera via a USB power supply in the study. There were no problems or noises whatsoever.
Therefore I will send the Imou back to the seller on Aliexpress and continue to use my Amcrest AD110. So I can't say anything more about it now.

The fact that some sensors are no longer displayed under Home Assistant could be due to an API change.

Yep - if the device isn't recognized as a doorbell via the deviceType or other DB identifier, then it should be expected that none of the doorbell-related sensors would be provided.
I've never heard of a crackling/whistling issue before.   Since it doesn't happen with USB power, the transformer power is the likely cause.  But may be too late for your DB6I as it was either damaged out of the box or damaged after connection.   Good choice to send it back.
Have you looked into the Reolink doorbell availability in Germany?   It appears to be overtaking the AD410 in home automation/RTSP popularity lately.

I did try the <camera_ip>/cgi-bin/magicBox.cgi?action=getSystemInfo, however it never got past the authentication. I put in the same username and password that I use in the app and keeps asking for me to login.

I put in the same username and password that I use in the app

@starfleetk - if you are referring to the SmartHome app, that ID/pw is the cloud login, which uses your email as the ID.   The Amcrest camera/doorbell always uses 'admin' as the userid, and whatever p/w you used during its initial setup.

appAutoStart=true deviceType=AD410 hardwareVersion=1.00 processor=SSC339 serialNumber=Z17A6F9xxxxx updateSerial=DB6I updateSerialCloudUpgrade=DB6I:01:04:07:7C:41:00:01:10:01:01:04:400:03:03:00:00:08:01:00:00:100:FF
Glad to see you're here @GaryOkie.
I do believe is most likely is a firmware issue.  I'm doing a lot of work on things in this area, my front gate, and during one of the mandatory factory resets I accepted the upgrade.
I'm about done with it.  Honestly, I've worked more on this than any 10 other cameras combined.  Mandatory factory reset, meaning having to open the internet to it, to change the wifi password!!!  A constant battle with this thing is just not worth it.

BTW  Using Smart PSS you can change the admin user name, and much more.

@Phrede - per your OP, we first need to understand why this integration is identifying the doorbell as a "generic RTSP" device.
I've taken a quick look at the code and see that vto.py  (the low level I/O Dahua device driver) is not using getSystemInfo, but instead:  <camera_ip>/cgi-bin/magicBox.cgi?action=getDeviceType   which should simply return device=AD410      (In some cases it has to get device type via "updateSerial" )
I also saw this code has done a good job of logging device attributes.    So in the logger config, please try enabling  INFO (or debug) loglevel and report devicetype/serial results.
custom_components.dahua: info

I put in the same username and password that I use in the app

@starfleetk - if you are referring to the SmartHome app, that ID/pw is the cloud login, which uses your email as the ID. The Amcrest camera/doorbell uses 'admin' as the userid, and whatever p/w you specified during its initial setup.
When you configured the doorbell for this custom Dahua integration, the local camera credentials is what you must use, not the app/cloud login.

Thanks for that. It is working as it should for me now. I will have to remember that for next time if I have to add that doorbell again.

<camera_ip>/cgi-bin/magicBox.cgi?action=getDeviceType

type=AD410
When I saw that I removed and reinstalled and things appear normal again.
It looks like the IR isn't going on,  I need to check it further.
I appreciate your help


[Refactor] Port TruthfulQA (mc1 only)

This PR implements only the multiple choice task with single answers (mc1) of TruthfulQA. The benchmark also provides a multiple choice task with multiple answers and a generation task, which are implemented in master and will probably require some changes to the codebase to be ported (i.e., a possibility to define multiple targets for a task).
The results of scripts/write_out.py are identical, the metrics for pythia-160m and llama-7b are below (note: compare master mc1 metric with big-refactor acc):
python main.py --model hf-causal --model_args pretrained=EleutherAI/pythia-160m --tasks truthfulqa_mc --batch_size 32
hf-causal (pretrained=EleutherAI/pythia-160m), limit: None, provide_description: False, num_fewshot: 0, batch_size: 32
|    Task     |Version|Metric|Value |   |Stderr|
|-------------|------:|------|-----:|---|-----:|
|truthfulqa_mc|      1|mc1   |0.2436|±  | 0.015|
|             |       |mc2   |0.4444|±  | 0.015|

accelerate launch main.py --model hf --model_args pretrained=EleutherAI/pythia-160m --tasks truthfulqa_mc1 --batch_size 32
hf (pretrained=EleutherAI/pythia-160m), limit: None, num_fewshot: 0, batch_size: 32
|     Task     |Version|Filter| Metric |Value |   |Stderr|
|--------------|-------|------|--------|-----:|---|-----:|
|truthfulqa_mc1|Yaml   |none  |acc     |0.2485|±  |0.0151|

python main.py --model hf-causal --model_args pretrained=huggyllama/llama-7b --batch_size 4 --tasks truthfulqa_mc
hf-causal (pretrained=huggyllama/llama-7b), limit: None, provide_description: False, num_fewshot: 0, batch_size: 4
|    Task     |Version|Metric|Value |   |Stderr|
|-------------|------:|------|-----:|---|-----:|
|truthfulqa_mc|      1|mc1   |0.2203|±  |0.0145|
|             |       |mc2   |0.3408|±  |0.0132|

accelerate launch main.py --model hf --model_args pretrained=huggyllama/llama-7b --tasks truthfulqa_mc1 --batch_size 4
hf (pretrained=huggyllama/llama-7b), limit: None, num_fewshot: 0, batch_size: 4
|     Task     |Version|Filter| Metric |Value |   |Stderr|
|--------------|-------|------|--------|-----:|---|-----:|
|truthfulqa_mc1|Yaml   |none  |acc     |0.2191|±  |0.0145|

@nopperl can you talk more about what the multiple-target task would look like? Is it simply assigning score 1 if a prediction is in a set of valid targets?

@lintangsutawika exactly! For the mc2 task, there are multiple true and false choices. You can see the targets here: https://huggingface.co/datasets/truthful_qa/viewer/multiple_choice/validation (in the mc2_targets column). The master branch implementation runs a loglikelihood Request for each choice and uses the sum of all true probabilities over the sum of all probabilities as metric (see https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/truthfulqa.py#L112-L159).


'error': 'unauthorized_client'

I want to access the data of a polar device. I receive the authorization code but when trying to get an access token I get 'error': 'unauthorized_client'.
`authorize_url = 'https://flow.polar.com/oauth2/authorization'
redirect_uri = 'http://localhost:8000/callback'
scope = 'accesslink.read_all'
params = {
'response_type': 'code',
'client_id': client_id,
'redirect_uri': redirect_uri,
'scope': scope
}
url = f"{authorize_url}?{'&'.join([f'{k}={v}' for k,v in params.items()])}"
webbrowser.open(url)
class CallbackHandler(http.server.SimpleHTTPRequestHandler):
code = None
def do_GET(self):
    parsed_url = urlparse.urlparse(self.path)
    query = urlparse.parse_qs(parsed_url.query)
    CallbackHandler.code = query.get('code')
    self.send_response(200)
    self.send_header('Content-type', 'text/html')
    self.end_headers()
    self.wfile.write(
        b'
Authorization code received!
')

with socketserver.TCPServer(("", 8000), CallbackHandler) as httpd:
httpd.handle_request()
headers = {
'Content-Type': 'application/x-www-form-urlencoded',
'Authorization': f'Basic {client_id}:{client_secret}',
'Accept': 'application/json',
}
data = {
'grant_type': 'authorization_code',
'code': CallbackHandler.code,
}
r = requests.post('https://polarremote.com/v2/oauth2/token',
data=data, headers=headers)
print(r.json()) `

Same issue here with:
import requests
Replace these with your own client ID, secret, and redirect URL
client_id = 'xxx'
client_secret = 'xxx'
redirect_url = 'xxx'
Step 1: Get the authorization URL
auth_url = f'https://flow.polar.com/oauth2/authorization?response_type=code&client_id={client_id}&redirect_uri={redirect_url}'
print(f'Visit this URL to authorize your application: {auth_url}')
Step 2: Get the authorization code from the user
auth_code = input('Enter the authorization code: ')
Step 3: Exchange the authorization code for an access token
token_url = 'https://polarremote.com/v2/oauth2/token'
token_data = {
'grant_type': 'authorization_code',
'code': auth_code,
'redirect_uri': redirect_url,
'client_id': client_id,
'client_secret': client_secret
}
print(token_data)  # Add this line to print the token_data dictionary
token_response = requests.post(token_url, data=token_data)
print(token_response.status_code)  # Add this line to print the response status code
print(token_response.headers)  # Add this line to print the response headers
token_json = token_response.json()
print(token_json)  # Add this line to print the JSON response
access_token = token_response.json()['access_token']
Step 4: Get the user's Polar user ID
user_id_url = 'https://www.polaraccesslink.com/v3/users'
headers = {'Authorization': f'Bearer {access_token}'}
user_id_response = requests.get(user_id_url, headers=headers)
user_id = user_id_response.json()['polar_user_id']
Step 5: Get the user's heart rate data during activities
activities_url = f'https://www.polaraccesslink.com/v3/users/{user_id}/exercise'
activities_response = requests.get(activities_url, headers=headers)
activities = activities_response.json()['exercises']
for activity in activities:
print(f'Activity: {activity["start_time"]}')
print(f'Heart rate: {activity["heart_rate"]["average"]} bpm')
Step 6: Get the user's sleep stages during the night
sleep_url = f'https://www.polaraccesslink.com/v3/users/{user_id}/sleep'
sleep_response = requests.get(sleep_url, headers=headers)
sleep_data = sleep_response.json()['sleeps']
for sleep in sleep_data:
print(f'Sleep: {sleep["date"]}')
print(f'Sleep stages: {sleep["sleep_stages"]}')

user_id_url = 'https://www.polaraccesslink.com/v3/users'
headers = {'Authorization': f'Bearer {access_token}'}
user_id_response = requests.get(user_id_url, headers=headers) I do exactly the same as here but I get a 405 reponse

Hi! Please make sure that you follow correct authentication flow. There has been some improvements to the documentation and it should describe it more clearly now. Pay attention especially to the fact that user needs to be registered (7) before the data is accessible. More information here: https://www.polar.com/accesslink-api/#polar-accesslink-api
Here are the steps

Create Polar Flow account if you don't have one already
Go to https://admin.polaraccesslink.com/ and log in using your Polar Flow credentials
Fill your application/service information and create client
After client is created, remember to write down OAuth2 client id and secret for later use
Obtain authorization code for the user using authorization endpoint
Use authorization code to obtain access token for the user using token endpoint
Register user using register user endpoint to be able to access user's data
Start developing the API

Also make sure that you are using correct HTTP method, error 405 indicates that incorrect method is used.


inject.js - 'Inject' is already declared in the upper scope

Refactor Method:
As this is an issue of shadowing an upper scope variable, the refactor method will be to rename the secondary inject variable so it no longer matches the initial 'inject' declaration

Tested in src/tests/injector.spec.ts


fork sync and fork CI workflows

Pull Request Test Coverage Report for Build<PHONE_NUMBER>2
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.02%) to 43.93%

Totals

Change from base Build<PHONE_NUMBER>4:
0.02%

Covered Lines:
6521

Relevant Lines:
14844

💛  - Coveralls


About us issues fixed

Pull Request
Title
About us issues fixed
Description
About us button link added in home page and about us scrollbar issue fixed
Related Issues
Fixes #78
Changes Made
bug fixes
Checklist

[X] I have tested the changes locally
[ ] Documentation has been updated (if necessary)
[X] Changes are backward-compatible

can you give a screen recording real quick, it's easier to check that way

About us:

Main page:


[SPARK-44985][CORE] Use toString instead of stacktrace for task reaper threadDump

What changes were proposed in this pull request?

In addition to the stack trace, the toString function includes information on the thread's ID, name, state, locks, and held locks.
Why are the changes needed?

improve debug messages for task reaper
Does this PR introduce any user-facing change?

yes, log changes
How was this patch tested?

existing ut
Was this patch authored or co-authored using generative AI tooling?

no

cc @zhengruifeng @cloud-fan @HyukjinKwon @dongjoon-hyun, thanks

Merged to master.

Thank you @dongjoon-hyun