HDFS-16203. Discover datanodes with unbalanced block pool usage by th…

JIRA: HDFS-16203
Discover datanodes with unbalanced volume usage by the standard deviation.
In some scenarios, we may cause unbalanced datanode disk usage:

Repair the damaged disk and make it online again.
Add disks to some Datanodes.
Some disks are damaged, resulting in slow data writing.
Use some custom volume choosing policies.

In the case of unbalanced disk usage, a sudden increase in datanode write traffic may result in busy disk I/O with low volume usage, resulting in decreased throughput across datanodes.
We need to find these nodes in time to do diskBalance, or other processing. Based on the volume usage of each datanode, we can calculate the standard deviation of the volume usage. The more unbalanced the volume, the higher the standard deviation.
We can display the result on the Web of namenode, and then sorting directly to find the nodes where the volumes usages are unbalanced.
This interface is only used to obtain metrics and does not adversely affect namenode performance.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 41s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
jshint
0m  1s

jshint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
12m 43s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
22m  3s

trunk passed

+1 :green_heart:
compile
5m 44s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
5m  9s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
1m 19s

trunk passed

+1 :green_heart:
mvnsite
2m 35s

trunk passed

+1 :green_heart:
javadoc
1m 46s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
2m 17s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
6m  7s

trunk passed

+1 :green_heart:
shadedclient
17m  3s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 29s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 10s

the patch passed

+1 :green_heart:
compile
5m 29s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
5m 29s

the patch passed

+1 :green_heart:
compile
5m  9s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
5m  9s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
1m  8s
/results-checkstyle-hadoop-hdfs-project.txt
hadoop-hdfs-project: The patch generated 1 new + 122 unchanged - 0 fixed = 123 total (was 122)

+1 :green_heart:
mvnsite
2m 16s

the patch passed

+1 :green_heart:
javadoc
1m 28s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
2m  1s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
6m 16s

the patch passed

+1 :green_heart:
shadedclient
17m  6s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m 29s

hadoop-hdfs-client in the patch passed.

-1 :x:
unit
238m 50s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 49s

The patch does not generate ASF License warnings.

361m 25s

Reason
Tests

Failed junit tests
hadoop.hdfs.TestRollingUpgrade

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3366

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell jshint

uname
Linux 67e808c87838 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / d9bcf5eff7d52eda97c537e80751d7371a101a6f

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/1/testReport/

Max. process+thread count
3100 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@jojochuang @tasanuma @Hexiaoqiao @ferhui @ayushtkn  Could you please review this PR. Thanks.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 39s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
jshint
0m  0s

jshint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
12m 57s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
20m 27s

trunk passed

+1 :green_heart:
compile
4m 54s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
4m 34s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
1m 13s

trunk passed

+1 :green_heart:
mvnsite
2m 21s

trunk passed

+1 :green_heart:
javadoc
1m 39s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
2m  6s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
5m 35s

trunk passed

+1 :green_heart:
shadedclient
16m 17s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 28s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m  3s

the patch passed

+1 :green_heart:
compile
4m 46s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
4m 46s

the patch passed

+1 :green_heart:
compile
4m 28s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
4m 28s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  5s

hadoop-hdfs-project: The patch generated 0 new + 113 unchanged - 9 fixed = 113 total (was 122)

+1 :green_heart:
mvnsite
2m  3s

the patch passed

+1 :green_heart:
javadoc
1m 22s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 55s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
5m 42s

the patch passed

+1 :green_heart:
shadedclient
16m  5s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m 20s

hadoop-hdfs-client in the patch passed.

+1 :green_heart:
unit
231m 52s

hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 48s

The patch does not generate ASF License warnings.

346m  8s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3366

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell jshint

uname
Linux b2667f9ca18e 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 58f0a2e2f44bacf4b98d236228b185df34501ae4

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/2/testReport/

Max. process+thread count
3200 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 57s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
jshint
0m  0s

jshint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
12m 33s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
20m 40s

trunk passed

+1 :green_heart:
compile
5m 13s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
4m 59s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
1m 16s

trunk passed

+1 :green_heart:
mvnsite
2m 20s

trunk passed

+1 :green_heart:
javadoc
1m 35s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
2m  6s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
5m 35s

trunk passed

+1 :green_heart:
shadedclient
18m 12s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 27s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 14s

the patch passed

+1 :green_heart:
compile
5m 17s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
5m 17s

the patch passed

+1 :green_heart:
compile
4m 57s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
4m 57s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  3s

hadoop-hdfs-project: The patch generated 0 new + 113 unchanged - 9 fixed = 113 total (was 122)

+1 :green_heart:
mvnsite
2m  9s

the patch passed

+1 :green_heart:
javadoc
1m 23s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 54s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
5m 48s

the patch passed

+1 :green_heart:
shadedclient
18m  1s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m 19s

hadoop-hdfs-client in the patch passed.

+1 :green_heart:
unit
235m  9s

hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 48s

The patch does not generate ASF License warnings.

355m 21s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3366

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell jshint

uname
Linux 82d18bc354e2 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9 23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 9f2a186f9e219167278537b7f0d4d0312f110942

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/3/testReport/

Max. process+thread count
3458 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Hi @jojochuang @tasanuma @ayushtkn @ferhui @Hexiaoqiao , could you please review this PR? Thanks a lot.

Hi @aajisaka @goiri @virajjasani , could you please also review this? Thank you.

@ayushtkn @ferhui @Hexiaoqiao Could you please review the code? Thanks a lot.
The standard deviation is calculated by referring to FSNamesystem#getNodeUsage. The key change is FSNamesystem#getBlockPoolUsedPercentStdDev, :
  public float getBlockPoolUsedPercentStdDev(StorageReport[] storageReports) {
    ArrayList<Float> usagePercentList = new ArrayList<>();
    float totalUsagePercent = 0.0f;
    float dev = 0.0f;

    if (storageReports.length == 0) {
      return dev;
    }

    for (StorageReport s : storageReports) {
      usagePercentList.add(s.getBlockPoolUsagePercent());
      totalUsagePercent += s.getBlockPoolUsagePercent();
    }

    totalUsagePercent /= storageReports.length;
    Collections.sort(usagePercentList);
    for (Float usagePercent : usagePercentList) {
      dev += (usagePercent - totalUsagePercent)
          * (usagePercent - totalUsagePercent);
    }
    dev = (float) Math.sqrt(dev / usagePercentList.size());
    return dev;
  }

Not sure about the way of standard deviation, I will take a look.
DataNodes also show on Router UI, do you test it?

Not sure about the way of standard deviation, I will take a look.
DataNodes also show on Router UI, do you test it?

Thanks @ferhui for your review. We currently only display it on the Namenode Web. If we need to display the router UI, I can open a separate JIRA.

Overall it looks good.
DataNode Info shows on namenode UI and Router UI, It is reasonable to add this into router UI.
Maybe it is like namenode, just lines code changed. We can review it here.

Ok, I'll add it to router UI and do some related tests. Thanks.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m  0s

Docker mode activated.

-1 :x:
patch
0m 19s

https://github.com/apache/hadoop/pull/3366 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hadoop/pull/3366

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/5/console

versions
git=2.17.1

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Overall it looks good.
DataNode Info shows on namenode UI and Router UI, It is reasonable to add this into router UI.
Maybe it is like namenode, just lines code changed. We can review it here.

Hi @ferhui , I added this into router UI and did some tests.
There are two NS in our test cluster. NS1 supports this feature, and NS2 does not. In this case, the Router UI is shown as follows:

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 40s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
jshint
0m  1s

jshint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
12m 44s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
20m 22s

trunk passed

+1 :green_heart:
compile
4m 58s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
4m 40s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
1m 13s

trunk passed

+1 :green_heart:
mvnsite
3m  7s

trunk passed

+1 :green_heart:
javadoc
2m 19s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
3m  7s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
6m 46s

trunk passed

+1 :green_heart:
shadedclient
14m 35s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 27s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 33s

the patch passed

+1 :green_heart:
compile
4m 49s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
4m 49s

the patch passed

+1 :green_heart:
compile
4m 28s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
4m 28s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  5s

hadoop-hdfs-project: The patch generated 0 new + 117 unchanged - 9 fixed = 117 total (was 126)

+1 :green_heart:
mvnsite
2m 36s

the patch passed

+1 :green_heart:
javadoc
1m 55s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
2m 43s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
7m  1s

the patch passed

+1 :green_heart:
shadedclient
14m 38s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m 23s

hadoop-hdfs-client in the patch passed.

-1 :x:
unit
234m 54s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

-1 :x:
unit
20m 48s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
hadoop-hdfs-rbf in the patch passed.

+1 :green_heart:
asflicense
0m 53s

The patch does not generate ASF License warnings.

378m 10s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes

hadoop.hdfs.server.federation.router.TestRouterRPCClientRetries

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/6/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3366

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell jshint

uname
Linux bb759ab19afd 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 0532517dd35ca3a01652f4fd1e77fe771eedff81

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/6/testReport/

Max. process+thread count
2975 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/6/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 40s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
jshint
0m  0s

jshint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
12m 50s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
20m 12s

trunk passed

+1 :green_heart:
compile
4m 52s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
4m 31s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
1m 14s

trunk passed

+1 :green_heart:
mvnsite
3m  6s

trunk passed

+1 :green_heart:
javadoc
2m 18s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
3m  4s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
6m 48s

trunk passed

+1 :green_heart:
shadedclient
14m 34s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 25s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 37s

the patch passed

+1 :green_heart:
compile
4m 46s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
4m 46s

the patch passed

+1 :green_heart:
compile
4m 25s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
4m 25s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
1m  6s
/results-checkstyle-hadoop-hdfs-project.txt
hadoop-hdfs-project: The patch generated 1 new + 120 unchanged - 9 fixed = 121 total (was 129)

+1 :green_heart:
mvnsite
2m 43s

the patch passed

+1 :green_heart:
javadoc
1m 54s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
2m 42s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
6m 59s

the patch passed

+1 :green_heart:
shadedclient
14m 19s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m 20s

hadoop-hdfs-client in the patch passed.

-1 :x:
unit
240m  4s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
unit
22m 26s

hadoop-hdfs-rbf in the patch passed.

+1 :green_heart:
asflicense
0m 46s

The patch does not generate ASF License warnings.

384m  6s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/7/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3366

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell jshint

uname
Linux 698a23a1a331 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 0b102603c8fdd9a33bb6a4bb77beb8d9591b8fa7

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/7/testReport/

Max. process+thread count
3341 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/7/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 44s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
jshint
0m  1s

jshint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
12m 49s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
20m 23s

trunk passed

+1 :green_heart:
compile
4m 57s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
4m 35s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
1m 12s

trunk passed

+1 :green_heart:
mvnsite
3m  5s

trunk passed

+1 :green_heart:
javadoc
2m 20s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
3m  8s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
6m 49s

trunk passed

+1 :green_heart:
shadedclient
14m 36s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 27s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 36s

the patch passed

+1 :green_heart:
compile
4m 51s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
4m 51s

the patch passed

+1 :green_heart:
compile
4m 28s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
4m 28s

the patch passed

+1 :green_heart:
blanks
0m  1s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  4s

hadoop-hdfs-project: The patch generated 0 new + 120 unchanged - 9 fixed = 120 total (was 129)

+1 :green_heart:
mvnsite
2m 40s

the patch passed

+1 :green_heart:
javadoc
1m 57s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
2m 47s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
7m  3s

the patch passed

+1 :green_heart:
shadedclient
15m 16s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m 18s

hadoop-hdfs-client in the patch passed.

+1 :green_heart:
unit
241m 16s

hadoop-hdfs in the patch passed.

+1 :green_heart:
unit
21m 27s

hadoop-hdfs-rbf in the patch passed.

+1 :green_heart:
asflicense
0m 51s

The patch does not generate ASF License warnings.

386m  2s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/8/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3366

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell jshint

uname
Linux fdb214245f36 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 70e73b3f3604f5e52328bb692ea38ad9f9d57c2c

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/8/testReport/

Max. process+thread count
3170 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/8/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 50s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
jshint
0m  0s

jshint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
12m 54s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
22m 43s

trunk passed

+1 :green_heart:
compile
5m 39s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
5m 14s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
1m 11s

trunk passed

+1 :green_heart:
mvnsite
3m 13s

trunk passed

+1 :green_heart:
javadoc
2m 33s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
3m 16s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
7m 25s

trunk passed

+1 :green_heart:
shadedclient
16m 43s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 29s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 52s

the patch passed

+1 :green_heart:
compile
5m 30s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
5m 30s

the patch passed

+1 :green_heart:
compile
4m 53s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
4m 53s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m 15s

hadoop-hdfs-project: The patch generated 0 new + 120 unchanged - 9 fixed = 120 total (was 129)

+1 :green_heart:
mvnsite
2m 55s

the patch passed

+1 :green_heart:
javadoc
2m  0s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
2m 54s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
7m 45s

the patch passed

+1 :green_heart:
shadedclient
15m 57s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m 28s

hadoop-hdfs-client in the patch passed.

-1 :x:
unit
299m 59s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

-1 :x:
unit
36m 47s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
hadoop-hdfs-rbf in the patch passed.

+1 :green_heart:
asflicense
0m 48s

The patch does not generate ASF License warnings.

470m 39s

Reason
Tests

Failed junit tests
hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy

hadoop.hdfs.TestRollingUpgrade

hadoop.hdfs.TestErasureCodingExerciseAPIs

hadoop.hdfs.tools.TestECAdmin

hadoop.hdfs.TestFileChecksum

hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl

hadoop.hdfs.TestParallelShortCircuitReadUnCached

hadoop.hdfs.TestDatanodeReport

hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy

hadoop.hdfs.TestDecommissionWithStriped

hadoop.hdfs.TestSetrepDecreasing

hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes

hadoop.fs.contract.router.web.TestRouterWebHDFSContractOpen

hadoop.fs.contract.router.TestRouterHDFSContractConcat

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/9/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3366

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell jshint

uname
Linux 328f0e7dc735 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / ea87838c996e94248fcef94e2eec25c098a44f3c

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/9/testReport/

Max. process+thread count
3368 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/9/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Those failed unit tests are unrelated to the change and they work fine locally.
@tasanuma @ferhui Could you please help review this PR again. Thank you very much.

Overall it looks good.
DataNode Info shows on namenode UI and Router UI, It is reasonable to add this into router UI.
Maybe it is like namenode, just lines code changed. We can review it here.

Hi @ferhui , I added this into router UI and did some tests.
There are two NS in our test cluster. NS1 supports this feature, and NS2 does not. In this case, the Router UI is shown as follows:

There are two NS in our test cluster. NS1 supports this feature, and NS2 does not. In this case, the Router UI is shown as this picture.

@tomscut Thanks.
Is Related UT fixed? If it's been fixed, this PR looks good to me.

@tomscut Thanks.
Is Related UT fixed? If it's been fixed, this PR looks good to me.

Yes, this related failed UT TestRouterRPCClientRetries#testNamenodeMetricsSlow is already fixed. Thanks for your review again.

@tomscut Sorry for being late, but I have one more request. Could you refactor the existing method?
  public Map<String, DatanodeStorageReport[]> getDatanodeStorageReportMap(
      DatanodeReportType type) throws IOException {
    return getDatanodeStorageReportMap(type, true, -1);
  }

@tomscut Sorry for being late, but I have one more request. Could you refactor the existing method?
  public Map<String, DatanodeStorageReport[]> getDatanodeStorageReportMap(
      DatanodeReportType type) throws IOException {
    return getDatanodeStorageReportMap(type, true, -1);
  }

Thanks for your review, I udpated the code.

+1, pending Jenkins.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 51s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
jshint
0m  0s

jshint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
12m 33s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
29m  8s

trunk passed

+1 :green_heart:
compile
6m 35s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
6m  1s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
1m 33s

trunk passed

+1 :green_heart:
mvnsite
3m 29s

trunk passed

+1 :green_heart:
javadoc
2m 44s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
3m 22s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
8m 24s

trunk passed

+1 :green_heart:
shadedclient
40m 35s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 30s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m  8s

the patch passed

+1 :green_heart:
compile
5m 39s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
5m 39s

the patch passed

+1 :green_heart:
compile
4m 41s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
4m 41s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  6s

hadoop-hdfs-project: The patch generated 0 new + 120 unchanged - 9 fixed = 120 total (was 129)

+1 :green_heart:
mvnsite
2m 40s

the patch passed

+1 :green_heart:
javadoc
1m 56s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
2m 44s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
7m  2s

the patch passed

+1 :green_heart:
shadedclient
28m 20s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m 19s

hadoop-hdfs-client in the patch passed.

-1 :x:
unit
242m 39s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
unit
21m 58s

hadoop-hdfs-rbf in the patch passed.

+1 :green_heart:
asflicense
0m 51s

The patch does not generate ASF License warnings.

443m 36s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped

hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/10/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3366

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell jshint

uname
Linux fcc8b33d75e1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 982272cade7a11c06a2aabf24dc505fadf5eb587

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/10/testReport/

Max. process+thread count
3178 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3366/10/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

The failed unit tests TestBlockTokenWithDFSStriped and TestBalancerWithHANameNodes are unrelated to the change, and work fine locally.

Merged it. Thanks for the contribution, @tomscut. Thanks for reviewing it, @ferhui.

Thanks @tasanuma @ferhui for your review and merge.


x/tools/gopls: panic in internal/golang.activeParameter

Using gopls golang/tools@e603756b2be22dc76f8a447c322fddce24fbb865:
goroutine 11966428 gp=0x140197a5180 m=45 mp=0x1402512c008 [running]:
panic({0x101c5cf20?, 0x102415050?})
	/usr/local/go/src/runtime/panic.go:804 +0x154 fp=0x14026e293f0 sp=0x14026e29340 pc=0x100f97294
runtime.panicmem(...)
	/usr/local/go/src/runtime/panic.go:262
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:900 +0x300 fp=0x14026e29450 sp=0x14026e293f0 pc=0x100f99760
golang.org/x/tools/gopls/internal/golang.activeParameter(0x101d98488?, 0x1400d9ae540?, 0xa0?, 0x101880df1?)
	/Users/dan/go/src/golang.org/x/tools/gopls/internal/golang/signature_help.go:192 +0x18 fp=0x14026e294b0 sp=0x14026e29460 pc=0x1017ec4b8
golang.org/x/tools/gopls/internal/golang.builtinSignature({0x101d98488?, 0x1400d9ae540?}, 0x1403f35e5a0, 0x0, {0x101880df1?, 0x0?}, 0x6c328350)
	/Users/dan/go/src/golang.org/x/tools/gopls/internal/golang/signature_help.go:183 +0x158 fp=0x14026e29570 sp=0x14026e294b0 pc=0x1017ec358
golang.org/x/tools/gopls/internal/golang.SignatureHelp({0x101d98488?, 0x1400d9ae3f0?}, 0x1403f35e5a0, {0x101d9b1e0, 0x1401cfcc900}, {0x108f6b4?, 0x1?})
	/Users/dan/go/src/golang.org/x/tools/gopls/internal/golang/signature_help.go:123 +0x98c fp=0x14026e297d0 sp=0x14026e29570 pc=0x1017ebcec
golang.org/x/tools/gopls/internal/server.(*server).SignatureHelp(0x1400018c780, {0x101d984c0, 0x1400d74abe0}, 0x1400d9ae300)
	/Users/dan/go/src/golang.org/x/tools/gopls/internal/server/signature_help.go:31 +0x194 fp=0x14026e298f0 sp=0x14026e297d0 pc=0x101859674
golang.org/x/tools/gopls/internal/protocol.serverDispatch({0x101d984c0, 0x1400d74abe0}, {0x101db3920, 0x1400018c780}, 0x1400d9ae2d0, {0x101d98680, 0x14031016500})
	/Users/dan/go/src/golang.org/x/tools/gopls/internal/protocol/tsserver.go:712 +0x3d5c fp=0x14026e29bd0 sp=0x14026e298f0 pc=0x1012c694c
golang.org/x/tools/gopls/internal/lsprpc.(*streamServer).ServeStream.ServerHandler.func3({0x101d984c0, 0x1400d74abe0}, 0x1400d9ae2d0, {0x101d98680, 0x14031016500})
	/Users/dan/go/src/golang.org/x/tools/gopls/internal/protocol/protocol.go:160 +0x74 fp=0x14026e29c30 sp=0x14026e29bd0 pc=0x101862384
golang.org/x/tools/gopls/internal/lsprpc.(*streamServer).ServeStream.handshaker.func4({0x101d984c0, 0x1400d74abe0}, 0x1400d9ae2d0, {0x101d98680, 0x14031016500})
	/Users/dan/go/src/golang.org/x/tools/gopls/internal/lsprpc/lsprpc.go:509 +0x6b8 fp=0x14026e29ee0 sp=0x14026e29c30 pc=0x101862198
golang.org/x/tools/gopls/internal/protocol.Handlers.MustReplyHandler.func1({0x101d984c0, 0x1400d74abe0}, 0x1404d3c5b60, {0x101d98680, 0x14031016500})
	/Users/dan/go/src/golang.org/x/tools/internal/jsonrpc2/handler.go:35 +0xc0 fp=0x14026e29f40 sp=0x14026e29ee0 pc=0x1012b49c0
golang.org/x/tools/gopls/internal/protocol.Handlers.AsyncHandler.func2.2()
	/Users/dan/go/src/golang.org/x/tools/internal/jsonrpc2/handler.go:103 +0x90 fp=0x14026e29fd0 sp=0x14026e29f40 pc=0x1012b47f0
runtime.goexit({})
	/usr/local/go/src/runtime/asm_arm64.s:1223 +0x4 fp=0x14026e29fd0 sp=0x14026e29fd0 pc=0x100fa0464
created by golang.org/x/tools/gopls/internal/protocol.Handlers.AsyncHandler.func2 in goroutine 109
	/Users/dan/go/src/golang.org/x/tools/internal/jsonrpc2/handler.go:100 +0x19c

Think I was in the middle of changing something like x.y(len(z), nil) to x.y(int64(len(z)), nil).
(I don't see this in local telemetry data so filing this issue)

Related Issues and Documentation

x/tools/gopls: panic in MethodSets: nil data for package #59138 (closed)
x/tools/gopls: panic related to go.mod #42949 (closed)
gopls: panic #42077 (closed)
x/tools/gopls: panic during ast walk for semantic token #47964 (closed)
x/tools/gopls: automated issue report (crash) - unhandled basic type #40956 (closed)
x/tools/gopls: crash during SignatureHelp #40230 (closed)
x/tools/gopls: automated issue report (crash) #40449 (closed)
x/tools/gopls: panic in internal/lsp/source.pathLinkAndSymbolName #40479 (closed)
x/tools/gopls: InlayHint: nil panic due to ast.CallExpr.Fun with no type #67142 (closed)
x/tools/gopls: calling outgoingCalls from an var func panics  #43376 (closed)

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

Thanks for the report. This looks like a recent tegression caused by https://go.dev/cl/605983, which changed callExpr to become potentially nil.

Thanks for the fix!
Checking: is this something that telemetry should have caught and logged locally? I have similar stacks in my local data from the 2023-11-01 timeframe but not since then.

@danp yes this should have been caught by telemetry, though it wouldn't have been uploaded due to the development version.
I'll write a test to ensure we're capturing local crashes. There may be some nondeterminism (races at process exit) that we need to sort out.


Stuck on 'Connecting to Steam as '

Seems to have happened today.
Tried from 2 different machines and 2 different accounts, both hang at the same spot (login). Left for 5 minutes, no progress, errors or timeout etc.
Input:
.\steamcli.exe --username=<username>--password=<password> --app-download --targetdir=D:\test --branch=public --os=windows --appid=233780 --depotid=233781 --synctarget

Output:
Connecting to Steam as <username>...

Version:
PS C:\Users\Cyruz\Downloads> .\steamcli.exe --version
steamcli <IP_ADDRESS>

Hey, thanks for reporting this. The issue has been fixed in the underlying library, I'll try to get a new version of SteamCLI out with it today.

Update published at https://github.com/BytexDigital/SteamCLI/releases/tag/<IP_ADDRESS>
Let me know if it works for you

Verified and working, ty!


[NOTABUG] Feature Ideas

A friend showed me this, and I wanted to jot down some thoughts/ideas that could make this really neat.

Kerberos Support: This would also require action on Proxmox's part, but in a perfect world you should be able to login to either a Windows or *nix host and use the Kerberos ticket generated by AD or whatever IdM solution you use to have the VDI client login to Proxmox, and then delegate the TGT to the VDI session host via SPICE or RDP (if Windows).

RDP Support: It would be neat if there was logic to allow you to connect to Windows VDI clients using RDP if the host is also running RDP. This would especially be useful with my point above, that way you can just login with AD to the VDI session host.

Anyways, just my two cents worth. If I had any actual good programming skills, I'd try to help with this, but the way I am now I'd just get in the way.

2. RDP seems like an interesting idea, but there needs to be a mechanism for determining what IP/hostname to connect to and I'm not sure the best way to go about that.

IP addresses are available for VMs with qemu-guest-agent installed and running:

API docs: https://pve.proxmox.com/pve-docs/api-viewer/index.html#/nodes/{node}/qemu/{vmid}/agent/network-get-interfaces
Also accessible from from proxmoxer python wrapper: https://github.com/proxmoxer/proxmoxer/issues/36#issuecomment-702335540

Not every Windows VM has RDP ("home edition"), and a Linux VM can have RDP ("xrdp") installed.
Probably a good idea to require presence of string rdp in field tags in /nodes/{node}/qemu/{vmid}/config (or something similar). Auto-detection by just attempting the connection could generate noise in logs or even trigger fail2ban.
This can be set using qm set <vmid> --tags rdp.
Unfortunately it looks like Proxmox handles tags as a single string (that cannot contain =), instead of more generic format key1=valueA;key2=valueB,valueC,valueD;key3withoutvalue.
Proxmox accepts qm set 2001 --tags "rdp;rdp_ip-<IP_ADDRESS>", but using - instead of = is a hack.
Using rdp;rdp_if-eth2,eth3 in tags could work for VMs that have only some interfaces connected to accessible networks (e.g. do not attempt connecting to addresses for interfaces eth0,eth1,eth4,.. or tun0,... or any other interface).
For multiple possible IP addresses, client should select the first address from the list which is also on the same subnet as one of the client's interfaces (prefer LAN over VPN for rdp;rdp_if-eth0,tun0 when client is in LAN), if no IP address is on the same subnet, try connecting to the addresses in listed order.
Combining rdp;rdp_ip-<IP_ADDRESS> or rdp;rdp_ip-vm.example.com could work for VMs without qemu-guest-agent.

I'm late to the conversation but it has to do with why I came here in the first place.  Proxmox has LDAP functionality and you can join it to a domain/realm (I'm using FreeIPA which is upstream Red Hat IdM and it Just Works ™)
That said, am I understanding correctly that PVE (local) auth is all that will work with this right now?  I tried pam auth which is a local system user and that didn't work.  Replacing that with 'ldap' in the config and no joy.

You have to use the realm name, which is configured in ProxMox:
[image: image.png]
On Thu, May 5, 2022 at 11:58 PM hidrawlic @.***> wrote:

I'm late to the conversation but it has to do with why I came here in the
first place. Proxmox has LDAP functionality and you can join it to a
domain/realm (I'm using FreeIPA which is upstream Red Hat IdM and it Just
Works ™)
That said, am I understanding correctly that PVE (local) auth is all that
will work with this right now? I tried pam auth which is a local system
user and that didn't work. Replacing that with 'ldap' in the config and no
joy.
—
Reply to this email directly, view it on GitHub
https://github.com/joshpatten/PVE-VDIClient/issues/8#issuecomment-1119260436,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABIP64UHSZHLRQFYU3AC7SDVISRHLANCNFSM5S75ILFA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

You have to use the name of the realm in the config file, which is in the authentication section of proxmox:

Ha.  Caught up in my own cleverness.  I had tried that already; unfortunately I forgot that the realm name needs to be in all-caps.


[BUG] Jobs page returns a 404 error

Describe the bug.
The jobs link https://www.asyncapi.com/jobs, available on the footer returns 404 page.
Expected behavior
Even if jobs are not present, we should show a valid page mentioning that jobs are currently not available, there should be a valid page shown instead of page not available
Screenshots

How to Reproduce

Open AsyncAPI Website - https://www.asyncapi.com/
Navigate to footer
Click on Jobs

🥦 Browser
Google Chrome
👀 Have you checked for similar open issues?

[X] I checked and didn't find similar issue

🏢 Have you read the Contributing Guidelines?

[X] I have read the Contributing Guidelines

Are you willing to work on this issue ?
Yes I am willing to submit a PR!

@aditygrg2 would like to work on this issue
We just need to remove the label from the footerlist.js

I'm willing to contribute.

Is this the expected behaviour, coz I get this page when I click Jobs from the footer.
I can make a PR if something needs to be changed.

Hi, i want to work on this issue. Can I have a sample template of the design/structure for the jobs page.

We need to remove the jobs page
So we need to just remove the jobs from footer

I can do that for you.Assign me this I'll make a PR asap.


Use SLE15 for the deployer

Uses SLE15 for the deployer.

Removes 2 lines from the velum Gemfile.lock for bundle 2.0 compatibility
Adds an extra vendor from the devel:languages:ruby for zypper
Drops rng-tools from the required packages as it doesnt seem to be used
Adds the needed repos for SLES15-JeOS
Changes the openstack image used from Leap 15.0 to SLES15-JeOS

SLE15-SP1 support may come later on monday

@Itxaka to some extend this conflicts with my leap 15.1 enablement work. can we decide in which order we move forward?

@Itxaka to some extend this conflicts with my leap 15.1 enablement work. can we decide in which order we move forward?

I was hoping we could support both leap && SLE, but as far as I understood, the product will only support SLE once released no?
To be fair this is not clear enough yet, what do we need to support in the final Cloud10 release? Cant we work on leap && SLE in parallel?

I was hoping we could support both leap && SLE, but as far as I understood, the product will only support SLE once released no?

Yes. The product will only support SLES.

To be fair this is not clear enough yet, what do we need to support in the final Cloud10 release? Cant we work on leap && SLE in parallel?

We need to support SLES, but having openSUSE working has a couple of advantages (like being able to use it for CI stuff)

Yes. The product will only support SLES.

Then its clear, we need to jump on the SLE train asap :)

We need to support SLES, but having openSUSE working has a couple of advantages (like being able to use it for CI stuff)

But we can use SLE for CI with no problems, which is also better because we test what customers will use.

So I just checked, and we could work with this in parallel no issues, @dirkmueller patch will only conflict with this in the deploy_on_openstack_oshnode_image var. The rest is not only compatible but some of it needed (ansible_python_interpreter for once)

I thought the move to sle 15 was only expected for caasp4, won't this be repeated work?

I thought the move to sle 15 was only expected for caasp4, won't this be repeated work?

This is for the deployer in preparation for cloud10 release. In any case, this PR is a testbed, a new one will come with 15-sp1 support based off this

Else it looks okay to me.

Moving to SP1 so this can be closed: https://github.com/SUSE-Cloud/socok8s/pull/628


Migrations is missing

Hi, this lib breaks my test suit, it reports:
django.db.utils.ProgrammingError: relation "auth_user" does not exist"

If i run python manage.py makemigrations django_cas_ng && python manage.py migrate it works again.
pip freeze:
boto==2.38.0
click==6.0
Django==1.8.7
django-cas-ng==3.5.3
django-debug-toolbar==1.3.0
django-filter==0.11.0
django-mama-cas==1.2.0
django-mptt==0.7.4
django-nose==1.4.2
django-reversion==1.9.3
django-rosetta==0.7.6
django-storages-redux==1.3
django-suit==0.2.15
django-wysiwyg-redactor==0.4.9
djangorestframework==3.3.1
djangorestframework-gis==0.9.6
ecdsa==0.13
Fabric==1.10.0
geopy==1.11.0
gitdb==0.6.4
GitPython==1.0.1
Jinja2==2.8
MarkupSafe==0.23
microsofttranslator==0.5
nose==1.3.7
paramiko==1.16.0
pipdeptree==0.4.3
polib==1.0.7
psycopg2==2.6.1
pycrypto==2.6.1
python-cas==1.1.0
python-dotenv==0.1.3
requests==2.8.1
six==1.10.0
smmap==0.9.0
sqlparse==0.1.18
Unipath==1.0
wheel==0.24.0

I try build a migration history soon

did you ran python manage.py syncdb ?

Yep, I have tried that, but i need to run makemigrations first, otherwise it does not work.

Migration are new in django 1.7 but django-cas-ng is suppose to work with django 1.5 and 1.6, thus I do not think migrations will be added to the repository.
Normally, django create the models tables for the apps without migrations as it did in version previous 1.7.
I think that you experienced is describe here https://docs.djangoproject.com/en/1.8/topics/migrations/#dependencies : id est, django-cas-ng model have a foreign key to user auth model. django-cas-ng is an unmigrated but the default auth model is in a migrated app and thus database creation are not happening in the right order.
The solution may be to run python manage.py migrate a first time before added django-cas-ng to INSTALLED_APPS``` and then add django-cas-ng to INSTALLED_APPSand run againpython manage.py migrate```.
The foreign key is here to allow cascade deletion: If a user is deleted all his ProxyGrantingTicket are deleted. As ProxyGrantingTicket allow django to request ProxyTicket (cas be seen as OTP passwords) for the user, it has seemed me safer to do it like this.
As the django doc says it is a bad idea, I may send a pull request deleting this foreign key and do ProxyGrantingTicket cleaning in a management command.
Can you confirm that the problem you encounter can be worked around by doing a two step migration ?

I did not have any problems to migrate django-cas-ng when first installing it (dependencies were already migrated). It is when ./manage.py test is creating the test-database the migration break.
So yes, i can confirm that a two step migration is working and that the assumptions in your comment is correct.
But i don´t understand why we can´t just add migrations to the app? Would it break backward compatibility?

@mikaelengstrom: It wouldn't. You would just have migrations work for Django 1.7+. 1.8 is now LTS, so it shouldn't be a problem.

until that happens, just manage.py makemigrations django_cas_ng and then
manage.py migrate

Fixed by #78 with change 4c4cf87a.


Auto-add hacktoberfest-accepted in the last week of October

Hi, thanks for the proposal this would be great from a maintenance perspective, but Hacktoberfest ToS doesn't allow it afaik, see this section:

Your PR/MR must be accepted by a maintainer before Hacktoberfest ends on October 31 (in any time zone, UTC-12 thru UTC+14) to be counted.


Implement lesson commands

Implement list and edit command for lessons
Fixes #194
Fixes #195

Codecov Report

Merging #196 into master will decrease coverage by 4.58%.
The diff coverage is 1.37%.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5bf79a5...2008e61. Read the comment docs.


x/mobile: memory

What version of Go are you using (go version)?

$ go version

Does this issue reproduce with the latest release?
What operating system and processor architecture are you using (go env)?
go env Output
$ go env

What did you do?

What did you expect to see?
What did you see instead?

No bug reported.


Support for secret sharing scheme

Is your feature request related to a problem? Please describe.
Secret sharing Scheme is a must-have for FATE project.
Describe the solution you'd like
Do R&D on implementing secret sharing operations, such as:

Create beaver triple
Add, Multiply, Division, Compare, and others

Having secret sharing operations been created, then

Implement secret sharing based LR
Implement secret sharing based FTL

These works do not need to be full-fledged for industrial applications. However, they should be able to help us create various secure federated learning algorithms/prototypes.

This is an important feature. Secret sharing based LR(hetero & homo) can work with mingchao & hartin.  They also focus on this in the next version and some work will be reused.
Also some numeric operator on secret-share will be implemented in secureprotol module.

Beaver triples maybe not a better solution which needs a trusted third party.
We can implement multiplication triplets by offline on HE, which don't need a trusted third party.

so , secret sharing feature now is not implemented?


Snackbar on clicking add to bookmarks.

Summary:
A snackbar with an option to directly open the bookmarks activity should be displayed when the user presses the add to the bookmarks icon. Similarly, when the user removes an item from bookmarks a snack bar should be shown notifying that the item has been removed from bookmarks.
Steps to reproduce:
Open a photo from the explore section. Click on the star icon to add to bookmarks.
Now a snackbar should be shown saying "Item added to bookmarks" along with an action to open the bookmarks activity.
Device and Android version:
Nokia 7 Plus, Android 10
Commons app version:
2.12.3-debug-master
Would you like to work on the issue?
Yes.

I have already sent a pr for the issue. I waited until the enhancement label had been added. There were no other comments, so once the label was added, I thought that it was fine to work on it since the changes were direct.
Apologies for the misunderstanding.

@misaochan @maskaravivek @nicolas-raoul how should we proceed with this issue. The thing I had in mind has been done in the pr I sent. If anything changes are needed, please let me know.

Showing a snack bar is good, although,  personally I don't see the need to add a link to go to the bookmarks screen.

@misaochan @maskaravivek @nicolas-raoul how should we proceed with this issue. The thing I had in mind has been done in the pr(tests are failing which will be fixed) I sent. If anything changes are needed, please let me know.

I agree with @ashishkumar468 , I don't think the link in the snack bar is necessary, as Bookmarks is directly accessible from the nav drawer.

Ok, I will remove that action.
I had another question regarding the failing tests.
Since I changed the signature of the updateBookmarks() function, they need to be changed in the test too and for that, we might need to make quite a few changes. So I can either make those changes or figure out a different way to show the snack bar.
Which option should I proceed with?


android: manually convert Format set to String

Because Set.toString() doesn't exist on all Android versions.

I was looking at your code, wanted to suggest a few things and ended up simply doing it at bit differently. May I ask which Android version is lacking the toString() function?

Ah, nice! I like your implementation better, too! 👍
Unfortunately, I can't really tell what version excactly, but I ran into this on an old device running Android 7 (API level 24). According to the documentation Set.toString(), which is inherited from AbstractCollection.toString(), should exist since API 1.
I guess Set.toString() was added with Android 8.0 (API level 26) which updated Java language support and brought support for "several additional OpenJDK Java APIs".


dotenv::var()'s error should report the variable that caused it

Currently this forwards the underlying std::env::VarError which has very unhelpful error messages: https://doc.rust-lang.org/src/std/env.rs.html#274-283
Namely, it doesn't report the variable that caused the error, which is useless for almost all intents and purposes unless you only ever invoke exactly it once in your application; even then, if someone tries to use your application that isn't familiar with this, they just get environment variable not found or environment variable was not valid unicode: <generic Utf8Error message>.
Unfortunately, this can't be changed in the stdlib for backcompat purposes since VarError is an enum, but I think dotenv::var() should definitely return a more useful error here, which gives it additional utility as well.

It's generally considered bad practice to include the arguments in your error type. Users want to be able to return errors about local values, and don't want to make redundant allocations.
User facing errors, on the other hand, do need this context. Your UI can include logic to report missing variables itself:
pub fn var<K: AsRef<OsStr>>(key: K) -> String {
    let s = key.as_ref();
    match env::var(&s) {
        Ok(s) => s,
        Err(e) => {
            eprintln!("Couldn't get {}: {}", s.to_string_lossy(), e);
            std::process::exit(1);
        }
    }
}

async-std reports context in its I/O errors and this was generally considered a very nice UX improvement: https://docs.rs/async-std/1.6.2/src/async_std/fs/file.rs.html#117
We do try to wrap calls to env::var() where possible to add this context but it's difficult to enforce. There's no lint that I know of to warn people when they should be using a custom-defined function rather than a library-defined function.
Eliminating allocations where possible is a noble effort but we have to consider the context of where this function is usually called. Does it often appear in hot loops? Is it called millions of times over the course of a process's life? I'm willing to bet the answer to both is "no".

Alocations should be avoided if they are redundant, but in this case, I think they are justified.
crate::Error not reporting the variable that caused the error forces some of us to create our custom implementation, which limits the usefulness of this crate.
I believe that errors should be able to have costful operations if it improves debugging for the developer or final user, because these errors usually occur only once and the program exits.

I've been thinking about how to make errors more helpful in dotenvy.
There's an issue here and a discussion here. Feedback and suggestions are welcome!


memory leak

Hi, why is there a memory leak in this code? and how to prevent it

https://github.com/pinqtape/kartikkk

You're saying that there is a memory leak in postter.js but hyper-express is not serving anything from postter.js. In fact, I am not even sure why you have hyper-express in postter.js as the job of postter.js just seems to be to send requests to hyper-express. What even is the point of creating a hyper-express server in postter.js without any routes and then listening it on port 6500?
The getter.js in which you have the hyper-express route parsing the responses as json is actually working phenomenally with very good memory usage of just ~80 MB.
 So you aren't seeing a memory leak but a really fast memory exhaustion from all those fetch requests being made to a local server. Keep in mind, 1000 / 5 = 200 aka. your postter.js is sending out ~200 requests per second to hyper-express. This type of request ratre is nothing for hyper-express and as you can see from your getter.js, hyper-express has no trouble handling these requests. It's your postter.js which is struggling because fetch is very memory intensive, especially when you're firing off 200 requests / second.
Please review your code again carefully to notice the mistakes.

I just wanted to reproduce everything as it was in my production, you are right about hyper-express in postter, the leak is due to node-fetch, but I do not know how to solve it... thank you!


Obfuscation

How can I trust this tool with all this obfuscation?!!

What obfuscation? The code is completely open for anyone to review.


Affinity for gateways

Gateway pods are typically run on dedicated high capacity nodes. For this to work, gateway pods need:

node selectors
affinity/anti-affinity (beyond existing arch selection)
custom annotations (for external LBs)

@rshriram: our customer requested that we place affinity on ingress gateway. Please don't edit the label.

Can this issue be expanded for other plane components eg. pilot (all of them would be nice with defaults in helm)

what is the istioctl install equivalent of this?


HA won't stop

I'm trying to stop HA and sometimes it just won't stop, it keeps hanging on closing the database.
I have this issue on Windows 8.1.
Sometimes it works, sometimes it doesn't.
I'm trying to close HA either by CTRL-C in the command prompt or by the homeassistant.stop service

When stopping home Assistant it will finish all current en queued jobs. So
check your logs to make sure you don't have a lot of pending jobs from some
broken platform, because in that case it won't be able to quit.
On Tue, Jan 12, 2016, 03:23 Bart274<EMAIL_ADDRESS>wrote:

I'm trying to stop HA and sometimes it just won't stop, it keeps hanging
on closing the database
I have this issue on Windows 81
Sometimes it works, sometimes it doesn't
I'm trying to close HA either by CTRL-C in the command prompt or by the
homeassistantstop service
—
Reply to this email directly or view it on GitHub
https://github.com/balloob/home-assistant/issues/885.

I when its hung on closing database I have always been able to issue a second ctrl-c to stop it the process it is hung on to finish closing promptly.

Is this still an issue with the latest dev branch?

This is till valid for 0.19.0.dev0, it hangs here:
16-05-01 11:18:07 homeassistant.core: Stopping
16-05-01 11:18:07 homeassistant.core: Bus:Handling <Event homeassistant_stop[L]>
16-05-01 11:18:07 homeassistant.components.recorder: Closing database
```

That log doesn't mean it hangs there. It can also be that that is the last
printed thing. It usually is zwave that us slow stopping
On Sun, May 1, 2016, 10:19 Hernán<EMAIL_ADDRESS>wrote:

This is till valid for 0.19.0.dev0, it hangs here:
16-05-01 11:18:07 homeassistant.core: Stopping
16-05-01 11:18:07 homeassistant.core: Bus:Handling <Event homeassistant_stop[L]>
16-05-01 11:18:07 homeassistant.components.recorder: Closing database
—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/home-assistant/home-assistant/issues/885#issuecomment-216044483


adding mix to the mix

Have you ever considered setting up exercises with mix in the mix?
As a new elixir user, I was interested in adding credo to my project to see how stylee my code was but it says "add to your mix.exs" and it made me wonder how mix works into an elixir project?
I know other languages have a different directory structure (e.g. java).  Setting up the initial directory structure for exercises that would support mix or even with a default mix.exs file might  be nice.
Thoughts?

There has been some discussion around this.  (See Issue #218 and Issue #245 for examples).
I am personally neutral -- I think it's nice for beginners to focus on just the language instead of having to learn the ecosystem, but I understand that it's probably more idiomatic to use mix than not.

Yeah - i see how the system is nice and simple - not mixing in any build tools.
I know the Java track uses gradle but other tracks i've run (ruby, clojure, javascript) don't have anything of note.
just thought i'd ask.
thanks for the response


getting 'can not process border-bottom of null' error

I have<EMAIL_ADDRESS>(latest) and<EMAIL_ADDRESS>(latest) when I got the error.  Downgraded to<EMAIL_ADDRESS>and<EMAIL_ADDRESS>the error went away.
 ping me for the source that shows this issue.

Apparently this was resolved sometime around 3.0.0-alpha.6


TypeError: zone.onError is not a function

This is what I see on Kadira, on Chrome / Windows 7.

We are using a very old zones version. I think we can do a update today.
@mnmtanish can we?


XML name should be used if present in JsonProperty annotation

Based on some troubleshooting work I did with @rickle-msft. It seems like we don't properly handle cases where the XML name is different than the property name.
Sounds like @lmazuel is planning on adding test coverage in autorest.testserver so we may want to wait for that.

I do that today and send an email, thanks!

@RikkiGibson Done in testserver 2.5.17.
Python tests: https://github.com/Azure/autorest.python/pull/63

I hoped I'd fixed this problem in 265 but I introduced a new one-- I think it's necessary to use CompositeType.XmlProperties.Name instead of CompositeType.XmlName. Will try to get to this one today..

Hey @RikkiGibson
What change in the modeler? Could you describe your scenario, so I test it in Python? If it works, it's definitely a Java issue :p. If it doesn't, might be a Java/Python issue or modeler. But my current feeling is that modeler has no trouble.
Thoughts?

@lmazuel and I hashed it out. I have basically concluded that this is a corner case where OpenAPI limitations interact with the AutoRest modeler such that it's not practical to make the AutoRest modeler just do the convenient thing for us.


impl Visitor and FromStream for IgnoredAny

closes #2
i pretty much just copied over the implementation from serde here.  I've commented out a few types that don't seem to be reflected in your Visitor trait, if there are no plans to add those functions to the trait then we can delete that code.
let me know if there's anything else (documentation, tests, etc) you'd like in addition to this!

also - I recreated serde's IgnoredAny struct rather than importing it from their crate, to avoid the extra dependency.  alternatively, can implement your traits for their struct, but i figure this way was preferred.


Tokenization Key or Client token was invalid

I am generating a token from "https://api.sandbox.paypal.com/v1/oauth2/token", and after that setting the token in requestOneTimePayment and getting [Error: Tokenization Key or client token was invalid.]

can you solve this problem

didnt get any solution

same issue
any help?

@Prashantrao331
Was this on android?  I'm curious if it's the android version of this iOS error
https://github.com/smarkets/react-native-paypal/issues/15
Regardless I think the issue is due to a bad token rather than this library
see this stack overflow answer by a Braintree employee
It may be worth regenerating a new token and trying again.  Make sure there aren't any extra spaces or weird string encoding in your token

this gave me access token but still got the same error
`import requests
d = {"grant_type" : "client_credentials"}
h = {"Accept": "application/json", "Accept-Language": "en_US"}
cid = "ASOGsGWr7yxepDuthbkKL-WoGNVAS7O0XlZ2ejcWsBA8ZXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
secret = "EJTKAFEYfN9IaVHc4Y-MECzgBivt2MfW6rcyfbVky0T07yRwuuTdXOczuCoEIXXXXXXXXXXXXXXX"
r = requests.post('https://api.paypal.com/v1/oauth2/token', auth=(cid, secret), headers=h, data=d).json()
access_token = r['access_token']`

this gave me access token but still got the same error
import base64
import requests
from backend.config import Config
def get_paypal_access_token(client_id=Config.PAYPAL_CLIENT_ID, secret=Config.PAYPAL_SECRET):
url = "https://api.sandbox.paypal.com/v1/oauth2/token"
payload = 'grant_type=client_credentials'
encoded_auth = base64.b64encode((Config.PAYPAL_CLIENT_ID + ':' + Config.PAYPAL_SECRET).encode())
headers = {
'Authorization': f'Basic {encoded_auth.decode()}',
'Content-Type': 'application/x-www-form-urlencoded'
}
r = requests.request("POST", url, headers=headers, data=payload)
assert r.status_code == 200
return r.json()["access_token"]

i was also able to get access token through api.sandbox, but still no success.

Hey everyone, I've updated the README and created a new example app included in the new version of the package (3.1.0).  Please check that out and see if it helps diagnose and fix this bug.  Message me here if you are still experiencing issues and I will reopen

Hey everyone, I've updated the README and created a new example app included in the new version of the package (3.1.1). Please check that out and see if it helps diagnose and fix this bug. Message me here if you are still experiencing issues and I will reopen

still facing this issue

@nzankich  please help


TMPL syntax highlighting

First of all thank you for this amazing plugin. I'm kinda new to Neovim; and I'm trying to setup a good Go development environment. Everything seems to work as expected except the files .tmpl or .tmpl.html or .html.tmpl.
Any advice as to get proper experience with those files? 🙏
Thank you very much!

Pushed a change for it.

@ray-x that seemed to do that trick. There are some caveats, but I guess the next step would be to have a proper treesitter plugin. I will look into it; nonetheless thank you very much!!!


Redact passwords from tasks fetched from the TaskQueue

Fixes #15882

Description

Active tasks fetched from TaskQueue's in memory state may reveal sensitive information.
This doesn't happen for payloads fetched from the metadata store, as they are read using a json mapper with an added mixin for redacting credentials.
This PR leans on a similar change in the TaskQueue where a payload is serialized and then deserialized.

Release note

Key changed/added classes in this PR

TaskQueue

This PR has:

[ ] been self-reviewed.

[ ] using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)

[ ] added documentation for new or modified features or behaviors.
[ ] a release note entry in the PR description.
[ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
[ ] added or updated version, license, or notice information in licenses.yaml
[ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
[ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
[ ] added integration tests.
[ ] been tested in a test Druid cluster.

Thank you for the reviews @kfaraz @cryptoe


Auditor: Allow users to run migrations with docker container

Closes #765
I also added a little bit more information to the documentation (how to set up the PostgreSQL database using Docker)

Codecov Report

Merging #766 (70f1305) into main (c445b5b) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #766   +/-   ##
=======================================
  Coverage   63.15%   63.15%
=======================================
  Files          44       44
  Lines        5607     5607
  Branches       93       93
=======================================
  Hits         3541     3541
  Misses       2066     2066

Components
Coverage Δ

apel_plugin
74.80% <ø> (ø)

auditor
82.18% <ø> (ø)

slurm_collector
65.17% <ø> (ø)

slurm_epilog_collector
0.00% <ø> (ø)

htcondor_collector
∅ <ø> (∅)

priority_plugin
35.57% <ø> (ø)


Vm trace changes

MERGE AFTER https://github.com/ethereum/remix/pull/21
bug fix: retrieve code for internal calls

I'm afraid I don't have anything very constructive to offer, having never seen the code before, other than noting formatting inconsistencies.

I will create a new that fit with coding standards


Pre-compute and persist change points

This addition is focused on:

Only pre-compute change points for logged in user. The /default/ URLs work like before.
We split the pure change point math from the GithubReport. The API between the two parts is AnalyzedSeries, and that is essentially what is cached in MongoDB.

Also decrease the client side caching to 10 minutes.

Didn't even try to make tests work, but this might just about work now so wanted to share.

Didn't even try to make tests work, but this might just about work now so wanted to share.

Do you still want me to review even without working tests? Are you looking for a review of the general approach?

The latter, but tests are working now. I just didn't push a commit that moves forward the hunter submodule. All test failures are because of that.


Incorrect service grafana dashboard with traffic split

With traffic splitting enabled during canary deployment, service dashboard show incorrect success percentages.
Success rate varies from 50% to 150%.
https://snapshot.raintank.io/dashboard/snapshot/mimQl4yS42oWPWpaW0na0rDozt5jRYLf

This appears to be caused by, at least in part, the difference in use of irate and rate in the numerator and denominator.

Closing, based on istio/installer/pull/166.
Please re-open if that does not resolve issue.


[Enhance] Optimize IO for flow_warp

This version of flow_warp is faster than original one and the consistency of results between two implementations has been checked.

Fix #878

Hi @NK-CS-ZZL ！First of all, we want to express our gratitude for your significant PR in the mmediting project. Your contribution is highly appreciated, and we are grateful for your efforts in helping improve this open-source project during your personal time. We believe that many developers will benefit from your PR.
We would also like to invite you to join our Special Interest Group (SIG) private channel on Discord, where you can share your experiences, ideas, and build connections with like-minded peers. To join the SIG channel, simply message moderator— OpenMMLab on Discord or briefly share your open-source contributions in the #introductions channel and we will assist you. Look forward to seeing you there! Join us ：https://discord.gg/raweFPmdzG
If you have WeChat，welcome to join our community on WeChat. You can add our assistant ：openmmlabwx. Please add "mmsig + Github ID" as a remark when adding friends：）
Thank you again for your contribution❤


Make backcalls hushed by default?

It occured to me that it doesn't really make sense for backcalls to return by default, since generally speaking there's little to no reason to ever have an (async) callback return (unless we're talking about early termination, but you'd still have to write an explicit return in LiveScript for that). Sure, the LS website has this example:
x <- map _, [1 to 3]
x * 2

But it feels rather contrived to me, plus if you really wanted to do something like that you could still explicitly return. Or am I missing some real usecases on backcalls where returns are actually useful?

They have non-async purposes as mentioned, so I think they should be left as-is for consistency.


Settings

Btw, when you are ready for me to re-review, please request it. Will wait for you 👍

Did you see my comment? It wasn't addressed as requested and just merged. Was that a mistake?

@zpoken? ⬆


fix: environments data source test flake

~The environments data source is the most recent victim of the "total count wrong" test flake. Before this, the SLO data source test had this before and it flaked so much I removed it #493.~
~This writes a helper to compare that the number of items in an attribute is at least what we expect it to be and restores the SLO data source total count check.~
Remove the flakey all count comparison -- but continue to run a data source without a filter.

I'm skeptical that this is a test worth keeping. "There's probably some stuff in there" doesn't seem too useful to me. 🤔

@dstrelau fair! I'll just rip out the test for all 👍🏻

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 78.50%. Comparing base (0233e88) to head (9501307).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #525      +/-   ##
==========================================
- Coverage   78.57%   78.50%   -0.08%
==========================================
  Files          73       73
  Lines        5606     5606
==========================================
- Hits         4405     4401       -4
- Misses        910      914       +4
  Partials      291      291

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


[Webpack 4] React-router does not render any routes when I build with NODE_ENV=production

Do you want to request a feature or report a bug?

What is the current behavior?
If the current behavior is a bug, please provide the steps to reproduce.
Hello,
When I build with NODE_ENV=production, React-Router does not render any routes. Otherwise, I works in developpment.
build script
"build": "rm -rf ./dist && NODE_ENV=production webpack --config build/webpack.config.js -p"

.babel-rc
{
  "presets": [
    ["@babel/preset-env", {
      "modules": false,
      "targets": {
        "browsers": ["last 2 versions"]
      },
      "development": {
        "plugins": [
          "babel-plugin-styled-components"
        ]
      },
      "production": {
        "plugins": [
          "babel-plugin-styled-components"
        ]
      },
      "exclude": ["transform-regenerator", "transform-async-to-generator"]
    }]
  ],
  "plugins": [
    ["@babel/plugin-transform-react-jsx", {"pragma": "h"}],
    ["@babel/plugin-proposal-object-rest-spread"],
    ["transform-react-remove-prop-types"],
    ["babel-plugin-styled-components"],
    ["syntax-dynamic-import"],
    ["module:fast-async", {
      "compiler": {
        "promises": true,
        "generators": false
      },
      "runtimePattern": "home.js",
      "spec": true
    }]
  ]
}

webpack.config.js
const config = require('../config.js');
const path = require('path');
const webpack = require('webpack');
const htmlWebpackPlugin = require('html-webpack-plugin');
const TidyHtmlWebpackPlugin = require('tidy-html-webpack-plugin');
const CopyWebpackPlugin = require('copy-webpack-plugin');
const FriendlyErrorsPlugin = require('friendly-errors-webpack-plugin');
const OptimizeCSSPlugin = require('optimize-css-assets-webpack-plugin');
const PreloadWebpackPlugin = require('preload-webpack-plugin');
const { InjectManifest } = require('workbox-webpack-plugin');
const autoprefixer = require('autoprefixer');
const ProgressBarPlugin = require('progress-bar-webpack-plugin');
const MiniCSSExtractPlugin = require('mini-css-extract-plugin');
const UglifyJsPlugin = require('uglifyjs-webpack-plugin')
const production = process.env.NODE_ENV === 'production';

const sw = path.join(__dirname, '../src/sw.js');
const plugins = [
  new MiniCSSExtractPlugin({
    filename: '[name].[contenthash].css'
  }),
  ProgressBarPlugin()
];

const devServer = {
  contentBase: config.contentBase,
  hot: true,
  hotOnly: true,
  historyApiFallback: true,
  port: config.port.front,
  compress: production,
  inline: !production,
  hot: !production,
  stats: {
    assets: true,
    children: false,
    chunks: false,
    hash: true,
    modules: false,
    publicPath: false,
    timings: true,
    version: false,
    warnings: true
  }
}

if (production) {
  plugins.push(
    new webpack.optimize.OccurrenceOrderPlugin(),
    new htmlWebpackPlugin({
      template: config.template,
      minify: {
        removeComments: true
      },
      //cache: true,
      // make it work consistently with multiple chunks
      chunksSortMode: 'dependency'
    }),
    // preload main bundles
    // prefetch should be done with webpack
    // when native support for prefetch will land
    new PreloadWebpackPlugin({
      rel: 'preload',
      include: 'initial'
    }),
    // new TidyHtmlWebpackPlugin(),
    new CopyWebpackPlugin([
      {
        from: path.resolve(__dirname, '../src/assets/'),
        to: path.resolve(__dirname, '../dist/assets/')
      },
      {
       from: path.resolve(__dirname, '../src/manifest.json'),
       to: path.resolve(__dirname, '../dist/manifest.json')
      },
      {
        from: path.resolve(__dirname, '../src/third_party/idb-keyval.min.js'),
        to: path.resolve(__dirname, '../dist/third_party/idb-keyval.min.js')
      }
    ]),
    new InjectManifest({
      swSrc: sw
    })
  );
} else {
  plugins.push(
    new webpack.HotModuleReplacementPlugin(), // hot reload
    new htmlWebpackPlugin({ // generate index.html
      template: config.template,
    }),
    new FriendlyErrorsPlugin()
  );
};

const common = {
  devtool: config.devtool,
  mode: production ? 'production' : 'development',
  // do not continue build if any errors
  bail: true,
  // watch mode
  watch: !production,
  entry: {
    app: config.entry.front
  },
  output: {
    path: path.resolve('dist'),
    filename: production ? '[name].bundle.[hash].js' : '[name].bundle.js',
    publicPath: '/'
  },
  resolve: {
    extensions: ['.js', '.jsx', '.css', '.scss'],
    alias: {
      // in order to use css-transition-group
      // you have to aliase react and react-dom
      react: 'preact-compat',
			'react-dom': 'preact-compat',
			'react-addons-css-transition-group': 'preact-css-transition-group',
      components: config.componentsPath,
      routes: config.routesPath,
      src: config.staticPath
    }
  },
  module: {
    rules: [{
      test: /\.(css|scss)$/,
      use: [
        MiniCSSExtractPlugin.loader,
        {
          loader: 'css-loader', options: {minimize: true}
        },
        {
          loader: 'sass-loader'
        }
      ]
    },{
      test: /\.(js|jsx)$/,
      exclude: /node_modules/,
      include: path.resolve(__dirname, "../src"),
      loader: 'babel-loader'
    },
    {
      test: /\.(.png|svg)$/,
      // compress images
      loader: 'image-webpack-loader',
      // Ensure this loader run before svg-loader or url-loader
      enforce: 'pre'
    },
    // {
    //   test: /\.svg$/,
    //   loader: 'svg-url-loader',
    //   options: {
    //     // Inline image smaller than 10kb
    //     limit: 10 * 1024,
    //     // remove quotes from url
    //     // https://developers.google.com/web/fundamentals/performance/webpack/decrease-frontend-size
    //     noquotes: true
    //   }
    // },
    {
      test: /\.(png|svg)$/,
      loader: production ? 'file-loader' : 'url-loader',
      query: {
        limit: 10 * 1024,
        name: '[name]-[hash:7].[ext]'
      }
    }]
  },
  optimization: {
    concatenateModules: production,
    nodeEnv: process.env.NODE_ENV,
    minimize: production,
    // optimization.minimizer overrides default optimization
    // in webpack 4.
    // plugin optimizer should be put here
    // not in plugins anymore as before
    minimizer: [
      // minify js
      new UglifyJsPlugin({
        cache: true,
        parallel: true,
        sourceMap: false
      }),
      // Compress extracted CSS.
      // Possible duplicated CSS from differents components can be deduped.
      new OptimizeCSSPlugin({
        cssProcessorOptions: {
          safe: true
        }
      })
    ],
    runtimeChunk: true,
    splitChunks: {
      chunks: 'all'
    }
  },
  performance: {
    hints: production ? 'warning' : false
  },
  plugins,
  devServer
};

module.exports = common;

What is the expected behavior?
As it works in dev, it should work in prod.
If this is a feature request, what is motivation or use case for changing the behavior?
Please mention other relevant information such as the browser version, Node.js version, webpack version, and Operating System.

Thanks for your report. It would be great if you reduce your issue to a small reproducible example. Best put this example into a github repository together with instructions how to get to the problem.

Clone this repository :
https://github.com/Mathieu-R/streamwave
Run npm install and npm run build.

@Mathieu-R There is a lot of things happening on that repo:
After building and serving dist directory I get an empty page without errors, but if you immediately hit Cmd+Shift+R it would load correctly and redirect you to http://localhost:5000/auth, it seems related with the ServiceWorker that is running.
I am also getting this error after building your site using development config:

Removing call that install service worker completely solves the problem.
That issue is not related with webpack.

Well I removed the service-worker and still have the same problem.
The problem disappear as soon as I disable uglifyjs.
Could you reopen the issue ?

That seems to work. Thanks :) !

@Mathieu-R @wyze How could I ever thank you. I've been stuck on this issue for over a month now. The compress.unused: false totally fixed it. I'm curious why this is occuring though. I originally thought it was just for dynamic imports, but when I tried it for regular static imports in react-router, it didn't work as well.


Add 'make start-with-vnext' command

Add 'make start-with-vnext' command that includes the 'current' version in onlyIncludeVersions in the docusaurus config, so that one can visualize the 'next' version of the locally rendered website (which doesn't show up in 'make start' since this PR: https://github.com/tigera/docs/pull/1052/)

Product Version(s):

Issue:

Link to docs preview:

SME review:

[ ] An SME has approved this change.

DOCS review:

[ ] A member of the docs team has approved this change.

Additional information:

Merge checklist:

[ ] Deploy preview inspected wherever changes were made
[ ] Build completed successfully
[ ] Test have passed

@ronanc-tigera @HugoJLFerreira Anything to add?

LGTM


Firebug breaks Animations side panel in other themes

The Animations side panel display is broken when you switch to the dark or light theme.

Sebastian

I can't reproduce this, I think it's fixed by
https://github.com/firebug/firebug.next/commit/d7327505c5d81da41312b43d743ee9881c1af0db
Honza

Closing, feel free to reopen.
Honza

Yes, it's ok now.
Sebastian


Listener is not active

c:\Program Files\nodejs\node_modules\steam-session\dist\helpers.js:17
let err = new Error(errorMessage || resultMsg);
^
Error: RateLimitExceeded

You exceeded the rate limit

Why can't the Error event be intercepted, but crash the program directly?

It can be. That Error comes from the promise returned by startWithCredentials being rejected.


Introduce switch logic

Introduces switch logic for #310
Moved assertion up to ensure it works as intended
Import switch and add it to all

Example code to test function where we have a scene with only the asset hero loaded
import avalon.io as io
import avalon.api as api

host = api.registered_host()

# Example 1:
# Swap subset modelDefault of hero for modelLod0 of hero
asset = io.find_one({"type": "asset", "name": "hero"})
subset = io.find_one({"type": "subset",
                      "name": "modelLod0",
                      "parent": asset["_id"]})

version = io.find_one({"type": "version",
                       "parent": subset["_id"]},
                        sort=[('name', -1)])

representation = io.find_one({"type": "representation", "parent": version["_id"]})

container = list(host.ls())[0]
api.switch(container, representation)

# Example 2:
# Scene contains model of hero but model needs to be of villain
asset = io.find_one({"type": "asset", "name": "villain"})
subset = io.find_one({"type": "subset", "name": "modelDefault",
                      "parent": asset["_id"]})

version = io.find_one({"type": "version",
                       "parent": subset["_id"]},
                      sort=[('name', -1)])

representation = io.find_one({"type": "representation",
                              "parent": version["_id"]})

container = list(host.ls())[0]

api.switch(container, representation)

This looks good. Just to be sure, this is functional - right? All that's needed is to implement a switch method on the Loader classes that the studio's  configuration has implemented, correct?

Aside from that, I think the next step is to have one of the tools expose this switch functionality to the artist. Which is slightly more involved and could maybe become a PR of its own to keep this one lightweight? @mottosso what do you think?

Just to be sure, this is functional - right?

Yes it is functional, as you said, a switch method is needed in a Loader in order to make it work.

Here is an example of how it can work. You can fill in all three of the fields to switch to a specific representation.

 I believe this is mergable. :muscle:

Also, should we bump the minor version of Avalon? (I believe we haven't done that recently with other PRs).
I also noticed just now that the Avalon version is set to 5.0.0 - any reason it's already at 5?
Should we maybe bump this to 5.1.0 then?
@mottosso what do you think?

Yeah it's semantic versioning and major version increments is for when it breaks backwards compatibility. That's how many times that has happened so far. Ideally it wouldn't happen so frequently, but I'd recommend we keep doing it as both a reminder and notifier for others using Avalon.

I bumped the minor version. I'd say, @aardschok, feel free to merge once the checks have passed.


[9.x] Collection Helper data_get Not Using Default Value

Currently the function will only return default value for the parameter $default. This fix corrects the issue on * segments and will now return the value specified in $default.

@tehomen There is already a parsing before the process gets there, so if it needs to get the default value, it has already taken it, no need for it to be there. It is not included because it will return null under all conditions.

@tehomen There is already a parsing before the process gets there, so if it needs to get the default value, it has already taken it, no need for it to be there. It is not included because it will return null under all conditions.

When I call the function like so:
data_get([ 'items' => [ [ 'name' => 'foo', 'id' => 12 ], [ 'id' => 13 ] ] ], 'items.*.name', 'bob');

I would get back an array with [ 'foo', null ] even though I specified to return 'bob' by default.

https://github.com/laravel/framework/blob/4d46bda6f19d490080512fa053065081f737bf2f/tests/Support/SupportHelpersTest.php#L102
https://github.com/laravel/framework/blob/4d46bda6f19d490080512fa053065081f737bf2f/tests/Support/SupportHelpersTest.php#L105
https://github.com/laravel/framework/blob/4d46bda6f19d490080512fa053065081f737bf2f/tests/Support/SupportHelpersTest.php#L117
If you change the null parts to irrelevant, the errors you get will be gone.

This seems to break the test suite. Please resubmit this with passing tests.


Windows Build Error

Describe the bug

set CGO_ENABLED=1
set GOOS=windows
set GOARCH=amd64
set CC=x86_64-w64-mingw32-gcc
go build
error info:
no Go files in D:\HackBrowserData

Desktop (please complete the following information):

OS Name: Windows 10
Browser Name:
Browser Version:

go build ./...