Tone Curve Request

Hi,
I have an iOS project that uses GPUImage to load .acv files to manipulate images. I was just wondering if it would be possible to implement the Tone Curve feature in GPUImage3?
Thanks

It should be possible to make this work, the problem was that I never ported across the original Objective-C logic for the tone curve input to Swift, so it consequently wasn't brought over to the new Metal version. It didn't help things that someone else contributed the original version, so I wasn't as familiar with how the core logic works.
It's on our list of filters to bring over, though.

That would be awesome, I’m just worried that ios14 could be the end of OpenGL support as it’s been deprecated for a while now... thanks for the quick reply!


HDFS-17172. Support FSNamesystemLock Parameters reconfigurable

Description of PR
https://issues.apache.org/jira/browse/HDFS-17172
For Namesystem lock some parameters such as
"dfs.namenode.lock.detailed-metrics.enabled",
"dfs.namenode.read-lock-reporting-threshold-ms",
"dfs.namenode.write-lock-reporting-threshold-ms"
we will support reconfigurable these parameters without namenode restart is convenient for us to check the metrics operation of FSNamesystemLock.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 51s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  1s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

-1 :x:
mvninstall
54m 11s
/branch-mvninstall-root.txt
root in trunk failed.

+1 :green_heart:
compile
1m 25s

trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
compile
1m 15s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m 11s

trunk passed

+1 :green_heart:
mvnsite
1m 23s

trunk passed

+1 :green_heart:
javadoc
1m  9s

trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 33s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 24s

trunk passed

-1 :x:
shadedclient
39m  7s

branch has errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 12s

the patch passed

+1 :green_heart:
compile
1m 19s

the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javac
1m 19s

the patch passed

+1 :green_heart:
compile
1m  9s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
1m  9s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
1m  2s
/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs-project/hadoop-hdfs: The patch generated 25 new + 186 unchanged - 0 fixed = 211 total (was 186)

+1 :green_heart:
mvnsite
1m 17s

the patch passed

+1 :green_heart:
javadoc
0m 57s

the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 30s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 25s

the patch passed

-1 :x:
shadedclient
38m 59s

patch has errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
244m 23s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 46s

The patch does not generate ASF License warnings.

401m 48s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6002/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6002

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux f7caad9a33ce 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / e1a5b8c2b5810a04f9871e883cd10bf8d64877f5

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6002/1/testReport/

Max. process+thread count
2522 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6002/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 42s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
43m 17s

trunk passed

+1 :green_heart:
compile
1m 27s

trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
compile
1m 21s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m 16s

trunk passed

+1 :green_heart:
mvnsite
1m 29s

trunk passed

+1 :green_heart:
javadoc
1m 11s

trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 40s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 23s

trunk passed

-1 :x:
shadedclient
34m 54s

branch has errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 13s

the patch passed

+1 :green_heart:
compile
1m 15s

the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javac
1m 15s

the patch passed

+1 :green_heart:
compile
1m 10s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
1m 10s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  2s

the patch passed

+1 :green_heart:
mvnsite
1m 15s

the patch passed

+1 :green_heart:
javadoc
0m 56s

the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 27s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 15s

the patch passed

-1 :x:
shadedclient
34m 44s

patch has errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
224m 29s

hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 57s

The patch does not generate ASF License warnings.

363m 25s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6002/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6002

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux cbbdda35134c 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / c5ecc2957858868a299e8d5dd4b1958fc532264c

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6002/2/testReport/

Max. process+thread count
3541 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6002/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Hi Sirs @Hexiaoqiao  @ayushtkn @ZanderXu @tomscut Could you please help me review this pr when you have free time ? Thanks a lot~

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 40s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
41m 54s

trunk passed

+1 :green_heart:
compile
1m 24s

trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
compile
1m 21s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m 17s

trunk passed

+1 :green_heart:
mvnsite
1m 30s

trunk passed

+1 :green_heart:
javadoc
1m 11s

trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 43s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 19s

trunk passed

+1 :green_heart:
shadedclient
36m  5s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 15s

the patch passed

+1 :green_heart:
compile
1m 18s

the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javac
1m 18s

the patch passed

+1 :green_heart:
compile
1m 11s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
1m 11s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  1s

the patch passed

+1 :green_heart:
mvnsite
1m 17s

the patch passed

+1 :green_heart:
javadoc
0m 57s

the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 27s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 12s

the patch passed

+1 :green_heart:
shadedclient
35m 52s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
221m 28s

hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 58s

The patch does not generate ASF License warnings.

361m  6s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6002/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6002

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 321cf8a27ea4 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / fd706d8698ab794acfadb4d21a5d45d301bee4d5

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6002/3/testReport/

Max. process+thread count
3476 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6002/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@haiyang1987 Thanks for your works. I am confused why we do not enable it by default and remove to config, any performance impact? Thanks.

Thanks Sir @Hexiaoqiao help me review it.
The current actual scene enabling dfs.namenode.lock.detailed-metrics.enabled will not have a large overhead on performance.
here it is mainly considered from the perspective of maintainable clusters, currently that this feature is disabled by default and can be used as needed  to turn this feature on and off without restarting the namenode.
Sir suggestion is to remove dfs.namenode.lock.detailed-metrics.enabled directly from config, with default metricsEnabled set to true ?

@haiyang1987 thanks for reporting this.  It's good to have. Patch lgtm.

@haiyang1987 Thanks, IMO, if no performance impact, I suggest to enable it by default without any config which involve some cost to end user, what do you think about? cc @brahmareddybattula

Sir @Hexiaoqiao Thanks review .
I thought about it, currently metricsEnabled is enabled in small-scale situations where qps is relatively low, it will have little impact on performance. If it is a large-scale cluster with relatively high qps, there will still be some performance overhead;
We still keep this configuration temporarily, let users choose whether to enable it according to the actual situation， what do you think about?

Hi Sir @Hexiaoqiao Could you please help me review this pr again when you have free time ? Thanks a lot~

Hi @Hexiaoqiao excuse me sir. can you help me take a look at this PR when you have time? Thank you very much~

Hi @ayushtkn sir，can you help me take a look at this PR when you have time.
any suggestions for this change? Thanks a lot~

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 51s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
50m 23s

trunk passed

+1 :green_heart:
compile
1m 24s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
1m 14s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m 10s

trunk passed

+1 :green_heart:
mvnsite
1m 23s

trunk passed

+1 :green_heart:
javadoc
1m  8s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 35s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 23s

trunk passed

+1 :green_heart:
shadedclient
40m 46s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 13s

the patch passed

+1 :green_heart:
compile
1m 18s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
1m 18s

the patch passed

+1 :green_heart:
compile
1m  8s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
1m  8s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
1m  0s
/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs-project/hadoop-hdfs: The patch generated 25 new + 185 unchanged - 0 fixed = 210 total (was 185)

+1 :green_heart:
mvnsite
1m 13s

the patch passed

+1 :green_heart:
javadoc
0m 56s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 26s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 24s

the patch passed

+1 :green_heart:
shadedclient
40m 14s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
239m  7s

hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 49s

The patch does not generate ASF License warnings.

396m 11s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6002/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6002

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 4303ee801794 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / e10a62fc035ba14d43aac08ccab3637293caec83

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6002/4/testReport/

Max. process+thread count
2462 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6002/4/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 52s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
50m 36s

trunk passed

+1 :green_heart:
compile
1m 24s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
1m 14s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m 11s

trunk passed

+1 :green_heart:
mvnsite
1m 24s

trunk passed

+1 :green_heart:
javadoc
1m  8s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 37s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 21s

trunk passed

+1 :green_heart:
shadedclient
40m 29s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m  9s

the patch passed

+1 :green_heart:
compile
1m 16s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
1m 16s

the patch passed

+1 :green_heart:
compile
1m  8s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
1m  8s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  0s

the patch passed

+1 :green_heart:
mvnsite
1m 14s

the patch passed

+1 :green_heart:
javadoc
0m 59s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 30s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 22s

the patch passed

+1 :green_heart:
shadedclient
40m  7s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
245m 49s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 41s

The patch does not generate ASF License warnings.

402m 27s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6002/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6002

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux a27193d368c6 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / e2a4822a505a254050fb967960f772c1beaa04f9

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6002/5/testReport/

Max. process+thread count
2424 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6002/5/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

+1 from my side. Try to trigger CI manually. Let's wait what it will say.

Thanks @Hexiaoqiao for your comment.
seem have some conflicts and i will fix later.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
11m 58s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
43m 19s

trunk passed

+1 :green_heart:
compile
1m 23s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
1m 17s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m  8s

trunk passed

+1 :green_heart:
mvnsite
1m 26s

trunk passed

+1 :green_heart:
javadoc
1m  6s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 42s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 16s

trunk passed

+1 :green_heart:
shadedclient
35m 28s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 14s

the patch passed

+1 :green_heart:
compile
1m 14s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
1m 14s

the patch passed

+1 :green_heart:
compile
1m 12s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
1m 12s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 58s

the patch passed

+1 :green_heart:
mvnsite
1m 16s

the patch passed

+1 :green_heart:
javadoc
0m 54s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 28s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 19s

the patch passed

+1 :green_heart:
shadedclient
34m 47s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
208m 49s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 42s

The patch does not generate ASF License warnings.

357m 54s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6002/7/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6002

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 7fb0d6c9f1c7 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 200643ea65cd415775a34ed66c5eb0cf7f661fd6

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6002/7/testReport/

Max. process+thread count
3834 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6002/7/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

The failed unit test seems unrelated to this PR. Committed to trunk. Thanks @haiyang1987 and @ZanderXu !

Thanks @Hexiaoqiao @ZanderXu @brahmareddybattula help me review and merge it.


[move-prover] Concretized the FixedPoint32 spec, and enabled the tests of FixedPoint32

Previously, the behavior of FixedPoint32 was abstracted away, and the
tests of FixedPoint32 were disabled because Z3 had an issue of solving
non-linear integer arithmetic (NIA).
In the recent releases of Z3, the NIA solver in Z3 was reimplemented, and the
performance seems to be improved and stabilized. Hopefully, this will enable Prover
reason about something related to the FixedPoint32 arithmetic.
This PR does the followings:

add some test cases for non-linear integer arithmetic (in functional/arithm.move)
unabstract (or concretize) the FixedPoint32 spec (in FixedPoint32.move)
enable the tests for the FixedPoint32 (in functional/fixed_point_arithm.move)

The performance impact of "un-abstracting" the FixedPoint32 behavior
seems to be negligible. Here are the running times of cargo test -p move-prover --release before and after the change.

before the change (abstracting the FixedPoint32 behavior)

(tested 3 times) 223 secs, 230 secs, 253 secs

after the change (without abstracting the FixedPoint32 behavior)

(tested 3 times) 235 secs, 243 secs, 250 secs

The Diem Framework does not heavily use the FixedPoin32 module currently. It is
used only in the following places:

The modules such as XUS, XDX and TreasuryComplianceScripts
create and assign a FixedPoint32 number.
The Diem module uses the FixedPoint32 multipication one time.

Motivation
To reason about things related to the FixedPoint32 arithmetic
Have you read the Contributing Guidelines on pull requests?
yes
Test Plan
cargo test

/land


[Enquiry] concept of channel fee

AFAIK, each channel has its own fee rate, which is "fee_per_kw"
For the following setup:
         A   ----(HIGH Fee rate) ----  B  ----------------  C

B want to charge A for more fee, he raises the fee of their channel

A pays to C
A pays to B
C pays to A

The high fee rate only affects case 1, am i correct?
Thanks in advance.

Questions are better asked on IRC or Slack. He'll raise the fee from channel B -> C.


Not able to make AlertDialog full width

By Using AlertDialog , it not taking full width to screen.
showAlertDialog(BuildContext context) {
  // Create button
  Widget okButton = FlatButton(
    child: Text("OK"),
    onPressed: () {
      Navigator.of(context).pop();
    },
  );

  // Create AlertDialog
  AlertDialog alert = AlertDialog(
    title: Text("Simple Alert"),
    content: Text("This is an alert message."),
    actions: [
      okButton,
    ],
  );

  // show the dialog
  showDialog(
    context: context,
    builder: (BuildContext context) {
      return alert;
    },
  );
}

Hi @mehtahardikr
Please check https://stackoverflow.com/questions/53913192/flutter-change-the-width-of-an-alertdialog
It seems you are asking for assistance on personal code. Please see https://flutter.dev/community for resources and asking questions like this,
you may also get some help if you post it on Stack Overflow and if you need help with your code, please see https://www.reddit.com/r/flutterhelp/
Closing, as this isn't an issue with Flutter itself. If you disagree, please write in the comments and I will reopen it.
Thank you

@TahaTesser  : that advised to copy flutter framework file which has fixed sized of width. Please repoen it.
thanks.

there is issue with dialog.dart file as well.
final EdgeInsets effectivePadding = MediaQuery.of(context).viewInsets + (insetPadding ?? const EdgeInsets.all(0.0));
now value of insetPadding
this.insetPadding = _defaultInsetPadding,
const EdgeInsets _defaultInsetPadding = EdgeInsets.symmetric(horizontal: 40.0, vertical: 24.0);
here not able to pass insetPadding as its take default value.

@mehtahardikr
There are multiple solutions in the link I have given, one of them is this

complete code sample
import 'package:flutter/material.dart';

void main() => runApp(MyApp());

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Material App',
      theme: ThemeData.dark(),
      home: Home(),
    );
  }
}

class Home extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text('Material App Bar'),
      ),
      floatingActionButton: FAB(),
    );
  }
}

class FAB extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return FloatingActionButton.extended(
      label: Text('Open Dialog'),
      icon: Icon(Icons.add),
      onPressed: () {
        showDialog(
            context: context,
            builder: (_) => new AlertDialog(
                  insetPadding:
                      EdgeInsets.symmetric(horizontal: 15, vertical: 40),
                  shape: RoundedRectangleBorder(
                      borderRadius: BorderRadius.all(Radius.circular(10.0))),
                  content: Builder(
                    builder: (context) {
                      // Get available height and width of the build area of this widget. Make a choice depending on the size.
                      var height = MediaQuery.of(context).size.height;
                      var width = MediaQuery.of(context).size.width;

                      return Container(
                        height: height - 40,
                        width: width - 5,
                      );
                    },
                  ),
                ));
      },
    );
  }
}

It seems you are asking for assistance on personal code. Please see https://flutter.dev/community for resources and asking questions like this,
you may also get some help if you post it on Stack Overflow and if you need help with your code, please see https://www.reddit.com/r/flutterhelp/
Thank you


ARROW-9600: [Rust] pin proc macro

The previous PR that was merged for this did not correctly pin the version and did not check in the corresponding generated code for that version.

@alamb could you review please?


updated Creating a Custom Form Field doc ngControl section

If the component already implements ControlValueAccessor, injecting NgControl in the constructor (as described in the docs) may fail.  Added an alternate approach to get past this problem.

Signed CLA.

Revision added.

Good call, done.


Added Error, Payload and Input Helpers (main)

Follow up PR for #4349 that targets #main instead of #develop

/AzurePipelines run


Add web tests for the Application Counts

What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #
Does this PR introduce a user-facing change?:

NONE

/approve


All attempts to get a Google authentication bearer token failed

I create a new environment with python3.8 and have installed all of the packages showed in requirements.txt and run PYTHONPATH=. python flan/v2/run_example.py, however, I failed for the following reasons:
2023-03-17 15:00:13.007874: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory
2023-03-17 15:00:13.007925: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1835] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2023-03-17 15:00:13.008442: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-03-17 15:00:16.014237: W tensorflow/core/platform/cloud/google_auth_provider.cc:184] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with "Not found: Could not locate the credentials file.". Retrieving token from GCE failed with "Not found: Error executing an HTTP request: HTTP response code 410 with body '<html>
  <head>
    <meta http-equiv='refresh' content='0; url=http://metadata/' />
  </head>
</html>
'".
2023-03-17 15:01:17.240986: E tensorflow/core/platform/cloud/curl_http_request.cc:614] The transmission  of request 0x558cf720ec00 (URI: https://www.googleapis.com/storage/v1/b/t5-data/o/vocabs%2Fcc_all.32000%2Fsentencepiece.model?fields=size%2Cgeneration%2Cupdated) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.046775 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
2023-03-17 15:02:18.712983: E tensorflow/core/platform/cloud/curl_http_request.cc:614] The transmission  of request 0x558cf720ec00 (URI: https://www.googleapis.com/storage/v1/b/t5-data/o/vocabs%2Fcc_all.32000%2Fsentencepiece.model?fields=size%2Cgeneration%2Cupdated) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.044402 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
2023-03-17 15:03:20.425956: E tensorflow/core/platform/cloud/curl_http_request.cc:614] The transmission  of request 0x558cf720ec00 (URI: https://www.googleapis.com/storage/v1/b/t5-data/o/vocabs%2Fcc_all.32000%2Fsentencepiece.model?fields=size%2Cgeneration%2Cupdated) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.032221 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
Traceback (most recent call last):
  File "flan/v2/run_example.py", line 93, in <module>
    dataset = selected_mixture.get_dataset(
  File "xx/lib/python3.8/site-packages/seqio/dataset_providers.py", line 1278, in get_dataset
    self._check_compatible_features()
  File "xx/lib/python3.8/site-packages/seqio/dataset_providers.py", line 1235, in _check_compatible_features
    if task.output_features[name].vocabulary != feature.vocabulary:
  File "xx/lib/python3.8/site-packages/seqio/vocabularies.py", line 330, in __eq__
    their_md5 = hashlib.md5(other.sp_model).hexdigest()
  File "xx/lib/python3.8/site-packages/seqio/vocabularies.py", line 248, in sp_model
    self._load_model()
  File "xx/lib/python3.8/site-packages/seqio/vocabularies.py", line 216, in _load_model
    self._sp_model = f.read()
  File "xx/lib/python3.8/site-packages/tensorflow/python/lib/io/file_io.py", line 119, in read
    length = self.size() - self.tell()
  File "xx/lib/python3.8/site-packages/tensorflow/python/lib/io/file_io.py", line 98, in size
    return stat(self.__name).length
  File "xx/lib/python3.8/site-packages/tensorflow/python/lib/io/file_io.py", line 871, in stat
    return stat_v2(filename)
  File "xx/lib/python3.8/site-packages/tensorflow/python/lib/io/file_io.py", line 887, in stat_v2
    return _pywrap_file_io.Stat(compat.path_to_str(path))
KeyboardInterrupt

And the intalled packages are shown below:(cuda 11.6)
Package                  Version
------------------------ ---------------------
absl-py                  0.12.0
astunparse               1.6.3
attrs                    21.2.0
Babel                    2.9.1
cachetools               4.2.2
certifi                  2021.5.30
charset-normalizer       2.0.4
clang                    5.0
click                    8.0.1
colorama                 0.4.4
dill                     0.3.4
editdistance             0.5.3
filelock                 3.0.12
flatbuffers              1.12
frozendict               2.3.5
future                   0.18.2
gast                     0.4.0
gin-config               0.4.0
google-auth              1.35.0
google-auth-oauthlib     0.4.5
google-pasta             0.2.0
googleapis-common-protos 1.53.0
grpcio                   1.39.0
h5py                     3.1.0
huggingface-hub          0.0.12
idna                     3.2
importlib-resources      5.12.0
iniconfig                1.1.1
joblib                   1.0.1
keras                    2.6.0
Keras-Preprocessing      1.1.2
Levenshtein              0.13.0
Markdown                 3.3.4
mesh-tensorflow          0.1.19
nltk                     3.6.2
numpy                    1.19.5
oauthlib                 3.1.1
opt-einsum               3.3.0
packaging                21.0
pandas                   1.3.2
pip                      23.0.1
pluggy                   0.13.1
portalocker              2.3.0
promise                  2.3
protobuf                 3.17.3
py                       1.10.0
pyasn1                   0.4.8
pyasn1-modules           0.2.8
pyparsing                2.4.7
pytest                   6.2.4
python-dateutil          2.8.2
pytz                     2021.1
PyYAML                   5.4.1
regex                    2021.8.3
requests                 2.26.0
requests-oauthlib        1.3.0
rouge-score              0.0.4
rsa                      4.7.2
sacrebleu                2.0.0
sacremoses               0.0.45
scikit-learn             0.24.2
scipy                    1.7.1
sentencepiece            0.1.96
seqio                    0.0.6
setuptools               67.6.0
six                      1.15.0
t5                       0.9.2
tabulate                 0.8.9
tensorboard              2.6.0
tensorboard-data-server  0.6.1
tensorboard-plugin-wit   1.8.0
tensorflow               2.6.0
tensorflow-datasets      4.4.0
tensorflow-estimator     2.6.0
tensorflow-hub           0.12.0
tensorflow-metadata      1.2.0
tensorflow-text          2.6.0
termcolor                1.1.0
tfds-nightly             4.4.0.dev202108200109
threadpoolctl            2.2.0
tokenizers               0.10.3
toml                     0.10.2
torch                    1.9.0
tqdm                     4.62.1
transformers             4.9.2
typing-extensions        <IP_ADDRESS>
urllib3                  1.26.6
Werkzeug                 2.0.1
wheel                    0.40.0
wrapt                    1.12.1
yapf                     0.32.0
zipp                     3.15.0

How could I solve this problem? Is this a network error or python environmental support problem?

@Zcchill I'm not sure what would cause this. I'm submitting an issue to Tensorflow Datasets to investigate. You may want to try to downloading the prerequisite raw datasets using tf.load() as suggested in #37, in case the problem is related to that issue.

Looks like I encountered a similar issue. I created a new conda environment and only installed the packages in flan/v2/requirements.txt, and the error message is:
2023-03-20 22:07:14.323104: W tensorflow/core/platform/cloud/google_auth_provider.cc:184] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with "NOT_FOUND: Could not locate the credentials file.". Retrieving token from GCE failed with "FAILED_PRECONDITION: Error executing an HTTP request: libcurl code 6 meaning 'Couldn't resolve host name', error details: Could not resolve host: metadata".
Traceback (most recent call last):
  File "flan/v2/run_example.py", line 94, in <module>
    dataset = selected_mixture.get_dataset(
  File "/afs/crc.nd.edu/user/z/zzhang23/anaconda3/envs/flan/lib/python3.8/site-packages/seqio/dataset_providers.py", line 1703, in get_dataset
    self._check_compatible_features()
  File "/afs/crc.nd.edu/user/z/zzhang23/anaconda3/envs/flan/lib/python3.8/site-packages/seqio/dataset_providers.py", line 1650, in _check_compatible_features
    and task.output_features[name].vocabulary != feature.vocabulary
  File "/afs/crc.nd.edu/user/z/zzhang23/anaconda3/envs/flan/lib/python3.8/site-packages/seqio/vocabularies.py", line 445, in __eq__
    their_md5 = hashlib.md5(other.sp_model).hexdigest()
  File "/afs/crc.nd.edu/user/z/zzhang23/anaconda3/envs/flan/lib/python3.8/site-packages/seqio/vocabularies.py", line 359, in sp_model
    self._load_model()
  File "/afs/crc.nd.edu/user/z/zzhang23/anaconda3/envs/flan/lib/python3.8/site-packages/seqio/vocabularies.py", line 314, in _load_model
    self._sp_model = f.read()
  File "/afs/crc.nd.edu/user/z/zzhang23/anaconda3/envs/flan/lib/python3.8/site-packages/tensorflow/python/lib/io/file_io.py", line 118, in read
    length = self.size() - self.tell()
  File "/afs/crc.nd.edu/user/z/zzhang23/anaconda3/envs/flan/lib/python3.8/site-packages/tensorflow/python/lib/io/file_io.py", line 96, in size
    return stat(self.__name).length
  File "/afs/crc.nd.edu/user/z/zzhang23/anaconda3/envs/flan/lib/python3.8/site-packages/tensorflow/python/lib/io/file_io.py", line 908, in stat
    return stat_v2(filename)
  File "/afs/crc.nd.edu/user/z/zzhang23/anaconda3/envs/flan/lib/python3.8/site-packages/tensorflow/python/lib/io/file_io.py", line 924, in stat_v2
    return _pywrap_file_io.Stat(compat.path_to_str(path))
tensorflow.python.framework.errors_impl.FailedPreconditionError: Error executing an HTTP request: libcurl code 77 meaning 'Problem with the SSL CA cert (path? access rights?)', error details: error setting certificate verify locations:  CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none
         when reading metadata of gs://t5-data/vocabs/cc_all.32000/sentencepiece.model

Strange that it's trying to download that vocabulary. I don't see it referenced in your code.

I have met the same problem, unable to get the  metadata
2023-05-02 20:02:02.843261: W tensorflow/tsl/platform/cloud/google_auth_provider.cc:184] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with "NOT_FOUND: Could not locate the credentials file.". Retrieving token from GCE failed with "ABORTED: All 10 retry attempts failed. The last failure: Error executing an HTTP request: HTTP response code 502".

How did you solve it

My error:
All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with "Not found: Could not locate the credentials file.". Retrieving token from GCE failed with "Failed precondition: Error executing an HTTP request: libcurl code 6 meaning 'Couldn't resolve host name', error details: Could not resolve host: metadata".

 export CURL_CA_BUNDLE=/etc/ssl/certs/ca-bundle.crt
solved it

2023-06-23 15:49:57.853557: W tensorflow/tsl/platform/cloud/google_auth_provider.cc:184] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with "NOT_FOUND: Could not locate the credentials file.". Retrieving token from GCE failed with "FAILED_PRECONDITION: Error executing an HTTP request: libcurl code 6 meaning 'Couldn't resolve host name', error details: Could not resolve host: metadata.google.internal".


add an option to keep or reuse the automated genesis image

When using automated genesis generation, it builds then cleanups the docker images every-time, allow reusing this image would save us time specially in local development setup.

actually it does this, it only removes the container.


[docs] add postgresml to list of external repos

Tonight I learned about postgresml, a project that allows you to train LightGBM models (and many other types of machine learning models) using SQL. postgresml  available as Postgres extension.
https://github.com/postgresml/postgresml
Here's the LightGBM binding: https://github.com/postgresml/postgresml/blob/e39d7e36ac285bb27496896bff33c87054e0d90b/pgml-extension/src/bindings/lightgbm.rs
cc postgresml maintainers: @montanalow @levkk

Nice! Let me know if there's anything we can do to help. I think the Rust bindings for LightGBM could use a little more love in general, but the MVP is looking good so far.


Add estimation time

Similar as in vanitygen, it would be great if vanityeth had some ability to estimate the probability and search time.

Agreed, we could be waiting forever ;)


add ES translation of AngularJS Waypoints

Translation to Spanish of the Angular.js Waypoints.

Good job with this translation.

Will leave it like this by now and then patch it once the rest of the translation is done.

Great work here, @justinian336!


Update pickadate jquery plugin api based on spec

I made several changes based on the documentation at http://amsul.ca/pickadate.js/api/
That includes adding some properties, and removing some definitions for things that don't actually exist.

@theodorejb could you review this PR?

Looks good to me, though once merged I'll probably follow it up with another pull request to bring the definitions in line with a number of other API additions.

@theodorejb thanks!
@tomtheisen thanks mate!


Feature form configuration annotation support

Checklist

[x] I have read the project's CONTRIBUTING document
[x] I have checked my code with the project's static analysis tooling
[x] I have formatted my code with the project's IDEA code-style configuration
[x] I have checked my code for misspellings
[x] I have organized my changes in easy-to-follow commits
[x] My change requires a change to the documentation
[ ] I have updated the documentation accordingly -> Waiting for PR for documentation to be merged
[x] I have added tests to cover my changes
[x] All new and existing tests pass.

@agrancaric can you please add related issue to the PR description? Thanks :)

@agrancaric can you please add related issue to the PR description? Thanks :)

Added it, it seems I've accidentally deleted it when undoing something.


Spark 3883: SSL support for HttpServer and Akka

Test build #25165 has started for   PR 3571 at commit b50e7dc.

This patch does not merge cleanly.

@vanzin I read about Yarn and Mesos deployments and found that making those namespaces does not make much sense because you cannot have different configurations in a single actor system.
For standalone deployment, it will work as I said before - the user can provide his own configuration in SparkConf, but to fetch it, the worker configuration will be used and then it will be overridden by the user configuration.
For YARN, we still need some settings to fetch the configuration from the driver. What I did was to repeat the same solution as it is applied for spark.auth namespace - SSL settings are required before the actual configuration is applied (the same for secret token - as it is implemented right now).
What do you think? I really want to move this PR forward.

Test build #25166 has started for   PR 3571 at commit 82a9faa.

This patch merges cleanly.

Test build #25166 has finished for   PR 3571 at commit 82a9faa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class SSLOptions(enabled: Boolean = false,
    val classServer                                   = new HttpServer(outputDir, new SecurityManager(conf), classServerPort, "HTTP class server", conf)

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25166/
Test FAILed.

@vanzin the logs are available now - could you help with this?

Test build #25165 has finished for   PR 3571 at commit b50e7dc.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
case class SSLOptions(enabled: Boolean = false,
    val classServer                                   = new HttpServer(outputDir, new SecurityManager(conf), classServerPort, "HTTP class server", conf)

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25165/
Test FAILed.

@jacek-lewandowski I was able to hit something that looks like the error Jenkins hit locally. While I try to take a look, there are two other issues I ran into:

You're using File.toPath() which was added in Java 7, and Spark needs to work against Java 6
The tests configure SSL to use TLSv1.2, while Java 6 only seems to have TLSv1.

Ahhh !!! That can be the problem, I always tried with Java 7. Thanks @vanzin, I'll try with Java 6 now.

According to the first point - would it ever compile if it was an issue?

No, it wouldn't. I think the jenkins builders might be using Java 7 (bad jenkins!). I had to modify your code to compile it locally.

Well.. I get a different exception with Java 6 : java.security.NoSuchAlgorithmException: TLSv1.2 SSLContext not available

Let me try once again...

I mentioned that in my second bullet. Java 6 doesn't have TLSv1.2. After I fix that I get something like this:
Exception in thread "Thread-40" java.lang.IllegalStateException: Trying to deserialize a serialized ActorRef without an ActorSystem in scope. Use 'akka.serialization.Serialization.currentSystem.withValue(system) { ... }'
	at akka.actor.SerializedActorRef.readResolve(ActorRef.scala:407)

How did you get this? I changed to SSLv3 (only for the tests) and AkkaUtilsTest passed...

Test build #25171 has started for   PR 3571 at commit 07396e6.

This patch merges cleanly.

How did I get it? I ran the test. I changed it to TLSv1, but that shouldn't make a difference.
Are you absolutely sure you don't have anything in your environment that could be affecting this? Like a SPARK_* env variable pointing to some other config, or some file you forgot to add to your repo, or something like that.

Will check, however, as you may see, SSL with Jetty test passes and it uses the same configuration. What OS do you have? I'm gonna test it on a separate machine, or even setup a clean VM.

The tests create a file unit-tests.log it target directory. It would very helpful if I can access that file.

Ok, I've replicated on a different machine with different OS

If I remove TLS_RSA_WITH_AES_256_CBC_SHA from the list of ciphers in SSLSampleConfigs.scala, most tests pass for me. "remote fetch ssl on - untrusted server" fails, but that doesn't seem to be because of the code, but because it's expecting the wrong exception (so a test issue).
Another interesting point: if I comment out spark.ssl.enabledAlgorithms altogether, then the tests seem to timeout.

BTW that cipher is not listed here: https://docs.oracle.com/javase/6/docs/technotes/guides/security/SunProviders.html

If you're curious you could try to use -Djavax.net.debug=all to see what's going on; both ciphers seem to exist in 1.6 so it's weird that the second one is causing problems.

Test build #25171 has finished for   PR 3571 at commit 07396e6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class SSLOptions(enabled: Boolean = false,
    val classServer                                   = new HttpServer(outputDir, new SecurityManager(conf), classServerPort, "HTTP class server", conf)

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25171/
Test FAILed.

Haha, I found probably the same thing :) (https://groups.google.com/forum/#!topic/akka-user/x07N35BH7es)

It seems that Java on OSX has some different cipher suite...

Test build #25174 has started for   PR 3571 at commit 23a834d.

This patch merges cleanly.

Test build #25174 has finished for   PR 3571 at commit 23a834d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class SSLOptions(enabled: Boolean = false,
    val classServer                                   = new HttpServer(outputDir, new SecurityManager(conf), classServerPort, "HTTP class server", conf)

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25174/
Test FAILed.

Test build #25203 has started for   PR 3571 at commit b5c965d.

This patch merges cleanly.

Test build #25203 has finished for   PR 3571 at commit b5c965d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class SSLOptions(enabled: Boolean = false,
    val classServer                                   = new HttpServer(outputDir, new SecurityManager(conf), classServerPort, "HTTP class server", conf)

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25203/
Test PASSed.

Test build #25215 has started for   PR 3571 at commit 937d839.

This patch merges cleanly.

Test build #25215 has finished for   PR 3571 at commit 937d839.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
    val classServer                                   = new HttpServer(outputDir, new SecurityManager(conf), classServerPort, "HTTP class server", conf)

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25215/
Test PASSed.

@jacek-lewandowski this is looking good and tests now pass! I left a few minor comments. On the yarn side, it would be nice to automatically distribute the key/trust store files to the containers, but that can be done separately.

Thanks @vanzin for the review. I'll apply your comments promptly.
Key/trust stores can be distributed by attaching these files to the application - am I right? The framework (Yarn in this case) should take care of distribution of them.

Yes, the framework can distribute the files, but if would be nice for Spark to automatically do it if SSL is enabled, instead of requiring the user to manually list the files in the spark-submit command line.

I'd leave distributing key stores for a separate ticket. It should cover Yarn, Mesos and standalone installation. It seems to be trivial for Yarn according to what we said. I don't know yet how it would work for Mesos and I do know that it wouldn't work for standalone because the files are fetched too late (by SparkContext). So the implementation of a versatile solution wouldn't be trivial.

I had to rebase and force push

Test build #25548 has started for   PR 3571 at commit 60caf8c.

This patch merges cleanly.

Test build #25548 has finished for   PR 3571 at commit 60caf8c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25548/
Test PASSed.

@vanzin I found one bug in my recent changes which I will fix in few hours

Test build #25603 has started for   PR 3571 at commit a703c9b.

This patch merges cleanly.

@vanzin can we move forward with this PR ?

Test build #25603 has finished for   PR 3571 at commit a703c9b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25603/
Test PASSed.

@jacek-lewandowski I can only review the code, you need a committer to be able to move forward. e.g. @andrewor14 @JoshRosen

@vanzin I've applied your comments. I've tested on a 3 node Spark cluster, tried to submit application from OSX to Linux cluster in both client and cluster deploy mode (in standalone mode). Everything seems working fine.

Test build #25661 has started for   PR 3571 at commit 271d166.

This patch merges cleanly.

Test build #25661 has finished for   PR 3571 at commit 271d166.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25661/
Test PASSed.

@andrewor14 can you take a look tell me what's next?

Actually @JoshRosen - can you take a look at this one?

Hi @jacek-lewandowski,
Thanks for bringing this up to date.  I took a quick pass through and left some minor comments.
Just to clarify: this only adds SSL support for internal HttpServer and Akka traffic, and not the Spark web UI?  When we last discussed this in #2739, I think the idea was that SSLOptions could use namespaced configs in order to allow the web UI to use different SSL configurations than, say, Akka.  I see that there's some namespace support built into this patch (the ns argument to parse); is this support sufficient to support HTTPs in the UI?  Also, does it support scenarios where I want to enable SSL only for the UI or only for Akka?  Settings like spark.ssl.enabled sound like they're systemwide settings, so we should think through how these might interact with different UI configurations, etc.  I'm not asking to implement SSL for the UI in this patch, but I'd like to just make sure that the SSLOptions configuration code will be compatible with it.
It would be great if you could add a short summary of this PR's changes to the PR description, since that description will become this PR's commit message.
There's a big block comment at the top of SecurityManager.scala which should be updated to reflect this PR's changes (it currently says " We currently do not support SSL (https) ...").
It would also be great to add a small section to the security documentation (docs/security.md) to mention how to configure this.  The documentation should mention the relevant Spark options, describe how/why someone would use the useNodeLocalConf setting, etc.  It could also contain a pointer to external instructions for generating your own keystore / truststores, etc., since this isn't a trivial process.  The new configuration options should also be documented in docs/configuration.md alongside the other security configurations.  In addition, the documentation should describe how the key stores are / aren't distributed depending on the choice of cluster manager.  If this works in fundamentally different ways on different cluster managers, then the docs should make this clear so users know what to expect.

@JoshRosen thanks a lot for your review and comments - will apply them soon. I've been waiting with the documentation until someone accept this approach.
For Spark UI SSL support - I didn't do that because there is https://github.com/apache/spark/pull/1980 (actually mentioned by you in https://github.com/apache/spark/pull/2739). However the answer for your question is yes - SSLOptions can be created at different namespace and, since it can configure Jetty (createJettySslContextFactory method), it can be used for Spark UI as well. Actually, I considered spark.ssl.xxx for Akka and internal HttpServer (for file sharing) and spark.ui.ssl.xxx for UI. We would just need to add:
private[spark] val sslUIOptions = SSLOptions.parse(sparkConf, "spark.ui.ssl")

to the SecurityManager class.

private[spark] val sslUIOptions = SSLOptions.parse(sparkConf, "spark.ui.ssl")

In a previous version of the patch, we talked about how to have namespaced options and also a fallback "global" config. That seems more flexible (e.g., spark.ssl.enabled = true would apply also for the UI, and you could use spark.ssl.ui.enabled = false to disable SSL only for the UI).
From what I remember from my last read, that is not possible with the current code, right? You'd need to explicitly set both the global and the spark.ui.ssl.* options to be able to enable SSL for both.

@vanzin it is low hanging fruit to achieve that

Test build #25979 has started for   PR 3571 at commit 054c3d8.

This patch does not merge cleanly.

@JoshRosen I've applied your comments but the documentation sutff (which i'm working on). Could you tell we whether it you gonna merge it eventually into master?

Test build #25982 has started for   PR 3571 at commit 4eda60c.

This patch merges cleanly.

Test build #25982 has finished for   PR 3571 at commit 4eda60c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25982/
Test PASSed.

Test build #25979 has finished for   PR 3571 at commit 054c3d8.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25979/
Test FAILed.

@JoshRosen what is going on with these test? I can see that Jenkins run them twice and the second comment says they failed, however it has a lower build number than the previously mentioned build which passed the tests. Also the failing tests seems to have nothing to do with this PR.

Jenkins, retest this please.

@jacek-lewandowski I think that we may have had an intermittent period of flakiness on Jenkins or a possible build-break, so I'm retesting it now.

Test build #26260 has started for   PR 3571 at commit 4eda60c.

This patch does not merge cleanly.

The SSL changes themselves here look good, so I think the last major thing to resolve is how we're going to handle tiered configurations.  I like @vanzin's idea of having the global spark.ssl setting be overridden by component-specific configurations, since this will be future-proof in terms of adding new components (e.g. if we add new network pieces, they'll be covered by the umbrella SSL configuration).
If we take care of that and the documentation, then I think we should be able to pull this in.
I think the idea of having per-component overrides is backwards-compatible with the approach here, so unless you anticipate any API-breakage that would result from adding that change later, I think it might be fine to stick with the global "all-or-nothing" SSL configuration then enhance it later with the per-component overrides.

Test build #26260 has finished for   PR 3571 at commit 4eda60c.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26260/
Test PASSed.

@JoshRosen I'll add a way to create SSLOptions which treats another SSLOptions as a set of defaults. I don't expect any API changes since the only thing you will have to do is to create another SSLOptions objects in SecurityManager.

 The second commit is for what I mentioned in the previous comment - the sample usage is in the test suite.

Test build #26314 has started for   PR 3571 at commit 1a1ea89.

This patch merges cleanly.

Test build #26314 has finished for   PR 3571 at commit 1a1ea89.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26314/
Test PASSed.

After thinking more about this, I decided to refactor things a little bit to make it more consistent.

Test build #26317 has started for   PR 3571 at commit 37d8221.

This patch merges cleanly.

Test build #26317 has finished for   PR 3571 at commit 37d8221.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26317/
Test PASSed.

@vanzin @JoshRosen wdyt about the recent changes?

Thanks for the nice organization of the commits here, since it makes it really easy to see what's changed since the last review.  Nice work on the test cases for configuration overriding, too.
I have a couple of super-minor comments (which I'll leave inline), but overall these latest changes look good to me and I like the configuration design, so I think we can go ahead and finish the docs and merge this soon for 1.3.

AFAIK TypeSafe configuration notify about missing properties by throwing an exception. That's why this Try. I just thought that this is cleaner form than try...catch block.

Ok @JoshRosen I've found a way to do this.

Test build #26510 has started for   PR 3571 at commit b7ba4c7.

This patch merges cleanly.

Test build #26510 has finished for   PR 3571 at commit b7ba4c7.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26510/
Test FAILed.

Test build #26512 has started for   PR 3571 at commit 2532668.

This patch merges cleanly.

Test build #26512 has finished for   PR 3571 at commit 2532668.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26512/
Test PASSed.

@jacek-lewandowski Thanks for updating this.  I'd like to merge this patch today so that it's included in the first 1.3.0 snapshot / preview.  If you have time, it would be great if you could add some documentation for the new configurations (and update the big block comment at the top of SecurityManager.scala to reflect the changes here).  If you're too busy to do this, though, just let me know and we can merge this without documentation and add it before the first release candidate (this patch touches a bunch of core stuff, so I want the code changes here to be merged so they're exercised during our first rounds of testing).

Test build #26544 has started for   PR 3571 at commit 9ef4ed1.

This patch merges cleanly.

@JoshRosen I've just added the documentation. I hope there is enough of it :)

@jacek-lewandowski Thanks for adding docs; I'll look over them now.

Test build #26544 has finished for   PR 3571 at commit 9ef4ed1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26544/
Test FAILed.

Jenkins, retest this please.

Overall, this looks good to me, so I'm going to just wait for Jenkins to pass then pull this in for 1.3.  We can fix any documentation touch-ups or other things during the QA period.

Test build #26548 has started for   PR 3571 at commit 9ef4ed1.

This patch merges cleanly.

Test build #26548 has finished for   PR 3571 at commit 9ef4ed1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26548/
Test FAILed.

I don't get why the tests failed. I only added documentation!

Jenkins, retest this please.

Test build #26556 has started for   PR 3571 at commit 9ef4ed1.

This patch merges cleanly.

Test build #26556 has finished for   PR 3571 at commit 9ef4ed1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26556/
Test PASSed.

I've fixed the documentation formatting nits myself, so I'm going to merge this into master (1.3.0).  Thanks @jacek-lewandowski for all of your hard work on this feature and to @vanzin for your help reviewing this patch!


[Visualization utils] Add visualization utils for plotting images(plain, with bounding boxes and segmentation masks)

This PR adds the following export utils for visualization of images

draw_bounding_boxes -  draw bounding boxes on give images.
draw_segmentation_masks - draw masks on given images.
plot_image_gallery - plots given images
plot_bounding_box_gallery - plots images with bounding boxes(y_true and y_pred).
plot_segmentation_mask_gallery - plots images with masks(y_true and y_pred).

Reference: https://colab.research.google.com/gist/sineeli/8b2caa962f0c33d9450a4e5e70ec2d43/-keras3-visualization-utils.ipynb

Codecov Report
Attention: Patch coverage is 26.00000% with 185 lines in your changes missing coverage. Please review.

Project coverage is 59.79%. Comparing base (49a3063) to head (d7939f6).
Report is 4 commits behind head on master.

Files with missing lines
Patch %
Lines

keras/src/visualization/draw_bounding_boxes.py
17.39%
57 Missing :warning:

keras/src/visualization/plot_image_gallery.py
23.63%
42 Missing :warning:

keras/src/visualization/draw_segmentation_masks.py
15.90%
37 Missing :warning:

...rc/visualization/plot_segmentation_mask_gallery.py
28.12%
23 Missing :warning:

...ras/src/visualization/plot_bounding_box_gallery.py
38.23%
21 Missing :warning:

...eras/api/_tf_keras/keras/visualization/__init__.py
0.00%
5 Missing :warning:

:exclamation:  There is a different number of reports uploaded between BASE (49a3063) and HEAD (d7939f6). Click for more details.
HEAD has 6 uploads less than BASE

Flag
BASE (49a3063)
HEAD (d7939f6)

keras
4
1

keras-torch
1
0

keras-tensorflow
1
0

keras-jax
1
0

Additional details and impacted files
@@             Coverage Diff             @@
##           master   #20401       +/-   ##
===========================================
- Coverage   82.30%   59.79%   -22.51%
===========================================
  Files         506      514        +8
  Lines       46735    46996      +261
  Branches     7344     7389       +45
===========================================
- Hits        38463    28103    -10360
- Misses       6488    17153    +10665
+ Partials     1784     1740       -44

Flag
Coverage Δ

keras
59.79% <26.00%> (-22.36%)
:arrow_down:

keras-jax
?

keras-numpy
59.79% <26.00%> (-0.20%)
:arrow_down:

keras-tensorflow
?

keras-torch
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Error when trying to run bot (IndexError)

Traceback (most recent call last):
File "hq_main.py", line 25, in
raise e
File "hq_main.py", line 22, in
USER_ID = settings[1].split("=")[1]
IndexError: list index out of range

Have you filled out your conn_settings.txt? If you have, please post the log file here.

Same error with me. I filled both in settings ( I was just trying to check, the game wasn't on )

User IDs are being reset. @U-r-Unknown, please post the log file here if you are still having the issue.

I am having this exact issue. Here is the log file if it helps.
CRITICAL:root:Settings read error: ['BEARER_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOjE5MzA3NDE2LCJ1c2VybmFtZSI6ImEzMTRzMTciLCJhdmF0YXJVcmwiOiJodHRwczovL2QyeHUxaGRvbWgzbnJ4LmNsb3VkZnJvbnQubmV0L2RlZmF1bHRfYXZhdGFycy9VbnRpdGxlZC0xXzAwMDFfYmx1ZS5wbmciLCJ0b2tlbiI6InNzUzlSTiIsInJvbGVzIjpbXSwiY2xpZW50IjoiaU9TLzEuMy41IGI4OCIsImd1ZXN0SWQiOm51bGwsInYiOjEsImlhdCI6MTUyNjkzMjA3MCwiZXhwIjoxNTM0NzA4MDcwLCJpc3MiOiJoeXBlcXVpei8xIn0.4s8bTLFk84uCvQ_zJ4ZRyVJfiMkX12zo8tK2zvDpA4E', '', 'USER_ID=19307416']

I had this problem but fixed it just now by removing the blank line i had between the Bearer_token and UserID lines.

how tf yall find ur bearer token and userid

You need to

get a program to inspect network traffic (I used charles)
2)install the ssl certificate on your phone
2.5)configure your phones http proxy to use your computer ip with port 8888(charles walks you through it)
3Go to your phone settings and make sure you flip it on
4)In charles enable SSL proxing for the HQ api
it should be there in plain text.

I can just give you an id but they change them every so often so you will need new ones periodically.

The need to figure that out has not arrived yet. But i would imagine you just need to get the new bearer  and ID strings.

Issue should have been solved in commit https://github.com/Exaphis/HackQ-Trivia/commit/9346ee572c079cdfe9cb3bd022f2fe807b6bd7b2. Let me know if you have any more problems.


ci: add attestation to container images and SBOMs

Add in-toto attestation to all image-building ci workflows, for both SBOMs (image.yml) and images themselves (all 3 workflows). Allow necessary permissions to attest (attestations, packages).

Attestations bind some subject (a named artifact along with its digest) to a SLSA build provenance predicate using the in-toto format.

A verifiable signature is generated for the attestation using a short-lived Sigstore-issued signing certificate. If the repository initiating the GitHub Actions workflow is public, the public-good instance of Sigstore will be used to generate the attestation signature. If the repository is private/internal, it will use the GitHub private Sigstore instance.

ref to https://github.com/actions/attest-build-provenance
well, I am open for SLSA, sigstore or any openSSF kind structure in CI approach.
But, or well...
https://github.com/sustainable-computing-io/kepler/actions/runs/10931316693/job/30346044739?pr=1789
it should not in PR image build, IMO, it's valuable as user will use release image or daily image(which is latest tag image).
I am not sure if we need additional project configuration for sigstore, @rootfs , could you please advise?
at least within this PR, it seems we need a github token.
https://github.com/actions/attest-build-provenance?tab=readme-ov-file#inputs
or @arthurus-rex , ref to github token settings, it seems PR(as PR owner is from a fork repo) can't use upstream repo's github token setting as security reason, please update code, remove/update settings for this github action. I suppose we don't need push to reg for each PR.

@SamYuan1990 sounds good, I think removing the attestation from image_pr.yml makes sense.  The attestation in general was suggested to me by @dave-tucker (tagged in case he has any input/suggestions here).  I will update this PR over the next 1-2 days.

@SamYuan1990 sounds good, I think removing the attestation from image_pr.yml makes sense. The attestation in general was suggested to me by @dave-tucker (tagged in case he has any input/suggestions here). I will update this PR over the next 1-2 days.
Follow-up question: Is there a reason we are pushing to registry at all for PRs?

IMO for PR level,
https://github.com/sustainable-computing-io/kepler/blob/main/.github/workflows/pull_request.yml#L50
the push image is set to false, so for each PR we should not push image.
https://github.com/sustainable-computing-io/kepler/blob/main/.github/workflows/image_pr.yml#L4 is working with workflow_dispatch. so when we need to debug/test a pr image/pr version, someone with manual build the image and upload it.
https://quay.io/repository/sustainable_computing_io/kepler?tab=tags&tag=latest which same as image tags you see on quay io.

I removed the attestation on PR and restricted permissions where possible.

the code change look good to me, @arthurus-rex , as push-to-registry is set, I am not sure if @rootfs, @vprashar2929  or other kepler maintainer can have a link to preview the output?


PublishTestResults Task fails

The official task from Azure to publish test results fails. This happens because the task checks if dotnet is installed by running the command dotnet --version. This arg is not available in the dotnetcore package installed in the image of the runner.
/usr/bin/dotnet --version
The command could not be loaded, possibly because:
  * You intended to execute a .NET application:
      The application '--version' does not exist.
  * You intended to execute a .NET SDK command:
      No .NET SDKs were found.

I understand you are trying to run a task using in backend .NET CLI. The task tries to run /usr/bin/dotnet --version, which fails.
Questions:

Is any task using .NET Core working fine before this one?
Can you please give me details about the project version and container distro you are using?

Hello @clemlesne
thank you for your feedback. I managed to solve this by manually installing the required SDK for dotnet:
           - bash: |
               microdnf remove aspnetcore-runtime-6.0
               microdnf install dotnet-sdk-8.0
               microdnf install dotnet-runtime-8.0
             displayName: :Install > Runtime

           - task: PublishTestResults@2
             inputs:
               testResultsFormat: 'JUnit'
               testResultsFiles: 'CheckovReport.xml'
               failTaskOnFailureToPublishResults: true
               failTaskOnMissingResultsFile: true
               testRunTitle: 'Checkov Results'
             displayName: "Publish > Checkov scan results"

Regarding your questions:

None of the dotnet tasks were working
I installed the latest helm chart and I am using the ubi8 image
I am using the PublishResults task from Azure and the GitTools custom tasks
Pipelines yaml definitions is as follows:

# Terraform pipeline

trigger:
  - main

variables:
  tf_version: "latest" # what version of terraform should be used
  cost_increase_alert_percentage: 50 # if the difference in costs is higher than x% then you will need to manually validate the deployment

pool:
  name: "test"

stages:
  # Run Checkov, upload results and optionally halt if there any problems found
  ######################################################################

  - stage: "runCheckov"
    displayName: "Checkov - Scan Terraform files"
    jobs:
      - job: "runCheckov"
        displayName: "Checkov > Pull, run and publish results of Checkov scan"
        steps:
          - bash: |
              python -m pip install checkov
              echo $PWD
            workingDirectory: $(System.DefaultWorkingDirectory)
            displayName: "Install  > bridgecrew/checkov"
          - bash: |
              checkov --file *.tf . --output junitxml --soft-fail > $(System.DefaultWorkingDirectory)/CheckovReport.xml
              echo $PWD
              ls -lah .
            workingDirectory: $(System.DefaultWorkingDirectory)
            displayName: :Run > Checkov"
          - bash: |
              microdnf remove aspnetcore-runtime-6.0
              microdnf install dotnet-sdk-8.0
              microdnf install dotnet-runtime-8.0
            displayName: :Install > Runtime

          - task: PublishTestResults@2
            inputs:
              testResultsFormat: 'JUnit'
              testResultsFiles: 'CheckovReport.xml'
              failTaskOnFailureToPublishResults: true
              failTaskOnMissingResultsFile: true
              testRunTitle: 'Checkov Results'
            displayName: "Publish > Checkov scan results"

Hope this is helpful,
Thank you

Did you tried to install the .NET SDK with the task UseDotNet@2 (like in the doc)?

Using the task you mentioned it works. Thank you for the help


Log collector

In what area(s)?

/area administration
/area ansible
/area autoscaling
/area configuration
/area cyclecloud
/area documentation
/area image
/area job-scheduling

/area monitoring

/area ood
/area remote-visualization
/area user-management

Describe the feature

It would be nice if we could view text logs without having to log into the individual VMs. We are having an issue with the slurm scheduler and it's quite hard to debug without seeing the logs. If Grafana Loki were configured it would be really nice.
cc @ltalirz

Thanks for the suggestion, having a log collector is definitively a nice to have

If this gets implemented, it would be really nice if it also grabbed the AD logs from the windows server. It's super hard to debug AD failures so that would be a really common use-case

Hi @xpillons , we are realizing more and more that we need a log collector to run az-hop reliably, and we may be able to contribute a prototype implementation.
If you have any specific suggestions / hints / things to keep in mind, please let us know.

if possible the goal is to leverage Azure Monitor and Log Analytics. This will avoid having to maintain another infrastructure for log analysis.

Just mentioning that this is also a severity "high" recommendation by the monitoring advisor
"Log Analytics agent should be installed on virtual machines"

Defender for Cloud collects data from your Azure virtual machines (VMs) to monitor for security vulnerabilities and threats. Data is collected using the Log Analytics agent, formerly known as the Microsoft Monitoring Agent (MMA), which reads various security-related configurations and event logs from the machine and copies the data to your Log Analytics workspace for analysis. This agent is also required if your VMs are used by an Azure managed service such as Azure Kubernetes Service or Azure Service Fabric. We recommend configuring auto-provisioning to automatically deploy the agent. If you choose not to use auto-provisioning, manually deploy the agent to your VMs using the instructions in the remediation steps.

Will look into this


Update ipfs-search URL, source URL and description

Can you re-generate the readme file?

There you go. Build seemed to fail though...


Suggestions for same hostname multiple path(s)

Just wondering if you might have any template examples that support URL/path based routing for different apps sharing the same DNS/Hostname? With my current template Nginx refuses to start because I end up with many server blocks that have the same hostname.
I'm currently running Marathon-lb but am investigating using Nixy/Nginx. We route many microservices for the same Hostname to different paths (/common/ms1, /common/ms2, etc) which is achievable with MLB by assigning apps the same VHOST label with different PATH labels for the path routing.
That would require generating a single server block shared by many apps with multiple location blocks routing to each app in marathon.
Any suggestions/pointers would be greatly appreciated, thanks!

Hi @kholloway, let me take a look and come back about that. Should not be that hard to fix :)

You can now take a look at https://github.com/martensson/nixy/blob/master/nginx-path.tmpl

Ahh nice trick with the servername, didn't realize you could do that with Nginx.
That covers most of my uses cases with one exception. Two different apps with same URL path but different host names. As long as my paths are all different then this config doesn't really care about my host name.
Thanks!

Mixed path and host based config makes things a bit more tricky, but not impossible. The nice thing with using templates and nginx is the endless possibilities it gives ;)

Very true! Thanks again!!


Title: Add Multi-language Support Icon in the Header Description and improve some styles

Currently, the multi-language switcher is functional, but it lacks an intuitive icon in the header. Adding an icon for language selection will improve the user experience by making the feature more accessible and visible.

Currently, the multi-language switcher is functional, but it lacks an intuitive icon in the header. Adding an icon for language selection will improve the user experience by making the feature more accessible and visible.

Oky  I will assign you do it 👍


feat: add hybrid-HPCS support

Description
Add the Hybrid-HPCS support
Types of changes in this PR
Changes that affect the core Terraform module or submodules

[ ] Bug fix
[x] New feature
[ ] Dependency update

Changes that don't affect the core Terraform module or submodules

[ ] Examples or tests (addition or updates of examples or tests)
[ ] Documentation update
[ ] CI-related update (pipeline, etc.)
[ ] Other

Release required?
Identify the type of release. For information about the changes in a semantic versioning release, see Release versioning.

[ ] No release
[ ] Patch release (x.x.X)
[x] Minor release (x.X.x)
[ ] Major release (X.x.x)

Release notes content
Adds Hybrid HPCS support.
Run the pipeline
If the CI pipeline doesn't run when you create the PR, the PR requires a user with GitHub collaborators access to run the pipeline.
Run the CI pipeline when the PR is ready for review and you expect tests to pass. Add a comment to the PR with the following text:
/run pipeline

Checklist for reviewers

[ ] If relevant, a test for the change is included or updated with this PR.
[ ] If relevant, documentation for the change is included or updated with this PR.

Merge actions for mergers

Use a relevant conventional commit message that is based on the PR contents and any release notes provided by the PR author. The commit message determines whether a new version of the module is needed, and if so, which semver increment to use (major, minor, or patch).
Merge by using "Squash and merge".

/run pipeline

@vburckhardt @jojustin - Added the required code to implement hybrid HPCS as per the information provided and added an example for the same. Example readme may have more information once we start test. Kindly review.

/run pipeline


Build mechanism update

Make build script compatible with latest unity version
Use more system variables
make steam path dynamic (has to be set in the script if not standard)

Nice work, man. There are more changes included, not just the "build mechanism". Have a look at the files.

I see that, I created this branch from my development instead of the Bruce branch ....
I need to rework it.


po_lines_location.sql

will fix #556

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Could we have a complete description for the pull request:  https://github.com/folio-org/folio-analytics/blob/main/CONTRIBUTING.md#2-commits-and-pull-requests

Could we have a complete description for the pull request: https://github.com/folio-org/folio-analytics/blob/main/CONTRIBUTING.md#2-commits-and-pull-requests

I think this is my bad. I just advised arthur to enter the related issue.
@arthur-aguilera
Please, add something like: This ports the derived table po_lines_locations to Metadb
Thanks!


Incorrect IP address displayed

I'm also using netconnectd plugin as a fallback for communication. OctoPrint Anywhere only displays the IP address that the host AP interface would have, if it were up, which would only be if no external connection were available.
ipOnConnect plugin still displays the correct IP address, always. Might want to check how that plugin finds IP address.
Most important would be the address of the interface with the default route.

Thank you so much @sdgray for pointing me to ipOnConnect plugin. I'll definitely check how it obtains the correct IP.

@sdgray do you have more than 1 wifi network interfaces on your Pi? Probably an internal one and a wifi dongle? I installed netconnectd but can't reproduce the same problem.
A list of steps to reproduce this problem will be appreciated.

No, only the two interfaces total (ethernet and wifi).
On Thu, Jul 20, 2017 at 8:50 AM, Kenneth Jiang<EMAIL_ADDRESS>wrote:

@sdgray https://github.com/sdgray do you have more than 1 wifi network
interfaces on your Pi? Probably an internal one and a wifi dongle? I
installed netconnectd but can't reproduce the same problem.
A list of steps to reproduce this problem will be appreciated.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kennethjiang/OctoPrint-Anywhere/issues/16#issuecomment-316692842,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKRPSeMuS8d2CKGQsVQTMbS5cXH8YjO5ks5sP00UgaJpZM4OPf2W
.

Ah yes you are using ethernet too!
I don't know if I want to use the way ipOnConnect detects IP address. As you have observed it only returns the IP of the default route. However I can see some use cases when this is not good enough. For instance, ethernet is attached to default gateway and the user wants to know the IP of Pi's wifi because the laptop is also on wifi network.
Ideally I should find out all routable IPs of OctoPrint and display them in OctoPrint Anywhere. But apparently there is a bug somewhere that prevents the code from doing it correctly.
Can you login to your Rasbperry Pi to run hostname -I and tell me if the output has all IP addresses, including the one of default route?

Here is all I can think to give you right now.
I don't know if I want to use the way ipOnConnect detects IP address. As
you have observed it only returns the IP of the default route. However I
can see some use cases when this is not good enough. For instance, ethernet
is attached to default gateway and the user wants to know the IP of Pi's
wifi because the laptop is also on wifi network.
Ideally I should find out all routable IPs of OctoPrint and display them
in OctoPrint Anywhere. But apparently there is a bug somewhere that
prevents the code from doing it correctly.
Can you login to your Rasbperry Pi to run hostname -I and tell me if the
output has all IP addresses, including the one of default route?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kennethjiang/OctoPrint-Anywhere/issues/16#issuecomment-316695685,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKRPSUq7YgwPI-yt46nzLPZpift15ok7ks5sP0-ZgaJpZM4OPf2W
.

Thank you @sdgray for the detailed information. I just released a new version that hopefully will correctly obtain and display all these ip addresses. Please upgrade and let me know if it actually works (I don't have an ethernet cable or router so I can't test it myself).

That works just fine. All IPs listed!
On Thu, Jul 20, 2017 at 11:01 AM, Kenneth Jiang<EMAIL_ADDRESS>wrote:

Thank you @sdgray https://github.com/sdgray for the detailed
information. I just released a new version that hopefully will correctly
obtain and display all these ip addresses. Please upgrade and let me know
if it actually works (I don't have an ethernet cable or router so I can't
test it myself).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kennethjiang/OctoPrint-Anywhere/issues/16#issuecomment-316730986,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKRPSWTwddqtueA78R7tgOTHSYKaNWThks5sP2uxgaJpZM4OPf2W
.

Hooray! Thank you so much @sdgray for reporting and helping me with this bug!


Premium Stripe integration

Fixes Issue
TODO

[x] Premium feature: hide navbar/footer
[x] Deploy to preview
[ ] Lock down Stripe webhook
[ ] Premium feature: Link detailed stats page
[x] Premium feature: Profile referrer pie graph + top referrers (data https://github.com/EddieHubCommunity/LinkFree/pull/8654)
[ ] Light/Dark mode check for new pages
[ ] Deploy to production

[ ] Set environment variables for production
[ ] Stripe webhook url

Admin

[ ] List Premium users

Pricing page

[ ] premium plan to have badge 30 days free
[ ] feature description
...

IDEAS

[x] data: referrer
[x] location stats

Changes proposed

Check List (Check all the applicable boxes)

[ ] My code follows the code style of this project.
[ ] My change requires changes to the documentation.
[ ] I have updated the documentation accordingly.
[ ] All new and existing tests passed.
[ ] This PR does not contain plagiarized content.
[ ] The title of my pull request is a short description of the requested changes.

Screenshots

Note to reviewers

Repo feature for home page

Repo feature for home page

My project got listed at last number in this image 😀

Hey,

are

Apologies, for closing. I clicked on the wrong Keyboard shortcuts.
Here are a couple of feedbacks after using testing it.
1) Getting a 500 when I click on the Account button.
2) When I entered my email, it gives me an error. Now it says

3) When I clicked on continue (from above), it sent me an email to do the payment and manage. Now, I am done with everything (adding the card and done with the payment), the continue button is not working.

Dark mode is not supported on Pricing page:

Learn more about Premium should redirect to the pricing page not searching for user

Thank you for testing and the feedback ❤

Pricing page in dark mode is looking good to me:

Thanks @schmelto for re-testing 👍

Location stats are bug for dark mode:

Thank you @schmelto ❤


transcribe: 9.41.0 -> 9.41.2

Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Seems like they pulled https://www.seventhstring.com/xscribe/downlo/xscsetup-9.41.0.tar.gz so transcribe fails to build until this is merged.


How do I access the landmark points in the java layer?

I am trying to send POST requests to a server based on gestures (hand tracking example) recognized through the landmark points. I saw that the landmark points are available in most calculators but transferring them to the java layer (specifically MainActivity.java) seems tricky. Any suggestions would be great!

Try to revise calculator itself sending out vector(or any other compatible forms) represents landmark points
Then adding PacketCallback like this to get the data from graph
At least this works for me
processor.getGraph().addPacketCallback("output_vector", new PacketCallback() {
       @Override
       public void process(Packet packet) {
              float[] dest = PacketGetter.getFloat32Vector(packet);
              // Do something with the landmark point
       }
});

@lcycoding   how to add "output_vector" in graph?  would you please share more detail?  thx very much!!!

I do know how to get face detector result too...

@liyuming1978 in your node.cc
How about just adding outputs in GetContract()
  if (cc->Outputs().HasTag("OUT_VEC")) {
    cc->Outputs().Tag("OUT_VEC").Set<std::vector<float>>();
  }

And in process()
 cc->Outputs().Tag("OUT_VEC").Add(output_vecs.release(),
                                         cc->InputTimestamp());

The other part is pure c code part, just code it...

@lcycoding  thx so much!  that works,  simple and rough.:)

@lcycoding  thx so much!  that works,  simple and rough.:)

  Have you implemented it? Can you share how do access the landmark points in the java layer?