[HDFS-13694]Making md5 computing being in parallel with image loading

Signed-off-by: sunlisheng<EMAIL_ADDRESS>

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
25
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
1179
trunk passed

+1
compile
63
trunk passed

+1
checkstyle
45
trunk passed

+1
mvnsite
72
trunk passed

+1
shadedclient
811
branch has no errors when building and testing our client artifacts.

+1
javadoc
50
trunk passed

0
spotbugs
176
Used deprecated FindBugs config; considering switching to SpotBugs.

-1
findbugs
174
hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant findbugs warnings.

_ Patch Compile Tests _

+1
mvninstall
64
the patch passed

+1
compile
60
the patch passed

+1
javac
60
the patch passed

-0
checkstyle
38
hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 28 unchanged - 0 fixed = 29 total (was 28)

+1
mvnsite
71
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
765
patch has no errors when building and testing our client artifacts.

+1
javadoc
49
the patch passed

-1
findbugs
186
hadoop-hdfs-project/hadoop-hdfs generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)

_ Other Tests _

-1
unit
5199
hadoop-hdfs in the patch failed.

+1
asflicense
60
The patch does not generate ASF License warnings.

8978

Reason
Tests

FindBugs
module:hadoop-hdfs-project/hadoop-hdfs

Should org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader$DigestThread be a static inner class?  At FSImageFormatProtobuf.java:inner class?  At FSImageFormatProtobuf.java:[lines 176-203]

Failed junit tests
hadoop.hdfs.server.diskbalancer.TestDiskBalancer

hadoop.hdfs.web.TestWebHdfsTimeouts

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1010

JIRA Issue
HDFS-13694

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 7727bfa4491b 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / b76b843

Default Java
1.8.0_212

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/1/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/1/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/1/artifact/out/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/1/testReport/

Max. process+thread count
3517 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/1/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
29
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
1036
trunk passed

+1
compile
61
trunk passed

+1
checkstyle
38
trunk passed

+1
mvnsite
64
trunk passed

+1
shadedclient
722
branch has no errors when building and testing our client artifacts.

+1
javadoc
47
trunk passed

0
spotbugs
172
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
170
trunk passed

_ Patch Compile Tests _

+1
mvninstall
61
the patch passed

+1
compile
57
the patch passed

+1
javac
57
the patch passed

-0
checkstyle
37
hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 28 unchanged - 0 fixed = 29 total (was 28)

+1
mvnsite
62
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
721
patch has no errors when building and testing our client artifacts.

+1
javadoc
47
the patch passed

+1
findbugs
175
the patch passed

_ Other Tests _

-1
unit
4991
hadoop-hdfs in the patch failed.

+1
asflicense
36
The patch does not generate ASF License warnings.

8408

Reason
Tests

Failed junit tests
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks

hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup

hadoop.hdfs.web.TestWebHdfsTimeouts

hadoop.hdfs.server.diskbalancer.TestDiskBalancer

hadoop.hdfs.server.datanode.TestDirectoryScanner

hadoop.hdfs.qjournal.server.TestJournalNodeSync

hadoop.hdfs.server.namenode.snapshot.TestSnapshot

hadoop.tracing.TestTracing

hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff

hadoop.hdfs.server.balancer.TestBalancerRPCDelay

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1010

JIRA Issue
HDFS-13694

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 86b570d993b9 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 041e7a7

Default Java
1.8.0_212

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/2/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/2/testReport/

Max. process+thread count
4599 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/2/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
28
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
1053
trunk passed

+1
compile
58
trunk passed

+1
checkstyle
39
trunk passed

+1
mvnsite
63
trunk passed

+1
shadedclient
719
branch has no errors when building and testing our client artifacts.

+1
javadoc
53
trunk passed

0
spotbugs
170
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
167
trunk passed

_ Patch Compile Tests _

+1
mvninstall
67
the patch passed

+1
compile
58
the patch passed

+1
javac
58
the patch passed

-0
checkstyle
37
hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 28 unchanged - 0 fixed = 29 total (was 28)

+1
mvnsite
64
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
737
patch has no errors when building and testing our client artifacts.

+1
javadoc
48
the patch passed

-1
findbugs
174
hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

_ Other Tests _

-1
unit
6193
hadoop-hdfs in the patch failed.

+1
asflicense
40
The patch does not generate ASF License warnings.

9648

Reason
Tests

FindBugs
module:hadoop-hdfs-project/hadoop-hdfs

Should org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader$DigestThread be a static inner class?  At FSImageFormatProtobuf.java:inner class?  At FSImageFormatProtobuf.java:[lines 176-203]

Failed junit tests
hadoop.hdfs.server.diskbalancer.TestDiskBalancer

hadoop.hdfs.TestWriteReadStripedFile

hadoop.hdfs.TestHdfsAdmin

hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy

hadoop.hdfs.TestMaintenanceState

hadoop.hdfs.tools.TestDFSZKFailoverController

hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy

hadoop.hdfs.TestDistributedFileSystemWithECFileWithRandomECPolicy

hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy

hadoop.hdfs.web.TestWebHdfsTimeouts

hadoop.hdfs.TestRollingUpgradeDowngrade

hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1010

JIRA Issue
HDFS-13694

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 2d2ecc60b973 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 041e7a7

Default Java
1.8.0_212

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/3/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/3/artifact/out/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/3/testReport/

Max. process+thread count
3937 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/3/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
30
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
1077
trunk passed

+1
compile
61
trunk passed

+1
checkstyle
45
trunk passed

+1
mvnsite
69
trunk passed

+1
shadedclient
790
branch has no errors when building and testing our client artifacts.

+1
javadoc
55
trunk passed

0
spotbugs
161
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
159
trunk passed

_ Patch Compile Tests _

+1
mvninstall
60
the patch passed

+1
compile
56
the patch passed

+1
javac
56
the patch passed

-0
checkstyle
41
hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 28 unchanged - 0 fixed = 29 total (was 28)

+1
mvnsite
64
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
733
patch has no errors when building and testing our client artifacts.

+1
javadoc
49
the patch passed

-1
findbugs
169
hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

_ Other Tests _

-1
unit
4826
hadoop-hdfs in the patch failed.

+1
asflicense
28
The patch does not generate ASF License warnings.

8369

Reason
Tests

FindBugs
module:hadoop-hdfs-project/hadoop-hdfs

Should org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader$DigestThread be a static inner class?  At FSImageFormatProtobuf.java:inner class?  At FSImageFormatProtobuf.java:[lines 176-203]

Failed junit tests
hadoop.hdfs.TestDatanodeDeath

hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes

hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean

hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy

hadoop.hdfs.server.diskbalancer.TestDiskBalancer

hadoop.hdfs.TestDFSStripedOutputStream

hadoop.hdfs.web.TestWebHdfsTimeouts

hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1010

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux a71b8f184110 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 049bb42

Default Java
1.8.0_212

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/4/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/4/artifact/out/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/4/testReport/

Max. process+thread count
4668 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/4/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@goiri  I have updated this patch as your comments. The exists unit test is coverage in this area.Could you have time to help review this patch? Thank you.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
30
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
1047
trunk passed

+1
compile
62
trunk passed

+1
checkstyle
41
trunk passed

+1
mvnsite
70
trunk passed

+1
shadedclient
758
branch has no errors when building and testing our client artifacts.

+1
javadoc
53
trunk passed

0
spotbugs
157
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
156
trunk passed

_ Patch Compile Tests _

+1
mvninstall
60
the patch passed

+1
compile
55
the patch passed

+1
javac
55
the patch passed

-0
checkstyle
38
hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 28 unchanged - 0 fixed = 30 total (was 28)

+1
mvnsite
60
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
712
patch has no errors when building and testing our client artifacts.

+1
javadoc
42
the patch passed

+1
findbugs
165
the patch passed

_ Other Tests _

-1
unit
5728
hadoop-hdfs in the patch failed.

+1
asflicense
47
The patch does not generate ASF License warnings.

9190

Reason
Tests

Failed junit tests
hadoop.hdfs.TestDatanodeDeath

hadoop.hdfs.TestReplication

hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy

hadoop.hdfs.server.diskbalancer.TestDiskBalancer

hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy

hadoop.hdfs.TestFileChecksumCompositeCrc

hadoop.hdfs.TestDFSStripedOutputStream

hadoop.hdfs.web.TestWebHdfsTimeouts

hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1010

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux ed36d96eea0e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / a79bdf7

Default Java
1.8.0_212

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/5/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/5/testReport/

Max. process+thread count
5279 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/5/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@goiri I have updated this patch. Could you continue to help review this patch? Thank you.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
35
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
1136
trunk passed

+1
compile
63
trunk passed

+1
checkstyle
44
trunk passed

+1
mvnsite
67
trunk passed

+1
shadedclient
763
branch has no errors when building and testing our client artifacts.

+1
javadoc
48
trunk passed

0
spotbugs
177
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
173
trunk passed

_ Patch Compile Tests _

+1
mvninstall
65
the patch passed

+1
compile
57
the patch passed

+1
javac
57
the patch passed

-0
checkstyle
35
hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 28 unchanged - 0 fixed = 30 total (was 28)

+1
mvnsite
63
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
710
patch has no errors when building and testing our client artifacts.

+1
javadoc
48
the patch passed

+1
findbugs
168
the patch passed

_ Other Tests _

-1
unit
4888
hadoop-hdfs in the patch failed.

+1
asflicense
39
The patch does not generate ASF License warnings.

8455

Reason
Tests

Failed junit tests
hadoop.hdfs.TestDatanodeDeath

hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes

hadoop.hdfs.qjournal.server.TestJournalNodeSync

hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy

hadoop.hdfs.server.diskbalancer.TestDiskBalancer

hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA

hadoop.hdfs.TestDFSStorageStateRecovery

hadoop.hdfs.TestDFSStripedOutputStream

hadoop.hdfs.web.TestWebHdfsTimeouts

hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/7/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1010

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 6fe5028fc878 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / be80334

Default Java
1.8.0_212

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/7/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/7/testReport/

Max. process+thread count
4925 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/7/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
128
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
1373
trunk passed

+1
compile
79
trunk passed

+1
checkstyle
48
trunk passed

+1
mvnsite
84
trunk passed

+1
shadedclient
986
branch has no errors when building and testing our client artifacts.

+1
javadoc
65
trunk passed

0
spotbugs
192
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
189
trunk passed

_ Patch Compile Tests _

+1
mvninstall
79
the patch passed

+1
compile
73
the patch passed

+1
javac
73
the patch passed

-0
checkstyle
44
hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 28 unchanged - 0 fixed = 30 total (was 28)

+1
mvnsite
84
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
906
patch has no errors when building and testing our client artifacts.

+1
javadoc
59
the patch passed

+1
findbugs
208
the patch passed

_ Other Tests _

-1
unit
6774
hadoop-hdfs in the patch failed.

+1
asflicense
35
The patch does not generate ASF License warnings.

11304

Reason
Tests

Failed junit tests
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks

hadoop.hdfs.server.namenode.TestDecommissioningStatus

hadoop.hdfs.server.diskbalancer.TestDiskBalancer

hadoop.hdfs.server.datanode.TestDirectoryScanner

hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure

Subsystem
Report/Notes

Docker
Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/6/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1010

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux e8a49be6f122 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / be80334

Default Java
1.8.0_212

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/6/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/6/testReport/

Max. process+thread count
2728 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/6/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@goiri  Thanks for your comments. I update this patch. Could you continue to review it? Thank you.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
69
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
1112
trunk passed

+1
compile
59
trunk passed

+1
checkstyle
42
trunk passed

+1
mvnsite
67
trunk passed

+1
shadedclient
795
branch has no errors when building and testing our client artifacts.

+1
javadoc
50
trunk passed

0
spotbugs
166
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
164
trunk passed

_ Patch Compile Tests _

+1
mvninstall
61
the patch passed

+1
compile
56
the patch passed

+1
javac
56
the patch passed

-0
checkstyle
37
hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 28 unchanged - 0 fixed = 30 total (was 28)

+1
mvnsite
62
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
774
patch has no errors when building and testing our client artifacts.

+1
javadoc
49
the patch passed

+1
findbugs
168
the patch passed

_ Other Tests _

-1
unit
6234
hadoop-hdfs in the patch failed.

+1
asflicense
33
The patch does not generate ASF License warnings.

9904

Reason
Tests

Failed junit tests
hadoop.tracing.TestTracing

hadoop.hdfs.TestMultipleNNPortQOP

hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA

hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem
Report/Notes

Docker
Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/8/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1010

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux ec764a9e9631 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 4a21224

Default Java
1.8.0_212

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/8/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/8/testReport/

Max. process+thread count
2731 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/8/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@goiri  I have added Javadoc explaining the overall idea and updated patch.Could you help review this patch? If have no question, please merge it trunk.Thank you.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
140
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
1762
trunk passed

+1
compile
80
trunk passed

+1
checkstyle
60
trunk passed

+1
mvnsite
93
trunk passed

+1
shadedclient
1040
branch has no errors when building and testing our client artifacts.

+1
javadoc
66
trunk passed

0
spotbugs
202
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
199
trunk passed

_ Patch Compile Tests _

+1
mvninstall
70
the patch passed

+1
compile
61
the patch passed

+1
javac
61
the patch passed

-0
checkstyle
50
hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 28 unchanged - 0 fixed = 30 total (was 28)

+1
mvnsite
77
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
887
patch has no errors when building and testing our client artifacts.

+1
javadoc
52
the patch passed

+1
findbugs
182
the patch passed

_ Other Tests _

-1
unit
6291
hadoop-hdfs in the patch failed.

+1
asflicense
34
The patch does not generate ASF License warnings.

11192

Reason
Tests

Failed junit tests
hadoop.hdfs.server.datanode.TestDirectoryScanner

hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes

Subsystem
Report/Notes

Docker
Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/9/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1010

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 2bc04a6e7af5 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / d203045

Default Java
1.8.0_212

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/9/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/9/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/9/testReport/

Max. process+thread count
2700 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/9/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Can we fix the checkstyle warnings?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
129
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
1278
trunk passed

+1
compile
65
trunk passed

+1
checkstyle
46
trunk passed

+1
mvnsite
71
trunk passed

+1
shadedclient
844
branch has no errors when building and testing our client artifacts.

+1
javadoc
52
trunk passed

0
spotbugs
170
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
168
trunk passed

_ Patch Compile Tests _

+1
mvninstall
64
the patch passed

+1
compile
59
the patch passed

+1
javac
59
the patch passed

+1
checkstyle
40
the patch passed

+1
mvnsite
65
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
811
patch has no errors when building and testing our client artifacts.

+1
javadoc
50
the patch passed

+1
findbugs
180
the patch passed

_ Other Tests _

-1
unit
6893
hadoop-hdfs in the patch failed.

+1
asflicense
48
The patch does not generate ASF License warnings.

10948

Reason
Tests

Failed junit tests
hadoop.hdfs.tools.TestDFSZKFailoverController

hadoop.hdfs.server.datanode.TestDirectoryScanner

hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA

hadoop.hdfs.server.namenode.ha.TestHAAppend

Subsystem
Report/Notes

Docker
Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/10/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1010

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 519877ca3588 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 2f4b37b

Default Java
1.8.0_212

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/10/testReport/

Max. process+thread count
2579 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1010/10/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Thank @goiri for your suggestion.  I have fixed the checkstyle warnings. Could you have time to review this patch ? Thank you.

LGTM
+1

Hi @goiri, If you think this patch no problem, could you help merge it to trunk ? Thank you.

Merged and updated the JIRA.
I'll update the patch there for completeness.


PMID:21151990 decreased microtubule bundle formation

decreased microtubule bundle formation

decreased microtubule bundle formation FYPO:0007150
edit file: 91b2bb7789f39baa6501546993a86ff9a05f0dca
release: e96bf3a739a2ced95bb4b13538c820dcfd2276a0


plot shaded quantiles instead of random samples

implementation introduced in d17cfb9 and works for the sep/pa plots in 2280f82. Doesn't produce proper result for the ra/dec plot, but closing for now


Relative import path "@olli/kvdex/ext/zod" not prefixed with.... on deno deploy

Is this a deno deploy bug because it works perfectly locally? If it is, i will open an issue about in their feedback repo and ping the discord channel.
I even committed deno.lock file to git and the issue still persisted.
I'll give jsr:@olli/kvdex/ext/zod a try after this and see how it goes

So using jsr:@olli/kvdex/ext/zod works instead of @olli/kvdex/ext/zod which is weird. I think it's a deno deploy bug.
I'm reporting the issue there as well

Thats very strange, I think it might be a bug on Deploy when importing package modules then. But I will have to check as well.

Wait, I read your issue wrong the first time. You have to use the jsr: specifier when importing from JSR. So that's not weird at all, and it's not an issue.

I think it could be that the import map doesn't do what you expect it to.
You would maybe have to add a separate entry for "@olli/kvdex/ext/zod": "jsr<EMAIL_ADDRESS>

I think it could be that the import map doesn't do what you expect it to. You would maybe have to add a separate entry for "@olli/kvdex/ext/zod": "jsr<EMAIL_ADDRESS>
Done. Thanks
I thought this update fixed it! Probably didn't take into account jsr modules at that time


ReferenceError: window is not defined

Error in Next.js typescript app.
error: ReferenceError: window is not defined
"next": "12.2.5",
"react": "18.2.0",
"react-dancing-lines": "^1.1.4",

I'm sure OP has resolved this by now, but if anyone else runs into this on nextjs you can dynamically import the component.
const DynamicDancingLines = dynamic(() => import('react-dancing-lines'), {     ssr: false, // Prevents server-side rendering of the component   });


Have maximum width for the central column in the layout. Right now it keeps becoming longer as the width increases. Check examples of twitter and Facebook:

Resolved in #2011


Your work is wonderful!

Mattermost core team here, your work looks terrific!
Would you be open to having us share about your work with the community?
We try to list and tweet about projects that can be of benefit to users,

Hey @it33 happy to have you share it around. I just fixed the CI pipeline and v2 now includes the latest v1.2.0-rc2 pre-release.

The CI pipeline doesn't yet run any smoke tests on the deployment, it stops at just running a single node and confirms that that works.
  Started updating job mattermost > mattermost/0 (canary). Done (00:00:09)

Unfortunately because testflight ci job exposes credentials I've made it private. But here's a look at it passing at moment:

@it33 if you have any smoke tests that we can run against a deployment to confirm it is working after an upgrade, let me know and I'll add them to our pipeline to test our releases before we cut them.

I'd like to test upgrades - from v1 to v2 etc in future too.

Correction: v3 now includes v1.2.0 that just came out https://github.com/cloudfoundry-community/mattermost-boshrelease/releases/tag/v3

@drnic, fantastic!!
We don't have smoke tests documented, but we should (ticket filed. Each month we push our RC candidates to https://gitlab.mattermost.com/community and we have nightly builds on our community server (please join us!): https://pre-release.mattermost.com/core/
We do have an upgrade guide, and would love your help testing. We ship on the 16th of the month, and aim to get Release Candidates out 5 working days prior. Working on documenting all this better...
Super excited about your work!


Crashes all the time after updating

Crash once open in mcSierra:

VSCode Version: 1.13.1 (379d2efb5539b09112c793d3d9a413017d736f89)
OS Version: Mac OSX 10.12.5

Steps to Reproduce:

Terminal
Execute code --disable-extensions --verbose
Result: hangs with this message - [main 7:38:34 AM] Sending env to running instance...

Update
I removed Code folder in Application Support and it works now!

All code here, close this.


ORC-833: Calculate nextVector Batch Size Once

What changes were proposed in this pull request?
The number of iterations of the loops in RunLengthIntegerReaderV2 nextVector look at two distinct, but known, values. Instead of check each value in each loop, pre-compute the batch size.
Why are the changes needed?
Code clarity, and if the optimizer doesn't already do it automagically, saves a few branches.
How was this patch tested?
No changes in functionality.  Uses existing unit tests.

Merged to main.


Match lines containing question mark

The append_if_no_line resource was not correctly identifying the
presence of a line if that line had a question mark, due to the question
mark being interpreted as part of the regex. As a result the line would
be appended every chef run. This commit adds the question mark character
to the list of characters to escape in Line::Helper.escape_string.

This change is

Thank you, nice catch.

added a line to the changelog as @damacus suggested


RFC - Add an owning "borrowed" pointer type &move

Rendered
Big remaining questions:

When is the memory deallocated when a &move is stored somewhere? Ideally when it's not stored, the memory is released as soon as possible (i.e. at the end of the statement), but when stored that would have to be deferred until the end of the original scope.

I like it. The only thing I really disagree with is having to use mut to be able to mutate through &move pointers; it feels like more trouble than it's worth. I always want to be able to turn an &move into an &mut; &move is strictly a superset of &mut; so having that would be weird, imo.
The other thing is in your unresolved questions, you ask about box destructuring. I don't think these have anything to do with each other.

The "rendered" link is broken.
How does this compare with #965? (cc #998)

@ubsan - The mut requirement came out of an IRC conversation, see example:
let v = Vec::new();
let v2 = &move v;
v2.push(123);

As for box de-structuring... it's probably unrelated, just was a similar case of special-cased Box
@kennytm Fixed the link, will read those issues in a bit.

I might just be too new to the language to see what this solves.  I'd love to see a simple but meaningful example of what I can express that I couldn't express before.  The examples in the RFC aren't explaining why I want it in a way that I can get it.
I'm not familiar with the difficulty that arises with Box<FnOnce>.  Other than that, I see a thing that works mostly like &mut.  Why wouldn't I just use regular moves and/or Box?
Partially this is curiosity.  But other people will ask this question if the RFC is approved, so I feel justified in asking it now.

How do you avoid the unsafety of deallocate? At the moment, the trait is safe to implement, and yet the deallocate implementation has access to dropped data.

@camlorn With this, you can now owned iterate over an array, for example.

@ubsan What do you mean by "owned iterate over an array", and how does it differ from into_iter?

@bstrie for x in [a, b, c] { drop(x); }

How exactly would this RFC allow for iterating over an owned array?
IntoIterator takes self, not &move self, and we can't change that without breaking backwards compatiblity.
For most usecases, it would be better if we allowed passing DSTs by-value (which under the hood would generate the same code as &move). But if we do that, what usecases remain left for a language-level &move?

@dgrunwald impl<T> IntoIterator for &move [T]. &move is strictly more powerful; you can't currently return "by-value" DSTs, but with this, you could.

@thepowersgang Make sure to add a section about *move, and how it will change Unique.

@camlorn
The problem with Box<FnOnce()> is this: you can't call it. FnOnce::call takes self by value, but a trait object such as FnOnce() is unsized, so you get an error when trying to make the call. However, using boxed closures is desirable when you don't know the concrete type, and specifically Box<FnOnce()> for its only-called-once guarantee. Right now we have the unstable FnBox that can paper over this if you need those guarantees (if you look at the source, you'll see that FnBox::call takes self: Box<Self> which works due to compiler magic and moves the whole box, avoiding the error). There are various ways to fix it, but until we decide which one is perfect we'll have the unstable kludge.
@bstrie
Note that [1, 2, 3].into_iter() is still an Iterator<Item=&i32>.

I'm not sure if I buy this as a solution for the Box<FnOnce> issue. It's my understanding that we already have mechanisms in the compiler for having functions which take by-value self take a pointer to self. Today, this alleviates the need for pushing a large structure onto the stack whenever we want to consume it by-value. I think it would be possible to extend that to all trait methods consuming self by-value which are not marked explicitly as where Self: Sized.  This is equivalent to the &move proposed here, but it doesn't require adding any complexity to the language itself -- I think it's better to have it done under the hood.

@rphmeier Sure Box works in the compiler, but this is about implementing Box in a library on top of lower-level primitives.

@eddyb I don't fundamentally disagree with the concept of a DerefMove trait, I just don't think that it necessarily goes hand-in-hand with an &move type.
The way we move out of Box today consumes the box and returns the value. To me, that implies a trait like this:
trait DerefMove: DerefMut {
    fn deref_move(self) -> Self::Target;
}

with an implementation for Box like
impl<T: Sized> DerefMove for Box<T> {
    fn deref_move(self) -> T {
        let val = ptr::read(&self.0);
        let (size, align) = (mem::size_of::<T>(), mem::align_of::<T>());
        if size > 0 {
            alloc::deallocate(self.0, size, align);
        }
        mem::forget(self) // skip destructor
        val
    }
}

The compiler can special-case this trait like it does Deref and DerefMut, allowing for Box to be implemented as a library type. Anything related to methods on DSTs which consume self can be done under the hood (at trans time) by passing a pointer to self rather than pushing self onto the stack.
Although the signature of DerefMove inherently doesn't allow for moving unsized types, CoerceUnsized<T> will only work if the struct has one and only one unsizeable field. That means the compiler has to be aware of which field is unsized.
We can pass a pointer to self to functions which expect self by-value (trans will make it work)
These two pieces of information together can enable calling by-value self methods on trait objects by
a) marking the unsized field on the pointer type as already dropped (so we don't drop it again)
b) passing the data pointer of the unsized field to the function found in the field's vtable.
This will cause some interactions with destructors for pointer types which interact with the data they wrap. This is why today we don't allow moving out of types which implement Drop. I think with non-zeroing drop or without drop flags that this strategy wouldn't work anyway.
At the same time, I do think that this could be made to work under the hood without any &move type.

@rphmeier  The problem with returning by value there is determining when the underlying allocation is released (and allowing a library type to release that memory)

@rphmeier So you'd be replacing the need for DerefMove with a CoerceUnsized-specific hack?
But that only works if the pointer is to T, not SomeWrapper<T>.
You could add another hack to also adjust the pointer. That seems so messy.
And still doesn't let you return an owning reference to a sub-field or anything like that.

Oh, by the way; moving out should not deallocate immediately, but at end of scope. This would allow for people to write
{
  let v = vec![1, 2, 3]
  let x = &move v[..];
  x[1] // still allowed, v is not deallocated yet
}
{
  let v = vec![1, 2, 3];
  let x = &move v[..] as *move _;
  unsafe { (*x)[2] } // still allowed, v is not deallocated yet
}

This means there's no confusing rules about lets or anything, and you can guarantee that the thing will get dropped at end of scope. This is useful for something like:
struct Matrix {
  indices: *move [usize],
  buffer: *move [u8],
}

impl DerefMove for Matrix {
  fn deref_move(&mut self) -> &move [u8] {
    &move *self.buffer
  }
  fn deallocate(&mut self) {
    Box::from_raw(indices);
    Box::from_raw(buffer);
  }
}

@ubsan I agree, the deallocation should be where drops would occur atm, which also makes the borrows simpler to work with (they just have to stop being used at some point before the drop/deallocation).

Make sure to point out you cannot move out of things that impl DerefMove.

Also... why not just use the already existing Drop machinery? We can remove deallocate, and just use Drop::drop (everything else would be the same, just you'd impl Drop).

@ubsan I think the distinction between Drop::drop and DerefMove::deallocate is such that Drop cleans up after its fields, while deallocate needs to assume the cleanup has been done already.

@rphmeier But drop could just assume that the cleanup has been done already by DerefMove, and, if you don't call DerefMove in scope, it will just call it at end of scope before dropping.

Yeah, I think that's the only way this really can work. I was wondering above about the claim that DerefMove is equivalent to Drop, and the only way that's really true is if DerefMove gets automatically called before Drop when the type isn't moved out of.
I'm also for the idea that for any T: Sized, you should be able to move out of an &move T by dereferencing it. Right now you can move out of an &mut T with mem::replace, but since &move T is supposed to be strictly more powerful it should allow direct moving out.
fn bar(_: String) {}
trait Foo {
    fn consume(&move self);
}

impl Foo for String {
    fn consume(&move self) {
        bar(*self);
    }
}

When you drop an &move T, it should call T's destructor. When you move out of an &move T, you just get T by-value.
Then it's not just that the Drop impl assumes that cleanup has been done. It should assume that it no longer has the responsibility of cleaning up the moved-out field.

Appears worth mentioning OwningRef by @Kimundi (docs), although I suppose this serves kinda opposite role.

@rphmeier Yes, that is how it should be, imo.

  Obviously people think it's valuable, but no one is doing a good job saying why.  I took the discussion to IRC and no one could explain it.  Additionally, at least one person (can't remember who ATM) echoed my sentiment.
So my question is: are new people to the language supposed to learn this?  It feels like the answer is yes, but perhaps it's like C++ rvalue references where it only matters for library developers.  If people like me who are just doing day-to-day Rust are supposed to learn it, then I ask this: if we can't explain it now, why do we think we can explain it later?
I'm new to Rust, but am not new to programming.  Specifically, I'm at the point where I believe I have a complete mental model of current Rust but not to the point where that applies simply.  I no longer seem to misunderstand what the borrow checker is telling me.  It would be accurate to say that I am an advanced C++ developer.  If this is a feature I am supposed to be using in future with any frequency (and it seems to be) and no one can explain it to someone at my level, then consider me against it.

@camlorn You could argue that you could have a *mut Self -> *mut Self::Target DerefMove which also works in conjuction with functions receiving DSTs by value.
That's the minimal requirement, I suppose.
But from there, you can try to add to it, to make it safe.
My experiments in API design led me to this contraption which is pretty close to maximally flexible for my usecases, but not as advanced as @Ericson2314's ideas.
At the same time, I have no idea how to add a surface syntax to that model.
I'm really not sure what we should do here. Hence why this is a RFC.
Should we just start with a very unsafe DerefMove that works with DSTs?

@camlorn To get some intuition, you can think of it like this. (Warning: intuition. Not deeply technical and precise explanation.) The borrow checker is fundamentally concerned with four things: (a) can I read from this? (b) can I write to this? (c) can I move out of this -- deinitialize it? (d) do I need to initialize this -- move into it? When working with function-local variables (and struct fields, etc.) directly, all four of these are available. You can move out of variables, you can re-initialize them, you can read and write, all as long as the borrow checker can verify that there aren't any conflicts. Indirectly, though -- working through a reference -- only two of those are available: read (&) and write (&mut). This is filling in the hole of being able to pass the right to move out of a variable (or field, etc.) around indirectly via a reference. This matters because there are some things you can only do through references, such as working with unsized types (array slices and trait objects -- like Box<FnOnce>).

@eddyb
I'm not sure what you're trying to tell me.  My issue here is that no one is sufficiently explaining the problem or why I want this tool in my toolbox, not that I have a problem with the proposed solution or think we should do it another way.
@glaebhoerl
Thanks.  I think I follow what it technically does now, though I'm not quite sure I see the use of it.  Still,  my principle objection is the difficulty in explaining it and the lack of good motivation in the RFC from my perspective, and you've shown that this is unwarranted.  I will now step back and allow the smart people who understand the problems debate under the assumption that eventually someone will write a very good example showing the why of it  for the book if it's approved.
I've been following the RFCs because it's fun for like 6 months now and this is the first one where I outright couldn't understand it from the RFC text.  I only had slight problems with specialization.  Consequently I thought it was probably worth being insistent on the this needs explanation part of the discussion.

@camlorn I do admit that some of us have internalized "we need an owning reference to a borrowed place" for years and the first need I can think of is "being able to move Box to a library while keeping its ability to move out of a dereference, and even extend that to work with unsized values", which is probably a bit obscure.
The second that comes to mind, and which probably affects more code in the wild, is having a common interface over by-value iteration of [T; N], Box<[T]> and Vec<T> - that requires a function that takes ownership over the elements of the array/slice/vector ("owned [T]") and returns an iterator (ByValueIter).
But you also have to keep track of the lifetime so that the iterator doesn't outlive the place where you got the owned slice from, so you end up with &'a move [T] -> ByValueIter<'a>.
Now if we had generics over constant integers, you could argue that you don't need of this ownership manipulation, you can just make one by-value iterator for each of those 3 categories of types ([T; N], Box<[T]> and Vec<T>), and have it own the whole allocation.
There are several options in this area of development of Rust's semantics (including my Own contraption) and none of them truly better than all others, is what I'm trying to say here.
It's not clear that the language could benefit from this feature, unless maybe reduced to bare minimums.
It can be hard to explain why you may "want this tool in your toolbox" if it's not clear that you do.

@camlorn #965 has some more examples which may help too.

#965 does help. Everyone else having my problem should go there too.  Thanks.  I'm starting to see the why of it, but I think I need to see it in practice before I fully grasp the motivation.
One interesting point of the linked issue is that this helps avoid unnecessary copies.  Is this still a possible use?  If so, I want to go elsewhere and start discussing compiler optimizations with a possible eye to working on this one.

@camlorn thanks again for bringing this issue up. I've agreed 100% but hadn't had the time to leave a  comment saying so.
The Ember RFC process has a "how do we teach this" section, which is specifically designed to make sure that this kind of thing is considered, I really like it.

@camlorn yes indeed, the extra copying / allocation is still a problem.
Also, for one more very convoluted example, in https://github.com/RustOS-Fork-Holding-Ground/libfringe/blob/params/src/context.rs#L27 and https://github.com/RustOS-Fork-Holding-Ground/libfringe/blob/params/src/context.rs#L57 I have a thread context (only for suspended threads) "owning" (with Unique) the allocated stack, and the closure used to spawn the thread receiving a &mut to that stack. This wrong the thread should own itself when alive, and Unique is of course unsafe (and has no lifetime).
With &move I can give the closure real ownership of the new thread, and when the thread exits, I can safely move the &move into the suspended thread context to show it now owns it. (when the thread suspends not exits, I'd still need to transmute &mut -> &move because the thread assumes it will always wake up even if it is killed and doesn't think its given up ownership.)
Also, the init closure, a DST, is written to the allocated stack not boxed separately, so the FnOnce + &move stuff already mentioned would be nice.

@Ericson2314 pls don't use transmute to turn &mut into &move: unsafe { &move *(ptr as *mut _ as *move _)

@ubsan Well yeah I do a variation on yours for Unique today https://github.com/RustOS-Fork-Holding-Ground/libfringe/blob/params/src/context.rs#L108, and would do that for &move tomorrow.

Ah much simpler: https://github.com/RustOS-Fork-Holding-Ground/libfringe/blob/params/src/fat_args.rs#L34-L36
After passing a pointer to args for the incoming thread, the incoming thread is responsible for moving them out not the outgoing thread. If I had &move I wouldn't need the forget and *mut cast.

One interesting point of the linked issue is that this helps avoid unnecessary copies. Is this still a possible use?

It certainly is. To fix a stack overflow in one unit test which involves moving large FnOnce closures I had to use pointers/memcpy/forget because LLVM was unable to elide the copies between (inlined!) function calls. &move would solve this.

@Ericson2314 Thanks :), I'm not sure why I even said anything.

@mahkoh and anyone else who can answer:
Going briefly off topic in the interest of saving time, is there any theoretical reason the compiler can't optimize this before LLVM and where should I take the discussion (besides setting up to build Rust on my machine)?  I have some vague, ten-thousand-foot view ideas as to how to do it. Sort of.  And it could be interesting.

@ubsan well I should of said "cast" rather than "transmute", no use being imprecise when I'm choosing the longer, more obscure, word :)!

Obviously this request has a long history (it's also, to some extent, tied in with wanting to move DST by value). We discussed this some in the @rust-lang/lang meeting but before I write that comment I wanted to just give my personal position at the moment. (I've moved around from time to time.)
First, as a generate note, I think that using & here is not the right approach. All &T types indicate that the T is borrowed, but &move T would indicate that the T is owned. Besides the learning hurdle ("wait...so it's borrowed...but it's owned..."), this semantic mismatch also shows up elsewhere. For example, you can "reborrow" both &T and &mut T references: but it wouldn't make sense to permit "reborrowing" an &move reference. If fact, a &move expression isn't a borrow at all: logically, it's a move, even though the memory stays put. Moreover, I think we should be extremely careful with adding built-in syntax. There is no easier way to make the language feel complex. This is especially true for a pointer type whose uses are relatively advanced and limited.
The points raised in the previous paragraph suggest that using a type like Moved<'a, T> would be a better choice. Still, whenever I've started to dig into this design, I wind up feeling that something just like Moved<'a, T> is likely not enough. The premise of such a type is that there is a memory slot that contains a T, and you own the T but not the slot -- so presumably there is some outer context that is going to cleanup the slot. For a stack variable, that's easy enough; if we wanted to use Moved<'a, T> to handle DerefMove or something though, we'd need a more general cleanup mechanism. This starts to sound like a ShallowDrop trait, and hence a little like the placer protocol -- it's not obvious to me that this shallow drop trait would be safe, for one thing.
And of course you can go further. If you're trying to work with data in the heap, maybe you don't need the lifetime, for example, you just need a way to free the pointer when you're done. This more or less brings you to eddyb's Own designs, which make things more flexible still. The key idea there is that you would not force a lifetime like 'a, but rather a type that indicates where the value resides, and which can thus encapsulate any cleanup required. So you might have Own<T, Stack<'a>> (no cleanup requires, but cannot escape 'a), or Own<T, Heap> (memory owned by a Box in the heap; when done, we must shallow free this box). This is a very powerful design, but also very complex, and at this point is where I start to get worried about the ergonomics and learnability. After all, we're tinkering here with Rust's "crown jewels" -- ownership and borrowing! -- so it's really important that we tread carefully and think things through deeply.
It would be good, I think, to start slower, by assembling the use cases we aim to address. How many of them are concerned with unsafe code aiming to make maximum efficiency? How many of them are things we expect users to use on a common basis? Where is compiler support needed? And so forth.
I've definitely felt the need for features like this (e.g., I think Rayon could benefit in some places). But I also don't see this as the most pressing problem that Rust faces right now, personally. I'd like to accumulate some more examples and take our time to think on it.

OK, now for the more official comment. :) So yes we talked about this in the @rust-lang/lang meeting a bit. The general consensus was that we'd be inclined to postpone this RFC under the (existing) issue, #998, to be revisited at some later point, when more "design bandwidth" is available. Therefore, I'm going to close this RFC. Thanks @thepowersgang!

(Not sure if I should respond here or #998)
@nikomatsakis It is a big extension right at the core of Rust, but I think that's precisely why I believe new syntax is warranted. Moreover, I see big overlap between this (including out pointers), fixing Drop (c.f. eyepatch, by-value), and, perhaps, non-lexical lifetimes too, and I think together these things do constitute a big priority for the language's evolution.

I still haven't taken the time to fully understand @eddyb's proposal, but I am suspicious because it looks like unsafe encapsulation tricks (safe interface, unsafe impl) when features I mention above are about making impl and interface safe---so its apples and oranges basically.

As to ergonomics/learnability my thoughts are: Yes were making the language more complex, but we're also making it more regular. Because nobody is proposing breaking backwards compat, "beginner's Rust" will stay the same. But I think the trade-off of simplicity for regularity will actually benefit programmers moving to the intermediate/advanced stage---like the point where you realize moving out of Box is hard to write yourself but haven't learned its actually impossible to write yourself is really confusing.

For example, you can "reborrow" both &T and &mut T references: but it wouldn't make sense to permit "reborrowing" an &move reference.

Actually it does. One is left with scratch pointer which, when written to, yields another &move.

In fact, with moving temporarily out of &mut allowed, it to would indicate "borrowing the location, owning its contents". In other places I mentioned there are 4 possible non-aliased mut reference types:

(init at beginning of lifetime, uninit at beginning lifetime) ⨯ (init at end of lifetime, uninit at end of lifetime)

All of these fill a "borrowing the location, owning its contents" role, just with the above restrictions on the initialization status.

This is especially true for a pointer type whose uses are relatively advanced and limited.

Kinda already said this, but I think it's important to take in account "transitive usage". The code that can be made safe with this is together used ubiquitously, even by beginners.

@Ericson2314

(Not sure if I should respond here or #998)

Yeah, I don't know :) probably #998 is better, since I think the primary question here is not details of this RFC, but rather how to prioritize this change. To be honest, I'd personally prefer to discuss on internals.rust-lang.org, since I think GH doesn't manage sprawling comment threads very well (and throw a link onto #998).

Actually it does [make sense to reborrow an &move]. One is left with scratch pointer which, when written to, yields another &move.

Hmm. I suppose that's true. It's not how borrowing of other things works, but it could be fit in I imagine. Interesting! (It's still rather different from other borows, in that it leaves the value in a distinct state from how it began, but I agree you could make it fit better than I thought initially.)

Moreover, I see big overlap between this (including out pointers), fixing Drop (c.f. eyepatch, by-value), and, perhaps, non-lexical lifetimes too, and I think together these things do constitute a big priority for the language's evolution.

I'm not sure that &move would solve all of those distinct issues :), though certainly it is a better fit for "by-value Drop" than by-value (which is, in fact, a bad fit, hence the reason we chose against adopting it, long ago). But I still feel like it's premature to be adopting an RFC at the moment: we need to get a better handle on the use cases we're aiming at with this design, as well as wrestling some with the mental model that would result.

So, I've been thinking a lot about this RFC over the last few weeks, and I've decided that I was wrong to close it. I didn't intend to "shut down" conversation on the topic, but I think that this is how it came across. I do have lots of doubts about introducing &move -- not the core underlying concept, but the actual syntax -- but I think I should have just tried to air those concerns here, rather than trying to move the conversation elsewhere (e.g. an issue). (Separately, I have some frustrations that we need a better way to coalesce conversations that occur across different forums. But anyway.) So I'm sorry about that.
Anyway, I decided to post this comment last night and at that time I had intended to re-open the RFC (or ask for suggestions about where to move the conversation to). But when I awoke this morning I noticed that in the meantime RFC 1646 has been opened, so perhaps I'll just comment there? (Thoughts?)

This RFC was pretty badly specified, and to be honest the new one is far better :) (I was feeling a little unwell when I wrote this). @nikomatsakis  Comments on the syntax should probably be raised on the new RFC pull (maybe as a summary of IRC/forum discussion?)


How does the routing class load modules

How does the routing class load modules

What do you mean by modules?
On Fri, 12 Apr 2019, 09:37 Liunk<EMAIL_ADDRESS>wrote:

How does the routing class load modules
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/thephpleague/route/issues/241, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAP3E2ISVNDHUATUUCJAP5DPQBC2ZANCNFSM4HFOOTRA
.


Xata Vector Search implementation is not working

I tried everything and everytime it returned an empty array. The only way to make it work was using my own implementation:
import { OpenAIEmbeddings } from '@langchain/openai'
import { getXataClient } from './xata.js'

const embeddings = new OpenAIEmbeddings({
    batchSize: 1536,
})

const xata = getXataClient()

const embedding = await embeddings.embedQuery('Hello World')
const records = await xata.db.vectors.vectorSearch('embedding', embedding)

for (const record of records) {
    console.log(record.content, record.xata)
}

CC @tsg would you be ok with having a look?

I tried everything and everytime it returned an empty array.

I will double check with the integration tests that it is still running, but one thing to keep in mind is that if you add a vector and then immediately search for it, it might not be indexed yet because the indexing happens async.
The integration tests do something like this to workaround: https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-community/src/vectorstores/tests/xata.int.test.ts#L98-L101
Could it be that this is the issue?
Thanks for the ping @jacoblee93 .

I confirmed that the integration tests still work, for reference I ran (after unskipping):
OPENAI_API_KEY=sk-.... XATA_API_KEY='xau_....' XATA_DB_URL='https://....eu-west-1.xata.sh/db/langchaintest' yarn test:single './src/vectorstores/tests/xata.int.test.ts'

@rodjosh let me know if I can help in any way, you can also find me in the Xata discord if you need help troubleshooting.


Replacing Make/Kbuild with CMake

This PR implements RFC: Replacing Make/Kbuild with CMake
The Gerrit feedback has been addressed, but the work is not complete. It is uploaded in an incomplete form to enable collaboration and early feedback.
Work In Progress
Blocker TODOs
TODOs that must be fixed before the PR is merged.

[x] Remove KBuild
[x] Update SDK to 0.9.2 when it is ready
[x] Support Device tree
[x] Support generating isr_tables
[x] Support generating offsets.c
[x] Integrate with sanitycheck.

[ ] Verify that all samples are similar enough[0] before and after CMake

[x] Integrate with Zephyr CI.
[ ] Update docs.

[x] getting_started

[x] getting_started.rst
[x] linux.rst
[x] win.rst
[x] mac.rst

[ ] Update build instructions throughout docs using 'make` (~140 uses)
[x] Document how to develop CMakeLists.txt code

[x] Check if all Make CLI arguments are supported by an equivalent CMake feature

[x] Support the equivalent of make help
[x] Integrate with qemu
[x] Integrate with openocd
[x] Support debugserver
[x] Support debug
[x] Support flash
[x] Support ram_report
[x] Support rom_report
[x] Support W=
[x] Support htmldocs

[x] Support configuring the build for code size, speed, unoptimzed, etc.
[x] Port kernel
[x] Port arch

[x] arm
[x] arc
[x] nios2
[x] riscv32
[x] x86
[x] xtensa

[x] Port boards

[x] arm
[x] x86
[x] xtensa
[x] riscv32
[x] arc
[x] nios2

[x] Port ext
[x] Port subsys
[x] Port samples
[x] Port tests
[x] Have CMake recursively call CMake to build host tools instead of
requiring the user to first invoke a seperate CMake build system to
build the host tools. Host tools will be provided in the SDK, non-SDK users will have to manually build Kconfig and add it to path.
[x] Support default BOARD's in application CMakeLists.txt code.[1]
[x] Fix the CLI usability problem: -DCMAKE_TOOLCHAIN_FILE=$ZEPHYR_BASE/cmake/toolchain.cmake
[x] Test with

[x] Linux,
[x] OSX,
[x] Windows + MinGW

[x] Test with

[x] Make

[ ] Port all supported toolchains

[x] zephyr
[x] gccarmemb
[x] issm
[x] espressif
[x] riscv32 (The riscv32 toolchain is deprecated)
[ ] xcc (Requires RHEL 4 or 5)
[x] xtools

Non-blocker TODOs
TODOs that must be fixed, but may be delayed until after the PR is merged.

[ ] Support building Zephyr on Windows natively (without MinGW)
[ ] Test with Ninja
[x] Check if it is acceptable to use an older version of
CMake. Currently minumum required version is 3.7, but the latest
Ubuntu LTS ships with 3.5. Supporting an older version would make it
easier for users to get started with Zephyr.
[ ] Ensure that verbose and normal builds are logging the appropriate amount of information.
[ ] Port this business logic that is present in many applications:

ifeq ($(CONFIG_NET_L2_BT), y)
QEMU_EXTRA_FLAGS = -serial unix:/tmp/bt-server-bredr
else
include $(ZEPHYR_BASE)/samples/net/common/Makefile.ipstack
endif

Un-classified TODOs
TODOs in need of being classified as Blocker or non-blocker.

[ ] Evaluate how application CMakeLists.txt code should integrate
with Zephyr CMakeLists.txt code. Currently they interact in a manner
very similair to how KBuild worked, but even though this worked for
KBuild, it might not be the best solution for a CMake build system.
In particular, consider the "bootloader use-case" of a multi-program
build.
[x] Support KBuild's ld-option (The ability to test if the linker
supports a flag).
[x] Introduce more CMake extentions to reduce boilerplate and enforce
conventions in driver/ subsys/ ext/ CMakeLists.txt code.
[x] Integrate more Kconfig targets than menuconfig

Getting started
Install CMake (>=3.7)
Install Python 3 (>=3.4)
Optionally install Ninja, recommended for most users.
To enable Ninja add the flag -GNinja when calling cmake.
Windows
Open Windows Command Prompt:

set ZEPHYR_BASE=
set ZEPHYR_GCC_VARIANT=gccarmemb
set GCCARMEMB_TOOLCHAIN_PATH=
set BOARD=nrf52_pca10040

cd samples\bluetooth\beacon
cmake -Bbuild -H.
cmake --build build

Linux
$ source zephyr-env.sh
$ cd $ZEPHYR_BASE/scripts
$ mkdir build && cd build
$ cmake .. && make
$ export PREBUILT_HOST_TOOLS=$(pwd)
$ export BOARD=nrf52_pca10040
$ export ZEPHYR_SDK_INSTALL_DIR=
$ export ZEPHYR_GCC_VARIANT=zephyr
$ cd $ZEPHYR_BASE/samples/bluetooth/beacon
$ cmake -Bbuild -H.
$ cmake --build build
[0] Binary identical FW's before and after the CMake transition would
make verification easy. But it may not be what we really want.
[1] https://github.com/zephyrproject-rtos/zephyr/pull/1107#issuecomment-322463484

Why want to replace kconfig with cmake? I think kconfig is good and easy to use .

Of the blocker todos, everything is done AFAICT except the documentation.
Carles: samples//*.rst (except samples/net)
@jukkar: samples/net//.rst
@mbolivar: boards/**/.rst
Anything else I'm missing?

Why Install CMake (>=3.7) ? Current apt version is way lower

@Zarkopafilis I believe it's due to requiring add_custom_command() to use DEPFILE, which was introduced in CMake 3.7. The getting started guide gives you a step by step guide on installing CMake from cmake.org, and it's quite simple.

Ok. Sidenote: On windows with MSYS it works with cmake 3.6.2.... on a
docker container with an ubuntu base image it did not.
On Wed, Nov 15, 2017 at 11:47 PM, Carles Cufí<EMAIL_ADDRESS>wrote:

@Zarkopafilis https://github.com/zarkopafilis I believe it's due to
requiring add_custom_command() to use DEPFILE, which was introduced in
CMake 3.7. The getting started guide gives you a step by step guide on
installing CMake from cmake.org, and it's quite simple.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/zephyrproject-rtos/zephyr/issues/1488#issuecomment-344739340,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFL1Mo-6TJjdc7c_2uTWgb_XHCUkwBuVks5s21wKgaJpZM4PWIUg
.

--
Best Regards,
Theodoros 'Zarkopafilis' Ntakouris
+30 698 92 90 272 (Greece)

@Zarkopafilis that's because the CMake scripts detect that you're running MSYS and lower the required version. It's an exception in the case of MSYS, which doesn't support Ninja. This is temporary since we will soon switch to native CMake with Ninja and no MSYS at all on Windows.


TaxonWorks download importing incorrectly into ChecklistBank

The latest version of the TaxonWorks Global Lepidoptera Index dataset gets fragmented into many siblings of Lepidoptera on import into CLB. It's currently uploaded as a private test dataset here:
https://www.checklistbank.org/dataset/60320/classification
The data displays correctly in TW. I initially assumed that recent changes had corrupted their COLDP exports, but I generated the following tree directly from the export via parent relationships, so I believe the issue is in CLB.
GLITree.txt

When I look at the subgenus Acontia (Acontarache) Berio, 1977 it claims to have an invalid parentID 267837.

That ID does exist on Taxon.txt, but is not interpreted as it claims the nameID 23340 is invalid:
https://www.checklistbank.org/dataset/60320/verbatim/589108

This seems to be true, there is no such name record in the archive.
There are lots of records that span over several lines in the Name.txt. A TSV file does not allow that and the line breaking reference strings are likely the cause. Example from line 52426:
20235	20235	Ctenognophos	Prout, 1915	genus	Ctenognophos					3300	384	1915	true	ICZN	http://purl.obolibrary.org/obo/NOMEN_0000224		"233370.00|ORIGINAL GENUS UNDETERMINED|Valid name|7554.00|Geometroidea-Geometridae-Ennominae-181A-202602|Q:\Geometroidea\Geometridae\Ennominae\181A\|FC-Geometroidea-Geometridae-Ennominae-181A-202602.JPG|BC-Geometroidea-Geometridae-Ennominae-181A-202602.JPG|Fletcher (1979) stated:-
The date of publication of the part of Seitz containing page 384 in vol. 4 is stated on page 377, in both the German and English editions, to be 9.ix.1915.

The part of Seitz containing page 459 of vol. 4 (Suppl.) has appeared only in the German edition; its date of publication is stated on page 457 to be 16.x.1941. Ctenognophos was originally proposed as a subgenus of Gnophos Treitschke, 1825.
Ctenognophos was originally proposed as a subgenus of Gnophos Treitscke, 1825.

Parsons et al. (1999) included 17 species and 8 subspecies (inc. nominates) in the genus CTENOGNOPHOS.|13334|19371|Gnophos|eolaria|Guenée|1857|[INDIA]|Nord de l'Inde [northern India]|Syntypes - 2 males|(BMNH, London)|19553|by subsequent designation"	2017-07-16T21:55:16Z	Ian Kitching
20236	20236	Bizia	Walker, 1860	genus	Bizia					6336	261	1860	true	ICZN	http://purl.obolibrary.org/obo/NOMEN_0000224		233371.00|ORIGINAL GENUS UNDETERMINED|Valid name|3938.00|Geometroidea-Geometridae-Ennominae-181A-202603|Q:\Geometroidea\Geometridae\Ennominae\181A\|FC-Geometroidea-Geometridae-Ennominae-181A-202603.JPG|BC-Geometroidea-Geometridae-Ennominae-181A-202603.JPG|Parsons et al. (1999) included 2 species in the genus BIZIA.|25134|25133|Bizia|aexaria|Walker|1860|CHINA (NORTH)|Locality unknown|Holotype - male|(BMNH, London)|by monotypy	2017-07-16T21:55:16Z	Ian Kitching

@gdower a sidenote: It would be good not to use .tsv or .txt as the file suffix when tabs are used as the delimiter. CSV should have a comma as the delimiter: https://github.com/CatalogueOfLife/coldp/blob/master/README.md#data-files

Closing as this is a data formatting issue, not a backend one.

@dhobern @gdower you can spot skipped records due to bad numbers of columns or empty rows through the previous line skipped issue.

Note that the RFC 4180 CSV format allows for multiline content when quoted properly.

https://github.com/CatalogueOfLife/coldp/blob/master/README.md#quoting-and-escaping

Thanks Markus. @gdower - should I raise a TW issue?

@mdoering I've just realised why I wasn't seeing the problem in my code. I use Pandas in my Python library and it handles stray line breaks on import. This means I should be able to write a really trivial processor to fix TW exports until this gets addressed there.

I'm also going to try to adjust the TW exporter to sanitize the line breaks (and change the file extension from csv to tsv), but it would be more user friendly if CLB can handle delimiter issues.

@dhobern, if you configure your datasets like this it does fix the issue as well:

Joining the lines fixes the problem. @mdoering Many CSV libraries handle this kind of thing automatically. Should CLB do so too?
If a row includes fewer tabs/commas than the header row, try reading and joining the next line. That was all that was needed here.

I was wondering about that, probably good to have in place. On the other hand the text files are wrongly encoded and it might be good to know that. I guess we can still flag those records once we have successfully read them. I will look into this. Are you aware of a java library that does that? CLB uses the univocity.parsers.

Sorry, no - the last time I did something like this in Java, I wrote my own class that really only reflects what I needed at the time:
https://github.com/dhobern/utils/blob/master/src/main/java/io/github/dhobern/utils/CSVReader.java

Looking at TW multiline data I see it is actually properly quoted. The problem is that this is a CSV feature only and instead of the filename suggests it is a tab delimited TSV instead which should escape new lines and carriage returns with \n and \r.

I have implemented support for stray away multilines as long as they end up being exactly the same number of columns as the header tells. It would not remove the TW quotes in TSV files though.

I did modify the TW exporter to strip \t, \r\n and \n in remarks, replacing them with spaces. I changed the file extension to tsv. I was also able to import @dhobern's export into dev with the csv options configured to have delimiter=\t, escape=", and enclosure=" (as above) but that extra dataset configuration should be unnecessary once the csv code gets deployed in TW.

Good! But should these chars not better be escaped or replaced in every field, not just remarks?

Because of validation on the other fields, I don't think they will have tabs or line breaks. Tabs and linebreaks are allowed in notes and that was the field that was breaking Donald's export.


Added prompt text to make the generated code responsive

Generated code is not always responsive, so added a prompt text that tells the llm to make sure the code is responsive and preview works correctly.

Unfortunately not accepting prompt changes at the moment without some serious evals on why the change is better. Feel free to use this prompt on your local repo.


Wrong envelope calculated for ReliefFeature

Set up a new 3DCityDB 4.0 instance with SRID = 25832
Import this dataset: http://schemas.opengis.net/citygml/examples/2.0/building/Building_LOD2-EPSG25832.gml
Recreate all BBOXes with core:_CityObject as top-level feature
Export as CityGML dataset

In the exported CityGML dataset, the BBOX of the ReliefFeature will be as follows, which is wrong:
<gml:Envelope srsName="urn:ogc:def:crs:EPSG::25832" srsDimension="3">
  <gml:lowerCorner>458880.0 5438352.6 112.0</gml:lowerCorner>
  <gml:upperCorner>458880.0 5438352.6 112.0</gml:upperCorner>
</gml:Envelope>

If you repeat the above steps, but use a feature class filter in step 2 to just import the ReliefFeature without the Building, then the BBOX will be correctly calculated as follows:
<gml:Envelope srsName="urn:ogc:def:crs:EPSG::25832" srsDimension="3">
  <gml:lowerCorner>458868.0 5438343.0 112.0</gml:lowerCorner>
  <gml:upperCorner>458892.0 5438362.0 114.0</gml:upperCorner>
</gml:Envelope>

Many thanks for the bug report. Fixed in https://github.com/3dcitydb/3dcitydb/commit/0c762eea33b85544c017283a3d9f984d610f8d1f

:thumbsup:


directory structure for multiple slide sets

I'm just starting out with markdeep-slides, but am already very excited!
I'd like to have a bunch of slide sets, one for each chapter in my 'book'.
They all have accompanying files, so each chapter should be its own subdirectory.
With the current setup in the repo that means I need to copy the markdeep-slides folder into each of the chapter folders. That's a pain to maintain if there are changes to the folder and is a lot of wasted space. Instead, I'd like to move the markdeep-slides folder one up in the directory structure so it becomes a sibling of Chapter1, Chapter2, Chapter3, etc.
It seemed pretty straightforward. I simply edited the final section to prepend the paths with '../'. And it mostly works, but somehow the fonts are not the correct ones and it's not obvious why. So this issue is to get assistance in fixing the fonts. Here is my final demo.md.html section:
<!-- Markdeep slides stuff -->
<script>
    markdeepSlidesOptions = {
        aspectRatio: 16 / 9,
        theme: 'simple',
        fontSize: 28,
        diagramZoom: 1.0,
        totalSlideNumber: false,
        progressBar: true,
        breakOnHeadings: false,
        slideChangeHook: (oldSlide, newSlide) => {},
        modeChangeHook: (newMode) => {}
    };
</script>
<link rel="stylesheet" href="../markdeep-slides/lib/markdeep-relative-sizes/1.11/relativize.css">
<link rel="stylesheet" href="../markdeep-slides/markdeep-slides.css">
<script src="../markdeep-slides/markdeep-slides.js"></script>

<!-- Markdeep stuff -->
<script>
    markdeepOptions = {
        tocStyle: 'none',
        detectMath: false,
        onLoad: function() {
            initSlides();
        }
    };
</script>
<style class="fallback">body{visibility:hidden;white-space:pre;font-family:monospace}</style>
<script src="../markdeep-slides/lib/markdeep/1.11/markdeep.min.js" charset="utf-8"></script>
<script>window.alreadyProcessedMarkdeep||(document.body.style.visibility="visible")</script>

That's certainly an interesting question!
I first suspected that this behavior was due to Same-origin policy, but it isn't – instead, it's because how theme loading works:

In the markdeepSlidesOptions variable, you've left the theme option set to its default value 'simple'.
As a result, markdeep-slides tries to load "markdeep-slides/themes/" + options.theme + ".css" (see line 36 of markdeep-slides.js) – which doesn't exist because you've moved the markdeep-slides directory one level up.
The fix is to specify the full path of the stylesheet: theme: '../markdeep-slides/themes/simple.css',. That should manage to load the theme (and the fonts, as well).


TypeError: DataCollatorForSeq2Seq got an unexpected keyword argument 'noise_type'

Hello, when I run run_summarization.py, I get the following error:
Traceback (most recent call last):
File "run_summarization.py", line 702, in
main()
File "run_summarization.py", line 538, in main
data_collator = DataCollatorForSeq2Seq(
TypeError: init() got an unexpected keyword argument 'noise_type'
noise_type is randomutterance2, but where is it?

Hello, when I run run_summarization.py, I get the following error: Traceback (most recent call last): File "run_summarization.py", line 702, in main() File "run_summarization.py", line 538, in main data_collator = DataCollatorForSeq2Seq( TypeError: init() got an unexpected keyword argument 'noise_type'
noise_type is randomutterance2, but where is it?

The code don't update noise and data augment, you need to add it.

I am facing the same issue, were you able to fix it? I didn't get the answer provided here, what is the code to add noise, could you give the code for reproducibility, or point us to a source

I am facing the same issue, were you able to fix it? I didn't get the answer provided here, what is the code to add noise, could you give the code for reproducibility, or point us to a source

The author don't update the code about the part of data augment. Now you can understand that the code only finish the part of semi-training. The 'noise_type' is useless in this repo.

I am facing the same issue, were you able to fix it? I didn't get the answer provided here, what is the code to add noise, could you give the code for reproducibility, or point us to a source

The author don't update the code about the part of data augment. Now you can understand that the code only finish the part of semi-training. The 'noise_type' is useless in this repo.

Were you able to reproduce the results from the paper?

I am facing the same issue, were you able to fix it? I didn't get the answer provided here, what is the code to add noise, could you give the code for reproducibility, or point us to a source

The author don't update the code about the part of data augment. Now you can understand that the code only finish the part of semi-training. The 'noise_type' is useless in this repo.

Were you able to reproduce the results from the paper?

Because I use different data-augment methods and dataset. I only tell you the 2-stage semi-training is efficient.

I am facing the same issue, were you able to fix it? I didn't get the answer provided here, what is the code to add noise, could you give the code for reproducibility, or point us to a source

The author don't update the code about the part of data augment. Now you can understand that the code only finish the part of semi-training. The 'noise_type' is useless in this repo.

Were you able to reproduce the results from the paper?

Because I use different data-augment methods and dataset. I only tell you the 2-stage semi-training is efficient.

Do you have a repository that you can point me to for the data augmentation code, that would be really helpful

I am facing the same issue, were you able to fix it? I didn't get the answer provided here, what is the code to add noise, could you give the code for reproducibility, or point us to a source

The author don't update the code about the part of data augment. Now you can understand that the code only finish the part of semi-training. The 'noise_type' is useless in this repo.

Were you able to reproduce the results from the paper?

Because I use different data-augment methods and dataset. I only tell you the 2-stage semi-training is efficient.

Do you have a repository that you can point me to for the data augmentation code, that would be really helpful

https://github.com/makcedward/nlpaug
I use this repo, I hope it will solve your problem.


[usecase-wrt] Add test for Custom URL Scheme

Impacted tests(approved): new 1, update 0, delete 0
Unit test platform: Crosswalk for Android 20.49.527.0
Unit test result summary: pass 1, fail 0, block 0
BUG=https://crosswalk-project.org/jira/browse/CTS-609

@yhe39 PTAL

LGTM


Refactor Data Loader for Alternate Pricing API Use

Right now loader.py has DataReader(stock, 'yahoo', start, end) sprinkled in it, but it would be great to  easily pass a dataloader function to fetch pricing from other APIs. _load_raw_yahoo_data becomes _load_raw_data with an additional source kwarg for passing in a loading function that returns pricing data formatted just like yahoo api data. I've had to implement this already for my own use.

@cyniphile can you provide an example of another data source you'd like to be able to load from via a reader?

Any EOD pricing API that is not yahoo; it could be Bloomberg, Barchart, Xignite.

You also have CSI and metastock as well.
On Tue, 8 Dec 2015, 13:39 Luke Schiefelbein<EMAIL_ADDRESS>wrote:

Any EOD pricing API that is not yahoo; it could be Bloomberg, Barchart,
Xignite.
—
Reply to this email directly or view it on GitHub
https://github.com/quantopian/zipline/issues/889#issuecomment-162740521.

There is new API for ingesting data for use in zipline. For information about how to use this api or add your own data, see: http://www.zipline.io/bundles.html


Research: High Availability of Keptn

Action Item
Currently, we cannot deploy Keptn to be resilient against pod or node crashes. As a first step, we should scale up the replica count of the keptn core services.
Acceptance Criteria
This research should verify that:

[ ] Only one replica process a message (avoiding duplicate) - which should be guaranteed by the NATs queue groups
[ ] How to do that for the shipyard-controller and what changes are required
[ ] Resource quotas impact - what services can we merge? A good candidate is the ligth-house service. Mind that, backward compatibility must be a priority.
[ ] What do we need to change in the API gateway Nginx configuration? E.g., load balancing among different replicas, sticky sessions, etc

DoD

[x] Create a POC with deployments that have multiple replicas
[ ] Create a POC for the shipyard-controller to run with multiple replicas
[ ] Create a POC for merging services together
[ ] Do manual testing (and run the integration tests)

Notes

MongoDB is out of scope
Helm- and Jmeter-service are out of scope

Notes from the meeting:

Api Gateway
-Core service:
-- shipyard controller
-- resource service
Api Service:
-- secret service
-- api service
Default Usecase service:
-- approval service
-- lighthouse service
-- remediation service
Webhook Service
MongoDB Datastore
MongoDB
NATS

picking the "Api Service"

Approval service is now merged with remediation service, ligthouse configuration handler is also merged,  the other two handlers have dependancies on https://github.com/keptn/enhancement-proposals/pull/56
PoC

Api-service update:
Api-service and secret-service are merged, manually tested and working. Secret service was implemented using gin-gonic, but api-service not, therefore there was a need to reimplement it using the mentioned framework. The reimplementation is not completely done in terms of clean code (no managers for handlers) and without unit tests. Also there was a need to adapt charts (keptn-endpoints.conf) and roleBindings, as currently we have only 1 service instead of 2. Poc implementation available here:
https://github.com/keptn/keptn/pull/6838

Notes on api- and secret-service. I would not go ahead with merging these two because:

semantically serve a very different purpose
security concerns:

auth and secrets are in the same place
we are handling secrets in the entry-point for the remote exec plane
we would need to give a very open RBAC to the entry-point for the remote exec plane


Uncaught ReferenceError: SocialShareButton is not defined at HTMLAnchorElement.onclick

Hello,
I have installed the gem ok and get the social share buttons to show, but when I click (in dev environment) nothing happens…
I get the following error in the console -
Uncaught ReferenceError: SocialShareButton is not defined at HTMLAnchorElement.onclick (VM7889 62:777) onclick @ VM7889 62:777
Any help appreciated!

Add this to your application.js
//= require social-share-button

what if i am using webpack

i am using webpack too.
what should i do?

Having same problem. Anyone know?

Also using Webpack on Rails 6. any solutions yet?

Thanks @hopewise but should have specified I am also using Webpack :)

@gareth326771 Any solution for this issue if we are using webpack?

@gareth326771 Any solution for this issue if we are using webpack?

I have the same issue, using webpack as well.

I too have the same issue using webpacker

Same :(

Hi guys I put together a blog about this,  not the greatest blog, but definitely solves the issue and references where I got the information,
Check it out.
https://sapphiregemtech.com/blogs/2

@DyllanCullingworth you saved my life!!!

Any official solution to use it with rails 6? @DyllanCullingworth website is down.

Still no solution
@maariavld, could you explain what you did to make it work?

@gabrielgomeso I ended up not using it in the end and I forgot how I made it work, but it was with an article written by @DyllanCullingworth.
I ended up using this, as my use case was quite simple, to share online a "premade" post:
<a href="https://twitter.com/intent/tweet?url=<your_link>&text=<your_text>"  onclick="javascript:window.open(this.href,'', 'menubar=no,toolbar=no,resizable=yes,scrollbars=yes,height=240,width=600');return false;" class= "your_css_class">
<a href="https://www.facebook.com/sharer/sharer.php?u=<your_link>" onclick="javascript:window.open(this.href,'', 'menubar=no,toolbar=no,resizable=yes,scrollbars=yes,height=240,width=600');return false;" class="<your_css_class>">

A workaround is to copy then content of this file in a javascript/vendor/social-share-button.js file and then import this file in javascript/application.js.


Both single GPU and multi GPUs give same error: "AttributeError: module 'torch.cuda' has no attribute '_UntypedStorage'"

I'm trying to train Yolact model with my own custom COCO dataset. I wanted to increase batch_size so I tried multiple GPUs and followed your instructions. I took this error and I couldn't find any solution. After that I tried with single GPU because I thought this related with multiple GPUs but I took same error again.
P.S.: I used this model before on both Windows and Ubuntu. I couldn't train for this time
There is all error
File "./train.py", line 504, in
train()
File "./train.py", line 270, in train
for datum in data_loader:
File "path/python3.7/site-packages/torch/utils/data/dataloader.py", line 652, in next
data = self._next_data()
File "path/python3.7/site-packages/torch/utils/data/dataloader.py", line 1330, in _next_data
idx, data = self._get_data()
File "path/python3.7/site-packages/torch/utils/data/dataloader.py", line 1296, in _get_data
success, data = self._try_get_data()
File "/pathpython3.7/site-packages/torch/utils/data/dataloader.py", line 1134, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "path/python3.7/multiprocessing/queues.py", line 113, in get
return _ForkingPickler.loads(res)
File "path/python3.7/site-packages/torch/multiprocessing/reductions.py", line 303, in rebuild_storage_fd
shared_cache[fd_id(fd)] = StorageWeakRef(storage)
File "path/python3.7/site-packages/torch/multiprocessing/reductions.py", line 65, in setitem
self.free_dead_references()
File "path/python3.7/site-packages/torch/multiprocessing/reductions.py", line 70, in free_dead_references
if storage_ref.expired():
File "path/python3.7/site-packages/torch/multiprocessing/reductions.py", line 35, in expired
return torch.Storage._expired(self.cdata)  # type: ignore[attr-defined]
File "path/python3.7/site-packages/torch/storage.py", line 753, in _expired
return eval(cls.module)._UntypedStorage._expired(*args, **kwargs)
AttributeError: module 'torch.cuda' has no attribute '_UntypedStorage'

I have found a solution. I guess this happens because of num_workers. I've set num_worker to 0 and it still running! I hope this solution helps you too.

I met this problem too. It may be caused by the version of Pytorch. I find it works on 1.10.0 but not on 1.12.1

Maybe the main problem is pytorch itself, pytorch1.12.0 have the problem. We can try updating it to 1.13.0.
https://github.com/pytorch/pytorch/issues/88839


Reverse order of legend for taxa_abundance_bar_plot

Hi @sbslee
Are you able to advise how I can reverse the order of the legend of a taxa_abundance_bar_plot?
I've tried using:
handles, labels = ax2.get_legend_handles_labels()
ax2.legend(handles[::-1], labels[::-1])
But this results in a missing legend.
Thanks!

@mishkb,
As you can see below, your approach is correct and should work:
import dokdo
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
sns.set()

qzv_file = '/Users/sbslee/Desktop/dokdo/data/moving-pictures-tutorial/taxa-bar-plots.qzv'

fig, [ax1, ax2] = plt.subplots(1, 2, figsize=(20, 10))

dokdo.taxa_abundance_bar_plot(
    qzv_file,
    figsize=(10, 7),
    level=6,
    count=8,
    legend_short=True,
    ax=ax1
)

dokdo.taxa_abundance_bar_plot(
    qzv_file,
    figsize=(10, 7),
    level=6,
    count=8,
    legend_short=True,
    ax=ax2
)

handles, labels = ax2.get_legend_handles_labels()
ax2.legend(handles[::-1], labels[::-1])

plt.tight_layout()
plt.savefig('test.png')

Did you run the code that creates the new legend within the same cell or in a separate cell in Jupyter Notebook? They should be run together.
Hope this helps.

Thanks @sbslee - I'm glad I was on the right track. I can get the reverse order to work as you have shown above, and legends are included in each plot.
But I can't get it to work when I'm using 'artist_kwargs=dict(legend_only=True, legend_loc='upper left') - to display the stacked bar chart with no legend and the legend only the second plot. I'm guessing there is a conflict with the artist?

@mishkb,
Glad to hear things are working now!
Also, please note that the artist_kwargs argument has been completely deprecated in 1.11.0. As shown in the changelog and in this issue, the deprecation process began in 1.10.0 and was completed in 1.11.0. Therefore, you should not use the argument anymore.
You can still plot the legend only by following this tutorial. I just found that there is a typo ("We can ameliorate the issue by plotting the legend separately with legend_only=True."), but please ignore it :) I will fix this ASAP, but the code is still correct:
handles, labels = ax2.get_legend_handles_labels()

ax2.clear()
ax2.legend(handles, labels)
ax2.axis('off')

Finally,  you can specify the location of a legend with:
ax.legend(..., loc='upper left')

Thanks again @sbslee for your quick response. I knew i had read that you were deprecating kwargs but hadn't updated my file (you are making frequent updates - which is great!). I have the latest version of Dokdo.
FYI - I also just found that 'plt.legend(bbox_to_anchor=(x0, y0)' or 'plt.legend(bbox_to_anchor=(x0, y0)' also works to move the legend out of the plot area. So you don't need to set-up two plots (one without legend, one without plot).

@mishkb,
Glad to hear we are on the same page now. And yea, sorry about the frequent updates. I know it can be challenging for end users to stay up to date (been there, done that), but I believe those are worth it :) I promise future updates will be less disrupting for sure -- completely getting rid of artist_kwargs was a massive task and that's why it took two separate versions to accomplish it.
As for utilizing bbox_to_anchor, I am aware of that approach but personally I like to create a separate plot just for the legend because it allows me to precisely control the width and height ratios when I need to plot multiple plots within single figure. But if you have single bar chart and want to plot its legend separately, please be my guest and use bbox_to_anchor for that! Thanks for sharing your approach.


Windows server possible?

I am trying to get juice working but no luck. They did have windows server. The use case is for homelab environment where I pass through my GPU to a windows VM and want to share the power back to other VMs.

A Windows server is definitely possible! The current server may already work on Windows however we haven't been testing with it. I'll leave this ticket open for official Windows support.