HDFS-13616. Batch listing of multiple directories

One of the dominant workloads for external metadata services is listing of partition directories. This can end up being bottlenecked on RTT time when partition directories contain a small number of files. This is fairly common, since fine-grained partitioning is used for partition pruning by the query engines.
A batched listing API that takes multiple paths amortizes the RTT cost. Initial benchmarks show a 10-20x improvement in metadata loading performance.

A rebase of the original patch by @umbrant . There are not too many conflicts when doing the rebasing. IMO the original patch is already in a very good shape. I'll start from here and see if people have any comments.
cc @jojochuang

Glad that this is being picked back up! Thanks Chao!

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 35s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
prototool
0m  0s
prototool was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 4 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
0m 25s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
18m  6s
trunk passed

+1 :green_heart:
compile
16m 57s
trunk passed

+1 :green_heart:
checkstyle
3m  8s
trunk passed

+1 :green_heart:
mvnsite
4m 38s
trunk passed

+1 :green_heart:
shadedclient
21m 50s
branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
4m 52s
trunk passed

+0 :ok:
spotbugs
1m 20s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
8m 38s
trunk passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 24s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 11s
the patch passed

+1 :green_heart:
compile
16m 18s
the patch passed

-1 :x:
cc
16m 18s
root generated 2 new + 24 unchanged - 2 fixed = 26 total (was 26)

-1 :x:
javac
16m 18s
root generated 1 new + 1869 unchanged - 0 fixed = 1870 total (was 1869)

-0 :warning:
checkstyle
3m  8s
root: The patch generated 15 new + 1199 unchanged - 0 fixed = 1214 total (was 1199)

+1 :green_heart:
mvnsite
4m 40s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  3s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
12m 54s
patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
4m 59s
the patch passed

+1 :green_heart:
findbugs
9m 14s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
9m 10s
hadoop-common in the patch passed.

+1 :green_heart:
unit
2m 18s
hadoop-hdfs-client in the patch passed.

-1 :x:
unit
87m 38s
hadoop-hdfs in the patch failed.

+1 :green_heart:
unit
7m 33s
hadoop-hdfs-rbf in the patch passed.

-1 :x:
asflicense
1m  3s
The patch generated 1 ASF License warnings.

239m 30s

Reason
Tests

Failed junit tests
hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1725

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc prototool

uname
Linux efb692705f75 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / b033c68

Default Java
1.8.0_222

cc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/1/artifact/out/diff-compile-cc-root.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/1/artifact/out/diff-compile-javac-root.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/1/artifact/out/diff-checkstyle-root.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/1/testReport/

asflicense
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/1/artifact/out/patch-asflicense-problems.txt

Max. process+thread count
3938 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/1/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 35s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
prototool
0m  0s
prototool was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 4 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
1m  6s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
18m  0s
trunk passed

+1 :green_heart:
compile
16m 57s
trunk passed

+1 :green_heart:
checkstyle
3m  8s
trunk passed

+1 :green_heart:
mvnsite
4m 39s
trunk passed

+1 :green_heart:
shadedclient
22m 17s
branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
4m 55s
trunk passed

+0 :ok:
spotbugs
1m 18s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
8m 43s
trunk passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 37s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 10s
the patch passed

+1 :green_heart:
compile
17m 11s
the patch passed

-1 :x:
cc
17m 11s
root generated 5 new + 21 unchanged - 5 fixed = 26 total (was 26)

-1 :x:
javac
17m 11s
root generated 1 new + 1870 unchanged - 0 fixed = 1871 total (was 1870)

-0 :warning:
checkstyle
3m  4s
root: The patch generated 18 new + 1199 unchanged - 0 fixed = 1217 total (was 1199)

+1 :green_heart:
mvnsite
4m 28s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  3s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
12m 53s
patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
4m 52s
the patch passed

+1 :green_heart:
findbugs
9m  9s
the patch passed

_ Other Tests _

-1 :x:
unit
8m 49s
hadoop-common in the patch failed.

-1 :x:
unit
0m 50s
hadoop-hdfs-client in the patch failed.

-1 :x:
unit
0m 41s
hadoop-hdfs in the patch failed.

-1 :x:
unit
0m 31s
hadoop-hdfs-rbf in the patch failed.

-1 :x:
asflicense
0m 48s
The patch generated 1 ASF License warnings.

145m 11s

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1725

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc prototool

uname
Linux 0b5d40582342 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 8d82c44

Default Java
1.8.0_222

cc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/2/artifact/out/diff-compile-cc-root.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/2/artifact/out/diff-compile-javac-root.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/2/artifact/out/diff-checkstyle-root.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/2/testReport/

asflicense
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/2/artifact/out/patch-asflicense-problems.txt

Max. process+thread count
1574 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/2/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

One thing I'm not totally sure is the case where the input srcs is empty. Currently it throws the exception:
FileNotFoundException("One or more paths do not exist.")

which I think is not very accurate. Is FileNotFoundException appropriate here?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 36s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
prototool
0m  0s
prototool was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 4 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
1m  6s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
18m 10s
trunk passed

-1 :x:
compile
5m 42s
root in trunk failed.

+1 :green_heart:
checkstyle
2m 56s
trunk passed

+1 :green_heart:
mvnsite
4m  1s
trunk passed

+1 :green_heart:
shadedclient
20m 33s
branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
4m 10s
trunk passed

+0 :ok:
spotbugs
1m  8s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
7m 53s
trunk passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 24s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m  7s
the patch passed

-1 :x:
compile
5m 39s
root in the patch failed.

-1 :x:
cc
5m 39s
root in the patch failed.

-1 :x:
javac
5m 39s
root in the patch failed.

-0 :warning:
checkstyle
2m 53s
root: The patch generated 18 new + 1199 unchanged - 0 fixed = 1217 total (was 1199)

+1 :green_heart:
mvnsite
3m 39s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  3s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
13m  3s
patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
4m  1s
the patch passed

+1 :green_heart:
findbugs
8m 24s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
8m 58s
hadoop-common in the patch passed.

-1 :x:
unit
2m  6s
hadoop-hdfs-client in the patch failed.

+1 :green_heart:
unit
87m 24s
hadoop-hdfs in the patch passed.

+1 :green_heart:
unit
7m 22s
hadoop-hdfs-rbf in the patch passed.

+1 :green_heart:
asflicense
0m 42s
The patch does not generate ASF License warnings.

210m 38s

Reason
Tests

Failed junit tests
hadoop.hdfs.protocol.TestReadOnly

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1725

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc prototool

uname
Linux f784a3824497 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 76bb297

Default Java
1.8.0_222

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/3/artifact/out/branch-compile-root.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/3/artifact/out/patch-compile-root.txt

cc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/3/artifact/out/patch-compile-root.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/3/artifact/out/patch-compile-root.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/3/artifact/out/diff-checkstyle-root.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/3/testReport/

Max. process+thread count
4277 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/3/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 35s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
prototool
0m  0s
prototool was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 5 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
1m  6s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
18m  2s
trunk passed

-1 :x:
compile
5m 42s
root in trunk failed.

+1 :green_heart:
checkstyle
2m 57s
trunk passed

+1 :green_heart:
mvnsite
3m 59s
trunk passed

+1 :green_heart:
shadedclient
20m 23s
branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
4m 10s
trunk passed

+0 :ok:
spotbugs
1m  9s
Used deprecated FindBugs config; considering switching to SpotBugs.

-1 :x:
findbugs
2m 45s
hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant findbugs warnings.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 24s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m  6s
the patch passed

-1 :x:
compile
5m 35s
root in the patch failed.

-1 :x:
cc
5m 35s
root in the patch failed.

-1 :x:
javac
5m 35s
root in the patch failed.

-0 :warning:
checkstyle
2m 55s
root: The patch generated 18 new + 1199 unchanged - 0 fixed = 1217 total (was 1199)

+1 :green_heart:
mvnsite
3m 41s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  3s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
13m  2s
patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
4m  1s
the patch passed

+1 :green_heart:
findbugs
8m 19s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
8m 56s
hadoop-common in the patch passed.

+1 :green_heart:
unit
2m  5s
hadoop-hdfs-client in the patch passed.

-1 :x:
unit
87m 52s
hadoop-hdfs in the patch failed.

+1 :green_heart:
unit
7m 25s
hadoop-hdfs-rbf in the patch passed.

+1 :green_heart:
asflicense
0m 44s
The patch does not generate ASF License warnings.

210m 41s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.namenode.TestFsck

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1725

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc prototool

uname
Linux<PHONE_NUMBER>f4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 11cd5b6

Default Java
1.8.0_222

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/4/artifact/out/branch-compile-root.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/4/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/4/artifact/out/patch-compile-root.txt

cc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/4/artifact/out/patch-compile-root.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/4/artifact/out/patch-compile-root.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/4/artifact/out/diff-checkstyle-root.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/4/testReport/

Max. process+thread count
3973 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/4/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 11s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+0 :ok:
prototool
0m  1s
prototool was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 5 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
1m 11s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
22m 41s
trunk passed

+1 :green_heart:
compile
19m 54s
trunk passed

+1 :green_heart:
checkstyle
3m 23s
trunk passed

+1 :green_heart:
mvnsite
4m 27s
trunk passed

+1 :green_heart:
shadedclient
22m 59s
branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
4m 59s
trunk passed

+0 :ok:
spotbugs
1m 27s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
9m 48s
trunk passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 24s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 28s
the patch passed

+1 :green_heart:
compile
17m 59s
the patch passed

-1 :x:
cc
17m 59s
root generated 2 new + 24 unchanged - 2 fixed = 26 total (was 26)

-1 :x:
javac
17m 59s
root generated 1 new + 1870 unchanged - 0 fixed = 1871 total (was 1870)

-0 :warning:
checkstyle
3m 34s
root: The patch generated 18 new + 1201 unchanged - 0 fixed = 1219 total (was 1201)

+1 :green_heart:
mvnsite
4m 47s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  3s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
15m  2s
patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
5m 27s
the patch passed

+1 :green_heart:
findbugs
10m 24s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
9m 33s
hadoop-common in the patch passed.

+1 :green_heart:
unit
2m  8s
hadoop-hdfs-client in the patch passed.

-1 :x:
unit
109m 10s
hadoop-hdfs in the patch failed.

-1 :x:
unit
7m 58s
hadoop-hdfs-rbf in the patch failed.

+1 :green_heart:
asflicense
0m 55s
The patch does not generate ASF License warnings.

278m 53s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots

hadoop.hdfs.TestFileChecksum

hadoop.hdfs.TestFileChecksumCompositeCrc

hadoop.hdfs.server.namenode.TestRedudantBlocks

hadoop.hdfs.TestReconstructStripedFile

hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1725

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml cc prototool

uname
Linux 54d09aa0d437 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / ef59ffd

Default Java
1.8.0_232

cc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/5/artifact/out/diff-compile-cc-root.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/5/artifact/out/diff-compile-javac-root.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/5/artifact/out/diff-checkstyle-root.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/5/testReport/

Max. process+thread count
3288 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1725/5/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Hello, I encountered a bug when using the batch method, when I input a directory with more than 1000 files in it and 2 replications of each file's data block, only the first 500 files of this directory are returned and then it stops. I think it should be hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java in getBatchedListing() method to modify, as follows.
for (; srcsIndex < srcs.length; srcsIndex++) {
String src = srcs[srcsIndex];
HdfsPartialListing listing;
try {
DirectoryListing dirListing = getListingInt(dir, pc, src, indexStartAfter, needLocation);
if (dirListing == null) {
throw new FileNotFoundException("Path " + src + " does not exist");}
listing = new HdfsPartialListing(srcsIndex, Lists.newArrayList(dirListing.getPartialListing()));
numEntries += listing.getPartialListing().size();
lastListing = dirListing;
            } catch (Exception e) {
                if (e instanceof AccessControlException) {
                    logAuditEvent(false, operationName, src);}
                listing = new HdfsPartialListing(srcsIndex,
                        new RemoteException(e.getClass().getCanonicalName(), e.getMessage()));
                lastListing = null;
                LOG.info("Exception listing src {}", src, e);}
            listings.put(srcsIndex, listing);

          //My modification
            (lastListing.getRemainingEntries()!=0)
            {
                 break;
            }

            if (indexStartAfter.length != 0)
            {
                indexStartAfter = new byte[0];
            }
            // Terminate if we've reached the maximum listing size
            if (numEntries >= dir.getListLimit()) {
                break;
            }
        }

The reason for this bug is mainly that the result returned by the getListingInt(dir, pc, src, indexStartAfter, needLocation) method will limit both the number of files in the directory as well as the number of data blocks and replications of the files at the same time. But the getBatchedListing() method will only exit the loop if the number of returned results is greater than 1000.
Looking forward to your reply

Hello, I encountered a bug when using the batch method, when I input a directory with more than 1000 files in it and 2 replications of each file's data block, only the first 500 files of this directory are returned and then it stops. I think it should be hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java in getBatchedListing() method to modify, as follows.
for (; srcsIndex < srcs.length; srcsIndex++) {
String src = srcs[srcsIndex];
HdfsPartialListing listing;
try {
DirectoryListing dirListing = getListingInt(dir, pc, src, indexStartAfter, needLocation);
if (dirListing == null) {
throw new FileNotFoundException("Path " + src + " does not exist");}
listing = new HdfsPartialListing(srcsIndex, Lists.newArrayList(dirListing.getPartialListing()));
numEntries += listing.getPartialListing().size();
lastListing = dirListing;
            } catch (Exception e) {
                if (e instanceof AccessControlException) {
                    logAuditEvent(false, operationName, src);}
                listing = new HdfsPartialListing(srcsIndex,
                        new RemoteException(e.getClass().getCanonicalName(), e.getMessage()));
                lastListing = null;
                LOG.info("Exception listing src {}", src, e);}
            listings.put(srcsIndex, listing);

          //My modification
            (lastListing.getRemainingEntries()!=0)
            {
                 break;
            }

            if (indexStartAfter.length != 0)
            {
                indexStartAfter = new byte[0];
            }
            // Terminate if we've reached the maximum listing size
            if (numEntries >= dir.getListLimit()) {
                break;
            }
        }

The reason for this bug is mainly that the result returned by the getListingInt(dir, pc, src, indexStartAfter, needLocation) method will limit both the number of files in the directory as well as the number of data blocks and replications of the files at the same time. But the getBatchedListing() method will only exit the loop if the number of returned results is greater than 1000.
Looking forward to your reply.

    Hello, I encountered a bug when using the batch method, when I input a directory with more than 1000 files in it and 2 replications of each file's data block, only the first 500 files of this directory are returned and then it stops. I think it should be hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java in getBatchedListing() method to modify, as follows.

` for (; srcsIndex < srcs.length; srcsIndex++) {
String src = srcs[srcsIndex];
HdfsPartialListing listing;
try {
DirectoryListing dirListing = getListingInt(dir, pc, src, indexStartAfter, needLocation);
if (dirListing == null) {
throw new FileNotFoundException("Path " + src + " does not exist");}
listing = new HdfsPartialListing(srcsIndex, Lists.newArrayList(dirListing.getPartialListing()));
numEntries += listing.getPartialListing().size();
lastListing = dirListing;
            } catch (Exception e) {
                if (e instanceof AccessControlException) {
                    logAuditEvent(false, operationName, src);}
                listing = new HdfsPartialListing(srcsIndex,
                        new RemoteException(e.getClass().getCanonicalName(), e.getMessage()));
                lastListing = null;
                LOG.info("Exception listing src {}", src, e);}
            listings.put(srcsIndex, listing);

          //My modification
            (lastListing.getRemainingEntries()!=0)
            {
                 break;
            }

            if (indexStartAfter.length != 0)
            {
                indexStartAfter = new byte[0];
            }
            // Terminate if we've reached the maximum listing size
            if (numEntries >= dir.getListLimit()) {
                break;
            }
        }`
   The reason for this bug is mainly that the result returned by the getListingInt(dir, pc, src, indexStartAfter, needLocation) method will limit both the number of files in the directory as well as the number of data blocks and replications of the files at the same time. But the getBatchedListing() method will only exit the loop if the number of returned results is greater than 1000.
    Looking forward to your reply.

Hello, I encountered a bug when using the batch method, when I input a directory with more than 1000 files in it and 2 replications of each file's data block, only the first 500 files of this directory are returned and then it stops. I think it should be hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java in getBatchedListing() method to modify, as follows.

		```

for (; srcsIndex < srcs.length; srcsIndex++) {
String src = srcs[srcsIndex];
HdfsPartialListing listing;
try {
DirectoryListing dirListing = getListingInt(dir, pc, src, indexStartAfter, needLocation);
if (dirListing == null) {
throw new FileNotFoundException("Path " + src + " does not exist");}
listing = new HdfsPartialListing(srcsIndex, Lists.newArrayList(dirListing.getPartialListing()));
numEntries += listing.getPartialListing().size();
lastListing = dirListing;
} catch (Exception e) {
if (e instanceof AccessControlException) {
logAuditEvent(false, operationName, src);}
listing = new HdfsPartialListing(srcsIndex,
new RemoteException(e.getClass().getCanonicalName(), e.getMessage()));
lastListing = null;
LOG.info("Exception listing src {}", src, e);}
listings.put(srcsIndex, listing);
          //My modification
            (lastListing.getRemainingEntries()!=0)
            {
                 break;
            }

            if (indexStartAfter.length != 0)
            {
                indexStartAfter = new byte[0];
            }
            // Terminate if we've reached the maximum listing size
            if (numEntries >= dir.getListLimit()) {
                break;
            }
        }

    The reason for this bug is mainly that the result returned by the getListingInt(dir, pc, src, indexStartAfter, needLocation) method will limit both the number of files in the directory as well as the number of data blocks and replications of the files at the same time. But the getBatchedListing() method will only exit the loop if the number of returned results is greater than 1000.
    Looking forward to your reply.

@fanlinqian best to file an HDFS issue on the apache jira server.


Potential Cost Optimizations doesn't populate in 8.2 in Infra dashboard

Describe the bug
There is a bug in 8.2 that was introduced where Potential Cost Optimizations doesn't populate data in the Azure Advisor Dashboard.
To Reproduce
Steps to reproduce the behavior:
Run Report using template
Expected behavior
Potential Cost Optimizations shows data
Screenshots

Fix
To fix this you need to delete the Subscription Name is not (Blank) filter on Azure Advisor page with nothing selected.

Then data populates.

How did you get it to even work or not have access issues?

Good point @timmcfadden , there're some advisor cost optimizations that at the API level don't have a Subscription assigned, and this is why are not appearing.


README.md updated

Coverage remained the same at 92.982% when pulling 56a92266daaf625eeddfd35cc09dd6f16c365c4d on development into 37f1c716c65c06b52e2e30bf449e4d430a6c3e10 on master.

Coverage remained the same at 92.982% when pulling 82dab0b43517d2d152c12f69ae1dacdac827ddff on development into 37f1c716c65c06b52e2e30bf449e4d430a6c3e10 on master.


Reimplementation

https://github.com/jonashaag/django-mail-templated-simple
Just posting this here if someone else has issues with serialization or django-celery-email.
Maybe you could explain why you had to make your implementation so complicated? I'm not saying you should change it. Just wondering...

Finally mail_templated was reimplemented too :)

I had a look at the reimplementation, and while it looks like some of the render related complexity has been cleaned up, the code still contains an insane amount stuff going on, most of which is non-trivial to understand.
How can you even come up with a solution so complicated for a problem so simple? Why do you need all that "change any parameters anytime later" flexibility? What do you even need state for? You see what sub classing Django's email class brings you there? Issues with integrating into other code, hence complexity.
Plus, the new clean parameter to __init__ follows the principle of most surprise, because if render=False, clean=True is passed, then the class behaves as if render=True, without any kind of warning. The render=False, clean=True case should be illegal and throw an error. But you can get rid of this complexity simply by not making a subclass in the first place.
What common use cases are you trying to cover with all that complicated stuff in this library? Isn't the problem description simply "I don't want to have email templates in the Python code" or "I want an easy way to have multiple email templates to choose from"?
How come my implementation doesn't do any subclassing, doesn't have issues with integration into other code, doesn't invent any new interfaces, doesn't need state? It doesn't even contain any real case distinctions (if statements), because the problem it solves is trivial, so the solution should be trivial as well!
 How many hours have you spent implementing all that stuff, trying to figure out all the corner cases, writing tests for all the cases, debugging incompatibilities with other software, etc? I have spent no more than a few hours on the whole project, and that includes reading up all of the mail-related Django documentation, setting up CI, publishing to PyPI etc.
Please tell my if my criticism annoys you, in which case I'll shut up and leave you alone :-)

@jonashaag

I had a look at the reimplementation, and while it looks like some of the render related complexity has been cleaned up

Thanks to your previous criticism :)

Plus, the new clean parameter to init follows the principle of most surprise, because if render=False, clean=True is passed, then the class behaves as if render=True, without any kind of warning. The render=False, clean=True case should be illegal and throw an error.

 Thanks. I've fixed this in the last version.

But you can get rid of this complexity simply by not making a subclass in the first place.

No needs, this is done by you already ;)

the code still contains an insane amount stuff going on, most of which is non-trivial to understand.

That's interesting. Actually, you should not understand the implementation. You should just understand how to use this. Are there any unclear things in the API?
What about other questions, well, you ask too many questions at once. I have answers for all of these questions, but do you really want to read them?
If you want to discuss this, let's review just one of many examples. Let's use your implementation to send 10000 email messages at once.
class CompareAppsTestCase(TestCase):

    def test1(self):
        start = time()
        mail_templated_simple.send_mass_mail('email.tpl', tuple(
            ({'foo': foo}<EMAIL_ADDRESS><EMAIL_ADDRESS>            for foo in range(1000)))
        print 'mail_templated_simple: {}s'.format(time() - start)

    def test2(self):
        start = time()
        message = mail_templated.EmailMessage('email.tpl')
        for foo in range(1000):
            message.from_email, message.to =<EMAIL_ADDRESS><EMAIL_ADDRESS>            message.context = {'foo': foo}
        print 'mail_templated: {}s'.format(time() - start)

Result:
mail_templated_simple: 7.39179491997s
mail_templated: 0.00233292579651s

10000 is not a big number. Now imagine we send 1 million messages, and imagine we use some slow (and probably third-party) storage for the email templates.
The answer is that the developer who uses your app will need to reimplement either your app or storage class. My app is ready for this and many other situations. Just because it can (c)

Heh, there was a small mistake in the example ))) Fixed.

One thing that I'm really in doubt about is those arguments render and clear. While they are pretty simple, they add more ways to do same things. This seems like not a Python way. This is a kind of universal function that can cook and drive and do whatever you want. This is a bad design pattern. I'll probably remove them in the next version.

Please let me know what you think about this change
https://github.com/artemrizhov/django-mail-templated/pull/22/files

That's interesting. Actually, you should not understand the implementation. You should just understand how to use this. Are there any unclear things in the API?

I always have to look at the implementation of all third party code at some point in the project, either because it has bugs, behaviour I didn't expect (this case back in 2014), incomplete documentation (or haven't found it in the docs), … I usually do a quick check on the quality and complexity of all third party code just to be sure I don't make myself dependent on some badly designed API.

What about other questions, well, you ask too many questions at once. I have answers for all of these questions, but do you really want to read them?

I do, best with examples for each, but I understand if you don't want to spend time explaining all that stuff to me.

Now imagine we send 1 million messages, and imagine we use some slow (and probably third-party) storage for the email templates.

Note that this case hasn't been considered in my implementation yet, but it is trivial to fix: Add the ability to pass a loaded template.
- def email_message(template_name, …):
-    template = get_template(template_name)
+ def email_message(template, …):
+    if isinstance(template, str):
+        template = get_template(template)

In my benchmarks, this speeds up mail-templated-simple's performance from 4x slower to about 1.3x slower.

Note that this case hasn't been considered in my implementation yet, but it is trivial to fix: Add the ability to pass a loaded template.

Sure, all such thing are trivial to fix, because the task is simple and Python is almost perfect. I just wanted to show how (and why) you just added new condition to your code.
To be honest, I don't use mail_templated for a long time because it is not needed in my current projects. This is just a kind of pet project. It is small enough to do and finish tasks on a weekend, but it still gives me significant experience, mostly because I can finish work quickly and see the result. This is also why I tend to add features that probably are not needed for anybody ))) It is hard to understand what may be needed when you don't use this yourself. My actual goal is not covering all the cases, but making the app flexible enough so that all problems may be solved without modifying the source code. So feedbacks from another developers are very important for me. Thank you for your feedbacks.

That's interesting. Actually, you should not understand the implementation. You should just understand how to use this. Are there any unclear things in the API?

I always have to look at the implementation of all third party code at some point in the project

And yes, I realized that I was wrong on this statement. I usually check the code of third-party libs too, and for the same reasons. I just wanted to say that I'm focused on the API. What about the code, I'm always trying to do my best to make it good as much as possible :) I feel like if the idea of the API is clear, then my code should not be too complex to understand.


Does it work on browser?

If I change requests to fetch API will it work?

This doesn't work in a frontend environment. If you use next.js or something like that that has a built-in server, you can use it there.


endpoins-operator安装在非kube-system的namespace下，无get leases权限

E0415 04:55:01.359523       1 leaderelection.go:325] error retrieving resource lock kube-system/sealyun-endpoints-operator-leader-election: leases.coordination.k8s.io "sealyun-endpoints-operator-leader-election" is forbidden: User "system:serviceaccount:deepflow:opensource-endpoints-operator" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system"

2f3e971b4b3d723df896459b5a69c49aff186bb4 已修复


Bulk generation

Hi,
I was wondering how I obtain the scripts for generating bulk from my single cell data (bulk_generation.py)?
Best,
Ida

Hi Ida,
just have a look in the scaden/preprocessing directory in this repo.
They are unfortunately not yet included in the main tool, sorry for that.
Bw,
Kevin

@idalarsson a quick follow up, because a different user had an issue:
you may want to use the scripts from the 'dev' branch, which has not been merged in the master branch yet.
the scripts from the main branch can lead to dependency issues with pandas.
Let me know if you encounter any problems!

Thank you! :)

Hello again,
I've now managed to get bulk_generation.py to work in my data, but encounter an error when trying to create the h5ad-file. The error message is:
Celltypes: [1, 2, 3, 4, 5, 6]
Traceback (most recent call last):
  File "create_h5ad_file.py", line 98, in <module>
    y = sort_celltypes(y, labels, celltypes)
  File "create_h5ad_file.py", line 55, in sort_celltypes
    idx = [labels.index(x) for x in ref_labels]
  File "create_h5ad_file.py", line 55, in <listcomp>
    idx = [labels.index(x) for x in ref_labels]
ValueError: 1 is not in list

Does this tell you anything?

I'm receiving the same type of error as @idalarsson. Did you solve the problem?

I'm receiving the same type of error as @idalarsson. Did you solve the problem?

Ah @idalarsson sorry for not reaching out again I didn't get any notification apparently because the issue is closed :/
@nicolasgoedert97 Have you tried running Scaden with the example data that can be generated?
Did you check that the format fits?
This seems like to be a formatting issue leading to Scaden not being able to parse the celltypes correctly.
Let me know if all of that works and if the format looked good!
I'm reopening this so I'm sure to be notifiede.

Ah @idalarsson sorry for not reaching out again I didn't get any notification apparently because the issue is closed :/
@nicolasgoedert97 Have you tried running Scaden with the example data that can be generated?
Did you check that the format fits?
This seems like to be a formatting issue leading to Scaden not being able to parse the celltypes correctly.
Let me know if all of that works and if the format looked good!
I'm reopening this so I'm sure to be notifiede.

@KevinMenden yup It was a formatting error for me. Applying the same format as in example data solved the issue.

@KevinMenden yup It was a formatting error for me. Applying the same format as in example data solved the issue.

Perfect, glad to hear that 👍

Perfect, glad to hear that 👍


ENYO-3774: Fix containers as they relate to Popup-ContextualPopup

Issue Resolved / Feature Added
This PR resolves issues related to spotlight-navigation using Popup and ContextualPopup. There are some underling spotlight container changes that were needed in order to add this functionality.
Resolution
This PR introduces spotlight behavior that allows/denies navigation based on an active container _lastContainerId. Measures are taken to only change the active container under specific scenarios. Also, spottable components can no longer be spotted via a pointer if they exist outside of a container using a restrict: 'self-only' container config.
In these cases we naturally prevent forwarding click events, since the component isn't spotted/active. A byproduct of this is - in our test suite when attempting to call component.simulate('click'). The click event isn't being forwarded because the component isn't spotted, unless you to call component.simulate('focus') first.
Additional Considerations
There are some edge-case scenarios to keep in mind when dealing with 5-way behavior in Popup. Behaviors can be dictated by the following individual (or combination of) props:
noAutoDismiss, scrimType, spotlightRestrict, then behavior is further affected if the Popup contains (or does not contain) spottable components.
Links
There are a series of in-progress PRs that this PR was dependent upon in order to work. The following PRs were incorporated into this fix:
ENYO-3864 - https://github.com/enyojs/enact/pull/594
ENYO-3987 - https://github.com/enyojs/enact/pull/605
ENYO-3970 - https://github.com/enyojs/enact/pull/599
ENYO-3905 - https://github.com/enyojs/enact/pull/600
Enact-DCO-1.0-Signed-off-by: Jeremy Thomas<EMAIL_ADDRESS>

Replaced by #691


Channel sidebar remains open on mobile device

Summary
Channel sidebar remains open on mobile device can't hide it which makes it unusable in portrait orientation of device.
Environment Information

Device Name: moto g(7)
OS Version: Android 9
Mattermost App Version: 1.20.0

Steps to reproduce
In device portrait orientation channel sidebar remains open no ability to hide it.
Expected behavior
Channel sidebar should be allowed to hide by sweep left gesture (as in previous app version).

@atomekk for some reason your phone is being detected as a tablet :S

I have the same issue with the same phone. Can we set it to mobile version manually?

do you know what is the size of that phone?

For some reason the app is detecting your diagonalSizeInches to be above 6.9

do you know what is the size of that phone?
According to spec it is:
Screen: 6.2" IPS LCD, 1080 x 2270px resolution (19:9); ~405 ppi.

Such option should be controlled by user settings and not unreliable auto detect IMHO.

really strange as
double widthInches = 1080 / 405;
double heightInches = 2280 / 405;
double diagonalSizeInches = Math.sqrt(Math.pow(widthInches, 2) + Math.pow(heightInches, 2));

diagonalSizeInches ~6.2

I have a Motorola G7 as well, running Android 9 and experiencing the same issue. Referred to this issue via https://twitter.com/Mattermost/status/1142874791554273280
The issue started after installing the 2 latest updates I believe.

Version 1.19.0 works fine btw.

Version 1.19.0 works fine btw.

It works fine on 1.19.0 because new update was introduced in 1.20.0 https://github.com/mattermost/mattermost-mobile/blob/master/CHANGELOG.md#tablet-improvements

Hey guys, I've built a quick app to get the screen metrics to see if there is something weird going on on those devices.
I've uploaded the apk in the Mattermost community server and you can access the [permalink here] (https://community.mattermost.com/core/pl/utqwrg8m53nsmxbk1xiawfr63c). It would be great if you can download, install and share a screenshot or the values that you get in the same thread as the permalink.
The app looks like this:

Thanks.

@enahum

Thanks for this @atomekk
So looking at the results your device is actually reporting ~320ppi instead of ~405ppi as the spec mention for some reason.
obviously by doing the math with 320ppi it actually yields a 7.9" screen size instead of 6.2" which is why is being detected as a tablet.
the way we get the ppi is with this https://developer.android.com/reference/android/util/DisplayMetrics.html#xdpi
So I wonder why is this happening, is it a manufacturer issue? is your phone rooted? do you have a custom rom?
Also just in case something that we are not sure but the question was brought up, does the phone has a notch? and if that is the case which one is it?

@enahum
Its brand new model from my mobile operator no custom rom not rooted.
By notch you mean that cam blended in screen on top?

@migbot looking at this last comment from @atomekk and as per our discussion, this seems to be that the manufacturer is reporting the wrong ppi's right?

If that is the case we'll need to add a specific check for the device model, but I really want to try and avoid that at all costs.

Why just not move that options to user setting: channel side bar fixed [x] IMHO?

My wife and I both also have the issue on Motorola G7 with Android Pie.

https://drive.google.com/file/d/12zOQnPWLejYQR1CPojQiAIfTMCqghmVj/view?usp=drivesdk

Hey guys @atomekk @olsonsp4c and @markmaas can I ask you to provide your exact model as it is shown in the phone preferences?

Moto G7 - Hardware verison PVT - Model Number XT1962-1
Scott
On Fri, Jun 28, 2019 at 9:30 AM Elias Nahum<EMAIL_ADDRESS>wrote:

Hey guys @atomekk https://github.com/atomekk @olsonsp4c
https://github.com/olsonsp4c and @markmaas https://github.com/markmaas
can I ask you to provide your exact model as it is shown in the phone
preferences?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mattermost/mattermost-mobile/issues/2911?email_source=notifications&email_token=AMOWLSWEYY7DHZ5DL6ROTYLP4YVBNA5CNFSM4H2WRRWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY2MYJY#issuecomment-506776615,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMOWLSUIB5OYZEVPRD2X4ZTP4YVBNANCNFSM4H2WRRWA
.

Ok guys another POC to see if this reports isTablet as NO.. please let me know how it goes
The build can be found here https://community.mattermost.com/core/pl/sa4zigmsgp8zmmuh385r944krc

That works for me!
On Fri, Jun 28, 2019, 12:01 PM Elias Nahum<EMAIL_ADDRESS>wrote:

Ok guys another POC to see if this reports isTablet as NO.. please let me
know how it goes
The build can be found here
https://community.mattermost.com/core/pl/sa4zigmsgp8zmmuh385r944krc
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mattermost/mattermost-mobile/issues/2911?email_source=notifications&email_token=AMOWLSUZQLN5TQQ2WRQRJELP4ZGY3A5CNFSM4H2WRRWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY2YY5Y#issuecomment-506825847,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMOWLSW4KV332OPMBMOHFZ3P4ZGY3ANCNFSM4H2WRRWA
.

@olsonsp4c can you share a screenshot or the values?

Here you go
On Sat, Jun 29, 2019, 4:55 AM Elias Nahum<EMAIL_ADDRESS>wrote:

@olsonsp4c https://github.com/olsonsp4c can you share a screenshot or
the values?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mattermost/mattermost-mobile/issues/2911?email_source=notifications&email_token=AMOWLSQBBX6EANORSDPFHWDP445QRA5CNFSM4H2WRRWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY3WVIQ#issuecomment-506948258,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMOWLSVTCXLTDT4JBHVKFRLP445QRANCNFSM4H2WRRWA
.

Hi all.  Just got a G7 myself, so I can help test.  Looks like isTablet is still yes on the latest test app.
GitHub doesn't seem to have an attach button for the screenshots(?), but the values reported are as follows:
Pixel w=1080 h=2270
Inches w=3.4 h=7.1
DPI x=319.0 y=320.3
Diagonal 7.9"
Is tablet yes
I'll get to a real computer in a bit to upload the screenshots.  The APK version does show 1.0, so maybe I got the original one? Not sure if you bothered to increment version numbers on a one off POC app :)
Thanks for the work on this.  For what it's worth, I noticed that YouTube in a browser (Firefox Focus) also appears to run a tablet layout.  Having second thoughts about this phone...

Hi @bnelsonfs thanks for your help on this. in the above permalink there should an apk attached, and you should also see in the screen when running the app fields for Model, manufacturer and brand in addition to what you shared.
If you are using your phone to browse GitHub I really haven't tried to upload an image from there, but in your computer you should be able to do so. (no rush but when you get a chance).
Also is weird that YouTube also has a tablet layout, that brings me to believe that there is an issue on that phone and that the manufacturer is not setting the right values of the screen.

@enahum Here are the screenshots for ya.

I've opened a ticket with Lenovo with a link back to this issue as well, and let you know if anything comes of that.

@bnelsonfs I've built another apk adding the model to a black list to see if it is now detected as a phone instead of a tablet. The apk can be found here https://community.mattermost.com/core/pl/6g45i6rcdjrqpg448xadwe5kay can you please confirm?

Thank you again @enahum for your work here.  Here are the results from the latest build.  Unfortunately, I really need a functional phone for work, so I'm returning the phone today for a different model, and won't be able to run further tests for you.
More testing shows that ~50% of websites will display the desktop version of the site, rather than mobile, making even general browsing on the internet a poor experience.  I really don't think this is an issue with Mattermost, but a glaring bug with the G7.

Lenovo directed me to the forums for support:
As per checking here, there's no way to change the screen resolution of the phone since it's already the default setting of the device. For further assistance, you can check our support forum.

That is ludicrous
On Mon, Jul 1, 2019, 11:33 PM Bryan Nelson<EMAIL_ADDRESS>wrote:

Lenovo directed me to the forums for support:
As per checking here, there's no way to change the screen resolution of the phone since it's already the default setting of the device. For further assistance, you can check our support forum.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mattermost/mattermost-mobile/issues/2911?email_source=notifications&email_token=AMOWLSSF7VF23NKWNKEAY7LP5LSAZA5CNFSM4H2WRRWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZADM4Y#issuecomment-507524723,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMOWLSXC6V4YMTSP2S6GX2DP5LSAZANCNFSM4H2WRRWA
.

So it is a manufacturer issue then.. I'm really sorry guys, but everything we tried to make it work doesn;'t seem to help.
I brought up this issue with a PM and we will see how feasible is to make this an opt-in/opt-out option, probably as a help wanted ticket, so if any of you want to volunteer and help that would be awesome 😎

I'll help however I can
On Tue, Jul 2, 2019, 6:15 AM Elias Nahum<EMAIL_ADDRESS>wrote:

So it is a manufacturer issue then.. I'm really sorry guys, but everything
we tried to make it work doesn;'t seem to help.
I brought up this issue with a PM and we will see how feasible is to make
this an opt-in/opt-out option, probably as a help wanted ticket, so if any
of you want to volunteer and help that would be awesome 😎
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mattermost/mattermost-mobile/issues/2911?email_source=notifications&email_token=AMOWLSU2XDE64EQERKB5KV3P5NBFNA5CNFSM4H2WRRWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZBCEKI#issuecomment-507650601,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMOWLSQBVNUMDITZIYVTA7DP5NBFNANCNFSM4H2WRRWA
.

@bnelsonfs @olsonsp4c we've discussed increasing the required screen size for the fixed sidebar to 8" diagonal, which should be large enough to solve the issue found with the G7 (even considering the incorrectly reported screen resolution by the manufacturer). @enahum is going to build another version of the screen metrics app with this change and then we can have your help retesting to verify your device is not reported as a tablet. If this solved it we'll get a new app version shipped as soon as possible.
In addition to that, we discussed opening a feature ticket for a sidebar preference setting to turn off the fixed width sidebar if a device is detected as a tablet.

Thank you!
Scott
On Wed, Jul 3, 2019 at 12:54 PM Eric Sethna<EMAIL_ADDRESS>wrote:

@bnelsonfs https://github.com/bnelsonfs @olsonsp4c
https://github.com/olsonsp4c we've discussed increasing the required
screen size for the fixed sidebar to 8" diagonal, which should be large
enough to solve the issue found with the G7 (even considering the
incorrectly reported screen resolution by the manufacturer). @enahum
https://github.com/enahum is going to build another version of the
screen metrics app with this change and then we can have your help
retesting to verify your device is not reported as a tablet. If this solved
it we'll get a new app version shipped as soon as possible.
In addition to that, we discussed opening a feature ticket for a sidebar
preference setting to turn off the fixed width sidebar if a device is
detected as a tablet.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mattermost/mattermost-mobile/issues/2911?email_source=notifications&email_token=AMOWLSQNXWJQUCAQ2NUB7NLP5TYURA5CNFSM4H2WRRWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZFMBVQ#issuecomment-508215510,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMOWLSQ65XLAEE7EOGDXPV3P5TYURANCNFSM4H2WRRWA
.

@bnelsonfs @olsonsp4c @atomekk @esethna here is the build that should report that is a tablet only on 8" + screen sizes https://community.mattermost.com/core/pl/dbymxtmr5pyr3f51dqpp1qemor
please confirm if this works

Here you go!
On Thu, Jul 4, 2019, 9:42 AM Elias Nahum<EMAIL_ADDRESS>wrote:

@bnelsonfs https://github.com/bnelsonfs @olsonsp4c
https://github.com/olsonsp4c @atomekk https://github.com/atomekk
@esethna https://github.com/esethna here is the build that should
report that is a tablet only on 8" + screen sizes
https://community.mattermost.com/core/pl/dbymxtmr5pyr3f51dqpp1qemor
please confirm if this works
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mattermost/mattermost-mobile/issues/2911?email_source=notifications&email_token=AMOWLST6ZJ54FIKF3Q7DXH3P5YK4VA5CNFSM4H2WRRWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZHXDYI#issuecomment-508522977,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMOWLSQ574KW7I3ZLJUNFZ3P5YK4VANCNFSM4H2WRRWA
.

@olsonsp4c I don't see anything with your post other than here you go!

I attached a screenshot. Let me know if it doesn't work
On Thu, Jul 4, 2019, 2:31 PM Elias Nahum<EMAIL_ADDRESS>wrote:

@olsonsp4c https://github.com/olsonsp4c I don't see anything with your
post other than here you go!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mattermost/mattermost-mobile/issues/2911?email_source=notifications&email_token=AMOWLSTIBIVVF535Y6JBBS3P5ZMY3A5CNFSM4H2WRRWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZIC7IY#issuecomment-508571555,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMOWLSU2HCFFBULKMUBX7MTP5ZMY3ANCNFSM4H2WRRWA
.

@olsonsp4c I don't see any screenshots, probably attaching files when replying to the email does not work

[image: Screenshot_20190704-135844.png]
On Thu, Jul 4, 2019 at 5:57 PM Elias Nahum<EMAIL_ADDRESS>wrote:

@olsonsp4c https://github.com/olsonsp4c I don't see any screenshots,
probably attaching files when replying to the email does not work
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mattermost/mattermost-mobile/issues/2911?email_source=notifications&email_token=AMOWLSVYYN5SGA5QTJKGQZTP52E5PA5CNFSM4H2WRRWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZIIAAA#issuecomment-508592128,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMOWLSSFXV3UETOU47C7J5TP52E5PANCNFSM4H2WRRWA
.

ᐧ

@olsonsp4c hahahahaha still nothing dude.. you can either upload the image directly in Github or just report if the field for is Tablet says No

well, that is interesting! Tablet does say NO. That time I had directly
inserted the screenshot in the response! Oh well.
Scott
On Fri, Jul 5, 2019 at 7:14 AM Elias Nahum<EMAIL_ADDRESS>wrote:

@olsonsp4c https://github.com/olsonsp4c hahahahaha still nothing dude..
you can either upload the image directly in Github or just report if the
field for is Tablet says No
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mattermost/mattermost-mobile/issues/2911?email_source=notifications&email_token=AMOWLSQ2VHP6EM43AVNN3ULP55CKBA5CNFSM4H2WRRWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZJPXXI#issuecomment-508754909,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMOWLSSGRFNRV5JZWHIDISTP55CKBANCNFSM4H2WRRWA
.

Thanks for confirming that it says No

Any word on the opt-in/opt-out feature? Mattermost has been unusable for a while for me :(

@lunarthegrey the opt-in/out feature will be a help wanted ticket, for now we added a workaround on a dot release, you can download and install the release candidate here https://releases.mattermost.com/mattermost-mobile-beta/1.20.2/204/Mattermost_Beta.apk

@enahum Just installed that rc and its working OK

I can't open the file
On Sat, Jul 6, 2019, 1:40 PM Elias Nahum<EMAIL_ADDRESS>wrote:

@lunarthegrey https://github.com/lunarthegrey the opt-in/out feature
will be a help wanted ticket, for now we added a workaround on a dot
release, you can download and install the release candidate here
https://releases.mattermost.com/mattermost-mobile-beta/1.20.2/204/Mattermost_Beta.apk
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mattermost/mattermost-mobile/issues/2911?email_source=notifications&email_token=AMOWLSQVSYCC6HMZEKFPZH3P6DYKFA5CNFSM4H2WRRWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZK7FWY#issuecomment-508949211,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMOWLSXQEQLXR7DHTSKCU2LP6DYKFANCNFSM4H2WRRWA
.

I did get this to work.
On Sat, Jul 6, 2019, 1:40 PM Elias Nahum<EMAIL_ADDRESS>wrote:

@lunarthegrey https://github.com/lunarthegrey the opt-in/out feature
will be a help wanted ticket, for now we added a workaround on a dot
release, you can download and install the release candidate here
https://releases.mattermost.com/mattermost-mobile-beta/1.20.2/204/Mattermost_Beta.apk
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mattermost/mattermost-mobile/issues/2911?email_source=notifications&email_token=AMOWLSQVSYCC6HMZEKFPZH3P6DYKFA5CNFSM4H2WRRWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZK7FWY#issuecomment-508949211,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMOWLSXQEQLXR7DHTSKCU2LP6DYKFANCNFSM4H2WRRWA
.

Hello guys. I'm facing this problem also on a Meizu M6 Note. The provided RC didn't work also :/
Here goes my metrics screenshot:

@RaphaSS this is very unfortunate and I really don't know what else can be done here as your device is reporting to have a diagonal size of 13.8" which is definitely going to be considered as a tablet.
Have you tried reaching out to the manufacturer?

@RaphaSS this is very unfortunate and I really don't know what else can be done here as your device is reporting to have a diagonal size of 13.8" which is definitely going to be considered as a tablet.
Have you tried reaching out to the manufacturer?

No @enahum. I haven't. I don't see how they could be of any help, actually. I wonder if the test for the phone to be considered a tablet is missing something like an additional variable? My phone is not that big to be considered a tablet haha.
I'll probably stick to an older version for a while and have a look into the code to see if I can help.

@RaphaSS help is always appreciate it ;)

some problem on Wileyfox Swift 2 X
waiting fix

Since the device reporting as tablet sized is out of our control and could be affecting multiple device types, we have a fix up that adds a setting to enable/disable the permanent sidebar: https://github.com/mattermost/mattermost-mobile/pull/3060


i want to show multiple button in toast and perform action when click on button

Hi @govind1530 , Toast is limited to single button. You can use Alert from 'react-native' for multiple button.

Hi @govind1530 , closing for now due to inactivity. Let us know if the issue still persists.


Add Search

Based off of 'message cache' PRs.

Add search UI and functionality
Make MessageList more reusable. Use for search results

Automated message from Dropbox CLA bot
@borisyankov, it looks like you've already signed the Dropbox CLA. Thanks!

I am also not a fan of putting a search button in the left panel, and in the right panel for users.
Indeed having a search button on the main screen and everything organized in tabs and zero or one side bar is the norm.

We can revisit whether to change the design to tabs at some point soon, but I don't see how that's related to this. Search can be in the main screen with or without tabs. Look at Slack for an example of an app without tabs but with a search icon in the navigation bar.

In order for a design to work on all mobile devices, and not just 5.5" phones (which I admit also having) we need to ensure focused views, with singular purpose.
We are already trying to fit a Stream + Topic in the nav bar, and it doesn't work that great on small devices.
We have at least 5-6 types of 'lists' of things. They do not naturally fit a left-right panel approach. That is consistent with most other mobile apps too, not unique to Zulip.

@borisyankov I actually use the smaller 4.7" iPhone :)
Yeah I'm not sure what the right answer here is. I know you're in favor of a tab-based approach and there are certainly some advantages to that. My main concern is context. Suppose you have two tabs, "Streams" and "Messages". The "Streams" tab lists all your subscriptions and lets you narrow to a particular stream. But then narrowing will land you onto the "Messages" tab and it's not clear from a user-interaction standpoint that the two screens are coupled. Usually tabs are really nice for independent screens.
Another option is to move the topic and/or stream out of the navigation bar (below it perhaps) and add a search icon to the navigation bar.
@timabbott thoughts?

A few ideas:

What if we had one of the entries in the left sidebar like "Starred messages" be "Search"?
We could put it in our main ... menu that has stuff like "About the app/legal/etc." in it.

If none of these work, I'd probably just put it in the navbar and shrink the font size so things still fit...

I rebased and merged for 5th-6th time.
There is no good reason not to merge.
Currently it is Tim's first suggestion, can be changed later, or course.


Capability of Rearranging and Moving Tabs Accross Multiple Windows

It will be instrumental if Terminal's tabs can be rearranged/reordered and can be move across windows like the browsers tab.
As of now, Terminal's tab can't be dragged and reordered like tabs browser tabs. I often simultaneously work on the terminal's multiple tabs, and sometimes It becomes essential to rearrange the tabs for ease of switching the tabs.

Hi, this is being tracked at #1256


test-branch/execute-job/comment-from-wrong-user-on-pr/d0fb8b81-e4bf-4e6b-8e51-82a0790fbd10

PR for testing 564488b42076d7de41bfbb3f1cbb0d92a13b16d9

/canonical/self-hosted-runners/run-workflows 564488b42076d7de41bfbb3f1cbb0d92a13b16d9


Scheduler design - Input needed

I spent a little time talking with @trxcllnt yesterday about scheduler design. I've also glossed over this briefly with @jhusain and @benjchristensen
I have a REALLY, REALLY primitive impl in the codebase now, but it needs work. In particular it needs a better design.
Right now it's a global scheduler. But you'll notice I added a dispose() method to it. This is because I initially created it as a class, and thought scheduling might be instance based. This would make scheduling more flexible. However, global scheduling is very useful for things like Framework lifecycles (@igorminar, @machty).
I'm looking for input as to how to make this flexible but easy to understand for authors.

 Meaning it doesn't provide gaps for other code execution. This is a bare-metal simplistic impl. I think for most browser-based use cases, adding gaps ala MicroTask scheduling probably isn't completely necessary because the event stream sources themselves will be async, but there's definitely a need for that type of scheduler.

@blesh sounds like you meant to provide a link to the code, no?

Closing this for now. I feel like the Scheduling is in an okay place.


Import Basic Choropleth Feature

To use, run grunt serve and then naviate to http://localhost:8082/examples/choropleth. Note that I've currently disabled multipolygon rendering (line 169 of core/choroplethFeature.js), to demonstrate the performance issues that arise between including and excluding multipolygons.

Looking good, thanks.  We have some strict style rules enforced by jscs you'll need to keep in mind.  You should probably also rebase out https://github.com/OpenGeoscience/geojs/commit/06c8ce30804f526a7b8a913c247b4c0e0e7655ca (a reverted commit from my PR).
I'll fix the selenium tests (a firefox update I think :angry:) and look into the performance issues.

Gotcha, I'll make those style fixes + rebase now.

Looking very good @dcjohnsto. Just has some minor questions.

@dcjohnston The feature classes are designed to have a d3-like interface rather than enforce any specific storage semantics.  Much like d3, we want to support even things like this:
polygon
  .data(d3.range(10))
  .coordinates(function (_, i) {
    // return the ith polygon from some external or derived data source
  });

I hadn't thought about this issue before now, but really choropleths have been supported since the polygon feature was added.  Now that I look into this PR more closely, I think we could do one of two things:

Add a choropleth feature that derives from polygon and just simplifies the interface.  I'm not sure that doing this would be of much value though because there isn't much in the way of simplification possible except for providing different default styles.
Move this code over to a "reader" or a utility method of some sort.  It would make sense to default to geojson for this and provide coloring algorithms as options.

Add a choropleth feature that derives from polygon and just simplifies the interface. I'm not sure that doing this would be of much value though because there isn't much in the way of simplification possible except for providing different default styles.

Same here. I think not having it derived from polygon feature is okay and I am somewhat inclined towards it.

Move this code over to a "reader" or a utility method of some sort. It would make sense to default to geojson for this and provide coloring algorithms as options.

+1

Reading through all of that, now sure if I should be generalizing the choropleth feature to accept data formats of arbitrary structure (as opposed to only geojson), or bumping the class to some method on the geojson reader, thereby implying that the data is to be of a specific structure.

Reading through all of that, not sure if I should be generalizing the choropleth feature to accept data formats of arbitrary structure (as opposed to only geojson), or bumping the class to some method on the geojson reader, thereby implying that the data is to be of a specific structure.

geojs code does not expect a particular format (format agnostic) and I think we should keep it that way as we may need to support other formats and not just geojson in the future. The default function for value can expect d.value and if the user wants to change it then can do so. We can always have adapters that can make it easy for the users.

Agreed, OK I'll generalize using the advice of @aashish24 + @jbeezley

As I mentioned, we already have a generic choropleth class by way of the polygon.  Because this is a common use case, I think this would be more useful as part of a geojson reader with sane defaults for choropleth coloring and interaction.

@jbeezley that is a interesting point. GeoJSON reader produces points, lines and polygons but it does not really create a look up table of anysort for coloring. Also, Choropleth as a high level (as a feature will be useful from the user point of view). If we sub-class from polygon feature, wouldn't that take care of code duplication problem? I understand that user can put pieces together now but I think it would be easier if we have a feature that I can use with minimal effort.

Ideally a lookup table should be available as a utility method or class.  It shouldn't be feature specific.
I think features should be generic and configurable (i.e. not just geojson), this implies using them will take more effort.  What I'm suggesting is a method that accepts geojson containing either an array of polygon geometries or a multipolygon geometry.  There can be some basic options as well for coloring algorithms and lookup tables, but it shouldn't be more generic than that.
This method might best be placed as a reader which is an overloaded term in the current code as readers can generate features from objects as well.  Another possibility is a utility method in geo.util or even as a set of "simple" methods under a new namespace.  I'm open to other ideas, but I don't think creating a new feature (subclassed or not) for this purpose is a good idea.

Ideally a lookup table should be available as a utility method or class. It shouldn't be feature specific.

+1

This method might best be placed as a reader which is an overloaded term in the current code as readers can generate features from objects as well. Another possibility is a utility method in geo.util or even as a set of "simple" methods under a new namespace. I'm open to other ideas, but I don't think creating a new feature (subclassed or not) for this purpose is a good idea.

@jbeezley since @dcjohnston already has a feature, lets go ahead and finish it. We can think about more general option as a next task. This is something we need to deliver to one of our client so I need this soon.

In that case, I will echo @dcjohnston's confusion on what exactly it is that you want.  I have concerns pushing code just to check a box on a task list with the intention of completely changing it later.

In that case, I will echo @dcjohnston's confusion on what exactly it is that you want. I have concerns pushing code just to check a box on a task list with the intention of completely changing it later.

My only suggestion for this branch is not to use assume geojson feature (for value accessor). And we can fix the performance issue than that would be great. I think this feature is useful as it is. The question of sub-classing or having some code in util is something we can discuss later.

Also, if we can put some of these function in util class and this feature uses them, that also sounds good to me as well.

The geometries in this class assume the data is given as geojson, and those aren't configurable.  Why would the default value accessor assume anything different?  You can always change it using the choropleth method if it is.

okay, @dcjohnston.How about this, let you, @jbeezley and me talk on monday? If that's okay. Until then hold on working on this branch. I think we need bit more thinking.

Sounds good! Sorry for the confusion.

Oh please don't :smile: I think what you have done is great plus you know GeoJS now more than before. I have an idea that will address some of the concerns we discussed here and hopefully won't require much work. It is important to discuss some of these things (this is true for new features in general).

Quick update: I reimplemented the multipolygon rendering, so that each multipolygon creates only one new feature, however the performance is about as bad as before. I was wondering, would it make sense to group all the polygons that have the same fill value (based on choropleth scale), and then have each of these groups of identically-colored polygons be added to a single polygon feature? This seems like the path to fewest features, which could help with performance.

I think you implemented it correctly.  Can you post or a link the geojson file you are testing with?

http://eric.clst.org/wupl/Stuff/gz_2010_us_040_00_20m.json

The performance gap i mentioned above is that given two states, for example, that have the same fill value (i.e. same scalar value), they each get their own polygon feature. It seems like it would be better for those two states to be passed into a single new polygon feature, so that there would be exactly one polygon feature per color.

I'm looking into it now.  I think I know what you mean, but you do have to be careful with how you use the term feature to avoid confusion.  A polygon feature is one or more geojson polygon geometries.  I think you are talking about merging two polygon geometries into a single geometry if they are displayed with the same color.  That would probably cause a few problems internally, and I doubt it would do much to improve the performance.
I'll let you know what I figure out.

I merged this branch with master because there were some incompatibilities with the vgl you were using.  With the new branch head, I'm not seeing any performance issues with the file you linked.  Could you give another try?  Make sure to git submodule update to ensure you are using the correct version of vgl.

Blazing fast :fire: ! Working great with the updated VGL. I'm plugging in the .scalar method to separate geodata from the scalar values used for coloring, and then I'll look at core/gl/choroplethFeature, since that definitely needs work. Thanks

This is awesome @dcjohnston!

Quick question: for the .scalar method, do we want to allow non bijective correspondance between scalar data and geo-data? I ask because, while it seems like it would offer more flexibility from the end-user's perspective, I'm not sure if there's a consistent best solution for how to aggregate the scalar values for the given feature. For instance, if we allows a "state" to have 10 scalar data points, do we determine it's color based on the average of these, the median of these, the mode? We could set average as default, but at this point, by adding a separate method for inputting scalar values, we're going to have a ton of customizations that the user will need to be aware of when the defaults aren't appropriate. I don't think there's anything inherently bad with that, but it seemed like the point of the feature was to make a quick and out of the box choropleth.

Quick question: for the .scalar method, do we want to allow non bijective correspondance between scalar data and geo-data? I ask because, while it seems like it would offer more flexibility from the end-user's perspective, I'm not sure if there's a consistent best solution for how to aggregate the scalar values for the given feature. For instance, if we allows a "state" to have 10 scalar data points, do we determine it's color based on the average of these, the median of these, the mode? We could set average as default, but at this point, by adding a separate method for inputting scalar values, we're going to have a ton of customizations that the user will need to be aware of when the defaults aren't appropriate. I don't think there's anything inherently bad with that, but it seemed like the point of the feature was to make a quick and out of the box choropleth.

Can we not have the flexibility so that user can see all of those 10 scalars for a particular boundary? so
for a csv like this: id scalar1 scalar2 scalar3 we get scalar1, scalar2, scalar3 in the function and then let user do whatever they want to do with that (via object).

Well, I think we can add a "custom aggregator" no problem. But it's necessary to define some sort of aggregation method, because otherwise how would you define the value to put into the color scalefor the region?
In that last commit I'm just using average, but we could just make it another conf. option, defaulting to 'mean', but accepting 'median', and 'custom', which would allow the user to pass in a custom function to determine the aggregation method (maybe certain subsets of the scalar data could be more heavily weighted, for instance).

In that last commit I'm just using average, but we could just make it another conf. option, defaulting to 'mean', but accepting 'median', and 'custom', which would allow the user to pass in a custom function to determine the aggregation method (maybe certain subsets of the scalar data could be more heavily weighted, for instance).

Having another conf option or second argument to scalar might work. What you think of seconfd argument to scalar? If none provided you can assume average.

@dcjohnston the tests are failing because of the baseline missing is that correct? @jbeezley are you going to upload new baseline?

There are also some style issues, but I can take care of the baseline.

@dcjohnston @jbeezley any update on the baseline and style issues?

I'll take a look at it today.

The changes and API looks great to me. Thanks @dcjohnston @jbeezley. I updated the branch with current master. Let's wait for the buildbot and then if all good, we can merge it.

LGTM :+1: :clap: @dcjohnston I am going to merge it (not sure if you are around).

Wonderful! I'm happy to get some of my code in such a cool project. Looking forward to building out more features soon!

Thanks @dcjohnston, we're happy to have the help.


Upgrade rust toolchain for libs/libsymbolization

Signed-off-by: Zhigao Tong<EMAIL_ADDRESS>What problem does this PR solve?
Issue Number: close #6014
What is changed and how it works?
M1 pro was published on 2021/10, maybe there is something incompatible.
Upgrade rust toolchain of libsymbolization to nightly-2022-07-31 like pingcap/tidb-engine-ext.
Check List
Tests

[ ] Unit test
[ ] Integration test
[ ] Manual test (add detailed scripts or steps below)
[ ] No code

Side effects

[ ] Performance regression: Consumes more CPU
[ ] Performance regression: Consumes more Memory
[ ] Breaking backward compatibility

Documentation

[ ] Affects user behaviors
[ ] Contains syntax changes
[ ] Contains variable changes
[ ] Contains experimental features
[ ] Changes MySQL compatibility

Release note

None

/merge

/hold

/merge

/unhold

/merge

/rebuild

/run-integration-test

/rebuild

/run-unit-test

/merge

/run-unit-test

/run-unit-test

/run-unit-test


ModelView Dialog onClose event

Useful for cleaning up resources used by the dialog when it's been closed.

Closing older extension API bugs.  Please log a customer-facing bug for the impacted scenario, or reactive if you are planning to address in an upcoming release.


Schedule long-running test failure - TestPersistentVolume

Steps to reproduce
Seen during long running test run: https://github.com/radius-project/radius/issues/7175
Observed behavior
cli.go:418: [rad]     {
    cli.go:418: [rad]       "code": "ResourceDeploymentFailure",
    cli.go:418: [rad]       "message": "Failed",
    cli.go:418: [rad]       "target": "/planes/radius/local/resourceGroups/kind-radius/providers/Applications.Core/containers/volume-azkv-ctnr",
    cli.go:418: [rad]       "details": [
    cli.go:418: [rad]         {
    cli.go:418: [rad]           "code": "Internal",
    cli.go:418: [rad]           "message": "PUT [https://management.azure.com/subscriptions/***/resourceGroups/radtest-efdf64ef15-e2e-all/providers/Microsoft.ManagedIdentity/userAssignedIdentities/corerp-resources-volume-azure-keyvault-volume-azkv-ctnr/federatedIdentityCredentials/volume-azkv-ctnr\n--------------------------------------------------------------------------------\nRESPONSE](https://management.azure.com/subscriptions/***/resourceGroups/radtest-efdf64ef15-e2e-all/providers/Microsoft.ManagedIdentity/userAssignedIdentities/corerp-resources-volume-azure-keyvault-volume-azkv-ctnr/federatedIdentityCredentials/volume-azkv-ctnr/n--------------------------------------------------------------------------------/nRESPONSE) 409: 409 Conflict\nERROR CODE: ConcurrentFederatedIdentityCredentialsWritesForSingleManagedIdentity\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"ConcurrentFederatedIdentityCredentialsWritesForSingleManagedIdentity\",\n    \"message\": \"Too many Federated Identity Credentials are written concurrently for the managed identity '/subscriptions/***/resourcegroups/radtest-efdf64ef15-e2e-all/providers/microsoft.managedidentity/userassignedidentities/corerp-resources-volume-azure-keyvault-volume-azkv-ctnr'. Concurrent Federated Identity Credentials writes under the same managed identity are not supported.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
    cli.go:418: [rad]         }
    cli.go:418: [rad]       ]
    cli.go:418: [rad]     }
    cli.go:418: [rad]   ]
    cli.go:418: [rad] }

logs_105812.zip
Desired behavior
This test should pass without running into concurrent issues.
Workaround
It's possible this could be fixed with retries, but it may also be something that the managed identity service would need to change. There's been similar issues logged with other repos that have some workarounds if this issue comes up more and more: https://github.com/Azure/bicep/issues/10928
rad Version
long-running test
Operating system
No response
Additional context
No response
Would you like to support us?

[ ] Yes, I would like to support you

AB#11209

Initial Investigation: It seems that handling creation of managed identities through the code and not through bicep files. So, adding the dependsOn field isn’t as simple as changing a bicep file for Radius. Probably a change is required in how we create managed identities in federated identity handler or in azure client. cc @sk593

To clarify, this investigation was only based on one of the workarounds (adding the dependsOn field) in the linked issue. Since we don't create managed identities through bicep files, we'd need to change code to mimic this

Initial Investigation: It seems that handling creation of managed identities through the code and not through bicep files. So, adding the dependsOn field isn’t as simple as changing a bicep file for Radius. Probably a change is required in how we create managed identities in federated identity handler or in azure client. cc @sk593

To clarify, this investigation was only based on one of the workarounds (adding the dependsOn field) in the linked issue. Since we don't create managed identities through bicep files, we'd need to change code to mimic this

We must use optimistic concurrency using etag to avoid this problem in this code - https://github.com/radius-project/radius/blob/275b861093057c9ded559b2ad36487647cab81a1/pkg/corerp/handlers/azure_federatedidentity.go#L150-L160


Cannot include code block in numbered list. Callout works a bit better.

Describe the bug
It is about the numbered lists where I can not insert a code block style. The code block style is greyed out in a numbered list.
Steps To Reproduce
See first screenshot. I did copy that from the Docker website.
Bookstack does display this without any problems.
However if I try to reproduce this manually, it is NOT possible.

Create numbered list
shift enter
try to insert a code block. Impossible. Greyed out in "Formats" menu, OR the code block takes the complete part of the numbered list as input, which is of course not the intention.

Expected behavior
I can insert a code block in a numbered/bulleted list.
Screenshots
If applicable, add screenshots to help explain your problem.

Your Configuration (please complete the following information):

Exact BookStack Version (Found in settings): 0.26.2
PHP Version: 7.3
Hosting Method (Nginx/Apache/Docker): Docker, linuxserver/bookstack image.

Additional context

Thanks for the clear description of this @MarsWarrior.
As it stands, Code blocks are block level elements which are currently not supported within list items due to the formatting and editor complexities it can introduce, but I can see why you may want this feature.
Since this has already been requested in #534 I'm going to close this off as a duplicate. Adding a :+1: to the original post there will help show support.