HDDS-2121. Create a shaded ozone filesystem (client) jar

We need a shaded Ozonefs jar that does not include Hadoop ecosystem components (Hadoop, HDFS, Ratis, Zookeeper).
A common expected use case for Ozone is Hadoop clients (3.2.0 and later) wanting to access Ozone via the Ozone Filesystem interface. For these clients, we want to add Ozone file system jar to the classpath, however we want to use Hadoop ecosystem dependencies that are provided and already expected to be in the client classpath.
Note that this is different from the legacy jar which bundles a shaded Hadoop 3.2.0.
See: https://issues.apache.org/jira/browse/HDDS-2121

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
37
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

-1
mvninstall
33
hadoop-ozone in trunk failed.

-1
compile
20
hadoop-ozone in trunk failed.

+1
mvnsite
0
trunk passed

+1
shadedclient
950
branch has no errors when building and testing our client artifacts.

+1
javadoc
158
trunk passed

_ Patch Compile Tests _

-1
mvninstall
35
hadoop-ozone in the patch failed.

-1
compile
26
hadoop-ozone in the patch failed.

-1
javac
26
hadoop-ozone in the patch failed.

+1
mvnsite
0
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
2
The patch has no ill-formed XML file.

+1
shadedclient
672
patch has no errors when building and testing our client artifacts.

+1
javadoc
153
the patch passed

_ Other Tests _

-1
unit
128
hadoop-hdds in the patch failed.

-1
unit
27
hadoop-ozone in the patch failed.

+1
asflicense
31
The patch does not generate ASF License warnings.

2520

Reason
Tests

Failed junit tests
hadoop.ozone.container.ozoneimpl.TestOzoneContainer

hadoop.ozone.container.keyvalue.TestKeyValueContainer

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1452

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml

uname
Linux b538116f5112 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 85b1c72

Default Java
1.8.0_222

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/2/artifact/out/branch-mvninstall-hadoop-ozone.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/2/artifact/out/branch-compile-hadoop-ozone.txt

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/2/artifact/out/patch-mvninstall-hadoop-ozone.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/2/artifact/out/patch-compile-hadoop-ozone.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/2/artifact/out/patch-compile-hadoop-ozone.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/2/artifact/out/patch-unit-hadoop-hdds.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/2/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/2/testReport/

Max. process+thread count
412 (vs. ulimit of 5500)

modules
C: hadoop-ozone/ozonefs-lib-current U: hadoop-ozone/ozonefs-lib-current

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/2/console

versions
git=2.7.4 maven=3.3.9

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
77
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

-1
mvninstall
30
hadoop-ozone in trunk failed.

-1
compile
19
hadoop-ozone in trunk failed.

+1
mvnsite
0
trunk passed

+1
shadedclient
1056
branch has no errors when building and testing our client artifacts.

+1
javadoc
163
trunk passed

_ Patch Compile Tests _

-1
mvninstall
30
hadoop-ozone in the patch failed.

-1
compile
21
hadoop-ozone in the patch failed.

-1
javac
21
hadoop-ozone in the patch failed.

+1
mvnsite
0
the patch passed

+1
whitespace
1
The patch has no whitespace issues.

+1
xml
2
The patch has no ill-formed XML file.

+1
shadedclient
733
patch has no errors when building and testing our client artifacts.

+1
javadoc
150
the patch passed

_ Other Tests _

-1
unit
157
hadoop-hdds in the patch failed.

-1
unit
26
hadoop-ozone in the patch failed.

+1
asflicense
29
The patch does not generate ASF License warnings.

2734

Reason
Tests

Failed junit tests
hadoop.ozone.container.keyvalue.TestKeyValueContainer

hadoop.ozone.container.ozoneimpl.TestOzoneContainer

Subsystem
Report/Notes

Docker
Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1452

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml

uname
Linux 407f30b66d7b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 85b1c72

Default Java
1.8.0_212

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/1/artifact/out/branch-mvninstall-hadoop-ozone.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/1/artifact/out/branch-compile-hadoop-ozone.txt

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/1/artifact/out/patch-mvninstall-hadoop-ozone.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/1/artifact/out/patch-compile-hadoop-ozone.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/1/artifact/out/patch-compile-hadoop-ozone.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/1/artifact/out/patch-unit-hadoop-hdds.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/1/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/1/testReport/

Max. process+thread count
306 (vs. ulimit of 5500)

modules
C: hadoop-ozone/ozonefs-lib-current U: hadoop-ozone/ozonefs-lib-current

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/1/console

versions
git=2.7.4 maven=3.3.9

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

The build seems to be failing in Jenkins. However I am able to compile Ozone locally with your patch.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
51
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

-1
mvninstall
34
hadoop-ozone in trunk failed.

-1
compile
22
hadoop-ozone in trunk failed.

+1
mvnsite
0
trunk passed

+1
shadedclient
1150
branch has no errors when building and testing our client artifacts.

+1
javadoc
178
trunk passed

_ Patch Compile Tests _

-1
mvninstall
35
hadoop-ozone in the patch failed.

-1
compile
24
hadoop-ozone in the patch failed.

-1
javac
24
hadoop-ozone in the patch failed.

+1
mvnsite
0
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
1
The patch has no ill-formed XML file.

+1
shadedclient
799
patch has no errors when building and testing our client artifacts.

+1
javadoc
178
the patch passed

_ Other Tests _

-1
unit
250
hadoop-hdds in the patch failed.

-1
unit
29
hadoop-ozone in the patch failed.

+1
asflicense
35
The patch does not generate ASF License warnings.

3069

Reason
Tests

Failed junit tests
hadoop.hdds.scm.container.placement.algorithms.TestSCMContainerPlacementRackAware

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1452

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml

uname
Linux 48f13474670c 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / eefe9bc

Default Java
1.8.0_222

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/4/artifact/out/branch-mvninstall-hadoop-ozone.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/4/artifact/out/branch-compile-hadoop-ozone.txt

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/4/artifact/out/patch-mvninstall-hadoop-ozone.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/4/artifact/out/patch-compile-hadoop-ozone.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/4/artifact/out/patch-compile-hadoop-ozone.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/4/artifact/out/patch-unit-hadoop-hdds.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/4/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/4/testReport/

Max. process+thread count
553 (vs. ulimit of 5500)

modules
C: hadoop-ozone/ozonefs-lib-current U: hadoop-ozone/ozonefs-lib-current

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/4/console

versions
git=2.7.4 maven=3.3.9

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
2872
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

-1
mvninstall
38
hadoop-ozone in trunk failed.

-1
compile
24
hadoop-ozone in trunk failed.

+1
mvnsite
0
trunk passed

+1
shadedclient
1210
branch has no errors when building and testing our client artifacts.

+1
javadoc
168
trunk passed

_ Patch Compile Tests _

-1
mvninstall
30
hadoop-ozone in the patch failed.

-1
compile
22
hadoop-ozone in the patch failed.

-1
javac
22
hadoop-ozone in the patch failed.

+1
mvnsite
0
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
1
The patch has no ill-formed XML file.

+1
shadedclient
727
patch has no errors when building and testing our client artifacts.

+1
javadoc
152
the patch passed

_ Other Tests _

+1
unit
270
hadoop-hdds in the patch passed.

-1
unit
25
hadoop-ozone in the patch failed.

+1
asflicense
30
The patch does not generate ASF License warnings.

5807

Subsystem
Report/Notes

Docker
Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1452

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml

uname
Linux 629d1f742a64 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / f580a87

Default Java
1.8.0_222

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/5/artifact/out/branch-mvninstall-hadoop-ozone.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/5/artifact/out/branch-compile-hadoop-ozone.txt

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/5/artifact/out/patch-mvninstall-hadoop-ozone.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/5/artifact/out/patch-compile-hadoop-ozone.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/5/artifact/out/patch-compile-hadoop-ozone.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/5/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/5/testReport/

Max. process+thread count
467 (vs. ulimit of 5500)

modules
C: hadoop-ozone/ozonefs-lib-current U: hadoop-ozone/ozonefs-lib-current

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/5/console

versions
git=2.7.4 maven=3.3.9

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
2470
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

-1
mvninstall
33
hadoop-ozone in trunk failed.

-1
compile
21
hadoop-ozone in trunk failed.

+1
mvnsite
0
trunk passed

+1
shadedclient
981
branch has no errors when building and testing our client artifacts.

-1
javadoc
50
hadoop-ozone in trunk failed.

_ Patch Compile Tests _

-1
mvninstall
33
hadoop-ozone in the patch failed.

-1
compile
25
hadoop-ozone in the patch failed.

-1
javac
25
hadoop-ozone in the patch failed.

+1
mvnsite
0
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
1
The patch has no ill-formed XML file.

+1
shadedclient
667
patch has no errors when building and testing our client artifacts.

-1
javadoc
49
hadoop-ozone in the patch failed.

_ Other Tests _

-1
unit
277
hadoop-hdds in the patch failed.

-1
unit
29
hadoop-ozone in the patch failed.

+1
asflicense
33
The patch does not generate ASF License warnings.

5053

Reason
Tests

Failed junit tests
hadoop.hdds.scm.container.placement.algorithms.TestSCMContainerPlacementRackAware

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/6/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1452

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml

uname
Linux fef328d63025 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 5db32b8

Default Java
1.8.0_222

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/6/artifact/out/branch-mvninstall-hadoop-ozone.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/6/artifact/out/branch-compile-hadoop-ozone.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/6/artifact/out/branch-javadoc-hadoop-ozone.txt

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/6/artifact/out/patch-mvninstall-hadoop-ozone.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/6/artifact/out/patch-compile-hadoop-ozone.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/6/artifact/out/patch-compile-hadoop-ozone.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/6/artifact/out/patch-javadoc-hadoop-ozone.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/6/artifact/out/patch-unit-hadoop-hdds.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/6/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/6/testReport/

Max. process+thread count
519 (vs. ulimit of 5500)

modules
C: hadoop-ozone/ozonefs-lib-current U: hadoop-ozone/ozonefs-lib-current

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1452/6/console

versions
git=2.7.4 maven=3.3.9

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

I assume all the test failures are unrelated. There is one SmokeTest failure for MapReduce.
@bharatviswa504 ok to commit this?

`$ ./test.sh
Removing network hadoop32_default
WARNING: Network hadoop32_default not found.
Creating network "hadoop32_default" with the default driver
Creating hadoop32_scm_1      ... done
Creating hadoop32_datanode_1 ... done
Creating hadoop32_datanode_2 ... done
Creating hadoop32_datanode_3 ... done
Creating hadoop32_nm_1       ... done
Creating hadoop32_rm_1       ... done
Creating hadoop32_s3g_1      ... done
Creating hadoop32_om_1       ... done
3 datanodes are up and registered to the scm
hadoop32-createmrenv :: Create directories required for MR test
Create test volume, bucket and key                                    | PASS |
Create user dir for hadoop                                            | PASS |
hadoop32-createmrenv :: Create directories required for MR test       | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
Output:  /tmp/smoketest/hadoop32/result/robot-hadoop32-hadoop32-createmrenv-scm.xml
hadoop32-hadoopo3fs :: Test ozone fs with hadoopfs
Test hadoop dfs                                                       | PASS |
hadoop32-hadoopo3fs :: Test ozone fs with hadoopfs                    | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
Output:  /tmp/smoketest/hadoop32/result/robot-hadoop32-hadoop32-hadoopo3fs-rm.xml
hadoop32-mapreduce :: Execute MR jobs
Execute PI calculation                                                | PASS |
Execute WordCount                                                     | PASS |
hadoop32-mapreduce :: Execute MR jobs                                 | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
Output:  /tmp/smoketest/hadoop32/result/robot-hadoop32-hadoop32-mapreduce-rm.xml
Stopping hadoop32_s3g_1      ... done
Stopping hadoop32_datanode_1 ... done
Stopping hadoop32_nm_1       ... done
Stopping hadoop32_datanode_3 ... done
Stopping hadoop32_datanode_2 ... done
Stopping hadoop32_om_1       ... done
Stopping hadoop32_scm_1      ... done
Stopping hadoop32_rm_1       ... done
Removing hadoop32_s3g_1      ... done
Removing hadoop32_datanode_1 ... done
Removing hadoop32_nm_1       ... done
Removing hadoop32_datanode_3 ... done
Removing hadoop32_datanode_2 ... done
Removing hadoop32_om_1       ... done
Removing hadoop32_scm_1      ... done
Removing hadoop32_rm_1       ... done
Removing network hadoop32_default
Log:     /Users/bviswanadham/workspace/myozone/hadoop/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone-mr/hadoop32/result/log.html
Report:  /Users/bviswanadham/workspace/myozone/hadoop/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone-mr/hadoop32/result/report.html`
Ran it locally test passed, the test failure is not related to this patch.
I will commit this to the trunk. Thank You @elek for the improvement.


can not open it

can not open it with MI 8SE sos!

Please explain in detail.

I cloned your project directly, and it was stuck on the loading interface.

Ok. You need to create separate firebase project.


Resubmit: Port #9488 and part of #9463

Resubmit of #9606 and fixed the template problem.

Test failures:
gRPC_performance_pull_requests - #9499


Derivative problem

Hello,
I`m trying to run following query against my database:
select derivative("server01.value" + "server02.value") from "server01" INNER JOIN "server02" group by time(1m)
However I don`t get any output, graph, message etc. Literally nothing. Is this a bug, or wrong query?
If this is a bug, then is there any work around?
My InfluxDB version is 0.8.6.
Thank you

Possible duplicate of #438.

@allertec try adding fill(0) to your query.
I find without fill(0) or fill(null) there is nothing returned.


fix(chart): calculate Gauge Chart intervals correctly when min value is set

SUMMARY

Fixes #21328.
If you create a Gauge Chart with interval bounds and a minimum value set, the color boundaries are placed incorrectly. This PR changes the boundary calculation to take the minimum value into account during normalization by using the formula described here.
I've also updated the part of the test file where the interval calculation gets tested.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Before (boundary between green and red should be at value 10 but isn't):

After:

TESTING INSTRUCTIONS

Create a new Gauge chart, e.g. with exported_stats example dataset
Select AVG(daily_members_posting_messages) as Metric
On Customize tab, set INTERVAL BOUNDS to 10, 16.53 and INTERVAL COLORS to 3, 6
Set MIN to 8
Now the boundary between green and red gets updated with the rest of the chart

ADDITIONAL INFORMATION

[x] Has associated issue: #21328
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

I was hoping to spin up a test environment here, but the docker build is failing. A rebase might fix that if warranted. Thanks @villebro for testing locally.

Thanks @villebro for the feedback! I've renamed the function and added some tests specifically for it, please have a look and let me know what you think.
@rusackas I've rebased the branch, maybe you can try the docker build again.

Thanks @goto-loop for the great tests and addressing the function name! Fingers crossed CI passes 🤞


Implement Area[2D/3D].has_overlapping_[bodies/areas]

Most of my uses of get_overlapping_bodies() are to find whether there are any collisions at all by using is_empty(). The problem is that this has to resize an array and dump all bodies into it, which is less performant :/  So I implemented this method that avoids all this heavy-lifting, as I think it will be useful to a lot of users.

6.5x faster than !get_overlapping_bodies.is_empty() (7,000 runs: 10.3ms --> 1.6ms)

8x faster than get_overlapping_bodies() != [] (7,000 runs: 12.4ms --> 1.6ms)

Although it's about performance, it's still a feature proposal I suppose?

Thanks!


docker login fail and cannot pull any image

Describe the bug
When I executed the command docker compose up --build -d, the command was not completed and was forcibly terminated in the middle. This situation has been happening since then. When I try to login, I cannot log in, and I cannot pull any images. .The same thing happened after reinstalling orbstack. I also tried using a proxy or deleting osxkeychain, etc., but it still couldn't be used normally.
When I switch to docker-desktop, I can log in normally, pull the image, and build and run.
As a result, I had to switch back to docker desktop.
To Reproduce

Run docker compose up --build -d
When pulling the image is halfway through execution, it is forcibly interrupted.

I don’t know if the above steps can be reproduced, but this is how I did it.
Expected behavior
can log in normally, pull normally, build images, and run containers.
Diagnostic report (REQUIRED)
OrbStack info:
Version: 1.2.0
Commit: 6bebbe07a1ebc2afb1ce113700b8514527316eda (v1.2.0)
System info:
macOS: 13.5.2 (22G91)
CPU: arm64, 8 cores
CPU model: Apple M1
Model: MacBookAir10,1
Memory: 16 GiB
Full report: https://orbstack.dev/_admin/diag/orbstack-diagreport_2023-12-28T02-53-49.277431Z.zip

Screenshots and additional context (optional)
use obtstack

use docker desktop

docker login runs on macOS, not in OrbStack. EOF means there was an issue with your proxy, likely http://<IP_ADDRESS>:7890.
Also, if you're using Surge, see #529

Regarding the link you gave me, I do have a local proxy, but the tun mode is not turned on. This has never happened in previous versions. I now manually change the proxy to none and it is indeed better. Thank you so much


Add option to select jp-tok region

Description

Ensure that all services in the stack can deploy in that region.
If this works - add support for Japan jp-tok region at the stack level https://github.com/terraform-ibm-modules/stack-retrieval-augmented-generation/blob/df61899f03da8e2f98ed26948eb93a92a8a375b7/ibm_catalog.json#L219

@vburckhardt Watsonx SaaS DA only supports us-south and eu-de
https://github.com/terraform-ibm-modules/terraform-ibm-watsonx-saas-da/blob/d243f4db0960e337c45e6c940659bdcdf2256748/variables.tf#L14

@Soaib024 - I think watson services can be deployed in jp however, and if so, the approach here would be to update the watsonx saas da.

Occupied with some other works, so unassigning for now

From @shemau :

Today, the only service that does not support jp-tok is watson.governance. It only supports eu-de and us-south.

So I think we cannot add jp-tok support to the stack

indeed, only eu-de and us-south supported https://github.com/terraform-ibm-modules/terraform-ibm-watsonx-saas-da/blob/main/main.tf#L180

No support in watson.governance so we cannot support in the stack - closing


[AMBARI-24260] Ambari shows success when Decommission/Recommission op…

…erations fail (dsen)
What changes were proposed in this pull request?
Added the state change revert if the decommission failed
How was this patch tested?
Manually

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/3097/
Test PASSed.

Closing PR. The issue will be fixed on the UI side.


Wikidata: extract shape-based API abstraction into package

plugin-wikidata has some code that allows for it to fetch nested entities from the Wikidata without SPARQL, through as few requests as possible. This code should be extracted into a reusable component. Optionally support for ShEx (shape expressions) would be nice, as they are more standard than the current nested object format, though also a bit more overkill.

@Daniel-Mietchen I mentioned the data structure used to specify which entities to fetch but I just found that there'a actually a nearly complete machine-readable mapping in the repository already: https://github.com/citation-js/citation-js/blob/main/packages/plugin-wikidata/src/props.json


Revert "fixed requires main queue setup warning"

Reverts Adyen/adyen-react-native#204

@pinpong
This PR crashes iOS build, because NSObject do not override requiresMainQueueSetup.
Fix:
final internal class AdyenCSE: NSObject, RCTBridgeModule {


Horizontal scrolling and truncation for "Typed Errors" page on mobile

https://arrow-kt.io/learn/typed-errors/

The "Coroutines" page exhibits the same. Let me know if you want me to file this as a separate issue.
https://arrow-kt.io/learn/coroutines/


Setting "allowTLS: true" fails the helm install

Describe the bug
Using Amazon DocumentDB and setting allowTLS: true results in the following error
Helm install failed: Deployment in version "v1" cannot be handled as a Deployment: json: cannot unmarshal bool into Go struct field EnvVar.spec.template.spec.containers.env.value of type string Last Helm

To Reproduce
Set values

testkube-api:
  mongodb:
    dsn: "<dsn>"
    allowDiskUse: false
    dbType: docdb
    allowTLS: true

Expected behavior
The helm chart should install without any errors
Version / Cluster

Which testkube version? 1.12.5
Which Helm Chart version? 1.12.1
What Kubernetes cluster? AWS EKS
What Kubernetes version? 1.23

Additional context
The helm chart installs if allowTLS is not set.
testkube-api:
  mongodb:
    dsn: "<dsn>"
    allowDiskUse: false
    dbType: docdb

thank you @michaelsatish will try to reproduce

@ypoplavs Thank you so much for the fix. I am able to use DocumentDB with TLS.


Installing locally

With the imminent closure of Google apps free tier, I'm looking for something to be able  to read my gmail takout mbox files (I don't want to import them into another mail viewer)
This is it| It's just perfect for what I am looking for.
Except: I'd like to run it locally.
I've had a quick look at dropping the files behind a webserver, and accessing the index.html in the public folder. Which doesn't work for me. I'm not really familiar with js, or the requirements here.
Would it be possible for some simple outline pointers, to get me up and running with a local install (for privacy and security reasons)?

Sure! Building it is a bit of a pain. I can probably make a release .zip file. It’s probably not working because you’re missing some files that get created when it’s built.
Let me throw something together…

Any update on this? Would prefer to run this locally if possible.


Smarter file retrieval

[x] generate rosetta folder structure as JSON at build time and use that, instead of querying the file system on demand
[x] pull markdown files directly from Github instead of local file system (should be faster + caching/CDN)

Done as part of https://github.com/abstractalgo/rosetta/commit/e3ec09fd65edc4605d635cf317f1655cba6afa6b

First item done as part of https://github.com/abstractalgo/rosetta/commit/6d1b13255d2e4d5f264ff5c4cd1d6278e29f53e5


Context issue when HTTPS is used in MITM mode

Fixed the issue which goproxy doesn't provide correct HTTP request information from the context when HTTPS MITM is used.

@g3rk6 Thanks! Great catch, since I usually use the request function parameter, I didn't check the context.


Java config docs: default-value should be immutable

I think it would be worth changing the wording to make it clearer that you're not talking about the user setting the default to a new value, but instead modifying the underlying object, i.e. the risk isn't that they might say myConfigKey.setDefaultValue(newValue), but rather myListConfigKey.getDefaultValue().add(foo). Perhaps something like:
If supplying an object as a default value, it is important that the object is immutable. Otherwise, it risks users of the blueprint modifying the underlying object, which would affect blueprints that are subsequently deployed.

Thanks for the addition @aledsage, do you think this should replace @bostko's #171?

LGTM, merging.


Member access level not updated on transfer

Problem
When a user is a member in Group A as a Maintainer, and the user is a member of Group B as a Guest and then Group B is transferred into the Group A namespace, the member roles are not updated correctly and the user is left with Maintainer access in Group A and Guest access in Group B. This should not be happening as the access level should propagate into descendants starting at the parent

Note this issue also applies to a project transfer
To verify:
Create 2 groups Group A and Group B, add<EMAIL_ADDRESS>as a Maintainer on Group A, add<EMAIL_ADDRESS>as a Guest on Group B. Now transfer Group B so that it's parent is Group A.  Visit the members page of Group A and you should see<EMAIL_ADDRESS>as a Maintainer. Now visit the members page of Group B and you should see<EMAIL_ADDRESS>as a Guest
Solution
In the transfer service for groups and projects, if and only if the group has successfully been transferred to a new parent, we want to get the Member relationships for the new parent and it's ancestors. For each one of these member relationships we want go through the descendants of the new parent and update the memberships so that the access level is ascending through the descendants. This should ensure that at no point is the access level higher in an ancestor membership than it is in a descendant membership
Couple of suggestions:

To get the namespace and ancestor member relationships we can user the Member.for_namespace_and_ancestors scope which will accept a project namespace or group

To get the descendant relationships we can use namespace.self_and_descendants

We should most likely be able to use the update_descendant_memberships method in the member model to update descendant memberships for a user

We will probably want to set this up as a job that fires off once the parent has been updated and runs in the background as there could be a very large number of members

Break this issue up into smaller pieces ie; updating of group transfer memberships, updating of project transfer memberships, setting up job to update descendant memberships

Acceptance criteria
After transferring a group or project under a new parent where the user was a member in the previous parent or it's ancestors, and is also a member in the new parent or it's ancestors, any membership for the user in the parent and it's descendants should be in descending order ie; the membership access level in a subgroup or project should be equal to or greater than the highest access level in the parent or it's ancestors

Fixed.


Any unwanted assumptions in relationship between METS files and storage manifests?

Scenario: given an identifier (e.g., b12345678), I want to make something that requires that identifier's METS file(s) and the contents of other files linked from the METS (such as METS-ALTO).
One instance of this scenario is creating an IIIF Manifest, given an identifier and the storage service API.
We need to make sure we know what assumptions we're making about how this will be consumed.
"files": [
    {
      "type": "File",
      "checksum": "a20eee40d609a0abeaf126bc7d50364921cc42ffacee3bf20b8d1c9b9c425d6f",
      "path": "data/b24923333.xml"      // << How do I know this is the METS file? Convention?
    },
    {..},
    {..},
    {
      "type": "File",
      "checksum": "17c0147413b0ba8099b000fc91f8bc4e67ce4f7d69fb5c2be632dfedb84aa502",
      "path": "data/alto/b24923333_001.xml"
//     ^^^^ can I always infer this from METS? - xlink:href="alto/b24923333_001.xml"
    }
]
....

....
"locations": [
  {..},
  {..},
  {
    // how do I know I should get files from this location and not the others?
    // (what marks this as the access bucket? locationType.id?)
    "type": "DigitalLocation",
    "locationType": {
      "id": "aws-s3-standard-ia",
      "label": "AWS S3 Standard IA",
      "type": "LocationType"
    },
    "url": "s3://accessbucket/digitised/b24923333/"  // gives me the root
  }
]

Process:

Fetch the storage manifest for the given identifier.
Identify the METS entry in the storage manifest. How? Convention that the METS will have the path data/{identifier}.xml?
Fetch the METS file from S3 by concatenating the bucket url with the path (how do I know which bucket url to pick?)
Process the METS file, following links and fetching further files (e.g., manifestations, ALTO).
I'm walking the METS XML structure, not the storage manifest. Links in the METS are relative to the current file. Can I assume that I know the S3 path of any file from its relative METS link, because that layout has been preserved in the bag? If I can't assume this, I need to find the path in the storage manifest. But then, what do I search for, besides path?

Closing as 💀


support pydantic v2

Also supports pydantic v1.
pydantic v2 provides v1 api through pydantic.v1 import, so we can support both versions by trying to import from pydantic.v1 and fallback to old behaviour.
See https://docs.pydantic.dev/latest/migration/#continue-using-pydantic-v1-features.
pydantic == 2.0.0 does not work due to https://github.com/pydantic/pydantic/issues/6361, so we need to blacklist that specific version.

@skshetry is there a reason to support both?

@skshetry is there a reason to support both?

Studio still relies on MLEM and therefore on Pydantic v1. Also I did not want to block any DVC users that are using pydantic v2 (or, pydantic v1 even).
Also, using pydantic v2 APIs means changing code, whereas if we use v1 APIs for now, we can support both and is working perfectly fine.

sounds good. can we run tests with both then?

I consider pydantic v1 support as best effort from now on. I don’t think we need to test both.

sound good, I think Studio has enough tests to detect failure if there is a regression in pydantic v1


Duplicate if-else branches found

Godot version:
Master branch commit e141845bfb3e12a9213c8682d5b2589ce5305f70

Issue description:

The following duplicate if-else branches were found by cppcheck:
[drivers/gles3/rasterizer_scene_gles3.cpp:3050] -> [drivers/gles3/rasterizer_scene_gles3.cpp:3047]: (style, inconclusive) Found duplicate branches for 'if' and 'else'.
[editor/property_editor.cpp:2272] -> [editor/property_editor.cpp:2269]: (style, inconclusive) Found duplicate branches for 'if' and 'else'.
[modules/mono/editor/bindings_generator.cpp:2075] -> [modules/mono/editor/bindings_generator.cpp:2072]: (style, inconclusive) Found duplicate branches for 'if' and 'else'.
[modules/visual_script/visual_script.cpp:1705] -> [modules/visual_script/visual_script.cpp:1703]: (style, inconclusive) Found duplicate branches for 'if' and 'else'.

Anyone want to make a PR to fix this?

I'll try my hand at this soon.


separation of values from thermostat and boiler

I kind of understand that currently MQTT is made to be as compatible as it possible with otmonitor. But, as result, it has the same fundamental issue as otmonitor has (and issue which home assistant integration does not have).
There is no separation for values coming from boiler and thermostat.
For example, there is boiler setpoint control. There are two values coming, one from thermostat (what it wants this set point to be) and one from boiler (what is actual setpoint on boiler). If you override this set point (via topic ctrlsetpt), then, effectively, you have two different values, for one topic (TSet).
Proper way is to separate values coming from thermostat and from boiler to different topics.

I'm encountering similar problems, also i haven't found a way to detect who is in control (ie. The thermostat of otgw) separating these values  as Antst is suggesting wil help with this

@emkookmer if you override the Tset value, the thermostat is no longer in control. It's being overridden by the Gateway.
Could you share some logs @antst and @emkookmer to show me what you mean.
Btw what we can do is detect who is controlling the boiler, the gateway or the thermostat. That one is quite easy to add in a future release. Let's do some discussion. If you have joined the Discord community you can talk to me and Schelte on this topic.

Love to keep improving the project, so feedback is a must.

Actually, otmonitor has the same issue. (which it didn't have 4 years ago, BTW)

@antst You have reported this multiple times in bug reports against otmonitor as well, but then never respond to questions for more details. I have tested your claims and cannot reproduce them. So please stop spreading rumors about alleged bugs in otmonitor unless you can actually show some evidence.

@antst please share logs to show what you mean. It is important to capture the message from port 24 on the ESP. Make sure the OTmessage logging is turned on.
You may want to turn on the MQTT logging too that way we can see what you mean.
Like I said before I think there is only one true value for Tset.
But I love to see if we can have logs to show what you mean.
Kinds regards
Robert


[TASK] Fix CVE issues for v1.7.0

What's the task? Please describe

Investigate CVE issues of the Longhorn component images to see if there are outstanding CVE issues that need to be fixed.
Describe the sub-tasks

https://github.com/longhorn/longhorn/blob/master/deploy/longhorn-images.txt
Additional context

None

This is a general task. Not a blocker for RC releases.
cc @derekbit

Analyse (WIP)

longhornio/openshift-origin-oauth-proxy:4.14
longhornio/openshift-origin-oauth-proxy:4.14 (redhat 8.6)
=========================================================
Total: 25 (HIGH: 25, CRITICAL: 0)
@derekbit , @mantissahz , is this within our scope for CVE?

We just mirrored it from https://quay.io/repository/openshift/origin-oauth-proxy?tab=tags and I think we can update it to 4.15. And we had some tests on OKD 4.15 (https://github.com/longhorn/longhorn/issues/8300)

Action Plan

longhornio/longhorn-engine: Update dependencies.

grpc_health_probe v0.4.28

longhornio/longhorn-instance-manager: Update dependencies.

grpc_health_probe v0.4.28

longhornio/support-bundle-kit Update base image.

Release v0.0.40

longhornio/csi-attacher: Update version.

Current highest available minor version: v4.6.1

longhornio/csi-resizer: Update version.

Current highest available minor version: v1.11.1.

longhornio/csi-node-driver-registrar: Update version.

Current highest available minor version: v2.10.1.

longhornio/openshift-origin-oauth-proxy: Update version.

This is not in scope, however, we can still update to the suggested version: 4.15. Ref: https://github.com/longhorn/longhorn/issues/8976#issuecomment-2230147085

Won't Fix

longhornio/backing-image-manager:

No CVE vulnerability detected.

longhornio/longhorn-manager:

go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc is an indirect dependency, and the fixed version is not included in the current latest Kubernetes v1.30.3 release. Its included in the incoming v1.31.0 release.

longhornio/longhorn-share-manager:

No CVE vulnerability detected.

longhornio/longhorn-ui:

No CVE vulnerability detected.

longhornio/csi-provisioner:

The current version is the same as the highest available minor version: v4.0.1.

longhornio/csi-snapshotter:

The current version is the same as the highest available minor version: v7.0.2.

longhornio/livenessprobe:

The current version is the same as the highest available minor version: v2.12.0.

All PRs have been merged. This doesn't require effort from @longhorn/qa , any issues introduced should be detected in the daily regression run. Closing.


chore: rename needs reproduction label

Description
We did this change in the Vite repo too. All other labels are "needs something".
After merging this PR, the label needs to be renamed.
We can also add needs test and needs documentation that are useful in the Vite repo at least.

the label has been renamed to needs reproduction now


Allow dashes in devtainer names

@EdwardSalkeld (or @MarkFitzpatrickNN) please review.

Thanks @EdwardSalkeld. Great to get this done.


Take our Check out of workflowengine app and put it into an app of our own?

After #26, we can clean up so that our code addition is done as a Nextcloud app (based on the MFA Checker app we already have), so that we don't have to change the https://github.com/nextcloud/server code
I think https://github.com/nextcloud/server/blob/master/apps/workflowengine/lib/Manager.php#L684 should allow this.

And for the frontend part, something like: https://github.com/nextcloud/files_accesscontrol/blob/master/js/admin.js

In https://github.com/nextcloud/server/blob/master/apps/workflowengine/lib/Manager.php#L673 you can see it loads built-in checks and registered checks.

Related PR: https://github.com/pondersource/server/pull/240

The failed actions related to my changes passed after merging Nextcloud to my branch. But some actions failed now for phpUnit and Cypress.  phpUnit failed because on action pipeline got an error when wanted to install phpUnit : 403: rate limit exceeded . And for Cypress two tests failed for checking some colors and got a timeout. this section of the site is not related to any of my changes.

By some retries Actions about phpUnit passed and only remains Cypress failures.

So what is the way forward?

We must run this actions locally to debug the progress and can see the Cypress screenshots from point of issue.

Hm, as discussed in https://github.com/pondersource/server/pull/240#issuecomment-1367919484 I think we can probably just ignore this?

@navid-dada can you elaborate on why you changed the title?

Btw 'Check' in the issue title refers to OCP\WorkflowEngine\ICheck

So which one will we add it to?

mfachecker
MohammadReza's newly generated app
the existing file_accesscontrol app

For reference, in our current dev environment, we are adding the code straight into the workflow engine app: apps/workflowengine/lib/Check/MfaVerified.php

As discussed with @navid-dada just now it seems there is no good way (yet) for a Nextcloud app to register a Check. So if that's the case then we can either leave our check where it is, or propose such a mechanism upstream? Should we discuss this with Nextcloud support?

What happens if we just inject our Check into the OCA\WorkflowEngine\Check namespace? Will workflowengine not just be able to find it? How does it list them anyway?

I think checks are loaded by https://github.com/nextcloud/server/blob/1daab5751d632e4d1dac60d28fed3b28af346b54/apps/workflowengine/composer/composer.json#L10 but I haven't found any registry of them yet, apart from that. Maybe they are just referred to by their class name?

Ah yes, I think that's the case, look! https://github.com/nextcloud/server/blob/047acc32ad07de4bb481c267b9f730ce644efbd1/apps/workflowengine/lib/Migration/Version2000Date20190808074233.php#L59

So we have 4 options:

leave things as they are
create a "config var has a certain value" check, and use that in our app
do everything in our app
leave our PR to the workflowengine code fully usable, and also create an app that uses the check in the correct way (to avoid misconfiguration by server admins)

But we think the frontend edits would still be needed in nextcloud/server...

We agree option 4 would be better than option 1

Advantage of option 4: it's less difficult for us to implement
Advantage of option 2: it will be more useful at the work flow engine level, and we're not asking the server admin to use this check anyway, so it would be fine if it's a bit more low-level.

Disadvantage of option 2: we would then probably also support other data types, etcetera, and it's not clear how useful it would be.

Another disadvantage of option 2: in general, server admins don't have know knowledge session variable names and values

So maybe I would prefer option 4 as my top choice.
@navid-dada what would your preference be?
@mrvahedi68 what do you think?
Let's try to take a decision about this.

Great! So nothing left to do here then.


Why one token corresponds to multiple token ids

I think one token corresponds to "yes" and the other one is  "yes". The special character "" represents space in the sentencepiece tokenizer.


Handling memory-mapped images

Hi @mkitti, thanks for the great package! I was wondering if you had an tips for getting memory-mapped TiffImages.jl images to work with Napari.jl. It would be great if I could pass a memory-mapped TIFF object to Napari.jl and it could request data per-slice instead of all at once.

What is the current difficulty? What would you like it to look like from the Python?
I would take a look at the approach I took in https://github.com/mkitti/NumPyArrays.jl. Generally, if you have a memory mapped TIFF image, you should be able to obtain a pointer to the data that can be then wrapped into a NumPy-like array.

Taking a closer look, what you probably want to do is override Napari.view_image, Napari.add_image, d/or anNapari.Image.primitive_array for your type. See
https://github.com/mkitti/Napari.jl/blob/main/src/image.jl

Given a XYZ image, the tricky part is that TiffImages loads each XY slice independently so I would need to be able to overload a function that lets me load per XY slice. Is that possible?
Reinterpreting and permuting the dims should be fine.

That might require more support on the Python side if you want to make the I/O completely lazy. I have not looked at what the update mechanism is that they are using for dask or zarr, but we would basically need a Python class that forwards indexing requests to Julia's getindex. A "jlarray". We would need to look into what API Napari is using to work with dask or zarr.

I'm still confused what the current challenge is though. You have a memory mapped array. This is already lazy.
So you are memory mapping each XY slice. For each XY slice you essentially have a pointer, and that pointer can be passed to NumPy which will represent it as an array. What is currently preventing that from happening?

Sorry, I'm returning back to this now that https://github.com/tlnagy/TiffImages.jl/pull/79 has been merged and TiffImages.jl has true mmap capabilities. This should make a lot of the stuff I was talking about above irrelevant. A memory-mapped TIFF is now a vector of chunks which are the different TIFF slices on disk.
The type of these chunks are (for an example 1024x1024 Gray{N0f16} image)
julia> typeof(diskimg.chunks[1])
Base.ReinterpretArray{Gray{N0f16}, 2, UInt8, Base.ReshapedArray{UInt8, 3, SubArray{UInt8, 1, Vector{UInt8}, Tuple{UnitRange{UInt64}}, true}, Tuple{}}, true}

which is basically from inner to outer -> the raw bytes on disk -> reshaped to a (2, 1024, 1024) array -> reinterpreted as Gray{N0f16}. This should definitely work with Napari.jl.
Currently, trying to call @view_image on the chunks gives me:
julia> @view_image diskimg.chunks[1]
ERROR: MethodError: no method matching NumPyArray(::Base.ReinterpretArray{Gray{N0f16}, 2, UInt8, Base.ReshapedArray{UInt8, 3, SubArray{UInt8, 1, Vector{UInt8}, Tuple{UnitRange{UInt64}}, true}, Tuple{}}, true})
Closest candidates are:
  NumPyArray(::StridedArray{T}, ::Bool) where T<:Union{Bool, Float16, Float32, Float64, Int16, Int32, Int64, Int8, Ptr{PyCall.PyObject_struct}, UInt16, UInt32, UInt64, UInt8, PyCall.PyObject, ComplexF32, ComplexF64} at ~/.julia/packages/NumPyArrays/vqSo5/src/NumPyArrays.jl:88
  NumPyArray(::PyCall.PyArray{T, N}) where {T, N} at ~/.julia/packages/NumPyArrays/vqSo5/src/NumPyArrays.jl:75
  NumPyArray(::AbstractArray{T}) where T<:Union{Bool, Float16, Float32, Float64, Int16, Int32, Int64, Int8, Ptr{PyCall.PyObject_struct}, UInt16, UInt32, UInt64, UInt8, PyCall.PyObject, ComplexF32, ComplexF64} at ~/.julia/packages/NumPyArrays/vqSo5/src/NumPyArrays.jl:87
  ...
Stacktrace:
 [1] view_image(::Base.ReinterpretArray{Gray{N0f16}, 2, UInt8, Base.ReshapedArray{UInt8, 3, SubArray{UInt8, 1, Vector{UInt8}, Tuple{UnitRange{UInt64}}, true}, Tuple{}}, true}; kwargs::Base.Pairs{Symbol, String, Tuple{Symbol}, NamedTuple{(:name,), Tuple{String}}})
   @ Napari.Layers.Image ~/.julia/packages/Napari/vaVWX/src/image.jl:220
 [2] top-level scope
   @ REPL[29]:1

Which looks like it isn't properly recursing down into the wrapped images (it tries to immediately cast the NumPyArray).
julia> NumPyArray(diskimg.chunks[4].parent)
ERROR: conversion to pointer not defined for SubArray{UInt8, 1, Vector{UInt8}, Tuple{UnitRange{UInt64}}, true}
<snip>

julia> NumPyArray(diskimg.chunks[4].parent.parent)
ERROR: conversion to pointer not defined for SubArray{UInt8, 1, Vector{UInt8}, Tuple{UnitRange{UInt64}}, true}
<snip>

julia> NumPyArray(diskimg.chunks[4].parent.parent.parent)
11334290172-element NumPyArray{UInt8, 1}:
 0x49
 0x49
 0x2b
 0x00
 0x08
 0x00
 0x00
 0x00
 0x10
 0x00
 0x20
 0x00
    ⋮
 0x2e
 0x6a
 0x6c
 0x20
 0x76
 0x30
 0x2e
 0x36
 0x2e
 0x31
 0x00

The last conversion appears to create a NumPy pointer to the entire TIFF image on disk. I'm having a hard time following why exactly the NumPyArray conversions are failing given that SubArrays, ReinterpretArrays, ReshapedArrays are all supported. Any help would be appreciated.

A quick MWE using tlnagy/exampletiffs
julia> using Napari, TiffImages

julia> img = TiffImages.load("exampletiffs/4D-series.ome.tif", mmap = true);

julia> @view_image img.chunks[1]
ERROR: MethodError: no method matching NumPyArrays.NumPyArray(::Base.ReinterpretArray{ColorTypes.Gray{FixedPointNumbers.Q0f7}, 2, UInt8, Base.ReshapedArray{UInt8, 2, SubArray{UInt8, 1, Vector{UInt8}, Tuple{UnitRange{Int64}}, true}, Tuple{}}, true})
Closest candidates are:
  NumPyArrays.NumPyArray(::StridedArray{T}, ::Bool) where T<:Union{Bool, Float16, Float32, Float64, Int16, Int32, Int64, Int8, Ptr{PyCall.PyObject_struct}, UInt16, UInt32, UInt64, UInt8, PyCall.PyObject, ComplexF32, ComplexF64} at ~/.julia/packages/NumPyArrays/vqSo5/src/NumPyArrays.jl:88
  NumPyArrays.NumPyArray(::PyCall.PyArray{T, N}) where {T, N} at ~/.julia/packages/NumPyArrays/vqSo5/src/NumPyArrays.jl:75
  NumPyArrays.NumPyArray(::AbstractArray{T}) where T<:Union{Bool, Float16, Float32, Float64, Int16, Int32, Int64, Int8, Ptr{PyCall.PyObject_struct}, UInt16, UInt32, UInt64, UInt8, PyCall.PyObject, ComplexF32, ComplexF64} at ~/.julia/packages/NumPyArrays/vqSo5/src/NumPyArrays.jl:87
  ...
Stacktrace:
 [1] view_image(::Base.ReinterpretArray{ColorTypes.Gray{FixedPointNumbers.Q0f7}, 2, UInt8, Base.ReshapedArray{UInt8, 2, SubArray{UInt8, 1, Vector{UInt8}, Tuple{UnitRange{Int64}}, true}, Tuple{}}, true}; kwargs::Base.Pairs{Symbol, String, Tuple{Symbol}, NamedTuple{(:name,), Tuple{String}}})
   @ Napari.Layers.Image ~/.julia/packages/Napari/vaVWX/src/image.jl:220
 [2] top-level scope
   @ REPL[16]:1

The original problem is that you are using UInt64 rather Int64 to index to the array. Try using Int64 ranges instead to subindex instead.
Issue:
https://github.com/JuliaLang/julia/issues/42120
Fix by me:
https://github.com/JuliaLang/julia/pull/43262

Okay, that helps! Trying with the chunk still fails, but now we're a level up and the parent of the chunk now loads:
julia> @view_image diskimg.chunks[4]
ERROR: MethodError: no method matching NumPyArray(::Base.ReinterpretArray{Gray{N0f16}, 2, UInt8, Base.ReshapedArray{UInt8, 3, SubArray{UInt8, 1, Vector{UInt8}, Tuple{UnitRange{Int64}}, true}, Tuple{}}, true})
Closest candidates are:
  NumPyArray(::StridedArray{T}, ::Bool) where T<:Union{Bool, Float16, Float32, Float64, Int16, Int32, Int64, Int8, Ptr{PyCall.PyObject_struct}, UInt16, UInt32, UInt64, UInt8, PyCall.PyObject, ComplexF32, ComplexF64} at ~/.julia/packages/NumPyArrays/vqSo5/src/NumPyArrays.jl:88
  NumPyArray(::PyCall.PyArray{T, N}) where {T, N} at ~/.julia/packages/NumPyArrays/vqSo5/src/NumPyArrays.jl:75
  NumPyArray(::AbstractArray{T}) where T<:Union{Bool, Float16, Float32, Float64, Int16, Int32, Int64, Int8, Ptr{PyCall.PyObject_struct}, UInt16, UInt32, UInt64, UInt8, PyCall.PyObject, ComplexF32, ComplexF64} at ~/.julia/packages/NumPyArrays/vqSo5/src/NumPyArrays.jl:87
  ...
Stacktrace:
 [1] view_image(::Base.ReinterpretArray{Gray{N0f16}, 2, UInt8, Base.ReshapedArray{UInt8, 3, SubArray{UInt8, 1, Vector{UInt8}, Tuple{UnitRange{Int64}}, true}, Tuple{}}, true}; kwargs::Base.Pairs{Symbol, String, Tuple{Symbol}, NamedTuple{(:name,), Tuple{String}}})
   @ Napari.Layers.Image ~/.julia/packages/Napari/vaVWX/src/image.jl:220
 [2] top-level scope
   @ REPL[9]:1

julia> @view_image diskimg.chunks[4].parent
PyObject Viewer(axes=Axes(visible=False, labels=True, colored=True, dashed=False, arrows=True), camera=Camera(center=(0.0, 511.5, 511.5), zoom=0.9936035156249999, angles=(0.0, 0.0, 90.0), perspective=0.0, interactive=True), cursor=Cursor(position=(1.0, 1.0, 0.0), scaled=True, size=1, style=<CursorStyle.STANDARD: 'standard'>), dims=Dims(ndim=3, ndisplay=2, last_used=0, range=((0.0, 2.0, 1.0), (0.0, 1024.0, 1.0), (0.0, 1024.0, 1.0)), current_step=(1, 512, 512), order=(0, 1, 2), axis_labels=('0', '1', '2')), grid=GridCanvas(stride=1, shape=(-1, -1), enabled=False), layers=[<Image layer '(diskimg.chunks[4]).parent' at 0x7f6d1284ac40>], scale_bar=ScaleBar(visible=False, colored=False, ticks=True, position=<Position.BOTTOM_RIGHT: 'bottom_right'>, font_size=10.0, unit=None), text_overlay=TextOverlay(visible=False, color=array([0.5, 0.5, 0.5, 1. ]), font_size=10.0, position=<TextOverlayPosition.TOP_LEFT: 'top_left'>, text=''), overlays=Overlays(interaction_box=InteractionBox(points=None, show=False, show_handle=False, show_vertices=False, selection_box_drag=None, selection_box_final=None, transform_start=<napari.utils.transforms.transforms.Affine object at 0x7f6cfeb25d90>, transform_drag=<napari.utils.transforms.transforms.Affine object at 0x7f6cfeb25df0>, transform_final=<napari.utils.transforms.transforms.Affine object at 0x7f6cfeb25e50>, transform=<napari.utils.transforms.transforms.Affine object at 0x7f6cfeb25eb0>, allow_new_selection=True, selected_vertex=None)), help='', status='Ready', tooltip=Tooltip(visible=False, text=''), theme='dark', title='napari', mouse_move_callbacks=[<function InteractionBoxMouseBindings.initialize_mouse_events.<locals>.mouse_move at 0x7f6d1285a310>], mouse_drag_callbacks=[<function InteractionBoxMouseBindings.initialize_mouse_events.<locals>.mouse_drag at 0x7f6d12844790>], mouse_double_click_callbacks=[], mouse_wheel_callbacks=[<function dims_scroll at 0x7f6d271d7940>], _persisted_mouse_event={}, _mouse_drag_gen={}, _mouse_wheel_gen={}, keymap={'Shift': <function InteractionBoxMouseBindings.initialize_key_events.<locals>.hold_to_lock_aspect_ratio at 0x7f6d128443a0>, 'Control-Shift-R': <function InteractionBoxMouseBindings._reset_active_layer_affine at 0x7f6d12844ee0>, 'Control-Shift-A': <function InteractionBoxMouseBindings._transform_active_layer at 0x7f6d12844ca0>})

Forcibly calling Napari.Layers.Image.primitive_type seems to fix the problem:
julia> @view_image Napari.Layers.Image.primitive_array(diskimg.chunks[4])
PyObject Viewer(axes=Axes(visible=False, labels=True, colored=True, dashed=False, arrows=True), camera=Camera(center=(0.0, 511.5, 511.5), zoom=0.9936035156249999, angles=(0.0, 0.0, 90.0), perspective=0.0, interactive=True), cursor=Cursor(position=(1.0, 1.0, 0.0), scaled=True, size=1, style=<CursorStyle.STANDARD: 'standard'>), dims=Dims(ndim=3, ndisplay=2, last_used=0, range=((0.0, 1.0, 1.0), (0.0, 1024.0, 1.0), (0.0, 1024.0, 1.0)), current_step=(0, 512, 512), order=(0, 1, 2), axis_labels=('0', '1', '2')), grid=GridCanvas(stride=1, shape=(-1, -1), enabled=False), layers=[<Image layer 'Napari.Layers.Image.primitive_array(diskimg.chunks[4])' at 0x7f6d0d9d8a60>], scale_bar=ScaleBar(visible=False, colored=False, ticks=True, position=<Position.BOTTOM_RIGHT: 'bottom_right'>, font_size=10.0, unit=None), text_overlay=TextOverlay(visible=False, color=array([0.5, 0.5, 0.5, 1. ]), font_size=10.0, position=<TextOverlayPosition.TOP_LEFT: 'top_left'>, text=''), overlays=Overlays(interaction_box=InteractionBox(points=None, show=False, show_handle=False, show_vertices=False, selection_box_drag=None, selection_box_final=None, transform_start=<napari.utils.transforms.transforms.Affine object at 0x7f6cfea46b20>, transform_drag=<napari.utils.transforms.transforms.Affine object at 0x7f6cfea46b80>, transform_final=<napari.utils.transforms.transforms.Affine object at 0x7f6cfea46be0>, transform=<napari.utils.transforms.transforms.Affine object at 0x7f6cfea46c40>, allow_new_selection=True, selected_vertex=None)), help='', status='Ready', tooltip=Tooltip(visible=False, text=''), theme='dark', title='napari', mouse_move_callbacks=[<function InteractionBoxMouseBindings.initialize_mouse_events.<locals>.mouse_move at 0x7f6d0d99c700>], mouse_drag_callbacks=[<function InteractionBoxMouseBindings.initialize_mouse_events.<locals>.mouse_drag at 0x7f6d0d99c040>], mouse_double_click_callbacks=[], mouse_wheel_callbacks=[<function dims_scroll at 0x7f6d271d7940>], _persisted_mouse_event={}, _mouse_drag_gen={}, _mouse_wheel_gen={}, keymap={'Shift': <function InteractionBoxMouseBindings.initialize_key_events.<locals>.hold_to_lock_aspect_ratio at 0x7f6d0d9899d0>, 'Control-Shift-R': <function InteractionBoxMouseBindings._reset_active_layer_affine at 0x7f6d0d9898b0>, 'Control-Shift-A': <function InteractionBoxMouseBindings._transform_active_layer at 0x7f6d0d989820>})

I guess that supposed I should use primitive_array before trying to use NumPyArray here:
https://github.com/mkitti/Napari.jl/blob/b2c1469599ce14911efb3a90286c48c59d2dda2d/src/image.jl#L216-L223
I'm curious what the output of @which Napari.Layers.Image.primitive_array(diskimg.chunks[4]) is

julia> @which Napari.Layers.Image.primitive_array(diskimg.chunks[4])
primitive_array(A::AbstractArray{C}) where {F, T<:(FixedPoint{F}), C<:Colorant{T, 1}} in Napari.Layers.Image at /home/tlnagy/.julia/packages/Napari/vaVWX/src/image.jl:48

I guess now the big problem is how to overload getindex or similar so that we can load the right chunk from the disk. I'm not sure what the best approach this would be.

That's just https://github.com/mkitti/Napari.jl/blob/b2c1469599ce14911efb3a90286c48c59d2dda2d/src/image.jl#L51
function primitive_array(A::AbstractArray{C}) where C <: Colorant{T,1} where T <: FixedPoint{F} where F
    # 1. Grab channel view which gets us Array{T,3}. This is a Base.ReinterpretArray
    # 2. Convert to Array{F,3}, still a Base.ReinterpretArray
    reinterpret(F,channelview(A))
end

Yeah, it looks like it's just converting it to a type that's compatible with Python since the FixedPointNumber is a Julia specific thing:
julia> NumPyArrays.PYARR_TYPES
Union{Bool, Float16, Float32, Float64, Int16, Int32, Int64, Int8, Ptr{PyCall.PyObject_struct}, UInt16, UInt32, UInt64, UInt8, PyCall.PyObject, ComplexF32, ComplexF64}

What are your thoughts on the getindex problem?


Fix scripts for publish

Pull Request Test Coverage Report for Build 440001893

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 93.837%

Totals

Change from base Build 434744707:
0.0%

Covered Lines:
419

Relevant Lines:
434

💛  - Coveralls


Can you provide an official implementation of FSDP based on keras distributed API?

I want to train gemma-7b, but I can't achieve fsdp. Hope keras-team can provide an implementation

Hi @pass-lin ,
Could you please refer data_parallel API and also distribution and let us know if it helps.

FSDP https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html

Hi @pass-lin ,
Could you please refer data_parallel API and also distribution and let us know if it helps.

Hello, based on your information, I implemented the following code (single-machine multi-card environment)
devices = np.array(jax.devices()).reshape(1, -1, 1)
mesh = keras.distribution.DeviceMesh(devices.shape, ('dp', 'fsdp', 'mp'), devices=jax.devices())
layout_map = keras.distribution.LayoutMap(mesh)
parallel = keras.distribution.ModelParallel(mesh, layout_map)
keras.distribution.set_distribution(parallel)

But this is incorrect and cannot run. I want to know why this is the case.

Hi @pass-lin ,
Could you please submit a minimal repro code for this ? Thanks!

Hi @pass-lin ,
Could you please submit a minimal repro code for this ? Thanks!
I’m not quite understanding why these commands above are resulting in errors


Where in the proposed folder structure do I put shared files like filters and services?

Where in the proposed folder structure do I put shared files like filters and services?
It looks to me like the project is set up to have one folder or module per controller. But where do I put services that are shared between them? Filters? Directives?

I agree with @tgallant, but I want to make one minor clarification. ngbp is set up to have one file per module, which often works out to one module per folder. That applies to one controller per folder, but also to one filter, one service, etc. So it's more broad is my point.
I'll go ahead and close this, but please follow up if you have more questions!

It's still unclear to me where I can put application specific services, filters and directives. Neither the app or the common directory seem to apply.
src/app

Apart from app.js and its accompanying tests (discussed below), this directory is filled with subdirectories corresponding to high-level sections of the application, often corresponding to top-level routes.

This is not the case, these do not correspond to a high-level section of the application, let alone to routes.
src/common

Every component contained here should be drag-and-drop reusable in any other  project; they should depend on no other components that aren't similarly drag-and-drop reusable.

Definitely not true in my case, these items are very much application specific and don't make sense in other applications.
Also asked on StackOverflow

I'll tell you what we do where I work, maybe it will be helpful. Note that I'm not saying this is The Right Way, but it seems to work pretty well and be pretty manageable from a maintenance and upgradeability perspective.
Although we've modified the ngbp setup a bit, the following details work with the vanilla ngbp.
our application
Our application is reasonably sizable:

many different screens/pages/routes
each screen has some level of shared state (e.g. date fields, text filters)
each screen has multiple interactions (e.g., adjust the dates, change the filters)
some state is only shared with some screens (e.g., maybe one date is shared between A and B, and another is shared between B and C)
easily over 100 each of directives and factories, and probably around 20-50 controllers
lots of old gross code (mixing factories and directives, not testable, etc.)
multiple apps (each a new clone of ngbp) that share common code from src/common (we basically do ln -s on the folder)

Anyway, it's a sizable app, and we keep at least one full time developer on it at all times, just upgrading and fixing bugs.
folder structure
We have it set up with src/app and src/common set up as the wording implies. The src/common has stuff that's "drag-and-drop reusable", while the src/app has application-specific code.
Here's what that actually looks like (with some actual examples):
src/common/autocomplete
	autocomplete.js # directives and factories specific to an input text autocomplete thing
	autocomplete.less # styles, namespaced to the autocomplete directive
src/common/jquery-dropdown
	jquery-dropdown.js # a third-party plugin we use, but modify heavily and wrapped to angularjs
	jquery-dropdown.less # the styles to touch up our mods
src/common/fallback-image
	fallback-image.js # a directive that sets a fallback image if `ng-src` doesn't load
src/common/loading-bar
	loading-bar.js # directive of a loading bar with extra options
	loading-bar.less # some style for the loading bar
src/common/days-between-dates
	days-between-dates.js # the factory
	days-between-dates.spec.js # actual tests
# etc.

Some observations:

Some of these are really small files. For example, the days-between-dates is only 11 lines.
You'll notice our test coverage is pretty poor. That's the fault of big old software, and it makes me sad.
The things are generally drag-and-drop, but obviously there's some small choices (e.g. the LESS files) that make it not entirely generic.
Although the naming scheme is redundant (having my-module/my-module.js used to bother me), what it allows is easier debugging in Chrome. (With the developer console open, you can hit command/control+o and type in the file name to open it. Easier than having 100 files named index.js or similar.)
Not everything could be immediately plucked out and

Inside the src/app we've got the application-specific code. This includes "screens" (we use the wonderful ui-router for states) and services that are explicitly for this application.
The pages look like:
src/app/splash
	/login # handle login requests (not behind password)
		login.html
		login.less
		login.js # this has the routing setup and a controller
	/password-reset # handle password resets and the email links go here
		password-reset.html
		password-reset-request.html
		password-reset.js # routing setup and controller
src/app/overlap # an example page showing overlapping data
	overlap.js # the routing setup and controller
	overlap.html
	overlap.less
	overlap-model.js # factories that handle $http.get() so that isn't in the controller
	/company-address # a directive that is only used within `overlap`
		company-address.js
		company-address.spec.js #tests
	/refinements # a child route of the parent `overlap` path
		refinements.js # routing and controller
		refinements.html
		refinement.less
src/app/audit
	audit.js # a service that we use in some of our screens to log user interaction
src/app/session
	session.js # a service we use to store things in-session and sync with the server
	session.spec.js # tests for the service
src/app
	app.js # the startup code

Observations:

All the application logic sits in src/app, meaning routes and controllers for pages
Directives and factories that are specific to a route/page go inside that folder (easier to find)
We put all our $http.get() logic inside factories, and those then go in NAME-model.js, we've found this easy to understand and maintain

other thoughts
In general we've found this layout to work well with our team, and to be maintainable. After having been through one big Angular upgrade (1.0 to 1.3) and then another big Bootstrap upgrade (2.1 to 3.3), I feel like this is a pretty solid setup.
Of course, YMMV, and this is only one opinion, based on only a couple applications. You may find that your application structure has much more code shared between pages. We found that ~70% of our directives and service were bound to a single page, and ~20% were abstractable enough to strip them down and put them in src/common to be shared between applications.
What you seem to be wondering about is the (for us, anyway) remaining 10%. For us, we found that they could be put into folders and such, like the src/app/session code. (That's a couple factories that sync'd session and server state. It was app-wide, not for a page.)
in conclusion
I don't think that answers your question directly, but maybe it will inform your design decisions.
If you have more questions, feel free to ask.


TypeScript definitions are out of date

CC @ikatyang @ColinBradley
See also DefinitelyTyped/DefinitelyTyped#33780
The DefinitelyTyped TS definitions haven't been updated since the 1.0 release last fall, and the library's API has expanded a bit since then. With the next minor release I'll be deprecating some of the endpoints and adding new ones thanks to #99.
Any suggestions for how to make it easier to keep these definitions up to date?

The best way is to rewrite the package in Typescript but it requires a huge effort. If you only want to keep the definitions up to date, you can actually include definitions in your own package. You can do this by add types in package.json, so you don't need to make a pull request every time.
{
   "types": "./lib/main.d.ts"
}

Any suggestions for how to make it easier to keep these definitions up to date?

Rewrite in typescript 😛

@akosyakov Would you be willing and/or interested in implementing a PoC of the JSDoc annotations? That sounds like a decent idea.

Had a quick try at this, and immediately hit microsoft/TypeScript#7546. But it looks like this might finally be getting solved as a part of microsoft/TypeScript#30555?

Yeah, I'm going to keep this open if someone that actually uses TS wants to have a go at it. With the following tsconfig.json I managed to get it down to just 160 errors, but then found at least the following specific pain points that I'm uninterested in fixing, at least for now:

The AST Node does not initialise its fields, so that the undefined ones are never set. At least Pair makes use of this to allow using get/set for its commentBefore
In particular at the CST level the library abuses labels for debug logging with babel-plugin-trace. Either that would need to get dropped, or the Babel plugin would need to run before the source goes to Typescript.

If someone's interested in setting it up, a possible workflow to generate the d.ts files could be something like *.js -> Babel -> *.ts -> tsc -> *.d.ts, presuming that JSDoc comments are also sufficient for typing TS files. I guess the alternative is that someone manually maintains the d.ts files?
{
  "compilerOptions": {
    "outDir": "./ts/",
    "strictNullChecks": false,
    "lib": ["es2017"],
    "module": "es6",
    "target": "es6",
    "allowJs": true,
    "checkJs": true,
    "types": []
  },
  "include": ["./src/**/*"]
}

I don't think others are doing like this. One of the common ways is to just create .d.ts files manually like what you do with the documentations. When there are public API update, update the related .d.ts files.
Using JSDoc seems to very different to maintain. At that point, you may as well write it in TypeScript.

Joining the party as I'm trying to use it on TypeScript, and for the time being I indeed have to work with either @ts-ignore, or alternatively use old definitions, which is terrible and does not match the documentation :(

Ping @benjamingr, who said he might be interested in looking at this.

Sure, is the context generating the type information?
Would dts-gen not suffice for starters :] ?

I've got background in Typescript and YAML, and this implementation interests me.   I'd like to have a look at doing this by adding type annotations to the existing implementation, and having TSC generate the definitions file.
Made some progress with the approach enabled by Babel 7 using @babel/preset-typescript for code generation, and tsc for error checking.   But there are a few parts of the code I don't understand.   I'll post them as separate questions.


Mantis database checks failed

I installed Mantis as adviced and was checking if the databases were installed properly using mantis check_sql
mantis's latest release is: 1.5.5

Cython correctly compiled!
Checking extra files
Passed installation check on: /home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/Resources/essential_genes
Passed installation check on: /home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/Resources/NCBI/gc.prt.dmp
Passed installation check on: /home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/Resources/Taxonomy.db
Checking references installation
Failed installation check on [path unavailable]: /home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/References/NOG/
Passed installation check on: /home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/References/NCBI/
Passed installation check on: /home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/References/pfam/
Passed installation check on: /home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/References/kofam/
Failed installation check on [invalid referecence type]: /home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/References/tcdb/
Installation check failed on:
/home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/References/NOG/
/home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/References/tcdb/

------------------------------------------
--------INSTALLATION CHECK FAILED!--------
------------------------------------------
Traceback (most recent call last):
  File "/home/subudhak/miniconda3/envs/mantis/bin/mantis", line 11, in <module>
    sys.exit(main())
  File "/home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/mantis/__main__.py", line 218, in main
    check_installation(mantis_config=mantis_config, check_sql=True, no_taxonomy=no_taxonomy)
  File "/home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/mantis/assembler.py", line 27, in check_installation
    mantis.check_installation(check_sql=check_sql)
  File "/home/subudhak/miniconda3/envs/mantis/lib/python3.10/site-packages/mantis/assembler.py", line 628, in check_installation
    raise InstallationCheckNotPassed
mantis.exceptions.InstallationCheckNotPassed: Installation check not passed! Make sure you've setup the databases

Hello @Rohit-Satyam
could you please share your config file?
Regards,
Pedro

Hi @PedroMTQ. It appears like some database failed to download previously (maybe due to internet issues). I reran the mantis setup command and now it works.


Poll the react-native community regarding bundle splitting via Metro

Ask the react-native community (Twitter, Discord) about bundle-splitting using Metro.
Is anyone doing it?
If not, and we built it, would they use it?
How does it fit into a Hermes world with byte-code bundles that favor a single mult-app bundle.
For those interested in module federation, how is this different? Should module federation be pursued concurrently as part of the same workstream?
Use the results to inform bundle-splitting scenario development, and to set the priority of this work.
Note the most enthusiastic respondents, and consider partnering with them as contributors and/or early adopters.

I think callstack/react-native-multibundle may be useful.


Upgrade ember-frost-test dependency

Upgrade to the version release due to the resolution of https://github.com/ciena-frost/ember-frost-test/issues/38

Will be address by #37


ensureNotInGC will break with 2.071

Must use something else than GC.free(1);

Yes it will stop working, however it may not be needed anymore:
https://github.com/D-Programming-Language/druntime/pull/1508/files#diff-6f1ab0423fff9dcd084ecf9a677dc426R3261

This means that you can always free in the destructor, but if you do it while a GC collections is running it will be ignored.

Yep, we'll switch to GC.malloc most probably, unless there is a way to tell if we are in a GC collection.

Actually ensureNotInGC was still using GC.malloc(1) in GFM.


[IoTHub] summary and description should not contain the same content

IoTHub copy/paste the same content into "summary" and "description". This should not be the case, since this leads to bad documentation:

In our next swagger version publish we will fix this. Please assign to me.

https://msazure.visualstudio.com/One/_workitems/edit/4979397


docker-compose in DinD

After running DinD image I noticed that docker-compose is not installed.
I tried to install with:
curl -L "https://github.com/docker/compose/releases/download/1.23.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

But this seems not to work.
How can I get docker-compose running?

Hi,
This repository is not maintained anymore; it's here only for historical reasons.
But you can check the docker:dind images, that replace it and (I think) provide similar features!


If input types are incorrect an error should be passed in the callback

I was just debugging a situation where I would have something like
contract.isAdmin.call(address, function(error, output){
	if (error) cb(error, null);
	cb(null, output.values.ret)
});

can the callback was not being called. No error was being passed instead an uncaught exception kept appearing "undefined is not a function" of course isAdmin.call was a function it turned out the argument i was passing in address had a string in it and not an address.
Having an uncaught exception makes these situations difficult to work with. (Yes I should have run my own type checks or something but still the callback should still be called with an error to handle it gracefully.)

I can't be a full param check in each function to guard against every possible error. If it is unclear how a function should be called, or if there are optional params, then I'll try and be more clear in the docs, though that will be handled case-by-case.

I think you misunderstood this is not with the library functions its with the generated solidity functions. The input type matching is something you already do. Im just asking that a useful error is passed to the callback if the arguments fail to match inputs.

There's no explicit check of parameters types. If the the formatting happens to throw an error, for whatever reason, it will catch it and return it in the callback. That's all. If you want to make sure that you're passing the correct data in

... Unit test your code

Thats what im saying. The callback was not called at all. It just throws an uncaught undefined is not a function and stops execution

Let me try to make this more clear lets say in solidity I have a function
	function addAdmin(address target) returns (bool success){
		//test for admin permissions
		if (isAdmin(msg.sender) == false) return false;

		admin[target] = true;
		return true;
	}

this is going to have an abi entry like:
{
     "constant":false,
     "inputs":[{"name":"target","type":"address"}],
     "name":"addAdmin",
     "outputs":[{"name":"success","type":"bool"}],
     "type":"function"
}

So if I call
contract.isAdmin.call(target, function(error, output){
    if (error) cb(error, null);
    cb(null, output.values.success)
});

but "target" is not an address (If instead its something inconvertible to the address type such as a string "dennis") the callback is NOT called with an error. the callback is not called AT ALL. Instead an uncaught exception in contract.isAdmin.call() is thrown saying "undefined is not a function"  which is clearly not the error that is really occurring.
I'm not asking for checks on every input argument. I'm asking that if the input formatters fail to format that the callback is called with a meaningful error message,
And for the record I found this bug BECAUSE I WAS  unit testing.

then pass it an address? i don't see what the issue here is. you don't pass a javascript method the proper number of arguments its behavior will generally be undefined. if you're worried about users calling your methods which passes them on un-checked to support libs, add the guards yourself.
here. maybe this will help: https://www.commonsensemedia.org/app-reviews/kodable-pro

You said yourself that errors from the input formatters are passed to the callback. I'm telling you that its not happening the callback is not being called with the error. If I pass a function a value and its not able to handle it properly it should still pass the error to be handled. the function is defined but its throwing an error instead of passing one.

if you don't call it with the right number of params the behavior is undefined. undefined means it does not have a clear behavior. it could return an error, it could not, it could throw an error. it could do nothing. that's how javascript methods generally works, since javascript is not a strongly typed language. That is not a bug, that is the default behavior unless you change it - which i'm not going to do, because it's extra work and it's just stupid hand-holding. send it the correct number of parameters and you'll be fine. if it's unclear how the function in question works, then I will see if the docs needs changing.
don't open this issue again. never open an issue in a github repository in a repo not managed by yourself if the actual manager(s) has closed it. that's extremely annoying, but then again - you're extremely annoying. if you think an issue has been closed wrongly, then explain why and ask to have it re-opened.

It has been closed incorrectly because its a bug. You clearly have not been listening its not that im not sending it the right number of parameters. Its that the parameter types can not be interpretted by  by the input formatters.
Since you would rather attack me personally then look into it ill do it myself.

if the formatters produce an error, you get the error in the callback.

then something is going wrong because in the above passing something that can't be converted to an address. like "dennis" causes an error to be thrown and the callback not called (at all. with or without and error)
Hence Bug.

Fine. Give me a reproducible example then, and I'll look at it. Both the call and transact methods should catch errors during formatting and send them to the callback function, given that the correct number of  params are used. If that is not true i will bug-flag it again.
re-opening.

hahaha wow. i got this all backwards it seems. except one thing....

To be fair to you. I am unable to replicate the exact same code in a new application. I'm looking into the npm dependencies (I have run npm update and it didn't fix it) so I think it might be a dependency of one of the dependencies not getting updated. I have run into this issue before when running npm update. just going to nuke the node_modules folder and re install.
Closing because I can't reproduce in fresh installs. Behaviour as for described situation appears to be as expected.


Looking for CSR from ecdsa

Hi, I'm looking for CSR with ecdsawithsha256 and I followed this library this is only for signature from EC so could you help me that how can I generate CSR (PEM format) certificate via using this library?

I've generated CSR with elliptic curve 256, and uploaded on github https://github.com/aleemrazzaq/CSR_EC_256
Thanks.

Heya!
Looks great! If you get around to adding usage and examples to the readme, and add test cases, let me know and I'll create a "Projects Using..." section on my readme, and link to it.

Ricmoo, it would be great if you'll.
Thanks


Feature/unsubscribe offers

Description
Documentation of offer unsubscription flow (backend)
Why
Newly released function. FE follows soon.
Issue
n/a
Checklist

[x] I have followed the contributing guidelines
[x] I have commented my code, particularly in hard-to-understand areas

@VPrasannaK94  please review the technical documentation of the endpoints.


downgrade bazel genproto for internal compat

Codecov Report

Merging #3 into master will increase coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master       #3      +/-   ##
==========================================
+ Coverage   54.41%   54.42%   +0.01%
==========================================
  Files         110      110
  Lines        8276     8274       -2
==========================================
  Hits         4503     4503
+ Misses       3333     3331       -2
  Partials      440      440

Impacted Files
Coverage Δ

bazel/execution/service.go
41.22% <ø> (+0.61%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5bb796f...3d33856. Read the comment docs.


Relabel MONDO:0013128

The ClinGen Kidney CDWG would like to request the following name change:
Mondo term (ID and Label): MONDO:0013128  familial juvenile hyperuricemic nephropathy type 2
Suggested new label: Autosomal dominant tubulointerstitial kidney disease due to mutations in REN (ADTKD-REN)
PMID: 25738250

@tsneddon

This class is part of a series, and I think we prefer to keep the naming consistent. Would you be okay if I added the new label as a synonym instead?

Sure, thanks Nicole.


Custom cookie name configuration spec

In session_spec.rb:33's custom cookie name spec, we are setting up cookie_domain  configuration and that makes the test suite green. We have the cookie_domain configuraiton spec on [session_spec.rb:278] (https://github.com/thoughtbot/clearance/blob/master/spec/clearance/session_spec.rb#L278), so I belive this is to test cookie_name configuration.
This commit changes the spec to test out the cookie name has been set properly in the header

I was a bit confused by your description here. After looking at the change, you are basically making sure that the cookie name isn't something like "custom_token123" by adding the =. Is this correct?

Sorry to confuse you, Custom cookie name configuration spec passes for both of the following scenario:

Clearance.configuration.cookie_domain="custom_token", then header hash is

{"Set-Cookie"=> "remember_token=7af2815d2161; domain=custom_token; path=/; expires=Sat, 22 Apr 2017 20:20:33 -0000" }

Clearance.configuration.cookie_name="custom_token" then header hash is

{"Set-Cookie"=>"custom_token=e92ae87ba13400dff6067bca58c55b8fada24c11; path=/; expires=Sat, 22 Apr 2017 20:19:19 -0000"}

In first case we are setting up the custom_token as domain, but that's still passign the test, so I wanted to be more specific, that  custom_token is present as custom_token=[token] to make sure we are testing cookie_name configuration is working.

I just updated the commit to make the test less confusing :)
Clearance.configuration.cookie_name="custom_cookie_name" instead of custom_token

Ahh, I see now. Thanks


publish arm64 wheels

Windows ARM64 is a thing now, so we should publish wheels for it.
Probably means switching build system to https://github.com/pypa/cibuildwheel.

This has been done since v1.0.0-beta.9.


Update testing to use latest dev rpc/horizon with protocol 22

What
Update testing to use latest dev rpc/horizon with protocol 22.
Tracking main for rpc.
Tracking protocol-22 branch for horizon and friendbot.
Why
For the @stellar/devx team to be able to use something to test with.
Merging
This change is not intended for merging because it references versions of rpc, horizon, and friendbot that have not yet been released.

The arm64 packages aren't finished yet, but you can run the amd64, and it'll emulate on arm64:
docker run --platform linux/amd64 stellar/quickstart:pr632-v445-testing-amd64 --protocol-version 22 --local

I was able to build this locally and update the protocol version to 22, but I get the following when trying to do an install:

❌ error: transaction simulation failed: cannot create budget from network configuration

I see the same. Here's the full output:
❯ stellar contract install --wasm target/wasm32-unknown-unknown/release/test_add_u64.wasm --network local --source me
ℹ  Simulating install transaction…
❌ error: transaction simulation failed: cannot create budget from network configuration

Caused by:
    HostError: Error(Budget, InternalError)
    DebugInfo not available

Here's the output of the rpc logs:
time="2024-10-10T13:52:10.453Z" level=info msg="starting JSONRPC request" json_req=18 method=simulateTransaction pid=488 req=36 subsys=jsonrpc
time="2024-10-10T13:52:10.454Z" level=info msg="finished JSONRPC request" duration="907.584µs" json_req=36 pid=488 req=36 status=error subsys=jsonrpc

You were pointing at main instead of the protocol22 branch. I am now getting an XDR length error. However, the transactions do seem to succeed.

@leighmcculloch I updated the commits again to the RPC that updated the rust deps. I then fixed some bugs I found in the client and updated that PR and the CLI to use it. Everything seems to be working locally! So once we get the images rebuilt we should be able to test it in CI!

docker run \
    -p 8000:8000 \
    stellar/quickstart:pr632-v449-testing \
    --protocol-version 22 \
    --local


Fix deferred handling in register logfile method

This fixes two bugs:

The error callback was called with the wrong number of arguments.

On retrying, this method would generate a new deferred and forget
about the old one.

Coverage remained the same at 75.54% when pulling 81b85654ab084b91d63aa83cc8c8316542752353 on guidow:fix_deferred_on_error into 9de89671fc70ef91bb5fcdbfc14be15ff395a53b on pyfarm:master.

Thanks, good catch.


permission denied

I am using ubuntu18.04 and opencv-3.4.4, I did every steps but still I got this in below;

Ps: I dont use anaconda..

Try to use
sudo ./linux_mark.sh

not worked but I solved, here:
chmod +x linux_mark.sh
./linux_mark.sh

try to re-run !make file
%cd /your_path/
!sed -i 's/OPENCV=0/OPENCV=1/g' Makefile
!sed -i 's/GPU=0/GPU=1/g' Makefile
!sed -i 's/CUDNN=0/CUDNN=1/g' Makefile
!sed -i "s/ARCH= -gencode arch=compute_60,code=sm_60/ARCH= ${ARCH_VALUE}/g" Makefile
!make


inaccurate recording rule

What did you do?
I wanted to calculate container CPU saturation. Without a recording rule I would do something like this:
rate(container_cpu_usage_seconds_total{pod!="", container!=""}[1m]) / ON (namespace, pod, container) GROUP_LEFT kube_pod_container_resource_limits_cpu_cores
Then I wanted to set up a recording rule for this query while retaining the ability to change the range selector, so I wrote the recording rule like this:
  - record: container_cpu_saturation_total
    expr: container_cpu_usage_seconds_total{pod!="", container!=""} / ON(namespace, pod, container) GROUP_LEFT kube_pod_container_resource_limits_cpu_cores

So I could do something like this:
rate(container_cpu_saturation_total[1m])
rate(container_cpu_saturation_total[5m])
rate(container_cpu_saturation_total[15m])

What did you expect to see?
I expect the original PromQL and the one with the recording rule to yield mostly identical results. Perhaps a margin of error no more than a few percents?
What did you see instead? Under which circumstances?
This is the actual result I got:

The difference is rather large. With recording rule, the returned values are always higher, sometimes going over 100% which should not be possible.
I tested this on both v2.15.2 and v2.17.1, both versions have this issue (the screenshot is from v2.15.2)
Environment

System information:

Linux 3.10.0-1062.el7.x86_64 x86_64

Prometheus version:

I tested this on both v2.15.2 and v2.17.1, both versions have this issue (the screenshot is from v2.15.2)

Prometheus configuration file:

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 30s

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - <IP_ADDRESS>:6007
    scheme: http
    timeout: 10s
    api_version: v1

rule_files:
- /config/*.recording.yml
- /config/*.alerting.yml

scrape_configs:

(other irrelevant config)

- job_name: cadvisor
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics/cadvisor
  scheme: https
  kubernetes_sd_configs:
  - role: node
  tls_config:
    ca_file: /secrets/kubelet/ca
    cert_file: /secrets/kubelet/cert
    key_file: /secrets/kubelet/key
    insecure_skip_verify: false

- job_name: kube-state-metrics
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - role: service
    namespaces:
      names:
      - kube-system
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: kube-state-metrics
    replacement: $1
    action: keep

here is another screenshot with graph instead of instant query

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.


[NEEDS TICKET] update react-scripts to v5

Working, but beware that this changes the console output when developing locally. The webpack bundle output is shown now.

I just added a few things to the .yarnrc; these make implicitly defined dependencies explicit. It gets rid of some yarn warnings when installing, and also makes us [mostly] PnP-ready.

I just added a few things to the .yarnrc; these make implicitly defined dependencies explicit. It gets rid of some yarn warnings when installing, and also makes us [mostly] PnP-ready.

Thanks, @zarsky-broad. What does "PnP-ready" mean?

@cahrens check it out: https://yarnpkg.com/features/pnp

@cahrens check it out: https://yarnpkg.com/features/pnp

@zarsky-broad Thanks! What are your thoughts about when (and how) we should try going to PnP?


Boxes  in Our Tours section of Our Tours page are not aligned

What is the bug*
A clear and concise description of what the bug is.
Boxes  in Our Tours section of Our Tours page are not aligned
Expected behavior
A clear and concise description of what you expected to happen.
Boxes should be aligned to give an aesthetically pleasant look
Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Allow me to work on this issue under hacktoberfest

Good catch @pandeatul27
Go ahead 👍
Show your support by stargazing the repo 🌟


Augment the lwaftr subcommands tests

As mentioned in a PR, we need more subcommands tests. After #412 we have a good start and a nice infrastructure, let's put it to good use. :-)
This issue is meant to record ideas and proposals about specific tests for each subcommand, like the one linked above. Please add anything you consider interesting, then we'll create other issues for each new group of tests we'll implement.

Comment by @dpino linked above:

Ideally the monitor test needs more test cases: keyword 'none', set by IPv4 address and setting by PID. I'm fine if those test cases are added in a new PR.

Ideally, there would be coverage for every option of every command. In practice, that's probably a bit heavyweight for this project, but adding coverage for any command / combination of commands which has had a known regression (at least from now on) strikes me as a good minimum.

Closing this issue since the monitor tool is barely used and it's likely to be gone in future versions. As for other commands that might be added in the future they must add their own tests.


Emit original event

Passing original event as an emitter option. This way we can handle the event on our side and stopPropagation ( for example ).
Same goal as https://github.com/zenorocha/clipboard.js/pull/475 but I think it's a more broad solution since we can handle the event the way we want.
Open to suggestions, ofc

@zenorocha any thoughts on this ?

@zenorocha thoughts on this ?

what is the status? I need the same feature¡, here +1

@zenorocha thoughts on this ?
Can it be merged?


Don't change rootElement when received node is the same

Issue:
storybook-html's render is always rewriting the root element's content even when it's not a different component. This appeared to be a problem for dynamic class changes (as in changing a prop in a AngularJS component) as it is removing the dom node and adding it again which kills stuff like animations (since from the browser's perspective this is not a change in the class attribute and it is a new node).
Props to @svipatov for finding the issue and help developing this PR.
What I did
Checked the reference of the received DOM node, if it is the same then the node was already mutated and there is no need to rewrite the root element.
How to test

Is this testable with Jest or Chromatic screenshots?
It could be, but I did not find any test for the changed file (I can add it if it's wanted)
Does this need a new example in the kitchen sink apps?
No
Does this need an update to the documentation?
No

If your answer is yes to any of these, please make sure to include it in your PR.

Can you maybe share a usecase ? I can think of knobs changes, but for this, we have a forceRender parameter that can be helpful. Here is the example in @storybook/react

Maybe forceRender is a bit misleading name, but it means that the story was rerendered forcibly (and not navigated). In other words, when knobs are changed, this parameter will be true.

Yeah, from your explanation it seems forceRender will do the trick. We'll change it and update/reopen the PR


Is this production-ready?

Thank you for these awesome components. I was wondering whether this is production ready? There are large number of open issues, so was wondering before usinging it in a project.

Yup - would also love to know.

I believe it is. I've used it in 3-4 production projects.

There still seems to be a lot of churn on this project and I'm considering using it on a large scale enterprise webapp.  Suggestions?

Please move this discussion the the appropriate channels. Thanks.