[SUPPORT] Row with _hoodie_is_deleted=True stored into target table

Describe the problem you faced
I am using the _hoodie_is_deleted column but some rows are written to the target table shouldn't.
When both the "Insert" and "Delete" (the delete being for said inserted row) are in the same "source" parquet file, the "Delete" is stored into the target table.
To Reproduce
Steps to reproduce the behavior:

Create a table using one parquet file (containing only Inserts) as the source
Generate a parquet file that stores the following incremental changes:

a Delete for one of the row that's currently in the target table
an Insert of a new row
a Delete of this new row that was just inserted

Add value to the _hoodie_is_deleted column accordingly and process/write the dataframe to the target table. If I follow the above, the rows will have the following values in this column, repsectively: True, False, True
The row that already was in the target table and deleted in the last update is indeed deleted
This other row whose Insert and Delete operations were stored in the same source parquet file (and so the Insert and Delete are in the same dataframe that was just processed) is present in the target table.

Expected behavior
The row whose Insert and Delete operations were stored in the same source parquet file shouldn't be written to the target table.

Hey @nsivabalan , I'm using upsert. I'm using a timestamp for hoodie.datasource.write.precombine.field.
So, in the case of a dataframe being processed that would contain an Insert and a Delete for the same record (so same key value), only the Delete is taken into account, and since there is nothing to be deleted in the target table, it looks as if it is inserting this Delete..
Any advice on the best way to handle this case?
Thank you

sorry. is your requirement, if we have insert and delete for the same record within one batch thats being ingested, you prefer final snapshot in hudi to show the insert record and not delete record?

sorry. is your requirement, if we have insert and delete for the same record within one batch thats being ingested, you prefer final snapshot in hudi to show the insert record and not delete record?

Hi @nsivabalan , no, I would expect the two to cancel each other out. For example: say someone creates a new user in a table, but realises they already had them under another name. They delete this user that was just created 1 minute before, and wouldn't expect the target table to show a row related to this duplicate user.

if the precombine value in delete record is higher than insert record, then delete will take precedence. if not, insert will take precedence.

Yes, it does seem to take precedence, but doesn't work as I would expect:

if you process a file that contains a delete for a row that's already in the target table, it is simply going to delete this row, which is what I would expect so that's totally fine.
if you process a file that contains an insert, and the delete for this same row, then as you said, the delete takes precedence, yes. But the target table will contain a row with the values from the delete, whereas this row should not be inserted into the target table in any way (same as above).

yes. unfortunately, this behavior can't be fixed easily.
For eg,

insert rec1 at time t0
delete rec1 at time t100
again re-insert rec1 at time t100000

lets say between (2) and (3), there were 100s of commits, hudi may not keep remembering every record it has ever seen. So, after (2), whenever next update happens, hudi will remove rec1 from its storage. and so later if rec1 is ingested again, hudi will consider it as an insert record.
If not, hudi has to keep track of every record that got inserted and deleted forever.
Dont' think makes sense for a large analytical storage system.
wrt your statement "But the target table will contain a row with the values from the delete, whereas this row should not be inserted into the target table in any way (same as above)." : the delete record will be ingested to hudi, but will have "_hoodie_is_deleted" set to true. But during next merge or compaction, the record will be removed. This record will be part of storage only for a short duration. Also, there are other ways to trigger deletes. Please check out the details here.  not all of them storage the value with _hoodie_is_deleted as true.

If you don't have any follow up questions, I will go ahead and close the issue for now. do let us know if you need any more clarifications. thanks!


[Storage] Fix #10620 in azure cli to support ADLS

This checklist is used to make sure that common guidelines for a pull request are followed.

For new extensions:

[ ] My extension description/summary conforms to the Extension Summary Guidelines.

If this PR is for a new extension or change to an existing extension, use the following to try out the changes in this PR:
docker run -it microsoft/azure-cli:latest
export EXT=<NAME>
pip install --upgrade --target ~/.azure/cliextensions/$EXT "git+https://github.com/Juliehzl/azure-cli-extensions.git@ADLS#subdirectory=src/$EXT&egg=$EXT"


fix: outdated link to foundry config docs

What ❔
I've found an outdated link pointing to oundry config documentation. This PR replaces the outdated link with a working link.
Why ❔
To keep external resources approachable.
Checklist

[x] PR title corresponds to the body of PR (we generate changelog entries from PRs).
[x] Documentation comments have been added / updated.

@vladbochok @StanislavBreadless is there anything from my side that needs to be done?

@krauspt Hey, we are sorry for the long wait. Could you please resolve the conflicts? (Basically replicate your change into the following file: https://github.com/matter-labs/era-contracts/blob/main/l1-contracts/foundry.toml)

@StanislavBreadless sorry, I wasn't able to do it within this PR. I opened https://github.com/matter-labs/era-contracts/pull/172. Thank you


Suggested refactor for model.h/cc

I'm working on integrating this into my engine, which uses OpenSceneGraph rather than raw openGL functions. Because of this, I can't use the parts of model.h/cc that deal with opengl resources, but I would like to use the glsl header factory part. I'd like to suggest splitting model.h/cc into two classes, one class to generate the glsl header from the atmosphere params, and a different class (in different files) to setup the GL resources (textures etc). That way people can more easily use one without the other.

After spending more time looking at the code, I decided to use uniforms at render-time instead of generating constants with the header factory. The main reason being uniforms give more flexibility and it looks like I can fold most of the constants into uniforms anyway. I guess the main purpose of the header factory is to speed up the precomputation step with constant folding, with less benifit at render-time. So I'm not actually sure how useful my suggestion is after all.


Add missing import, remove unused imports

azure.core.pipelines.policies.universal is missing an import of typing.IO. This adds that and removes the module's unused imports.

Can one of the admins verify this patch?

Codecov Report

Merging #5337 into master will decrease coverage by 3.93%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #5337      +/-   ##
==========================================
- Coverage   52.26%   48.32%   -3.94%
==========================================
  Files       10892     6556    -4336
  Lines      233804   188636   -45168
==========================================
- Hits       122191    91167   -31024
+ Misses     111613    97469   -14144

Impacted Files
Coverage Δ

...ure-core/azure/core/pipeline/policies/universal.py
75.48% <100%> (-0.62%)
:arrow_down:

...afactory/azure/mgmt/datafactory/models/__init__.py
50.67% <0%> (-49.33%)
:arrow_down:

...ights/azure/applicationinsights/models/__init__.py
51% <0%> (-49%)
:arrow_down:

...e/cognitiveservices/vision/face/models/__init__.py
51.08% <0%> (-48.92%)
:arrow_down:

...services/language/textanalytics/models/__init__.py
51.16% <0%> (-48.84%)
:arrow_down:

...iveservices/search/entitysearch/models/__init__.py
51.25% <0%> (-48.75%)
:arrow_down:

azure-eventgrid/azure/eventgrid/models/__init__.py
51.25% <0%> (-48.75%)
:arrow_down:

...ration/azure/mgmt/datamigration/models/__init__.py
51.27% <0%> (-48.73%)
:arrow_down:

...ure/mgmt/recoveryservicesbackup/models/__init__.py
51.46% <0%> (-48.54%)
:arrow_down:

...ce/azure/mgmt/containerinstance/models/__init__.py
51.8% <0%> (-48.2%)
:arrow_down:

... and 4437 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4d81a2a...c5ff984. Read the comment docs.

I believe this is being resolved in #5021
I'm fine with merging this too.

Yes, looks like #5152 fixes it and is much more comprehensive, so I'm closing this.


Work update on styling template and other pages.

@pierre Okay! best. I will do this. thanks

@pierre Okay! best. I will do this. thanks


Add option to make the private key non-exportable

We have a company policy that the private key should not be exportable. This adds an option to make the key non-exportable.
I kept the behavior as before. By default the key will be exportable, only when the appsetting is set to "False" it will be non-exportable.
It might be preferable to set the default to non-exportable. I can't think of a reason to export a Let's Encrypt-certificate, as you can easily request a new one.

I think this is great for now to keep it backwards compatible,
I created an issue to make it non-exportable by default in the next major version: https://github.com/Lone-Coder/letsencrypt-win-simple/issues/369


fix update node engine to 16

This was missed 2 years ago when the version in the .nvmrc file was changed https://github.com/10up/wp-scaffold/commit/b1cd51f7ca900df5afb50ea96d4d6e8e1faf7269

Using node 16 results in a failure to build
Now using node v16.20.2 (npm v8.19.4)
npm ERR! code EBADENGINE
npm ERR! engine Unsupported engine
npm ERR! engine Not compatible with your version of node/npm<EMAIL_ADDRESS>npm ERR! notsup Not compatible with your version of node/npm<EMAIL_ADDRESS>npm ERR! notsup Required: {"node":">=18.12.0"}
npm ERR! notsup Actual:   {"npm":"8.19.4","node":"v16.20.2"}

Manually editing the file to version 18 resolves the issue


Update version: WiresharkFoundation.Wireshark version 4.2.4

Checklist for Pull Requests

[X] Have you signed the Contributor License Agreement?
[X] Is there a linked Issue?
Resolve https://github.com/microsoft/winget-pkgs/issues/146941

Manifests

[X] Have you checked that there aren't other open pull requests for the same manifest update/change?

[ ] Have you tested your manifest locally with winget install --manifest <path>?
[X] Does your manifest conform to the 1.6 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

For moderators
This account is automated by Github Actions and the source code was created by CoolPlayLin. If you have any questions about any pull request, don't hesitate to ping Exorcism0666.

[!warning]
With the new version of Komac, numerous Pull Requests may contain bugs or unwanted elements. If such issues arise, please tag russellbanks so that these bugs can be addressed in future updates.

[!important]
Please carefully review these Pull Request before merging. If it is a Pull Request for removing incorrect content and the URLs are issue, free upon manual checking, please close this Pull Request directly. (It is best to inform Exorcism0666 of the closure.)

良い一日をお過ごしください！


293 streams save xy point data to csv

post processing pipeline is now wrapped up in a single function call. Should make things a bit easier going forward....
Next step will be processing that 2d data a bit more richly so that reprojection error can filter out erroneous points

the csv files that are getting saved in here are getting to be a bit much....I need to start controlling that a bit more tightly...


Roll back Timo Jakkola’s crossing penalties, as they are covered with Eero’s edge traversal times

closed in #16


HDDS-1054. List Multipart uploads in a bucket

This Jira is to implement in ozone to list of in-progress multipart uploads in a bucket.
[https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListMPUpload.html]

See: https://issues.apache.org/jira/browse/HDDS-1054

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
73
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

0
mvndep
30
Maven dependency ordering for branch

+1
mvninstall
599
trunk passed

+1
compile
371
trunk passed

+1
checkstyle
73
trunk passed

+1
mvnsite
0
trunk passed

+1
shadedclient
997
branch has no errors when building and testing our client artifacts.

+1
javadoc
164
trunk passed

0
spotbugs
432
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
629
trunk passed

_ Patch Compile Tests _

0
mvndep
21
Maven dependency ordering for patch

+1
mvninstall
558
the patch passed

+1
compile
371
the patch passed

+1
cc
371
the patch passed

+1
javac
371
the patch passed

+1
checkstyle
77
the patch passed

+1
mvnsite
0
the patch passed

-1
whitespace
0
The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1
shadedclient
740
patch has no errors when building and testing our client artifacts.

+1
javadoc
166
the patch passed

-1
findbugs
443
hadoop-ozone generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

_ Other Tests _

+1
unit
347
hadoop-hdds in the patch passed.

-1
unit
225
hadoop-ozone in the patch failed.

+1
asflicense
39
The patch does not generate ASF License warnings.

6300

Reason
Tests

FindBugs
module:hadoop-ozone

Dead store to multipartKey in org.apache.hadoop.ozone.om.KeyManagerImpl.listMultipartUploads(String, String, String)  At KeyManagerImpl.java:org.apache.hadoop.ozone.om.KeyManagerImpl.listMultipartUploads(String, String, String)  At KeyManagerImpl.java:[line 1312]

Failed junit tests
hadoop.ozone.om.ratis.TestOzoneManagerRatisServer

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1277

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc

uname
Linux 6ec7387069c5 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / cf5d895

Default Java
1.8.0_222

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/1/artifact/out/whitespace-eol.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/1/artifact/out/new-findbugs-hadoop-ozone.html

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/1/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/1/testReport/

Max. process+thread count
1272 (vs. ulimit of 5500)

modules
C: hadoop-ozone/common hadoop-ozone/client hadoop-ozone/ozone-manager hadoop-ozone/s3gateway hadoop-ozone/dist U: hadoop-ozone

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/1/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
41
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

0
mvndep
11
Maven dependency ordering for branch

+1
mvninstall
591
trunk passed

+1
compile
353
trunk passed

+1
checkstyle
59
trunk passed

+1
mvnsite
0
trunk passed

+1
shadedclient
785
branch has no errors when building and testing our client artifacts.

+1
javadoc
149
trunk passed

0
spotbugs
427
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
622
trunk passed

_ Patch Compile Tests _

0
mvndep
16
Maven dependency ordering for patch

+1
mvninstall
547
the patch passed

+1
compile
355
the patch passed

+1
cc
355
the patch passed

+1
javac
355
the patch passed

+1
checkstyle
64
the patch passed

+1
mvnsite
0
the patch passed

-1
whitespace
0
The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1
shadedclient
626
patch has no errors when building and testing our client artifacts.

+1
javadoc
149
the patch passed

-1
findbugs
444
hadoop-ozone generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

_ Other Tests _

+1
unit
309
hadoop-hdds in the patch passed.

-1
unit
170
hadoop-ozone in the patch failed.

+1
asflicense
34
The patch does not generate ASF License warnings.

5670

Reason
Tests

FindBugs
module:hadoop-ozone

Dead store to multipartKey in org.apache.hadoop.ozone.om.KeyManagerImpl.listMultipartUploads(String, String, String)  At KeyManagerImpl.java:org.apache.hadoop.ozone.om.KeyManagerImpl.listMultipartUploads(String, String, String)  At KeyManagerImpl.java:[line 1313]

Failed junit tests
hadoop.ozone.om.ratis.TestOzoneManagerRatisServer

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1277

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc

uname
Linux 08d3de8a4e7f 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 9b8359b

Default Java
1.8.0_222

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/2/artifact/out/whitespace-eol.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/2/artifact/out/new-findbugs-hadoop-ozone.html

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/2/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/2/testReport/

Max. process+thread count
1371 (vs. ulimit of 5500)

modules
C: hadoop-ozone/common hadoop-ozone/client hadoop-ozone/ozone-manager hadoop-ozone/s3gateway hadoop-ozone/dist U: hadoop-ozone

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/2/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

In the list, we have different parameters like key-marker, max-uploads and few other parameters.
Are we planning to do this in a new Jira?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
115
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

0
mvndep
32
Maven dependency ordering for branch

+1
mvninstall
767
trunk passed

+1
compile
445
trunk passed

+1
checkstyle
86
trunk passed

+1
mvnsite
0
trunk passed

+1
shadedclient
1129
branch has no errors when building and testing our client artifacts.

+1
javadoc
201
trunk passed

0
spotbugs
554
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
828
trunk passed

_ Patch Compile Tests _

0
mvndep
29
Maven dependency ordering for patch

+1
mvninstall
724
the patch passed

+1
compile
443
the patch passed

+1
cc
443
the patch passed

+1
javac
443
the patch passed

+1
checkstyle
76
the patch passed

+1
mvnsite
0
the patch passed

-1
whitespace
0
The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1
shadedclient
827
patch has no errors when building and testing our client artifacts.

+1
javadoc
216
the patch passed

-1
findbugs
546
hadoop-ozone generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

_ Other Tests _

+1
unit
433
hadoop-hdds in the patch passed.

-1
unit
303
hadoop-ozone in the patch failed.

+1
asflicense
46
The patch does not generate ASF License warnings.

7713

Reason
Tests

FindBugs
module:hadoop-ozone

Dead store to multipartKey in org.apache.hadoop.ozone.om.KeyManagerImpl.listMultipartUploads(String, String, String)  At KeyManagerImpl.java:org.apache.hadoop.ozone.om.KeyManagerImpl.listMultipartUploads(String, String, String)  At KeyManagerImpl.java:[line 1313]

Failed junit tests
hadoop.ozone.om.ratis.TestOzoneManagerRatisServer

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1277

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc

uname
Linux eecbd90521e2 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 6244502

Default Java
1.8.0_222

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/3/artifact/out/whitespace-eol.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/3/artifact/out/new-findbugs-hadoop-ozone.html

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/3/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/3/testReport/

Max. process+thread count
1295 (vs. ulimit of 5500)

modules
C: hadoop-ozone/common hadoop-ozone/client hadoop-ozone/ozone-manager hadoop-ozone/s3gateway hadoop-ozone/dist U: hadoop-ozone

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/3/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
63
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

0
mvndep
28
Maven dependency ordering for branch

+1
mvninstall
600
trunk passed

+1
compile
346
trunk passed

+1
checkstyle
64
trunk passed

+1
mvnsite
0
trunk passed

+1
shadedclient
848
branch has no errors when building and testing our client artifacts.

+1
javadoc
158
trunk passed

0
spotbugs
433
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
636
trunk passed

_ Patch Compile Tests _

0
mvndep
19
Maven dependency ordering for patch

+1
mvninstall
581
the patch passed

+1
compile
374
the patch passed

+1
cc
374
the patch passed

+1
javac
374
the patch passed

-0
checkstyle
37
hadoop-ozone: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

+1
mvnsite
0
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
661
patch has no errors when building and testing our client artifacts.

+1
javadoc
153
the patch passed

-1
findbugs
441
hadoop-ozone generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

_ Other Tests _

+1
unit
383
hadoop-hdds in the patch passed.

-1
unit
329
hadoop-ozone in the patch failed.

+1
asflicense
34
The patch does not generate ASF License warnings.

6160

Reason
Tests

FindBugs
module:hadoop-ozone

Dead store to multipartKey in org.apache.hadoop.ozone.om.KeyManagerImpl.listMultipartUploads(String, String, String)  At KeyManagerImpl.java:org.apache.hadoop.ozone.om.KeyManagerImpl.listMultipartUploads(String, String, String)  At KeyManagerImpl.java:[line 1313]

Failed junit tests
hadoop.ozone.om.ratis.TestOzoneManagerRatisServer

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1277

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc

uname
Linux a8ec28791adb 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 10ec31d

Default Java
1.8.0_222

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/4/artifact/out/diff-checkstyle-hadoop-ozone.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/4/artifact/out/new-findbugs-hadoop-ozone.html

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/4/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/4/testReport/

Max. process+thread count
1354 (vs. ulimit of 5500)

modules
C: hadoop-ozone/common hadoop-ozone/client hadoop-ozone/ozone-manager hadoop-ozone/s3gateway hadoop-ozone/dist U: hadoop-ozone

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/4/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

In the list, we have different parameters like key-marker, max-uploads and few other parameters.
Are we planning to do this in a new Jira?

Yes. I pagination is not yet implemented. This is the minimal implementation to support docker registry. Will create an other jira to add all the pagination magic.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
0
Docker mode activated.

-1
patch
10
https://github.com/apache/hadoop/pull/1277 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hadoop/pull/1277

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/6/console

versions
git=2.17.1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
0
Docker mode activated.

-1
patch
10
https://github.com/apache/hadoop/pull/1277 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hadoop/pull/1277

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/7/console

versions
git=2.17.1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
76
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

0
mvndep
14
Maven dependency ordering for branch

+1
mvninstall
616
trunk passed

+1
compile
386
trunk passed

+1
checkstyle
69
trunk passed

+1
mvnsite
0
trunk passed

+1
shadedclient
926
branch has no errors when building and testing our client artifacts.

+1
javadoc
175
trunk passed

0
spotbugs
471
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
709
trunk passed

_ Patch Compile Tests _

0
mvndep
21
Maven dependency ordering for patch

+1
mvninstall
610
the patch passed

+1
compile
409
the patch passed

+1
cc
409
the patch passed

+1
javac
409
the patch passed

+1
checkstyle
95
the patch passed

+1
mvnsite
0
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
808
patch has no errors when building and testing our client artifacts.

+1
javadoc
198
the patch passed

+1
findbugs
685
the patch passed

_ Other Tests _

+1
unit
374
hadoop-hdds in the patch passed.

-1
unit
262
hadoop-ozone in the patch failed.

+1
asflicense
40
The patch does not generate ASF License warnings.

6658

Reason
Tests

Failed junit tests
hadoop.ozone.om.TestKeyManagerUnit

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/8/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1277

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc

uname
Linux fc03450f9b59 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / c749f62

Default Java
1.8.0_222

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/8/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/8/testReport/

Max. process+thread count
1237 (vs. ulimit of 5500)

modules
C: hadoop-ozone/common hadoop-ozone/client hadoop-ozone/ozone-manager hadoop-ozone/s3gateway hadoop-ozone/dist U: hadoop-ozone

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/8/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
34
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

0
mvndep
12
Maven dependency ordering for branch

+1
mvninstall
597
trunk passed

+1
compile
383
trunk passed

+1
checkstyle
68
trunk passed

+1
mvnsite
0
trunk passed

+1
shadedclient
917
branch has no errors when building and testing our client artifacts.

+1
javadoc
167
trunk passed

0
spotbugs
470
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
701
trunk passed

_ Patch Compile Tests _

0
mvndep
22
Maven dependency ordering for patch

+1
mvninstall
593
the patch passed

+1
compile
394
the patch passed

+1
cc
394
the patch passed

+1
javac
394
the patch passed

+1
checkstyle
81
the patch passed

+1
mvnsite
0
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
756
patch has no errors when building and testing our client artifacts.

+1
javadoc
171
the patch passed

+1
findbugs
639
the patch passed

_ Other Tests _

+1
unit
303
hadoop-hdds in the patch passed.

-1
unit
1591
hadoop-ozone in the patch failed.

+1
asflicense
45
The patch does not generate ASF License warnings.

7646

Reason
Tests

Failed junit tests
hadoop.ozone.TestOzoneConfigurationFields

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/9/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1277

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc

uname
Linux 376784018ab2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 8c0759d

Default Java
1.8.0_222

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/9/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/9/testReport/

Max. process+thread count
5019 (vs. ulimit of 5500)

modules
C: hadoop-ozone/common hadoop-ozone/client hadoop-ozone/ozone-manager hadoop-ozone/s3gateway hadoop-ozone/dist U: hadoop-ozone

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/9/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
39
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

0
mvndep
14
Maven dependency ordering for branch

+1
mvninstall
614
trunk passed

+1
compile
388
trunk passed

+1
checkstyle
99
trunk passed

+1
mvnsite
0
trunk passed

+1
shadedclient
922
branch has no errors when building and testing our client artifacts.

+1
javadoc
178
trunk passed

0
spotbugs
431
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
639
trunk passed

_ Patch Compile Tests _

0
mvndep
26
Maven dependency ordering for patch

+1
mvninstall
561
the patch passed

+1
compile
423
the patch passed

+1
cc
423
the patch passed

+1
javac
423
the patch passed

+1
checkstyle
83
the patch passed

+1
mvnsite
0
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
778
patch has no errors when building and testing our client artifacts.

+1
javadoc
192
the patch passed

+1
findbugs
770
the patch passed

_ Other Tests _

+1
unit
310
hadoop-hdds in the patch passed.

-1
unit
2359
hadoop-ozone in the patch failed.

+1
asflicense
56
The patch does not generate ASF License warnings.

8614

Reason
Tests

Failed junit tests
hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion

hadoop.ozone.client.rpc.TestContainerStateMachineFailures

hadoop.ozone.client.rpc.TestCommitWatcher

hadoop.ozone.om.TestSecureOzoneManager

hadoop.ozone.client.rpc.TestWatchForCommit

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/10/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1277

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc

uname
Linux 371835c48863 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 915cbc9

Default Java
1.8.0_222

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/10/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/10/testReport/

Max. process+thread count
4430 (vs. ulimit of 5500)

modules
C: hadoop-ozone/common hadoop-ozone/client hadoop-ozone/ozone-manager hadoop-ozone/s3gateway hadoop-ozone/dist U: hadoop-ozone

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/10/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

When i run the command I see below output, whereas it is not showing up bucketName, keyMarker other fields.
bash-4.2$ aws s3api --endpoint http://s3g:9878 list-multipart-uploads --bucket b1234 --prefix mpu
{
"Uploads": [
{
"Initiator": {
"DisplayName": "Not Supported",
"ID": "NOT-SUPPORTED"
},
"Initiated": "2019-09-03T18:39:35.916Z",
"UploadId": "24eea7f4-52db-4a0f-978a-f06cb7a57657-102730037717565440",
"StorageClass": "STANDARD",
"Key": "mpukey",
"Owner": {
"DisplayName": "Not Supported",
"ID": "NOT-SUPPORTED"
}
},
{
"Initiator": {
"DisplayName": "Not Supported",
"ID": "NOT-SUPPORTED"
},
"Initiated": "2019-09-03T18:39:37.816Z",
"UploadId": "81c0e5c2-db11-4b11-a5f7-81f48bdbfb04-102730037842083841",
"StorageClass": "STANDARD",
"Key": "mpukey1",
"Owner": {
"DisplayName": "Not Supported",
"ID": "NOT-SUPPORTED"
}
},
{
"Initiator": {
"DisplayName": "Not Supported",
"ID": "NOT-SUPPORTED"
},
"Initiated": "2019-09-03T18:39:39.259Z",
"UploadId": "4aab75b8-1954-4e8a-a658-0d403bcbc42f-102730037936717826",
"StorageClass": "STANDARD",
"Key": "mpukey2",
"Owner": {
"DisplayName": "Not Supported",
"ID": "NOT-SUPPORTED"
}
}
]
}

Have few comments in place.
Can we open Jira's for all TODO's in this Jira for tracking purpose?
1. Audit support for the new method.

2. Pagination support and other query parameters support.

3. If replication type will not be handled in this Jira, can you open Jira for this one too.

Sure. Audit + the replication type support is added to this patch. I created HDDS-2130 for the pagination support.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
1208
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

0
mvndep
77
Maven dependency ordering for branch

-1
mvninstall
32
hadoop-ozone in trunk failed.

-1
compile
19
hadoop-ozone in trunk failed.

+1
checkstyle
100
trunk passed

+1
mvnsite
0
trunk passed

+1
shadedclient
937
branch has no errors when building and testing our client artifacts.

-1
javadoc
13
hadoop-hdds in trunk failed.

-1
javadoc
15
hadoop-ozone in trunk failed.

0
spotbugs
159
Used deprecated FindBugs config; considering switching to SpotBugs.

-1
findbugs
23
hadoop-ozone in trunk failed.

_ Patch Compile Tests _

0
mvndep
23
Maven dependency ordering for patch

-1
mvninstall
31
hadoop-ozone in the patch failed.

-1
compile
22
hadoop-ozone in the patch failed.

-1
cc
22
hadoop-ozone in the patch failed.

-1
javac
22
hadoop-ozone in the patch failed.

-0
checkstyle
24
hadoop-hdds: The patch generated 5 new + 9 unchanged - 1 fixed = 14 total (was 10)

-0
checkstyle
71
hadoop-ozone: The patch generated 373 new + 2410 unchanged - 15 fixed = 2783 total (was 2425)

+1
mvnsite
0
the patch passed

+1
whitespace
1
The patch has no whitespace issues.

+1
shadedclient
726
patch has no errors when building and testing our client artifacts.

-1
javadoc
14
hadoop-hdds in the patch failed.

-1
javadoc
15
hadoop-ozone in the patch failed.

-1
findbugs
23
hadoop-ozone in the patch failed.

_ Other Tests _

-1
unit
163
hadoop-hdds in the patch failed.

-1
unit
25
hadoop-ozone in the patch failed.

+1
asflicense
29
The patch does not generate ASF License warnings.

4264

Reason
Tests

Failed junit tests
hadoop.ozone.container.ozoneimpl.TestOzoneContainer

hadoop.ozone.container.keyvalue.TestKeyValueContainer

Subsystem
Report/Notes

Docker
Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1277

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc

uname
Linux 86494a8d4639 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 6a9f7ca

Default Java
1.8.0_222

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/branch-mvninstall-hadoop-ozone.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/branch-compile-hadoop-ozone.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/branch-javadoc-hadoop-hdds.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/branch-javadoc-hadoop-ozone.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/branch-findbugs-hadoop-ozone.txt

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/patch-mvninstall-hadoop-ozone.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/patch-compile-hadoop-ozone.txt

cc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/patch-compile-hadoop-ozone.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/patch-compile-hadoop-ozone.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/diff-checkstyle-hadoop-hdds.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/diff-checkstyle-hadoop-ozone.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/patch-javadoc-hadoop-hdds.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/patch-javadoc-hadoop-ozone.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/patch-findbugs-hadoop-ozone.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/patch-unit-hadoop-hdds.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/testReport/

Max. process+thread count
306 (vs. ulimit of 5500)

modules
C: hadoop-hdds/common hadoop-ozone/client hadoop-ozone/common hadoop-ozone/dist hadoop-ozone/ozone-manager hadoop-ozone/s3gateway U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/11/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

/retest

/retest

/retest

/retest

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
43
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 4 new or modified test files.

_ trunk Compile Tests _

0
mvndep
48
Maven dependency ordering for branch

-1
mvninstall
33
hadoop-ozone in trunk failed.

-1
compile
23
hadoop-ozone in trunk failed.

+1
checkstyle
124
trunk passed

+1
mvnsite
0
trunk passed

+1
shadedclient
1003
branch has no errors when building and testing our client artifacts.

+1
javadoc
164
trunk passed

0
spotbugs
184
Used deprecated FindBugs config; considering switching to SpotBugs.

-1
findbugs
26
hadoop-ozone in trunk failed.

_ Patch Compile Tests _

0
mvndep
32
Maven dependency ordering for patch

-1
mvninstall
35
hadoop-ozone in the patch failed.

-1
compile
27
hadoop-ozone in the patch failed.

-1
cc
27
hadoop-ozone in the patch failed.

-1
javac
27
hadoop-ozone in the patch failed.

-0
checkstyle
31
hadoop-hdds: The patch generated 5 new + 9 unchanged - 1 fixed = 14 total (was 10)

-0
checkstyle
97
hadoop-ozone: The patch generated 448 new + 2401 unchanged - 92 fixed = 2849 total (was 2493)

+1
mvnsite
0
the patch passed

-1
whitespace
0
The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1
shadedclient
765
patch has no errors when building and testing our client artifacts.

-1
javadoc
91
hadoop-ozone generated 9 new + 249 unchanged - 7 fixed = 258 total (was 256)

-1
findbugs
28
hadoop-ozone in the patch failed.

_ Other Tests _

-1
unit
247
hadoop-hdds in the patch failed.

-1
unit
31
hadoop-ozone in the patch failed.

+1
asflicense
35
The patch does not generate ASF License warnings.

3706

Reason
Tests

Failed junit tests
hadoop.hdds.scm.container.placement.algorithms.TestSCMContainerPlacementRackAware

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/12/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1277

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc

uname
Linux 4392a4c17837 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / c474e24

Default Java
1.8.0_222

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/12/artifact/out/branch-mvninstall-hadoop-ozone.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/12/artifact/out/branch-compile-hadoop-ozone.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/12/artifact/out/branch-findbugs-hadoop-ozone.txt

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/12/artifact/out/patch-mvninstall-hadoop-ozone.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/12/artifact/out/patch-compile-hadoop-ozone.txt

cc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/12/artifact/out/patch-compile-hadoop-ozone.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/12/artifact/out/patch-compile-hadoop-ozone.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/12/artifact/out/diff-checkstyle-hadoop-hdds.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/12/artifact/out/diff-checkstyle-hadoop-ozone.txt

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/12/artifact/out/whitespace-eol.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/12/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/12/artifact/out/patch-findbugs-hadoop-ozone.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/12/artifact/out/patch-unit-hadoop-hdds.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/12/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/12/testReport/

Max. process+thread count
534 (vs. ulimit of 5500)

modules
C: hadoop-hdds/common hadoop-ozone/client hadoop-ozone/common hadoop-ozone/dist hadoop-ozone/ozone-manager hadoop-ozone/s3gateway U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/12/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
3434
Docker mode activated.

_ Prechecks _

+1
dupname
2
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 4 new or modified test files.

_ trunk Compile Tests _

0
mvndep
44
Maven dependency ordering for branch

-1
mvninstall
31
hadoop-ozone in trunk failed.

-1
compile
22
hadoop-ozone in trunk failed.

+1
checkstyle
125
trunk passed

+1
mvnsite
0
trunk passed

+1
shadedclient
897
branch has no errors when building and testing our client artifacts.

-1
javadoc
47
hadoop-ozone in trunk failed.

0
spotbugs
164
Used deprecated FindBugs config; considering switching to SpotBugs.

-1
findbugs
27
hadoop-ozone in trunk failed.

_ Patch Compile Tests _

0
mvndep
29
Maven dependency ordering for patch

-1
mvninstall
33
hadoop-ozone in the patch failed.

-1
compile
25
hadoop-ozone in the patch failed.

-1
cc
25
hadoop-ozone in the patch failed.

-1
javac
25
hadoop-ozone in the patch failed.

-0
checkstyle
29
hadoop-hdds: The patch generated 6 new + 8 unchanged - 2 fixed = 14 total (was 10)

-0
checkstyle
96
hadoop-ozone: The patch generated 370 new + 2478 unchanged - 15 fixed = 2848 total (was 2493)

+1
mvnsite
0
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
660
patch has no errors when building and testing our client artifacts.

-1
javadoc
51
hadoop-ozone in the patch failed.

-1
findbugs
27
hadoop-ozone in the patch failed.

_ Other Tests _

+1
unit
249
hadoop-hdds in the patch passed.

-1
unit
29
hadoop-ozone in the patch failed.

+1
asflicense
32
The patch does not generate ASF License warnings.

6687

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/13/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1277

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc

uname
Linux 68f3b34ec47c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 1029060

Default Java
1.8.0_222

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/13/artifact/out/branch-mvninstall-hadoop-ozone.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/13/artifact/out/branch-compile-hadoop-ozone.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/13/artifact/out/branch-javadoc-hadoop-ozone.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/13/artifact/out/branch-findbugs-hadoop-ozone.txt

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/13/artifact/out/patch-mvninstall-hadoop-ozone.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/13/artifact/out/patch-compile-hadoop-ozone.txt

cc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/13/artifact/out/patch-compile-hadoop-ozone.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/13/artifact/out/patch-compile-hadoop-ozone.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/13/artifact/out/diff-checkstyle-hadoop-hdds.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/13/artifact/out/diff-checkstyle-hadoop-ozone.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/13/artifact/out/patch-javadoc-hadoop-ozone.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/13/artifact/out/patch-findbugs-hadoop-ozone.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/13/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/13/testReport/

Max. process+thread count
498 (vs. ulimit of 5500)

modules
C: hadoop-hdds/common hadoop-ozone/client hadoop-ozone/common hadoop-ozone/dist hadoop-ozone/ozone-manager hadoop-ozone/s3gateway U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1277/13/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

+1 LGTM.
Thank You @elek for the contribution.
I will commit this to the trunk.


Travel to Maraxsis

Due to its position, the path to go to Maraxsis would best be from nauvis.
Or if it's intended as a sort of dificulty, moving it to the other side of the star, it would still justify a path of 30 000 km

For now, you could install this one: https://mods.factorio.com/mod/Starmap_Nexuz

I mean this as a default path


How to systemd on docker container Centos 7.2

I have docker infra running in dev env and i want to manage my docker docker container for use case, testing, POC purpose that my docker container should be enable with systemd
So that i can use init process  systemctl command to configure my docker container dynamic ,,,
Atul

Closing this ticket due to its age, and the impending refactor. If you think this is in error, feel free to reopen. Thanks!


Added negation, fixed CAST, Added SUM DISTINCT

Added a negation term
fixed CAST function and SqlTypes enum
added SUM(DISTINCT ...)

Coverage decreased (-0.1%) to 97.978% when pulling d83d655bb7a543605130d953ef9fa7116c285fff on function_improvements into a5e08d172af0cf7e263c95456ee8699c3a2e3183 on master.


Sendable DebugCallback

[x] Added an entry to CHANGELOG_VULKANO.md or CHANGELOG_VK_SYS.md if knowledge of this change could be valuable to users
[ ] Updated documentation to reflect any user-facing changes - in this repository
[ ] Updated documentation to reflect any user-facing changes - PR to the guide that fixes existing documentation invalidated by this PR.

On the master branch the DebugCallback struct can not be sent between threads due to the user_callback field being non Sendable.
It's useful to be able to instantiate Vulkano on one thread and then send the instances to a worker thread for use, but right now this would require the worker thread to instantiate it's own DebugCallback, which could cause two to be registered at the same time which results in double logging.
This change simply adds the Send trait bound to the user_callback field to fix this. The DebugCallback::new function already required the Send trait bound, so I do not believe this can break any existing code. I also don't believe this introduces any unsoundness since the field is only there to keep the callback object alive.
Including error to aid searchability:
error[E0277]: `(dyn for<'r, 's> std::ops::Fn(&'r instance::debug::Message<'s>) + 'static)` cannot be sent between threads safely
   --> vulkano/src/instance/debug.rs:356:9
    |
356 |         thread::spawn(move || {
    |         ^^^^^^^^^^^^^ `(dyn for<'r, 's> std::ops::Fn(&'r instance::debug::Message<'s>) + 'static)` cannot be sent between threads safely
    |
    = help: the trait `std::marker::Send` is not implemented for `(dyn for<'r, 's> std::ops::Fn(&'r instance::debug::Message<'s>) + 'static)`
    = note: required because of the requirements on the impl of `std::marker::Send` for `std::ptr::Unique<(dyn for<'r, 's> std::ops::Fn(&'r instance::debug::Message<'s>) + 'static)>`
    = note: required because it appears within the type `std::boxed::Box<(dyn for<'r, 's> std::ops::Fn(&'r instance::debug::Message<'s>) + 'static)>`
    = note: required because of the requirements on the impl of `std::marker::Send` for `std::ptr::Unique<std::boxed::Box<(dyn for<'r, 's> std::ops::Fn(&'r instance::debug::Message<'s>) + 'static)>>`
    = note: required because it appears within the type `std::boxed::Box<std::boxed::Box<(dyn for<'r, 's> std::ops::Fn(&'r instance::debug::Message<'s>) + 'static)>>`
    = note: required because it appears within the type `instance::debug::DebugCallback`
    = note: required because it appears within the type `std::result::Result<instance::debug::DebugCallback, instance::debug::DebugCallbackCreationError>`
    = note: required because it appears within the type `[closure@vulkano/src/instance/debug.rs:356:23: 358:10 callback:std::result::Result<instance::debug::DebugCallback, instance::debug::DebugCallbackCreationError>]`

Reading through vulkan documentation I see nothing where the callback should be pinned to a single thread; therefore, I don't see an issue accepting this.


怎么添加适配新的游戏，比如说小丑牌

我想在新的游戏上进行测试

Hi. Thanks for reaching out. We provide an overall perspective on how to adapt the framework to a new game: https://github.com/BAAI-Agents/Cradle/discussions/55 .
And we are still trying to re-organize the code to make it easier to understand and adapt to new game.


Scheduler pod for dask is also a preemptible instance on GCP

When using preemptible instances with dask, the scheduler pod is also a preemptible instance type.
While dask can handle some workers being killed, the scheduler instance being killed would cause all progress in the job to fail.

It seems that preemptible instances have been more stable recently. Also there haven't been any other reports of problems here. Closing.


Confusing link states for not reachable instances

Feature/Improvement
If an instance becomes unreachable after successfully being assigned to another instance, the link state is not updated but listed as assigned in both components (the unreachable one and the one it was connected to). In my opinion the correct thing to do would be to update the linkstate to outdated since we can't know anything about this connection without reaching the instance.

Referring #122

Closed with #122

Closed with #122


[GSSOC'23] Restricting Contact Us Form to Accept Only String Values for Name Field

the contact us form should only accept name if it is a string

I am a full-stack developer, so adding this feature is not a big task for me. so please assign me this and I am also a GSSOC 2023 contributor

Hello, I did these types of features in my personal project. please assign me this issue.
I'm GSSOC'23 contributor.


Better convert OBS-tq and OBS-tn; better handle \ca and \va fields

Also better handles unmarked text inside footnotes.

Pull Request Test Coverage Report for Build 155

105 of 174 (60.34%)  changed or added relevant lines in 6 files are covered.
8 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.5%) to 81.807%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

linters/obs_notes_linter.py
8
15
53.33%

converters/md2html_converter.py
4
35
11.43%

tx_usfm_tools/singlehtmlRenderer.py
27
58
46.55%

Files with Coverage Reduction
New Missed Lines
%

tx_usfm_tools/singlehtmlRenderer.py
8
48.58%

Totals

Change from base Build 151:
-0.5%

Covered Lines:
5423

Relevant Lines:
6629

💛  - Coveralls


Installer Not Completing Successfully When Run in Hive Pods

We're getting failed clusters where all masters are left in NotReady state due to cni not being configured when we run the installer in Hive pods specifically. If we run externally even with the same image via podman, the cluster comes up fine.
Suspect this points to a problem in how Hive's install manager is executing the installer.
End of a failed Hive provision:
level=info msg="API v1.11.0+d4cacc0 up"                                                                                [0/3239]
level=debug msg="added kube-controller-manager.1566fa9c9ff3b86e: ip-10-0-10-83_b3efcb53-e801-11e8-8d14-0e50234f79ea became leader"
level=debug msg="added kube-scheduler.1566fa9cca7f44e9: ip-10-0-10-83_b3feb2d0-e801-11e8-98c8-0e50234f79ea became leader"
level=warning msg="RetryWatcher - getting event failed! Re-creating the watcher.  Run 'export KUBECONFIG=/output/auth/kubeconfig' to manage your cluster."
level=info msg="After exporting your kubeconfig, run 'oc -h' for a list of OpenShift client commands."
time="2018-11-14T11:58:23Z" level=info msg="uploading cluster metadata"
time="2018-11-14T11:58:23Z" level=info msg="uploaded cluster metadata configmap" configMapName=dgoodwin1-metadata
time="2018-11-14T11:58:23Z" level=info msg="uploading admin kubeconfig"
time="2018-11-14T11:58:23Z" level=info msg="uploaded admin kubeconfig secret" secretName=dgoodwin1-admin-kubeconfig

Running in Podman with:
sudo podman run -ti --rm -e AWS_ACCESS_KEY_ID=SNIP -e AWS_SECRET_ACCESS_KEY=SNIP -e OPENSHIFT_INSTALL_CLUSTER_NAME="dgoodwin2" -e OPENSHIFT_INSTALL_BASE_DOMAIN="SNIP" -e OPENSHIFT_INSTALL_EMAIL_ADDRESS="SNIP" -e OPENSHIFT_INSTALL_PASSWORD="password" -e  OPENSHIFT_INSTALL_SSH_PUB_KEY="SNIP" -e OPENSHIFT_INSTALL_PULL_SECRET="SNIP" -e OPENSHIFT_INSTALL_PLATFORM=aws -e OPENSHIFT_INSTALL_AWS_REGION=us-east-1 -v /home/dgoodwin/go/src/github.com/openshift/installer/output:/output:Z registry.svc.ci.openshift.org/openshift/origin-v4.0:installer create cluster --log-level=debug

The podman install comes up healthy with different output near the end:
WARNING Failed to connect events watcher: Get https://dgoodwin2-api.new-installer.openshift.com:6443/api/v1/namespaces/kube-system/events?resourceVersion=1940&watch=true: dial tcp <IP_ADDRESS>:6443: connect: co
nnection refused
DEBUG added openshift-master-controllers.1566fde611cb83a1: controller-manager-nnvhs became leader
DEBUG added kube-controller-manager.1566fde67693b5ec: ip-10-0-17-165_06eba434-e80a-11e8-9e6c-02b5eb23c7ce became leader
DEBUG added bootstrap-complete: cluster bootstrapping has completed
INFO Destroying the bootstrap resources...
DEBUG Stopping RetryWatcher.
INFO Using Terraform to destroy bootstrap resources...

Our code to execute can be seen here: https://github.com/openshift/hive/blob/master/contrib/pkg/installmanager/installmanager.go#L246
This was working previously but may have gone bad with installer changes around monitoring the logs from the bootstrap node.
Appears to be the root cause of https://github.com/openshift/cluster-network-operator/issues/35
CC @wking

Looks like:

The Failed to connect lines are truncated ;).
The installer listens to the bootstrap node fine, but it's failing to connect to any of the other masters (via several load-balancers) after the bootstrap bootkube takes itself down.

Maybe ingress rules for the load balancer?

@jhernand I think we've got a fix coming. I will test some clusters Monday morning and update the environment if everything is ok.


Update proto files for 0.8.0 release

Describe the proposal
Update proto files for 0.8.0 release

Reopening to track an additional proto change for bindings.


Finicky order dependence in --calendar arg, only accepted if supplied BEFORE subcommand name

As seen in #637, --calendar=Foo SUBCOMMAND works but SUBCOMMAND --calendar=Foo fails with parsing errors. Can/should that be more permissive about the order of args?

Had a look to understand what's happening here...
First off, the repro is brittle and kind of obscure. It's happening because gcalcli's parser relies on having unparsed args fed through to a second handle_unparsed pass but argparse for some inexplicable reason has logic for ambiguous arg cases that assumes that "if it contains a space, it was meant to be a positional" (discussed in python/cpython#66623).
It might be fixable if instead of relying on those arguments to remain "unparsed" for a 2nd pass, we explicitly pass a --calendar parser through to the root parser + every level that should support --calendar. Seems a little brutalist, but OTOH if we did that for everything we could clean up the whole handle_unparsed hack.


Docs: Update API documentation

The API documentation is outdated, especially the section about card creation. Needs an update

Hi, I'd love to have a look at the existing API documentation. I might be interested in working on this update.

I am interested in working on this

I am interested in working on this

Hi,
Can I have a look at the existing  API document? I may be able be able to help you out. The POST method for the API may need updates.
~Thanks


Allow ability to provide more than one mutually exclusive option

Allow having more than one mutually exclusive option. If the number of options is more than one make them radios if its only one then it will still be a checkbox

@rmccar  Could do with some more info here to understand the use cases, and user research etc.

Heres the details from the Runner card
Goal
To allow the respondent to provide more than one mutually exclusive option
Why do we want to do this?
Often there is a need for more than one mutually exclusive answer option, such as "Don't know" and "Prefer not to say. These are mutually exclusive answer options which appear under a checkbox answer type.
When the mutually exclusive checkbox is selected, the above checkboxes are unchecked as a result.
User story
As a respondent
I want to be able to be able to give an accurate answer as to why any of the previous checkbox answers do not apply.
so that I can provide accurate data.
Hypothesis
By adding another mutually exclusive answer option
we will be able to collect more specific information
because often there is a 'don't know' or 'prefer not to say' answer option.
Prototype
https://eq-v2-prototypes.herokuapp.com/utilities-social-survey/v1/about-your-household/benefits-received
Assumptions
1 mutually exclusive option would be a checkbox, while multiple mutually exclusive options would need to be radios
There would be one 'OR' in between first answers and mutually exclusive options
User research and testing
Related survey
BICS, all social surveys
Acceptance criteria (user perspective)
GIVEN a respondent is answering a survey
WHEN any answer type is used
AND more than one mutually exclusive answer option is available
THEN the respondent will select the relevant mutually exclusive radio option
AND the non mutually exclusive answer(s) will be unchecked
GIVEN a respondent is answering a survey
WHEN any answer type is used
AND more than one mutually exclusive answer option is available
THEN the respondent will select/enter the relevant answer option
AND any selected mutually exclusive radio will be unselected
Number of mutually exclusive options is configurable and there is no limit
Acceptance criteria (tech perspective)
** Technical details**:

DS component needs updating to support radio options for mutually exclusive options. (Implemented via a separate card)
Validator to support both Checkbox/Radio for the exclusive answer.
Make the appropriate changes to runner to support a single checkbox option or multiple radio options.

Ensure validation and summaries behave as expected.
Radio options should be cleared.
Ensure the submission payload is as expected.
New functional tests.

communicate schema config changes to the Author team

xxx
Out of scope
Input answer with a mutually exclusive answer to the input fields: https://eq-v2-prototypes.herokuapp.com/utilities-social-survey/v1/about-your-household/contact-you

Clearing mutually exclusive radio options when you do not have JS.

@rmccar Is there a UR report for this we can see?

@rmccar Is there a UR report for this we can see?

Ill send it to you


Remove the oom score in systemd unit. cri-containerd sets it by itself.

Remove OOMScore from systemd units. The default -999 oom score is set in https://github.com/kubernetes-incubator/cri-containerd/pull/347.
Signed-off-by: Lantao Liu<EMAIL_ADDRESS>

/test pull-cri-containerd-node-e2e

/lgtm

/test pull-cri-containerd-node-e2e

/test all

/test all


Remove  OrcFiltersAdapter

What changes were proposed in this pull request?
Call OrcFilters.createFilter directly and remove OrcFiltersAdapter.
How was this patch tested?
exists ut.

cc @zhixingheyi-tian ~


[full node] Merge Validator & Full Node Binaries

Since full nodes are a precursor to becoming validators, it makes sense to ship one binary which starts as a full node and can evolve as needed

(pasting my comments from the slack thread on this: https://mysten-labs.slack.com/archives/C03AW7J5Q5U/p1652906469866469)
There are already a bunch of existing differences between full node and validator:

full node has additional indexes (https://github.com/MystenLabs/sui/pull/2053) that a lot of people felt shouldn’t be part of the validator
full node is shortly going to use tx replaying instead of object syncing in its follower implementation - will validator also switch to tx replaying?
full node serves the read-only gateway api, and according to @Xun Li will also become the write side of the gateway.

this raises further questions about the role of gateway as a sparse-but-long-lived app-specific data store vs the role of a node as a complete-but-shorter-lived universal data store. @Sam Blackshear was asking about this in a different thread yesterday

managing these differences within a single process (e.g. turning off extended indexing when the node becomes a validator) is possible but it doesn’t seem like the easiest way to proceed to me (edited)

Hi Ade, can we use this diagram in the full node docs?
https://mysten-labs.slack.com/archives/C03AW7J5Q5U/p1652906469866469


new persistence api

Rewrite of persistence api and added ravendb persistence plugin. Maybe ravendb/couchebase should be moved outside ofta the core framework into its own repos?

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Daniel Söderberg seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

Im not 100% sure I got the versioning right. Byt maybe that is not set in the projects.

Can you describe the changes to start a discussion about it?
How is snapshotting handled now?

I have removed the "GetSnapshotInterval" method and property from Persistence. I also removed the bool property "Recovering". I think snapshots should be triggered by the actor itself instead when snapshots are needed.
I've introduced the following messages to be send to the actor on recovery/persisting:

RecoveryStarted
RecoveryCompleted
RecoverSnapshot *contains persisted snapshot object
RecoverEvent *contains persisted event object
PersistedEvent *contains the persisted event object

"IProviderState" has been expanded with methods to handle "DeleteEventsAsync" and "DeleteSnapshotsAsync" so the actor could clean up stored events/snapshots that are not needed anymore.

Letting the actor control snapshotting makes sense IMO. It is more related to the needs of the actor than the needs of the persistence.
About versioning, don't worry. It's currently set by the build script and we only bump it after merging to master.

Anyone time ro review my changes?

The change looks good to me.
cc @cpx86

Looks good! I like the introduction of the recovery messages, so the actor can handle those differently if necessary :)

Just fix the merge conflict and I'll merge 👍

Merged, thanks!


Calling MLXRandom.seed causes crash in 0.18.0

I haven't been able to reproduce this in LLMEval from mlx-swift-examples for some reason, but when I run the following line in my app using mlx-swift 0.18.0:
MLXRandom.seed(UInt64(Date.timeIntervalSinceReferenceDate * 1000))

I get this crash on macOS:

And on iOS:

I can't reproduce with a unit test:
    func testSeed() {
        MLXRandom.seed(UInt64(Date.timeIntervalSinceReferenceDate * 1000))
    }

so perhaps it is something unique to the runtime environment.  Could this be happening:

when the process is exiting?

it looks like something in the C++ runtime is being torn down

concurrently?

this looks like locking code so I wouldn't expect a problem

Can you give more of the backtrace?

On my M3 MacBook Pro with 16 GB of RAM, the crash doesn't happen with Qwen 2.5 1.5B or Gemma 2 2B, but it does with Gemma 2 9B, Llama 3.1 8B and Mistral 7B.
Here is another screenshot with more of the backtrace.

OK, so that looks like a call to set the seed, it takes the lock in RandomState (new in 0.18.0) and then sets state which frees the old value.
I presume you are setting the random state at the beginning of the inference so this is not a shutdown situation?

Yes, I set the seed and then call this:
let result = await modelContainer.perform { model, tokenizer in
        MLXLLM.generate(...

You have a "chatMessages" parameter -- does this happen on the first prompt evaluation or after a few back and forth messages?

The crash happens when calling MLXRandom.seed. The program exits at that point, so no prompt evaluation is happening. When I comment out the line with MLXRandom.seed, it works as expected.

To clarify: I was trying to ask if the program was trying to shut down (e.g. cmd-q) at this time -- I have seen crashes like this where the C++ machinery was being torn down while an evaluation was going on at shutdown time.

I'm not sure. How can I find out if that's the case?

You have a "chatMessages" parameter -- does this happen on the first prompt evaluation or after a few back and forth messages?

It happens on the first evaluation with one of the larger models I mentioned above. I can do one evaluation on a smaller one and it works, and then if I switch to a larger one it crashes. These are models that should fit in memory and work on the previous version of mlx-swift.

If you weren't trying to exit the application when the crash happened, it probably isn't that.  Generally you would have to look at all the threads (the crash log in ~/Library/Logs/DiagnosticReports would contain all of them) and infer what the process is doing.

It happens on the first evaluation with one of the larger models I mentioned above. I can do one evaluation on a smaller one and it works, and then if I switch to a larger one it crashes. These are models that should fit in memory and work on the previous version of mlx-swift.

OK, I am going to try and reproduce with LLMEval in the examples and ModelConfiguration.llama3_1_8B_4bit

This is happening while the app is still running and I'm trying to generate text.

I can't reproduce it in LLMEval with Llama 3.1 8B.  Is it possible for you to send me a crashlog?  It will be named something like Local Chat-2024-DATE+TIME.ips
If you don't want to attach it here but are ok with sending it directly, you can send it to<EMAIL_ADDRESS>

Unfortunately I can't find any logs corresponding to this crash in ~/Library/Logs/... (I think they won't be stored there when the app is run from Xcode) or in ~/Library/Developer/Xcode/DerivedData or in the place mentioned in this article: https://developer.apple.com/news/?id=nra79npr

Yeah, that is right -- if it dies in the debugger it won't produce one.  I think you can:

launch it without a debugger attached (turn off debug executable in the run configuration):

You can probably do this by just launching from Finder as well (if you can find your executable!)

when it crashes you can detach from it

Got it, thanks! I'll send this to you by email.

The other threads are all idle (so nothing obviously threading related).
This is off the main thread, so it won't have a deep stack.  The crash log indicates a depth of 510, but that may be an artificial cutoff.  The screenshot above shows a stack depth of ~10000.  There was a change a while back:

https://github.com/ml-explore/mlx/pull/982

to address this issue specifically, but this is specifically recursion in this area:
468 Local Chat.debug.dylib        	       0x105fa0680 mlx::core::array::~array() + 676 (array.cpp:205)
469 Local Chat.debug.dylib        	       0x105f9fcb0 mlx::core::array::~array() + 28 (array.cpp:173)
470 Local Chat.debug.dylib        	       0x105f1bf14 std::__1::allocator<mlx::core::array>::destroy[abi:de180100](mlx::core::array*) + 28 (allocator.h:176)
471 Local Chat.debug.dylib        	       0x105f1bed8 void std::__1::allocator_traits<std::__1::allocator<mlx::core::array>>::destroy[abi:de180100]<mlx::core::array, void>(std::__1::allocator<mlx::core::array>&, mlx::core::array*) + 32 (allocator_traits.h:311)
472 Local Chat.debug.dylib        	       0x105f1be94 std::__1::vector<mlx::core::array, std::__1::allocator<mlx::core::array>>::__base_destruct_at_end[abi:de180100](mlx::core::array*) + 104 (vector:926)
473 Local Chat.debug.dylib        	       0x105f1bd9c std::__1::vector<mlx::core::array, std::__1::allocator<mlx::core::array>>::__clear[abi:de180100]() + 28 (vector:920)
474 Local Chat.debug.dylib        	       0x105f1bd0c std::__1::vector<mlx::core::array, std::__1::allocator<mlx::core::array>>::__destroy_vector::operator()[abi:de180100]() + 60 (vector:490)
475 Local Chat.debug.dylib        	       0x105f1bc84 std::__1::vector<mlx::core::array, std::__1::allocator<mlx::core::array>>::~vector[abi:de180100]() + 44 (vector:501)
476 Local Chat.debug.dylib        	       0x105f171f8 std::__1::vector<mlx::core::array, std::__1::allocator<mlx::core::array>>::~vector[abi:de180100]() + 28 (vector:501)
477 Local Chat.debug.dylib        	       0x105fa0fd4 mlx::core::array::ArrayDesc::~ArrayDesc() + 404 (array.cpp:265)
478 Local Chat.debug.dylib        	       0x105fa11d8 mlx::core::array::ArrayDesc::~ArrayDesc() + 28 (array.cpp:237)
479 Local Chat.debug.dylib        	       0x105eefa30 std::__1::allocator<mlx::core::array::ArrayDesc>::destroy[abi:de180100](mlx::core::array::ArrayDesc*) + 28 (allocator.h:176)
480 Local Chat.debug.dylib        	       0x105eefa08 void std::__1::allocator_traits<std::__1::allocator<mlx::core::array::ArrayDesc>>::destroy[abi:de180100]<mlx::core::array::ArrayDesc, void>(std::__1::allocator<mlx::core::array::ArrayDesc>&, mlx::core::array::ArrayDesc*) + 32 (allocator_traits.h:311)
481 Local Chat.debug.dylib        	       0x105eef9d4 void std::__1::__shared_ptr_emplace<mlx::core::array::ArrayDesc, std::__1::allocator<mlx::core::array::ArrayDesc>>::__on_zero_shared_impl[abi:de180100]<std::__1::allocator<mlx::core::array::ArrayDesc>, 0>() + 48 (shared_ptr.h:284)
482 Local Chat.debug.dylib        	       0x105eef6c8 std::__1::__shared_ptr_emplace<mlx::core::array::ArrayDesc, std::__1::allocator<mlx::core::array::ArrayDesc>>::__on_zero_shared() + 24 (shared_ptr.h:287)
483 Local Chat.debug.dylib        	       0x105ef4958 std::__1::__shared_count::__release_shared[abi:de180100]() + 64 (shared_ptr.h:157)
484 Local Chat.debug.dylib        	       0x105ef48f8 std::__1::__shared_weak_count::__release_shared[abi:de180100]() + 28 (shared_ptr.h:186)
485 Local Chat.debug.dylib        	       0x105ef48c8 std::__1::shared_ptr<mlx::core::array::ArrayDesc>::~shared_ptr[abi:de180100]() + 64 (shared_ptr.h:648)
486 Local Chat.debug.dylib        	       0x105eef07c std::__1::shared_ptr<mlx::core::array::ArrayDesc>::~shared_ptr[abi:de180100]() + 28 (shared_ptr.h:646)
487 Local Chat.debug.dylib        	       0x105fa0680 mlx::core::array::~array() + 676 (array.cpp:205)
488 Local Chat.debug.dylib        	       0x105f9fcb0 mlx::core::array::~array() + 28 (array.cpp:173)

The triggering condition seems to be a long chain of calls to random that get released and deallocated -- we saw something like that when working on the Stable Diffusion example.
@angeloskath or @awni any thoughts?

Could this:
477 Local Chat.debug.dylib        	       0x105fa0fd4 mlx::core::array::ArrayDesc::~ArrayDesc() + 404 (array.cpp:265)
478 Local Chat.debug.dylib        	       0x105fa11d8 mlx::core::array::ArrayDesc::~ArrayDesc() + 28 (array.cpp:237)

be a sign that ArrayDesc is recursing on one of its member variables?  We have:
  for (array& a : inputs) {
    if (a.array_desc_.use_count() == 1) {
      for_deletion.push_back(std::move(a.array_desc_));
    }
  }

but it doesn't have one for siblings (which isn't new, but perhaps we are triggering that case)?

A recent change in Random does now call:
extern "C" mlx_tuple_array_array mlx_random_split(mlx_array key, mlx_stream s) {
  RETURN_MLX_C_ARRAYPAIR(mlx::core::random::split(key->ctx, s->ctx));
}

where previously it called:
mlx_random_split_equal_parts(mlx_array key, int num, mlx_stream s) {
  RETURN_MLX_C_ARRAY(mlx::core::random::split(key->ctx, num, s->ctx));
}

The implementations of those are very different:
std::pair<array, array> split(const array& key, StreamOrDevice s /* = {} */) {
  auto stream = to_stream(s);
  auto out = mlx::core::split(random::split(key, 2, stream), 2, stream);
  return {reshape(out[0], {2}, stream), reshape(out[1], {2}, stream)};
}

array split(const array& key, int num, StreamOrDevice s /* = {} */) {
  return bits({num, 2}, 4, key, s);
}

@DePasqualeOrg you can try this branch as a workaround: https://github.com/davidkoski/mlx-swift/tree/workaround-139
We will continue to investigate, but I think I know the triggering condition at least.

This should be fixed on MLX main: https://github.com/ml-explore/mlx-swift/issues/139


Galaxy does not handle correctly branch renaming

Check https://github.com/ansible/galaxy/issues/2535

Looks like it worked.

I can see it works, but it's getting only version without v preffix on galaxy. I guess it should work anyways, but in same cases may break some dependencies :thinking:
https://galaxy.ansible.com/aalaesar/install_nextcloud

I can see it works, but it's getting only version without v preffix on galaxy. I guess it should work anyways, but in same cases may break some dependencies, so it's just good to have it in mind thinking https://galaxy.ansible.com/aalaesar/install_nextcloud
BTW I'd prefer as it is right now without v - just version (most of the roles are versioned this way) ;)

I also noticed that, and was going to comment that vast majority of roles in Ansible Galaxy does not a v. I consider it an unintentional bugfix XD


setuptools: ModuleNotFoundError: No module named 'six'

Building setuptools in the standard way I've used forever fails:
# /usr/pkg/bin/python3.6  setup.py  build
Traceback (most recent call last):
  File "setup.py", line 11, in <module>
    import setuptools
  File ".../setuptools-34.1.1/setuptools/__init__.py", line 10, in <module>
    from six.moves import filter, map
ModuleNotFoundError: No module named 'six'

It looks like setuptools now added a six dependency, which breaks distributions that build from scratch, because six uses setuptools for installation (dependency cycle).

That's right. This change was made intentionally as part of the Setuptools 34 release - dropping support for self install and upgrade. We've removed the hack that vendors setuptools dependencies. You'll want to install the dependencies with pip first. See bootstrap.py for one technique for installing the dependencies.

I'm sorry if I'm slow to understand, but I have more questions. I don't see bootstrap.py documented, and reading it, it imports pip. pip also needs setuptools to install.
What is the recommended packaging procedure if you only have python installed to get to packages for pip and setuptools?

https://python-packaging-user-guide.readthedocs.io/installing/#install-pip-setuptools-and-wheel

For anyone else stumbling over this issue, my solution was:

install the setuptools version that is coming with python, using python -m ensurepip --user
build and install appdirs, six, pyparsing, and packaging with this version
build setuptools
delete the --user installation of the old version

I got the same problem after I upgrade python2.7.10 to python2.7.13.
None of the above works for me.
Finally, I fixed this by download six package directly from pypi,
then go to the download directory,
and manually installed it by using python set-up.py install.

I wish that dependency would be lifted when Python 3 is being used.

That dependency will be lifted when Python 2.7 support is dropped. Until then, six is the library that enables a unified codebase for all versions.


sprintf format

Hi Daniel,
I must say firstly awesome work you have done, great examples.
I am trying to get the full digits of a clock to display, initial values are int's using sprintf to format so that leading zero is not truncated and displayed.  for example 07:01  and not 7 : 1
Any ideas on where I may be going wrong?
char bufHR[3]  = "";
sprintf(bufHR, "%2d", RTC_Hour); // format to 2 decimal places for display output
display.drawString(15, 20, bufHR);
Display works but still only displays single digit and not leading zero.
Thanks
Dans
PS: I will try different formats such as: "%02d"  any other suggestions please let me know.

@danbicks
%02d is the trick
sprintf(bufHR, "%02d", RTC_Hour); // format to 2 decimal places for display output

Thanks Buddy,
Already added Woo looks so cool.
Dans

Cool. Now I learned something as well. I thought sprintf is not available for Arduino environment. Can we close this issue then?

Hi Squix,
Yep all closed, PS have you managed to get compass display code working?
Sneak download pretty please :)
Cheers buddy
Dans


tigervnc: fix compatibility with xorgserver 1.20.7

(cherry picked from commit f216b03d5b01490aadc54e5e9d5e3cb607816262)
Motivation for this change
fix tigervnc build with nixos-20.03
Things done

[X] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[X] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[X] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

ZHF: #80379

Funny, I just made #80633 10 minutes ago.

Duplicate of #80551

Why did I not find #80551 ?


Add a tool to clean up non-live instances

The tool will simply remove all instances that are not in LIVEINSTANCES from INSTANCES and CONFIGS/PARTICIPANT.
Use with caution!

Codecov Report

Merging #3061 into master will decrease coverage by 0.07%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #3061      +/-   ##
============================================
- Coverage     70.44%   70.36%   -0.08%
  Complexity        4        4
============================================
  Files           934      934
  Lines         43498    43503       +5
  Branches       5907     5907
============================================
- Hits          30642    30612      -30
- Misses        10803    10836      +33
- Partials       2053     2055       +2

Impacted Files
Coverage Δ
Complexity Δ

...core/operator/dociditerators/AndDocIdIterator.java
51.72% <0%> (-31.04%)
0% <0%> (ø)

...e/impl/dictionary/LongOnHeapMutableDictionary.java
82.22% <0%> (-13.34%)
0% <0%> (ø)

...lix/EmptyBrokerOnlineOfflineStateModelFactory.java
86.66% <0%> (-13.34%)
0% <0%> (ø)

...impl/dictionary/DoubleOnHeapMutableDictionary.java
68.88% <0%> (-13.34%)
0% <0%> (ø)

...ore/query/scheduler/resources/ResourceManager.java
87.09% <0%> (-9.68%)
0% <0%> (ø)

...kedin/pinot/core/util/SortedRangeIntersection.java
83.82% <0%> (-7.36%)
0% <0%> (ø)

.../impl/dictionary/FloatOnHeapMutableDictionary.java
86.66% <0%> (-6.67%)
0% <0%> (ø)

...mpl/dictionary/DoubleOffHeapMutableDictionary.java
78.18% <0%> (-5.46%)
0% <0%> (ø)

...nsport/netty/PooledNettyClientResourceManager.java
85.41% <0%> (-4.17%)
0% <0%> (ø)

...om/linkedin/pinot/transport/netty/NettyServer.java
80% <0%> (-4%)
0% <0%> (ø)

... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 65c9779...89de87c. Read the comment docs.

@kishoreg Yes, we should call Helix.deleteInstance() ideally. The reason I added this small tool is because when we replace hardware we don't call that method, and ZK INSTANCES dir will never be cleaned up. Before running this tool, we need to make sure all instances in IdealStates are live. I agree that this tool is a little risky to use. Will close the PR and use it just for a record.


Fix args handling

[x] pass unicode (wide strings) on windows
[x] add wx::App::args() fn to retrieve args  after wx handled
[x] some testing

Checked WxArgs should work with existing command line parsing crates:

clap
getopts

They have parse method can be used with String (requires Into<OsString> or AsRef<OsStr> that String implement)