HADOOP-16085: use object version or etags to protect against inconsistent read after replace/overwrite

This is a new PR to replace #675 .  The only difference vs what is on #675 is a merge of the latest code from trunk, resolution of minor merge conflicts when doing so, and squash down to a single commit.
#675 is being closed as-is to preserve the discussion there.
I have performed a full test run on this (against us-west-2 with a bucket with versioning enabled) which passed without any failing tests:
mvn -T 1C verify -Dparallel-tests -DtestsThreadCount=8 -Ds3guard -Ddynamo

Tests run: 860, Failures: 0, Errors: 0, Skipped: 145

I've pushed a commit that adds retries as discussed in https://github.com/apache/hadoop/pull/675#issuecomment-488614814
The retries happen in S3AInputStream if the version doesn't match on initial open.  There are no retries if the version doesn't match on re-open (during seek() backwards).
Retries also happen for rename() and select().
Testing was added in ITestS3ARemoteFileChanged.  I used Mockito.spy() on the s3 client to stub in inconsistent responses until a threshold of retries is met.
I've run the full test suite (against a bucket with versioning enabled in us-west-2):
mvn -T 1C verify -Dparallel-tests -DtestsThreadCount=8 -Ds3guard -Ddynamo

[ERROR] Tests run: 896, Failures: 0, Errors: 2, Skipped: 145

The two errors were in ITestDirectoryCommitMRJob and  ITestS3GuardConcurrentOps, which succeeded when run individually:
mvn -T 1C verify -Dtest=skip -Dit.test=ITestDirectoryCommitMRJob -Ds3guard -Ddynamo
mvn -T 1C verify -Dtest=skip -Dit.test=ITestS3GuardConcurrentOps -Ds3guard -Ddynamo

https://github.com/apache/hadoop/pull/675#issuecomment-488614814 suggests possibly different retry settings for these scenarios.  I haven't done that, at least yet.  Perhaps that can be carved off as another issue.  Similarly, I haven't implemented the HADOOP-13293 proposal.  I'm open to those things but would like to get the rest of this settled (merged) first if possible.

thanks, I'm checking this out and going to test/review it locally. with the goal of getting it in this week. I might do some changes to the PR locally and push them up as a branch for you to cherry-pick in, as that is potentially easier than me just adding a large set of bits of homework for you to do. Would that be OK? it should save time all round

Great, thanks Steve!

I might do some changes to the PR locally and push them up as a branch for you to cherry-pick in, as that is potentially easier than me just adding a large set of bits of homework for you to do. Would that be OK? it should save time all round

Sure, that sounds good to me.

Right, I've done my edits and will put it up as a PR alongside that: if you cherry pick my patch in here, then I'll close/delete that one and this will have everything in.
That patch is me just going through my review comments and doing them.
I am seeing failures with testRenameEventuallyConsistentFile on some options, despite my efforts to understand it.  Can you test and make sure it is still good for you?

See #803

Thanks Steve!  I'll have a look over it and see what's up with testRenameEventuallyConsistentFile.

I looked over your changes and they all made sense to me.  Thanks for cleaning up my mistakes :) . I've fast-forwarded this PR branch to include your commit.
The change to add the annotation that labeled the parameters on the parameterized tests is especially nice.  Embarrassingly I hadn't learned about that one yet.
After pulling your changes in I re-ran testRenameEventuallyConsistentFile again and all permutations succeeded against a bucket with object versioning enabled.  I did see a reproducible failure on (etag, client) against a bucket with versioning disabled.  I'll dig into that further.  Which permutations are failing for you?

The latest commit fixes the test failures I was seeing against a bucket with versioning disabled.  In the (etag,client) case for testRenameEventuallyConsistentFile, stubbing of inconsistent responses from AmazonS3.copyObject() was incorrect.  For that case we should never see a "precondition failed" response since we don't pass any eTag or versionId qualification on the request.
There were a few other failures in test methods that require versioning since I hadn't copied the code that executes the assumption to make sure versioning is there.
I ran the full ITestS3ARemoteFileChanged once each against a bucket with versioning enabled and a bucket with versioning disabled and all tests either succeeded or were skipped as expected.
I will run the full test suite again against a bucket with versioning disabled just in case there are somehow other failures to uncover there since previously I only ran the suite against a bucket with versioning enabled.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
57
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 25 new or modified test files.

_ trunk Compile Tests _

0
mvndep
78
Maven dependency ordering for branch

+1
mvninstall
1254
trunk passed

+1
compile
1276
trunk passed

+1
checkstyle
162
trunk passed

+1
mvnsite
135
trunk passed

+1
shadedclient
1043
branch has no errors when building and testing our client artifacts.

+1
javadoc
109
trunk passed

0
spotbugs
67
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
193
trunk passed

_ Patch Compile Tests _

0
mvndep
24
Maven dependency ordering for patch

+1
mvninstall
79
the patch passed

+1
compile
1104
the patch passed

+1
javac
1104
the patch passed

-0
checkstyle
146
root: The patch generated 29 new + 70 unchanged - 4 fixed = 99 total (was 74)

+1
mvnsite
123
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
3
The patch has no ill-formed XML file.

+1
shadedclient
671
patch has no errors when building and testing our client artifacts.

+1
javadoc
89
the patch passed

+1
findbugs
207
the patch passed

_ Other Tests _

+1
unit
562
hadoop-common in the patch passed.

+1
unit
270
hadoop-aws in the patch passed.

+1
asflicense
44
The patch does not generate ASF License warnings.

7587

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-794/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/794

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle

uname
Linux cad520a187b1 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 3418bbb

Default Java
1.8.0_191

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/3/artifact/out/diff-checkstyle-root.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/3/testReport/

Max. process+thread count
1387 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/3/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
30
Docker mode activated.

_ Prechecks _

+1
dupname
2
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 25 new or modified test files.

_ trunk Compile Tests _

0
mvndep
66
Maven dependency ordering for branch

+1
mvninstall
1039
trunk passed

+1
compile
1020
trunk passed

+1
checkstyle
141
trunk passed

+1
mvnsite
132
trunk passed

+1
shadedclient
1004
branch has no errors when building and testing our client artifacts.

+1
javadoc
106
trunk passed

0
spotbugs
67
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
185
trunk passed

_ Patch Compile Tests _

0
mvndep
23
Maven dependency ordering for patch

+1
mvninstall
76
the patch passed

+1
compile
958
the patch passed

+1
javac
958
the patch passed

-0
checkstyle
144
root: The patch generated 29 new + 70 unchanged - 4 fixed = 99 total (was 74)

+1
mvnsite
125
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
3
The patch has no ill-formed XML file.

+1
shadedclient
685
patch has no errors when building and testing our client artifacts.

+1
javadoc
105
the patch passed

+1
findbugs
203
the patch passed

_ Other Tests _

+1
unit
528
hadoop-common in the patch passed.

+1
unit
285
hadoop-aws in the patch passed.

+1
asflicense
53
The patch does not generate ASF License warnings.

6939

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-794/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/794

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle

uname
Linux de5a731ca399 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 0c5fa2e

Default Java
1.8.0_191

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/4/artifact/out/diff-checkstyle-root.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/4/testReport/

Max. process+thread count
1448 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/4/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

I will run the full test suite again against a bucket with versioning disabled just in case there are somehow other failures to uncover there since previously I only ran the suite against a bucket with versioning enabled.

I completed the test run (us-west-2, bucket with versioning disabled):
mvn -T 1C verify -Dparallel-tests -DtestsThreadCount=8 -Ds3guard -Ddynamo

[ERROR] Tests run: 896, Failures: 1, Errors: 3, Skipped: 189

The 3 errors and 1 failure were spread across ITestS3AContractGetFileStatusV1List, ITestDirectoryCommitMRJob, and ITestS3GuardToolDynamoDB, which succeeded when ran individually.
mvn -T 1C verify -Dtest=skip -Dit.test=ITestS3AContractGetFileStatusV1List -Ds3guard -Ddynamo
mvn -T 1C verify -Dtest=skip -Dit.test=ITestDirectoryCommitMRJob -Ds3guard -Ddynamo
mvn -T 1C verify -Dtest=skip -Dit.test=ITestS3GuardToolDynamoDB -Ds3guard -Ddynamo

The change to add the annotation that labeled the parameters on the parameterized tests is especially nice. Embarrassingly I hadn't learned about that one yet.

I only learned about it last week. Note, for any code which creates FS paths via calls to getMethodName, you need to have strings which can form valid paths (no spaces etc)

One thing which needs changing: the error message. Currently it says "on position -X" but now it should describe what operation failed
org.apache.hadoop.fs.s3a.RemoteFileChangedException: copy `s3a://hwdev-steve-ireland-new/test/testRenameEventuallyConsistentDirectory[bogus-bogus]/sourcedir/inconsistent': ETag change detected while reading at position 0. Expected 9d4c55e9fcea10126a50efe60111e522 got b81ae3a09fdd252ca7b30bddfa3cf2bd

One thing which needs changing: the error message. Currently it says "on position -X" but now it should describe what operation failed

Ah, yes, the position mention is kind of misleading.  It does mention the operation, but that is easy to overlook:
org.apache.hadoop.fs.s3a.RemoteFileChangedException: copy ...

new exception will include the operation and only include a position if >= 0:

org.apache.hadoop.fs.s3a.RemoteFileChangedException: copy `s3a://hwdev-steve-ireland-new/test/testRenameEventuallyConsistentDirectory[bogus-bogus]/sourcedir/inconsistent': ETag change detected on copy. Expected 9d4c55e9fcea10126a50efe60111e522 got b81ae3a09fdd252ca7b30bddfa3cf2bd

new exception will include the operation and only include a position if >= 0:

Sounds good to me.  Are you crafting a commit for that?  If not I can do it here in a bit.

here: #807
-move CopyOutcome into fs.s3a.impl package
-add new (failing) test for directory renaming where one of the files underneath is eventually consistent.
-pull out commonanity from test methods (spyOnFilesystem(), expectReadFailure())
-intercept() calls provide details on failure, primarily just by returning the result of the operation expected to fail.
-ChangeDetectionPolicy-raised exceptions on copy failures don't include position any more
-cache/restore metastore so even on test failures mockito is involved on teardown
Tests: yes, some fail, testRenameEventuallyConsistent* depending on the options. I think its really down to "how many failures are being generated"...but I don't want to blindly change them until they work, not without knowing why

Perfect.  I had a quick review of your changes and as before they look good.  The additional logging in the tests should be especially helpful.
I'll give the tests a run to presumably reproduce the failures and then help explain and fix them.  I have to step away from it for a little bit here but I should be able to come back to it a little later today.

That'd be great. If you turn @debug logging on for the tests you can get a better view of what's happening in terms of when an operation is being called.
Now some other issues

if the versionId is known, should it be included in the getObjectMetadata() Call? Right now it's being checked on the client, but GetObjectMetadataRequest does take a version ID which could be passed in

the copy operation has moved from once() to retry(), something I've updated in the @Retry tag.

But:

getObjectMetdata() Already does retry: it shouldn't be wrapped in case of extended connection failures amplifying delays
I Believe the transfer manager is meant to do its own recovery. (but I've also known it to fail once with a 200 + Error) state

I'm worried that the addition of retry resilience may be putting a double wrapper around existing failure conditions and their retries.

Given that getObjectMetadata() only needs the new retry to handle its eventualness, if we passed the ChangeTracker into that method, the existing retry-invoked operation could both set the version ID and check on the client side for problems. It would then be pulled out into its own once() call just to translate exceptions.

And for the transfer, we could have it retrying with a (new) retry policy which only retries on 412 responses, so any other failure doesn't trigger a retry.

BTW, note that the CopyOutcome class only stores SdkBaseExceptions raised in the copy, as does the method to await the copy to complete. I've not put that through all the way through the other bits of the patch, but think it makes sense as other failures shouldn't need to be considered by the change tracker.

Good catches.  This retry stuff is a bit tricky.  I obviously overlooked the fact that getObjectMetadata() was retrying.  That also needs to be considered in S3AFileSystem.select() where I added a layer of retries to wait for consistency, but it is wrapping getObjectMetadata() as well.
A getObjectMetadata() that takes the ChangeTracker as you suggest seems like it could apply in both copyFile() and select().  It seems like a good approach to try.  I assume maybe you are already taking a shot at that?  If not, I can give it a whirl.

BTW, note that the CopyOutcome class only stores SdkBaseExceptions raised in the copy, as does the method to await the copy to complete. I've not put that through all the way through the other bits of the patch, but think it makes sense as other failures shouldn't need to be considered by the change tracker.

Yep, agree.

Some weird stuff is going on in testRenameEventuallyConsistentDirectory contributing to the test failure (investigating etag-server mode).
First, listFilesAndEmptyDirectories() returns S3ALocatedFileStatus that doesn't have the etag.  When combined with the stubbing to return the old eTag, this causes ChangeTracker to get seeded with the old eTag and as such the first several retry passes are going past changeTracker.processMetadata() because the old eTag actually looks to ChangeTracker like it is correct.
Since the code gets past that, it gets to transfers.copy().  That fails with NullPointerException.  That seems like a bug in the AWS SDK.  The NPE happens because com.amazonaws.services.s3.transfer.internal.CopyCallable.copyInOneChunk() calls s3.copyObject(copyObjectRequest), which returns null when the ETag constraint isn't met.  The method goes on assuming that would have never returned null, resulting in the NPE.
The NPE causes a trip back through the retry loop.  The same thing happens again until the inconsistent metadata stub count is exceeded and then any retry after that causes the ChangeTracker to trigger RemoteFileChangedException when the correct eTag passes through since it thinks the old eTag is actually the right one.  The test ultimately fails with that RemoteFileChangedException when the overall retry limit is met.
I need to look closer at why listFilesAndEmptyDirectories() doesn't return the status containing the etag.

Great investigation!

I was thinking it was time to update the AWS SDK anyway; lets try that and if the AWS bug still exists, file it
I thought I changed things to only catch AWS exceptions; I guess if they are wrapping the NPE it'll come through.

If you look at why the listing isn't returning the etag, I'll do the SDK update.
regarding the change tracker on GetObjectMetadata -I've not started that, so go ahead

If you look at why the listing isn't returning the etag, I'll do the SDK update.
regarding the change tracker on GetObjectMetadata -I've not started that, so go ahead

Will do.  I'll be investigating the listing issue first.

I was thinking it was time to update the AWS SDK anyway; lets try that and if the AWS bug still exists, file it

Yeah, looks to me like it is still an issue.  It happens here.  I thought to come back to logging it.  If you want to get that done that would be great!

I guess if they are wrapping the NPE it'll come through.

Yep, it gets wrapped.

I'll be investigating the listing issue first.

The listing doesn't return the eTag because innerListFiles() calls listing.createFileStatusListingIterator() which reconciles the native S3 listing against the S3Guard listing and the implementation of that (in Listing.FileStatusListingIterator.next()) prefers the native S3 listing S3AFileStatus over the S3Guard status.  This used to make sense since when a file existed in both S3 and S3Guard, S3Guard probably wouldn't have had any better information about the file.
With these changes, we should be preferring the status from S3Guard just as we do on file open.

I've pulled in your patch from #807 and added https://github.com/apache/hadoop/pull/794/commits/c924d740b56467a3f8047d3f768219d5f24971ce to fix the test failures.  All the ITestS3ARemoteFileChanged permutations are succeeding for me (on both buckets with and without versioning enabled).
I still need to complete the refactoring to use ChangeTracker in getObjectMetadata() to get rid of the layered retries problem.

https://github.com/apache/hadoop/pull/794/commits/348aa61a9b7bcafe067c3bdf374b07d08e06867d has the refactoring to use ChangeTracker inside getObjectMetadata() to get rid of the layered retries.  All the ITestS3ARemoteFileChanged tests are passing for me on that.

if the versionId is known, should it be included in the getObjectMetadata() Call?

I haven't done that yet.  Seems like it is probably a good idea.  I'll try that now.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
30
Docker mode activated.

_ Prechecks _

+1
dupname
2
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 25 new or modified test files.

_ trunk Compile Tests _

0
mvndep
62
Maven dependency ordering for branch

+1
mvninstall
1036
trunk passed

+1
compile
1008
trunk passed

+1
checkstyle
144
trunk passed

+1
mvnsite
131
trunk passed

+1
shadedclient
1026
branch has no errors when building and testing our client artifacts.

+1
javadoc
103
trunk passed

0
spotbugs
60
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
179
trunk passed

_ Patch Compile Tests _

0
mvndep
21
Maven dependency ordering for patch

+1
mvninstall
75
the patch passed

+1
compile
992
the patch passed

+1
javac
992
the patch passed

-0
checkstyle
130
root: The patch generated 32 new + 70 unchanged - 4 fixed = 102 total (was 74)

+1
mvnsite
115
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
3
The patch has no ill-formed XML file.

+1
shadedclient
607
patch has no errors when building and testing our client artifacts.

+1
javadoc
86
the patch passed

+1
findbugs
181
the patch passed

_ Other Tests _

+1
unit
503
hadoop-common in the patch passed.

+1
unit
277
hadoop-aws in the patch passed.

+1
asflicense
37
The patch does not generate ASF License warnings.

7685

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-794/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/794

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle

uname
Linux bd4ca8898ff0 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 36267b6

Default Java
1.8.0_191

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/5/artifact/out/diff-checkstyle-root.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/5/testReport/

Max. process+thread count
1508 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/5/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

So, now I've added the change to add versionId on the getObjectMetadata() calls:
https://github.com/apache/hadoop/pull/794/commits/16a428926fc6b62bfdaae81bb654e582d8ff4169
Following that, I looked into the NPE in copyObject() a bit further.  There is already an issue logged:
https://github.com/aws/aws-sdk-java/issues/1644
The issue seems to suggest that a 412 response would never actually happen, which implies our simulation of this type of failure is invalid.  I need to digest it a bit further.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
29
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 25 new or modified test files.

_ trunk Compile Tests _

0
mvndep
21
Maven dependency ordering for branch

+1
mvninstall
1022
trunk passed

+1
compile
1009
trunk passed

+1
checkstyle
141
trunk passed

+1
mvnsite
127
trunk passed

+1
shadedclient
993
branch has no errors when building and testing our client artifacts.

+1
javadoc
107
trunk passed

0
spotbugs
69
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
184
trunk passed

_ Patch Compile Tests _

0
mvndep
23
Maven dependency ordering for patch

+1
mvninstall
76
the patch passed

+1
compile
987
the patch passed

+1
javac
987
the patch passed

-0
checkstyle
145
root: The patch generated 38 new + 70 unchanged - 4 fixed = 108 total (was 74)

+1
mvnsite
126
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
2
The patch has no ill-formed XML file.

+1
shadedclient
692
patch has no errors when building and testing our client artifacts.

+1
javadoc
107
the patch passed

+1
findbugs
203
the patch passed

_ Other Tests _

+1
unit
514
hadoop-common in the patch passed.

+1
unit
288
hadoop-aws in the patch passed.

+1
asflicense
51
The patch does not generate ASF License warnings.

6882

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-794/7/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/794

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle

uname
Linux 071484dc2602 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 7c2d7c0

Default Java
1.8.0_191

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/7/artifact/out/diff-checkstyle-root.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/7/testReport/

Max. process+thread count
1390 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/7/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

The issue seems to suggest that a 412 response would never actually happen, which implies our simulation of this type of failure is invalid. I need to digest it a bit further.

After discovering this I was wondering how I never realized it sooner.  For example, why did ITestS3ARemoteFileChanged.testRenameChangedFile() not surface it?
The answer lies in the fact that S3AFileSystem.copyFile() has ChangeTracker test the ObjectMetadata before it calls transfers.copy().  Because of this, the tests generate RemoteFileChangedException there instead of getting to the NPE induced AWS exception.
As such, to see the NPE AWS exception, the caller would have to hit a scenario where getObjectMetadata() is showing the correct object version but then transfers.copy() does not.  This seems like it should be a pretty rare scenario, but that's not to say it couldn't happen.
I'm going to go ahead and change the stubbing in ITestS3ARemoteFileChanged where I had copyObject throwing the 412 exception to return null instead like the real impl would.  I'll make sure there is a test that expects the behavior that comes along with that (which won't be RemoteFileChangedException since I don't think we want to write code that assumes an NPE means that), with comments linking back to the AWS SDK issue.
Additionally, I will add something to the troubleshooting doc to show a trace for this kind of problem.

Hey @steveloughran -  I think I've covered everything previously discussed now if you can have a look again?
One thing I did notice with my last update is that the exception message from copyFile() does not qualify the paths.  That's previous behavior, but I'm guess was a mistake?  Should I make an update here to qualify those?  You can see an example in what I added to s3guard.md.
I'll be running the full test suite again and posting the results.

BTW, note that the CopyOutcome class only stores SdkBaseExceptions raised in the copy, as does the method to await the copy to complete. I've not put that through all the way through the other bits of the patch

Realized I forgot that.  57ad29b58c83e17e313d57c6dd09310c6b3cad18 covers it.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
37
Docker mode activated.

_ Prechecks _

+1
dupname
2
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 25 new or modified test files.

_ trunk Compile Tests _

0
mvndep
61
Maven dependency ordering for branch

+1
mvninstall
1051
trunk passed

+1
compile
1131
trunk passed

+1
checkstyle
132
trunk passed

+1
mvnsite
116
trunk passed

+1
shadedclient
955
branch has no errors when building and testing our client artifacts.

+1
javadoc
87
trunk passed

0
spotbugs
61
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
177
trunk passed

_ Patch Compile Tests _

0
mvndep
21
Maven dependency ordering for patch

+1
mvninstall
71
the patch passed

+1
compile
964
the patch passed

+1
javac
964
the patch passed

-0
checkstyle
142
root: The patch generated 39 new + 70 unchanged - 4 fixed = 109 total (was 74)

+1
mvnsite
127
the patch passed

+1
whitespace
1
The patch has no whitespace issues.

+1
xml
3
The patch has no ill-formed XML file.

+1
shadedclient
695
patch has no errors when building and testing our client artifacts.

+1
javadoc
104
the patch passed

+1
findbugs
198
the patch passed

_ Other Tests _

+1
unit
513
hadoop-common in the patch passed.

+1
unit
284
hadoop-aws in the patch passed.

+1
asflicense
52
The patch does not generate ASF License warnings.

7860

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-794/8/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/794

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle

uname
Linux 6eb65570ba50 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 5847e00

Default Java
1.8.0_191

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/8/artifact/out/diff-checkstyle-root.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/8/testReport/

Max. process+thread count
1389 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/8/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 25 new or modified test files.

_ trunk Compile Tests _

0
mvndep
98
Maven dependency ordering for branch

+1
mvninstall
1218
trunk passed

+1
compile
1270
trunk passed

+1
checkstyle
152
trunk passed

+1
mvnsite
130
trunk passed

+1
shadedclient
1045
branch has no errors when building and testing our client artifacts.

+1
javadoc
95
trunk passed

0
spotbugs
68
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
196
trunk passed

_ Patch Compile Tests _

0
mvndep
22
Maven dependency ordering for patch

+1
mvninstall
81
the patch passed

+1
compile
1297
the patch passed

+1
javac
1297
the patch passed

-0
checkstyle
151
root: The patch generated 39 new + 70 unchanged - 4 fixed = 109 total (was 74)

+1
mvnsite
129
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
3
The patch has no ill-formed XML file.

+1
shadedclient
702
patch has no errors when building and testing our client artifacts.

+1
javadoc
106
the patch passed

+1
findbugs
204
the patch passed

_ Other Tests _

+1
unit
581
hadoop-common in the patch passed.

+1
unit
296
hadoop-aws in the patch passed.

+1
asflicense
51
The patch does not generate ASF License warnings.

9383

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-794/9/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/794

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle

uname
Linux c75e51fa5147 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 5847e00

Default Java
1.8.0_191

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/9/artifact/out/diff-checkstyle-root.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/9/testReport/

Max. process+thread count
1387 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/9/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

I'll do another run with a bucket with object versioning disabled.

That's done now.  Like when I ran with versioning enabled, the only test with an error was ITestDirectoryCommitMRJob>AbstractITCommitMRJob.testMRJob, which succeeded on an individual run:
[ERROR] Errors:
[ERROR]   ITestDirectoryCommitMRJob>AbstractITCommitMRJob.testMRJob:133->AbstractFSContractTestBase.assertIsDirectory:327 » FileNotFound
[INFO]
[ERROR] Tests run: 933, Failures: 0, Errors: 1, Skipped: 205

I'm travelling right now & can't review this until wed/thur -it's got priority as I want this in.
The failure of the MR commit is interesting as I'm seeing it too -I need to understand why that test is being flaky. Big Q: is it only with this patch which makes it surface? If you could do a test run with your settings against trunk, that'd help isolate us.
BTW PR #818 does the SDK update. Again, having a play with that would be good in general.
w.r.t qualifying paths in errors, there's no consistency. I like to get the full FS URI as it helps debug things.

Thanks Steve, hope you have a great trip.
I've seen the ITestDirectoryCommitMRJob failure before.  I noted it on my PR for HADOOP-16221.
It looks to me like you logged the issue as HADOOP-16207 back on March 26.  The only change I worked on that got in before that was for HADOOP-15625.  I suppose it is possible that it is somehow related to that.  I can try a run from the commit just prior to that.  I'll also have a closer look at that test to see if I can deduce what might be going wrong.

Ben: my etag-server version of the NPE test is failing on the client. Probably there's some config quirk surfacing where the options aren't being cleared. Changing the text of the exception to be clear it's on the client

org.apache.hadoop.fs.s3a.RemoteFileChangedException: copy `s3a://hwdev-steve-ireland-new/test/rename-eventuallyNPE.dat': ETag change detected on client during copy. Expected b81ae3a09fdd252ca7b30bddfa3cf2bd got 9d4c55e9fcea10126a50efe60111e522

	at org.apache.hadoop.fs.s3a.impl.ChangeDetectionPolicy.onChangeDetected(ChangeDetectionPolicy.java:306)
	at org.apache.hadoop.fs.s3a.impl.ChangeTracker.processNewRevision(ChangeTracker.java:287)
	at org.apache.hadoop.fs.s3a.impl.ChangeTracker.processMetadata(ChangeTracker.java:259)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$5(S3AFileSystem.java:1587)
	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:406)
	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:369)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1579)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$copyFile$13(S3AFileSystem.java:3030)
	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:3028)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:1258)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:1119)
	at org.apache.hadoop.fs.s3a.ITestS3ARemoteFileChanged.lambda$testRenameEventuallyConsistentFileNPE$6(ITestS3ARemoteFileChanged.java:573)
	at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:498)
	at org.apache.hadoop.fs.s3a.ITestS3ARemoteFileChanged.testRenameEventuallyConsistentFileNPE(ITestS3ARemoteFileChanged.java:570)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)

Created #824 with my changes.
@bgaborg and I worked out that the reason your tests were working but our test runs failing were because you'd explicitly set authoritative = false in your config files, whereas we were going with the default, and for hadoop-aws test runs, auth is automatically set to true (we should look at that elsewhere).
Also made some other changes about error reporting and how assertions on exceptions in test failures are examined. FWIW, here's my (best practises on test design)[https://github.com/steveloughran/formality/blob/master/styleguide/styleguide.md], a two keys ones being:

never hard code the string to check for in an exception if you have a choice, just add some constant in src/main and refer to it in both places
stack traces are too precious to lose.

With these test we have consistent test runs. Let's see what Yetus says.
We've been discussing how we ought to actually switch between auth and nonauth mode in some of the parameterized tests so we can actually explore both options; the value of getObjectMetadata() calls expected in the tests would be: isAuth? 0 : 1. Do you want to try that?
once we've got that test coverage down and we are all happy with our test suites working locally, we should be done

@bgaborg and I worked out that the reason your tests were working but our test runs failing were because you'd explicitly set authoritative = false in your config files, whereas we were going with the default, and for hadoop-aws test runs, auth is automatically set to true (we should look at that elsewhere).

Hmmm, strange.  I haven't actually set anything about auth mode in my config.  I'll have to look at this again.  Unfortunately I have some other things going on here that are probably going to require most of my attention today.  I will get back to this as quickly as I can.

OK, I'll do the following

tweak the tests as proposed to switch auth options
my proposed change to Listing.java
test against unversioned bucket

as soon as you can after that, if you can do a test run and are happy then we will declare victory here, get it in, and I can then worry about merge conflicts with HADOOP-15183 in that PR alone.

Also:

Adding something to the test docs about versioned stores.
s3guard bucket info now prints the change policy

bin/hadoop s3guard bucket-info s3a://hwdev-steve-ireland-new/
Filesystem s3a://hwdev-steve-ireland-new
Location: eu-west-1
Filesystem s3a://hwdev-steve-ireland-new is using S3Guard with store DynamoDBMetadataStore{region=eu-west-1, tableName=hwdev-steve-ireland-new, tableArn=arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new}
Authoritative S3Guard: fs.s3a.metadatastore.authoritative=false
Metadata Store Diagnostics:
	ARN=arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new
	billing-mode=per-request
	description=S3Guard metadata store in DynamoDB
	name=hwdev-steve-ireland-new
	persist.authoritative.bit=true
	read-capacity=0
	region=eu-west-1
	retryPolicy=ExponentialBackoffRetry(maxRetries=9, sleepTime=250 MILLISECONDS)
	size=108591
	status=ACTIVE
	table={AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeType: S}],TableName: hwdev-steve-ireland-new,KeySchema: [{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,CreationDateTime: Thu Oct 11 18:51:14 BST 2018,ProvisionedThroughput: {LastIncreaseDateTime: Tue Oct 30 16:48:45 GMT 2018,LastDecreaseDateTime: Tue Oct 30 18:00:03 GMT 2018,NumberOfDecreasesToday: 0,ReadCapacityUnits: 0,WriteCapacityUnits: 0},TableSizeBytes: 108591,ItemCount: 682,TableArn: arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new,TableId: a7b0728a-f008-4260-b2a0-ff3dd03367d1,}
	write-capacity=0
The "magic" committer is supported

S3A Client
	Signing Algorithm: fs.s3a.signing-algorithm=
	Endpoint: fs.s3a.endpoint=s3-eu-west-1.amazonaws.com
	Encryption: fs.s3a.server-side-encryption-algorithm=none
	Input seek policy: fs.s3a.experimental.input.fadvise=normal
	Change Detection Source: fs.s3a.change.detection.source=etag
	Change Detection Mode: fs.s3a.change.detection.mode=server
Delegation token support is disabled

I've looked at all your changes and they look good to me. I'll pull them into this PR shortly and repeat my test runs. You've definitely made some good improvements. The doc added to ITestRemoteFileChanged is especially nice.

for hadoop-aws test runs, auth is automatically set to true

Where is that happening? In my quick search I am not seeing it. It looks to me as though you would need to pass -Dauth to switch it on (something I have not done in any of my runs). If you've been running the tests with auth mode on then that does explain why tests were passing for me but not for you though and your fix looks like a good one.

moved from two spaces after a "." to one. Sorry.

No need to apologize - I'm not offended :) I'm struggling slightly to unlearn the two spaces thing. I'm not sure why, but I've only become aware of it recently and the behavior is so ingrained in me. I'm young enough not to have ever typed anything real on a typewriter yet for whatever reason in school I learned the two spaces.

I pushed a fast-forward to the latest commit from #824 into this PR.
Also, I completed my first full test suite run against a bucket with versioning disabled:
mvn -T 1C verify -Dparallel-tests -DtestsThreadCount=8 -Ds3guard -Ddynamo

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   ITestS3AMiscOperations.testEmptyFileChecksums:147->Assert.assertEquals:118->Assert.failNotEquals:834->Assert.fail:88 checksums expected:<etag: "e1cdb893a982e62a4de587b1e3cbf7fb"> but was:<etag: "bc7a796a8fb5cd2d8bcf7f3c6479c188">
[ERROR]   ITestS3AMiscOperations.testNonEmptyFileChecksumsUnencrypted:199->Assert.assertEquals:118->Assert.failNotEquals:834->Assert.fail:88 checksums expected:<etag: "d96315e900853a75d6e4998a48a23c66"> but was:<etag: "52ce70aa15620acd1703adcc8dff8a91">
[ERROR] Errors:
[ERROR]   ITestDirectoryCommitMRJob>AbstractITCommitMRJob.testMRJob:133->AbstractFSContractTestBase.assertIsDirectory:327 » FileNotFound
[INFO]
[ERROR] Tests run: 933, Failures: 2, Errors: 1, Skipped: 205

The failures in ITestS3AMiscOperations seem to be related to the fact that I recently turned on default encryption on the bucket (to quiet some warnings I was getting from some best practice auditing my org does against buckets).  I turned that back off and re-ran that test individually and it succeeded.  I'll log a new JIRA to track that.
mvn -T 1C verify -Dtest=skip -Dit.test=ITestS3AMiscOperations -Ds3guard -Ddynamo

The failure from ITestDirectoryCommitMRJob is one that has been discussed before.  Re-running that test individually yielded success.
mvn -T 1C verify -Dtest=skip -Dit.test=ITestDirectoryCommitMRJob -Ds3guard -Ddynamo

Now I'll run the suite one more time against a bucket with versioning enabled.

The failures in ITestS3AMiscOperations seem to be related to the fact that I recently turned on default encryption on the bucket (to quiet some warnings I was getting from some best practice auditing my org does against buckets). I turned that back off and re-ran that test individually and it succeeded. I'll log a new JIRA to track that.

Logged HADOOP-16085.

Now I'll run the suite one more time against a bucket with versioning enabled.

Done with that now too:
mvn -T 1C verify -Dtest=skip -Dit.test=ITestS3AMiscOperations -Ds3guard -Ddynamo

[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   ITestDirectoryCommitMRJob>AbstractITCommitMRJob.testMRJob:133->AbstractFSContractTestBase.assertIsDirectory:327 » FileNotFound
[INFO]
[ERROR] Tests run: 933, Failures: 0, Errors: 1, Skipped: 145

As before, ITestDirectoryCommitMRJob succeeds individually.
mvn -T 1C verify -Dtest=skip -Dit.test=ITestDirectoryCommitMRJob -Ds3guard -Ddynamo

I'm happy with everything here now.  Thanks so much for all of your help!  Let me know if there's anything more you'd like to see.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
34
Docker mode activated.

_ Prechecks _

+1
dupname
2
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 25 new or modified test files.

_ trunk Compile Tests _

0
mvndep
56
Maven dependency ordering for branch

+1
mvninstall
1048
trunk passed

+1
compile
1030
trunk passed

+1
checkstyle
143
trunk passed

+1
mvnsite
132
trunk passed

+1
shadedclient
1019
branch has no errors when building and testing our client artifacts.

+1
javadoc
109
trunk passed

0
spotbugs
69
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
185
trunk passed

_ Patch Compile Tests _

0
mvndep
24
Maven dependency ordering for patch

+1
mvninstall
79
the patch passed

+1
compile
985
the patch passed

+1
javac
985
the patch passed

-0
checkstyle
148
root: The patch generated 21 new + 70 unchanged - 4 fixed = 91 total (was 74)

+1
mvnsite
126
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
3
The patch has no ill-formed XML file.

+1
shadedclient
687
patch has no errors when building and testing our client artifacts.

+1
javadoc
85
the patch passed

+1
findbugs
200
the patch passed

_ Other Tests _

+1
unit
513
hadoop-common in the patch passed.

+1
unit
281
hadoop-aws in the patch passed.

+1
asflicense
43
The patch does not generate ASF License warnings.

6925

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-794/10/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/794

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle

uname
Linux 6871fdcb5766 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 4cb3da6

Default Java
1.8.0_212

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/10/artifact/out/diff-checkstyle-root.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/10/testReport/

Max. process+thread count
1410 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-794/10/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

OK, I'm happy with this, and am doing my preflight tests

get the latest PR and retest
tag, rebase onto trunk & retest (may need changes, though I hope not)
if changes => new PR and yetus
if not: merge

happy; rebase and retest. Tests in the final run failing were related to SSL (HADOOP-16050) and again, an intermittent v1 listing one.
We've seen that v1 list failure before so it's not related, at the same time I'm somewhat worried about hints of inconsistencies as well as the "why is this only the v1 test". Not letting it stop me here though
+1

The changes from this were merged into trunk as https://github.com/apache/hadoop/commit/a36274d69947648dbe82721220cc5240ec5d396d.  As such, I think it makes sense to close this out.


Track fault detection using DL

Pull Request for DL-Simplified 💡
Issue Title :  Track fault detection using DL

Info about the related issue (Aim of the project) : To create deep learning models for railway track fault detection.
Name: Dammala Uday Kiran
GitHub ID: the-silent-geek
Email ID<EMAIL_ADDRESS>Idenitfy yourself: (Mention in which program you are contributing in. Eg. For a JWOC 2022 participant it's, JWOC Participant) : contributor SWOC 24

Closes: #224
Describe the add-ons or changes you've made 📃
Added dataset, accuracy plots, and 5 models for fault detection.
Type of change ☑
What sort of change have you made:

[x] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[x] Code style update (formatting, local variables)
[x] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested? ⚙
Describe how it has been tested
Describe how have you verified the changes made
Checklist: ☑

[x] My code follows the guidelines of this project.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly wherever it was hard to understand.
[x] I have made corresponding changes to the documentation.
[x] My changes generate no new warnings.
[x] I have added things that prove my fix is effective or that my feature works.
[x] Any dependent changes have been merged and published in downstream modules.

Sorry, I think the other folder with the same name has been deleted. I will add it to the PR.

I have added the previous folder and all its components


Is this compatible with d3 v4?

what all the libraries to be changed if it is compatible with d3 v4?
Thanks & Regards
Govardhan

Current C3 uses D3 v5
What a time to be alive


Get-RubrikDatabase -Relic:$false returns both Relics and non-Relics

Current Behavior:
Provide information about the failure by issuing the command using the -Verbose command.
PS O:\> Get-RubrikDatabase -Name abc -Hostname xyz -PrimaryClusterID 1ae6a92a-cb07-433e-be9e-87227dca6c9b -Relic:$false -Verbose
VERBOSE: Validate the Rubrik token exists
VERBOSE: Found a Rubrik token for authentication
VERBOSE: Gather API Data for Get-RubrikDatabase
VERBOSE: Selected 1.0 API Data for Get-RubrikDatabase
VERBOSE: Load API data for
VERBOSE: Description: Returns a list of summary information for Microsoft SQL databases.
VERBOSE: Build the URI
VERBOSE: URI = https://url/api/v1/mssql/db
VERBOSE: Build the query parameters for is_relic,instance_id,primary_cluster_id,effective_sla_domain_id
VERBOSE: Query = Found limit flag
VERBOSE: URI = https://url/api/v1/mssql/db?primary_cluster_id=1ae6a92a-cb07-433e-be9e-87227dca6c9b&limit=9999
VERBOSE: Performing the operation "Returns a list of summary information for Microsoft SQL databases." on target "".
VERBOSE: Submitting the request
VERBOSE: GET https://url/api/v1/mssql/db?primary_cluster_id=1ae6a92a-cb07-433e-be9e-87227dca6c9b&limit=9999 with 0-byte payload
VERBOSE: received 504779-byte response of content type application/json; charset=UTF-8
VERBOSE: Formatting return value
VERBOSE: Filter the results
VERBOSE: Filter match = Instance
VERBOSE: Filter match = Name
VERBOSE: Filter match = Hostname
VERBOSE: Filter match = SLA

availabilityGroupId          : MssqlAvailabilityGroup:::31a8afc2-6c08-4d76-8d65-47aff962776d
effectiveSlaDomainId         : 5f6be6dd-24c6-48de-918f-4918b1e38229
primaryClusterId             : 1ae6a92a-cb07-433e-be9e-87227dca6c9b
unprotectableReasons         : {}
effectiveSlaSourceObjectId   : MssqlAvailabilityGroup:::31a8afc2-6c08-4d76-8d65-47aff962776d
effectiveSlaSourceObjectName : xyz
configuredSlaDomainId        : INHERIT
isLogShippingSecondary       : False
effectiveSlaDomainName       : LAB_DLY
copyOnly                     : False
id                           : MssqlDatabase:::6740c632-2c87-4736-9f43-568d355af9b2
isInAvailabilityGroup        : True
isLiveMount                  : False
configuredSlaDomainName      : Inherit
replicas                     : {@{instanceId=MssqlInstance:::5ed23ad2-721b-4d4d-a3ab-8230a303d3d7; instanceName=abc; recoveryModel=FULL; state=ONLINE; hasPermissions=True; isStandby=False; recoveryForkGuid=A26CEABC-9B0F-4092-80E8-83683DDC22A9; isArchived=False; isDeleted=False; availabilityInfo=; rootProperties=}, @{instanceId=MssqlInstance:::de49eecd-2222-49f1-9c6c-236e41202f43; instanceName=abc; recoveryModel=FULL; state=ONLINE; hasPermissions=True; isStandby=False;
                               recoveryForkGuid=A26CEABC-9B0F-4092-80E8-83683DDC22A9; isArchived=False; isDeleted=False; availabilityInfo=; rootProperties=}}
slaAssignment                : Derived
rootProperties               : @{rootType=MssqlAvailabilityGroup; rootId=MssqlAvailabilityGroup:::31a8afc2-6c08-4d76-8d65-47aff962776d; rootName=xyz}
logBackupRetentionHours      : 168
isRelic                      : False
name                         : abc
logBackupFrequencyInSeconds  : 900

availabilityGroupId          : MssqlAvailabilityGroup:::31a8afc2-6c08-4d76-8d65-47aff962776d
effectiveSlaDomainId         : UNPROTECTED
primaryClusterId             : 1ae6a92a-cb07-433e-be9e-87227dca6c9b
unprotectableReasons         : {}
effectiveSlaSourceObjectId   : MssqlDatabase:::d67209c5-57fd-4c40-b1a6-ba3d33e049d0
effectiveSlaSourceObjectName : abc
configuredSlaDomainId        : 5f6be6dd-24c6-48de-918f-4918b1e38229
isLogShippingSecondary       : False
effectiveSlaDomainName       : Unprotected
copyOnly                     : False
id                           : MssqlDatabase:::d67209c5-57fd-4c40-b1a6-ba3d33e049d0
isInAvailabilityGroup        : True
isLiveMount                  : False
configuredSlaDomainName      : LAB_DLY
replicas                     : {@{instanceId=MssqlInstance:::5ed23ad2-721b-4d4d-a3ab-8230a303d3d7; instanceName=abc; recoveryModel=FULL; state=ONLINE; hasPermissions=True; isStandby=False; recoveryForkGuid=D46FDD2B-FD77-4FC6-88FF-3776B93B744F; isArchived=True; isDeleted=True; availabilityInfo=; rootProperties=}, @{instanceId=MssqlInstance:::de49eecd-2222-49f1-9c6c-236e41202f43; instanceName=abc; recoveryModel=FULL; state=ONLINE; hasPermissions=True; isStandby=False;
                               recoveryForkGuid=D46FDD2B-FD77-4FC6-88FF-3776B93B744F; isArchived=True; isDeleted=True; availabilityInfo=; rootProperties=}}
slaAssignment                : Direct
rootProperties               : @{rootType=MssqlAvailabilityGroup; rootId=MssqlAvailabilityGroup:::31a8afc2-6c08-4d76-8d65-47aff962776d; rootName=xyz}
logBackupRetentionHours      : 168
isRelic                      : True
name                         : abc
logBackupFrequencyInSeconds  : 900

Expected Behavior:
We should not see the 'isRelic : True' result when using the -Relic:$false switch.
Similarly, the -Relic switch defaults to true (and even if I force it to -Relic:$true) I get both the True and False results.
Expected behavior would be that the -Relic switch actually displays only relics or non-relics when specified.
Steps to Reproduce:
Please provide detailed steps for reproducing the issue.

Run Get-RubrikDatabase -Relic:$false on a SQL DB in a SQL AG that has relics.

Context:
Please provide any relevant information about your setup. This is important in case the issue is not reproducible except for under certain conditions.

Rubrik PowerShell Module Version: <IP_ADDRESS>5
PowerShell Version: 5.1.15063.1805
Operating System: Windows 10 1703
Rubrik CDM: 5.0.0-p2-1122

Failure Logs
Please include any relevant log snippets or files here.

@cmckeeg The -Relic parameter is a switch parameter, meaning it's either on or off.
For example
'Get-RubrikDatabase -Relic'  - you will only see relics
'Get-RubrikDatabase' - you will see both relics and non-relics
Running Get-RubrikDatabase -Relic:$false will display all results as well (the false won't help in the case as it's not a boolean.
Currently, everything is functioning as expected - to get a list of non-relic database you could run
Get-RubrikDatabase | where {$_.isRelic -eq $false}
Does this help at all?
@jaapbrasser - do you see value in switching this to a boolean and using the filter functionality instead?

@mwpreston Thank you! I now see in the Get-Help for Get-RubrikDatabase that -Relic is not a boolean. Using the 'where' to filter it works for us in my brief testing.
@mwpreston @jaapbrasser It would be nice if the Get-RubrikDatabase -Relic switch were boolean, but that's just because it meets my needs and cuts down on length of code.

@cmckeeg I'll raise a PR for this to work towards a solution that works with the switch parameter as you described


publication gets closed on cpp publisher

I am building a small network with two modules on C++ and one on Go. The only problem I have is with the cpp publisher ( the client works fine )
I have separated the publication creation method and the message sender since I need to send messages as the origin message is ready ( its a stream simulator ). What I'm doing here is to return the shared ptr and pass it to the message method every time there is a new message that must be sent. I'm getting -4 code on the publication when offering / sending the message, it says the publication is closed. Why ?
if you can shed some light in to this I'd very much appreciated. Thanks !
#include "pubsub.h"

std::atomic<bool> running(true);

void sigIntHandler(int)
{
    running = false;
}

std::shared_ptr<aeron::Publication> PubSub::runPublisher()
{

    try{

        Settings settings;

        std::cout << "Publishing to channel " << settings.channel << " on Stream ID " << settings.streamId << std::endl;

        aeron::Context context;

        if (!settings.dirPrefix.empty())
        {
            context.aeronDir(settings.dirPrefix);
        }

        context.newPublicationHandler(
            [](const std::string &channel, std::int32_t streamId, std::int32_t sessionId, std::int64_t correlationId)
            {
                std::cout << "Publication: " << channel << " " << correlationId << ":" << streamId << ":" << sessionId << std::endl;
            });

        std::shared_ptr<Aeron> aeron = Aeron::connect(context);
        signal(SIGINT, sigIntHandler);
        // add the publication to start the process
        std::int64_t id = aeron->addPublication(settings.channel, settings.streamId);

        std::shared_ptr<Publication> publication = aeron->findPublication(id);

        publication = aeron->findPublication(id);
        // wait for the publication to be valid
        while (!publication)
        {
            std::this_thread::yield();
            publication= aeron->findPublication(id);
        }

        const std::int64_t channelStatus = publication->channelStatus();

        std::cout << "Publication channel status (id=" << publication->channelStatusId() << ") "
                << (channelStatus == ChannelEndpointStatus::CHANNEL_ENDPOINT_ACTIVE ? "ACTIVE" : std::to_string(channelStatus))
                << std::endl;

        std::cout << running << std::endl;

        return publication;

    }
    catch (const SourcedException &e)
    {
        std::cerr << "FAILED: " << e.what() << " : " << e.where() << std::endl;
        // return -1;
    }
    catch (const std::exception &e)
    {
        std::cerr << "FAILED: " << e.what() << " : " << std::endl;
        // return -1;
    }
}

void PubSub::sendMessage(std::string &jsonStringMessage,std::shared_ptr<aeron::Publication> &publication)

{

    const std::int64_t channelStatus = publication->channelStatus();

    std::cout << channelStatus << std::endl;

    AERON_DECL_ALIGNED(buffer_t buffer, 16);
    concurrent::AtomicBuffer srcBuffer(&buffer[0], buffer.size());

    char *message;
    message = &jsonStringMessage[0];

    if (running)
    {
        std::cout << "it's running" << std::endl;
        // std::cout << jsonStringMessage << std::endl;
        const int messageLen = ::snprintf(message, sizeof(message), PRId64);
        const std::int64_t result = publication->offer(srcBuffer, 0, messageLen);

        std::cout << result << std::endl;

        if (result > 0)
        {
            std::cout << "yay!" << std::endl;
        }
        else if (BACK_PRESSURED == result)
        {
            std::cout << "Offer failed due to back pressure" << std::endl;
        }
        else if (NOT_CONNECTED == result)
        {
            std::cout << "Offer failed because publisher is not connected to a subscriber" << std::endl;
        }
        else if (ADMIN_ACTION == result)
        {
            std::cout << "Offer failed because of an administration action in the system" << std::endl;
        }
        else if (PUBLICATION_CLOSED == result)
        {
            std::cout << "Offer failed because publication is closed" << std::endl;
        }
        else
        {
            std::cout << "Offer failed due to unknown reason " << result << std::endl;
        }

        if (!publication->isConnected())
        {
            std::cout << "No active subscribers detected" << std::endl;
        }
    }
}

The Publication is owned by an Aeron client. If the Aeron client goes out of scope then all its associated resources are closed.

Thank you for the hint ... with the client you mean the context ?

I mean the Aeron client.


⚠ Bofrost.de has degraded performance

In 0f3fca1, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1370 ms

Resolved: Bofrost.de performance has improved in e641029.


Question about basecalling accuracy

Hi:
I have some doubts about basecalling results,wish to get your guidance.
I used the trained model that you have already published，the test dataset is human-dataset.After running many times using different dataset,I never get high accuracy.Accuracy is 50% or less.Do you have any suggestions for this?

I need further details as to how you're running rodan and on what dataset. Specifically command line options which includes how you're checking accuracy.


Configure Tests command overrides project testpaths

Type: Bug
Behaviour
Discover tests is overriding standard testpaths after configuration with the "Configure Tests" command.
By forcing the selection of the directory, the pytest --collect-only test discovery is run with a single directory as the argument, which overrides any testpaths set in external tools, for example pyproject.toml
If the command allows skipping of the directory selection, and doesn't set the related pytestArgs setting, test discovery will run normally.
Steps to reproduce:

Configure project with pyproject.toml with [tools.pytest.ini_options] that defines testspaths = ["some","list","of/different","testpaths"]
Open in vscode with python plugin
Select "Python: Configure Tests" from action menu, or via "Testing" -> button
Select pytest
Select root directory
Discover tests takes a long time in large directory trees (e.g. .venv, libs, etc) because the configured testpaths are being overridden.

Diagnostic data
From the Python output logs:
....
2024-06-27 07:39:05.246 [info] > ./.bluegen-venv/bin/python -m pytest -p vscode_pytest --collect-only .
2024-06-27 07:39:05.246 [info] cwd: .
2024-06-27 07:39:07.900 [info] ============================= test session starts ==============================
....

From the settings:
{
        ...
        "python.testing.pytestEnabled": true,
	"python.testing.pytestArgs": [
		"."
	],
        ...
}

Extension version: 2024.8.1
VS Code version: Code 1.90.2 (5437499feb04f7a586f677b155b039bc2b3669eb, 2024-06-18T22:33:48.698Z)
OS version: Linux x64 6.8.0-36-generic
Modes:
Remote OS version: Linux x64 6.8.0-36-generic

Hi! If you delete all your pytest args (or more specifically the ".") does this fix the problem? Go to settings.json and look for "python.testing.pytestArgs"

Yes, it solves the problem, and it's what I did as a workaround.
It just took a while to figure out why the paths were being ignored, and why the discovery took so long.

@Jurriaan-BlueGen understood thanks! We have the setup steps which include selecting what folder your tests are in but we should be more explicit about that answer being used to update / change the args


Is there an apk to test this?

Hi, I was wondering if there's an apk that we can run on Android to test out without having to build it ourselves.

Not yet


Innovation - Using Objects

Week 2 Review Ticket
Using Objects Tech Talk

Code.org OOP Unit 1

Following Point Guideline --> Issue showing how team has a shared repository and has plans to split up work.

2.7+/3
Graded by: Dylan Luo
Completed all of the assignment requirements, as he created a Java menu with interesting features, finished Code.org lessons up to 15, and worked well with me during partner programming.