HADOOP-17771. S3AFS creation fails "Unable to find a region via the region provider chain."

Contributed by Steve Loughran.
Change-Id: I94284178d27a48947e7c0942a7c8565379de7e9b

Testing in progress, in a setup where

the failure mode is replicated
fs.s3a.endpoint is not set globally or for the test bucket.

cloudstore regions
This verifies that the local system is set up to fail without the patch.
> bin/hadoop jar $CLOUDSTORE regions

Determining AWS region for SDK clients
======================================

Determining region using AwsEnvVarOverrideRegionProvider
========================================================

Use environment variable AWS_REGION
2021-06-22 13:12:20,509 [main] INFO  extra.Regions (StoreDurationInfo.java:<init>(53)) - Starting: AwsEnvVarOverrideRegionProvider.getRegion()
2021-06-22 13:12:20,512 [main] INFO  extra.Regions (StoreDurationInfo.java:close(100)) - AwsEnvVarOverrideRegionProvider.getRegion(): duration 0:00:004
region is not known

Determining region using AwsSystemPropertyRegionProvider
========================================================

System property aws.region
2021-06-22 13:12:20,513 [main] INFO  extra.Regions (StoreDurationInfo.java:<init>(53)) - Starting: AwsSystemPropertyRegionProvider.getRegion()
2021-06-22 13:12:20,513 [main] INFO  extra.Regions (StoreDurationInfo.java:close(100)) - AwsSystemPropertyRegionProvider.getRegion(): duration 0:00:000
region is not known

Determining region using AwsProfileRegionProvider
=================================================

Region info in ~/.aws/config
2021-06-22 13:12:20,535 [main] INFO  extra.Regions (StoreDurationInfo.java:<init>(53)) - Starting: AwsProfileRegionProvider.getRegion()
2021-06-22 13:12:20,550 [main] INFO  extra.Regions (StoreDurationInfo.java:close(100)) - AwsProfileRegionProvider.getRegion(): duration 0:00:015
region is not known

Determining region using InstanceMetadataRegionProvider
=======================================================

EC2 metadata; will only work in AWS infrastructure
2021-06-22 13:12:20,551 [main] INFO  extra.Regions (StoreDurationInfo.java:<init>(53)) - Starting: InstanceMetadataRegionProvider.getRegion()
2021-06-22 13:12:20,552 [main] INFO  extra.Regions (StoreDurationInfo.java:close(100)) - InstanceMetadataRegionProvider.getRegion(): duration 0:00:001
WARNING: Provider raised an exception com.amazonaws.AmazonClientException: AWS_EC2_METADATA_DISABLED is set to true, not loading region from EC2 Instance Metadata service
region is not known

Region was NOT FOUND
====================

WARNING: AWS region was not determined through SDK region chain
WARNING: This may not work
2021-06-22 13:12:20,554 [main] INFO  util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 50:

hadoop fs -ls without patch
 bin/hadoop fs -ls s3a://stevel-usw2/
 ...
 ls: initializing  on s3a://stevel-usw2/: com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.

hadoop fs -ls with patch
(+iostats to log @ info; auditor logging enabled)
2021-06-22 13:16:51,907 [s3a-transfer-stevel-usw2-unbounded-pool2-t1] DEBUG impl.LoggingAuditor (LoggingAuditor.java:beforeExecution(327)) - [16] 9f36cb47-8dea-4e46-8e5a-e4b5ed3c0304-00000007 Executing op_list_status with {object_list_request '' size=5000, mutating=false}; https://audit.example.org/hadoop/1/op_list_status/9f36cb47-8dea-4e46-8e5a-e4b5ed3c0304-00000007/?op=op_list_status&pr=stevel&ps=3878272c-5b04-487e-be1f-0562bc7e4b21&cm=FsShell&id=9f36cb47-8dea-4e46-8e5a-e4b5ed3c0304-00000007&t0=1&fs=9f36cb47-8dea-4e46-8e5a-e4b5ed3c0304&t1=16&ts=1624364211897
2021-06-22 13:16:53,469 [shutdown-hook-0] INFO  statistics.IOStatisticsLogging (IOStatisticsLogging.java:logIOStatisticsAtLevel(269)) - IOStatistics: counters=((audit_request_execution=1)
(audit_span_creation=4)
(object_list_request=1)
(op_get_file_status=1)
(op_glob_status=1)
(op_list_status=1)
(store_io_request=2));

gauges=();

minimums=((object_list_request.min=1550)
(op_get_file_status.min=2)
(op_glob_status.min=8)
(op_list_status.min=1563));

maximums=((object_list_request.max=1550)
(op_get_file_status.max=2)
(op_glob_status.max=8)
(op_list_status.max=1563));

means=((object_list_request.mean=(samples=1, sum=1550, mean=1550.0000))
(op_get_file_status.mean=(samples=1, sum=2, mean=2.0000))
(op_glob_status.mean=(samples=1, sum=8, mean=8.0000))

The list status went through
Storediag with patch
And trimmed storediag; note how bucket HEAD returns the region along with the 401 x-amz-bucket-region: us-west-2. this is how the SDK determines which real endpoint to use.
Store Diagnostics for stevel (auth:SIMPLE) on stevel-mbp15-13176.local/<IP_ADDRESS>
====================================================================================

Diagnostics for filesystem s3a://stevel-usw2/
=============================================

S3A FileSystem Connector
ASF Filesystem Connector to Amazon S3 Storage and compatible stores
https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html

Hadoop information
==================

  Hadoop 3.4.0-SNAPSHOT
  Compiled by stevel on 2021-05-26T16:24Z
  Compiled with protoc 3.7.1
  From source with checksum a6a33d523c6f35e2a28e9a9ecf703a27

Determining OS version
======================

Darwin stevel-mbp15-13176.local 20.5.0 Darwin Kernel Version 20.5.0: Sat May  8 05:10:33 PDT 2021; root:xnu-7195.121.3~9/RELEASE_X86_64 x86_64

Selected System Properties
==========================

aws.accessKeyId = (unset)
aws.secretKey = (unset)
aws.sessionToken = (unset)
aws.region = (unset)
com.amazonaws.regions.RegionUtils.fileOverride = (unset)
com.amazonaws.regions.RegionUtils.disableRemote = (unset)
com.amazonaws.sdk.disableCertChecking = (unset)
com.amazonaws.sdk.ec2MetadataServiceEndpointOverride = (unset)
com.amazonaws.sdk.enableDefaultMetrics = (unset)
com.amazonaws.sdk.enableInRegionOptimizedMode = (unset)
com.amazonaws.sdk.enableThrottledRetry = (unset)
com.amazonaws.services.s3.disableImplicitGlobalClients = (unset)
com.amazonaws.services.s3.enableV4 = (unset)
com.amazonaws.services.s3.enforceV4 = (unset)

Environment Variables
=====================

AWS_ACCESS_KEY_ID = (unset)
AWS_ACCESS_KEY = (unset)
AWS_SECRET_KEY = (unset)
AWS_SECRET_ACCESS_KEY = (unset)
AWS_SESSION_TOKEN = (unset)
AWS_REGION = (unset)
AWS_S3_US_EAST_1_REGIONAL_ENDPOINT = (unset)
AWS_CBOR_DISABLE = (unset)
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI = (unset)
AWS_CONTAINER_CREDENTIALS_FULL_URI = (unset)
AWS_CONTAINER_AUTHORIZATION_TOKEN = (unset)
AWS_EC2_METADATA_DISABLED = "true"
AWS_EC2_METADATA_SERVICE_ENDPOINT = (unset)
AWS_MAX_ATTEMPTS = (unset)
AWS_RETRY_MODE = (unset)
HADOOP_CONF_DIR = "/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/etc/hadoop"
HADOOP_CREDSTORE_PASSWORD = (unset)
HADOOP_HEAPSIZE = (unset)
HADOOP_HEAPSIZE_MIN = (unset)
HADOOP_HOME = "/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT"
HADOOP_LOG_DIR = (unset)
HADOOP_OPTIONAL_TOOLS = "hadoop-azure,hadoop-aws,hadoop-openstack"
HADOOP_OPTS = "-Djava.net.preferIPv4Stack=true  -Dyarn.log.dir=/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/logs -Dyarn.log.file=hadoop.log -Dyarn.home.dir=/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT -Dyarn.root.logger=INFO,console -Dhadoop.log.dir=/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT -Dhadoop.id.str=stevel -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender"
HADOOP_SHELL_SCRIPT_DEBUG = (unset)
HADOOP_TOKEN = (unset)
HADOOP_TOKEN_FILE_LOCATION = (unset)
HADOOP_TOOLS_HOME = (unset)
HADOOP_TOOLS_OPTIONS = (unset)
HDP_VERSION = (unset)
LOCAL_DIRS = (unset)
PYSPARK_DRIVER_PYTHON = (unset)
SPARK_HOME = (unset)
SPARK_CONF_DIR = (unset)
SPARK_SCALA_VERSION = (unset)
YARN_CONF_DIR = (unset)

Security
========

Security Enabled: false
Keytab login: false
Ticket login: false
Current user: stevel (auth:SIMPLE)
Token count: 0

Hadoop Options
==============

fs.defaultFS = "file:///" [core-default.xml]
fs.default.name = "file:///"
fs.trash.classname = (unset)
fs.trash.interval = "0" [core-default.xml]
fs.trash.checkpoint.interval = "0" [core-default.xml]
hadoop.tmp.dir = "/tmp/hadoop-stevel" [core-default.xml]
hdp.version = (unset)
yarn.resourcemanager.address = "<IP_ADDRESS>:8032" [yarn-default.xml]
yarn.resourcemanager.principal = (unset)
yarn.resourcemanager.webapp.address = "<IP_ADDRESS>:8088" [yarn-default.xml]
yarn.resourcemanager.webapp.https.address = "<IP_ADDRESS>:8090" [yarn-default.xml]
mapreduce.input.fileinputformat.list-status.num-threads = "1" [mapred-default.xml]
mapreduce.jobtracker.kerberos.principal = (unset)
mapreduce.job.hdfs-servers.token-renewal.exclude = (unset)
mapreduce.application.framework.path = (unset)
fs.iostatistics.logging.level = "info" [core-site.xml]

Security Options
================

dfs.data.transfer.protection = (unset)
hadoop.http.authentication.simple.anonymous.allowed = "true" [core-default.xml]
hadoop.http.authentication.type = "simple" [core-default.xml]
hadoop.kerberos.min.seconds.before.relogin = "60" [core-default.xml]
hadoop.kerberos.keytab.login.autorenewal.enabled = "false" [core-default.xml]
hadoop.security.authentication = "simple" [core-default.xml]
hadoop.security.authorization = "false" [core-default.xml]
hadoop.security.credential.provider.path = (unset)
hadoop.security.credstore.java-keystore-provider.password-file = (unset)
hadoop.security.credential.clear-text-fallback = "true" [core-default.xml]
hadoop.security.key.provider.path = (unset)
hadoop.security.crypto.jceks.key.serialfilter = (unset)
hadoop.rpc.protection = "authentication" [core-default.xml]
hadoop.tokens = (unset)
hadoop.token.files = (unset)

Selected Configuration Options
==============================

fs.s3a.session.token = (unset)
fs.s3a.server-side-encryption-algorithm = (unset)
fs.s3a.server-side-encryption.key = (unset)
fs.s3a.aws.credentials.provider = "
    org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,
    org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,
    com.amazonaws.auth.EnvironmentVariableCredentialsProvider,
    org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
  " [core-default.xml]
fs.s3a.endpoint = (unset)
fs.s3a.endpoint.region = (unset)
fs.s3a.signing-algorithm = (unset)
fs.s3a.acl.default = (unset)
fs.s3a.attempts.maximum = "20" [core-default.xml]
fs.s3a.authoritative.path = "/" [fs.s3a.bucket.stevel-usw2.authoritative.path via [core-site.xml]]
fs.s3a.block.size = "32M" [core-default.xml]
fs.s3a.buffer.dir = "/tmp/hadoop-stevel/s3a" [core-default.xml]
fs.s3a.bulk.delete.page.size = (unset)
fs.s3a.change.detection.source = "etag" [core-default.xml]
fs.s3a.change.detection.mode = "server" [core-default.xml]
fs.s3a.change.detection.version.required = "true" [core-default.xml]
fs.s3a.connection.ssl.enabled = "true" [core-default.xml]
fs.s3a.connection.maximum = "48" [core-default.xml]
fs.s3a.connection.establish.timeout = "5000" [core-site.xml]
fs.s3a.connection.request.timeout = "0" [core-default.xml]
fs.s3a.connection.timeout = "5000" [core-site.xml]
fs.s3a.custom.signers = (unset)
fs.s3a.directory.marker.retention = (unset)
fs.s3a.downgrade.syncable.exceptions = (unset)
fs.s3a.etag.checksum.enabled = "false" [core-default.xml]
fs.s3a.experimental.input.fadvise = (unset)
fs.s3a.experimental.aws.s3.throttling = (unset)
fs.s3a.experimental.optimized.directory.operations = (unset)
fs.s3a.fast.buffer.size = (unset)
fs.s3a.fast.upload.buffer = "bytebuffer" [core-site.xml]
fs.s3a.fast.upload.active.blocks = "4" [core-default.xml]
fs.s3a.impl.disable.cache = (unset)
fs.s3a.list.version = "2" [core-default.xml]
fs.s3a.max.total.tasks = "32" [core-default.xml]
fs.s3a.multipart.size = "8388608" [core-site.xml]
fs.s3a.paging.maximum = "5000" [core-default.xml]
fs.s3a.multiobjectdelete.enable = "true" [core-default.xml]
fs.s3a.multipart.purge = "false" [core-site.xml]
fs.s3a.multipart.purge.age = "3600000" [core-site.xml]
fs.s3a.paging.maximum = "5000" [core-default.xml]
fs.s3a.path.style.access = "false" [core-site.xml]
fs.s3a.proxy.host = (unset)
fs.s3a.proxy.port = (unset)
fs.s3a.proxy.username = (unset)
fs.s3a.proxy.password = (unset)
fs.s3a.proxy.domain = (unset)
fs.s3a.proxy.workstation = (unset)
fs.s3a.readahead.range = "524288" [core-site.xml]
fs.s3a.retry.limit = "7" [core-default.xml]
fs.s3a.retry.interval = "500ms" [core-default.xml]
fs.s3a.retry.throttle.limit = "20" [core-default.xml]
fs.s3a.retry.throttle.interval = "100ms" [core-default.xml]
fs.s3a.ssl.channel.mode = "default" [fs.s3a.bucket.stevel-usw2.ssl.channel.mode via [core-site.xml]]
fs.s3a.s3.client.factory.impl = (unset)
fs.s3a.threads.max = "80" [core-site.xml]
fs.s3a.threads.keepalivetime = "60" [core-default.xml]
fs.s3a.user.agent.prefix = (unset)
fs.s3a.metadatastore.impl = "org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore" [core-default.xml]
fs.s3a.metadatastore.authoritative = "false" [core-default.xml]
fs.s3a.metadatastore.authoritative.dir.ttl = (unset)
fs.s3a.metadatastore.fail.on.write.error = "true" [core-default.xml]
fs.s3a.metadatastore.metadata.ttl = "15m" [core-default.xml]
fs.s3a.s3guard.consistency.retry.interval = "2s" [core-default.xml]
fs.s3a.s3guard.consistency.retry.limit = "7" [core-default.xml]
fs.s3a.s3guard.ddb.table = (unset)
fs.s3a.s3guard.ddb.region = "eu-west-2" [core-site.xml]
fs.s3a.s3guard.ddb.background.sleep = "25ms" [core-default.xml]
fs.s3a.s3guard.ddb.max.retries = "9" [core-default.xml]
fs.s3a.s3guard.ddb.table.capacity.read = "0" [core-default.xml]
fs.s3a.s3guard.ddb.table.capacity.write = "0" [core-default.xml]
fs.s3a.s3guard.ddb.table.create = "false" [core-default.xml]
fs.s3a.s3guard.ddb.throttle.retry.interval = "100ms" [core-default.xml]
fs.s3a.s3guard.local.max_records = (unset)
fs.s3a.s3guard.local.ttl = (unset)
fs.s3a.committer.name = "magic" [core-site.xml]
fs.s3a.committer.magic.enabled = "true" [core-default.xml]
fs.s3a.committer.staging.abort.pending.uploads = (unset)
fs.s3a.committer.staging.conflict-mode = "append" [core-default.xml]
fs.s3a.committer.staging.tmp.path = "tmp/staging" [core-default.xml]
fs.s3a.committer.threads = "8" [core-default.xml]
fs.s3a.committer.staging.unique-filenames = "false" [core-site.xml]
mapreduce.outputcommitter.factory.scheme.s3a = "org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory" [mapred-default.xml]
mapreduce.fileoutputcommitter.marksuccessfuljobs = (unset)
fs.s3a.delegation.token.binding = (unset)
fs.s3a.delegation.token.secondary.bindings = (unset)
fs.s3a.audit.referrer.enabled = (unset)
fs.s3a.audit.referrer.filter = (unset)
fs.s3a.audit.reject.out.of.span.operations = "true" [core-site.xml]
fs.s3a.audit.request.handlers = (unset)
fs.s3a.audit.service.classname = (unset)

Endpoints
=========

Attempting to list and connect to public service endpoints,
without any authentication credentials.
This is just testing the reachability of the URLs.
If the request fails with any network error it is likely
to be configuration problem with address, proxy, etc

If it is some authentication error, then don't worry so much
-look for the results of the filesystem operations

Endpoint: https://stevel-usw2.s3.amazonaws.com/
===============================================

Canonical hostname s3-us-west-2-w.amazonaws.com
  IP address <IP_ADDRESS>
Proxy: none

Connecting to https://stevel-usw2.s3.amazonaws.com/

Response: 403 : Forbidden
HTTP response 403 from https://stevel-usw2.s3.amazonaws.com/: Forbidden
Using proxy: false
Transfer-Encoding: chunked
null: HTTP/1.1 403 Forbidden
Server: AmazonS3
x-amz-request-id: FFH2R0SM4MM8SZR2
x-amz-id-2: nRlKBuQRLD+7AOPpbsRzvASmnZUa4iZSLuqNQbKfyCjnTb3A53jNIIemlR2/ZsZQaveGyJ69p7Y=
Date: Tue, 22 Jun 2021 12:18:43 GMT
x-amz-bucket-region: us-west-2
Content-Type: application/xml

<?xml version="1.0" encoding="UTF-8"?>
<Error>AccessDenied<Message>Access Denied</Message><RequestId>FFH2R0SM4MM8SZR2</RequestId><HostId>nRlKBuQRLD+7AOPpbsRzvASmnZUa4iZSLuqNQbKfyCjnTb3A53jNIIemlR2/ZsZQaveGyJ69p7Y=</HostId></Error>

Test filesystem s3a://stevel-usw2/
==================================

Trying some list and read operations
2021-06-22 13:18:45,526 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(53)) - Starting: Creating filesystem s3a://stevel-usw2/
2021-06-22 13:18:47,017 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(100)) - Creating filesystem s3a://stevel-usw2/: duration 0:01:492
S3AFileSystem{uri=s3a://stevel-usw2, workingDir=s3a://stevel-usw2/user/stevel, inputPolicy=normal, partSize=8388608, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=524288, blockSize=33554432, multiPartThreshold=134217728, serverSideEncryptionAlgorithm='NONE', blockFactory=ByteBufferBlockFactory{buffersOutstanding=0}, auditManager=Service ActiveAuditManagerS3A in state ActiveAuditManagerS3A: STARTED, auditor=LoggingAuditor{ID='1cdd7da8-c418-40c5-ae44-b0398d1b98c5', headerEnabled=true, rejectOutOfSpan=true}}, metastore=NullMetadataStore, authoritativeStore=false, authoritativePath=[s3a://stevel-usw2/], useListV1=false, magicCommitter=true, boundedExecutor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=192, available=192, waiting=0}, activeCount=0}, unboundedExecutor=java.util.concurrent.ThreadPoolExecutor@1b5bc39d[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], credentials=AWSCredentialProviderList[refcount= 1: [TemporaryAWSCredentialsProvider, SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider, org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider@655a5d9c], delegation tokens=disabled, DirectoryMarkerRetention{policy='delete'}, instrumentation {S3AInstrumentation{}}}
Implementation class class org.apache.hadoop.fs.s3a.S3AFileSystem
2021-06-22 13:18:47,019 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(53)) - Starting: GetFileStatus s3a://stevel-usw2/
root entry S3AFileStatus{path=s3a://stevel-usw2/; isDirectory=true; modification_time=0; access_time=0; owner=stevel; group=stevel; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null
2021-06-22 13:18:47,026 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(100)) - GetFileStatus s3a://stevel-usw2/: duration 0:00:007
2021-06-22 13:18:47,026 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(53)) - Starting: First 25 entries of listStatus(s3a://stevel-usw2/)
2021-06-22 13:18:47,040 [s3a-transfer-stevel-usw2-unbounded-pool2-t1] DEBUG impl.LoggingAuditor (LoggingAuditor.java:beforeExecution(327)) - [22] 1cdd7da8-c418-40c5-ae44-b0398d1b98c5-00000006 Executing op_list_status with {object_list_request '' size=5000, mutating=false}; https://audit.example.org/hadoop/1/op_list_status/1cdd7da8-c418-40c5-ae44-b0398d1b98c5-00000006/?op=op_list_status&pr=stevel&ps=1ac8a7d7-8609-4698-811a-bf617bbe594b&cm=StoreDiag&id=1cdd7da8-c418-40c5-ae44-b0398d1b98c5-00000006&t0=1&fs=1cdd7da8-c418-40c5-ae44-b0398d1b98c5&t1=22&ts=1624364327026
s3a://stevel-usw2/ : scanned 0 entries
2021-06-22 13:18:48,830 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(100)) - First 25 entries of listStatus(s3a://stevel-usw2/): duration 0:01:804
2021-06-22 13:18:48,830 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(53)) - Starting: First 25 entries of listFiles(s3a://stevel-usw2/)
2021-06-22 13:18:48,835 [s3a-transfer-stevel-usw2-unbounded-pool2-t2] DEBUG impl.LoggingAuditor (LoggingAuditor.java:beforeExecution(327)) - [23] 1cdd7da8-c418-40c5-ae44-b0398d1b98c5-00000007 Executing op_list_files with {object_list_request '' size=5000, mutating=false}; https://audit.example.org/hadoop/1/op_list_files/1cdd7da8-c418-40c5-ae44-b0398d1b98c5-00000007/?op=op_list_files&pr=stevel&ps=1ac8a7d7-8609-4698-811a-bf617bbe594b&cm=StoreDiag&id=1cdd7da8-c418-40c5-ae44-b0398d1b98c5-00000007&t0=1&fs=1cdd7da8-c418-40c5-ae44-b0398d1b98c5&t1=23&ts=1624364328832
Files listing provided by: FunctionRemoteIterator{FileStatusListingIterator[Object listing iterator against s3a://stevel-usw2/; listing count 1; isTruncated=false; counters=((object_list_request=1) (object_list_request.failures=0) (object_continue_list_request.failures=0) (object_continue_list_request=0));
gauges=();
minimums=((object_list_request.failures.min=-1) (object_continue_list_request.min=-1) (object_list_request.min=185) (object_continue_list_request.failures.min=-1));
maximums=((object_list_request.failures.max=-1) (object_continue_list_request.failures.max=-1) (object_continue_list_request.max=-1) (object_list_request.max=185));
means=((object_list_request.mean=(samples=1, sum=185, mean=185.0000)) (object_list_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_continue_list_request.mean=(samples=0, sum=0, mean=0.0000)) (object_continue_list_request.failures.mean=(samples=0, sum=0, mean=0.0000)));
]}
2021-06-22 13:18:49,029 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(100)) - First 25 entries of listFiles(s3a://stevel-usw2/): duration 0:00:199

Security and Delegation Tokens
==============================

Security is disabled
Filesystem s3a://stevel-usw2 does not/is not configured to issue delegation tokens (at least while security is disabled)
2021-06-22 13:18:49,030 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(53)) - Starting: probe for a directory which does not yet exist s3a://stevel-usw2/dir-d2dc8c76-a357-418a-8bc3-6d45d256c7d9
2021-06-22 13:18:49,034 [main] DEBUG impl.LoggingAuditor (LoggingAuditor.java:beforeExecution(327)) - [1] 1cdd7da8-c418-40c5-ae44-b0398d1b98c5-00000008 Executing op_get_file_status with {action_http_head_request 'dir-d2dc8c76-a357-418a-8bc3-6d45d256c7d9' size=0, mutating=false}; https://audit.example.org/hadoop/1/op_get_file_status/1cdd7da8-c418-40c5-ae44-b0398d1b98c5-00000008/?op=op_get_file_status&p1=dir-d2dc8c76-a357-418a-8bc3-6d45d256c7d9&pr=stevel&ps=1ac8a7d7-8609-4698-811a-bf617bbe594b&cm=StoreDiag&id=1cdd7da8-c418-40c5-ae44-b0398d1b98c5-00000008&t0=1&fs=1cdd7da8-c418-40c5-ae44-b0398d1b98c5&t1=1&ts=1624364329030
2021-06-22 13:18:49,277 [main] DEBUG impl.LoggingAuditor (LoggingAuditor.java:beforeExecution(327)) - [1] 1cdd7da8-c418-40c5-ae44-b0398d1b98c5-00000008 Executing op_get_file_status with {object_list_request 'dir-d2dc8c76-a357-418a-8bc3-6d45d256c7d9/' size=2, mutating=false}; https://audit.example.org/hadoop/1/op_get_file_status/1cdd7da8-c418-40c5-ae44-b0398d1b98c5-00000008/?op=op_get_file_status&p1=dir-d2dc8c76-a357-418a-8bc3-6d45d256c7d9&pr=stevel&ps=1ac8a7d7-8609-4698-811a-bf617bbe594b&cm=StoreDiag&id=1cdd7da8-c418-40c5-ae44-b0398d1b98c5-00000008&t0=1&fs=1cdd7da8-c418-40c5-ae44-b0398d1b98c5&t1=1&ts=1624364329030
2021-06-22 13:18:49,465 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(100)) - probe for a directory which does not yet exist s3a://stevel-usw2/dir-d2dc8c76-a357-418a-8bc3-6d45d256c7d9: duration 0:00:436
Tests ar read only
JVM: memory=127129952
2021-06-22 13:18:49,474 [shutdown-hook-0] INFO  statistics.IOStatisticsLogging (IOStatisticsLogging.java:logIOStatisticsAtLevel(269)) - IOStatistics: counters=((action_http_head_request=1)
(audit_request_execution=4)
(audit_span_creation=5)
(object_list_request=3)
(object_metadata_request=1)
(op_get_file_status=2)
(op_get_file_status.failures=1)
(op_list_files=1)
(op_list_status=1)
(store_io_request=5));

gauges=();

minimums=((action_http_head_request.min=241)
(object_list_request.min=185)
(op_get_file_status.failures.min=435)
(op_get_file_status.min=3)
(op_list_files.min=189)
(op_list_status.min=1801));

maximums=((action_http_head_request.max=241)
(object_list_request.max=1787)
(op_get_file_status.failures.max=435)
(op_get_file_status.max=3)
(op_list_files.max=189)
(op_list_status.max=1801));

means=((action_http_head_request.mean=(samples=1, sum=241, mean=241.0000))
(object_list_request.mean=(samples=3, sum=2157, mean=719.0000))
(op_get_file_status.failures.mean=(samples=1, sum=435, mean=435.0000))
(op_get_file_status.mean=(samples=1, sum=3, mean=3.0000))
(op_list_files.mean=(samples=1, sum=189, mean=189.0000))
(op_list_status.mean=(samples=1, sum=1801, mean=1801.0000)))

No new tests here. The only way to test would be to destroy ~/.aws/config and & skip if the env vars/syssprops resolved or running in EC2. If we could set the resolve chain I'd have patched that to always return null...

Test run with -Dparallel-tests -DtestsThreadCount=7 -Dmarkers=keep -no failures!
rerunning with dynamo & scale

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 39s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
30m 37s

trunk passed

+1 :green_heart:
compile
0m 46s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
0m 40s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
0m 31s

trunk passed

+1 :green_heart:
mvnsite
0m 46s

trunk passed

+1 :green_heart:
javadoc
0m 27s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 34s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
1m 11s

trunk passed

+1 :green_heart:
shadedclient
14m  8s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 36s

the patch passed

+1 :green_heart:
compile
0m 38s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
0m 38s

the patch passed

+1 :green_heart:
compile
0m 32s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
0m 32s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 21s

the patch passed

+1 :green_heart:
mvnsite
0m 35s

the patch passed

+1 :green_heart:
javadoc
0m 16s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 25s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
1m  9s

the patch passed

+1 :green_heart:
shadedclient
14m  1s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m  2s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 33s

The patch does not generate ASF License warnings.

72m 26s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3133

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux e50f6390e4ad 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 474d2a9696ad2427a0425da1f3532ab88c56589b

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/1/testReport/

Max. process+thread count
714 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

tested with  -Dparallel-tests -DtestsThreadCount=7 -Dmarkers=keep -Dscale -Ds3guard -Ddynamo
transient read buffer underflow failure; one extra S3Guard write than expected. Both of those surface when there are too many records
[ERROR] Failures:
[ERROR]   ITestS3AContractUnbuffer>AbstractContractUnbufferTest.testUnbufferOnClosedFile:83->AbstractContractUnbufferTest.validateFullFileContents:132->AbstractContractUnbufferTest.validateFileContents:139->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 failed to read expected number of bytes from stream. This may be transient expected:<1024> but was:<515>
[ERROR]   ITestCommitOperations.testBulkCommitFiles:723->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Number of records written after commit #2; first commit had 4; first commit ancestors CommitContext{operationState=AncestorState{operation=Commitid=44; dest=s3a://stevel-london/fork-0007/test/DELAY_LISTING_ME/testBulkCommitFiles/out; size=6; paths={s3a://stevel-london/fork-0007/test/DELAY_LISTING_ME/testBulkCommitFiles/out s3a://stevel-london/fork-0007/test/DELAY_LISTING_ME/testBulkCommitFiles/out/file1 s3a://stevel-london/fork-0007 s3a://stevel-london/fork-0007/test s3a://stevel-london/fork-0007/test/DELAY_LISTING_ME/testBulkCommitFiles s3a://stevel-london/fork-0007/test/DELAY_LISTING_ME}}}; second commit ancestors: CommitContext{operationState=AncestorState{operation=Commitid=44; dest=s3a://stevel-london/fork-0007/test/DELAY_LISTING_ME/testBulkCommitFiles/out; size=8; paths={s3a://stevel-london/fork-0007/test/DELAY_LISTING_ME/testBulkCommitFiles/out s3a://stevel-london/fork-0007/test/DELAY_LISTING_ME/testBulkCommitFiles/out/file1 s3a://stevel-london/fork-0007 s3a://stevel-london/fork-0007/test/DELAY_LISTING_ME/testBulkCommitFiles/out/subdir s3a://stevel-london/fork-0007/test s3a://stevel-london/fork-0007/test/DELAY_LISTING_ME/testBulkCommitFiles s3a://stevel-london/fork-0007/test/DELAY_LISTING_ME/testBulkCommitFiles/out/subdir/file2 s3a://stevel-london/fork-0007/test/DELAY_LISTING_ME}}}: s3guard_metadatastore_record_writes expected:<2> but was:<3>
[INFO]

One thought here: would you ever want the s3a connector to fall back to that bundled region lookup sequence?
I'm wondering in particular if it makes a difference in routing/billing on EC2 deployments?
As of Hadoop 3.3.1 if region=null, endpoint=null the Ec2 metadata is used to provide the region info (this is new).
With this patch. if endpoint = null we switch to saying region = us-east-1
will that do bad things for signing/routing HTTP connections in an EC2 deployment in different regions?
For example, if I am running in AWS ireland, will this cause requests to go to us-east-1, even if they then end up redirected back to eu-west-1.
this could mean connections are slower to set up, risk of remote data transfer and billing (though the redirections should fix that, right?), and if the rules for a deployment prevent out-of-region network traffic, will this break. I think we have hit problems related to this in the past.
put differently: is anything special happening with the default  "null" endpoint and Ec2 metadata region name provision which we need to know about and support? If so, we could allow the region to be set to "" or maybe "ec2" and have that revert to the resolve chain

maybe I could test this by setting the sysprop aws.region to something invalid. If the region resolution is going through the chain then this would get picked up ahead of ~/.config or Ec2 and so, being invalid, fail somehow. And if we weren't using that chain, all would be good.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 34s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
markdownlint
0m  1s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
33m  6s

trunk passed

+1 :green_heart:
compile
0m 51s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
0m 37s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
0m 28s

trunk passed

+1 :green_heart:
mvnsite
0m 42s

trunk passed

+1 :green_heart:
javadoc
0m 26s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 34s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
1m 11s

trunk passed

+1 :green_heart:
shadedclient
14m 57s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 36s

the patch passed

+1 :green_heart:
compile
0m 37s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
0m 37s

the patch passed

+1 :green_heart:
compile
0m 32s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
0m 32s

the patch passed

-1 :x:
blanks
0m  0s
/blanks-eol.txt
The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

-0 :warning:
checkstyle
0m 21s
/results-checkstyle-hadoop-tools_hadoop-aws.txt
hadoop-tools/hadoop-aws: The patch generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2)

+1 :green_heart:
mvnsite
0m 36s

the patch passed

+1 :green_heart:
javadoc
0m 16s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 25s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
1m 10s

the patch passed

+1 :green_heart:
shadedclient
13m 59s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m  5s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 34s

The patch does not generate ASF License warnings.

75m 37s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3133

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint

uname
Linux 2e43df8d71e4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 88a6c0f100b372b3eda775c2b23c0e0bb0a29cf5

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/2/testReport/

Max. process+thread count
722 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Latest version does let you switch to the region resolution process if you really want to; this actually lets me do a test by setting sysprops to verify that the region is picked up that way.
Also the SDK exceptions are being converted to IOEs.
Tested s3 london, -Dparallel-tests -DtestsThreadCount=7 -Dmarkers=delete -Dscale; all good.
I just realised that I'd set the fs.s3a.endpoint property though; I'll have to rerun without any endpoint or region set for the test bucket.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 34s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
markdownlint
0m  1s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
30m 51s

trunk passed

+1 :green_heart:
compile
0m 46s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
0m 39s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
0m 31s

trunk passed

+1 :green_heart:
mvnsite
0m 46s

trunk passed

+1 :green_heart:
javadoc
0m 27s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 36s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
1m  9s

trunk passed

+1 :green_heart:
shadedclient
14m 22s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 36s

the patch passed

+1 :green_heart:
compile
0m 38s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
0m 38s

the patch passed

+1 :green_heart:
compile
0m 32s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
0m 32s

the patch passed

-1 :x:
blanks
0m  0s
/blanks-eol.txt
The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1 :green_heart:
checkstyle
0m 20s

the patch passed

+1 :green_heart:
mvnsite
0m 36s

the patch passed

+1 :green_heart:
javadoc
0m 16s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 25s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
1m 12s

the patch passed

+1 :green_heart:
shadedclient
13m 58s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m  4s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 35s

The patch does not generate ASF License warnings.

72m 57s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3133

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint

uname
Linux 994fb767e1cf 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / ac0a6104c97d948f004cac78bea24b8dba36b605

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/3/testReport/

Max. process+thread count
720 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Latest patch warns user on fallback, with the LogExactlyOnce class to stop it being over-noisy if someone really, really wants to use this "feature".
Also the latest stack trace is in, as well as the hadoop-3.3.1 one. I've also added the workaround info to the JIRA description as it'll probably be the first entry google will find for this

Tests in progress, s3 london, endpoint and region unset, -Dparallel-tests -DtestsThreadCount=7 -Dmarkers=keep

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 34s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
markdownlint
0m  1s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
30m 40s

trunk passed

+1 :green_heart:
compile
0m 47s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
0m 41s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
0m 31s

trunk passed

+1 :green_heart:
mvnsite
0m 46s

trunk passed

+1 :green_heart:
javadoc
0m 27s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 35s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
1m 10s

trunk passed

+1 :green_heart:
shadedclient
14m 12s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 37s

the patch passed

+1 :green_heart:
compile
0m 37s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
0m 37s

the patch passed

+1 :green_heart:
compile
0m 30s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
0m 30s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 20s

the patch passed

+1 :green_heart:
mvnsite
0m 35s

the patch passed

+1 :green_heart:
javadoc
0m 17s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 25s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
1m  9s

the patch passed

+1 :green_heart:
shadedclient
14m  7s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m  4s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 35s

The patch does not generate ASF License warnings.

72m 39s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3133

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint

uname
Linux 79ac4e35adbe 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 34525f1d7b429ff71f76c3d1ded5ea4441120239

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/4/testReport/

Max. process+thread count
546 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/4/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 34s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
markdownlint
0m  1s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
30m 10s

trunk passed

+1 :green_heart:
compile
0m 45s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
0m 37s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
0m 31s

trunk passed

+1 :green_heart:
mvnsite
0m 46s

trunk passed

+1 :green_heart:
javadoc
0m 23s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 32s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
1m 12s

trunk passed

+1 :green_heart:
shadedclient
14m 10s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 36s

the patch passed

+1 :green_heart:
compile
0m 38s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
0m 38s

the patch passed

+1 :green_heart:
compile
0m 31s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
0m 31s

the patch passed

-1 :x:
blanks
0m  0s
/blanks-eol.txt
The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1 :green_heart:
checkstyle
0m 21s

the patch passed

+1 :green_heart:
mvnsite
0m 38s

the patch passed

+1 :green_heart:
javadoc
0m 16s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 25s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
1m 12s

the patch passed

+1 :green_heart:
shadedclient
14m  4s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m  3s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 35s

The patch does not generate ASF License warnings.

71m 49s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3133

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint

uname
Linux cd5626ccf857 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 2cdceaa45ce4311a55d9af628c855b8dde314212

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/5/testReport/

Max. process+thread count
545 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/5/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 40s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
29m 35s

trunk passed

+1 :green_heart:
compile
0m 46s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
0m 40s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
0m 32s

trunk passed

+1 :green_heart:
mvnsite
0m 46s

trunk passed

+1 :green_heart:
javadoc
0m 27s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 36s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
1m 10s

trunk passed

+1 :green_heart:
shadedclient
14m 21s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 37s

the patch passed

+1 :green_heart:
compile
0m 38s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
0m 38s

the patch passed

+1 :green_heart:
compile
0m 31s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
0m 31s

the patch passed

-1 :x:
blanks
0m  0s
/blanks-eol.txt
The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1 :green_heart:
checkstyle
0m 21s

the patch passed

+1 :green_heart:
mvnsite
0m 35s

the patch passed

+1 :green_heart:
javadoc
0m 17s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 26s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
1m 10s

the patch passed

+1 :green_heart:
shadedclient
14m  0s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m  3s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 34s

The patch does not generate ASF License warnings.

71m 46s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/6/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3133

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint

uname
Linux 08fb3df3a830 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 6decb38ff0bce0d82ccc9357cd268a1f8767fc32

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/6/testReport/

Max. process+thread count
543 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/6/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 39s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
markdownlint
0m  1s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
29m 43s

trunk passed

+1 :green_heart:
compile
0m 46s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
0m 39s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
0m 32s

trunk passed

+1 :green_heart:
mvnsite
0m 46s

trunk passed

+1 :green_heart:
javadoc
0m 28s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 34s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
1m 12s

trunk passed

+1 :green_heart:
shadedclient
14m 13s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 35s

the patch passed

+1 :green_heart:
compile
0m 37s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
0m 37s

the patch passed

+1 :green_heart:
compile
0m 32s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
0m 32s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 20s

the patch passed

+1 :green_heart:
mvnsite
0m 35s

the patch passed

+1 :green_heart:
javadoc
0m 16s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 25s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
1m  9s

the patch passed

+1 :green_heart:
shadedclient
14m 13s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m  5s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 34s

The patch does not generate ASF License warnings.

71m 54s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/7/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3133

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint

uname
Linux 7a0c55ffbf3a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 993eea471208a0a65d7e3c12ca3c8813942d0883

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/7/testReport/

Max. process+thread count
656 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3133/7/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Test run in progress.  I  have commented the region field in ~/.aws/config
I see failures like
[ERROR] testCreateWithRenewer(org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationTokens)  Time elapsed: 0.827 s  <<< ERROR! com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region. 	at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:462) 	at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424) 	at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46) 	at org.apache.hadoop.fs.s3a.auth.delegation.SessionTokenBinding.maybeInitSTS(SessionTokenBinding.java:312) 	at org.apache.hadoop.fs.s3a.auth.delegation.SessionTokenBinding.prepareSTSClient(SessionTokenBinding.java:341) 	at org.apache.hadoop.fs.s3a.auth.delegation.SessionTokenBinding.createTokenIdentifier(SessionTokenBinding.java:362) 	at org.apache.hadoop.fs.s3a.auth.delegation.SessionTokenBinding.createTokenIdentifier(SessionTokenBinding.java:63) 	at org.apache.hadoop.fs.s3a.auth.delegation.AbstractDelegationTokenBinding.createDelegationToken(AbstractDelegationTokenBinding.java:142) 	at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.lambda$createDelegationToken$0(S3ADelegationTokens.java:435) 	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499) 	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444) 	at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.createDelegationToken(S3ADelegationTokens.java:433) 	at org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationTokens.testCreateWithRenewer(ITestSessionDelegationTokens.java:230)

After taking a glance. Not debugged in detail.
@steveloughran  I think STS tests still require the region to be set in ~/.aws/config as the stack trace is different.

The STS stuff has always needed it; look in testing.md and elsewhere for references to the option. Maybe the SDK provided the fallback if the default endpoint was used. That is not a regression, and all the docs discuss it.
I don't want to fix that here as its clearly not an issue in actual deployments; this patch can focus on the regression
Set up your region for testing and it will go away
 <property>
   <name>fs.s3a.assumed.role.sts.endpoint</name>
   <value>${sts.london.endpoint}</value>
 </property>
 <property>
   <name>fs.s3a.assumed.role.sts.endpoint.region</name>
   <value>${sts.london.region}</value>
 </property>

@mukund-thakur see HADOOP-16565 for the STS behaviour; no regressions there & that patch provides some diagnostics.

@steveloughran  Thanks. Tests run fine after setting the sts region and endpoints. So we are good.

thanks, merging!

merged into trunk; building and testing for 3.3


sap.m.Text misalignment in sap.m.Table

OpenUI5 version: 1.109.0
Browser/version (+device/version): Chrome 108.0.5359.124
Any other tested browsers/devices(OK/FAIL): Edge - FAIL.
URL (minimal example if possible): https://jsbin.com/nedubuyaxi/edit?html,output
User/password (if required and possible - do not post any confidential information here): -
Steps to reproduce the problem:

Open the jsbin link

What is the expected result? -- Label and Text are aligned inside the HBox of the ColumnListItem of the Table control.
What happens instead? -- sap.m.Text is misaligned
Any other information? (attach screenshot if possible)

This issue was introduced after the following commit: https://github.com/SAP/openui5/commit/0f9144a6dbf4873dc2f1d477af0819cce1d44e9e. If line-height: inherit; is removed, the text is aligned properly.
Is not reproducible in 1.108, only appears in 1.109.

Would you accept one of the workarounds as actual solutions?

Removing the outer HBox instead of nested: https://jsbin.com/yecatay/edit?html,output
Keeping the outer HBox but using sap.m.ObjectAttribute/.ObjectStatus instead of an inner HBox and a combination of sap.m.Label and sap.m.Text: https://jsbin.com/nodepih/edit?html,output

@boghyon, thanks, that indeed works! Actually, the real table is more complex and this bug/feature results in many misalignments across the different cells. Not sure if it will be fixed, but I decided to temporary patch a css with line-height: 1.5rem !important; for .sapMListTbl .sapMFlexItem .sapMText (I know, it's not a recommended way).

@whydrae Thanks for sharing your finding. I've created an internal incident<PHONE_NUMBER>. The status of the issue will be updated here in GitHub.


Optimizing images automatically

New feature: Optimizing images automatically.
I have seen most of the pull request fail because of large images. Topic images should not exceed 75000 bytes https://travis-ci.org/github/explore/builds/445250712, so why not to include ImgBot?
ImgBot will crawl through all the images files and losslessly compress them. This will make the file size go down, but leave the dimensions and quality untouched.
This should be great because the size of the repository will notoriously go down.
Example of this: https://github.com/abranhe/explore/pull/1/files

Reducing the file size would be good but we still require the images to be certain dimensions so I'm not convinced this would save much work. Is there anything that will also fix the dimensions?

Is there anything that will also fix the dimensions?

No, I don't think ImgBot would fix the dimensions, it will keep the dimensions untouched, but still there are many other PRs that are submitted with a correct image dimension (288x288) but still with a large size (over 7.5 kb).
I don't think ImgBot  would do anything else other than reduce a little bit the images size, so it's up to you guys if you want to integrate it to the repo.

Bình chọn khi có vấn đề gì thì làm lai cho chuẩn mực  , và lợi ích chung cho tất cả xã hội chúng ta đặt lên hàng đầu .

Thanks @abranhe! Passing for now.


soc: xilinx: zynq7000: add pinctrl driver + digilent zybo board

Add pinctrl driver for the Xilinx Zynq-7000 series SoC.
Add pinctrl support to the Xilinx Zynq-7000 PS UART driver.
Add board definition for the Digilent Zybo to get code coverage for the newly added pinctrl driver.

Hi Henrik,
just a few quick initial comments after just having received the notification and before leaving for the rest of the day:

Regarding the altered SLCR mapping in the SoC code: do you intend to permanently leave the SLCR unlocked, or did I miss on-demand locking in the pinctrl driver? As far as I remember, Xilinx encourages re-locking it once a write operation on the SLCR registers has been performed. Please note that the GEM driver accesses the SLCR in order to change the clock dividers upon link speed chage reported by the PHY, and once this has been done, the registers are re-locked. This already happens in the initial setup of the driver, so once the GEM is initialized, the registers will be locked.
Similarly to what I wrote in the CAN driver PR regarding a clock controller DT node or driver the other day, I don't really get the point of a pin controller driver for the Zynq or UltraScale. From my point of view, all the I/O and clock configuration (disregard special scenarios like the afforementioned GEM link speed change) comes from Vivado rather than the Zephyr config and thanks to the boot mechanism including the bitstream programming, Zynqs and Ultrascales can be considered fully set up with regards to I/O and clocks by the time Zepyhr boots - all that has been handled by the FSBL. Why should we meddle with the MIO pin assignments from within Zephyr afterwards?

Regarding the altered SLCR mapping in the SoC code: do you intend to permanently leave the SLCR unlocked, or did I miss on-demand locking in the pinctrl driver? As far as I remember, Xilinx encourages re-locking it once a write operation on the SLCR registers has been performed. Please note that the GEM driver accesses the SLCR in order to change the clock dividers upon link speed chage reported by the PHY, and once this has been done, the registers are re-locked. This already happens in the initial setup of the driver, so once the GEM is initialized, the registers will be locked.

Yes. The intent is to leave the SLCR permanently unlocked. I know that the registers allows for it to be locked again, but I have not seen anywhere that this is a recommended by Xilinx? Do you have a reference to this? What would the benefit be? Any code, that can write to the SLCRs, can just unlock them?
Leaving the SLCRs unlocked has a couple of benefits:

No need for locking (by semaphore or mutex) access to the SLCRs (if we use unlock -> modify -> lock, we need to guard against the thread having unlocked the SLCRs being pre-empted by another thread also unlocking-modifying-locking the SLCRs, causing the SLCRs to be locked when the first thread attempts to write to them). This scheme would require a custom syscon driver for the Zynq-7000 SLRCs for very little gain.
No locking means we can use the generic syscon driver for accessing the SLCRs

The Linux kernel also unlocks the Zynq-7000 SLCRs at boot time and leaves the unlocked: https://elixir.bootlin.com/linux/latest/source/arch/arm/mach-zynq/slcr.c#L220
I was not aware of the GEM driver accessing the SLCRs directly. This needs to be updated (but by using a clock control driver, not by accessing the SLCRs directly, see my next paragraph).

Similarly to what I wrote in the CAN driver PR regarding a clock controller DT node or driver the other day, I don't really get the point of a pin controller driver for the Zynq or UltraScale. From my point of view, all the I/O and clock configuration (disregard special scenarios like the afforementioned GEM link speed change) comes from Vivado rather than the Zephyr config and thanks to the boot mechanism including the bitstream programming, Zynqs and Ultrascales can be considered fully set up with regards to I/O and clocks by the time Zepyhr boots - all that has been handled by the FSBL. Why should we meddle with the MIO pin assignments from within Zephyr afterwards?

If your use-case is what you describe above, you probably do not need to alter the pinctrl settings from Zephyr. In that case, you can just leave CONFIG_PINCTRL=n and no Zephyr code will touch the pinctrl settings.
We have a different use-case. We have a modular hardware design:

The CPU board containing the Zynq-7000 is generic and programmed at the factory with a generic, minimalistic FSBL + U-Boot in QSPI.
At boot time, the FSBL applies a minimal configuration and passes control to U-Boot.
U-Boot probes the hardware and determines the type of base board(s) the CPU board is connected to, loads and boots the corresponding OS + software configuration from external media.
The OS then configures clocking, sets up pinctrl, enables peripherals, loads the bitstream corresponding to the hardware, etc.

For this use-case we need both pinctrl and clock control. Pinctrl is introduced here, next up is a proper clock control driver. The Linux kernel includes both a pinctrl and a clock control driver for the Zynq-7000 to meet these needs.
Other use-cases for pinctrl + clock control:

When using Zephyr on a generic Zynq-7000 development board using the standard U-Boot configuration for that board, a pinctrl + clock control driver would allow for the OS the configure the development board using shield definitions/devicetree overlays to accommodate to the various attached Pmods, shields, and other extensions.
Since the FSBL + U-Boot is often programmed into e.g. QSPI flash at the factory, they can be difficult to update over-the-air (at least without risking bricking the device due to a failed update). Having full pinctrl + clock control support within the OS allows "patching" wrong settings in the deployed FSBL from an OS update received OTA.

Yes. The intent is to leave the SLCR permanently unlocked. I know that the registers allows for it to be locked again, but I have not seen anywhere that this is a recommended by Xilinx? Do you have a reference to this?

Admittedly, it's not stated explicitly in the TRMs, for me, an obvious clue was that the PS7Init code re-locks the registers after each initialization step. The closest to an official statement regarding the issue is probably Xilinx's Answer Record 47570, which refers to an early version of the Boot ROM not re-locking the registers before exiting. The record states: "Leaving the SLCR registers unlocked exposes them to being accidentally overwritten". Although protection of the registers against rogue write accesses is the obvious purpose, it's admittedly more likely that those will try to write to NULL rather than somewhere beyond 0xF8000000. If leaving the unlocked helps against the need for an extra mutex, fine, let's go for that.

I was not aware of the GEM driver accessing the SLCRs directly. This needs to be updated (but by using a clock control driver, not by accessing the SLCRs directly, see my next paragraph).

That sounds like a good idea for unified and abstracted clock configuration access, and most of the peripherals contained in the Zynq/UltraScale are simple enough that a one-time config will do, but if you're already working on that driver or are about to start working on it, could you please consider the following for the GEM(s):

The clock controller driver should either have a means of obtaining the frequency of the PLL that drives the respective GEM and setting the pre-scalers without needing any knowledge about what the clock source is (effectively, allowing the modification of DIVISOR/DIVISOR1 in GEMx_CLK_CTRL without having to set SRCSEL and CLKACT as well) or something like a set_clock_frequency(hz) function which converts a frequency value into DIVISOR/DIVISOR1 values. This is required for TX clock re-configuration upon link speed change.
The driver or rather its DT integration should allow the specification of EMIO clock sources independent of the ususal ARM/DDR/IOPLL -> DIVISOR -> DIVISOR1 structure so that GEMx_CLK_CTRL.SRCSEL can be set to EMIO and DIVISOR/DIVISOR1 re-calculation still works. This of course applies to any peripheral which can be clocked via EMIO.
For the GEM, clocking information should be optional. When using the GEM in conjunction with the GMII-to-RGMII IP Core, clock frequency information is neither available nor required. Still, the clock source must be configurable to EMIO.

Also, there should be consideration for the fact that a Zynq/UltraScale based system may contain IP Core based peripherals whose clocking and I/O configurations are completely outside the scope of Zephyr, such as the AXI GPIO controller IP core. Therefore, with regards to this example, requiring pin control and/or clock control information for any GPIO controller might be too broad.

The OS then configures clocking, sets up pinctrl, enables peripherals, loads the bitstream corresponding to the hardware, etc.

Do you already have a bitstream loader module for Zephyr? I wrote one in C many years ago, IIRC that was for a Virtex FPGA. In case the bitstream interface still looks similar in the Zynq (block-wise loading, plus a few flags like done, CRC error etc.), I could have a look at whether this code can be ported or not in case you don't have a loader module yet.

The tests/kernel/mp/kernel.multiprocessing test case is failing with an undefined reference to arch_start_cpu. As far as I know, Zephyr does not yet support SMP on aarch32, but yet soc/arm/xilinx_zynq7000/xc7zxxx/Kconfig.defconfig.series sets CONFIG_MP_NUM_CPUS=2 (causing this test to be selected).
@microbuilder What would be the correct solution here? Set CONFIG_MP_NUM_CPUS=1 for the dual-core Zynq-7000 Cortex-A9 for now even though it has two cores?

@microbuilder What would be the correct solution here? Set CONFIG_MP_NUM_CPUS=1 for the dual-core Zynq-7000 Cortex-A9 for now even though it has two cores?

I need to look at the Zynq-7000 in a bit more detail, but that would probably be my suggestion as an interim solution.

@microbuilder What would be the correct solution here? Set CONFIG_MP_NUM_CPUS=1 for the dual-core Zynq-7000 Cortex-A9 for now even though it has two cores?

I need to look at the Zynq-7000 in a bit more detail, but that would probably be my suggestion as an interim solution.

@microbuilder Thanks. I have added a commit setting CONFIG_MP_NUM_CPUS=1 until we have aarch32 SMP support in the tree.

The failing compliance check is due to the added pinctrl_soc_pin_t typedef.


How to figure out that the promoted App being installed or not?

It seems that the PNTrackingManager only provides the way to track the "impression" of an Ad. And if the ad be clicked, it redirects the user to the App Store to show the App page. So, my question is, how the pubnative knows that the specific promoted App actually be installed on the user's device, and then track it with the "conversion"?
In other word, how the "conversion" be confirmed?

Hello ‘stoncle’
We do not track the conversion directly but when the user clicks to the store they activate the 'click url'
The click url sends information about the device to us. When the user installs and opens the application this information is sent to the game publisher and then to us by a 3rd party. We confirm both sets data match up. and then we record the conversion.

Hello @stoncle
The click sends information about the device to us. When the user installs and opens the application this information is sent to the game publisher and then to us by a 3rd party. We confirm both sets data match up. and then we record the conversion.

Hello ‘stoncle’
The click sends information about the device to us. When the user installs and opens the application this information is sent to the advertiser and then to us by a 3rd party. We confirm both sets data match up. and then we record the conversion.

Just to complete the answer,
Since the conversion is a server to server interaction between the advertiser app and our servers, impressions and clicks can only be confirmed by the publisher (aka, the user of the library), that's why impressions and clicks are handled in the PNTrackingManager and conversions are not.

So, is that means the target App has built-in codes to send device info to the advertiser?

Exactly

ok, thanks to your reply and your contribution to bring pubnative!


Split Build Files from Terraform Code

Hi,
I was wondering if there is a way to use this terraform module if the artifact files (usually in .next-tf) wasn't part of the terraform.
Ideally the files within .next-tf would be uploaded to a S3 project from a separate solution and then in our terraform code solution it can be referenced the bucket (and prefix key) path for there the files are.

Hi, this would probably work today.
You could create an initial deployment first and then uploading the static-files.zip & proxy-config.json manually to S3.
You simply would have to find out the bucket names of this two buckets (1 & 2):

The only thing is that the Terraform state and the proxy-config.json will be out of sync after the first manual deployment.
The static-files.zip is not tracked by Terraform (And is also deleted after successful deployment).

thanks this answers what I needed to know to deal with my issue :)


Remove blob batch

Pull Request Checklist
Please leverage this checklist as a reminder to address commonly occurring feedback when submitting a pull request to make sure your PR can be reviewed quickly:
See the detailed list in the contributing guide.

[x] C++ Guidelines
[x] Doxygen docs
[x] Unit tests
[x] No unwanted commits/changes
[x] Descriptive title/description

[x] PR is single purpose
[x] Related issue listed

[x] Comments in source
[x] No typos
[x] Update changelog
[x] Not work-in-progress
[x] External references or docs updated
[x] Self review of PR done
[x] Any breaking changes?

/azp run cpp - storage

/azp run cpp - storage

What about these things:
https://github.com/Azure/azure-sdk-for-cpp/blob/c3484c5d4daa38257c633f74cf86e44ed1bd68fc/sdk/storage/azure-storage-blobs/inc/azure/storage/blobs/blob_service_client.hpp#L188-L190
https://github.com/Azure/azure-sdk-for-cpp/blob/0643eeba5de821c0585223fdb160a3cf1a8fb1b0/sdk/storage/azure-storage-blobs/inc/azure/storage/blobs/protocol/blob_rest_client.hpp#L655-L658
Please go through and make sure the removal of blob batch is complete.

What about these things:
https://github.com/Azure/azure-sdk-for-cpp/blob/c3484c5d4daa38257c633f74cf86e44ed1bd68fc/sdk/storage/azure-storage-blobs/inc/azure/storage/blobs/blob_service_client.hpp#L188-L190
https://github.com/Azure/azure-sdk-for-cpp/blob/0643eeba5de821c0585223fdb160a3cf1a8fb1b0/sdk/storage/azure-storage-blobs/inc/azure/storage/blobs/protocol/blob_rest_client.hpp#L655-L658
Please go through and make sure the removal of blob batch is complete.

@Jinming-Hu you marked these changes for the February milestone, is it meant for the next release rather than the upcoming Jan release?

@Jinming-Hu you marked these changes for the February milestone, is it meant for the next release rather than the upcoming Jan release?

@Jinming-Hu you marked these changes for the February milestone, is it meant for the next release rather than the upcoming Jan release?

Nah, it's a mistake.

@Jinming-Hu you marked these changes for the February milestone, is it meant for the next release rather than the upcoming Jan release?

Nah, it's a mistake.

What about these things:
https://github.com/Azure/azure-sdk-for-cpp/blob/c3484c5d4daa38257c633f74cf86e44ed1bd68fc/sdk/storage/azure-storage-blobs/inc/azure/storage/blobs/blob_service_client.hpp#L188-L190
https://github.com/Azure/azure-sdk-for-cpp/blob/0643eeba5de821c0585223fdb160a3cf1a8fb1b0/sdk/storage/azure-storage-blobs/inc/azure/storage/blobs/protocol/blob_rest_client.hpp#L655-L658
I don't see any uses or calls to those types. How come it is being built successfully?
Please go through and make sure the removal of blob batch is complete.

Yeah, some of them were left out. I'm gonna fix it.

What about these things:
https://github.com/Azure/azure-sdk-for-cpp/blob/c3484c5d4daa38257c633f74cf86e44ed1bd68fc/sdk/storage/azure-storage-blobs/inc/azure/storage/blobs/blob_service_client.hpp#L188-L190
https://github.com/Azure/azure-sdk-for-cpp/blob/0643eeba5de821c0585223fdb160a3cf1a8fb1b0/sdk/storage/azure-storage-blobs/inc/azure/storage/blobs/protocol/blob_rest_client.hpp#L655-L658
I don't see any uses or calls to those types. How come it is being built successfully?
Please go through and make sure the removal of blob batch is complete.

Yeah, some of them were left out. I'm gonna fix it.

Nah, it's a mistake.

Yeah, some of them were left out. I'm gonna fix it.

Sounds good 👍
Let's prioritize getting this change in for the current release quickly :)

Nah, it's a mistake.

Yeah, some of them were left out. I'm gonna fix it.

Sounds good 👍
Let's prioritize getting this change in for the current release quickly :)


AB test copy-on-write in pandas

I'm curious if we'll see performance improvements here
xref https://github.com/dask/dask/issues/10218
cc @phofl

We talked about this offline a bit. The theory so far is that we could possibly see substantial improvements since some keys may now actually share memory since some keys would just be views instead of deepcopies.
However a concern was raised that this would also mess with distributed instrumentation about memory, particularly managed vs unmanaged memory. This would not be a dealbreaker of course.

However a concern was raised that this would also mess with distributed instrumentation about memory, particularly managed vs unmanaged memory

Just to clarify, the concern here is that distributed may overestimate things because sizeof isn't aware of views vs. non-views?

However a concern was raised that this would also mess with distributed instrumentation about memory, particularly managed vs unmanaged memory

Just to clarify, the concern here is that distributed may overestimate things because sizeof isn't aware of views vs. non-views?

Yes, I think we'll need some variant of sizeof that returns a dict of {buffer id: nbytes} and a change in SpillBuffer to match

Let's keep the discussion on pitfalls etc. on https://github.com/dask/dask/issues/10218

Here is a link to the benchmarks: https://github.com/coiled/benchmarks/actions/runs/4809354163
Results are a bit mixed, but A/B tests don't cover most of the methods where CoW would make a difference

FWY I typically run at least 5, better if 7, runs in A/B. 3 aren't enough to get any statistically significant measure and can easily lead to incorrect conclusions.

Thx, will rerun with a higher repeat setting

In the above link the dashboard is lacking the confidence report due to a bug that's been resolved in main.
I've generated it locally.
Note how the very impressive speedup on the new test test_dataframe_cow_chain is accompanied by an equally extreme increase in memory usage.

The memory increase is weird, but haven’t looked deeper. The chain is zero copy with cow in pandas…


Display row number

Is there a way to display the row numbers? One of my tables is ranking assets and it would be very helpful for the users to see if an asset is ranked, for example, 28 or 12. Thanks!

See the customBodyRender/customBodyRenderLite methods. customBodyRender has a rowIndex on the tableMeta object and customBodyRenderLite takes in a rowIndex parameter. You can use this value for the display.

Thanks. For future searches here's the code I used. I added a column as the first one:
    {
        name: '',
        label: '',
        options: {filter: false,
            customBodyRender : (value, tableMeta, update) => {
                let rowIndex = Number(tableMeta.rowIndex) + 1;
                return ( <span>{rowIndex}</span> )
            }
        },
    },


WIP: Add support for the RPL routing protocol

This adds the RPL routing protocol from RFC6550. This routing protocol is designed for Lossy Low-power Networks.

[x] Add wire representation of the RPL headers and the options (section 6).

[x] Add field getters
[x] Add field setters
[x] Add parse functions
[x] Add header_len functions
[x] Add emit functions
[x] Add tests!

[x] Update the support of extension headers in 6LoWPAN (currently only for Hop By Hop, which is needed in RPL).

Currently, smoltcp has support for compressed UDP, uncompressed IPv6 headers, but no support for compressed IPv6 extension headers. These headers are mostly carried unmodified in the bytes immediately following the LOWPAN_NHC octet, with two important exceptions: Length field and Next Header field RFC6282 § 4.2. I saw Contiki-OS using this for their RPL protocol where they used a compressed Hop-by-Hop extension header.

[x] Add the Trickle timer mechanism from RFC6202.

The Trickle algorithm allows nodes in low-power and lossy networks to exchange information in a highly robust, energy efficient, simple, and scalable manner. Dynamically adjusting transmission windows allows Trickle to spread new information on the scale of link-layer transmission times while sending only a few messages per hour when information does not change. A simple suppression mechanism and transmission point selection allow Trickle's communication rate to scale logarithmically with density.

[ ] Add Objective Functions:

The Objective Function (OF) defines how RPL nodes select and optimize routes within a RPL Instance. The following OFs are supported by Contiki-OS:
[x] Objective Function Zero RFC6552
[ ] Minimum Rank with Hysteresis Objective Function RFC6719 (Not sure if this will be added in this PR)

[x] Support for Mode of Operation 0: no downward routes maintained.
[ ] Support for Mode of Operation 1: upward routes, without storing mode
[ ] Support for Mode of Operation 2: upward routes, with storing mode (no multicast)
[ ] Support for Mode of Operation 3: upward routes, with storing mode (multicast)

Closing this PR since I'm breaking this down into multiple PRs. I have a tracking issue for RPL #769 .


Correct modin samples

Existing Sample Changes
Description
This PR corrects Modin related samples. Changes are next:

Change config setting technique for more appropriate for python file.
Remove compression parameter from read_csv function (csv file compression is inferred automatically and explicit notion of compression parameter causes "default-to-pandas" behavior and performance degradation).
Remove nrows parameter from read_csv function to avoid "default-to-pandas" behavior (nrows parameter is not supported yet in Modin with OmniSci backend).
DataFrame.query function calls replaced with equivalents to avoid "default-to-pandas" behavior and avoid performance degradation (DataFrame.query is not supported yet in Modin with OmniSci backend).
Alternative installation guidance was fixed, sample.json file was corrected accordingly (similarly to https://github.com/oneapi-src/oneAPI-samples/pull/742).
Warnings were disabled to avoid unneeded output cluttering.
Fixed minor typos.

External Dependencies
List any external dependencies created as a result of this change.
Type of change
Please delete options that are not relevant. Add a 'X' to the one that is applicable.

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Implement fixes for ONSAM Jiras

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[x] Command Line
[ ] oneapi-cli
[ ] Visual Studio
[ ] Eclipse IDE
[ ] VSCode
[ ] When compiling the compliler flag "-Wall -Wformat-security -Werror=format-security" was used

Hi @JoeOster ! Could you take a look on this and on the https://github.com/oneapi-src/oneAPI-samples/pull/769 PRs?

Please submit to development branch but not the master branch

Please submit to development branch but not the master branch

@praveenkk123 target branch was changed.

@louie-tsai  @praveenkk123  please approve for 2022.1 since the change for census is important (after README edit i requested as conda install command currently listed is not latest)

@amyskov can you also ensure that the conda install commands are correct for the README of the sample? conda install modin should be good i believe .
https://github.com/oneapi-src/oneAPI-samples/blob/master/AI-and-Analytics/Getting-Started-Samples/IntelModin_GettingStarted/README.md

@raoberman verified conda install command was added.

Additional changes:

nrows parameter was removed from read_csv function to avoid "default-to-pandas" behavior (nrows parameter is not supported yet in Modin with OmniSci backend).
Warnings were disabled to avoid unneeded output cluttering.

@amyskov please re-submit this PR to the master branch. See Praveen and I's email for info

@amyskov please re-submit this PR to the master branch. See Praveen and I's email for info

@raoberman PR was re-summited and rebased.

Modin configurations were changed to avoid triggering https://github.com/modin-project/modin/issues/4017 issue. Also installation instructions were slightly modified to speed-up packages installation.

Hi @praveenkk123!
For some reason Samples validation (Linux) test failed with strange output, while if i run steps commands from sample.json file tests passes (as i understood this file contains CI settings). Can you please provide me with instructions to reproduce CI fail? Or maybe you can give me some suggestions how i can tune CI to pass tests?

@amyskov please rebase on the current master, we recently merged #863 that should fix CI testing

@praveenkk123 Census sample CI config was changed as well, but jupyter nbconvert was used instead of runipy because of hardcoded timeout value in runipy (see https://github.com/jupyter/jupyter_client/issues/197 for details), that raised error during long IO step execution. Can you, please, restart the CI?

@praveenkk123 merge conflict was fixed, could you please restart the CI?

All tests passed. Can I merge this?

All tests passed. Can I merge this?

Yes, sure.


⚠ Bofrost.de has degraded performance

In d6202d6, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1540 ms

Resolved: Bofrost.de performance has improved in d9502cb.


No More Changes?

Hi, i'm probably going to work on the plugin when i will have more time because for the moment i have school. So there should be new updates soon 😄 !

i would look to Update you repo with permissions with my codes? You accept Pule requests?

Yes absolutly!

I have made 2 Pull Requests :)

I'm looking them 😄

Merged! Thanks for the pull request 😛

Can add me, so i can push without pullrequests? :-)

I'm sorry but i prefer reviewing the pull requests before merging them 😄


[FEATURE REQ] Set content-type when using Azure Storage Resource

Is your feature request related to a problem? Please describe.
When I use the azure-spring-boot-starter-storage and upload file/blob, it's not possible to set the content type. For example, when I try to upload a PDF file and to preview it using the blob/file URL, it's not preview-able.
Describe the solution you'd like
When uploading the blob/file, it's possible for me to specify, or the program can infer the file's content type via its suffix.
Describe alternatives you've considered
None
Additional context
None
Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[x] Description Added
[x] Expected solution specified

Closing this in #25253.


[freexl,libspatialite,spatialite-tools] Update, [minizip] Enable DLL, [libkml] Update minizip import

Touching multiple ports in one PR:
Updates freexl, libspatialite, spatialite-tools which are all from https://www.gaia-gis.it/gaia-sins/.
freexl 2.0.0 is needed for updating libspatialite to 5.1.0.
minizip is now used by freexl, and static linkage became a limitation with much wider effect than before. The DEF file generation was done (almost) as in port nettle. There aren't many updates to this part of zlib, so the maintenance debt should be acceptable.
The tiny libkml change was an outstanding issue from a past minizip change, now bundled to the rebuild which is triggered by the minizip modificiation.

[x] Changes comply with the maintainer guide
[x] SHA512s are updated for each updated download
[x] The "supports" clause reflects platforms that may be fixed by this new version
[x] Any fixed CI baseline entries are removed from that file.
[x] Any patches that are no longer applied are deleted from the port's directory.
[x] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[x] Only one version is added to each modified port's versions file.

All features passed with following triplets:
x86-windows
x64-windows
x64-windows-static

Usage test passed on x64-windows.


Create design for username/password authentication on Android

This is to be used both for Yinbi wallet as well as for Messenger. For Messenger, it should allow the user to store their identity key and contact list on our servers for later recovery. It should also allow users add contacts using their username.

cc: @atavism


Implement email/password login for API server

New developers don't have a way to sign into their local API server without getting We Vote Twitter or Facebook auth information. (This stopped a developer last night at Code for SF!) We would like a new developer to be able to create an account (with email address + password), and then go into the database to activate that account so they can sign into their own local WeVoteServer.

Verified!


Sinistea Wiki : biome not working

Command needs to be entirely rewritten, this plugin is mad jank

Closing, as we no longer use this plugin!


fix(docs): broken link for community contribution lifecycle and processes

Which problem is this PR solving?
The current link is broken https://github.com/open-telemetry/community/blob/main/CONTRIBUTING.md
Fixes # (issue)
Short description of the changes
Fixes the broken link for contribution processes which were moved in https://github.com/open-telemetry/community/pull/2051
Type of change
Please delete options that are not relevant.

[x] This change is a documentation update

How Has This Been Tested?

[x] View modified page https://github.com/trivikr/opentelemetry-js/blob/5c851779de309ea4b4add7214f40f2a388fa544e/CONTRIBUTING.md

Checklist:

[x] Followed the style guidelines of this project
[x] Documentation has been updated

The changelog CI failed

Please add a CHANGELOG entry, or add the "Skip Changelog" label if not required.

This is a minor change, for which CHANGELOG entry can be skipped and Skip changelog label can be added.

@trentm I've updated the links in current repo.
The opentelemetry-proto is a submodule, and it was fixed there in https://github.com/open-telemetry/opentelemetry-proto/pull/573

@trentm I've updated the links in current repo.
The opentelemetry-proto is a submodule, and it was fixed there in open-telemetry/opentelemetry-proto#573 It'll be fixed in this repo when the submodule is updated.

Thanks, I created an issue to track this - the submodule is quite outdated (#5208)


Update bower_components to vendor and self_service to ui/service paths

Unprocessed service dependencies are now built into a folder vendor this was previously called bower_components.
Updated rewrite rule to point to correct folder (self_service no longer exists)
👀 of @himdel @Fryguy @simaishi and all 👨👩👧👦  welcome! (but required from former 😉 )
Links

https://github.com/ManageIQ/manageiq-ui-service/pull/296#discussion_r85765514

LGTM.. @simaishi Please merge if you are good with these changes.

LGTM too, though I guess we should either merge https://github.com/ManageIQ/manageiq/pull/12240 first, or close that one in favor of this one.. (same change except for the bower_components -> vendor bit)


Add Tensor.roll

https://pytorch.org/docs/stable/generated/torch.roll.html

closing for now, we want to match torch and prioritize the commonly used API


Operator doesn't handle errors well

What happens?
During the testing of the new examples, a couple of problems were observed:

The operator sometimes gets stuck and doesn't start new jobs if the previous didn't complete successfully. A restart is required.
When there are errors with the repo (no access, wrong branch, missing files, missing template image) and the config doesn't get applied, the GitOpsConfig CRs often can't be deleted afterwards. The Finalizer has to be disabled to remove it.
Behavior affects other namespaces also.

Not sure if these are related. I'll keep documenting more behaviors as I do more testing.
What were you expecting to happen?
The Operator gracefully handles any issues, doesn't get "stuck" , and the CR can always be deleted without having to "patch" the resource.
Steps to reproduce:

Use the examples and modify one of the CRs to not being able to pull the repo. Leave the trigger as change.
Apply the CR and let it spin.
Try deleting the CR
Try applying an other CR in a different namespace

Here's some sample code for the finalizer: https://github.com/hossainemruz/k8s-initializer-finalizer-practice

With regards to "errors with the repo": I think fixing this part of the above issue is not sustainable with the current approach, which basically requires the .yaml files to be fetched correctly in order to be able to delete them, i.e. to even know what to delete:
https://github.com/KohlsTechnology/eunomia/blob/12c2afb630a77f36ebdd8f6dee1a3ea90fc93618/template-processors/base/bin/resourceManager.sh#L32-L36
So, when the git repo can't even be fetched, the current approach just doesn't have anything to grab onto.
My suggestion is to revisit this after #24. I believe adding labels to any pods created with eunomia should allow us to just confidently purge anything even remotely connected to a particular label when performing a delete. This would allow us to ignore any problems with git, or whatever else.
This might be somewhat tricky with regards to ResourceHandlingMode = "None" and ResourceDeletionMode = "None". However, I would suggest that we introduce a completely separate path for handling delete in template-processors, that would not require going through gitPull.sh etc. at all. Any "special" templating engine, which requires Mode = "None", would then have to override a separate "deleteResources.sh" script too.
@seanmalloy @Smiley73 what do you think? Does it make sense to you, or did I miss something? I'm still only learning eunomia...

@akavel i'm find with waiting on this fix until #24 is implemented.

Tried to reproduce it using hello-world-yaml example, but came across another issues as described in #202

Further trying to reproduce the original issue I managed to avoid #202 but came across another one: #206.

One idea is to use Semantic Resource Validation as described in Kubernetes Operators Best Practice in order not to let a fishy CR get to the API server.


fix full light data is not sent

Fixed an issue where full light data was not sent when re-logging in.

Before

After

Is there a reason to store a cached full light packet in dynamic chunk? Looks like the only place it's used is in the chunk packet, where it will be cached

Is there a reason to store a cached full light packet in dynamic chunk? Looks like the only place it's used is in the chunk packet, where it will be cached

You're right, I don't think that's necessary.
I'll fix it.

Fixed