Bridged network seems not to work with systemd 235

Required information

Distribution: archlinux
Distribution version: -
The output of "lxc info":

config:
  core.https_address: '[::]:9000'
  core.trust_password: true
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
api_status: stable
api_version: "1.0"
auth: trusted
auth_methods:
- tls
public: false
environment:
  addresses:
  - <IP_ADDRESS>:9000
  - <IP_ADDRESS>:9000
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIIFfTCCA2WgAwIBAgIRAKKJ2Ons1VgTSOKHl/ydIvowDQYJKoZIhvcNAQELBQAw
    OjEcMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEaMBgGA1UEAwwRcm9vdEB6
    ZXVzLmltcGwuaXQwHhcNMTYwOTE3MjEyMzMwWhcNMjYwOTE1MjEyMzMwWjA6MRww
    GgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRowGAYDVQQDDBFyb290QHpldXMu
    aW1wbC5pdDCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAOsdrPNjK+mC
    N1DPx7IARLii7mKH4OPgllqPtb/Jb6QXamfj/uE18hVucDduModCWt+BkzvMvSG8
    vt9JlfgVeQ8qCvyhbyfNC4eJIULN/q1PUDX4bTaE/uJbrkc+e2nXdanOfJfe0yns
    EZ1QlVHfcj4cLqwDV6WW2bdq0cCPCVb6s2zR52oHjzYxUXJ+n5C/kGJErlw856XB
    8FGkdF7XwwZkwjN8o45/TLd6pFsO881d4q/UgYCXWaLpxJqUlP1iUjXx6hrHPhOE
    QLfdF/Epr6G764ynRJ3I5h3M5W+MTlMkCc75Ih4TLxaZoyjHmzmAn/cEr4hygpi5
    Z9PF6vmDsJqFxqSsDs49vK3RXtsuY1GbKOcZgc1rjVEUE6q+y65Yo3leqSS83eYN
    Tc326Sx3Y/F2eUfeFwF328+5fLRD2Vdey3G+K3FemUoie1BGQVoz1BaTuV5ipHOQ
    vKVxP97C/RhmEB5OCPMzDz1xj0KgJ/FOAglrsobcQhE395hOPkrrr0R99eNwGVTK
    pY22DSpDWAmTz5YDPxGsQHaAvzLFldqfhaGTqXsF8x4aUFcqvUPr2g35xKw6FqQK
    m68t3yxGuz0d6VkFRkhZvBhOOLBHqtrNUWcwaTO7bADWGkJYWC/NMFw2S9stNsob
    Hd+kkeYM9Iui3zWISal1IfTmyqtC8QpRAgMBAAGjfjB8MA4GA1UdDwEB/wQEAwIF
    oDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMEcGA1UdEQRAMD6C
    DHpldXMuaW1wbC5pdIIQNjIuMTQxLjQ4LjE4OC8xM4IcZmU4MDo6MWM2ZTozN2Zm
    OmZlMzE6MWVlZi82NDANBgkqhkiG9w0BAQsFAAOCAgEA2JCnP2sNPNF9v4LZE5la
    a+sDlwi5Uv+7B2EFqNvgDU08ofUJZmNVMneYBX0mZC2g8NkNiBFfPdN2VM1VTlKz
    zJjTKifWxSq5nrz0bIfo4Sxw8Xjz4FiKGyhDw/Tja9c+rRfDh+bGb2v6yKGCAThZ
    NwFJ42cBtwDzLsXrm9BNgRiFuSV3lnOa8gtheJsMuc+9uttbN9E+Qv8yVI9uw1RZ
    fo65+KU49YFGeEe4EpCF3YSmH1f2Mf8enmFmovmVHGDrfvnst0TvQvNqZS6t3Sa2
    hMRAsAzb0FAd4i5oW59RapXk7QfcHwt8gmpAm7FG88NMylBxERA6Bn0+CfGpluE2
    gWQCR0ZED0FdxPMHMGvHGHv2PF4Ooim0fwSK6vIpxPBpb2+RjtmwJlFYoaoiAhZJ
    UM8A15VWpD2v6CaNVaOF5v85I0q8kSoCPEAcXYBfmoGb3KBP7EUcjgg5V3/qyGME
    sAuwScWcJmfFdhDuMbJGTQU5ORemhd8gFfEo3VaV1h4JJWozGgcEXWR+4gwMozJu
    MQIeX5FDv8gzlR97H2GwuN2mWBNriJFUEw751Xtb9L1DBWY+kjhglnV0UAtQFA6X
    vNsKIqcQHhZ2QQus56Mu5TmjErhZ6bq1bxqpMr4NDwfj7w3KPMAMk9eSwsIIeLQA
    lBonK0AV5xqH1cFm+QTMeQ8=
    -----END CERTIFICATE-----
  certificate_fingerprint: 9b4865d46819654f6ffcecdb9f9b78bbeb514067c0458d091847e39073596eda
  driver: lxc
  driver_version: 2.1.0
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.12.8-2-userns
  server: lxd
  server_pid: 230
  server_version: "2.17"
  storage: btrfs
  storage_version: "4.13"

Issue description
The containers systemd-networkd can not start:
Nov 01 20:57:10 arch-systemd systemd[1]: systemd-networkd.service: Service has no hold-off time, scheduling restart.
Nov 01 20:57:10 arch-systemd systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
Nov 01 20:57:10 arch-systemd systemd[1]: Stopped Network Service.
Nov 01 20:57:10 arch-systemd systemd[1]: systemd-networkd.service: Start request repeated too quickly.
Nov 01 20:57:10 arch-systemd systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
Nov 01 20:57:10 arch-systemd systemd[1]: Failed to start Network Service.

Steps to reproduce

lxc launch images:archlinux arch-systemd
lxc exec arch-systemd -- journalctl -u systemd-networkd

Information to attach

[ ] Any relevant kernel output (dmesg)
[x] Container log (lxc info NAME --show-log)

lxc info arch-systemd --show-log
Name: arch-systemd
Remote: unix://
Architecture: x86_64
Created: 2017/11/01 20:57 UTC
Status: Running
Type: persistent
Profiles: default
Pid: 27404
Ips:
  eth0:	inet6	fe80::216:3eff:fe67:84d3	vethHJFFVN
  lo:	inet	<IP_ADDRESS>
  lo:	inet6	::1
Resources:
  Processes: 12
  CPU usage:
    CPU usage (in seconds): 1
  Memory usage:
    Memory (current): 125.38MB
    Memory (peak): 129.30MB
  Network usage:
    lo:
      Bytes received: 0B
      Bytes sent: 0B
      Packets received: 0
      Packets sent: 0
    eth0:
      Bytes received: 3.96kB
      Bytes sent: 866B
      Packets received: 40
      Packets sent: 11

Log:

            lxc<PHONE_NUMBER>5709.645 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc<PHONE_NUMBER>5709.645 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc<PHONE_NUMBER>5709.765 WARN     lxc_cgfsng - cgroups/cgfsng.c:chown_cgroup_wrapper:1492 - Error chmoding /sys/fs/cgroup/unified//lxc/arch-systemd: No such file or directory
            lxc<PHONE_NUMBER>5709.779 ERROR    lxc_network - network.c:lxc_setup_network_in_child_namespaces:2974 - WARNING: using "lxc.network.*" keys to define networks is DEPRECATED, please switch to using "lxc.net.[i].* keys"
            lxc<PHONE_NUMBER>5709.818 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc<PHONE_NUMBER>5709.818 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.

[x] Container configuration (lxc config show NAME --expanded)

architecture: x86_64
config:
  image.architecture: amd64
  image.description: Archlinux current amd64 (20171101_01:27)
  image.os: Archlinux
  image.release: current
  image.serial: "20171101_01:27"
  volatile.base_image: 8e3ed7df523661657f523629fa98239d3cf3d9c794207869d3b91080a4e0f85f
  volatile.eth0.hwaddr: 00:16:3e:67:84:d3
  volatile.idmap.base: "0"
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.power: RUNNING
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

[ ] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
[ ] Output of the client with --debug
[ ] Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)

This looks like another keyring issue and is related to systemd. Can you try setting:
lxc profile set default security.syscalls.blacklist "keyctl errno 38"

and see if that fixes the container when restarting?

$ lxc profile get default security.syscalls.blacklist
keyctl errno 38

I restarted the container. But still no ip. And systemd-networkd states the same error message.

There are a few potential issues here:

Using anything but current git master, security.syscalls.blacklist is broken...
Even with that fix, if you're using the LXD snap package, then snapd effectively disables our ability to generate custom seccomp profiles.

@stgraber , is there a specific commit for the security.syscalls.blacklist fix?
(Just FI: I don't use the snap package)

5b33b998a8894611ec24fab72f5f7c64d27ebffe

@brauner @stgraber You probably are already aware, but 235 will be the version come 18.04 bionic, and I know how Stephane loves his LTS versions.... :) happy happy joy joy
ver 235 does however add some nice new features, including support within systemd-networkd for route tables and VRFs

@brauner , @stgraber , I applied the commit to my running version and in combination with security.syscalls.blacklist  I get an ip again.

Ubuntu's systemd is or will be patched to behave properly when keyctl fails with a permission issue.
@xnox attempted to upstream that particular fix but upstream systemd wasn't interested...

We do not intend to have LXD block that syscall by default as there is ongoing work to have keyctl actually work from inside the container. The right thing to do is for a caller to keyctl to look at the return value they get from the kernel and act accordingly...

In ubuntu v235 is in proposed, but currently fails its tests on all architectures. The keyctl permission fails has been reworked again, and is in progress merged upstream...

@xnox upstream as in systemd?

Closing this LXD issue as we fixed the seccomp issue in master.
As mentioned above, the proper fix for this should be in systemd/kernel to either allow unprivileged keyctl operations or to properly deal with the kernel refusing those operations inside a user namespace.
There seems to be some progress on both of those options and we'll certainly be keeping an eye on this.

So the fix for this will be applied to the LXD host machine? Or is there something I will need to do inside the container for the fix to work properly?

Right, you'll want to modify your default profile on the host with the security.syscalls.blacklist config entry above.


My Settings does not show up on latest version of October CMS

I have update my October CMS to April 13 cf1988e . My Settings is not visible on the left sidebar

Ok old instructions, please update the documentation to let us know that it is already located here Wysiwyg Editors:

Now settings for editor is back in sidebar in CMS settings.


don't push the stashes if there isn't any

If you did a git fire in a repo without any stash you would get this error:
fatal: ambiguous argument 'stash': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

Thanks for fixing this, @bricewge!


Vmware Import Problem

ISSUE TYPE

Bug Report

COMPONENT NAME

Vmware

CLOUDSTACK VERSION

4.19

SUMMARY

Storage is not seen while importing [ <IP_ADDRESS> Working.]

@omurozlu Is that storage added as primary storage in CloudStack, cluster-wide or zone-wide?

hello @sureshanaparti,
I tried it both ways. (zone + cluster)

Hi @omurozlu I did quick code check, seems issue with datastore path comparison here
https://github.com/apache/cloudstack/blob/45d267ccbf2749c547cbbbac4a2cb1f3351dcaf2/server/src/main/java/org/apache/cloudstack/vm/UnmanagedVMsManagerImpl.java#L554  (updated in https://github.com/apache/cloudstack/pull/8445)

hello @sureshanaparti,
im check api post process.
Do I need a manual process to do this?

hello @sureshanaparti, im check api post process. Do I need a manual process to do this?

@omurozlu Manual process does the same comparison, while importing root and data disks of the instance. Is root disk of the instance, exists on that storage?

Hi @omurozlu - is the storage pool a datastore cluster type? Can you upload the management server logs for the import operation?

hello @sureshanaparti,
same zone added

hello @nvazquez ,
datastore cluster type VMFS
management server log this :
2024-02-28 15:48:59,347 DEBUG [c.c.a.ApiServer] (qtp1278254413-352:ctx-5219e9e4 ctx-3aeefedf) (logid:15f58092) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"683aa0c3-d606-11ee-8b6a-0050568d528d"}]' is allowed to perform API calls: <IP_ADDRESS>/0,::/0 2024-02-28 15:48:59,349 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278254413-352:ctx-5219e9e4 ctx-3aeefedf) (logid:15f58092) Account [Account [{"accountName":"admin","id":2,"uuid":"683aa0c3-d606-11ee-8b6a-0050568d528d"}]] is Root Admin or Domain Admin, all APIs are allowed. 2024-02-28 15:48:59,350 WARN  [o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278254413-352:ctx-5219e9e4 ctx-3aeefedf) (logid:15f58092) Project is null, ProjectRoleBasedApiAccessChecker only applies to projects, returning API [importUnmanagedInstance] for user [User {"username":"admin","uuid":"683bac0e-d606-11ee-8b6a-0050568d528d"}.] as allowed. 2024-02-28 15:48:59,351 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1278254413-352:ctx-5219e9e4 ctx-3aeefedf) (logid:15f58092) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2024-02-28 15:48:59,351 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1278254413-352:ctx-5219e9e4 ctx-3aeefedf) (logid:15f58092) API rate limiting is disabled. We will not use ApiRateLimitService. 2024-02-28 15:48:59,353 WARN  [c.c.a.d.ParamGenericValidationWorker] (qtp1278254413-352:ctx-5219e9e4 ctx-3aeefedf) (logid:15f58092) Received unknown parameters for command importUnmanagedInstance. Unknown parameters : temppath importsource hypervisor host zoneid diskpath 2024-02-28 15:48:59,366 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-10:ctx-aa2475ff job-142) (logid:4377266b) Add job-142 into job monitoring 2024-02-28 15:48:59,374 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp1278254413-352:ctx-5219e9e4 ctx-3aeefedf) (logid:15f58092) submit async job-142, details: AsyncJobVO: {id:142, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.vm.ImportUnmanagedInstanceCmd, cmdInfo: {"temppath":"","migrateallowed":"true","httpmethod":"GET","ctxAccountId":"2","details[0].memory":"32768","cmdEventType":"VM.IMPORT","hypervisor":"vmware","host":"","ctxStartEventId":"373","ctxDetails":"{\"interface com.cloud.offering.ServiceOffering\":\"04a0bc45-5669-4ef7-8214-ea7bf8e4e87f\",\"interface com.cloud.org.Cluster\":\"07cc5a91-8306-4ae9-9a49-8b89965ea23f\"}","details[0].cpuNumber":"8","details[0].cpuSpeed":"1000","clusterid":"07cc5a91-8306-4ae9-9a49-8b89965ea23f","importsource":"unmanaged","nicnetworklist[0].network":"871e547e-d26a-43c0-9d7b-0c1f33a3edb8","serviceofferingid":"04a0bc45-5669-4ef7-8214-ea7bf8e4e87f","response":"json","ctxUserId":"2","displayname":"testvm","name":"WinTestYeni","zoneid":"3e34a0b7-ba5e-44f1-ab7a-8598549095dc","diskpath":"","nicnetworklist[0].nic":"Network adapter 1","username":""}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:<PHONE_NUMBER>, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null} 2024-02-28 15:48:59,374 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-10:ctx-aa2475ff job-142) (logid:36097674) Executing AsyncJobVO: {id:142, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.vm.ImportUnmanagedInstanceCmd, cmdInfo: {"temppath":"","migrateallowed":"true","httpmethod":"GET","ctxAccountId":"2","details[0].memory":"32768","cmdEventType":"VM.IMPORT","hypervisor":"vmware","host":"","ctxStartEventId":"373","ctxDetails":"{\"interface com.cloud.offering.ServiceOffering\":\"04a0bc45-5669-4ef7-8214-ea7bf8e4e87f\",\"interface com.cloud.org.Cluster\":\"07cc5a91-8306-4ae9-9a49-8b89965ea23f\"}","details[0].cpuNumber":"8","details[0].cpuSpeed":"1000","clusterid":"07cc5a91-8306-4ae9-9a49-8b89965ea23f","importsource":"unmanaged","nicnetworklist[0].network":"871e547e-d26a-43c0-9d7b-0c1f33a3edb8","serviceofferingid":"04a0bc45-5669-4ef7-8214-ea7bf8e4e87f","response":"json","ctxUserId":"2","displayname":"testvm","name":"WinTestYeni","zoneid":"3e34a0b7-ba5e-44f1-ab7a-8598549095dc","diskpath":"","nicnetworklist[0].nic":"Network adapter 1","username":""}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:<PHONE_NUMBER>, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null} 2024-02-28 15:48:59,375 DEBUG [c.c.a.ApiServlet] (qtp1278254413-352:ctx-5219e9e4 ctx-3aeefedf) (logid:15f58092) ===END===  <IP_ADDRESS> -- GET  name=WinTestYeni&clusterid=07cc5a91-8306-4ae9-9a49-8b89965ea23f&displayname=testvm&zoneid=3e34a0b7-ba5e-44f1-ab7a-8598549095dc&importsource=unmanaged&hypervisor=vmware&host=&username=&diskpath=&temppath=&serviceofferingid=04a0bc45-5669-4ef7-8214-ea7bf8e4e87f&details[0].cpuNumber=8&details[0].cpuSpeed=1000&details[0].memory=32768&migrateallowed=true&nicnetworklist[0].nic=Network+adapter+1&nicnetworklist[0].network=871e547e-d26a-43c0-9d7b-0c1f33a3edb8&command=importUnmanagedInstance&response=json 2024-02-28 15:48:59,378 WARN  [c.c.a.d.ParamGenericValidationWorker] (API-Job-Executor-10:ctx-aa2475ff job-142 ctx-8dae85a2) (logid:36097674) Received unknown parameters for command importUnmanagedInstance. Unknown parameters : temppath importsource hypervisor host zoneid diskpath 2024-02-28 15:48:59,386 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-10:ctx-aa2475ff job-142 ctx-8dae85a2) (logid:36097674) Access granted to Account [{"accountName":"admin","id":2,"uuid":"683aa0c3-d606-11ee-8b6a-0050568d528d"}] to Service offering {"id":37,"name":"Custom","uuid":"04a0bc45-5669-4ef7-8214-ea7bf8e4e87f"}. by AffinityGroupAccessChecker 2024-02-28 15:48:59,388 TRACE [o.a.c.a.c.a.v.ImportUnmanagedInstanceCmd] (API-Job-Executor-10:ctx-aa2475ff job-142 ctx-8dae85a2) (logid:36097674) nic, 'Network adapter 1', goes on net, '871e547e-d26a-43c0-9d7b-0c1f33a3edb8' 2024-02-28 15:48:59,397 DEBUG [c.c.a.ApiServlet] (qtp1278254413-364:ctx-71cf1a66) (logid:5b984267) ===START===  <IP_ADDRESS> -- GET  jobId=36097674-1d79-401d-ba43-130fde33d3c4&command=queryAsyncJobResult&response=json 2024-02-28 15:48:59,397 DEBUG [c.c.a.ApiServlet] (qtp1278254413-364:ctx-71cf1a66) (logid:5b984267) Two factor authentication is already verified for the user 2, so skipping 2024-02-28 15:48:59,402 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-199:ctx-09d81f14) (logid:99c7322f) Seq 1-3508304109721617290: Executing request 2024-02-28 15:48:59,402 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-199:ctx-09d81f14) (logid:36097674) Executing resource command GetUnmanagedInstancesCommand: []. 2024-02-28 15:48:59,403 DEBUG [c.c.a.ApiServer] (qtp1278254413-364:ctx-71cf1a66 ctx-f1129f61) (logid:5b984267) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"683aa0c3-d606-11ee-8b6a-0050568d528d"}]' is allowed to perform API calls: <IP_ADDRESS>/0,::/0 2024-02-28 15:48:59,405 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278254413-364:ctx-71cf1a66 ctx-f1129f61) (logid:5b984267) Account [Account [{"accountName":"admin","id":2,"uuid":"683aa0c3-d606-11ee-8b6a-0050568d528d"}]] is Root Admin or Domain Admin, all APIs are allowed. 2024-02-28 15:48:59,407 WARN  [o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278254413-364:ctx-71cf1a66 ctx-f1129f61) (logid:5b984267) Project is null, ProjectRoleBasedApiAccessChecker only applies to projects, returning API [queryAsyncJobResult] for user [User {"username":"admin","uuid":"683bac0e-d606-11ee-8b6a-0050568d528d"}.] as allowed. 2024-02-28 15:48:59,407 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1278254413-364:ctx-71cf1a66 ctx-f1129f61) (logid:5b984267) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker. 2024-02-28 15:48:59,408 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1278254413-364:ctx-71cf1a66 ctx-f1129f61) (logid:5b984267) API rate limiting is disabled. We will not use ApiRateLimitService. 2024-02-28 15:48:59,418 DEBUG [c.c.a.ApiServlet] (qtp1278254413-364:ctx-71cf1a66 ctx-f1129f61) (logid:5b984267) ===END===  <IP_ADDRESS> -- GET  jobId=36097674-1d79-401d-ba43-130fde33d3c4&command=queryAsyncJobResult&response=json 2024-02-28 15:48:59,419 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-199:ctx-09d81f14 esx02.flexapp360.cloud, job-142, cmd: GetUnmanagedInstancesCommand) (logid:36097674) find VM hypervisor name: WinTestYeni on host 2024-02-28 15:48:59,419 INFO  [c.c.h.v.m.HostMO] (DirectAgent-199:ctx-09d81f14 esx02.flexapp360.cloud, job-142, cmd: GetUnmanagedInstancesCommand) (logid:36097674) VM hypervisor name: WinTestYeni not found in host cache 2024-02-28 15:48:59,419 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-199:ctx-09d81f14 esx02.flexapp360.cloud, job-142, cmd: GetUnmanagedInstancesCommand) (logid:36097674) load VM cache on host 2024-02-28 15:48:59,459 DEBUG [c.c.a.ApiServlet] (qtp1278254413-352:ctx-944e5120) (logid:b2962708) ===START===  <IP_ADDRESS> -- GET  clusterid=07cc5a91-8306-4ae9-9a49-8b89965ea23f&page=1&pagesize=10&command=listUnmanagedInstances&response=json 2024-02-28 15:48:59,459 DEBUG [c.c.a.ApiServlet] (qtp1278254413-352:ctx-944e5120) (logid:b2962708) Two factor authentication is already verified for the user 2, so skipping 2024-02-28 15:48:59,460 DEBUG [c.c.a.ApiServlet] (qtp1278254413-1586:ctx-b445e3bf) (logid:e3503aed) ===START===  <IP_ADDRESS> -- GET  listall=true&clusterid=07cc5a91-8306-4ae9-9a49-8b89965ea23f&page=1&pagesize=10&command=listVirtualMachines&response=json 2024-02-28 15:48:59,460 DEBUG [c.c.a.ApiServlet] (qtp1278254413-1586:ctx-b445e3bf) (logid:e3503aed) Two factor authentication is already verified for the user 2, so skipping 2024-02-28 15:48:59,466 DEBUG [c.c.a.ApiServer] (qtp1278254413-352:ctx-944e5120 ctx-3e42fa19) (logid:b2962708) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"683aa0c3-d606-11ee-8b6a-0050568d528d"}]' is allowed to perform API calls: <IP_ADDRESS>/0,::/0 2024-02-28 15:48:59,466 DEBUG [c.c.a.ApiServer] (qtp1278254413-1586:ctx-b445e3bf ctx-74422909) (logid:e3503aed) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"683aa0c3-d606-11ee-8b6a-0050568d528d"}]' is allowed to perform API calls: <IP_ADDRESS>/0,::/0 2024-02-28 15:48:59,468 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278254413-352:ctx-944e5120 ctx-3e42fa19) (logid:b2962708) Account [Account [{"accountName":"admin","id":2,"uuid":"683aa0c3-d606-11ee-8b6a-0050568d528d"}]] is Root Admin or Domain Admin, all APIs are allowed. 2024-02-28 15:48:59,468 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278254413-1586:ctx-b445e3bf ctx-74422909) (logid:e3503aed) Account [Account [{"accountName":"admin","id":2,"uuid":"683aa0c3-d606-11ee-8b6a-0050568d528d"}]] is Root Admin or Domain Admin, all APIs are allowed. 2024-02-28 15:48:59,470 WARN  [o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278254413-1586:ctx-b445e3bf ctx-74422909) (logid:e3503aed) Project is null, ProjectRoleBasedApiAccessChecker only applies to projects, returning API [listVirtualMachines] for user [User {"username":"admin","uuid":"683bac0e-d606-11ee-8b6a-0050568d528d"}.] as allowed. 2024-02-28 15:48:59,470 WARN  [o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278254413-352:ctx-944e5120 ctx-3e42fa19) (logid:b2962708) Project is null, ProjectRoleBasedApiAccessChecker only applies to projects, returning API [listUnmanagedInstances] for user [User {"username":"admin","uuid":"683bac0e-d606-11ee-8b6a-0050568d528d"}.] as allowed.

Thanks @omurozlu - I have quickly tested imports on NFS cluster and zone wide storage and couldn't hit the issue. It seems related to datastore cluster type, will try to replicate it using datastore cluster

4.18 version working but 4.19 not working

Hi @omurozlu - I have tested 4.19.0 creating a VM with a ROOT volume on a datastore cluster, unmanaging it and import it back and it succeeds.
I have checked your logs and the images you have shared and I think the problem is that the datastore cluster in which the ROOT volume of the VM is located, is not added to CloudStack.
From the cmk output, CloudStack is looking for datastore FA360-DS02-ESX:
"Storage pool for disk Hard disk 1(123-2000) with datastore: FA360-DS02-ESX not found in zone ID: 3e34a0b7-ba5e-44f1-ab7a-8598549095dc"

However, from the images above, I only see one datastore with name FA360-DS01-ESX.
Can you double check the names on vCenter and CloudStack? In case you have renamed the datastore on vCenter, you should rename it as well on CloudStack for them to match

Hello again @nvazquez ,
I now tested again (reset system)

this first tested
==============
and im again reset system

all time give same error

Sorry @omurozlu I wrongly tested datastore cluster pools, but I see you are using VMFS pools instead. Can you share what's the output of this SQL query?
select * from storage_pool;

@nvazquez,

Thanks @omurozlu can you please upload the management server logs around the whole import operation? I want to compare the received unmanaged instance disk's information to the information of the pool in the database

hello again @nvazquez,
im again re install and I showed you everything I did
https://flexapp360.cloud/video.gif
management-server.log

Thanks @omurozlu - can you please test the fix https://github.com/apache/cloudstack/pull/8759?

Hello @nvazquez,
Im reinstall cloudstack and im used this repo
[cloudstack] name=cloudstack baseurl=https://download.cloudstack.org/testing/nightly/2024-03-07/el8/4.19 enabled=1 gpgcheck=0
=> CloudStack <IP_ADDRESS>-SNAPSHOT
but now same error give me

Hi @omurozlu, this fix is not yet merged so the nightly version does not contain the fix

@nvazquez hello,
I have to wait for the version update ?
sorry i dont know how build project :)

@omurozlu I can give you the steps to build:
yum install git maven rpm-build (could be more required, can check executing the last step if it requires more packages)

git clone https://github.com/apache/cloudstack.git
git clone https://github.com/shapeblue/cloudstack-nonoss.git
cp cloudstack-nonoss/*.jar cloudstack/deps
cd cloudstack
git fetch origin
git fetch origin pull/8759/head:pr/8759
git checkout pr/8759
cd deps/
./install-non-oss.sh
cd ../packaging
./package.sh -d centos8

@nvazquez,
im step step make but give me error all time.

@omurozlu you can consider running these commands on a VM different than the management server and copying the built commands once they are finished. For the failure above I suspect you are missing the Java SDK 11, you can try installing openjdk-11

Btw I've updated the last command, it was missing -p noredist

Hello again @nvazquez,
im again trying,
1- ) install centos 7 and update
2-) install java-11-openjdk, java-11-openjdk-devel, gcc, glibc-devel, nodejs
3-) change java version
4-)
`yum install git maven rpm-build (could be more required, can check executing the last step if it requires more packages)
git clone https://github.com/apache/cloudstack.git
git clone https://github.com/shapeblue/cloudstack-nonoss.git
cp cloudstack-nonoss/*.jar cloudstack/deps
cd cloudstack
git fetch origin
git fetch origin pull/8759/head:pr/8759
git checkout pr/8759
cd deps/
./install-non-oss.sh
cd ../packaging
FLAGS=-DskipTests ./package.sh -p noredist -d centos8`
and still give me error

Your information is very valuable, thank you

Hi @omurozlu the fix has been merged today and will be available on the 15/03 nightly packages on the 4.19 branch

hello @nvazquez

Hi @omurozlu the fix has been merged today and will be available on the 15/03 nightly packages on the 4.19 branch

im tested this version and working.
I have a few different questions
1-) When will this update be merged?
2-) Or how can I build the stable version + this pr ?

Thanks @omurozlu - I will close this issue as per your confirmation that is fixed.
The fix is already merged, it will be available on the next minor release for 4.19 which will be 4.19.1 and onwards. 4.19.1 can be expected on Q2 of this year (https://lists.apache.org/thread/0zg248b66fxtw1x99bmysjm1g7rygnmw).
To build 4.19.0 + this fix you can try the following:
git checkout <IP_ADDRESS>
Modify the file changed on the PR: https://github.com/apache/cloudstack/pull/8759/files
cd packaging
FLAGS=-DskipTests ./package.sh -p noredist -d centos8


[Event Change] codeunit 12 "Gen. Jnl.-Post Line"

Please change OnBeforePostDeferralPostBuffer event as below :
local procedure PostDeferralPostBuffer(GenJournalLine: Record "Gen. Journal Line")
    var
        DeferralPostBuffer: Record "Deferral Posting Buffer";
        GLEntry: Record "G/L Entry";
        PostDate: Date;
//NEW CODE
        IsHandled: Boolean;
//NEW CODE
    begin
//EVENT CHANGE
        IsHandled := false;
        OnBeforePostDeferralPostBuffer(GenJournalLine, IsHandled);
        IF IsHandled then
            exit;
//EVENT CHANGE

        with GenJournalLine do begin
...

 [IntegrationEvent(true, false)]
    local procedure OnBeforePostDeferralPostBuffer(var GenJournalLine: Record "Gen. Journal Line"; IsHandled: Boolean)
    begin
    end;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.


Buffer_Size in ILI9341_Draw_Colour_Burst() should be fixed

Thank you for your great work!
BTW, I think following line should be fixed
Buffer_Size = Size;
to
Buffer_Size = Size * 2;
It also causes that ILI9341_Draw_Char() works not properly.
When Size > 1 drawn character seems not clean.
https://github.com/martnak/STM32-ILI9341/blob/2ed24b6fbbd080bb79a21d21ba395926898ce31a/Src/ILI9341/ILI9341_STM32_Driver.c#L354

Changing the buffer_size in ILI9341_Draw_Colour_Burst() to Size * 2 was the solution!
I was getting weird effects when trying to use vertical lines, they changed color mid way.

This is most definitely a bug causing stack corruption when the requested Size < BURST_MAX_SIZE.  The fix above worked for me as well.


Niivue cleanup after useEffect unmount.

          @hari7696 , please reopen if you can provide a minimal React demo showing the bad behaviour.

Originally posted by @hanayik in https://github.com/niivue/niivue/issues/1009#issuecomment-2191647050
I am not able to reopen the issue, so creating a new one with reference.
Hi @hanayik @neurolabusc
Thanks for the inputs, here is the standalone code that simulates the issue I am facing.
I hosted the code here : https://devmetavision3d.rc.ufl.edu/  in "dummy" tab

standalone code for dummy.js
import React, { useState, useEffect } from 'react';
import { Niivue } from '@niivue/niivue';
import { Dropdown, DropdownToggle, DropdownMenu, DropdownItem } from 'reactstrap';

function DummyRender() {
  const [fileName, setSelectedFile] = useState("");
  const [dropdownOpen, setDropdownOpen] = useState(false);
  const [isattached , setIsAttached] = useState(false);
  const nv1 = new Niivue();
  const nv2 = new Niivue();

  const files = ["5x_ADP.nii.gz","5x_AMP.nii.gz","5x_Aarachidonic.acid.nii.gz","5x_Alanine.nii.gz","5x_Ascorbic.Acid.nii.gz","5x_Aspartate.nii.gz","5x_Carnosine.nii.gz","5x_Citric.Acid.nii.gz","5x_Docosahexaenoic.acid.nii.gz","5x_Fructose.1.6.Bisphosphate.nii.gz","5x_G6P.nii.gz","5x_GMP.nii.gz","5x_GSH.nii.gz","5x_Glucose.nii.gz","5x_Glutamate.nii.gz","5x_Glutamine.nii.gz","5x_Glycerol.3.phosphate.nii.gz","5x_Glycerophosphorylethanolamine.nii.gz","5x_HEME.nii.gz","5x_Hypoxanthine.nii.gz","5x_IMP.nii.gz","5x_Inosine.nii.gz","5x_LPA..20.1..nii.gz","5x_LPA..20.4..nii.gz","5x_LPA..22.4..nii.gz","5x_LPA..22.6..nii.gz","5x_LPA.18.0..nii.gz","5x_LPA.18.1..nii.gz","5x_LPE..16.1..nii.gz"]

  useEffect( () => {
     nv1.attachTo('gl1');
     nv2.attachTo('gl2');
    setIsAttached(true);
  }, [fileName]);

  useEffect(() => {

    console.log("Main  func invoked")
    if (isattached) {
      console.log("volumes loading")
      const volumeList1 = [
        {
          url: `https://devmetavision3d.rc.ufl.edu/data/Alzhemier/Disease/Inverted/${fileName}`,
          opacity: 0.9,
          visible: true,
          colormap: 'blue2red',
        }
      ];
      nv1.loadVolumes(volumeList1);
      nv1.updateGLVolume();
      nv1.setSliceType(nv1.sliceTypeAxial);

      nv2.loadVolumes(volumeList1);
      nv2.updateGLVolume();
      nv2.setSliceType(nv2.sliceTypeRender);
    }

    return () => {
      for (let i = 0; i < nv1.volumes.length; i++) {
        nv1.removeVolumeByIndex(i);
      }

      for (let i = 0; i < nv2.volumes.length; i++) {
        nv2.removeVolumeByIndex(i);
      }
      setIsAttached(false);
      console.log("Volumes removed");
    };
  }, [fileName]);

  const toggleDropdown = () => setDropdownOpen(prevState => !prevState);

  return (

            <Dropdown isOpen={dropdownOpen} toggle={toggleDropdown} style={{ textAlign: 'left', width: '300px', padding: '10px' }}>
                <DropdownToggle caret  style={{ width: '400px' }}>
                    {fileName || "Select a file"}
                </DropdownToggle>
                {/* The right prop is omitted or set to false for left alignment */}
                <DropdownMenu className="scrollable-menu">
                    {files.map((file, index) => (
                        <DropdownItem
                            key={index}
                            onClick={() => setSelectedFile(file)}
                        >
                            {file}
                        </DropdownItem>
                    ))}
                </DropdownMenu>
            </Dropdown>

Wild Type Brain

          <canvas id="gl1" style={{ width: '500px', height: '400px' }}></canvas>

          <canvas id="gl2" style={{ width: '500px', height: '400px' }}></canvas>

  );
}

export default DummyRender;

Behaviour observed.

After every image switch, the the Memory usage [ Windows Task Manager] is kept on increasing. I am adding screen grabs taken between image switches.

I monitored the page performance as well, I dont see any exponential increase.

@hari7696 I think @hanayik is the best person to provide feedback. However, I notice that the console log of your demo reports you are using  niivue version from 5 months ago (0.38.10). I doubt it is related to your issue, but there are numerous useful enhancements in the current stable release (0.43.4).

@hari7696 a few minor suggestions about your images.

I would avoid double periods in filenames, so I would suggest gaa_PI.38.5.nii.gz rather than gaa_PI.38.5..nii.gz
I note that all of these images use the float64 datatype, but actual data precision is very low. To my eye, you could use uint8 (0..255), but to be on the cautious side you could use int16 (-32768..32767). Using the Python script below I converting the image gaa_PI.38.5 to uint8 reduced compressed file size from 71.8mb to 7.1mb, and reduces the uncompressed size by a factor of 8. If you know that these images will only be compressed once, but downloaded many times you can compress the image to 6.6mb using zopfli (pigz -n -11 img.nii). Smaller files and more compact datatype will make your elegant tool much more responsive.

import nibabel as nib
import numpy as np

# Load the NIfTI file
fnm = 'gaa_PI.38.5..nii.gz'
nii = nib.load(fnm)
img = nii.get_fdata()

# Set non-finite values to zero
img[~np.isfinite(img)] = 0

# Scale the image to use the full range of 0 to 255
min_val = np.min(img)
max_val = np.max(img)

if max_val > min_val:  # Ensure there is a range to scale
    img = (img - min_val) / (max_val - min_val) * 255
else:
    img = np.zeros_like(img)  # If no range, set the image to all zeros

# Convert the image to uint8
img = img.astype(np.uint8)

# Save the new NIfTI file
nii.set_data_dtype(np.ubyte)
nii = nib.Nifti1Image(img, nii.affine, nii.header)
nii.header['scl_slope'] = 1.0
nii.header['scl_inter'] = 0.0
outnm = 'gaa_PI.38.5u8.nii.gz'
nib.save(nii, outnm)

@neurolabusc  Thanks, I'll certianly conisder dropping the precision.


Check if the code for the DataGrids in x-item-supply-platform is ok with Simon

I put the nutrients, physical characteristics and allergens in Datagrid inside the file itemSupplyPlatform/model.js
What the Datagrids look like for the moment

For the code, I used a generic function setDataGrid (used for the generation of the nutrients, allergens.. Datagrids)
The function is called 3 times in the method setData

@simonLouvet can we close this one?


Add "Incubating" to the project name in docs

"Incubating" is required to be add after the project name, e.g. "EventMesh (Incubating)" in the doc

@kezhenxu94 Good reminder, you can request a pr if you wish, thanks.


Remove node_modules from repository

I'm afraid the only way to fix this is to restart the git repository from scratch 👉👌

noooooooooooooooes :D

tried a git -r --cached on nodes modules. I assume it has worked pretty well, but can you confirm @vitaminwater? thx

No, this only removes node_modules from further commits, but it is still present in the previous commits. thus still there somewhere in the .git for eternity.
No more seriously, I remember that there is a way to remove something from ALL commits (even previous), but it requires to tell everyone to rm and re-clone the repository.

So the question is: Could @raphisky handles such a legacy in his carreer? 🤡


Preparing for Laravel 5.3

Laravel 5.3 is currently slated for a release around the same time as Laracon (July 27-29th), which means it could very well be less than a month away (I assume they'd want to release before the conference). It will only support PHP 5.6+, 7.0+, and HHVM. As we previously decided to track the latest release of Laravel, this kinda matters. I think we should start building now with the system packages that Laravel 5.3 will use so that we're as prepared as we can be before we actually update the framework and deal with that stuff. Then, when 5.3 is released, we should just have to update the dependency in composer.json and work on the things in the "code stuff" section and be up and running quickly (in theory).
Build/CI

[ ] Update CircleCI config to a PHP version of 5.6.x

[ ] Update CircleCI Project Config (on site) to use the Ubuntu 14.04 VM (waiting for @euantorano to say OK on doing that; I doubt it will break anything)
(This will update available versions of Node, PHP, etc. to modern ones, which will reduce errors echoed during testing)

[ ] Build project (latest master commit and important PRs) on CircleCI after the updates to make sure nothing breaks due to the environment changes
(Build req'd after changing VM version to make the update take effect)
[ ] Fix stuff if it breaks

Code

[ ] Query Builder will now return a Collection instead of an array. Any uses of array functions or array-like interaction should be changed to use Collection stuff.

Having trouble grepping to see how badly this will affect us.
Potentially affected files:

[ ] ./app/Console/Commands/RecountCommand.php
[ ] ./app/Database/Repositories/Eloquent/ForumRepository.php
[ ] ./app/Database/Repositories/Eloquent/PollVoteRepository.php
[ ] ./app/Database/Repositories/Eloquent/PostRepository.php
[ ] ./app/Database/Repositories/Eloquent/ProfileFieldOptionRepository.php
[ ] ./app/Database/Repositories/Eloquent/ProfileFieldRepository.php
[ ] ./app/Database/Repositories/Eloquent/TopicRepository.php
[ ] ./app/Database/Repositories/Eloquent/UserProfileFieldRepository.php
[ ] ./app/Http/Controllers/ModerationController.php
[ ] ./app/Http/Controllers/SearchController.php
[ ] ./app/Http/Requests/Account/UpdateProfileRequest.php
[ ] ./app/Likes/Database/Repositories/Eloquent/LikesRepository.php
[ ] ./app/Presenters/ModerationLogPresenter.php
[ ] ./app/Presenters/ProfileFieldGroupPresenter.php

[ ] Route::controller() will no longer be a thing. There is one call to Route::controllers() in routes.php  (Source: https://github.com/laravel/framework/pull/10777).

[ ] ./app/Http/routes.php:72 (maybe; assuming it will matter)

[ ] Collection::whereLoose() method was removed since Collection::where() now supports operators.
Supported: = == === <= >= < > <> != !==

[ ] ./app/Database/Models/User.php:132
[ ] ./app/Database/Repositories/Eloquent/ConversationRepository.php:221

I have no problem with the upgrade when it happens, but we need to make sure we don't end up using a version that isn't going to work on the majority of hosts. 5.2 requires PHP >5.5.9, 5.3 requires >5.6. We've already hanged our minds a few times about version requirements, so the benefits of the updated framework need to outweigh that IMO.

There is way more to change. I think it would be easier and better to install a fresh Laravel 5.3 installation and add each functionality bit by bit. Some many things are too old or nearly deprecated.

@matthijs110 We'll evaluate it when 5.3 is released. We managed to upgrade to 5.2 without too much hassle in the end. The Laravel dev usually put out  good upgrade guide.

So 5.3 is out... https://laravel.com/docs/5.3/upgrade#upgrade-5.3.0

Yep, I'll be looking at going through the upgrade soon, hopefully this weekend. I also need to try and fix our errors in Travis related to HHVM.

HHVM isn't supported by 5.3 so we don't need Travis to check HHMV surely...?

@JordanMussi It's not officially supported, but it would be nice to know that we can run on it to be sure. As far as I know the base 5.3 framework will work just fine on HHVM as it stands.

@euantorano Ah fair enough! :grinning:


HADOOP-16806: AWS AssumedRoleCredentialProvider needs ExternalId add

Description of PR
This applies an externalId value to the STSAssumeRoleSessionCredentialsProvider.Builder, if provided in the Hadoop config field fs.s3a.assumed.role.externalid.
This allows for AWS resources to have a trust policy for sts:AssumeRole that can match on the externalId which is now provided as part of the assume role request, in order to solve the confused deputy problem
I'm happy to take guidance on an improved unit test or any other changes. I'm relatively unfamiliar with the Hadoop unit testing framework.
How was this patch tested?
Manual testing, and now running in a production SaaS offering.
For code changes:

[x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

which s3 region did you run the hadoop-aws integration tests against?

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m 11s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
markdownlint
0m  1s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
37m 39s

trunk passed

+1 :green_heart:
compile
1m  3s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
0m 54s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
0m 53s

trunk passed

+1 :green_heart:
mvnsite
1m  1s

trunk passed

+1 :green_heart:
javadoc
0m 48s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 50s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
1m 33s

trunk passed

+1 :green_heart:
shadedclient
25m 15s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 40s

the patch passed

+1 :green_heart:
compile
0m 44s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javac
0m 44s

the patch passed

+1 :green_heart:
compile
0m 35s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
0m 35s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 24s

the patch passed

+1 :green_heart:
mvnsite
0m 42s

the patch passed

+1 :green_heart:
javadoc
0m 24s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 30s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
1m 17s

the patch passed

+1 :green_heart:
shadedclient
20m 41s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
3m  2s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 51s

The patch does not generate ASF License warnings.

102m  8s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4753/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4753

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint

uname
Linux d194f91c3a26 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 5afe1ed91e7e89c4a36167241bb5c94d870dfdf7

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4753/1/testReport/

Max. process+thread count
700 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4753/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

I've updated the test, and I attest that I ran all of the hadoop-aws tests, as well as integration tests against us-east-1.
However, I was unable to successfully run ITestDelegatedMRJob.testJobSubmissionCollectsTokens, I believe due to https://issues.apache.org/jira/browse/HADOOP-17784, although it may also be possible that I don't have the assume role configured correctly for this test.
This seems unrelated to the feature at hand, but I can revisit if required.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 59s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
markdownlint
0m  1s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
38m 11s

trunk passed

+1 :green_heart:
compile
1m  1s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
0m 56s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
0m 53s

trunk passed

+1 :green_heart:
mvnsite
1m  2s

trunk passed

+1 :green_heart:
javadoc
0m 50s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 50s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
1m 36s

trunk passed

+1 :green_heart:
shadedclient
21m  3s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 39s

the patch passed

+1 :green_heart:
compile
0m 43s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javac
0m 43s

the patch passed

+1 :green_heart:
compile
0m 38s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
0m 38s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 28s

the patch passed

+1 :green_heart:
mvnsite
0m 42s

the patch passed

+1 :green_heart:
javadoc
0m 24s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 32s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
1m 15s

the patch passed

+1 :green_heart:
shadedclient
20m 38s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
3m  3s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 51s

The patch does not generate ASF License warnings.

98m 23s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4753/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4753

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint

uname
Linux 6303de1777b9 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 7e12347b0a4d19d0528ada1723aeed0b07d9c871

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4753/2/testReport/

Max. process+thread count
722 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4753/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

I was unable to successfully run ITestDelegatedMRJob.testJobSubmissionCollectsTokens

got a stack trace?

Tried with and without an fs.s3a.assumed.role.arn setting. I'm not sure if I'm expected to set up this dataset manually, or if there's some magic behind the scenes that should make it available?
[ERROR] testJobSubmissionCollectsTokens[0](org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob)  Time elapsed: 10.84 s  <<< ERROR!
org.apache.hadoop.fs.s3a.AWSBadRequestException: getFileStatus on s3a://landsat-pds/scene_list.gz: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 16AXE4CXSVZBN8BH; S3 Extended Request ID: 3F4W4GXGPmUqh9fgojrQoGZYInEGXbKez4G3LAzmbSvL/1nyyyAq0kfXrr5LszB5B6wg7Inr6SM=; Proxy: null), S3 Extended Request ID: 3F4W4GXGPmUqh9fgojrQoGZYInEGXbKez4G3LAzmbSvL/1nyyyAq0kfXrr5LszB5B6wg7Inr6SM=:400 Bad Request: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 16AXE4CXSVZBN8BH; S3 Extended Request ID: 3F4W4GXGPmUqh9fgojrQoGZYInEGXbKez4G3LAzmbSvL/1nyyyAq0kfXrr5LszB5B6wg7Inr6SM=; Proxy: null)
	at org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob.testJobSubmissionCollectsTokens(ITestDelegatedMRJob.java:281)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 16AXE4CXSVZBN8BH; S3 Extended Request ID: 3F4W4GXGPmUqh9fgojrQoGZYInEGXbKez4G3LAzmbSvL/1nyyyAq0kfXrr5LszB5B6wg7Inr6SM=; Proxy: null)
	at org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob.testJobSubmissionCollectsTokens(ITestDelegatedMRJob.java:281)

[ERROR] testJobSubmissionCollectsTokens[1](org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob)  Time elapsed: 7.758 s  <<< ERROR!
org.apache.hadoop.fs.s3a.AWSBadRequestException: getFileStatus on s3a://landsat-pds/scene_list.gz: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: W01GKMAXC95KHVPR; S3 Extended Request ID: OGWbDa4Lv6wwE9mBU6+QV+cYjZOgkRT3PFAvful1QZykKZ2t1ql/bNJaeiTZd/GjxFyE52/itms=; Proxy: null), S3 Extended Request ID: OGWbDa4Lv6wwE9mBU6+QV+cYjZOgkRT3PFAvful1QZykKZ2t1ql/bNJaeiTZd/GjxFyE52/itms=:400 Bad Request: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: W01GKMAXC95KHVPR; S3 Extended Request ID: OGWbDa4Lv6wwE9mBU6+QV+cYjZOgkRT3PFAvful1QZykKZ2t1ql/bNJaeiTZd/GjxFyE52/itms=; Proxy: null)
	at org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob.testJobSubmissionCollectsTokens(ITestDelegatedMRJob.java:281)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: W01GKMAXC95KHVPR; S3 Extended Request ID: OGWbDa4Lv6wwE9mBU6+QV+cYjZOgkRT3PFAvful1QZykKZ2t1ql/bNJaeiTZd/GjxFyE52/itms=; Proxy: null)
	at org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob.testJobSubmissionCollectsTokens(ITestDelegatedMRJob.java:281)

maybe its a permissions thing. show the full stack trace for the failed tests, please

[ERROR] testJobSubmissionCollectsTokens[0](org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob)  Time elapsed: 15.048 s  <<< ERROR!
org.apache.hadoop.fs.s3a.AWSBadRequestException: getFileStatus on s3a://landsat-pds/scene_list.gz: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: D2F7W48K28SJYAQ8; S3 Extended Request ID: jFvfXqdgBlubfaH3svMYtDxgAqD9Ij9RZ/saXipGPoRXDeVUDT/ApOXfYINPOYfG/U+AjZyhfzc=; Proxy: null), S3 Extended Request ID: jFvfXqdgBlubfaH3svMYtDxgAqD9Ij9RZ/saXipGPoRXDeVUDT/ApOXfYINPOYfG/U+AjZyhfzc=:400 Bad Request: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: D2F7W48K28SJYAQ8; S3 Extended Request ID: jFvfXqdgBlubfaH3svMYtDxgAqD9Ij9RZ/saXipGPoRXDeVUDT/ApOXfYINPOYfG/U+AjZyhfzc=; Proxy: null)
	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:241)
	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:172)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3567)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3473)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getFileStatus$24(S3AFileSystem.java:3450)
	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2341)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2360)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3448)
	at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:115)
	at org.apache.hadoop.fs.Globber.doGlob(Globber.java:349)
	at org.apache.hadoop.fs.Globber.glob(Globber.java:202)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$globStatus$32(S3AFileSystem.java:4387)
	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2341)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2360)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.globStatus(S3AFileSystem.java:4380)
	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:310)
	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:278)
	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:432)
	at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:310)
	at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:327)
	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:200)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1677)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1674)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1919)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1674)
	at org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob.testJobSubmissionCollectsTokens(ITestDelegatedMRJob.java:281)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: D2F7W48K28SJYAQ8; S3 Extended Request ID: jFvfXqdgBlubfaH3svMYtDxgAqD9Ij9RZ/saXipGPoRXDeVUDT/ApOXfYINPOYfG/U+AjZyhfzc=; Proxy: null), S3 Extended Request ID: jFvfXqdgBlubfaH3svMYtDxgAqD9Ij9RZ/saXipGPoRXDeVUDT/ApOXfYINPOYfG/U+AjZyhfzc=
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1372)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$11(S3AFileSystem.java:2511)
	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2499)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2479)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3545)
	... 46 more

The other test stack trace is basically the same

It looks like the issue is in retrieving s3a://landsat-pds/scene_list.gz. The corresponding role should have full S3 access at this point.
I've enabled the low-level request tracing. Here's what it looks like for the first parameterized test:
testJobSubmissionCollectsTokens[0](org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob)
2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "HEAD /scene_list.gz HTTP/1.1[\r][\n]"
2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Host: landsat-pds.s3.amazonaws.com[\r][\n]"
2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "amz-sdk-invocation-id: <redacted>[\r][\n]"
2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "amz-sdk-request: attempt=1;max=21[\r][\n]"
2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "amz-sdk-retry: 0/0/500[\r][\n]"
2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Authorization: AWS4-HMAC-SHA256 Credential=<redacted>/20220823/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;referer;user-agent;x-amz-content-sha256;x-amz-date, Signature=<redacted>[\r][\n]"
2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Content-Type: application/octet-stream[\r][\n]"
2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Referer<EMAIL_ADDRESS>2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "User-Agent: Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.262 Linux/5.10.104-linuxkit OpenJDK_64-Bit_Server_VM/25.342-b07 java/1.8.0_342 kotlin/1.4.10 vendor/Private_Build cfg/retry-mode/legacy[\r][\n]"
2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]"
2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "X-Amz-Date: 20220823T161525Z[\r][\n]"
2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Connection: Keep-Alive[\r][\n]"
2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "[\r][\n]"
2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "HTTP/1.1 400 Bad Request[\r][\n]"
2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "x-amz-request-id: ZMY1169S2RC7NEHP[\r][\n]"
2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "x-amz-id-2: ENtHr5DR7HX+qkh5FfYQublaU82ykB/SD5fAvR5kC6JZJibFVLH6Rq+F/EutrE3dAL1uTz6yad8=[\r][\n]"
2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "Content-Type: application/xml[\r][\n]"
2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "Date: Tue, 23 Aug 2022 16:15:25 GMT[\r][\n]"
2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "Server: AmazonS3[\r][\n]"
2022-08-23 16:15:25,937 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "Connection: close[\r][\n]"
2022-08-23 16:15:25,937 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "[\r][\n]"

I've also tried adding this to my auth-keys, which likewise does not work:
<property>
  <name>fs.s3a.bucket.landsat-pds.aws.credentials.provider</name>
  <value>org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider</value>
</property>

I've seen mention in HADOOP-13551 and HADOOP-18340 that folks have run into similar issues with this test, although I haven't been able to determine what, if anything, was done to correct it.

Unfortunately, the errors can be a bit vague for S3 HeadObject (used by getFileStatus). There was a similar issue when the credentials had expired in HADOOP-18353.
...
Authorization: AWS4-HMAC-SHA256 Credential=<redacted>/20220823/us-east-1/s3/aws4_request
...

The signature is signed for us-east-1, but the landsat-pds bucket is in us-west-2. Do you have fs.s3a.endpoint.region set? If so, try adding the following to your auth-keys.xml:
<property>
  <name>fs.s3a.bucket.landsat-pds.endpoint.region</name>
  <value>us-west-2</value>
</property>

Thanks @dannycjones, that did it!
All integration tests have now passed in this PR.
I was indeed using us-east-1 in fs.s3a.endpoint.region, which would explain the issue. Might make sense to add some additional instructions in hadoop-aws/testing.md under Testing against different regions. I can include that as part of this PR, or file a separate one.

need to fix hadoop-tools/hadoop-aws/src/test/resources/core-site.xml ; it is set there

Specifying us-west-2 for both landsat-pds and usgs-landsat allowed my tests to pass with my the various region settings in auth-keys.xml set to us-east-1.
There is an issue with ITestS3ABucketExistence#testAccessPointProbingV2,testAccessPointRequired with another region enabled.
[ERROR] testAccessPointProbingV2(org.apache.hadoop.fs.s3a.ITestS3ABucketExistence)  Time elapsed: 10.476 s  <<< ERROR!
java.lang.IllegalArgumentException: The region field of the ARN being passed as a bucket parameter to an S3 operation does not match the region the client was configured with. Provided region: 'eu-west-1'; client region: 'us-east-1'.
	at org.apache.hadoop.fs.s3a.ITestS3ABucketExistence.lambda$testAccessPointProbingV2$12(ITestS3ABucketExistence.java:172)
	at org.apache.hadoop.fs.s3a.ITestS3ABucketExistence.expectUnknownStore(ITestS3ABucketExistence.java:103)
	at org.apache.hadoop.fs.s3a.ITestS3ABucketExistence.testAccessPointProbingV2(ITestS3ABucketExistence.java:171)

[ERROR] testAccessPointRequired(org.apache.hadoop.fs.s3a.ITestS3ABucketExistence)  Time elapsed: 0.889 s  <<< ERROR!
java.lang.IllegalArgumentException: The region field of the ARN being passed as a bucket parameter to an S3 operation does not match the region the client was configured with. Provided region: 'eu-west-1'; client region: 'us-east-1'.
	at org.apache.hadoop.fs.s3a.ITestS3ABucketExistence.lambda$testAccessPointRequired$14(ITestS3ABucketExistence.java:189)
	at org.apache.hadoop.fs.s3a.ITestS3ABucketExistence.expectUnknownStore(ITestS3ABucketExistence.java:103)
	at org.apache.hadoop.fs.s3a.ITestS3ABucketExistence.testAccessPointRequired(ITestS3ABucketExistence.java:188)

Commenting out my region settings allows these tests to pass, however.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  2s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
markdownlint
0m  1s

markdownlint was not available.

+0 :ok:
xmllint
0m  1s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
39m 30s

trunk passed

+1 :green_heart:
compile
0m 53s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
0m 48s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
0m 46s

trunk passed

+1 :green_heart:
mvnsite
0m 52s

trunk passed

+1 :green_heart:
javadoc
0m 36s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 38s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
1m 21s

trunk passed

-1 :x:
shadedclient
2m 32s

branch has errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 39s

the patch passed

+1 :green_heart:
compile
0m 38s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javac
0m 38s

the patch passed

+1 :green_heart:
compile
0m 31s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
0m 31s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 22s

the patch passed

+1 :green_heart:
mvnsite
0m 37s

the patch passed

+1 :green_heart:
javadoc
0m 20s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 31s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
1m 14s

the patch passed

-1 :x:
shadedclient
2m  5s

patch has errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
3m  1s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 40s

The patch does not generate ASF License warnings.

60m 44s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4753/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4753

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint xmllint

uname
Linux cea56b748b9e 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / a00586ceca062495da9ed740c631d8c1e1a89b3f

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4753/3/testReport/

Max. process+thread count
272 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4753/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  8s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
40m 43s

trunk passed

+1 :green_heart:
compile
0m 54s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
0m 46s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
0m 47s

trunk passed

+1 :green_heart:
mvnsite
0m 56s

trunk passed

+1 :green_heart:
javadoc
0m 47s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 39s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
1m 34s

trunk passed

-1 :x:
shadedclient
2m 48s

branch has errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 39s

the patch passed

+1 :green_heart:
compile
0m 44s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javac
0m 44s

the patch passed

+1 :green_heart:
compile
0m 35s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
0m 35s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 25s

the patch passed

+1 :green_heart:
mvnsite
0m 41s

the patch passed

+1 :green_heart:
javadoc
0m 23s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 30s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
1m 11s

the patch passed

-1 :x:
shadedclient
2m  3s

patch has errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m 52s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 38s

The patch does not generate ASF License warnings.

63m  4s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4753/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4753

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint xmllint

uname
Linux 44800ade0261 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 176a7f1d03134711fee9fe309180dd393168a349

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4753/4/testReport/

Max. process+thread count
264 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4753/4/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 58s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
37m 56s

trunk passed

+1 :green_heart:
compile
0m 54s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
0m 54s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
0m 45s

trunk passed

+1 :green_heart:
mvnsite
0m 56s

trunk passed

+1 :green_heart:
javadoc
0m 39s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 42s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
1m 29s

trunk passed

+1 :green_heart:
shadedclient
21m  3s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 40s

the patch passed

+1 :green_heart:
compile
0m 43s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javac
0m 43s

the patch passed

+1 :green_heart:
compile
0m 37s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
0m 37s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 23s

the patch passed

+1 :green_heart:
mvnsite
0m 38s

the patch passed

+1 :green_heart:
javadoc
0m 19s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 28s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
1m 14s

the patch passed

+1 :green_heart:
shadedclient
20m 34s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
3m  0s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 43s

The patch does not generate ASF License warnings.

96m 50s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4753/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4753

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint xmllint

uname
Linux 2ce4a0aaf86b 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / ddd85fb2188dd773cab52b7d21a27bd46fd8450f

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4753/5/testReport/

Max. process+thread count
662 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4753/5/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Is there anything left to do on this PR @steveloughran ?


fix(content-docs): suppress git error on multiple occurrences

Motivation
https://github.com/facebook/docusaurus/pull/6937#discussion_r833133765

Have you read the Contributing Guidelines on pull requests?

Test Plan
I wrote tests.

Related PRs
https://github.com/facebook/docusaurus/pull/6937
https://github.com/cardano-foundation/developer-portal/pull/557

I created the temp folder and changed the temporary files to there (with a .gitignore) because I noticed that when the test fails, the temporary file may be left there.

Thanks, that looks nice ;)


bump rustc to 1.73

This matches the RUST_VERSION used in the pull request CI pipeline.

@bacongobbler apologies, I missed this PR.  This can be closed per https://github.com/fermyon/platform-plugin/pull/20 but I have one more follow-up in https://github.com/fermyon/platform-plugin/pull/21.  Then the release pipeline should finally be happy 😅


Fix custom markings

The main fix in this PR is for custom markings.  The changes which had been made to @CustomMarking to support the new extension-based custom markings are backed out.  The new way to do this is to create a normal extension via @CustomExtension and use the extension with a marking definition, the same way as for any other object.  @CustomMarking is now only used for the old-style custom markings.
Other misc changes include:

Update the extensions jupyter notebook to correct documentation w.r.t. the above changes
Update unit tests for custom markings w.r.t. the above changes
Update extension registration to check toplevel property names in the same way as nested property names
Add a unit test for an interesting oddball case where an extension can enable itself.  The library supports this for now.

Codecov Report

Merging #521 (295037f) into master (10956d3) will decrease coverage by 0.21%.
The diff coverage is 93.61%.

@@            Coverage Diff             @@
##           master     #521      +/-   ##
==========================================
- Coverage   87.20%   86.98%   -0.22%
==========================================
  Files         153      153
  Lines       17911    17523     -388
==========================================
- Hits        15619    15243     -376
+ Misses       2292     2280      -12

Impacted Files
Coverage Δ

stix2/v21/common.py
46.42% <0.00%> (-5.19%)
:arrow_down:

stix2/custom.py
87.34% <66.66%> (+1.79%)
:arrow_up:

stix2/registration.py
78.66% <83.33%> (+0.58%)
:arrow_up:

stix2/test/v21/test_custom.py
100.00% <100.00%> (ø)

stix2/v21/observables.py
36.00% <0.00%> (-0.32%)
:arrow_down:

stix2/v21/sdo.py
60.73% <0.00%> (-0.21%)
:arrow_down:

stix2/test/test_properties.py
99.50% <0.00%> (-0.03%)
:arrow_down:

stix2/test/v20/test_datastore_filesystem.py
99.20% <0.00%> (-0.02%)
:arrow_down:

stix2/test/v21/test_datastore_filesystem.py
99.22% <0.00%> (-0.02%)
:arrow_down:

stix2/test/test_hashes.py
100.00% <0.00%> (ø)

... and 76 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 10956d3...295037f. Read the comment docs.


Crash due to node name starting with number

Hi
I noticed that if the node xform names start with a number the exporter crashes inside usd with an illformed sdfPath.
Warning: in _InitWithString at line 106 of c:\users\d_chr\downloads\usd-master\pxr\usd\lib\sdf\path.cpp -- Ill-formed SdfPath </root/00095036_LACK_N_cff_tbl_90x55_white001_1_0>: syntax error
This is the section in the gltf file,notice the name of each xform.
"nodes":[{"mesh":0,"translation":[-0.00169372559,0.0,0.00169372559],"rotation":[0.0,0.0,0.0,-1.0],"scale":[1.0,1.0,1.0],"name":"00095036-LACK_N_cff_tbl_90x55_white001:1"},{"mesh":1,"translation":[-0.00169372559,0.0,0.00169372559],"rotation":[0.0,0.0,0.0,-1.0],"scale":[1.0,1.0,1.0],"name":"00095036-LACK_N_cff_tbl_90x55_white001:2"},{"mesh":2,"translation":[-0.00169372559,0.0,0.00169372559],"rotation":[0.0,0.0,0.0,-1.0],"scale":[1.0,1.0,1.0],"name":"00095036-LACK_N_cff_tbl_90x55_white001:3"}]
The error manifests in gltf2usd.py in line 95. in  _convert_node_to_xform
xformPrim = UsdGeom.Xform.Define(self.stage, '{0}/{1}'.format(usd_xform.GetPath(), GLTF2USDUtils.convert_to_usd_friendly_node_name(node.get_name())))
Just by changing the xform name with starting a letter no error is thrown. This must be a limitation of the usd sdk it can not process xform node names that start with a number. For example this passes. note the character before the name.
"nodes":[{"mesh":0,"translation":[-0.00169372559,0.0,0.00169372559],"rotation":[0.0,0.0,0.0,-1.0],"scale":[1.0,1.0,1.0],"name":"d00095036-LACK_N_cff_tbl_90x55_white001:1"},{"mesh":1,"translation":[-0.00169372559,0.0,0.00169372559],"rotation":[0.0,0.0,0.0,-1.0],"scale":[1.0,1.0,1.0],"name":"d00095036-LACK_N_cff_tbl_90x55_white001:2"},{"mesh":2,"translation":[-0.00169372559,0.0,0.00169372559],"rotation":[0.0,0.0,0.0,-1.0],"scale":[1.0,1.0,1.0],"name":"d00095036-LACK_N_cff_tbl_90x55_white001:3"}]
Thanks
Dimi

I had the same issue and tried to make a substitution in the former commit. https://github.com/kcoley/gltf2usd/commit/396ea0b0d70cf83f0a5b2776c4eee0ab2fcd5ea8

Hi
i added a substitution code fix in gltf2usdUtils.py in convert_to_usd_friendly_node_name
name = re.sub(r'\.|\b \b|-\b|:|\(|\)|[ \t]|-', '_', name) # replace '.',' ','-',':','/','\','(',')' and ':' with '_'
if str(name[0]).isdigit():
   name = "d"+name;

        return re.sub('//', '/', name)

and it processed.

Sorry, when merging my changes, it overwrote @sulydeni 's fix.  I'll add it to gltf2usdUtils


Support yofi.toml as config filename

Allows editors to more easily syntax highlight. Didn't open an issue beforehand since this feature is so minor.

Thanks for your contribution, however I would really prefer avoiding that patch as the config path becomes more obscure, e.g having both yofi.config & yofi.toml leads to a troubles in figuring out which one is used. I agree the toml extension would be clearer choice but having two implicit defaults or changing one completely lacks of good motivation so far.
Besides, I suppose 99% of editors allows to specify language/syntax for arbitrary file extensions, doesn't it?


Startup and verification workflow is not well documented

This is a bind long-winded, but I decided to roll these thoughts into a single ticket since they mostly address the same point from different angles.
I'm happy to provide some documentation PRs if/when I get pantalaimon working for myself and understand these missing bits.
Also related: https://github.com/matrix-org/pantalaimon/issues/155
TL/DR
As a Matrix end user (not familiar with Matrix internals, but able to run a UNIX daemon in general), the steps to take after installing Pantalaimon and creating my config file are not very clear.
How do I get to a point where pantalaimon is en/decrypting my messages, and my encrypted message history is visible in my client?
Missing step-by-step instructions
The "Usage" chapter in the README and/or the man pages should answer questions like:

When first starting and connecting to the daemon, do I need to take steps in panctl - if so, what and in which order?
Are there alternatives panctl (see DBUS/UI comments below)?
Do I need to take steps in my existing Matrix clients - if so, what and in which order?

The animated gif at the top or the README alludes to some kind of workflow, but obviously an animation like this is impossible to follow.
DBUS / UI?
This paragraph in README.md is confusing:

Pantalaimon contains a dbus based UI that can be used to control the daemon. The dbus based UI is completely optional and needs to be installed with the daemon:
pip install pantalaimon[ui]

Does this simply mean that installing pantalaimon[ui] makes pantalaimon expose a DBUS service that panctl can connect to? Then it's not a UI, it's an API.
Why is it optional? AFAIK verifying your login is necessary in order to use any encryption features - what is the alternative way to handle the verification?

What does the "If user interaction is required pantalaimon will send out OS notifications which the user can react to." part in the pantalaimon(8) man page mean in practice?

Desktop notifications that will appear in KDE/GNOME/etc?
Or just the DBUS communication with panctl?

panctl man page
The man page for panctl only covers syntax, not semantics or purpose:

What does each of these commands do? This should be put into adequate context for an end user.
What do each of the arguments mean?

What is a pan-user? How do I know what my pan-user is? (The pantalaimon(8) man page briefly defines the term, but not where to find the value)
Same for user_id
device_id seems obvious, but which device's ID in a given situation?

I just got done with an attempt to setup alphapapa/ement.el.
I had to get the inital verivications using panctl with trial-and-error but the process was like this

Open Element and request to verify the pantalaimon session
Run panctl
Run start-verification @user:domain.tld @user:domain.tld device_id where the device_id is found in the privacy settings in the Element settings
Match the emojis
Run confirm-verification @user:domain.tld @user:domain.tld device_id

And I still have the issue where Emacs has Unable to decrypt: The sender's device has not sent us the keys for this message everywhere although I connected as document in the ement.el documentation.
Also sending requires you to run send-anyways @user:domain.tld !STUFF:matrix.org and I only managed to do this by hitting tab.

@adham-omran Did you ever get this working? Trying to setup right now and start-verification is doing nothing.

And I still have the issue where Emacs has Unable to decrypt: The sender's device has not sent us the keys for this message everywhere although I connected as document in the ement.el documentation.

I don't use Pantalaimon myself, but from what I've gathered, verification is only one of the required steps.  You also need to send/receive the appropriate keys.
And we really, really need these procedures to be documented, correctly, in one place, for all users.  I've watched so many users stumble through the process, but it seems that none of them will take a few moments to write down what they did to save the next person the trouble.

I just set it up. Here are the steps for Linux, as of 2024-02-18, using pantalaimon-0.10.5 installed from pip:

Ensure you have an OS keyring installed. In my case I installed gnome-keyring. You may also want a GUI like seahorse to inspect the keyring. (pantalaimon will work without a keyring but your client will have to log in with the password every time pantalaimon is restarted, instead of being able to reuse the access token from the previous successful login.)

Create ~/.config/pantalaimon/pantalaimon.conf as the README says, except that the Homeserver = URL should use port 443, not port 8448 as the README says. (Pantalaimon is a client, not a federating server, so 443 is the correct port.)

In case you have prior experiments, clean the slate by deleting the ~/.local/share/pantalaimon directory.

Start pantalaimon.

Connect a client to the ListenAddress:ListenPort you specified in pantalaimon.conf, eg to <IP_ADDRESS>:8009, using the same username and password you would've used to login to your homeserver directly.

The login should succeed, but at this point all encrypted messages will fail to decrypt. This is fine.

Start another client that you were already using for your encrypted chats previously. In my case this was app.element.io, so the rest of the steps here assume that.

Run panctl. At the prompt, run start-verification <user ID> <user ID> <Element's device ID>. <user ID> here is the full user ID like @arnavion:arnavion.dev. If you only have the one Element session, panctl will show you the device ID as an autocomplete hint so you don't have to look it up. If you do need to look it up, go to Element -> profile icon -> All Settings -> Sessions, expand the "Current session" item, and the "Session ID" is the device ID.

panctl will print emojis. In Element you will see a popup with matching emojis. Click the "They match" button. It will now change to a popup like "Waiting for other client to confirm..."

In panctl, run confirm-verification <user ID> <user ID> <Element's device ID>, ie the same command as before but with confirm-verification instead of start-verification.

At this point, if you look at all your sessions in Element (profile icon -> All Settings -> Sessions), you should see "pantalaimon" in the "Other sessions" list as a "Verified" session.

Export the E2E room keys that Element was using via profile icon -> Security & Privacy -> Export E2E room keys. Pick any password and then save the file to some path.

Back in panctl, run import-keys <user ID> <path of file> <password you used to encrypt the file>

I restarted pantalaimon here for good measure. I don't think it's necessary.

Close and restart the client you had used in step 5, ie the one you want to connect to pantalaimon. Now, finally, you should be able to see the encrypted chats be decrypted.

Delete the E2E room keys backup file from step 12. You don't need it any more.

If you have other unverified sessions from pantalaimon, you can sign out of them too.

You will probably have to repeat steps 12-16 any time you start a new encrypted chat in Element.

@Arnavion Thanks, that's a very concise step-by-step guide!

This is fixed thanks to @Arnavion's comment which has now been added to the README.


sope: 5.5.0 -> 5.5.1

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/sope/versions.
meta.description for sope is: "An extensive set of frameworks which form a complete Web application server environment"
meta.homepage for sope is: "https://github.com/inverse-inc/sope"
meta.changelog for sope is: ""
Updates performed

Version update

To inspect upstream changes

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
Warning: no invocation of /nix/store/l143iyc0fz2bs1szwwf494jm1z0q16h8-sope-5.5.1/bin/load-EOAdaptor had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/l143iyc0fz2bs1szwwf494jm1z0q16h8-sope-5.5.1/bin/wod had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/l143iyc0fz2bs1szwwf494jm1z0q16h8-sope-5.5.1/bin/connect-EOAdaptor had a zero exit code or showed the expected version
0 of 3 passed binary check by having a zero exit code.
0 of 3 passed binary check by having the new version present in output.
found 5.5.1 with grep in /nix/store/l143iyc0fz2bs1szwwf494jm1z0q16h8-sope-5.5.1
found 5.5.1 in filename of file in /nix/store/l143iyc0fz2bs1szwwf494jm1z0q16h8-sope-5.5.1

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

2 x86_64-linux rebuild(s)
0 x86_64-darwin rebuild(s)

First fifty rebuilds by attrpath
sogo
sope

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/l143iyc0fz2bs1szwwf494jm1z0q16h8-sope-5.5.1 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A sope https://github.com/r-ryantm/nixpkgs/archive/065cab8b661425b0c80036bfcb4b4d350ca7c15b.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/l143iyc0fz2bs1szwwf494jm1z0q16h8-sope-5.5.1
ls -la /nix/store/l143iyc0fz2bs1szwwf494jm1z0q16h8-sope-5.5.1/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

2 packages built:

sogo
sope

Maintainer pings
cc @ajs124 @dasJ for testing.

@jonringer sogo and sope should be bumped together. For the last few updates, I've pushed the sogo update to the r-ryantm branch and merged it afterwards.


tweaks.no_sneak_bypass

unsure if a block besides magma is affected.
but the same reasoning probably applies.

If this were Alpha, I'd complain that patching this method will make farmland trample when sneaking, but quite a while ago now that got changed to only happen when falling onto farmland.

If this were Alpha, I'd complain that patching this method will make farmland trample when sneaking, but quite a while ago now that got changed to only happen when falling onto farmland.