Getting error /var/scratch/fetch_and_run.sh: Is a directory soon after launch of workflow

Starting the workflow
Running the workflow as follows:
$ agc workflow run read --args workflow_inputs/read.inputs.json --context ctx1

Which yields
2021-09-29T15:53:37+10:00 𝒊   Running workflow. Workflow name: 'read', Arguments: 'workflow_inputs/read.inputs.json', Context: 'ctx1'
47c28bec-58b1-4aaf-abab-b7a011944e5b

Checking the status
Checking the status with
$ agc workflow status --context-name ctx1 --workflow-name read --run-id 47c28bec-58b1-4aaf-abab-b7a011944e5b --verbose

yields
2021-09-29T16:14:46+10:00 𝒊   Showing workflow run(s). Max Runs: 20
2021-09-29T16:14:46+10:00 ↓  EstablishWesConnection(https://ghe0nhjr4c.execute-api.ap-southeast-2.amazonaws.com/prod/ga4gh/wes/v1)
2021-09-29T16:14:46+10:00 ↓  Ping WES endpoint until we get an answer:
2021-09-29T16:14:46+10:00 ↓  Attempt 1 of 3
2021-09-29T16:14:46+10:00 ↓  Connected to WES endpoint
WORKFLOWINSTANCE        ctx1    47c28bec-58b1-4aaf-abab-b7a011944e5b    true    EXECUTOR_ERROR  2021-09-29T05:53:39Z    read

Checking the logs
$ agc logs workflow read --run 47c28bec-58b1-4aaf-abab-b7a011944e5b --verbose

which yields
2021-09-29T16:18:49+10:00 𝒊   Showing the logs for 'read'
2021-09-29T16:18:49+10:00 ↓  Showing logs for workflow run '47c28bec-58b1-4aaf-abab-b7a011944e5b'
2021-09-29T16:18:49+10:00 ↓  EstablishWesConnection(https://ghe0nhjr4c.execute-api.ap-southeast-2.amazonaws.com/prod/ga4gh/wes/v1)
2021-09-29T16:18:49+10:00 ↓  Ping WES endpoint until we get an answer:
2021-09-29T16:18:49+10:00 ↓  Attempt 1 of 3
2021-09-29T16:18:49+10:00 ↓  Connected to WES endpoint
Wed, 29 Sep 2021 15:55:42 +1000 /bin/bash: /var/scratch/fetch_and_run.sh: Is a directory

I've found the following related issues:

https://github.com/broadinstitute/cromwell/issues/4687
https://github.com/aws-samples/aws-genomics-workflows/issues/142

Hi @alexiswl ,
This is a symptom of the underlying EC2 instance not being provisioned correctly when it is launched. Provisioning logs are sent to CloudWatch to enable debugging in such a scenario. If you know the instance-id, a you can get the provisioning logs at the following log-group:
/aws/ecs/container-instance/agc

in the log-stream:
/aws/ecs/container-instanc/agc/<instance-id>/cloud-init-output.log

Thank you, I've seen this a few times now so next time it happens I'll checkout the the log and provide with more information.

Don't seem to be able to make sense of this:
...
+ printenv
--
INSTALLED_ARTIFACTS_S3_ROOT_URL=s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts
OS=4.14.243-185.433.amzn2.x86_64
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
PWD=/opt
ARTIFACTS_NAMESPACE=agc
SHLVL=3
_=/usr/bin/printenv
+ [[ 4.14.243-185.433.amzn2.x86_64 =~ amzn1 ]]
+ [[ 4.14.243-185.433.amzn2.x86_64 =~ amzn2 ]]
+ systemctl enable amazon-ssm-agent
+ systemctl start amazon-ssm-agent
Job for amazon-ssm-agent.service canceled.
Sep 30 00:23:02 cloud-init[3887]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/runcmd [1]
Sep 30 00:23:02 cloud-init[3887]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
Sep 30 00:23:02 cloud-init[3887]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
ci-info: no authorized ssh keys fingerprints found for user ec2-user.
Cloud-init v. 19.3-44.amzn2 finished at Thu, 30 Sep 2021 00:23:02 +0000. Datasource DataSourceEc2.  Up 30.46 seconds

systemctl start amazon-ssm-agent is called by provision.sh (from s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts/ecs-additions/provision.sh
Having had a look at the logs (even with termination protection enabled the ec2 instance still shuts down after failing to launch correctly so I needed to be quick), with one of the instances, I jumped in and tried starting the ssm agent myself and everything seemed to be fine. Could this be a race condition that's causing some problems?
[root@ip-10-0-105-184 ~]# systemctl start amazon-ssm-agent
[root@ip-10-0-105-184 ~]# echo $?

Cloud init output log

Click to expand!
<html>
<body>
<!--StartFragment-->

Cloud-init v. 19.3-44.amzn2 running 'init-local' at Tue, 02 Nov 2021 05:07:11 +0000. Up 3.35 seconds.
--
Cloud-init v. 19.3-44.amzn2 running 'init' at Tue, 02 Nov 2021 05:07:11 +0000.  Up 5.37 seconds.
Loaded plugins: priorities, update-motd, upgrade-helper --> glibc-common-2.26-48.amzn2.x86_64 from installed removed (updateinfo) --> glibc-2.26-55.amzn2.x86_64 from amzn2-core removed (updateinfo) --> glibc-common-2.26-55.amzn2.x86_64 from amzn2-core removed (updateinfo) --> docker-20.10.7-1.amzn2.x86_64 from installed removed (updateinfo) --> libblkid-2.30.2-2.amzn2.0.4.x86_64 from installed removed (updateinfo) --> systemd-sysv-219-78.amzn2.0.15.x86_64 from amzn2-core removed (updateinfo) --> grubby-8.28-23.amzn2.0.3.x86_64 from amzn2-core removed (updateinfo) --> 1:grub2-tools-efi-2.06-2.amzn2.0.6.x86_64 from amzn2-core removed (updateinfo) --> 7:device-mapper-libs-1.02.146-4.amzn2.0.2.x86_64 from installed removed (updateinfo) --> curl-7.76.1-4.amzn2.0.1.x86_64 from installed removed (updateinfo) --> runc-1.0.0-2.amzn2.x86_64 from amzn2extra-ecs removed (updateinfo) --> glibc-all-langpacks-2.26-55.amzn2.x86_64 from amzn2-core removed (updateinfo) --> openldap-2.4.44-23.amzn2.0.1.x86_64 from installed removed (updateinfo) --> glibc-all-langpacks-2.26-48.amzn2.x86_64 from installed removed (updateinfo) --> 1:grub2-common-2.06-2.amzn2.0.6.noarch from amzn2-core removed (updateinfo) --> containerd-1.4.6-3.amzn2.x86_64 from amzn2extra-ecs removed (updateinfo) --> systemd-libs-219-78.amzn2.0.15.x86_64 from amzn2-core removed (updateinfo) --> runc-1.0.0-1.amzn2.x86_64 from installed removed (updateinfo) --> glibc-locale-source-2.26-48.amzn2.x86_64 from installed removed (updateinfo) --> libuuid-2.30.2-2.amzn2.0.4.x86_64 from installed removed (updateinfo) --> systemd-219-78.amzn2.0.14.x86_64 from installed removed (updateinfo) --> libfdisk-2.30.2-2.amzn2.0.4.x86_64 from installed removed (updateinfo) --> libcrypt-2.26-48.amzn2.x86_64 from installed removed (updateinfo) --> 1:grub2-pc-modules-2.06-2.amzn2.0.3.noarch from installed removed (updateinfo) --> libblkid-2.30.2-2.amzn2.0.5.x86_64 from amzn2-core removed (updateinfo) --> libmount-2.30.2-2.amzn2.0.5.x86_64 from amzn2-core removed (updateinfo) --> 1:grub2-2.06-2.amzn2.0.6.x86_64 from amzn2-core removed (updateinfo) --> glibc-minimal-langpack-2.26-55.amzn2.x86_64 from amzn2-core removed (updateinfo) --> 1:grub2-common-2.06-2.amzn2.0.3.noarch from installed removed (updateinfo) --> 1:grub2-tools-minimal-2.06-2.amzn2.0.3.x86_64 from installed removed (updateinfo) --> 1:grub2-efi-x64-ec2-2.06-2.amzn2.0.3.x86_64 from installed removed (updateinfo) --> 1:grub2-pc-modules-2.06-2.amzn2.0.6.noarch from amzn2-core removed (updateinfo) --> libsmartcols-2.30.2-2.amzn2.0.4.x86_64 from installed removed (updateinfo) --> 1:openssl-libs-1.0.2k-19.amzn2.0.8.x86_64 from amzn2-core removed (updateinfo) --> 1:openssl-libs-1.0.2k-19.amzn2.0.7.x86_64 from installed removed (updateinfo) --> 7:device-mapper-libs-1.02.170-6.amzn2.5.x86_64 from amzn2-core removed (updateinfo) --> 1:grub2-tools-2.06-2.amzn2.0.3.x86_64 from installed removed (updateinfo) --> libcurl-7.76.1-4.amzn2.0.1.x86_64 from installed removed (updateinfo) --> 1:grub2-tools-extra-2.06-2.amzn2.0.6.x86_64 from amzn2-core removed (updateinfo) --> systemd-sysv-219-78.amzn2.0.14.x86_64 from installed removed (updateinfo) --> glibc-minimal-langpack-2.26-48.amzn2.x86_64 from installed removed (updateinfo) --> 1:grub2-tools-minimal-2.06-2.amzn2.0.6.x86_64 from amzn2-core removed (updateinfo) --> amazon-linux-extras-2.0.1-1.amzn2.noarch from amzn2-core removed (updateinfo) --> util-linux-2.30.2-2.amzn2.0.4.x86_64 from installed removed (updateinfo) --> glib2-2.56.1-9.amzn2.0.2.x86_64 from amzn2-core removed (updateinfo) --> libcurl-7.76.1-7.amzn2.0.2.x86_64 from amzn2-core removed (updateinfo) --> 1:grub2-tools-2.06-2.amzn2.0.6.x86_64 from amzn2-core removed (updateinfo) --> 1:grub2-2.06-2.amzn2.0.3.x86_64 from installed removed (updateinfo) --> 1:grub2-pc-2.06-2.amzn2.0.3.x86_64 from installed removed (updateinfo) --> libuuid-2.30.2-2.amzn2.0.5.x86_64 from amzn2-core removed (updateinfo) --> grubby-8.28-23.amzn2.0.2.x86_64 from installed removed (updateinfo) --> systemd-219-78.amzn2.0.15.x86_64 from amzn2-core removed (updateinfo) --> 1:grub2-efi-x64-ec2-2.06-2.amzn2.0.6.x86_64 from amzn2-core removed (updateinfo) --> glibc-2.26-48.amzn2.x86_64 from installed removed (updateinfo) --> util-linux-2.30.2-2.amzn2.0.5.x86_64 from amzn2-core removed (updateinfo) --> 1:openssl-1.0.2k-19.amzn2.0.7.x86_64 from installed removed (updateinfo) --> 1:openssl-1.0.2k-19.amzn2.0.8.x86_64 from amzn2-core removed (updateinfo) --> docker-20.10.7-3.amzn2.x86_64 from amzn2extra-ecs removed (updateinfo) --> curl-7.76.1-7.amzn2.0.2.x86_64 from amzn2-core removed (updateinfo) --> systemd-libs-219-78.amzn2.0.14.x86_64 from installed removed (updateinfo) --> amazon-linux-extras-2.0.0-1.amzn2.noarch from installed removed (updateinfo) --> glib2-2.56.1-9.amzn2.0.1.x86_64 from installed removed (updateinfo) --> libfdisk-2.30.2-2.amzn2.0.5.x86_64 from amzn2-core removed (updateinfo) --> 7:device-mapper-1.02.146-4.amzn2.0.2.x86_64 from installed removed (updateinfo) --> libmount-2.30.2-2.amzn2.0.4.x86_64 from installed removed (updateinfo) --> glibc-locale-source-2.26-55.amzn2.x86_64 from amzn2-core removed (updateinfo) --> openldap-2.4.44-23.amzn2.0.2.x86_64 from amzn2-core removed (updateinfo) --> 1:grub2-pc-2.06-2.amzn2.0.6.x86_64 from amzn2-core removed (updateinfo) --> libcrypt-2.26-55.amzn2.x86_64 from amzn2-core removed (updateinfo) --> containerd-1.4.6-2.amzn2.x86_64 from installed removed (updateinfo) --> 7:device-mapper-1.02.170-6.amzn2.5.x86_64 from amzn2-core removed (updateinfo) --> libsmartcols-2.30.2-2.amzn2.0.5.x86_64 from amzn2-core removed (updateinfo)
1 package(s) needed (+0 related) for security, out of 38 available
Resolving Dependencies
--> Running transaction check
---> Package ca-certificates.noarch 0:2020.2.41-70.0.amzn2.0.1 will be updated
---> Package ca-certificates.noarch 0:2021.2.50-72.amzn2.0.1 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================ Package             Arch       Version                    Repository      Size
================================================================================
Updating: ca-certificates     noarch     2021.2.50-72.amzn2.0.1     amzn2-core     380 k
Transaction Summary
================================================================================
Upgrade  1 Package
Total download size: 380 k
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction  Updating   : ca-certificates-2021.2.50-72.amzn2.0.1.noarch                1/2   Cleanup    : ca-certificates-2020.2.41-70.0.amzn2.0.1.noarch              2/2   Verifying  : ca-certificates-2021.2.50-72.amzn2.0.1.noarch                1/2   Verifying  : ca-certificates-2020.2.41-70.0.amzn2.0.1.noarch              2/2
Updated:  ca-certificates.noarch 0:2021.2.50-72.amzn2.0.1
Complete!
Loaded plugins: priorities, update-motd, upgrade-helper
Package sed-4.2.2-5.amzn2.0.2.x86_64 already installed and latest version
Package amazon-ssm-agent-3.0.1124.0-1.amzn2.x86_64 already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package amazon-cloudwatch-agent.x86_64 0:1.247347.4-1.amzn2 will be installed
---> Package btrfs-progs.x86_64 0:4.15.1-1.amzn2.0.1 will be installed
--> Processing Dependency: libzstd.so.1()(64bit) for package: btrfs-progs-4.15.1-1.amzn2.0.1.x86_64
--> Processing Dependency: liblzo2.so.2()(64bit) for package: btrfs-progs-4.15.1-1.amzn2.0.1.x86_64
---> Package git.x86_64 0:2.32.0-1.amzn2.0.1 will be installed
--> Processing Dependency: perl-Git = 2.32.0-1.amzn2.0.1 for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: git-core-doc = 2.32.0-1.amzn2.0.1 for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: git-core = 2.32.0-1.amzn2.0.1 for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl >= 5.008 for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: emacs-filesystem >= 27.1 for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(warnings) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(strict) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(lib) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(Term::ReadKey) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(IPC::Open2) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(Git::I18N) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(Git) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(Getopt::Long) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(File::Spec) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(File::Find) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(File::Basename) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: /usr/bin/perl for package: git-2.32.0-1.amzn2.0.1.x86_64
---> Package jq.x86_64 0:1.5-1.amzn2.0.2 will be installed
--> Processing Dependency: libonig.so.2()(64bit) for package: jq-1.5-1.amzn2.0.2.x86_64
---> Package unzip.x86_64 0:6.0-43.amzn2 will be installed
--> Running transaction check
---> Package emacs-filesystem.noarch 1:27.2-4.amzn2.0.1 will be installed
---> Package git-core.x86_64 0:2.32.0-1.amzn2.0.1 will be installed
---> Package git-core-doc.noarch 0:2.32.0-1.amzn2.0.1 will be installed
---> Package libzstd.x86_64 0:1.3.3-1.amzn2.0.1 will be installed
---> Package lzo.x86_64 0:2.06-8.amzn2.0.4 will be installed
---> Package oniguruma.x86_64 0:5.9.6-1.amzn2.0.4 will be installed
---> Package perl.x86_64 4:5.16.3-299.amzn2.0.1 will be installed
--> Processing Dependency: perl-libs = 4:5.16.3-299.amzn2.0.1 for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Socket) >= 1.3 for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Scalar::Util) >= 1.10 for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl-macros for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl-libs for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(threads::shared) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(threads) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(constant) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Time::Local) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Time::HiRes) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Storable) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Socket) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Scalar::Util) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Pod::Simple::XHTML) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Pod::Simple::Search) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Filter::Util::Call) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(File::Temp) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(File::Path) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Exporter) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Carp) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: libperl.so()(64bit) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
---> Package perl-Getopt-Long.noarch 0:2.40-3.amzn2 will be installed
--> Processing Dependency: perl(Pod::Usage) >= 1.14 for package: perl-Getopt-Long-2.40-3.amzn2.noarch
--> Processing Dependency: perl(Text::ParseWords) for package: perl-Getopt-Long-2.40-3.amzn2.noarch
---> Package perl-Git.noarch 0:2.32.0-1.amzn2.0.1 will be installed
--> Processing Dependency: perl(Error) for package: perl-Git-2.32.0-1.amzn2.0.1.noarch
---> Package perl-PathTools.x86_64 0:3.40-5.amzn2.0.2 will be installed
---> Package perl-TermReadKey.x86_64 0:2.30-20.amzn2.0.2 will be installed
--> Running transaction check
---> Package perl-Carp.noarch 0:1.26-244.amzn2 will be installed
---> Package perl-Error.noarch 1:0.17020-2.amzn2 will be installed
---> Package perl-Exporter.noarch 0:5.68-3.amzn2 will be installed
---> Package perl-File-Path.noarch 0:2.09-2.amzn2 will be installed
---> Package perl-File-Temp.noarch 0:0.23.01-3.amzn2 will be installed
---> Package perl-Filter.x86_64 0:1.49-3.amzn2.0.2 will be installed
---> Package perl-Pod-Simple.noarch 1:3.28-4.amzn2 will be installed
--> Processing Dependency: perl(Pod::Escapes) >= 1.04 for package: 1:perl-Pod-Simple-3.28-4.amzn2.noarch
--> Processing Dependency: perl(Encode) for package: 1:perl-Pod-Simple-3.28-4.amzn2.noarch
---> Package perl-Pod-Usage.noarch 0:1.63-3.amzn2 will be installed
--> Processing Dependency: perl(Pod::Text) >= 3.15 for package: perl-Pod-Usage-1.63-3.amzn2.noarch
--> Processing Dependency: perl-Pod-Perldoc for package: perl-Pod-Usage-1.63-3.amzn2.noarch
---> Package perl-Scalar-List-Utils.x86_64 0:1.27-248.amzn2.0.2 will be installed
---> Package perl-Socket.x86_64 0:2.010-4.amzn2.0.2 will be installed
---> Package perl-Storable.x86_64 0:2.45-3.amzn2.0.2 will be installed
---> Package perl-Text-ParseWords.noarch 0:3.29-4.amzn2 will be installed
---> Package perl-Time-HiRes.x86_64 4:1.9725-3.amzn2.0.2 will be installed
---> Package perl-Time-Local.noarch 0:1.2300-2.amzn2 will be installed
---> Package perl-constant.noarch 0:1.27-2.amzn2.0.1 will be installed
---> Package perl-libs.x86_64 4:5.16.3-299.amzn2.0.1 will be installed
---> Package perl-macros.x86_64 4:5.16.3-299.amzn2.0.1 will be installed
---> Package perl-threads.x86_64 0:1.87-4.amzn2.0.2 will be installed
---> Package perl-threads-shared.x86_64 0:1.43-6.amzn2.0.2 will be installed
--> Running transaction check
---> Package perl-Encode.x86_64 0:2.51-7.amzn2.0.2 will be installed
---> Package perl-Pod-Escapes.noarch 1:1.04-299.amzn2.0.1 will be installed
---> Package perl-Pod-Perldoc.noarch 0:3.20-4.amzn2 will be installed
--> Processing Dependency: perl(parent) for package: perl-Pod-Perldoc-3.20-4.amzn2.noarch
--> Processing Dependency: perl(HTTP::Tiny) for package: perl-Pod-Perldoc-3.20-4.amzn2.noarch
---> Package perl-podlators.noarch 0:2.5.1-3.amzn2.0.1 will be installed
--> Running transaction check
---> Package perl-HTTP-Tiny.noarch 0:0.033-3.amzn2 will be installed
---> Package perl-parent.noarch 1:0.225-244.amzn2.0.1 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================ Package                   Arch     Version                  Repository    Size
================================================================================
Installing: amazon-cloudwatch-agent   x86_64   1.247347.4-1.amzn2       amzn2-core    41 M btrfs-progs               x86_64   4.15.1-1.amzn2.0.1       amzn2-core   771 k git                       x86_64   2.32.0-1.amzn2.0.1       amzn2-core   126 k jq                        x86_64   1.5-1.amzn2.0.2          amzn2-core   154 k unzip                     x86_64   6.0-43.amzn2             amzn2-core   179 k
Installing for dependencies: emacs-filesystem          noarch   1:27.2-4.amzn2.0.1       amzn2-core    67 k git-core                  x86_64   2.32.0-1.amzn2.0.1       amzn2-core   4.8 M git-core-doc              noarch   2.32.0-1.amzn2.0.1       amzn2-core   2.7 M libzstd                   x86_64   1.3.3-1.amzn2.0.1        amzn2-core   203 k lzo                       x86_64   2.06-8.amzn2.0.4         amzn2-core    60 k oniguruma                 x86_64   5.9.6-1.amzn2.0.4        amzn2-core   127 k perl                      x86_64   4:5.16.3-299.amzn2.0.1   amzn2-core   8.0 M perl-Carp                 noarch   1.26-244.amzn2           amzn2-core    19 k perl-Encode               x86_64   2.51-7.amzn2.0.2         amzn2-core   1.5 M perl-Error                noarch   1:0.17020-2.amzn2        amzn2-core    32 k perl-Exporter             noarch   5.68-3.amzn2             amzn2-core    29 k perl-File-Path            noarch   2.09-2.amzn2             amzn2-core    27 k perl-File-Temp            noarch   0.23.01-3.amzn2          amzn2-core    56 k perl-Filter               x86_64   1.49-3.amzn2.0.2         amzn2-core    76 k perl-Getopt-Long          noarch   2.40-3.amzn2             amzn2-core    56 k perl-Git                  noarch   2.32.0-1.amzn2.0.1       amzn2-core    43 k perl-HTTP-Tiny            noarch   0.033-3.amzn2            amzn2-core    38 k perl-PathTools            x86_64   3.40-5.amzn2.0.2         amzn2-core    83 k perl-Pod-Escapes          noarch   1:1.04-299.amzn2.0.1     amzn2-core    52 k perl-Pod-Perldoc          noarch   3.20-4.amzn2             amzn2-core    87 k perl-Pod-Simple           noarch   1:3.28-4.amzn2           amzn2-core   216 k perl-Pod-Usage            noarch   1.63-3.amzn2             amzn2-core    27 k perl-Scalar-List-Utils    x86_64   1.27-248.amzn2.0.2       amzn2-core    36 k perl-Socket               x86_64   2.010-4.amzn2.0.2        amzn2-core    49 k perl-Storable             x86_64   2.45-3.amzn2.0.2         amzn2-core    78 k perl-TermReadKey          x86_64   2.30-20.amzn2.0.2        amzn2-core    31 k perl-Text-ParseWords      noarch   3.29-4.amzn2             amzn2-core    14 k perl-Time-HiRes           x86_64   4:1.9725-3.amzn2.0.2     amzn2-core    45 k perl-Time-Local           noarch   1.2300-2.amzn2           amzn2-core    24 k perl-constant             noarch   1.27-2.amzn2.0.1         amzn2-core    19 k perl-libs                 x86_64   4:5.16.3-299.amzn2.0.1   amzn2-core   685 k perl-macros               x86_64   4:5.16.3-299.amzn2.0.1   amzn2-core    44 k perl-parent               noarch   1:0.225-244.amzn2.0.1    amzn2-core    12 k perl-podlators            noarch   2.5.1-3.amzn2.0.1        amzn2-core   112 k perl-threads              x86_64   1.87-4.amzn2.0.2         amzn2-core    50 k perl-threads-shared       x86_64   1.43-6.amzn2.0.2         amzn2-core    39 k
Transaction Summary
================================================================================
Install  5 Packages (+36 Dependent packages)
Total download size: 61 M
Installed size: 263 M
Downloading packages:
--------------------------------------------------------------------------------
Total                                               57 MB/s \|  61 MB  00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction  Installing : perl-HTTP-Tiny-0.033-3.amzn2.noarch                         1/41   Installing : 1:perl-parent-0.225-244.amzn2.0.1.noarch                    2/41   Installing : perl-Pod-Perldoc-3.20-4.amzn2.noarch                        3/41   Installing : perl-podlators-2.5.1-3.amzn2.0.1.noarch                     4/41   Installing : perl-Encode-2.51-7.amzn2.0.2.x86_64                         5/41   Installing : 1:perl-Pod-Escapes-1.04-299.amzn2.0.1.noarch                6/41   Installing : perl-Text-ParseWords-3.29-4.amzn2.noarch                    7/41   Installing : perl-Pod-Usage-1.63-3.amzn2.noarch                          8/41   Installing : 4:perl-macros-5.16.3-299.amzn2.0.1.x86_64                   9/41   Installing : perl-Filter-1.49-3.amzn2.0.2.x86_64                        10/41   Installing : perl-Carp-1.26-244.amzn2.noarch                            11/41   Installing : 4:perl-libs-5.16.3-299.amzn2.0.1.x86_64                    12/41   Installing : perl-constant-1.27-2.amzn2.0.1.noarch                      13/41   Installing : perl-Time-Local-1.2300-2.amzn2.noarch                      14/41   Installing : perl-PathTools-3.40-5.amzn2.0.2.x86_64                     15/41   Installing : perl-Scalar-List-Utils-1.27-248.amzn2.0.2.x86_64           16/41   Installing : perl-threads-shared-1.43-6.amzn2.0.2.x86_64                17/41   Installing : perl-threads-1.87-4.amzn2.0.2.x86_64                       18/41   Installing : 4:perl-Time-HiRes-1.9725-3.amzn2.0.2.x86_64                19/41   Installing : perl-Socket-2.010-4.amzn2.0.2.x86_64                       20/41   Installing : perl-Storable-2.45-3.amzn2.0.2.x86_64                      21/41   Installing : perl-File-Temp-0.23.01-3.amzn2.noarch                      22/41   Installing : perl-File-Path-2.09-2.amzn2.noarch                         23/41   Installing : 1:perl-Pod-Simple-3.28-4.amzn2.noarch                      24/41   Installing : perl-Getopt-Long-2.40-3.amzn2.noarch                       25/41   Installing : perl-Exporter-5.68-3.amzn2.noarch                          26/41   Installing : 4:perl-5.16.3-299.amzn2.0.1.x86_64                         27/41   Installing : git-core-2.32.0-1.amzn2.0.1.x86_64                         28/41   Installing : git-core-doc-2.32.0-1.amzn2.0.1.noarch                     29/41   Installing : perl-TermReadKey-2.30-20.amzn2.0.2.x86_64                  30/41   Installing : 1:perl-Error-0.17020-2.amzn2.noarch                        31/41   Installing : oniguruma-5.9.6-1.amzn2.0.4.x86_64                         32/41   Installing : libzstd-1.3.3-1.amzn2.0.1.x86_64                           33/41   Installing : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 34/41   Installing : perl-Git-2.32.0-1.amzn2.0.1.noarch                         35/41   Installing : git-2.32.0-1.amzn2.0.1.x86_64                              36/41   Installing : lzo-2.06-8.amzn2.0.4.x86_64                                37/41   Installing : btrfs-progs-4.15.1-1.amzn2.0.1.x86_64                      38/41   Installing : jq-1.5-1.amzn2.0.2.x86_64                                  39/41   Installing : unzip-6.0-43.amzn2.x86_64                                  40/41
create group cwagent, result: 0
create user cwagent, result: 0  Installing : amazon-cloudwatch-agent-1.247347.4-1.amzn2.x86_64          41/41   Verifying  : perl-Exporter-5.68-3.amzn2.noarch                           1/41   Verifying  : 4:perl-5.16.3-299.amzn2.0.1.x86_64                          2/41   Verifying  : perl-Filter-1.49-3.amzn2.0.2.x86_64                         3/41   Verifying  : lzo-2.06-8.amzn2.0.4.x86_64                                 4/41   Verifying  : 4:perl-macros-5.16.3-299.amzn2.0.1.x86_64                   5/41   Verifying  : perl-Pod-Usage-1.63-3.amzn2.noarch                          6/41   Verifying  : perl-Carp-1.26-244.amzn2.noarch                             7/41   Verifying  : perl-PathTools-3.40-5.amzn2.0.2.x86_64                      8/41   Verifying  : perl-HTTP-Tiny-0.033-3.amzn2.noarch                         9/41   Verifying  : 4:perl-libs-5.16.3-299.amzn2.0.1.x86_64                    10/41   Verifying  : perl-Encode-2.51-7.amzn2.0.2.x86_64                        11/41   Verifying  : amazon-cloudwatch-agent-1.247347.4-1.amzn2.x86_64          12/41   Verifying  : perl-threads-shared-1.43-6.amzn2.0.2.x86_64                13/41   Verifying  : 1:perl-Pod-Simple-3.28-4.amzn2.noarch                      14/41   Verifying  : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 15/41   Verifying  : perl-File-Temp-0.23.01-3.amzn2.noarch                      16/41   Verifying  : perl-constant-1.27-2.amzn2.0.1.noarch                      17/41   Verifying  : perl-Time-Local-1.2300-2.amzn2.noarch                      18/41   Verifying  : git-core-2.32.0-1.amzn2.0.1.x86_64                         19/41   Verifying  : perl-Scalar-List-Utils-1.27-248.amzn2.0.2.x86_64           20/41   Verifying  : 1:perl-Pod-Escapes-1.04-299.amzn2.0.1.noarch               21/41   Verifying  : perl-threads-1.87-4.amzn2.0.2.x86_64                       22/41   Verifying  : perl-podlators-2.5.1-3.amzn2.0.1.noarch                    23/41   Verifying  : perl-TermReadKey-2.30-20.amzn2.0.2.x86_64                  24/41   Verifying  : git-core-doc-2.32.0-1.amzn2.0.1.noarch                     25/41   Verifying  : perl-Git-2.32.0-1.amzn2.0.1.noarch                         26/41   Verifying  : libzstd-1.3.3-1.amzn2.0.1.x86_64                           27/41   Verifying  : 4:perl-Time-HiRes-1.9725-3.amzn2.0.2.x86_64                28/41   Verifying  : jq-1.5-1.amzn2.0.2.x86_64                                  29/41   Verifying  : 1:perl-Error-0.17020-2.amzn2.noarch                        30/41   Verifying  : perl-Socket-2.010-4.amzn2.0.2.x86_64                       31/41   Verifying  : perl-Storable-2.45-3.amzn2.0.2.x86_64                      32/41   Verifying  : unzip-6.0-43.amzn2.x86_64                                  33/41   Verifying  : oniguruma-5.9.6-1.amzn2.0.4.x86_64                         34/41   Verifying  : 1:perl-parent-0.225-244.amzn2.0.1.noarch                   35/41   Verifying  : btrfs-progs-4.15.1-1.amzn2.0.1.x86_64                      36/41   Verifying  : perl-Pod-Perldoc-3.20-4.amzn2.noarch                       37/41   Verifying  : git-2.32.0-1.amzn2.0.1.x86_64                              38/41   Verifying  : perl-Text-ParseWords-3.29-4.amzn2.noarch                   39/41   Verifying  : perl-File-Path-2.09-2.amzn2.noarch                         40/41   Verifying  : perl-Getopt-Long-2.40-3.amzn2.noarch                       41/41
Installed:  amazon-cloudwatch-agent.x86_64 0:1.247347.4-1.amzn2                             btrfs-progs.x86_64 0:4.15.1-1.amzn2.0.1                                         git.x86_64 0:2.32.0-1.amzn2.0.1                                                 jq.x86_64 0:1.5-1.amzn2.0.2                                                     unzip.x86_64 0:6.0-43.amzn2
Dependency Installed:  emacs-filesystem.noarch 1:27.2-4.amzn2.0.1                                      git-core.x86_64 0:2.32.0-1.amzn2.0.1                                            git-core-doc.noarch 0:2.32.0-1.amzn2.0.1                                        libzstd.x86_64 0:1.3.3-1.amzn2.0.1                                              lzo.x86_64 0:2.06-8.amzn2.0.4                                                   oniguruma.x86_64 0:5.9.6-1.amzn2.0.4                                            perl.x86_64 4:5.16.3-299.amzn2.0.1                                              perl-Carp.noarch 0:1.26-244.amzn2                                               perl-Encode.x86_64 0:2.51-7.amzn2.0.2                                           perl-Error.noarch 1:0.17020-2.amzn2                                             perl-Exporter.noarch 0:5.68-3.amzn2                                             perl-File-Path.noarch 0:2.09-2.amzn2                                            perl-File-Temp.noarch 0:0.23.01-3.amzn2                                         perl-Filter.x86_64 0:1.49-3.amzn2.0.2                                           perl-Getopt-Long.noarch 0:2.40-3.amzn2                                          perl-Git.noarch 0:2.32.0-1.amzn2.0.1                                            perl-HTTP-Tiny.noarch 0:0.033-3.amzn2                                           perl-PathTools.x86_64 0:3.40-5.amzn2.0.2                                        perl-Pod-Escapes.noarch 1:1.04-299.amzn2.0.1                                    perl-Pod-Perldoc.noarch 0:3.20-4.amzn2                                          perl-Pod-Simple.noarch 1:3.28-4.amzn2                                           perl-Pod-Usage.noarch 0:1.63-3.amzn2                                            perl-Scalar-List-Utils.x86_64 0:1.27-248.amzn2.0.2                              perl-Socket.x86_64 0:2.010-4.amzn2.0.2                                          perl-Storable.x86_64 0:2.45-3.amzn2.0.2                                         perl-TermReadKey.x86_64 0:2.30-20.amzn2.0.2                                     perl-Text-ParseWords.noarch 0:3.29-4.amzn2                                      perl-Time-HiRes.x86_64 4:1.9725-3.amzn2.0.2                                     perl-Time-Local.noarch 0:1.2300-2.amzn2                                         perl-constant.noarch 0:1.27-2.amzn2.0.1                                         perl-libs.x86_64 4:5.16.3-299.amzn2.0.1                                         perl-macros.x86_64 4:5.16.3-299.amzn2.0.1                                       perl-parent.noarch 1:0.225-244.amzn2.0.1                                        perl-podlators.noarch 0:2.5.1-3.amzn2.0.1                                       perl-threads.x86_64 0:1.87-4.amzn2.0.2                                          perl-threads-shared.x86_64 0:1.43-6.amzn2.0.2
Complete!
Cloud-init v. 19.3-44.amzn2 running 'modules:final' at Tue, 02 Nov 2021 05:07:31 +0000. Up 23.81 seconds.
****** processing amazon-cloudwatch-agent ******
/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source file:/opt/aws/amazon-cloudwatch-agent/etc/config.json --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp
Start configuration validation...
/opt/aws/amazon-cloudwatch-agent/bin/config-translator --input /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json --input-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --output /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
2021/11/02 05:07:31 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp ...
Valid Json input schema.
I! Detecting run_as_user...
No csm configuration found.
No metric configuration found.
Configuration validation first phase succeeded
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
Configuration validation second phase succeeded
Configuration validation succeeded
amazon-cloudwatch-agent has already been stopped
Created symlink from /etc/systemd/system/multi-user.target.wants/amazon-cloudwatch-agent.service to /etc/systemd/system/amazon-cloudwatch-agent.service.
Redirecting to /bin/systemctl restart amazon-cloudwatch-agent.service
You can now run: /usr/bin/aws --version
/usr/bin/aws
Completed 29 Bytes/17.7 KiB (31 Bytes/s) with 8 file(s) remainingdownload: s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts/ecs-additions/ecs-additions-cromwell.sh to ecs-additions/ecs-additions-cromwell.sh
Completed 29 Bytes/17.7 KiB (31 Bytes/s) with 7 file(s) remainingCompleted 486 Bytes/17.7 KiB (526 Bytes/s) with 7 file(s) remainingdownload: s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts/ecs-additions/awscli-shim.sh to ecs-additions/awscli-shim.sh
Completed 486 Bytes/17.7 KiB (526 Bytes/s) with 6 file(s) remainingCompleted 2.6 KiB/17.7 KiB (2.7 KiB/s) with 6 file(s) remaining    download: s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts/ecs-additions/provision.sh to ecs-additions/provision.sh
Completed 2.6 KiB/17.7 KiB (2.7 KiB/s) with 5 file(s) remainingCompleted 7.9 KiB/17.7 KiB (8.3 KiB/s) with 5 file(s) remainingdownload: s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts/ecs-additions/fetch_and_run.sh to ecs-additions/fetch_and_run.sh
Completed 7.9 KiB/17.7 KiB (8.3 KiB/s) with 4 file(s) remainingCompleted 9.3 KiB/17.7 KiB (9.8 KiB/s) with 4 file(s) remainingCompleted 14.6 KiB/17.7 KiB (15.3 KiB/s) with 4 file(s) remainingdownload: s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts/ecs-additions/get-amazon-ebs-autoscale.sh to ecs-additions/get-amazon-ebs-autoscale.sh
Completed 14.6 KiB/17.7 KiB (15.3 KiB/s) with 3 file(s) remainingdownload: s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts/ecs-additions/ecs-additions-cromwell-linux2-worker.sh to ecs-additions/ecs-additions-cromwell-linux2-worker.sh
Completed 14.6 KiB/17.7 KiB (15.3 KiB/s) with 2 file(s) remainingCompleted 15.9 KiB/17.7 KiB (16.5 KiB/s) with 2 file(s) remainingdownload: s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts/ecs-additions/ecs-additions-common.sh to ecs-additions/ecs-additions-common.sh
Completed 15.9 KiB/17.7 KiB (16.5 KiB/s) with 1 file(s) remainingCompleted 17.7 KiB/17.7 KiB (17.8 KiB/s) with 1 file(s) remainingdownload: s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts/ecs-additions/ecs-additions-step-functions.sh to ecs-additions/ecs-additions-step-functions.sh
++ uname -r
+ export OS=4.14.243-185.433.amzn2.x86_64
+ OS=4.14.243-185.433.amzn2.x86_64
++ dirname /opt/ecs-additions/provision.sh
+ BASEDIR=/opt/ecs-additions
+ printenv
INSTALLED_ARTIFACTS_S3_ROOT_URL=s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts
OS=4.14.243-185.433.amzn2.x86_64
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
PWD=/opt
ARTIFACTS_NAMESPACE=agc
SHLVL=3
_=/usr/bin/printenv
+ [[ 4.14.243-185.433.amzn2.x86_64 =~ amzn1 ]]
+ [[ 4.14.243-185.433.amzn2.x86_64 =~ amzn2 ]]
+ systemctl enable amazon-ssm-agent
+ systemctl start amazon-ssm-agent
Job for amazon-ssm-agent.service canceled.
Nov 02 05:07:39 cloud-init[3972]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/runcmd [1]
Nov 02 05:07:39 cloud-init[3972]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
Nov 02 05:07:39 cloud-init[3972]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
ci-info: no authorized ssh keys fingerprints found for user ec2-user.
Cloud-init v. 19.3-44.amzn2 finished at Tue, 02 Nov 2021 05:07:39 +0000. Datasource DataSourceEc2.  Up 32.22 seconds

<!--EndFragment-->
</body>
</html>

ECS Agent log

Click to expand!
<html>
<body>
<!--StartFragment-->

level=info time=2021-11-02T05:07:45Z msg="Successfully got ECS instance credentials from provider: EC2RoleProvider" module=instancecreds.go
--
level=info time=2021-11-02T05:07:45Z msg="Loading configuration" module=agent.go
level=info time=2021-11-02T05:07:45Z msg="Amazon ECS agent Version: 1.55.3, Commit: 67d8b9ab" module=agent.go
level=info time=2021-11-02T05:07:45Z msg="Successfully got ECS instance credentials from provider: EC2RoleProvider" module=instancecreds.go
level=info time=2021-11-02T05:07:45Z msg="Successfully got ECS instance credentials from provider: EC2RoleProvider" module=instancecreds.go
level=info time=2021-11-02T05:07:45Z msg="Image excluded from cleanup: amazon/amazon-ecs-pause:0.1.0" module=docker_image_manager.go
level=info time=2021-11-02T05:07:45Z msg="Image excluded from cleanup: amazon/amazon-ecs-pause:0.1.0" module=docker_image_manager.go
level=info time=2021-11-02T05:07:45Z msg="Image excluded from cleanup: amazon/amazon-ecs-agent:latest" module=docker_image_manager.go
level=info time=2021-11-02T05:07:45Z msg="Creating root ecs cgroup: /ecs" module=init_linux.go
level=info time=2021-11-02T05:07:45Z msg="Creating cgroup /ecs" module=cgroup_controller_linux.go
level=info time=2021-11-02T05:07:45Z msg="Event stream ContainerChange start listening..." module=eventstream.go
level=info time=2021-11-02T05:07:45Z msg="Loading state!" module=state_manager.go
level=info time=2021-11-02T05:07:45Z msg="eni watcher has been initialized" module=watcher_linux.go
level=info time=2021-11-02T05:07:45Z msg="Registering Instance with ECS" module=agent.go
level=info time=2021-11-02T05:07:45Z msg="Remaining mem: 7732" module=client.go
level=info time=2021-11-02T05:07:45Z msg="Registered container instance with cluster!" module=client.go
level=info time=2021-11-02T05:07:45Z msg="Registration completed successfully. I am running as 'arn:aws:ecs:ap-southeast-2:843407916570:container-instance/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/1517990654334753a65f2aba6305b319' in cluster 'TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063'" module=agent.go
level=info time=2021-11-02T05:07:45Z msg="Event stream DeregisterContainerInstance start listening..." module=eventstream.go
level=info time=2021-11-02T05:07:45Z msg="Beginning Polling for updates" module=agent.go
level=info time=2021-11-02T05:07:45Z msg="Initializing stats engine" module=engine.go
level=info time=2021-11-02T05:07:45Z msg="Pull behavior is set to always use cache. Disabling cleanup" module=docker_image_manager.go
level=info time=2021-11-02T05:07:45Z msg="Establishing a Websocket connection to https://ecs-a-3.ap-southeast-2.amazonaws.com/ws?agentHash=67d8b9ab&agentVersion=1.55.3&clusterArn=TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063&containerInstanceArn=arn%3Aaws%3Aecs%3Aap-southeast-2%3A843407916570%3Acontainer-instance%2FTaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063%2F1517990654334753a65f2aba6305b319&dockerVersion=DockerVersion%3A+20.10.7&protocolVersion=2&sendCredentials=true&seqNum=1" module=client.go
level=info time=2021-11-02T05:07:45Z msg="NO_PROXY set:<IP_ADDRESS>,<IP_ADDRESS>,/var/run/docker.sock" module=client.go
level=info time=2021-11-02T05:07:45Z msg="Establishing a Websocket connection to https://ecs-t-3.ap-southeast-2.amazonaws.com/ws?agentHash=67d8b9ab&agentVersion=1.55.3&cluster=TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063&containerInstance=arn%3Aaws%3Aecs%3Aap-southeast-2%3A843407916570%3Acontainer-instance%2FTaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063%2F1517990654334753a65f2aba6305b319&dockerVersion=20.10.7" module=client.go
level=info time=2021-11-02T05:07:46Z msg="Connected to TCS endpoint" module=handler.go
level=info time=2021-11-02T05:07:46Z msg="Connected to ACS endpoint" module=acs_handler.go
level=info time=2021-11-02T05:07:59Z msg="Waiting for any previous stops to complete" sequence=2 taskARN="arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3"
level=info time=2021-11-02T05:07:59Z msg="Wait over; ready to move towards desired status" desiredStatus="RUNNING" taskARN="arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3"
level=info time=2021-11-02T05:07:59Z msg="Creating cgroup /ecs/47362f1eb6a64888ba9b3e0420dfd2c3" module=cgroup_controller_linux.go
level=info time=2021-11-02T05:07:59Z msg="Transitioned resource" taskARN="arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3" resource="cgroup" failedStatus="CREATED"
level=info time=2021-11-02T05:07:59Z msg="Managed task got resource" resource="cgroup" status="CREATED" taskARN="arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3"
level=info time=2021-11-02T05:07:59Z msg="Task engine [arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3]: pulling image ubuntu:latest for container default concurrently" module=docker_task_engine.go
level=info time=2021-11-02T05:07:59Z msg="Task engine [arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3]: recording timestamp for starting image pulltime: 2021-11-02 05:07:59.596964767 +0000 UTC m=+14.207393454" module=docker_task_engine.go
level=info time=2021-11-02T05:08:05Z msg="Adding image name- ubuntu:latest to Image state- sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" module=types.go
level=info time=2021-11-02T05:08:05Z msg="Updating container reference default in Image State - sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" module=types.go
level=info time=2021-11-02T05:08:06Z msg="Task engine [arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3]: finished pulling image ubuntu:latest for container default in 6.418695052s" module=docker_task_engine.go
level=info time=2021-11-02T05:08:06Z msg="Handling container change event" taskARN="arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3" container="default" runtimeID="" status="PULLED"
level=info time=2021-11-02T05:08:06Z msg="Task engine [arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3]: creating container: default" module=docker_task_engine.go
level=info time=2021-11-02T05:08:06Z msg="Task engine [arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3]: created container name mapping for task:  default -> ecs-cromwell_ubuntu_latest_7bf253eba2cefcdbbd770873f42a6867e967ce27-2-default-cc9ae9e9f7f39db69801" module=docker_task_engine.go
level=info time=2021-11-02T05:08:06Z msg="Task engine [arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3]: created docker container for task: default -> 339f03b60a4e94ff29651a9427e1b02f074ad85321b1443c5e34f4903c12eb81" module=docker_task_engine.go
level=info time=2021-11-02T05:08:06Z msg="Task engine [arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3]: created docker container for task: default -> 339f03b60a4e94ff29651a9427e1b02f074ad85321b1443c5e34f4903c12eb81, took 235.163476ms" module=docker_task_engine.go
level=info time=2021-11-02T05:08:06Z msg="Handling container change event" container="default" runtimeID="339f03b60a4e94ff29651a9427e1b02f074ad85321b1443c5e34f4903c12eb81" status="CREATED" taskARN="arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3"
level=info time=2021-11-02T05:08:06Z msg="Container change also resulted in task change" taskARN="arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3" container="default" runtimeID="339f03b60a4e94ff29651a9427e1b02f074ad85321b1443c5e34f4903c12eb81" desiredStatus="RUNNING" knownStatus="CREATED"
level=info time=2021-11-02T05:08:06Z msg="Task engine [arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3]: starting container: default (Runtime ID: 339f03b60a4e94ff29651a9427e1b02f074ad85321b1443c5e34f4903c12eb81)" module=docker_task_engine.go
level=info time=2021-11-02T05:08:06Z msg="Handling container change event" taskARN="arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3" container="default" runtimeID="339f03b60a4e94ff29651a9427e1b02f074ad85321b1443c5e34f4903c12eb81" status="RUNNING"
level=info time=2021-11-02T05:08:06Z msg="Container change also resulted in task change" container="default" runtimeID="339f03b60a4e94ff29651a9427e1b02f074ad85321b1443c5e34f4903c12eb81" desiredStatus="RUNNING" knownStatus="RUNNING" taskARN="arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3"
level=info time=2021-11-02T05:08:06Z msg="TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3 -> RUNNING, Known Sent: NONE, PullStartedAt: 2021-11-02 05:07:59.596964767 +0000 UTC m=+14.207393454, PullStoppedAt: 2021-11-02 05:08:06.015680232 +0000 UTC m=+20.626108927, ExecutionStoppedAt: 0001-01-01 00:00:00 +0000 UTC, container change: arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3 default -> RUNNING, Known Sent: NONE] sent: false" module=task_handler_types.go
level=info time=2021-11-02T05:08:06Z msg="DockerGoClient: Starting streaming metrics for container 339f03b60a4e94ff29651a9427e1b02f074ad85321b1443c5e34f4903c12eb81" module=docker_client.go
level=info time=2021-11-02T05:08:06Z msg="Task engine [arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3]: started docker container for task: default -> 339f03b60a4e94ff29651a9427e1b02f074ad85321b1443c5e34f4903c12eb81, took 389.877577ms" module=docker_task_engine.go
level=info time=2021-11-02T05:08:06Z msg="Managed task at steady state" taskARN="arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3" knownStatus="RUNNING"
level=info time=2021-11-02T05:08:06Z msg="Managed task at steady state" knownStatus="RUNNING" taskARN="arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3"
level=info time=2021-11-02T05:08:06Z msg="Handling container change event" taskARN="arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3" container="default" runtimeID="339f03b60a4e94ff29651a9427e1b02f074ad85321b1443c5e34f4903c12eb81" status="STOPPED"
level=info time=2021-11-02T05:08:06Z msg="Task [arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3]: recording execution stopped time. Essential container [default] stopped at: 2021-11-02 05:08:06.77076548 +0000 UTC m=+21.381194168" module=task.go
level=info time=2021-11-02T05:08:06Z msg="Container change also resulted in task change" runtimeID="339f03b60a4e94ff29651a9427e1b02f074ad85321b1443c5e34f4903c12eb81" desiredStatus="RUNNING" knownStatus="STOPPED" taskARN="arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3" container="default"
level=info time=2021-11-02T05:08:06Z msg="api/task: Updating task desired status to stopped because of container: [default]; task: [cromwell_ubuntu_latest_7bf253eba2cefcdbbd770873f42a6867e967ce27:2 arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3, TaskStatus: (STOPPED->RUNNING) N Containers: 1, N ENIs 0]" module=task.go
level=info time=2021-11-02T05:08:06Z msg="TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3 -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2021-11-02 05:07:59.596964767 +0000 UTC m=+14.207393454, PullStoppedAt: 2021-11-02 05:08:06.015680232 +0000 UTC m=+20.626108927, ExecutionStoppedAt: 2021-11-02 05:08:06.77076548 +0000 UTC m=+21.381194168, container change: arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3 default -> STOPPED, Exit 126, , Known Sent: RUNNING] sent: false" module=task_handler_types.go
level=info time=2021-11-02T05:08:06Z msg="Container [339f03b60a4e94ff29651a9427e1b02f074ad85321b1443c5e34f4903c12eb81]: Stopping stats collection" module=container.go
level=info time=2021-11-02T05:08:06Z msg="Managed task has reached stopped; waiting for container cleanup" taskARN="arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3"
level=info time=2021-11-02T05:10:06Z msg="Cleaning up task's containers and data" taskARN="arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3"
level=info time=2021-11-02T05:10:06Z msg="Task engine [arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3]: removing container: default" module=docker_task_engine.go
level=info time=2021-11-02T05:10:06Z msg="Removing Container Reference: default from Image State- sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" module=types.go
level=info time=2021-11-02T05:10:06Z msg="Task engine [arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3]: resource cgroup cleanup complete" module=docker_task_engine.go
level=info time=2021-11-02T05:10:06Z msg="Task engine [arn:aws:ecs:ap-southeast-2:843407916570:task/TaskBatchComputeEnviron-5881d57c8398998_Batch_ac3a75fc-8cbb-3503-b163-a4d3081c7063/47362f1eb6a64888ba9b3e0420dfd2c3]: finished removing task data, removing task from managed tasks" module=docker_task_engine.go

<!--EndFragment-->
</body>
</html>

It does look like something is going wrong with provisioning. Can you say more about the conditions where you see this?

Is it always in the same region?
Does this only ever happen with the read workflow or do you see it in other workflows?
Approximately how frequently does it happen?
Does it happen during the first step of a workflow or during a "scatter" step?

Is it always in the same region?

We're only using the ap-southeast-2 region.

Does this only ever happen with the read workflow or do you see it in other workflows?

Have only tried the read workflow.

Approximately how frequently does it happen?

Everytime.

Does it happen during the first step of a workflow or during a "scatter" step?

There is no scatter step in this workflow.

The read workflow is this workflow here: https://github.com/aws/amazon-genomics-cli/blob/main/examples/demo-wdl-project/workflows/read/main.wdl

I've dug a little further into this,
After playing around with packages/cdk/lib/constants.ts and adding in the line
sed -i 's/systemctl enable amazon-ssm-agent/systemctl --output=verbose enable amazon-ssm-agent' "/opt/ecs-additions/provision.sh"

I discovered something quite unique:
  | 2021-11-17T10:37:31.668+11:00Copy+ systemctl --output=verbose enable amazon-ssm-agent | + systemctl --output=verbose enable amazon-ssm-agent
-- | -- | --
  | 2021-11-17T10:37:31.668+11:00CopyFailed to execute operation: File exists | Failed to execute operation: File exists

Might be related to either https://github.com/aws/amazon-ssm-agent/issues/282 or https://github.com/aws-samples/aws-digital-content-creation-render-environment/issues/13
I can confirm that we do update the AWS-SSM agent on all our instances.
Prior to running provision.sh, the following code will re-install amazon-ssm-agent has fixed this issue
# As done here: https://github.com/umccr/aws_parallel_cluster/blob/main/ami/base.yml
- systemctl stop amazon-ssm-agent
- systemctl disable amazon-ssm-agent
- curl --output amazon-ssm-agent.rpm https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
- rpm --install --force --upgrade --replacepkgs amazon-ssm-agent.rpm
- rm amazon-ssm-agent.rpm
- systemctl enable amazon-ssm-agent
- systemctl start amazon-ssm-agent

The cloud output log is now as follows:

Click to expand!

loud-init v. 19.3-44.amzn2 running 'init-local' at Wed, 17 Nov 2021 04:56:56 +0000. Up 4.16 seconds.
--
Cloud-init v. 19.3-44.amzn2 running 'init' at Wed, 17 Nov 2021 04:56:57 +0000.  Up 6.77 seconds.
Loaded plugins: priorities, update-motd, upgrade-helper
Existing lock /var/run/yum.pid: another copy is running as pid 3515.
Another app is currently holding the yum lock; waiting for it to exit...  The other application is: yum    Memory :  31 M RSS (245 MB VSZ)    Started: Wed Nov 17 04:56:59 2021 - 00:00 ago    State  : Running, pid: 3515
Another app is currently holding the yum lock; waiting for it to exit...  The other application is: yum    Memory :  80 M RSS (371 MB VSZ)    Started: Wed Nov 17 04:56:59 2021 - 00:02 ago    State  : Running, pid: 3515
Another app is currently holding the yum lock; waiting for it to exit...  The other application is: yum    Memory : 104 M RSS (394 MB VSZ)    Started: Wed Nov 17 04:56:59 2021 - 00:04 ago    State  : Running, pid: 3515
Another app is currently holding the yum lock; waiting for it to exit...  The other application is: yum    Memory : 154 M RSS (444 MB VSZ)    Started: Wed Nov 17 04:56:59 2021 - 00:06 ago    State  : Running, pid: 3515
Another app is currently holding the yum lock; waiting for it to exit...  The other application is: yum    Memory : 155 M RSS (446 MB VSZ)    Started: Wed Nov 17 04:56:59 2021 - 00:08 ago    State  : Running, pid: 3515 --> 1:system-release-2-13.amzn2.x86_64 from installed removed (updateinfo) --> docker-20.10.7-1.amzn2.x86_64 from installed removed (updateinfo) --> systemd-sysv-219-78.amzn2.0.15.x86_64 from amzn2-core removed (updateinfo) --> grubby-8.28-23.amzn2.0.3.x86_64 from amzn2-core removed (updateinfo) --> 1:grub2-tools-efi-2.06-2.amzn2.0.6.x86_64 from amzn2-core removed (updateinfo) --> openldap-2.4.44-23.amzn2.0.1.x86_64 from installed removed (updateinfo) --> curl-7.76.1-4.amzn2.0.1.x86_64 from installed removed (updateinfo) --> libsmartcols-2.30.2-2.amzn2.0.5.x86_64 from amzn2-core removed (updateinfo) --> runc-1.0.0-2.amzn2.x86_64 from amzn2extra-ecs removed (updateinfo) --> systemd-219-78.amzn2.0.15.x86_64 from amzn2-core removed (updateinfo) --> 7:device-mapper-libs-1.02.146-4.amzn2.0.2.x86_64 from installed removed (updateinfo) --> glibc-all-langpacks-2.26-48.amzn2.x86_64 from installed removed (updateinfo) --> grubby-8.28-23.amzn2.0.2.x86_64 from installed removed (updateinfo) --> 1:grub2-common-2.06-2.amzn2.0.6.noarch from amzn2-core removed (updateinfo) --> containerd-1.4.6-3.amzn2.x86_64 from amzn2extra-ecs removed (updateinfo) --> libcrypt-2.26-56.amzn2.x86_64 from amzn2-core removed (updateinfo) --> systemd-libs-219-78.amzn2.0.15.x86_64 from amzn2-core removed (updateinfo) --> runc-1.0.0-1.amzn2.x86_64 from installed removed (updateinfo) --> glibc-locale-source-2.26-48.amzn2.x86_64 from installed removed (updateinfo) --> libfdisk-2.30.2-2.amzn2.0.4.x86_64 from installed removed (updateinfo) --> libcrypt-2.26-48.amzn2.x86_64 from installed removed (updateinfo) --> 1:grub2-pc-modules-2.06-2.amzn2.0.3.noarch from installed removed (updateinfo) --> libmount-2.30.2-2.amzn2.0.4.x86_64 from installed removed (updateinfo) --> libblkid-2.30.2-2.amzn2.0.4.x86_64 from installed removed (updateinfo) --> 1:grub2-2.06-2.amzn2.0.6.x86_64 from amzn2-core removed (updateinfo) --> 1:openssl-1.0.2k-19.amzn2.0.10.x86_64 from amzn2-core removed (updateinfo) --> 7:device-mapper-1.02.170-6.amzn2.5.x86_64 from amzn2-core removed (updateinfo) --> 1:grub2-common-2.06-2.amzn2.0.3.noarch from installed removed (updateinfo) --> glibc-minimal-langpack-2.26-56.amzn2.x86_64 from amzn2-core removed (updateinfo) --> 1:grub2-tools-minimal-2.06-2.amzn2.0.3.x86_64 from installed removed (updateinfo) --> 1:grub2-efi-x64-ec2-2.06-2.amzn2.0.3.x86_64 from installed removed (updateinfo) --> 1:grub2-pc-modules-2.06-2.amzn2.0.6.noarch from amzn2-core removed (updateinfo) --> libsmartcols-2.30.2-2.amzn2.0.4.x86_64 from installed removed (updateinfo) --> 1:system-release-2-14.amzn2.x86_64 from amzn2-core removed (updateinfo) --> 1:openssl-libs-1.0.2k-19.amzn2.0.7.x86_64 from installed removed (updateinfo) --> 7:device-mapper-libs-1.02.170-6.amzn2.5.x86_64 from amzn2-core removed (updateinfo) --> 1:grub2-tools-2.06-2.amzn2.0.3.x86_64 from installed removed (updateinfo) --> libcurl-7.76.1-4.amzn2.0.1.x86_64 from installed removed (updateinfo) --> 1:openssl-libs-1.0.2k-19.amzn2.0.10.x86_64 from amzn2-core removed (updateinfo) --> 1:grub2-tools-extra-2.06-2.amzn2.0.6.x86_64 from amzn2-core removed (updateinfo) --> systemd-sysv-219-78.amzn2.0.14.x86_64 from installed removed (updateinfo) --> glibc-minimal-langpack-2.26-48.amzn2.x86_64 from installed removed (updateinfo) --> dracut-config-ec2-2.0-1.amzn2.noarch from installed removed (updateinfo) --> 1:grub2-tools-minimal-2.06-2.amzn2.0.6.x86_64 from amzn2-core removed (updateinfo) --> amazon-linux-extras-2.0.1-1.amzn2.noarch from amzn2-core removed (updateinfo) --> util-linux-2.30.2-2.amzn2.0.4.x86_64 from installed removed (updateinfo) --> glib2-2.56.1-9.amzn2.0.2.x86_64 from amzn2-core removed (updateinfo) --> libcurl-7.76.1-7.amzn2.0.2.x86_64 from amzn2-core removed (updateinfo) --> 1:grub2-tools-2.06-2.amzn2.0.6.x86_64 from amzn2-core removed (updateinfo) --> 1:grub2-2.06-2.amzn2.0.3.x86_64 from installed removed (updateinfo) --> 1:grub2-pc-2.06-2.amzn2.0.3.x86_64 from installed removed (updateinfo) --> libuuid-2.30.2-2.amzn2.0.5.x86_64 from amzn2-core removed (updateinfo) --> dracut-config-ec2-2.0-2.amzn2.noarch from amzn2-core removed (updateinfo) --> 1:grub2-efi-x64-ec2-2.06-2.amzn2.0.6.x86_64 from amzn2-core removed (updateinfo) --> glibc-2.26-56.amzn2.x86_64 from amzn2-core removed (updateinfo) --> glibc-2.26-48.amzn2.x86_64 from installed removed (updateinfo) --> util-linux-2.30.2-2.amzn2.0.5.x86_64 from amzn2-core removed (updateinfo) --> 1:openssl-1.0.2k-19.amzn2.0.7.x86_64 from installed removed (updateinfo) --> systemd-219-78.amzn2.0.14.x86_64 from installed removed (updateinfo) --> docker-20.10.7-3.amzn2.x86_64 from amzn2extra-ecs removed (updateinfo) --> curl-7.76.1-7.amzn2.0.2.x86_64 from amzn2-core removed (updateinfo) --> systemd-libs-219-78.amzn2.0.14.x86_64 from installed removed (updateinfo) --> amazon-linux-extras-2.0.0-1.amzn2.noarch from installed removed (updateinfo) --> glib2-2.56.1-9.amzn2.0.1.x86_64 from installed removed (updateinfo) --> libfdisk-2.30.2-2.amzn2.0.5.x86_64 from amzn2-core removed (updateinfo) --> 7:device-mapper-1.02.146-4.amzn2.0.2.x86_64 from installed removed (updateinfo) --> libmount-2.30.2-2.amzn2.0.5.x86_64 from amzn2-core removed (updateinfo) --> libblkid-2.30.2-2.amzn2.0.5.x86_64 from amzn2-core removed (updateinfo) --> openldap-2.4.44-23.amzn2.0.2.x86_64 from amzn2-core removed (updateinfo) --> 1:grub2-pc-2.06-2.amzn2.0.6.x86_64 from amzn2-core removed (updateinfo) --> libuuid-2.30.2-2.amzn2.0.4.x86_64 from installed removed (updateinfo) --> containerd-1.4.6-2.amzn2.x86_64 from installed removed (updateinfo) --> glibc-locale-source-2.26-56.amzn2.x86_64 from amzn2-core removed (updateinfo) --> glibc-common-2.26-56.amzn2.x86_64 from amzn2-core removed (updateinfo) --> glibc-all-langpacks-2.26-56.amzn2.x86_64 from amzn2-core removed (updateinfo) --> glibc-common-2.26-48.amzn2.x86_64 from installed removed (updateinfo)
1 package(s) needed (+0 related) for security, out of 40 available
Resolving Dependencies
--> Running transaction check
---> Package ca-certificates.noarch 0:2020.2.41-70.0.amzn2.0.1 will be updated
---> Package ca-certificates.noarch 0:2021.2.50-72.amzn2.0.1 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================ Package             Arch       Version                    Repository      Size
================================================================================
Updating: ca-certificates     noarch     2021.2.50-72.amzn2.0.1     amzn2-core     380 k
Transaction Summary
================================================================================
Upgrade  1 Package
Total download size: 380 k
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction  Updating   : ca-certificates-2021.2.50-72.amzn2.0.1.noarch                1/2   Cleanup    : ca-certificates-2020.2.41-70.0.amzn2.0.1.noarch              2/2   Verifying  : ca-certificates-2021.2.50-72.amzn2.0.1.noarch                1/2   Verifying  : ca-certificates-2020.2.41-70.0.amzn2.0.1.noarch              2/2
Updated:  ca-certificates.noarch 0:2021.2.50-72.amzn2.0.1
Complete!
Loaded plugins: priorities, update-motd, upgrade-helper
Package sed-4.2.2-5.amzn2.0.2.x86_64 already installed and latest version
Package amazon-ssm-agent-3.0.1124.0-1.amzn2.x86_64 already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package amazon-cloudwatch-agent.x86_64 0:1.247347.4-1.amzn2 will be installed
---> Package btrfs-progs.x86_64 0:4.15.1-1.amzn2.0.1 will be installed
--> Processing Dependency: libzstd.so.1()(64bit) for package: btrfs-progs-4.15.1-1.amzn2.0.1.x86_64
--> Processing Dependency: liblzo2.so.2()(64bit) for package: btrfs-progs-4.15.1-1.amzn2.0.1.x86_64
---> Package git.x86_64 0:2.32.0-1.amzn2.0.1 will be installed
--> Processing Dependency: perl-Git = 2.32.0-1.amzn2.0.1 for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: git-core-doc = 2.32.0-1.amzn2.0.1 for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: git-core = 2.32.0-1.amzn2.0.1 for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl >= 5.008 for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: emacs-filesystem >= 27.1 for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(warnings) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(strict) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(lib) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(Term::ReadKey) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(IPC::Open2) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(Git::I18N) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(Git) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(Getopt::Long) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(File::Spec) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(File::Find) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: perl(File::Basename) for package: git-2.32.0-1.amzn2.0.1.x86_64
--> Processing Dependency: /usr/bin/perl for package: git-2.32.0-1.amzn2.0.1.x86_64
---> Package jq.x86_64 0:1.5-1.amzn2.0.2 will be installed
--> Processing Dependency: libonig.so.2()(64bit) for package: jq-1.5-1.amzn2.0.2.x86_64
---> Package unzip.x86_64 0:6.0-43.amzn2 will be installed
--> Running transaction check
---> Package emacs-filesystem.noarch 1:27.2-4.amzn2.0.1 will be installed
---> Package git-core.x86_64 0:2.32.0-1.amzn2.0.1 will be installed
---> Package git-core-doc.noarch 0:2.32.0-1.amzn2.0.1 will be installed
---> Package libzstd.x86_64 0:1.3.3-1.amzn2.0.1 will be installed
---> Package lzo.x86_64 0:2.06-8.amzn2.0.4 will be installed
---> Package oniguruma.x86_64 0:5.9.6-1.amzn2.0.4 will be installed
---> Package perl.x86_64 4:5.16.3-299.amzn2.0.1 will be installed
--> Processing Dependency: perl-libs = 4:5.16.3-299.amzn2.0.1 for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Socket) >= 1.3 for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Scalar::Util) >= 1.10 for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl-macros for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl-libs for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(threads::shared) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(threads) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(constant) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Time::Local) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Time::HiRes) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Storable) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Socket) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Scalar::Util) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Pod::Simple::XHTML) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Pod::Simple::Search) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Filter::Util::Call) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(File::Temp) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(File::Path) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Exporter) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: perl(Carp) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
--> Processing Dependency: libperl.so()(64bit) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
---> Package perl-Getopt-Long.noarch 0:2.40-3.amzn2 will be installed
--> Processing Dependency: perl(Pod::Usage) >= 1.14 for package: perl-Getopt-Long-2.40-3.amzn2.noarch
--> Processing Dependency: perl(Text::ParseWords) for package: perl-Getopt-Long-2.40-3.amzn2.noarch
---> Package perl-Git.noarch 0:2.32.0-1.amzn2.0.1 will be installed
--> Processing Dependency: perl(Error) for package: perl-Git-2.32.0-1.amzn2.0.1.noarch
---> Package perl-PathTools.x86_64 0:3.40-5.amzn2.0.2 will be installed
---> Package perl-TermReadKey.x86_64 0:2.30-20.amzn2.0.2 will be installed
--> Running transaction check
---> Package perl-Carp.noarch 0:1.26-244.amzn2 will be installed
---> Package perl-Error.noarch 1:0.17020-2.amzn2 will be installed
---> Package perl-Exporter.noarch 0:5.68-3.amzn2 will be installed
---> Package perl-File-Path.noarch 0:2.09-2.amzn2 will be installed
---> Package perl-File-Temp.noarch 0:0.23.01-3.amzn2 will be installed
---> Package perl-Filter.x86_64 0:1.49-3.amzn2.0.2 will be installed
---> Package perl-Pod-Simple.noarch 1:3.28-4.amzn2 will be installed
--> Processing Dependency: perl(Pod::Escapes) >= 1.04 for package: 1:perl-Pod-Simple-3.28-4.amzn2.noarch
--> Processing Dependency: perl(Encode) for package: 1:perl-Pod-Simple-3.28-4.amzn2.noarch
---> Package perl-Pod-Usage.noarch 0:1.63-3.amzn2 will be installed
--> Processing Dependency: perl(Pod::Text) >= 3.15 for package: perl-Pod-Usage-1.63-3.amzn2.noarch
--> Processing Dependency: perl-Pod-Perldoc for package: perl-Pod-Usage-1.63-3.amzn2.noarch
---> Package perl-Scalar-List-Utils.x86_64 0:1.27-248.amzn2.0.2 will be installed
---> Package perl-Socket.x86_64 0:2.010-4.amzn2.0.2 will be installed
---> Package perl-Storable.x86_64 0:2.45-3.amzn2.0.2 will be installed
---> Package perl-Text-ParseWords.noarch 0:3.29-4.amzn2 will be installed
---> Package perl-Time-HiRes.x86_64 4:1.9725-3.amzn2.0.2 will be installed
---> Package perl-Time-Local.noarch 0:1.2300-2.amzn2 will be installed
---> Package perl-constant.noarch 0:1.27-2.amzn2.0.1 will be installed
---> Package perl-libs.x86_64 4:5.16.3-299.amzn2.0.1 will be installed
---> Package perl-macros.x86_64 4:5.16.3-299.amzn2.0.1 will be installed
---> Package perl-threads.x86_64 0:1.87-4.amzn2.0.2 will be installed
---> Package perl-threads-shared.x86_64 0:1.43-6.amzn2.0.2 will be installed
--> Running transaction check
---> Package perl-Encode.x86_64 0:2.51-7.amzn2.0.2 will be installed
---> Package perl-Pod-Escapes.noarch 1:1.04-299.amzn2.0.1 will be installed
---> Package perl-Pod-Perldoc.noarch 0:3.20-4.amzn2 will be installed
--> Processing Dependency: perl(parent) for package: perl-Pod-Perldoc-3.20-4.amzn2.noarch
--> Processing Dependency: perl(HTTP::Tiny) for package: perl-Pod-Perldoc-3.20-4.amzn2.noarch
---> Package perl-podlators.noarch 0:2.5.1-3.amzn2.0.1 will be installed
--> Running transaction check
---> Package perl-HTTP-Tiny.noarch 0:0.033-3.amzn2 will be installed
---> Package perl-parent.noarch 1:0.225-244.amzn2.0.1 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================ Package                   Arch     Version                  Repository    Size
================================================================================
Installing: amazon-cloudwatch-agent   x86_64   1.247347.4-1.amzn2       amzn2-core    41 M btrfs-progs               x86_64   4.15.1-1.amzn2.0.1       amzn2-core   771 k git                       x86_64   2.32.0-1.amzn2.0.1       amzn2-core   126 k jq                        x86_64   1.5-1.amzn2.0.2          amzn2-core   154 k unzip                     x86_64   6.0-43.amzn2             amzn2-core   179 k
Installing for dependencies: emacs-filesystem          noarch   1:27.2-4.amzn2.0.1       amzn2-core    67 k git-core                  x86_64   2.32.0-1.amzn2.0.1       amzn2-core   4.8 M git-core-doc              noarch   2.32.0-1.amzn2.0.1       amzn2-core   2.7 M libzstd                   x86_64   1.3.3-1.amzn2.0.1        amzn2-core   203 k lzo                       x86_64   2.06-8.amzn2.0.4         amzn2-core    60 k oniguruma                 x86_64   5.9.6-1.amzn2.0.4        amzn2-core   127 k perl                      x86_64   4:5.16.3-299.amzn2.0.1   amzn2-core   8.0 M perl-Carp                 noarch   1.26-244.amzn2           amzn2-core    19 k perl-Encode               x86_64   2.51-7.amzn2.0.2         amzn2-core   1.5 M perl-Error                noarch   1:0.17020-2.amzn2        amzn2-core    32 k perl-Exporter             noarch   5.68-3.amzn2             amzn2-core    29 k perl-File-Path            noarch   2.09-2.amzn2             amzn2-core    27 k perl-File-Temp            noarch   0.23.01-3.amzn2          amzn2-core    56 k perl-Filter               x86_64   1.49-3.amzn2.0.2         amzn2-core    76 k perl-Getopt-Long          noarch   2.40-3.amzn2             amzn2-core    56 k perl-Git                  noarch   2.32.0-1.amzn2.0.1       amzn2-core    43 k perl-HTTP-Tiny            noarch   0.033-3.amzn2            amzn2-core    38 k perl-PathTools            x86_64   3.40-5.amzn2.0.2         amzn2-core    83 k perl-Pod-Escapes          noarch   1:1.04-299.amzn2.0.1     amzn2-core    52 k perl-Pod-Perldoc          noarch   3.20-4.amzn2             amzn2-core    87 k perl-Pod-Simple           noarch   1:3.28-4.amzn2           amzn2-core   216 k perl-Pod-Usage            noarch   1.63-3.amzn2             amzn2-core    27 k perl-Scalar-List-Utils    x86_64   1.27-248.amzn2.0.2       amzn2-core    36 k perl-Socket               x86_64   2.010-4.amzn2.0.2        amzn2-core    49 k perl-Storable             x86_64   2.45-3.amzn2.0.2         amzn2-core    78 k perl-TermReadKey          x86_64   2.30-20.amzn2.0.2        amzn2-core    31 k perl-Text-ParseWords      noarch   3.29-4.amzn2             amzn2-core    14 k perl-Time-HiRes           x86_64   4:1.9725-3.amzn2.0.2     amzn2-core    45 k perl-Time-Local           noarch   1.2300-2.amzn2           amzn2-core    24 k perl-constant             noarch   1.27-2.amzn2.0.1         amzn2-core    19 k perl-libs                 x86_64   4:5.16.3-299.amzn2.0.1   amzn2-core   685 k perl-macros               x86_64   4:5.16.3-299.amzn2.0.1   amzn2-core    44 k perl-parent               noarch   1:0.225-244.amzn2.0.1    amzn2-core    12 k perl-podlators            noarch   2.5.1-3.amzn2.0.1        amzn2-core   112 k perl-threads              x86_64   1.87-4.amzn2.0.2         amzn2-core    50 k perl-threads-shared       x86_64   1.43-6.amzn2.0.2         amzn2-core    39 k
Transaction Summary
================================================================================
Install  5 Packages (+36 Dependent packages)
Total download size: 61 M
Installed size: 263 M
Downloading packages:
--------------------------------------------------------------------------------
Total                                               61 MB/s \|  61 MB  00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction  Installing : perl-HTTP-Tiny-0.033-3.amzn2.noarch                         1/41   Installing : 1:perl-parent-0.225-244.amzn2.0.1.noarch                    2/41   Installing : perl-Pod-Perldoc-3.20-4.amzn2.noarch                        3/41   Installing : perl-podlators-2.5.1-3.amzn2.0.1.noarch                     4/41   Installing : perl-Encode-2.51-7.amzn2.0.2.x86_64                         5/41   Installing : 1:perl-Pod-Escapes-1.04-299.amzn2.0.1.noarch                6/41   Installing : perl-Text-ParseWords-3.29-4.amzn2.noarch                    7/41   Installing : perl-Pod-Usage-1.63-3.amzn2.noarch                          8/41   Installing : 4:perl-macros-5.16.3-299.amzn2.0.1.x86_64                   9/41   Installing : perl-Filter-1.49-3.amzn2.0.2.x86_64                        10/41   Installing : perl-Carp-1.26-244.amzn2.noarch                            11/41   Installing : 4:perl-libs-5.16.3-299.amzn2.0.1.x86_64                    12/41   Installing : perl-constant-1.27-2.amzn2.0.1.noarch                      13/41   Installing : perl-Time-Local-1.2300-2.amzn2.noarch                      14/41   Installing : perl-PathTools-3.40-5.amzn2.0.2.x86_64                     15/41   Installing : perl-Scalar-List-Utils-1.27-248.amzn2.0.2.x86_64           16/41   Installing : perl-threads-shared-1.43-6.amzn2.0.2.x86_64                17/41   Installing : perl-threads-1.87-4.amzn2.0.2.x86_64                       18/41   Installing : 4:perl-Time-HiRes-1.9725-3.amzn2.0.2.x86_64                19/41   Installing : perl-Socket-2.010-4.amzn2.0.2.x86_64                       20/41   Installing : perl-Storable-2.45-3.amzn2.0.2.x86_64                      21/41   Installing : perl-File-Temp-0.23.01-3.amzn2.noarch                      22/41   Installing : perl-File-Path-2.09-2.amzn2.noarch                         23/41   Installing : 1:perl-Pod-Simple-3.28-4.amzn2.noarch                      24/41   Installing : perl-Getopt-Long-2.40-3.amzn2.noarch                       25/41   Installing : perl-Exporter-5.68-3.amzn2.noarch                          26/41   Installing : 4:perl-5.16.3-299.amzn2.0.1.x86_64                         27/41   Installing : git-core-2.32.0-1.amzn2.0.1.x86_64                         28/41   Installing : git-core-doc-2.32.0-1.amzn2.0.1.noarch                     29/41   Installing : perl-TermReadKey-2.30-20.amzn2.0.2.x86_64                  30/41   Installing : 1:perl-Error-0.17020-2.amzn2.noarch                        31/41   Installing : oniguruma-5.9.6-1.amzn2.0.4.x86_64                         32/41   Installing : libzstd-1.3.3-1.amzn2.0.1.x86_64                           33/41   Installing : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 34/41   Installing : perl-Git-2.32.0-1.amzn2.0.1.noarch                         35/41   Installing : git-2.32.0-1.amzn2.0.1.x86_64                              36/41   Installing : lzo-2.06-8.amzn2.0.4.x86_64                                37/41   Installing : btrfs-progs-4.15.1-1.amzn2.0.1.x86_64                      38/41   Installing : jq-1.5-1.amzn2.0.2.x86_64                                  39/41   Installing : unzip-6.0-43.amzn2.x86_64                                  40/41
create group cwagent, result: 0
create user cwagent, result: 0  Installing : amazon-cloudwatch-agent-1.247347.4-1.amzn2.x86_64          41/41   Verifying  : perl-Exporter-5.68-3.amzn2.noarch                           1/41   Verifying  : 4:perl-5.16.3-299.amzn2.0.1.x86_64                          2/41   Verifying  : perl-Filter-1.49-3.amzn2.0.2.x86_64                         3/41   Verifying  : lzo-2.06-8.amzn2.0.4.x86_64                                 4/41   Verifying  : 4:perl-macros-5.16.3-299.amzn2.0.1.x86_64                   5/41   Verifying  : perl-Pod-Usage-1.63-3.amzn2.noarch                          6/41   Verifying  : perl-Carp-1.26-244.amzn2.noarch                             7/41   Verifying  : perl-PathTools-3.40-5.amzn2.0.2.x86_64                      8/41   Verifying  : perl-HTTP-Tiny-0.033-3.amzn2.noarch                         9/41   Verifying  : 4:perl-libs-5.16.3-299.amzn2.0.1.x86_64                    10/41   Verifying  : perl-Encode-2.51-7.amzn2.0.2.x86_64                        11/41   Verifying  : amazon-cloudwatch-agent-1.247347.4-1.amzn2.x86_64          12/41   Verifying  : perl-threads-shared-1.43-6.amzn2.0.2.x86_64                13/41   Verifying  : 1:perl-Pod-Simple-3.28-4.amzn2.noarch                      14/41   Verifying  : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 15/41   Verifying  : perl-File-Temp-0.23.01-3.amzn2.noarch                      16/41   Verifying  : perl-constant-1.27-2.amzn2.0.1.noarch                      17/41   Verifying  : perl-Time-Local-1.2300-2.amzn2.noarch                      18/41   Verifying  : git-core-2.32.0-1.amzn2.0.1.x86_64                         19/41   Verifying  : perl-Scalar-List-Utils-1.27-248.amzn2.0.2.x86_64           20/41   Verifying  : 1:perl-Pod-Escapes-1.04-299.amzn2.0.1.noarch               21/41   Verifying  : perl-threads-1.87-4.amzn2.0.2.x86_64                       22/41   Verifying  : perl-podlators-2.5.1-3.amzn2.0.1.noarch                    23/41   Verifying  : perl-TermReadKey-2.30-20.amzn2.0.2.x86_64                  24/41   Verifying  : git-core-doc-2.32.0-1.amzn2.0.1.noarch                     25/41   Verifying  : perl-Git-2.32.0-1.amzn2.0.1.noarch                         26/41   Verifying  : libzstd-1.3.3-1.amzn2.0.1.x86_64                           27/41   Verifying  : 4:perl-Time-HiRes-1.9725-3.amzn2.0.2.x86_64                28/41   Verifying  : jq-1.5-1.amzn2.0.2.x86_64                                  29/41   Verifying  : 1:perl-Error-0.17020-2.amzn2.noarch                        30/41   Verifying  : perl-Socket-2.010-4.amzn2.0.2.x86_64                       31/41   Verifying  : perl-Storable-2.45-3.amzn2.0.2.x86_64                      32/41   Verifying  : unzip-6.0-43.amzn2.x86_64                                  33/41   Verifying  : oniguruma-5.9.6-1.amzn2.0.4.x86_64                         34/41   Verifying  : 1:perl-parent-0.225-244.amzn2.0.1.noarch                   35/41   Verifying  : btrfs-progs-4.15.1-1.amzn2.0.1.x86_64                      36/41   Verifying  : perl-Pod-Perldoc-3.20-4.amzn2.noarch                       37/41   Verifying  : git-2.32.0-1.amzn2.0.1.x86_64                              38/41   Verifying  : perl-Text-ParseWords-3.29-4.amzn2.noarch                   39/41   Verifying  : perl-File-Path-2.09-2.amzn2.noarch                         40/41   Verifying  : perl-Getopt-Long-2.40-3.amzn2.noarch                       41/41
Installed:  amazon-cloudwatch-agent.x86_64 0:1.247347.4-1.amzn2                             btrfs-progs.x86_64 0:4.15.1-1.amzn2.0.1                                         git.x86_64 0:2.32.0-1.amzn2.0.1                                                 jq.x86_64 0:1.5-1.amzn2.0.2                                                     unzip.x86_64 0:6.0-43.amzn2
Dependency Installed:  emacs-filesystem.noarch 1:27.2-4.amzn2.0.1                                      git-core.x86_64 0:2.32.0-1.amzn2.0.1                                            git-core-doc.noarch 0:2.32.0-1.amzn2.0.1                                        libzstd.x86_64 0:1.3.3-1.amzn2.0.1                                              lzo.x86_64 0:2.06-8.amzn2.0.4                                                   oniguruma.x86_64 0:5.9.6-1.amzn2.0.4                                            perl.x86_64 4:5.16.3-299.amzn2.0.1                                              perl-Carp.noarch 0:1.26-244.amzn2                                               perl-Encode.x86_64 0:2.51-7.amzn2.0.2                                           perl-Error.noarch 1:0.17020-2.amzn2                                             perl-Exporter.noarch 0:5.68-3.amzn2                                             perl-File-Path.noarch 0:2.09-2.amzn2                                            perl-File-Temp.noarch 0:0.23.01-3.amzn2                                         perl-Filter.x86_64 0:1.49-3.amzn2.0.2                                           perl-Getopt-Long.noarch 0:2.40-3.amzn2                                          perl-Git.noarch 0:2.32.0-1.amzn2.0.1                                            perl-HTTP-Tiny.noarch 0:0.033-3.amzn2                                           perl-PathTools.x86_64 0:3.40-5.amzn2.0.2                                        perl-Pod-Escapes.noarch 1:1.04-299.amzn2.0.1                                    perl-Pod-Perldoc.noarch 0:3.20-4.amzn2                                          perl-Pod-Simple.noarch 1:3.28-4.amzn2                                           perl-Pod-Usage.noarch 0:1.63-3.amzn2                                            perl-Scalar-List-Utils.x86_64 0:1.27-248.amzn2.0.2                              perl-Socket.x86_64 0:2.010-4.amzn2.0.2                                          perl-Storable.x86_64 0:2.45-3.amzn2.0.2                                         perl-TermReadKey.x86_64 0:2.30-20.amzn2.0.2                                     perl-Text-ParseWords.noarch 0:3.29-4.amzn2                                      perl-Time-HiRes.x86_64 4:1.9725-3.amzn2.0.2                                     perl-Time-Local.noarch 0:1.2300-2.amzn2                                         perl-constant.noarch 0:1.27-2.amzn2.0.1                                         perl-libs.x86_64 4:5.16.3-299.amzn2.0.1                                         perl-macros.x86_64 4:5.16.3-299.amzn2.0.1                                       perl-parent.noarch 1:0.225-244.amzn2.0.1                                        perl-podlators.noarch 0:2.5.1-3.amzn2.0.1                                       perl-threads.x86_64 0:1.87-4.amzn2.0.2                                          perl-threads-shared.x86_64 0:1.43-6.amzn2.0.2
Complete!
Cloud-init v. 19.3-44.amzn2 running 'modules:final' at Wed, 17 Nov 2021 04:57:27 +0000. Up 35.52 seconds.
****** processing amazon-cloudwatch-agent ******
/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source file:/opt/aws/amazon-cloudwatch-agent/etc/config.json --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp
Start configuration validation...
/opt/aws/amazon-cloudwatch-agent/bin/config-translator --input /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json --input-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --output /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
2021/11/17 04:57:28 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp ...
Valid Json input schema.
I! Detecting run_as_user...
No csm configuration found.
No metric configuration found.
Configuration validation first phase succeeded
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
Configuration validation second phase succeeded
Configuration validation succeeded
amazon-cloudwatch-agent has already been stopped
Created symlink from /etc/systemd/system/multi-user.target.wants/amazon-cloudwatch-agent.service to /etc/systemd/system/amazon-cloudwatch-agent.service.
Redirecting to /bin/systemctl restart amazon-cloudwatch-agent.service
You can now run: /usr/bin/aws --version
/usr/bin/aws
Completed 1.4 KiB/17.7 KiB (1.4 KiB/s) with 8 file(s) remainingdownload: s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts/ecs-additions/ecs-additions-cromwell-linux2-worker.sh to ecs-additions/ecs-additions-cromwell-linux2-worker.sh
Completed 1.4 KiB/17.7 KiB (1.4 KiB/s) with 7 file(s) remainingCompleted 3.5 KiB/17.7 KiB (3.4 KiB/s) with 7 file(s) remainingdownload: s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts/ecs-additions/provision.sh to ecs-additions/provision.sh
Completed 3.5 KiB/17.7 KiB (3.4 KiB/s) with 6 file(s) remainingCompleted 4.7 KiB/17.7 KiB (4.6 KiB/s) with 6 file(s) remainingdownload: s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts/ecs-additions/ecs-additions-common.sh to ecs-additions/ecs-additions-common.sh
Completed 4.7 KiB/17.7 KiB (4.6 KiB/s) with 5 file(s) remainingCompleted 5.2 KiB/17.7 KiB (5.0 KiB/s) with 5 file(s) remainingdownload: s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts/ecs-additions/awscli-shim.sh to ecs-additions/awscli-shim.sh
Completed 5.2 KiB/17.7 KiB (5.0 KiB/s) with 4 file(s) remainingCompleted 10.4 KiB/17.7 KiB (10.1 KiB/s) with 4 file(s) remainingdownload: s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts/ecs-additions/get-amazon-ebs-autoscale.sh to ecs-additions/get-amazon-ebs-autoscale.sh
Completed 10.4 KiB/17.7 KiB (10.1 KiB/s) with 3 file(s) remainingCompleted 15.8 KiB/17.7 KiB (15.3 KiB/s) with 3 file(s) remainingCompleted 15.9 KiB/17.7 KiB (15.3 KiB/s) with 3 file(s) remainingdownload: s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts/ecs-additions/ecs-additions-cromwell.sh to ecs-additions/ecs-additions-cromwell.sh
Completed 15.9 KiB/17.7 KiB (15.3 KiB/s) with 2 file(s) remainingdownload: s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts/ecs-additions/fetch_and_run.sh to ecs-additions/fetch_and_run.sh
Completed 15.9 KiB/17.7 KiB (15.3 KiB/s) with 1 file(s) remainingCompleted 17.7 KiB/17.7 KiB (16.6 KiB/s) with 1 file(s) remainingdownload: s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts/ecs-additions/ecs-additions-step-functions.sh to ecs-additions/ecs-additions-step-functions.sh
Removed symlink /etc/systemd/system/multi-user.target.wants/amazon-ssm-agent.service.  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  0 26.2M    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  1 26.2M    1  475k    0     0   262k      0  0:01:42  0:00:01  0:01:41  261k 24 26.2M   24 6590k    0     0  2352k      0  0:00:11  0:00:02  0:00:09 2352k 55 26.2M   55 14.6M    0     0  3949k      0  0:00:06  0:00:03  0:00:03 3948k 90 26.2M   90 23.8M    0     0  5120k      0  0:00:05  0:00:04  0:00:01 5119k100 26.2M  100 26.2M    0     0  5400k      0  0:00:04  0:00:04 --:--:-- 6480k
warning: amazon-ssm-agent.rpm: Header V4 RSA/SHA1 Signature, key ID 693eca21: NOKEY
++ uname -r
+ export OS=4.14.243-185.433.amzn2.x86_64
+ OS=4.14.243-185.433.amzn2.x86_64
++ dirname /opt/ecs-additions/provision.sh
+ BASEDIR=/opt/ecs-additions
+ printenv
INSTALLED_ARTIFACTS_S3_ROOT_URL=s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts
OS=4.14.243-185.433.amzn2.x86_64
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
PWD=/opt
ARTIFACTS_NAMESPACE=agc
SHLVL=3
_=/usr/bin/printenv
+ [[ 4.14.243-185.433.amzn2.x86_64 =~ amzn1 ]]
+ [[ 4.14.243-185.433.amzn2.x86_64 =~ amzn2 ]]
+ systemctl enable amazon-ssm-agent
+ systemctl start amazon-ssm-agent
+ trap 'ecs enable' INT ERR EXIT
+ set +e
+ ecs disable
+ [[ 4.14.243-185.433.amzn2.x86_64 =~ amzn1 ]]
+ [[ 4.14.243-185.433.amzn2.x86_64 =~ amzn2 ]]
+ case $1 in
+ systemctl stop ecs
+ systemctl stop docker
Warning: Stopping docker.service, but it can still be activated by:  docker.socket
+ set -e
++ aws ssm get-parameter --name /agc/_common/installed-artifacts/s3-root-url --query Parameter.Value --output text
+ ARTIFACT_S3_ROOT_URL=s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts
+ cd /opt
+ sh /opt/ecs-additions/get-amazon-ebs-autoscale.sh --install-version dist_release --artifact-root-url s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts --file-system btrfs
+ INSTALL_VERSION=dist_release
+ FILESYSTEM=btrfs
++ cat
+ USAGE='Retrive and install Amazon EBS Autoscale    /opt/ecs-additions/get-amazon-ebs-autoscale.sh [options]
Options    -i, --install-version       [dist_release] \| release \| latest \| develop            Version of Amazon EBS Autoscale to install.                            "dist_release" uses '\''aws s3 cp'\'' to retrieve a tarball from an S3 bucket.                    requires setting --artifact-root-url to an S3 URL.                "release" uses '\''curl'\'' or '\''aws s3 cp'\'' to retrieve a tarball from a publicly                     accessible location - i.e. an upstream distribution.                    requires setting --artifact-root-url to either an S3 or HTTP URL.                                "latest" uses '\''curl'\'' to retrieve the latest released version of                     amazon-ebs-autoscale from GitHub                                "develop" uses '\''git'\'' to clone the source code of amazon-ebs-autoscale                    from GitHub.        -a, --artifact-root-url     s3://... \| https:// ...            Root URL where amazon-ebs-autoscale tarballs can be retrieved.            Required if --install-version is "dist_release" or "release".        -f, --file-system           [btrfs] \| lvm.ext4            File system to use        -h, --help            Print help and exit'
+ PARAMS=
+ ((  6  ))
+ case "$1" in
+ INSTALL_VERSION=dist_release
+ shift 2
+ ((  4  ))
+ case "$1" in
+ ARTIFACT_ROOT_URL=s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts
+ shift 2
+ ((  2  ))
+ case "$1" in
+ FILE_SYSTEM=btrfs
+ shift 2
+ ((  0  ))
+ eval set -- ''
++ set --
+ cd /opt
+ dist_release
+ [[ ! -n s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts ]]
+ [[ s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts =~ ^s3.* ]]
+ aws s3 cp --no-progress s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts/amazon-ebs-autoscale.tgz .
download: s3://healthai-public-assets-us-east-1/batch/1.0.2/artifacts/amazon-ebs-autoscale.tgz to ./amazon-ebs-autoscale.tgz
+ tar -xzvf amazon-ebs-autoscale.tgz
amazon-ebs-autoscale/
amazon-ebs-autoscale/CODE_OF_CONDUCT.md
amazon-ebs-autoscale/LICENSE
amazon-ebs-autoscale/bin/
amazon-ebs-autoscale/uninstall.sh
amazon-ebs-autoscale/config/
amazon-ebs-autoscale/install.sh
amazon-ebs-autoscale/shared/
amazon-ebs-autoscale/README.md
amazon-ebs-autoscale/.gitignore
amazon-ebs-autoscale/CONTRIBUTING.md
amazon-ebs-autoscale/.github/
amazon-ebs-autoscale/templates/
amazon-ebs-autoscale/service/
amazon-ebs-autoscale/.git/
amazon-ebs-autoscale/.git/config
amazon-ebs-autoscale/.git/objects/
amazon-ebs-autoscale/.git/HEAD
amazon-ebs-autoscale/.git/info/
amazon-ebs-autoscale/.git/logs/
amazon-ebs-autoscale/.git/description
amazon-ebs-autoscale/.git/hooks/
amazon-ebs-autoscale/.git/refs/
amazon-ebs-autoscale/.git/index
amazon-ebs-autoscale/.git/packed-refs
amazon-ebs-autoscale/.git/refs/heads/
amazon-ebs-autoscale/.git/refs/tags/
amazon-ebs-autoscale/.git/refs/remotes/
amazon-ebs-autoscale/.git/refs/remotes/origin/
amazon-ebs-autoscale/.git/refs/remotes/origin/HEAD
amazon-ebs-autoscale/.git/refs/heads/master
amazon-ebs-autoscale/.git/hooks/commit-msg.sample
amazon-ebs-autoscale/.git/hooks/pre-rebase.sample
amazon-ebs-autoscale/.git/hooks/pre-commit.sample
amazon-ebs-autoscale/.git/hooks/applypatch-msg.sample
amazon-ebs-autoscale/.git/hooks/fsmonitor-watchman.sample
amazon-ebs-autoscale/.git/hooks/pre-receive.sample
amazon-ebs-autoscale/.git/hooks/prepare-commit-msg.sample
amazon-ebs-autoscale/.git/hooks/post-update.sample
amazon-ebs-autoscale/.git/hooks/pre-merge-commit.sample
amazon-ebs-autoscale/.git/hooks/pre-applypatch.sample
amazon-ebs-autoscale/.git/hooks/pre-push.sample
amazon-ebs-autoscale/.git/hooks/update.sample
amazon-ebs-autoscale/.git/hooks/push-to-checkout.sample
amazon-ebs-autoscale/.git/logs/HEAD
amazon-ebs-autoscale/.git/logs/refs/
amazon-ebs-autoscale/.git/logs/refs/heads/
amazon-ebs-autoscale/.git/logs/refs/remotes/
amazon-ebs-autoscale/.git/logs/refs/remotes/origin/
amazon-ebs-autoscale/.git/logs/refs/remotes/origin/HEAD
amazon-ebs-autoscale/.git/logs/refs/heads/master
amazon-ebs-autoscale/.git/info/exclude
amazon-ebs-autoscale/.git/objects/pack/
amazon-ebs-autoscale/.git/objects/info/
amazon-ebs-autoscale/.git/objects/pack/pack-a277d5a47b12f45a0ffbbe18d7da11d79f82f933.idx
amazon-ebs-autoscale/.git/objects/pack/pack-a277d5a47b12f45a0ffbbe18d7da11d79f82f933.pack
amazon-ebs-autoscale/service/upstart/
amazon-ebs-autoscale/service/systemd/
amazon-ebs-autoscale/service/systemd/uninstall.sh
amazon-ebs-autoscale/service/systemd/install.sh
amazon-ebs-autoscale/service/systemd/ebs-autoscale.service
amazon-ebs-autoscale/service/upstart/uninstall.sh
amazon-ebs-autoscale/service/upstart/install.sh
amazon-ebs-autoscale/service/upstart/ebs-autoscale.conf
amazon-ebs-autoscale/templates/cloud-init-userdata.yaml
amazon-ebs-autoscale/.github/PULL_REQUEST_TEMPLATE.md
amazon-ebs-autoscale/shared/utils.sh
amazon-ebs-autoscale/config/ebs-autoscale.logrotate
amazon-ebs-autoscale/config/ebs-autoscale.json
amazon-ebs-autoscale/bin/ebs-autoscale
amazon-ebs-autoscale/bin/create-ebs-volume
+ install btrfs
+ local filesystem=btrfs
+ local docker_storage_driver=btrfs
+ case $filesystem in
+ docker_storage_driver=btrfs
+ local 'docker_storage_options=DOCKER_STORAGE_OPTIONS="--storage-driver btrfs"'
+ cp -au /var/lib/docker /var/lib/docker.bk
+ rm -rf /var/lib/docker/buildkit /var/lib/docker/containers /var/lib/docker/image /var/lib/docker/network /var/lib/docker/overlay2 /var/lib/docker/plugins /var/lib/docker/runtimes /var/lib/docker/swarm /var/lib/docker/tmp /var/lib/docker/trust /var/lib/docker/volumes
+ sh /opt/amazon-ebs-autoscale/install.sh -d /dev/xvdba -f btrfs -m /var/lib/docker
+ awk -v 'docker_storage_options=DOCKER_STORAGE_OPTIONS="--storage-driver btrfs"' '{ sub(/DOCKER_STORAGE_OPTIONS=.*/, docker_storage_options); print }' /etc/sysconfig/docker-storage
+ mv -f /opt/amazon-ebs-autoscale/docker-storage /etc/sysconfig/docker-storage
+ cp -au /var/lib/docker.bk/buildkit /var/lib/docker.bk/containers /var/lib/docker.bk/image /var/lib/docker.bk/network /var/lib/docker.bk/overlay2 /var/lib/docker.bk/plugins /var/lib/docker.bk/runtimes /var/lib/docker.bk/swarm /var/lib/docker.bk/tmp /var/lib/docker.bk/trust /var/lib/docker.bk/volumes /var/lib/docker
+ cd /opt
+ sh /opt/ecs-additions/ecs-additions-common.sh
32+0 records in
32+0 records out
4294967296 bytes (4.3 GB) copied, 21.8533 s, 197 MB/s
Setting up swapspace version 1, size = 4 GiB<PHONE_NUMBER> bytes)
no label, UUID=0249631f-b515-4423-9b99-033aec269245
Filename				Type		Size	Used	Priority
/swapfile                              	file    	4194300	0	-2
+ [[ -n '' ]]
+ ecs enable
+ [[ 4.14.243-185.433.amzn2.x86_64 =~ amzn1 ]]
+ [[ 4.14.243-185.433.amzn2.x86_64 =~ amzn2 ]]
+ case $1 in
+ systemctl start docker
+ systemctl enable --now --no-block ecs
ci-info: no authorized ssh keys fingerprints found for user ec2-user.
Cloud-init v. 19.3-44.amzn2 finished at Wed, 17 Nov 2021 04:58:59 +0000. Datasource DataSourceEc2.  Up 127.03 seconds

<br class="Apple-interchange-newline">

@markjschreiber just wondering if we can get an ETA on this being resolved?

We are internally investigating if we can drop the SSM agent from the Batch workers. I'll keep you posted.