Error when running inside Docker

First of all, this looks like a great project! As someone who as created much of the code logic here independently, I know how much work was put into this project! Much appreciated!

We are currently using a custom preparation script to modify the pi lite os img pre distribution. We are trying to switch over to using sdm.
However when running sdm inside a docker container, we are getting the following error
* Mount IMG '/user/project/2023-12-11-raspios-bookworm-arm64-lite.img'
mount: /mnt/sdm: special device /dev/loop0p2 does not exist.
? Error mounting IMG '/user/project/2023-12-11-raspios-bookworm-arm64-lite.img'

Is sdm is using loop devices for mounting? It is definitely possible to mount the boot and main partition inside a docker container.
This might be an opportunity to solidify the code used for mounting in sdm. Would appreciate some feedback!

Here is the relevant (working) code from our script:
# ...
echo "Analysing ISO..."
isoSectorSize=$(fdisk -l $ISO_FULL_PATH | sed -n -e '/^Sector size/p' | grep -o -E '[0-9]+' | head -1 | sed -e 's/^0\+//')
isoBootStart=$(fdisk -l $ISO_FULL_PATH | grep "FAT32" | awk '{print $2}')
isoBootEnd=$(fdisk -l $ISO_FULL_PATH | grep "FAT32" | awk '{print $3}')
isoBootOffset=$(($isoSectorSize*$isoBootStart))
isoBootSizeLimit=$(($isoSectorSize*$isoBootEnd))
isoMainStart=$(fdisk -l $ISO_FULL_PATH | grep "Linux$" | awk '{print $2}')
isoMainEnd=$(fdisk -l $ISO_FULL_PATH | grep "Linux$" | awk '{print $3}')
isoMainOffset=$(($isoSectorSize*$isoMainStart))
isoMainSizeLimit=$(($isoSectorSize*$isoMainEnd))

echo "Mounting Boot Partition..."
sudo mkdir /mnt/boot
sudo mount -v -o offset=$isoBootOffset,sizelimit=$isoBootSizeLimit -t auto $ISO_FULL_PATH /mnt/boot
# ...
echo "Unmounting..."
sudo umount /mnt/boot
sudo rm -rf /mnt/boot

echo "Mounting Main Partition..."
sudo mkdir /mnt/main
sudo mount -v -o offset=$isoMainOffset,sizelimit=$isoMainSizeLimit -t ext4 $ISO_FULL_PATH /mnt/main
# ...
echo "Unmounting..."
sudo umount /mnt/main
sudo rm -rf /mnt/main
# ...

Relevant spot in the code is the domount function
https://github.com/gitbls/sdm/blob/e810ec5fde930539a4044e9c5f268c4f697e3f0f/sdm-cparse#L1173

Yes, indeed, sdm uses a loop device to mount the IMG.
Before worrying about loop devices inside docker, here's some homework for you. Please test to see if it's possible to run systemd-nspawn inside docker.  If that fails, does chroot work?
And, assuming one of the above two does work, how would sdm determine that it's running inside docker?

Hello and thank you for the quick reply!

Any reason that loop devices need to be used? Directly mounting should work just fine everywhere

I wasn't able to get systemd-nspawn working, but this test script ran just fine

Hello and thank you for the quick reply!

Any reason that loop devices need to be used? Directly mounting should work just fine everywhere

I wasn't able to get systemd-nspawn working, but this whacky test script for chroot ran just fine

mkdir /user/project/jail
mkdir /user/project/jail/bin
mkdir /user/project/jail/lib64
mkdir /user/project/jail/lib
mkdir /user/project/jail/lib/x86_64-linux-gnu
cd /user/project/jail
cp -v /bin/bash /user/project/jail/bin/bash
ldd /bin/bash
cp -v /lib/x86_64-linux-gnu/libtinfo.so.6 /user/project/jail/lib/x86_64-linux-gnu/
cp -v /lib/x86_64-linux-gnu/libdl.so.2 /user/project/jail/lib/x86_64-linux-gnu/
cp -v /lib/x86_64-linux-gnu/libc.so.6 /user/project/jail/lib/x86_64-linux-gnu/
cp -v /lib64/ld-linux-x86-64.so.2 /user/project/jail/lib64
chroot /user/project/jail /bin/bash

Why would sdm need to know that it's running inside of a docker container. Ideally the logic should be the same? But the easiest way to check for it (if this is indeed necessary for some reason) would be to check for existence of /.dockerenv

I will look into removing the need for a loop device
good that chroot works
sdm would need to know that if it had to do something special for being in a docker container. Hoping it will not be necessary, but was asking ahead in case the loop device issue isn't the only challenge

If you want to continue making progress/further testing of sdm in docker, you could copy the IMG contents to a directory accessible in docker and then use the --directory switch. sdm doesn't use a loop device when mounting a directory.

Gotcha, all those make sense!

If you want to continue making progress/further testing of sdm in docker, you could copy the IMG contents to a directory accessible in docker and then use the --directory switch. sdm doesn't use a loop device when mounting a directory.

I was looking into that, but didn't fully understand how it works. How do I extract the img into a directory? Mainly that there are two partitions in the image, how do I deal with that? Could I just mount the main partition and point the "--directory" at the mounted partition?

The code you provided doesn't work. It's close, but the 2nd mount gets an overlapping loop device error.
Also, JFYI it creates an automatic loop device (at least for the first mount). If automatic loop devices don't work in docker this could be a non-starter approach.
w/ssdy/work# ISO_FULL_PATH=/ssd/work/2023-12-05-raspios-bookworm-arm64-lite.img ./x
Analysing ISO...
Mounting Boot Partition...
mkdir: cannot create directory ‘/mnt/boot’: File exists
mount: /dev/loop0 mounted on /mnt/boot.
Mounting Main Partition...
mount: /mnt/main: overlapping loop device exists for /ssd/work/2023-12-05-raspios-bookworm-arm64-lite.img.
Unmounting...
umount: /mnt/main: not mounted.

Thank you for investigating! Great findings! It'll be a few day before I can look into this again, but based on what you are saying, there are a few things to look into
a) Can we get umount to work with non overlapping loop devices?
b) Why does umount work in docker with an implicit loop device, but using loop devices explicitly does not.
I should have time to take another look after the weekend. Cheers, L~

The problem is not umount. sdm needs to mount the root partition first to decide whether the boot partition should be mounted at /boot or /boot/firmware.
So, sdm mounts the root partition, and then it mounts the boot partition.
The 2nd mount (boot partition) fails with an overlap error.
sdm requires both partitions to be mounted for the entire customize process for various reasons. For instance,'apt upgrade' will fail for certain updates if the boot partition is not mounted.

Gotcha, all those make sense!

If you want to continue making progress/further testing of sdm in docker, you could copy the IMG contents to a directory accessible in docker and then use the --directory switch. sdm doesn't use a loop device when mounting a directory.

I was looking into that, but didn't fully understand how it works. How do I extract the img into a directory? Mainly that there are two partitions in the image, how do I deal with that? Could I just mount the main partition and point the "--directory" at the mounted partition?

Here's a sketch of how to do it:
sudo sdm --mount /path/to/img.img
# Now in sdm --mount
rsync -a /mnt/sdm/ /path/to/newdir
exit # Exit the sdm --mount session

Then you can mount /path/to/newdir in your docker container somehow and use sdm --directory /path/to/img to operate on it.
This is the 2nd time you've mentioned "just the main partition". As I explained in another reply, sdm needs BOTH root and boot mounted. sdm itself doesn't have any requirements around the boot partition, except for the plugins that operate on it (bootconfig and quietness, for example), and the partition needs to be accessible during apt upgrade since any package could decide to update an existing file or provide a new file to the boot partition.

Apologies for the late reply. Hope you had a great Christmas time!

The code you provided doesn't work [...]

Here is the fixed code (the previous code was incorrect):
ISO_FULL_PATH=...
ROOT=/mnt/root

echo "Analysing ISO..."
INFO=($(fdisk --bytes -lo Start,Size "${ISO_FULL_PATH}" | tail -n 2))
BOOT_START=$((${INFO[0]} * 512))
BOOT_SIZE=${INFO[1]}
ROOT_START=$((${INFO[2]} * 512))
ROOT_SIZE=${INFO[3]}

echo "Mounting Image..."
sudo mkdir -p "${ROOT}"
sudo mount -o loop,offset=${ROOT_START},sizelimit=${ROOT_SIZE} -t ext4 "${ISO_FULL_PATH}" "${ROOT}"
sudo mount -o loop,offset=${BOOT_START},sizelimit=${BOOT_SIZE} -t vfat "${ISO_FULL_PATH}" "${ROOT}/boot"

# DO WORK HERE
# ...

echo "Unmounting Image..."
sudo umount -f "${ROOT}/boot/"
sudo umount -f "${ROOT}"

Alright, this is getting us somewhere. I've mounted the pi os image partitions as described above inside docker and then ran sudo cp -r "${ROOT}" "/tmp/root".
This resulted in the following
> Command Line: /usr/local/bin/sdm --customize --plugin user:adduser=bls|password=mypassword
  --plugin L10n:host --plugin disables:piwiz --regen-ssh-host-keys --restart --directory /tmp/root
* Host Information
   Hostname: 26657deefa68
   uname: Linux 26657deefa68 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023
  x86_64 x86_64 x86_64 GNU/Linux
   os-release Name: Ubuntu
   Version: 20.04.4 (focal)
   Like OS: debian
   sdm Version: V10.2
* IMG Information
   Name: /tmp/root
   Date: 2023-12-11
   RasPiOS Version: 12
   RasPiOS Architecture: 64-bit aarch64
   os-release Version: 12
% sdm will use systemd-nspawn on this 'x86-64' host
  Retry the command with --chroot if this fails
  sdm will run more slowly due to qemu emulation
* Plugins included:
   * user
       Args: adduser=bls|password=mypassword
   * L10n
       Args: host
   * disables
       Args: piwiz
> Directory '/tmp/root' has 43664416 1K-blocks (44.7GB, 41.6GiB) free
> Copy sdm to /usr/local/sdm in the Directory
* Start Phase 0 image customization
> Run Plugins Phase '0'
> Run Plugin 'user' (/mnt/sdm/usr/local/sdm/plugins/user) Phase 0 with arguments:
  'adduser=bls|password=mypassword'
* Plugin user: Start Phase 0
> Plugin user: Keys/values found:
   adduser: bls
   password: mypassword
> Plugin user: Create user 'bls' home directory '/home/bls' so available for other plugins
* Plugin user: Complete Phase 0
> Run Plugin 'L10n' (/mnt/sdm/usr/local/sdm/plugins/L10n) Phase 0 with arguments: 'host'
* Plugin L10n: Start Phase 0
> Plugin L10n: Keys/values found:
   host:
> Plugin L10n: Read L10n configuration from host system
% /etc/default/keyboard missing; Keyboard set to 'us'
> Plugin L10n: Load Localization (L10N) settings from running system
> Plugin L10n:   Keymap:       us
> Plugin L10n:   Locale:       en_US.UTF-8
> Plugin L10n:   Timezone:     Etc/UTC
> Plugin L10n:   WiFi Country:
> Plugin L10n: Save L10n configuration
? Prefix L10n: Unrecognized keymap 'us'
* Plugin L10n: Complete Phase 0
> Run Plugin 'disables' (/mnt/sdm/usr/local/sdm/plugins/disables) Phase 0 with arguments:
  'piwiz'
* Plugin disables: Start Phase 0
> Plugin disables: Keys/values found:
   piwiz:
* Plugin disables: Complete Phase 0
* Phase 0 Completed
* Enter image '/tmp/root' for Phase 1
Failed to open system bus: No such file or directory
Attempted to remove disk file system under "/run/systemd/nspawn/propagate/sdm", and we can't allow that.
umount: /mnt/sdm/boot (overlay) unmounted
umount: /mnt/sdm (overlay) unmounted

When running sdm directly on the mounted partitions I get:
sudo sdm --customize --plugin 'user:adduser=bls|password=mypassword' --plugin L10n:host --plugin disables:piwiz --regen-ssh-host-keys --restart --directory /mnt/raspbian
? Directory '/mnt/raspbian' is already mounted

Would be great if we could fix that, since copying everything over takes a lot of time and resources.
Let me know if you need anything else from my side. Cheers!

Thanks! Looks promising. Will have an in-depth look into it.

I tried to play around with systemd-nspawn some more. In particular adding --register=no --keep-unit to prevent registration with system bus. But it really wasn't playing nice - issues with cgroup that I didn't fully understand and some other nonsense.
So here is the command from above, but with the --chroot option:
sudo sdm --batch --chroot --customize --plugin 'user:adduser=bls|password=mypassword' --plugin L10n:host --plugin disables:piwiz --regen-ssh-host-keys --restart --directory /tmp/root

Output
* Start Configuration
> Command Line: /usr/local/bin/sdm --batch --chroot --customize --plugin
  user:adduser=bls|password=mypassword --plugin L10n:host --plugin disables:piwiz
  --regen-ssh-host-keys --restart --directory /tmp/root
* Host Information
   Hostname: b4714aceeea9
   uname: Linux b4714aceeea9 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023
  x86_64 x86_64 x86_64 GNU/Linux
   os-release Name: Ubuntu
   Version: 20.04.4 (focal)
   Like OS: debian
   sdm Version: V10.2
* IMG Information
   Name: /tmp/root
   Date: 2023-12-11
   RasPiOS Version: 12
   RasPiOS Architecture: 64-bit aarch64
   os-release Version: 12
% sdm will use chroot per --chroot on this 'x86-64' host
  sdm will run more slowly due to qemu emulation
* Plugins included:
   * user
       Args: adduser=bls|password=mypassword
   * L10n
       Args: host
   * disables
       Args: piwiz
> Directory '/tmp/root' has 48395116 1K-blocks (49.6GB, 46.2GiB) free
> Copy sdm to /usr/local/sdm in the Directory
* Start Phase 0 image customization
> Run Plugins Phase '0'
> Run Plugin 'user' (/mnt/sdm/usr/local/sdm/plugins/user) Phase 0 with arguments:
  'adduser=bls|password=mypassword'
* Plugin user: Start Phase 0
> Plugin user: Keys/values found:
   adduser: bls
   password: mypassword
> Plugin user: Create user 'bls' home directory '/home/bls' so available for other plugins
* Plugin user: Complete Phase 0
> Run Plugin 'L10n' (/mnt/sdm/usr/local/sdm/plugins/L10n) Phase 0 with arguments: 'host'
* Plugin L10n: Start Phase 0
> Plugin L10n: Keys/values found:
   host:
> Plugin L10n: Read L10n configuration from host system
% /etc/default/keyboard missing; Keyboard set to 'us'
> Plugin L10n: Load Localization (L10N) settings from running system
> Plugin L10n:   Keymap:       us
> Plugin L10n:   Locale:       en_US.UTF-8
> Plugin L10n:   Timezone:     Etc/UTC
> Plugin L10n:   WiFi Country:
> Plugin L10n: Save L10n configuration
? Prefix L10n: Unrecognized keymap 'us'
* Plugin L10n: Complete Phase 0
> Run Plugin 'disables' (/mnt/sdm/usr/local/sdm/plugins/disables) Phase 0 with arguments:
  'piwiz'
* Plugin disables: Start Phase 0
> Plugin disables: Keys/values found:
   piwiz:
* Plugin disables: Complete Phase 0
* Phase 0 Completed
* Enter image '/tmp/root' for Phase 1
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
* Start Phase 1 image customization
> Directory '/tmp/root' has 48388560 1K-blocks (49.5GB, 46.1GiB) free at start of Phase 1 image customization
> Configure and enable sdm FirstBoot service (sdm-firstboot)
> Start 'apt update'
> Directory '/tmp/root' has 48388556 1K-blocks (49.5GB, 46.1GiB) free at start of 'apt update'
> Directory '/tmp/root' has 48274888 1K-blocks (49.4GB, 46.0GiB) free at end of 'apt update'
> Run Plugins Phase '1'
> Run Plugin 'user' (/usr/local/sdm/plugins/user) Phase 1 with arguments:
  'adduser=bls|password=mypassword'
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
* Plugin user: Start Phase 1
> Plugin user: Add user 'bls'
> Plugin user: Add groups to user 'bls':
  dialout,cdrom,floppy,audio,video,plugdev,users,adm,sudo,users,input,netdev,spi,i2c,gpio
> Plugin user: Enable sudo for user 'bls'
> Plugin user: Copy skel files for user 'bls'
> Plugin user: Set user 'bls' home directory '/home/bls' owner to 'bls:bls'
> Plugin user: Set password for user 'bls'
* Plugin user: Complete Phase 1
> Run Plugin 'L10n' (/usr/local/sdm/plugins/L10n) Phase 1 with arguments: 'host'
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
* Plugin L10n: Start Phase 1
% Plugin L10n: L10n item 'keymap' was not set
> Plugin L10n: Set locale to 'en_US.UTF-8'
Generating locales (this might take a while)...
  en_US.UTF-8... done
Generation complete.
> Plugin L10n: Set timezone to 'Etc/UTC'
* Plugin L10n: Complete Phase 1
> Run Plugin 'disables' (/usr/local/sdm/plugins/disables) Phase 1 with arguments: 'piwiz'
* Plugin disables: Start Phase 1
* Plugin disables: Complete Phase 1
* Phase 1 post-app installation/configuration
> Start 'apt upgrade'
> Directory '/tmp/root' has 48269892 1K-blocks (49.4GB, 46.0GiB) free at start of 'apt upgrade'
> Directory '/tmp/root' has 48269888 1K-blocks (49.4GB, 46.0GiB) free at end of 'apt upgrade'
> Start 'apt autoremove'
> Directory '/tmp/root' has 48269888 1K-blocks (49.4GB, 46.0GiB) free at start of 'apt autoremove'
> Directory '/tmp/root' has 48269888 1K-blocks (49.4GB, 46.0GiB) free at end of 'apt autoremove'
> Directory '/tmp/root' has 48269888 1K-blocks (49.4GB, 46.0GiB) free at end of Phase 1 image customization
* Phase 1 Completed
> Run Plugins Phase 'post-install'
> Run Plugin 'user' (/usr/local/sdm/plugins/user) Phase post-install with arguments:
  'adduser=bls|password=mypassword'
* Plugin user: Start Phase post-install
* Plugin user: Complete Phase post-install
> Run Plugin 'L10n' (/usr/local/sdm/plugins/L10n) Phase post-install with arguments: 'host'
> Run Plugin 'disables' (/usr/local/sdm/plugins/disables) Phase post-install with arguments:
  'piwiz'
* Plugin disables: Start Phase post-install
> Plugin disables: Disable running piwiz and userconfig on first boot per --disable piwiz
* Plugin disables: Complete Phase post-install
> Run graphics post-install configuration
> Plugin sdm-phase1: No known Display Manager found
> Plugin sdm-phase1: Set RasPiOS to B1:'console no login' for First Boot
> Enable SSH
> Plugin phase1: Enable SSH service via /boot/ssh
> Directory '/tmp/root' has 48269880 1K-blocks (49.4GB, 46.0GiB) free at end of image customization
* Customization complete
* Batch Mode exit
umount: /mnt/sdm/boot (overlay) unmounted
umount: /mnt/sdm (overlay) unmounted

This already looks really good.

A couple of questions:

Confirming that this is only when mounting IMG files, and you don't need it for mounting, for instance, devices from within the docker container
What's with this: /bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8) ?

Yes, the only requirement for us is to be able to edit an image file.
I am not sure. It might be because the docker container isn't set up correctly, but my fix attempts didn't work. I'll try some more and get back to you. This is a very harmless error and shouldn't cause any issues.

Thx.
 It seems to work well in my initial non-docker test, but haven't done much with it yet.

Just tested this, and if delete this line by running sudo sed -i '1184d' /usr/local/sdm/sdm-cparse, I don't need to copy the mount into a directory and can instead just work against the mounted directory!
While certainly not ideal, this makes it somewhat usable for me now.

Just tested this, and if delete this line by running sudo sed -i '1184d' /usr/local/sdm/sdm-cparse, I don't need to copy the mount into a directory and can instead just work against the mount itself!
While certainly not ideal, this makes it somewhat usable for me now.

Presumably this is to enable you to keep making progress, and not related to anything I'm doing?

Presumably this is to enable you to keep making progress, and not related to anything I'm doing?

Sort of. Maybe there should be an option to allow "--directory" to be pointed to an already mounted device? I'm assuming the line in question exists as a check, however for me it prevents my (current) use case where I want to run sdm against a mounted image.

For your planning purposes, I'm thinking later next week I should be able to release the next sdm update.

Sounds good! I am unblocked for now, which is great.

Yea, the check is there just to be careful. You've just proven that having it already mounted isn't problematic wrt doing a mount/bind on it.
I'm a bit uncomfortable removing it completely, as there could be unforseen issues. Thinking changing it to a %warning and continuing would be a better solution than full removal. Thoughts?

@gitbls I always like being explicit, so how about an additional flag that only takes mount points?
It looks like --mount is already used, so maybe --mount-point /path/to/mount/point?
Instead of running mount --bind $dimg $SDMPT you could just set them equal and run chroot / systemd-nspawn against the passed mount point directly?

It might also be time to refactor  and add commands instead of flags on sdm, but I don't know enough about the syntax yet. E.g. sdm modify ..., sdm mount ..., sdm unmount etc. This would be a much bigger undertaking and breaking change though, so for another time

While I understand your point about explicitness, adding --mount-point without getting rid of --directory doesn't seem like a win to me. It adds yet another thing to test/document/explain with questionable value from my perspective.
Someone else mentioned replacing switches with commands. I don't disagree with the sentiment, but there are other commands that do this. For instance: dpkg and update-alternatives.
There's nothing broken because of it and changing it would break literally everyone's sdm scripts. IOW it's a "How to piss users off and not create any new value for them" kind of change...Something that a certain company in Redmond has been known to do 🙄.

@gitbls Yeah, I don't disagree with any of that:

Breaking change = bad ✅
Adding features / complexity that no-one needs / uses = bad ✅

Changing switches to commands doesn't have to be a breaking change. You could slowly transition and deprecate the old way. But I would only do this if it is clearly better (i.e. you have distinct commands that don't join together somehow). Also it's really a question of time invested vs value gained. It was just a suggestion because some flags seemed to me like they are actually disjoined commands (and maybe they're not)
So I'm happy with just making it a warning until more people are running into it. You can revisit when the first person opens a ticket "why am I getting this warning?" =)

@simlu Would you mind sharing your Dockerfile?

For sure, here are the files needed to run this locally
project/script.sh
# put any logic you want to run here
# ...

docker/Dockerfile
FROM cimg/node:18.0

RUN sudo apt update
# sdm install script
RUN curl -L https://raw.githubusercontent.com/gitbls/sdm/master/EZsdmInstaller | bash
# used by sdm for qemu emulation
RUN sudo apt install qemu-user-static binfmt-support
# used to resize image partitions
RUN sudo apt install udev

and
manage.sh
#!/bin/bash

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

echo "Starting docker container..."
docker build \
  --build-arg COMMAND=". script.sh" \
  -t sdm_build-container \
  --network="host" \
  $SCRIPT_DIR/docker/. &&
docker run \
  --rm \
  -v $SCRIPT_DIR/project:/user/project \
  -it --privileged sdm_build-container

Note that I couldn't get partition resizing to work with sdm, so I've did it custom for now.
Maybe we can revisit that once you release a new sdm version that uses the new mounting logic.

Note that I couldn't get partition resizing to work with sdm, so I've did it custom for now.
Maybe we can revisit that once you release a new sdm version that uses the new mounting logic.

In order to get a few other things done, I postponed modifying the mounting logic for the following:

Anything having to do with burning
The new parted plugin
sdm --shrink

I think that's it.  If the list is missing something, let me know after you've tested it*
These will get done, of course, just ran out of time to do them all at once.

I'm hoping to get the release out today USPT, but it may slip into the weekend if I get hung up on final improvements/testing.

No rush, I am no longer blocked on anything.
Happy to test and provide some demo code for documentation purpose once this is released.

Must pushed V11. Please take it for a spin! Thx!

In-depth look into the remaining issues WRT running in Docker. Here's my current findings:

Burning to an SSD/SD Card: OK and should work in V11, pending your testing
Burning to a file: Does not work and can't work. I need a handle to a device. In the case of an IMG, it needs a loop device. But the only way to get a loop device that is docker-compatible is to mount the disk using -o loop,offset=,sizelimit=. Fine, but then both sfdisk and fdisk fail to set the Disk ID on the burned output file because there's a disk mounted:

> Set new disk ID '390cb582' on 'x.img'
sfdisk: /dev/loop0: failed to set disklabel ID
% sfdisk failed; Trying fdisk to set Disk ID
This disk is currently in use - repartitioning is probably a bad idea.
It's recommended to umount all file systems, and swapoff all swap partitions on this disk.

The device contains 'ext4' signature and it may remain on the device.
It is recommended to wipe the device with wipefs(8) or fdisk --wipe, in order to avoid possible collisions.

parted plugin operating on an IMG: Same issue as burning to a file
parted plugin operating on an SSD/SD Card: OK and should work in V11, pending your testing

If two problematic scenarios are important to you, feel free to figure out how to make them work in Docker!

Closing due to lack of activity. OP, please re-open if needed. Thx!

@simlu  @gitbls
Hey guys sorry for respawning a closed issue But i just want to ge this working on my mac in docker
I Have a question for you guys
I have tried to make this work in docker on my m1 macbook pro. But i cant see to manage to do it,
So i was wondering if it is possible when the host is a machine is a mac, or do you have to run debian/ubuntu as the host manchine.
This is what i have tried based on simlus successfull attempt, sorry if it is a bit long to read i want to go in detail
This is my host machine foler structure:
[img]
drwxr-xr-x sebastian staff 192 B Sat May 11 21:40:14 2024  .
drwxr-xr-x sebastian staff 288 B Sat May 11 07:23:53 2024  ..
.rw-r--r-- sebastian staff 394 B Sat May 11 04:41:03 2024  develop.sh
drwxr-xr-x sebastian staff 256 B Sat May 11 21:38:26 2024  docker
.rwxr-xr-x sebastian staff 429 B Sat May 11 03:03:39 2024  manage.sh
drwxr-xr-x sebastian staff 160 B Sat May 11 04:30:37 2024  project

content of develop.sh (used to run the container fast )

#!/bin/bash

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
echo $SCRIPT_DIR
echo "Starting docker container..."
# Run for test
#docker run -v $SCRIPT_DIR/docker:/user/project/docker -it --privileged sdm_build-container
docker run --rm -v /Users/seb/dev/hobby/auto-setup-docker/docker:/home/circleci/project -it --privileged sdm_build-container

content of manage.sh (used tobild the image)

#!/bin/bash

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

echo "Starting docker container..."
# Run for test
docker build \
  --build-arg COMMAND=". script.sh" \
  --platform linux/arm64/v8 \
  -t sdm_build-container \
  --network="host" \
  $SCRIPT_DIR/docker/. &&
docker run \
  --rm \
  --storage-opt =10G \
  -v $SCRIPT_DIR/project:/user/project \
  -it --privileged sdm_build-container \

Content from docker directory(this is the mounted volume in the container)

drwxr-xr-x sebastian staff 256 B  Sat May 11 21:38:26 2024  .
drwxr-xr-x sebastian staff 160 B  Sat May 11 21:46:27 2024  ..
.rw-r--r-- sebastian staff 2.4 KB Sat May 11 05:26:32 2024  .bashrc
.rw-r--r-- sebastian staff 340 B  Sat May 11 05:59:58 2024  .bashrc2
.rw-r--r-- sebastian staff 287 B  Sat May 11 04:13:17 2024  Dockerfile
.rw-r--r-- sebastian staff 2.6 GB Sat May 11 05:46:06 2024  first.img
drwxr-xr-x sebastian staff  96 B  Fri May 10 21:23:12 2024  mount-script.sh
drwxr-xr-x sebastian staff 832 B  Sat May 11 04:12:25 2024  sdm

the first.img is a renamed --> 2023-12-11-raspios-bookworm-arm64-lite.img.xz
The .bashrc includes some simple scripts to try to work on the file
*bashrc
export SDM_BOOT_PATH="/home/circleci/project/sdm"
export ISO_FULL_PATH="/home/circleci/project/first.img"
export ROOT=/mnt/raspbian
export INFO=($(fdisk --bytes -lo Start,Size "${ISO_FULL_PATH}" | tail -n 2))
export BOOT_SIZE=${INFO[1]}
export ROOT_START=$((${INFO[2]} * 512))
export ROOT_SIZE=${INFO[3]}
export BOOT_START=$((${INFO[0]} * 512))
export BOOT_START_HARD=4194304
export BOOT_SIZE_HARD=268435456
export ROOT_START_HARD=272629760
export ROOT_SIZE_HARD=1832910848

This is what i have tried From inside the container on before hand
sudo sdm --mount /path/to/img.img # the raspberry pi image is in the container
# Now in sdm --mount
rsync -a /mnt/sdm/ /path/to/newdir
exit # Exit the sdm --mount session
# I have then tried to copy the newdir folder out of the container and to the host machine

This is where the error commes
I running sudo sh develoop.sh
Dockerfile  first.img  mount-script.sh  sdm
circleci@5b1a079029dc:~/project$ source .bashrc2
circleci@5b1a079029dc:~/project$ ls
Dockerfile  first.img  mount-script.sh  sdm
circleci@5b1a079029dc:~/project$ mount_image
Mounting Image...
circleci@5b1a079029dc:~/project$ ls
Dockerfile  first.img  mount-script.sh  sdm
circleci@5b1a079029dc:~/project$ run_sdm
% sdm will use qemu 'aarch64'
% Add and enable qemu binfmt for 'aarch64' processor architecture
% sdm will use chroot per --chroot on this 32-bit x86-64 host
* Enter Directory '/home/circleci/project/sdm'
/usr/local/sdm/sdm-cparse: line 1839: /usr/bin/stty: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1839: /usr/bin/stty: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1839: /usr/bin/stty: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1839: /usr/bin/stty: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1839: /usr/bin/stty: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1839: /usr/bin/stty: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1093: /usr/bin/mount: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1093: /usr/bin/mount: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1093: /usr/bin/mount: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1093: /usr/bin/mount: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1097: /usr/sbin/chroot: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 984: /usr/bin/grep: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 985: /usr/bin/umount: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1685: /usr/bin/findmnt: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1687: /usr/bin/findmnt: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1493: /usr/bin/findmnt: cannot execute binary file: Exec format errorhttps://github.com/gitbls
/usr/local/sdm/sdm-cparse: line 1493: /usr/bin/findmnt: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 1706: /usr/bin/sync: cannot execute binary file: Exec format error

The run_sdm
is a bash script in the .bashrc file
run_sdm() {

    if [ $# -lt 1 ]; then
        directory=$TEMP_ROOT
    else
        directory="$1"     # Use the passed argument
    fi
sudo sdm --batch --chroot --customize --plugin 'user:adduser=bls|password=mypassword' --plugin L10n:host --plugin disables:piwiz --regen-ssh-host-keys --restart --directory $directory

when i run this command i get much errors in the
/usr/local/bin/sdm: line 825: /usr/bin/cp: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 157: /usr/bin/chown: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 158: /usr/bin/chmod: cannot execute binary file: Exec format error
/usr/local/bin/sdm: line 828: /usr/bin/chown: cannot execute binary file: Exec format error
* Start Configuration
/usr/local/sdm/sdm-cparse: line 397: /usr/bin/file: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 50: /usr/bin/fold: cannot execute binary file: Exec format error

* Host Information
/usr/local/bin/sdm: line 840: /usr/bin/hostname: cannot execute binary file: Exec format error
   Hostname:
/usr/local/bin/sdm: line 841: /usr/bin/uname: cannot execute binary file: Exec format error
   uname:
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
   os-release Name:
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
   Version:
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
   sdm Version: V12.0
* IMG Information
   Name: /home/circleci/project/sdm
/usr/local/sdm/sdm-cparse: line 376: /usr/bin/grep: cannot execute binary file: Exec format error
   Date:
   RasPiOS Version:
   RasPiOS Architecture: 32-bit armhf
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
   os-release Version:

First of all i am so sorry that this post is all over the place, but i wanted to ensure you that i have tried much(and more than is described here go get this to work) but do you guys now if it is even possible to run sdm in a docker container if the host machine is a mac
have tried to mount the image then Rsynced the directory the director to the host machine, and then mount that folder onto the container. But when i try to run any commands the docker breaks and

@odi89 I don't have a Mac, and I don't use docker. If you're willing to try various things I suggest, then we can probably figure out what's going on.
One immediate question:. You said it's an M1 Mac.  You also said that first.img is a copy of an arm64 RasPiOS. Yet, in the log output I see:
   Date:
   RasPiOS Version:
   RasPiOS Architecture: 32-bit armhf

Please clarify what you're actually using.

@gitbls
Thank you for your response i will try you suggest and will try any suggestions you have.
And further down in the process i will try to document my comments so it is easier for you to help and maybe some
others who tries to get sdm to run insed docker will find it helpfull/usefull.
Since your last comment. You are asbolutely right, first.img was wrong image.
I now downloaded 2024-03-15-raspios-bookworm-arm64-lite.img and try again
Okay this is the process
Started the docker container
mounted the ISO following your steps
sudo sdm --mount 2024-03-15-raspios-bookworm-arm64-lite.img
rsync -a /mnt/sdm/ /tmp/root
exit # Exit the sdm --mount session

Then from outside the container i synced the root directory back to my host machine(mac ) so i dont have to do this every time
# From host machine
##/Users/sebastian/dev/hobby/auto-setup-docker/docker
sudo docker cp  082dabf64821:/tmp/root ./sdm # This copies the mounted isofile from inside the container --> to the hostmachine in a folder called /sdm

Back inside the container i tried to run the newly mountes iso image from /tmp/root but stil got this
run_sdm_from_directory # Bash script running ls sudo sdm --batch --chroot --customize --plugin 'user:adduser=bls|password=mypassword' --plugin L10n:host --plugin disables:piwiz --regen-ssh-host-keys --restart --directory $directory

/usr/local/sdm/sdm-cparse: line 50: /usr/bin/fold: cannot execute binary file: Exec format error

* Host Information
/usr/local/bin/sdm: line 840: /usr/bin/hostname: cannot execute binary file: Exec format error
   Hostname:
/usr/local/bin/sdm: line 841: /usr/bin/uname: cannot execute binary file: Exec format error
   uname:
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
   os-release Name:
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
   Version:
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
   sdm Version: V12.0
* IMG Information
   Name: /tmp/root
/usr/local/sdm/sdm-cparse: line 376: /usr/bin/grep: cannot execute binary file: Exec format error
   Date:
   RasPiOS Version:
   RasPiOS Architecture: 32-bit armhf
/usr/local/sdm/sdm-cparse: line 367: /usr/bin/grep: cannot execute binary file: Exec format error
   os-release Version:
% sdm will use qemu 'aarch64'
% Add and enable qemu binfmt for 'aarch64' processor architecture
% sdm will use chroot per --chroot on this 32-bit x86-64 host
* Plugins selected:
   * user
       Args: adduser=bls|password=mypassword
   * L10n
       Args: host
   * disables
       Args: piwiz
/

I will no experiment with other images tried to download the This 2024-03-15-raspios-bookworm-arm64-full.img.xz version to test with that
Is it something that i am missing?
This is the content of (maybe)  some relevant file
manage.sh
#!/bin/bash

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

echo "Starting docker container..."

# Run for test
#docker run -v $SCRIPT_DIR/docker:/user/project/docker -it --privileged sdm_build-container
docker run --rm -v /Users/sebastian/dev/hobby/auto-setup-docker/docker:/home/circleci/project -it --privileged sdm_build-container

Dockerfile
FROM cimg/node:18.0
# Install dependencies
RUN sudo apt-get update && sudo apt-get install -y \
    curl \
    qemu-user-static \
    binfmt-support \
    udev
# Install sdm
RUN curl -L https://raw.githubusercontent.com/gitbls/sdm/master/EZsdmInstaller | bash

What is the result if you try to use sdm on the directory tree outside of docker (but still on the Mac)? IOW just remove docker completely from the equation.

❯ sudo sdm --batch --chroot --customize --plugin 'user:adduser=bls|password=mypassword' --plugin L10n:host --plugin disables:piwiz --regen-ssh-host-keys --restart --directory ./sdm
Password:
/usr/local/bin/sdm: line 311: --plugin: command not found
/usr/local/bin/sdm: line 315: ${1,,}: bad substitution
/usr/local/bin/sdm: line 365: ${dimg,,}: bad substitution
? No command specified (--aptmaint --burn, --customize, --explore, --mount, --runonly)

As I mentioned, I don't have a Mac, nor am I looking over your shoulder to see what you might be doing wrong, so you're going to have to resolve this issue yourself. To me, the errors you showed above look like the main sdm file (sdm) is corrupted somehow.
Another approach to using sdm on a Mac is to use a Debian VM, as ljames8 did here: https://github.com/gitbls/sdm/issues/175

Thank you for your time<3

@odi89

Thank you for your time<3
Sorry...you're using two technologies that I don't, most importantly, the Mac.

If you're interested in trying to chase down why sdm doesn't run on the Mac, I'm willing to help, but it will take some investigation and likely a fair amount of work/testing on your part. The failure you showed with the two sdm commands running on Bash on the Mac (sans docker) indicate something strange and fundamental is wrong.
One last question: What version of bash is on your mac? bash --version

sorry. for late respons, did not see this.
I solved most of my use cases, but i did not get to use sdm.
But i would love to help out, and i would love to get sdm working in docker on mac or just nativly on mac.
you are right, Mac now uses zsh as default shell.
i tried to use bash instead, but same problem
auto-setup-docker/docker on  main [!] on ☁<EMAIL_ADDRESS>❯ sdm mount 2024-03-15-raspios-bookworm-arm64-full.img
? Please run as root: sudo /usr/local/bin/sdm mount 2024-03-15-raspios-bookworm-arm64-full.img

auto-setup-docker/docker on  main [!] on ☁<EMAIL_ADDRESS>❯ sudo sdm 2024-03-15-raspios-bookworm-arm64-full.img
/usr/local/bin/sdm: line 315: ${1,,}: bad substitution
/usr/local/bin/sdm: line 365: ${dimg,,}: bad substitution
? No command specified (--aptmaint --burn, --customize, --explore, --mount, --runonly)

auto-setup-docker/docker on  main [!] on ☁<EMAIL_ADDRESS>❯ bash
bash-5.2$ sudo sdm mount 2024-03-15-raspios-bookworm-arm64-
/usr/local/bin/sdm: line 315: ${1,,}: bad substitution
/usr/local/bin/sdm: line 365: ${dimg,,}: bad substitution
? No command specified (--aptmaint --burn, --customize, --explore, --mount, --runonly)
bash-5.2$ sudo sdm mount 2024-03-15-raspios-bookworm-arm64-full.img
/usr/local/bin/sdm: line 315: ${1,,}: bad substitution
/usr/local/bin/sdm: line 365: ${dimg,,}: bad substitution
? No command specified (--aptmaint --burn, --customize, --explore, --mount, --runonly)
bash-5.2$

again, sorry for late reply! Will check this thread for further information=)
Cheers, from your Mac guy in Norway

@odi89  Would you please try the following commands on the console and post the output here?
# start a bash shell from zsh
which bash
/bin/bash
# Print the bash version
bash-5.2$ bash --version
bash-5.2$ sudo sdm --version
# Now let's see if the ${var,,} construct works
bash-5.2$ s="ABCDEFG"
bash 5.2$ echo ${s,,}

Thx!

@gitbls
So sorry for taking so long. And thanks for keeping up.
I did manage to get this to work to my liking
Running and customizing images but not burning the image to disk.
Using a docker container with debian and theese dependencies (described in another thread)
    sudo \
    curl \
    unzip \
    fdisk \
    dosfstools \
    rsync \
    parted \
    kpartx \
    qemu-utils \
    qemu-user-static \
    qemu-system-arm \
    qemu-efi-aarch64 \
    ipxe-qemu \
    qemu-efi-arm \
    qemu-system-gui \
    systemd-container \
    binfmt-support \
    util-linux \
    xz-utils \
    zip \
    bzip2 \
    file \
    less \
    && rm -rf /var/lib/apt/lists/*

I'm running into the same issue, I tried installing Bash 5.2 via Homebrew and still getting the same error

I'm running into the same issue, I tried installing Bash 5.2 via Homebrew and still getting the same error

Sorry about that. If you're on a Mac, I have no explanation other than there's an issue with the bash you're using, since sdm works perfectly fine on RasPiOS, Windows WSL, X64-based Ubuntu and Debian.
To me, the fact that this error only shows up on Macs is highly indicative of a problem with the Mac's Bash.
I've asked previously in this thread for some information that would be helpful for me to identify the problem, but so far nobody has provided it.
If you're interested and willing to work with me to chase this down, then please start by providing the information I requested here.

@gitbls Sorry for the huge delay getting back to you with this!

start a bash shell from zsh
bash-3.2$ which bash

/bin/bash

Print the bash version
bash-3.2$ bash --version

GNU bash, version 3.2.57(1)-release (arm64-apple-darwin23)
Copyright (C) 2007 Free Software Foundation, Inc.

bash-3.2$ getopt --version

 --

bash-3.2$ sudo sdm --version

/usr/local/bin/sdm: line 315: ${1,,}: bad substitution
/usr/local/bin/sdm: line 365: ${dimg,,}: bad substitution
? No command specified (--aptmaint --burn, --customize, --explore, --mount, --runonly)

Now let's see if the ${var,,} construct works
bash-3.2$ s="ABCDEFG"
bash-3.2$ echo ${s,,}

bash: ${s,,}: bad substitution

If I switch to the Homebrew version...

% which bash

/opt/homebrew/bin/bash

% bash
bash-5.2$ bash --version

GNU bash, version 5.2.32(1)-release (aarch64-apple-darwin23.4.0)
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

bash-5.2$ getopt --version

--

bash-5.2$ sudo sdm --version

/usr/local/bin/sdm: line 315: ${1,,}: bad substitution
/usr/local/bin/sdm: line 365: ${dimg,,}: bad substitution
? No command specified (--aptmaint --burn, --customize, --explore, --mount, --runonly)

bash-5.2$ s="ABCDEFG"
bash-5.2$ echo ${s,,}

abcdefg

...I also tried installing getopt from Homebrew:

% brew install gnu-getopt
% echo 'export PATH="/opt/homebrew/opt/gnu-getopt/bin:$PATH"' >> ~/.bashrc
% echo 'export PATH="/opt/homebrew/opt/gnu-getopt/bin:$PATH"' >> ~/.zshrc
closed all shells...

bash-5.2$ getopt --version

getopt from util-linux 2.40.2%

...But same error when I run sudo sdm --version.

@gitbls Sorry for the huge delay getting back to you with this!

bash-3.2$ which bash

/bin/bash

bash-3.2$ bash --version

GNU bash, version 3.2.57(1)-release (arm64-apple-darwin23)
Copyright (C) 2007 Free Software Foundation, Inc.

bash-3.2$ getopt --version

 --

bash-3.2$ sudo sdm --version

/usr/local/bin/sdm: line 315: ${1,,}: bad substitution
/usr/local/bin/sdm: line 365: ${dimg,,}: bad substitution
? No command specified (--aptmaint --burn, --customize, --explore, --mount, --runonly)

bash-3.2$ s="ABCDEFG"
bash-3.2$ echo ${s,,}

bash: ${s,,}: bad substitution

If I switch to the Homebrew version...

% which bash

/opt/homebrew/bin/bash

% bash
bash-5.2$ bash --version

GNU bash, version 5.2.32(1)-release (aarch64-apple-darwin23.4.0)
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

bash-5.2$ getopt --version

--

bash-5.2$ sudo sdm --version

/usr/local/bin/sdm: line 315: ${1,,}: bad substitution
/usr/local/bin/sdm: line 365: ${dimg,,}: bad substitution
? No command specified (--aptmaint --burn, --customize, --explore, --mount, --runonly)

bash-5.2$ s="ABCDEFG"
bash-5.2$ echo ${s,,}

abcdefg

...I also tried installing getopt from Homebrew:

% brew install gnu-getopt
% echo 'export PATH="/opt/homebrew/opt/gnu-getopt/bin:$PATH"' >> ~/.bashrc
% echo 'export PATH="/opt/homebrew/opt/gnu-getopt/bin:$PATH"' >> ~/.zshrc

...Closed all shells...

bash-5.2$ getopt --version

getopt from util-linux 2.40.2%

...But same error when I run sudo sdm --version.

Simply put: I don't understand what point you're trying to make.
I see that the default bash is super-old ( GNU bash, version 3.2.57(1)-release (arm64-apple-darwin23)), while Debian Bookworm has 5.2.15(1), so that's clearly a problem.
I don't  understand what "getopt" is, or has anything to do with any of this.
The issue with sdm on MacOS is only going to get fixed if someone wants to step up and work closely with me on this, committing to test things that I ask and provide timely responses.

@gitbls Apologies, I was simply trying to provide as detailed an error case as possible given that's what you asked for here.
I tried it with both the system Bash as well as one installable from Homebrew because the version of Bash that ships with MacOS is brutally old and a lot of stuff attached to with it is incompatible with what you'd expect in the average Linux distribution; you can install a more recent Bash from Homebrew (5.2.32 as per the output), but it will generally lack a lot of the niceties you'd expect (ergo, installing gnu-getopt, which is ultimately the bash completion extensions provided by getopt, I think?). I only investigated it because you specifically asked for that in that snippet you requested people run to troubleshoot.
I'm happy to troubleshoot it with you but I've sort of moved on to running sdm exclusively in Linux, where it works pretty flawlessly; mainly providing the output there because I remembered this issue and thought it might be helpful because nobody else seemed to be as per your request.

Apologies if I misunderstood your intent. As you might infer, I'm frustrated by this issue.
Since the ${symbol,,} construct works in isolation from the command line but fails in sdm, I'll need to do a bit more code review, and hope to find a few simple tests that I will ask you to try.
Could you confirm the sdm version that you were using?
Thanks!

No worries, totally understood! I can get that it's especially annoying given how well sdm runs literally everywhere else and this is an issue very much due to Apple's lacklustre UNIX implementation!
According to my reading of the /usr/local/bin/sdm's source (I can't run sudo sdm --version because it crashes due to the aforementioned issue; looking at line 196), I think I'm on 12.6.


[DT] Move to use ResponseError

Fixes: https://github.com/Azure/azure-sdk-for-net/issues/25054
Same as PR https://github.com/Azure/azure-sdk-for-net/pull/24943, It uses ResponseError instead of DocumentTranslationError. It follows the structure recommended by the archboard when presenting errors.
It also fixes method names in the ModelFactory class and adds respective tests.

API changes have been detected in Azure.AI.Translation.Document. You can review API changes here

/check-enforcer override


Problem getting the first and last name

I am calling getResourceOwner($ token); as indicated by the repo, but this returns me null, for the first_name and last_name.
The Repo indicates that by default it brings name and email.
I also tried adding it to the scope, but it doesn't work for me either.
This does not return.
= ( Help!

Did you tried to go to appleid.apple.com and delete your app from the used apps list.
Then register as a new user to your app.
You should see your name details on $_POST / $_GET (Query params or Body) and then you can call ->getFirstName() / ->getLastName(). - when you requested the scope.
It's bad but from Apple: You'll get the name only once at first registration.

Did you tried to go to appleid.apple.com and delete your app from the used apps list.
Then register as a new user to your app.
You should see your name details on $_POST / $_GET (Query params or Body) and then you can call ->getFirstName() / ->getLastName(). - when you requested the scope.
It's bad but from Apple: You'll get the name only once at first registration.

Thank you very much you can get it through $ _POST
I tried with getLastName () but it returned null. An example of how to obtain it in this way would be helpful.

Thank you very much you can get it through $ _POST
I tried with getLastName () but it returned null. An example of how to obtain it in this way would be helpful.

You must ensure that the application is deleted on appleid.apple.com!
So when your application name is "Example app" then it shouldnt be in the Application list on appleid.apple.com.
Then register by requesting the scope.
https://github.com/patrickbussmann/oauth2-apple#managing-scopes
$options = [
    'scope' => ['name', 'email']
];

$authorizationUrl = $provider->getAuthorizationUrl($options);

And then simply...
printf('Hello %s %s!', $user->getFirstName(), $user->getLastName());

You must ensure that the application is deleted on appleid.apple.com!
So when your application name is "Example app" then it shouldnt be in the Application list on appleid.apple.com.
Then register by requesting the scope.
https://github.com/patrickbussmann/oauth2-apple#managing-scopes
$options = [
    'scope' => ['name', 'email']
];

$authorizationUrl = $provider->getAuthorizationUrl($options);

And then simply...
printf('Hello %s %s!', $user->getFirstName(), $user->getLastName());


Shell extension has no effect / not shown in side or top bar

Hello,
I was very happy to find this extension, thank you for that. It looks great in the screenshots. Unfortunately, I can't seem to make it work. Turning it on has no effect, I can't see any (graphical) hint of it.
I am using fedora 39 with Gnome 45. I am using Dash to panel but there is no difference in turning it off. I can provide more information if you'd like.

Sure thing, more info would be great, looking glass and/or journal


Add "shortest path to closest point"

Hi there,
Thanks for your very nice library! It is very easy to use, although I modified Grid to use my underlying data structure directly.
I was wondering if you could add a version (e.g., GetPathToClosest() or an option to GetPath) that would return a path to the closest point to the destination, if the destination itself could not be reached, please. This would be useful, for example, when using it for pathfinding for monsters who are extremely numerous and are "mobbing" the player, assuming of course that only one monster can be in a cell and hence would BlockCell their locations.
The shortest path, if multiple end up equidistant, would be preferred, of course.
(Yes, I know a non-infinite cell cost could be used as well.)
Thanks,
Doug

Hey Doug,
Great to hear that you're using the library. I think this is a good feature to add. I think the logic will be quite different than from the normal A* path finding so I will have to think about the implementation. But we can probably do something with keeping track of the path that got the closest 'as the crow flies' distance to the desired location.  But there is a problem:
At the moment there is no distinction between a path blocked by an immovable object, or by an object that might move out of the way. So we could end up with a path that leads to a wall, meaning the agent will get near the object it desires to be, but even after fighting cannot reach it.

Posting this here instead of on #14
The general idea of 'best' path, from what I can tell, is taking the node that had the lowest H score in the case of no full path found. I just made some changes on my end to test this out and I think it works.

In Grid.cs, "GetPath" becomes:

public Position[] GetPath(Position start, Position end, Offset[] movementPattern, int iterationLimit, out bool fullPathNotFound)
{
        var current = PathFinder.FindPath(this, start, end, movementPattern, iterationLimit, out fullPathNotFound);
        [...]
}

In Pathfinder.cs "FindPath" method becomes:

public static List<Position> FindPath(Grid grid, Position start, Position end, Offset[] movementPattern, int iterationLimit, out bool fullPathNotFound)
{
            ClearStepList();

            fullPathNotFound = false;

            if (start == end)
            {
                return new List<Position> { start };
            }

            var head = new MinHeapNode(start, ManhattanDistance(start, end));
            MinHeapNode lowestHNode = head;
            var open = new MinHeap();
            open.Push(head);

            var costSoFar = new float[grid.DimX * grid.DimY];
            var cameFrom = new Position[grid.DimX * grid.DimY];

            while (open.HasNext() && iterationLimit > 0)
            {
                // Get the best candidate
                var currentNode = open.Pop();
                var current = currentNode.Position;
                MessageCurrent(current, PartiallyReconstructPath(grid, start, current, cameFrom));
                if (currentNode.ExpectedCost < lowestHNode.ExpectedCost)
                    lowestHNode = currentNode;
                if (current == end)
                {
                    return ReconstructPath(grid, start, end, cameFrom);
                }

                Step(grid, open, cameFrom, costSoFar, movementPattern, current, end);

                MessageClose(current);

                --iterationLimit;
            }

            fullPathNotFound = true;
            return ReconstructPath(grid, start, lowestHNode.Position, cameFrom);

            //return null;
        }

Also in my project, if the path kept coming back with "fullPathNotFound==true" (ie 2 in a row), then I would tell the unit to do something else, since obviously it can't make it to whatever it is trying to.

Hey cclog, Thanks for this. I think its a great idea to implement it like something like this. Its gonna take me a few weeks to get back to this issue (very busy weeks here, I'm helping with the organisation of a conference). So I wanted to add this comment here to show its not forgotten :).

I've thought about this more and more and I think every game has a different idea of "shortest to closest point", like what the closest point is (a closed door 10 meters away or an impenetrable wall 1 meter from the objective). For open areas that are congested with enemies its more clear. So I'm still not entirely sure what to do with this.
Maybe I can come up with an idea on how to give developers the tools to measure different types of closest points, and then let them plot a path there?

Hi Roy,
This is an interesting idea. However, perhaps it would be best to hear from other users of your library to discern whether and how they might prefer this, rather than just taking one person's opinion.
I "solved" the problem "for now" by making nothing "impassible" (infinite value) but making game-impassible things very high path cost (like closed doors, other mobile objects) and letting the pathing algo do its thing and then not allowing the path to be followed at the appropriate point.
Cheers!


[wasm] Include syslog.h only if available

wasi-libc does not provide syslog.h, so we should only include it if it is available. syslog APIs are only used for debugging code guarded under DEBUG_URL_MEMORY_USAGE, so it is safe to exclude it.

@swift-ci test


Remove extra line

Thanks for contributing. I dislike writing multiple instructions on a single line. It's a deliberate choice to write the .visit() call on a second line.


feat(values) Support string values for affinity and topologySpreadConstraints

What this PR does
This PR adds support for both string and object/array values for affinity and topologySpreadConstraints. String values will be fed through helms tpl function which allows users to use both helpers and chart values. This can be used for example to select a specific version or kong deployment.
Checklist

[x] PR is based off the current tip of the main branch.
[x] Changes are documented under the "Unreleased" header in CHANGELOG.md
[x] Commits follow the Kong commit message guidelines

Hi, thank you for your comment! The reason tpl is used is because I want to use .Chart.version and the release name for affinity and topology constraints. I have multiple Kong deployments on a single cluster which need to be balanced across nodes

Are you able to handle those out-of-band? If you need multiple values.yaml configurations, you can generate them prior to installing the chart.

This change would allow us to handle these things without introducing additional tooling.

Understood, but unless there's demonstrated need from a number of users for the same functionality we default ask you put that additional tooling in place.


Update README with Project Details

Add a comprehensive description to the README, including instructions on how to run the application, the features it includes, and a link to the walkthrough video.
Acceptance Criteria: README file should include:
Project description
How to install dependencies
Instructions to run the application with npm start
Walkthrough video link
Any known limitations or notes on additional features

The README file has been updated with comprehensive project details. It now includes the following:

Project Description: A brief overview of the project, its purpose, and key features.
Dependency Installation: Instructions on how to install all necessary dependencies.
Running the Application: Step-by-step instructions on how to start the application using npm start.
Walkthrough Video Link: A link to the walkthrough video demonstrating the application's functionality.
Additional Notes: Information on any known limitations or notes on additional features.

All acceptance criteria have been fulfilled, making the README more informative and user-friendly. Based on these updates, I am closing this issue as resolved.


Downloadable Models for different datasets

@riedelcastro where can we host those
@dirkweissenborn SQuAD
@pminervini SNLI
@mitchelljeff KBP?

@pminervini the embeddings are not stored with the model anyways. What @riedelcastro suggested is to convert embeddings that need to be downloaded anyways to mmap, such that we do not have to load all of them at once.

@pminervini @TimDettmers any updates on an SNLI model? Can someboday train one?

@pminervini could you replace the old fastqa model with the newer version that uses memory maps? It is the same dropbox link.

@dirkweissenborn done

@pminervini sorry the model was only renewed now, can you do it again?
https://www.dropbox.com/s/wasdpmmpuywpdlm/fastqa.zip?dl=0

@dirkweissenborn done!

@pminervini when you have time, could you also upload the memory map embeddings (see the doc on how to create them), and download them with the glove download script? (Maybe zipping and unzipping accordingly)? This would streamline the setup.

@undwie sure!

Great! let me know when it's up!

@undwie here you go: http://data.neuralnoise.com/jack/embeddings/

Most awesome!
On Tue, Oct 24, 2017 at 5:50 PM Pasquale Minervini<EMAIL_ADDRESS>wrote:

@undwie https://github.com/undwie here you go:
http://data.neuralnoise.com/jack/embeddings/
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/uclmr/jack/issues/261#issuecomment-339056975, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AdMriKktKC9BEo-IMXZcLHrEqnDYp57zks5svhVMgaJpZM4Pu0tu
.

@undwie the different options for the pre-trained embeddings are now word2vec, glove, memory_map_dir - my understanding is that memory_map_dir is like glove but faster, is that right ?

Have a look at the documentation. Memory map loads embeddings lazily from
disk, only when needed, instead of pre-loading everything into memory.
On Tue, Oct 24, 2017 at 8:19 PM Pasquale Minervini<EMAIL_ADDRESS>wrote:

@undwie https://github.com/undwie the different options for the
pre-trained embeddings are now word2vec, glove, memory_map_dir - my
understanding is that memory_map_dir is like glove but faster, is that
right ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/uclmr/jack/issues/261#issuecomment-339101688, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AdMriNBCrg1Cwk1qmG0ulDmrt28IzQZVks5svjg9gaJpZM4Pu0tu
.

@undwie thank you - I've been going through the documentation, but I was thinking it was tied to GloVe.

No, you can load embeddings from any other format, and then save it in a memory map directory.

@dirkweissenborn did you manage to train fastqa with the memory map files? For me things got much slower at training time, even though startup and memory footprint at test time is much better. But I am not sure right now if the slow training comes from the embeddings, or from other changes.

@riedelcastro I did not train the model but only changed the configuration yaml in the shared resources to use mem_map. However, I found that training time was not affected by that much (10%), if at all. I have the mem map embeddings on SSD which should make a difference compared to an HDD. Anyway, I could try to provide a quick fix for that by caching embeddings in the input module. Over time, training should become faster then. What do you think?

If it only hurts you by ~5%, this looks okay for now. Maybe something else is going on for me. On that note, I still haven't been able to reproduce your F1 results. I am always ending up somewhere at 72. @mitchelljeff, have you trained FastQA on squad with results in the F1=74% range?

@riedelcastro can you bin/jack-eval.py the pre-trained model on the SQuAD dev set, just to be sure that it is really the training that is the problem for you?
Btw., I get similar results with my pyTorch implementation, so the results seem solid.

Yes, I will do this. But in any case, the training does produce different intermediate outputs it seems.

I never got TF to be completely deterministic, either. However, results shouldn't change too much.

Yeah. It's weird: for me, I might have some variance, but I relatively consistently get around 72%.

Did you ever train it on a GPU? I am really curious now...

Not yet. Interestingly, in this run, after 10 iterations, I get to about 74. I guess it is indeed very non-deterministic. On the plus side, the memory map embeddings seem to "work" in the sense that you can train with them to get similar results. This wasn't properly tested before.


Some graph type not rendering, despite manual creating working on mermaid.ink

Hello,
I can run "graph TD" without any problem, so I assume plugin installation is OK.
Upon running "sequence Diagram", the img tag is generated but the image link returns a "not found image".
Could be that the link is truncated?
Example source:
`
sequenceDiagram
participant Alice
participant Bob
Alice->>John: Hello John, how are you?
loop Healthcheck
John->>John: Fight against hypochondria
end
Note right of John: Rational thoughts
John-->>Alice: Great!
John->>Bob: How about you?
Bob-->>John: Jolly good!
`

Please update to v1.1.1 and try again

... and we're good ! congratulations (again)!


Fix HACS installation issue

If you try to install https://github.com/themooer1/keepsmile_homeassistant using HACS, it will be failed. You can find these lines in logs:
This error originated from a custom integration.

Logger: custom_components.hacs
Source: custom_components/hacs/repositories/base.py:986
integration: HACS (documentation, issues)
First occurred: 9:09:04 AM (4 occurrences)
Last logged: 9:26:24 AM

<Integration themooer1/keepsmile_homeassistant> Failed to download https://github.com/themooer1/keepsmile_homeassistant/releases/download/main/keepsmileled.zip

This PR fixes the issue

Thanks for the work on this, still does not work for me...am I missing something b

still not working, can you please fix


Rename developer-facing Types

For example, I suggest to rename:

VDomModifier to Modifier

Any more suggestions?

I think that is a good idea, I do not like the VDomModifier name either. What would you do about VNode? VNode has the V prefix, but Modifier would not. Would VModifier make sense or should we have Node or is Node a too generic name.

I think VNode is fine. It never felt too long to write. Node seems to be indeed to generic.

I think we should have VNode and VModifier. It communicates more clearly that they belong to the same Concept.
We should not rename VNode to Node since that would cause collisions with dom.Node which is often used in the same context.

I think we should have VNode and VModifier. It communicates more clearly that they belong to the same Concept. We should not rename VNode to Node since that would cause collisions with dom.Node which is often used in the same context.

@busti agreed to everything you said!

I have this change hidden in a huge PR - again - that is #441. There I changed VDomModifier to Modifier. Should I change it to VModifier then? I like that is symmetric to VNode.

Yes, just do it.


chore: updated readme on private end point

PR summary
updated readme on private end points.
Fixes: 5450
PR Checklist
Please make sure that your PR fulfills the following requirements:

[x] The commit message follows the Go Commit Message Guidelines.
[ ] Tests for the changes have been added (for bug fixes / features)
[x] Docs have been added / updated (for bug fixes / features)

PR Type

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] New tests
[ ] Build/CI related changes
[x] Documentation content changes
[ ] Other (please describe)

What is the current behavior?
More explanation on private end point was missing.
What is the new behavior?
Given more details on private end point configuration
Does this PR introduce a breaking change?

[ ] Yes
[x] No

Other information

Signed-off-by: Nitish Kulkarni<EMAIL_ADDRESS>

:tada: This PR is included in version 0.1.6 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Env var interpolation in repositories

Would it be possible to enhance the env var interpolation feature, so that we could safely git-commit a helmfile referencing private chart repositories which requires authn?
For example, we can use private git repos as helm chart repos like described in https://github.com/kmzfs/helm-repo-in-github#how-to-use-it-as-a-helm-repo.
However, it requires you to include a sensitive information(=auth credential) in repository urls.
Simply doing it results in a charts.yaml like:
respositories:
- name: your-private-git-repo-hosted-charts
  url<EMAIL_ADDRESS>
which ins't very good to be committed into your git repo.
If the repositories section had a support for env var interpolation, it would look like:
respositories:
- name: your-private-git-repo-hosted-charts
  url: "https://{{ env<EMAIL_ADDRESS>
This makes helmfile safe enough to be committed into a git repo.
Note that running helm sync would result in printing the repository url containing the token like:
exec: helm repo add your-private-git-repo-hosted-charts<EMAIL_ADDRESS>
I'd like to fix it, too. However, it could be done in an another github issue.

Should we close this one? What's the practice around here?

@mumoshu This might be closed as well?

@cmeury Yes, thanks for the reminder ☺


Some prerequisites of running pipeline in a local environment are missing

Desription
When following the Running pipelines in a local environment tutorial to step Running a notebook pipeline locally, a user may see the following error message:

The error details is:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/elyra/pipeline/processor_local.py", line 191, in process
    papermill.execute_notebook(
  File "/usr/local/lib/python3.9/site-packages/papermill/execute.py", line 118, in execute_notebook
    raise_for_execution_errors(nb, output_path)
  File "/usr/local/lib/python3.9/site-packages/papermill/execute.py", line 230, in raise_for_execution_errors
    raise error
papermill.exceptions.PapermillExecutionError:
---------------------------------------------------------------------------
Exception encountered at "In [2]":
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-2-dd0d2a848dc3> in <module>
      1 # Define required imports
----> 2 import pandas as pd
      3 import numpy as np
      4 import sys
      5 import re

ModuleNotFoundError: No module named 'pandas'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/elyra/pipeline/processor_local.py", line 74, in process
    operation_processor.process(operation)
  File "/usr/local/lib/python3.9/site-packages/elyra/pipeline/processor_local.py", line 197, in process
    raise RuntimeError(f'Internal error executing {filepath}: {ex}') from ex
RuntimeError: Internal error executing /Users/xuanyu/examples/pipelines/hello_world/Part 1 - Data Cleaning.ipynb:
---------------------------------------------------------------------------
Exception encountered at "In [2]":
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-2-dd0d2a848dc3> in <module>
      1 # Define required imports
----> 2 import pandas as pd
      3 import numpy as np
      4 import sys
      5 import re

ModuleNotFoundError: No module named 'pandas'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/tornado/web.py", line 1704, in _execute
    result = await result
  File "/usr/local/lib/python3.9/site-packages/elyra/pipeline/handlers.py", line 89, in post
    response = await PipelineProcessorManager.instance().process(pipeline)
  File "/usr/local/lib/python3.9/site-packages/elyra/pipeline/processor.py", line 70, in process
    res = await asyncio.get_event_loop().run_in_executor(None, processor.process, pipeline)
  File<EMAIL_ADDRESS>line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/site-packages/elyra/pipeline/processor_local.py", line 79, in process
    raise RuntimeError(f'Error processing operation {operation.name}.') from ex
RuntimeError: Error processing operation Part 1 - Data Cleaning.

Possible cause and a potential fix
The reason is that the package pandas was not installed. When we open the notebook Part 1 - Data Cleaning.ipynt, we'll see this screen:

Installing pandas by running pip3 install pandas can resolve this problem.
Suggestion
It is a small thing and can be easily fixed. However, I don't find anywhere in the documentation of installing Elyra or the documentation of Running pipelines in a local environment that says that pandas is one of the prerequites.
So, we may want to tell a user at Prerequisites section that they need to make sure that pandas along with other packages like numpy should be there before running the pipeline.
Reproduce steps
Environment:

macOS Big Sur.
Python 3.9.1

Steps:

install Elyra by running pip install --upgrade "elyra>=2.0.1" && jupyter lab build
following the tutorial: https://github.com/elyra-ai/examples/tree/master/pipelines/hello_world#running-a-notebook-pipeline-locally.
make sure the Runtime Image is Pandas 1.1.1, just like the screenshot from the tutorial
run the pipeline

Hi @xuanyuwang , thank you for opening this issue.
I think it's a good idea to let users know about pandas dependency in the hello world example prerequisites.
@ptitzler ?

In general I agree. However, to keep things simple and portable the first cell in each tutorial notebook includes one or more !pip install, which should install the prerequisite packages in the kernel where the notebook is running. For example, in https://github.com/elyra-ai/examples/blob/master/pipelines/hello_world/Part 1 - Data Cleaning.ipynb

In what kind of environment are you running?

In general I agree. However, to keep things simple and portable the first cell in each tutorial notebook includes one or more !pip install, which should install the prerequisite packages in the kernel where the notebook is running. For example, in https://github.com/elyra-ai/examples/blob/master/pipelines/hello_world/Part 1 - Data Cleaning.ipynb

In what kind of environment are you running?

I'm running on

macOS 11.2.1
Darwin Kernel Version 20.3.0 x86_64
Python 3.9.1
pip 21.0.1 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)

One problem in my environment for the command !pip install PyGithub pandas is that I do NOT have pip available. Instead, I have pip3.
If I change the command to !pip3 install PyGithub pandas > /dev/null 2>&1, then I don't see the problem.

Excellent! It appears that in your environment there is no pip symlink defined for pip3. We ran into a similar problem in the context of a completely different issue. To solve the problem for everyone we need to update all tutorial notebooks in https://github.com/elyra-ai/examples/tree/master/pipelines to use pip3 instead of pip, because that command is always available in a Python 3 environment.
Would you be interested in making those changes and creating a pull request?
The easiest and fastest way to apply such a change without any side effects would be to open every .ipynb file with a text editor and perform a global replace of  !pip install with !pip3 install.

Excellent! It appears that in your environment there is no pip symlink defined for pip3. We ran into a similar problem in the context of a completely different issue. To solve the problem for everyone we need to update all tutorial notebooks in https://github.com/elyra-ai/examples/tree/master/pipelines to use pip3 instead of pip, because that command is always available in a Python 3 environment.
Would you be interested in making those changes and creating a pull request?
The easiest and fastest way to apply such a change without any side effects would be to open every .ipynb file with a text editor and perform a global replace of !pip install with !pip3 install.

Sure I'd like to make a pull request. I'll let you know when the PR is ready.


Trouble installing

I've been trying to install polybar from the Arch User Repository, but each time I keep getting this error message.
/tmp/yaourt-tmp-norbzilla/aur-polybar/src/polybar/lib/i3ipcpp/include/i3ipc++/ipc-util.hpp:147:57: error: dynamic exception specifications are deprecated in C++11 [-Werror=deprecated]  std::shared_ptr<buf_t>   i3_recv(const int32_t  sockfd) throw (invalid_header_error, eof_error);                                                          ^~~~~ In file included from /tmp/yaourt-tmp-norbzilla/aur-polybar/src/polybar/lib/i3ipcpp/src/ipc.cpp:9:0: /tmp/yaourt-tmp-norbzilla/aur-polybar/src/polybar/lib/i3ipcpp/include/i3ipc++/ipc-util.hpp:163:131: error: dynamic exception specifications are deprecated in C++11 [-Werror=deprecated]  int32_t  sockfd, const ClientMessageType  type, const std::string&  payload = std::string()) throw (invalid_header_error, eof_error);                                                                                ^~~~~ cc1plus: all warnings being treated as errors make[2]: *** [lib/i3ipcpp/CMakeFiles/i3ipc++.dir/build.make:63: lib/i3ipcpp/CMakeFiles/i3ipc++.dir/src/ipc.cpp.o] Error 1 make[1]: *** [CMakeFiles/Makefile2:440: lib/i3ipcpp/CMakeFiles/i3ipc++.dir/all] Error 2 make: *** [Makefile:130: all] Error 2 ==> ERROR: A failure occurred in build().     Aborting... ==> ERROR: Makepkg was unable to build polybar.
I don't know if it's my computer or not, but I wanted to let you know.

It's not just your computer.  It's occurring on my freshly installed Arch Linux + Gnome/GDM + i3 install.

I think this is due to GCC that has been updated five days ago. The current version is now 7.1.1.
From [url=https://www.reddit.com/r/archlinux/comments/6e0g7c/gcc_711_is_in_core/]this thread on reddit[/url] :
[quote]So if you are having troubles building polybar after that - switch to clang temporary. Install clang and add this to the PKGBUILD before cmake in the build() section:
export CC=/usr/bin/clang
export CXX=/usr/bin/clang++
this might work for other packages too, as a workaround for everything that relies on now deprecated stuff in GCC[/quote]

@Dupond Yeah, add those two lines to the top of the build function/procedure in the PKGBUILD.

This indeed allowed the source to be built and the package from the AUR to be installed.  Thanks.

So if you are having troubles building polybar after that - switch to clang temporary. Install clang and add this to the PKGBUILD before cmake in the build() section:
export CC=/usr/bin/clang
export CXX=/usr/bin/clang++

This sounds really good - I have developed a useful Polybar on a VM but can not now install it to my laptop, I guess this is the 'Linux Experience' ;-).
Could I ask for a little more detail (I really am a Linux noob!) - where would I find the PKGBUILD?

@LinuxNoob24 If you use pacaur or yaourt, it will ask you if you want to edit PKGBUILD, select Y or yes.  Edit that with your favorite editor and go through the rest of the steps at the arch wiki.

@LinuxNoob24 If you use pacaur or yaourt, it will ask you if you want to edit PKGBUILD, select Y or yes.  Edit that with your favorite editor and go through the rest of the steps at the arch wiki.

@aurieh - that is brilliant - thank you so much for your help, I can now complete my project.

@Dupond yeah, I will fix the issues but I currently don't have much time over for it. Might be better to update the PKGBUILD and add clang as a dep until its been fixed.

For you who are installing polybar using the AUR; I have updated the PKGBUILD so that clang will be used until the gcc issues have been fixed.

Closing as a duplicate of #572

ERROR: A failure occurred in build().
Aborting...
ERROR: Makepkg was unable to build polybar.
i got this error when trying to install polybar on my freshly installed manajro i3.what am i supposed to do?

@shahules Please don't post in a closed issue unless it's related. Open a new issue with more information (output of makepkg or cmake and make)


feat: update ES6 config per Gutenberg ESLint config

This PR migrates ESLint ES6 rules out of Gutenberg and into the eslint-plugin-wordpress package and the wordpress/es6 configuration.
Please see the companion pull request for Gutenberg linked below for further reference
cc @gziolo

Now I'm wondering if most of the rules already match in here with what we have in Gutenberg. If yes it seems like a very easy and straightforward change 👍
Can we rename/alias es6 to es-next?

At the moment the only rules in this ES6 config are from ESLint's ES6 rules:

https://eslint.org/docs/rules/#ecmascript-6

As there has never been a release of this config there is no issue in renaming or aliasing it...
After reading a some ESLints I think renaming it to es-next would be best as ESLint has no plan of creating specific ES7, ES8 configurations so grouping by individual ECMAScript release is not really required and es-next can cover this entire category:

https://eslint.org/docs/user-guide/configuring

"ecmaVersion - set to 3, 5 (default), 6, 7, 8, or 9 to specify the version of ECMAScript syntax you want to use. You can also set to 2015 (same as 6), 2016 (same as 7), 2017 (same as 8), or 2018 (same as 9) to use the year-based naming."

Sounds great 👍

Done, renames ES6 to esnext

Done, renamed ES6 to esnext in #142


Fix Okta connector tests

Hi @tovbinm,
Could you please tell me if I can close this issue?
@huongtn has just told me that this task is resolved.


deps: upgrade golangci-lint and action versions

$ golangci-lint run ./...
WARN [runner] The linter 'structcheck' is deprecated (since v1.49.0) due to: The owner seems to have abandoned the linter. Replaced by unused.
WARN [runner] The linter 'deadcode' is deprecated (since v1.49.0) due to: The owner seems to have abandoned the linter. Replaced by unused.
WARN [runner] The linter 'varcheck' is deprecated (since v1.49.0) due to: The owner seems to have abandoned the linter. Replaced by unused.

taking a look

@ysugimoto should be good now, after this PR (go mod tidy issue)

I will create another PR to upgrade to use go1.21 after this one.


In-App Purchase

Is it possible to create and/or manage In-App Purchases using fastlane?

That's not possible right now, but would be part of the deliver project.

Great!

Closing in favour of https://github.com/KrauseFx/deliver/issues/35, please subscribe to https://github.com/KrauseFx/deliver/issues/35


Debug output when searching for operations to perform

@jmazzitelli this helped me to find out that I need tho send Redeploy rather than redeploy.

Sorry, you were right again. I thought for some reason that resource.toString() shows also the supported operations, which it does not. 01467f2 is completely adopting of your proposal.


Use onDidStopChanging to reduce resource consumption

I use Atom as my writing editor, so I end up with some pretty long markdown files. I find that wordcount eats up CPU since it re-counts the whole document on every keystroke - when you are typing fast and the document is long I found this causes a spike in CPU usage.
Using onDidStopChanging instead of onChange would help a lot, since it would prevent the wordcount re-running as often.
Another alternative would be to add an option option to only do the wordcount on save

Constant recalculation should be solved with a debounce like the lodash version for example: https://lodash.com/docs#debounce
PRs welcome <3

Wouldn't that basically do the same thing as onDidStopChanging ? That function also just waits until 330milliseconds after the last keystroke before invoking, my first instinct is that the result would be the same as using debounce?

Well in the default configuration it would basically be the same, I just want to be sure the func will be executed which could be done with the maxWait parameter. As described in their docs:

[options.maxWait] (number): The maximum time func is allowed to be delayed before it’s invoked.

A little update on that: I meant actually throttle https://lodash.com/docs#throttle


Wrong calculations & cut off text

When using padding within the content it seems that text get's cut off so I assume something is calculated wrong. See the last word in the screenshot:

Also, a weird space is visible below the expanded content. This get's especially visible in the last item of an accordion:

This is my code:
class HelpIndex extends React.Component {
  static navigationOptions = {
    headerTitle: "Hilfe",
  };

  _renderSectionTitle(section) {
    return <View style={styles.content} />;
  }

  _renderHeader(section) {
    return (
      <View style={styles.header}>
        <Txt medium style={styles.headerText}>{section.title}</Txt>
      </View>
    );
  }

  _renderContent(section) {
    return (
      <View style={styles.content}>
        <View style={styles.contentInner}>
          <Txt style={styles.body}>{section.body}</Txt>
        </View>
      </View>
    );
  }

  render() {
    return (
      <Container>
        <ScrollContainer withPadding>
            <Accordion
              sections={faq}
              renderSectionTitle={this._renderSectionTitle}
              renderHeader={this._renderHeader}
              renderContent={this._renderContent}
              touchableProps={{
                underlayColor: 'transparent',
              }}
            />
        </ScrollContainer>
      </Container>
    );
  }
}

Applied styles:
export default StyleSheet.create({
  content: {
    backgroundColor: 'rgba(57,47,82,0.1)',
  },
  contentInner: {
    padding: 20,
  },
  container: {
    backgroundColor: '#fff',
  },
  headerText: {
    fontSize: 16,
    lineHeight: 24,
    color: '#59568B',
  },
  header: {
    padding: 20,
    borderBottomWidth: 1,
    borderBottomColor: 'rgba(57,47,82,0.1)',
  },
  body: {
    fontSize: 14,
    lineHeight: 22,
    color: '#392F52',
  },
});

Hey @iRoachie thanks, haha!
You can find the snack here:
https://snack.expo.io/@mxmtsk/collapsible-test

Hi mxmtsk,  in fact I'm having the same issue with margin, I managed to arrange my view using padding instead.

i have same issue

@Taglohner I haven't used any margin though...
@iRoachie Did you have a chance to take a look at this? We're close to rolling out our app and this is one of the last bugs we encounter

This is such a weird bug, seems like it's an issue with lineHeight, cause without setting it, it works as expected.
Everything seems to work after adding flex: 1 to the contentInner style. https://snack.expo.io/rJj30SMEQ

@mxmtsk are you sure this is not just a ScrollContainer issue? I had the same problem recently where I passed padding to style instead of contentContainerStyle.
Here's the link to the ScrollContainer issue

@iRoachie I can confirm that your "workaround" does work, so the issue is resolved for me (but maybe not for this library in general) - thanks!
@WarHatch Yes I'm sure, I'm not passing any styling to ScrollView in my component

Awesome glad I could help. Not sure how we could fix in the future, but I'll look into it more closely if someone else brings it up. Beautiful app man!

@iRoachie Thank you very much. Feel free to add it as an example for the component of how it can be styled if you like it so much.

Sure thing @mxmtsk let me know when your app goes live!

For anyone suffering with this strange bug...
I'm using Styled Components with React Native,
and giving flex: 1 styling to content container didn't work.
Instead, this one worked.
  flex: 1;
  flex-direction: row;
  flex-wrap: wrap;

Hope this can help somebody using Styled Components.


Validate and fix REX-Ray service scripts

When using systemd and ScaleIO, a user must manually update the rexray service to start after the ScaleIO scini service, otherwise REX-Ray will fail to start every time the system boots. There is a chance this can affect other drivers that we support now as well as ones we plan to support in the future.

@akutz @codenrhoden - do we want to add support for starting REX-Ray after other dependent services start or do we want to just make the REX-Ray service auto restart?
For instance, with ScaleIO, we could do either of the following:
start rexray service after scini service:
[Unit]
Description=rexray
Before=docker.service
After=scini.service

[Service]
EnvironmentFile=/etc/rexray/rexray.env
ExecStart=/usr/bin/rexray start -f
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process

[Install]
WantedBy=docker.service

Or we could make rexray service auto restart:
[Unit]
Description=rexray
Before=docker.service

[Service]
EnvironmentFile=/etc/rexray/rexray.env
ExecStart=/usr/bin/rexray start -f
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process

Restart=on-failure
RestartSec=5

[Install]
WantedBy=docker.service

Any opinions?

Auto restart is a terrible idea and we should not even begin to consider supporting it IMO. We should suss out possible dependencies. That's the only sane path forward.

@akutz I agree that auto-restart isn't the best solution for this specific issue, however I believe there are use-cases as to why we should support it outside of this scope.

As of right now, I can only think of ScaleIO having a service dependency.
@codenrhoden does Ceph present any service depency?

Sure. My response was specific to the question at hand, not in general. Thank you for clarifying :)

@codenrhoden does Ceph present any service dependency?

No. Since the rbd kernel module doesn't have a background process/service attached to it, there is nothing depend on. the module needs to be installed (e.g. modprobe), but that's expected to be taken care of more along the lines of /etc/modules or equivalent.

Will we be updating script to take account of ScaleIO dependency?

Hi @cduchesne,
I would like to move this back to 0.8.0 as there's no reason it should take longer than a few minutes to do via the unit file's dependency section. Is there a reason you moved it in the first place?

You're more that welcome to move it back and push the quick fix - I had moved it aside only due to transition to a containerized option in which the systemd service becomes less relevant.

Except this pretends that the containerized option, which when you speak of it in this context and relevant to the other feature designated for 0.8.0 is Docker specific, is the primary or even recommend approach. It's not either IMO. Not while Dockers plugin model is so deficient.
--
-a

On Jan 26, 2017, at 4:12 PM, Chris Duchesne<EMAIL_ADDRESS>wrote:
You're more that welcome to move it back and push the quick fix - I had moved it aside only due to transition to a containerized option in which the systemd service becomes less relevant.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

my $0.02... working with 3 ScaleIO Node cluster i had to use
Restart=always
RestartSec=5

if the cluster is starting up and MDM2 loads first before finding MDM1 and scini.service starts then it fails. The first on-failure is clean so it doesn't try and restart it.


Update http4s-async-http-client, ... to 0.21.22

Updates

org.http4s:http4s-async-http-client
org.http4s:http4s-circe

from 0.21.6 to 0.21.22.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.http4s" } ]

labels: library-update, semver-patch

Superseded by #79.


Summary ResponseParam doesn't work with JsonPropertyName

Hi. First of all, thanks for the awesome framework. At Rage4, we are moving most of our codebase to the FE.
We are not sure if it is a bug or not, but we have noticed the following behaviour:

Create a response record (SampleResponse) with, let's say, an IsActive property (boolean)
Add the JsonPropertyName attribute to be serialized to 'is_active'.
Add the Summary with s.ResponseParam(o => o.IsActive, "Entity status");

I would expect that the Swagger would have the correct description. Unfortunately, it's empty.
Is there any workaround? Something similar to s.Params, so it would accept the dictionary?
Thanks. Piotr

Thanks for reporting this.
It has been fixed in v<IP_ADDRESS>-beta.
Hoping to do a final release around 3rd of November.


Setting test mode will load the correct endpoint

Looks good. Do you by chance have any link to the docs to verify?

@barryvdh sure, they have pretty old URLs and designs and most banks send this in a PDF, you can correlate here http://www.migssupport.com/Testing/migsvpcphp/vpc_php_serverhost.html https://www.anz.com/australia/business/merchant/pdf/entrust-curl.pdf https://migs-mtf.mastercard.com.au/ma/

Thanks!


[isort] Add classes Config Option

ref https://github.com/charliermarsh/ruff/issues/1819

Looks great - thanks!


Implement several examples to the client application

I would like to implement several examples to the client application.
Would you be interested ?
Like this for example : #114

That would be awesome! You can see some Vue examples here; https://github.com/devworkshops/vuejs-minnesota-2019/tree/master/northwind-traders/src/views.
Let me know if you’re still keen.

As per above comment, I will be happy to accept a pull request for additional client apps or examples. Thanks!


correct various spelling errors in README

like it says - fix spelling errors

OK @gegles let's see if this is better...


Need to add Google font Roboto in the font list

Hello,
I want to add google font roboto in the font list. How can i add? I have added the 'robot' in the font array. But font is not loading. For that i have to include the style sheet link.
<link href=https://fonts.googleapis.com/css?family=Roboto rel=stylesheet>
Where i can add this? Is this possible? Please help me

In the configuration:
E.g.
font: ['Arial', 'tahoma', 'Courier New,Courier', 'Roboto','PlayfairDisplay','Dancing Script','Great Vibes'],

Hi, @subrahmanyaadiga
Just include the style in the same HTML as the editor.

Thanks @JiHong88 @MRB60


document webpack@5's support of workers

Just note - webpack@5 support workers (web workers and Node.js worker threads) out of box, you don't need this loader, it was deprecated someday (need improve docs)
/cc @chenxsan Do we have examples/guide for web workers and etc stuff?
Originally posted by @alexander-akait in https://github.com/webpack-contrib/worker-loader/issues/301#issuecomment-741898222

Let's close in favor https://github.com/webpack/webpack.js.org/issues/4341 (I think with example it will be easy to write)


Toggle on/off streamreader test if needed

Toggle on/off streamreader test if needed
By default it ON, hence should not affect any current tests.
To toggle ON no change required.
To toggle OFF use:
smoke_test.py --no-streamreader

To be used when calling from builder currently. Since we do not install ffmpeg currently.

This is very sensitive change, and I am not in favor of this, because the reason this test was added is to makes sure that FFmpeg installation (a version different from the one used for build) is properly integrated, which constitutes the most basic smoke test.
If this option is abused, the binary being tested will be in a bad shape. Can you elaborate on usecases?

This is to be used when calling from https://pytorch.org/get-started/locally/ and release validation workflows in https://github.com/pytorch/builder. Which do not install ffmpeg  currently.


ARROW-3318: [C++] Push down read-all-batches operation on RecordBatchReader into C++

Codecov Report

Merging #3097 into master will increase coverage by 1.09%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #3097      +/-   ##
==========================================
+ Coverage   87.06%   88.15%   +1.09%
==========================================
  Files         492      434      -58
  Lines       69104    65341    -3763
==========================================
- Hits        60164    57603    -2561
+ Misses       8843     7738    -1105
+ Partials       97        0      -97

Impacted Files
Coverage Δ

cpp/src/arrow/record_batch.h
100% <ø> (ø)
:arrow_up:

cpp/src/arrow/ipc/ipc-read-write-test.cc
99.12% <100%> (-0.01%)
:arrow_down:

cpp/src/arrow/record_batch.cc
90.57% <100%> (+0.89%)
:arrow_up:

python/pyarrow/ipc.pxi
67.53% <100%> (-0.83%)
:arrow_down:

cpp/src/plasma/thirdparty/ae/ae.c
72.03% <0%> (-0.95%)
:arrow_down:

cpp/src/arrow/util/thread-pool-test.cc
98.91% <0%> (-0.55%)
:arrow_down:

go/arrow/array/table.go

go/arrow/math/uint64_amd64.go

go/arrow/internal/testing/tools/bool.go

go/arrow/internal/bitutil/bitutil.go

... and 54 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a06624e...03216b3. Read the comment docs.


boards: nordic: nrf54h20dk: VPRs board runner config

Added basic support for west debug for nrf54h20
RISC-V cpus: nrf54h20_cpuppr and nrf54h20_cpuppr.
Note external jlink probe needs to be used.

@anangl @gmarull @bjarki-andreasen @tmon-nordic could you please test this on H20 DK?


Add .blend backup files to .gitignore

Additional Minor Changes:

Remove unnecessary back slash in .gitignore
Add comments in .gitignore
Remove .blend backup file

Looks good, thanks.


Remove eclipse settings from git

:tada: This issue has been resolved in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket: