official rpios bookworm issue

The official rpios bookworm release has changed the mount point /boot to /boot/firmware.  The current rpi-clone can back up, but the backup media is not bootable because the cmdline.txt file is not updated with the new PARTUUID.
/boot/cmdline.txt -> /boot/firmware/cmdline.txt
Consider adding lines like this after these variable assignments  "cmdline_txt=${clone}/boot/cmdline.txt"
if [ -h $cmdline_txt ] ; then
cmdline_txt=${clone}/boot/firmware/cmdline.txt
fi
This tests if the /boot/cmdline.txt is a link and changes it to the actual file.

Since I've updated to bookworm as well, looking at potential necessary fixes. I am looking at where to put this test.
Would this be right after line 1733?

Thanks, I'll look into this. For now it does not seem to be needed in my install. But it may be worthwhile for a future fresh install... A pity that rpi-clone does not seem to be maintained for quite some time anymore, and we have to rely on individual changes and updates such as the one pointed out by you.

I'll create a fix in my fork when I'm back home next week.

Rebuilt a Raspberry 4-System from scratch yesterday. After work I was surprised to find out that the backups I used to do with rpi-clone would not boot.
So I found your issues-discussion here and changed rpi-clone by only adding:
if [ -h $cmdline_txt ] ; then
cmdline_txt=${clone}/boot/firmware/cmdline.txt
fi
after line 1733.
To use it again in line 1741 will probably not work because you just changed cmdline_txt to point to a file (rather than a link) after line 1733 and check again whether it points to a link (which will always fail?). Line 1741 deals with cmdline.boot and not cmdline.txt? And from my poor understanding of what the code is doing here I think further changes are not needed here. Don't know about other parts of the code where /boot/cmdline.txt is used.
Bottom line: I cloned with just three lines added, it booted fine.
Hope that helps.

I just created a fix in my fork so rpi-clone creates a bootable clone for bookworm. I tested the fix with a lite 32 bit image only. Would be great if if somebody tests the fix for other images and reports any issues.

The changes around line 1733 are sufficient to make a normal backup bootable.  One of the other two changes are in an area to change fstab and cmdline.txt from device names to PARTUUID: line 974.  The other change seems to be when someone has a boot (or now firmware) partition on the SD card while the root partition is on a USB drive.: line 1741.

The issue arises when cmdline.txt is edited via 'sed -i'. That command ends up replacing the symlink /boot/cmdline.txt with the edited content, but the copy in /boot/firmware/cmdline.txt, which is the one that needs changing, is unchanged.
To fix, realpath(1) can be used to resolve the file to the actual target. Following patch applies this for all instances where 'sed -i' is used.
--- rpi-clone/rpi-clone 2023-10-22 20:01:42.068868892 -0700
+++ rpi-clone-bookworm/rpi-clone        2023-10-22 13:37:47.691962245 -0700
@@ -971,7 +971,7 @@
                cp $fstab_tmp $fstab
                printf "Your original fstab is backed up to $fstab_save\n"

-               cmdline_txt=/boot/cmdline.txt
+               cmdline_txt=`realpath /boot/cmdline.txt`
                cmdline_save=$cmdline_txt.${PGM}-save
                if [ -f $cmdline_txt ] && grep -q "$src_root_dev" $cmdline_txt
                then
@@ -1729,8 +1729,8 @@

 # Fix PARTUUID or device name references in cmdline.txt and fstab
 #
-fstab=${clone}/etc/fstab
-cmdline_txt=${clone}/boot/cmdline.txt
+fstab=`realpath ${clone}/etc/fstab`
+cmdline_txt=`realpath ${clone}/boot/cmdline.txt`

 if [ -f $cmdline_txt ]
 then
@@ -1738,7 +1738,7 @@
        then
                qecho "Leaving SD to USB boot alone."
                cp $cmdline_txt ${clone}/boot/cmdline.boot
-               cmdline_txt=${clone}/boot/cmdline.boot
+               cmdline_txt=`dirname $cmdline_txt`/cmdline.boot
        fi
        if grep -q $src_disk_ID $cmdline_txt
        then

I realize that realpath is not required for fstab ... currently. However, a similar problem would arise if /etc/fstab was symlinked. So I though it prudent to fix that as well.

However, a similar problem would arise if /etc/fstab was symlinked.

I see your point. But I don't expect /etc/fstab to become symlinked. It's a file as long as Linux exists. That's why I decided to remove realpath for /etc/fstab in my PR.

I fixed this by adding --follow-symlinks to the sed command

:cry: Unfortunately rpi-clone misses an important feature to help if there are any issues - a debug log :cry:
Would be great to have a debug log of your restore in order to help you. Maybe somebody adds the missing debug feature in rpi-clone :wink:

😢 Unfortunately rpi-clone misses an important feature to help if there are any issues - a debug log 😢
Would be great to have a debug log of your restore in order to help you. Maybe somebody adds the missing debug feature in rpi-clone 😉

No worries. I have an Pi 4B on hand so I'm using the SD card copier utility instead. Still not as handy for backups, but it'll work well enough to migrate from one microSD card to another. Here's hoping that doesn't fail.

I use raspiBackup to backup my Raspberries  Just give it a try :wink:

I'll look into that, thanks!

Hi, I tried everything in this post but my cloned card doesn't boot. I run rpi-clone on a pi4 (waiting for pi5 arrival) with Pi OS "bookworm" 64 bit. Nothing to do, I have to give-up. The clone with GUI official script works, as usual.

Syncing file systems (can take a long time)
Syncing mounted partitions:
  Mounting /dev/sda2 on /mnt/clone
  => rsync // /mnt/clone with-root-excludes ...
  Mounting /dev/sda1 on /mnt/clone/boot/firmware
  => rsync /boot/firmware/ /mnt/clone/boot/firmware  ...

Editing /mnt/clone/boot/cmdline.txt PARTUUID to use 59dc1bbf
Editing /mnt/clone/etc/fstab PARTUUID to use 59dc1bbf

as others pointed out it did do the correct edit but just didn't store it at the right target folder.  It stored it in  /mnt/clone/boot/cmdline.txt  instead of  /mnt/clone/boot/firmware/cmdline.txt . I was able to manually put in the USB drive and set the right partition value in the firmware folder and it works.

@framps thanks, your version works without issue!

In verband met de feestdagen ben ik tot en met vrijdag 29 december 2023 afwezig en zal mijn email beperkt kunnen lezen en beantwoorden.
Ik wens u het allerbeste voor 2024 en fijne feestdagen!
Met vriendelijke groet,
FSE Turnstiles b.v.
Jos de Vries
PS: Neem voor urgente storingen contact met ons op door een email te sturen naar @.***

@framps thanks, your version works without issue!

On which Raspberry Pi OS version?

@framps thanks, your version works without issue!

On which Raspberry Pi OS version?

Debian 1:6.1.63-1+rpt1 (2023-11-24) aarch64 with Kernel: pi5 6.1.0-rpi7-rpi-2712

which is the final path to apply to rpi-clone on a Rasp 3B+ (32bit) bookworm? I am getting confused...

  T± s4≡G:.ä±
-rwxr-xr-x 1 pab  pab<PHONE_NUMBER> Mar 16  2043 ''$'\002''.'$'\a''ü'$'\006'
d????????? ? ?    ?             ?            ? '▓"'$'\003''∩'$'\002''>'$'\002''.üá'$'\003'
-rwxr-xr-x 1 pab  pab         164 Mar 17  2019  wpa_supplicant.conf_Salty
d????????? ? ?    ?             ?            ? ''$'\001''αy░▄Φ'$'\035''≡.|± '
-rwxr-xr-x 1 pab  pab         172 Jan  1  2067 'σ▒≥'$'\016''╚.'$'\004'
-rwxr-xr-x 1 pab  pab<PHONE_NUMBER> Feb  1  2043 ''$'\003''Φ8.'$'\a''ü'$'\006'
-rwxr-xr-x 1 pab  pab   520388128 Jan  4  2096 ''$'\004''Φ'$'\b''É ~'$'\004\037''.'$'\004''φ'$'\b'
-rwxr-xr-x 1 pab  pab<PHONE_NUMBER> Mar 16  2043 ''$'\004\037\003''Φ   '$'\177''.'$'\b''ü'$'\006'
pab@pab:~ $

Looks like your SD card is broken.

Looks like your SD card is broken.

It seems very unlikely, as it worked every day before the upgrade... I will check and maybe reformat it.

during rsync I am getting these errors:
rsync: [receiver] mkstemp "/mnt/clone/boot/firmware/.bcm2712-rpi-5-b.dtb.SlzdW6" failed: Read-only file system (30)
rsync: [receiver] mkstemp "/mnt/clone/boot/firmware/.cmdline.txt.g9a4Sq" failed: Read-only file system (30)
rsync: [receiver] mkstemp "/mnt/clone/boot/firmware/.config.txt.QA8F5F" failed: Read-only file system (30)
rsync: [receiver] mkstemp "/mnt/clone/boot/firmware/.config.txt.bak.fK9cPg" failed: Read-only file system (30)
rsync: [receiver] mkstemp "/mnt/clone/boot/firmware/.fixup_cd.dat.DoHYL7" failed: Read-only file system (30)
rsync: [receiver] mkstemp "/mnt/clone/boot/firmware/.fixup_db.dat.IfOnvM" failed: Read-only file system (30)
rsync: [receiver] mkstemp "/mnt/clone/boot/firmware/.initramfs7.btIE0Q" failed: Read-only file system (30)
rsync: [receiver] mkstemp "/mnt/clone/boot/firmware/.issue.txt.Q6vgxm" failed: Read-only file system (30)
rsync: [receiver] mkstemp "/mnt/clone/boot/firmware/.kernel7.img.hWtAb9" failed: Read-only file system (30)
rsync: [receiver] mkstemp "/mnt/clone/boot/firmware/.start.elf.i72DIe" failed: Read-only file system (30)
rsync: [receiver] mkstemp "/mnt/clone/boot/firmware/.start_cd.elf.nY5NIh" failed: Read-only file system (30)
rsync: [receiver] mkstemp "/mnt/clone/boot/firmware/.start_db.elf.WnMyIt" failed: Read-only file system (30)
rsync: [receiver] mkstemp "/mnt/clone/boot/firmware/.start_x.elf.WR0DqO" failed: Read-only file system (30)

Your SD card is broken. I suggest to use a brand new SD card.

Your SD card is broken. I suggest to use a brand new SD card.

I suspect a problem in the USB->SD adapter. The SD is quite new.

2. my fork

Replaced SD. It is working, thanks.
So, should I just use your fork or is there any other patch/change to apply?

Just use my fork ab your'e fine :wink:

@framps
I am using your fork on a Raspi 3B+ 32.bit:
PRETTY_NAME="Raspbian GNU/Linux 12 (bookworm)"
NAME="Raspbian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

I had a look in the cloned fstab, and I have found a PARTUUID different than the one in the clone cmdline.txt.
This is from cloned fstab:
PARTUUID=93a7c455-01  /boot/firmware           vfat    defaults,flush    0       2
PARTUUID=93a7c455-02  /               ext4    defaults,noatime  0       1

and this from cloned cmdline.txt:
root=PARTUUID=b4a2e918-02
The PARTUUID in cmdline.txt of the running system is: b4a2e918-02
It doesn't seem to be correct...

@framps
This is the /boot folder in the running system. There is no link to /boot/firmware/cmdline.txt, but the systems boots fine:
ll /boot
total 17569
drwxr-xr-x  3 root root     4096 Dec 23 10:55 .
drwxr-xr-x 19 root root     4096 Dec 27 14:27 ..
-rw-r--r--  1 root root   205495 Nov 24 17:11 config-6.1.0-rpi7-rpi-v7
drwxr-xr-x  4 root root     4608 Jan  1  1970 firmware
-rw-r--r--  1 root root 10738708 Dec 23 10:54 initrd.img-6.1.0-rpi7-rpi-v7
lrwxrwxrwx  1 root root       17 Dec 23 10:49 overlays -> firmware/overlays
-rw-r--r--  1 root root       83 Nov 24 17:11 System.map-6.1.0-rpi7-rpi-v7
-rw-r--r--  1 root root  7023344 Nov 24 17:11 vmlinuz-6.1.0-rpi7-rpi-v7

This is not a clean installation, but an upgrade from bullseye

I never upgrade my RPis and it's also not recommended by the foundation. Looks like there is some issue with the upgrade.

I cannot do a clean install as it will be a major PITA. On my Raspi there is a NodeRed installation and many additional services all configured, so it is not a matter of just make a $HOME backup. Tenths of files in /etc are involved.

During rpi-clone execution, I see these messages:
Editing /boot/firmware/cmdline.txt PARTUUID to use 93a7c455
Editing /mnt/clone/etc/fstab PARTUUID to use 93a7c455

but then, only fstab has been edited, not cmdline.txt

By adding the symlinks in /boot:
sudo ln -s /boot/firmware/cmdline.txt /boot/cmdline.txt
sudo ln -s /boot/firmware/config.txt /boot/config.txt

rpi-clone has edite the "real" cmdline.txt, not the one in the clone!!

Ok, I will give up with rpi-clone.. Thanks.

BTW: I wrote a script which syncs the fstab and cmdline.txt with the actual UUIDs of the SD card.
Maybe this is mitigates your issue with rpi-clone. Just call this script when rpi-clone finished and it will fix the incorrect UUIDs.

Does the script takes into consideration the new /boot/firmware mount point?

Yes. See this commit I created this morning.

I have run rpi-clone with '-x' option to have max verbosity, and this is what I found:
+ fstab=/mnt/clone/etc/fstab
++ realpath /mnt/clone/boot/cmdline.txt                     <---------------------
+ cmdline_txt=/boot/firmware/cmdline.txt                  <---------------------
+ '[' -f /boot/firmware/cmdline.txt ']'
+ (( leave_sd_usb_boot && SD_slot_dst ))
+ grep -q b4a2e918 /boot/firmware/cmdline.txt
+ qecho 'Editing /boot/firmware/cmdline.txt PARTUUID to use 93a7c455'
+ (( !quiet ))
+ echo 'Editing /boot/firmware/cmdline.txt PARTUUID to use 93a7c455'
Editing /boot/firmware/cmdline.txt PARTUUID to use 93a7c455              <---------------------
+ sed -i s/b4a2e918/93a7c455/ /boot/firmware/cmdline.txt
+ (( leave_sd_usb_boot && SD_slot_boot ))
+ grep -q b4a2e918 /mnt/clone/etc/fstab
+ qecho 'Editing /mnt/clone/etc/fstab PARTUUID to use 93a7c455'
+ (( !quiet ))
+ echo 'Editing /mnt/clone/etc/fstab PARTUUID to use 93a7c455'
Editing /mnt/clone/etc/fstab PARTUUID to use 93a7c455
+ sed -i s/b4a2e918/93a7c455/g /mnt/clone/etc/fstab
+ rm -f /mnt/clone/etc/udev/rules.d/70-persistent-net.rules
++ e2label /dev/sda2

As you can see, cmdline_txt is pointing to the active cmdline.txt ! How can that be if the code line is this:
cmdline_txt=$(realpath ${clone}/boot/cmdline.txt)
Look at the lines I have highlighted

Line 1730 on rpi-clone should be:
cmdline_txt=${clone}/$( realpath /boot/cmdline.txt)

With that mod, it is working fine. Correct UUIDs are in the cloned system and no modification occurred in the running system. Can you apply the fix in your code (if you agree)?

That's interesting:
Your upgraded system does not have the link and therefore realpath doesn't point to the linked file on the clone.
On a non upgraded system the link already exists and therefore realpath resolves to the linked file.
I think the other suggested fix to add --follow-symlinks in the sed commands is a better way to go now because it's easier to understand what's going on there.
I'll change the code to use --follow-symlinks and execute a test with a plain bookworm. If everything is OK I'll commit the change in my fork.

I have manually added the symlink. Without it, no changes were done in the cloned cmdline.txt.
Does my proposed line work in any case also on a fresh install?

I believe this syntax is not correct:
cmdline_txt=$(realpath ${clone}/boot/cmdline.txt)
as you are getting the realpath of /mnt/clone/boot/cmdline.txt, which will find the symlink to /boot/firmware/cmdline.txt
my proposed change is getting the realpath from /boot/cmdline.txt (which is /boot/firmware/cmdline.txt) and prepend to it /mnt/clone.

Does my proposed line work in any case also on a fresh install?

As far as I understand - yes. But I will switch to --follow-symlinks now. See my previous reply.

When cloning a few SD cards in a row I've noticed that /boot/firmware becomes unmounted and then rpi-clone doesn't update the UUID at all, so I had to make sure that it was mounted each time.. I assume rpi-clone is unmounting it ?

@bputtick , AFAIK rpi-clone is not unmounting any current partition!

@bputtick , AFAIK rpi-clone is not unmounting any current partition! Just the ones it clones to.

Yeh I've not really investigated it, but it happened more than once and I was just cloning, replacing SD card and repeating and nothing else.
Just thought I'd mention it, in case someone else experienced it.

I just committed the --follow-symlinks change and successfully tested rpi-clone on a fresh install.

I just committed the --follow-symlinks change and successfully tested rpi-clone on a fresh install.

I felt this was the safest fix, at least until they remove the symlink they put in for backwards compatibly

I just committed the --follow-symlinks change and successfully tested rpi-clone on a fresh install.

But line 1730 is still there, with wrong syntax...

...and it is still modifying original file!!!
Editing /boot/firmware/cmdline.txt PARTUUID to use 93a7c455
Editing /mnt/clone/etc/fstab PARTUUID to use 93a7c455

I have re-instated my correct line 1630, and it is working back again:
Editing /mnt/clone/boot/firmware/cmdline.txt PARTUUID to use 93a7c455
Editing /mnt/clone/etc/fstab PARTUUID to use 93a7c455

But line 1730 is still there, with wrong syntax...

:+1: Good catch. I now added --follow-symlinks in every sed call.

but if you "follow symlink" on the cloned partition, it will point to /boot/firmware and NOT to /mnt/clone/boot/firmware

In this line cmdline_txt contains /mnt/clone/boot/cmdline.txt
sed -i --follow-symlinks "s/${src_disk_ID}/${dst_disk_ID}/" "$cmdline_txt"
and when you follow that symlink, it will then point to /boot/firmware/cmdline.txt

Ok. So you're saying the --follow-symlink approach does not work for you and thus realpath is the only way to fix your issue?
I unfortunately have no upgraded bookworm in order to test your scenario :cry:

Why you didn't just keep your previous version and change that 1730 line?

Looks like I should revert the --follow-symlinks approach ...

Looks like I should revert the --follow-symlinks approach ...

Why, if your previous version was working?

Can you send me the output of ll /boot on a fresh installed bookworm?

I agree. Maybe someone is willing to investigate on this. I'll apply the fix in my fork as soon as there is one available.

I investigated the whole morning and found the solution, but I don't understand why you didn't agree...

Ok, leave it as it is. I am managing by myself.
Thank you.

Editing /mnt/clone/boot/cmdline.txt PARTUUID to use 93a7c455
Editing /mnt/clone/etc/fstab PARTUUID to use 93a7c455

As you can see, it is NOT following to /mnt/clone/boot/firmware

Can you send me the output of ls -la /boot on a fresh installed bookworm?

pi@raspberrypi-bookworm64-lite-cm4:~ $ ls -la /boot
total 102028
drwxr-xr-x  4 root root     4096 Dec 28 14:36 .
drwxr-xr-x 19 root root     4096 Dec 29 15:44 ..
lrwxrwxrwx  1 root root       20 Oct 10 05:39 cmdline.txt -> firmware/cmdline.txt
-rw-r--r--  1 root root   230394 Oct 27 16:31 config-6.1.0-rpi6-rpi-2712
-rw-r--r--  1 root root   230383 Oct 27 16:31 config-6.1.0-rpi6-rpi-v8
-rw-r--r--  1 root root   230416 Nov 24 17:11 config-6.1.0-rpi7-rpi-2712
-rw-r--r--  1 root root   230405 Nov 24 17:11 config-6.1.0-rpi7-rpi-v8
lrwxrwxrwx  1 root root       19 Oct 10 05:39 config.txt -> firmware/config.txt
-rw-r--r--  1 pi   pi         94 Dec 14 18:32 deletedKernels.txt
drwxr-xr-x  3 root root     4096 Jan  1  1970 firmware
drwxr-xr-x  3 root root     4096 Jan  1  1970 firmware.bak
-rw-r--r--  1 root root 17125444 Dec 28 14:36 initrd.img-6.1.0-rpi6-rpi-2712
-rw-r--r--  1 root root 17124794 Dec 28 14:36 initrd.img-6.1.0-rpi6-rpi-v8
-rw-r--r--  1 root root 17126006 Dec 28 14:35 initrd.img-6.1.0-rpi7-rpi-2712
-rw-r--r--  1 root root 17124187 Dec 28 14:35 initrd.img-6.1.0-rpi7-rpi-v8
lrwxrwxrwx  1 root root       17 Oct 10 05:38 overlays -> firmware/overlays
-rw-r--r--  1 pi   pi         94 Dec 16 23:40 raspiHandleKernels.krnl
-rw-r--r--  1 root root       83 Oct 27 16:31 System.map-6.1.0-rpi6-rpi-2712
-rw-r--r--  1 root root       83 Oct 27 16:31 System.map-6.1.0-rpi6-rpi-v8
-rw-r--r--  1 root root       83 Nov 24 17:11 System.map-6.1.0-rpi7-rpi-2712
-rw-r--r--  1 root root       83 Nov 24 17:11 System.map-6.1.0-rpi7-rpi-v8
-rw-r--r--  1 root root  8731342 Oct 27 16:31 vmlinuz-6.1.0-rpi6-rpi-2712
-rw-r--r--  1 root root  8729756 Oct 27 16:31 vmlinuz-6.1.0-rpi6-rpi-v8
-rw-r--r--  1 root root  8762266 Nov 24 17:11 vmlinuz-6.1.0-rpi7-rpi-2712
-rw-r--r--  1 root root  8760390 Nov 24 17:11 vmlinuz-6.1.0-rpi7-rpi-v8

As you can see, it is NOT following to /mnt/clone/boot/firmware

In the echo command there is no realpath nor --follow-symlink used. In the corresponding sed command --follow-symlink ist used :wink:
If realpath is used the echo command will echo the correct filename and use the correct filename in sed.
So both ways work correctly but for the --symlink case the echo command echos not the final target filename.

Aaaaaah! That /boot directrory!
I fixed my symlinks in /boot. Now it works, thanks!

@framps Have you reverted it back to realpath??? SHould I have to test everything again?

Replaced SD. It is working, thanks.

SD cards fail in weird ways, often not all at once, either.
I think the reason @fmarzocca & @framps are seeing different things is there are a lot of underlying changes in the specific case of the Raspberry Pi OS v11.x to v12.x update that apt doesn't - and some that it can't - account for. I "successfully" updated a 3B+ from 11 to 12 myself, only to run into apt update problems later because packages, dependencies, etc. weren't located in the filesystem where Raspberry Pi OS v12 apt expected them to be. I wound up having to clean install to resolve the issues.
@fmarzocca I don't know the exact reason a clean install won't work for you, but in my case I'd avoided doing so due to an application, UniFi Controller, with dependencies that can't be fulfilled (not by default, at least) by later versions of Raspberry Pi OS. Fortunately, I found instructions online on how to do so with a clean install. You may wind up having to do something similar or - as is possible with UniFi Controller - migrate to a 1st party appliance that has already solved that problem.

@fmarzocca I don't know the exact reason a clean install won't work for you,

I didn't say that it won't work, but that it is a major PITA. My server has a complex NodeRed installation (which can be easily backed up, but it also have several (maybe 7 or 8) additional services that have been carefully tuned up during years (i.e.: apcupsd, gammu, email, mosquitto and many others). It is not just a matter of backup0 the home folder. To do a complete new installation I should save all the settings and  stop the server for several days (I don't have full time free) and this will completely stop the domotics.  And I am sure I will forgot something. That's why I took the way of an upgrade which - I may be honest - is fully working for ten days without problems. I have fiuxed this last issue with rpi-clone.

Have you reverted it back to realpath

Yes. That way the echo displays the symlinked filename.
So now the code uses realpath again with your suggested modification in line 1730 :wink:

I "successfully" updated a 3B+ from 11 to 12 myself,

@jdrch 32-bit or 64-bit?

I "successfully" updated a 3B+ from 11 to 12 myself,

@jdrch 32-bit or 64-bit?

32-bit.
All subsequent deployments have been 64-bit.

Following the link from this comment
"I think that's not dependent on the new HW but the SW required by RPi5. As far as I know RPi5 requires bookworm and rpi-clone works with bookworm if you apply the patch listed in https://github.com/billw2/rpi-clone/issues/168."
rpi-clone fails for me when trying on my RPI 5
What is the patch? It seems like this is still in flux.

What is the patch? It seems like this is still in flux.

@phillipredshirt use this fork, it has been patched and working.

It's still not working.  I get:
`Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done
Begin: Waiting for root file system ... Begin: Running /scripts/local-block ...done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
same line about 15 times then
done.
Gave up waiting for root file system device. Common problems .
...
ALERT! PARTUUID=20ffxxxx-02 does not exist. Dropping to a shell!
`
That's the PARTUUID of the SD card.

@fmarzocca I used the fork and it didn't change anything and then I used the -l switch on the command like this
sudo rpi-clone -l sda
and now I can't boot from the SD card either. It get the same error but points to the PARTUUID of the SSD.
So, I kept both the SD card and the SSD and it boots. I think my pi is a little mixed up now.

Thank you @framps !! It took me some digging to understand what was going wrong. Once I knew I also found this issue here and the fork which worked very well when cloning to usb. I finally could boot it, because the cmdline.txt was updated properly on the usb. Please @billw2 look into the fork or at least make a note in the readme that current RaspianOS based on bookworm have this problem.

Thank you @framps !! It took me some digging to understand what was going wrong. Once I knew I also found this issue here and the fork which worked very well when cloning to usb. I finally could boot it, because the cmdline.txt was updated properly on the usb. Please @billw2 look into the fork or at least make a note in the readme that current bookworm systems have this problem.

Reasonably sure this repo has been abandoned.

Not working with Bookworm at all for me, even with the patch. when I do
sudo rpi-clone sda -v -f
when it boots it ends up saying
Gave up waiting for root file system device...

and sits in a Busybox shell with an initramfs prompt.
Using
sudo rpi-clone sda -v
after booting it ends up like
Trying partition: 0
lba: 8192 oem:  'mkfs.fat' volume: ' NO NAME   '
...
Firmware not found

Tried with 2 different SD cards. What am I doing wrong? This worked fine on another Pi running buster.

@teq0 Which model Pi? AFAIK the Bookworm base images for the 3B, 4*, and possibly 5 are sufficiently different to break the script?

Pi 4. And it's the lite image.
Does Raspi Imager change the image based on the machine? I've burned a bunch of Bookworm Lite cards and told it it was for Pi 4 but they run fine in 3 and 3+s.
I'll try cloning Bookworm on a 3 and see what happens.

Ah, on 2nd thought I could be wrong. I think you wind up with 2 different images if you update from 11.x vs. clean install 12.x, though.

I may be wrong but I guess my fork of rpi-clone fails if Bookworm was not a fresh install but an upgrade from Bullseye. In #174 I helped the guy to check the UUIDs and they didn't match even rpi-clone updates /boot/cmdline.txt. But if /boot/cmdline.txt is a file and no link to /boot/firmware/cmdline.txt rpi-clone updates /boot/cmdline.txt but not /boot/firmware/cmdline.txt. But this file is not used during boot and therefore boot fails because of UUID mismatch.
Maybe somebody who runs an upgraded Bookworm can verify whether my guess is correct?

fails if Bookworm is not a fresh install but an upgrade from Bullseye

This has been my experience too.

Maybe somebody who runs an upgraded Bookworm can verify whether my guess is correct?

The irony of that request is because the 11.x to 12.x upgrade path isn't officially supported, upgraded 12.x installations eventually run into breaking issues, such apt being unable to satisfy dependencies.
In other words, as horrible as it sounds, I don't think supporting upgraded 12.x installations is worthwhile, as it's pretty much impossible to guarantee the config/folder structure/locations the script is operating on.

11.x to 12.x upgrade path isn't officially supported

That's why I install a fresh OS every time I want to upgrade my systems and I don't have a system handy which was upgraded to verify my guess.
Actually it's not a big deal to check whether /boot/cmdline.txt is a file and create a link to /boot/firmware/cmdline.txt. But I frankly don't like to fix this under the cover in rpi-clone. Maybe it's worth to check and write an error message and nobody blames rpi-clone .

That's why I install a fresh OS every time I want to upgrade my systems and I don't have a system handy which was upgraded to verify my guess.

Requiring clean installs for major version upgrades is pretty retrograde and anachronistic of the Raspberry Pi Foundation (I'm not blaming @framps, @billw2, or anyone else) in 2023 when the vast majority of major OSes, including the one Raspberry Pi OS is based on, support in-place upgrades.
I appreciate the Foundation warning users of the risks of in-place upgrades, but the fact there aren't supported reflects poorly on them and them only.

Maybe it's worth to check and write an error message and nobody blames rpi-clone .

Correct. This is a Raspberry Pi OS, and therefore a Raspberry Pi Foundation problem. Except they don't see it as a problem.
I'd say rpi-clone should probably have an option to toggle between cloning for the following environments:

v12.x, clean installed
all versions up to 11.x

As I said in a previous comment, I don't think it's fair to developers to ask them to support 11.x to higher major version in-place upgrades if the underlying OS itself doesn't support them.

I have upgraded 4 Raspi 3B+ from bullseye to bookworm, and I am happily using @framps' fork, provided to execute (after kernel update) the following:
cd /boot
sudo ln -s firmware/cmdline.txt ./cmdline.txt
sudo ln -s firmware/config.txt ./config.txt

This proves an upgraded OS misses the links which are new to Bookworm and everybody who upgraded has to execute these commands first and then rpi-clone will create a bootable clone.

This proves an upgraded OS misses the links which are new to Bookworm and everybody who upgraded has to execute these commands first and then rpi-clone will create a bootable clone.

Correct.

I am suprised this this works.  I would have thought that a bullseye to bookworm upgrade would retain the old /boot mountpount, creating the /boot/firmware directory within the /boot filesystem.  I was under the impression that vfat filesystems do not support unix symbolic links.

Keep in mind the boot partition is mounted on Bookworm now in /boot/firmware and /boot is located on the root filesystem which is formatted with ext4 and supports symlinks.
pi@raspberrypi-bookworm64-lite:~ $ ls -la /boot
total 39292
drwxr-xr-x  3 root root     4096 Jan  8 21:33 .
drwxr-xr-x 18 root root     4096 Dec 11 05:55 ..
-rw-r--r--  1 root root       83 Nov 24 17:11 System.map-6.1.0-rpi7-rpi-2712
-rw-r--r--  1 root root       83 Nov 24 17:11 System.map-6.1.0-rpi7-rpi-v8
lrwxrwxrwx  1 root root       20 Dec 11 05:37 cmdline.txt -> firmware/cmdline.txt
-rw-r--r--  1 root root   230416 Nov 24 17:11 config-6.1.0-rpi7-rpi-2712
-rw-r--r--  1 root root   230405 Nov 24 17:11 config-6.1.0-rpi7-rpi-v8
lrwxrwxrwx  1 root root       19 Dec 11 05:37 config.txt -> firmware/config.txt
drwxr-xr-x  3 root root     4096 Jan  1  1970 firmware
-rw-r--r--  1 root root 11108528 Dec 11 05:54 initrd.img-6.1.0-rpi7-rpi-2712
-rw-r--r--  1 root root 11108073 Dec 11 05:54 initrd.img-6.1.0-rpi7-rpi-v8
lrwxrwxrwx  1 root root       18 Dec 11 05:54 issue.txt -> firmware/issue.txt
lrwxrwxrwx  1 root root       17 Dec 11 05:36 overlays -> firmware/overlays
-rw-r--r--  1 root root  8762266 Nov 24 17:11 vmlinuz-6.1.0-rpi7-rpi-2712
-rw-r--r--  1 root root  8760390 Nov 24 17:11 vmlinuz-6.1.0-rpi7-rpi-v8

pi@raspberrypi-bookworm64-lite:~ $ mount | grep mmcblk0p
/dev/mmcblk0p2 on / type ext4 (rw,noatime)
/dev/mmcblk0p1 on /boot/firmware type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)

I see unofficial rpios upgrade bullseye to bookworm instructions that do not mention upgrading to the new kernel or say that is is optional.  Upgrading the kernel will remount /boot to /boot/firmware like a new installation of bookworm, but likely will not add the symbolic links.  I just upgraded a 32bit lite rpios using the "edit apt sources" method.  While the upgrade to bookworm works, I was left with this:
pi@raspberrypi:/boot $ mount |grep mmcblk0
/dev/mmcblk0p2 on / type ext4 (rw,noatime)
/dev/mmcblk0p1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
pi@raspberrypi:/boot $ df -h / /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        59G  2.1G   54G   4% /
/dev/mmcblk0p1  255M   51M  205M  20% /boot
pi@raspberrypi:/etc $ cat os-release |head -1
PRETTY_NAME="Raspbian GNU/Linux 12 (bookworm)"
pi@raspberrypi:/etc $ ls -la /boot/config.txt /boot/cmdline.txt
-rwxr-xr-x 1 root root  102 Dec  4 21:57 /boot/cmdline.txt
-rwxr-xr-x 1 root root 2075 Dec  4 21:39 /boot/config.txt
In this specific case, the symbolic links not only are not present, but probably are not requred for rpi-clone to work.

Bookworm uses /boot/firmware/cmdline.txt during boot and if you modify /boot/cmdline.txt which is loand no link to /boot/firmware/cmdline.txt this update will be ignored.
That's the root cause for this issue because rpi-clone fails to update the UUID in /boot/firmware/cmdline.txt but instead updates /boot/cmdline.txt.

that do not mention upgrading to the new kernel or say that is is optional

The fact that's the case tells you how broken the in-place upgrade path is.

Bookworm uses /boot/firmware/cmdline.txt during boot and if you modify /boot/cmdline.txt which is no link to /boot/firmware/cmdline.txt this update will be ignored.
That's the root cause for this issue because rpi-clone fails to update the UUID in /boot/firmware/cmdline.txt but instead updates /boot/cmdline.txt.

when upgrading to bookworm it creates the symlink and when rpi-clone is run it copies the symlink, but when it uses sed to replace the UUID which replaces the symlink with a file. What i've not tested is if you install bookworm from scratch does it bother with the symlink?

@framps thanks for maintaining your fork / patchset! I've also pulled it into my fork, so hopefully anyone looking for a working tool can grab one of the forks (I had been maintaining my fork for the NVMe support, so since it's referred to a lot I decided to pull in your changes instead of just archiving it... would love to find a way for there to be one 'official' place for the project to reside!
Alternatively, does your rpiBackup tool potentially have room for cloning functionality built in? (Last time I looked, I think you had to do some sort of backup + restore to another drive, which was more complex than a straight clone).f

so since it's referred to a lot I decided to pull in your changes instead of just archiving it...

NP. Glad you pulled my changes. Whoever uses rpi-clone should be able to use rpi-clone with Bookworm and future OS releases. Whether it's grabbed from your fork or mine :smiley:

would love to find a way for there to be one 'official' place for the project to reside!

Agree. Given the fact I maintain raspiBackup I don't volunteer to maintain rpi-clone - even I was asked to do this already.

Alternatively, does your rpiBackup tool potentially have room for cloning functionality built in? (

raspiBackup was designed to create backups - not to clone any systems. Indeed you can clone a system by creating a backup and restore this backup. But that's a workaround and not convenient.
I frankly think most of the guys actually want to create a backup instead of a clone. Maybe I'm wrong ... If there is a strong requirement to get a cloning capability in raspiBackup I'll think about this new feature.

@framps heh, I want both! :D
But for me, the cloning is often useful just to test different media on the same Pi without having to re-image (though sometimes imaging fresh is faster anyway).
For most people, I think full backups are all they'd ever need, short of a better mechanism to upgrade major versions (something I still never recommend on any Linux OS!).

the cloning is often useful just to test different media on the same Pi

I see your point but that's a special use case you have :smiley:
Most of the folks using raspiBackup just want to have a backup in case either the SD card fails or something went wrong when they do any upgrades or manually do any config updates and which causes the system fail to boot.

short of a better mechanism to upgrade major versions (something I still never recommend on any Linux OS!)

Neither do I

But for me, the cloning is often useful just to test different media on the same Pi without having to re-image (though sometimes imaging fresh is faster anyway).

@geerlingguy  I've found the easiest way to do this is to use the SD copier utility on a separate Pi. Most of the time if you're having storage media problems on a Pi it's probably unstable anyway, which complicates using it as a clone source. Now, the latter should be possible, but it's not necessarily ideal.
@framps I appreciate raspibackup in the context of the relative scarcity of dedicated Pi backup and restore tools. That said, if you ask me, an ideal backup tool would be a mix of recording partitions and corresponding UUIDs with something like restic for the actual files. That way you get incremental, rapidly restored backups. My main issue with raspibackup as a solution is it has all the inconveniences of a clone operation without restoration convenience (read: the ability to just slap the target disk into the Pi and boot right away) thereof.
This is why I prefer rpi-clone. It's not perfect or incremental, but at least the restore process is as easy as swapping out physical disks.

I frankly think most of the guys actually want to create a backup instead of a clone. Maybe I'm wrong ... If there is a strong requirement to get a cloning capability in raspiBackup I'll think about this new feature.

I use the Raspi with NodeRed to control a complex Home Automation. In addition to NR there are many other modules installed and configured (i.e.: mosquitto, gammu-sms, etc...). My main target is to get a clone in order to swap the SDs in case of failure and reduce the system downtime to few minutes. This is a security issue for me. A backup won't work that way.

My main target is to get a clone in order to swap the SDs in case of failure and reduce the system downtime to few minutes.

I'm now convinced a clone feature makes sense. I just created an issue for this in my repo.

My main target is to get a clone in order to swap the SDs in case of failure and reduce the system downtime to few minutes.

Isn't that goal achieved with rpi-clone? That is what I'm using it for as well, but it only updates what has changed between runs.

Isn't that goal achieved with rpi-clone? That is what I'm using it for as well, but it only updates what has changed between runs.

Yes, it is. I was answering to a previous question from @framps, if you look at the quoted text.

Yes, it is. I was answering to a previous question from @framps, if you look at the quoted text.

It sounded like the cloning feature was being asked for in rpiBackup, so my comment was more geared towards "if it's already done in rpi-clone, why re-create the cloning function in rpiBackup".

Another vote of thanks for making the fork available!    I had been making periodic "clones" of my new bookworm system, but didn't realize I was not protected because the clones would not actually boot.   The clones created with the @framps fork will now boot.  Thanks!

but didn't realize I was not protected because the clones would not actually boot.

I think a lot of folks will detect this fact when they actually need the clone to boot. Fortunately the clone only has inconsistent UUIDs which break the boot and can be fixed manually.
I helped folks with this boot issue multiple times in the past when they cloned their system manually and didn't use rpi-clone. Finally I wrote syncUUIDs :smiley:

An update to package  raspberrypi-sys-mods released today (2024-01-29) breaks this again.
In the install transcript,
Setting up raspberrypi-sys-mods (20240123) ...
Replacing transitional /boot/cmdline.txt symlink...
Replacing transitional /boot/config.txt symlink...

Sure enough, /boot/cmdline.txt is no longer a symlink to firmware/cmdline.txt; instead a regular file with contents
DO NOT EDIT THIS FILE

The file you are looking for has moved to /boot/firmware/cmdline.txt

So back to the drawing board for a new fix...

Thank you very much for the update :+1:

So back to the drawing board for a new fix...

Not a big deal but nevertheless some code change is required again. I will update my fork accordingly.

@framps - Can you either file that as a separate PR here or just link to your commit here and I'll make sure that's also pulled in on my fork too? Thanks!
There's so much documentation (blogs, forum posts, etc.) still pointing to /boot files :O

I created a fix in a new branch in my fork and executed some initial tests. Looks good so far. Would be great if this fix is reviewed by other folks before I merge the fix into my main branch.
@geerlingguy

There's so much documentation (blogs, forum posts, etc.) still pointing to /boot files :O

Unfortunately :cry: I frankly don't understand why they don't keep the link. That's a much more backward compatible change.

That's from a backward compatibility view

The Foundation has made it pretty clear from their Bookworm release announcement that backwards compatibility isn't a priority for them.
Competing x86-64 options with higher performance and first party Debian support approaching Raspberry Pi pricing should fix that problem.

backwards compatibility isn't a priority for them.
Ok. Well - I don't link this decision. But why do they added link first and then removed this link?

why do they added the link first and then removed this link now?

If we're lucky, they may have commented the PR/commit containing the change.

Nope. Just read the commit comment

Would be great if this fix is reviewed and tested by other folks before I merge the fix into my main branch.

Tested OK.

Just to keep you posted: I created an issue and asked for the rational of this update
@fmarzocca Thank you very much for your verification of my fix. In the meantime I found other ways to fix this issue - just check my latest commit in this branch.
But I think it's just fine tuning from a coding perspective. I'll merge the fix into my master branch in one week if there is no feedback any more.

@framps IMHO I prefer if [[ -d /boot/firmware ]];

just check my latest commit.

I took a look at this; couple of comments

I like the idea of extracting from /etc/fstab or mount. You might consider
awk '{ if ($2 ~ "^/boot") print substr($2, 2)}' /etc/fstab
or
mount | awk '{ if ($3 ~ "^/boot") print substr($3, 2)}'
which are slightly less "punctuation salad" IMO ;-)

On line 988 you have a case inconsistency (cmdLineDir vs. cmdlineDir)

@zwolfpack I like your awk proposal :+1: and updated the code accordingly.

On line 988 you have a case inconsistency (cmdLineDir vs. cmdlineDir)

Good catch :+1: Fixed right now.

If you are accepting suggestions, could you update the version line in rpi-clone?  And maybe even the attribution comment near the top to indicate your github url?

@zwolfpack Just to make the awk call more robust: What do you think about
awk '$2 ~ "^\/boot" && $3 == "vfat" { print substr($2, 2); exit}' /etc/fstab

@ProblemChild4901 Done.

@zwolfpack Just to make the awk call more robust: What do you think about
awk '$2 ~ "^\/boot" && $3 == "vfat" { print substr($2, 2); exit}' /etc/fstab

Good idea to add the exit.
If you want, you could add a check that the line isn't commented out as well
awk '$1 != "^#" && $2 ~ "^/boot" && $3 == "vfat" { print substr($2, 2); exit }' /etc/fstab

you could add

Added :wink:

@framps Can we consider stable latest commit? Are you going to merge it in master?

I just removed the function call and use a variable now. I consider the code stable but will wait until this WE to give some additional time for any other valuable feedback.

I just merged the bookworm fix branch into master :-)

Thank you!

Good catch :+1: Done

I wish I could redirect my fork to yours — is that possible (outside of just a README change)? Don't want to try maintaining a parallel fork especially since it would be great to increase visibility on your other project.

Good catch 👍 Done

@framps Another one: in readme the installation procedure points to:
$ git clone https://github.com/billw2/rpi-clone.git
You should change the url, otherwise users are always downloading the old one!

I wish I could redirect my fork to yours — is that possible (outside of just a README change)?

Sure. But please note: I will neither start working on the issues in this repo nor merge any PRs into my fork. I started to work on my fork just to get rpi-clone to run successfully again on Bookworm. But as far as I can see that's the intention of your fork also: Just to add minor missing features (nvme support) and to keep rpi-clone up and running in current and future RaspbianOS releases.

Don't want to try maintaining a parallel fork

If you want to maintain rpi-clone, work on the PRs and issues just merge my Bookworm fix into your fork and I'll change the all existing links to your fork :wink:
Besides that I spent some time and estimated the effort to add clone capability to raspiBackup. It doesn't make sense because raspiBackup is designed to create backups and no clones. You can actually clone a system if you create a backup and then restore the backup to another device. But this makes sense only if you already have a backup. So folks who need clone functionality should use rpi-clone and those who need backup functionality may decide to use raspiBackup.

You should change the url, otherwise users are always downloading the old one!

Good catch. I just updated the link and added an xref to Bills repo.

@geerlingguy Do you want new PRs and issues to be created in your repo or should still everything be created in Bills repo? I think it makes sense to use your repo from now on.

@framps - Yeah, that might be good to do, I've turned on notifications. And like I've mentioned, the maintenance would be mostly simple things—so if anyone needs attention on something that breaks or the like, please feel free to mention me in a PR!

It doesn't make sense because raspiBackup is designed to create backups and no clones.

I have just remembered I created a helper script a long time ago called raspiBackupRestore2Image which creates a dd backup image file from the just created backup and stores the backup in the backup directory. You even optionally can reduce the dd backup size with pishrink. It's  not a big deal to modify the script to restore the backup on another backup device instead of a dd file.
As of now every helper script of raspiBackup is unsupported - but I know raspiBackupRestore2Image is used a lot of time and works perfectly.
I can imagine a feature request for raspiBackup where the restore capability of a just created backup into a dd file in the backup partition or on a device. Latter will actually be a clone onto another device combined with the creation of a next backup version. That way the primary feature of rpi-clone - clone a system for backup purposes - will be added to raspiBackup. Any thoughts on this?

I spent the last week to create a prototype based on current raspiBackup release which now supports cloning. Details how to install the prototype and how to provide feedback are available here.

@framps
OK - I'm confused. I was under the impression that this repo (the original rpi-clone) had been (or would be) archived as it was not being maintained. Geerling advertised his fork as a "friendly fork", and I assumed that was accurate.
So, could I ask, "What is the plan?"... will this (original) repo continue on, or are we all supposed to be on board with the geerling repo?

Jeff fortunately volunteered to take over maintenance of this abandoned repository. Would be great if @billw2 updates this repo and points to Jeffs clone. Not sure whether he still follows his repo.
are we all supposed to be on board with the geerling repo?

If you want to get any updates/fixes on rpi-clone - yes :wink:
I was inspired by rpi-clone and added clone support in my prototype of raspiBackup. It's a different backup tool and it's primary purpose is to create backups. I think the clone feature is a very useful additional feature which is missing in raspiBackup and that's why I created the prototype :wink:

Yeah, basically @framps has a great backup tool, with some added cloning functionality. And it has more active development. I'm maintaining a copy of rpi-clone minimally, just to make sure basic cloning functionality continues working on newer Pi OS releases. It is meant only for cloning.

OK - thanks for the input!
BTW, have either of you taken a look at/evaluated the usb-boot tool? Like rpi-clone it's able to copy a "live" image from an RPi to an NVME card - or other media. Just curious if you were aware - or compared - the two.

Given there is no feedback on the clone feature in raspiBackup I added in a prototype this feature will not make it into next release. It was a nice experience to add this feature but it's unfortunately orthogonal to the design of raspiBackup and therefore the prototype is just a hack and will not make it :-(

Can someone clarify for the slower amongst us - I've been on hols for 3 weeks and so only now noticed the issue with the latest 64 bit OS on rPI 5 (it was working maybe 5 weeks ago). Can create ssd (on usb adaptors) clones but they will not boot. Should I be using the @framps version and it so which link?

@scargill use https://rpi-clone.jeffgeerling.com

@scargill use https://rpi-clone.jeffgeerling.com

Right I went to your page as you suggested - and as use pi  I got rid of any traces of rpi-clone and  used the link
curl https://raw.githubusercontent.com/geerlingguy/rpi-clone/master/install | sudo bash
Remembering I'm doing this on a new RPI5 with the latest (a few days ago) Bookworm 64 bit from the RPI site.
For years I've used the original rpi-clone on various versions of PI OS 32 bit up to and including Bullseye - indeed I'm using it now with Bullseye 32 bit on the RPI4.
So the new RPI5 - Bookworm 64 bit - standard setup - decent SD and SSD I've used reliably several times.
I ran your rpi-clone as instructed - stopped the Pi5, took out the SD, turned on the RPI5, the SSD blue and reMobaxterm on Win 11 PC) to access the RPi5 - no change from my normal access of RPi4 with Bullseye.
Any help to get this working REALLY appreciated as without rpi-clone I'm FAR too likely to make fatal mistakes moving my massive Node-Red/MQTT/Zigbee2MQTT etc setup over to the new RPI5....

Has anyone tried this: https://github.com/seamusdemora/RonR-RPi-image-utils?

Yes. It's a nice tool to create a clone also. But in contrast to rpi-clone the clone is stored in a dd image and you have to restore the dd image first before you can use it. With rpi-clone you have a cold standy because the clone is already stored on an SD card or USB disk and can be used immediately.
In addition don't use the code from this repo ! The code is outdated and it's  not the repo of Ron, the author of the code. He doesn't maintain his code on github. If you want to use his code grab the latest code directly from his thread in https://forums.raspberrypi.com/

@framps

Yes. It's a nice tool to create a clone also. But in contrast to rpi-clone the clone is stored in a dd image and you have to restore the dd image first before you can use it. With rpi-clone you have a cold standy because the clone is already stored on an SD card or USB disk and can be used immediately.

Huh!?!?  That is poppycock... every word of it is BS.

In addition don't use the code from this repo ! The code is outdated and it's not the repo of Ron, the author of the code. He doesn't maintain his code on github. If you want to use his code grab the latest code directly from his thread in https://forums.raspberrypi.com/

This must be Slag Seamus Day in your world!
Everything you've said in this post is provably and objectively UNTRUE. No idea what you've got a hard-on about, but please find someone else to invent stories about!

Prove every word is BS :wink:

Hey folks I didn't mean to start an argument. I came across the script I asked about in an XDA post and was curious since I'd never heard of it before. Currently only @framps' tool is actively developed by the original developer, so there's that.

Proof that every word is BS 😉

How do you figure that?

Either proof my statements in https://github.com/billw2/rpi-clone/issues/168#issuecomment-2082106387 are wrong or shut up.

@framps

OK... first: "you have to restore the dd image first before you can use it."

No - that's utter nonsense... you can do a loop mount on the .img file to copy or edit any file you want. There's even a utility in image-utils that will do this for you: image-mount. And of course you can keep an image-backup-produced on a USB, SD, NAS, local drive - whatever. Finally - it is not a "dd image"... image-backup uses rsync to copy files from the "live" system to the *.img file. Only a nincompoop would try to use dd on a live, mounted filesystem! You have shown more ignorance in this one statement than I would have imagined humanly possible... where do you get these nonsense ideas??

Second: "In addition don't use the code from this repo ! The code is outdated..."

Wrong again... If you'll actually bother to "look", you will see that the files from the GitHub repo are exactly the same as the latest posted version from RonR on the RPi forum page. You don't need to take my word for this... you can download the zip file from the forum & clone the GitHub repo - and then do a diff on the two. But you've not done that have you?...  No - I didn't think so.
So - are you now going to "shut up", and retract your ignorance?  ...  I suspect not, but we shall see.

@jdrch

Hey folks I didn't mean to start an argument. I came across the script I asked about in an XDA post and was curious since I'd never heard of it before. Currently only @framps' tool is actively developed by the original developer, so there's that.

Which tool is that?... framps' tool I mean.

Which tool is that?... framps' tool I mean.

@seamusdemora scroll up, it's linked to in the thread.

@jdrch : Do you mean a fork he created of this repo (rpi-clone)??

@seamusdemora https://github.com/framps/raspiBackup/

@seamusdemora
Great you now started to explain in detail what's wrong for you in my statements above instead of just grumbling :+1:
Please note: I'm not saying image-backup is a bad tool. It's used by a lot of folks. It's a nice tool similar to rpi-clone which is also used by a lot of folks. image-backup has another design and because of this has other users.

(a) you can do a loop mount on the .img file to copy or edit any file you want.

Sure. You don't have to restore the whole backup if you just need some files to restore. losetup is the tool which helps to access any individual files in the backup.

(b) And of course you can keep an image-backup-produced image file on a USB, SD, NAS, local drive - whatever.

Sure. That's a difference to rpi-clone where you can only have a device as a target instead of a directory. There is a PR out there for rpi-clone to support loop devices. Then rpi-clone also will be able to store a clone in any directory.

(c) Finally - it is not a "dd image"... image-backup uses rsync to copy files from the "live" system to the *.img file.

Sure. rsync is used to populate an image mounted with losetup - not dd.
Please read my statement from above again carefully :wink:

But in contrast to rpi-clone the clone is stored in a dd image and you have to restore the dd image first before you can use it. With rpi-clone you have a cold standy because the clone is already stored on an SD card or USB disk and can be used immediately.

(a) I didn't talk about partial restore of some data. I talked about a full restore of the whole clone. If you want to do this with image-backup you have to restore the backup on Linux with dd first (I didn't write anything about creation of the backup with dd). In contrast rpi-clone has the backup already available and no restore is required. That's one of the major difference for me between rpi-clone and image-backup.
(b) I didn't write this cannot be done with image-backup. I wrote rpi-clone creates the clone on a device like a SD card or SSD which then can be used immediately to boot the system. That's something image-backup cannot do.
(c) I didn't write dd is used to create the backup. I wrote you have to use dd to restore the backup. And keep in mind: The result of image-backup is an dd image which can be accessed with losetup and be restored with dd on Linux and other tools like Etcher, Rufus or win32diskimager on Windows. That's something I learned a lot of folks like because they can restore the backup on Windows instead on Linux. Mostly because they don't have another Linux system available - even the Raspberry  runs with Linux and can be used to restore a backup or they feel much more comfortable on Windows than on Linux.
Please read my statement from above again carefully :wink:

In addition don't use the code from this repo ! The code is outdated and it's not the repo of Ron, the author of the code. He doesn't maintain his code on github. If you want to use his code grab the latest code directly from his thread in https://forums.raspberrypi.com/

Your repo was updated 3 months ago. In the mean time Ron added multiple new versions of his code in the forum. So your repo is outdated. And I've seen multiple times folks grabbed Rons code from some github repos (not sure whether it was yours or others) and reported an issue in the forum thread and Ron just answered it's already fixed: Just grab my latest code from my first post in this thread.

The updated RPI-clone vis backing up directories..... RPI-clone is so fast, who cares about all the detail about which directories need backing up – it does the lot – just changes – typically. Using my aliases - a typical clone to SD involves the word CLONE and that;'s it - can do a clone blind drunk - often in a minute - no thought - no effort.... when I refer to rpi-clone, today on RPI and Bookworm of course, I'm referring to this.  curl https://raw.githubusercontent.com/geerlingguy/rpi-clone/master/install | sudo bash

The updated RPI-clone vis backing up directories..... RPI-clone is so fast, who cares about all the detail about which directories need backing up – it does the lot – just changes – typically.

image-backup is also very fast because it uses rsync to backup only changes. I don't want to have a flame war rpi-clone against image-backup here :flushed: . Both are nice and heavily used tools. I just want to make sure everybody understands the major differences between them and then can decide which tool is the best one for his Raspberries.

Both are nice and heavily used tools. I just want to make sure everybody understands the major differences between them and then can decide which tool is the best one for his Raspberries

Agreed. The more options we have, the better.

@framps

The updated RPI-clone vis backing up directories..... RPI-clone is so fast, who cares about all the detail about which directories need backing up – it does the lot – just changes – typically.

image-backup is also very fast because it uses rsync to backup only changes. I don't want to have a flame war rpi-clone against image-backup here 😳 . Both are nice and heavily used tools. I just want to make sure everybody understands the major differences between them and then can decide which tool is the best one for his Raspberries.

Is this what you call a "graceful retreat"? LOL

No. It's just a statement to make sure we're on the same page :wink:

Ah - so is that an admission that you were mistaken?
But after your BS rant - NO... We are not now, nor will we ever be "on the same page".  You have issues, my friend - you should get some help.

<((((*>

Oh-h-h!... look everyone - he's so clever.  :)

Is there nyone who can help me on this issue?


Add open boundaries

based on #437 and #440

Add to news.md and update examples doc page.

Update NEWS.md and README.md. Also add pictures to the docs for the examples or add it to the list here #495.


Unnecessary escape character apparently

Unnecessary escape character: \]. (no-useless-escape)

The regex:
/Expected property "1" of type \[Buffer\], got String "foobar"/

Is this one for eslint? Or am I misunderstanding?
edit:  This apparently works... is it even sane though? It feels so odd.
/Expected property "1" of type \[Buffer], got String "foobar"/

This only started happening in a recent update.

The no-useless-escape rule got stricter in the latest version of eslint. I think they're right that it's technically useless to escape ] but I agree that this is a tad bit aggressive.
This might be the source of the issue: https://github.com/eslint/eslint/issues/7424
Could you open an issue on the ESLint repo?

One good argument for why ESLint should not warn about \] is that regex's "Unicode mode" (which @mathiasbynens recently described as "strict mode" for regex literals) doesn't warn about it.
/\a/u - throws because a is always an invalid escape sequence. Prevents user from making a mistake.
/\)/u - does not throw even though ) does not need to be escaped. This is because ) sometimes does need to be escaped, and it doesn't hurt to just allow it. Very unlikely this is a user mistake.
This let's users keep things nice and parallel in their regex, like: /\[Buffer\]/u. My opinion: if it's good enough for regex "Unicode mode", then it's good enough for ESLint. 👍

I opened an issue: https://github.com/eslint/eslint/issues/7656

Thanks for championing this @feross

Sure. If they end up adding a way to specify exceptions to this rule, we can open a new issue for that.

FYI, the original eslint issue was fixed: https://github.com/eslint/eslint/issues/7656


Update DINO CPU due dates

https://github.com/jlpteaching/comparch/blob/e3e5e32f9fca4b875f6ccde1c242666489476588/syllabus.md#L257-L262

@takekoputa please take the info from here (https://trello.com/b/STcHN7zh/ecs-154b-201a-wq22/calendar/2022/01) and update these assignments.
We'll also need to change the links, but I'll get that info to you soon.

@powerjg Assignment 4 and 5 due dates are not in the trello board.

@powerjg Assignment 4 and 5 due dates are not in the trello board.

You can put 2 weeks between each as a placeholder for now.


Add Dask Contrib docs

This PR adds a documentation page describing the Dask Contrib GitHub Organization.
I have kinda arbitrarily made up all the criteria and stuff in here with things I consider sensible to give the community some guiding points to keep in mind. However some discussion and feedback here would be much appreciated.
For example I've written it with pretty formal language, but perhaps we want to make it more informal. Perhaps we want to add/remove criteria. Perhaps we don't want any criteria at all.

Note that we should also agree on who has the ability to +1 repos for inclusion and who has write access on the org (the same set as for the dask org?). Presumably the original maintainers of the new contrib package would carry forward their permissions to the moved repo.

I haven't looked at this for a while. Folks are very welcome to push commits here.

So does dask-contrib function as a way to prototype new features for dask?

So does dask-contrib function as a way to prototype new features for dask?

If you follow the conversation above, you will see that we are both talking about "incubation" (repos that may eventually move to the dask/ org) and "adjunct" (repos that add a specific thing outside of what we may consider a core concern of dask). Either of these may be experimental, but being in dask-contrib already assumes a certain level of readiness. For pure POCs/prototypes, there is no need to move them to dask-contrib.
This is not a place for changes that are meant to be affected on the core repos, dask/dask and dask/distributed. We use PRs for that, which might be marked as draft.

Ah I see, sorry for the confusion on my part. Thanks again for the clarification @martindurant, much appreciated.

I moved the project we were talking about here just to get started.
https://github.com/bparbhu?tab=projects
Should this topic continue under this project or should I move the project to someplace under the dask-contrib banner?
I'm not sure what would be best.
Thanks guys.


Fix: ParsingNeuralNetwork and ParserGenerator

Adding a small change to the ParsingNeuralNetwork build method which now also accepts dai.NNModelDescription (next to dai.NNArchive). This is done to allow abstracted pipeline creation as shown in the DAI example HERE.
Moreover, ParserGenerator is moved one level higher to prevent errors with getting parser classes from the globals (globals().get(parser_name)).

Following changes in DAI (v3_develop branch of depthai-core) I'm changing the ParsingNeuralNetwork.build() to now accept both dai.NNModelDescription and dai.NNArchive


Still busy

Only I can play one time... second time, system halt.
0|MagicMir | hotword 1 alexa
0|MagicMir | transiting to state: listening
0|MagicMir | transiting to state: busy

0|MagicMir | hotword 1 alexa
0|MagicMir | transiting to state: listening
0|MagicMir | transiting to state: busy
0|MagicMir | transiting to state: idle
0|MagicMir | hotword 1 alexa
0|MagicMir | transiting to state: listening
0|MagicMir | transiting to state: busy
0|MagicMir | transiting to state: idle
0|MagicMir | hotword 1 alexa
0|MagicMir | transiting to state: listening
0|MagicMir | transiting to state: busy
0|MagicMir | transiting to state: idle
0|MagicMir | hotword 1 alexa
0|MagicMir | transiting to state: listening
0|MagicMir | transiting to state: busy

Are your speakers working?
0|MagicMir | transiting to state: listening
0|MagicMir | transiting to state: busy
0|MagicMir | transiting to state: idle

This indicates to me that it did the full loop: listening busy (which is speaking to you), and idle, back the the beginning.

Yes it working.
I ask to Alexa two or three times, after that hangs.
Regards,

What are your OS and specs? For me, I can ask as many times as I want

Raspberry pi 3 using Raspbian Lite.

I will try it on my Raspberry Pi over the weekend
I am running it on:
MacOS Sierra
Rasberry Pi 3 with Raspbian + PIXEL

@dolanmiu Do you tried?

Yes, I have tried
Sorry about the delay
works fine for me, I tried it 6 times in a row on my Raspberry Pi, and it worked every single time
Just a quick check, on your Pi, can you run:
$ rec t.wav

If you can't, then there is something wrong with your mic
Raspberry Pi has weird issues with the mic:
https://www.raspberrypi.org/forums/viewtopic.php?t=13088&p=332703

I have noticed the same on my mac that I can get alexa respond to me like 3 times. Then it stops responding 🤔

@henrikra
Found the issue:
https://github.com/electron/electron/issues/9201

Hmm very interesting since when that error is in the log the Alexa won't hear me anymore. At electron they said it is not their fault. But MagicMirror is done with Electron so what was your idea that it can be MagicMirror bug? :D

Because magic mirror uses Electron, and if they somehow don't utilize Electron properly, perhaps it messes up? just speculation :P

This is fixed in #38


security issues

@liara/cli is dependent on packages that have security issues.
in the following screenshot you can see my project npm audit check before installing @liara/cli and after installing it.

also running npm audit fix wont help it either.
for more details and reproduce problem:

install latest @liara/cli package
and run npm audit

Hello,
thank you for bringing up that issue. These security issues pertain to the latest versions of the dependencies, and currently, there are no available fixes for them. We will promptly update our CLI once updates for these dependencies become available.

It is also important to note that the Liara CLI should be installed globally, rather than as part of the project dependencies.
npm i -g @liara/cli

It is also important to note that the Liara CLI should be installed globally, rather than as part of the project dependencies.
npm i -g @liara/cli

why it is recommended?

@imkarimkarim Hi, thank you for bringing up the issue.
We’ve just released a new beta version that addresses the reported vulnerabilities. We highly recommend updating to this version to benefit from the fixes.
You can install the beta version by running the following command:
npm i -g @liara/cli@beta

Feel free to reach out if you have any questions or run into any issues after updating.

@imkarimkarim Hi, thank you for bringing up the issue. We’ve just released a new beta version that addresses the reported vulnerabilities. We highly recommend updating to this version to benefit from the fixes.
You can install the beta version by running the following command:
npm i -g @liara/cli@beta

Feel free to reach out if you have any questions or run into any issues after updating.

thank you for taking care of the issue. i will switch to beta version and let you know if encountered any issue.
thanks again 🫡


Use %v to format nil argument in Fatalf

Trivial suggestion to fix arguments to t.Fatalf in tests.

Fixes the following go vet reports:
./search_aggs_test.go:709: T.Fatalf format %q has arg dateHistoRes.Buckets[0].KeyAsString of wrong type *string
./search_aggs_test.go:721: T.Fatalf format %q has arg dateHistoRes.Buckets[1].KeyAsString of wrong type *string
./search_test.go:355: T.Fatalf format %q has arg hit.Source of wrong type *encoding/json.RawMessage


#52 Create test cases

Summary
It closes #52
Extra info

Needs more than 2 reviewers? y
Needs more than 10 minutes for review? y
Needs to execute in order to review? y

Images
Further Work
Discussion of test case

Do something...

https://github.com/nodkz/mongodb-memory-server
mongodb-memory-server 에 대해서(mockgoose라는 dummy database 만드는 lib이 사용 불가 되었어요 )


Add BulkGet support for SQLite and Postgres state stores

Description
This PR implements proper BulkGet for the SQLite and Postgres (so, also CockroachDB) state stores. Also adds a conformance test for BulkGet.
This is particularly helpful for actor reminders. When BulkGet isn't implemented, the runtime performs multiple Get operations in sequence, causing more I/O than needed. A "proper" BulkGet performs a single SELECT query that retrieves multiple rows.
Issue reference
Related to dapr/dapr#6121

Thanks for the review @JoshVanL


Add -codec execution option

Hi @mpgn!
In this PR I'd like to add --codec option to execution methods. I've been always suffering from the inability to specify my own codec for exec output readability (English is not native language for most of my customers), so I had to modify CME source every time I deploy a new Kali instance. So I though if there's a chance to put this in the upstream to make my like a bit easier :sweat_smile:

Great !


silodiff on a few example cases to be sure the silo files are correct

Following #107 :
Implemented: "Dumb test": with parallel_IO = 'T' we want to be sure the parallel files are actually created and that they do not create NaNs or an out-of-bounds number. You can do the latter via a call to h5dump on the files.
What remains: "Smarter test": Do a silodiff on golden silo files and compare them to the created ones.

Closing this as the Silo build system does not support the simultaneous build of Silo binaries.


Integrate with 47deg CI/CD

The standard 47deg CI/CD setup should be brought into this repository so that the microsite and artifacts can be properly deployed and published.

Done! Thanks @sloshy


Prep the / (root) endpoint for Liveness and Readiness

Liveness and Readiness can be effectively implemented if the service ends gracefully in the event of -

Bad configuration
Services that it depends on are not available

In the PR, Redis is being checked for availability.
We should perhaps add similar checks for both Topology Inventory Service and Host Inventory Service
e2edeploy PR - https://github.com/RedHatInsights/e2e-deploy/pull/342

Codecov Report

Merging #42 into master will decrease coverage by 0.4%.
The diff coverage is 25%.

@@            Coverage Diff             @@
##           master      #42      +/-   ##
==========================================
- Coverage   40.27%   39.86%   -0.41%
==========================================
  Files          14       14
  Lines         437      449      +12
==========================================
+ Hits          176      179       +3
- Misses        261      270       +9

Impacted Files
Coverage Δ

server.py
0% <0%> (ø)
:arrow_up:

collector/utils.py
48.71% <30%> (-6.46%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0c0c67f...da199a1. Read the comment docs.


feat: convert custom category to cdkV2

Description of changes

added new imports and changed package.json for new lib

Issue #, if available

Description of how you validated changes
Checklist

[ ] PR description included
[ ] yarn test passes
[ ] Tests are changed or added
[ ] Relevant documentation is changed or added (and PR referenced)
[ ] New AWS SDK calls or CloudFormation actions have been added to relevant test and service IAM policies

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

Merging #11079 (a7ecee7) into cdkv2 (24a811d) will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##            cdkv2   #11079   +/-   ##
=======================================
  Coverage   49.57%   49.57%
=======================================
  Files         682      682
  Lines       32848    32848
  Branches     6705     6705
=======================================
+ Hits        16285    16286    +1
+ Misses      15083    15082    -1
  Partials     1480     1480

Impacted Files
Coverage Δ

...category-custom/src/utils/generate-cfn-from-cdk.ts
44.44% <ø> (ø)

...li/src/domain/amplify-usageData/getUsageDataUrl.ts
100.00% <0.00%> (+12.50%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Closing in favour of https://github.com/aws-amplify/amplify-cli/pull/10988


Support for showing the slowest N tests

I see that here https://nexte.st/book/slow-tests.html - it is possible to set a limit for what is marked as slow. I think it would also be nice to show the slowest N (10?) tests without having to know the limit up front. This would be pretty easy to implement externally after #20 (just pipe to sort), but I bet it would look a lot nicer if internal.

Hi there --
Thanks for the feature request! This is currently not supported, and the way I'd like to add support for it is by logging previous runs and letting users replay them, show statistics about them and so on. For example:
cargo nextest run
cargo nextest replay slow-tests

However, that would be a big chunk of work that I don't expect to do any time soon.
Meanwhile, you could probably use the JUnit support along with some open source tooling that consumes them to build this out.


Component#isInitializing returns wrong value when direction is REPLACE

isInitializing is returning false for REPLACE transitions, even if this is the first time that the component is created.

Addressed by #105


Fix sierra class hash calculation

TITLE
Description
Description of the pull request changes and motivation.
Checklist

[ ] Linked to Github Issue
[ ] Unit tests added
[ ] Integration tests added.
[ ] This change requires new documentation.

[ ] Documentation has been added/updated.

just in case you didn't have enough approves


Run demo on WindowOS

I ran demo on WindowOS and got this issue: "Unable to load DLL 'NcnnDotNetNative' or one of its dependencies: The specified module could not be found. (0x8007007E)".

@AzukaPans
Could you tell me more detail?

If use .NET Framework

You should build as x64 rather than Any CPU

If use UltraFaceDotNet.GPU

It depends on NcnnDotNet.GPU and NcnnDotNet.GPU requires Visual C++ 2017 Redistributable Package and Vulkan SDK. You should install Vulkan SDK.

@AzukaPans
Could you tell me more detail?

If use .NET Framework

You should build as x64 rather than Any CPU

If use UltraFaceDotNet.GPU

It depends on NcnnDotNet.GPU and NcnnDotNet.GPU requires Visual C++ 2017 Redistributable Package and Vulkan SDK. You should install Vulkan SDK.

@takuya-takeuchi
I use .NET Framework, branch source: develop, version not GPU.
I tried to build as x64 but got the same issue.
Maybe I missed ncnndotnet source code in src folder.
I will close this question if that issue is fixed.

@takuya-takeuchi
Do I have to use CMake application to build NcnnDotNet.Native source in src ?
I have done steps below:

Using Microsoft Visual Studio 2019 to build UltraFaceDotNet.sln and NcnnDotNet.sln with Release mode as x64.
Following guideline in examples/Demo.

Have you figured out how to resolve your issue @AzukaPans ?

@AzukaPans
Sorry late reply.
I guess VS 2017 shares same binary of 2019. So it should work.
Could you check the this exmaple?
#8.zip
This is work on dev machine.
https://github.com/takuya-takeuchi/NcnnDotNet/issues/8#issuecomment-573445271
This issue looks like to wrong.

@AzukaPans
Sorry. I did explain wrong.
https://github.com/takuya-takeuchi/UltraFaceDotNet/tree/master/examples/Demo needs NcnnDotNet library.
But this example links UltraFaceDotNet project from git submodule.
So you should delete UltraFaceDotNet link and add nuget package.
Or copying NcnnDotNet and UltraFaceDotNet all libs to application bin directory manually.
It should works.

I encountered this issue, and here's my workaround for it.

remove project reference from NcnnDotNet and UltraFaceDotNet.
install NcnnDotNet and UltraFaceDotNet from nuget.
Build as x64. (It should not build on AnyCPU)
NcnnDotNetNative will be in UltraFaceDotNet/examples/Demo/bin/Release/netcoreapp3.1/runtimes/win-x64/native/NcnnDotNetNative.dll
Copy NcnnDotNetNative to folder where Demo.exe is in.

I think this (https://github.com/takuya-takeuchi/NcnnDotNet/issues/8) could be fixed by changing
https://github.com/takuya-takeuchi/NcnnDotNet/blob/master/nuget/nuspec/NcnnDotNet.CPU.nuspec#L30-L33
target to content\NcnnDotNetNative.dll.

This issue could be fixed by latest version


Convert to TF 0.12. Add tests. Add Codefresh test pipeline

what

Port module to Terraform 0.12
Pin all providers
Add example for testing
Add bats and terratest for the example
Add Codefresh badge to point to the test pipeline in terraform-modules project
Update README

why

Module currently does not work with 0.12. Much easier syntax
Better regression control
Automatically test the example on every commit and pull request
Provision resources on AWS in the test account and check the outputs for the correct values
terraform-modules project contains pipelines for all terraform modules

related

Closes #40

The example file looks like it is not for Elasticache but instead is for Elasticsearch.
Additionally, the link from the readme to the Example goes to a 404


Tutorial home

Fixes #22.

lgtm, I don't suppose folks will want, say their ~/.Renvron and these tutorial files? Either way, I guess we could still just prep the env in that working_dir_host.

Users might expect to have their regular environment. However, part of Shameema's specification for this tutorial mode is a clean environment; so no environment files, startup files, or pre-installed modules.


JSON is too deeply nested (SystemStackError)

I am having an issue and I can only reproduce it in my jenkins instance so I assume its something that is related to the system, ruby, and gems installed. Here is the relevant output from my jenkins job with some extra information such as ruby version and gems added for debug purposes.
https://gist.github.com/majormoses/721d610a1a11c0ffde9e1a1aa594cba1
Here is the simple json that was used to test:
https://gist.github.com/majormoses/e3245bf85ec2b1d1248ea2159f6b11f0
Any useful insight as where to look would be greatly appreciated.

This might be a bug on the OJ side.  Two like it have been reported over the years:
https://github.com/ohler55/oj/issues/50
https://github.com/ohler55/oj/issues/69

@shortdudey123 https://github.com/ohler55/oj/issues/333#issuecomment-279038941 where would you like to start?

The referenced OJ issue have nothing to do with this. They are old and not even for the same code used for the SAJ parser. Completely a red herring.
Can someone provide me with the hander used for the SAJ parser and the JSON being parsed? Is it the same short one in the gist? https://gist.github.com/majormoses/e3245bf85ec2b1d1248ea2159f6b11f0

Looks like you were faster on the comment. :-)

Here is a simple script I tried. It ran with no problems locally. Is it representative of the use in this issue?
require 'stringio'
require 'oj'
require 'jsonlint'

json = %|{
  "id": "babrams",
  "ssh_keys": [
    "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDMCKMvMJ2yb13pHLsIPqi2xHOBKFZKa8+FM1FUqNbIxKeq3LLw+4WiXLK60DxxS6hmOnXD+FcWNaykkGLyGQeYHxsHynsXo9BPaG/ewaAp5SDU/zAIAaex15s/zvo5l+5Pq9OwXYtFRmfezk3ImCx7SZ8sMmHiFHYD8d38XBlGX53kLSFm5HLEopEvSCRTUyTj+tPIspgYR6IvCTdXnamO9FT8Rkeqw+mqjX9sVTaLuuqwQZlRFRMslrrJbSfv+7XvyKsjOsmAlkEYRlpHbUCxUh2Hc5q2Wfm+acOHPkkUPX8kLeT2vW+Bd/9LlPi9BN0dbmazGPbf5kv02MRNQNeUrdRfdzRIOG4tUEv154msF7QdEuy9W4pv9p0z2rNOqOQEw9HPhMiAkftIVGnvvGRj9+jIARIVzV5gAfVm2DQbPJClr0tGNCfzHmndt6FddawubXFPvFNrKgdC38Ts0Jzl1F3aWGHT8UyURDbezrTGpxg+Cqq4YUXIZfrrqB8nzF8qK3eMW2Tcxdy2m+fFBzQeHlozBSP55dcdjekdQrcVcwkYux4jecJ9BU++DjWtMtY93LgVL5BnHixS4ybo7loCndYkpsI6ZZm9oLVxHsjeoaM9D9iYoN28LIlALBm/dnfCh92G/H40v/X25DMIvRqcfnE31gsOCJ85A29twSC+Cw==<EMAIL_ADDRESS>    "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCy3cbPekJYHAIa8J1fOr2iIqpx/7pl4giJYAG7HCfsunRRUq3dY1KhVw1BlmMGIDzNwcuNVIfBS5HS/wREqbHMXxbwAjWrMwUofWd09CTuKJZiyTLUC5pSQWKDXZrefH/Fwd7s+YKk1s78b49zkyDcHSnKxjN+5veinzeU+vaUF9duFAJ9OsL7kTDEzOUU0zJdSdUV0hH1lnljnvk8kXHLFl9sKS3iM2LRqW4B6wOc2RbXUnx+jwNaBsq1zd73F2q3Ta7GXdtW/q4oDYl3s72oW4ySL6TZfpLCiv/7txHicZiY1eqc591CON0k/Rh7eR7XsphwkUstoUPQcBuLqQPA529zBigD7A8PBmeHISxL2qirWjR2+PrEGn1b0yu8IHHz9ZgliX83Q4WpjXvJ3REj2jfM8hiFRV3lA/ovjQrmLLV8WUAZ8updcLE5mbhZzIsC4U/HKIJS02zoggHGHZauClwwcdBtIJnJqtP803yKNPO2sDudTpvEi8GZ8n6jSXo/N8nBVId2LZa5YY/g/v5kH0akn+/E3jXhw4CICNW8yICpeJO8dGYMOp3Bs9/cRK8QYomXqgpoFlvkgzT2h4Ie6lyRgNv5QnUyAnW43O5FdBnPk/XZ3LA462VU3uOfr0AQtEJzPccpFC6OCFYWdGwZQA/r1EZQES0yRfJLpx+uZQ== babrams@babrams-Serval-WS"
  ],
  "htpasswd": "$1$i2xUX9a4$6LwYbCk4K6JErTDdaiZy50",
  "groups": [
    "devops"
  ],
  "shell": "\/bin\/bash",
  "comment": "Ben Abrams"
}|

linter = JsonLint::Linter.new
linter.check_stream(StringIO.new(json))

puts "errors: #{linter.errors}"

@majormoses Sorry to hear you're running into issues.
I tried reproducing this issue under OS X 10.12 with the same Ruby version, bundler version, and set of Gem you're using in your Gemfile, but the command completes successfully.
I also tried reproducing in a trusty64 virtual machine, but that also completed successfully.
It seems like some detail is missing here. I put a script in a gist that might help collect some forensics. Mind running that on your jenkins instance and sending the output over?
https://gist.github.com/dougbarth/84e6ccc3825d92aaad61fc2ab4e7fd59
Reproducing this in a clean environment (preferably in a VM) might be possible with guidance from that output.

@dougbarth https://gist.github.com/majormoses/721d610a1a11c0ffde9e1a1aa594cba1#file-ruby_details

What is the status on this? Anyone?

waiting for further troubleshooting advice...

@dougbarth ping any thoughts?

@dougbarth @ohler55 I tried messing again with this on our jenkins instance and after playing around I found something curious it only is re-produceable with Make (though I can not replicate locally sadly).
here is a gist that might maybe provide some more insight: https://gist.github.com/majormoses/a83a68426af45931adc0f5e0d466c305

so I think I found it! so it actually was the stacksize...after bumping this it worked I updated the comments on the gist to reflect the process...My guess is the reason I did not see this locally is that I have a newer version of make that likely more efficient?

Must be a very small stack size. The JSON is not very deep at all. Maybe the Jenkins VM or machine is small.

it was 8192 and bumped it to 16384

I still think there is a bug but its likely within that version of make itself.

It that in bytes? If so that is more like an embedded processor setting.

yes thats in bytes.

Wow, that is extreme. Glad you found the issue.

ya I dont like this "fix" at all, very few times should an application need such level of recursion...but hopefully we will be able to get onto a newer version of ubuntu soon and will see if we can replicate with a newer version of make. Do you think we should open new issues to do some profiling to see if we can turn anything up?

With the json you provided the stack depth is not deep at all but on a 64bit machine 8192 byte does not give you very many pointers and variables. A stack allocation of 4096 for a buffer uses up half the stack. 4096 is a single page. Usually the stack size is measured in MB.

@ohler55 sorry I misspoke that is in KB.

$ ulimit -a | grep stack
stack size              (kbytes, -s) 8192

Glad to hear you found a lead on the issue. I tried recreating the problem myself in a Docker container: https://gist.github.com/dougbarth/160de6ac13120103bfb1bd505901f6e1
Note: I'm not using the exact same Ruby version, but it's Ruby 2.2 and is using the same set of gems
At dramatically smaller stack sizes than you're using (failures start around 38KB), the program eventually fails with a SystemStackError, but I can't get it to fail at the call to Oj.saj_parse.
8192 seems to be the default stack size limit, so I'm not sure why you're running into this issue at that size.
It seems like this issue must be specific to something on your Jenkins server.

Ya it feels very wrong to me to increase it (at least in this case) but I can not find anything on the Jenkins node pointing to anything enlightening. I will try to spend some time next week if I can spare it to try digging deeper. @dougbarth could you verify what version of make you tested with? So far that is the only thing that makes even a shred of sense to me is that there is a bug in the version of Make on our Jenkins node that effects this in a very odd way.

Looks to be the same version as your Jenkins server.
root@506320a3eb4e:/jsonlint# make --version
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for x86_64-pc-linux-gnu

hmm that kills my theory...

So I think I figured out what was going on and I don't think it was caused directly by this gem. It has to do with how our Makefile was structured. I was able to reproduce it on travis as well so it was not longer a wonky jenkins in question. Basically every time you call a target from another target it spawns another make process. It looked something like this:
rubocop:
  # do my rubocop things
 foodcritic:
  # do my foodcritic things
chefjsonlint:
  # do my json linting of chef objects
ci: rubocop foodcritic chefjsonlint

My guess is that by the time it gets through spawning all those make processes makes it's stack size at that point too close to normal/sane limits before running it. When I have some more time later I will try to profile it to better understand how much is being used by what.


[Bug]: ts-loader runs into an error with rspack

System Info
System:
OS: Linux 6.2 Ubuntu 20.04.6 LTS (Focal Fossa)
CPU: (32) x64 AMD EPYC 7763 64-Core Processor
Memory: 113.67 GB / 125.78 GB
Container: Yes
Shell: 5.0.17 - /bin/bash
Binaries:
Node: 20.11.0 - /usr/local/share/nvm/versions/node/v20.11.0/bin/node
Yarn: 1.23.19 - /usr/bin/yarn
npm: 10.2.4 - /usr/local/share/nvm/versions/node/v20.11.0/bin/npm
Watchman: 20240317.093234.0 - /usr/local/bin/watchman
Browsers:
Chrome: 124.0.6367.60
npmPackages:
@rspack/cli: 0.6.2 => 0.6.2
@rspack/core: 0.6.2 => 0.6.2
Details
So running into the below error when using ts-loader for some files. (for other files I am using builtin:swc-loader) .
ERROR in ../../someFile.graphql.ts
  × Module build failed:
  ╰─▶   × TypeError: Cannot read properties of undefined (reading 'buildMeta')
        │     at setModuleMeta (/workspaces/teams-modular-packages/node_modules/ts-loader/dist/index.js:62:31)
        │     at makeSourceMapAndFinish (/workspaces/teams-modular-packages/node_modules/ts-loader/dist/index.js:56:5)
        │     at successLoader (/workspaces/teams-modular-packages/node_modules/ts-loader/dist/index.js:39:5)

And this error is coming from the following piece of code in ts-loader:
function setModuleMeta(loaderContext, instance, fileVersion) {
    // _module.meta is not available inside happypack
    if (!instance.loaderOptions.happyPackMode &&
        loaderContext._module.buildMeta !== undefined) {
        // Make sure webpack is aware that even though the emitted JavaScript may be the same as
        // a previously cached version the TypeScript may be different and therefore should be
        // treated as new
        loaderContext._module.buildMeta.tsLoaderFileVersion = fileVersion;
    }
}

I do not run into this error with webpack.
Currently I am getting around this issue with setting happyPackMode as true for rspack.
Reproduce link
No response
Reproduce Steps
try using ts-loader for some .tsx. file -> should repro this error.

LoaderContext does not support _module yet, but PR is welcome

LoaderContext does not support _module yet, but PR is welcome

I would like to give it a try.

@coder-xiaotian Adding custom fields to buildMeta does not quite reasonable while we need to pass the value back and forth from rust to node. This is not related to this issue, but exposing a readonly _module is fine.

@coder-xiaotian Adding custom fields to buildMeta does not quite reasonable while we need to pass the value back and forth from rust to node. This is not related to this issue, but exposing a readonly _module is fine.@coder-xiaotian 在 buildMeta 中添加自定义字段不太合理，而我们需要在 rust 和节点之间来回传递值。这与此问题无关，但公开只读 _module 就可以了。

Ok, Get it.

Any follow-ups here? Would love to see typescript errors integrated with hot refresh

Any follow-ups here? Would love to see typescript errors integrated with hot refresh

I have committed my code to fix ts-loader error.


Error: decoding to str: need a bytes-like object, tuple found

I recently installed carml, and executing any command results in an error. Stack trace attached (with -d option):
carml -d stream -L
Error: decoding to str: need a bytes-like object, tuple found
Traceback (most recent call last):
File "/home/piyush/.local/lib/python3.8/site-packages/twisted/internet/defer.py", line 661, in callback
self._startRunCallbacks(result)
File "/home/piyush/.local/lib/python3.8/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
self._runCallbacks()
File "/home/piyush/.local/lib/python3.8/site-packages/twisted/internet/defer.py", line 857, in _runCallbacks
current.result = callback(  # type: ignore[misc]
File "/home/piyush/.local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1750, in gotResult
current_context.run(_inlineCallbacks, r, gen, status)
---  ---
File "/home/piyush/.local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
result = current_context.run(gen.send, result)
File "/home/piyush/.local/lib/python3.8/site-packages/carml/cli.py", line 156, in _startup
print("Connected via {}".format(str(tor.protocol.transport.addr, "utf8")))
builtins.TypeError: decoding to str: need a bytes-like object, tuple found
It seems like there is some change in the way tor returns certain data that the carml's current implementation is not able to decode.

Can you say how you installed this?
I did a pip install carml in a new virtualenv (ended up with Python 3.9.2) and the above command worked. Can you dump your python version and results of pip freeze (if it's in a virtualenv) please?


Using provider config context

I'm not sure if this is already possible, but I'd like use the config context of clusters managed by the provider. My use-case is as follows:

I'm creating a cluster using this provider, terraform-provider-eksctl.
I'm creating various resources in the cluster using the Kubernetes provider.

The README mentions a computed output but I'm unsure how to use it.
My current workaround simply entails using an explicit dependency via depends_on for my Kubernetes resources. However, I can see this being a problem in a multi-context scenario.
It seems that eksctl utils write-kubeconfig may be useful.

@dalberto Thanks for trying this proider! FYI, recent versions of the eksctl provider outputs kubeconfig_path that can be used by e.g. helmfile resource.
To be clear, Helmfile resource works because it can configure which kubeconfig to use "per resource", not "per provider" like the kubernetes provider does. Probably the situation is the same for the helm provider, right?
For that reason, I was also considering to implement dedicated terraform provider/resource for triggering kubectl/helm operations, that supports setting kubeconfig per tf resource.  I've tested the newest (v0.6.0) and can confirm this is useful. However, it seems like I'd also need the config_context in order to provide enough authentication information for the built-in provider.
I believe a provider/resource that allows per-resource operations would be useful, but for now I'd be satisfied with configuring the built-in provider and have those changes apply globally to all resources.
Relatedly, is it possible to get the generated cluster name? I noticed that the provider supplies a suffix and so the actual cluster name is different. Alternatively, is it possible to disable the suffix generation? I can see a workaround in which cluster auth/connection information can be derived from a data source, but the full cluster name would be required. I've also run into issues provisioning resources where the "real" cluster name does not match the prefix supplied (the alb-ingress-controller for example). For now, I've resorted to deriving the "real" name by:
"${var.cluster_name_prefix}-${eksctl_cluster.cluster.id}"

similar to how it's done https://github.com/mumoshu/terraform-provider-eksctl/blob/429cb984608ccea309133077ab8f7cdbcec988cd/pkg/resource/cluster/cluster.go#L265

Relatedly, is it possible to get the generated cluster name? I noticed that the provider supplies a suffix and so the actual cluster name is different.

Yes. If you use thew new eksctl_cluster rather than old/renamed eksctl_cluster_deplyoment the name you specified is the exact cluster name.

With the exact cluster name, the aws_eks_cluster_auth data source can be used to configure the Kubernetes provider.
While somewhat inconvenient, according the the Kubernetes provider docs, it is a best practice to have two separate apply commands: one for the cluster and one for the resources within the cluster.
For posterity I did the following:
data "aws_eks_cluster" "cluster_data" {
  name = eksctl_cluster.cluster.name
}

data "aws_eks_cluster_auth" "cluster_auth" {
  name = eksctl_cluster.cluster.name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster_data.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster_data.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.cluster_auth.token
  load_config_file       = false
}

Subsequently,
terraform apply -target='<path>.eksctl_cluster.cluster'
terraform apply -target='<path>.k8s_resources'

Creates a cluster.
Creates resources unambiguously in that cluster.

Thanks again for your help and your work on this provider.


Fix [object Object] over icon text in siteList

What happened (or feature request):
Fix [object Object] over icon text in siteList
How to reproduce this:
Mouseover on a site icon in the site listing

PR #155 was merged


0.98.3 has killed the recorder and logbook

0.98.3
Home Assistant release with the issue:

0.98.2
Last working Home Assistant release (if known):
Debian Hass.io
Operating environment (Hass.io/Docker/Windows/etc.):

Component/platform:

Description of problem:
No states recorded and no history
Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

Wed Sep 04 2019 20:33:39 GMT+1000 (AEST)
Error executing query: 'latin-1' codec can't encode character '\u2013' in position 3: ordinal not in range(256)
Traceback (if applicable):

Additional information:

I downgraded to 0.98.2 and everything was ok again. Then upgraded to 0.98.3 again and it seems ok now as well. I did restart the 0.98.3 that produced the error and it was showing the error again. After downgrade/upgrade it seems ok......

It seems ok now after the re-install so I'm going to close this.


Filenames not showing

Subject of the issue
The same situation as #10 but there are no filenames in a Summary too.
Please, help to solve problem: https://github.com/Limych/HomeAssistantConfiguration/actions/runs/769191693
Your environment

OS:
Packages:
Env:

Steps to reproduce
Tell us how to reproduce this issue.  Please provide a working and simplified example.
🎉 BONUS POINTS for creating a minimal reproduction and uploading it to GitHub.  This will get you the fastest support.  🎉

Expected behavior
What should happen?
Actual behavior
What happens instead?

Unfortunately, the problem still persists.

Thanks 😊

No problem!


tweak: enable lazy loading of agenix

See https://discourse.doomemacs.org/t/common-config-anti-patterns/119#loading-packages-too-early-3
Sorry, had to do it… ;-P
Jokes aside, thanks a lot for explaining the anti-patterns!

It's already lazy-loaded; it uses :mode. There's a pinned notice in that post:

Glad it could help, in any case!

Thanks again!


Deprecate all EC_KEY_* functions

From point 2 of the Technical Items still to be done list:
Deprecation List Proposal: DH_, DSA_, ECDH_, ECDSA_, EC_KEY_, RSA_, RAND_METHOD_.

Does not include macros defining useful constants (e.g. SHA512_DIGEST_LENGTH).
Excluded from Deprecation: EC_, DSA_SIG_, ECDSA_SIG_.
There might be some others.
Review for exceptions.

This is finished


Crash that is preventing me from viewing compatibility issues with a set of unrelated mods

https://pastebin.com/zfkUHwA6

Yeah - this is a tough one to resolve.  The DS core mod ran, but DS itself isn't loaded because the startup is stillborn.  You figure Forge would stop further execution in the client, but for whatever reason it continues and wants to play music and other stuff. :\  For your particular circumstance you could remove DS and it's coremod, resolve the dependency issue, and put it back in.
This problem is something I am looking at for 3.5.  Not sure how I can resolve at the moment.

Pushed a BETA of v3.5 to CurseForge.  Note that the new version has a library dependency on OreLib (another library mod I made).  Make sure you read the release notes prior to upgrade.


measurement

Hello, is there a way to implement measurements on panoramas with pannellum? Maybe someone do something like that?

There's no such functionality built-in, but you could probably build something using a custom event handler for mousedown / mouseup / touchstart / touchend events.


minor error handling fix

I propose the following correction. I think it will be better

Something I hurried =)