[BUG] Upgrade stuck with first node in Post-draining state

Describe the bug
Upgrading from v1.1.2 to v1.2.0, first node to be updated gets stuck in post-draining state.

To Reproduce
Steps to reproduce the behavior:

Create a version object to enable v1.2.0 upgrade, run pre-check script with all pass, then upgrade.
Had to edit the Secret fleet-agent-bootstrap to contain the FQDN url instead of VIP url since we use Letsencrypt, SAN doesn't contain the IP, from this comment: https://github.com/harvester/harvester/issues/4519#issuecomment-1715692409
Upgrade from v1.1.1 to  v1.1.2 rendered rancher-logging-crd in Pending-Upgrade state, this had to be deleted so that it could be recreated by the upgrade which seems to work fine. Status for it was ErrApplied(1) [Cluster fleet-local/local: cannot re-use a name that is still in use]

Expected behavior
Upgrade to finish successfully
Support bundle
E-mailed the support bundle to harvester-support-bundle [ at ] suse.com
Environment

Harvester ISO version: v1.2.0
Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630): Baremetal using Supermicro's Microcloud

Additional context
From kubectl --context pre-harvester01 -n harvester-system logs -f jobs/hvst-upgrade-f79bg-post-drain-pre-harvester01-01:
++ date +%Y%m%d%H%M%S
+ elemental_upgrade_log=/usr/local/upgrade_tmp/elemental-upgrade-20230913120242.log
+ local ret=0
+ mount --bind /usr/local/bin/elemental /host/usr/bin/elemental
+ chroot /host elemental upgrade --logfile /usr/local/upgrade_tmp/elemental-upgrade-20230913120242.log --directory /tmp/tmp.n2JZm1dxDv --config-dir /tmp/tmp.N6rn4F6mKM
Flag --directory has been deprecated, 'directory' is deprecated please use 'system' instead
INFO[2023-09-13T12:02:42Z] Starting elemental version 0.3.1
INFO[2023-09-13T12:02:42Z] reading configuration form '/tmp/tmp.N6rn4F6mKM'
ERRO[2023-09-13T12:02:42Z] Invalid upgrade command setup undefined state partition
elemental upgrade failed with return code: 33
+ ret=33
+ '[' 33 '!=' 0 ']'
+ echo 'elemental upgrade failed with return code: 33'
+ cat /host/usr/local/upgrade_tmp/elemental-upgrade-20230913120242.log
INFO[2023-09-13T12:02:42Z] Starting elemental version 0.3.1
INFO[2023-09-13T12:02:42Z] reading configuration form '/tmp/tmp.N6rn4F6mKM'
ERRO[2023-09-13T12:02:42Z] Invalid upgrade command setup undefined state partition
+ exit 33
+ clean_up_tmp_files
+ '[' -n /host/tmp/tmp.n2JZm1dxDv ']'
+ echo 'Try to unmount /host/tmp/tmp.n2JZm1dxDv...'
+ umount /host/tmp/tmp.n2JZm1dxDv
Try to unmount /host/tmp/tmp.n2JZm1dxDv...
+ '[' -n /host/usr/bin/elemental ']'
+ echo 'Try to unmount /host/usr/bin/elemental...'
+ umount /host/usr/bin/elemental
Try to unmount /host/usr/bin/elemental...
Clean up tmp files...
+ echo 'Clean up tmp files...'
+ '[' -n '' ']'
+ '[' -n /host/usr/local/upgrade_tmp/tmp.OoKOTjupTE ']'
+ echo 'Try to remove /host/usr/local/upgrade_tmp/tmp.OoKOTjupTE...'
+ rm -vf /host/usr/local/upgrade_tmp/tmp.OoKOTjupTE
Try to remove /host/usr/local/upgrade_tmp/tmp.OoKOTjupTE...
removed '/host/usr/local/upgrade_tmp/tmp.OoKOTjupTE'
[Wed Sep 13 12:02:42 UTC 2023] Running "upgrade_node.sh post-drain" errors, will retry after 10 minutes (6 retries)...

Seems like there's a problem when running the elemental command.

Also, could you try the command lsblk -f on the first node?
And post the result here. Thanks!

Also, could you try the command lsblk -f on the first node? And post the result here. Thanks!

NAME   FSTYPE FSVER LABEL           UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0  ext2   1.0   COS_ACTIVE      e4b427ea-e63e-4338-860d-45ae04ae3103    1.5G    44% /
sda    ext4   1.0   HARV_LH_DEFAULT 41cc7d2a-b2db-4f19-9c59-3f1580d6ef5f    1.6T     0% /var/lib/harvester/defaultdisk
sdb    ext4   1.0                   5211eca8-58f6-47b1-8841-61dd03198e06    1.6T     0% /var/lib/harvester/extra-disks/0334af669935687f11987a42483d511d
sdc    ext4   1.0                   8778694f-ad8d-492f-923e-fb6db69491d0    1.6T     0% /var/lib/harvester/extra-disks/c83455052794a782c6b407555ee176c0
sdd
├─sdd1
├─sdd2 ext4   1.0   COS_OEM         bc450f48-9f7d-4320-9fba-239d48c77b5f     52M     0% /oem
├─sdd3 ext4   1.0   COS_STATE       be760e78-f20b-4539-bf3f-01fcd5d799df   11.3G    18% /run/initramfs/cos-state
├─sdd4 ext4   1.0   COS_RECOVERY    d8b7bb54-8329-434d-b4ac-46a9dba1faf8
└─sdd5 ext4   1.0   COS_PERSISTENT  ab10ce23-7599-4e75-b1dc-0d73a381c2f9   33.7G    59% /var/lib/longhorn
                                                                                /var/crash
                                                                                /var/lib/cni
                                                                                /var/lib/wicked
                                                                                /var/lib/kubelet
                                                                                /var/lib/rancher
                                                                                /var/log
                                                                                /usr/libexec
                                                                                /root
                                                                                /opt
                                                                                /home
                                                                                /etc/pki/trust/anchors
                                                                                /etc/cni
                                                                                /etc/iscsi
                                                                                /etc/ssh
                                                                                /etc/rancher
                                                                                /etc/systemd
                                                                                /usr/local
sde    ext4   1.0                   48b3a081-7f51-4da9-af14-1049d3198099    1.6T     0% /var/lib/harvester/extra-disks/a581c1e14a1b5125ac156cff3196e7da

/usr/local/upgrade_tmp/elemental-upgrade-20230913120242.log

pre-harvester01-01:~ # cat /usr/local/upgrade_tmp/elemental-upgrade-20230913120242.log
INFO[2023-09-13T12:02:42Z] Starting elemental version 0.3.1
INFO[2023-09-13T12:02:42Z] reading configuration form '/tmp/tmp.N6rn4F6mKM'
ERRO[2023-09-13T12:02:42Z] Invalid upgrade command setup undefined state partition

@egrist There should be a button to Download log in the upgrade dialog, can you attach or send the log to us? Thanks.

I clicked the 'Download Log', but it's been 15mins, the button is still grayed out after clicking it. Will send it when I get it downloaded.

something missed when calling elemental-toolkit ? https://github.com/rancher/elemental-toolkit.git
direct: error code
ERRO[2023-09-13T12:02:42Z] Invalid upgrade command setup undefined state partition
elemental upgrade failed with return code: 33
+ ret=33

			spec, err := config.ReadUpgradeSpec(cfg, cmd.Flags()) //========================
			if err != nil {
				cfg.Logger.Errorf("Invalid upgrade command setup %v", err) //========================
				return elementalError.NewFromError(err, elementalError.ReadingSpecConfig) //========================
			}

func ReadUpgradeSpec(r *v1.RunConfig, flags *pflag.FlagSet) (*v1.UpgradeSpec, error) {
	upgrade, err := config.NewUpgradeSpec(r.Config)
	if err != nil {
		return nil, fmt.Errorf("failed initializing upgrade spec: %v", err)
	}
	vp := viper.Sub("upgrade")
	if vp == nil {
		vp = viper.New()
	}
	// Bind upgrade cmd flags
	bindGivenFlags(vp, flags)
	// Bind upgrade env vars
	viperReadEnv(vp, "UPGRADE", constants.GetUpgradeKeyEnvMap())

	err = vp.Unmarshal(upgrade, setDecoder, decodeHook)
	if err != nil {
		r.Logger.Warnf("error unmarshalling UpgradeSpec: %s", err)
	}
	err = upgrade.Sanitize()  //========================
	r.Logger.Debugf("Loaded upgrade UpgradeSpec: %s", litter.Sdump(upgrade))
	return upgrade, err
}

// Error reading the config for the command
const ReadingSpecConfig = 33 //========================

// Sanitize checks the consistency of the struct, returns error
// if unsolvable inconsistencies are found
func (i *InstallSpec) Sanitize() error {
	if i.Active.Source.IsEmpty() && i.Iso == "" {
		return fmt.Errorf("undefined system source to install")
	}
	if i.Partitions.State == nil || i.Partitions.State.MountPoint == "" {   //========================
		return fmt.Errorf("undefined state partition")
	}

cc @bk201
  # adjust new active.img size
  cat > $tmp_elemental_config_dir/config.yaml <<EOF
upgrade:
  system:
    size: 3072
EOF

  mount --bind $new_elemental_cli $target_elemental_cli
  chroot $HOST_DIR elemental upgrade \
    --logfile "$elemental_upgrade_log" \
    --directory ${tmp_rootfs_mount#"$HOST_DIR"} \
    --config-dir ${tmp_elemental_config_dir#"$HOST_DIR"} || ret=$?
  if [ "$ret" != 0 ]; then

@egrist please get such info from your first node  01-01
~ #  (sudo -i) cat /run/initramfs/cos-state/state.yaml
# Autogenerated file by elemental client, do not edit

date: "2023-09-11T09:10:16Z"
oem:
    label: COS_OEM
persistent:
    label: COS_PERSISTENT
recovery:
    label: COS_RECOVERY
    recovery:
        source: dir:///run/rootfsbase
        label: COS_SYSTEM
        fs: ext2
state:
    label: COS_STATE
    active:
        source: dir:///run/rootfsbase
        label: COS_ACTIVE
        fs: ext2
    passive:
        source: dir:///run/rootfsbase
        label: COS_PASSIVE
        fs: ext2

@egrist please get such info from your first node 01-01
~ # (sudo -i) cat /run/initramfs/cos-state/state.yaml
# Autogenerated file by elemental client, do not edit

date: "2023-09-11T09:10:16Z"
oem:
    label: COS_OEM
persistent:
    label: COS_PERSISTENT
recovery:
    label: COS_RECOVERY
    recovery:
        source: dir:///run/rootfsbase
        label: COS_SYSTEM
        fs: ext2
state:
    label: COS_STATE
    active:
        source: dir:///run/rootfsbase
        label: COS_ACTIVE
        fs: ext2
    passive:
        source: dir:///run/rootfsbase
        label: COS_PASSIVE
        fs: ext2

Here you go:
pre-harvester01-01:~ # cat /run/initramfs/cos-state/state.yaml
# Autogenerated file by elemental client, do not edit

date: "2023-09-13T08:31:42Z"
state:
    active:
        source: dir:///tmp/tmp.01deNrXNEC
        label: COS_ACTIVE
        fs: ext2
    passive: null

A side note, all day yesterday I tried to Download Log from the UI, also 2 colleagues of mine with no success, developer console only shows a GET every now and then and I don't know why it doesn't work, hence I cannot provide it as requested :(

@egrist Is https://github.com/harvester/harvester/issues/4526#issuecomment-1718850386 the lsblk -f output from the first node ? could you take it again, and if anything different, please re-post it, thanks.

@egrist Is #4526 (comment) the lsblk -f output from the first node ? could you take it again, and if anything different, please re-post it, thanks.

Sure thing:
NAME   FSTYPE FSVER LABEL           UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0  ext2   1.0   COS_ACTIVE      e4b427ea-e63e-4338-860d-45ae04ae3103    1.5G    44% /
sda    ext4   1.0   HARV_LH_DEFAULT 41cc7d2a-b2db-4f19-9c59-3f1580d6ef5f    1.6T     0% /var/lib/harvester/defaultdisk
sdb    ext4   1.0                   5211eca8-58f6-47b1-8841-61dd03198e06    1.6T     0% /var/lib/harvester/extra-disks/0334af669935687f11987a42483d511d
sdc    ext4   1.0                   8778694f-ad8d-492f-923e-fb6db69491d0    1.6T     0% /var/lib/harvester/extra-disks/c83455052794a782c6b407555ee176c0
sdd
├─sdd1
├─sdd2 ext4   1.0   COS_OEM         bc450f48-9f7d-4320-9fba-239d48c77b5f     52M     0% /oem
├─sdd3 ext4   1.0   COS_STATE       be760e78-f20b-4539-bf3f-01fcd5d799df   11.3G    18% /run/initramfs/cos-state
├─sdd4 ext4   1.0   COS_RECOVERY    d8b7bb54-8329-434d-b4ac-46a9dba1faf8
└─sdd5 ext4   1.0   COS_PERSISTENT  ab10ce23-7599-4e75-b1dc-0d73a381c2f9   32.6G    60% /var/lib/longhorn
                                                                                /var/crash
                                                                                /var/lib/cni
                                                                                /var/lib/wicked
                                                                                /var/lib/kubelet
                                                                                /var/lib/rancher
                                                                                /var/log
                                                                                /usr/libexec
                                                                                /root
                                                                                /opt
                                                                                /home
                                                                                /etc/pki/trust/anchors
                                                                                /etc/cni
                                                                                /etc/iscsi
                                                                                /etc/ssh
                                                                                /etc/rancher
                                                                                /etc/systemd
                                                                                /usr/local
sde    ext4   1.0                   48b3a081-7f51-4da9-af14-1049d3198099    1.6T     0% /var/lib/harvester/extra-disks/a581c1e14a1b5125ac156cff3196e7da

I made a diff on the two (the one posted before and this one):
11c11
< └─sdd5 ext4   1.0   COS_PERSISTENT  ab10ce23-7599-4e75-b1dc-0d73a381c2f9   33.7G    59% /var/lib/longhorn
---
> └─sdd5 ext4   1.0   COS_PERSISTENT  ab10ce23-7599-4e75-b1dc-0d73a381c2f9   32.6G    60% /var/lib/longhorn

The next point will be:
cat /run/initramfs/cos-state/state.yaml
cat /host/run/initramfs/cos-state/state.yaml

inside the upgrade job, when it is still existing
jobs/hvst-upgrade-f79bg-post-drain-pre-harvester01-01:

kubectl --context pre-harvester01 -n harvester-system exec --stdin --tty jobs/hvst-upgrade-f79bg-post-drain-pre-harvester01-01 -- /bin/bash
pre-harvester01-01:/ # cat /run/initramfs/cos-state/state.yaml
cat: /run/initramfs/cos-state/state.yaml: No such file or directory

pre-harvester01-01:/ # cat /host/run/initramfs/cos-state/state.yaml
# Autogenerated file by elemental client, do not edit

date: "2023-09-13T08:31:42Z"
state:
    active:
        source: dir:///tmp/tmp.01deNrXNEC
        label: COS_ACTIVE
        fs: ext2
    passive: null

contents of /run/, initramfs seems to be missing
pre-harvester01-01:/ # ls /run/
mount  secrets	zypp.pid

From the source code, this bug looks to be:
(1) note the difference of the `/run/initramfs/cos-state/state.yaml`

normal v120 cluster:

charv41:~ # cat /run/initramfs/cos-state/state.yaml
# Autogenerated file by elemental client, do not edit

date: "2023-09-11T09:10:16Z"
...
state:
    label: COS_STATE  ///////===========================> note this label
    active:
        source: dir:///run/rootfsbase
        label: COS_ACTIVE
        fs: ext2
    passive:
        source: dir:///run/rootfsbase
        label: COS_PASSIVE
        fs: ext2

current upgrade failure, it is missing /// label field

pre-harvester01-01:~ # cat /run/initramfs/cos-state/state.yaml
# Autogenerated file by elemental client, do not edit

date: "2023-09-13T08:31:42Z"
state:
    active:
        source: dir:///tmp/tmp.01deNrXNEC
        label: COS_ACTIVE
        fs: ext2
    passive: null

(2) code process

a. LoadInstallState();  InstallState file content to struct InstallState

due to the `label` field missing, the value is ""

b. // NewElementalPartitionsFromList fills an ElementalPartitions instance from given
// partitions list. First tries to match partitions by partition label, if not,
// it tries to match partitions by filesystem label
func NewElementalPartitionsFromList(pl PartitionList, state *InstallState) ElementalPartitions {
	ep := ElementalPartitions{}

	lm := map[string]string{
		constants.EfiPartName:        constants.EfiLabel,  COS_GRUB
		constants.OEMPartName:        constants.OEMLabel,  COS_OEM
		constants.RecoveryPartName:   constants.RecoveryLabel, COS_RECOVERY
		constants.StatePartName:      constants.StateLabel,  state: COS_STATE
		constants.PersistentPartName: constants.PersistentLabel, COS_PERSISTENT
	}
	if state != nil {
		for k := range lm {
			if state.Partitions[k] != nil {
				lm[k] = state.Partitions[k].FSLabel  //====> the original value of lm["state"] COS_STATE is changed to empty ""
			}
		}
	}

	ep.BIOS = pl.GetByName(constants.BiosPartName)
	ep.EFI = pl.GetByNameOrLabel(constants.EfiPartName, lm[constants.EfiPartName])
	ep.OEM = pl.GetByNameOrLabel(constants.OEMPartName, lm[constants.OEMPartName])
	ep.Recovery = pl.GetByNameOrLabel(constants.RecoveryPartName, lm[constants.RecoveryPartName])
	ep.State = pl.GetByNameOrLabel(constants.StatePartName, lm[constants.StatePartName])   /// state, "", here we get  nil point
	ep.Persistent = pl.GetByNameOrLabel(constants.PersistentPartName, lm[constants.PersistentPartName])

	return ep
}

c. the Sanitize complains the ep, as it is nil

// Sanitize checks the consistency of the struct, returns error
// if unsolvable inconsistencies are found
func (u *UpgradeSpec) Sanitize() error {

		if u.Partitions.State == nil || u.Partitions.State.MountPoint == "" {
			return fmt.Errorf("undefined state partition")
		}
...

d. the struct definition

type UpgradeSpec struct {
	RecoveryUpgrade bool   `yaml:"recovery,omitempty" mapstructure:"recovery"`
	Active          Image  `yaml:"system,omitempty" mapstructure:"system"`
	Recovery        Image  `yaml:"recovery-system,omitempty" mapstructure:"recovery-system"`
	GrubDefEntry    string `yaml:"grub-entry-name,omitempty" mapstructure:"grub-entry-name"`
	Passive         Image
	Partitions      ElementalPartitions
	State           *InstallState
}

// Sanitize checks the consistency of the struct, returns error
// if unsolvable inconsistencies are found
func (u *UpgradeSpec) Sanitize() error {
	if u.RecoveryUpgrade {
		if u.Partitions.Recovery == nil || u.Partitions.Recovery.MountPoint == "" {
			return fmt.Errorf("undefined recovery partition")
		}
		if u.Recovery.Source.IsEmpty() {
			return fmt.Errorf("undefined upgrade source")
		}
	} else {
		if u.Partitions.State == nil || u.Partitions.State.MountPoint == "" {
			return fmt.Errorf("undefined state partition")
		}
		if u.Active.Source.IsEmpty() {
			return fmt.Errorf("undefined upgrade source")
		}
	}

d.
// InstallState tracks the installation data of the whole system
type InstallState struct {
	Date       string                     `yaml:"date,omitempty"`
	Partitions map[string]*PartitionState `yaml:",omitempty,inline"`
}

// PartState tracks installation data of a partition
type PartitionState struct {
	FSLabel string                 `yaml:"label,omitempty"`
	Images  map[string]*ImageState `yaml:",omitempty,inline"`
}

// ImageState represents data of a deployed image
type ImageState struct {
	Source         *ImageSource `yaml:"source,omitempty"`
	SourceMetadata interface{}  `yaml:"source-metadata,omitempty"`
	Label          string       `yaml:"label,omitempty"`
	FS             string       `yaml:"fs,omitempty"`
}

@egrist : please try to edit this yaml, add label: COS_STATE under state (same level as active), and kill the current pending POD
pre-harvester01-01:~ # cat /run/initramfs/cos-state/state.yaml
# Autogenerated file by elemental client, do not edit

date: "2023-09-13T08:31:42Z"
state:
    label: COS_STATE /// ====> add to here
    active:
        source: dir:///tmp/tmp.01deNrXNEC
        label: COS_ACTIVE
        fs: ext2
    passive: null

@egrist
I am calling help from elemental-toolkit.  They will join.

@w13915984028 That file seems stored in a read only mount on the first host where this file exists, what's the correct procedure to edit it?
pre-harvester01-01:~ # mount -l | grep /run/initramfs/cos-state
/dev/sdd3 on /run/initramfs/cos-state type ext4 (ro,relatime) [COS_STATE]

Hi @egrist, you can run sudo mount -o remount,rw /run/initramfs/cos-state to remount the state partition and be able to write, and then sudo mount -o remount,ro /run/initramfs/cos-state when done.

I took the content from one of my Harvester v112 cluster, for further reference:
harv31:~ # cat /run/initramfs/cos-state/state.yaml
# Autogenerated file by elemental client, do not edit

date: "2023-05-19T09:08:15Z"
oem:
    label: COS_OEM
persistent:
    label: COS_PERSISTENT
recovery:
    label: COS_RECOVERY
    recovery:
        source: dir:///run/rootfsbase
        label: COS_SYSTEM
        fs: ext2
state:
    label: COS_STATE
    active:
        source: dir:///run/rootfsbase
        label: COS_ACTIVE
        fs: ext2
    passive:
        source: dir:///run/rootfsbase
        label: COS_PASSIVE
        fs: ext2
harv31:~ #

@egrist : please try to edit this yaml, add label: COS_STATE under state (same level as active), and kill the current pending POD
pre-harvester01-01:~ # cat /run/initramfs/cos-state/state.yaml
# Autogenerated file by elemental client, do not edit

date: "2023-09-13T08:31:42Z"
state:
    label: COS_STATE /// ====> add to here
    active:
        source: dir:///tmp/tmp.01deNrXNEC
        label: COS_ACTIVE
        fs: ext2
    passive: null

So I added 'label: COS_STATE' under 'state' and then deleted the job 'hvst-upgrade-f79bg-post-drain-pre-harvester01-01' (which I though you meant)
Nothing really happens, still stuck in 'Post-draining'.

@egrist Is there a new POD named hvst-upgrade-f79bg-post-drain-pre-harvester01-* created ?

@w13915984028 nope

@w13915984028 nope

kubectl --context pre-harvester01 get pods -A | grep f79bg
harvester-system            hvst-upgrade-f79bg-apply-manifests-7hd57                  0/1     Completed   0               4d20h
harvester-system            hvst-upgrade-f79bg-upgradelog-downloader-d8db6d4-xpl95    1/1     Running     0               4d21h
harvester-system            virt-launcher-upgrade-repo-hvst-upgrade-f79bg-cm6zq       1/1     Running     0               4d21h

@bk201 @starbops
The post-drain job instead of POD was deleted by accident, could we manually re-create the job to retry the post-drain process ?

@bk201 @starbops
The post-drain job instead of POD was deleted by accident, could we manually re-create the job to retry the post-drain process ?

I'd see if we can get the job manifest from the user's support bundle, remove non-relevant fields, and apply it again. It's better to get an SB again to see the current upgrade object and CAPI cluster state.

I'll send a new support bundle

Probably I missed it somewhere in the discussion, but I am wondering about the firmware and the partition table type. Could it be this error raised using MDOS partition types instead of GPT? The expectation from elemental-toolkit is that MDOS partition tables are used in BIOS context only.
The expectation in GPT partition tables is that even if the filesystem label data is lost somehow partitions can still be found by the partition label, which is not customizable and hence it is always an elemental hardcoded value.

@davidcassany thanks. when any information is needed, @egrist will help to get from his server.
Harvester team now tries workaround to continue the stucking upgrade.

Hi @egrist, I regenerate your post-drain yaml file as follows
apiVersion: batch/v1
kind: Job
metadata:
  labels:
    harvesterhci.io/node: pre-harvester01-01
    harvesterhci.io/upgrade: hvst-upgrade-f79bg
    harvesterhci.io/upgradeComponent: node
    harvesterhci.io/upgradeJobType: post-drain
  name: hvst-upgrade-f79bg-post-drain-pre-harvester01-01
  namespace: harvester-system
  ownerReferences:
  - apiVersion: harvesterhci.io/v1beta1
    kind: Upgrade
    name: hvst-upgrade-f79bg
    uid: fb197b26-d8c1-4b42-b7a1-38f02ff6dae8
spec:
  backoffLimit: 6
  completionMode: NonIndexed
  completions: 1
  parallelism: 1
  suspend: false
  template:
    metadata:
      creationTimestamp: "null"
      labels:
        harvesterhci.io/upgrade: hvst-upgrade-f79bg
        harvesterhci.io/upgradeComponent: node
        harvesterhci.io/upgradeJobType: post-drain
        job-name: hvst-upgrade-f79bg-post-drain-pre-harvester01-01
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values:
                - pre-harvester01-01
      containers:
      - args:
        - post-drain
        command:
        - do_upgrade_node.sh
        env:
        - name: HARVESTER_UPGRADE_NAME
          value: hvst-upgrade-f79bg
        - name: HARVESTER_UPGRADE_NODE_NAME
          value: pre-harvester01-01
        - name: HARVESTER_UPGRADE_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        image: rancher/harvester-upgrade:v1.2.0
        imagePullPolicy: IfNotPresent
        name: apply
        resources: {}
        securityContext:
          capabilities:
            add:
            - CAP_SYS_BOOT
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /host
          name: host-root
      dnsPolicy: ClusterFirstWithHostNet
      hostIPC: true
      hostNetwork: true
      hostPID: true
      restartPolicy: Never
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: harvester
      serviceAccountName: harvester
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        key: node.kubernetes.io/unschedulable
        operator: Exists
      - effect: NoExecute
        key: node-role.kubernetes.io/control-plane
        operator: Exists
      - effect: NoSchedule
        key: kubevirt.io/drain
        operator: Exists
      - key: CriticalAddonsOnly
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
      - effect: NoSchedule
        key: kubernetes.io/arch
        operator: Equal
        value: amd64
      - effect: NoSchedule
        key: kubernetes.io/arch
        operator: Equal
        value: arm64
      - effect: NoSchedule
        key: kubernetes.io/arch
        operator: Equal
        value: arm
      volumes:
      - hostPath:
          path: /
          type: Directory
        name: host-root
  ttlSecondsAfterFinished: 604800

Please use kubectl apply  to re-apply it.
Then please help to add --debug to the elemental command. (Please refer to the following steps)
You could find the related pod from the following command
$ kubectl get pod --selector=harvesterhci.io/upgradeJobType=post-drain -n harvester-system

# you will see your post-drain job, should be named `hvst-upgrade-f79bg-post-drain-pre-harvester01-01`

Then go into this pod
$ kubectl exec -it pods/<the above name> -n harvester-system -- /bin/bash

Then try to modify the bash script /usr/local/bin/upgrade_node.sh like following:
chroot $HOST_DIR elemental upgrade \   <-- this line replace to following

chroot $HOST_DIR elemental upgrade --debug \

This job will repeatedly execute for 10 minutes if failure.
So keep waiting for the next run.
Thanks!
cc @bk201

@Vicente-Cheng Since I edited state.yaml on the first node and added the job, it completed successfully quite fast so I didn't get the time to edit the upgrade_node.sh script. However it continued to node number 2 which is very similar to the other nodes, the same problem occured. On node number 2 I added the --debug flag in the upgrade_node.sh, and then fetched the log which I'll post below. After it failed, I edited the state.yaml, deleted the pod (not the actual job :P) so that a new pod started and completed successfully.
Log with chroot $HOST_DIR elemental upgrade --debug:
+++ dirname /usr/local/bin/upgrade_node.sh
++ cd /usr/local/bin
++ pwd
+ SCRIPT_DIR=/usr/local/bin
+ source /usr/local/bin/lib.sh
++ UPGRADE_NAMESPACE=harvester-system
++ UPGRADE_REPO_URL=http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso
++ UPGRADE_REPO_VM_NAME=upgrade-repo-hvst-upgrade-f79bg
++ UPGRADE_REPO_RELEASE_FILE=http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/harvester-release.yaml
++ UPGRADE_REPO_SQUASHFS_IMAGE=http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/rootfs.squashfs
++ UPGRADE_REPO_BUNDLE_ROOT=http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/bundle
++ UPGRADE_REPO_BUNDLE_METADATA=http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/bundle/metadata.yaml
++ CACHED_BUNDLE_METADATA=
++ HOST_DIR=/host
+ UPGRADE_TMP_DIR=/host/usr/local/upgrade_tmp
+ mkdir -p /host/usr/local/upgrade_tmp
+ case $1 in
+ command_post_drain
+ wait_repo
++ get_repo_vm_status
++ kubectl get virtualmachines.kubevirt.io upgrade-repo-hvst-upgrade-f79bg -n harvester-system '-o=jsonpath={.status.printableStatus}'
+ [[ Running == \R\u\n\n\i\n\g ]]
+ curl -sfL http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/harvester-release.yaml
harvester: v1.2.0
harvesterChart: 1.2.0
os: Harvester v1.2.0
kubernetes: v1.25.9+rke2r1
rancher: v2.7.5
monitoringChart: 102.0.0+up40.1.2
loggingChart: 102.0.0+up3.17.10
kubevirt: 0.54.0-1504<IP_ADDRESS>
minUpgradableVersion: 'v1.1.2'
rancherDependencies:
  fleet:
    chart: 102.1.0+up0.7.0
    app: 0.7.0
  fleet-crd:
    chart: 102.1.0+up0.7.0
    app: 0.7.0
  rancher-webhook:
    chart: 2.0.5+up0.3.5
    app: 0.3.5
+ detect_repo
++ mktemp --suffix=.yaml
+ release_file=/tmp/tmp.NYKakgbfbj.yaml
+ download_file http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/harvester-release.yaml /tmp/tmp.NYKakgbfbj.yaml
+ local url=http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/harvester-release.yaml
+ local output=/tmp/tmp.NYKakgbfbj.yaml
+ echo 'Downloading the file from "http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/harvester-release.yaml" to "/tmp/tmp.NYKakgbfbj.yaml"...'
+ local i=0
+ [[ 0 -lt 100 ]]
Downloading the file from "http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/harvester-release.yaml" to "/tmp/tmp.NYKakgbfbj.yaml"...
+ curl -sSfL http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/harvester-release.yaml -o /tmp/tmp.NYKakgbfbj.yaml
+ break
++ yq -e e .harvester /tmp/tmp.NYKakgbfbj.yaml
+ REPO_HARVESTER_VERSION=v1.2.0
++ yq -e e .harvesterChart /tmp/tmp.NYKakgbfbj.yaml
+ REPO_HARVESTER_CHART_VERSION=1.2.0
++ yq -e e .os /tmp/tmp.NYKakgbfbj.yaml
+ REPO_OS_PRETTY_NAME='Harvester v1.2.0'
+ REPO_OS_VERSION=v1.2.0
++ yq -e e .kubernetes /tmp/tmp.NYKakgbfbj.yaml
+ REPO_RKE2_VERSION=v1.25.9+rke2r1
++ yq -e e .rancher /tmp/tmp.NYKakgbfbj.yaml
+ REPO_RANCHER_VERSION=v2.7.5
++ yq -e e .monitoringChart /tmp/tmp.NYKakgbfbj.yaml
+ REPO_MONITORING_CHART_VERSION=102.0.0+up40.1.2
++ yq -e e .loggingChart /tmp/tmp.NYKakgbfbj.yaml
+ REPO_LOGGING_CHART_VERSION=102.0.0+up3.17.10
++ yq -e e .rancherDependencies.fleet.chart /tmp/tmp.NYKakgbfbj.yaml
+ REPO_FLEET_CHART_VERSION=102.1.0+up0.7.0
++ yq -e e .rancherDependencies.fleet.app /tmp/tmp.NYKakgbfbj.yaml
+ REPO_FLEET_APP_VERSION=0.7.0
++ yq -e e .rancherDependencies.fleet-crd.chart /tmp/tmp.NYKakgbfbj.yaml
+ REPO_FLEET_CRD_CHART_VERSION=102.1.0+up0.7.0
++ yq -e e .rancherDependencies.fleet-crd.app /tmp/tmp.NYKakgbfbj.yaml
+ REPO_FLEET_CRD_APP_VERSION=0.7.0
++ yq -e e .rancherDependencies.rancher-webhook.chart /tmp/tmp.NYKakgbfbj.yaml
+ REPO_RANCHER_WEBHOOK_CHART_VERSION=2.0.5+up0.3.5
++ yq -e e .rancherDependencies.rancher-webhook.app /tmp/tmp.NYKakgbfbj.yaml
+ REPO_RANCHER_WEBHOOK_APP_VERSION=0.3.5
++ yq -e e .kubevirt /tmp/tmp.NYKakgbfbj.yaml
+ REPO_KUBEVIRT_VERSION=0.54.0-1504<IP_ADDRESS>
++ yq e .minUpgradableVersion /tmp/tmp.NYKakgbfbj.yaml
+ REPO_HARVESTER_MIN_UPGRADABLE_VERSION=v1.1.2
+ '[' -z v1.2.0 ']'
+ '[' -z 1.2.0 ']'
+ '[' -z v1.2.0 ']'
+ '[' -z v1.25.9+rke2r1 ']'
+ '[' -z v2.7.5 ']'
+ '[' -z 102.0.0+up40.1.2 ']'
+ '[' -z 102.0.0+up3.17.10 ']'
+ '[' -z 102.1.0+up0.7.0 ']'
+ '[' -z 0.7.0 ']'
+ '[' -z 102.1.0+up0.7.0 ']'
+ '[' -z 0.7.0 ']'
+ '[' -z 2.0.5+up0.3.5 ']'
+ '[' -z 0.3.5 ']'
+ '[' -z 0.54.0-1504<IP_ADDRESS> ']'
++ mktemp --suffix=.yaml
+ CACHED_BUNDLE_METADATA=/tmp/tmp.2zOqKu5naF.yaml
+ download_file http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/bundle/metadata.yaml /tmp/tmp.2zOqKu5naF.yaml
+ local url=http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/bundle/metadata.yaml
+ local output=/tmp/tmp.2zOqKu5naF.yaml
+ echo 'Downloading the file from "http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/bundle/metadata.yaml" to "/tmp/tmp.2zOqKu5naF.yaml"...'
+ local i=0
Downloading the file from "http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/bundle/metadata.yaml" to "/tmp/tmp.2zOqKu5naF.yaml"...
+ [[ 0 -lt 100 ]]
+ curl -sSfL http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/bundle/metadata.yaml -o /tmp/tmp.2zOqKu5naF.yaml
+ break
+ set_max_pods
+ cat
+ wait_rke2_upgrade
++ echo -n v1.25.9+rke2r1
++ sed 's/-rc[[:digit:]]*//g'
+ REPO_RKE2_VERSION_WITHOUT_RC=v1.25.9+rke2r1
++ get_node_rke2_version
++ kubectl get node pre-harvester01-02 -o yaml
++ yq -e e .status.nodeInfo.kubeletVersion -
+ '[' v1.25.9+rke2r1 = v1.25.9+rke2r1 ']'
+ clean_rke2_archives
+ yq -e -o=json e .images.rke2 /tmp/tmp.2zOqKu5naF.yaml
+ jq -r '.[] | [.list, .archive] | @tsv'
+ IFS='	'
+ read -r list archive
++ basename /harvester/images/rke2-images.linux-amd64-v1.25.9-rke2r1.tar.zst
+ rm -f /host/var/lib/rancher/rke2/agent/images/rke2-images.linux-amd64-v1.25.9-rke2r1.tar.zst
++ basename /harvester/images-lists/rke2-images.linux-amd64-v1.25.9-rke2r1.txt
+ rm -f /host/var/lib/rancher/rke2/agent/images/rke2-images.linux-amd64-v1.25.9-rke2r1.txt
+ IFS='	'
+ read -r list archive
++ basename /harvester/images/rke2-images-multus.linux-amd64-v1.25.9-rke2r1.tar.zst
+ rm -f /host/var/lib/rancher/rke2/agent/images/rke2-images-multus.linux-amd64-v1.25.9-rke2r1.tar.zst
++ basename /harvester/images-lists/rke2-images-multus.linux-amd64-v1.25.9-rke2r1.txt
+ rm -f /host/var/lib/rancher/rke2/agent/images/rke2-images-multus.linux-amd64-v1.25.9-rke2r1.txt
+ IFS='	'
+ read -r list archive
+ kubectl taint node pre-harvester01-02 kubevirt.io/drain-
error: taint "kubevirt.io/drain" not found
+ true
+ convert_nodenetwork_to_vlanconfig
+ '[' -z '' ']'
+ detect_upgrade
++ kubectl get upgrades.harvesterhci.io hvst-upgrade-f79bg -n harvester-system -o yaml
+ upgrade_obj='apiVersion: harvesterhci.io/v1beta1
kind: Upgrade
metadata:
  annotations:
    harvesterhci.io/replica-replenishment-wait-interval: "600"
  creationTimestamp: "2023-09-13T09:39:02Z"
  finalizers:
  - wrangler.cattle.io/harvester-upgrade-controller
  generateName: hvst-upgrade-
  generation: 28
  labels:
    harvesterhci.io/latestUpgrade: "true"
    harvesterhci.io/upgradeState: UpgradingNodes
  name: hvst-upgrade-f79bg
  namespace: harvester-system
  resourceVersion: "37124388"
  uid: fb197b26-d8c1-4b42-b7a1-38f02ff6dae8
spec:
  image: ""
  logEnabled: true
  version: v1.2.0
status:
  conditions:
  - status: Unknown
    type: Completed
  - lastUpdateTime: "2023-09-13T09:39:18Z"
    status: "True"
    type: LogReady
  - lastUpdateTime: "2023-09-13T09:40:54Z"
    status: "True"
    type: ImageReady
  - lastUpdateTime: "2023-09-13T09:42:29Z"
    status: "True"
    type: RepoReady
  - lastUpdateTime: "2023-09-13T10:17:16Z"
    status: "True"
    type: NodesPrepared
  - lastUpdateTime: "2023-09-13T10:22:00Z"
    status: "True"
    type: SystemServicesUpgraded
  - status: Unknown
    type: NodesUpgraded
  imageID: harvester-system/harvester-iso-zwxj6
  nodeStatuses:
    pre-harvester01-01:
      state: Succeeded
    pre-harvester01-02:
      state: Post-draining
    pre-harvester01-03:
      state: Images preloaded
    pre-harvester01-04:
      state: Images preloaded
  previousVersion: v1.1.2
  repoInfo: |
    release:
        harvester: v1.2.0
        harvesterChart: 1.2.0
        os: Harvester v1.2.0
        kubernetes: v1.25.9+rke2r1
        rancher: v2.7.5
        monitoringChart: 102.0.0+up40.1.2
        minUpgradableVersion: v1.1.2
  upgradeLog: hvst-upgrade-f79bg-upgradelog'
++ echo 'apiVersion: harvesterhci.io/v1beta1
kind: Upgrade
metadata:
  annotations:
    harvesterhci.io/replica-replenishment-wait-interval: "600"
  creationTimestamp: "2023-09-13T09:39:02Z"
  finalizers:
  - wrangler.cattle.io/harvester-upgrade-controller
  generateName: hvst-upgrade-
  generation: 28
  labels:
    harvesterhci.io/latestUpgrade: "true"
    harvesterhci.io/upgradeState: UpgradingNodes
  name: hvst-upgrade-f79bg
  namespace: harvester-system
  resourceVersion: "37124388"
  uid: fb197b26-d8c1-4b42-b7a1-38f02ff6dae8
spec:
  image: ""
  logEnabled: true
  version: v1.2.0
status:
  conditions:
  - status: Unknown
    type: Completed
  - lastUpdateTime: "2023-09-13T09:39:18Z"
    status: "True"
    type: LogReady
  - lastUpdateTime: "2023-09-13T09:40:54Z"
    status: "True"
    type: ImageReady
  - lastUpdateTime: "2023-09-13T09:42:29Z"
    status: "True"
    type: RepoReady
  - lastUpdateTime: "2023-09-13T10:17:16Z"
    status: "True"
    type: NodesPrepared
  - lastUpdateTime: "2023-09-13T10:22:00Z"
    status: "True"
    type: SystemServicesUpgraded
  - status: Unknown
    type: NodesUpgraded
  imageID: harvester-system/harvester-iso-zwxj6
  nodeStatuses:
    pre-harvester01-01:
      state: Succeeded
    pre-harvester01-02:
      state: Post-draining
    pre-harvester01-03:
      state: Images preloaded
    pre-harvester01-04:
      state: Images preloaded
  previousVersion: v1.1.2
  repoInfo: |
    release:
        harvester: v1.2.0
        harvesterChart: 1.2.0
        os: Harvester v1.2.0
        kubernetes: v1.25.9+rke2r1
        rancher: v2.7.5
        monitoringChart: 102.0.0+up40.1.2
        minUpgradableVersion: v1.1.2
  upgradeLog: hvst-upgrade-f79bg-upgradelog'
++ yq e .status.previousVersion -
+ UPGRADE_PREVIOUS_VERSION=v1.1.2
+ detect_node_current_harvester_version
+ NODE_CURRENT_HARVESTER_VERSION=
+ local harvester_release_file=/host/etc/harvester-release.yaml
+ '[' -f /host/etc/harvester-release.yaml ']'
++ yq e .harvester /host/etc/harvester-release.yaml
NODE_CURRENT_HARVESTER_VERSION is: v1.1.2
The UPGRADE_PREVIOUS_VERSION is v1.1.2, NODE_CURRENT_HARVESTER_VERSION is v1.1.2, will check nodenetwork upgrade node option
+ NODE_CURRENT_HARVESTER_VERSION=v1.1.2
There is nothing to do in nodenetwork
+ echo 'NODE_CURRENT_HARVESTER_VERSION is: v1.1.2'
+ echo 'The UPGRADE_PREVIOUS_VERSION is v1.1.2, NODE_CURRENT_HARVESTER_VERSION is v1.1.2, will check nodenetwork upgrade node option'
+ test v1.1.2 '!=' v1.0.3
+ test v1.1.2 '!=' v1.0.3
+ echo 'There is nothing to do in nodenetwork'
+ return
+ upgrade_os
+ trap clean_up_tmp_files EXIT
++ source /host/etc/os-release
+++ NAME='SLE Micro'
+++ VERSION=5.3
+++ VERSION_ID=5.3
+++ PRETTY_NAME='Harvester v1.1.2'
+++ ID=sle-micro-rancher
+++ ID_LIKE=suse
+++ ANSI_COLOR='0;32'
+++ CPE_NAME=cpe:/o:suse:sle-micro-rancher:5.3
+++ VARIANT=Harvester
+++ VARIANT_ID=Harvester-20230420
+++ GRUB_ENTRY_NAME='Harvester v1.1.2'
++ echo Harvester v1.1.2
+ CURRENT_OS_VERSION='Harvester v1.1.2'
+ '[' 'Harvester v1.2.0' = 'Harvester v1.1.2' ']'
+ '[' -n '' ']'
++ mktemp -p /host/usr/local/upgrade_tmp
+ tmp_rootfs_squashfs=/host/usr/local/upgrade_tmp/tmp.JbyjvexfYr
+ download_file http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/rootfs.squashfs /host/usr/local/upgrade_tmp/tmp.JbyjvexfYr
+ local url=http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/rootfs.squashfs
+ local output=/host/usr/local/upgrade_tmp/tmp.JbyjvexfYr
+ echo 'Downloading the file from "http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/rootfs.squashfs" to "/host/usr/local/upgrade_tmp/tmp.JbyjvexfYr"...'
+ local i=0
+ [[ 0 -lt 100 ]]
Downloading the file from "http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/rootfs.squashfs" to "/host/usr/local/upgrade_tmp/tmp.JbyjvexfYr"...
+ curl -sSfL http://upgrade-repo-hvst-upgrade-f79bg.harvester-system/harvester-iso/rootfs.squashfs -o /host/usr/local/upgrade_tmp/tmp.JbyjvexfYr
+ break
++ mktemp -d -p /host/tmp
+ tmp_rootfs_mount=/host/tmp/tmp.tl898qOpsa
+ mount /host/usr/local/upgrade_tmp/tmp.JbyjvexfYr /host/tmp/tmp.tl898qOpsa
++ mktemp -d -p /host/tmp
+ tmp_elemental_config_dir=/host/tmp/tmp.J0dMMEmbEv
+ cat
+ new_elemental_cli=/usr/local/bin/elemental
+ target_elemental_cli=/host/usr/bin/elemental
++ date +%Y%m%d%H%M%S
+ elemental_upgrade_log=/usr/local/upgrade_tmp/elemental-upgrade-20230919113309.log
+ local ret=0
+ mount --bind /usr/local/bin/elemental /host/usr/bin/elemental
+ chroot /host elemental upgrade --debug --logfile /usr/local/upgrade_tmp/elemental-upgrade-20230919113309.log --directory /tmp/tmp.tl898qOpsa --config-dir /tmp/tmp.J0dMMEmbEv
Flag --directory has been deprecated, 'directory' is deprecated please use 'system' instead
DEBU[2023-09-19T11:33:09Z] Starting elemental version 0.3.1 on commit bab71d15
INFO[2023-09-19T11:33:09Z] reading configuration form '/tmp/tmp.J0dMMEmbEv'
DEBU[2023-09-19T11:33:09Z] Full config loaded: &v1.RunConfig{
  Reboot: false,
  PowerOff: false,
  EjectCD: false,
  Config: v1.Config{
    Logger: &v1.logrusWrapper{ // p0
      Logger: &logrus.Logger{
        Out: &io.multiWriter{},
        Hooks: logrus.LevelHooks{},
        Formatter: &logrus.TextFormatter{
          ForceColors: true,
          DisableColors: false,
          ForceQuote: false,
          DisableQuote: false,
          EnvironmentOverrideColors: false,
          DisableTimestamp: false,
          FullTimestamp: true,
          TimestampFormat: "",
          DisableSorting: false,
          SortingFunc: ,
          DisableLevelTruncation: false,
          PadLevelText: false,
          QuoteEmptyFields: false,
          FieldMap: logrus.FieldMap(nil),
          CallerPrettyfier: ,
        },
        ReportCaller: false,
        Level: 5,
        ExitFunc: os.Exit,
        BufferPool: nil,
      },
    },
    Fs: &vfs.osfs{}, // p1
    Mounter: &mount.Mounter{},
    Runner: &v1.RealRunner{ // p2
      Logger: p0,
    },
    Syscall: &v1.RealSyscall{},
    CloudInitRunner: &cloudinit.YipCloudInitRunner{},
    ImageExtractor: v1.OCIImageExtractor{},
    Client: &http.Client{},
    Platform: &v1.Platform{
      OS: "linux",
      Arch: "x86_64",
      GolangArch: "amd64",
    },
    Cosign: false,
    Verify: false,
    CosignPubKey: "",
    LocalImage: false,
    Arch: "",
    SquashFsCompressionConfig: []string{
      "-comp",
      "xz",
      "-Xbcj",
      "x86",
    },
    SquashFsNoCompression: false,
    CloudInitPaths: []string{
      "/system/oem",
      "/oem/",
      "/usr/local/cloud-config/",
    },
    Strict: false,
  },
}
DEBU[2023-09-19T11:33:09Z] Loaded upgrade UpgradeSpec: &v1.UpgradeSpec{
  RecoveryUpgrade: false,
  Active: v1.Image{
    File: "",
    Label: "",
    Size: 3072,
    FS: "",
    Source: &v1.ImageSource{},
    MountPoint: "",
    LoopDevice: "",
  },
  Recovery: v1.Image{
    File: "/run/cos/recovery/cOS/transition.img",
    Label: "COS_SYSTEM",
    Size: 0,
    FS: "ext2",
    Source: &v1.ImageSource{},
    MountPoint: "/run/cos/transition",
    LoopDevice: "",
  },
  GrubDefEntry: "",
  Passive: v1.Image{
    File: "",
    Label: "",
    Size: 0,
    FS: "",
    Source: nil,
    MountPoint: "",
    LoopDevice: "",
  },
  Partitions: v1.ElementalPartitions{
    BIOS: nil,
    EFI: nil,
    OEM: &v1.Partition{
      Name: "p.oem",
      FilesystemLabel: "COS_OEM",
      Size: 64,
      FS: "ext4",
      Flags: nil,
      MountPoint: "/oem",
      Path: "/dev/sdd2",
      Disk: "/dev/sdd",
    },
    Recovery: &v1.Partition{
      Name: "p.recovery",
      FilesystemLabel: "COS_RECOVERY",
      Size: 8192,
      FS: "ext4",
      Flags: nil,
      MountPoint: "/run/cos/recovery",
      Path: "/dev/sdd4",
      Disk: "/dev/sdd",
    },
    State: nil,
    Persistent: &v1.Partition{
      Name: "p.persistent",
      FilesystemLabel: "COS_PERSISTENT",
      Size: 97213,
      FS: "ext4",
      Flags: nil,
      MountPoint: "/etc/systemd",
      Path: "/dev/sdd5",
      Disk: "/dev/sdd",
    },
  },
  State: &v1.InstallState{
    Date: "2023-09-13T08:43:10Z",
    Partitions: map[string]*v1.PartitionState{
      "state": &v1.PartitionState{
        FSLabel: "",
        Images: map[string]*v1.ImageState{
          "active": &v1.ImageState{
            Source: &v1.ImageSource{},
            SourceMetadata: nil,
            Label: "COS_ACTIVE",
            FS: "ext2",
          },
          "passive": nil,
        },
      },
    },
  },
}
ERRO[2023-09-19T11:33:09Z] Invalid upgrade command setup undefined state partition
+ ret=33
+ '[' 33 '!=' 0 ']'
elemental upgrade failed with return code: 33
+ echo 'elemental upgrade failed with return code: 33'
+ cat /host/usr/local/upgrade_tmp/elemental-upgrade-20230919113309.log
DEBU[2023-09-19T11:33:09Z] Starting elemental version 0.3.1 on commit bab71d15
INFO[2023-09-19T11:33:09Z] reading configuration form '/tmp/tmp.J0dMMEmbEv'
DEBU[2023-09-19T11:33:09Z] Full config loaded: &v1.RunConfig{
  Reboot: false,
  PowerOff: false,
  EjectCD: false,
  Config: v1.Config{
    Logger: &v1.logrusWrapper{ // p0
      Logger: &logrus.Logger{
        Out: &io.multiWriter{},
        Hooks: logrus.LevelHooks{},
        Formatter: &logrus.TextFormatter{
          ForceColors: true,
          DisableColors: false,
          ForceQuote: false,
          DisableQuote: false,
          EnvironmentOverrideColors: false,
          DisableTimestamp: false,
          FullTimestamp: true,
+ exit 33
+ clean_up_tmp_files
          TimestampFormat: "",
          DisableSorting: false,
          SortingFunc: ,
          DisableLevelTruncation: false,
          PadLevelText: false,
          QuoteEmptyFields: false,
          FieldMap: logrus.FieldMap(nil),
          CallerPrettyfier: ,
        },
        ReportCaller: false,
        Level: 5,
        ExitFunc: os.Exit,
        BufferPool: nil,
+ '[' -n /host/tmp/tmp.tl898qOpsa ']'
      },
+ echo 'Try to unmount /host/tmp/tmp.tl898qOpsa...'
    },
+ umount /host/tmp/tmp.tl898qOpsa
    Fs: &vfs.osfs{}, // p1
    Mounter: &mount.Mounter{},
    Runner: &v1.RealRunner{ // p2
      Logger: p0,
    },
    Syscall: &v1.RealSyscall{},
    CloudInitRunner: &cloudinit.YipCloudInitRunner{},
    ImageExtractor: v1.OCIImageExtractor{},
    Client: &http.Client{},
    Platform: &v1.Platform{
      OS: "linux",
      Arch: "x86_64",
      GolangArch: "amd64",
    },
    Cosign: false,
    Verify: false,
    CosignPubKey: "",
    LocalImage: false,
    Arch: "",
    SquashFsCompressionConfig: []string{
      "-comp",
      "xz",
      "-Xbcj",
      "x86",
    },
    SquashFsNoCompression: false,
    CloudInitPaths: []string{
      "/system/oem",
      "/oem/",
      "/usr/local/cloud-config/",
    },
    Strict: false,
  },
}
DEBU[2023-09-19T11:33:09Z] Loaded upgrade UpgradeSpec: &v1.UpgradeSpec{
  RecoveryUpgrade: false,
  Active: v1.Image{
    File: "",
    Label: "",
    Size: 3072,
    FS: "",
    Source: &v1.ImageSource{},
    MountPoint: "",
    LoopDevice: "",
  },
  Recovery: v1.Image{
    File: "/run/cos/recovery/cOS/transition.img",
    Label: "COS_SYSTEM",
    Size: 0,
    FS: "ext2",
    Source: &v1.ImageSource{},
    MountPoint: "/run/cos/transition",
    LoopDevice: "",
  },
  GrubDefEntry: "",
  Passive: v1.Image{
    File: "",
    Label: "",
    Size: 0,
    FS: "",
    Source: nil,
    MountPoint: "",
    LoopDevice: "",
  },
  Partitions: v1.ElementalPartitions{
    BIOS: nil,
    EFI: nil,
    OEM: &v1.Partition{
      Name: "p.oem",
      FilesystemLabel: "COS_OEM",
      Size: 64,
      FS: "ext4",
      Flags: nil,
      MountPoint: "/oem",
      Path: "/dev/sdd2",
      Disk: "/dev/sdd",
    },
    Recovery: &v1.Partition{
      Name: "p.recovery",
      FilesystemLabel: "COS_RECOVERY",
      Size: 8192,
      FS: "ext4",
      Flags: nil,
      MountPoint: "/run/cos/recovery",
      Path: "/dev/sdd4",
      Disk: "/dev/sdd",
    },
    State: nil,
    Persistent: &v1.Partition{
      Name: "p.persistent",
      FilesystemLabel: "COS_PERSISTENT",
      Size: 97213,
      FS: "ext4",
      Flags: nil,
      MountPoint: "/etc/systemd",
      Path: "/dev/sdd5",
      Disk: "/dev/sdd",
    },
  },
  State: &v1.InstallState{
    Date: "2023-09-13T08:43:10Z",
    Partitions: map[string]*v1.PartitionState{
      "state": &v1.PartitionState{
        FSLabel: "",
        Images: map[string]*v1.ImageState{
          "active": &v1.ImageState{
            Source: &v1.ImageSource{},
            SourceMetadata: nil,
            Label: "COS_ACTIVE",
            FS: "ext2",
          },
          "passive": nil,
        },
      },
    },
  },
}
ERRO[2023-09-19T11:33:09Z] Invalid upgrade command setup undefined state partition
Try to unmount /host/tmp/tmp.tl898qOpsa...
+ '[' -n /host/usr/bin/elemental ']'
+ echo 'Try to unmount /host/usr/bin/elemental...'
Try to unmount /host/usr/bin/elemental...
+ umount /host/usr/bin/elemental
Clean up tmp files...
Try to remove /host/usr/local/upgrade_tmp/tmp.JbyjvexfYr...
+ echo 'Clean up tmp files...'
+ '[' -n '' ']'
+ '[' -n /host/usr/local/upgrade_tmp/tmp.JbyjvexfYr ']'
+ echo 'Try to remove /host/usr/local/upgrade_tmp/tmp.JbyjvexfYr...'
+ rm -vf /host/usr/local/upgrade_tmp/tmp.JbyjvexfYr
removed '/host/usr/local/upgrade_tmp/tmp.JbyjvexfYr'
[Tue Sep 19 11:33:09 UTC 2023] Running "upgrade_node.sh post-drain" errors, will retry after 10 minutes (2 retries)...

Just chiming in that I hit the same issue, and was able to successfully upgrade thanks to the workarounds mentioned here. Thanks all!

summarize the issue and workaround:
(1) Upgrade hang on `post-drain` of certain node, the POD log has "elemental upgrade failed with return code: 33"

the error log looks like:

kubectl --context pre-harvester01 -n harvester-system logs -f jobs/hvst-upgrade-f79bg-post-drain-pre-harvester01-01:

....

ERRO[2023-09-13T12:02:42Z] Invalid upgrade command setup undefined state partition
elemental upgrade failed with return code: 33
+ ret=33
...

(2) Confirm the issue via

pre-harvester01-01:~ # cat /run/initramfs/cos-state/state.yaml
# Autogenerated file by elemental client, do not edit

date: "2023-09-13T08:31:42Z"
state:
    label: COS_STATE /// if there is no such line, then the bug will happen
    active:
        source: dir:///tmp/tmp.01deNrXNEC
        label: COS_ACTIVE
        fs: ext2
    passive: null

(3) Workaround:

a. remount the initramfs to be r/w
sudo mount -o remount,rw /run/initramfs/cos-state to remount the state partition and be able to write,

b. add `label: COS_STATE` to /run/initramfs/cos-state/state.yaml

c.remount the initramfs to be read only
sudo mount -o remount,ro /run/initramfs/cos-state when done.

d. delete the current stuck POD (NOT the JOB), e.g.;  the replaced POD will continue and success
kubectl delete pod -n harvester-system hvst-upgrade-f79bg-post-drain-pre-harvester-*...

Detailed analysis can be found on this comment
TL;DR, the incomplete state.yaml and the older state partition name p.state would cause this problem.

The incomplete state.yaml would cause elemental can not find the state partition with the correct filesystem label COS_STATE.
The old partition name p.state would cause elemental can not find the state partition either.

That only affected the harvester version v1.1.1 and the early version. (This means if you upgrade from the v1.1.1 or early version, you will have a chance to encounter this issue.)
The quick checking you can do before the upgrade.

check the state partition name
# Finding the state partition
harvester-node-0:~ # lsblk  -f |grep COS_STATE
├─vda4 ext4   1.0   COS_STATE       de068d8f-76be-443a-94c1-1bb830e00961    9.6G    29% /run/initramfs/cos-state
harvester-node-0:~ # udevadm info /dev/vda4 |grep PART
...
E: PARTNAME=state
...
E: ID_PART_ENTRY_NAME=state
...

If the partition name is state, we would not encounter this issue. Or you need to check your state.yaml.

Check the state.yaml
# Autogenerated file by elemental client, do not edit

date: "2023-09-23T17:59:24Z"
...
state:
    label: COS_STATE <-- this is used for filesystem label
    active:
        source: dir:///run/rootfsbase
        label: COS_ACTIVE
        fs: ext2
    passive:
        source: dir:///run/rootfsbase
        label: COS_PASSIVE
        fs: ext2
...

If the content of state.yaml is like the above, the elemental command could find the state partition with the correct filesystem label COS_STATE
The elemental command could not find the state partition with the following state.yaml
# Autogenerated file by elemental client, do not edit

date: "2023-09-13T08:31:42Z"
state:
    # we are missing `label` here.
    active:
        source: dir:///tmp/tmp.01deNrXNEC
        label: COS_ACTIVE
        fs: ext2
    passive: null

The simple work around is to delete the state.yaml. If we do not contain any state.yaml, the elemental command will use the default setup. So we could still find the state partition with the default filesystem label COS_STATE.

@Vicente-Cheng Please check if we need this for 1.1.3. Thanks!

@Vicente-Cheng We need to add this and its workaround to known issues

@Vicente-Cheng for testing this I validated that the workaround worked in: v1.1.2 ->  Version: v1.2-b5acb14e-head
Setting up a storage-network I was able to see the bundle issue, implemented the workaround when encountering it, then continued the upgrade.
Do you feel we should go ahead and close this out as the workaround does indeed work when customers encounter the bundle issue with a storage-network?
Artifacts:

Known issue PR: https://github.com/harvester/docs/pull/464

@Vicente-Cheng for testing this I validated that the workaround worked in: v1.1.2 -> Version: v1.2-b5acb14e-head
Setting up a storage-network (900 VLAN) utilizing a combination of openvswitch and custom network to funnel ipxe-vm traffic w/ tagged vlans back out to router {all on port 15 as in the screenshot over dedicated nic on laptop to support ipxe vms} I was able to see the bundle issue, implemented the workaround when encountering it, then continued the upgrade.
Do you feel we should go ahead and close this out as the workaround does indeed work when customers encounter the bundle issue with a storage-network?

Hi @irishgordo,
I thought this issue was related to the incomplete state.yaml and could be verified with the Test plan from https://github.com/harvester/harvester/pull/4566.
Could you elaborate more about how you verify this?
Thanks!

@Vicente-Cheng  thanks for clarifying that, I had missed the test plan on 4566 - I've gone through and started to rework with the test plan.
I am hitting a validation issue though.
Validation Issue:
I was able to conduct validating the state.yml file from v1.1.1 -> v1.1.2, to see how it is shaped.
But encountering an issue when trying to upgrade from v1.1.2 -> master-head (that has fix for state.yaml).
Noticed the mcc-rancher-monitoring-crd bundle was seeing an issue but had tried to patch with workaround
Though other bundles all seem to be in an errored state:
╭─mike at suse-workstation-team-harvester in ~/Downloads/test-temp-upgrade
╰─○ kubectl get bundle -A
NAMESPACE     NAME                                          BUNDLEDEPLOYMENTS-READY   STATUS
fleet-local   fleet-agent-local                             1/1
fleet-local   local-managed-system-agent                    1/1
fleet-local   mcc-harvester                                 0/1                       ErrApplied(1) [Cluster fleet-local/local: another operation (install/upgrade/rollback) is in progress]
fleet-local   mcc-harvester-crd                             0/1                       ErrApplied(1) [Cluster fleet-local/local: another operation (install/upgrade/rollback) is in progress]
fleet-local   mcc-local-managed-system-upgrade-controller   1/1
fleet-local   mcc-rancher-logging                           1/1
fleet-local   mcc-rancher-logging-crd                       1/1
fleet-local   mcc-rancher-monitoring                        1/1
fleet-local   mcc-rancher-monitoring-crd                    0/1                       ErrApplied(1) [Cluster fleet-local/local: another operation (install/upgrade/rollback) is in progress]

fleet-agent-pod.log
fleet-controller-log.log
helm-list-all-namespaces.log
kubectl-deployment-grep-fleet.log
rancher.log
rancher-webhook.log
supportbundle_c090f030-6095-4ef0-906b-94364f022113_2023-10-04T19-12-35Z.zip
Would this be something that a Helm Rollback would work to fix?

I'll re-test :smile:  and implement this helm rollback workaround:
https://github.com/harvester/harvester/issues/3616#issuecomment-1489892688

@Vicente-Cheng revalidation passes from:
v1.1.1 -> v1.1.2 -> v1.2.1-rc1 :smile:

This looks good, I'll go ahead and close this out :smile:


Clean up scripts and move into folder

Maybe we should have something that can find those fetchGit's and update them automatically.

You mean across repos? If we agree on a file name, like this always being in a file called closure.nix or whatever, this wouldn't be hard at all.

You can create tags with some github api, even.

You're the jq warrior around here. All yours.


Plot world map

not ready for merging yet!! just so you can see the code @michaelweinold

So both plotting files are geopandas-based? No plotly implementation yet?

Please mark relevant comments as 'resolved' - so that I know they are resolved 😇


remove codeium related code

PR #328 introduced the capability to connect to codeium. At that time, the Codeium chrome extension didn't work on codepod.io out-of-box.
As of today, the chrome extension seems to work fine with a whitelist config (Xinyi's PR https://github.com/Exafunction/codeium-chrome/pull/5). Therefore, PR #328 is deprecated and we are removing related code in this PR.
To use the chrome extension on CodePod.io, add these to the whitelist:
https:\/\/app\.codepod\.io\/.*

For local development, add:
http:\/\/localhost:3000\/.*

Note: DB schema changed. Need to run pnpm dlx prisma migrate dev.


Improve unit testing by adding a framework and including code coverage

The current version of fiveam requires ASDF>3.1 and therefore, does not work on CLISP (at least out of the box). fiveam leading to incompatibility is probably not worth it given the smallish size of the project.

Oh that is a bummer... Well using some small macro's already improves the structure enough I think, and the outuput is also a bit nicer.


make serve timeout and watch timeout in client handler tunable

for #1551

@yichengq
I think we should:

make a default timeout for normal key request -> 30 second at most.
no default timeout for long pooling watching

I do not want it to be configurable via command line.
I have talked about this with @barakmich. We decided to make cluster wide configuration (heartbeat, election + this) to be configurable via HTTP API.
/cc @philips

@xiang90 No default timeout for watching may lead to fds leaking: https://github.com/coreos/etcd/commit/084dcb5596d52a3a1386d59acced24286137e385
I don't know the actual reasons for fds leaking though.
/cc @philips

to make it configurable, we need to add cluster-wide configuration (http endpoint) first.
i will update timeout value in another PR.


version update required

This doesnot work with latest version, have tried it on both 6.2.4 and 6.3.1 no luck.  Think so version upgrade is required..

This versions are now available. Great :)


[Improvement]:Introduce a gradually increasing optimizing retry interval when failed to optimize

Search before asking

[X] I have searched in the issues and found no similar issues.

What would you like to be improved?
#1103 introduced a retry mechanism to prevent optimizing from being blocked.
However, if the exception cannot be automatically recovered after retrying, it will continue to retry, resulting in wasted resources.
How should we improve?
We should introduce a gradually increasing optimizing retry interval when failed to optimize, so if the exception cannot be automatically recovered after retrying, the occupied resources will gradually decrease.
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Subtasks
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

It only affect 0.4.x


Selector Separator Newline

I want to disable, that selectors get their own lines. I tried it with this configuration:
"beautify.options": {
        "css": {
            "selector_separator_newline": false
        }
    }

Hi,
the correct extension configuration is:
"beautify.options": {         "newline_between_rules": false     }
for more options visit js-beautify

No, @mike7515. This question is asking about selector_separator_newline, not newline_between_rules.


IONIQ 5 Charging State is not Identified

Personal experience shows IONIQ 5 is not identifying when charging, reports in Discord suggest same exists for EV6.
Posting this to track this as an issue and open discussion for potential remedy.
Quick triage of the issue myself using data from https://github.com/Esprit1st/Hyundai-Ioniq-5-Torque-Pro-PIDs suggests that there are atleast 3 data points for charging:
000_HV_Charging,Charging,0x220101,{j:7},0,1,,7E4
000_Normal Charge Port,AC Plug,0x220101,{j:5},0,1,,7E4
000_Rapid Charge Port,CCS Plug,0x220101,{j:6},0,1,,7E4

My limited CAN hacking might be to blame here, but this data, suggests that bits 5 and 6 are used for identifying AC (5) and DC (6) Charging.
Referring to the following code snippet
https://github.com/nickn17/evDash/blob/0f283ccd2a4504c21a9e30930ed9786360d909e9/src/CarHyundaiIoniq5.cpp#L415-L424
It appears that this may be correctly identified, but it is polling 220105 instead of 220101.
Unfortunately my previous attempts to compile this code have failed, limiting my testing ability, so presently I can only post issue for review.

Fixed. Pls test. I don't have ioniq5
https://github.com/nickn17/evDash/commit/2597f7b61d7972ac539b0895555207682b9de717

Can confirm 85bf56f92fee92fb69185cec0235f2a8c9dd954d resolved this :D


notebook example by Raymond Dunn

Notebook example for headless closed-loop acquisition. Also included a short python module I wrote for peak finding, used in the example. Edited applications.rst to standardize on "pycro-manager" based on base repo readme.

Cool! Looks like the notebook isn't correctly compiling into docs though. I'm guessing there's somewhere in the notebook that where the name of a specific kernel on your local system is specified, and the automatic build system doesn't have access to it. Maybe theres some way to remove that reference?

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/nbsphinx.py", line 1021, in parse
    rststring, resources = exporter.from_notebook_node(nb, resources)
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/nbsphinx.py", line 832, in from_notebook_node
    nb, resources = pp.preprocess(nb, resources)
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/nbconvert/preprocessors/execute.py", line 79, in preprocess
    self.execute()
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/nbclient/util.py", line 74, in wrapped
    return just_run(coro(*args, **kwargs))
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/nbclient/util.py", line 53, in just_run
    return loop.run_until_complete(coro)
  File "/home/docs/.pyenv/versions/3.7.9/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/nbclient/client.py", line 524, in async_execute
    async with self.async_setup_kernel(**kwargs):
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/async_generator/_util.py", line 34, in __aenter__
    return await self._agen.asend(None)
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/nbclient/client.py", line 482, in async_setup_kernel
    await self.async_start_new_kernel(**kwargs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/nbclient/client.py", line 394, in async_start_new_kernel
    await ensure_async(self.km.start_kernel(extra_arguments=self.extra_arguments, **kwargs))
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/nbclient/util.py", line 85, in ensure_async
    result = await obj
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/jupyter_client/manager.py", line 571, in start_kernel
    kernel_cmd, kw = self.pre_start_kernel(**kw)
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/jupyter_client/manager.py", line 262, in pre_start_kernel
    kernel_cmd = self.format_kernel_cmd(extra_arguments=extra_arguments)
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/jupyter_client/manager.py", line 181, in format_kernel_cmd
    cmd = self.kernel_spec.argv + extra_arguments
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/jupyter_client/manager.py", line 87, in kernel_spec
    self._kernel_spec = self.kernel_spec_manager.get_kernel_spec(self.kernel_name)
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/jupyter_client/kernelspec.py", line 235, in get_kernel_spec
    raise NoSuchKernel(kernel_name)
jupyter_client.kernelspec.NoSuchKernel: No such kernel named Python 3.6.12 64-bit ('pycromanager': conda)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/sphinx/cmd/build.py", line 284, in build_main
    app.build(args.force_all, filenames)
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/sphinx/application.py", line 345, in build
    self.builder.build_update()
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 319, in build_update
    len(to_build))
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 332, in build
    updated_docnames = set(self.read())
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 438, in read
    self._read_serial(docnames)
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 460, in _read_serial
    self.read_doc(docname)
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 504, in read_doc
    doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/sphinx/io.py", line 325, in read_doc
    pub.publish()
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/docutils/core.py", line 218, in publish
    self.settings)
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/sphinx/io.py", line 113, in read
    self.parse()
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/docutils/readers/__init__.py", line 77, in parse
    self.parser.parse(self.input, document)
  File "/home/docs/checkouts/readthedocs.org/user_builds/pycro-manager/envs/164/lib/python3.7/site-packages/nbsphinx.py", line 1032, in parse
    str(e))
nbsphinx.NotebookError: NoSuchKernel in closedloop_headless_demo.ipynb:
No such kernel named Python 3.6.12 64-bit ('pycromanager': conda)

Notebook error:
NoSuchKernel in closedloop_headless_demo.ipynb:
No such kernel named Python 3.6.12 64-bit ('pycromanager': conda)

I'm guessing there's somewhere in the notebook that where the name of a specific kernel on your local system is specified, and the automatic build system doesn't have access to it.

good catch I've never really dug into ipynb metadata. removed that ref

It looks like you need to change the heading levels within your notebook. Right now they are all top level, which creates a bunch of extra stuff in the table of contents
Also, I think it might be good to note in the description in applications.rst that you've used this primarily for neuroscience (right?), but that it could be applied to other things as well

Looks great. One last nitpicky thing: could you capitalize the c in "closed-loop" in the title of your notebook so its consistent with other applications?

not picky at all! I forgot to make those last edits - looks like i snuck them in just before your last msg


Missing support for "justify-content: space-between;"

Let me start off with saying that this is fantastic. A polyfill for flexbox is what is missing to make my life complete (as a webdev).
I just wanted to add this here so it doesn't get forgotten: "justify-content: space-between;" would be nice, it does wonders for some layouts.

This has now been added in Flexibility v1! Thank you so much for your kind words. It means a lot when you know this could really help people and so you’re coding and coding and suddenly it’s 3am. So it’s really great to read that someone appreciates what we do.


Prism stops automatically reloading after detecting invalid schema

Describe the bug
Prism's feature to automatically reload the swagger spec stops working after I upload a bad spec.
To Reproduce

Run Prism with dynamic responses enabled using the attached swagger swagger-spec.json.zip...

$ /usr/local/bin/prism mock -d -h <IP_ADDRESS> /swagger/swagger-spec.json &
[6:08:26 PM] › [CLI] …  awaiting  Starting Prism…
[6:08:26 PM] › [CLI] ℹ  info      GET        http://<IP_ADDRESS>:4010/hello/tempore
[6:08:26 PM] › [CLI] ▶  start     Prism is listening on http://<IP_ADDRESS>:4010

Test Prism with curl to verify it is working correctly...

$ curl -H "Authorization: Bearer 123456"  http://localhost:4010/hello/john
[6:12:40 PM] › [HTTP SERVER] get /hello/john ℹ  info      Request received
[6:12:40 PM] ›     [NEGOTIATOR] ℹ  info      Request contains an accept header: */*
[6:12:40 PM] ›     [VALIDATOR] ✔  success   The request passed the validation rules. Looking for the best response
[6:12:40 PM] ›     [NEGOTIATOR] ✔  success   Found a compatible content for */*
[6:12:40 PM] ›     [NEGOTIATOR] ✔  success   Responding with the requested status code 200
{"greeting":"Krista"}/ $

Change line 45 of the swagger spec document from "x-faker": "name.firstName", to "x-faker": "Hello, name.firstName",
Note that Prism automatically reloads the swagger spec

$ [6:21:59 PM] › [HTTP SERVER] ℹ  info      Restarting Prism...
[6:21:59 PM] › [HTTP SERVER] ℹ  info      Loading the updated operations...
[6:21:59 PM] › [CLI] …  awaiting  Starting Prism…
[6:21:59 PM] › [CLI] ℹ  info      GET        http://<IP_ADDRESS>:4010/hello/aut
[6:21:59 PM] › [CLI] ▶  start     Prism is listening on http://<IP_ADDRESS>:4010

Again test Prism with curl to verify it is working correctly. Note the error in the log and the null value in the response JSON...

$ curl -H "Authorization: Bearer 123456"  http://localhost:4010/hello/john
[6:23:30 PM] › [HTTP SERVER] get /hello/john ℹ  info      Request received
[6:23:30 PM] ›     [NEGOTIATOR] ℹ  info      Request contains an accept header: */*
[6:23:30 PM] ›     [VALIDATOR] ✔  success   The request passed the validation rules. Looking for the best response
[6:23:30 PM] ›     [NEGOTIATOR] ✔  success   Found a compatible content for */*
[6:23:30 PM] ›     [NEGOTIATOR] ✔  success   Responding with the requested status code 200
[6:23:30 PM] ›     [VALIDATOR] ✖  error     Violation: response.body.greeting should be string
{"greeting":null}/ $

Change line 45 of the swagger spec document from "x-faker": "Hello, name.firstName", back to its original value of "x-faker": "name.firstName",
Observe Prism does NOT automatically reload the swagger spec
Again test Prism with curl to verify it is working correctly...

$ curl -H "Authorization: Bearer 123456"  http://localhost:4010/hello/john
[6:28:24 PM] › [HTTP SERVER] get /hello/john ℹ  info      Request received
[6:28:24 PM] ›     [NEGOTIATOR] ℹ  info      Request contains an accept header: */*
[6:28:24 PM] ›     [VALIDATOR] ✔  success   The request passed the validation rules. Looking for the best response
[6:28:24 PM] ›     [NEGOTIATOR] ✔  success   Found a compatible content for */*
[6:28:24 PM] ›     [NEGOTIATOR] ✔  success   Responding with the requested status code 200
[6:28:24 PM] ›     [VALIDATOR] ✖  error     Violation: response.body.greeting should be string
{"greeting":null}/ $

Prism is still serving up the invalid spec

Expected behavior
Prism reloads the corrected swagger spec.
Environment (remove any that are not applicable):

Prism version: 4.1.0
OS: Linux fc5ba140ab3e 4.19.76-linuxkit #1 SMP Tue May 26 11:42:35 UTC 2020 x86_64 Linux
Browser: curl 7.67.0 (x86_64-alpine-linux-musl) libcurl/7.67.0 OpenSSL/1.1.1g zlib/1.2.11 nghttp2/1.40.0

Hey @slessard,
I cannot reproduce the error, and I can't find any flaw in the process.
Chokidar reloads the file as soon as it changes, and does not resonate at all about the schema or invalid content.
Here's a video from my computer: https://sendeyo.com/en/f76416d7b2 (pwd: test)
I can see from your text that you're using Linux; the only thing that comes to my mind is that in the case you're on Docker, the notify mechanism might not work and so you should enable the polling by running your container with the CHOKIDAR_USEPOLLING  env variable set to true.
Please provide more details and reopen if necessary!


[alerta] There is no option for "severity"

There's no option to specify the severity of an alert which makes the integration pretty useless. Could you please add this method?

@madshall The severity is mapped from the Kapacitor Alert Level to the Alerta Severity.
As a result specifying the severity explicitly would not allow it to be dynamic based on the data being evaluated. The mappings are as follows:

Kapacitor Alert Level
Alerta Severity

OK
OK

Info
Informational

Warn
Warning

Crit
Critical

The other Severities listed here http://docs.alerta.io/en/latest/api/alert.html#severity-table are not possible to define within Kapacitor. If you specifically needed one of those severities then we can look for a solution.
Example:
stream
 ....
 .alert()
   .info(lambda: "value" > 5)
   .warn(lambda: "value" > 10)
   .crit(lambda: "value" > 15)
   .alerta()

So for a given value the alerta severity would be as follows:

Value
Alert Severity

1
ok

6
informational

13
warning

23
criticial


Improve gostudy.py

Hi,
Changed gostudy.py slightly due to redundancy in the code and large use of if/else statements. This should make it slightly more efficient. Additionally, I apologise for the edits on app.py, I had the Black formatter on and it auto-formatted, and I noticed it just when I made this PR so just ignore app.py (it should be formatted better now though). Just double check gostudy.py

Resolved the conflicts arising from merging PR #67

@emergenitro @juzcallmekaushik anyone able to make the requested changes?

?

I am working on the recode rn.. I will add it into the AR24

Alright, closing this PR as per your comment.


warnings and errors

Hi, I'm sorry to bother you. I'm trying to use OPRF of volepsi in https://github.com/asu-crypto/mPSI,  but I encountered a difficulty that has been bothering me for a long time. Could you please give me some help?
I build volepsi successfully and install it as follows.
python3 build.py --install=/home/user/MyFile/mPSI5/mPSI/libVolepsi
Then, I wrote the following code in the file frontend/CMakeLists.txt in https://github.com/asu-crypto/mPSI.
find_package(volePSI REQUIRED HINTS "../libVolepsi")
target_link_libraries(frontend.exe visa::volePSI)

When I cmake it, I encountered these warnings.
user@user-virtual-machine:~/MyFile/mPSI5/mPSI/build$ cmake ..
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found Boost: /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost (found version "1.64.0") found components: system thread missing components: chrono date_time atomic
Found Boost library at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_system.a;/home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_thread.a
Found Boost library at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib
Found Boost include at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost
Found Boost system library at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_system.a
Found Boost thread library at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_thread.a
frontend is using boost at: /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_system.a;/home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_thread.a
-- VOLEPSI_THIRDPARTY_DIR=/home/user/MyFile/mPSI5/mPSI/libVolepsi
SPARSEHASH_INCLUDE_DIRS=/home/user/MyFile/mPSI5/mPSI/libVolepsi/include

libOTe_options=silentot;silent_vole;circuits;sse;boost;bitpolymul;no_asan;no_pic;sodium;sodium_montgomery
-- SODIUM_INCLUDE_DIRS:  /home/user/MyFile/mPSI5/mPSI/libVolepsi/include
-- SODIUM_LIBRARIES:  /home/user/MyFile/mPSI5/mPSI/libVolepsi/lib/libsodium.a
-- SODIUM_MONTGOMERY:  ON

macoroConfig.cmake : /home/user/MyFile/mPSI5/mPSI/libVolepsi/lib/cmake/macoro/macoroConfig.cmake
CMake Warning (dev) at libVolepsi/lib/cmake/coproto/coprotoFindDeps.cmake:130 (option):
  Policy CMP0077 is not set: option() honors normal variables.  Run "cmake
  --help-policy CMP0077" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

  For compatibility with older versions of CMake, option is clearing the
  normal variable 'Boost_USE_MULTITHREADED'.
Call Stack (most recent call first):
  libVolepsi/lib/cmake/coproto/coprotoFindDeps.cmake:160 (FIND_BOOST)
  libVolepsi/lib/cmake/coproto/coprotoConfig.cmake:15 (include)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsDepHelper.cmake:215 (find_package)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsDepHelper.cmake:225 (FIND_COPROTO)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsConfig.cmake:26 (include)
  libVolepsi/lib/cmake/libOTe/libOTeConfig.cmake:12 (find_package)
  libVolepsi/lib/cmake/volePSI/findDependancies.cmake:104 (find_package)
  libVolepsi/lib/cmake/volePSI/findDependancies.cmake:118 (FIND_LIBOTE)
  libVolepsi/lib/cmake/volePSI/volePSIConfig.cmake:26 (include)
  frontend/CMakeLists.txt:30 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at libVolepsi/lib/cmake/coproto/coprotoFindDeps.cmake:142 (option):
  Policy CMP0077 is not set: option() honors normal variables.  Run "cmake
  --help-policy CMP0077" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

  For compatibility with older versions of CMake, option is clearing the
  normal variable 'Boost_USE_DEBUG_RUNTIME'.
Call Stack (most recent call first):
  libVolepsi/lib/cmake/coproto/coprotoFindDeps.cmake:160 (FIND_BOOST)
  libVolepsi/lib/cmake/coproto/coprotoConfig.cmake:15 (include)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsDepHelper.cmake:215 (find_package)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsDepHelper.cmake:225 (FIND_COPROTO)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsConfig.cmake:26 (include)
  libVolepsi/lib/cmake/libOTe/libOTeConfig.cmake:12 (find_package)
  libVolepsi/lib/cmake/volePSI/findDependancies.cmake:104 (find_package)
  libVolepsi/lib/cmake/volePSI/findDependancies.cmake:118 (FIND_LIBOTE)
  libVolepsi/lib/cmake/volePSI/volePSIConfig.cmake:26 (include)
  frontend/CMakeLists.txt:30 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at libVolepsi/lib/cmake/coproto/coprotoFindDeps.cmake:146 (find_package):
  Policy CMP0144 is not set: find_package uses upper-case <PACKAGENAME>_ROOT
  variables.  Run "cmake --help-policy CMP0144" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  CMake variable BOOST_ROOT is set to:

    /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/

  For compatibility, find_package is ignoring the variable, but code in a
  .cmake module might still use it.
Call Stack (most recent call first):
  libVolepsi/lib/cmake/coproto/coprotoFindDeps.cmake:160 (FIND_BOOST)
  libVolepsi/lib/cmake/coproto/coprotoConfig.cmake:15 (include)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsDepHelper.cmake:215 (find_package)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsDepHelper.cmake:225 (FIND_COPROTO)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsConfig.cmake:26 (include)
  libVolepsi/lib/cmake/libOTe/libOTeConfig.cmake:12 (find_package)
  libVolepsi/lib/cmake/volePSI/findDependancies.cmake:104 (find_package)
  libVolepsi/lib/cmake/volePSI/findDependancies.cmake:118 (FIND_LIBOTE)
  libVolepsi/lib/cmake/volePSI/volePSIConfig.cmake:26 (include)
  frontend/CMakeLists.txt:30 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found Boost: /home/user/MyFile/mPSI5/mPSI/libVolepsi/lib/cmake/Boost-1.77.0/BoostConfig.cmake (found suitable version "1.77.0", minimum required is "1.77.0") found components: system thread regex
--

Boost_LIB: Boost::system;Boost::thread;Boost::regex
-- Boost_INC: /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost

-- LIBDIVIDE_INCLUDE_DIRS:  /home/user/MyFile/mPSI5/mPSI/libVolepsi/include
-- bitpolymulConfig.cmake : /home/user/MyFile/mPSI5/mPSI/libVolepsi/lib/cmake/bitpolymul
volePSI_INCLUDE_DIRS=/home/user/MyFile/mPSI5/mPSI/libVolepsi/include;/home/user/MyFile/mPSI5/mPSI/libVolepsi/include
volePSI_LIBRARIES=/home/user/MyFile/mPSI5/mPSI/libVolepsi/lib/libvolePSI.a
-- Configuring done (1.3s)
CMake Warning (dev) at /snap/cmake/1384/share/cmake-3.29/Modules/FindBoost.cmake:2503 (set_target_properties):
  Policy CMP0028 is not set: Double colon in target name means ALIAS or
  IMPORTED target.  Run "cmake --help-policy CMP0028" for policy details.
  Use the cmake_policy command to set the policy and suppress this warning.

  The link interface of target "Boost::thread" contains:

    Boost::chrono

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.

Call Stack (most recent call first):
  frontend/CMakeLists.txt:25 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done (0.2s)
-- Build files have been written to: /home/user/MyFile/mPSI5/mPSI/build

After that, I make it and encounter the following error. Could you please tell me what I can do to solve this problem or how can I use OPRF of volepsi in https://github.com/asu-crypto/mPSI? Thank you very much.
user@user-virtual-machine:~/MyFile/mPSI5/mPSI/build$ make
...
[100%] Linking CXX executable /home/user/MyFile/mPSI5/mPSI/bin/frontend.exe
/usr/bin/ld: Can't find -lBoost::chrono: no such file or directory
/usr/bin/ld: Can't find -lBoost::date_time: no such file or directory
/usr/bin/ld: Can't find -lBoost::atomic: no such file or directory
collect2: error: ld returned 1 exit status
make[2]: *** [frontend/CMakeFiles/frontend.exe.dir/build.make:211：/home/user/MyFile/mPSI5/mPSI/bin/frontend.exe] error 1
make[1]: *** [CMakeFiles/Makefile2:305：frontend/CMakeFiles/frontend.exe.dir/all] error 2
make: *** [Makefile:91：all] error 2

try increasing your cmake version, currently 2.8. Change it to something like 3.20.

try increasing your cmake version, currently 2.8. Change it to something like 3.20.

Thank you for your reply. But the cmake version I use is 3.29.3.
user@user-virtual-machine:~/MyFile/mPSI5/mPSI/build$ cmake --version
cmake version 3.29.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).

I tried to change the first line of CMakeLists.txt from cmake_minimum_required (VERSION 2.8) to
cmake_minimum_required (VERSION 3.15)
cmake_policy(SET CMP0144 NEW)

but the following error occurred.
user@user-virtual-machine:~/MyFile/mPSI5/mPSI/build$ cmake ..
-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found Boost: /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost (found version "1.64.0") found components: system thread missing components: chrono date_time atomic
Found Boost library at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_system.a;/home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_thread.a
Found Boost library at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib
Found Boost include at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost
Found Boost system library at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_system.a
Found Boost thread library at /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_thread.a
frontend is using boost at: /home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_system.a;/home/user/MyFile/mPSI5/mPSI/thirdparty/linux/boost/stage/lib/libboost_thread.a
-- VOLEPSI_THIRDPARTY_DIR=/home/user/MyFile/mPSI5/mPSI/libVolepsi
SPARSEHASH_INCLUDE_DIRS=/home/user/MyFile/mPSI5/mPSI/libVolepsi/include

libOTe_options=silentot;silent_vole;circuits;sse;boost;bitpolymul;no_asan;no_pic;sodium;sodium_montgomery
-- SODIUM_INCLUDE_DIRS:  /home/user/MyFile/mPSI5/mPSI/libVolepsi/include
-- SODIUM_LIBRARIES:  /home/user/MyFile/mPSI5/mPSI/libVolepsi/lib/libsodium.a
-- SODIUM_MONTGOMERY:  ON

macoroConfig.cmake : /home/user/MyFile/mPSI5/mPSI/libVolepsi/lib/cmake/macoro/macoroConfig.cmake
CMake Error at libVolepsi/lib/cmake/Boost-1.77.0/BoostConfig.cmake:141 (find_package):
  Found package configuration file:

    /home/user/MyFile/mPSI5/mPSI/libVolepsi/lib/cmake/boost_regex-1.77.0/boost_regex-config.cmake

  but it set boost_regex_FOUND to FALSE so package "boost_regex" is
  considered to be NOT FOUND.  Reason given by package:

  No suitable build variant has been found.

  The following variants have been tried and rejected:

  * libboost_regex.so.1.77.0 (shared, Boost_USE_STATIC_LIBS=ON)

  * libboost_regex.a (release runtime, Boost_USE_DEBUG_RUNTIME=TRUE)

Call Stack (most recent call first):
  libVolepsi/lib/cmake/Boost-1.77.0/BoostConfig.cmake:258 (boost_find_component)
  /snap/cmake/1384/share/cmake-3.29/Modules/FindBoost.cmake:594 (find_package)
  libVolepsi/lib/cmake/coproto/coprotoFindDeps.cmake:146 (find_package)
  libVolepsi/lib/cmake/coproto/coprotoFindDeps.cmake:160 (FIND_BOOST)
  libVolepsi/lib/cmake/coproto/coprotoConfig.cmake:15 (include)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsDepHelper.cmake:215 (find_package)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsDepHelper.cmake:225 (FIND_COPROTO)
  libVolepsi/lib/cmake/cryptoTools/cryptoToolsConfig.cmake:26 (include)
  libVolepsi/lib/cmake/libOTe/libOTeConfig.cmake:12 (find_package)
  libVolepsi/lib/cmake/volePSI/findDependancies.cmake:104 (find_package)
  libVolepsi/lib/cmake/volePSI/findDependancies.cmake:118 (FIND_LIBOTE)
  libVolepsi/lib/cmake/volePSI/volePSIConfig.cmake:26 (include)
  frontend/CMakeLists.txt:30 (find_package)

Hmm, so it's hard to say what exactly is the issue. Seems boost is being built without the regex library or at least without the correct settings.
It also seems you are using an older version of the library?
What project are you trying to build? It appears your including volepsi as a sub project?

Hmm, so it's hard to say what exactly is the issue. Seems boost is being built without the regex library or at least without the correct settings.
It also seems you are using an older version of the library?
What project are you trying to build? It appears your including volepsi as a sub project?

Thank you again for your reply.
I want to use OPRF of volepsi in https://github.com/asu-crypto/mPSI. I build volepsi successfully and install it to /home/user/MyFile/mPSI5/mPSI/libVolepsi by python3 build.py --install=/home/user/MyFile/mPSI5/mPSI/libVolepsi. Then I link it to frontend.exe in the file frontend/CMakeLists.txt using the following code. I want to use  OPRF of volepsi in frontend/tpsi.h.
find_package(volePSI REQUIRED HINTS "../libVolepsi")
target_link_libraries(frontend.exe visa::volePSI)

After that, I build the https://github.com/asu-crypto/mPSI and the above error occurs.
https://github.com/asu-crypto/mPSI use boost 1.64, but volepsi use boost 1.77. I'm not sure if the error is related to the boost version. I have no idea how to solve it.

Are you building both versions? Like does mpsi download and build its own boost?

Are you building both versions? Like does mpsi download and build its own boost?

Yes, I'm afraid this is what caused the error. I am considering if I can use only one boost. What can I do?

Correct, you can not mix version. You'd need to upgrade to a common version. For example, make mpsi depend on the same version of libOTe, or maybe simpler, depend on volepsi.

Correct, you can not mix version. You'd need to upgrade to a common version. For example, make mpsi depend on the same version of libOTe, or maybe simpler, depend on volepsi.

Thank you very much for your help. I try that's way and make it 100%.

I build the project by python3 build.py -DVOLE_PSI_ENABLE_BOOST=ON.
I find the following code in many places in volepsi, but I'm not sure about the effect of ‘useSilver’ and ‘useQC’, and how to use it. Can you please tell me more about it?
#ifdef ENABLE_INSECURE_SILVER
        type = cmd.isSet("useSilver") ? oc::MultType::slv5 : type;
    #endif
#ifdef ENABLE_BITPOLYMUL
        type = cmd.isSet("useQC") ? oc::MultType::QuasiCyclic : type;
#endif

I find silver can improve performance in https://github.com/Visa-Research/volepsi/issues/10#issuecomment-1344552585. Then, I try the following tests by -useSilver or -useQC, but the efficiency doesn't seem to change. Why is that?
If I want to improve performance, what can I do about it.
I set the number of trials 5, I'm wondering why the first one took more time than the remaining four?
user@user-virtual-machine:~/MyFile/volepsi$ ./out/build/linux/frontend/frontend -perf -psi -malicious -v -t 5 -nt 1 -nn 15
nt 1 fakeBase 0 n 32768
Label     Time (ms)  diff (ms)
__________________________________
end            68.2     68.182  **********
begin          68.4      0.232
end            90.3     21.888  *******
begin          90.3      0.002
end           108.7     18.398  *******
begin         108.7      0.002
end           127.0     18.288  *******
begin         127.0      0.002
end           145.2     18.160  *******

7734281 3241345
user@user-virtual-machine:~/MyFile/volepsi$ ./out/build/linux/frontend/frontend -perf -psi -malicious -v -t 5 -nt 1 -nn 15 -useSilver
nt 1 fakeBase 0 n 32768
Label     Time (ms)  diff (ms)
__________________________________
end            68.9     68.922  **********
begin          69.2      0.235
end            90.2     21.008  *******
begin          90.2      0.002
end           109.2     19.072  *******
begin         109.2      0.004
end           129.7     20.456  *******
begin         129.7      0.003
end           148.2     18.527  *******

7734281 3241345
user@user-virtual-machine:~/MyFile/volepsi$ ./out/build/linux/frontend/frontend -perf -psi -malicious -v -t 5 -nt 1 -nn 15 -useQC
nt 1 fakeBase 0 n 32768
Label     Time (ms)  diff (ms)
__________________________________
end           108.5    108.527  **********
begin         108.8      0.230
end           151.3     42.541  ********
begin         151.3      0.002
end           190.7     39.370  ********
begin         190.7      0.002
end           229.5     38.862  ********
begin         229.5      0.002
end           268.5     38.932  ********

5373961 2940225

Another thing I want to confirm is whether volePSI uses OKVS in CCS 2022-Blazing Fast PSI from Improved OKVS and Subfield VOLE by default.
Looking forward to your reply.

Silver is an insecure construction. By default, you can't use it even if you set useSilver. You have to compile the library with special settings to allow it to be used.
Not sure about useQC, but that should be slower than the default which is Expand Convolute codes, https://eprint.iacr.org/2023/882
So for all these vole based protocols, you first generate a sparse vector e, and then compress it by a random looking matric G by computing Ge. The question of which G should be used is what silver, QC, Expand Convolute answers. Each specifies a different G.
Slightly more concretely, the choice value in the vole correlation is the Ge value.
I'm away from my computer so cant look at the code right now. But the default version should be the fastest version. I think you should expend maybe a 4x slowdown compared to what's in the blazing fast psi paper. Not actually sure what the slowdown is, need to look into it to be sure. Basically the main cost will be to generate a vole of size 1.3n. So you should look at the paper above. You should expend psi times slightly slower than the time reported to just generate the vole.

Silver is an insecure construction. By default, you can't use it even if you set useSilver. You have to compile the library with special settings to allow it to be used.
Not sure about useQC, but that should be slower than the default which is Expand Convolute codes, https://eprint.iacr.org/2023/882
So for all these vole based protocols, you first generate a sparse vector e, and then compress it by a random looking matric G by computing Ge. The question of which G should be used is what silver, QC, Expand Convolute answers. Each specifies a different G.
Slightly more concretely, the choice value in the vole correlation is the Ge value.
I'm away from my computer so cant look at the code right now. But the default version should be the fastest version. I think you should expend maybe a 4x slowdown compared to what's in the blazing fast psi paper. Not actually sure what the slowdown is, need to look into it to be sure. Basically the main cost will be to generate a vole of size 1.3n. So you should look at the paper above. You should expend psi times slightly slower than the time reported to just generate the vole.

That's very useful, thank you very much for your reply.


Bug Fix: Fix XSS in Magic button and To Table operations

Fixes the XSS bugs as reported in #483 and #484 .

Cheers @j433866


(M/F)ake it green

move all code from conftest.py into our only integration test file - integration/test_update.py, because it's not used anywhere else
xfail the integration test so that we can start requesting green tests in all PR from now on

Hello CentOS CI, wake up!

Oh man! So, next time when I will need to add something to conftest I'd need to create it, AGAIN?!
Just kidding.

Oh man! So, next time when I will need to add something to conftest I'd need to create it, AGAIN?!

hope I'll understand your sense of humor one day

Looks like that tests are failing on my behavioral change to is_valid. Sorry!

np, #99

hope I'll understand your sense of humor one day

I haven't met such person, yet.


Added support for multiquery calls on fql method

multiquery support for fql calls

Looks great, I'll merge.  Thanks guys!


Make firstRange and lastRange mirror macOS Foundation more

When the searched Data is empty, on macOS nil is returned, on Linux an empty Range is returned.
This PR fixes the discrepancy.

This looks fine but could you also add a test case please.

@spevans Done!

@swift-ci test

@Frizlab There are some other issues with the CI at the moment that are being fixed in https://github.com/apple/swift-corelibs-foundation/pull/2994
Once that is merged I will retest your PRs

@swift-ci test linux

@swift-ci test

⛵


Added school: Nonsuch

@VulcanWM First stage of verification passed, official domain is nonsuchschool.org as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision.

@VulcanWM Thank you for your patience while waiting for a review and processing your request. Pull request merged.

i have a question, my school email is only allowed emails from certain organisations and jetbrains doesn't seem to be one of them.
is there another way to verify my email?