rbd: check for clusterid mapping in genVolFromVolumeOptions()

This commit adds capability to genVolFromVolumeOptions() to fetch
mapped clusted-id & mon-ips for mirrored PVC on secondary cluster
which may have different cluster-id.
This is required for NodeStageVolume().
We also don't need to check for mapping during volume create requests,
so it can be disabled by passing a bool checkClusterIDMapping as false.
Signed-off-by: Rakshith R<EMAIL_ADDRESS>

/retest ci/centos/upgrade-tests-cephfs

/retest ci/centos/upgrade-tests-cephfs

Sep  6 11:34:37.762: FAIL: failed to create user cephcsi-cephfs-provisioner with error etcdserver: request timed out
https://jenkins-ceph-csi.apps.ocp.ci.centos.org/blue/rest/organizations/jenkins/pipelines/upgrade-tests-cephfs/runs/3000/nodes/91/log/?start=0

Hope you have already tested mirror functionality to avoid regressions.

If the current changes are fine, I'll do some final tests with different cluster-ids.
@humblec , does the changes look okay from your end?

Hope you have already tested mirror functionality to avoid regressions.

If the current changes are fine, I'll do some final tests with different cluster-ids.
@humblec , does the changes look okay from your end?

Good enough @Rakshith-R , not sure why this has been moved to draft though.

Current canary image with secondary cluster id rook-ceph1
[rakshith@fedora rbd]$ k get configmaps rook-ceph-csi-config -o yaml
apiVersion: v1
data:
  csi-cluster-config-json: '[{"clusterID":"rook-ceph1","monitors":["<IP_ADDRESS>:6789"]}]'

  [rakshith@fedora rbd]$ k get configmaps rook-ceph-csi-mapping-config -o yaml
apiVersion: v1
data:
  csi-mapping-config-json: '[{"ClusterIDMapping":{"rook-ceph":"rook-ceph1"},"RBDPoolIDMapping":[{"1":"1"}]}]'

1157-11ec-867a-0242ac110005 GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-05d02323-5e89-4a50-8893-c4ff5840912f/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"clusterID":"rook-ceph","csi.storage.k8s.io/pv/name":"pvc-05d02323-5e89-4a50-8893-c4ff5840912f","csi.storage.k8s.io/pvc/name":"rbd-pvc","csi.storage.k8s.io/pvc/namespace":"rook-ceph","imageFeatures":"layering","imageFormat":"2","imageName":"csi-vol-3879d00b-1157-11ec-867a-0242ac110005","journalPool":"replicapool","pool":"replicapool","storage.kubernetes.io/csiProvisionerIdentity":"1631180988965-8081-rook-ceph.rbd.csi.ceph.com"},"volume_id":"0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005"}
E0909 11:09:41.456864  112271 rbd_util.go:1089] ID: 24 Req-ID: 0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005 failed getting mons (failed to fetch monitor list using clusterID (rook-ceph): missing configuration for cluster ID "rook-ceph")
E0909 11:09:41.456899  112271 utils.go:185] ID: 24 Req-ID: 0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005 GRPC error: rpc error: code = Internal desc = failed to fetch monitor list using clusterID (rook-ceph): missing configuration for cluster ID "rook-ceph"

With this pr
I0909 11:16:07.743398  140126 utils.go:181] ID: 5 Req-ID: 0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005 GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-05d02323-5e89-4a50-8893-c4ff5840912f/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"clusterID":"rook-ceph","csi.storage.k8s.io/pv/name":"pvc-05d02323-5e89-4a50-8893-c4ff5840912f","csi.storage.k8s.io/pvc/name":"rbd-pvc","csi.storage.k8s.io/pvc/namespace":"rook-ceph","imageFeatures":"layering","imageFormat":"2","imageName":"csi-vol-3879d00b-1157-11ec-867a-0242ac110005","journalPool":"replicapool","pool":"replicapool","storage.kubernetes.io/csiProvisionerIdentity":"1631180988965-8081-rook-ceph.rbd.csi.ceph.com"},"volume_id":"0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005"}
I0909 11:16:07.743576  140126 cluster_mapping.go:157] ID: 5 Req-ID: 0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005 found new clusterID mapping "rook-ceph1" for existing clusterID "rook-ceph"
I0909 11:16:07.743657  140126 rbd_util.go:1191] ID: 5 Req-ID: 0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005 setting disableInUseChecks: false image features: [layering] mounter: rbd
I0909 11:16:07.755573  140126 omap.go:87] ID: 5 Req-ID: 0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005 got omap values: (pool="replicapool", namespace="", name="csi.volume.3879d00b-1157-11ec-867a-0242ac110005"): map[csi.imageid:1429998da44c csi.imagename:csi-vol-3879d00b-1157-11ec-867a-0242ac110005 csi.volname:pvc-05d02323-5e89-4a50-8893-c4ff5840912f csi.volume.owner:rook-ceph]
E0909 11:16:07.755845  140126 util.go:247] kernel 4.19.202 does not support required features
I0909 11:16:07.887983  140126 cephcmds.go:62] ID: 5 Req-ID: 0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005 command succeeded: rbd [device list --format=json --device-type krbd]
I0909 11:16:07.973578  140126 rbd_attach.go:329] ID: 5 Req-ID: 0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005 rbd: map mon <IP_ADDRESS>:6789
I0909 11:16:08.174316  140126 cephcmds.go:62] ID: 5 Req-ID: 0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005 command succeeded: rbd [--id csi-rbd-node -m <IP_ADDRESS>:6789 --keyfile=***stripped*** map replicapool/csi-vol-3879d00b-1157-11ec-867a-0242ac110005 --device-type krbd --options noudev]
I0909 11:16:08.174341  140126 nodeserver.go:400] ID: 5 Req-ID: 0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005 rbd image: 0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005/replicapool was successfully mapped at /dev/rbd0
I0909 11:16:08.174427  140126 mount_linux.go:463] Attempting to determine if disk "/dev/rbd0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/rbd0])
I0909 11:16:08.181456  140126 mount_linux.go:466] Output: ""
I0909 11:16:08.293051  140126 mount_linux.go:463] Attempting to determine if disk "/dev/rbd0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/rbd0])
I0909 11:16:08.299898  140126 mount_linux.go:466] Output: "DEVNAME=/dev/rbd0\nTYPE=ext4\n"
I0909 11:16:08.299921  140126 mount_linux.go:357] Checking for issues with fsck on disk: /dev/rbd0
I0909 11:16:08.339074  140126 mount_linux.go:453] Attempting to mount disk /dev/rbd0 in ext4 format at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-05d02323-5e89-4a50-8893-c4ff5840912f/globalmount/0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005
I0909 11:16:08.339116  140126 mount_linux.go:180] Mounting cmd (mount) with arguments (-t ext4 -o _netdev,defaults /dev/rbd0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-05d02323-5e89-4a50-8893-c4ff5840912f/globalmount/0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005)
I0909 11:16:08.357826  140126 nodeserver.go:334] ID: 5 Req-ID: 0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005 rbd: successfully mounted volume 0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005 to stagingTargetPath /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-05d02323-5e89-4a50-8893-c4ff5840912f/globalmount/0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005
I0909 11:16:08.357932  140126 utils.go:188] ID: 5 Req-ID: 0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005 GRPC response: {}

The important logs : I0909 11:16:07.743576  140126 cluster_mapping.go:157] ID: 5 Req-ID: 0001-0009-rook-ceph-0000000000000001-3879d00b-1157-11ec-867a-0242ac110005 found new clusterID mapping "rook-ceph1" for existing clusterID "rook-ceph"
Works with same cluster id too I0909 11:44:46.151528  140126 cluster_mapping.go:157] ID: 81 Req-ID: 0001-0009-rook-ceph-0000000000000001-1a77bfb2-1162-11ec-bb4a-0242ac110003 found new clusterID mapping "rook-ceph" for existing clusterID "rook-ceph"
With above tests, this pr is completely functional and no regression :smile:
cc @humblec

This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes?

resolved conflicts, PTAL

@mergifyio rebase

/retest ci/centos/mini-e2e-helm/k8s-1.20

/retest ci/centos/mini-e2e-helm/k8s-1.20

https://jenkins-ceph-csi.apps.ocp.ci.centos.org/blue/organizations/jenkins/mini-e2e-helm_k8s-1.20/detail/mini-e2e-helm_k8s-1.20/2438/pipeline/
etcdserver: request timed out

/retest ci/centos/mini-e2e-helm/k8s-1.22

/retest ci/centos/mini-e2e-helm/k8s-1.22

failed due to #2204
https://jenkins-ceph-csi.apps.ocp.ci.centos.org/blue/rest/organizations/jenkins/pipelines/mini-e2e-helm_k8s-1.22/runs/397/nodes/97/log/?start=0

/retest ci/centos/mini-e2e-helm/k8s-1.22

/retest ci/centos/mini-e2e-helm/k8s-1.22

https://jenkins-ceph-csi.apps.ocp.ci.centos.org/blue/organizations/jenkins/mini-e2e-helm_k8s-1.22/detail/mini-e2e-helm_k8s-1.22/406/pipeline
failed due to #2264


3D Projection of phantoms with shapes of elliptical cylinder and cuboid don't have tilting rectangular shapes whatever Euler angles set

Code that randomly generates 3D phantoms including elliptical cylinders only
# Import the necessary libraries.
# ~~~python
import os
import numpy as np
import matplotlib.pyplot as plt
import tifffile
import random
from copy import deepcopy
import numpy as np

from tomophantom import TomoP3D

# Define the path and name of the `.dat` file that will contain the phantoms model description. If the file exists, the new models will be appended.python

path_1 = '/home/mli/tomograms/pycharm_demos/single_density/without_Gaussian_shapes/normal_test/without_noise/1024*1024_with_oversampling=2/Generate_Training_Data/Phantom3DLibrary_For_att_without_Gaussian_3.dat'
path_2 = '/home/mli/tomograms/pycharm_demos/single_density/without_Gaussian_shapes/normal_test/without_noise/1024*1024_with_oversampling=2/Generate_Training_Data/Phantom3DLibrary_For_phase_without_Gaussian_3.dat'

# Generate the models:

number_of_phantoms = 1000

# components = ['gaussian','paraboloid', 'ellipsoid', 'cone', 'cuboid', 'elliptical_cylinder']
# the unused components are not uniform in density.

components = ['elliptical_cylinder']
mu = [0.328]
delta = [481]

for n in range(1, number_of_phantoms + 1):
    num_of_components = random.randint(1, 5)

    objects_1 = []
    objects_2 = []
    component_subset = random.choices(components, k=random.randint(1, 3))
    for i in range(num_of_components):
        obj_1 = random.choice(component_subset)
        obj_2 = deepcopy(obj_1)
        upper_lim_half_width = random.choice([0.5])

        c0 = 0
        m0 = mu[c0]
        d0 = delta[c0]

        if obj_1 == 'elliptical_cylinder' or 'cuboid':

            p_or_n_x = random.choices([-1, 1], k=1)
            p_or_n_y = random.choices([-1, 1], k=1)
            p_or_n_z = random.choices([-1, 1], k=1)

            x0 = round(random.uniform(-1.0*p_or_n_x[0], -0.75*p_or_n_x[0]), 2)
            y0 = round(random.uniform(-1.0*p_or_n_y[0], -0.75*p_or_n_y[0]), 2)
            z0 = round(random.uniform(-1.0*p_or_n_z[0], -0.75*p_or_n_z[0]), 2)
            a = round(random.uniform(0.3, upper_lim_half_width), 3)
            b = round(random.uniform(0.3, upper_lim_half_width), 3)
            c = round(random.uniform(0.3, upper_lim_half_width), 3)
            # alpha = round(random.uniform(-180, 180), 2)
            alpha = 125
            # beta = round(random.uniform(-180, 180), 2)
            beta = 70
            # gamma = round(random.uniform(-180, 180), 2)
            gamma = 50
            print('ellip')

        else:
            x0 = round(random.uniform(-0.5, 0.5), 2)
            y0 = round(random.uniform(-0.5, 0.5), 2)
            z0 = round(random.uniform(-0.5, 0.5), 2)
            a = round(random.uniform(0.01, upper_lim_half_width), 3)
            b = round(random.uniform(0.01, upper_lim_half_width), 3)
            c = round(random.uniform(0.01, upper_lim_half_width), 3)
            alpha = round(random.uniform(-180, 180), 2)
            beta = round(random.uniform(-180, 180), 2)
            gamma = round(random.uniform(-180, 180), 2)

        obj_1 += ' {c0} {x0} {y0} {x0} {a} {b} {c} {alpha} {beta} {gamma}'.format(c0=m0,
                                                                                # round(random.uniform(0.1,1),2),
                                                                                x0=x0,
                                                                                y0=y0,
                                                                                z0=z0,
                                                                                a=a,
                                                                                b=b,
                                                                                c=c,
                                                                                alpha=alpha,
                                                                                beta=beta,
                                                                                gamma=gamma)

        obj_2 += ' {c0} {x0} {y0} {x0} {a} {b} {c} {alpha} {beta} {gamma}'.format(c0=d0,
                                                                                # round(random.uniform(0.1,1),2),
                                                                                x0=x0,
                                                                                y0=y0,
                                                                                z0=z0,
                                                                                a=a,
                                                                                b=b,
                                                                                c=c,
                                                                                alpha=alpha,
                                                                                beta=beta,
                                                                                gamma=gamma)

        objects_1.append(obj_1)
        objects_2.append(obj_2)

    phantom_string_1 = '''#----------------------------------------------------
# random phantom number {num}
Model : {num};
Components : {comp};
TimeSteps : 1;
'''.format(num=n, comp=num_of_components)

    phantom_string_2 = '''#----------------------------------------------------
# random phantom number {num}
Model : {num};
Components : {comp};
TimeSteps : 1;
'''.format(num=n, comp=num_of_components)

    for obj in objects_1:
        phantom_string_1 += 'Object : ' + obj + '\n'

    with open(path_1, 'a') as file:
        file.write(phantom_string_1)

    for obj in objects_2:
        phantom_string_2 += 'Object : ' + obj + '\n'

    with open(path_2, 'a') as file:
        file.write(phantom_string_2)

Code that make projections of 3D randomly generated phantoms
import os
import numpy as np
import matplotlib.pyplot as plt
import tifffile
from sklearn import preprocessing
from tomophantom import TomoP3D

# Define the path and name of the `.dat` file that will contain the phantoms model description. If the file exists, the new models will be appended.python

path_1 = '/home/mli/tomograms/pycharm_demos/single_density/without_Gaussian_shapes/normal_test/without_noise/1024*1024_with_oversampling=2/Generate_Training_Data/Phantom3DLibrary_For_att_without_Gaussian_3.dat'
path_2 = '/home/mli/tomograms/pycharm_demos/single_density/without_Gaussian_shapes/normal_test/without_noise/1024*1024_with_oversampling=2/Generate_Training_Data/Phantom3DLibrary_For_phase_without_Gaussian_3.dat'

# Define the image size.

N_size = 1024

#Horiz_det = int(np.sqrt(2)*N_size) # detector column count (horizontal)
Horiz_det = N_size # detector column count (horizontal)
print("Horiz_det: ", Horiz_det)

Vert_det = N_size # detector row count (vertical) (no reason for it to be > N)
print("Vert_det: ", Vert_det)

# angles_num = int(0.5*np.pi*N_size); # angles number
angles_num = 5
print("angles_num: ", angles_num)

angles = np.linspace(0.0, 179.9, angles_num, dtype='float32') # in degrees
print("angles: ", angles)

# Created files
os.makedirs('../projection_2', exist_ok=True)
os.makedirs('../projection_2/tif_format', exist_ok=True)
os.makedirs('../projection_2/npy_format', exist_ok=True)

# Create the directories to where the files will be saved in tif format. For attenuation
os.makedirs('../projection_2/tif_format/attenuation_projection_in_tif_format', exist_ok=True)

# Create the directories to where the files will be saved in tif format. For phase
os.makedirs('../projection_2/tif_format/phase_projection_in_tif_format', exist_ok=True)

# Create the directories to where the files will be saved in npy format. For attenuation
os.makedirs('../projection_2/npy_format/attenuation_projection_in_npy_format', exist_ok=True)

# Create the directories to where the files will be saved in npy format. For phase
os.makedirs('../projection_2/npy_format/phase_projection_in_npy_format', exist_ok=True)

# Create the directories to where the files contain the stacks of attenuation and phase together and save in .npy format
os.makedirs('../projection_2/npy_format/projection_stack_in_npy_format', exist_ok=True)

# Generate the phantoms from models *a* to *b* from the *Phantom3DLibrary3.dat* library using TomoP3D.ModelSino.
pixel_size = 0.7e-6
scaling_factor = pixel_size * 100

# Projection
N = 1024
# For attenuation
projData3D_analyt_list_For_attenuation = []
for model in range(901, 1001):  # For phantom No.1 ~ 1000
    projData3D_analyt_attenuation = TomoP3D.ModelSino(model, N_size, Horiz_det, Vert_det, angles,
                                                      path_1) * scaling_factor
    print('projection shape: ', projData3D_analyt_attenuation.shape)
    projData3D_analyt_list_For_attenuation.append(projData3D_analyt_attenuation)

# For phase
projData3D_analyt_list_For_phase = []
for model in range(901, 1001):  # For phantom No.1 ~ 99
    projData3D_analyt_phase = TomoP3D.ModelSino(model, N_size, Horiz_det, Vert_det, angles, path_2) * scaling_factor
    projData3D_analyt_list_For_phase.append(projData3D_analyt_phase)

# At this step you may: extract one slice for each projection sinogram, extract one angle
# Save in tiff picure format

# For attenuation
for model in range(1, 101):      # For phantom No.1 ~ 1000
    projected_3D = projData3D_analyt_list_For_attenuation[model-1]
    projected_slice = projected_3D[::, 0, ::]
    tifffile.imsave('../projection_2/tif_format/attenuation_projection_in_tif_format/{:05d}.tif'.format(model+900), projected_slice.astype(np.float32))
    np.save('../projection_2/npy_format/attenuation_projection_in_npy_format/{:05d}.npy'.format(model+900), projected_slice.astype(np.float32))

# For phase
for model in range(1, 101):      # For phantom No.1 ~ 1000
    projected_3D = projData3D_analyt_list_For_phase[model-1]
    projected_slice = projected_3D[::, 0, ::]
    tifffile.imsave('../projection_2/tif_format/phase_projection_in_tif_format/{:05d}.tif'.format(model+900), projected_slice.astype(np.float32))
    np.save('../projection_2/npy_format/phase_projection_in_npy_format/{:05d}.npy'.format(model+900), projected_slice.astype(np.float32))

# At this step you may: extract one slice for each projection sinogram (both attenuation and phase), extract one angle
# Stack them together and save in .npy format

for model in range(1, 101):      # For phantom No.1 ~ 1000
    projected_slice = [0, 0]
    projected_3D_attenuation = projData3D_analyt_list_For_attenuation[model-1]
    projected_3D_phase = projData3D_analyt_list_For_phase[model-1]
    projected_slice[0] = projected_3D_attenuation[::, 0, ::]
    projected_slice[1] = projected_3D_phase[::, 0, ::]
    projected_slice = np.array(projected_slice)
    np.save('../projection_2/npy_format/projection_stack_in_npy_format/{:05d}.npy'.format(model+900), projected_slice.astype(np.float32))

I see what you are trying to do.
The python interface is not very nice, but I guess what you want to achieve I would do with TomoP3D.ObjectSino
Notice that the paramenter objlist is a list with dictionaries describing each object in the phantom along these lines:

obj1 = {}
obj1['C0'] = c0
obj1['x0'] = x0
...
obj2 = {} ...

objlist = [obj1, obj2,]

I also see that the angles phi2 and phi3 will not be used, probably a bug.

I see what you are trying to do.
The python interface is not very nice, but I guess what you want to achieve I would do with TomoP3D.ObjectSino
Notice that the paramenter objlist is a list with dictionaries describing each object in the phantom along these lines:
obj1 = {}
obj1['C0'] = c0
obj1['x0'] = x0
...
obj2 = {} ...

objlist = [obj1, obj2,]

I also see that the angles phi2 and phi3 will not be used, probably a bug.

Thanks for suggestion. I will do it.
Since ph12 and phi3 will not be used (be zero in practice),  it means the 3D projection images will still remain like the rectangular shapes without any tilting even I set phi2 and phi3 that makes phantom has a certain tilting angle.

Apparently yes. But it should be an easy fix.

I made a PR #88  which should fix the bug.

@lokeaichirou I just merged #88 which should fix the bug. It's being built now. In a few hours you should be able to find the updated version.
Please install as conda install -c ccpi/label/dev tomophantom to get it.

Hi @lokeaichirou ,
I removed two of Euler angles on purpose. The reason is that the 3D analytical projection data generator is not finished so providing these angles to the function will not give you the right answer. It is a limitation at the moment but I might find some time to complete it.
The 3D phantom, however can be generated by providing all 3 Euler angles. The objects can be controlled fully. So to avoid errors with someone using all 3 angles for the phantom and for the data I have removed them. So it is a hack rather than a bug, sorry.
If you really need a full 3D control (tilting, rolling, panning), then it is possible to pass all angles while generating a phantom and then use a numerical forward projector (e.g. Astra toolbox) to generate data.
thanks

Hi @lokeaichirou ,
I removed two of Euler angles on purpose. The reason is that the 3D analytical projection data generator is not finished so providing these angles to the function will not give you the right answer. It is a limitation at the moment but I might find some time to complete it.
The 3D phantom, however can be generated by providing all 3 Euler angles. The objects can be controlled fully. So to avoid errors with someone using all 3 angles for the phantom and for the data I have removed them. So it is a hack rather than a bug, sorry.
If you really need a full 3D control (tilting, rolling, panning), then it is possible to pass all angles while generating a phantom and then use a numerical forward projector (e.g. Astra toolbox) to generate data.
thanks
Hi @dkazanc . Many thanks to your answer, it;s helpful. May I ask is it possible to use the tomophantom to generate random 3D phantoms first and then use astra to make their projections?because my project requires me to generate random 3D phantoms, and tomophantom is a tool can do that.

Hi @lokeaichirou, sure you can do it with this piece of code
This is to generate a 'foam' type phantoms i.e. with spheres, ellipses and Gaussians. One probably can make a similar random generation with cubes but it is not implemented.
You probably can modify it in a way that you generate objects using all 3 angles and then use Astra to generate projections.


Github Actions failed

I found that after this PR, GitHub actions failed because the Cairo program verifier failed at ch10_01_1_checkcomp_checkfmt.cairo .
https://github.com/cairo-book/cairo-book.github.io/actions/runs/5157228054
Starting Cairo programs verifier...

---- ch10_01_1_checkcomp_checkfmt.cairo ----

Diff found in file ch10_01_1_checkcomp_checkfmt.cairo:
 --- original
+++ modified
@@ -5,16 +5,13 @@

 impl PotionAdd of Add<Potion> {
     fn add(lhs: Potion, rhs: Potion) -> Potion {
-        Potion {
-            health: lhs.health + rhs.health,
-            mana: lhs.mana + rhs.mana,
-        }
+        Potion { health: lhs.health + rhs.health, mana: lhs.mana + rhs.mana,  }
     }
 }

 fn main() {
-    let health_potion: Potion = Potion { health: 100, mana: 0};
-    let mana_potion: Potion = Potion { health: 0, mana: 100};
+    let health_potion: Potion = Potion { health: 100, mana: 0 };
+    let mana_potion: Potion = Potion { health: 0, mana: 100 };
     let super_potion: Potion = health_potion + mana_potion;
     // Both potions were combined with the `+` operator.
     assert(super_potion.health == 100, '');

---

Some Cairo programs have errors, please check the list above.

The reason is ch10_01_1_checkcomp_checkfmt.cairo can not pass the fmt check.
I fixed it in https://github.com/cairo-book/cairo-book.github.io/pull/192 .

Will be fixed in https://github.com/cairo-book/cairo-book.github.io/pull/188 .


New version?

When will the new version with promotional content be released? Currently, the GitHub action is not functioning correctly.

We need to merge the #116 as well in order to release a new version of the action

👍

Done https://github.com/Schrodinger-Hat/youtube-to-anchorfm/releases/tag/v2.4.0


Refs #33263 -- Added warning to BaseDeleteView when delete() method is overridden.

ticket-33263

I'm in favor, I think this will save some headaches to a bunch of developers.

@ngnpope Thanks :+1:

Thanks y'all for reviews :star2:


Add support for Office 365.

I have modified the manifest so it is active on the Office 365 editions as well. This is especially useful for them , as admins often disable the native feature.

Thanks @homelyseven250! It seems like the layout is slightly different for Outlook 365 (the ad containers are probably missing) so the stylesheet is hiding the message list instead. This basically makes the whole page unusable so I'd like to figure out what is going wrong there before merging this PR.
Also, are you saying that the built in dark mode is sometimes disabled?


Is pretrained model available?

I had read the paper. It is very interesting idea and a step forward from TabPFN.
Now I am looking at the MotherNet API and would like to try it on my internal data. However, it is quite frustrated since I cannot found any pretrained model. Am I missing something? or can you open-source a pretrained model? Thank you so much.

Hey! sorry for the slow response. There's an example here:
https://github.com/microsoft/mothernet/blob/762f63b821ed864eced123e09f7b9d26dc18b0b2/mothernet/tests/test_follow_sklearn_interface.py#L47
Using
from mothernet.prediction import MotherNetClassifier
from mothernet.utils import get_mn_model
model_string = "mn_d2048_H4096_L2_W32_P512_1_gpu_warm_08_25_2023_21_46_25_epoch_3940_no_optimizer.pickle"
    model_path = get_mn_model(model_string)
classifier = MotherNetClassifier(path=model_path)

yields the model from the arXiv paper.

Thank you for your information. I'll try it later.


Enabled dynamically adding settings paths via TwillAppSettings Facade

Description
This change allows for the dynamic registration of settings paths in case any capsules, packages, or anything else that wants to have isolated settings in the twill dashboard, and considers publishing to the main directory to be cumbersome.

Moving this one to a draft until the other one gets merged, that will be easier and ensure avoiding the custom code issue


fixing formatting issues

...leftover issues from earlier merged PR

cherry-picked changes from this PR to https://github.com/accurics/terrascan/pull/624


Liveness probe failed

I am trying to deploy ExceptionLess in K8S, But I've deployed Redis and ElasticSearch separately, So I changed it ex-setup.sh file ,Now any POD describe: Liveness probe failed: Get http://<IP_ADDRESS>:80/health，Maybe My helm configuration is wrong ?
Many Thanks.

``helm repo add stable https://kubernetes-charts.storage.googleapis.com/
helm repo add jetstack https://charts.jetstack.io
helm repo update
kubectl create namespace ex-dev
install exceptionless app
APP_TAG="2.9.1"
API_TAG="6.1.0"
ELASTIC_CONNECTIONSTRING="http://<IP_ADDRESS>:9200"
EMAIL_CONNECTIONSTRING="smtps://user%40domain.com:password@smtp.domain.com:465"
QUEUE_CONNECTIONSTRING="provider=azurestorage;$QUEUECONNECTIONSTRING"
REDIS_CONNECTIONSTRING="<IP_ADDRESS>:6379"
STORAGE_CONNECTIONSTRING="provider=azurestorage;$STORAGECONNECTIONSTRING"
helm install ex-dev ./exceptionless --namespace ex-dev --values ex-dev-values.yaml
--set "app.image.tag=$APP_TAG"
--set "api.image.tag=$API_TAG"
--set "jobs.image.tag=$API_TAG"
--set "elasticsearch.connectionString=$ELASTIC_CONNECTIONSTRING"
--set "email.connectionString=$EMAIL_CONNECTIONSTRING"
--set "queue.connectionString=$QUEUE_CONNECTIONSTRING"
--set "redis.connectionString=$REDIS_CONNECTIONSTRING"
--set "storage.connectionString=$STORAGE_CONNECTIONSTRING"
--set "config.EX_Scope=dev"

Did you set your connection strings to elastic correctly? Exec into the pod and curl the health endpoint or hit the health end point of the api

I'm not sure , Redis and  ElasticSearch on the public net . after the problem, curl the Health Exec into the POD the result is something like this
curl -I http://<IP_ADDRESS>/health
curl: (7) Failed to connect to <IP_ADDRESS> port 80: Connection refused

redis and elastic should not be exposed publicly.  I'd make sure the port is mapped it might be on 5000 locally in the container

Thanks , Public IP and Port is only in the learning phase. I will try to modify the mapping port, I also notice your new release, I will continue to try

Did you get this working?

Hi, thank you very much. It hasn't been solved yet, but I am busy with other work. I must have passed too many wrong arguments. Maybe Next week I'll be able to follow up on this

Thanks, please keep us updated.

Were you able to resolve this issue?

Hi，thank you niemyjski !  I rewrote the 'yml' and installed it successfully .I think my PVC and PV are still not perfect. I will continue to improve them if I have time . thank you very much !

Let me know if this shouldn't be closed and we'll reopen it. But please feel free to open a discussion with us or submit a pr for any changes you think we should make based on your findings.

Our manager asked for deployment ‘ExceptionLess’ in K8S , but I personally think it should be deployed separately.If deploying 'docker-compose mode
' separately is an excellent choice.'Helm mode' is too comprehensive and too automatically deployed.maybe it has a lot to do with my ability, which is my personal feeling.
My expectation for deployment is simple configuration,Easy to add or subtract applications.
Thank you for your constant attention to my question !!
From: Blake Niemyjski
Date: 2020-11-18 22:26
To: exceptionless/Exceptionless
CC: H; Author
Subject: Re: [exceptionless/Exceptionless] Liveness probe failed (#705)
Let me know if this shouldn't be closed and we'll reopen it. But please feel free to open a discussion with us or submit a pr for any changes you think we should make based on your findings.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

If you won't have very many events coming through the system and you won't be needing to store hundreds of thousands or millions of events I'd probably just do docker-compose. If you are going to be doing more than that we strongly recommend using k8s and that's what we use in production and dev. It can scale to every level and just works. There is a learning curve to k8s and I'm learning more every time I use it, but it is clearly the future and it's available on all major cloud providers!

We just started using K8S，not too much experience, lots of failures，It must be fully tested in an experimental environment before it can be used.
I would do better if I managed K8S full-time, but more unrelated work is affecting me .It's not good for me or the company.
I hope to get to the other end of the K8S curve as soon as possible too. Thank you !


[vulkan-*] Fix dependency versions

Fixes #40603

[ ] Changes comply with the maintainer guide.
[x] SHA512s are updated for each updated download.
[x] The "supports" clause reflects platforms that may be fixed by this new version.
[x] Any fixed CI baseline entries are removed from that file.
[x] Any patches that are no longer applied are deleted from the port's directory.
[x] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[x] Only one version is added to each modified port's versions file.

Similarly to Boost's multiple ports, Vulkan-related ports require that their version stay in sync (this represents the multiple "1.3.290.0" version contraints).
I believe this part complies with the Version constraints in ports chapter maintener guide.
I also added constraints for a few third-party dependencies (jsoncpp, valijson, etc.) because each of the vulkan repos have a scripts/known_good.json file that list these required dependencies and their version.
The last sentence of the "Version constraints in ports" chapter is a bit unclear so I am not sure if this part complies with it.
Last but not least, I did not touche the vulkan-sdk-components port as this PR is more of a hotfix and this port need a bit more investigation.
Notably, some dependencies listed in the SDK's config.json file are not available on vcpkg (MoltenVK, etc.) and some other do not have the exact version available on vcpkg (directx-dxc, etc.)

@microsoft-github-policy-service agree

It will be hard to maintain these ports without automatic update of these constraints.

These ports are usually updated at the same time (e.g. #40107) so is this really an issue?
For the third party dependency constraints, it may be slightly more tedious as we need to check the known_good.json for every port, so we could drop this part of the PR if necessary.
For first party dependencies however, it's just a matter of replacing the common version (1.3.290.0 here) in the dependencies in addition to the version field.

For the third party dependency constraints, it may be slightly more tedious as we need to check the known_good.json for every port, so we could drop this part of the PR if necessary.

And there's no mention of it in any of the ports.
I propose extending ports/vulkan-sdk-components/get-vulkan-sdk-components-info.ps1 to also generate manifest files.

I propose extending ports/vulkan-sdk-components/get-vulkan-sdk-components-info.ps1 to also generate manifest files.

Since the known_good.json files are easily machine-readable, this probably is the cleanest fix.
I have never used Powershell though, so I am not the right person for this task unfortunately.

Please note that I am not an official maintainer of vcpkg so you can keep this PR opened for review.


make input_names and output_names fields of InferenceSession public

Codecov Report

Merging #16 (1eaa488) into main (9b30ed5) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main      #16   +/-   ##
=======================================
  Coverage   87.98%   87.98%
=======================================
  Files           3        3
  Lines         383      383
=======================================
  Hits          337      337
  Misses         46       46

Impacted Files
Coverage Δ

src/highlevel.jl
91.02% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9b30ed5...1eaa488. Read the comment docs.


The test server always returns "503 Over Quota"

For example:
$ curl http://youtube-dl.appspot.com/

<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>503 Over Quota</title>

Error

Over Quota

This application is temporarily over its serving quota.  Please try again later.

Seems the test server is too popular.

you are running server on Google App Engine. and there is limitation on GAE. you can fix this by installing youtube-dl-server many times on GAE and change url of server  in your application if one app utilize its quota.

The test server runs with the free service, so its limited by its quotas. I'm not going to pay for it. But since a lot of request come from clients with user agent NativeHost and Apache-HttpClient/UNAVAILABLE (java 1.4), I think i'll try to block them.

Thanks @jaimeMF. This is just a remainder that http://youtube-dl.appspot.com/ is down. Now it's working again :)


Develop offline (spark-based) ground truth patch generation pipeline for extractions

While patch extractions happen through openeo-gfmap, matching ground truth patches through rasterization of the ground truth (communication with RDM API, see  here) should happen offline in a spark script.
Result should be matching unitemporal ground truth patches with STAC metadata so it can be loaded and merged with the other openeo cubes.

Not planned because we will directly interact with RDM API to collect ground truth and generate a ground truth global point-based geoparquet.


[WIP] Add unit tests for application

Tests for:

[ ] getEnvVars
[x] FrontEndServices
[x] RequestHandler

Developer's Certificate of Origin 1.1
   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

I’m trying to test NotebookParser.getEnvVars with the NBTestUtils.DEFAULT_CONTENT from @jupyterlab/testutils, and I’m running into issues because NotebookParser.findInCode is assuming that the source field of each cell is a string  and calling code.split, but it’s actually an array. Should I just avoid using the jupyterlab testutils and make my own? I’m calling JSON.stringify(NBTestUtils.DEFAULT_CONTENT), maybe that’s the issue?
This is the content of NBTestUtils.DEFAULT_CONTENT:

Show `NBTestUtils.DEFAULT_CONTENT`
{
   "cells":[
      {
         "cell_type":"code",
         "execution_count":1,
         "metadata":{
            "tags":[

            ]
         },
         "outputs":[
            {
               "name":"stdout",
               "output_type":"stream",
               "text":[
                  "hello world\n",
                  "0\n",
                  "1\n",
                  "2\n"
               ]
            },
            {
               "name":"stderr",
               "output_type":"stream",
               "text":[
                  "output to stderr\n"
               ]
            },
            {
               "name":"stdout",
               "output_type":"stream",
               "text":[
                  "some more stdout text\n"
               ]
            }
         ],
         "source":[
            "import sys\n",
            "sys.stdout.write('hello world\\n')\n",
            "sys.stdout.flush()\n",
            "for i in range(3):\n",
            "    sys.stdout.write('%s\\n' % i)\n",
            "    sys.stdout.flush()\n",
            "sys.stderr.write('output to stderr\\n')\n",
            "sys.stderr.flush()\n",
            "sys.stdout.write('some more stdout text\\n')\n",
            "sys.stdout.flush()"
         ]
      },
      {
         "cell_type":"markdown",
         "metadata":{
            "tags":[

            ]
         },
         "source":[
            "# Markdown Cell\n",
            "\n",
            "$ e^{ \\pm i\\theta } = \\cos \\theta \\pm i\\sin \\theta + \\beta $\n",
            "\n",
            "*It* **really** is!"
         ]
      },
      {
         "cell_type":"code",
         "execution_count":2,
         "metadata":{
            "tags":[

            ]
         },
         "outputs":[
            {
               "ename":"SyntaxError",
               "evalue":"invalid syntax (<ipython-input-2-6c5185427360>, line 1)",
               "output_type":"error",
               "traceback":[
                  "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-2-6c5185427360>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    this is a syntax error\u001b[0m\n\u001b[0m                   ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
               ]
            }
         ],
         "source":[
            "this is a syntax error"
         ]
      },
      {
         "cell_type":"code",
         "execution_count":null,
         "metadata":{
            "tags":[

            ]
         },
         "outputs":[

         ],
         "source":[
            "print('test')"
         ]
      },
      {
         "cell_type":"code",
         "execution_count":4,
         "metadata":{
            "tags":[

            ]
         },
         "outputs":[
            {
               "data":{
                  "text/latex":[
                     "The mass-energy equivalence is described by the famous equation\n",
                     " \n",
                     "$$E=mc^2$$\n",
                     " \n",
                     "discovered in 1905 by Albert Einstein. \n",
                     "In natural units ($c$ = 1), the formula expresses the identity\n",
                     " \n",
                     "\\begin{equation}\n",
                     "E=m\n",
                     "\\end{equation}"
                  ],
                  "text/plain":[
                     "<IPython.core.display.Latex object>"
                  ]
               },
               "execution_count":4,
               "metadata":{

               },
               "output_type":"execute_result"
            }
         ],
         "source":[
            "from IPython.display import Latex\n",
            "Latex('''The mass-energy equivalence is described by the famous equation\n",
            " \n",
            "$$E=mc^2$$\n",
            " \n",
            "discovered in 1905 by Albert Einstein. \n",
            "In natural units ($c$ = 1), the formula expresses the identity\n",
            " \n",
            "\\\\begin{equation}\n",
            "E=m\n",
            "\\\\end{equation}''')"
         ]
      },
      {
         "cell_type":"code",
         "execution_count":null,
         "metadata":{
            "collapsed":true
         },
         "outputs":[

         ],
         "source":[

         ]
      }
   ],
   "metadata":{
      "anaconda-cloud":{

      },
      "kernelspec":{
         "display_name":"Python [default]",
         "language":"python",
         "name":"python3"
      },
      "language_info":{
         "codemirror_mode":{
            "name":"ipython",
            "version":3
         },
         "file_extension":".py",
         "mimetype":"text/x-python",
         "name":"python",
         "nbconvert_exporter":"python",
         "pygments_lexer":"ipython3",
         "version":"3.5.2"
      }
   },
   "nbformat":4,
   "nbformat_minor":1
}


Chapter-7 Request

I love the book!!!
One request, not issue, for Chapter 7: It'd be great if there is an example of building a ensemble of different kinds of "regressions" such SVM regression, random forest regression, etc.... using the housing data.
The book has only the classifier ensemble example. Thank you.

Hi Victor,
Thanks for your kind words! :)
For regression, one simple option is to create multiple regressors and do a weighted sum of their predictions (e.g., weighted by their r2 score on the validation set). Another approach is to use stacking (checkout the solution to exercise 7.9: it shouldn't be hard to switch it from classification to regression).
Here's an implementation of the first option:
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_diabetes
from sklearn.linear_model import Ridge
from sklearn.ensemble import RandomForestRegressor, ExtraTreesRegressor
from sklearn.base import BaseEstimator, RegressorMixin, clone

X, y = load_diabetes(return_X_y=True)

X_train_val, X_test, y_train_val, y_test = train_test_split(X, y, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X, y, random_state=42)

class WeightedMeanEnsemble(BaseEstimator, RegressorMixin):
    def __init__(self, estimators, weights=None):
        self.estimators = estimators
        self.weights = weights

    def fit(self, X, y, **params):
        self.estimators_ = [clone(reg).fit(X, y, **params)
                            for name, reg in self.estimators]
        return self

    def predict(self, X):
        predictions = np.array([estimator.predict(X) for estimator in self.estimators_])
        return np.average(predictions, axis=0, weights=self.weights)

weighted_mean_ensemble = WeightedMeanEnsemble([
    ("ridge", Ridge(random_state=42)),
    ("rnd_forest_reg", RandomForestRegressor(random_state=42)),
    ("extra_trees_reg", ExtraTreesRegressor(random_state=42)),
])

weighted_mean_ensemble.fit(X_train, y_train)

print(weighted_mean_ensemble.score(X_val, y_val))
# 0.47083442135168646

weights = [reg.score(X_val, y_val) for reg in weighted_mean_ensemble.estimators_]
print(weights)
# [0.4384009113704955, 0.4084100811626866, 0.4379750225843305]

print(weighted_mean_ensemble.score(X_test, y_test))
# 0.47083442135168646

weighted_mean_ensemble.weights = weights

print(weighted_mean_ensemble.score(X_test, y_test))
# 0.47130267314071506  = very slight improvement

[reg.score(X_test, y_test) for reg in weighted_mean_ensemble.estimators_]
# [0.4384009113704955, 0.4084100811626866, 0.4379750225843305]

Notice that the ensemble performs better than all the individual regressors, and the weighted ensemble performs better than the first ensemble (which gives all estimators the same weight).
Hope this helps!


Login Modal

Hello there, I see that you need a login feature, I can help you with its frontend, Basically, upon clicking the login button, it will open a modal where you can add your email address and password and a login button to move further to authentication which you can add in future.
Kindly assign this issue, happy to contribute @AkhileshThite

Great! go ahead.

Hello there, I see that you need a login feature, I can help you with its frontend, Basically, upon clicking the login button, it will open a modal where you can add your email address and password and a login button to move further to authentication which you can add in future.
Kindly assign this issue, happy to contribute @AkhileshThite

Just make sure it matches with the theme, keep it simple.


Mention important additional parameters for engine config in README

@mfuntowicz  Thanks for looking! Wonder if anyone with correct permissions can merge this?

Thanks a lot @zaycev your contribution, looking forward for the next one 🤗


fix(tests): use legacy rules files for legacy tests

Legacy tests are designed to test Falco features, not the ruleset. Most of these tests already use ad-hoc rules files, not those provided by the tested Falco package. This commit also aligns the same behavior for legacy tests requiring the default rulesets. This is particularly relevant after we split falco_rules.yaml in multiple files, which are not shipped anymore by default.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: leogr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, leogr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Andreagit97,leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/hold

I'm fixing some failing tests. Hold on please :pray:

/hold cancel


feat(release): Reuse semantic release.

Coverage remained the same at 100.0% when pulling 80e733487161d4c5c44f77a2e881d3cbfa5ac545 on reuse-semantic-release into f7c8bad121acefe1804e7969b4d969e4476f77d7 on master.

Coverage remained the same at 100.0% when pulling 80e733487161d4c5c44f77a2e881d3cbfa5ac545 on reuse-semantic-release into f7c8bad121acefe1804e7969b4d969e4476f77d7 on master.


COOK-65 correct jar name for ubuntu

fixes COOK-65
corrects the name of the installed mysql_connector jar on Ubuntu
dpkg -L libmysql-java | grep /usr/share/java.*.jar
/usr/share/java/mysql-connector-java-5.1.16.jar
/usr/share/java/mysql-connector-java.jar
/usr/share/java/mysql.jar

failures unrelated... looks like latest rubocop added some rules

style fixes in https://github.com/caskdata/hadoop_cookbook/pull/217

:+1: LGTM


[IntelliSense explain mode] Not sure if it has value to show in Extensions View / Settings GUI

#75500

Toggle Explain Mode in editor
Extensions View trigger autocomplete

What is Explain Mode?
How do you go somewhere else?

There is a link to issue at the very top

While those suggestions aren't from extensions it's still the same feature, explaining how suggestions happen. This is a debug feature and when enabled, folks should know what it means and how to turn if off again


Get the optimized bytecode from IR

Feature request
Can numba get the optimized bytecode from intermediate representation (IR)?

Feature request
Can numba get the optimized bytecode from intermediate representation (IR)?

Probably not, no. Although, to be able to answer your question, please do say which optimized bytecode and which intermediate representation? If you want to create Python Bytecode from Numba IR, that's a no. If you want to create Python bytecode from LLVM IR, that's also a no.

Got it, thanks anyway.


"Albești", "Buimăceni" and "Coștiugeni" are listed in the drop-down menu

Description:
"Albești" , "Buimăceni" and "Coștiugeni" are listed in the drop-down menu alongside its corresponding county and commune, according to the law
Precondition:
The website is up an running.

Step 1
Type "Albești" in the search bar.

Expected results
The location is listed in the drop-down menu as "Albești, Botoșani (Albești)".

Step 2
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 3
Type "Buimăceni" in the search bar.

Expected results
The location is listed in the drop-down menu as "Buimăceni, Botoșani (Albești)".

Step 4
Press "x" button.

Expected results
The location was deleted from the search bar.

ATTENTION:
STEP1:
Request - "Albești, Botoșani (Albești)"
In dropbox, the request is displayed correctly. - "Albești, Botoșani (Albești)"
After you click on the text, what is displayed in the window this is different - "Albești, BOTOȘANI (Albești)"
STEP3:
Request - "Buimăceni, Botoșani (Albești)"
In dropbox, the request is displayed correctly. - "Buimăceni, Botoșani (Albești)"
After you click on the text, what is displayed in the window this is different - "Buimăceni, BOTOȘANI (Albești)"
STEP3:
Request - "Coștiugeni, Botoșani (Albești)"
In dropbox, the request is displayed correctly. - "Coștiugeni, Botoșani (Albești)"
After you click on the text, what is displayed in the window this is different - "Coștiugeni, BOTOȘANI (Albești)"


SQL-backed pipeline/ETL process

A wealth of info from Stefan here: https://lists.okfn.org/pipermail/okfn-labs/2014-December/001542.html

This is/was definitely a good idea, but GoodTables has a nice sweet spot working with files and we won't change the scope of this library.


can't connect: message type: closed

Hi,
I've been using this project for a while without problems but since the last 2 days I've not been able to connect.
I get every time the following message on Chrome console while trying to connect to he API:
got message  {from: "meta", type: "closed"}
Above I get the following message:
WebSocket connection to 'ws://localhost:2019/' failed: Connection closed before receiving a handshake response
This happens every time I click on the Click to connect to API button.
Backend, API server and HTTP server seems to be running fine.

Hmm that's odd... Someone else had a similar error at #5 and #48. Which operating system do you use? Might it be a firewall issue for you?

Same here, it's been working well but it stopped. getting a
["Cmd",{"type":"update"}]

Not getting the ref like usual when trying to generate the QR code

@sigalor I am using Ubuntu 17.10
I followed the steps on #5 and everything seems fine.
Accesing http://<IP_ADDRESS>:2019/ (<IP_ADDRESS> is the machine I am running the service on) shows a page with the following text:

Upgrade Required

@UrbanSwati Thanks for this info, now I know the cause of @nilsburg's error. Looking at the generateQRCode function of backend/whatsapp.py, you can see that the script currently wants the server to use WA Web version 0.2.9229, but apparently, the server does not support it anymore. Just exchange 9229 by 9547, because when you look at the websocket log of original WA Web using the Chrome developer console, you can see it requesting version 0.2.9547, thus this is the newest one.
A question to everyone: Is it possible to query the server for the newest WA Web version directly, i.e. without possibly receiving the ["Cmd",{"type":"update"}] message beforehand? This would allow the Python script to not rely on using a hardcoded version number.
@nilsburg, the Upgrade Required text is absolutely plausible, because a WebSocket server is listening on port 2019. Accessing it with a HTTP request does not really make sense.

Hi,
Yes! It's now working, but it seems that the JSON object with the messages has changed quite a bit so I will have to adapt my app.
Thank you very much for your help!

I now realise that the error I was getting on chrome was my fault because the client script was trying to connect to localhost:2019 while the service was launched on another machine. Sorry for the mistake.
But updating the WA Web version solved my problem... so thanks again!


API improvement

Changes

Provide shortcut accessors on the Merry instance to access the namespace.
Support async error handling suite.
Improve _except API. Now it can directly be used as a deco to register a handler for BaseException. Add a new kwarg "_as" to create an accessor on the namespace to access the caught error instance.
Improve _else, which should come with any _excepts or a RuntimeError is raised.
Make _try, _except, _else and _finally share the same signature (bypass the same args, kwargs to all of them).
Clean up g namespace after each error handling for less possibilities to memory leak.

All error handling suites are made to share the same signature therefore they can be methods of the same class, instance level context should rather be a member data than a Merry.Namespace member.
This is useful for example in the class-based Namespace of python-socketio.
class MyClass:
    # static member
    m = Merry()

    @m._try
    def method_a(self):
        pass

    @m._except  # all BaseException
    def handle_all_error(self):
        pass

Merry.Namespace should only be accessible inside error handling clauses to avoid memory leak.
Please also take a look at new test cases for other examples of the new changes.
Thanks!

Refer to #2

Oops, it seems that new improvements are only supported above py35.
Please let me know if there is any mishandled :)
Thank you.

Thanks. Unfortunately you have combined a lot of things into a single PR, which makes it hard for me to review and test. What I think I'm going to do when I have a moment is separate your improvements so that I can check them out individually.

@miguelgrinberg

Unfortunately you have combined a lot of things into a single PR

I see. I should have separated them by features before sending the PR. 😂
I can split the features into 6 PRs if you wish, make it easier for you to review.
(then feel free to close this one)

Provide shortcut accessors on the Merry instance to access the namespace.
Support async error handling suite.
Improve _except API. Now it can directly be used as a deco to register a handler for BaseException. Add a new kwarg "_as" to create an accessor on the namespace to access the caught error instance.
Improve _else, which should come with any _excepts or a RuntimeError is raised.
Make _try, _except, _else and _finally share the same signature (bypass the same args, kwargs to all of them).
Clean up g namespace after each error handling for less possibilities to memory leak.

I'm going to close this PR, if you submit the features separately I'll be happy to review them.


Need to keep track of part submitters

When a post request is sent to the submit_part view, the currently logged in user must be found in the Users table and referenced in the Part object as a ForeignKey.

The key is to save the incoming form data with the commit=False option. This prevents updating the database while the author field is still null.


chore: release 1.4.0

:robot: I have created a release beep boop
1.4.0 (2024-06-18)
Features

snowflake: support key-pair authentication (#103) (011a7fd)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/rudderlabs/sqlconnect-go/releases/tag/v1.4.0 :sunflower:


Add 'path' to Event

Hello 👋
Could you provide documentation in the description backing this change? How do we know this is the correct definition?
Also, style-wise, we prefer Array<T> over T[] in the libdefs 👍

(Seems like it's a non-standard actually)[https://stackoverflow.com/questions/39245488/event-path-undefined-with-firefox-and-vue-js].
Apologies, closing.

No worries, thanks for checking!


add Harbor webhook

Description of your changes
add harbor webhook trigger
I have:

[x] Read and followed KubeVela's contribution process.
[ ] Related Docs updated properly. In a new feature or configuration option, an update to the documentation is necessary.
[x] Run make reviewable to ensure this PR is ready for review.
[x] Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested
it has tested by local cluster & harbor and its passed ginkgo usecase test
Special notes for your reviewer
/cc @barnettZQG @wonderflow @FogDong

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: K1ngram4 sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


Cannot upgrade calico from 3.4 to 3.23

I'm following the guide https://projectcalico.docs.tigera.io/maintenance/kubernetes-upgrade#upgrading-an-installation-that-uses-an-etcd-datastore to upgrade calico from 3.4 to 3.23. The calico-kube-controllers becomes Ready, but the calico-node couldn't reach the "Ready" state.
calico-kube-controllers-86bc6c487d-ppgnm   1/1     Running   0          2m57s
calico-node-cbz6n                          0/1     Running   0          3m4s

Expected Behavior

The upgrade process should be smoothly.
Current Behavior

The calico-node stuck in restarting Felix. Here are some logs of calico-node:
2022-05-24 14:00:45.514 [WARNING][2155] felix/l3_route_resolver.go 632: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-05-24 14:00:45.514 [WARNING][2155] felix/l3_route_resolver.go 632: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-05-24 14:00:45.514 [WARNING][2155] felix/l3_route_resolver.go 632: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-05-24 14:00:45.514 [WARNING][2155] felix/l3_route_resolver.go 632: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-05-24 14:00:45.514 [WARNING][2155] felix/l3_route_resolver.go 632: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-05-24 14:00:45.514 [WARNING][2155] felix/l3_route_resolver.go 632: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-05-24 14:00:45.514 [WARNING][2155] felix/l3_route_resolver.go 632: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-05-24 14:00:45.514 [WARNING][2155] felix/l3_route_resolver.go 632: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-05-24 14:00:45.514 [WARNING][2155] felix/l3_route_resolver.go 632: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-05-24 14:00:45.515 [WARNING][2155] felix/l3_route_resolver.go 632: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-05-24 14:00:45.515 [WARNING][2155] felix/l3_route_resolver.go 632: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-05-24 14:00:45.515 [WARNING][2155] felix/l3_route_resolver.go 935: Some nodes share IP address, route calculation may choose wrong node. cidr=::/128 nodes=[]string{"lasvai-staging-capos-k8s-master01", "lasvai-staging-capos-k8s-master02", "lasvai-staging-capos-k8s-master03", "vai-computing-con-027", "vai-computing-con-028", "vai-computing-con-029", "vai-computing-con-030", "vai-computing-con-031", "vai-computing-con-038", "vai-computing-con-039", "vai-computing-con-040"}
2022-05-24 14:00:45.515 [WARNING][2155] felix/l3_route_resolver.go 935: Some nodes share IP address, route calculation may choose wrong node. cidr=::/128 nodes=[]string{"lasvai-staging-capos-k8s-master01", "lasvai-staging-capos-k8s-master02", "lasvai-staging-capos-k8s-master03", "vai-computing-con-027", "vai-computing-con-028", "vai-computing-con-029", "vai-computing-con-030", "vai-computing-con-031", "vai-computing-con-038", "vai-computing-con-039", "vai-computing-con-040", "vai-computing-con-041"}
2022-05-24 14:00:45.515 [WARNING][2155] felix/l3_route_resolver.go 935: Some nodes share IP address, route calculation may choose wrong node. cidr=::/128 nodes=[]string{"lasvai-staging-capos-k8s-master01", "lasvai-staging-capos-k8s-master02", "lasvai-staging-capos-k8s-master03", "vai-computing-con-027", "vai-computing-con-028", "vai-computing-con-029", "vai-computing-con-030", "vai-computing-con-031", "vai-computing-con-038", "vai-computing-con-039", "vai-computing-con-040", "vai-computing-con-041", "vai-computing-con-042"}
2022-05-24 14:00:45.515 [WARNING][2155] felix/l3_route_resolver.go 935: Some nodes share IP address, route calculation may choose wrong node. cidr=::/128 nodes=[]string{"lasvai-staging-capos-k8s-master01", "lasvai-staging-capos-k8s-master02", "lasvai-staging-capos-k8s-master03", "vai-computing-con-027", "vai-computing-con-028", "vai-computing-con-029", "vai-computing-con-030", "vai-computing-con-031", "vai-computing-con-038", "vai-computing-con-039", "vai-computing-con-040", "vai-computing-con-041", "vai-computing-con-042", "vai-computing-con-043"}
2022-05-24 14:00:45.515 [WARNING][2155] felix/l3_route_resolver.go 935: Some nodes share IP address, route calculation may choose wrong node. cidr=::/128 nodes=[]string{"lasvai-staging-capos-k8s-master01", "lasvai-staging-capos-k8s-master02", "lasvai-staging-capos-k8s-master03", "vai-computing-con-027", "vai-computing-con-028", "vai-computing-con-029", "vai-computing-con-030", "vai-computing-con-031", "vai-computing-con-038", "vai-computing-con-039", "vai-computing-con-040", "vai-computing-con-041", "vai-computing-con-042", "vai-computing-con-043", "vai-computing-con-044"}
2022-05-24 14:00:45.534 [WARNING][2155] felix/daemon.go 1209: IPIP and/or VXLAN encapsulation changed, need to restart.
2022-05-24 14:00:45.534 [WARNING][2155] felix/daemon.go 715: Felix is shutting down reason="encapsulation changed"

Your Environment

Calico version: v3.4 -> v3.23
Orchestrator version (e.g. kubernetes, mesos, rkt): k8s v1.19.9
Operating System and version: Linux vai-computing-con-040 4.15.0-133-generic #137~16.04.1-Ubuntu SMP Fri Jan 15 02:55:18 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

My calico.yaml:
---
# Source: calico/templates/calico-config.yaml
# This ConfigMap is used to configure a self-hosted Calico installation.
kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  # Configure this with the location of your etcd cluster.
  etcd_endpoints: "https://<IP_ADDRESS>:2379,https://<IP_ADDRESS>:2379,https://<IP_ADDRESS>:2379"
  # If you're using TLS enabled etcd uncomment the following.
  # You must also populate the Secret below with these files.
  etcd_ca: "/var/lib/kubernetes/ca.pem"
  etcd_cert: "/var/lib/kubernetes/kubernetes.pem"
  # chmod 644 /var/lib/kubernetes/kubernetes-key.pem
  etcd_key: "/var/lib/kubernetes/kubernetes-key.pem"
  # Typha is disabled.
  typha_service_name: "none"
  # Configure the backend to use.
  calico_backend: "bird"

  # Configure the MTU to use for workload interfaces and tunnels.
  # By default, MTU is auto-detected, and explicitly setting this field should not be required.
  # You can override auto-detection by providing a non-zero value.
  veth_mtu: "0"

  ipv4_range: "<IP_ADDRESS>/16"
  ipip_env: "always"

  # The CNI network configuration to install on each node. The special
  # values in this config will be automatically populated.
  cni_network_config: |-
    {
      "name": "k8s-pod-network",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "calico",
          "log_level": "info",
          "log_file_path": "/var/log/calico/cni/cni.log",
          "etcd_endpoints": "__ETCD_ENDPOINTS__",
          "etcd_key_file": "__ETCD_KEY_FILE__",
          "etcd_cert_file": "__ETCD_CERT_FILE__",
          "etcd_ca_cert_file": "__ETCD_CA_CERT_FILE__",
          "mtu": __CNI_MTU__,
          "ipam": {
              "type": "calico-ipam"
          },
          "policy": {
              "type": "k8s"
          },
          "kubernetes": {
              "kubeconfig": "__KUBECONFIG_FILEPATH__"
          }
        },
        {
          "type": "portmap",
          "snat": true,
          "capabilities": {"portMappings": true}
        },
        {
          "type": "bandwidth",
          "capabilities": {"bandwidth": true}
        }
      ]
    }

---
# Source: calico/templates/calico-kube-controllers-rbac.yaml

# Include a clusterrole for the kube-controllers component,
# and bind it to the calico-kube-controllers serviceaccount.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-kube-controllers
rules:
  # Pods are monitored for changing labels.
  # The node controller monitors Kubernetes nodes.
  # Namespace and serviceaccount labels are used for policy.
  - apiGroups: [""]
    resources:
      - pods
      - nodes
      - namespaces
      - serviceaccounts
    verbs:
      - watch
      - list
      - get
  # Watch for changes to Kubernetes NetworkPolicies.
  - apiGroups: ["networking.k8s.io"]
    resources:
      - networkpolicies
    verbs:
      - watch
      - list
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-kube-controllers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-kube-controllers
subjects:
  - kind: ServiceAccount
    name: calico-kube-controllers
    namespace: kube-system
---

---
# Source: calico/templates/calico-node-rbac.yaml
# Include a clusterrole for the calico-node DaemonSet,
# and bind it to the calico-node serviceaccount.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-node
rules:
  # The CNI plugin needs to get pods, nodes, and namespaces.
  - apiGroups: [""]
    resources:
      - pods
      - nodes
      - namespaces
    verbs:
      - get
  # EndpointSlices are used for Service-based network policy rule
  # enforcement.
  - apiGroups: ["discovery.k8s.io"]
    resources:
      - endpointslices
    verbs:
      - watch
      - list
  - apiGroups: [""]
    resources:
      - endpoints
      - services
    verbs:
      # Used to discover service IPs for advertisement.
      - watch
      - list
  # Pod CIDR auto-detection on kubeadm needs access to config maps.
  - apiGroups: [""]
    resources:
      - configmaps
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - nodes/status
    verbs:
      # Needed for clearing NodeNetworkUnavailable flag.
      - patch

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: calico-node
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-node
subjects:
  - kind: ServiceAccount
    name: calico-node
    namespace: kube-system

---
# Source: calico/templates/calico-node.yaml
# This manifest installs the calico-node container, as well
# as the CNI plugins and network config on
# each master and worker node in a Kubernetes cluster.
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: calico-node
  namespace: kube-system
  labels:
    k8s-app: calico-node
spec:
  selector:
    matchLabels:
      k8s-app: calico-node
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      labels:
        k8s-app: calico-node
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      hostNetwork: true
      tolerations:
        # Make sure calico-node gets scheduled on all nodes.
        - effect: NoSchedule
          operator: Exists
        # Mark the pod as a critical add-on for rescheduling.
        - key: CriticalAddonsOnly
          operator: Exists
        - effect: NoExecute
          operator: Exists
      serviceAccountName: calico-node
      # Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force
      # deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.
      terminationGracePeriodSeconds: 0
      priorityClassName: system-node-critical
      initContainers:
        # This container installs the CNI binaries
        # and CNI network config file on each node.
        - name: install-cni
          image: cubox.prod.hulu.com/proxy/calico/cni:v3.23.1
          command: ["/opt/cni/bin/install"]
          envFrom:
            - configMapRef:
                # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
                name: kubernetes-services-endpoint
                optional: true
          env:
            # Name of the CNI config file to create.
            - name: CNI_CONF_NAME
              value: "10-calico.conflist"
            # The CNI network config to install on each node.
            - name: CNI_NETWORK_CONFIG
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: cni_network_config
            # The location of the etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            # CNI MTU Config variable
            - name: CNI_MTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # Prevents the container from sleeping forever.
            - name: SLEEP
              value: "false"
          volumeMounts:
            - mountPath: /host/opt/cni/bin
              name: cni-bin-dir
            - mountPath: /host/etc/cni/net.d
              name: cni-net-dir
            - mountPath: /var/lib/kubernetes
              name: var-lib-k8s
              readOnly: true
            #- mountPath: /etc/calico/kube/kubeconfig
            #  name: etc-kubeconfig
            #  readOnly: true
          securityContext:
            privileged: true
      containers:
        # Runs calico-node container on each Kubernetes node. This
        # container programs network policy and routes on each
        # host.
        - name: calico-node
          image: docker.io/calico/node:v3.23.1
          envFrom:
            - configMapRef:
                # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
                name: kubernetes-services-endpoint
                optional: true
          env:
            # The location of the etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            # Location of the CA certificate for etcd.
            - name: ETCD_CA_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_ca
            # Location of the client key for etcd.
            - name: ETCD_KEY_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_key
            # Location of the client certificate for etcd.
            - name: ETCD_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_cert
            # Set noderef for node controller.
            - name: CALICO_K8S_NODE_REF
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            # Choose the backend to use.
            - name: CALICO_NETWORKING_BACKEND
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: calico_backend
            # Cluster type to identify the deployment type
            - name: CLUSTER_TYPE
              value: "k8s,bgp"
            # Auto-detect the BGP IP address.
            - name: IP
              value: ""
            # Enable IPIP
            - name: CALICO_IPV4POOL_IPIP
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: ipip_env
            # Enable or Disable VXLAN on the default IP pool.
            - name: CALICO_IPV4POOL_VXLAN
              value: "Never"
            # Enable or Disable VXLAN on the default IPv6 IP pool.
            - name: CALICO_IPV6POOL_VXLAN
              value: "Never"
            # Set MTU for tunnel device used if ipip is enabled
            - name: FELIX_IPINIPMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # Set MTU for the VXLAN tunnel device.
            - name: FELIX_VXLANMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # Set MTU for the Wireguard tunnel device.
            - name: FELIX_WIREGUARDMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # The default IPv4 pool to create on startup if none exists. Pod IPs will be
            # chosen from this range. Changing this value after installation will have
            # no effect. This should fall within `--cluster-cidr`.
            - name: CALICO_IPV4POOL_CIDR
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: ipv4_range
            # Disable file logging so `kubectl logs` works.
            - name: CALICO_DISABLE_FILE_LOGGING
              value: "true"
            # Set Felix endpoint to host default action to ACCEPT.
            - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
              value: "ACCEPT"
            # Disable IPv6 on Kubernetes.
            - name: FELIX_IPV6SUPPORT
              value: "false"
            - name: FELIX_HEALTHENABLED
              value: "true"
          securityContext:
            privileged: true
          resources:
            requests:
              cpu: 250m
          lifecycle:
            preStop:
              exec:
                command:
                  - /bin/calico-node
                  - -shutdown
          livenessProbe:
            exec:
              command:
                - /bin/calico-node
                - -felix-live
                - -bird-live
            periodSeconds: 10
            initialDelaySeconds: 10
            failureThreshold: 6
            timeoutSeconds: 10
          readinessProbe:
            exec:
              command:
                - /bin/calico-node
                - -felix-ready
                - -bird-ready
            periodSeconds: 10
            timeoutSeconds: 10
          volumeMounts:
            # For maintaining CNI plugin API credentials.
            - mountPath: /host/etc/cni/net.d
              name: cni-net-dir
              readOnly: false
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /run/xtables.lock
              name: xtables-lock
              readOnly: false
            - mountPath: /var/run/calico
              name: var-run-calico
              readOnly: false
            - mountPath: /var/lib/calico
              name: var-lib-calico
              readOnly: false
            - mountPath: /var/lib/kubernetes
              name: var-lib-k8s
            #- mountPath: /etc/calico/kube/kubeconfig
            #  name: etc-kubeconfig
            #  readOnly: true
            - name: policysync
              mountPath: /var/run/nodeagent
            # For eBPF mode, we need to be able to mount the BPF filesystem at /sys/fs/bpf so we mount in the
            # parent directory.
            - name: sysfs
              mountPath: /sys/fs/
              # Bidirectional means that, if we mount the BPF filesystem at /sys/fs/bpf it will propagate to the host.
              # If the host is known to mount that filesystem already then Bidirectional can be omitted.
              mountPropagation: Bidirectional
            - name: cni-log-dir
              mountPath: /var/log/calico/cni
              readOnly: true
      volumes:
        # Used by calico-node.
        - name: lib-modules
          hostPath:
            path: /lib/modules
        - name: var-run-calico
          hostPath:
            path: /var/run/calico
        - name: var-lib-calico
          hostPath:
            path: /var/lib/calico
        - name: xtables-lock
          hostPath:
            path: /run/xtables.lock
            type: FileOrCreate
        - name: sysfs
          hostPath:
            path: /sys/fs/
            type: DirectoryOrCreate
        # Used to install CNI.
        - name: cni-bin-dir
          hostPath:
            path: /opt/cni/bin
        - name: cni-net-dir
          hostPath:
            path: /etc/cni/net.d
        # Used to access CNI logs.
        - name: cni-log-dir
          hostPath:
            path: /var/log/calico/cni
        # Mount in the etcd TLS secrets with mode 400.
        - name: var-lib-k8s
          hostPath:
            path: /var/lib/kubernetes
        #- name: etc-kubeconfig
        #  hostPath:
        #    path: /etc/calico/kube/kubeconfig
        # Used to create per-pod Unix Domain Sockets
        - name: policysync
          hostPath:
            type: DirectoryOrCreate
            path: /var/run/nodeagent
---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-node
  namespace: kube-system

---
# Source: calico/templates/calico-kube-controllers.yaml
# See https://github.com/projectcalico/kube-controllers
apiVersion: apps/v1
kind: Deployment
metadata:
  name: calico-kube-controllers
  namespace: kube-system
  labels:
    k8s-app: calico-kube-controllers
spec:
  # The controllers can only have a single active instance.
  replicas: 1
  selector:
    matchLabels:
      k8s-app: calico-kube-controllers
  strategy:
    type: Recreate
  template:
    metadata:
      name: calico-kube-controllers
      namespace: kube-system
      labels:
        k8s-app: calico-kube-controllers
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      tolerations:
        # Mark the pod as a critical add-on for rescheduling.
        - key: CriticalAddonsOnly
          operator: Exists
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      serviceAccountName: calico-kube-controllers
      priorityClassName: system-cluster-critical
      # The controllers must run in the host network namespace so that
      # it isn't governed by policy that would prevent it from working.
      hostNetwork: true
      containers:
        - name: calico-kube-controllers
          image: docker.io/calico/kube-controllers:v3.23.1
          env:
            # The location of the etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            # Location of the CA certificate for etcd.
            - name: ETCD_CA_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_ca
            # Location of the client key for etcd.
            - name: ETCD_KEY_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_key
            # Location of the client certificate for etcd.
            - name: ETCD_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_cert
            # Choose which controllers to run.
            - name: ENABLED_CONTROLLERS
              value: policy,namespace,serviceaccount,workloadendpoint,node
          volumeMounts:
            # Mount in the etcd TLS secrets.
            - mountPath: /var/lib/kubernetes
              name: var-lib-k8s
          livenessProbe:
            exec:
              command:
                - /usr/bin/check-status
                - -l
            periodSeconds: 10
            initialDelaySeconds: 10
            failureThreshold: 6
            timeoutSeconds: 10
          readinessProbe:
            exec:
              command:
                - /usr/bin/check-status
                - -r
            periodSeconds: 10
      volumes:
        # Mount in the etcd TLS secrets with mode 400.
        - name: var-lib-k8s
          hostPath:
            path: /var/lib/kubernetes

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-kube-controllers
  namespace: kube-system

---

# This manifest creates a Pod Disruption Budget for Controller to allow K8s Cluster Autoscaler to evict

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: calico-kube-controllers
  namespace: kube-system
  labels:
    k8s-app: calico-kube-controllers
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: calico-kube-controllers

@cy-zheng Facing the same issue. How did you manage to solve it?

I just had the same issue (after updating from 3.22 to 3.23). I managed to solve it by setting vxlanEnabled to false in FelixConfiguration. (I'm on an IPv6 only cluster and using no kind of encapsulation)


W/S moving conductor not continuously

Describe your bug here.
https://github.com/user-attachments/assets/227a83f4-a71d-4a2d-8acc-4b88424cd371
Command Prompt/Terminal logs ()
No response
Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows
Did you edit anything in this build? If so, mention or summarize your changes.
No

have you ever tried Shift + W/S?

#76


empty a

An experiment, please ignore.

version 1 commit 39428df787f53febdc617cbccbf0cb1bf6cacd0f

version 2 commit a56a92da07fa1a85abe665febd423938d0b46859

Comparison of version1 and version2 https://github.com/oVirt/ovirt-web-ui/compare/39428df...a56a92d


break: Remove out-of-place repair

Fix #409
This PR removes out-of-place repair, so that Sorald now only operates in-place. It is the final part of removing the mostly redundant features that hinder or slow down further improvement of Sorald.
Changes to production code are small. The only significant changes to production code are the removal of the --file-output-strategy argument and some related modifications to SoraldConfig, Repair and RepairCommand.
The rest of the changes are in the tests, and that is entirely to accommodate for the fact that the sorald-workspace no longer is created automatically (all repair is in-place!). Any test that repairs code must be executed on a copy of the test resources, so as not to directly modify the test resources themselves. Utility functions for this is provided in TestHelper.
Ps. Sorry about the size of this PR. Splitting it further is possible but would result in a large amount of extra work for me, as all changes are pretty tightly related.

I love removal of code. Thanks @slarse!

I love removal of code. Thanks @slarse!

It's the best!
For the record, if we ever find a need to re-implement out-of-place repair, we can do so very easily on top of the current architecture:

Out-of-place with all files preserved: Just copy the input directory/file to the desired destination and run in-place on the copy.

This is however so easily done by the user of sorald that I find it kind of pointless ($ cp -r target target-copy; sorald repair --original-files-path target-copy)

Out-of-place with only changed files preserved: Copy the input directory/file to the desired destination, run in-place on the copy, delete all files that were not written to.

Just putting that here in case someone decides this functionality is desirable.


Add an extra readout method that uses a firmware patch.

Hello,
This pull request tries to read the resolution values inside the MS213x firmware. Two new region fields are provided:
bertold -> Resolution on input (might be unstable, running long test now)
bertold_scaler -> Resolution on output of scaler (seems stable for many hours)
These readout methods also provide a counter that is incremented each time a frame is sent. It allows easy detection of freezes.
Good luck.

Thank you so much, Bertold! We'll test both regions today! :)


Add crate ndarray-rand

This is a stab at just ndarray-rand with four simple constructor methods.

cc @daniel-vainsencher
The f64 vs f32 issue is making me unhappy here. It would be the best if rand just implemented Normal for f32.

Pushed a new version that even removed the ::normal() constructor again. We'll add it later. Ideally, Normal from rand should just implement generation of any F: Float or something like that..

Wasted some time looking at this, but unfortunately don't have much smart to say. I see your frustration with rand's Normal being f64 only. Maybe there aren't any speed savings to be had (without too much numerical precision tradeoff) so they didn't bother?
One way to depend on the design of rand less (hence make it easier to change implementations later on) is to export only particular important convenience constructors like normal, binomial, rademacher, hence not expose IndependentSample etc in the interfaces. Ideally generic in NdFloat, but this isn't critical, IMO.

I thought that was a smart contribution about a possible rand-less API surface.

Glad that helps, I meant I'd hoped to have a fix rather than a workaround, but I don't see a simple one and don't have the bandwidth to try to improve rand.

Haha the more I fret over this, the harder it seems.
It's related to traits and common interfaces. It's really uncommon that a trait really fits your use case like a glove.
rand and num are immature, just like ndarray is.
Float is another trait that does not fit so well. Yet I want to use it because it's what I assume numerical code likes to use, and we want to be compatible with that as far as possible. As soon as we try to create the best possible library, we are drawn towards creating and controlling traits ourselves though.
I'll merge this, it has the right directory structure, so that I can fix up documentation on top of this.
But, no release of ndarray-rand yet.


support Allegro CL

This is the start to supporting Allegro CL. One big question I have is: Must I pin the vector storage? If I do
(excl:with-pinned-objects (,g) ...)

after g is created, then I get an error:
found non-atomic quad when atomicity required, quad =
(CALL FSLOT-ADDRESS-TYPED  ({380} {381} {379})
    -> ({378}))

In any case, at minimum, this patch allows magicl to compile.

I'm going to close this since I don't have Allegro CL access anymore.

I'm going to close this since I don't have Allegro CL access anymore.


Wildcard support

The plugin doesn't consider wildcard in ssh/config.
I have following configuration:
Host exx--example
  Hostname <IP_ADDRESS>

Host exx_ exx-*
  User root
  IdenityFile ~/.ssh/key
  ForwardAgent no

The plugin then tries to connect to the host with my current username and ask for a password.
If I add the settings directly to the host like:
Host exx--example
  Hostname <IP_ADDRESS>
  User root
  IdenityFile ~/.ssh/key
  ForwardAgent no

Everything works as expected and the key is also loaded from the ssh-agent. I need this kind of a setup as I have over 200 hosts in my configuration and the with the wildcard my config is already 1200 lines long.
Edit:
I guess the function parse_hosts_from_configs would needed to be extended to support this use-case.

@ohdearaugustin Thanks for the issue report!
This is an interesting one because the plugin itself doesn't pass the SSH opts (other than Port, User, and Path) from the host files specifically, it banks on the executables themselves to read from the SSH config for the specified host. etc, if you do ssh <host>  in your terminal, if the wildcards work they theoretically should work with the plugin as well...
Maybe what is happening here is because you don't have User defined for the individual hosts, then https://github.com/nosduco/remote-sshfs.nvim/blob/main/lua/remote-sshfs/connections.lua#L109 this line is overwriting it with your local user which is incorrect and thus asking for your password.
I'm debating on the best way to "detect" that it should not overwrite the username and use a different or wildcard one. Could be a configuration option, or potentially parse the SSH config files for any wildcards that match the current host and check if a User is specified. Probably would do that for Port and Path as well...
Open to suggestions! Will try and tinker with this today and early this week.

https://github.com/libfuse/sshfs?tab=readme-ov-file#how-to-use
With how SSHFS works under the hood, seems like parsing for the wildcard configuration and using the User (and other options) from that is the only way since SSHFS defaults to local user if not supplied...

I have now tried it with sshfs itself and it seems like to work with the wildcard.
I guess the best approach would be to not fallback on the local user in the lua script. This way we would not need to parse the whole ssh config in lua. Furthermore if no user is defined in the plugin or the ssh config sshfs automatically will fallback to the local user anyway.


⚠ Fetchers has degraded performance

In a715a8f, Fetchers ($STATUS_URL) experienced degraded performance:

HTTP code: 200
Response time: 426 ms

Resolved: Fetchers performance has improved in f1831fb.


Add an option to change the color scheme

Hey team! Please add your planning poker estimate with ZenHub @isabellamoak @vstevens14


Update rules are not processing declarations of mutable variables correctly

A statement
mutable arr = new Int[10]; // this is not updated

is not updated by the update rules of the qsfmt but instead merely left as is.
The same assigned to an immutable variable works fine:
let arr = new Int[10]; // this is updated correctly

Fixed with #1212


Package (count on) website http://nixos.org/nixpkgs is terribly outdated

The package count on http://nixos.org/nixpkgs/ says that
„The Nix Packages collection (Nixpkgs) is a set of about 2,500 packages for the Nix package manager,“
which is a very outdated information that might turn potential users away.
It would be a good idea to update it, or better yet, have a script constantly update the package count on the website.

This is now updated on http://nixos.org/nixpkgs/ (and this repo is obsolete)


Support for properties

Ideas
Require From<JsValue> & Into<JsValue>

#[property(get, set)] for property only
#[attribute(get, set)] for attribute + property

Setting a property could send a message like setting an attribute.
We need a way to retrieve a property value from  a RustComponentInstance.
See

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/get
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/set
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Reflect

Explore issue #20 first


[Feature] allow forcing the action to use Github API, even if local .git directory is present

Is this feature missing in the latest version?

[X] I'm using the latest release

Is your feature request related to a problem? Please describe.
I only want to do a shallow clone of the git repository (we use a monorepo, so full fetch is slow) in an initial step, and check the changed files using this action in a later step, using the Github API. However, the action checks if there is a .git directory present and then automatically performs a full fetch of the git repository, negating the performance benefit of the shallow clone.
Describe the solution you'd like?
An input that forces this action to use the Github API, even if a .git directory is present.
Describe alternatives you've considered?
Changing the order of the steps, i.e., first check the changed files using this action, and in a subsequent step perform the shallow clone. This is awkward and error-prone, however.
Anything else?
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Hi @gstokkink. have you considered running the action a separate job after which you can utilize the output?

That would slow things down and result in an explosion of jobs in my case, since I have to rely on matrix.

Hi, @gstokkink can you clarify what you mean by "since I have to rely on matrix.".
Here's a working example of what I mean by using multiple jobs: https://github.com/tj-actions/demo3/blob/main/.github/workflows/multi-job.yml

@jackton1 neither link works for me

Hi @gstokkink ive updated the link let me know if that works for you.

Hi, @gstokkink this is now available in the latest release via the use_rest_api input.

@jackton1 excellent, thanks! 🙏

@jackton1 looks like the warning for incompatible inputs is not working 100% correctly. I receive the following warnings when using the new use_rest_api input:
  Warning: Input "path" is not supported when using GitHub's REST API to get changed files
  Warning: Input "quotePath" is not supported when using GitHub's REST API to get changed files
  Warning: Input "diffRelative" is not supported when using GitHub's REST API to get changed files
  Warning: Input "recoverFilesSeparator" is not supported when using GitHub's REST API to get changed files
  Warning: Input "recoverFilesIgnoreSeparator" is not supported when using GitHub's REST API to get changed files
  Warning: Input "oldNewSeparator" is not supported when using GitHub's REST API to get changed files
  Warning: Input "oldNewFilesSeparator" is not supported when using GitHub's REST API to get changed files

However, I have not (explicitly) set any of these inputs. Maybe it's because these inputs default to a non-empty string, and https://github.com/tj-actions/changed-files/blob/main/src/main.ts#L282 therefore evaluates to a truthy value?

HI @gstokkink this should be resolved in the latest patch

@jackton1 thanks for the quick fix! 👍🏻 Good job on this action by the way, seems to work flawlessly. Only too bad the path argument does not work for the Github REST API approach, I'll see if I can whip up a PR for that. Useful for monorepos containing (and building) multiple services.

Thanks, Unfortunately using path  input in the context of a REST API call isn’t supported due to the fact that no local file is used. But I’m open to reviewing a proposed change