Translations and ideas for extensions

In addition to the take translation I added in my previous PR, there is some more that might be good to add. At least, I am using these myself. I can make a PR.

split. The syntax is different for numpy and tensorflow/torch. The former wants the number of splits or an array of locations of splits, whereas tensorflow/torch either want the number of splits or an array of split sizes. We can go from one format the other using np.diff
diff. This is implemented in tensorflow as tf.experimental.numpy.diff, and not implemented at all for torch. This also means I don't know what the cleanest way is to implement split mentioned above. Maybe just using np.diff and then convert to array of right backend if necessary?
linalg.norm, seems to work with tensorflow, but for torch we need to do _SUBMODULE_ALIASES["torch", "linalg.norm"] = "torch"
I didn't check these things for any other libraries.

Maybe a bit of an overly ambitious idea, but have you ever thought about baking in support for JIT? Right now it seems that for TensorFlow everything works with eager execution, and I'm not sure you can compile the computation graphs resulting from a series of ar.do calls.
PyTorch also support JIT to some extend with TorchScript
Numpy doesn' t have JIT, but there is Numba
Cupy has an interface with Numba that does seem to allow JIT.
JAX has support for JIT
Another thing is gradients. Several of these libraries have automatic gradients, and having an autoray interface for doing computations with automatic gradients would be fantastic as well (although probably also ambitious).
If you think these things are doable at all, I wouldn't mind spending some time to try to figure out how this could work.

Less ambitiously, you did mention in #3 that something along the lines of
with set_backend(like):
    ...

would be pretty nice. I can try to do this. This probably comes down to checking for a global flag in ar.do after the line
if like is None:

Regarding split I think torch has tensor_split which is the equivalent? Since the user interface is unified as the numpy version feel free to contribute any versions even if limited to just 1 other backend and specific syntaxes - we can incrementally improve from there.

autoray works with all the jit libraries that work via tracing (jax.jit, torch.jit.trace, tensorflow.function) - indeed this is one of the reasons for it to exist and how I use it. Similarly for autodiff - I use autoray with all the above and also e.g. autograd.
Do you mean 'baked in' like a unified autoray.jit(backend='torch')(fn) / autoray.gradient(fn(arrays), arrays) like interface? This is to some extent what quimb.tensor.TNOptimizer does.
I do have a private, experimental 'lazy' autoray module which is like a very minimal, lightweight computational graph - you can perform an entire autoray computation symbolically, do some basic simplifications/reuse, then optionally compute later.
The motivation is partly just that libraries like dask are very slow for large (100000+) computational graphs, but also maybe down the line to think about automatically chunking jit, checkpointing autodiff etc.

with_backend I am slightly in two minds about.  The negatives are it's just another way to do something one can already do - i.e. it doesn't really solve any problems I have come across so far.
The other is that it adds more overhead to do, which I feel is crucial to keep low since it often called millions+ of times. Though that might be cleverly avoided somethow.
However if it had some crucial use case and little to no overhead I could be tempted.

Regarding split I think torch has tensor_split which is the equivalent? Since the user interface is unified as the numpy version feel free to contribute any versions even if limited to just 1 other backend and specific syntaxes - we can incrementally improve from there.

RIght, tensor_split is indeed much more like the numpy split. I'll make a PR.

autoray works with all the jit libraries that work via tracing (jax.jit, torch.jit.trace, tensorflow.function) - indeed this is one of the reasons for it to exist and how I use it. Similarly for autodiff - I use autoray with all the above and also e.g. autograd.
Do you mean 'baked in' like a unified autoray.jit(backend='torch')(fn) / autoray.gradient(fn(arrays), arrays) like interface? This is to some extent what quimb.tensor.TNOptimizer does.
I do have a private, experimental 'lazy' autoray module which is like a very minimal, lightweight computational graph - you can perform an entire autoray computation symbolically, do some basic simplifications/reuse, then optionally compute later.
The motivation is partly just that libraries like dask are very slow for large (100000+) computational graphs, but also maybe down the line to think about automatically chunking jit, checkpointing autodiff etc.

I wasn't aware JIT worked so well with autray. Maybe it's actually a good idea to document this then? Same for autograd.
I was indeed thinking of something like function decorator @autoray_jit(backend). A unified interface like that would make benchmarking between libraries much easier. Support for Numba in particular would also be nice.

with_backend I am slightly in two minds about. The negatives are it's just another way to do something one can already do - i.e. it doesn't really solve any problems I have come across so far.
The other is that it adds more overhead to do, which I feel is crucial to keep low since it often called millions+ of times. Though that might be cleverly avoided somethow.
However if it had some crucial use case and little to no overhead I could be tempted.

I do think the overhead would be minimal, but I can't think of a particular convincing use case. If you're creating a lot of arrays it's annoying to always supply like=backend as an argument, and with_backend would create cleaner code. I thought of it mostly as a nice-to-have feature.

Another thing I just noticed is that when calling ar.do("where", X==0), for numpy and torch we get a list of tuples; one for each dimension of the array, whereas for tensorflow we get a (Nxd) array, with d the number of dimensions.
This can lead to cryptic error messages, so maybe it's a good idea to translate the output of this function to conform to numpy's form? Even transposing the resulting array would essentially fix it.

I wasn't aware JIT worked so well with autray. Maybe it's actually a good idea to document this then? Same for autograd.
I was indeed thinking of something like function decorator @autoray_jit(backend). A unified interface like that would make benchmarking between libraries much easier. Support for Numba in particular would also be nice.

Yeah I'll definitely have a think about this. One might even just want @autoray.jit with backend inference. Numba however is much more tricky as it's doesn't trace the function and then compile, it inspects the actual code.


Auto submit on copy from clipboard

When the code is copied from clipboard, if have the right length, it is submitted.

@fabiohfab can you please submit the changes without changing the format and indentation? Thanks!

Hi @fabiohfab We appreciate your effort in submitting the PR. Since we haven't hear back from your for a while, and PR https://github.com/Twotalltotems/react-native-otp-input/pull/58 that resolves the same issue had been merged. We are closing this PR. Thank you.


Forest colour changes between z11 and z12

z11
z12

Why does the forest colour change between z11 and z12?
Using the darker colour on higher zooms as well would look nicer and more consistent in my opinion.

Hi thanks for bringing this up. From z9 to z11 raster landcover data (https://worldcover2021.esa.int/) is used. From z12 OSM data is used. The transition isn't tuned up very well indeed. Will check.

To me it looks like changing the colour value for forest here would do the trick.
Using a colour similar to the one in the raster landcover data would also make the colour choices more consistent on the higher zoom levels, with a lighter colour for grass and a darker one for forest. And it would look more similar to OSM Carto then.

Very much appreciate for opening this ticket. The forest color you pointed to somehow was different than it should be (the color in carto style). Since the impact is quite visible and big (all forest and wood at z>=11), ideally we should render all forests again.
Closing the ticket and will start to render forest by areas.

Thanks!


Need assistance with vGPU not displaying after addons applied

I recently deployed Harvester v1.3.1 to our Supermicro X10DRT-PT with dual Xeon E5-2687W v4 @ 3.00GHz and 924G of ram.  I have a Tesla M60 installed that I am trying to setup for vGPU access:
  *-display UNCLAIMED        description: VGA compatible controller        product: GM204GL [Tesla M60]        vendor: NVIDIA Corporation        physical id: 0        bus info: pci@0000:84:00.0        version: a1        width: 64 bits        clock: 33MHz        capabilities: pm msi pciexpress vga_controller cap_list        configuration: latency=0        resources: memory:fa000000-faffffff memory:e0000000-efffffff memory:f0000000-f1ffffff ioport:f000(size=128)
 *-display UNCLAIMED        description: VGA compatible controller        product: GM204GL [Tesla M60]        vendor: NVIDIA Corporation        physical id: 0        bus info: pci@0000:85:00.0        version: a1        width: 64 bits        clock: 33MHz        capabilities: pm msi pciexpress vga_controller cap_list        configuration: latency=0        resources: memory:f9000000-f9ffffff memory:c0000000-cfffffff memory:d0000000-d1ffffff ioport:e000(size=128)
I have verified that SR-IOV is enabled via bios settings and I have deployed the [nvidia-driver-toolkit] and [pcidevices-controller] addons.   Here is the configuration I used for the [nvidia-driver-toolkit] addon.

Both the [nvidia-driver-toolkit] and [pcidevices-controller] addons show enabled.

But there are no SR-IOV GPU Devices shown

I can see SR-IOV Network devices so I believe the service is activated properly

Is there a specific step that I am missing to enable vGPU?  I have a support bundle that I can provide for my configuration.
Any assistance with this issue is very much appreciated.

Hi @psarinana-fsb, may you generate support-bundle for this issue?
https://docs.harvesterhci.io/v1.3/troubleshooting/harvester/#generate-a-support-bundle
cc @ibrokethecloud, could you take a look for this issue? Thank you.

Certainly, attached is the support bundle for my configuration
supportbundle_19b2afc0-d7ae-4be5-becc-c8b3d0ed9a04_2024-07-16T16-18-54Z.zip

this is likely because the Tesla M60 is based on the older Maxwell architecture. NVIDIA sriov vgpu capability is available on GPU's based on Ampere architecture or later

Thanks for the response.  I replaced the M60 with a T4 and I reinstalled Harvester.  I am still not seeing the vGPU resource.  Here is a screenshot of the PCI bios settings verifying SR-IOV:

I see the TESLA T4 in PCI devices in Harvester

And I have verified the addons are enabled

I am really not sure what I am missing here, I am providing another support bundle for this configuration.  Thanks again for looking into this
supportbundle_efc4c818-c19b-4c04-81d6-b9ab660e9719_2024-07-31T15-42-00Z.zip

Closed by mistake, just reopened

the Tesla T4 does not support SRIOV vGPU devices. It supports legacy vgpu devices as documented here: https://documentation.suse.com/sles/15-SP5/html/SLES-all/article-nvidia-vgpu.html


Implementing 'Append row' option

Working with this plugin, i missed an option to add a new row to table, so the user can perform an insertion too.
So i have made a little update on core script :)
Call $(element).editableTableWidget('appendrow') to append a new editable row to table

Thank you @davelima


[ASE] The colors of the edge pixels are darker than the center pixels. How can I rectify this problem?

Your dataset is really good! I'm trying to use ASE as my training data for the novel view synthesis task.
But I found a problem:
In the circle, the colors of the edge pixels (I'm not saying about the black border) are much darker than the center pixels(I have undistorted the images). So the corresponding pixels in the edges are not fully view-consistent between neighbor views.
Do you know how to rectify this inconsistent brightness problem?

Hello!
ASE is designed to produce accurate simulations of Aria output. The RGB camera has a very small fisheye lens on it. This means that we also simulate the Vignette of the Cameras. As you might be aware, most fisheye lenses produce a pronounced variation of brightness, more information can be found Here: (non affiliated link!)
The good news is that the variance in brightness is static. It could be reduced by creating a gradient (by inverting an image similar to this and multiplying it with the ASE image.
I hope that helps!

Thanks for your reply! However, I still do not know how to compute this gradient in ASE data accurately.

Hello!
ASE is designed to produce accurate simulations of Aria output. The RGB camera has a very small fisheye lens on it. This means that we also simulate the Vignette of the Cameras. As you might be aware, most fisheye lenses produce a pronounced variation of brightness, more information can be found Here: (non affiliated link!)
The good news is that the variance in brightness is static. It could be reduced by creating a gradient (by inverting an image similar to this and multiplying it with the ASE image.
I hope that helps!
Hi!
Is there any specific method to compute this relative illumination value of each pixel? I'm not familiar with fisheye cameras.

let me see if I can generate a gradient, standby!

Hi! Do you have any clue about relative illumination computation?

We calculate the distortion and then apply a Vignette, so the relative illumination is a function of combining a "normal" but distorted image with a vignette image.

This should re-flatten the lens based roll off.
The top right of the image is "up" so depending on how its applied you might need to rotate it to line it up.

Thanks for your help. I previously wrote the code about preprocessing ASE fisheye data including undistort and rotation.  I beg your help. Which line code should I revise to change the relative illumination?
import matplotlib.colors as colors
import matplotlib.pyplot as plt
import numpy as np
import plotly.graph_objects as go
from pathlib import Path
import os
from PIL import Image
from scipy.spatial.transform import Rotation as R
from projectaria_tools.projects import ase
from projectaria_tools.core import data_provider, calibration
from projectaria_tools.core.image import InterpolationMethod
from readers import read_points_file, read_trajectory_file, read_language_file
import cv2
from tqdm import tqdm
import os, sys, json
from multiprocessing import Pool

def distance_to_depth(K, dist, uv=None):
    if uv is None and len(dist.shape) >= 2:
        # create mesh grid according to d
        uv = np.stack(np.meshgrid(np.arange(dist.shape[1]), np.arange(dist.shape[0])), -1)
        uv = uv.reshape(-1, 2)
        dist = dist.reshape(-1)
        if not isinstance(dist, np.ndarray):
            import torch
            uv = torch.from_numpy(uv).to(dist)
    if isinstance(dist, np.ndarray):
        # z * np.sqrt(x_temp**2+y_temp**2+z_temp**2) = dist
        uvh = np.concatenate([uv, np.ones((len(uv), 1))], -1)
        uvh = uvh.T # N, 3
        temp_point = np.linalg.inv(K) @ uvh # 3, N
        temp_point = temp_point.T # N, 3
        z = dist / np.linalg.norm(temp_point, axis=1)
    else:
        uvh = torch.cat([uv, torch.ones(len(uv), 1).to(uv)], -1)
        temp_point = torch.inverse(K) @ uvh
        z = dist / torch.linalg.norm(temp_point, dim=1)
    return z

def transform_3d_points(transform, points):
    N = len(points)
    points_h = np.concatenate([points, np.ones((N, 1))], axis=1)
    transformed_points_h = (transform @ points_h.T).T
    transformed_points = transformed_points_h[:, :-1]
    return transformed_points

def aria_export_to_scannet(scene_id):
    src_folder = Path("/group/40033/public_datasets/3d_datasets/aria/ase_data/"+str(scene_id))
    trgt_folder = Path("/group/40033/public_datasets/3d_datasets/aria/ase_preprocessed_data/"+str(scene_id))
    trgt_folder.mkdir(parents=True, exist_ok=True)
    SCENE_ID = src_folder.stem
    print("SCENE_ID:", SCENE_ID)

    scene_max_depth = 0
    scene_min_depth = np.inf
    Path(trgt_folder, "intrinsic").mkdir(exist_ok=True)
    Path(trgt_folder, "pose").mkdir(exist_ok=True)
    Path(trgt_folder, "depth").mkdir(exist_ok=True)
    Path(trgt_folder, "color").mkdir(exist_ok=True)

    rgb_dir = src_folder / "rgb"
    depth_dir = src_folder / "depth"
    # Load camera calibration
    device = ase.get_ase_rgb_calibration()
    # Load the trajectory using read_trajectory_file()
    trajectory_path = src_folder / "trajectory.csv"
    trajectory = read_trajectory_file(trajectory_path)

    num_frames = len(list(rgb_dir.glob("*.jpg")))
    Path('./debug').mkdir(exist_ok=True)
    for frame_idx in range(num_frames):
        frame_id = str(frame_idx).zfill(7)
        rgb_path = rgb_dir / f"vignette{frame_id}.jpg"
        depth_path = depth_dir / f"depth{frame_id}.png"
        depth = Image.open(depth_path) # uint16
        rgb = cv2.imread(str(rgb_path), cv2.IMREAD_UNCHANGED)
        depth = np.array(depth)
        scene_min_depth = min(depth.min(), scene_min_depth)
        inf_value = np.iinfo(np.array(depth).dtype).max
        depth[depth == inf_value] = 0 # consider it as invalid, inplace with 0
        T_world_from_device = trajectory["Ts_world_from_device"][frame_idx] # camera-to-world
        assert device.get_image_size()[0] == 704
        # https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/advanced_code_snippets/image_utilities
        pinhole = calibration.get_linear_camera_calibration(
            # device.get_image_size()[0],
            # device.get_image_size()[1],
            # device.get_focal_lengths()[0],
            512,
            512,
            150,
            "camera-rgb",
            device.get_transform_device_camera() # important to get correct transformation matrix in pinhole_cw90
            )
        # distort image
        rectified_rgb = calibration.distort_by_calibration(np.array(rgb), pinhole, device, InterpolationMethod.BILINEAR)
        # raw_image = np.array(depth) # Will not work
        depth = np.array(depth).astype(np.float32) # WILL WORK
        rectified_depth = calibration.distort_by_calibration(depth, pinhole, device)

        rotated_image = np.rot90(rectified_rgb, k=3)
        rotated_depth = np.rot90(rectified_depth, k=3)
        increase_light = True
        if increase_light:
            rotated_image = cv2.cvtColor(rotated_image,cv2.COLOR_BGR2HSV)
            h,s,v = cv2.split(rotated_image)
            v1 = np.clip(cv2.add(1*v, 30), 0, 255)
            rotated_image = np.uint8(cv2.merge((h,s,v1)))
            rotated_image = cv2.cvtColor(rotated_image,cv2.COLOR_HSV2BGR)

        cv2.imwrite(str(Path(trgt_folder, "color", f"{frame_id}.jpg")), rotated_image)
        # TODO: check this
        plt.imsave(Path(f"./debug/debug_undistort_{frame_id}.png"), np.uint16(rotated_depth), cmap="plasma")
        # Get rotated image calibration
        pinhole_cw90 = calibration.rotate_camera_calib_cw90deg(pinhole)
        principal = pinhole_cw90.get_principal_point()
        cx, cy = principal[0], principal[1]
        focal_lengths = pinhole_cw90.get_focal_lengths()
        fx, fy = focal_lengths
        K = np.array([ # camera-to-pixel
            [fx, 0, cx],
            [0, fy, cy],
            [0, 0, 1.0]])

        c2w = T_world_from_device
        c2w_rotation = pinhole_cw90.get_transform_device_camera().to_matrix()
        c2w_final = c2w @ c2w_rotation   # right-matmul!
        cam2world = c2w_final
        # distance-to-depth
        rotated_depth = distance_to_depth(K, rotated_depth).reshape((rotated_depth.shape[0], rotated_depth.shape[1]))#.reshape((dpt.shape[0], dpt.shape[1]))
        rotated_depth = np.uint16(rotated_depth)

        cv2.imwrite(str(Path(trgt_folder, "depth", f"{frame_id}.png")), rotated_depth) # cmap="gray", vmin=0, vmax=255
        scene_max_depth = max(scene_max_depth, float(depth.max()))
        Path(trgt_folder, "min_depth.txt").write_text(f"{scene_min_depth * 1.0 / 1000}")
        Path(trgt_folder, "max_depth.txt").write_text(f"{scene_max_depth * 1.0 / 1000}")
        Path(trgt_folder, "intrinsic", "intrinsic_color.txt").write_text(f"""{K[0][0]} {K[0][1]} {K[0][2]} 0.00\n{K[1][0]} {K[1][1]} {K[1][2]} 0.00\n{K[2][0]} {K[2][1]} {K[2][2]} 0.00\n0.00 0.00 0.00 1.00""")
        Path(trgt_folder, "pose", f"{frame_id}.txt").write_text(f"""{cam2world[0, 0]} {cam2world[0, 1]} {cam2world[0, 2]} {cam2world[0, 3]}\n{cam2world[1, 0]} {cam2world[1, 1]} {cam2world[1, 2]} {cam2world[1, 3]}\n{cam2world[2, 0]} {cam2world[2, 1]} {cam2world[2, 2]} {cam2world[2, 3]}\n0.00 0.00 0.00 1.00""")

if __name__ == "__main__":
    aria_export_to_scannet(scene_id=0)

multiply it together with the rgb image just as its loaded:
        rgb = cv2.imread(str(rgb_path), cv2.IMREAD_UNCHANGED)
        anti_vignette = cv2.imread('path_to_anti_vignette.jpg')
        rgb = cv2.multiply(rgb,anti_vignette,scale=1.0)

that should flatten it out. (again, i'm not sure of the rotation, so you might need to rotate the anit-vignette image left by 90 degrees for it to line up properly. )

We calculate the distortion and then apply a Vignette, so the relative illumination is a function of combining a "normal" but distorted image with a vignette image.
This should re-flatten the lens based roll off.
The top right of the image is "up" so depending on how its applied you might need to rotate it to line it up.

Hi! it seems that this anti-vignette is normalized(min value is 0 and max value is 255 with data type np.uint8). I use it to change RGB images but the color overflows. Could you tell me how to reverse it to a true value?

import matplotlib.colors as colors
import matplotlib.pyplot as plt
import numpy as np
# import plotly.graph_objects as go
from pathlib import Path
import os
from PIL import Image
import cv2
import os, sys, json
scene_id = 0
vignette_path = Path("/group/40033/public_datasets/3d_datasets/aria/data/anti_vignette.png")
anti_vignette = cv2.imread(str(vignette_path)) # , cv2.IMREAD_UNCHANGED

src_folder = Path("/group/40033/public_datasets/3d_datasets/aria/ase_data/"+str(scene_id))
rgb_dir = src_folder / "rgb"

frame_idx = 0
frame_id = str(frame_idx).zfill(7)
rgb_path = rgb_dir / f"vignette{frame_id}.jpg"
rgb = cv2.imread(str(rgb_path), cv2.IMREAD_UNCHANGED)

rgb = cv2.multiply(rgb, anti_vignette,scale=1.0)
cv2.imwrite("./debug.jpg", rgb)

Hi! @captain-sysadmin  It seems that the given anti-vignette is normalized(min value is 0 and max value is 255 with data type np.uint8). If I use this anti-vignette to multiply RGB image, the RGB image will overflow (the value exceeds the data range of [0, 255]). Do you know how to resolve it?

Hello!
As you can see, because we are multiplying white(or very near white) with another colour other than black, we quickly overflow and clip.
You can try using cv2.[addWeighted](https://docs.opencv.org/4.x/d2/de8/group__core__array.html#ga10ac1bfb180e2cfda1701d06c24fdbd6) instead of multiply. So:
rgb = cv2.multiply(rgb, anti_vignette,scale=1.0)

becomes
alpha = 1.0
beta = 1.0
gamma = 1.0
rgb = cv2.addWeighted(rgb, alpha, anti_vignette, beta, gamma)

changing the alpha and beta would allow you to alter the mix between the two images, and gamma should allow you to control clipping

Thanks for your reply!
I can get the normal image.

But I still have a question. It seems the given code above will re-weight all three channels (RGB) with the anti-vignette image. So the image looks too bright.
I tried to re-weight the image in HSV space and revise only the V value. The brightness becomes normal. But it will get a checkboard artifact near the edge. Do you know how to alleviate this problem?

rgb = rgb.astype(np.float32) / 255           # go to 32-bit float on 0..1
anti_vignette = anti_vignette.astype(np.float32) / 255
new_rgb = cv2.cvtColor(rgb,cv2.COLOR_BGR2HSV)
h,s,v = cv2.split(new_rgb)
new_v = cv2.addWeighted(v, alpha, anti_vignette[:, :, 0], beta, gamma)
new_rgb = cv2.merge((h,s,new_v))
new_rgb = cv2.cvtColor(new_rgb, cv2.COLOR_HSV2BGR)
new_rgb = np.uint8(np.clip(new_rgb*255, 0, 255))
rgb = new_rgb

I tried to re-weight the image in HSV space and revise only the V value. The brightness becomes normal. But it will get a checkboard artifact near the edge. Do you know how to alleviate this problem?

Off the top of my head you might be able to lower the V of vignette before adding it to the RGB image?

Each channel of anti_vignette is equal (R=G=B). So V=max(R, G, B) of vignette is vignette[:, :, 0].
anti_vignette =  np.uint8(anti_vignette.astype(np.float32) * 1.0 / 3.0)
rgb = cv2.addWeighted(rgb, alpha, anti_vignette , beta, gamma)

The result is still a little white.

Please have a look at this
https://github.com/facebookresearch/projectaria_tools/pull/125

Thanks!

Please have a look at this #125


Functions Not Supported

(quant 30) is not supported by this project at all, and yet I don't think it is mentioned.
Is there any other syntax that is not supported and not listed?

I stopped working on javascript parser of Alda after I got Alda WASM port working.
I am not using Alda anymore personaly as I am more focused on live music today.


Get errors in quickstart

When following the steps in the quickstart, I get some errors.
To reproduce:

OS: ubuntu 20
git clone git@github.com:alan-turing-institute/privacy-sdg-toolbox.git
cd privacy-sdg-toolbox
poetry install (python 3.9)
then create a new notebook, run the notebook with the project .venv (I use VS code), and follow the steps in the quickstart

The specific errors I get
First, there is something wrong in this cell:
from sklearn.ensemble import RandomForestClassifier

attacker = tapas.attacks.ShadowModellingAtack(
   FeatureBasedSetClassifier(
      tapas.attacks.NaiveSetFeature() + tapas.attacks.HistSetFeature() + tapas.attacks.CorrSetFeature(),
      RandomForestClassifier(n_estimators = 100)
   ),
   label = "Groundhog"
)

RandomForestClassifier vs FeatureBasedSetClassifier? (why do we do the import first?)
FeatureBasedSetClassifier is never imported
either way, when running the cell, I get AttributeError: module 'tapas.attacks' has no attribute 'ShadowModellingAtack'

Second, when training the Groundhog attack:
attacker = tapas.attacks.GroundhogAttack()
attacker.train(threat_model, num_samples=1000)

I get a value error:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[19], line 3
      1 attacker = tapas.attacks.GroundhogAttack()
----> 3 attacker.train(threat_model, num_samples=1000)

File ~/repositories/projects/GANS/privacy-sdg-toolbox/tapas/attacks/shadow_modelling.py:87, in ShadowModellingAttack.train(self, threat_model, num_samples)
     84 synthetic_datasets, labels = threat_model.generate_training_samples(num_samples)
     86 # Fit the classifier to the data.
---> 87 self.classifier.fit(synthetic_datasets, labels)
     88 self.trained = True

File ~/repositories/projects/GANS/privacy-sdg-toolbox/tapas/attacks/set_classifiers.py:144, in FeatureBasedSetClassifier.fit(self, datasets, labels)
    143 def fit(self, datasets: list[Dataset], labels: list[int]):
--> 144     self.classifier.fit(self.features(datasets), labels)

File ~/repositories/projects/GANS/privacy-sdg-toolbox/tapas/attacks/set_classifiers.py:85, in SetFeature.__call__(self, *args, **kwargs)
     84 def __call__(self, *args, **kwargs):
---> 85     return self.extract(*args, **kwargs)

File ~/repositories/projects/GANS/privacy-sdg-toolbox/tapas/attacks/set_classifiers.py:108, in CombinedSetFeatures.extract(self, dataset)
    107 def extract(self, dataset: Dataset) -> np.array:
--> 108     return np.concatenate([f.extract(dataset) for f in self.features], axis=1)

File ~/repositories/projects/GANS/privacy-sdg-toolbox/tapas/attacks/set_classifiers.py:108, in (.0)
...
--> 101     cidx = [categories.index(c) for c in col_data]
    102     col_data_onehot[np.arange(len(col_data)), cidx] = 1
    104     return col_data_onehot

ValueError: nan is not in list

Hi, sorry for the late reply.

You are correct, the quickstart is missing tapas.attacks. before FeatureBasedSetClassifier.
There is also a typo (missing a t in ShadowModellingAttack)!

These will be fixed shortly.
I am a bit surprised by the second error. What dataset are you using? (we do not have support for NaNs at this point).

Hi, thanks! I just found that the second error was a mistake on my part from preparing the UK census data. Sorry!


About flyweight pattern

Hi! Thanks for the collection of design patterns in Go!
Unfortunately, you have an issue with the "flyweight" pattern here. Please take a look at this example in playground https://play.golang.org/p/fZM6N97Y9yX
In case if something will happen with golang playground I passed code here as well:
package main

import (
	"fmt"
)

type Flyweighter interface {
	GetName() string
	SetName(name string)
}

type FlyweightFactory struct {
	pool map[int]Flyweighter
}

func (f *FlyweightFactory) GetFlyweight(state int) Flyweighter {
	if f.pool == nil {
		f.pool = make(map[int]Flyweighter)
	}
	if _, ok := f.pool[state]; !ok {
		f.pool[state] = &ConcreteFlyweight{state: state}
	}
	return f.pool[state]
}

type ConcreteFlyweight struct {
	state int
	name  string
}

func (f *ConcreteFlyweight) GetName() string {
	return "My name: " + f.name
}

func (f *ConcreteFlyweight) SetName(name string) {
	f.name = name
}

func main() {
	factory := FlyweightFactory{}

	instance1 := factory.GetFlyweight(1)
	instance2 := factory.GetFlyweight(1)

	fmt.Println(instance1.GetName())
	fmt.Println(instance2.GetName())

	instance1.SetName("instance1")
	instance2.SetName("instance2")

	fmt.Println(instance1.GetName())
	fmt.Println(instance2.GetName())
}

I don't have much experience with that pattern but I believe we have the next options here:

Object shouldn't provide setters;
We should use something like COW here and each setter should create a new instance instead of mutating the current one.

Hi! You are right, I chose a bad example.
The "flyweight" can have mutable or immutable state, but not in this example.
I can provide a link to the wiki:
https://en.wikipedia.org/wiki/Flyweight_pattern#Mutable_or_immutable_extrinsic_state
I probably need to fix the example.


Implementation of OPHARV in SAMUCA

Hi @chporter,
Following your previous suggestion I've added the OPHARV subroutine in SAMUCA. Now the SAMUCA outputs are also printed on Summary, Evaluate and Overview outputs.
I've also added a calibration of CANEGRO for the Brazilian cultivar RB867515 that will be provided in the default DSSAT data for SAMUCA in file ESAL1401.SCX (here).
Please consider merging it into your samuca branch.
Cheers!

Hi @chporter, interesting to see that github apparently already handled the new commits I made after the pull request.
The pull request was made in 1st Apr but the last 4 commits in the list above were made in June when I fixed a bug in the initialization. So I think by accepting this pull request you'd automatically update your samuca branch and then you could pull request to DSSAT/dssat-csm-os afterwards.
Please let me know if you need the code to be adapted to v4.8
Cheers

Hi, Murilo;
Thanks! I’m just looking at the data now. I’ve pulled your latest code into my SAMUCA branch and I’m running it. All looks good so far. I like to handle the pull requests on my local machine first. Once I merge it into develop, the pull-request is automatically closed. Git is pretty amazing.
I don’t think the code will need to be adapted. But the genotype files will. I can do that – it doesn’t take much time.
Thanks!
Cheryl
From: Murilodsv<EMAIL_ADDRESS>Sent: Wednesday, 22 July, 2020 12:01
To: chporter/dssat-csm-os<EMAIL_ADDRESS>Cc: Porter, Cheryl H<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [chporter/dssat-csm-os] Implementation of OPHARV in SAMUCA (#4)
[External Email]
Hi @chporterhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_chporter&d=DwMCaQ&c=sJ6xIWYx-zLMB3EPkvcnVg&r=jin_7m5pUyWQ-Rq3_pHt6w&m=vnnejpqF5dOnLUU-sDENNsXxyHGOatR3kK122jP_0IY&s=nw8uT99CLKWMleozoxeHg_VmDCNZqL4JpULNF6q0YCE&e=, interesting to see that github apparently already handled the new commits I made after the pull request.
The pull request was made in 1st Apr but the last 4 commits in the list above were made in June when I fixed a bug in the initialization. So I think by accepting this pull request you'd automatically update your samuca branch and then you could pull request to DSSAT/dssat-csm-os afterwards.
Please let me know if you need the code to be adapted to v4.8
Cheers
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_chporter_dssat-2Dcsm-2Dos_pull_4-23issuecomment-2D662538414&d=DwMCaQ&c=sJ6xIWYx-zLMB3EPkvcnVg&r=jin_7m5pUyWQ-Rq3_pHt6w&m=vnnejpqF5dOnLUU-sDENNsXxyHGOatR3kK122jP_0IY&s=Aj0XOpk8mcIR27hXFaISqAmXl-cNWDFzahnrK4fjyUQ&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AACOFGMI6SAWAAPLXASWNODR44ELBANCNFSM4LY37HDA&d=DwMCaQ&c=sJ6xIWYx-zLMB3EPkvcnVg&r=jin_7m5pUyWQ-Rq3_pHt6w&m=vnnejpqF5dOnLUU-sDENNsXxyHGOatR3kK122jP_0IY&s=u8uhCxlxdS8OW-UEu7GpOcc0EHD9J9Mt31tH6Mh98aw&e=.

Awesome!
Thanks a lot =]


Error: wave-cli: Failed to download resource "wave-cli" (brew)

brew installation failed:
==> Tapping seqeralabs/tap
Cloning into '/opt/homebrew/Library/Taps/seqeralabs/homebrew-tap'...
remote: Enumerating objects: 25, done.
remote: Counting objects: 100% (25/25), done.
remote: Compressing objects: 100% (24/24), done.
remote: Total 25 (delta 2), reused 19 (delta 0), pack-reused 0
Receiving objects: 100% (25/25), 8.43 KiB | 4.21 MiB/s, done.
Resolving deltas: 100% (2/2), done.
Tapped 2 formulae (15 files, 28KB).
==> Fetching seqeralabs/tap/wave-cli
==> Downloading https://github.com/seqeralabs/wave-cli/releases/download/v1.2.0/
curl: (22) The requested URL returned error: 404

Error: wave-cli: Failed to download resource "wave-cli"
Download failed: https://github.com/seqeralabs/wave-cli/releases/download/v1.2.0/wave-1.2.0-macos-arm64

Probably ignore -- 1.2.0 isn't available yet.

I was not able to replicate it. Please reopen if the problem persists


[CT-2304] [Bug] run-operation cannot be run with --no-version-check flag

Is this a new bug in dbt-core?

[X] I believe this is a new bug in dbt-core
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior
As a temporary exception, I need to run my currently installed DBT (i.e. 1.3.2) despite the project-driven requirement for DBT 1.3.3. When running a macro via dbt run-operation I get the following response:
15:39:30  Encountered an error while reading the project:
15:39:30    ERROR: Runtime Error
  This version of dbt is not supported with the '...' package.
    Installed version of dbt: =1.3.2
    Required version of dbt for '...': ['=1.3.3']
  Check for a different version of the '...' package, or run dbt again with --no-version-check

When I then run dbt run-operation --no-version-check, this is not accepted and I get the following error:
dbt: error: unrecognized arguments: --no-version-check

Expected Behavior
dbt run-operation --no-version-check is successful, just like running dbt run ... --no-version-check works fine
Steps To Reproduce

Using DBT 1.3.2 installed (no specific config, just a random demo project)
dbt run ... --no-version-check -> ok
dbt run-operation ... --no-version-check -> error

Relevant log output
No response
Environment
- OS: Ubuntu 22.04
- Python: 3.10.6
- dbt: 1.3.2

Which database adapter are you using with dbt?
snowflake
Additional Context
No response

Thanks for reporting this @lucianmoldovanu !
Could you try this and see if it works for you?
dbt --no-version-check run-operation {...}

Instead of:
dbt run-operation {...} --no-version-check

Example
I was able to reproduce what you are seeing, but was also able to get it to work by changing the order of --no-version-check.
First, I added require-dbt-version: 0.0.0 to dbt_project.yml.
Then I created this dummy macro within macros/my_macros.sql:
{% macro do_nothing() %}
{% endmacro %}

As it should, this complained about the version :
dbt run-operation do_nothing

Same as for you, this didn't work for me though:
dbt run-operation do_nothing --no-version-check

But this did work:
dbt --no-version-check run-operation do_nothing

In this specific case, the solution might be as simple as adding @p.version_check to here.
In the general case, we'd like to have a way that it doesn't matter if an optional flag is supplied before or after a sub-command.
See https://github.com/dbt-labs/dbt-core/issues/6913 for more context (including a problem to solve).


Update ktfmt and detekt plugins

Routine update of code check plugins

@gino-m @scolsen  PTAL?

Daemon is failing here too..could it be related to this recent change?

Daemon is failing here too..could it be related to this recent change?

Filed a separate bug to tackle gradle failure issue. For this PR, I'll ensure that the checks are passing locally and will merge this PR.


Using NLTK for tokenization

Proposed title of article
Using NLTK for tokenisation (a data preprocessing method)
Proposed article introduction
Tokenisation is a data preprocessing method that involves dividing text- data into a set of meaningful pieces depending on the task at hand. These pieces are called tokens. NLTK (Natural Language Toollkit) is a natural language processing tool written in python programming language that provides libraries for sentence and word tokenisation.
Key takeaways
Readers will be able to install NLTK on various operating systems
Readers will be able to download datasets that NLTK provides
Readers will be able to tokenize text data
Readers will be informed on other things to use tokenized data for

Good afternoon and thank you for submitting your topic to the EngEd program.
After some careful consideration it struck us that this topic may be a bit over saturated throughout other blog sites and official documentations. We typically refrain from publishing content that is covered widely on the net or other blogs. We're more interested in original, practitioner-focused content that takes a deeper dive into programming-centric concepts.
But in order to approve the topic it has to serve value to the larger developer community at large. Please feel free to suggest an alternate topic to explore. 🚀
An option and a great way to write this as an in-depth article and make it more add value to the greater developer community at large would be to walk the reader through the USE of methods and functions by building a unique, different, useful project.
If you believe your article is unique enough - please present your case to explain how it will be different from what is currently out there?
https://www.google.com/search?q=NLTK+for+tokenization&rlz=1C1CHBF_enUS891US891&sourceid=chrome&ie=UTF-8

####Alternate proposed title of article
Permissioned blockchain based databases
Proposed article introduction
The most common blockchain based databases are the permissionless blockchain based databases. General implementation of a permissioned blockchain solution is a tough one which is why most people leverage on the permissionless part of blockchain.
Key takeaways
Readers will be able to understand general blockchain based databases
Readers will be able to differentiate between permissioned and permissionless blockchain based databases
Readers will be able to differentiate between traditional databases and permissioned blockchain databases
Readers will be able to choose the right permissioned blockchain database for a particular solution
A list of permissioned blockchain databases will be stated
Conclusion
At the end of this article, blockchain enthusiasts will be able to understand the general blockchain concept and then differentiate between permissioned blockchain databases and permissionless blockchain databases


LWRP behaves odd when trying to set an attribute to nil

Given the following cookbook/resources/manage.rb:
attribute :users,    :kind_of => [Hash, Symbol, NilClass], :default => :admins

When trying to use it like this:
cookbook_manage 'foo' do
  users nil
end

The provider gets new_resource.users set to :admins
After digging a bit in the code, I noticed that the LWRP resource attribute implementation uses Chef::Mixin::ParamsValidate#set_or_return, which uses nil instead of NULL_ARG like the rest of the code.

This is a breaking change which can only be fixed in Chef 13 and will be fixed in Chef 13.  The deprecation warnings are in there now.  We cannot fix it now because there is existing code in popular cookbooks which expects this code to retain the default value for the 'property' on the resource even if its not nil (so if the default is 'foo' then it retains 'foo', we cannot set it to nil):
variable = nil
resource "whatever" do
  property variable
end

That code should all be rewritten something like:
variable = nil
resource "whatever" do
  property variable unless variable.nil?
end

But until it is, that would be a breaking change.

@lamont-granquist may I ask how this will be fixed? I'd still  want it to work the way it works currently, so that i can avoid all the unless .nil? boilerplate when doing such as
modues.each_pair do |name, cfg|
  module name
    cfg[:url]
    cfg[:instances]
  end
end

where the values default to the default values if they are unset in the hash.


Conditionally render the ACCT alert expiry date when rendering the check your answers page

Context
We received two support tickets[1] from users who were unable to submit
their application. We can see issues[2] coming through to sentry on the
'Check your answers' page before submitting a referral.
Upon inspection, it is possiblle for an ACCT alert to have a missing
dateExpires property, so this change adds a null guard against that
scenario. We render an empty string, which aligns with how we handle it
elsewhere[3].
[1]
https://mojdt.slack.com/archives/C04NJ6BATUL/p1693409827077599?thread_ts=1693409758.827739&cid=C04NJ6BATUL
https://mojdt.slack.com/archives/C04NJ6BATUL/p1693411279302619?thread_ts=1693409758.827739&cid=C04NJ6BATUL
[2]
https://ministryofjustice.sentry.io/issues/4347240074/events/39b224f1720e4e10af3e3aa7d4e6d8dc/
[3]
https://github.com/ministryofjustice/hmpps-temporary-accommodation-ui/blob/main/server/views/applications/pages/assess-placement-risks-and-needs/prison-information/acct-alerts.njk#L27
Screenshots of UI changes

Release checklist
Release process documentation.
Pre-merge checklist

[ ] Are any changes required to dependent services for this change to work?
(eg. CAS API)
[ ] Have they been released to production already?

Post-merge checklist

[ ] Manually approve
release to test
[ ] Manually approve
release to preprod
[ ] Manually approve
release to prod

 Users can't submit referrals due to a missing date


Ability to filter confidence for predicted boxes on downloaded images when tagging

It would be really great to be able to change the confidence threshold on the fly to be able to display bounding boxes in VoTT that have only high or low confidence.

This looks very similar to #26, except #26 doesn't mention the low confidence. Did you want a way to have only low confidence predictions instead of either only high confidence or both low and high confidence? The latter two are now possible, but the first is not.


Keycloak SAML login stuck

The issues #1662 and #1694 were closed, but the problem that I'm observing was not resolved. I'm opening a new issue as I don't like commenting on closed issues.
As suggested by @hsluoyz I want to show you the broken flow in a short video:

Hint: The video ends with the Casdoor spinner turning and nothing happening anymore.
The following video shows the login using Casdoor directly (to illustrate that the application is working!):

I tested this on CentOS 7.8 (as this is the preferred platform of @leo220yuyaodog). This shows the build process:

@leo220yuyaodog Please tell me what part of the deploy script didn't work for you.
Again the example can be cloned from this repository:
https://github.com/fischermario/casdoor-example/tree/saml-keycloak-6.0.1
Please make sure that you have Docker installed according to this manual:
https://docs.docker.com/engine/install/centos/
Thank you!

@fischermario can you try directly login via KeyCloak from Casdoor's org login page (so your app is not involved and no 2nd jump)?
Like: https://door.casdoor.com/login/casbin

@hsluoyz
I just tried what you suggested (use Keycloak to log in to app-built-in). Unfortunately the result is the same. Do you want me to make a video of the process?

@fischermario yes, a video will be good

@fischermario

Because of nslookup failed, so I comment the dns check.

Then I meet the problem, can you help me?

About the crash screen, it seems that Casdoor failed to parse the samlResponse from keycloak. Does your script initialize origin in app.conf.  It is required for using saml sp.
https://github.com/fischermario/casdoor-example/blob/85331fa3d9fa9f6734247c8706fa68cab99723e4/conf/casdoor.conf-template#L18

@fischermario Oh, I reproduce your problem. Maybe the problem will be resolved. It caused by the wrong config. As below picture shows: the issue url need be same as the idp attribute entityID:

or will report error when login:


#148 add optional array of private registry secrets

#148

It looks like @CrystalMethod hasn't signed our Contributor License Agreement, yet.

The purpose of a CLA is to ensure that the guardian of a project's outputs has the necessary ownership or grants of rights over all contributions to allow them to distribute under the chosen licence.
Wikipedia

You can read and sign our full Contributor License Agreement here.
Once you've signed reply with [clabot:check] to prove it.
Appreciation of efforts,
clabot

[clabot:check]

@confluentinc It looks like @CrystalMethod just signed our Contributor License Agreement. :+1:
Always at your service,
clabot

I have tried to include the requested changes - please comment.

LGTM. Thanks a lot for the PR!

Thank you @CrystalMethod


ReflectionTypeLoadException

Though rare, if your working with a library that was built against something other than say 3.5 you'll see this exception thrown which will prevent the drop down list from populating anything.
Quick solution is just to add the following to the FilterTypes method:
            Type[] typeArray = null;
            try
            {
                typeArray = assembly.GetTypes();
            }
            catch(System.Reflection.ReflectionTypeLoadException e)
            {
                Debug.Log("Types could not be extracted from assembly:" + e.Message);
            }
            if (typeArray == null)
            {
                return;
            }

Hi, thanks for the feedback. It was fixed here: https://github.com/SolidAlloy/ClassTypeReference-for-Unity


Adjust pdf view temporarily for our needs

R: @samerce

Adjust pdf view temporarily for our needs

Remove border
Make scale to full width
Pass back dimension info

lgtm


The PactBrokerLoader enforces host/port combination, throws IllegalArgumentException if port not included

In summary it requires I provide a value for  "pactbroker.host" and "pactbroker.port", if not it throws an illegalArgumentException if the port is excluded.
So a url like "https://pact.broker.com" wont be resolved unless I include 443 in "pactbroker.port".

Released 3.6.0-rc.1 with this fix.


Use a single array in SelectedSelectionKeySet

Motivation:
SelectedSelectionKeySet currently uses 2 arrays internally and users are expected to call flip() to access the underlying array and switch the active array. However we do not concurrently use 2 arrays at the same time and we can get away with using a single array if we are careful about when we reset the elements of the array.
Modifications:

Introduce SelectedSelectionKeySetSelector which wraps a Selector and ensures we reset the underlying SelectedSelectionKeySet data structures before we select
The loop bounds in NioEventLoop#processSelectedKeysOptimized can be defined more precisely because we know the real size of the underlying array

Result:
Fixes https://github.com/netty/netty/issues/6058

@lightningMan @normanmaurer - FYI

pong…
Scott Mitchell<EMAIL_ADDRESS>周四14:58写道：

@lightningMan https://github.com/lightningMan @normanmaurer
https://github.com/normanmaurer - FYI
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/netty/netty/pull/6083#issuecomment-264093754, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABmkeiXPSx3dLkUegyvp0EEFdenc_ZNvks5rDnAKgaJpZM4LBEBM
.

@lightningMan Scott is letting you know that he refactored SelectedSelectionKeySet, you can review the code here https://github.com/netty/netty/pull/6083/files

@johnou thank you for your Friendly reminder, I have reviewed the code ^^

@Scottmitch I am still not 100 % convinced we need these change and I am a bit concerned about introduce a regression as we also have no tests for this class (as we had none before as well).
So let me cut a release first without this and then we can pick this one up and see... Ok ?

Yip I'm fine with delaying this until 4.1.8.
"Need" is a loaded statement :) Things work today so I guess in that sense I agree, however this simplifies and removes complexity that doesn't seem useful. SelectedSelectionKeySetSelector.java introduced in this PR ensures that we reset the array backing the Set at the appropriate time, and so in theory this should help with correctness (unless there are bugs in my implementation). Bounding the iteration and removing duplicate null checks may also clarify functionality.

@Scottmitch haha fair enough :) So let us pick this up as first thing after the release 👍

@Scottmitch if you want to pull this in can you please at least add a testcase for the Set impl ?

@normanmaurer - Fair enough. I added a few unit tests to verify expected behavior. Let me know if you want more.

@normanmaurer - comments addressed PTAL

4.1 (795f318) 4.0 (26124f6)


GUVNOR-2466: No Option to copy any drl file to a specific package

Depends on:
https://github.com/uberfire/uberfire-extensions/pull/269
https://github.com/droolsjbpm/kie-wb-common/pull/346
https://github.com/droolsjbpm/drools-wb/pull/168
https://github.com/droolsjbpm/optaplanner-wb/pull/60

+1 to merge (I cannot merge uf-ext, so all PRs will have to be merged by @csadilek)


Update README.md

是不是误操作了？
---Original---
From: @.>
Date: Wed, Dec 18, 2024 10:08 AM
To: @.>;
Cc: @.***>;
Subject: [CoderWanFeng/PyOfficeRobot] Update README.md (PR #47)
You can view, comment on, or merge this pull request online at:
  https://github.com/CoderWanFeng/PyOfficeRobot/pull/47
Commit Summary
a77a399  Update README.md
File Changes
(1 file)
 M     README.md     (3)

Patch Links:
https://github.com/CoderWanFeng/PyOfficeRobot/pull/47.patch
https://github.com/CoderWanFeng/PyOfficeRobot/pull/47.diff
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: @.***>

是不是误操作了？

对不起 点错了  学习者 还不太会用 打扰打扰


feat: make header responsive

fix #31
What do you think?

I love it! I was thinking about a dropout menu, but this is way better!
Maybe we can keep "Github" in the top bar, and move it to the right? That way we keep a bit separation between playground links, and testing-library links.
What do you think?

Yes we can do. I was inspired by docusaurus

I have fixed. Now it is as the image below

Nice! I'll first want to get #26 merged, it's almost there. This might need to get some conflicts resolved afterwards, but we'll see.
Thanks.


Add candlestick chart example

Please correct me if I am wrong, but there does not seem to be a candlestick chart example. It would be great if a sample chart could be added.

Do you mean in the examples repo, or in codepen?
In the examples folder, the D3 example is a candlestick.
You can interact with it if you:

Clone the repo and run npm install
Run npm run start
Open your browser to localhost:8080/examples/d3.html

The only D3 example is a candlestick plot.

In case you wanted a codepen example, I have created one:
Candlestick example on CodePen

Aw, great! I did not know that the candlestick was included in the example folder because its file name (D3 ) was not providing that chart type info. Could you please rename the file name in the example folder to be self-expalnatory? Thanks for the codepen example as well.

I've renamed the example file d3.html to d3_candlestick.html


Compose(typeahead): Remove text color change on hover.

Removes hover effect on compose typeahead which causes it to change text color in dark mode.

Fixes: #29842.

Screenshots and screen captures:

Before
After

Light Mode
Dark Mode

Self-review checklist

[x] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[x] Explains differences from previous plans (e.g., issue description).
[x] Highlights technical choices and bugs encountered.
[x] Calls out remaining decisions and concerns.
[x] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[x] Each commit is a coherent idea.
[x] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[x] Visual appearance of the changes.
[x] Responsiveness and internationalization.
[x] Strings and tooltips.
[x] End-to-end functionality of buttons, interactions and flows.
[x] Corner cases, error conditions, and easily imagined bugs.

What CSS generates the hover effect that you're overriding here? And does this bug affect the light theme?

What CSS generates the hover effect that you're overriding here?

This part of code in dark_theme.css generates the hover effect (line 29).
%dark-theme {
    & a:hover {
        color: hsl(200deg 79% 66%);

        & code {
            color: hsl(200deg 79% 66%);
        }
    }

And does this bug affect the light theme?

No, it does not affect the light theme.

@timabbott Your question above has been addressed.

OK. Merged, thanks @nimishmedatwal, though @karlstolley FYI -- that base a tag CSS is probably a thing we should try to move to a CSS variable to reduce its precedence; didn't want to block this on it because it could be thorny.


Feature/saa 1083 update waiting list endpoint

Codecov Report
Patch coverage: 98.70% and project coverage change: +0.10% :tada:

Comparison is base (c776f51) 85.20% compared to head (4fbf871) 85.31%.

Additional details and impacted files
@@             Coverage Diff              @@
##               main     #508      +/-   ##
============================================
+ Coverage     85.20%   85.31%   +0.10%
- Complexity     2347     2384      +37
============================================
  Files           294      295       +1
  Lines          8592     8662      +70
  Branches        545      551       +6
============================================
+ Hits           7321     7390      +69
  Misses          994      994
- Partials        277      278       +1

Files Changed
Coverage Δ

...ivitiesmanagementapi/service/WaitingListService.kt
92.37% <98.38%> (+5.70%)
:arrow_up:

...hmppsactivitiesmanagementapi/entity/WaitingList.kt
100.00% <100.00%> (ø)

...api/model/request/WaitingListApplicationRequest.kt
100.00% <100.00%> (ø)

...del/request/WaitingListApplicationUpdateRequest.kt
100.00% <100.00%> (ø)

...ntapi/resource/WaitingListApplicationController.kt
100.00% <100.00%> (ø)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Feat: Policy multiple datatypes and tag values

resolves #
This is a:

[x] documentation update
[x] bug fix with no breaking changes
[x] new functionality
[ ] a breaking change

All pull requests from community contributors should target the main branch (default).
Description & motivation
We are implementing Masking Policies to protect the PII in our Snowflake DWH and as we are using dbt we wanted to use it to be able to manage our tag-based masking policies. This package allows us to do that, but had some limitations which I have built new functionality for.
The first limitation was not being able to set multiple policies against a single tag. This is an issue for us because we have masking policies which do the exact same thing, but just have different input and output data types. For example, masking all date parts apart from the year value in a date such as a date of birth field. We have a DATE data type version and a TIMESTAMP_NTZ version, just in case any source systems we use have it as a datetime for whatever reason. Same thing for a generic NULL mask, we don't want to have a different tag and assigned policy for each different data type possible and have to figure out which tag to apply based on the data type of the column we're applying it to. We want a single tag that we apply every data type masking policy to and it just works, whichever data type the column is.
To fix this I have implemented a new optional list var in the project for policy data types. If you don't set it for any of the tags then it will default to how it worked before - looking for a MP with the same name as the tag. However, if you set a list of values for the tag in the var, then it will loop through those and add those to the suffix of the tag name and expect those as masking policies. I have added examples of this in the getting started guide.
The second limitation was not being able to set values for tags. We need this because we use the function system$get_tag_on_current_column() to be able to create a dynamic masking policy rather than creating multiple different masking policies. The example here is if we wanted to have a MP which only shows the last x characters, we don't want to create a masking policy for last 1 char, last 2 chars, last 3 chars, etc. Therefore this allows us to have a single masking policy and use the value assigned on the tag to make its application dynamic. Again, if you don't use the functionality that I have created in this new feature, then it defaults to how it worked before, assigning the name of the column as the tag value.
This is done by using the ~ separator when assigning the tag in the model schema yml file after the tag name, then defining the value. If you use the ~ character in any of your tag names, there is also an optional var defined which allows you to choose another character.
The unapply_mp_from_tags and drop_tags macros both did not work for their intended purpose (I'm basing this off their name), so I have fixed these and they now actually do as intended. This is as well as a few other macros and functions, which didn't seem to fully work as expected.
I also added some notes in the contribution guide as doing this in Windows, I came across multiple issues that took a while to figure out and resolve. I fixed the dbt-utils package version as I was having issues with running dbt deps, but this can be changed back if needed.
I have updated all documentation and tests, running through the integration tests multiple times to make sure it works as intended. Again, if you don't use any features in this release, then it works as it did prior to these new features being added.
Checklist

[ ] This code is associated with an Issue which has been triaged and accepted for development
[x] I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)

[x] Snowflake

[x] I have updated the README.md (if applicable)
[x] I have added tests & descriptions to my models (and macros if applicable)

Thanks @ChrisBRAC
The first sign looks impressive! Great job here and thanks for actively looking into this new feat!
Let me review and get back to you as soon as possible!
Thanks & sorry for this late resp 🙌

Thanks @ChrisBRAC again! I will recheck it and get back to you tmr. If it's all good, let go v1.1 right away!

I removed some of the information from the tag comment because it was related to the specific column it was found on in the config, however if it is applied to multiple columns, this is fairly useless info.

This is perfect, the prev intention is 1 tag - 1 masking policy, but with this new thing, I see it to be useless too

Also, not sure how useful the target name is in the comment of the tag either?

It's intentional to support multiple environments as we normally use dev,ci,uat,prod in the profiles.yml. Imagine when you'd like to manually clean up the tags, in order to mitigate the human mistake dropping prod tags, for example.

Run CI manually and get ✅ result (TODO: be able to run CI with User keypair instead)

Awesome works! All looks great to me except a minor thing that verify_if_.. models are having duplicates, perhaps due to multiple data types in one masking policy, which I believe can be resolved quickly.
Thank you very much @ChrisBRAC for your contribution.
cc: @il-dat

FYI @ChrisBRAC @thu-IL
I just pushed some commits to:

upgrade the dev deps
fix the unapply & drop macros

For (2), the original intention is to help for cleaning up the existing DWH tags&masking policies, including ones which are not managed by dbt-tags package. Since they're not perfectly working though, so let's move forward with current limitation (already mentioned in doacs) and I created #14 for handling this need later on.
@ChrisBRAC: Could you help to look at the changes again and Thu's comments, feel free to commit your new changes based on them if necessarily. I don't think there is a blocker now to release v1.1, but just let me know.
Thank you!

@il-dat
Don't know if you want to incorporate the unapply and drop macros as part of the poe dbt-tags-test run? I should've checked that it was actually in it as I assumed it was being tested and so missed these errors. Thanks for picking them up.
Thinking about that intention it might be risky to try and have a macro which removes all tags and MPs that aren't a part of this package, as it may unknowingly remove some manual tags to objects in Snowflake, non-pii related or it could remove some manually applied policies. So it might be worth keeping it to dbt-tags tags only for now?

@il-dat Don't know if you want to incorporate the unapply and drop macros as part of the poe dbt-tags-test run? I should've checked that it was actually in it as I assumed it was being tested and so missed these errors. Thanks for picking them up.
Thinking about that intention it might be risky to try and have a macro which removes all tags and MPs that aren't a part of this package, as it may unknowingly remove some manual tags to objects in Snowflake, non-pii related or it could remove some manually applied policies. So it might be worth keeping it to dbt-tags tags only for now?

@ChrisBRAC make sense with the first point, will add those as part of poe dbt-tags-test command or a similar one
For the 2nd point, it should be separate macros to support that purpose, and yes, it should be keeping it to dbt-tags tags only for now 👍

v1.1 should be GA now 🎉
release | docs | hub


Need a Font Awesome icon for Harvest

Want to provide users an icon for Harvesting.

@fossygirl @jeffmcaffer  what about one of the available leaf icons as a start?

@jeffmcaffer I like the leaf, any thoughts? 😄

I've searched into the fontawesome library and i hadn't found anything suitable for "harvesting".
So i've followed the suggestion by @jeffmcaffer , and these are what i've found:

@fossygirl is this icon supposed to be used in the Harvest Button into the Harvest Page only?

@jeffmcaffer @fossygirl any updates on this?

@fossygirl @jeffmcaffer any updates on this?

You can just ignore this for now, we’ll talk about it next week.
From: Simone Torrisi<EMAIL_ADDRESS>Reply-To: clearlydefined/website<EMAIL_ADDRESS>Date: Sunday, October 28, 2018 at 11:08 PM
To: clearlydefined/website<EMAIL_ADDRESS>Cc: Carol Smith<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [clearlydefined/website] Need a Font Awesome icon for Harvest (#260)
@fossygirlhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffossygirl&data=02|01|carols%40microsoft.com|91c3826d25cb4c63469208d63d64f162|72f988bf86f141af91ab2d7cd011db47|1|0|636763901092674554&sdata=glDYN3%2B9EmEprUIqtgSnLGdqD6i8s8B55OMgznfqMiE%3D&reserved=0 @jeffmcafferhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjeffmcaffer&data=02|01|carols%40microsoft.com|91c3826d25cb4c63469208d63d64f162|72f988bf86f141af91ab2d7cd011db47|1|0|636763901092674554&sdata=VF00g1jUwNCkQJZcaPT2pcqY3iDaonasjO8GYcvMGeU%3D&reserved=0 any updates on this?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fclearlydefined%2Fwebsite%2Fissues%2F260%23issuecomment-433798090&data=02|01|carols%40microsoft.com|91c3826d25cb4c63469208d63d64f162|72f988bf86f141af91ab2d7cd011db47|1|0|636763901092684562&sdata=RKbEbTrbkoOtt%2B74FJzspxONYC4X%2BwR8nkjMhBj54%2FA%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAQTxJJpaCudoqwwvHMZ26aa1fLfmZFrVks5upptagaJpZM4WMIeC&data=02|01|carols%40microsoft.com|91c3826d25cb4c63469208d63d64f162|72f988bf86f141af91ab2d7cd011db47|1|0|636763901092684562&sdata=HdUsG0YL%2FaIW27x9uKeWDnYN2eg%2BfiYYjzGcxcbXGCY%3D&reserved=0.

@jeffmcaffer Have you made a decision here? Let's just pick one and close this out.

@jeffmcaffer @fossygirl thanks for the decision.
I guess it should be added to the Harvest button in the Harvest page, is that right?

@storrisi @jeffmcaffer Is this implemented and we can close? This is a tiny thing and I'd really, really like to just be done with it.

@fossygirl i'm able to do it in a short, but i need to know where this icon should be added

@jeffmcaffer @storrisi What's the latest on this issue? I see that assigned to Jeff. Does that mean that Jeff will place the icon? If not, please re-assign to @storrisi and we'll get this sorted asap.

I'll close this.

@fossygirl Thanks for following up/Closing on this. Does that mean that the issue have been already addressed?


recordPath doesn't make any sense

Which version are you using?
1.9.3
Which operating system are you using?
Linux amd64 Docker
Describe how to replicate the issue
I'm running the following docker image of MediaMTX in Kubernetes: bluenviron/mediamtx:1.9.3
The config is set to
paths:
  all_others:
    record: yes
    recordPath: ./recordings/%Y-%m-%d/%path--%Y-%m-%d--%Hh%M
    recordSegmentDuration: 30m
    recordDeleteAfter: 0s

But when checking the files being created, I see
$ TZ=utc ls -l
total 68176
-rw-r--r-- 1 root root 13984850 Nov 21 19:34 mycamera--2024-11-21--19h04.mp4
-rw-r--r-- 1 root root 13977877 Nov 21 20:04 mycamera--2024-11-21--20h04.mp4
-rw-r--r-- 1 root root 13979416 Nov 21 20:34 mycamera--2024-11-21--21h04.mp4
-rw-r--r-- 1 root root 13961680 Nov 21 21:04 mycamera--2024-11-21--22h04.mp4
-rw-r--r-- 1 root root 13892649 Nov 21 21:34 mycamera--2024-11-21--23h04.mp4

The first filename makes sense. Recording started at 19h04 and took 30 minutes,  hense the modification date of 19:34. But then the follow up filenames don't make any sense anymore. Especially if you look at the last one. It's like it's from the future 😮
Server logs
No response
Network dump
No response

I think the problem is the recordSegmentDurction: 30m configuration. MediaMTX will create segments that are at least 30 minutes long. As the example configuration suggests, try a lower value, such as `1s. '

Thanks for reporting the issue, this is fixed by #4062


Make it possible to install openshift-routes in a different namespace than "cert-manager" using oc process

This is a continuation of https://github.com/cert-manager/openshift-routes/pull/32. This PR focuses on a very precise request coming from people relying on the oc process command to substitute variables such as $NAMESPACE in the static manifest.
To test this PR, run the following:
oc process -f https://raw.githubusercontent.com/maelvls/openshift-routes/different-namespace/deploy/static/cert-manager-openshift-routes.yaml \
  -p NAMESPACE=<NAMESPACE_NAME> \
  | oc create -f -

Closing as I wasn't able to find the time to finish this. Please feel free to take over!


EPIC: Add tests for all methods

Description
Add test for all methods in the framework

[x] App test
[x] Context test
[x] Group test
[x] Middleware test
[x] Route test
[x] Router test

Resolved


Cannot Run on Fedora 39

Due to the requirement of Speech Dispatcher v0.7.0, the project cannot effectively be run on modern Fedora distributions or its derivatives. Attempting to run a simple Hello world test will result in the following error:
error: failed to run custom build command for `speech-dispatcher-sys v0.7.0`

Caused by:
  process didn't exit successfully: `/home/[user]/Documents/Projects/Rust/HumanInteractionTests/target/debug/build/speech-dispatcher-sys-7bc85cd666e6bccd/build-script-build` (exit status: 101)
  --- stdout
  cargo:rustc-link-lib=speechd

  --- stderr
  wrapper.h:1:10: fatal error: 'speech-dispatcher/libspeechd.h' file not found
  thread 'main' panicked at /home/[user]/.cargo/registry/src/index.crates.io-6f17d22bba15001f/speech-dispatcher-sys-0.7.0/build.rs:22:10:
  called `Result::unwrap()` on an `Err` value: ClangDiagnostic("wrapper.h:1:10: fatal error: 'speech-dispatcher/libspeechd.h' file not found\n")
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

Distributions like Fedora and its derivatives are bundled with speech-dispatcher-0.11.5-2 or higher. A potential solution is to increase the Speech Dispatcher version requirement or check for multiple versions.

Please try and install this from the official fedora repo : speech-dispatcher-devel. Using: sudo dnf install speech-dispatcher-devel

This package worked. Thank you.


Add how to build the docs instructions

For new contributors it may be helpful to have instructions on building docs, this way one can try something before making the PR. If this does not sound like the expected workflow, we need such description to beware of the workflow.

Hey @srcolinas, that is a great idea! We should add it to the README. The easiest way to setup the project for development is to run
poetry install
```bash
We use `mkdocs` for the documentation, you can start the local server by running
```bash
mkdocs serve


chore(metrics): support and default to . separator

metrics-exporter-prometheus will replace any invalid character with _ to match prometheus data model.
See Behavior Section

Codecov Report

Merging #1018 (0d80d67) into main (01cb6c0) will decrease coverage by 0.01%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main    #1018      +/-   ##
==========================================
- Coverage   73.90%   73.90%   -0.01%
==========================================
  Files         300      300
  Lines       32916    32923       +7
==========================================
+ Hits        24328    24332       +4
- Misses       8588     8591       +3

Flag
Coverage Δ

unit-tests
73.90% <0.00%> (-0.01%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

crates/metrics/metrics-derive/src/expand.rs
84.30% <0.00%> (-2.22%)
:arrow_down:

crates/net/dns/src/resolver.rs
78.94% <0.00%> (ø)

crates/net/network/src/peers/manager.rs
83.42% <0.00%> (+0.18%)
:arrow_up:

crates/net/network/src/session/active.rs
84.65% <0.00%> (+0.36%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Partial Save Not Supported on DynamoDB2 Tables with Range Key

Partial Save doesn't seem to be supported for DynamDB2 tables that have a range key.

Fixed with #499


buildkite-agent service: declarative hooks and extraConfig option

Motivation for this change
I thought it would be better to be able to set hook scripts as module options rather than manually creating a directory with script files.
There are other possible settings in the buildkite-agent config file which aren't covered by module options, so I have added an extraConfig option to handle those.
There are also other minor tweaks to the module.
The option changes are backwards compatible.
Things done

[X] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[X] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[X] Tested NixOS module with Buildkite service.
[X] Fits CONTRIBUTING.md.

/cc @deepfire @zimbatm @domenkozar

Thanks @zimbatm @domenkozar :-)


Notice for max item limit is removed permanently if you keep typing

Describe the bug
The element that shows the message Only ${maxItemCount} values can be added after the item limit is reached disappears too early and cannot be triggered again. After the limit is reached, if you type one character the message is shown in the dropdown. If you type one or more extra characters, the element is removed completely and permanently (unless you reload the page). It does not re-appear after deleting the text and typing again, or after leaving and re-entering the input. The issue occurs with type text and type select-multiple.
To Reproduce
Steps to reproduce the behavior:

Go to https://choices-js.github.io/Choices/
Go to the first example under Text inputs.
Add items until the limit of 5 is reached.
Type one character to make the notice Only 5 values can be added appear.
Type another character to make the same notice disappear.

Expected behavior
I would expect that the notice stays even if you type more than one character and that it re-appears if you clear the search input and make another attempt of adding or searching an item.
Screenshots
Not applicable.
Choices version and bundle

Version: v11.0.2

Desktop (please complete the following information):

OS: Ubuntu 22.04
Browser: Firefox
Version: 130

Smartphone (please complete the following information):
Not applicable.
Additional context
None.

I will look into this early next week, as it looks like the logic on when or what a notice gets added needs more work.

I don't have time to create a PR for this, however if you throw a this._renderNotice(); before the early return for displayNotice it will solve the issue.


If there is no Docker installed running script closes PowerShell on Windows

When running an install script on a Windows box that has no docker a PowerShell is closed, and no error is displayed.

This makes it hard to figure out what the problem is.  Workaround is to turn transcript on prior to running the command:
start-transcipt -Path install.log
iwr https://windows.memgraph.com | iex


Deep linking opens share.android.js first

I am trying to redirect the app using Linking. When ever i open the url 'myapp://test/' in android it is going to componentDidMount method of share.android.js instead it should go to App.js componentDidMount. Works fine in IOS.
Here is my index.js
 AppRegistry.registerComponent('MyApp', () =>  require('./src/App').default);
 AppRegistry.registerComponent('myshareex', () =>  require('./share.android').default)

Problem in androidmainfest.xml
 <activity android:name="com.facebook.react.devsupport.DevSettingsActivity" />
This line should be added first. Closing the issue.


Perfect match only option when reconciling using details from other columns

Original author: thadguidry (October 07, 2010 23:39:24)
Refine should have capability for itself and extensions to add additional checkboxes called "Force", "Force Match" or "Perfect Match" next to Include? for each additional column that is included during reconcile.  This would help to force a perfect match and throw out irrelevant candidates that should not even be shown to the user because he/she chooses to perform perfect matches against those additional columns.
Example: Reconciling a City column and using an additional State column against the Freebase /location/location/containsby.  Checking an individual Force box next to the included State column would throw out returned candidates that do not match the extra property or properties.  Reconciling a City column that contains the cell "Miami" and has a corresponding cell of "Florida" in the State column for the row or record would throw out Miami(99) in Arizona as a candidate, even if the recon service returned "Miami" in Arizona with a score of (99) and help to automatch against the Miami(102) in Florida.
Original issue: http://code.google.com/p/google-refine/issues/detail?id=150

I'm interpreting this as "add the ability to force the service to use a property at the candidate retrieval stage", such that only results which agree on this property are considered. This is not supported by the API yet, so this would require specification work.

Related to Thad's suggestion in the reconciliation API specs: https://github.com/reconciliation-api/specs/issues/24

@wetneb Yeap, basically...make it more SQL-like (conditions specified by user, not service), where we give users more control of result processing, in general, rather than the service controlling a large portion of recon result processing.


API for Interacting with Docker Desktop Managed Kubernetes

How might an extension be able to interact with the Docker Desktop Kubenetes cluster (when one exists)?
Comments in the Extension SDK source show using ddClient.extension.host.cli.exec to invoke kubectl on the host, provided the kubectl binary is shipped as part of your extension. Would this be the recommended way to interact with the Docker Desktop managed Kubernetes cluster from an extension?

Hi @gilday, as of today there are no API methods available to interact with the Docker Desktop Kubernetes cluster.
If your extension needs to interact with Kubernetes clusters, then include the kubectl binary as part of your extension. You can find an example here.

@felipecruz91 Hello! An api for enabling Kubernetes on docker-desktop would enable a lot of functionality and ease-of-use. The use case is:

user clicks extension store
user finds useful extension, but it requires Kubernetes
user clicks "install extension" and we can enable Kubernetes for them if it is not already enabled
extension can then use kubectl normally as explained in the docs

Thank you!

A navigation intent for directing users to the Kubernetes preferences would be helpful too.


How can I decrypt the action and condition fields

Could you be so kind and let me know how could I decrypt/decompress the data in the "action" and "condition" fields. I am getting results, but I can't tell what's what. Much appreciated.

Action and Condition fields where?

Job Notification From CL Post
Hey I see your job Resume in CL
and I have a task For you To Do
It's a Short and Easy task Like Copy Paste
and if you able to do it i'll pay you.
So if Interested and to know more then
Reply below 'YES'<EMAIL_ADDRESS>


No repositories after restart and issues creating repository on existing S3 bucket

What happened?
Current Behavior:
Creating repository on already existing S3 storage results in the following error message:
failed to create repository: found lakeFS objects in the storage namespace(s3://test): storage namespace already in use

Steps to Reproduce:

Create a docker-compose.yml file with the following contents:version: '3.9'

services:
  minio:
    image: minio/minio:RELEASE.2023-06-29T05-12-28Z
    container_name: minio
    ports:
      - "9000:9000"
      - "9001:9001"
    entrypoint: ["minio", "server", "/data", "--console-address", ":9001"]

  postgres:
    image: postgres:14
    container_name: postgres
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
      interval: 10s
      retries: 5
      start_period: 5s
    restart: always

  minio-setup:
    image: minio/mc:RELEASE.2023-06-28T21-54-17Z
    container_name: minio-setup
    environment:
        - MC_HOST_lakefs=http://minioadmin:minioadmin@minio:9000
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      /usr/bin/mc mb lakefs/test;
      exit 0;
      "

  lakefs:
    image: treeverse/lakefs:0.104.0
    container_name: lakefs
    depends_on:
      - postgres
      - minio-setup
    ports:
      - "8000:8000"
    environment:
      - LAKEFS_DATABASE_TYPE=local
      - LAKEFS_BLOCKSTORE_TYPE=s3
      - LAKEFS_BLOCKSTORE_S3_FORCE_PATH_STYLE=true
      - LAKEFS_BLOCKSTORE_S3_ENDPOINT=http://minio:9000
      - LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_KEY_ID=minioadmin
      - LAKEFS_BLOCKSTORE_S3_CREDENTIALS_SECRET_ACCESS_KEY=minioadmin
      - LAKEFS_AUTH_ENCRYPT_SECRET_KEY=some random secret string
      - LAKEFS_STATS_ENABLED
      - LAKEFS_LOGGING_LEVEL
      - LAKECTL_CREDENTIALS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
      - LAKECTL_CREDENTIALS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
      - LAKECTL_SERVER_ENDPOINT_URL=http://localhost:8000
      - LAKEFS_DATABASE_CONNECTION_STRING=postgres://postgres:postgres@postgres-lakefs/postgres?sslmode=disable
    entrypoint: ["/bin/sh", "-c"]
    command:
        - |
          lakefs setup --local-settings --user-name docker --access-key-id AKIAIOSFODNN7EXAMPLE --secret-access-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY || true
          lakefs run --local-settings &
          wait-for -t 60 lakefs:8000 -- lakectl repo create lakefs://test s3://test || true
          wait

Start this docker container environment by running: docker compose up
Go to the test repository in the UI on http://localhost:8000 and upload a file, e.g. test.txt
Simulate a restart of LakeFS by running the following two commands in the terminal:docker compose down lakefs
docker compose up

Now open the LakeFS UI again, where you now can see that no repositories are found
Try to re-create repository in the UI by creating an empty repository with the following details:

Repository ID: test
Storage Namespace: s3://test
Default Branch: main

Expected Behavior
Expected Behavior:
Expecting behavior is that after a restart of LakeFS, the repositories are still visible in the UI. If they do not show up, make it possible to create repository on existing S3 storage.
Using the import functionality does not seem to satisfy the requirements as the files are of binary format that does not have the same naming scheme as previously uploaded.
lakeFS Version
0.104.0
Deplyoment
Local with min.io, and Azure Container Instance with blob storage
Affected Clients
No response
Relevant logs output
No response
Contact Details
No response

Hi @FredrikBakken!
The behaviour that you're observing is exactly as designed. lakeFS stores its metadata on object store (minio in this case), and restarting the lakeFS server does not remove that.
If you want to reset your environment then docker compose down && docker compose up will do that (i.e. recreate the minio container to delete its objects)
It's a nice idea lakeFS actually scanning the attached storage for repositories and pre-populating them in the UI, but that would be new functionality for us to build.
Can you elaborate a bit more on what your requirements/use-case is here please? I didn't quite follow your comment about the use of import and how it relates to the repository already existing in the object store.

Hi @rmoff !
Thank you for the quick follow-up to my Issue.
The problem I've encountered is related to LakeFS not being able to re-populate any repositories in the UI after the container is stopped (on purpose or after crashing) and then started again. In my case I have a self-hosted LakeFS container mapped to S3 buckets in min.io, once the LakeFS container is stopped and started the repositories defined in LakeFS before the restart does no longer appear in the UI (no repositories are listed in LakeFS). However, all the data and metadata objects can still be found inside the min.io buckets as I in this example only stopped and started LakeFS.
When I then try to re-create the repository in the LakeFS UI on the existing min.io buckets with data and metadata, I am getting the error message:
failed to create repository: found lakeFS objects in the storage namespace(s3://test): storage namespace already in use

The behaviour I'd want to see is that I can re-map the repositories to the S3 buckets in the restarted LakeFS container.
I hope this description is a bit easier to understand. Please let me know if some details still are unclear and I can create an even more detailed step-by-step example.

@rmoff , you've absolutely made my day with this explanation - thank you! 🚀
I now clearly understand where my misconfigurations came into causing chaos for my local set up. It also helped me understand why I was seeing a similar issues in Azure where environment variables were actually set up correctly, but since it was spun up together with PostgreSQL as a Terraform module the metadata was reset when it was taken down and spun up, while Storage Account with Container and data was not taken down and spun up with the rest - adding this as it might help others in Azure.
When it comes to the: docker compose down lakefs, I am not sure why it does not work for you. I have my terminal placed in the same directory as where the docker-compose.yml file exists. Maybe it is related to docker / docker compose versions? Mine are:
$ docker version
Client: Docker Engine - Community
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 20:35:18 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.4
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.5
  Git commit:       4ffc614
  Built:            Fri Jul  7 14:50:55 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ docker compose version
Docker Compose version v2.20.2

Another addition that I came over during my testing. I've encountered race conditions when trying to start the entire container environment with docker compose up, where I had some issues with trying to add the automated wait.
Instead I changed the depends_on to the following:
depends_on:
  postgres:
    condition: service_healthy
  minio-setup:
    condition: service_completed_successfully

Which seems to be a docker native solution to avoiding race conditions in this case.

@FredrikBakken thanks for the depends_on tip — i'll try and add that to one of the samples that we publish


[WIP] Fix SurveyGizmo::API::Question#sub_questions

Hi,
I think I noticed a small problem in SurveyGizmo::API::Question#sub_questions but I'd like to have your input before continuing the PR and writing some specs.
When I try to fetch all the questions of a survey containing sub questions with the following code:
SurveyGizmo::API::Question.all(survey_id: my_id, page_id: 1).inspect

it fails with this error:
/home/fchaynes/.rbenv/versions/2.1.7/lib/ruby/2.1.0/uri/common.rb:176:in `split': bad URI(is not URI?): /survey/1234567/surveyquestion/["0", 6] (URI::InvalidURIError)
	from /home/fchaynes/.rbenv/versions/2.1.7/lib/ruby/2.1.0/uri/common.rb:211:in `parse'
	from /home/fchaynes/.rbenv/versions/2.1.7/lib/ruby/2.1.0/uri/common.rb:747:in `parse'
	from /home/fchaynes/.rbenv/versions/2.1.7/lib/ruby/gems/2.1.0/gems/httparty-0.13.7/lib/httparty/request.rb:53:in `path='
	from /home/fchaynes/.rbenv/versions/2.1.7/lib/ruby/gems/2.1.0/gems/httparty-0.13.7/lib/httparty/request.rb:43:in `initialize'
	from /home/fchaynes/.rbenv/versions/2.1.7/lib/ruby/gems/2.1.0/gems/httparty-0.13.7/lib/httparty.rb:545:in `new'
	from /home/fchaynes/.rbenv/versions/2.1.7/lib/ruby/gems/2.1.0/gems/httparty-0.13.7/lib/httparty.rb:545:in `perform_request'
	from /home/fchaynes/.rbenv/versions/2.1.7/lib/ruby/gems/2.1.0/gems/httparty-0.13.7/lib/httparty.rb:476:in `get'
	from /home/fchaynes/Projects/github/survey-gizmo-ruby/lib/survey_gizmo/resource.rb:73:in `first'
	from /home/fchaynes/Projects/github/survey-gizmo-ruby/lib/survey_gizmo/api/question.rb:37:in `block in sub_questions'
	from /home/fchaynes/Projects/github/survey-gizmo-ruby/lib/survey_gizmo/api/question.rb:37:in `map'
	from /home/fchaynes/Projects/github/survey-gizmo-ruby/lib/survey_gizmo/api/question.rb:37:in `sub_questions'
	from /home/fchaynes/Projects/github/survey-gizmo-ruby/lib/survey_gizmo/resource.rb:62:in `block in all'
	from /home/fchaynes/Projects/github/survey-gizmo-ruby/lib/survey_gizmo/resource.rb:62:in `map'
	from /home/fchaynes/Projects/github/survey-gizmo-ruby/lib/survey_gizmo/resource.rb:62:in `all'
	from sg_gem.rb:9:in `<main>'

On the [SurveyGizmo API documentation] (http://apihelp.surveygizmo.com/help/article/link/surveyquestion-returned-fields) they say that sub_question_skus is a number but when I enable the debug mode in the gem, here's what I actually get (I removed irrelevant informations for clarity):
"Parsed SurveyGizmo Response:"
{
           "result_ok" => true,
         "total_count" => 1,
                "page" => 1,
         "total_pages" => 1,
    "results_per_page" => 1,
                "data" => [
        [0] {
                           "id" => 5,
                        "_type" => "SurveyQuestion",
                     "_subtype" => "group",
            [...]
            "sub_question_skus" => {
                                    "0" => 6,
                                    "1" => 8,
                "customer_satisfaction" => 9
            }
        }
    ]
}

so it seems to fail because we're not receiving just the question id but a hash with the sub question shortname for the key and the id for the value. BTW, it's the same thing with the v3.
Have you ever experienced this problem? Am I misunderstanding something or is the documentation incorrect?

@apurvis Any opinion on this?

That's very strange - I have not seen that type of response from SG before - only ever seen sub_question_skus be an array.  (The docs also seem kind of wrong because historically sub_question_skus is not a number)
I wonder if it has to do with the "group" subtype?

I don't think we can merge this though because it will break all the other surveys... I'm guessing this is an API bug.
This is the response type I see for my surveys:
                        "sub_question_skus" => [
                            [0] 14,
                            [1] 15,
                            [2] 16,
                            [3] 17
                        ]

or
                    "sub_question_skus" => [
                        [0] 8,
                        [1] 9,
                        [2] 10,
                        [3] 11,
                        [4] 12
                    ]

It's just plain arrays - and as far as I know you are the first person to run into this problem.
The recommended course of action is probably to contact your SG rep and/or go post about this on their community board:
https://community.surveygizmo.com/

Point being, if you want to submit a PR, it can't change existing functionality like this.  You'd have to put in some kind of hack
Also worth mentioning this comment:
https://github.com/RipTheJacker/survey-gizmo-ruby/blob/10dfc515dd22ce59c4bcc94205d7d88d3a7d1fb5/lib/survey_gizmo/api/survey.rb#L56-L57
because team has a sort of similar problem with its own data type, depending on whether you call .first or .all.  I would double check whether .first and .all have different response types for your problem question, in which case you could use that as a workaround like i did for team

PS really the right answer is for SurveyGizmo to not be broken when it comes to subquestions in the first place!  So definitely go voice your concern over that.  I have already told our rep and I believe others have as well because of issues with https://github.com/RipTheJacker/survey-gizmo-ruby/issues/36

@FabienChaynes  as a workaround i would probably do something like
def sub_questions
  sub_question_skus = sub_question_skus.values if sub_question_skus.is_a?(Hash)
  @sub_questions ||= sub_question_skus.map { |subquestion_id| Question.first(survey_id: survey_id, id: subquestion_id) }
                                           .each { |subquestion| subquestion.parent_question_id = id  }
end

I'm not sure how well that will fly with the Virtus attributes but I'm guessing you can solve that.

FWIW I just scanned all my surveys and it looks like we have never used the group question type so that might be the root of the problem.

Thanks for your feedback @apurvis!
Actually I was wrong when I said that we were receiving a Hash. If I inspect sub_question_skus in the sub_questions method it gives me: [["0", 6], ["foo", 8]]. So it's an array of [String, Integer] corresponding to the subquestion shortname and id.
FWICS, they return this format when at least one shortname (alias in the admin interface) is set in the questions.

ruby hashes are arrays of 2 element arrays under the hood ;)
Hash[ [ ["a", 100], ["b", 200] ] ]   #=> {"a"=>100, "b"=>200}

I suspect if the sub_question_skus attribute were a Hash, then Virtus would coerce this correctly; however given the type discrepancy you are ending up with the array of arrays.

Thank you for your advice :)

:+1:

there was a conflict so i merged this into https://github.com/RipTheJacker/survey-gizmo-ruby/pull/49 - i think you can close this PR.


[FLINK-26223] Enables INFO logging for ZooKeeper into separate file

What is the purpose of the change
Enables ZooKeeper INFO logs in test runs. I was wondering whether we should just create two log files: one with ZK logs enabled and one with ZK logs disabled. But I decided against it because it would generate a lot of redundant content. We can still match the logs through the timestamp
Brief change log

Added new file appender and made it being used by the ZK-related loggers

Verifying this change
Test run in AzureCI to verify that the logs are generated.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

Rebased to fix the architecturial test failure

A sample run separating the client and the server logs can be found in this [test build|https://dev.azure.com/mapohl/flink/_build/results?buildId=800&view=logs&j=9dc1b5dc-bcfa-5f83-eaa7-0cb181ddc267&t=511d2595-ec54-5ab7-86ce-92f328796f20&l=9566]

Failures unrelated to the changes:

FLINK-25771
FLINK-26387


Revise SE-0204 w/ core team feedback

This adds back the renaming of index(of:) and index(where:) to firstIndex(of:) and firstIndex(where:), per core team feedback.

Looks good, thanks!


Dataset Collection. Fixing problems with inlining

Solved problem with commints in #50:

Problem to extract body of one-line methods: one-line several statements in a row
public String getRemainingString() { return str.substring(index); }

Problem to inline methods with massive comments inside at the end of method declaration:

/*
        } else {
              pass
        }
*/

@Vitaly-Protasov can you please pull the latest master branch?

@Vitaly-Protasov can you please pull the latest master branch?

pulled

I fixed all valuable comments, the rest of them were subjective. I left them as it is


[Provisioning Devices] Provision over Bluetooth - Broadcast BT Beacon

Problem
<what's wrong or missing, please include any applicable:

expected behavior
actual behavior
steps to reproduce
system configuration
screenshots, images, logs, etc.

Proposed Solution
<suggested fix, suggested enhancement>

Done