bounding_box_query does not work on cellpose result stored in spatialdata

Dear scverse Team,
First of all, thank you for your effort on developping integrative frameworks for spatial transcriptomics data.
Now, I'm analyzing MERFISH data using spatialdata. When I use cellpose to segment cells, I am not able to crop my data using the bounding_box_query function. However, this function works fine for data from generated by other segmentation tools  such as baysor.
We suspect an incompatibility in the coordinate system but do not know how to check.
Could you please help us to solve the issue?
The code to crop the data:
from spatialdata import bounding_box_query
crop_sdata = bounding_box_query(sdata, min_coordinate=[8000,8000], max_coordinate=[8250,8250], axes=("x","y"), target_coordinate_system="microns",filter_table=True)
This is the error message:

LinAlgError                               Traceback (most recent call last)
Input In [7], in <cell line: 2>()
1 from spatialdata import bounding_box_query
----> 2 crop_sdata = bounding_box_query(sdata, min_coordinate=[8000,8000], max_coordinate=[8250,8250], axes=("x","y"), target_coordinate_system="microns",filter_table=True)
File /data/kdi_prod/.kdi/project_workspace_0/1928/acl/01.00/scripts/conda_sopa/lib/python3.10/functools.py:889, in singledispatch..wrapper(*args, **kw)
885 if not args:
886     raise TypeError(f'{funcname} requires at least '
887                     '1 positional argument')
--> 889 return dispatch(args[0].class)(*args, **kw)
File /data/kdi_prod/.kdi/project_workspace_0/1928/acl/01.00/scripts/conda_sopa/lib/python3.10/site-packages/spatialdata/_core/query/spatial_query.py:454, in _(sdata, axes, min_coordinate, max_coordinate, target_coordinate_system, filter_table)
452 for element_type in ["points", "images", "labels", "shapes"]:
453     elements = getattr(sdata, element_type)
--> 454     queried_elements = _dict_query_dispatcher(
455         elements,
456         bounding_box_query,
457         axes=axes,
458         min_coordinate=min_coordinate,
459         max_coordinate=max_coordinate,
460         target_coordinate_system=target_coordinate_system,
461     )
462     new_elements[element_type] = queried_elements
464 tables = _get_filtered_or_unfiltered_tables(filter_table, new_elements, sdata)
File /data/kdi_prod/.kdi/project_workspace_0/1928/acl/01.00/scripts/conda_sopa/lib/python3.10/site-packages/spatialdata/_core/query/spatial_query.py:398, in _dict_query_dispatcher(elements, query_function, **kwargs)
396 assert isinstance(d, dict)
397 if target_coordinate_system in d:
--> 398     result = query_function(element, **kwargs)
399     if result is not None:
400         # query returns None if it is empty
401         queried_elements[key] = result
File /data/kdi_prod/.kdi/project_workspace_0/1928/acl/01.00/scripts/conda_sopa/lib/python3.10/functools.py:889, in singledispatch..wrapper(*args, **kw)
885 if not args:
886     raise TypeError(f'{funcname} requires at least '
887                     '1 positional argument')
--> 889 return dispatch(args[0].class)(*args, **kw)
File /data/kdi_prod/.kdi/project_workspace_0/1928/acl/01.00/scripts/conda_sopa/lib/python3.10/site-packages/spatialdata/_core/query/spatial_query.py:728, in _(polygons, axes, min_coordinate, max_coordinate, target_coordinate_system)
720 _ = BoundingBoxRequest(
721     target_coordinate_system=target_coordinate_system,
722     axes=axes,
723     min_coordinate=min_coordinate,
724     max_coordinate=max_coordinate,
725 )
727 # get the four corners of the bounding box
--> 728 (intrinsic_bounding_box_corners, intrinsic_axes) = _get_bounding_box_corners_in_intrinsic_coordinates(
729     element=polygons,
730     axes=axes,
731     min_coordinate=min_coordinate,
732     max_coordinate=max_coordinate,
733     target_coordinate_system=target_coordinate_system,
734 )
736 bounding_box_non_axes_aligned = Polygon(intrinsic_bounding_box_corners)
737 indices = polygons.geometry.intersects(bounding_box_non_axes_aligned)
File /data/kdi_prod/.kdi/project_workspace_0/1928/acl/01.00/scripts/conda_sopa/lib/python3.10/site-packages/spatialdata/_core/query/spatial_query.py:102, in _get_bounding_box_corners_in_intrinsic_coordinates(element, axes, min_coordinate, max_coordinate, target_coordinate_system)
100 # transform the coordinates to the intrinsic coordinate system
101 intrinsic_axes = get_axes_names(element)
--> 102 transform_to_intrinsic = transform_to_query_space.inverse().to_affine_matrix(  # type: ignore[union-attr]
103     input_axes=axes, output_axes=intrinsic_axes
104 )
105 rotation_matrix = transform_to_intrinsic[0:-1, 0:-1]
106 translation = transform_to_intrinsic[0:-1, -1]
File /data/kdi_prod/.kdi/project_workspace_0/1928/acl/01.00/scripts/conda_sopa/lib/python3.10/site-packages/spatialdata/transformations/transformations.py:527, in Affine.inverse(self)
526 def inverse(self) -> BaseTransformation:
--> 527     inv = np.linalg.inv(self.matrix)
528     return Affine(inv, self.output_axes, self.input_axes)
File /data/kdi_prod/.kdi/project_workspace_0/1928/acl/01.00/scripts/conda_sopa/lib/python3.10/site-packages/numpy/linalg/linalg.py:556, in inv(a)
554 a, wrap = _makearray(a)
555 _assert_stacked_2d(a)
--> 556 _assert_stacked_square(a)
557 t, result_t = _commonType(a)
559 signature = 'D->D' if isComplexType(t) else 'd->d'
File /data/kdi_prod/.kdi/project_workspace_0/1928/acl/01.00/scripts/conda_sopa/lib/python3.10/site-packages/numpy/linalg/linalg.py:213, in _assert_stacked_square(*arrays)
211 m, n = a.shape[-2:]
212 if m != n:
--> 213     raise LinAlgError('Last 2 dimensions of the array must be square')
LinAlgError: Last 2 dimensions of the array must be square
The structure of our spatialdata:
SpatialData object with:
├── Images
│     └── 'output_region_0_z3': MultiscaleSpatialImage[cyx] (11, 96363, 78210), (11, 48181, 39105), (11, 24090, 19552), (11, 12045, 9776), (11, 6022, 4888)
├── Points
│     └── 'output_region_0_transcripts': DataFrame with shape: (19797064, 9) (2D points)
├── Shapes
│     ├── 'cellpose_boundaries': GeoDataFrame shape: (62498, 1) (2D shapes)
│     ├── 'output_region_0_polygons': GeoDataFrame shape: (196467, 1) (2D shapes)
│     └── 'sopa_patches': GeoDataFrame shape: (9, 1) (2D shapes)
└── Tables
└── 'table': AnnData (62498, 315)
with coordinate systems:
▸ 'microns', with elements:
output_region_0_z3 (Images), output_region_0_transcripts (Points), cellpose_boundaries (Shapes), output_region_0_polygons (Shapes), sopa_patches (Shapes)
Version of tools:
spatialdata 0.1.2,
cellpose 3.0.10,
cellpose was run on image patches created by the sopa framework version 1.1.0 after creating multiples patches of the image)
Please let me know if you need more informations.
Looking forward for your reply.
Regards,
Mamy

Hi thanks for the reporting the bug. I think here the easiest would be if you could subset the data by dropping all but the shapes that lead to the bug, and also try to subset the rows of the GeoDataFrame to reduce it to a very small dataset.
After this if you could write it to Zarr, zip it and send it to me via Zulip it would be easier for me to study the bug: https://scverse.zulipchat.com/#user/480560.

Actually, before that, could you paste here the content of .attrs in your shapes object? It could may be a problem with a transformation badly specified. Thanks!

Hello Luca,
Thanks for you rapid reply.
Here is the .attrs of the shape:
sdata.shapes['cellpose_boundaries'].attrs
{'transform': {'microns': Affine (x, y -> c, x, y)
[0. 0. 0.]
[1.07998399e-01 0.00000000e+00 4.20769397e+03]
[0.00000000e+00 1.07999088e-01 1.62009116e+03]
[0. 0. 1.]}}
Also, I 've tried to delete all other shapes except the cellpose one but I still had the same error:
del sdata.shapes['output_region_0_polygons']
del sdata.shapes['sopa_patches']
sdata
SpatialData object with:
├── Images
│     └── 'output_region_0_z3': MultiscaleSpatialImage[cyx] (11, 96363, 78210), (11, 48181, 39105), (11, 24090, 19552), (11, 12045, 9776), (11, 6022, 4888)
├── Points
│     └── 'output_region_0_transcripts': DataFrame with shape: (19797064, 9) (2D points)
├── Shapes
│     └── 'cellpose_boundaries': GeoDataFrame shape: (62498, 1) (2D shapes)
└── Tables
└── 'table': AnnData (62498, 315)
with coordinate systems:
▸ 'microns', with elements:
output_region_0_z3 (Images), output_region_0_transcripts (Points), cellpose_boundaries (Shapes)
Cropping this object leads exactly to the same error.
Regards,
Mamy

Thanks for sharing. I think that if you remove the c channel from the output of the transformation it will work. Was the transformation given with the c channel from cellpose?

Hello Luca,
Can you let us know how we can easily remove the c channel?
We tried to work directly on the transformation matrix of the 'cellpose_boundaries' but we didn't succeed to remove the error.
For example, we changed this
sdata.shapes['cellpose_boundaries'].attrs["transform"]
{'microns': Affine (x, y -> c, x, y)
[0. 0. 0.]
[1.07998399e-01 0.00000000e+00 4.20769397e+03]
[0.00000000e+00 1.07999088e-01 1.62009116e+03]
[0. 0. 1.]}
to this
sdata.shapes['cellpose_boundaries'].attrs["transform"]
{'microns': Affine (x, y -> c, x, y)
[0. 0.]
[   0.         4207.69396883]
[1.07999088e-01 1.62009116e+03]
[0. 1.]}
but still have the same error.
Thanks for your help!
Best,

Solution
Here is how you can remove the c channel and set the transformation inside the object again:
affine_without_c = affine_from_data.to_affine(input_axes=("x", "y"), output_axes=("x", "y"))
set_transformation(sdata[element_name], affine_without_c)

Alternative solution
You could have also constructed a new Affine transformation from a new affine matrix (you can see the code for doing it below).
Saving to disk
Please note that calling set_transformation only changes the transformation metadata in-memory. If you want to also save this to disk you need to call set_transformation with write_to_sdata set to the SpatialData object you are considering.
Alternatively you can call the method write_transformations() of the SpatialData object you are considering (please note that this method is not currently available in the version you can install from pip; we are making a new release this week).
Full example
Full example showing how to reproduce your bug and how to make it disappear (the example doesn't show how to save to disk).
##
import numpy.linalg
import pytest
from spatialdata.transformations import Affine, set_transformation, get_transformation
from spatialdata.datasets import blobs
from spatialdata import bounding_box_query

##
# data preparation
sdata = blobs()
element_name = "blobs_circles"

matrix = [
    [0.00000000e00, 0.00000000e00, 0.00000000e00],
    [1.07998399e-01, 0.00000000e00, 4.20769397e03],
    [0.00000000e00, 1.07999088e-01, 1.62009116e03],
    [0.00000000e00, 0.00000000e00, 1.00000000e00],
]

affine = Affine(matrix, input_axes=("x", "y"), output_axes=("c", "x", "y"))
set_transformation(sdata[element_name], affine)

##
# querying the data fails
with pytest.raises(numpy.linalg.LinAlgError):
    bounding_box_query(
        sdata[element_name],
        axes=("x", "y"),
        min_coordinate=[4000, 1000],
        max_coordinate=[5000, 2000],
        target_coordinate_system="global",
    )

##
# replacing the transformation with a matrix without the 'c' axis
# this is the same as the transformation above, here I am showing how to get it from the data
affine_from_data = get_transformation(sdata[element_name])

affine_without_c = affine_from_data.to_affine(input_axes=("x", "y"), output_axes=("x", "y"))
set_transformation(sdata[element_name], affine_without_c)

##
# querying the data succeeds
bounding_box_query(
    sdata[element_name],
    axes=("x", "y"),
    min_coordinate=[4000, 1000],
    max_coordinate=[5000, 2000],
    target_coordinate_system="global",
)

Thanks Luca for your reply.
I will try it and let you know.
Regards,
Mamy

Hello Luca,
Sorry to bring up this issue again.
Following your solution, indeed, the bounding_box_query works fine but the cellpose shapes seem not to be aligned at all to the original image even after removing the c in the transformation.
This is what I've done:
my sdata:
SpatialData object with:
├── Images
│     └── 'output_region_0_z3': MultiscaleSpatialImage[cyx] (11, 96363, 78210), (11, 48181, 39105), (11, 24090, 19552), (11, 12045, 9776), (11, 6022, 4888)
├── Points
│     └── 'output_region_0_transcripts': DataFrame with shape: (19797064, 9) (2D points)
├── Shapes
│     ├── 'baysor_boundaries': GeoDataFrame shape: (35330, 1) (2D shapes)
│     ├── 'cellpose_nuclei': GeoDataFrame shape: (21894, 1) (2D shapes)
│     ├── 'output_region_0_polygons': GeoDataFrame shape: (196467, 1) (2D shapes)
│     └── 'sopa_patches': GeoDataFrame shape: (9, 1) (2D shapes)
└── Tables
└── 'table': AnnData (21894, 315)
with coordinate systems:
▸ 'microns', with elements:
output_region_0_z3 (Images), output_region_0_transcripts (Points), baysor_boundaries (Shapes), cellpose_nuclei (Shapes), output_region_0_polygons (Shapes), sopa_patches (Shapes)
The coordinate before transformations:
print(sdata.images['output_region_0_z3'].attrs)
print(sdata.points['output_region_0_transcripts'].attrs)
print(sdata.shapes['cellpose_nuclei_6diam_0minArea'].attrs)
print(sdata.shapes['output_region_0_polygons'].attrs)
print(sdata.shapes['sopa_patches'].attrs)
{}
{'transform': {'microns': Identity }}
{'transform': {'microns': Affine (c, y, x -> c, x, y)
[1. 0. 0. 0.]
[0.00000000e+00 0.00000000e+00 1.07998399e-01 4.20769397e+03]
[0.00000000e+00 1.07999088e-01 0.00000000e+00 1.62009116e+03]
[0. 0. 0. 1.]}}
{'transform': {'microns': Identity }}
{'transform': {'microns': Identity }}
from spatialdata.transformations import Affine, set_transformation, get_transformation
Get transformation
affine_from_data = get_transformation(sdata.shapes['cellpose_nuclei_6diam_0minArea'], to_coordinate_system='microns')
affine_from_data
Affine (c, y, x -> c, x, y)
[1. 0. 0. 0.]
[0.00000000e+00 0.00000000e+00 1.07998399e-01 4.20769397e+03]
[0.00000000e+00 1.07999088e-01 0.00000000e+00 1.62009116e+03]
[0. 0. 0. 1.]
Create new transformation without c
affine_from_data_without_c = affine_from_data.to_affine(input_axes=("x","y"), output_axes=("x","y"))
affine_from_data_without_c
Affine (x, y -> x, y)
[1.07998399e-01 0.00000000e+00 4.20769397e+03]
[0.00000000e+00 1.07999088e-01 1.62009116e+03]
[0. 0. 1.]
Set transformation
set_transformation(sdata.shapes['cellpose_nuclei_6diam_0minArea'], affine_from_data_without_c)
sdata.shapes['cellpose_nuclei_6diam_0minArea'].attrs
{'transform': {'microns': Affine (c, y, x -> c, x, y)
[1. 0. 0. 0.]
[0.00000000e+00 0.00000000e+00 1.07998399e-01 4.20769397e+03]
[0.00000000e+00 1.07999088e-01 0.00000000e+00 1.62009116e+03]
[0. 0. 0. 1.],
'global': Affine (x, y -> x, y)
[1.07998399e-01 0.00000000e+00 4.20769397e+03]
[0.00000000e+00 1.07999088e-01 1.62009116e+03]
[0. 0. 1.]}}
Here, it creates a new transformation with 'global' name but keep the old one (microns). Then I remove the 'microns' transformation and rename the 'global' to 'microns':
sd.transformations.remove_transformation(sdata.shapes['cellpose_nuclei_6diam_0minArea'], to_coordinate_system = 'microns')
sdata.shapes['cellpose_nuclei_6diam_0minArea'].attrs
{'transform': {'global': Affine (x, y -> x, y)
[1.07998399e-01 0.00000000e+00 4.20769397e+03]
[0.00000000e+00 1.07999088e-01 1.62009116e+03]
[0. 0. 1.]}}
sdata.shapes['cellpose_nuclei_6diam_0minArea'].attrs["transform"]["microns"] = sdata.shapes['cellpose_nuclei_6diam_0minArea'].attrs["transform"]["global"]
sd.transformations.remove_transformation(sdata.shapes['cellpose_nuclei_6diam_0minArea'], to_coordinate_system = 'global')
At the end, I have the same except the c channel removed in the cellpose boundaries shape:
print(sdata.images['output_region_0_z3'].attrs)
print(sdata.points['output_region_0_transcripts'].attrs)
print(sdata.shapes['cellpose_nuclei_6diam_0minArea'].attrs)
print(sdata.shapes['output_region_0_polygons'].attrs)
print(sdata.shapes['sopa_patches'].attrs)
{}
{'transform': {'microns': Identity }}
{'transform': {'microns': Affine (x, y -> x, y)
[1.07998399e-01 0.00000000e+00 4.20769397e+03]
[0.00000000e+00 1.07999088e-01 1.62009116e+03]
[0. 0. 1.]}}
{'transform': {'microns': Identity }}
{'transform': {'microns': Identity }}
Using this object, the cropping works without error but the cellpose shapes plot seems to be not aligned to the other images in the object and looks weird.
sdata.pl.render_shapes('cellpose_nuclei_6diam_0minArea', outline=True, fill_alpha=0, outline_color='r').pl.render_shapes('output_region_0_polygons', outline=True, fill_alpha=0, outline_color='g').pl.show()

sdata.pl.render_shapes('cellpose_nuclei_6diam_0minArea', outline=True, fill_alpha=0, outline_color='r').pl.show()

Thanks for you help.
Regards,
Mamy

Thanks for sharing the details.
First, minor tip, I have formatted your message by enclosing the code in blocks
```python
your code here
```
and the same for the outputs (I have used the triple backticks, without the "python" string in the first line).
Now, back to your problem. I am afraid I can't pinpoint the cause of the problem here not having the data, but I can give some advice.

Where does the transformation matrix come from? I see that you have an element from SOPA in the data. Is that matrix constructed by SOPA? In that case you could be seeing a bug from SOPA and I would report it in the SOPA GitHub repository.
More of an "hack" than a solution, but may unlock you for this particular case. If there is only one matrix available (as it seems from your case), a good guess is that one the following will make the data aligned: either no matrix either the inverse matrix. I would try these two cases (by setting Identity for the nuclei, and by setting my_affine_transformation.inverse()). You can also try to construct an affine matrix with only translation. Quickly looking at your plot, the following matrix may work:

Affine (x, y -> x, y)
      [1 0.00000000e+00 4.20769397e+03]
      [0.00000000e+00 1 1.62009116e+03]
      [0. 0. 1.]}}

I don't get what is happening in the last plot. I would try zooming a bit more, or visualizing the data with napari_spatialdata, which offers an interactive exploration experience.

from napari_spatialdata import Interactive

Interactive(sdata)

I hope the above can help.

Hello Luca,
Thanks for the tips.

I suspect also a problem with sopa because the plot of the shape is really weird, I will post the issue in the SOPA GitHub too. However, element generated by other segmentation tool (baysor) run using SOPA does not have this transformation and this problem.
I will test these solutions and let you know
I tried to use napari-spatialdata but I have also an issue that I will post on the napari-spatialdata GitHub.

Regards,
Mamy

Ok perfect thanks for the information, we can follow up in the napari-spatialdata repository, and please when you post the issue in SOPA reference this issue so I can track the conversation.


Confusion in bower and package versions

Hi,
The bower version and the package version are different:
Bower: "version": "0.14.1"
Package: "version": "0.16.1"
bower angular-ui-select#*     mismatch Version declared in the json (0.14.1) is different than the resolved one (0.16.1)
This is an error? If not, why have this difference?
Thanks for this amazing plugin!

:+1:

+1

Bower uses tags to determine the version currently, and from what I understand, that is not likely to change.
Either the value should be removed from bower.json, or the number needs to be kept up to date with the releases.

@aaronroberson can you look at this for the next release?

Since our gulp task updates bower I will go ahead and bump the bower version up and keep them in sync from here on out. Is that ok with you as well @deeg?

@aaronroberson, yeah that sounds good. I think you could also just remove the version property if you didn't want to keep it up to date. I don't think it matters either way.


Add basic blacklist functionality

Example usecase
./suo5 -t 'https://example.com/suo5.aspx' -l '<IP_ADDRESS>:1080' -B 'polling.oastify.com,portswigger.net'
[INFO] 06-17 16:38 starting tunnel at <IP_ADDRESS>:1080

[Tunnel Info]
Target:  http://example.com/suo5.aspx
Proxy:   socks5://<IP_ADDRESS>:1080
Mode:    half

[INFO] 06-17 16:38 creating a test connection to the remote target
[INFO] 06-17 16:38 start connection to <IP_ADDRESS>:0
[DBUG] 06-17 16:38 recv dial response from server: length: 7
[INFO] 06-17 16:38 successfully connected to <IP_ADDRESS>:0
[DBUG] 06-17 16:38 recv socks5 reply: 0
[DBUG] 06-17 16:38 local conn closed, <IP_ADDRESS>:0
[INFO] 06-17 16:38 congratulations! everything works fine
[DBUG] 06-17 16:38 remote readwriter closed, <IP_ADDRESS>:0
[INFO] 06-17 16:38 connection closed, <IP_ADDRESS>:0
[SUCC] 06-17 16:38 drop connecction to polling.oastify.com:443
[SUCC] 06-17 16:38 drop connecction to polling.oastify.com:443
[SUCC] 06-17 16:39 drop connecction to portswigger.net:443

抱歉这些天一直没有评论，这个功能挺好的，可以在再改一下我合了

Hi, i change a bit so that it match the project coding style, now the syntax is like this
./suo5 -d --nh -t 'http://<IP_ADDRESS>/suo5.aspx' -l '<IP_ADDRESS>:1080' -E "google.com" -E "stackoverflow" -E "amazon.com"


circom support？

gnark supports circom？ if not，is there a plan for it?

Hello, no there is compatibility planned with Circom frontend.


Get search first zebrafish test set to pass

EDIT: This may be an issue regarding overlap. If so, that's currently out-of-scope.
When running the search-first zebrafish test set /data/active/fractal/3D/PelkmansLab/ZebrafishMultiplexing/cycle0 (number 2 gridless of https://github.com/fractal-analytics-platform/fractal/issues/213), parsing fails with the following error message:
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/dataflow/dflow.py", line 783, in _unwrap_futures
    kwargs[key] = dep.result()
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/dataflow/dflow.py", line 288, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/dataflow/dflow.py", line 483, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/app/errors.py", line 138, in reraise
    reraise(t, v, v.__traceback__)
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/six.py", line 719, in reraise
    raise value
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/app/errors.py", line 176, in wrapper
    return func(*args, **kwargs)  # type: ignore
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/fractal_server/app/runner/__init__.py", line 55, in _task_fun
    metadata_update = _callable(
  File "/net/nfs4/pelkmanslab-fileserver-jluethi/data/homes/jluethi/fractal_3repo/fractal-tasks-core/fractal_tasks_core/create_zarr_structure.py", line
157, in create_zarr_structure
    metadata = parse_metadata(fn.name)
  File "/net/nfs4/pelkmanslab-fileserver-jluethi/data/homes/jluethi/fractal_3repo/fractal-tasks-core/fractal_tasks_core/lib_parse_filename_metadata.py",
 line 58, in parse_metadata
    raise Exception(
Exception: Failure in metadata parsing of AssayPlate_Greiner_#655090, with img_date={img_date} and img_time={img_time}

Debug the parsing to figure out what's failing. Could be an issue when the metadata was created for this test set, could be a parsing problem.

I updated the parsing logic to speed it up (see here: )
As part of the refactor, the base problem appears to have been solved. But now, the overlap correction runs into an issue:
2022-10-14 15:44:30,935; INFO; removing FOV overlaps for well='B02'
2022-10-14 15:44:30,936; INFO; removing FOV overlaps for well='B03'
2022-10-14 15:44:30,937; INFO; removing FOV overlaps for well='B04'
2022-10-14 15:44:30,939; INFO; iteration=1, removing overlap between fov_id_1=2.0 and fov_id_2=1.0
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Cell In [9], line 1
----> 1 site_metadata, total_files = parse_yokogawa_metadata(mrf_path, mlf_path)
      2 total_files

File ~/Dropbox/Joel/FMI/Code/fractal/fractal-tasks-core/fractal_tasks_core/metadata_parsing.py:81, in parse_yokogawa_metadata(mrf_path, mlf_path)
     75 total_files = len(mlf_frame)
     76 # TODO: Check whether the total_files correspond to the number of
     77 # relevant input images in the input folder. Returning it for now
     78 # Maybe return it here for further checks and produce a warning if it does
     79 # not match
---> 81 site_metadata = remove_FOV_overlaps(site_metadata)
     83 return site_metadata, total_files

File ~/Dropbox/Joel/FMI/Code/fractal/fractal-tasks-core/fractal_tasks_core/lib_remove_FOV_overlaps.py:119, in remove_FOV_overlaps(df)
    117 is_y_equal = abs(ymin_1 - ymin_2) < tol and (ymax_1 - ymax_2) < tol
    118 if is_x_equal + is_y_equal != 1:
--> 119     raise Exception(
    120         "Two overlapping FOVs MUST share either a "
    121         "vertical boundary or an horizontal one, but "
    122         f"{is_x_equal=} and {is_y_equal=}"
    123     )
    125 # Compute and apply shift (either along x or y)
    126 if is_y_equal:

Exception: Two overlapping FOVs MUST share either a vertical boundary or an horizontal one, but is_x_equal=False and is_y_equal=False

I'll have to dig a bit deeper to understand why this would occur

My impression is that this is an issue that arises when there is overlap between images that can't be corrected with a simple shift in one direction. We'll may need to be able to handle this case, but I wasn't expecting to hit it here.
The well B04, which is the first one triggering this issue, does have a weird FOV setup. Here are the FOVs:

FOV 1 is at -1907.4; 514.5
FOV 2 is at -2385.3; 385.4
The pixel size for this experiment is 0.322101 in x & y, 1 in z, according to the metadata. @MaksHess is this correct? That's a fairly weird pixel size! At the CV7K, I expect to see 0.1625 and 0.33. Is that correct?
Given the pixel size and the image size of 2000x2000 pixels, FOV 2 thus extends into FOV1 (overlap in both X & Y) => not a regular overlap, but overlap like in scenario 4 here that we're not planning to cover:  https://github.com/fractal-analytics-platform/fractal-tasks-core/issues/11
@MaksHess Is this dataset acquired with overlap in some field of views?

I think Yokogawa does some internal scaling of the pixel size based on who-knows-what (theoretically it would be 0.325um).
Regarding the overlap: I did not acquire overlapping regions on purpose, but it's theoretically possible if two embryos detected by searchFirst are close together...

Ok, weird that the Yokogawa does some kind of pixel size scaling here. The CV7K returns pixel sizes like we would expect them. Do you think 0.322101 is actually a wrong entry or the actual truth for this dataset?
Alternatively, we could look into manually changing the metadata to avoid such overlaps (but misplace a few FOVs then). But I'd have some concerns with that, see below

Regarding the overlap: I did not acquire overlapping regions on purpose, but it's theoretically possible if two embryos detected by searchFirst are close together...

Two points here:

Do you have a dataset without any such overlap that we could test? Alternatively, I'd just test with the B02 & B03 wells only, because they don't appear to have any overlap
Broader discussion: Is imaging with potential overlap desirable? The way the search-first is implemented in the Liberali lab is the following: If two objects are very close to each other, it still acquires non-overlapping images by just shifting one FOV slightly off-center. My concern with imaging with overlap has multiple levels:
a) It becomes really hard to process whole wells when we have unpredictable regions of overlap. We could work around this by shifting FOVs to pretend they don't have overlap. We already do this for simple cases (e.g. a grid with predictable overlap). But in unpredictable setups, it's quite tricky to handle this
b) On the scientific side: you may have partial bleaching of one of the objects you image that you wouldn't account for. Also, elution may work worse if some regions are imaged multiple times, leading unpredicted elution issues in the actual experiment that don't show up in any controls. Thus, I'd strongly argue towards imaging non-overlapped were possible.

I think the scaling might be right (mb taking into account some deviation from 20x for their custom objective), in any case, an error of <1% sounds reasonable...

No unfortunately the dataset is on archive, if you can work with B02 & B03 that would be great... Another option would be to delete the single acquisition that is overlapping from the well.
I share your concerns with overlapping FOV's in general, mb long-term we should add constraints on that in the initial search-first detection. With my data it's possible that only edges of FOV's overlap without imaging an embryo twice though. Ideally those cases would be covered, could also be solved by a small shift :)

Makes sense. I'll test with B02 & B03 for the moment then (next week). And will think about a temporary mitigation strategy (e.g. a quick check we could run over the metadata file and then whether we could work around issues that come up with some small hack)

Ok, noticed another issue we're having with this test set: Our filename parsing currently makes assumptions of the filename structure that don't hold up here. We'll need to correct those. In the meantime, I'm renaming a (linked version of) these files into something without _ in the first part of the filename.
See this issue for details: https://github.com/fractal-analytics-platform/fractal-tasks-core/issues/189


Support 2024.3 eap

Closes https://github.com/getappmap/appmap-intellij-plugin/issues/790
The IDE's service of the HTTP server was throwing this exception when used with 2024.3 eap:
java.lang.NoSuchMethodError: 'void io.netty.channel.SingleThreadEventLoop.<init>(io.netty.channel.EventLoopGroup, java.util.concurrent.Executor, boolean, boolean)'

The problem is that our testDependency to org.mock-server:mockserver-junit-rule:5.15.0 pulls in netty-transport 4.1.x, but the IntelliJ SDK is using netty-transport 4.2.0-Alpha4. Because both versions have different method signatures, the SDK is throwing NoSuchMethodExceptions when used with netty-transport 4.1.x.
I've moved the failing test into a new submodule tests-integration, where the mockserver dependency is not made available.
In the PR, I've removed the dependency to mockserver because the same dependency is already pulled in by mockserver-junit-rule.
While debugging a problem in the sandbox IDE, I found a problem introduced by 2024.3 and documented it at https://github.com/getappmap/appmap-intellij-plugin/issues/790#issuecomment-2391419857

:tada: This PR is included in version 0.75.0 :tada:
The release is available on:

v0.75.0
GitHub release

Your semantic-release bot :package::rocket:


fake JSON model of hello.wat

convert WIT from textual to JSON model form

@AaronRobinsonMSFT FYI: @ilonatommy joined us to do this


Add a manage-podman script

Add ./scripts/manage-podman, a copy of ./scripts/manage that uses podman-compose instead of docker-compose.
This script can be used to run "manage" commands inside a container when using podman instead of docker.

Does this solve the toughbook situation or is this because we're moving to Podman anyways?

Not related to the thoughbook issue. I've used podman instead of docker during the training sessions and there were some thoughts about supporting podman as a rootless alternative to docker both for users and potentially for production. Not sure what's the current status of that. But so far, podman works perfectly as a drop-in replacement, once it's installed and configured correctly.

Not related to the thoughbook issue. I've used podman instead of docker during the training sessions and there were some thoughts about supporting podman as a rootless alternative to docker both for users and potentially for production. Not sure what's the current status of that. But so far, podman works perfectly as a drop-in replacement, once it's installed and configured correctly.

We are currently in the process of migrating the MUSCLE project to Podman on the servers. I think it does make sense to then also start using Podman locally during development to keep things consistent, but I'm not sure if that discussion has been started yet. If that will be the case, it would probably better to replace all references to docker-compose with podman-compose, making this PR obsolete. Perhaps you could also modify the existing manage script to use podman-compose and we can merge it after #722 has been merged?

I'm not sure how easy it is to use podman on Mac and Windows. Looks like it's build for Linux and uses a VM on Mac and WLS on Windows, so it might not quite as straightforward to use for development. Maybe it's better to allow a user to use either docker or podman, depending on what is more convenient, or would that be too much effort to maintain?

Yes, let's add support for Podman, but as an extra option next to Docker.

Yes, let's add support for Podman, but as an extra option next to Docker.

Does that mean we'll have to make Podman versions of the tother test/storybook/lint scripts too? Wouldn't it be nicer to update all the existing scripts with a parameter like -p that runs the same command using Podman?

Did you decide against this approach in the end?

Did you decide against this approach in the end?

We thought it might be better to nudge people into aliasing podman and podman-compose if they would like to use Podman, see also #803


macos build with with-dds-gateway fails (github workflow )

Required information
Operating system:
macos-10.15
Compiler version:
AppleClang <IP_ADDRESS>00032
Observed result or behaviour:
macos build fails in github workflow job (when with-dds-gateway is enabled)
[ 33%] Building CXX object CMakeFiles/iceoryx_dds_messages.dir/Mempool.cpp.o
ANTLR Tool version 4.4 used for code generation does not match the current runtime version 4.5ANTLR Tool version 4.4 used for code generation does not match the current runtime version 4.5In file included from /Users/runner/work/iceoryx/iceoryx/build/iceoryx_dds_messages/Mempool.cpp:1:
In file included from /Users/runner/work/iceoryx/iceoryx/build/iceoryx_dds_messages/Mempool.h:4:
In file included from /Users/runner/work/iceoryx/iceoryx/build/dependencies/install/include/ddscxx/dds/core/ddscore.hpp:22:
In file included from /Users/runner/work/iceoryx/iceoryx/build/dependencies/install/include/ddscxx/dds/core/types.hpp:28:
In file included from /Users/runner/work/iceoryx/iceoryx/build/dependencies/install/include/ddscxx/dds/core/macros.hpp:22:
/Users/runner/work/iceoryx/iceoryx/build/dependencies/install/include/ddscxx/dds/core/detail/macros.hpp:88:14: fatal error: 'boost/static_assert.hpp' file not found
#    include <boost/static_assert.hpp>
             ^~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make[2]: *** [CMakeFiles/iceoryx_dds_messages.dir/Mempool.cpp.o] Error 1
make[1]: *** [CMakeFiles/iceoryx_dds_messages.dir/all] Error 2
make: *** [all] Error 2

Expected result or behaviour:
Successful compilation & test run
Conditions where it occurred / Performed steps:
Enable option with-dds-gateway while invoking macos  build job in github workflow(using iceoryx_build_test.sh)

I guess the workaround is to install boost if you would like to use cyclonedds.

The ubuntu runners have boost installed but the macos ones do not ...


Zstack requirement to change sshd_config for PermitRootLogin to "yes" seems a security issue

During my initial attempts to install zstack I eventually found the documentation reference:
       http://zstack.org/tutorials/flat-network-ui.html

where it says:
= = = = = = = = =
Configure root user
The KVM host will need root user credentials of SSH, to allow Ansible to install necessary packages and to give the KVM agent full control of the host. As this tutorial use a single machine for both ZStack management node and KVM host, you will need to configure credentials for the root user.
CentOS:
sudo su
passwd root
Ubuntu:
You need to also enable root user in SSHD configuration.
1. sudo su
2. passwd root
3. edit /etc/ssh/sshd_config
4. comment out 'PermitRootLogin without-password'
5. add 'PermitRootLogin yes'
6. restart SSH: 'service ssh restart'
= = = = = = = = =
I know its stated that doing the above is required for Ansible... but in my mind and from everything I've read anywhere this permitting "root" login via ssh is NOT considered a Security "best practice".
This also seems a very odd config requirement for an IaaS Platform to require for obvious reasons.
Isn't that the whole reason why  "sudo" exists ?
Is there no way to resolve this by leaving PermitRootLogin to deny "root" login but utilize SUDO instead?
Brian

yeah. This version has not hardened everything with sudo. We will definitely fix this in future version.


Ability to download from proprietary git repos with authentication

Attempts to install from authenticated git repos return a 401 response, which bower doesn't currently handle.
This is a feature request to add support for authenticated repos. I would like our devs to have a '.bowerrc' something like...
{
  "directory": "app/bower_components",
  "registry": "https://www.company.com/bower/"
}

Our in-house bower repo then exposes the 'https://www.company.com/git/repo' URLs for our various proprietary JS components to our dev's bower clients.
IMHO, ideally, the bower client needs to do as the git client does and utilise the '.netrc' file to obtain auth creds in the case of a 401. This would save devs from having to duplicate their creds in multiple places.

As bower uses git directly, you can use any available auth mechanism: https://git-scm.com/docs/gitcredentials


Add configuration system

Would be great if you added a config to enable & disable certain features. Auto Config would make it fairly easy, and LibCD could be used to conditionally load recipes to prevent log spam.
Config options:

nether mushroom generation
nether cactus generation
overworld nether-ores
nether overworld-ores
armor/tool stats
ore rarity

Are you interested in doing this? If not, I can implement it and send in a PR.
Thanks!

I'm trying to find out how to make one for a while, without success. And I didn't know Auto Config.
I'm not against help in a PR to see a little how it's going. Especially, for new features or for my other mods. If it doesn't mind ?


perf: exploration of better matmul algorithms

This adds a notebook that explores different ways to do the V = Z.dot(Z.T.conj()) calculation.
Ideas that I haven't yet explored:

[ ] Using MAGMA (see eg. https://github.com/PSims/BayesEoR)
[ ] Using PyTorch
[ ] Proper NVIDIA profiling of all the functions

If you're going through all the efforts of writing a cublas implementation, might investigated compiled extensions like cython or a rust extension. Both have interop with numpy which is pretty straightforward


BUG: sparse: Validate explicit shape if given with dense argument to coo_matrix

Fixes gh-9919.
There are a couple routes we could take when validating the shape kwarg in this situation, but given the context of the original bug, it should be sufficient to raise an error whenever the given shape (intentional or not) doesn't match the shape of the dense matrix being converted (the first argument).
Note that once scipy starts converting code to be explicitly py3k-only, we could consider marking the shape and dtype arguments as keyword only. This would prevent these sorts of errors more broadly, though perhaps at the expense of breaking existing user code.

@mattip another random PyPy failure, haven't seen this before:
Traceback (most recent call last):
  File "setup.py", line 488, in <module>
    setup_package()
  File "setup.py", line 420, in setup_package
    import numpy
  File "/usr/local/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/usr/local/site-packages/numpy/core/__init__.py", line 40, in <module>
    from . import multiarray
  File "/usr/local/site-packages/numpy/core/multiarray.py", line 44, in <module>
    _reconstruct.__module__ = 'numpy.core.multiarray'
AttributeError: readonly attribute '__module__'
Exited with code 1

does that need a bug report somewhere, or is it a flake?

Note that once scipy starts converting code to be explicitly py3k-only, we could consider marking the shape and dtype arguments as keyword only. This would prevent these sorts of errors more broadly, though perhaps at the expense of breaking existing user code.

That would indeed break quite some code. There may be a couple of exceptions, but in general I think we can only use keyword-only arguments for new code.

This was fixed in #9840. Rebase the pr off master. It is using an old pypy version.

This was fixed in #9840. Rebase the pr off master. It is using an old pypy version.

Ah right. The failure didn't look familiar and it was a fresh PR, so I didn't expect that. Sorry for the noise.
What we need to do is fix CircleCI if we can, this is getting really annoying. It should test the merge commit, not the commit from the PR itself.

I tracked down the relevant CircleCI issue and explained the problem again: https://circleci.com/ideas/?idea=CCI-I-926. There's a voting feature, if people want to push the vote button that may be useful.

@perimosocordiae could you add the example from the issue as a regression test?

@rgommers Sure, tests added. I also rebased, so hopefully we'll see the pypy tests pass.

All green, in it goes. Thanks @perimosocordiae

It might be that this broke a test in pandas (a test constructing a sparse matrix started failing the last hours). The test basically does:
from scipy import sparse

row = [0, 3, 1, 0]
col = [0, 3, 1, 2]
data = [4, 5, 7, 9]
sp_array = sparse.coo_matrix(data, (row, col))

So using lists and not arrays to create a small dummy matrix. That worked before, but now started raising an error in check_shape (see eg https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=9238&view=logs&jobId=3a03f79d-0b41-5610-1aa4-b4a014d0bc70).
It's only in the testing code, not in actual functionality, so not a big problem for pandas (and it is also easy to fix the testing code to use arrays). But, just wanted to signal this (not sure what guarantees scipy gives on coercing the input).

Looking into it further, this is actually a wrong usage on our side, which was exactly the purpose of this PR to catch such shape mistakes ;-)
We are passing the data, (row, col) not inside a tuple, so gets interpreted as the shape.
So ignore my message above.

Glad to see the PR is already helping! :+1:


non-matching table and graphs in "Fixed/Free Spiking Correlation Histogram" under "freely moving analysis"

tables show lower values fro significant cells than what is in the graph.
test Experiment: I leave this open on Subiculum is at it large

The table and bar is confusing. I will add 2 more lines to the table displaying the Mean and SEM values.

The issue has been addressed in Issue #31. The table has also been expanded to display the mean and SEM values
As aside, from analysis of the data it appears that there are 2 files with 70+ cells whereas there are 3 files with 14-16 cells. This is causing the skewing in the mean statistics over all experiments relative to the total count percentages.


Relay support

How can this be made to work with a Relay client?

@stanonyime hi, this is an example repo. Check our up to date GraphQL ORM project: https://github.com/RisingStack/graffiti-mongoose
It supports Relay, more examples: https://github.com/RisingStack/graffiti-todo


Update Relative References

Purpose:

Make sure that the references to other files from the main README are accurate.

Cause:

Scaling
Changing folder structure
Renaming files

Bash
Main file

Closed today (8/11/22) via

#24
#25
#26
e92d89e1d5ae05ef44946eeb2a958a9d062b57dc


Inline method expands to multiple lines

Hi,
Seems to be the methods which are okay to be inline are spanned to multiple lines and spacing is adjusted with if and else  on PR #4690

https://github.com/plataformatec/devise/pull/4690#issuecomment-342037513


Build setting and runner missing

The code for build settings is missing, so it doesn't compile when I download the code:

Build\DocFxBuildRunner.cs
Build\DocFxBuildSettings.cs

Oh no. My dammit .gitignore ignored the files cause they're in a /Build folder.
They're now in the repo. 😄


CSRF cookie not set

Expected Behavior
When running in a local environment, a csrf cookie is set by my Django view.
Actual Behavior
However, after deploying to AWS Lambda, no csrf cookie is dropped.
Your Environment

Zappa version used: zappa==0.45.1
Python version: Python 3.6.2
Django version: Django==1.11.7

I just realized that I'm also using django-cors-headers ; that may have an impact. I'll look into it further and report back.

Re-opening; the problem has come up again...

Solved it by explicitly setting the csrf cookie:
`from django.utils.decorators import method_decorator
from django.views.decorators.csrf import ensure_csrf_cookie
import requests
class aView(TemplateView):
template_name = "template/template.html"
@method_decorator(ensure_csrf_cookie)
def get(self, request, *args, **kwargs):
    return super(aView, self).get(request, *args, **kwargs)

`


Request for raw cliff sensor data

It would be nice to add a topic for cliff state, including raw values, and information about which cliff(s) is/are triggered.
Discussed in https://github.com/iRobotEducation/create3_docs/discussions/62

Originally posted by guystam May 23, 2022
Where is /cliff_state topic?

Topic for cliff state triggered documented here: https://github.com/iRobotEducation/create3_docs/discussions/62#discussioncomment-2805572

Intensity request duplicated here: https://github.com/iRobotEducation/create3_docs/issues/320


Confusing instructions compared to previous step experiences

Misleading copy command via this link: https://learn.microsoft.com/en-us/azure/iot-edge/how-to-configure-iot-edge-for-linux-on-windows-networking?view=iotedge-1.4#check-dns-resolution
See attached screen shot.  I would break this dialogue box (?) into two sections.  The command should be entered is:
"resolvectl query microsoft.com", but the COPY command (which is typically used when following a tutorial) is incorrect, and copies what is the "output" of the command that you want to issue, rather than the command itself.
I would replace the "OUTPUT" (upper left) with "INPUT", and then put the result in a section which does NOT have a copy command.
I'm speaking as a newbie, but after banging my head on this a few times, I'm slowly realizing what I'm missed the first few times.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 750530e6-885c-339d-5683-3d79896c08d3
Version Independent ID: c3d56ec6-a78c-f795-45cd-aede44a8d43a
Content: Networking for Azure IoT Edge for Linux on Windows
Content Source: articles/iot-edge/how-to-configure-iot-edge-for-linux-on-windows-networking.md
Service: iot-edge
GitHub Login: @PatAltimore
Microsoft Alias: fcabrera

@RichardReukemaOptimus
Thanks for your feedback! We will investigate and update as appropriate.

Thanks for the feedback! I have assigned the issue to the content author to evaluate and update as appropriate.

Hi @RichardReukemaOptimus,
Thanks for the feedback. You're right. That is confusing. I changed the article to be more consistent. Highlighting the command in a code block and then showing the output. The change should publish today.
Thanks,
Pat
#please-close


Add no data + help modes to graph elements

@deadlyfingers - Please review this ask, or think about an alternative, not sure if this would be comfortable to define.
When no data is present enable elements like PieData, Timeline, etc... should present a "No Data" message and a link or InfoDrawer to show more data (like where that data should come from).
For example - Sentiment analysis - when no data is present, explain how when where this data can be added.

Hello is this issue still available?

Hi @DK308 - yes it is

Hi, is this still available?

@asantiagot most probably...


EF Core 2.1.3 Migrations not generating identical property mapping

When generating migrations for the following class:
public class User
{
        [DatabaseGenerated(DatabaseGeneratedOption.Identity)]
        [Key]
        public virtual Guid Id { get; set; }
        public virtual string FirstNames { get; set; }
        public virtual string LastName { get; set; }
        public virtual string EmailAddress { get; set; }
        public virtual int Status { get; set; }
        public virtual DateTime LastUpdated { get; set; }
        public virtual DateTime Created { get; set; }
        public virtual User CreatedBy { get; set; }
        public virtual User LastUpdatedBy { get; set; }
        public virtual bool IsDeleted { get; set; }
}

The following mapping is created:
migrationBuilder.CreateTable(
                name: "User",
                columns: table => new
                {
                    Id = table.Column<Guid>(nullable: false),
                    FirstNames = table.Column<string>(nullable: true),
                    LastName = table.Column<string>(nullable: true),
                    EmailAddress = table.Column<string>(nullable: true),
                    Status = table.Column<int>(nullable: false),
                    LastUpdated = table.Column<DateTime>(nullable: false),
                    Created = table.Column<DateTime>(nullable: false),
                    CreatedById = table.Column<Guid>(nullable: true),
                    IsDeleted = table.Column<bool>(nullable: false)
                },
                constraints: table =>
                {
                    table.PrimaryKey("PK_User", x => x.Id);
                    table.ForeignKey(
                        name: "FK_User_User_CreatedById",
                        column: x => x.CreatedById,
                        principalTable: "User",
                        principalColumn: "Id",
                        onDelete: ReferentialAction.Restrict);
                });

As you can see, the LastUpdatedBy column is not being mapped in the migration, whereas the CreatedBy column is. These two columns are seemingly the same to me, so why is one being mapped and not the other?
Further technical details
EF Core version: 2.1.3
Database Provider: Microsoft.EntityFrameworkCore.SqlServer
Operating system: MacOS
IDE: VS for Mac 2017
Thanks :).

@Jaffacakes82 What does the User class look like and do you have any configuration in OnModelCreating?

@ajcvickers the User class is at the top of my comment.
My OnModelCreating method just has a couple of table name bindings and a many to many mapping:
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
            modelBuilder.Entity<User>().ToTable("User");
            modelBuilder.Entity<Role>().ToTable("Role");
            modelBuilder.Entity<Permission>().ToTable("Permission");
            modelBuilder.Entity<RolePermission>().ToTable("RolePermission");

            modelBuilder.Entity<RolePermission>()
                .HasKey(t => new { t.RoleId, t.PermissionId });
}

CreatedBy & LastUpdatedBy are not scalar properties which would have corresponding columns. They are navigations. Since they are defined on User and also target User type, together they form a self-referencing foreign key navigation pair. One of those navigation CreatedBy is picked as dependent to principal navigation hence the shadow FK created is named CreatedById and there is not LastUpdatedById (since there is nothing to map to it).
If your intention is to have 2 foreign keys, 1 for each navigation then you need to configure them manually in fluent API to tell EF that, your navigations does not have inverse navigations and don't combine them in single FK.


Telemetry Error with Redix 0.10.2

I'm using pow 1.0.9 with redix 0.10.2.  I followed the Redis guide and when I run the tests I get the following error.  Any idea what the problem is?
     ** (ArgumentError) argument error
     code: assert PowRedisCache.get(@default_config, "key") == :not_found
     stacktrace:
       (stdlib) :ets.lookup(:telemetry_handler_table, [:redix, :pipeline])
       /Users/jackclark/workspace/ex338/deps/telemetry/src/telemetry_handler_table.erl:47: :telemetry_handler_table.list_for_event/1
       /Users/jackclark/workspace/ex338/deps/telemetry/src/telemetry.erl:114: :telemetry.execute/3
       lib/redix.ex:758: Redix.pipeline_without_checks/3
       lib/redix.ex:563: Redix.command/3
       (ex338) lib/ex338_web/pow_redis_cache.ex:34: Ex338Web.PowRedisCache.get/2
       test/ex338_web/pow_redis_cache_test.exs:10: (test)

Away from the laptop but my hunch is that redix has not been started before running the tests. This is a shot in the dark, but maybe if you add the following to your test_helper.exs similar to how you start ecto:
{:ok, _conn} = Redix.start_link(name: :redix)

If that’s not working then I’ll get back to this soon.

Updating test_helper.exs returns the following error:
** (MatchError) no match of right hand side value: {:error, {:already_started, #PID<0.410.0>}}

I forgot to mention I don't receive the error with {:redix, "~> 0.9.2"}, only when I update to the latest redix version {redix, "~> 0.10.2"}.

@axelclark I created a test app, added all your dependencies from ex338 and still the tests pass when using {redix, "~> 0.10.2"} ... Something strange is going on with your app setup that I can't put the finger on 🤔

@humancopy thanks!  It’s good to know it is within my app setup.
I’m guessing it has to do with how I’m passing options in to Redix in my lib/ex338/application.ex.  I will take a look later tonight.

You got pretty old app setup. You can fix this by updating your application/0 method in mix.exs to use the :extra_applications option instead of :applications:
  def application do
    [
      mod: {Ex338.Application, []},
      extra_applications: [
        :cowboy,
        :logger,
        :gettext,
        :phoenix,
        :phoenix_pubsub,
        :phoenix_html,
        :phoenix_ecto,
        :postgrex,
        :swoosh,
        :calendar,
        :phoenix_swoosh,
        :honeybadger,
        :scrivener_ecto,
        :runtime_tools
      ]
    ]
  end

Works for me when I set it up in your repo. You would have experienced the same issue if you upgrade to ecto 3.0. Most of the above apps can also be removed since they are automatically inferred (by default only :logger and :runtime_tools needs to be there).
This was a change in Elixir 1.4 (check the "Application inference" section): https://elixir-lang.org/blog/2017/01/05/elixir-v1-4-0-released/
Might be a good idea to check the release notes up to 1.8 just to see if you got your app all up to date 😄

@humancopy & @danschultzer thank you so much!  You both are great!


bity.com cashout functionality makes the user wait too long

Scope

With merging #164 into master, on Exchange.js there us now an option for the user to cash out into their bank account using bity.com
To cashout using bity.com, the burner-wallet does the following: (1) Estimates the ETH/$ rate, (2) creates a bity.com order, (3) sends some ETH to a designated bity.com ETH address
That process can take a lot of time
Right now, the user is presented with the <Loader /> component throughout all this process
While it is very likely for the user to be confused about the long waiting time, ironically it's really important that the process is not canceled by the user throughout. This means, the user needs to sit through the "cashout" loading screen without doing anything
This yields a very high likely hood of the process not completing in many cases
Other Exchange.js functionality pretends to run asynchronously (it doesn't), and gives the user regular feedback of what's happening (e.g. DAI => PDAI)

Deliverables

Write a serviceworker script for the bity.com cashout functionality.
Store all ephemeral values into localStorage to be able to pick up the cashout task at any time again
Notify the user through a loading bar in e.g. Exchange.js about the progress of the order
Consider that Exchange.js will likely go away in the future, so maybe a progress in history would make more sense?
The user should be able to start a cashout process now, exit the app, reload it, and the cashout process should at all times be able to resume and successfully finish

Gains for the project

User has piece of mind when cashing out

Roles
bounty gardener: @TimDaub / 10%
bounty worker: name / 75%
bounty reviewer: name / 15%

I would also consider testing concurent cashouts. I've had a case using localbitcoins, when I would make several transfers within 5 minute window frame :)


no-lockfile: true complaining lockfile doesn't exist?

Opening an issue here as wyvox/action-setup-pnpm doesn't have an issues section.
My Floating dependencies test in ember-shiki fails with the following error:
Run wyvox/action-setup-pnpm@v2
Run if [ ! -f "package.json" ]; then
--- Debug ---
inputs.pnpm-version:
packageManager:
volta.pnpm:
--- ----- ---
No pnpm-version override detected
package.json#volta.pnpm was not defined
package.json#packageManager () did not start with 'pnpm' or was not defined.
--
Resolved pnpm @ 8
Run echo "Detected option --no-lockfile. Lockfile will be deleted before install."
Detected option --no-lockfile. Lockfile will be deleted before install.
Run pnpm/action-setup@v2
Running self-installer...
Installation Completed!
Run actions/setup-node@v3
Found in cache @ /opt/hostedtoolcache/node/16.20.2/x64
Environment details
/home/runner/setup-pnpm/node_modules/.bin/pnpm store path --silent
/home/runner/setup-pnpm/node_modules/.bin/store/v3
Error: Dependencies lock file is not found in /home/runner/work/ember-shiki/ember-shiki. Supported file patterns: pnpm-lock.yaml

Action logs: https://github.com/IgnaceMaes/ember-shiki/actions/runs/6053509551/job/16450716280
PR: https://github.com/IgnaceMaes/ember-shiki/pull/16

It seems like after the install there is no generated lockfile? It didn't occur before so I'm quite puzzled what triggered this sudden failure.

Hm, well, the no-lockfile option does delete the lockfile. https://github.com/wyvox/action-setup-pnpm/blob/main/action.yml#L113
If you know the gh secrets to make it behave the same as pnpm-args: --no-lockfile, that'd be better
(Also, i just enabled issues. Apologies!)


Toolchain fix for older python version

This fixes the problem that I had building on wingtip for python 3.8.10 thanks to @henryleberre

No idea why this change messed with the benchmark time

@henryleberre can merge this whenever it is done

Thanks @wilfonba. The increase in benchmark time is due to:

Only running a few time steps
Storing benchmark numbers as integers (seconds).

I believe @sbryngelson will add more cases.

@wilfonba @henryleberre yeah it's in my queue


DMP-58 Set up App Insights - 2nd Attempt

…also updated in prep to push to new Azure resources.
Before creating a pull request make sure that:

[ ] commit messages are meaningful and follow good commit message guidelines
[ ] README and other documentation has been updated / added (if needed)
[ ] tests have been updated / new tests has been added (if needed)

Please remove this line and everything above and fill the following sections:
JIRA link (if applicable)
https://tools.hmcts.net/jira/browse/DMP-58
Change description
Work to add Azure Application Insights to the three microservices.
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[x] No

@scott-robertson1 I think these features need to be in a feature block. https://github.com/hmcts/darts-api/blob/c54a718760fab267e9b4090e0f62935dac45334c/infrastructure/main.tf#LL42C1-L45C55
I'm getting a pipeline error. https://sds-build.platform.hmcts.net/blue/organizations/jenkins/HMCTS%2Fdarts-api/detail/PR-42/59/pipeline/219
Cheers

@scott-robertson1 I think these features need to be in a feature block. https://github.com/hmcts/darts-api/blob/c54a718760fab267e9b4090e0f62935dac45334c/infrastructure/main.tf#LL42C1-L45C55
I'm getting a pipeline error. https://sds-build.platform.hmcts.net/blue/organizations/jenkins/HMCTS%2Fdarts-api/detail/PR-42/59/pipeline/219
Cheers

No, maybe not. https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_provider_registration

@scott-robertson1 I think these features need to be in a feature block. https://github.com/hmcts/darts-api/blob/c54a718760fab267e9b4090e0f62935dac45334c/infrastructure/main.tf#LL42C1-L45C55
I'm getting a pipeline error. https://sds-build.platform.hmcts.net/blue/organizations/jenkins/HMCTS%2Fdarts-api/detail/PR-42/59/pipeline/219
Cheers

No, maybe not. https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_provider_registration

https://discuss.hashicorp.com/t/at-least-1-features-blocks-are-required-issue/44511/2

@jackmaloney  and @davet1985, it looks as though we are at a point where we can now move forward with this. We Discussed this on Friday Dave. Once the vault secrets have been created on staging they'll be available for the PR, then we'll be able to check the PR With the references to those secrets exposed.

@jackmaloney and @davet1985, it looks as though we are at a point where we can now move forward with this. We Discussed this on Friday Dave. Once the vault secrets have been created on staging they'll be available for the PR, then we'll be able to check the PR With the references to those secrets exposed.

The branch needs updating and I think has some conflicts again. If you can sort that out I can review and we can potentially merge this part of it.

Morning @jackmaloney, branch has been updated. Let me know what the next stage of the process is. Thanks


Set TMPDIR value correctly if it's nil.

build.sh set TMPDIR  to string "getconf DARWIN_USER_TEMP_DIR" if TMPDIR is empty.
This causes instlation failure. (when TMPDIR is empty)
getconf DARWIN_USER_TEMP_DIR should be enclosed by $().

Thanks for the PR, @nakiwo! We'll need you to accept our Contributor License Agreement before we can merge your contributions.
Cheers,
The robots and humans at Realm

Thanks @nakiwo!


Contour deletes its ingress routes when etcd cluster is down.

What steps did you take and what happened:
[A clear and concise description of what the bug is.]
We are running Contour and Envoy in different pods, Contour runs as a Deployment and Envoy runs as a Daemonset.
Also, Envoy is running utilizing host networking within the Daemonset.
The issue we are seeing is that some of the Envoy's won't be able to connect to gRPC(etcd) cluster whose cluster is down state.
In case, if we update ingressroute then it's disable all existing Envoy's ingress active entries.
And Envoy configuration 'dynamic_active_clusters' is deleted.
// ingressroute file
$ cat ingress-test.yaml
apiVersion: contour.heptio.com/v1beta1
kind: IngressRoute
metadata:
  name: foo
  namespace: foobar
spec:
  virtualhost:
    fqdn: foobar.com
  routes:
    - match: /
      services:
        - name: service0
          port: 80
          healthCheck:
            path: /healthy
            intervalSeconds: 3
            timeoutSeconds: 1
            unhealthyThresholdCount: 3
            healthyThresholdCount: 5
          strategy: WeightedLeastRequest
+        - name: add_later_service1
+          port: 80
+          strategy: WeightedLeastRequest
+          healthCheck:
+            path: /healthy
+            intervalSeconds: 3
+            timeoutSeconds: 1
+            unhealthyThresholdCount: 3
+            healthyThresholdCount: 5

// envoy logs
// gRPC(etcd) cluster is down
[2019-05-27 06:35:18.959][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 13,
[2019-05-27 06:35:18.959][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 13,

// add `add_later_service1`
// envoy removes its route
[2019-05-27 06:35:18.988][000001][info][upstream] [source/server/lds_api.cc:72] lds: remove listener 'ingress_https'
[2019-05-27 06:35:18.988][000001][info][upstream] [source/server/lds_api.cc:72] lds: remove listener 'ingress_http'
[2019-05-27 06:35:19.192][000001][info][upstream] [source/common/upstream/cluster_manager_impl.cc:554] removing cluster foo/bar/80/buz
[2019-05-27 06:35:23.386][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 13,
[2019-05-27 06:35:23.386][000001][warning][upstream] [source/common/config/grpc_mux_impl.cc:268] gRPC config stream closed: 13,

// envoy configuration 'dynamic_active_clusters' is deleted when etcd cluster was down
--- config_dump_bad2    2019-05-29 08:14:25.541271926 +0900
+++ config_dump_normal  2019-05-29 07:53:07.589444053 +0900
@@ -263,6 +263,78 @@
      },
      "last_updated": "2019-05-27T06:59:32.046Z"
     }
+   ],
+   "dynamic_active_clusters": [
+    {
+     "version_info": "0",
+     "cluster": {
+      "name": "foo/bar/80/baz",
+      "type": "EDS",
+      "eds_cluster_config": {
+       "eds_config": {
+        "api_config_source": {
+         "api_type": "GRPC",
+         "grpc_services": [
+          {
+           "envoy_grpc": {
+            "cluster_name": "contour"
+           }
+          }
+         ]
+        }
+       },
+       "service_name": "foo/bar/port-80"
+      },
+      "connect_timeout": "0.250s",
+      "lb_policy": "LEAST_REQUEST",
+      "health_checks": [
+       {
+        "timeout": "1s",
+        "interval": "3s",
+        "unhealthy_threshold": 3,
+        "healthy_threshold": 5,
+        "http_health_check": {
+         "host": "contour-envoy-healthcheck",
+         "path": "/healthy"
+        }
+       }
+      ],
+      "common_lb_config": {
+       "healthy_panic_threshold": {}
+      },
+      "alt_stat_name": "foo_bar_80",
+      "drain_connections_on_host_removal": true
+     },
+     "last_updated": "2019-05-28T02:06:50.452Z"
+    },
+    {
...

What did you expect to happen:
If etcd cluster was down, Envoy pods did not update its traffic rule.
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:
Contour version: v0.11.0
Kubernetes version: (use kubectl version): 1.14.0
Kubernetes installer & version: 1.14.0
Cloud provider or hardware configuration: Xeon E5-2683v4 2.10GHz / 2CPU / 512GBMEM RAM / SATA SSD 240GB x 1 / 10G Base-T*2
OS (e.g. from /etc/os-release): Ubuntu 16.04.6 LTS
CNI : Calico v3.4

Thank you for raising this issue. I think there are two issues here. I want to talk about the second issue, envoy removing the cluster when you edit the ingressroute. What may be happening is the ingressroute object is not valid after editing and so the configuration associated with it is being removed. Do you think this is what could be happening?

I think this issue is caused by read-only mode on etcd cluster.
While etcd cluster was read-only mode, we update ingressroute.
And then, Envoy pods expect a valid etcd entry(ingressroute object) but there is no data.
So, Envoy pods removed the invalid ingressroute object.

Thank you for replying. If there is a problem with the etcd cluster underlying your k8s cluster, pretty much all bets are off. You need to address that problem first to restore the invariants that kubernetes relies on.

In this case, even if we notice that etcd cluster is down by failures or hypervisor problems, users may not notice that.
And we cannot stop Contour system soon after the etcd failure.
So, we should disable for users to update ingressroute while etcd cluster is down.

Contour relies on the invariants guaranteed by the Kubernetes api.
I’m sorry but asking contour to diagnose the state of the api server’s backing etcd is not something we will do.

On 30 May 2019, at 16:59, hmorikaw<EMAIL_ADDRESS>wrote:
In this case, even if we notice that etcd cluster is down by failures or hypervisor problems, users may not notice that.
And we cannot stop Contour system soon after the etcd failure.
So, we should disable for users to update ingressroute while etcd cluster is down.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

@hmorikaw did I understand the issue correctly? Is the etcd server you refer to the API server for your k8s cluster?
If so, as there is nothing that Contour can do when deployed on a misbehaving k8s cluster, I would like to close this issue. Please let me know what you think.

I mean some codes delete envoy routes if etcd cluster is read-only mode.
Does contour have some codes running like this behavior?

@hmorikaw thank you for your reply.
Before I can answer your question I need to know more about the role of this etcd server? Is this etcd server the etcd server that backs your k8s cluster?

@hmorikaw hello, would you be able to answer my previous question about the role of the faulty etcd server. Thank you

yes. This etcd cluster backs my k8s cluster's API server and k8s nodes.

@hmorikaw thank you for clarifying.
In that case my answer is the same as https://github.com/heptio/contour/issues/1128#issuecomment-498117102. If the underlying API sever is not behaving correctly then there is nothing Contour can do.

I see. Thank you for your reply. I'll expand etcd servers and close this issue.


React questions

Description
😉 In reference to #110.
What does your PR belong to?

[x] Questions / Answers
[x] Website
[ ] General / Things regarding the repository (like CI Integration)

Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[x] Enhancement (non-breaking improvement of a question)
[x] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] I have checked that the changes are working properly
[x] I have checked that there isn't any PR doing the same
[x] I have read the CONTRIBUTING document.

@atomiks, we should only tag as JS the snippets that are relevant to vanilla JS, as well as React then. Let's make the two PRs consistent before we merge them.


chore: check tags using GitHub API before cloning.

For analysis targets that would require the cloning of a GitHub repository, and have a provided version but no provided digest, this PR adds an additional pre-step that performs the Commit Finder tag matching process on the first page of tag results returned via the GitHub API for that repository.
One of the following outcomes result from this process:

The version matched a tag within the results -> No change
The version did not match a tag within the results, but there are more tags than the first page of results contains -> No change
The version did not match a tag within the results, and the tags within the results are an exhaustive list -> Do not clone the repository

Closes #644
Does not include #576

As we have discussed internally to stop exploring this approach, should we close this PR?


Have to flush the text writer after WriteStream.

You have to flush the text writer after calling WriteSteam or you will end up with an incomplete stream.

Hello lukekini
Text writers and streams are used in several places in FileHelpers.
And your statement is very general.
The following would help us considerably:

Indicate where the text writer and stream handling is erroneous.
Describe which methods and types of the library you use, how the library behaves and what your expectation is.

Thank you for supporting us, Matthias

Hello lukekini
Is there any chance that you can show us where the problem is? Is the problem similar to
http://stackoverflow.com/questions/40119550/filehelpers-writestream-only-writing-the-first-2048-characters-to-output/40119574#40119574 ?

Seems that we have to verify the behavior by ourselves.
Hopefully we get some support from Code Analyze / FxCop.

The FileHelperEngine write the stream in a using statement, so it will auto close and flush the stream
If you use the FileHelperAsyncEngine you MUST call the Close method or better yet use the using statement
using(engine.BeginWriteFile("TestOut.txt"))
{
    foreach(Customer cust in arrayCustomers)
    {
        engine.WriteNext(cust);
    }
}

http://www.filehelpers.net/docs/html/M_FileHelpers_FileHelperAsyncEngine_1_Close.htm
@Iukekini do you have more info about this problem ?


ClientClass.GetClassId() returns 0 every time

Once you got a fully loaded client ( ingame ) they are set.


Gradle build Error

Hi bro can't build for this project .
Could not find com.tencent.mm:AndResGuard-gradle-plugin:1.2.0.
Searched in the following locations:
https://jcenter.bintray.com/com/tencent/mm/AndResGuard-gradle-plugin/1.2.0/AndResGuard-gradle-plugin-1.2.0.pom
https://jcenter.bintray.com/com/tencent/mm/AndResGuard-gradle-plugin/1.2.0/AndResGuard-gradle-plugin-1.2.0.jar
Required by:
:ListenerMusicPlayer:unspecified

I have not used com.tencent.mm:AndResGuard-gradle-plugin:1.2.0.
Maybe you can delete this plugin or upgrade to the latest version.


joint_trajectory_controller should not use now() but rely on time passed in through the update() function

Background
joint_trajectory_controller should not use now() but rely on time passed in through the update() function.
The goal of this ticket is to remove the usage of node_->now() from joint_trajectory_controller.cpp e.g. here and replace it with the time parameter passed in here.
Any usage of node_->now() that is in the same call tree as update(), meaning it is a function called within update() should also be eliminated. If needed, extend those function's parameters with time.
Instructions
Hi, this is a good-first-issue issue. This means we've worked to make it more legible to people who either haven't contributed to our codebase before, or even folks who haven't contributed to open source before.
We're interested in helping you take the first step, and can answer questions and help you out along the way. Note that we're especially interested in contributions from underrepresented groups!
We know that creating a pull request is the biggest barrier for new contributors. This issue is for you 💝
If you have contributed before, consider leaving this PR for someone new, and looking through our general bug issues. Thanks!
🤔 What you will need to know.
Nothing. This issue is meant to welcome you to Open Source :) We are happy to walk you through the process.
📋 Step by Step

[ ] 🙋 Claim this issue: Comment below. If someone else has claimed it, ask if they've opened a pull request already and if they're stuck -- maybe you can help them solve a problem or move it along!

[ ] 🗄 Create a local workspace for making your changes and testing following these instructions, for Step3 use "Download Source Code" section with these instructions.

[ ] 🍴 Fork the repository using the handy button at the top of the repository page and clone it into ~/ws_ros2_control/src/ros-controls/ros2_controllers, here is a guide that you can follow (You will have to remove or empty the existing ros2_controllers folder before cloning your own fork)

[ ] Checkout a new branch using git checkout -b <branch_name>

[ ] 🤖 Apply pre-commit auto formatting, by running pip3 install pre-commit and running pre-commit install in the ros2_control repo.

[ ] 💾 Commit and Push your changes

[ ] 🔀 Start a Pull Request to request to merge your code into master. There are two ways that you can start a pull request:

If you are not familiar with GitHub or how to create a pull request, here is a guide you can follow on how GitHub works.

[ ] 🏁 Done Ask in comments for a review :)

Is someone else already working on this?
🔗- We encourage contributors to link to the original issue in their pull request so all users can easily see if someone's already started on it.
👥- If someone seems stuck, offer them some help!
🤔❓ Questions?
Don’t hesitate to ask questions or to get help if you feel like you are getting stuck. For example leave a comment below!
Furthermore, you find helpful resources here:

ROS2 Control Contribution Guide
ROS2 Tutorials
ROS Answers

Good luck with your first issue!

I would like to work on this issue @bmagyar

Hey Hi!
Sorry if this is a dumb question but I have one doubt, do we have a test suite to check if everything is working? I mean, to be sure that I can change the code without breaking anything?
Can you assign this to me as my first issue?

We have quite a few tests for controllers, never enough of course but there should be plenty of coverage for this job

Amazing, thank you @bmagyar !


Documentation inconsistent with UI labels

Often, throughout the Azure DevOps documentation the words used in the documentation do not match the labels in the UI, and this leads to tremendous confusion. For example, on this page, you describe "Portfolio, product, and sprint backlogs," but when you actually look in ADO in the process templates, the labels used in the UI are "Portfolio," "Requirements," and "Iteration." It would be much easier if you were consistent between documentation and UI labels.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: eca2985c-878e-ee5d-9df3-ad5755abd873
Version Independent ID: cf61abed-bd24-817f-7d28-f9f8a0002926
Content: What tasks do backlogs, boards, & plans support? - Azure Boards
Content Source: docs/boards/backlogs/backlogs-boards-plans.md
Product: devops
Technology: devops-agile
GitHub Login: @KathrynEE
Microsoft Alias: kaelli

@shawnhernan, @martinessiet - Thank you for this feedback. I've left this issue opened for as long as I have as I wasn't sure how to address it. I agree the PDF is of no use as it doesn't provide support for versioned content and instead includes all content for all versions. This is a known issue which is being investigated. Also, the problem with consistent terminology is being addressed and we hope to provide better and more consistent content in the future.


Error handler is not skipping past the error properly when handled on a higher level object

Source/destination types
public enum Color { Red, Green, Blue }

public class MyClass
{
    public IList<Color> Values { get; set; }
}

Source/destination JSON
{"Values":["Black", "Green"]}

Expected behavior
In the above example we have a class with a list of Color enum objects, and a Json with an invalid color ('Black').
Normally, when using an error handler and handling the error immediately on the original object level, the deserialization will succeed and the item will have the default value (0 or 'Red' in this case).
new JsonSerializerSettings() { Error = (sender, e) => { e.ErrorContext.Handled = true; }}

However, if we try to handle the error on a higher level object (e.g. on the MyClass object) like this:
new JsonSerializerSettings()
{
    Error = (sender, e) =>
    {
        if (e.CurrentObject is MyClass)
        {
            e.ErrorContext.Handled = true;
        }
    }
}

then the expected behavior is that the 'Values' list will be null, or at least that deserialization succeeds with some result.
Actual behavior
The actual behavior is that deserialization fails with exception, because the reader is not skipped properly after handling the error (in JsonSerializerInternalReader.HandleError).
The exception is:
Unexpected token when deserializing object: String. Path 'Values[1]'
Steps to reproduce
var myClass = JsonConvert.DeserializeObject<MyClass>("{\"Values\":[\"Black\", \"Red\"]}",
new JsonSerializerSettings()
{
    Error = (sender, e) =>
    {
        if (e.CurrentObject is MyClass)
        {
            e.ErrorContext.Handled = true;
        }
    }
});

This issue has a proposed fix with pull request #1550
The example above was actually fixed with commit f88ae288bc00836760ee8dfab526e4b944a191bb, however that commit only handled the Object case, while the PopulateDictionary still suffers from the same issue.
The added unit test in pull request #1550 demonstrates the failed case.


Utilizing Cuda cores for frame stabilization

As the frame size is increased  - Stabilization starts to experience delay.  This possibly could be improved by utilizing Cuda cores of Nvidia GPUs
Detailed Description
I have performed stabilization tests on live-feed with different resolutions and noticed that low resolution frames get processed smoothly even if the CPU (Ryzen 3600) has high utilization. When the resolution is increased to 720p Stabilized image starts to lag behind causing audio sync issues and the video skips frames as the delay grows too large.  Even when CPU load is reduced the stabilized feed might lag.
Context
Ability to handle 720p live-content with stabilizer would be great as current software in most mobile phones does not allow to use built-in stabilization. Many live streamers would benefit if a moderate gaming system would be capable of stabilizing at least 720p video feed smoothly.
Your Current Environment

VidGear version:  0.1.9
Branch: Development
Python version:  3.9.0
Operating System and version: Windows 10 2004
CPU: Ryzen 3600
GPU: Geforce 1070TI

Any Other Important Information
Video demonstrating the behavior with time stamps.
https://youtu.be/ysvaSvLdjLE

@Nizcik Can you provide raw un-stablized 720p footage here directly (GitHub comments now support video attachments) or YouTube or else way. I need to run some tests.

@Nizcik Can you provide raw un-stablized 720p footage here directly (GitHub comments now support video attachments) or YouTube or else way. I need to run some tests.

@abhiTronix Here you go. There was limit of 10mb uploading directly to github
https://mega.nz/folder/GARTySKI#i1Vm6HKB92AlLSzWJW-T0g
1080p footage was also 5 minutes, but it got very laggy at the end so I cut that to 1 minute. Let me know if you need more footage or if this is not good enough.
Happy new year!

@abhiTronix Here you go. There was limit of 10mb uploading directly to github
https://mega.nz/folder/GARTySKI#i1Vm6HKB92AlLSzWJW-T0g
1080p footage was also 5 minutes, but it got very laggy at the end so I cut that to 1 minute. Let me know if you need more footage or if this is not good enough.
Happy new year!

@Nizcik Could perform same test as before .i.e.  Video demonstrating the behavior with time stamps by installing latest vidgear that uses experimental Queue instead of deques, and see if this made any difference. You can install latest vidgear development branch as follows:
  # clone the repository and get inside
  git clone https://github.com/abhiTronix/vidgear.git && cd vidgear

  # checkout the latest development branch
  git checkout development

  # install normally
  pip install .

  # OR install with asyncio support
  pip install .[asyncio]

After installation run those test with 720p video input. Thanks in Advance.

@Nizcik Could perform same test as before .i.e.  Video demonstrating the behavior with time stamps by installing latest vidgear that uses experimental Queue instead of deques, and see if this made any difference. You can install latest vidgear development branch as follows:
  # clone the repository and get inside
  git clone https://github.com/abhiTronix/vidgear.git && cd vidgear

  # checkout the latest development branch
  git checkout development

  # install normally
  pip install .

  # OR install with asyncio support
  pip install .[asyncio]

After installation run those test with 720p video input. Thanks in Advance.

@Nizcik Any updates on this?

@Nizcik Any updates on this?

Sorry, I noticed your message only now. I made a quick test and it seemed that there is slight improvement. Still now when I loaded the cpu a bit more I experienced a lag. However with light CPU usage 720p 30fps seemed to stabilize in real time.
I will try to do a longer field test tomorrow and let you know.
Just a note, I experimented with lower resolutions also earlier and I got lag even with lowest resolutions then after maybe 10 minutes or so, and suddenly stabilization processing slowed down.
Anyways - Ill test this more.

Sorry, I noticed your message only now. I made a quick test and it seemed that there is slight improvement. Still now when I loaded the cpu a bit more I experienced a lag. However with light CPU usage 720p 30fps seemed to stabilize in real time.
I will try to do a longer field test tomorrow and let you know.
Just a note, I experimented with lower resolutions also earlier and I got lag even with lowest resolutions then after maybe 10 minutes or so, and suddenly stabilization processing slowed down.
Anyways - Ill test this more.

Still now when I loaded the cpu a bit more I experienced a lag.

@Nizcik CamGear now utilizes all CPU cores with this queue approach. That's why it is hanging a bit. I'll advise to remove
    # Show output window
    cv2.imshow("Output Frame", frame)

    # check for 'q' key if pressed
    key = cv2.waitKey(1) & 0xFF
    if key == ord("q"):
        break

from your code to improve performance significantly.

Just a note, I experimented with lower resolutions also earlier and I got lag even with lowest resolutions then after maybe 10 minutes or so, and suddenly stabilization processing slowed down.

Did you checked the CPU utilization? Add a CPU widget if on Windows 10, and a GPU monitor to see both CPU an GPU usage in real-time.

Still now when I loaded the cpu a bit more I experienced a lag.

@Nizcik CamGear now utilizes all CPU cores with this queue approach. That's why it is hanging a bit. I'll advise to remove
    # Show output window
    cv2.imshow("Output Frame", frame)

    # check for 'q' key if pressed
    key = cv2.waitKey(1) & 0xFF
    if key == ord("q"):
        break

from your code to improve performance significantly.

Just a note, I experimented with lower resolutions also earlier and I got lag even with lowest resolutions then after maybe 10 minutes or so, and suddenly stabilization processing slowed down.

Did you checked the CPU utilization? Add a CPU widget if on Windows 10, and a GPU monitor to see both CPU an GPU usage in real-time.

@Nizcik Any updates on this?

@Nizcik Any updates on this?

@Nizcik I'll just call it a day till my college is over, I just tested cupy and it sorta works on my AMD setup but I need a proper NVIDIA hardware to test it performance which I'll be purchasing at sometime in 2022. So stay tuned for any future updates.


[BEAM-11408] Integrate Python BigQuery sink with GroupIntoBatches

This is a parity of Java changes (#13496 and #13859).
Use GroupIntoBatches.WithShardedKey API to group and batch write before streaming to BigQuery service. Currently batching is done best-effort upon bundle finalization.
This PR adds an option to WriteToBigQuery to toggle between the existing and new implementation for both streaming inserts and file loads.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @pabloem
CC: @reuvenlax

Run Python PreCommit

Run Python 3.8 PostCommit

Thanks for the review!
I got the following lint errors which are supposed to be suppressed. Any ideas?
17:47:48 ERROR: /home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonLint_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/io/gcp/bigquery_test.py Imports are incorrectly sorted.
17:47:48 --- /home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonLint_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/io/gcp/bigquery_test.py:before	2021-02-23 01:43:30.069624
17:47:48 +++ /home/jenkins/jenkins-slave/workspace/beam_PreCommit_PythonLint_Commit/src/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/io/gcp/bigquery_test.py:after	2021-02-23 01:47:43.161598
17:47:48 @@ -38,6 +38,10 @@
17:47:48  import mock
17:47:48  import pytz
17:47:48  from nose.plugins.attrib import attr
17:47:48 +# Protect against environments where bigquery library is not available.
17:47:48 +# pylint: disable=wrong-import-order, wrong-import-position
17:47:48 +from parameterized import param
17:47:48 +from parameterized import parameterized

I think you can just move lines 80-81 to be 41-42, and that should be enough?

Run Python 3.8 PostCommit

Run Python 3.8 PostCommit

thanks @nehsyc ! this will be in Beam 2.29.0 - excited to set it to the wild : )


pyswagger can handle the properties of object type is empty or not ?

Hi mission,
I wonder whether pyswagger can handle this situation or not?
"plan":{
   "type":"object",
   "additionalProperties":true,
   "properties":{

   }
},
"oplans":{
   "items":{
      "$ref":"#/definitions/plan"
   },
   "type":"array",
   "description":"List of oplans"
},

When I run client, the parameter of plan can't be validate successfully.
Thanks,
Shuang

Hi, I try this schema
{
   "swagger":"2.0",
   "host":"http://test.com",
   "basePath":"/v1",
   "paths":{
      "/t":{
         "get":{
            "operationId":"kk",
            "parameters":[
               {
                  "in":"body",
                  "name":"body",
                  "schema":{
                     "$ref":"#/definitions/oplans"
                  }
               }
            ],
            "responses":{
               "default":{
                  "description":"void"
               }
            }
         }
      }
   },
   "definitions":{
      "plan":{
         "type":"object",
         "additionalProperties":true,
         "properties":{

         }
      },
      "oplans":{
         "items":{
            "$ref":"#/definitions/plan"
         },
         "type":"array",
         "description":"List of oplans"
      }
   }
}

with this code snippet:
from pyswagger import SwaggerApp

app = SwaggerApp.create('swagger.json')

op = app.op["kk"]
req_and_resp = op(body=[
    dict(a=1, b=2, c=3),
    dict(name='john', email='xx@gmail.com'),
])

and it didn't raise any exception. If it's not what you expect, could you provide your code snippet / schema for me to investigate?

Hi, mission,
I try this schema , I modify the code snippet:
from pyswagger import SwaggerApp
app = SwaggerApp.create('swagger.json')
op = app.op["kk"]
req, resp = op(body=[
dict(a=1, b=2, c=3),
dict(name='john', email='xx@gmail.com'),
])
print req._p["body"]["body"]
please see the body value of req, the data (a=1, b=2, c=3) and (name='john'<EMAIL_ADDRESS>is not validated and pass to body of req.
(virtualenv)-bash-4.1$ python test.py  {},{}

ya, u r right, that seems a bug.

I think I found the bug, for some reason, the cleanup of Model primitive is not called.
You could patch pyswagger.primitive.init.py (L224) from
if ret and cleanup and hasattr(ret, 'cleanup'):

to
if ret != None and hasattr(ret, 'cleanup'):

before I provide an official fix.

Ok, That's great, Thanks!
Shuang
2016-03-16 23:35 GMT+08:00 mission.liao<EMAIL_ADDRESS>
I think I found the bug, for some reason, the cleanup of Model primitive
is not called.
You could patch pyswagger.primitive.init.py (L224) from
if ret and cleanup and hasattr(ret, 'cleanup'):
to
if ret != None and hasattr(ret, 'cleanup'):
before I provide an official fix.
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/mission-liao/pyswagger/issues/73#issuecomment-197386469

Hi mission,
I want to confirm with you about this: After you fix this bug and the
default additionproperties value, whether this schema object can be
validate successfully?
"oplans": {
"items": {
"type":"object",
"additionalproperties": true
},
"type": "array",
"description": "List of oplans"
},
The "oplans" is an array of object, and the content of object can be
anything I want.
Thanks.
Shuang
2016-03-16 23:55 GMT+08:00 Willa luo<EMAIL_ADDRESS>
Ok, That's great, Thanks!
Shuang
2016-03-16 23:35 GMT+08:00 mission.liao<EMAIL_ADDRESS>
I think I found the bug, for some reason, the cleanup of Model primitive
is not called.
You could patch pyswagger.primitive.init.py (L224) from
if ret and cleanup and hasattr(ret, 'cleanup'):
to
if ret != None and hasattr(ret, 'cleanup'):
before I provide an official fix.
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/mission-liao/pyswagger/issues/73#issuecomment-197386469

I think your schema object should be ok, I'll include it as a testing case to make sure it works in your workaround. But remember that, "additionalProperties: true" is not spec-compliant and might be fixed in the future.


Datetime picker not fully showing inside a .table-responsive class NOT CLOSED

You closed my initial response #1420, but it is not closed, you did not reply to my comment and there is still an issue when nested within a .table-responsive class. Are you aware of this? Have you tried to fix this?
The comment on my closed thread is:
@Eonasdan I see no errors in my console log (Firefox), as far as I can tell your fiddle is exactly the same as mine and still does not work? Still does not show the date picker outside the table unless in the css you add .table-responsive { overflow:visible !important } in the css?
EDIT: I see I added the external CSS file incorrectly, but other than that, the picker does not show the date picker outside the table
https://jsfiddle.net/daza110/6abxn644/3/ ---- see that I have added the overflow CSS element, delete that, run it and then you will see the difference.
Also see my stack thread which no one has yet answered too.
http://stackoverflow.com/questions/34811084/bootstrap-datetimepicker-issue-instide-a-table-responsive-class

Any update on this. I am still facing the same issue.

Still not fixed still no response

Hola que tal... tengo el mismo problema...! alguien poddria ayudar porfavor!!!

Quick workaround
$('#picker').datetimepicker().on('dp.show', function() {
$(this).closest('.table-responsive').removeClass('table-responsive').addClass('temp');
}).on('dp.hide', function() {
$(this).closest('.temp').addClass('table-responsive').removeClass('temp')
});


[Question] Is it possible to manually operate browser in a container?

Hey.
Is there a way to operate browsers manually? It would be useful for the manual QA team if they can interact with browsers that are currently running in a container (since it could be installed and maintained so easily). I see that it is an option to create a session manually from UI. But it only shows the current browser's state in the container and the user can't interact with it (for example input URL with a keyboard and browse through the website).
Also, manual interaction is useful for automation purposes too. For example, if I'm debugging tests and want to inspect web elements of a particular screen with help of the browser's dev tools.

@7artur7 yes, Selenoid UI has a feature to create manual sessions.

@vania-pooh yep. But it doesn't allow me to interact with a browser. It is just like an image of what is going on inside of the container. Check the video https://monosnap.com/file/YFOsOAIqJJhh7h6btYm6yKRGctC9jg right click proposes to save an image and if inspect browser screen it is an image/png data. While my question how I can interact with this browser manually - input URL from keyboard, click on elements, inspect the shown elements with google dev tools.

@7artur7 it does - simply click on violet button after creating a manual session. By default VNC is read-only to not break running automated tests.

@vania-pooh thanks, it works.

Hello, there is purple "lock" button (middle), you can unlock and interact
with browser.
ср, 14 окт. 2020 г., 21:11 7artur7<EMAIL_ADDRESS>
Hey.
Is there a way to operate browsers manually? It would be useful for the
manual QA team if they can interact with browsers that are currently
running in a container (since it could be installed in maintained so
easily). I see that it is an option to create a session manually from UI.
But it only shows the current browser's state in the container and the user
can't interact with it manually (for example input URL and browse through
it).
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/aerokube/selenoid/issues/1007, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKY23JMHMOQXA55C75PW5DSKXSWJANCNFSM4SQ6ITWA
.


[Next.Js Documentation] - Improves performance

Description

Improves performance

Checklist

[x] I read the extension guidelines
[x] I read the documentation about publishing
[x] I ran npm run build and tested this distribution build in Raycast
[x] I checked that files in the assets folder are used by the extension itself
[x] I checked that assets used by the README are placed outside of the metadata folder

Hi @PrasadChinwal 👋
Could you resolve your own merge conflicts? 😄
Request a new review when you are ready. Feel free to contact me here or at Slack if you have any questions.