How to register multiple .ome.tif at the same time & issue registering .ome.tif

Hi @Yu-AnChen !
I just tried palom on two cycles of FISH and the registration so far seems really really impressive (in particular since one of the cycles is half out of focus). First of all, thank you so much for making this tool available!
I was wondering how it would be possible to register multiple .ome.tif images simultaneously. Additionally, the registration worked fine with one of my datasets (converted from .h5 to .ome.tif which was a big pain already), however I have issues with a smaller .ome.tif dataset. I get the below error message:
(palom) [boehm0002@sgi27 ]$python3 palom_dan_ome_R1toR2.py
2023-09-04 14:12:42.717 | WARNING  | palom.reader:auto_format_pyramid:64 - Unable to detect pyramid levels, it may take a while to compute thumbnails during coarse alignment
2023-09-04 14:12:42.729 | WARNING  | palom.reader:auto_format_pyramid:64 - Unable to detect pyramid levels, it may take a while to compute thumbnails during coarse alignment
Traceback (most recent call last):
  File "/scicore/home/rueegg/boehm0002/spatial_transcriptomics/dans_test1/HiPlex_Example_DJH/palom_dan_ome_R1toR2.py", line 27, in <module>
    ref_thumbnail=c1r.read_level_channels(THUMBNAIL_LEVEL, 2).compute(),
  File "/scicore/home/rueegg/boehm0002/miniconda3/envs/palom/lib/python3.10/site-packages/dask/threaded.py", line 89, in get
    results = get_async(
  File "/scicore/home/rueegg/boehm0002/miniconda3/envs/palom/lib/python3.10/site-packages/dask/local.py", line 511, in get_async
    raise_exception(exc, tb)
  File "/scicore/home/rueegg/boehm0002/miniconda3/envs/palom/lib/python3.10/site-packages/dask/local.py", line 319, in reraise
    raise exc
  File "/scicore/home/rueegg/boehm0002/miniconda3/envs/palom/lib/python3.10/site-packages/dask/local.py", line 224, in execute_task
    result = _execute_task(task, data)
  File "/scicore/home/rueegg/boehm0002/miniconda3/envs/palom/lib/python3.10/site-packages/zarr/core.py", line 821, in __getitem__
    result = self.get_basic_selection(pure_selection, fields=fields)
  File "/scicore/home/rueegg/boehm0002/miniconda3/envs/palom/lib/python3.10/site-packages/zarr/core.py", line 947, in get_basic_selection
    return self._get_basic_selection_nd(selection=selection, out=out,
  File "/scicore/home/rueegg/boehm0002/miniconda3/envs/palom/lib/python3.10/site-packages/zarr/core.py", line 990, in _get_basic_selection_nd
    return self._get_selection(indexer=indexer, out=out, fields=fields)
  File "/scicore/home/rueegg/boehm0002/miniconda3/envs/palom/lib/python3.10/site-packages/zarr/core.py", line 1285, in _get_selection
    self._chunk_getitem(chunk_coords, chunk_selection, out, out_selection,
  File "/scicore/home/rueegg/boehm0002/miniconda3/envs/palom/lib/python3.10/site-packages/zarr/core.py", line 1994, in _chunk_getitem
    cdata = self.chunk_store[ckey]
  File "/scicore/home/rueegg/boehm0002/miniconda3/envs/palom/lib/python3.10/site-packages/zarr/storage.py", line 738, in __getitem__
    return self._mutable_mapping[key]
  File "/scicore/home/rueegg/boehm0002/miniconda3/envs/palom/lib/python3.10/site-packages/tifffile/tifffile.py", line 12274, in __getitem__
    return self._getitem(key)
  File "/scicore/home/rueegg/boehm0002/miniconda3/envs/palom/lib/python3.10/site-packages/tifffile/tifffile.py", line 12951, in _getitem
    keyframe, page, chunkindex, offset, bytecount = self._parse_key(key)
  File "/scicore/home/rueegg/boehm0002/miniconda3/envs/palom/lib/python3.10/site-packages/tifffile/tifffile.py", line 13050, in _parse_key
    page = series[pageindex]
  File "/scicore/home/rueegg/boehm0002/miniconda3/envs/palom/lib/python3.10/site-packages/tifffile/tifffile.py", line 12135, in __getitem__
    return self._getitem(int(key))
  File "/scicore/home/rueegg/boehm0002/miniconda3/envs/palom/lib/python3.10/site-packages/tifffile/tifffile.py", line 12115, in _getitem
    return self.parent.pages._getitem(page.index + key)
  File "/scicore/home/rueegg/boehm0002/miniconda3/envs/palom/lib/python3.10/site-packages/tifffile/tifffile.py", line 7568, in _getitem
    self._seek(key)
  File "/scicore/home/rueegg/boehm0002/miniconda3/envs/palom/lib/python3.10/site-packages/tifffile/tifffile.py", line 7402, in _seek
    raise ValueError('seek of closed file')
ValueError: seek of closed file

Thanks ever so much for your help!

Hi @Boehmin thanks for the kind words :)
Palom might be able to read h5 files depending on the h5 file structure. If you have a small dataset to share I'll find time to take a look.
For the .ome.tif dataset that gave you the error, it looks to me that the file might have some issues. Would you be able to share that?
To register multiple files, our recommended approach is to register everything (cycles 2, 3, 4, and so on) to the first cycle. Please checkout the example script and lmk for any questions.

Hi @Yu-AnChen
Thanks for pointing me to the example script. I'll try this!
I have a smaller fused example h5 file (only one timepoint) and the ome tif dataset here. The .h5 consists of z-stacks so I do not know if that would be an issue for palom? If not that would be even better. Channel 4 is DAPI and channel 3 is a fiducial channel so in theory I could register over the fiducial channel as well if that helps?
Again, thanks for looking into this.

Thanks for sharing the images, I was able to reproduce the error when using your ome-tiff files as input. Will look into a solution there.
On the h5 dataset (seems to be generated with BigDataViewer), I was able to convert it into a pyramidal ome-tiff using palom. Note that installing h5py in the environment that you run palom is required.
Since there are Z-stacks, I did a maximum intensity projection on the Z-axis in the following script (the output file size is ~50 MB; it can be opened in QuPath)
import palom
import h5py
import dask.array as da
import pathlib

img_path = pathlib.Path('h5_small/small_fuse_fused.h5')
h5_img = h5py.File(img_path)

# X-Y pixel size at full resolution; unit is µm/pixel
PIXEL_SIZE = 1
CHANNEL_NAMES = ['marker-1', 'marker-2', 'marker-fiducial', 'DNA']

mosaics = [
    # max projection on the Z axis
    da.from_array(h5_img[f"t00000/{channel}/0/cells"]).max(axis=0)
    for channel in h5_img['t00000']
]

palom.pyramid.write_pyramid(
    mosaics=mosaics,
    output_path=img_path.parent / f"{img_path.stem}-pyramid.ome.tif",
    pixel_size=PIXEL_SIZE,
    channel_names=CHANNEL_NAMES,
    downscale_factor=2,
    compression='zlib',
    tile_size=1024,
    save_RAM=True
)

With the output ome-tiffs from the above script, I think you should be able to run palom without running into the ValueError: seek of closed file error in your first comment.

As for the issue of ValueError: seek of closed file, I've found a workaround. Please try it out on your end - I don't have a complete solution to it yet but hopefully that can get you up and running.

Since there are Z-stacks, I did a maximum intensity projection on the Z-axis in the following script (the output file size is ~50 MB; it can be opened in QuPath)

Sorry to hijack this, is palom working on 2D files only or did you do a projection to speed it up ?
Thanks !

@Yu-AnChen, I tried both solutions and they both worked! Thank you.
Also curious whether this could work in 3D. :)

The current image file reader and writer only works with images that have 2 or 3 dimensions. In our own application, it's (Channel/Band, Y, X). Palom could compute and find out the transformation/mapping between two (C, Z, Y, X) datasets and apply the transformation plane by plane, but the image file reader and writing will need some custom formatting.

Hi @Boehmin thanks for the kind words :)
Palom might be able to read h5 files depending on the h5 file structure. If you have a small dataset to share I'll find time to take a look.
For the .ome.tif dataset that gave you the error, it looks to me that the file might have some issues. Would you be able to share that?
To register multiple files, our recommended approach is to register everything (cycles 2, 3, 4, and so on) to the first cycle. Please checkout the example script and lmk for any questions.

Hello Yu-AnChen,
Sorry to hijack the thread and thanks for this great tool. I discovered your tool after hitting a wall with Ashlar, as my data is already stitched. I used your sample script in this thread and just passed in the location of 3 of my images (CyCIF runs, each DAPI + 3 other channels) as well as an output destination. My files were originally .VSI images, which I converted to OME-TIFF files using QuPath.
I get the following error:
2024-12-15 23:21:24.058 | DEBUG    | palom.register:cv2_feature_detect_and_match:288 - keypts L:5000, keypts R:5000 2024-12-15 23:21:24.168 | DEBUG    | palom.register:cv2_feature_detect_and_match:288 - keypts L:5000, keypts R:5000 2024-12-15 23:21:24.261 | DEBUG    | palom.register:cv2_feature_detect_and_match:288 - keypts L:5000, keypts R:5000 2024-12-15 23:21:24.355 | DEBUG    | palom.register:cv2_feature_detect_and_match:288 - keypts L:4165, keypts R:4165 2024-12-15 23:21:31.402 | INFO     | palom.align:compute_shifts:203 - Computing block-wise shifts Computing shifts: 100%|##############################################################################################################################################################################| 34021/34021 [01:49<00:00, 309.32it/s] Traceback (most recent call last):   File "D:\valis_container\palom_v1.py", line 74, in <module>     fig = aligner.plot_shifts()   File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom\lib\site-packages\palom\align.py", line 276, in plot_shifts     viz_shifts(shifts, self.grid_shape, ax=axs[1])   File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom\lib\site-packages\palom\align.py", line 90, in viz_shifts     dcenter = skimage.filters.threshold_triangle(distances[is_finite])   File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom\lib\site-packages\skimage\filters\thresholding.py", line 960, in threshold_triangle     arg_level = np.argmax(length) + arg_low_level   File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom\lib\site-packages\numpy\core\fromnumeric.py", line 1229, in argmax     return _wrapfunc(a, 'argmax', axis=axis, out=out, **kwds)   File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom\lib\site-packages\numpy\core\fromnumeric.py", line 59, in _wrapfunc     return bound(*args, **kwds) ValueError: attempt to get argmax of an empty sequence

Hi @moataz-youssef , no worries and thanks for giving it a try. Apologies that I should have included this detail in my recent updates - VSI files are now supported through slideio (github repo) in palom v2024.10.2. You can pass the path to the VSI file to the palom.reader.VsiReader (ref).
From the above message, I'm guessing when converting to ome-tiff via QuPath, the tile-size wasn't specified and the default is 256x256 pixels. I found that small tile size will break how it's currently filtering out tiles that failed the alignment.
If you could start from the VSI files, the tile size will be set to 1024x1024 pixels internally, and the error you are running into would likely be gone.
It would be helpful if you could share a thumbnail of your scan. And here's a jupyter notebook that might give you some ideas.

Thanks a lot @Yu-AnChen.
I actually specified in QuPath that the tile size is 1024 pixels, but I will double check, what the final tile size in the converted OME-TIF files was. I can also share the scans/converted files with you privately, if you have time for this.
Thanks a lot for this great tool, I will update you on my progress.

@Yu-AnChen your solution worked! Using palom.reader.VsiReader worked and the script finished. The output had only 3 channels from the 3 original files (each with DAPI + 3 channels), so I should expect 12 channels. I can't seem to find how to do this, other than to specify that the registration should be done on channel 0 (DAPI). Do you have any suggestions for me? Thanks a lot in advance.

And just to add, the jupyter notebook also worked great and gave me a composite OME-TIFF with the DAPI channels from the 3 VSI images. How can I add the other channels through this method as well?
I am really impressed with the results, they are comparable, if not better, than manual adjustment and elastic deformation using Warpy in QuPath and Fiji.

Thanks, and glad to hear that the alignment seemed to work well!
Would you be able to share the image or images from some test samples? I suspect that the num_channels property of the scene in the VSI file wasn't detected properly. If that's the case, I think it's also valuable to the slideio author.

Re: the jupyter notebook that only gives you the first channel of all the "cycles", try changing the reader.pyramid[0][0] in block In [11]: to reader.pyramid[0], this should give you all the channels detected in the output.

Re: the jupyter notebook that only gives you the first channel of all the "cycles", try changing the reader.pyramid[0][0] in block In [11]: to reader.pyramid[0], this should give you all the channels detected in the output.

Thanks for this suggestion. Removing one [0] from reader.pyramid, gave me 7 channels. Removing one [0] from r1.pyramid in the same block. The output of r1.pyramid is:
[dask.array<transpose, shape=(3, 61515, 79801), dtype=uint16, chunksize=(3, 1024, 1024), chunktype=numpy.ndarray>,  dask.array<transpose, shape=(3, 30757, 39900), dtype=uint16, chunksize=(3, 1024, 1024), chunktype=numpy.ndarray>,  dask.array<transpose, shape=(3, 15378, 19950), dtype=uint16, chunksize=(3, 1024, 1024), chunktype=numpy.ndarray>,  dask.array<transpose, shape=(3, 7689, 9975), dtype=uint16, chunksize=(3, 1024, 1024), chunktype=numpy.ndarray>,  dask.array<transpose, shape=(3, 3844, 4987), dtype=uint16, chunksize=(3, 1024, 1024), chunktype=numpy.ndarray>,  dask.array<transpose, shape=(3, 1922, 2493), dtype=uint16, chunksize=(3, 1024, 1024), chunktype=numpy.ndarray>,  dask.array<transpose, shape=(3, 961, 1246), dtype=uint16, chunksize=(3, 961, 1024), chunktype=numpy.ndarray>,  dask.array<transpose, shape=(3, 480, 623), dtype=uint16, chunksize=(3, 480, 623), chunktype=numpy.ndarray>,  dask.array<transpose, shape=(3, 240, 311), dtype=uint16, chunksize=(3, 240, 311), chunktype=numpy.ndarray>]
So I guess somehow, my images are seen as 3-channel images.
2024-12-17 21:41:19.247 | INFO     | palom.pyramid:write_pyramid:166 - Writing to combined_IF.ome.tif Assembling mosaic  1/ 3 (channel  1/ 3): 100%|###################################| 14275/14275 [04:47<00:00, 49.59it/s] Assembling mosaic  1/ 3 (channel  2/ 3): 100%|###################################| 14275/14275 [04:56<00:00, 48.22it/s] Assembling mosaic  1/ 3 (channel  3/ 3): 100%|###################################| 14275/14275 [04:56<00:00, 48.11it/s] Assembling mosaic  2/ 3 (channel  1/ 3): 100%|###################################| 24230/24230 [05:32<00:00, 72.81it/s] Assembling mosaic  2/ 3 (channel  2/ 3): 100%|###################################| 24230/24230 [05:25<00:00, 74.54it/s] Assembling mosaic  2/ 3 (channel  3/ 3): 100%|###################################| 24230/24230 [05:24<00:00, 74.72it/s] Assembling mosaic  3/ 3 (channel  1/ 3): 100%|###################################| 23834/23834 [05:47<00:00, 68.53it/s] Assembling mosaic  3/ 3 (channel  2/ 3): 100%|###################################| 23834/23834 [05:46<00:00, 68.76it/s] Assembling mosaic  3/ 3 (channel  3/ 3): 100%|###################################| 23834/23834 [05:47<00:00, 68.65it/s] 2024-12-17 22:31:30.615 | INFO     | palom.pyramid:write_pyramid:182 - Generating pyramid 2024-12-17 22:31:30.615 | INFO     | palom.pyramid:write_pyramid:185 -     Level 1 (30758 x 39901) Processing channel: 100%|################################################################| 9/9 [01:57<00:00, 13.04s/it] 2024-12-17 22:33:28.146 | INFO     | palom.pyramid:write_pyramid:185 -     Level 2 (15379 x 19951) Processing channel: 100%|################################################################| 9/9 [00:36<00:00,  4.02s/it] 2024-12-17 22:34:04.411 | INFO     | palom.pyramid:write_pyramid:185 -     Level 3 (7690 x 9976) Processing channel: 100%|################################################################| 9/9 [00:15<00:00,  1.74s/it] 2024-12-17 22:34:20.349 | INFO     | palom.pyramid:write_pyramid:185 -     Level 4 (3845 x 4988) Processing channel: 100%|################################################################| 9/9 [00:10<00:00,  1.22s/it] 2024-12-17 22:34:31.365 | INFO     | palom.pyramid:write_pyramid:185 -     Level 5 (1923 x 2494) Processing channel: 100%|################################################################| 9/9 [00:09<00:00,  1.09s/it] 2024-12-17 22:34:41.161 | INFO     | palom.pyramid:write_pyramid:185 -     Level 6 (962 x 1247) Processing channel: 100%|################################################################| 9/9 [00:09<00:00,  1.06s/it] 2024-12-17 22:34:50.687 | INFO     | palom.pyramid:write_pyramid:185 -     Level 7 (481 x 624) Processing channel: 100%|################################################################| 9/9 [00:09<00:00,  1.04s/it]
The registration of the 9 channels is just fantastic. So I have the 3 DAPIs + 6 other channels. Unfortunately, I can't distinguish which is which. Is there a way to add the channel names to the final mosaic OME-TIFF, either by supplying a csv file with the channel names and order or by querying the file name? This will be just awesome!
I will send you my images (6 in total from 6 staining cycles) as well as the jupyter notebook with the output of all blocks on the email linked to your profile.

By the way, installing napari in the same palom environment, to view the final file, breaks the palom packages.
In the environment I only had palom (according to your instructions on this github) and jupyter-lab installed. Once I install napari, I get the following errors:
TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. TIFFReadDirectory: Warning, Unknown field with tag 33560 (0x8318) encountered. C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\numpy\_core\fromnumeric.py:3596: RuntimeWarning: Mean of empty slice.   return _methods._mean(a, axis=axis, dtype=dtype, C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\numpy\_core\_methods.py:138: RuntimeWarning: invalid value encountered in divide   ret = ret.dtype.type(ret / rcount) 2024-12-17 23:09:33.224 | DEBUG    | palom.register:cv2_feature_detect_and_match:288 - keypts L:0, keypts R:0 2024-12-17 23:09:33.256 | DEBUG    | palom.register:cv2_feature_detect_and_match:288 - keypts L:0, keypts R:0 2024-12-17 23:09:33.287 | DEBUG    | palom.register:cv2_feature_detect_and_match:288 - keypts L:0, keypts R:0 2024-12-17 23:09:33.320 | DEBUG    | palom.register:cv2_feature_detect_and_match:288 - keypts L:0, keypts R:0 C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\palom\register.py:249: FutureWarning:plot_matchesis deprecated since version 0.23 and will be removed in version 0.25. Useskimage.feature.plot_matched_featuresinstead.   skimage.feature.plot_matches( 2024-12-17 23:09:34.622 | WARNING  | palom.register:feature_based_registration:153 - Feature matching failed. Returning identity matrix as placeholder 2024-12-17 23:09:35.552 | INFO     | palom.align:compute_shifts:203 - Computing block-wise shifts Computing shifts:  58%|################################3                       | 19471/33745 [00:10<00:07, 1822.06it/s] Traceback (most recent call last):   File "D:\valis_container\palom_v2.py", line 74, in <module>     aligner.compute_shifts()   File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\palom\align.py", line 210, in compute_shifts     shifts = shifts_da.compute()   File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\dask\base.py", line 372, in compute    (result,) = compute(self, traverse=False, **kwargs)   File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\dask\base.py", line 660, in compute    results = schedule(dsk, keys, **kwargs)   File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\palom\block_affine.py", line 24, in block_affine_dask     return block_affine(   File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\palom\block_affine.py", line 83, in block_affine     block_tform = skimage.transform.AffineTransform(   File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\skimage\transform\_geometric.py", line 1046, in __init__     b2 = translation[1] IndexError: index 1 is out of bounds for axis 0 with size 1
By installing palom in a fresh environment, the problem is solved.

I see, it's unfortunate that only the first 3 channels are detected. Let me look into it once I got your files. For adding channel names to the output ome-tiff, you'll need to specify it in the write_pyramid call since it's not parsed from the VSI file. For example to write a 3-cycle image -
palom.pyramid.write_pyramid(
    mosaics,
    "registered-image.ome.tif",
    r1.pixel_size,
    channel_names=[
        ["DAPI", "Marker A (cycle 1)", "Marker B (cycle 1)"],
        ["DAPI", "Marker C (cycle 2)", "Marker D (cycle 2)"],
        ["DAPI", "Marker E (cycle 3)", "Marker F (cycle 3)"],
    ],
    downscale_factor=2,
    compression="lzw",
    tile_size=1024,
    save_RAM=True,
)

If you open the output image in QuPath, the channel names should show up in the interface.
For napari, we use ome_types (github repo) to parse the channel name in the registered-image.ome.tif like so
import napari
import ome_types
import palom

ome = ome_types.from_tiff("registered-image.ome.tif")
channel_names = [cc.name for cc in ome.images[0].pixels.channels]

v = napari.Viewer()
reader = palom.reader.OmePyramidReader("registered-image.ome.tif")

v.add_image(reader.pyramid, channel_axis=0, name=channel_names)

By the way, installing napari in the same palom environment, to view the final file, breaks the palom packages.

The error seems to be due to the scikit-image version, you could try re-install palom (via python -m pip install --force-reinstall palom) in the env that it was broke. This should downgrade scikit-image to the compatible version and hopefully will run through.

Thanks. I am uploading the images now. These datasets are trial ones, for the actual experiments, we have a choice between saving in OME-TIFF or in VSI, but it will take sometime till we have them running.
Is there an option to skip the 2nd and 3rd DAPI channels when writing the mosaic OME-TIFF? This will save time and disk space, once the workflow is standardized.
Another question, the output mosaic OME-TIFF file from the jupyter notebook has the same pixel size as the original images, although the downscale_factor is set to 2. Shouldn't it be halved?

python -m pip install --force-reinstall palom

After running this, I get a different error:
  File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\numpy\_core\arrayprint.py", line 34, in <module>
    from . import numerictypes as _nt
  File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\numpy\_core\numerictypes.py", line 102, in <module>
    from ._type_aliases import (
  File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\numpy\_core\_type_aliases.py", line 38, in <module>
    allTypes[_abstract_type_name] = getattr(ma, _abstract_type_name)
AttributeError: module 'numpy.core.multiarray' has no attribute 'unsignedinteger'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\valis_container\palom_v2.py", line 7, in <module>
    import palom
  File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\palom\__init__.py", line 9, in <module>
    from . import (
  File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\palom\align.py", line 3, in <module>
    import skimage.exposure
  File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\skimage\__init__.py", line 157, in <module>
    from .util.dtype import (img_as_float32,
  File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\skimage\util\__init__.py", line 16, in <module>
    from ._montage import montage
  File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\skimage\util\_montage.py", line 4, in <module>
    from .. import exposure
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\skimage\_shared\lazy.py", line 62, in __getattr__
    return importlib.import_module(f'{package_name}.{name}')
  File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\skimage\exposure\__init__.py", line 6, in <module>
    from ._adapthist import equalize_adapthist
  File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\skimage\exposure\_adapthist.py", line 21, in <module>
    from ..color.adapt_rgb import adapt_rgb, hsv_value
  File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\skimage\color\__init__.py", line 1, in <module>
    from .colorconv import (convert_colorspace,
  File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\skimage\color\colorconv.py", line 407, in <module>
    rgbcie_from_rgb = rgbcie_from_xyz @ xyz_from_rgb
  File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\numpy\core\_internal.py", line 855, in array_ufunc_errmsg_formatter
    args_string = ', '.join(['{!r}'.format(arg) for arg in inputs] +
  File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom-clone\lib\site-packages\numpy\core\_internal.py", line 855, in <listcomp>
    args_string = ', '.join(['{!r}'.format(arg) for arg in inputs] +
RuntimeError: Unable to configure default ndarray.__repr__

After running this, I get a different error:

Woof, I see, will try to see what is the right sequence to set it up - it's annoying to have separate envs.

Is there an option to skip the 2nd and 3rd DAPI channels when writing the mosaic OME-TIFF? This will save time and disk space, once the workflow is standardized.

Yes you can slice the channels you want and append it to the mosaics. For example this will only take the second and third channels from the aligned_img
mosaics = []
aligned_img = palom.align.block_affine_transformed_moving_img(
    aligner.ref_img, reader.pyramid[0], aligner.block_affine_matrices_da
)
mosaics.append(aligned_img[1:3])

Another question, the output mosaic OME-TIFF file from the jupyter notebook has the same pixel size as the original images, although the downscale_factor is set to 2. Shouldn't it be halved?

The downscale_factor is for specifying the downsizing factor between image pyramid levels. Default is 4 which some tools will have a hard time to render it properly.

The downscale_factor is for specifying the downsizing factor between image pyramid levels. Default is 4 which some tools will have a hard time to render it properly.

Thanks. I think it is the same in QuPath as well (4).
Another question this time regarding brightfield images. The examples listed in the readme of this GitHub under SVS both are either pure brightfield images (Example 1) or a brightfield and IF image (Example 2). The question is, what is the output? How can you make a composite OME-TIFF of that? As far as I know, other WSI alignment tools, e.g. VALIS  or CODA, can't do that. Can you share an example?

Here's a brief doc for aligning "same section" H&E to Orion (an IF image) image (example output here). Note that the default channels that works for me might not work for your images, the key settings are --thumbnail_channel1 and 2 and --channel1 and 2.
As for the combined view, QuPath experts would prob know how to load multiple images and overlay them, if you figure out a way to do so, please share with us here :)
I personally use napari as you can add IF channels and R/G/B channels from the BF to the same viewer and toggle on/off, setting colormap, etc.
If you really want to generate one single file by merging the IF and BF images, palom-pyramid merge (source code) can combine multiple ome-tiffs into one output. The syntax is a bit exotic (via fire), see the following example
$ palom-pyramid merge "[r'C:\Users\me\Desktop\palom-align-he-test\merge-test-1.ome.tif', r'C:\Users\me\Desktop\palom-align-he-test\merge-test-2.ome.tif']"
2024-12-17 16:32:11.742 | INFO     | palom.reader:pixel_size:147 - Detected pixel size: 0.3250 µm
2024-12-17 16:32:11.750 | INFO     | palom.cli.pyramid_tools:merge_channels:54 -
    Processing:
        merge-test-1.ome.tif
        merge-test-2.ome.tif

2024-12-17 16:32:11.756 | INFO     | palom.pyramid:write_pyramid:166 - Writing to C:\Users\me\Desktop\palom-align-he-test\merged-merge-test-1-zlib.ome.tif

I decided to do some further troubleshooting:
To solve the problem of not all channels being read from the VSI file, I decided to convert again to OME-TIFF, but this time not through QuPath but by using the NGFF-Converter from Glenco. I ran the script first and got this error:
  mx_ref = affine(scale=1/self.ref_thumbnail_down_factor).params
C:\Users\yousseah\AppData\Local\anaconda3\envs\palom\lib\site-packages\skimage\transform\_geometric.py:899: RuntimeWarning: invalid value encountered in scalar multiply
  [sx * math.cos(rotation), -sy * math.sin(rotation + shear), 0],
C:\Users\yousseah\AppData\Local\anaconda3\envs\palom\lib\site-packages\skimage\transform\_geometric.py:900: RuntimeWarning: invalid value encountered in scalar multiply
  [sx * math.sin(rotation),  sy * math.cos(rotation + shear), 0],
C:\Users\yousseah\AppData\Local\anaconda3\envs\palom\lib\site-packages\palom\align.py:178: RuntimeWarning: divide by zero encountered in scalar divide
  mx_moving = affine(scale=1/self.moving_thumbnail_down_factor).params
Computing shifts:  42%|###########################################################9                                                                                | 235/561 [00:00<00:00, 2342.93it/s]C:\Users\yousseah\AppData\Local\anaconda3\envs\palom\lib\site-packages\palom\block_affine.py:68: RuntimeWarning: invalid value encountered in cast
  ).astype(int)
C:\Users\yousseah\AppData\Local\anaconda3\envs\palom\lib\site-packages\palom\block_affine.py:71: RuntimeWarning: invalid value encountered in cast
  ).astype(int)
Computing shifts: 100%|###############################################################################################################################################| 561/561 [00:00<00:00, 2223.51it/s]
C:\Users\yousseah\AppData\Local\anaconda3\envs\palom\lib\site-packages\palom\align.py:250: RuntimeWarning: divide by zero encountered in divide
  np.divide(self.ref_img.chunksize, self.ref_thumbnail_down_factor)
C:\Users\yousseah\AppData\Local\anaconda3\envs\palom\lib\site-packages\palom\align.py:257: UserWarning: Attempting to set identical low and high xlims makes transformation singular; automatically expanding.
  ax.imshow(
C:\Users\yousseah\AppData\Local\anaconda3\envs\palom\lib\site-packages\palom\align.py:257: UserWarning: Attempting to set identical low and high ylims makes transformation singular; automatically expanding.
  ax.imshow(
Traceback (most recent call last):
  File "D:\valis_container\palom_v2.2.py", line 77, in <module>
    fig = aligner.plot_shifts()
  File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom\lib\site-packages\palom\align.py", line 276, in plot_shifts
    viz_shifts(shifts, self.grid_shape, ax=axs[1])
  File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom\lib\site-packages\palom\align.py", line 90, in viz_shifts
    dcenter = skimage.filters.threshold_triangle(distances[is_finite])
  File "C:\Users\yousseah\AppData\Local\anaconda3\envs\palom\lib\site-packages\skimage\filters\thresholding.py", line 932, in threshold_triangle
    arg_low_level, arg_high_level = np.where(hist > 0)[0][[0, -1]]
IndexError: index 0 is out of bounds for axis 0 with size 0

So I checked the first file with QuPath and the OME-TIFF had all series in it. Using the NGFF-Converter, I did a second conversion, but this time I extracted only the series which is important (Series 2, counting from 0), the rest was ignored.
I ran the output this time with the jupyter notebook and voila! The 4 channels per image were detected:
 dask.array<from-zarr, shape=(4, 30757, 39900), dtype=uint16, chunksize=(1, 1024, 1024), chunktype=numpy.ndarray>,
 dask.array<from-zarr, shape=(4, 15378, 19950), dtype=uint16, chunksize=(1, 1024, 1024), chunktype=numpy.ndarray>,
 dask.array<from-zarr, shape=(4, 7689, 9975), dtype=uint16, chunksize=(1, 1024, 1024), chunktype=numpy.ndarray>,
 dask.array<from-zarr, shape=(4, 3844, 4987), dtype=uint16, chunksize=(1, 1024, 1024), chunktype=numpy.ndarray>,
 dask.array<from-zarr, shape=(4, 1922, 2493), dtype=uint16, chunksize=(1, 1024, 1024), chunktype=numpy.ndarray>,
 dask.array<from-zarr, shape=(4, 961, 1246), dtype=uint16, chunksize=(1, 961, 1024), chunktype=numpy.ndarray>,
 dask.array<from-zarr, shape=(4, 480, 623), dtype=uint16, chunksize=(1, 480, 623), chunktype=numpy.ndarray>,
 dask.array<from-zarr, shape=(4, 240, 311), dtype=uint16, chunksize=(1, 240, 311), chunktype=numpy.ndarray>,
 dask.array<from-zarr, shape=(4, 120, 155), dtype=uint16, chunksize=(1, 120, 155), chunktype=numpy.ndarray>]

And running the cells further resulted in getting all 12 channels :-D
2024-12-18 12:19:16.657 | INFO     | palom.pyramid:write_pyramid:166 - Writing to combined_IF.ome.tif
Assembling mosaic  1/ 3 (channel  1/ 4): 100%|####################################################################################################################| 14275/14275 [00:09<00:00, 1466.50it/s]
Assembling mosaic  1/ 3 (channel  2/ 4): 100%|####################################################################################################################| 14275/14275 [00:10<00:00, 1325.58it/s]
Assembling mosaic  1/ 3 (channel  3/ 4): 100%|####################################################################################################################| 14275/14275 [00:11<00:00, 1291.96it/s]
Assembling mosaic  1/ 3 (channel  4/ 4): 100%|####################################################################################################################| 14275/14275 [00:09<00:00, 1487.64it/s]
Assembling mosaic  2/ 3 (channel  1/ 4): 100%|####################################################################################################################| 24230/24230 [00:20<00:00, 1195.50it/s]
Assembling mosaic  2/ 3 (channel  2/ 4): 100%|####################################################################################################################| 24230/24230 [00:20<00:00, 1173.52it/s]
Assembling mosaic  2/ 3 (channel  3/ 4): 100%|####################################################################################################################| 24230/24230 [00:20<00:00, 1207.78it/s]
Assembling mosaic  2/ 3 (channel  4/ 4): 100%|####################################################################################################################| 24230/24230 [00:21<00:00, 1143.24it/s]
Assembling mosaic  3/ 3 (channel  1/ 4): 100%|####################################################################################################################| 23834/23834 [00:20<00:00, 1173.06it/s]
Assembling mosaic  3/ 3 (channel  2/ 4): 100%|####################################################################################################################| 23834/23834 [00:20<00:00, 1160.65it/s]
Assembling mosaic  3/ 3 (channel  3/ 4): 100%|####################################################################################################################| 23834/23834 [00:21<00:00, 1099.74it/s]
Assembling mosaic  3/ 3 (channel  4/ 4): 100%|####################################################################################################################| 23834/23834 [00:20<00:00, 1188.01it/s]
2024-12-18 12:26:11.806 | INFO     | palom.pyramid:write_pyramid:182 - Generating pyramid
2024-12-18 12:26:11.807 | INFO     | palom.pyramid:write_pyramid:185 -     Level 1 (30758 x 39901)
Processing channel: 100%|#################################################################################################################################################| 12/12 [04:19<00:00, 21.59s/it]
2024-12-18 12:30:31.175 | INFO     | palom.pyramid:write_pyramid:185 -     Level 2 (15379 x 19951)
Processing channel: 100%|#################################################################################################################################################| 12/12 [01:22<00:00,  6.91s/it]
2024-12-18 12:31:54.107 | INFO     | palom.pyramid:write_pyramid:185 -     Level 3 (7690 x 9976)
Processing channel: 100%|#################################################################################################################################################| 12/12 [00:39<00:00,  3.30s/it]
2024-12-18 12:32:34.068 | INFO     | palom.pyramid:write_pyramid:185 -     Level 4 (3845 x 4988)
Processing channel: 100%|#################################################################################################################################################| 12/12 [00:21<00:00,  1.78s/it]
2024-12-18 12:32:55.441 | INFO     | palom.pyramid:write_pyramid:185 -     Level 5 (1923 x 2494)
Processing channel: 100%|#################################################################################################################################################| 12/12 [00:15<00:00,  1.32s/it]
2024-12-18 12:33:11.360 | INFO     | palom.pyramid:write_pyramid:185 -     Level 6 (962 x 1247)
Processing channel: 100%|#################################################################################################################################################| 12/12 [00:15<00:00,  1.31s/it]
2024-12-18 12:33:27.090 | INFO     | palom.pyramid:write_pyramid:185 -     Level 7 (481 x 624)
Processing channel: 100%|#################################################################################################################################################| 12/12 [00:15<00:00,  1.27s/it]

Naming the channels worked in the jupyter notebook with:
    mosaics,
    'combined_IF_channel_names.ome.tif',
    r1.pixel_size,
    channel_names=[
        ["DAPI", "XX-488 (cycle 1)", "YY-550 (cycle 1)", "ZZ-647 (cycle 1)"],
        ["DAPI", "XX-488 (cycle 2)", "YY-550 (cycle 2)", "ZZ-647 (cycle 2)"],
        ["DAPI", "XX-488 (cycle 3)", "YY-550 (cycle 3)", "ZZ-647 (cycle 3)"],
    ],
    downscale_factor=2,
    compression='lzw',
    tile_size=1024,
    save_RAM=True
)

This is an awesome tool! Thanks a lot for the development and your support. Looking forward to what you will find out from the VSI files. I will try it out now by scripting instead of the jupyter notebook.

A small update, as I have communicated this not very clearly. Running the script gives me the output of just the first image (3 channels while using VSI and all 4 when using properly converted OME-TIFFs).
The jupyter notebook is the one that gave me first the 3 DAPI channels from all 3 images, then all channels from the moving images plus DAPI from the reference image, then finally all detected channels in one OME-TIFF (9 total when using VSI from a maximum of 12). Converting the VSI images to OME-TIFF and running the notebook gave me all 12 channels, with labels, according to your suggestion.
So the question is what exactly does the jupyter notebook does differently than the python script?

Here's a brief doc for aligning "same section" H&E to Orion (an IF image) image (example output here). Note that the default channels that works for me might not work for your images, the key settings are --thumbnail_channel1 and 2 and --channel1 and 2.
As for the combined view, QuPath experts would prob know how to load multiple images and overlay them, if you figure out a way to do so, please share with us here :)
I personally use napari as you can add IF channels and R/G/B channels from the BF to the same viewer and toggle on/off, setting colormap, etc.
If you really want to generate one single file by merging the IF and BF images, palom-pyramid merge (source code) can combine multiple ome-tiffs into one output. The syntax is a bit exotic (via fire), see the following example
$ palom-pyramid merge "[r'C:\Users\me\Desktop\palom-align-he-test\merge-test-1.ome.tif', r'C:\Users\me\Desktop\palom-align-he-test\merge-test-2.ome.tif']"
2024-12-17 16:32:11.742 | INFO     | palom.reader:pixel_size:147 - Detected pixel size: 0.3250 µm
2024-12-17 16:32:11.750 | INFO     | palom.cli.pyramid_tools:merge_channels:54 -
    Processing:
        merge-test-1.ome.tif
        merge-test-2.ome.tif

2024-12-17 16:32:11.756 | INFO     | palom.pyramid:write_pyramid:166 - Writing to C:\Users\me\Desktop\palom-align-he-test\merged-merge-test-1-zlib.ome.tif

I did align an IF image and HE together but it was a virtual 32-bit image in QuPath using Warpy. However, I did not save it as a OME-TIFF.
I will try your suggestions, and update.

Yes you can slice the channels you want and append it to the mosaics. For example this will only take the second and third channels from the aligned_img
mosaics = []
aligned_img = palom.align.block_affine_transformed_moving_img(
    aligner.ref_img, reader.pyramid[0], aligner.block_affine_matrices_da
)
mosaics.append(aligned_img[1:3])

Last question for the day :-)
Any advice on how to get only certain channels from one of the moving images? For the rest, I would like to take all channels.

Thanks for all the updates and testing! I was able to fixe the 3 channel instead of 4 channel bug (I hardcoded that thinking all the VSI are RGB bright-field images). Try the v2024.12.1. And here's the script with some comments hopefully is helpful to you. I noticed that there are 2 scenes (ROIs) in the scan - not sure if you can configure that during the imaging.
We should discuss your bright-field to IF registration in a separate issue - don't want to spam others' inbox in this issue.

Any advice on how to get only certain channels from one of the moving images? For the rest, I would like to take all channels.

You can slide the aligned_img to only take the channels you want and append that to the mosaics. The dimension in aligned_img is (channel, row, column). The slicing does not add compute time because the actual pixel data is only touched in the final write_pyramid step.

Thanks a lot. It works fantastically out of the box.
Assembling mosaic  1/ 6 (channel  1/ 4): 100%|########################################################################################################################################################| 14581/14581 [04:36<00:00, 52.70it/s]
Assembling mosaic  1/ 6 (channel  2/ 4): 100%|########################################################################################################################################################| 14581/14581 [04:34<00:00, 53.14it/s]
Assembling mosaic  1/ 6 (channel  3/ 4): 100%|########################################################################################################################################################| 14581/14581 [04:35<00:00, 52.99it/s]
Assembling mosaic  1/ 6 (channel  4/ 4): 100%|########################################################################################################################################################| 14581/14581 [04:38<00:00, 52.43it/s]
Assembling mosaic  2/ 6 (channel  1/ 4): 100%|########################################################################################################################################################| 24098/24098 [05:07<00:00, 78.47it/s]
Assembling mosaic  2/ 6 (channel  2/ 4): 100%|########################################################################################################################################################| 24098/24098 [05:06<00:00, 78.50it/s]
Assembling mosaic  2/ 6 (channel  3/ 4): 100%|########################################################################################################################################################| 24098/24098 [05:06<00:00, 78.56it/s]
Assembling mosaic  2/ 6 (channel  4/ 4): 100%|########################################################################################################################################################| 24098/24098 [05:06<00:00, 78.58it/s]
Assembling mosaic  3/ 6 (channel  1/ 4): 100%|########################################################################################################################################################| 24536/24536 [05:35<00:00, 73.15it/s]
Assembling mosaic  3/ 6 (channel  2/ 4): 100%|########################################################################################################################################################| 24536/24536 [05:38<00:00, 72.44it/s]
Assembling mosaic  3/ 6 (channel  3/ 4): 100%|########################################################################################################################################################| 24536/24536 [05:41<00:00, 71.75it/s]
Assembling mosaic  3/ 6 (channel  4/ 4): 100%|########################################################################################################################################################| 24536/24536 [06:04<00:00, 67.33it/s]
Assembling mosaic  4/ 6 (channel  1/ 4): 100%|########################################################################################################################################################| 24140/24140 [06:16<00:00, 64.18it/s]
Assembling mosaic  4/ 6 (channel  2/ 4): 100%|########################################################################################################################################################| 24140/24140 [06:16<00:00, 64.06it/s]
Assembling mosaic  4/ 6 (channel  3/ 4): 100%|########################################################################################################################################################| 24140/24140 [06:13<00:00, 64.67it/s]
Assembling mosaic  4/ 6 (channel  4/ 4): 100%|########################################################################################################################################################| 24140/24140 [06:16<00:00, 64.14it/s]
Assembling mosaic  5/ 6 (channel  1/ 4): 100%|########################################################################################################################################################| 24302/24302 [05:23<00:00, 75.06it/s]
Assembling mosaic  5/ 6 (channel  2/ 4): 100%|########################################################################################################################################################| 24302/24302 [06:43<00:00, 60.16it/s]
Assembling mosaic  5/ 6 (channel  3/ 4): 100%|########################################################################################################################################################| 24302/24302 [06:53<00:00, 58.72it/s]
Assembling mosaic  5/ 6 (channel  4/ 4): 100%|########################################################################################################################################################| 24302/24302 [05:19<00:00, 76.05it/s]
Assembling mosaic  6/ 6 (channel  1/ 2): 100%|#######################################################################################################################################################| 24220/24220 [02:49<00:00, 142.97it/s]
Assembling mosaic  6/ 6 (channel  2/ 2): 100%|#######################################################################################################################################################| 24220/24220 [02:49<00:00, 143.18it/s]
2024-12-19 15:02:02.306 | INFO     | palom.pyramid:write_pyramid:182 - Generating pyramid
2024-12-19 15:02:02.308 | INFO     | palom.pyramid:write_pyramid:185 -     Level 1 (30700 x 41028)
Processing channel: 100%|###################################################################################################################################################################################| 22/22 [05:08<00:00, 14.00s/it]
2024-12-19 15:07:10.564 | INFO     | palom.pyramid:write_pyramid:185 -     Level 2 (15350 x 20514)
Processing channel: 100%|###################################################################################################################################################################################| 22/22 [01:24<00:00,  3.83s/it]
2024-12-19 15:08:34.929 | INFO     | palom.pyramid:write_pyramid:185 -     Level 3 (7675 x 10257)
Processing channel: 100%|###################################################################################################################################################################################| 22/22 [00:29<00:00,  1.34s/it]
2024-12-19 15:09:04.734 | INFO     | palom.pyramid:write_pyramid:185 -     Level 4 (3838 x 5129)
Processing channel: 100%|###################################################################################################################################################################################| 22/22 [00:16<00:00,  1.34it/s]
2024-12-19 15:09:21.235 | INFO     | palom.pyramid:write_pyramid:185 -     Level 5 (1919 x 2565)
Processing channel: 100%|###################################################################################################################################################################################| 22/22 [00:13<00:00,  1.69it/s]
2024-12-19 15:09:34.292 | INFO     | palom.pyramid:write_pyramid:185 -     Level 6 (960 x 1283)
Processing channel: 100%|###################################################################################################################################################################################| 22/22 [00:12<00:00,  1.81it/s]
2024-12-19 15:09:46.483 | INFO     | palom.pyramid:write_pyramid:185 -     Level 7 (480 x 642)
Processing channel: 100%|###################################################################################################################################################################################| 22/22 [00:12<00:00,  1.83it/s]

I will start a separate issue with brightfield images, once I have some nice scans to play with. Thanks again.

By the way, installing napari in the same palom environment, to view the final file, breaks the palom packages.

I was able to install both palom and napari in the same conda environment (using micromamba, if you have a recent version of miniconda, you probably will not need mamba/micromamba (info)).
I used these to set up the env, replace micromamba with conda as needed.
micromamba create -y -n test-palom python=3.10 "scikit-image<0.20" scikit-learn "zarr<2.15" tifffile=2024.7.2 imagecodecs matplotlib tqdm scipy dask numpy "loguru=0.5.3" ome-types pydantic pint "yamale<5" fire termcolor openslide napari pyqt -c conda-forge -c sdvillal

micromamba activate test-palom

python -m pip install palom

# for viewing the ome-tiff output in napari
# drag-and-drop file in napari window and select "WSI Reader"
pip install git+https://github.com/yu-anchen/napari-wsi-reader.git

And here's my conda env lockfile (info) on my Windows machine.


fix: Add missing NSG rules for VCN-Native Pod Networking

Changed to reflect https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengnetworkconfig.htm#securitylistconfig:

Moved oci_core_network_security_group_security_rule.workers_egress_flannel to CNI-agnostic local.workers_egress for node to node.
Added oci_core_network_security_group_security_rule.workers_ingress_npn for pod to worker.
Added oci_core_network_security_group_security_rule.pods_ingress for CP to pod, worker to pod, and pod to pod (local.pods_ingress existed for them but was not yet referenced).
Added to TCP 6443/12250 to local.pods_egress for pod to CP.
Updated oci_core_network_security_group_security_rule.pods_egress_internet from tcp to all protocols.

Signed-off-by: Devon Crouse<EMAIL_ADDRESS>

@Djelibeybi Can you please check why @devoncrouse is required to sign OCA?

@devoncrouse you either need to make your OGHO membership public (if you're already a member) or you need to become a member of OGHO and make that membership public once you are.


Running final binary as sudo

...and getting

INFO:0] Initialize OpenCL runtime...
[ INFO:0] Successfully initialized OpenCL cache directory: /root/.cache/opencv/3.4.1/opencl_cache/
[ INFO:0] Preparing OpenCL cache configuration for context: 32-bit--Broadcom--VideoCore_IV_GPU--0_4
OpenCV(3.4.1) Error: Unknown error code -220 (OpenCL error CL_INVALID_VALUE (-30) during call: clGetProgramInfo(handle, CL_PROGRAM_BINARIES, sizeof(ptr), &ptr, NULL)) in getProgramBinary, file /build/opencv/src/opencv-3.4.1/modules/core/src/ocl.cpp, line 3752
[ WARN:0] Can't save OpenCL binary into cache: /root/.cache/opencv/3.4.1/opencl_cache/32-bit--Broadcom--VideoCore_IV_GPU--0_4/imgproc--filterSepRow_e99b92fca8604fe253f3c641802ce117.bin
OpenCV(3.4.1) /build/opencv/src/opencv-3.4.1/modules/core/src/ocl.cpp:3752: error: (-220) OpenCL error CL_INVALID_VALUE (-30) during call: clGetProgramInfo(handle, CL_PROGRAM_BINARIES, sizeof(ptr), &ptr, NULL) in function getProgramBinary
OpenCL error CL_INVALID_WORK_GROUP_SIZE (-54) during call: clEnqueueNDRangeKernel('row_filter_C1_D0', dims=2, globalsize=240x544x1, localsize=16x16x1) sync=false
OpenCV(3.4.1) Error: Unknown error code -220 (OpenCL error CL_INVALID_VALUE (-30) during call: clGetProgramInfo(handle, CL_PROGRAM_BINARIES, sizeof(ptr), &ptr, NULL)) in getProgramBinary, file /build/opencv/src/opencv-3.4.1/modules/core/src/ocl.cpp, line 3752
[ WARN:0] Can't save OpenCL binary into cache: /root/.cache/opencv/3.4.1/opencl_cache/32-bit--Broadcom--VideoCore_IV_GPU--0_4/imgproc--filterSepRow_e99b92fca8604fe253f3c641802ce117.bin
OpenCV(3.4.1) /build/opencv/src/opencv-3.4.1/modules/core/src/ocl.cpp:3752: error: (-220) OpenCL error CL_INVALID_VALUE (-30) during call: clGetProgramInfo(handle, CL_PROGRAM_BINARIES, sizeof(ptr), &ptr, NULL) in function getProgramBinary
OpenCL error CL_INVALID_WORK_GROUP_SIZE (-54) during call: clEnqueueNDRangeKernel('row_filter_C1_D0', dims=2, globalsize=240x544x1, localsize=16x16x1) sync=false
FPS  0.0205593, Objects: 0
OpenCL error CL_INVALID_WORK_GROUP_SIZE (-54) during call: clEnqueueNDRangeKernel('row_filter_C1_D0', dims=2, globalsize=240x544x1, localsize=16x16x1) sync=false
OpenCL error CL_INVALID_WORK_GROUP_SIZE (-54) during call: clEnqueueNDRangeKernel('row_filter_C1_D0', dims=2, globalsize=240x544x1, localsize=16x16x1) sync=false

Is it problem of what ? >: Not supported features, rights, opencl?
Made just in case

sudo chmod 777 /root/.cache/opencv/3.4.1/opencl_cache/32-bit--Broadcom--VideoCore_IV_GPU--0_4

and didnt work

You get two different error codes:

OpenCL error CL_INVALID_WORK_GROUP_SIZE (-54)

OpenCV seems to not be able to handle small work-group sizes, or at least has a lower bound larger than the supported work-group size of VC4CL.

Error: Unknown error code -220

This seem to be that the call to clGetProgramInfo failed with somehow both the error code CL_INVALID_VALUE (-30) and an unknown error code -220.

Fails here:Mailbox::executeQPU on sending ioctl, everything hangs
as

[VC4CL] Running work-group 0, 0, 0
[VC4CL] Mailbox buffer before:
[VC4CL] 0000: 0x00000028
[VC4CL] 0004: 0x00000000
[VC4CL] 0008: 0x00030011
[VC4CL] 000c: 0x00000010
[VC4CL] 0010: 0x00000010
[VC4CL] 0014: 0x0000000c
[VC4CL] 0018: 0x5e10df58
[VC4CL] 001c: 0x00000000
[VC4CL] 0020: 0x00007530
[VC4CL] 0024: 0x751687cc

Code is:
R"CLC(
__kernel void TestMagic(const int total, const int is_deeper_magic, const float alpha_s, const float fore_th, __global const float* gradx, __global const float* grady,
//in/out
__global float* BSx,  __global float* BSy, __global int* mapRes)
{
private const size_t i        = get_global_id(0);
private const size_t gpu_used = get_global_size(0);
        private const size_t elements_count = total / (gpu_used * 16);
        private const size_t offset = i * total / gpu_used;

        for (size_t k = 0; k < elements_count; ++k)
        {
           int16 mr           = vload16( k , mapRes + offset);
           const int16 twos   = 2;
           mr += twos;
           vstore16(mr, k, mapRes + offset);
        }
    }
    )CLC",

During compilation:

[VC4CL] Precompiling source with:
[VC4CL] Precompilation complete with status: 0
[VC4CL] Compilation log: [W] Sat Jul  7 03:04:50 2018: Warnings in precompilation:
[W] Sat Jul  7 03:04:50 2018: :90:9: warning: null character ignored
<U+0000>
^
1 warning generated.

Any ideas? >: QPULib examples work...(https://github.com/mn416/QPULib)

Sometimes, if the code generated for a kernel does something completely wrong, then the QPUs get into a hanged state. Looks like this is happening here

and how to deal with? is it compiler problem or my code?

Most likely a compiler problem. I found an issue which also occurs in the kernel given, and will try to fix it.

what exactly ? I could change my kernel maybe to skip this problem.

This commit does not fix issue, still hangs
https://github.com/doe300/VC4C/commit/f13120debba4d59da6b8ca432a22eaf64e338550

Yeah, the integer division (and modulo) is still wrong. I could not figure out the reason for this yesterday.

does it mean float div will work? I need that for atan2, which contains same integer div (for offset) and inside atan2 y/x ... btw in you souce it is some comment there, maybe you do x/y ?

Floating point division works, at least for the tests I ran, with worse-than-allowed accuracy in some cases. I don't know if the atan2 function works..

Yes, thats weird. Opencl on nvidia give a bit different results, then opencv-default(cpu?) does:
1st is opencv-default, 2nd is opencl-nvidia kernel:

DUMP:
x = 0; y = 2; a = 1.5708
x = -4; y = 4; a = 2.35636
x = 6; y = 8; a = 0.927403
x = 21; y = 13; a = 0.554224
x = 18; y = 16; a = 0.726774
x = 12; y = 6; a = 0.463683
x = 15; y = -13; a = 5.569
x = 8; y = -22; a = 5.06123
x = -9; y = -11; a = 4.02667
x = -11; y = 11; a = 2.35636
x = -2; y = 18; a = 1.68144
x = 4; y = 4; a = 0.785232
x = 4; y = -12; a = 5.0342
x = 7; y = -13; a = 5.20633
x = 8; y = 0; a = 0
x = 0; y = 10; a = 1.5708
DUMP:
x = 0; y = 2; a = 1.5708
x = -4; y = 4; a = 2.35619
x = 6; y = 8; a = 0.927295
x = 21; y = 13; a = 0.554308
x = 18; y = 16; a = 0.726642
x = 12; y = 6; a = 0.463648
x = 15; y = -13; a = 5.56909
x = 8; y = -22; a = 5.06116
x = -9; y = -11; a = 4.02666
x = -11; y = 11; a = 2.35619
x = -2; y = 18; a = 1.68145
x = 4; y = 4; a = 0.785398
x = 4; y = -12; a = 5.03414
x = 7; y = -13; a = 5.20633
x = 8; y = 0; a = 0
x = 0; y = 10; a = 1.5708

Used nvidia reference implementation (http://developer.download.nvidia.com/cg/atan2.html),on desktop it's ok.
float16 myatan2(float16 y, float16 x)
        {
          float16 t0, t1, t2, t3, t4;

          t3 = fabs(x);
          t1 = fabs(y);
          t0 = max(t3, t1);
          t1 = min(t3, t1);
          t3 = 1.f / t0;
          t3 = t1 * t3;

          t4 = t3 * t3;
          t0 =         - 0.013480470f;
          t0 = t0 * t4 + 0.057477314f;
          t0 = t0 * t4 - 0.121239071f;
          t0 = t0 * t4 + 0.195635925f;
          t0 = t0 * t4 - 0.332994597f;
          t0 = t0 * t4 + 0.999995630f;
          t3 = t0 * t3;

          t3 = (fabs(y) > fabs(x)) ? 1.570796327f - t3 : t3;
          t3 = (x < 0) ?  3.141592654f - t3 : t3;
          t3 = (y < 0) ? -t3 : t3;

          return t3;
        }

On RPI it works as well:

DUMP:
x = 0; y = 2; a = 1.5708
x = -4; y = 4; a = 2.35636
x = 6; y = 8; a = 0.927403
x = 21; y = 13; a = 0.554224
x = 18; y = 16; a = 0.726774
x = 12; y = 6; a = 0.463683
x = 15; y = -13; a = 5.569
x = 8; y = -22; a = 5.06123
x = -9; y = -11; a = 4.02667
x = -11; y = 11; a = 2.35636
x = -2; y = 18; a = 1.68144
x = 4; y = 4; a = 0.785232
x = 4; y = -12; a = 5.0342
x = 7; y = -13; a = 5.20633
x = 8; y = 0; a = 0
x = 0; y = 10; a = 1.5708
DUMP:
x = 0; y = 2; a = 1.5708
x = -4; y = 4; a = 2.3562
x = 6; y = 8; a = 0.927294
x = 21; y = 13; a = 0.554308
x = 18; y = 16; a = 0.72664
x = 12; y = 6; a = 0.463646
x = 15; y = -13; a = -0.714088
x = 8; y = -22; a = -1.22203
x = -9; y = -11; a = -2.25652
x = -11; y = 11; a = 2.3562
x = -2; y = 18; a = 1.68145
x = 4; y = 4; a = 0.785395
x = 4; y = -12; a = -1.24905
x = 7; y = -13; a = -1.07686
x = 8; y = 0; a = 0
x = 0; y = 10; a = 1.5708

Opencl on nvidia give a bit different results, then opencv-default

Depending on the compilation flags you specified, the NVIDIA code may use faster but inaccurate operations (e.g. due to -cl-fast-relaxed-math or -cl-mad-enable).

On RPI something is broken completely:

This looks like the atan2 function yielding wrong results. Some/a lot of the math functions in VC4CL are not correct or properly tested.

Used nvidia reference implementation [...]

Thanks for the link, this might come in very handy, if I can figure out its license...

Still can't make "select" work >: is it broken too ? Bcs I was dependant on in another kernel as well (that initial I simplified for test).

Ok made custom select for float16, for integers must be even simplier:
 float16 myselect(float16 afalse, float16 atrue, int16 condition)
        {
            //we have -1 = true in condition ...it should be so
            float16 cond = convert_float16(condition) * -1.f;
            float16 not_cond = 1.f - cond;
            return atrue * cond + afalse * not_cond;
        }

You need to re-trigger building of the PCH and BC for vc4cl-stdlib, which is done by the script located in ${CMAKE_BINARY_DIR}/build/postinst from the VC4C project for installed stdlib and make vc4cl-stdlib for stdlib sources (in which case you have to delete <VC4CLStdLib-root>/include/VC4CLStdLib.h.pch and <VC4CLStdLib-root>/include/VC4CLStdLib.bc).

Ok..so no compiler rebuild if pch is built as separated package like here https://github.com/alexzk1/vc4_stdlib_arch/blob/master/PKGBUILD ?
Btw, building VC4C gives many such warnings, is it ok ? (gcc 8)
usr/include/c++/8.1.0/bits/stl_vector.h: In member function 'std::vector Parser::parseArgumentList(const string&, size_t)':
/usr/include/c++/8.1.0/bits/stl_vector.h:1085:4: note: parameter passing for argument of type '__gnu_cxx::__normal_iterator<exprValue*, std::vector >' changed in GCC 7.1
In file included from /usr/include/c++/8.1.0/vector:69,

Well, the warnings are both libstdc++ internal warnings. I can try to disable the warning, but I don't think I can do anything about fixing them.

fmod(float16, float16) is still broken - pi hangs.

No kernel panic, but that red dots around... maybe because connected monitor? And hang

holdon...why it says "myfabs", I have removed that func and it's not on disk at all in sources.

Did such macro, hanged  immediately without any 1 success step

#define Z1 ((float16)(a, b.s0123, b.s456789ab, b.scde))
#define Z2 (b)
#define Z3 ((float16) (b.s123, b.s4567, b.s89abcdef, c))
#define Z4 ((float16)(d, e.s0123, e.s456789ab, e.scde))
#define Z5 (e)
#define Z6 ((float16)(e.s123, e.s4567, e.s89abcdef, f))
#define Z7 ((float16)(g, h.s0123, h.s456789ab, h.scde))
#define Z8 (h)
#define Z9 ((float16)(h.s123, h.s4567, h.s89abcdef, i))

Okey, this seems working - function calls without macroses as parameters, code is wrong, but works, so how is it possible? >:
 p1 = myselectf16(p1, p2, atest);
        p2 = myselectf16(p2, p1, atest);

        atest =  isless(fabs(angle - pi4), pi8); //45
        p1 = myselectf16(p1, p2, atest);
        p2 = myselectf16(p2, p1, atest);

        atest =  islessequal(angle, pi8) || islessequal(fabs(angle - pi1), pi8); //0
        p1 = myselectf16(p1, p2, atest);
        p2 = myselectf16(p2, p1, atest);

        atest =  isless(fabs(angle - pi34), pi8); //135
        p1 = myselectf16(p1, p2, atest);
        p2 = myselectf16(p2, p1, atest);

In another kernel I had usage of those Z's in ariphmetics like
float16 Gx = (Z7 + 2 * Z8 + Z9) - (Z1 + 2 * Z2 +Z3);
and it is working ok.

And maybe something else, replaced fmod by equivalent, but...on next run it was ok, and next-next run hanged again...

Did you reboot your Pi in between runs? Sometimes when the GPU hangs, it can only recovered by rebooting it.

No kernel panic, but that red dots around... maybe because connected monitor?

It is possible, that the graphics driver collides with VC4CL, at least if VC4CL is compiled with REGISTER_POKE_KERNELS on. If the flag is disabled, the VC4CL uses the kernel synchronization routines, the graphics driver uses too (but see #15, idk if this is still valid).

Okey, this seems working - function calls without macroses as parameters, code is wrong, but works, so how is it possible?

One theory: There is something wrong with vector shuffling (putting together vector from pieces of other vector), which you do in the macros.

One theory: There is something wrong with vector shuffling (putting together vector from pieces of other vector), which you do in the macros.

Yes, possibly. Assignign like float16  z1 = Z1; and using variable didnt work too. However as I said other kernels were working. But not sure if gave correct results actually.

Last commits (from 15th Jul) to compiler/std lib do not fix hangs. However.. I see (always saw actually) 4 fps reports which is done once per 10 frames, so kernel still processes at least 40 frames. I use static buffers in C++code to avoid copies/creations. Is it possibly some error happens in opencl memory managment? >:

So what does the "40 frames" it runs mean? How are they split up in work-groups and work-items?
And do you run every 10 frames separately (e.g. with a new call to clEnqueueNDKernel or all at once?
Also, what do you mean by "[...]static buffers in C++code to avoid copies/creations"? You at least once have to copy the data to the GPU memory.

Here, that is last run i did:
https://github.com/alexzk1/LitterBug-Algorithm/blob/Desktop_Litterbug/cl/kernels.cpp
It runs about 40 times openCl::sobel2magic
which currently start non_maximum kernel only for testing. For example, sobel kernel runs fine many times (if uncomented)

So, I replaced all those Z's by e (it is Z5) and compiled kernel worked...
Then I replaced all by Z5, and it works again.
Then I tried those and still hangs:
 #define Z1 ((float16)(a, b.s0, b.s1, b.s2, b.s3, b.s4, b.s5, b.s6, b.s7, b.s8, b.s9, b.sa, b.sb, b.sc, b.sd, b.se))
        #define Z2 (b)
        #define Z3 ((float16) (b.s1, b.s2, b.s3, b.s4, b.s5, b.s6, b.s7, b.s8, b.s9, b.sa, b.sb, b.sc, b.sd, b.se, b.sf, c))
        #define Z4 ((float16)(d, e.s0, e.s1, e.s2, e.s3, e.s4, e.s5, e.s6, e.s7, e.s8, e.s9, e.sa, e.sb, e.sc, e.sd, e.se))
        #define Z5 (e)
        #define Z6 ((float16)(e.s1, e.s2, e.s3, e.s4, e.s5, e.s6, e.s7, e.s8, e.s9, e.sa, e.sb, e.sc, e.sd, e.se, e.sf, f))
        #define Z7 ((float16)(g, h.s0, h.s1, h.s2, h.s3, h.s4, h.s5, h.s6, h.s7, h.s8, h.s9, h.sa, h.sb, h.sc, h.sd, h.se))
        #define Z8 (h)
        #define Z9 ((float16)(h.s1, h.s2, h.s3, h.s4, h.s5, h.s6, h.s7, h.s8, h.s9, h.sa, h.sb, h.sc, h.sd, h.se, h.sf, i))

I am currently working on writing tests for and fixing most of the standard-library functions. But I will have a look at the vector insertion/extraction today and see if I can figure out, why the macros do not work.

Not sure if it is macros-only problem, another kernel works with:

float16 Gx = (Z7 + 2 * Z8 + Z9) - (Z1 + 2 * Z2 + Z3);
float16 Gy = (Z3 + 2 * Z6 + Z9) - (Z1 + 2 * Z4 + Z7);

I also tested the vector-assembly similar to what the macros do and they work correctly, so this is not the issue. Also, it would be very strange if the question whether it hangs depends on the values (whether correct or not).
One thing you could try is to compile the kernels with different optimization levels (-O0, -O1 or-O2). This could give us a hint whether the error is in one of the optimizations or a general problem. Btw, the default level is -O2 so all stable/halfway fast optimizations are activated.

last time i used was
progs.build("-cl-opt-disable");
can u check my sources link (and see history of the tries) ?
bcs i'm lost. I'm not expert in paraller or whatever, but ...i try my best, and check 1st of all on desktop/nvidia - when it works (no warnings) and does proper result - i try on PI ...and get a problems...

output.txt
that was build on desktop ...well...initial warnings saw on pi... not sure if here though

I think problem rises pi is 32 bits while desktop is 64

to be honest, i pray to things - that all compiler warnings must be fixed - "trust in ir compiler"

progs.build("-cl-opt-disable");

VC4C does not (yet) head this option, so they stay enabled.
I'm pretty sure, that if a calculation works fine on NVIDIA and does not on VC4CL, then the problem is on the VC4CL part.
Did you compile the binaries (VC4C and VC4CL) as 64-bit program? If so, it might be a problem, but I can't say with certainty.

no. i compiled all as packages on rpi, then git pull my prog there, the tried
all links are above

So I found and fixed a bug assigning locals to registers (and handling conflicts) and also found another problem, which is not yet fixed:
In several kernels, you use the index -1 for accessing memory objects. I did not really check, whether this is correct from a functionality point, but the VC4C compiler does not support negative integers. I will have to add some extra handling for this.

Yes, I add extra memory prior call t kernel, so final index must be valid in any case. If not - it is my mistake somewhere then.

Also replacing by *(INPUT - 1) just in case.

That didn't do anything, since CLang converted it to the exact same instructions:)
But I now added support at least for constant negative indices, resulting in the kernels building and running in the emulator without any issues. Idk though, if the results are correct.

Okey, thanks. No hangs now on that kernel. I will test atan2 and kernels for proper results as well, but that will take some time.

Next question, this single line makes kernel 10x TIMES(!!) slower
vstore16(select(0, Z5, isless(p2, Z5) && isless(p1, Z5)), 0, ( __global float*)(N + dstPaddedIndex));
I would bet on vstore, bcs another lines use isless too. Any reason for that ? >: On NVIDA delay is less then time measure error.

This makes the kernel slower compared to what?
The otherwise same kernel with this line removed?
Then yes, it is definitively the vstore16, since memory access (esp. writing memory) is the bottleneck for VC4.

Such C++ code, prints time used inside:
 {
            TimeMeasure t("Kernel 2");
            kernel_non_maximum(cl::EnqueueArgs(queue, cl::NDRange(kw, kh), cl::NDRange(gpus)), bangle, bgm, bN).wait();
        }

So when line uncommented it is 100ms, when commented out - it is 7-8ms.

There is just one memory. The GPU directly accesses the CPU memory, it does not have any of its own.
For __private and __local memory, VC4C tries to optimize the memory access by either keeping the values in the registers or using the VPM (a small cache also used to write to RAM). For __constant memory, there is also an optimization trying to load the contents directly into registers.
__global memory is always located in the RAM. Also, if a memory location (the same parameter) is not written to, the loading of the data can also be sped up a bit.
On the host-side, since the memory is really shared, accessing a mapped buffer (e.g. via clEnqueueMapBuffer) accesses directly the physical addresses also accessed by the GPU resulting in no extra copying required.

So...there is no way to beat that ? >: Usual technique was split task into couple kernels, and keep passing buffers between them. And only final one is copied to sys memory. So, I run 3 kernels. As you see on 2nd kernel example - load all data from 1st + calculations = 8ms. Storing data to pass to 3rd = 90ms extra. I don't get why loads are much faster then storing, if it uses same unit linearly.

So what If you will check how that buffer was created, and if not accessed from host side (kinda gpu-only buffer) then you skip locks ? It is kernel's developer problem to keep all proper.

So...there is no way to beat that ?

No, that is a hardware limitation. Unless of course there is some other hardware component I don't know about, or there is some trick to be done with the known components...

I don't get why loads are much faster then storing, if it uses same unit linearly.

As I wrote in the post above, loads (if the same parameter is not written into, e.g. by const pointer) are optimized. They use a different hardware component, which is mainly for reading colors for texture processing, but can also read arbitrary data (and most importantly: in parallel!)

So what If you will check how that buffer was created, [...]

How would you do that? At compilation time I don't know which memory is really accessed host-side (at least not for __global memory). Also, using the VPM as cache requires locking too, but for a much shorter time-span, since the data does not need to be copied into RAM under lock.

Do you take in account "restrict" keyword too ? As I understood, it tells compiler no overlaps are expected on this pointer.
..ok well, is any way to take lock once for couple vstore? For example, store 16 of float16 at once in 1 lock ?

Do you take in account "restrict" keyword too ?

Currently no, I more or less assume all parameters to be restricted, which might not be right. But treating the correctly would decrease performance...

ok well, is any way to take lock once for couple vstore?

It is done. vstore16 writes (guarded by the hardware mutex) 16 elements first into VPM cache and then into RAM.

It is done. vstore16 writes (guarded by the hardware mutex) 16 elements first into VPM cache and then into RAM.
Well, I have loop there, which does vstore16 for float16 and loop itself is for 16 items. I was thinking to cache it, and store 16 of float16 (256 numbers) in 1 lock.

Ah okay.
There is an optimization for grouping stores into contiguous memory, but I don't think it currently works with loops, unless they are unrolled.

How big is VPM cache? What if you will do copy to ram on explicit call of cl::copy (or cache overflow)? Otherwise, if kernels are chained, you will have same address out/in and can keep using VPM then.

If I will lower amount of work items (i.e. will do deep nested loops in kernel), will it speed up vstore's somehow?

How big is VPM cache?

64 rows (actually more, but only 64 can be addressed) with 16 words (32-bit) per row, but addressing works per row.

What if you will do copy to ram on explicit call of cl::copy (or cache overflow)?

Using the VPM as cache and only writing back when the cache is full would be a useful optimization, but also very hard to realize (esp. dynamically, e.g. with loops). The cache space needs to be split into blocks per QPU (core) and for each core the number of remaining free cache lines need to be tracked...

Otherwise, if kernels are chained, you will have same address out/in and can keep using VPM then.

This would be very useful, but the compiler cannot know if the input comes from another kernel or the host.

If I will lower amount of work items (i.e. will do deep nested loops in kernel), will it speed up vstore's somehow?

I doubt it, it will probably stay the same.

Damn, then it kills all usage of GPU >: 1:10 ratio of calculation/storing result is crazy.

OpenCL only provides the async_work_group_copy and prefetch functions, which both are not really what you need and also both cannot be really optimized for the VC4.
One thing though I saw in his code, or more accurately didn't see in his code, was mutex locking...

Do you use TMU too ? Is it for load only?
https://petewarden.com/2014/08/07/how-to-optimize-raspberry-pi-code-using-its-gpu/

Yeah, for parameters which are only read and not written to, the TMU is used.

Can you add non-opencl function, which will accept ARRAY of values or pointer (and it's size) and will do vstore in single lock? So usage can be like
#ifdef V4C
float16 tmp[16];
v4c_vstore(tmp, 16, offset, ram_pointer)
#endif

What exactly do you mean?
What would the function definition look like? What should be the behavior for the vc4c_vstore?
With the current code you may be able to do the following:
 vc4cl_mutex_lock();
vc4cl_dma_write(ptr, data[0]);
vc4cl_dma_write(ptr + offset, data[1]);
vc4cl_dma_write(ptr + offset * 2, data[2]);
vc4cl_dma_write(ptr + offset * 3, data[3]);
// ...
vc4cl_mutex_unlock();

This uses internal VC4CLStdLib functions, but they might change at any time without warnings!

Yes,  I was thinking if using single mutex to store many values instead 1 will speed up process. vstore takes 1 value (variable), I was thinking to use array of variables, i.e. in my example it stores 256 floats at once.

Ok, tried that:
 void arrayOut(float16* arr, int sz, __global float16* out, int stride)
           {
                vc4cl_mutex_lock();
                for (int i = 0; i < sz; ++i)
                {
                   __global float* ptr = (__global float*)(out + stride * i);
                   vc4cl_dma_write(ptr + 0, arr[i].s0);
                   vc4cl_dma_write(ptr + 1, arr[i].s1);
                   vc4cl_dma_write(ptr + 2, arr[i].s2);
                   vc4cl_dma_write(ptr + 3, arr[i].s3);

                   vc4cl_dma_write(ptr + 4, arr[i].s4);
                   vc4cl_dma_write(ptr + 5, arr[i].s5);
                   vc4cl_dma_write(ptr + 6, arr[i].s6);
                   vc4cl_dma_write(ptr + 7, arr[i].s7);

                   vc4cl_dma_write(ptr + 8, arr[i].s8);
                   vc4cl_dma_write(ptr + 9, arr[i].s9);
                   vc4cl_dma_write(ptr + 10, arr[i].sa);
                   vc4cl_dma_write(ptr + 11, arr[i].sb);

                   vc4cl_dma_write(ptr + 12, arr[i].sc);
                   vc4cl_dma_write(ptr + 13, arr[i].sd);
                   vc4cl_dma_write(ptr + 14, arr[i].se);
                   vc4cl_dma_write(ptr + 15, arr[i].sf);

                }
                vc4cl_mutex_unlock();
           }

Gives next errors:
%call2 = tail call spir_func i32 bitcast (i32 ()* @vc4cl_dma_write to i32 (float addrspace(1), double))(float addrspace(1)* %1, double %conv) #10Failed to compiler kernels:
[W] Sun Jul 29 18:49:21 2018: 64-bit operations are not supported by the VideoCore IV architecture, further compilation may fail!
[W] Sun Jul 29 18:49:21 2018: 64-bit operations are not supported by the VideoCore IV architecture, further compilation may fail!
[W] Sun Jul 29 18:49:21 2018: 64-bit operations are not supported by the VideoCore IV architecture, further compilation may fail!
[E] Sun Jul 29 18:49:21 2018: Compiler threw exception: Parser: Unhandled type of indirect function call!

I do not understand why this error is thrown, vc4cl_dma_write does not return anything. Also, why does it take double arguments?
Did you declare the function before use? The error looks like it was not declared, and as OpenCL is basically C, any not declared function is assumed to be of type int (). You will need to declare void vc4cl_dma_write(volatile __global float*, float) in this case.

No, i didn't declare anything, including mutexes.
Maybe you could change texts to something more like GCC ? Like "function is implicitly declared" or so.

Maybe you could change texts to something more like GCC ? Like "function is implicitly declared" or so.

For that I would need to know that the function is not declared. This is already handled by CLang by converting it to this strange bit-cast function call. From a VC4C point of view, this is just a bit-cast function call which is not supported.

4 times slower total, than direct vstore16

I din't notice it before, but why do you write every element of the vector extra? You could also use vc4cl_dma_write(volatile __global float*, float16) to write the whole vector at once.
Also, the loop with the non-constant iteration count will be a problem. If you allow clang to unroll it (e.g. by converting sz to a macro or constant), this should be faster than using multiple vstore16s.

Well, this works slightly slower (about 20%) then original vstore16 each.
void vc4cl_dma_write(volatile __global float*, float16);
           vc4cl_mutex_lock();
           vc4cl_mutex_unlock();
           void arrayOut16(const float16* arr,  __global float16* out, const int stride)
           {
                vc4cl_mutex_lock();
                for (int i = 0; i < 16; ++i)
                {
                   __global float* ptr = (__global float*)(out + stride * i);
                   vc4cl_dma_write(ptr + 0, arr[i]);

                }
                vc4cl_mutex_unlock();
           }

Can you send me a full example kernel of the memory access you do? So I can better see what improvements I can make that will help you.

I tried 1st kernel there - with all vstrore disabled it takes 35ms, once vstrores are enabled - takes 200+ms. Not sure why they did such chip >: just useless.


Is it a practice project?

Is it a practice project?

No, it's a demo project for my YouTube Channel.  There is a link to the series in the README.md


CUPS does not check policy in Print-Job and Create-Job

In code of functions print_job() and create_job() (scheduler/ipp.c) call like cupsdCheckPolicy() of validate_user() is missing, but in Policy directive in cupsd.conf those IPP commands are exists.
So, any policy rules will be ignored for those commands.
I use cups 2.2.0 from official release.

The policy is checked in the add_job() function which is called by both create_job() and print_job().


Deployment tier can be null

After latest Gitlab update we encountered such error:
Oneduo\LaravelGitlabWebhookClient\Data\Environment::__construct(): Argument #3 ($deployment_tier) must be of type string, null given

This PR make $deployment_tier nullable to fix the issue.

Thank you !


First Implementation of Google Dagger for Android

Google Dagger is required for Firebase InAppMessaging in Xamarin GooglePlayServiceComponent Rep.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: GuillaumeSE sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

Seems like nuget download of nuget validation cake script is failing
Downloading script from https://raw.githubusercontent.com/xamarin/XamarinComponents/GoogleDaggerAndroid/.ci/validation.cake...
Invoke-WebRequest : 404: Not Found

Thank you!!


Promptable Segmentation

Dear authors,
Thanks for your fantastic work.
I am curious about whether Open3DIS supports promptable segmentation (e.g., text-conditional segmentation) like traditional 2D SAM. Based on your visualization shown in your GitHub page, it seems Open3DIS can output a mask that accurately corresponds to the input text prompt. May I ask when will you add this function to this repository?
Best wishes,
Ye

Hi @Yebulabula,
I have committed the promptable segmentation (Grounded SAM, 2D only) script for Open3DIS.
You can modify text query at here (For example, query 3 object categories: 'object1.object2.object3')
Below I use Grounded SAM for 'chair.table' prompt

If you have any concerns, please feel free to let me know.
Best.

Hi @Yebulabula,
The current class-agnostic 3D segmenters of Open3DIS, such as ISBNet, Mask3D, SoftGroup, etc., do not currently support promptable segmentation. For investigating on detailed design of end-to-end grounding backbones, you can refer to the ScanRefer benchmark.
Best.


Demote Not registering a Vert.x handler ... message to DEBUG

Just a minor thing, but the following message just adds noise if you're only using clients (like in my case):
INFO  [io.qua.cxf.dep.CxfEndpointImplementationProcessor] (build-96) Not registering a Vert.x handler for CXF as no WS endpoints were found at build time and no other extension requested it

I'll disable this logger in my project but it might be worth thinking about to demote this message to DEBUG.

I have added those messages due to https://github.com/apache/camel-quarkus/issues/4192 I thought it would be beneficial for the users to see how their server endpoints are mapped and why the possibly do not see other (like RESTeasy) endpoints to work.
I think you are right that in case there the Vert.x route is not deployed, it can be just a debug message. So please go ahead and send a PR if you like

@ppalaga I had a quick look and I'm not entirely sure what "in case the Vert.x route is not deployed" means here.
Do you mean if cxfRouteRegistrationRequestors is empty?

Yes, the vert.x route is deployed only when there are some CXF service endpoints present in the app. That's done in the true branch of the conditional
        if (!requestors.isEmpty()) {
            final Handler<RoutingContext> handler = recorder.initServer(infos, beanContainer.getValue(),
                    httpConfiguration);
            final String mappingPath = getMappingPath(cxfBuildTimeConfig.path);
            LOGGER.infof("Mapping a Vert.x handler for CXF to %s as requested by %s", mappingPath, requestors);
            routes.produce(RouteBuildItem.builder()
                    .route(mappingPath)
                    .handler(handler)
                    .handlerType(HandlerType.BLOCKING)
                    .build());
        } else {
            LOGGER.info(
                    "Not registering a Vert.x handler for CXF as no WS endpoints were found at build time and no other extension requested it");
        }

The else branch outputs the message you do not like. I am all for changing LOGGER.info("Not registering... to LOGGER.debug("Not registering...


XML parser raises IndexError if single quotes are used instead of double quotes

When parsing the following XML, IndexError: list index out of range will be raised. This is because the XML parser assumes that attributes are wrapped in double quotes, whereas single quotes are also valid.
<SpliceInfoSection tier='0'>
<TimeSignal>
<SpliceTime ptsTime='4'/>
</TimeSignal>
</SpliceInfoSection>

One to address at a later date.
 :)

So then  no attribute can contain a single qoute inside it, correct?
<anode  this='don't' >

this will work.
we replace =' with =" and ' [space] with " [space]
.replace("='",'="' ).replace("' ",'" ')

So then no attribute can contain a single qoute inside it, correct?
<anode  this='don't' >

This scenario is supported by escaping the ' in the attribute value to be '.
That should be added to escape and unescape.

I'm replacing single quotes around attributes with double quotes. I can't think of a single SCTE-35 value that contains single or double quotes or a contraction like" don't". We're good here.


AuthenticationBuilder extensions

Description
All AddCloudFoundry___ extensions to AuthenticationBuilder should process application configuration and service bindings
Fixes #1275
Quality checklist

[ ] Your code complies with our Coding Style.
[ ] You've updated unit and/or integration tests for your change, where applicable.
[ ] You've updated documentation for your change, where applicable.
If your change affects other repositories, such as Documentation, Samples and/or MainSite, add linked PRs here.
[ ] There's an open issue for the PR that you are making. If you'd like to propose a new feature or change, please open an issue to discuss the change or find an existing issue.
[ ] You've added required license files and/or file headers (explaining where the code came from with proper attribution), where code is copied from StackOverflow, a blog, or OSS.

/azp run Steeltoe.All


http: Multiple arg of Cids/IpfsPath

There are a few Options structs, which are basically the deserialized form the of the paths query string:
$ rg "TryFrom<&'a str> for" http/src/
http/src/v0/block/options.rs
14:impl<'a> TryFrom<&'a str> for RmOptions {

http/src/v0/pin/add.rs
55:impl<'a> TryFrom<&'a str> for AddRequest {

http/src/v0/refs/options.rs
42:impl<'a> TryFrom<&'a str> for RefsOptions {

http/src/v0/pin.rs
34:impl<'a> TryFrom<&'a str> for ListRequest {

These needed to be added because the plain serde_urlencoded at least did not support at the time deserializing for example this:
#[derive(Debug)]
pub struct AddRequest {
    arg: Vec<StringSerialized<Cid>>,
    recursive: bool,
    progress: bool,
}

This might be a limitation of serde "object model" or I just failed to use the library. Writing the TryFrom conversions seemed fast and flexible enough. If this arg: Vec<FromStr> member cannot be supported with serde_urlencoded we should probably just skip the particular field, and manually parse only the arg value. The best outcome of this would be a single custom filter for deserializing these "special" kinds of "options". Right now it would seem that each of these "special options" needs a custom filter creating method as well.
See for previous attempts: https://github.com/rs-ipfs/rust-ipfs/pull/349/files#r480903618

Also very likely that this issue comes from me misunderstanding how should serde_urlencoded be used, feel free to fix in other ways as well :+1:! Minimizing boilerplate and thus LOC needed per request representation (such as AddRequest above) is key.


filter scenario results by scenario id

https://vizzuality.atlassian.net/browse/MARXAN-670

@kgajowy I have lost the thread on how the changes in this PR so far end up in this test failure - could you please help lift my mental fog on this? 😬
FAIL scenario-solutions/get-all-solutions.e2e-spec.ts
  ● When getting scenario solution results › should resolve solutions

    expect(received).toMatchInlineSnapshot(properties, snapshot)

    Snapshot name: `When getting scenario solution results should resolve solutions 2`

    - Expected properties  - 6
    + Received value       + 6

      Object {
    -   "costValue": Any<Number>,
    -   "id": Any<String>,
    -   "missingValues": Any<Number>,
    -   "planningUnits": Any<Number>,
    -   "runId": Any<Number>,
    -   "scoreValue": Any<Number>,
    +   "costValue": null,
    +   "id": "2d508950-6d25-477c-af6e-db305f588beb",
    +   "missingValues": null,
    +   "planningUnits": null,
    +   "runId": null,
    +   "scoreValue": 4000,
      }

      25 |
      26 |     expect(response.body.data.length).toEqual(1);
    > 27 |     expect(response.body.data[0].attributes).toMatchInlineSnapshot(
         |                                              ^
      28 |       {
      29 |         costValue: expect.any(Number),
      30 |         id: expect.any(String),

      at Object.<anonymous> (scenario-solutions/get-all-solutions.e2e-spec.ts:27:46)
          at runMicrotasks (<anonymous>)

@kgajowy I have lost the thread on how the changes in this PR so far end up in this test failure - could you please help lift my mental fog on this? 😬
FAIL scenario-solutions/get-all-solutions.e2e-spec.ts
  ● When getting scenario solution results › should resolve solutions

    expect(received).toMatchInlineSnapshot(properties, snapshot)

    Snapshot name: `When getting scenario solution results should resolve solutions 2`

    - Expected properties  - 6
    + Received value       + 6

      Object {
    -   "costValue": Any<Number>,
    -   "id": Any<String>,
    -   "missingValues": Any<Number>,
    -   "planningUnits": Any<Number>,
    -   "runId": Any<Number>,
    -   "scoreValue": Any<Number>,
    +   "costValue": null,
    +   "id": "2d508950-6d25-477c-af6e-db305f588beb",
    +   "missingValues": null,
    +   "planningUnits": null,
    +   "runId": null,
    +   "scoreValue": 4000,
      }

      25 |
      26 |     expect(response.body.data.length).toEqual(1);
    > 27 |     expect(response.body.data[0].attributes).toMatchInlineSnapshot(
         |                                              ^
      28 |       {
      29 |         costValue: expect.any(Number),
      30 |         id: expect.any(String),

      at Object.<anonymous> (scenario-solutions/get-all-solutions.e2e-spec.ts:27:46)
          at runMicrotasks (<anonymous>)

on it!


sync vx-demo site documentation with packages

as referenced in #124 the demo is not in sync with the documentation
what about create static/docs links dynamically so we don't risk to miss them in the future?

I inserted all the links to pages that have a readme page with something other than the install instructions

🏆 thank you!