PyDIP, where should it go from here?

Currently PyDIP is a thin wrapper around the DIPlib library. Some things are still rather awkward. For example:

dip.Image doesn't have an iterator.
Conversion of an RGB image read through pyplot yields a vector image without color space information (solutions: assume RGB if there are 3 tensor elements? write a dip.readim function?).
Parameters that require an array (e.g. dip.Gauss' sigma input) cannot handle a scalar.
etc. etc.

In short, we need lots of usage, lots of feedback from users, and lots of time polishing the interface.
But ideally we'd take this to a different level, in a way similar to what DIPimage is. For example, we could write a GUI as exists in DIPimage. And maybe make the module more "Pythonic" by changing function names.
We also need to find a way to automatically generate documentation from Doxygen. One approach is here.

Worth mentioning with regard to PyDIP is that a small selection of functions, such as PeronaMalikDiffusion, use lambda as a keyword argument. Python does not support this.

@Wicloz Indeed, lambda is a keyword in Python. I'll rename the parameter.
Thanks!

Some random musings:

Documentation
There should be more of it.
GaussianNoise is documented as requiring random, but does not (at least in PyDIP).

Convert data type
a = dip.Image((256, 256), dt='UINT8')
b = dip.Image(a, dt='UINT8')

The second fails

Advanced indexing
a = dip.Image((256, 256))
print(a[0,[1, 3]])
print(np.asarray(a)[0,[1, 3]])

The first fails

Tensor indexing
Maybe use a(0) instead of a.TensorElement(0)?

Tensor vs dimension when converting from numpy
dip.Image(np.zeros((256, 256, 9)))  # Tensor image
dip.Image(np.zeros((256, 256, 10))) # 3D image

How to do this consistently?

Passing empty images
dip.Mean(a, dip.Image())

Should be able to be written as (one of?)
dip.Mean(a, ())
dip.Mean(a, [])
dip.Mean(a, None)

Should we rewrite indexing so it is done entirely in Python? I think that's the only way we're going to get indexing syntax that matches Python's. The conversion to/from a NumPy array is essentially free, so there's no major overhead there.
To convert data type, you can use dip.Convert. Your syntax doesn't exist in C++, and hence it's not currently present in the Python interface. I guess we could add that option to the constructor to match the functionality in DIPimage.
If you want to add or improve anything in the Python interface, go ahead! There's lots of work there! :)

Should we rewrite indexing so it is done entirely in Python? I think that's the only way we're going to get indexing syntax that matches Python's. The conversion to/from a NumPy array is essentially free, so there's no major overhead there.

I assume we still want a[x, y, z] indexing, right, instead of a[y, x, z]? The question is, do we want to index tensor elements as a[...,1]/a[1,...] or as a(1)[...]? The first one can be assigned to...

To convert data type, you can use dip.Convert. Your syntax doesn't exist in C++, and hence it's not currently present in the Python interface. I guess we could add that option to the constructor to match the functionality in DIPimage.

Maybe, but Convert is fine. I must admit it is difficult to keep all functions in my head sometimes.

If you want to add or improve anything in the Python interface, go ahead! There's lots of work to do there! :)

I'm thinking about it :-)
I'll see if I can tackle some of the easier stuff first.

What do you think about a(0) instead of a.TensorElement(0)?
I'm thinking of changing the higher dimension ordering in BufferToImage. I think it should go [y, x, z, w] instead of [w, z, y, x]; how about you?
The tensor/spatial magic could be kept as is, with additional dip.SpatialImage and dip.TensorImage functions?

I assume we still want a[x, y, z] indexing, right, instead of a[y, x, z]? The question is, do we want to index tensor elements as a[...,1]/a[1,...] or as a(1)[...]? The first one can be assigned to...

I don't like mixing tensor and spatial dimensions in indexing. Separating these out in DIPimage way back when was a good idea, I found it improved code logic quite a lot, and made working with color images easier for the uninitialized (i.e. those that hadn't yet gotten used to the [...,1] way).
This works:
b.TensorElement(0)[:,:] = a

I guess b(0)[:,:] is quite OK. b[:,:](0) would then work also, but wouldn't be assignable?
So the DIPimage syntax b{1}(2,3) is translated to Python as b(0)[1,2]. It'll be a bit getting used to, but it is shorter than the TensorElement() function.
Something like b[0][:,:] could be made to work as well, but I guess you'd always have to specify the tensor indexing? Because we can't have b[0] be ambiguous.

I'm thinking of changing the higher dimension ordering in BufferToImage. I think it should go [y, x, z, w] instead of [w, z, y, x]; how about you?

I thought NumPy did [w, z, y, x], which is why it's implemented in that way. If I'm wrong about that, feel free to fix it. [y, x, z, w] sounds very MATLABy.

The tensor/spatial magic could be kept as is, with additional dip.SpatialImage and dip.TensorImage functions?

Sounds good. dip.TensorImage would be dip.SpatialImage plus a call to dip.SpatialToTensor.
For dip.SpatialImage to trick dip.Image into not casting a short dimension to the tensor dimension, you could simply add a dimension of size 1 to the matrix.

Passing empty images
dip.Mean(a, dip.Image())

Should be able to be written as (one of?)
dip.Mean(a, ())
dip.Mean(a, [])
dip.Mean(a, None)

None seems the cleanest here. But it's explicitly disallowed in Pybind11.
https://github.com/pybind/pybind11/issues/642#issuecomment-634961927

So the DIPimage syntax b{1}(2,3) is translated to Python as b(0)[1,2]. It'll be a bit getting used to, but it is shorter than the TensorElement() function.

Right. And I didn’t even think about b(0)[...] being assignable, I was hung up on b(0) not being.

I thought NumPy did [w, z, y, x], which is why it's implemented in that way. If I'm wrong about that, feel free to fix it. [y, x, z, w] sounds very MATLABy.

I don’t think numpy cares much, except that it prints the first index vertically. It is disconnected from the storage order, which can be chosen by the user. I’ll figure out what imageio does and make sure the conversion to and from that is easy.

None seems the cleanest here. But it's explicitly disallowed in Pybind11.

I came across that issue. Let’s see what they suggest. We could always allow it in the DIPlib repo, but it would be nice to know what the consequences are.

I did not find any issues with the py::none change when running the standard examples. I therefore suggest the following patch:
 dependencies/pybind11/include/pybind11/cast.h |  4 ++--
 pydip/src/PyDIP_py.py                         | 25 +++++++++++++++++++++++++
 pydip/src/image.cpp                           | 22 ++++++++++++++++------
 3 files changed, 43 insertions(+), 8 deletions(-)

diff --git a/dependencies/pybind11/include/pybind11/cast.h b/dependencies/pybind11/include/pybind11/cast.h
index 605acb36..70aa770c 100644
--- a/dependencies/pybind11/include/pybind11/cast.h
+++ b/dependencies/pybind11/include/pybind11/cast.h
@@ -642,8 +642,8 @@ public:
         if (src.is_none()) {
             // Defer accepting None to other overloads (if we aren't in convert mode):
             if (!convert) return false;
-            value = nullptr;
-            return true;
+            //value = nullptr;
+            //return true;
         }

         auto &this_ = static_cast<ThisT &>(*this);
diff --git a/pydip/src/PyDIP_py.py b/pydip/src/PyDIP_py.py
index 69994e57..6b540d30 100644
--- a/pydip/src/PyDIP_py.py
+++ b/pydip/src/PyDIP_py.py
@@ -153,3 +153,28 @@ def Show(img, range=(), complexMode='abs', projectionMode='mean', coordinates=()

 PyDIP.Image.Show = Show
+
+
+def ScalarImage(a):
+   """Convert numpy.ndarray to scalar PyDIP.Image"""
+   img = PyDIP.Image(a)
+   if img.TensorElements() > 1:
+      if img.TensorStride() > 1:
+         img.TensorToSpatial(img.Dimensionality())
+      else:
+         img.TensorToSpatial(0)
+   return img
+
+
+def TensorImage(a, axis=-1):
+   """Convert numpy.ndarray to column tensor PyDIP.Image
+
+   Keyword arguments:
+   axis -- array axis to convert to Tensor; defaults to the last axis.
+   """
+   img = SpatialImage(a)
+   axis = -axis - 1
+   if axis < 0:
+     axis += img.Dimensionality()
+   img.SpatialToTensor(axis)
+   return img
diff --git a/pydip/src/image.cpp b/pydip/src/image.cpp
index b39f3f7b..e0be4ff9 100644
--- a/pydip/src/image.cpp
+++ b/pydip/src/image.cpp
@@ -112,9 +112,13 @@ dip::Image BufferToImage( py::buffer& buf ) {
    }};
    // Create an image with all of this.
    dip::Image out( dataSegment, info.ptr, datatype, sizes, strides, {}, 1 );
-   // If it's a 3D image and the first dimension has <10 pixels, let's assume it's a tensor dimension:
-   if(( sizes.size() > 2 ) && ( sizes[ 0 ] < 10 )) {
-      out.SpatialToTensor( 0 );
+   // If it's a 3D image and the first or last dimension has <10 pixels, let's assume it's a tensor dimension:
+   if(( sizes.size() > 2 ) && ( sizes[ 0 ] < 10 || sizes[ ndim-1 ] < 10 )) {
+      if ( sizes[ 0 ] < sizes[ ndim - 1 ]) {
+         out.SpatialToTensor( 0 );
+      } else {
+         out.SpatialToTensor( ndim-1 );
+      }
    }
    return out;
 }
@@ -222,6 +226,8 @@ void init_image( py::module& m ) {
    auto img = py::class_< dip::Image >( m, "Image", py::buffer_protocol(), "The class that encapsulates DIPlib images of all types." );
    // Constructor for raw (unforged) image, to be used e.g. when no mask input argument is needed
    img.def( py::init<>() );
+   img.def( py::init([](py::none const& ) { return dip::Image{}; }), "none"_a );
+   py::implicitly_convertible< py::none, dip::Image >();
    // Constructor, generic
    img.def( py::init< dip::UnsignedArray const&, dip::uint, dip::DataType >(), "sizes"_a, "tensorElems"_a = 1, "dt"_a = dip::DT_SFLOAT );
    // Constructor that takes a Sample
@@ -354,9 +360,13 @@ void init_image( py::module& m ) {
    img.def( "Imaginary", []( dip::Image const& image ) -> dip::Image { return image.Imaginary(); } );
    img.def( "QuickCopy", &dip::Image::QuickCopy );
    // These don't exist, but we need to have a function for operation[] too
-   img.def( "TensorElement", []( dip::Image const& image, dip::sint index ) -> dip::Image { return image[ index ]; }, "index"_a );
-   img.def( "TensorElement", []( dip::Image const& image, dip::uint i, dip::uint j ) -> dip::Image { return image[ dip::UnsignedArray{ i, j } ]; }, "i"_a, "j"_a );
-   img.def( "TensorElement", []( dip::Image const& image, dip::Range const& range ) -> dip::Image { return image[ range ]; }, "range"_a );
+   img.def( "__call__", []( dip::Image const& image, dip::sint index ) -> dip::Image { return image[ index ]; }, "index"_a );
+   img.def( "__call__", []( dip::Image const& image, dip::uint i, dip::uint j ) -> dip::Image { return image[ dip::UnsignedArray{ i, j } ]; }, "i"_a, "j"_a );
+   img.def( "__call__", []( dip::Image const& image, dip::Range const& range ) -> dip::Image { return image[ range ]; }, "range"_a );
+   // NOTE: Compatibility with beta PyDIP. Remove?
+   img.def("TensorElement", [](dip::Image const& image, dip::sint index) -> dip::Image { return image[index]; }, "index"_a);
+   img.def("TensorElement", [](dip::Image const& image, dip::uint i, dip::uint j) -> dip::Image { return image[dip::UnsignedArray{ i, j }]; }, "i"_a, "j"_a);
+   img.def("TensorElement", [](dip::Image const& image, dip::Range const& range) -> dip::Image { return image[range]; }, "range"_a);
    // Copy or write data
    img.def( "Pad", py::overload_cast< dip::UnsignedArray const&, dip::String const& >( &dip::Image::Pad, py::const_ ), "sizes"_a, "cropLocation"_a = "center" );
    img.def( "Copy", py::overload_cast<>( &dip::Image::Copy, py::const_ ));

Question: should we remove the TensorElement method? I think it's cleaner to do so, but it will break existing code.

Awesome! Thanks!
I don't know what to do with the TensorElement method. I never really liked it, it was just an easy solution at the time. But it's hard to know how many people might be using the Python bindings by now.

Please create a branch. I'd like to test a few things as well. For example make sure things don't break if you put None in random places where it's not expected.
Also, I have some changes I want to add concerning raw images and zero-size NumPy arrays, that sort of goes together with the None change.
Fantastic that you implemented casting of scalars to DimensionArray. It would be good to do that for dip::Pixel too. Scalars cast to dip::Sample, and dip::Sample casts to dip::Pixel, but I don't know if both steps are applied in the interface.
Many thanks!

Ok! Let's make sure it works in all cases... 3571a8a28fc0d3f0ab51cd93f8bede64bcad800d .

I can't get your changes to type_caster< dip::DimensionArray< Type >> to compile. For some reason value and value_conv are not recognized by the compiler. You can see these errors on Travis as well, but only for MacOS: https://travis-ci.org/github/DIPlib/diplib/builds/692600861

I tried some things until Xcode was happy. Not sure why it needs this, though.

Weird that I had this issue with GCC on Mac, not Xcode.
I've pushed some more changes to this branch. The DimensionArray type caster change turned dip.Image(50) into the same as dip.Image([50]), creating a 1D image with 50 pixels, rather than a 0D image with the value 50. We now distinguish the two again by changing the order of the constructors (the first one that matches the signature is used). Plus, Sample implicitly converts to Image, so now we can do things like dip.Modulo(out, 50) instead of dip.Modulo(out, dip.Image(50)).
There no longer is a constructor that takes a Pixel, there was no way of distinguishing that from the one that takes a DimensionArray. If we ever make an interface to the Pixel class (rather than casting to/from a Python list), we can put that constructor back.
Also, things like dip.Image(np.array((0,))) and np.array(dip.Image()) now work correctly.

Awesome! Do we want
dip.DrawBandlimitedBall(img, 30, (42, 60), 255, "filled", 5)

To work? In that case, scalars should cast to Pixel as well.
On as side note, I think we're going to need Python unit tests...

Yes, anything that makes usage easier is welcome.
And we certainly do need Python unit tests. Right now, Travis only compiles the Python interface on MacOS, the other 3 runs only compile DIPlib itself. So there's a lot to do to get to that point. But even if the tests are only run locally and on demand, it would help a lot when making important changes like the ones we're making here.
I've also been thinking about adding the out image argument to functions. What would it look like to have an optional out argument? Would that mean we'd have two different function signatures, like in C++? If someone passes a NumPy array for out, then it's cast to a dip.Image, this image must be Protected, otherwise the function could re-forge the image and write to something that is not the NumPy array. But the conversion function does't know it's an input or an output image. So what happens if we always protect images created from a buffer? Is there some place this could be problematic?

I committed the scalar to Pixel conversion. 77e33b9091d51cc961ee5b7536c65d53c42c5f47
I also tried Python's doctest on for size. I think it's quite a nice way of writing the tests; they can be used for reference as well. What do you think? cdb4675c983507faa59b1146cf84db8f3c342fb6

That’s an excellent way to write a test! Really cool. It might end up being a bit overly detailed in places, for a tutorial. But it’s a great way of ensuring that the documentation and the code don’t go out of synch. I really like that.

I integrated the unit tests into the build system. However, I had to disable CMAKE_BUILD_WITH_INSTALL_RPATH, otherwise PyDIP_bin cannot be loaded from the build tree. Do you have a solution for this?

It should be fine to disable that. I think I added it to make the install faster.
This was especially important when DIPimage had more than 100 MEX-files to re-link. We have a lot fewer now, so it's not such a big deal. Maybe we can set_target_properties("mex-file", BUILD_WITH_INSTALL_RPATH ON) or something like that?

Brilliant. I did the opposite:
set_target_properties(PyDIP_bin PROPERTIES BUILD_WITH_INSTALL_RPATH OFF)

@wcaarls : I've fixed the issue with the tensor iterator in cases of 0 stride: edbb27abae488e117f8c88f6f9506038d2100783
I've also rebased your branch pydip-experimental onto master, and at the same time removed some "oops" commits. If you want to continue working on this branch, please git pull --rebase, or simply delete your local branch and then pull a new copy. I've merged most stuff into master, except the new Python testing. I'll let you tweak that some more first if you want. By the way, I moved the test files to a separate directory to avoid confusion.
We need to enable assertions in Travis so that they can fire during testing. One of your new Python tests failed an assertion inside DIPlib code, which brought to light a bug: 1520820aa514ee7efd3d0950c11f5939b2ad9d10

Great!
I think the current state can be merged to master, although it makes sense to keep the branch around for further development, such as on the out / protect question.
Enabling assertions on Travis sounds like a good idea, is it a matter of passing -DDIP_ENABLE_ASSERT=ON in the script?

Feel free to merge. I merged a portion because I wanted to change the order of some commits, and push out the bug fixes right away.
Yes, -DDIP_ENABLE_ASSERT=ON as a parameter to CMake.
I see that the Travis configure script installs python3-dev on Linux, but somehow this is not found by the CMake configuration script.

Merged. I'm not sure what to do with the current state of the pydip-experimental branch, though. I'd just delete it and rebranch from master, but I'm sure you have a better idea...

This should work:
git checkout pydip-experimental
git rebase master

If you get nasty conflicts, you can cancel and do
git reset --hard HEAD~5  # or whatever number of commits
git rebase master

Nice by the way that you got all that Travis stuff working. Thanks!

I really don't grok git yet. I ended up doing
git checkout pydip-experimental
git reset --hard master
git push --force

to make pydip-experimental equal to master.
Travis took some round trips, but worked in the end. Note though that Windows is now compiling in 64-bits.

A summary of what happened in this thread:

dip.Image doesn't have an iterator.
➜ It doesn't need one.
Conversion of an RGB image read through pyplot yields a vector image without color space information (solutions: assume RGB if there are 3 tensor elements? write a dip.readim function?).
➜ This is still the case.
Parameters that require an array (e.g. dip.Gauss' sigma input) cannot handle a scalar.
➜ Solved.
Documentation: There should be more of it.
➜ There's a little bit more of it, but still way too little.
GaussianNoise is documented as requiring random, but does not in PyDIP.
➜ Functions that deviate from the C++ syntax like this one have specific documentation in Python.
Convert data type: a = dip.Image((256, 256), dt='UINT8'); b = dip.Image(a, dt='UINT8') fails.
➜ Use dip.Convert() instead.
Advanced indexing: a = dip.Image((256, 256)); a[0,[1, 3]] fails, but np.asarray(a)[0,[1, 3]] works.
➜ It's cheap to convert to NumPy array, for complex indexing like this. We're only supporting indexing as defined in C++ for now. We're open to changing this, but it won't be easy.
Tensor indexing: Maybe use a(0) instead of a.TensorElement(0)?
➜ Done.
Tensor vs dimension when converting from numpy:dip.Image(np.zeros((256, 256, 9)))  # Tensor image
dip.Image(np.zeros((256, 256, 10))) # 3D image

How to do this consistently?
➜ Added an optional argument axis to this constructor, to allow the user to pick which dimension (if any) to use as the tensor dimension. The default behavior hasn't changed.
Passing empty images: dip.Mean(a, dip.Image()). Should be able to be written as dip.Mean(a, ()), dip.Mean(a, []), or dip.Mean(a, None)?
➜ Done. None casts implicitly to a raw (empty) dip.Image.

Continue the discussion here: #105


Collaborative Whiteboard

@Sulagna-Dutta-Roy
Collaborative Whiteboard
Is your feature request related to a problem? Please describe.
The feature request is related to enabling real-time collaboration on a whiteboard. The problem it aims to solve is the inability for multiple users to work together on a digital whiteboard simultaneously. This limitation hinders collaborative brainstorming, visual communication, and remote teamwork.
Describe the solution you'd like
The solution involves creating a web-based whiteboard application that allows multiple users to draw, write, and interact with the canvas simultaneously. Each user should be able to see the changes made by others in real-time, fostering seamless collaboration.
Describe alternatives you've considered
An alternative solution could involve using dedicated collaboration software or integrating existing whiteboard tools with collaboration features. However, developing a custom solution allows for tailored functionality and seamless integration with other systems or platforms.

Already added to my repository #955

Closing this


IterableDataset: split by node and map may preprocess samples that will be skipped anyway

          There are two ways an iterable dataset can be split by node:

if the number of shards is a factor of number of GPUs: in that case the shards are evenly distributed per GPU
otherwise, each GPU iterate on the data and at the end keeps 1 sample out of n(GPUs) - skipping the others.

In case 2. it's therefore possible to have the same examples passed to prepare_dataset for each GPU.
This doesn't sound optimized though, because it runs the preprocessing on samples that won't be used in the end.
Could you open a new issue so that we can discuss about this and find a solution ?
Originally posted by @lhoestq in https://github.com/huggingface/datasets/issues/5360#issuecomment-1592729051

The number of shards is ds.n_shards. It corresponds generally to the number of files the dataset is made of, to be able to distribute to several nodes.
You don't end up with the same data per GPU. But all the samples are going through your preprocessing function you pass to map. They are just skipped afterwards to only keep 1 sample out of n(GPUs)

For each GPU, although see the same data in prepare_data(), the actual training data will not be the same in the end.
Is my understanding correct?

Yes exactly :)

Where can I print the actual training data for each GPU?

You should call print in the data_collator

I print out n_shards, and under multiple GPUs, this value is always 1.
Is this value correct?

Yes it's correct, and it explains why you always have the same data passed to your map function (the data can't be split).
But after being passed to map, each GPU keeps one example out of n(GPUs) so that you don't end up with duplicate data across GPUs

For each GPU, although see the same data in prepare_data(), the actual training data will not be the same in the end.
Is my understanding correct?

Yes exactly :)

Where can I print the actual training data for each GPU?

You should call print in the data_collator

OK, when printing the train data in the data collator, each GPU sees different data.
Thanks for your reply

Do we have a solution for this one? Or it's required to get "number of shards is a factor of number of GPUs: in that case the shards are evenly distributed per GPU"

For now it's required to have a number of shards that is a factor of the number of GPUs to not have all the workers process the same data (and then skip the right ones to not end up training on duplicate data).
It would be quite complex to implement a strategy that would utilize all the GPUs with an arbitrary number of shards even at the end of training


Query selector breaks element when used with hash mode routing

Using the storytelling element in a SPA with hash routing causes troubles, as it thinks the hash in the url is a selector:
Example url: https://mywebsite.com/sub/path/to/app/#/storytelling
Error: Uncaught (in promise) SyntaxError: Failed to execute 'querySelector' on 'DocumentFragment': '#/storytelling' is not a valid selector.
https://github.com/EOX-A/EOxElements/blob/e88e875cdf79806c66d44b9174f7be3be2905d7c/elements/storytelling/src/helpers/misc.js#L25-L29

Currently can't reproduce anymore :thinking:


[MM-49515] Show board insights for boards 'product'

Summary

Cherrypick of https://github.com/mattermost/mattermost-webapp/pull/11922
Ticket Link

https://mattermost.atlassian.net/browse/MM-49515
Release Note

NONE

Will try to auto merge this PR once all tests and checks are passing. This might take up to an hour.


Tooltip model is based on blockClosure outside context

The tooltips in are managed by the trait TToElementWithTooltip.
When you add a tooltip, you use one of the following methods:

tooltipString:
tooltipText:
tooltipContent:

All the methods use the tooltipContent:.
Here is the source code of tooltipContent:.
tooltipContent: anElement

	anElement ifNil: [
		self removeTooltipWindowManager.
		^ self ].
	self tooltipBuilder: [ :win :requestEventt |
		win root addChild: anElement ]

anElement here is the BlElement we want to display in the tooltip.
The element is not added directly in a "tooltip model" but as an outter context variable of a blockClosure.

It make it difficult to try to work with tooltip.
It is hard to understand what is the tooltip of an element using only the inspector.
It make it difficult to retreive information about the tooltip.

The advantage of having a BlockClosure is that you can generate on the fly the content of the tooltip.
This effect could be achieve by adding new event for tooltip:

An event that is trigger just before the tootltip is shown.
We could add an event that trigger after the tooltip is hidden in order to manage the destruction of the generated content.

the benefit of current implementation is that is is stateless
one can store the element in the user data maybe

Why being stateless is a benefit ?

the trait is used by ToElement level as TToElementWithContextMenu so, I guess one have to limit the number of inst variables for a ToElement.

" The advantage of having a BlockClosure is that you can generate on the fly the content of the tooltip. "
yes, exactly
"This effect could be achieve by adding new event for tooltip:
An event that is trigger just before the tootltip is shown.
We could add an event that trigger after the tooltip is hidden in order to manage the destruction of the generated content."
I don't get what you mean

@Nyan11 what are the serializer(s) that Pyramid uses to serialize?
I'm analysing a possible solution in Fuel using "clean" blocks. But not sure if you use Fuel.

Hello,
I used a custom one that generate executable code: https://github.com/Nyan11/Stash
It is inspired from STON.
Currently i only check if the block closure is clean before serializing it.
If it is not, i raise an Error.

Stash looks nice! But I only saw it in the web browser. I guess I have to load Pyramid to see it in action.

If you want to use it without loading Pyramid, you will need https://github.com/OpenSmock/Bloc-Serialization
Bloc-Serialization contains all the methods needed to serialize with Stash and Ston the BlElement and their properties (except for the  grid layout that is still mssing).
Once loaded, you can open a new space with BlElement inside, inspect one of the BlElement and modify it using the inspector.
For example:

BlElement new
background: (BlPaintBackground new
paint: (BlColorPaint new
color: (Color r: 1.0 g: 0.0 b: 0.0 alpha: 1.0);
yourself);
opacity: 1.0;
yourself);
border: (BlBorderBuilder new
paint: (BlColorPaint new
color: (Color r: 0.0 g: 1.0 b: 0.0 alpha: 1.0);
yourself);
width: 7;
build);
yourself


Make the extension also a web extension

It would be nice if the extension could also run in a web extension host. It should be investigated if this is possible. All the information should be available at https://code.visualstudio.com/api/extension-guides/web-extensions

Ok, I think this can be done with just a little bit of configuration, so I'll give it a try!


string-or-bytes-too-long (PYI053) flags and removes long strings in Literal, producing invalid type

In the example reproduction below,

List of keywords you searched for before creating this issue. Write them down here so that others can find this issue more easily and help provide feedback.
string-or-bytes-too-long PYI053

A minimal code snippet that reproduces the bug.

def foo(bar: typing.Literal["a", "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"]):...

becomes
import typing

def foo(bar: typing.Literal["a", ...]):...

Not only is this an invalid type form (https://github.com/microsoft/pyright/blob/main/docs/configuration.md#reportInvalidTypeForm) but even if autofixed to Literal["a"] that would fundamentally change the parameter's type.
I assume this is an oversight as it's meant to replace long default values and is not flagged by flake8-pyi .

The command you invoked (e.g., ruff /path/to/file.py --fix), ideally including the --isolated flag.
ruff check --isolated --select=PYI053

The current Ruff settings (any relevant sections from your pyproject.toml).
not relevant

The current Ruff version (ruff --version).
ruff 0.6.1

is not flagged by flake8-pyi

I can actually repro this bug with flake8-pyi as well on this snippet in a .pyi file:
from typing import Literal

x: Literal["fooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooO"]

We disable the rule at typeshed for our generated stubs; maybe that's what was hiding the bug? https://github.com/python/typeshed/blob/a7ff1be4394cef089dd62090a2e2ad10cc77cfe6/.flake8#L16

I merged #13002, but then immediately afterwards realised that it doesn't deal properly with type aliases. We still get two false positives on this snippet, even after #13002:
from typing import TypeAlias, Literal, Annotated

x: TypeAlias = Literal["fooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooO"]
y: TypeAlias = Annotated[int, "metadataaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"]

@dylwil3, don't suppose you'd be interested in a followup PR, would you? :-) The alternative approach I suggested in https://github.com/astral-sh/ruff/pull/13002#discussion_r1723034693 might work better here after all:

we'd want to add tracking in the semantic model for when we're entering Annotated slices, the same as we do when we're entering Literal slices
we'd want to skip PYI053 when we're inside any Annotated or Literal slice, not just when we're inside annotations.

Sounds good I'll give it a try!

@AlexWaygood  Can you not do
def f(x: int) --> "AnnotationsForClassesWithVeryLongNamesInQuotesAsReturnTypes"

?

@AlexWaygood Can you not do
def f(x: int) --> "AnnotationsForClassesWithVeryLongNamesInQuotesAsReturnTypes"

?

Well, there's no reason to ever use quoted annotations in stubs, since forward references can be used even with __future__ annotations enabled. We have another rule (PYI020) specifically for flagging any quoted annotation in a stub... but I suppose if you are going to use quoted annotations in stubs for some reason, and have PYI020 switched off for some reason, then maybe it makes sense for this rule to ignore veryyyyyyyyyyyyyyyyyyyy long quoted annotations.
In case it's not obvious, I changed my mind about this while writing this response out 😆 So it's definitely worth adding a comment about this to the code for the rule!

Hmm... for some reason in_typing_literal has no effect - it doesn't trip for either the argument annotation or typevar. Maybe I've done something wrong. I'll try to investigate later today.

Hmm... for some reason in_typing_literal has no effect - it doesn't trip for either the argument annotation or typevar. Maybe I've done something wrong. I'll try to investigate later today.

That is odd. I can't immediately see why that would be... don't worry too much about fixing it if you can't figure it out; feel free to ping me if you're stuck!

We disable the flake8-pyi rule at typeshed for our generated stubs; maybe that's what was hiding the bug?

Yep, you're right !

Sorry for the delay - I was derailed by my baby learning to crawl all of a sudden 🚀 🚀 🚀

Hmm... for some reason in_typing_literal has no effect - it doesn't trip for either the argument annotation or typevar. Maybe I've done something wrong. I'll try to investigate later today.

That is odd. I can't immediately see why that would be... don't worry too much about fixing it if you can't figure it out; feel free to ping me if you're stuck!

Mystery solved - I had an invalid .pyi file since I did not actually import Literal or Annotated 🤦


[ENHANCEMENT] Preventing VM Backup if the VM is under Migration

Is your enhancement related to a problem? Please describe.
If the VM Backup is triggered during the VM is under Migration, there are multiple situations may occur:

Filesystem freeze fail

It could happen if the VM Backup starts as two VM launcher pods are running
From compute log:compute {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"vm1","namespace":"default","pos":"server.go:269","reason":"Failed to freeze VMI, VMI is currently during migration","timestamp":"2024-09-20T02:39:05.371600Z","uid":"16b24dd6-babb-4168-aa97-498e81cf3b12"}
compute {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"vm1","namespace":"default","pos":"server.go:269","reason":"Failed to freeze VMI, VMI is currently during migration","timestamp":"2024-09-20T02:39:05.444526Z","uid":"16b24dd6-babb-4168-aa97-498e81cf3b12"}
compute {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"vm1","namespace":"default","pos":"server.go:269","reason":"Failed to freeze VMI, VMI is currently during migration","timestamp":"2024-09-20T02:39:05.531953Z","uid":"16b24dd6-babb-4168-aa97-498e81cf3b12"}
compute {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"vm1","namespace":"default","pos":"server.go:269","reason":"Failed to freeze VMI, VMI is currently during migration","timestamp":"2024-09-20T02:39:05.668165Z","uid":"16b24dd6-babb-4168-aa97-498e81cf3b12"}
compute {"component":"virt-launcher","kind":"","level":"error","msg":"Failed to freeze vmi","name":"vm1","namespace":"default","pos":"server.go:269","reason":"Failed to freeze VMI, VMI is currently during migration","timestamp":"2024-09-20T02:39:05.758315Z","uid":"16b24dd6-babb-4168-aa97-498e81cf3b12"}

Kubevirt rejects freeze fs during VM is under migration https://github.com/kubevirt/kubevirt/blob/1466b658f78b9b8bb9517ffb6dafd4b777f33fe6/pkg/virt-launcher/virtwrap/manager.go#L1597-L1599

LH SnapshotCR creation fail

From VMBackup manifest:apiVersion: harvesterhci.io/v1beta1
kind: VirtualMachineBackup
  name: vm-b1
  namespace: default
  resourceVersion: "548372"
  uid: 833dbced-110c-4259-956c-53047fde9faa
spec:
  source:
    apiGroup: kubevirt.io
    kind: VirtualMachine
    name: vm1
  type: backup
status:
  ....
  volumeBackups:
  - csiDriverName: driver.longhorn.io
    error:
      message: 'Failed to check and update snapshot content: failed to take snapshot
        of the volume pvc-0d63908e-8c78-40e6-9198-5701cd1c8b53: "rpc error: code =
        Internal desc = Bad response statusCode [500]. Status [500 Internal Server
        Error]. Body: [code=Internal Server Error, detail=, message=failed to create
        snapshot CR: cannot operate during migration] from [http://longhorn-backend:9500/v1/volumes/pvc-0d63908e-8c78-40e6-9198-5701cd1c8b53?action=snapshotCRCreate]"'
      time: "2024-09-20T02:50:05Z"
    name: vm-b1-volume-vm1-disk-0-d65jn
    ....
    readyToUse: false
    volumeName: disk-0
    volumeSize:<PHONE_NUMBER>0

LH rejects creating snapshot CR if the volume is under migration https://github.com/longhorn/longhorn-manager/blob/5ff1f83aefb2c17365264d7cc111e04415c9cef8/manager/snapshot.go#L31-L33

LH Backup fails to create the VA ticket

From LH Manager log:longhorn-manager-8v8nh:longhorn-manager time="2024-09-20T02:56:36Z" level=error msg="Failed to sync Longhorn backup" func=controller.handleReconcileErrorLogging file="utils.go:74" Backup=longhorn-system/backup-7b8dad8f64464a31 controller=longhorn-backup error="longhorn-backup: failed to sync backup longhorn-system/backup-7b8dad8f64464a31: handleAttachmentTicketCreation: failed to create/update attachment: admission webhook \"validator.longhorn.io\" denied the request: cannot attach migratable volume pvc-0d63908e-8c78-40e6-9198-5701cd1c8b53 to more than two nodes" node=harvester-node-0
longhorn-manager-8v8nh:longhorn-manager time="2024-09-20T02:56:36Z" level=error msg="Failed to sync Longhorn backup" func=controller.handleReconcileErrorLogging file="utils.go:74" Backup=longhorn-system/backup-7b8dad8f64464a31 controller=longhorn-backup error="longhorn-backup: failed to sync backup longhorn-system/backup-7b8dad8f64464a31: handleAttachmentTicketCreation: failed to create/update attachment: admission webhook \"validator.longhorn.io\" denied the request: cannot attach migratable volume pvc-0d63908e-8c78-40e6-9198-5701cd1c8b53 to more than two nodes" node=harvester-node-0

LH rejects creating the VA ticket if the volume is under migration https://github.com/longhorn/longhorn-manager/blob/5ff1f83aefb2c17365264d7cc111e04415c9cef8/webhook/resources/volumeattachment/validator.go#L121-L124

Describe the solution you'd like
VM Backup webhook should check if the VM is under migration

verified on v1.4-rc2 and works as expected.

verified on v1.4-rc2 and it works as expected


EDC 0.7.0+ → Azure Deployment →  PolicyMonitor Default Services→ Cannot unwrap ThreadPoolExecutor

EDC 0.7.0 Azure Deployment, Chart form HEAD
edc070-tractusx-connector-azure-vault-controlplane
INFO 2024-05-01T13:12:09.160275028 Started Commons Connection Pool
INFO 2024-05-01T13:12:09.160328128 Started PolicyMonitor Default Services
🍄[2024-05-01 13:12:09] [INFO   ] Cannot unwrap ThreadPoolExecutor for monitoring from java.util.concurrent.Executors$DelegatedScheduledExecutorService due to java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.util.concurrent.ScheduledExecutorService java.util.concurrent.Executors$DelegatedScheduledExecutorService.e accessible: module java.base does not "opens java.util.concurrent" to unnamed module @cb0ed20
INFO 2024-05-01T13:12:09.171962092 Started org.eclipse.edc.connector.policy.monitor.PolicyMonitorExtension
INFO 2024-05-01T13:12:09.172078491 Started API Core
INFO 2024-05-01T13:12:09.172142691 Started Jersey Micrometer Metrics
INFO 2024-05-01T13:12:09.172331491 Started Web DID
INFO 2024-05-01T13:12:09.17239629 Started org.eclipse.edc.iam.identitytrust.issuer.configuration.TrustedIssuerConfigurationExtension
INFO 2024-05-01T13:12:09.177329475 edc070-tractusx-connector-azure-vault-controlplane ready

Does 🍄 indicate some misconfiguration of the chart?

Any update on this? We also have same issue/exception comes up during start.

Issue got resolved by adding jvmargs. This was proposed solution from micrometer https://github.com/micrometer-metrics/micrometer/issues/2317
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
Not sure will it cause any security issue as this module open for all now.


[Bug] Variable substitution does not work in plan only mode

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Search before asking

[X] I have searched in the issues and found no similar issues.

Describe the bug
0: jdbc:hive2://<IP_ADDRESS>:10009/> set kyuubi.operation.plan.only.mode=parse;
10:21:15.802 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing fchen's query[5027f6c0-03bd-4541-861c-ef12da759ad4]: INITIALIZED_STATE -> PENDING_STATE, statement: set kyuubi.operation.plan.only.mode=parse
10:21:15.846 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing fchen's query[5027f6c0-03bd-4541-861c-ef12da759ad4]: PENDING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.plan.only.mode=parse
10:21:15.835 INFO org.apache.kyuubi.engine.spark.operation.ExecuteStatement: Processing fchen's query[b4e7c9d5-9a47-4185-8a0b-b4ad9655dc13]: INITIALIZED_STATE -> PENDING_STATE, statement: set kyuubi.operation.plan.only.mode=parse
10:21:15.839 INFO org.apache.kyuubi.engine.spark.operation.ExecuteStatement: Processing fchen's query[b4e7c9d5-9a47-4185-8a0b-b4ad9655dc13]: PENDING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.plan.only.mode=parse
10:21:15.841 INFO org.apache.kyuubi.engine.spark.operation.ExecuteStatement:
           Spark application name: kyuubi_USER_SPARK_SQL_fchen_default_6c5b23b2-e017-48c1-9f30-7bdee7f42815
                 application ID: local-1651803664826
                 application web UI: http://<IP_ADDRESS>:50619
                 master: local[*]
                 deploy mode: client
                 version: 3.2.0
           Start time: 2022-05-06T10:21:03.817
           User: fchen
10:21:15.876 INFO org.apache.kyuubi.engine.spark.operation.ExecuteStatement: Execute in full collect mode
10:21:15.896 INFO org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator: Code generated in 9.750672 ms
10:21:15.927 INFO org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator: Code generated in 9.754239 ms
10:21:15.933 INFO org.apache.kyuubi.engine.spark.operation.ExecuteStatement: Processing fchen's query[b4e7c9d5-9a47-4185-8a0b-b4ad9655dc13]: RUNNING_STATE -> RUNNING_STATE, statement: set kyuubi.operation.plan.only.mode=parse
10:21:15.933 INFO org.apache.kyuubi.engine.spark.operation.ExecuteStatement: Processing fchen's query[b4e7c9d5-9a47-4185-8a0b-b4ad9655dc13]: RUNNING_STATE -> FINISHED_STATE, statement: set kyuubi.operation.plan.only.mode=parse, time taken: 0.094 seconds
10:21:15.983 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[5027f6c0-03bd-4541-861c-ef12da759ad4] in FINISHED_STATE
10:21:15.983 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing fchen's query[5027f6c0-03bd-4541-861c-ef12da759ad4]: RUNNING_STATE -> FINISHED_STATE, statement: set kyuubi.operation.plan.only.mode=parse, time taken: 0.137 seconds
+----------------------------------+--------+
|               key                | value  |
+----------------------------------+--------+
| kyuubi.operation.plan.only.mode  | parse  |
+----------------------------------+--------+
1 row selected (8.99 seconds)
0: jdbc:hive2://<IP_ADDRESS>:10009/> set x=y;
10:21:24.797 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing fchen's query[9cd03fd0-a825-4f16-8412-14795c9e3f09]: INITIALIZED_STATE -> PENDING_STATE, statement: set x=y
10:21:24.834 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing fchen's query[9cd03fd0-a825-4f16-8412-14795c9e3f09]: PENDING_STATE -> RUNNING_STATE, statement: set x=y
10:21:24.850 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[9cd03fd0-a825-4f16-8412-14795c9e3f09] in FINISHED_STATE
10:21:24.850 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing fchen's query[9cd03fd0-a825-4f16-8412-14795c9e3f09]: RUNNING_STATE -> FINISHED_STATE, statement: set x=y, time taken: 0.016 seconds
+------+--------+
| key  | value  |
+------+--------+
| x    | y      |
+------+--------+
1 row selected (0.256 seconds)
0: jdbc:hive2://<IP_ADDRESS>:10009/> select '${x}';
10:21:30.636 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing fchen's query[65402727-35c6-4692-ab38-6bf4343b45dd]: INITIALIZED_STATE -> PENDING_STATE, statement: select '${x}'
10:21:30.690 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing fchen's query[65402727-35c6-4692-ab38-6bf4343b45dd]: PENDING_STATE -> RUNNING_STATE, statement: select '${x}'
10:21:30.694 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[65402727-35c6-4692-ab38-6bf4343b45dd] in FINISHED_STATE
10:21:30.694 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing fchen's query[65402727-35c6-4692-ab38-6bf4343b45dd]: RUNNING_STATE -> FINISHED_STATE, statement: select '${x}', time taken: 0.004 seconds
+----------------------------------------------------+
|                        plan                        |
+----------------------------------------------------+
| 'Project [unresolvedalias(, None)]
+- OneRowRelation
 |
+----------------------------------------------------+
1 row selected (0.075 seconds)

Affects Version(s)
master
Kyuubi Server Log Output
No response
Kyuubi Engine Log Output
No response
Kyuubi Server Configurations
No response
Kyuubi Engine Configurations
No response
Additional context
No response
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

parse mode means that engine only parse, analyze or optimize statements instead of executing them. Of course, it will not run the set x = y

According to #1059, parse mode means that engine only parse, analyze or optimize statements instead of executing them. Of course, it will not run the set x = y

The kyuubi.operation.plan.only.excludes configuration allows some commands to be executed.


[Add-ins] Add internet headers article

Docs Build status updates of commit 2eaa464:
:warning: Validation status: warnings

File
Status
Preview URL
Details

docs/add-ins/internet-headers.md
:warning:Warning

Details

docs/add-ins/toc.yml
:white_check_mark:Succeeded

docs/add-ins/internet-headers.md

[Warning] Missing attribute: title. Add a title string to show in search engine results.

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit d179fa1:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/add-ins/internet-headers.md
:white_check_mark:Succeeded

docs/add-ins/toc.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit 6fd64c3:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/add-ins/internet-headers.md
:white_check_mark:Succeeded

docs/add-ins/toc.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit 2c28ba7:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/add-ins/internet-headers.md
:white_check_mark:Succeeded

docs/add-ins/toc.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit d18d24a:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/add-ins/internet-headers.md
:white_check_mark:Succeeded

docs/add-ins/toc.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit 03d3c02:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/add-ins/internet-headers.md
:white_check_mark:Succeeded

docs/add-ins/toc.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit 2ccf6ca:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/add-ins/internet-headers.md
:white_check_mark:Succeeded

docs/add-ins/toc.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit e566648:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/add-ins/internet-headers.md
:white_check_mark:Succeeded

docs/add-ins/toc.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit 8c8d961:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/add-ins/internet-headers.md
:white_check_mark:Succeeded

docs/add-ins/toc.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit fc9c762:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/add-ins/internet-headers.md
:white_check_mark:Succeeded

docs/add-ins/toc.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit 45825a6:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/add-ins/internet-headers.md
:white_check_mark:Succeeded

docs/add-ins/toc.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit c99805f:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/add-ins/internet-headers.md
:white_check_mark:Succeeded

docs/add-ins/toc.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit 5fd8cfe:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/add-ins/internet-headers.md
:white_check_mark:Succeeded

docs/add-ins/toc.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit 08b0670:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/add-ins/internet-headers.md
:white_check_mark:Succeeded

docs/add-ins/toc.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit b446f39:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/add-ins/internet-headers.md
:white_check_mark:Succeeded

docs/add-ins/toc.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit f2789e4:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/add-ins/internet-headers.md
:white_check_mark:Succeeded

docs/add-ins/toc.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit 2bbf523:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/add-ins/internet-headers.md
:white_check_mark:Succeeded

docs/add-ins/toc.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit 8995d66:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/add-ins/internet-headers.md
:white_check_mark:Succeeded

docs/add-ins/toc.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


ENH: add remove method to scipy.spatial.cKDTree

Reference issue #12897
What does this implement/fix?
A simple remove method for cKDTree. This remove a given point (if it belongs to the tree) and update self.n.
If needed, leaves are collapsed into parent node (when parent_node.children <= leafsize) and self.size is updated.
For performance reasons, this method doesn't remove the point from self.data and it doesn't update maxes and mins but, if it is required, I can do it.
Additional information
The method accept only one point and doesn't perform any kind of rebalancing. I'm working about these.

Here you can see a very simple example with leafsize=1 and n=8. A point is removed, so two leaves are collapsed into parent.

Sorry, I wrote a piece of doc in the wrong way, now it should be ok.

Sorry guys, I don't get the error in "ARM - Test Group 1". It seems that ckdtree.size is not updated. Any guess?

Formatting of remove.cxx should be cleaned up to adhere to NumPy (snd SciPy) coding standards. It is a bit messy.
Also only one function should be exported into the global namespace.

The remove method of cKDTree is not threadsafe. It also renders the query methods no longer threadsafe.
It does not even help to retain the GIL for remove, as other threads might be doing a query on cKDTree in nogil context while it is being mutated. This will lead to race conditions or even a segfault. There is also the possibility of two concurrent calls to remove creating a race and possibly segfaulting.
You will therefore need to add a synchonization object to cKDTree that will:

Suspend a call to remove while query calls are running.
Suspend any query call while a call to remove is running.
Suspend a call to remove while another call to remove is running.
Suspend pickling of the cKDTree object while a call to remove is running.
Allow concurrent and reentrant query calls.
Suspend any access to the property cKDTree.tree while remove is running.
Prohibit any call to remove after the first access to cKDTree.tree.

(Possibly other things as well, but this is all I could think of now.)
A simple mutex is not sufficient as it would serialize query calls against each other. The synchronization problem is nightmarish, but not undoable.
Also we need test cases to make sure cKDTree remains threadsafe.
We cannot consider to merge this PR before the threadsafety issues are solved. It is absolutely mandatory that they be solved.

We cannot consider to merge this PR before the threadsafety issues are solved.

Shouldn't it be up to the user to not concurrently modify the tree. Or provide their own reader-writer lock if this is important to them.

If we were just tempering with Python objects I would concur, but here we are mutating the underlying C++ objects. We do synchronize access to C/C++/Fortran data elsewhere in SciPy, either with the GIL or some other means (e.g. threading.RLock).

That is why we have this:
https://github.com/scipy/scipy/blob/5f4c4d802e5a56708d86909af6e5685cd95e6e66/scipy/_lib/_threadsafety.py
(But here we need a different synchronization mechanism.)

I thought the reentrant locks were for fortran functions that have internal shared state and so independent function calls are not thread safe. Whereas in this case, the user has to consciously mutate state that they've shared between multiple threads.

I thought the reentrant locks were for fortran functions that have internal shared state and so independent function calls are not thread safe. Whereas in this case, the user has to consciously mutate state that they've shared between multiple threads.

When we do concurrent read-write to C++ structures that contain pointers, then all bets are off as to what might happen.

The remove method of cKDTree is not threadsafe. It also renders the query methods no longer threadsafe.
It does not even help to retain the GIL for remove, as other threads might be doing a query on cKDTree in nogil context while it is being mutated. This will lead to race conditions or even a segfault. There is also the possibility of two concurrent calls to remove creating a race and possibly segfaulting.
You will therefore need to add a synchonization object to cKDTree that will:

Suspend a call to remove while query calls are running.
Suspend any query call while a call to remove is running.
Suspend a call to remove while another call to remove is running.
Suspend pickling of the cKDTree object while a call to remove is running.
Allow concurrent and reentrant query calls.
Suspend any access to the property cKDTree.tree while remove is running.
Prohibit any call to remove after the first access to cKDTree.tree.

(Possibly other things as well, but this is all I could think of now.)
A simple mutex is not sufficient as it would serialize query calls against each other. The synchronization problem is nightmarish, but not undoable.
Also we need test cases to make sure cKDTree remains threadsafe.
We cannot consider to merge this PR before the threadsafety issues are solved. It is absolutely mandatory that they be solved.

Thanks for this precise analysis.
I suspect this will be hard for me to do it in a reasonable amount of time, but surely I will give it a shot.

When we do concurrent read-write on C++ structures that contain pointers, then all bets are off as to what might happen.

If you are querying the data structure concurrently with modifying it, then your results are non-deterministic; even with a reader-writer lock. So, either way I would say the user's code is broken and they need to introduce higher-level synchronization into their program.

It is better to be safe than sorry and synchronize the access. It is not like it will be difficult to do the appropriate synchronization, nor will it affect the performance of cKDTree.

Well locks aren't free. It's probably somewhere in the region of 10-100 ns overhead, so not huge, but still not free.
For me the question is: is okay for a user's broken multi-threaded code to segfault? And is it worth pessimising everyone's code to avoid that?
@rgommers is there any standard approach to multi-threading in SciPy?

For me the question is: is okay for a user's broken multi-threaded code to segfault? And is it worth pessimising everyone's code to avoid that?

@rgommers is there any standard approach to multi-threading in SciPy?

The tools in scipy/_lib/_threadsafety.py are used in multiple modules, that's probably as standard as it gets. I'm not sure that's exactly the same issue or not. E.g. here:
# ARPACK is not threadsafe or reentrant (SAVE variables), so we need a
# lock and a re-entering check.
_ARPACK_LOCK = ReentrancyLock("Nested calls to eigs/eighs not allowed:      "
                              "ARPACK is not re-entrant")

the user shouldn't have to know that the underlying Fortran code isn't reentrant; the user expects to be able to make calls to a function like eigs in multi-threaded/processing code.
In this case it's clear that a .remove call is modifying the data, so doing .remove plus a query in separate threads is arguably user error.
I haven't followed the details of the discussion here, hope the comment is somewhat helpful.

In this case it's clear that a .remove call is modifying the data, so doing .remove plus a query in separate threads is arguably user error.

So we should perhaps just document this in the docstring and state that the user has to provide the synchronization?

A notes section with a few warnings would be useful.

Don't query concurrently, the tree may be in an invalid state
Don't expect references to kdtree.tree from before the remove call to be up to date


Detect ccache and provide a default configuration

Summary:
Building native modules from source, may take a long time. Xcode already helps bring this down, by providing incremental builds, as long as the user doesn't delete their ios/build directory. But in some situations, i.e. when iterating the native code of an app or library or when the developer need to delete that ios/build directory, it's advantageous to use a compiler cache, such as ccache. This is already outlined in our "Speeding up your Build phase" guide.
But setting up an Xcode project to use Ccache with the correct configuration, isn't trivial in a way that doesn't require symlinking clang and clang++ or passing configuration via environment variables on every npm run ios invokation.
This PR takes it's inspiration from the existing guide on setting up Ccache for Xcode, but applies the build settings only if an installation of ccache is detected. It uses two shell scripts to wrap the call to ccache, which both injects a default CCACHE_CONFIGPATH environment variable (i.e. it won't override this if already provided, to allow for customisations on CI), pointing to a ccache.config which works well with React Native projects (it has the same values as the guide mentions).
For context, I posted about this change in the ios channel of the contributors Discord server, where I discussed it with @cipolleschi and @saadnajmi
Changelog:
[IOS] [ADDED] - Added better support for ccache, to speed up subsequent builds of native code. After installing ccache and running pod install, the Xcode project is injected with compiler and linker build settings pointing scripts that loads a default  Ccache configuration and invokes the ccache executable.
Test Plan:
I've tested this manually - would love some inspiration on how to automate this, if the reviewer thinks that's needed.
To test this locally:

Install Ccache and make sure the ccache executable is in your PATH (verify by running ccache --version)
Create a new template app instance and apply the changes of this PR to the node_modules/react-native package.
Run pod install in the ios directory.
Check the stats of Ccache (running ccache -s).
Run npm run ios or build the project from Xcode.
Check the Ccache stats again to verify ccache is intercepting compilation ("Cacheable calls" should ideally be 100%).
To check the speed gain:
a. Delete the ios/builds directory
b. Zero out the ccache stats (by running ccache -z)
c. Run pod install again (only needed if you ran the initial pod install with new architecture enabled RCT_NEW_ARCH_ENABLED=1).
d. Run npm run ios or build the project from Xcode.
e. This last step should be significantly faster and you should see "Hits" under "Local storage" in the ccache stats approach 100%.

cc @tido64 but he might be out till after new years


Remove version: Microsoft.Edge.Dev version 124.0.2478.10

[Automated] It returns code over 400 in all urls
Microsoft Reviewers: Open in CodeFlow

For moderators or/and anyone else

[!Caution]
As a result of several personal problems I've had over the last few months + my final exams coming up very soon, I'm currently unable to look at my GitHub notifications every day and maintain my executions correctly.
If you detect a certain problem on a certain pul request:

Either you can close the pull request directly without mentioning me.
Or you can send me a message on X (Twitter): @Exorcism0666

Thank you for your understanding, and I'm sorry if some pull requests can't be approved due to the situation I'm in.

This account is automated by Github Actions and the source code was created by CoolPlayLin. If you have any questions about any pull request, don't hesitate to ping Exorcism0666.

[!warning]
With the new version of Komac, numerous Pull Requests may contain bugs or unwanted elements. If such issues arise, please tag russellbanks so that these bugs can be addressed in future updates.

[!important]
Please carefully review these Pull Request before merging. If it is a Pull Request for removing incorrect content and the URLs are issue, free upon manual checking, please close this Pull Request directly. (It is best to inform Exorcism0666 of the closure.)

良い一日をお過ごしください！

URL: https://msedge.sf.dl.delivery.mp.microsoft.com/filestreamingservice/files/471ee1d1-4ef0-4b5d-bca9-af23023a00dd/MicrosoftEdgeDevEnterpriseX86.msi
Response Error: The remote server returned an error: (404) Not Found.
(Automated message - build 912)