Incorporate Lin-BASEX Method into PyAbel

Thomas Gerber of the Paul Scherrer Institut contacted us and offered to let his Lin-BASEX method be incorporated into PyAbel!
The method is already implemented in an iPython Notebook:
Download: https://www.psi.ch/sls/vuv/Station1_IntroEN/Lin_Basex0.7.zip
Reference: http://dx.doi.org/10.1063/1.4793404
PDF: http://www.nccr-must.ch/libraries.files/Gerber-2013.pdf
Amusing drawing of pig with a top hat: https://www.psi.ch/sls/vuv/pepico
I just started reading about this method this morning, but it seems very interesting and quite different from out other methods. Fundamentally, it expands the projection in terms of Legendre polynomials (or other polynomials optionally?), so I think that it assumes a "VMI-like" image. This limits the applicability of the algorithm somewhat, but provides the huge bonus of eliminating much of the noise from a VMI image.
In contrast to the Polar Onion Peeling method (https://github.com/PyAbel/PyAbel/issues/30), which I believe fits Legendre polynomials to the entire 2D image, the Lin-Basex method appears to take 1D projections of the original image at different angles and fits the Legendre polynomials to these. This serves to reduce the numerical complexity of the problem. To some degree, I think that this also makes the approximation that the angular distribution is slowly varying with angle. Presumably, the number of projections can be decreased to provide faster transforms at the cost of making the "slowly varying with angle" approximation stronger.

Thomas thanks for providing a new method to add to the PyAbel stable of Abel transforms.
Your ipython notebook code minus the matplotlib and notebook GUI statements should produce compact code. I have hacked on it for a little while, but it is not quite their yet.
I ran your notebook, with defaults, on the PyAbel example/data/O2-ANU1024.txt. The noise characteristics for the inverse transform, along the image center-line look to be very good (left image cf right)

The speed distribution cf hansenlaw reveals more noise, but this is probably due to the default parameter selection, and that I have misinterpreted what data is saved.

Early days.
Your Abel method looks neat, and it is great that it provides (comparative) anisotropy parameters.

From my perspective I am selfishly interested in the PES (photoelectron spectra) and PAD (photoelectron angular distributions). The notebook code appears to be geared to PAD.
Is the PES "Beta[0]"  or is there more involved to extract it?

In [22]:  After running the function "beta_solve" above, the array "Beta" contains all the data the evaluation can produce.
#Units are [counts/radius], were the radius of the Newton sphere is given in pixel.
#You may want to store Beta as a ".txt" file to use another software for visualization.

Use e.g.,np.savetxt('/Home/Your_path/Your_file_name.xls', Beta ,delimiter=',')
np.savetxt('/home/stg121/betaO2.txt',Beta)

@stggh, I'm guessing that the different Beta values are probably just the coefficients for each Legendre polynomial as a function of radius. I imagine that you are plotting Beta_2 or something, and that is why the noise looks so bad.
In general, a method like this that fits smooth functions to the PAD should provide very smooth transform (with no centerline noise!), with the cost that some artifacts may be introduced if the fitted functions are not a good description of the data. Legendre polynomials should be an excellent description of photoelectron datasets, but perhaps not flames or laser plumes.

Neat. Thanks!

I think that extracting the 1D angular integration (aka PES, but we should try to use general names in case non-spectroscopists use PyAbel) could be a bit more powerful than with the previous methods. Of course, we can simply generate the 2D array and integrate this. However, since we have the actual functions that we fit to the data, we should be able to generate an 1D angular integration with arbitrary sampling.

Lin-Basex was established with Velocity map images in mind.  Actually, the distribution according to Legendre polynomials follows from the intrinsic angular momentum mechanics describing the photo-electron or photo-ion process. In the paper we denote the isotropic distribution with beta0 and the second order distribution with beta2. In a single photon experiment there are only anisotropies up to second order. The interaction of 4 photons (Four wave mixing) yields unisotropies up to order 8, i.e beta0 to beta8.
Mathematically the Lin-Basex evaluation should be adoptable to more general applications. The only request is, that the "body" under investigation can be expanded into spherical harmonics. Up to now it has been shown that Lin-Basex works for real harmonic functions without phi-dependence, i.e. Y_l0. The more general case was not yet addressed. It is questionable whether it is of any benefit to expand a flame or a laser plume in terms of spherical functions. If however it is, lin-Basex would do its job.

@gerber-t, very well stated! (And welcome to GitHub!)

Actually, the distribution according to Legendre polynomials follows from the intrinsic angular momentum mechanics describing the photo-electron or photo-ion process.

Yes, the Legendre polynomials seems like the ideal way to describe most photoelectron distributions, since the physics is based on them in the first place.
To be clear, it is great to incorporate methods into PyAbel that are optimized for photoelectron/photoion spectroscopy. We should just make a point to describe the advantages/limitation of each method in general language, so that users from other fields can make educated decisions regarding the usage of each method.

The only request is, that the "body" under investigation can be expanded into spherical harmonics.

This is a good description to determine the applicability of the the Lin-Basex method.
Thanks again for helping us with this!

@DanHickstein and @gerber-t
My fork linbasex has "working" version of the Lin-Basex method, hacked from Thomas' ipython notebook code. It runs in the same way as the other Abel transform methods.
git clone https://github.com/stggh/PyAbel.git -b linbasex

will fetch you a copy.
Discussion points:

Name:
I called the method linbasex, since I am not a big fan of underscore characters. However, feel free to choose another name (lloking at you @DanHickstein)
Code:
I just hacked away at the ipython note book code to obtain a minimal running version.
There is a great deal of scope for restructuring, and separating the basis code (as is done for basex and the dasch algorithms). My apologies to @gerber-t, as at the moment its not the code you would hope for.
I have added some crude docs (for readthedocs), examples, `example_linbasex.py

@DanHickstein and @gerber-t
My fork linbasex has "working" version of the Lin-Basex method, hacked from Thomas' ipython notebook code. It runs in the same way as the other Abel transform methods.
git clone https://github.com/stggh/PyAbel.git -b linbasex

will fetch you a copy.
Discussion points:

Name:

I called the method linbasex, since I am not a big fan of underscore characters. However, feel free to choose another name (looking at you @DanHickstein).

Code:

I just hacked away at the ipython note book code to obtain a minimal running version.
There is a great deal of scope for restructuring, and separating the basis code (as is done for basex and the dasch algorithms). The linbasex code is a working starting point for improvement.
I have added some crude docs (for readthedocs), examples, example_linbasex.py and example_all_O2,py. Note, these are geared to my favourite O2- PES. I will put back @gerber-t's original example (if the image is not too big). I have written unit tests, but these are not fully operational.
Much of the remaining code excised from the original notebook may be placed elsewhere, eg. centering, stretching etc.  I believe the calculation provides more efficient evaluation of angular and radial integrals? @gerber-t or @DanHickstein I may need educating on these aspects.

Issues:
a. linbasex insists on square-odd-sized images. I believe, in the end "squareness" may not be a strict requirement (?).
b. PyAbel  <method>_transform(), by design, transforms the right-half (or top-right quadrant) of an image. basex handles this by duplicating the right-side into the left, transforming the full image, and returning the right-half.  The square image requirement of linbasex forces the use of quadrant analysis, throwing Q1, Q2, Q3 away, each time. This is inefficient.

@DanHickstein my preference would be to modify abel.Transform() to pass the full image to basex and linbasex_transform_full()?
c. The reconstructed image returned from `linbasex` generates a speed profile that is quite noisy. The output from `examples/example_all_O2.py` (`linbasex` cyan colour). Q: I am returning he correct fully reconstructed image? I have not played with the parameters.

Alone, example_linbasex.py the O2- spectrum does not look too bad:

Benchmark (Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz, running Anaconda Python). NB @gerber-t  this will be the worse case, pre-code improvements:

[etalon:/home/stg121]$ ipython3
Python 3.5.1 |Anaconda custom (64-bit)| (default, Dec  7 2015, 11:16:01)
Type "copyright", "credits" or "license" for more information.

IPython 4.1.2 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import abel

In [2]: abel.benchmark.AbelTiming()
Out[2]:
PyAbel benchmark run on x86_64

time in milliseconds

=========    Basis Abel implementations ==========

Implementation      n = 301           n = 501
----------------------------------------------------------
     basex_bs    6421.6           20292.4
onion_peeling_bs       6.1               7.2
three_point_bs      13.9              30.1
 two_point_bs       3.7               9.1

=========  Forward Abel implementations ==========

Implementation      n = 301           n = 501
----------------------------------------------------------
     direct_C      17.8              75.2
direct_Python      70.4             295.2
    hansenlaw       6.5              17.9

=========  Inverse Abel implementations ==========

Implementation      n = 301           n = 501
----------------------------------------------------------
        basex       2.4              20.7
     direct_C      18.2              77.7
direct_Python      70.7             297.3
    hansenlaw       6.8              16.3
     linbasex     162.8             476.9
 onion_bordas     244.9             789.3
onion_peeling       0.6               2.7
  three_point       0.6               2.5
    two_point       0.6               2.6

and the "standard plot" (linbasex is lurking in gray just below the basex curve):

The next steps will be code restructuring and efficiency improvements.

@gerber-t, here is the direct link to Steve's code: https://github.com/stggh/PyAbel/blob/linbasex/abel/linbasex.py
@stggh - Great work on this!
I take great joy that we have added so many new transform methods that your (once sensible) example_all_O2.py now looks delightfully ridiculous!  :)

The name linbasex is fine with me, though I worry that there may be a place named "Linba" where people are searching for physical affection. I'm actually not sure that "Lin-basex" is a great description of the algorithm. There are several method that fit basis functions to the data, and not all of them are called "BASEX". The most interesting part of this algorithm (to me) is that projections of the data are used in order to simplify the problem. I may have used a name like "polar_legendre". Or, we could just call it "gerber" after it's father :)

If you want to include @gerber-t's example image (I assume this is okay?), you can use something like ImageJ to re-size the image to something a small so that it doesn't take up much room in the repo.

a. Since the algorithm fits Legendre polynomials to the data, I would expect that it only makes sense to use it on datasets that are circular. I assure that data that lay outside of the largest circle are ignored by the algorithm? Basically, this means that the images should be square. Potentially, I suppose the fit could still take place with non-square datasets, some of the functions would just not be fit through their entire range. That doesn't seem like something most users would want to do. For now, I think that it's fine for the algorithm to insist on square images.
b. Regarding the quadrants and symmetry. I think that when Lin-basex uses only even Legendre polynomials, then it is enforcing both left/right and up/down symmetry. When it uses both even and odd Legendre polynomials, then I imaging that this is allowing the bottom of the image to be asymmetric with the top.
So, one could argue that, in the case of only-even Legendre polynomials, the algorithm is basically only using one quadrant anyways (since the result is averaged over all quadrants). I tend to like the idea to standardize all of the algorithms to just accept one quadrant at a time.
However, in this case, there might be some advantages to allowing Lin-basex to operate on the whole image. Potentially, if one wants to look at the coefficients of the basis functions, they can compare the coefficients from the odd functions with those from the even functions. If we only passed linbasex one quadrant, then there the coefficients for the odd functions would always be zero. But, I'm not sure if users would be interested in this functionality.
Are there other reasons that argue for passing linbasex the whole image? I don't think that averaging the image, sending a quadrant to linbasex_transform, and relying on linbasex_transform to re-assemble a full image is particularly inefficient (okay, it's not great, but probably not the bottleneck).
c. The issue with the speed distribution is interesting. I don't understand how the speed profile can look noisier than the other transform methods while the 2D transform looks like it has less noise. Can you send a large-format comparison of the 2D lin-basex transform with something like Hansenlaw? Are you converting the 2D transform to the speed distribution in the same manner?

Premature optimization is the root of all evil ;). Let's focus on getting the code readable, well commented, well documented, and short. That should make the path to optimization clear.

@all
I am overwhelmed at what speed and what performance you work on linBasex.
I have absolutely no concerns in renaming. I chose Lin- Basex to indicate "linear" Basex in contrast to polar Basex (p-Basex etc.). At that time I was also thinking at 1D-Basex. I am fine with all of your proposition.
There is no need for the test image (rather big) to post on your repo. The pictures you have and the comparison of different methods working on them is very adequate.
Indeed, if uneven polynomials are invoked the evaluation should be aware of top - down along the cylinder axis. However even then the two halfs about the symmetry axis could be added together.
In case of even polynomials all quadrants are equivalent.
I thought at VMI experimentalists. Usually it is already difficult to determine exactly the VMI center. Some of my code is dedicated to that and to checking symmetry. The projections at diffferent angles over the whole image give experimentalists a good feel about the correctness of their raw data. A charged particle VMI image reflect Newton spheres generated in a perfect instrument. So, for evaluating speed plots and anisotropies the raw-data have to be "round". If not they should be made "round" correcting  astigmatism, and possibly other distortions introduced by the ion/electron TOF apparatus (ion optics).
For the evaluation of VMI it is essential that anisotropies are not just reflecting bad optics.
The noise in speed plots is intriguing, The lstsq solver allows, according to linear algebra, for negative "Newton spheres" what is physically not satisfying. At the time of the paper I was trying linear programs (e.g. Chebyshev) to force only positive solutions. Unfortunately the linear equation system resulting in this approach needs conditioning again (what a that time, I was not aware of). So I got only unusable solutions. But maybe it is worth resuming on that.
Actually on does not make a mathematical crime if one is smoothing over the speed plot as tentatively indicated in the last few cells of my code. The decomposition into "Newton spheres" and addition of "newton spheres" is mathematically sound. Averaging over adjacent spheres will finally produce positive contributions as there are no negative "counts" in the raw data. That is obvious from the 2D reconstruction.
I guess that other methods use some form of regularization  in the determination of the speed plot, as for instance the one built into lstsq cond using an SVD decomposition. One could adjust cond to somewhat higher than 0.005 in my code without blurring out resolution. However, I think it is important that the user has the choice to fix things. A conditioning, one for all, may hide details to the experimentalists in special circumstances. So let him choose.

@DanHickstein @gerber-t
A good discussion, very informative, particularly, about the whole image symmetry and circularization. The latter is a key aspect which is to be addressed in future pre-processing PyAbel code.
For the moment, I will tidy up the linbasex code, and perform some further comparative studies.
Specific points:
0) I also like that there are more Abel methods to add to PyAbel, each new algorithm providing  improvements to the coding of the methods.

"linba" first time I have heard of this word/town. I do think that linbasex is a little ordinary, and so we should look to select a more representative name.
3a) Agreed. Image symmetry considerations are essential.
3b) Single quadrant analysis by linbasex makes logical sense.
3c) There is a hint of horizontal centre-line noise, which would be enhanced by the negative anisotropy parameter of O2-.
Yes, optimization can wait.
@gerber-t ) The implementation of linbasex has been greatly aided by your nice ipython notebook code.  Negative values, although, not physical reflect the uncertainties of the measurement. The original basex.exe code made the mistake of forcing only positive image values, resulting in a truncation of spectral intensities near the image center, with an "apparent" reduction of transform noise. I think @DanHickstein is correct, we should ensure your code is clean and readable, and then look to the physics/math if there remain any issues.  It is great that you provide such detailed insight.

One more remark about linbasex.
The method I propose is mostly aimed at VMI in Photo-electron/Photo-ion experiments.
A physicist is interested in what ions/electrons are produced at what energy and in what distributions.
Most inversion methods provide a 2D cross section through the reconstructed original 3D distributions. Beta's are then obtained by fitting Legendre polynomials to circular traces in the 2D cross section.
My approach delivers the beta's relevant to the physicist/chemist directly. So, as stated in my code, the array containing the beta's after solving the linear equation system contains all information one can get from an experiment.
The slices can the be calculated using the beta's for calculating the corresponding 3D Newton spheres from which a 2D section is shown in a plot. (You can "turn" around those spheres an depict any other slice at any other angle). However, the "slices" are nice for visualization but are not really suited to derive physical facts. Those are contained already in the beta's.
Thus, one should make available the beta's as output. The slices are nice to have for visualization and  for inspiration. (It would be a pity if someone approximates circular traces in the slices obtained with linbasex, having the beta's already at hand in a numpy array.)

Thus, one should make available the beta's as output.

This is a important point! Currently our method for calculating the anisotropy parameters only calculates the betas at one radius (well, averaged over a radial range) and it's reasonably slow. Getting accurate betas directly from the transform method – and getting all of the betas at every radial position! – will be appealing for many users. I would suggest the following interface:
linbasex_transform has a keyword argument return_betas=False which determines if the function returns IM_transform or IM_transform, betas. The keyword argument should probably default to False in order to keep a similar interface for all of the individual transform methods.
Most users will access the method through abel.Transform(data_2d, method='linbasex'), in which case, abel.Transform will call linbasex_transform with return_betas=True and store the betas in the self.anisotropy_parameters (or self.betas, or whatever we want to call it...).

@gerber-t be assured that we will not reduce the capabilities of your original code.
@DanHickstein your idea (as usual) is good.

only calculates the betas at one radius (well, averaged over a radial range) and it's reasonably slow

Not quite. It fits a 2nd order Legendre polynomial to the integrated radial intensity vs angle. Multiple radial ranges may be specified in a tuple. See examples/example_O2_PES_PAD.py.
def radial_integration(IM, radial_ranges=None):
             :
   radial_ranges : list of tuples
        integration ranges
        ``[(r0, r1), (r2, r3), ...]``
        Evaluate the intensity vs angle
        for the radial ranges ``r0_r1``, ``r2_r3``, etc.

[etalon:.+PyAbel/examples]$ python3 example_O2_PES_PAD.py
even pixel width image, make it odd and re-adjust image center
Calculating speed distribution:
Calculating angular distribution:
radial-range      anisotropy parameter (beta)
     93-111        +0.50+-0.29
    145-162        +0.08+-0.10
    255-280        -0.45+-0.03
    330-350        -0.82+-0.02
    350-370        -0.91+-0.01
    370-390        -0.93+-0.01
    390-410        -0.92+-0.01
    410-430        -0.88+-0.01

It has the advantage that the evaluated anisotropy parameter is accurate (although, this not necessarily reflected by the fit error estimate, which tends to be over estimated).
I have slowed up a little on the code changes for linbasex, to better understand the code, and  because of other diversions (work). However, I will put everything together during the weekend.

@stggh, I did not mean to criticize your fine anisotropy calculation :)
But, if you try to calculate the anisotropy for every radial pixel, then the results are noisy, and the calculation is reasonably slow, correct? I suspect that many people will want the averaging that your method provides, but we should certainly make the betas from the linebasex easily accessible.

No worries Dan, no offence taken.
Having looked at this I think that @gerber-t means that we should return the calculated Beta array.
The latest version (still to be tidied up) does this, with the parameter return_Beta, only used with linbasex_transform() wrapper, that is called by abel.Transform(method='linbasex').
The core function linbasex_transform_full() returns both the reconstructed image and Beta array.
There is a separate function to perform the integration (over a range of Newton spheres) int_beta().
Now, I must admit my lack of understanding on how the Beta array provides the anisotropy parameter (hence, the "I may need educating on these aspects" statement that I made above).
When I run the ipython notebook with the O2- VMI:
Region, Integrated Beta
|from| |to|  | β0 | β2  |  ...
  390   400 [ 0.13 -2.93]

The anisotropy parameter is β = -0.92(1) from abel.vmi.radial_integration() as listed above
Sorry to be the dummy here ...

Still working on linbasex.py and I will over the long weekend, in Oz.
I have separated the basis calculation, which is now stored in the same way as the other basis methods basex,  and dasch onion_peeling, two_point, three_point.  dasch.get_bs_cached now a general abel.tools.basis.get_bs_cached.  basex.get_bs_basex_cached() could also use this function (with modification for the multiple bases).
return_Beta will not yet work with abel.Transform(IM, method='linbasex', transform_options=dict(return_Beta=True)), because I have not thought of a smart way to handle this special case in abel/transform.py.
Operational, but not yet fully tested.

I guess there is some confusion about what I call beta's.
Imaging an PES experiment which produces electrons that are distributed on the surface of a sphere.
This sphere can be described by spherical functions. If all electrons have the same energy we expect them on a (Newton) sphere with radius i. This radius is projected to the CCD. The distribution on the CCD has (if optics are approriate) the same radius i. Now lets assume that the distribution on the Newton sphere has some anisotropy. We can describe the distribution on this sphere by spherical functions Ynm. Lets say xY00 + yY20. The 1D projection of those spheres produces just x Pi0(k) +yPi2(k) where Pi denotes Legendre Polynoms scaled to the interval i and k is the argument (pixel).
For one projection Lin Basex now solves for the parameters x and y.
If we look at another projection turned by an angle, the Basis Pi0 and Pi2 has to be modified because the projection of e.g., Y20 turned by an angle yields another function. It was shown that this function for e.g., P2 is just P2(a)Pi2(k) where a is the turning angle. Solving the equations for the 1D projection at angle (a) with this modified basis yields the same x and y parameters as before.
What I call beta0 is just the contribution of Y00's to the 3D electron distribution
beta2 the contribution of Y20's etc.
Or in the above example:
Beta0[i] = x --> what is just the "intensity" or the numbers of electrons with the same energy
Beta0[k] versus k --> speed plot.
Beta2[i]= y  --> How many electrons are distributed according to Y20. (The integral over Y20 is = zero)
Beta2[k] versus k --> anisotropy of each Newton sphere
etc.
So linbasex aims at the determination of contributions in terms of spherical functions calculating the weight of each Yl0. If we reconstruct the 3D object by adding all the Yl0 contributions we get the inverse Laplace transform of the image on the CCD from which we can derive "Slices".
In principle Dribinsky could proceed the same way. However his Basis set is not simply connected to the original "physical meaningful functions" the 3D object is made of. But a connection could be established by some algebra.

@gerber-t an excellent description, thank you!  I will add your comment to the docs.
My excuse ;-) is that I was confused as to the nature of the Beta parameters because the notebook analysis of the O2- VMI has beta2 extending beyond -1. No doubt this means a poor choice of analysis parameters.

We may examine these aspects once the code that I have hacked from your notebook is at an acceptable (for a PR) standard.
Thanks, again, you are a very helpful.

Just out of interest, I note that when playing with the jupyter notebook program, if threshold = 0 the anisotropy parameter vs radius looks just like the (previously) odd, original basex.exe code, calculation, [comment here] (https://github.com/PyAbel/PyAbel/issues/60#issuecomment-169220274), that I could make no sense of. Now I know, it is just a matter of throwing away some of the weaker Newton spheres, to give sensible behaviour (the plot above)

linbasex.py is still work in progress. The basic PyAbel functionality is present e.g.
LIM = abel.Transform(IM, method='linbasex', center='convolution',
                     center_options=dict(square=True),
                     transform_options=dict(an=[0, 90]))

produces all the attributes (LIM.transform, .linbasex.angular_integration, .linebasex.anisotropy_parameter) for the figure below, full code here

However, there are bugs, as it crashes if e.g. an=[0, 45, 90] (no doubt due to some short cuts on shapes). Also, I have not yet captured all the nuances of the parameters that are available.
Out of interest, if you are able to run this example, the reconstructed (inverse Abel transformed) images show detailed weak radial structure, which map to the 'pseudo' noise of the speed distribution.
Changes (from memory):

linbasex.py follows the template of the other transforms. abel.linbasex.linbasex_transform() will transform the Q0 quadrant. Unfortunately, in the same way as the other full-image algorithm basex, by contructing a full image, which is then transformed. There is also the whole image transform linbasex_transform_full(), which actually does all the work.

Basis sets are generated _bs_linbasex() and stored for re-use, abel.tools.get_bs_cached(method ...) common to the Dasch methods. The basis is saved as linbasex_basis_cols_rows_proj_pol.npy. This may not be unique enough. At the moment not a big deal, as this feature can be turned off, as for the other methods, to generate the basis each time.

As @gerber-t points out, linbasex trivially provides many physical parameters of interest, speed distribution, anisotropy parameter (others that I do not fully comprehend). At the moment inv_Dat (reconstructed image), Beta (array), and QLz (array) are returned, as a tuple, if return_Beta=True, otherwise, just the image (for compatibility with what the other methods return). abel.Transform() captures these parameters as attributes .transform, .angular_integration, .anisotropy_parameter, .projection (QLz).

Note, in abel.Transform() if the symmetry_axis parameter is not set the full image form linbasex.linbasex_transform_full() is used. This neater, and saves extracting tuples if return_Beta is set.

The convolution centering code, from the jupyter notebook, has been moved to abel.tools.center.center_by_convolution(), but is available in the usual way by the one function does all within `abel.Transform(...center='convolution'), as given above.  The stretching example code, could also be added, but might be better just in the docs.

As a comparative test, I have attempted to re-code the jupyter notebook using PyAbel calls. This is not finished.

Basically, the usual PyAbel calls work (including benchmark), but there may (are) bugs, and I have not yet fully understood the importance of each of @gerber-t's parameters. I will query later, if I get stuck.

I am still overwhelmed at what frantic speed and success you assess implementation of linbasex. Thank you very much. (My time budget for linbasex is at the moment a bit limited. I have some urgent other projects).
Indeed determination of anisotropy parameters is an issue.
I try to explain what I understood about it (Sorry, if I am breaking in open doors, what is easily possible with you, having a great routine in the subject ).
Physics are clear: The distribution of counts on a Newtonsphere with radius i are
counts on  i = Intensity(i) x(1 + Beta2,i. 1)
Beta[2,i] should be limited in the range [-1,2]. (Finite limits also for beta[4],... etc.)
An Abel transform provides the number and distributions of counts on a sphere. Retrieving beta[2], beta[4]... thus involves according to 1) always a division by Intensity(i).
Lets now assume the transform provides a "sphere k" due to noise with total numbers of counts = I(k) =0 (negative counts due to noise are possible). Beta[2,k] is not necessarily = 0. In this case we get an infinite beta, what is nonsense. The same may happen for all very week spheres.
Linbasex calculates all Beta's for each radius. As long as we deal with # counts this makes some physical sense as a plot versus angle shows, how many counts (including noise) have to be subtracted or added at a specific angle to the isotropic intensity (including noise). Division by this intensity produces  a mathematical result irrespective of physical constraints. A better choice to reflect physics is just to evaluate betas for those spheres that are clearly above noise. Moreover it may be appropriate to average intensities and betas over a range of some adjacent radii to smooth out noise in adjacent spheres. Linbasex leaves it to interpretation whether different adjacent spheres reflect some underlying physics or just reflecting shortcomings in apparative resolution. Though mathematically always correct one has to judge what "smoothing" means for the measurement.
If beta's are retrieved from sliced Abel inverted functions. A fit of the function 3 cos^2(phi) - 1  (second order case) is fitted to a circular trace of the slice with radius r and width dr. The same problem arises as described above: Eventually one divides by an intensity. We obtain random results for beta if the common intensity is close to noise intensity. Distrust is indicated when the resulting betas fall without the allowed limits.
Averaging is another issue. The function 3 cos^2(phi) - 1 is just good on one radius (one Newton sphere). If we evaluate a trace over a finite range of dr (what is usually done) we assume that all counts in this radius range are concentrated on one specific radius what may (physically) not be the case. The error inferred in this procedure stems from the fact that anisotropy is distributed according to a function Pn(cos(phi) witch is a Legendre polynomial of the variable cos(phi). However evaluation from slices  is done with functions Pn(cos(phi) with variable phi. In that variable Pn(cos(phi) is not a Legendre polynomial. Still it describes distribution on a specific infinitesimal narrow circle.

Great work @stggh, and thanks for all of the help @gerber-t!
I am still a bit confused: If the betas are the coefficients for the functions, then shouldn't it be necessary to sum all of the betas in order to get the total "intensity versus radius" (speed distribution)?

@DanHickstein
No, there is a Beta0[i] (isotropic contribution) for every Newton sphere with radius i. The same for beta2[i] anisotropic contribution (second order) to the same sphere i, etc.
Beta0[i] versus i is the speed plot.
Beta[0] is the number of counts having "speed" i. Whether it makes sense to have a Newton sphere for every pixel i is depending on experiment. Sometimes it my be better to bin several pixel to one sphere. But you will never add (i.e., bin) all together.

Okay, makes sense! Thanks for clarifying :)

@gerber-t, @DanHickstein  here is a bug that is present in the jupyter note book code. The size of the reconstructed image Slice is larger than the input image (I added LIM=Slice(...) and the shape statements). This is for the VMI_art1 image (but the shape change occurs for all images):

Just thought I would mention it in case you are able find the code issue before I do.
Possibly, the reshape?
    #arrange solutions into subarrays for each β.
    Beta = Sol[0].reshape((pol, len(Sol[0])//pol)

Since lin-basex has the coefficients for the functions, I think that it basically has an analytical description of the reconstructed image, and it could, in theory, produce a reconstruction with any arbitrary pixel dimensions. But yes, it would make the most sense to return an image of the same size.
And this "off-by-one" situation brings up the usual and important question of:
Where does lin-basex assume the center to be? Within a pixel (odd width images should be used) or between two pixels (even sized images should be used).

@stggh  Really great work. I am amazed with what virtuosity you scrutinize my code and bring it to speak.
True, the reconstruction images I first intended to be cast on any shape. That's possible as linbasex allows reconstruction analytically as @DanHickstein observed. Of course the same shape would be appropriate, though scalability may be useful as "viewing glass" for tiny details. (In view of typically coarse experimental images there are no details. That is only a hope and needs not be implemented).
The center is a crucial question. With limited coding skills I wanted to avoid non integer radii (how to use as index etc.) and therefore I choose a centre 0 and the radius to be i pixels. This is very arbitrary. Any other choice should work, too.
I am so happy that my idea of another transform method is in so good hands with you.

@gerber-t I realise that my comment may have appeared critical of your code. That was definitely not my intention. I was comparing the output of linbasex with `Lin_Basex0.7 and found the shape issue. Thanks, to your expert insight these issues are easier to address.
The returned PyAbel image shape started my involvement with PyAbel #34! This lead @DanHickstein and @rth to consider what the image center represents #39. All PyAbel methods (except onion_bordas) have the image center mid-pixel, which means an odd-column width image. As a consequence PyAbel will always return an odd-width image, which may differ from the input shape.
You are right

The center is a crucial question

Our "solution", the same as yours, is to assign a radial (float) grid, with the image center set mid-pixel, at r=0.
In linbasex (and Lin_Basex) I think the even image size arises from the doubling of array sizes,
    #Concatenate vectors to one matrix of bases
    Bpol = np.concatenate((B), axis=2)
    Basis = np.concatenate((Bpol), axis=0)

My question is whether Basis may/should be odd on all axes to provide the same shape as in odd-size input image?

@stggh
You point out some inconsistencies in my code. That is no critisism but an important remark. (Moreover I had no problem even with critisism, ... that is scientific discourse.  I myself consider my code as pedestrian as I was a newby in Python at that time).
Now, to solve the equations it is important that the column height of the projection arrays is exactly  the same as the column height of the Basis (else lstsq would not work). However the row length can be chosen at will (or according to physical/experimental considerations). The fact that in my code the row length of the basis is too long (pol X span+1) makes most likely no sense. (I forgot why it comes a long like this).
The contrary is more important: the row length could be shorter. If we knew, that a VMI were composed only of a small number of Newton spheres (either due to primary knowledge (i.e. "selection rules", energy levels etc.) or by inspection of a primarily evaluated VMI), we could be tented to use only this set of base vectors being projections of "present" Newton spheres. The row of Basis would then only contain those basis vectors having the corresponding radii.
In an earlier version, I had a code were all basis vectors were clipped out of the full basis except those with the radii corresponding to the present spheres (e.g. 8 basis vectors  in VMI_art1 with corresponding radius). Using an array [0,0,..0,r1,0,...0,r2,0,....]) indicating the "counting Newton spheres" with radius r1,r2, .. as index for basis vectors and omitting all "0". All noise is then averaged into those spheres, providing a very good performance.
In this sense also the centre problem can be addressed. In the centre we have contributions from all spheres projected to the VMI. Information about those spheres (with radius r > r_min >0) we get from out of centre counts (either by just omitting the basis vectors for the spheres with r < r_min, or more involved by omitting the data in the centre spot and shorting the basis vectors correspondingly).
Of physical interest are, when present, only the zero kinetic energy (ZEKE) contribution projected to the centre spot. The amount of ZEKE's could also be evaluated by the counts found in a few centre pixels  minus the counts from all other spheres falling into that spot.
So, for evaluation we could clip out the centre basis vectors, possibly making the lstsq solver more stable. After we have determined all other spheres with r > r_min we could do the calculation above to produce the ZEKE counts.
(That what I guessed and what I tried sometimes, but I did not get a clear answer nor a clear improvement. In any case it would be wise to have only one basis vector with radius 1, as one pixel has no "unisotropy" and needs no basis for P0,P2, etc.)  In my code there is still this clipping possibility --> clip under "Calculation of Base vectors").

(Edit: finally got this to upload)
@DanHickstein @gerber-t
I found the an=[0, 45, 90, 135] bug in the linbasex code, I stupidly flipped the sizes for pol and proj.
Running more angles demonstrates an improvement in appearance, cf an=[0, 90]:

Note, I am not sure the maximum angle allowed, does your example case 135 imply up to 180?
@gerber-t your description is exciting. Python loves boolean lists. I will test turning off some of the Newton spheres tomorrow.

@stggh
Yes angles can go to 360 but of course do not tell something new beyond 180.
(Note that the P2 contributions get zero at the "magic" angle 54.7. A VMI containing contributions only up to second order will look isotropic in the projection at the magic angle. That is as funny as magic).
If the number of projection  is increased the appearance should improve.  BASEX, using 2D information corresponds to Linbasex using about 500 projections. The advantage of linbasex is, that not so many projections are needed (typicall proj ~ pol). So, linbasex evaluation using a mathematically appropriate and correct basis set should eventually be much faster than BASEX.
One more remark about the number of basis vectors in other words.
Evaluating with the full basis (P0,P2, ... vectors for every Pixel) the method is synonym to a basis expanded Abel transform, i.e., it is a genuine Abel transform.
If our 3D object is "sparse" (PESler's know, i.e., contains a sparse set of Newton spheres) a sparse basis can be used, too. In this case one must have primary information about what "sparsity" is appropriate.
That means that an Abel transform can be simplified if primary information about the object is available. That is not the case with the other methods.

Wow! Wonderful discussion @stggh and @gerber-t!
I can see that there is a huge advantage of using the Legendre polynomials to reduce the "degrees of freedom" in the angular dimension. The idea about using fewer newton spheres in the radial dimension is interesting, but what is the advantage over just using one sphere for each radial pixel? Just to speed up the simulation? Or are you thinking that "thicker" newton spheres could be used in order to improve the signal-to-noise of the reconstruction.
The idea that, in general, you are better off fitting a representative basis set to your projection and using that for the transform is a good one. If the user is willing to make assumptions about the shape of their object, then huge improvements in signal-to-noise should be achievable.

If your PES experiment (I always think in terms of PES/PIMS and less in ABEL) is accurate an produces sharp Newton spheres within one pixel, then you are probably better off  by reducing the basis to those contribution only you expect to be there. Counts are then distributed on a few spheres only. Absolute noise increases in each sphere with sqrt(counts) but relative noise decreases with 1/sqrt(counts).
If your experiment is not accurate enough, you do better with "thick" Newton spheres binning your VMI to 2 or even 4 pixel. That is accommodated with the parameter inc (for increment).

@DanHickstein @gerber-t:   My progress on the linbasex implementation has slowed due to other commitments, although, I did work on the code over the weekend. linbasex is fully functional (in respect to the other PyAbel methods) and just about ready for a PR. All the variables used in the original jupyter notebook are mapped, including rcond, clip, and norm_range (also called clip in the notebook).  clip forced the mapping and return of the associated radial grid, and an additional label _{clip} for the basis file.
My excitement about limiting radial extend of Newton spheres has been tempered by the realisation that I do not fully understand the notebook code :-(. In the basis code, what do the 'ascii-art' beautiful COS and TRI arrays represent?  (see def _bs_linbasex(cols, an=[0, 90], un=[0, 2], inc=1, clip=0, **kwargs): at the bottom of linbasex.py). Here is a trace print for the small cols=5:
cols = 5
n = 3
In [15]: COS
Out[15]:
array([[-0. , -0. , -1. ],
       [-0. , -1. , -0.5],
       [-1. , -0. , -0. ],
       [ 0. ,  1. ,  0.5],
       [ 0. ,  0. ,  1. ]])

In [16]: TRI
Out[16]:
array([[ 0.,  0.,  1.],
       [ 0.,  1.,  1.],
       [ 1.,  1.,  1.],
       [ 0.,  1.,  1.],
       [ 0.,  0.,  1.]])

For the subsequent code:
    # inc: use only each inc other vector. Keep the base vector with full span
    COS = COS[:, ::-inc]
    TRI = TRI[:, ::-inc]

    COS = COS[:, ::-1]  # rearrange base vectors again in ascending order
    TRI = TRI[:, ::-1]

    if clip > 0:
        # clip first vectors (smallest Newton spheres) to avoid singularities
        COS=COS[:,clip:]
        # It is difficult to trace the effect on the SVD solver used below.
        TRI=TRI[:,clip:]  # usually no clipping works fine.

    # Calculate base vectors for each projection and each order.
    B = np.zeros((pol, proj, len(COS[:, 0]), len(COS[0, :])))

    Norm = np.sum(_bas(0, 1, COS, TRI), axis=0)  # calculate normalization
    cos_an = np.cos(np.radians(an))  # angles in radians

    for p in range(pol):
        for u in range(proj):
            B[p, u] = _bas(un[p], cos_an[u], COS, TRI)/Norm

    # concatenate vectors to one matrix of bases
    Bpol = np.concatenate((B), axis=2)
    Basis = np.concatenate((Bpol), axis=0)

The concatenate statements imply to me that it should be possible to transform only 1/2-side of an image, which would then match the other PyAbel methods. In addition, the inc and 'clip` point to trivial methods to radially limit the calculation of the basis. However, I did not achieve instant coding joy, and was diverted by other code (parameters etc).
Issues:

linbasex normalizes Beta[0] (speed distribution), and hence fails the "required" zero intensity image unit test listed in CONTRIBUTING.rst`:

Given an array of 0 elements, the reconstruction should also be a 0 array.

Q: Is there an absolute normalization for the Lin-Basex method?

Unit testing is proving a pain, related to the slight different shape from Beta[0] and normalization.

1/2 image processing, that I mentioned above. This would eliminate the need for wrapper functions in linbasex.py.

Currently Beta_convol is returned (convolved [0] = speed, [1] = anisotropy parameter), to match the calculation available from PyAbel. Is this sufficient, or should the raw Beta array be available?

@stggh @DanHickstein ,
In my code I use "Index", and "Tri" to define the base vectors. The base vectors are the projection of a spherical function Y_n0 with radius i to a 1 D function on a support with 2i length. The 1D projection is just a Legendre polynomial of order n calculated at i points. In the array of base vectors the column i contains the base vector of length 2i (symmetrical about i=0). We thus get a set of bi-triangular arrays, for each Legendre polynomial one array.
For the calculation I just filled a triangular matrix with row indices. The highest index in column i is also i.  Division of the all entries in a column by i provides the values k/i where k is the row index in column i (k=0 in the bottom row). These entries are equal to the cosines y/r in a circle of radius r, or in other words the arguments for calculating the Legendre polynomials P_n(cos(phi) = P_n(y/r) = P_n(k/i).
That is "straightforward" coding. I have no doubt that there are more elegant procedures.
"Concatenate" just makes from this triangular arrays (which is the array P_n(0) ...P_n(i/i = 1) a bi-triangular array ( P_n(-1) ...P_n( 1)).
Now, you are right that, the "concatenate" just makes a symmetrical base array spanning the full image. Half of it would do as it is symmetrical. (Two halves could be added, thereby reducing the number of finally to solve linear equations from 2N to N equations. A minor speed gain).
However, for making the projection of the measured VMI at different angles you need primarily the full image. Having made the projection it is up to the user to solve for  half length projections (that are symmetrical about 0) with the half length basis vectors  (symmetrical about 0). To what avail?
Indeed, in this sense linbasex is a bit different from the other inversion methods that work directly on the measured VMI whereas linbasex works with 1D projections of the rotated VMI that have to be established with the full image, though finally they could be truncated to half length due to symmetry.

Beta0 is normalized with a max norm. That is somewhat arbitrary. For understanding the intrinsic inversion method it is maybe better to think in un-normalized "counts". Calculation with "counts" produces 0 counts for a "0" image.

Normalization only concerns the definition of  "intensity" "counts",or whatever. Number of counts is the primary input. How you normalize is up to the user. (e.g., intensity =1 for the maximum number of counts in sphere i).

See above.  I guess you need a special treatment for linbasex as it starts from projections at different angles requesting the full image. The method does not work with projections of half the image turned to different angles. But what concerns programming I am of little help.

Return of "speed" and "Anisotropy" is fine. However,  linbasex provides more info. In experiments with a dipole interaction you come out with speed and anisotropy beta, nothing else. In nonlinear optics with 4 dipole interactions there are "anisotropies" of order 2,4,6,8 where anisotropy of order 2 ( my beta2) is the conventionally called anisotropy parameter "beta". A linbasex evaluation using Polynomials of higher order (pol >2, even polynomials) provides the higher order anisotropies beta4, beta6 and beta8. These may be relevant for certain experiments.

I am for holiday this week and the following weeks I will be burried with an other project. I will only comment sporadically, sorry.

Thanks, @gerber-t , as usual very instructive. Enjoy your holiday.

@DanHickstein I followed your advice and have a new branch linbasex-rebased that was constructed from the master, with the linbasex files (from my memory):

abel/linbasex.py (core method from @gerber-t's jupyter notebook),
abel/transform.py (linbasex calls. Note, full image is transformed unless symmetry_axis selected),
abel/benchmark.py (linbasex included),
abel/tools/basis.py (common basis read/write, moved from dasch file),
examples/example_linbasex.py (O2-),
examples/example_linbasex_hansenlaw.py (shown here),
examples/example_all_O2.py (updated),
doc/transform_methods/linbasex.rst,
doc/transform_methods.rst,
README.rst,
CHANGELOG.rst,
abel/tests/text_linbasex.py,
abel/tests/test_dasch_methods.py (basis function re-location).

It turns out the VMI-art1 is only 96k when bzip2 compressed, and so I have added it to the example/data/ directory, and made example/example_linbasex_hansenlaw.py. The output looks pretty good.

@gerber-t  it would be interesting to know the actual anisotropy values used to construct the test-sample image VMI-art1 (not sure whether this information is available in your paper - it's late in Oz).
The linbasex method should be "out there", so I think the code is ready for a PR.

Great work @stggh!! Another massive and wonderful effort!
I agree that if we have something that is working, we should incorporate it as quickly as possible. Go ahead and post the PR and I will look through it and see if there are any typos. Then we can merge it!
(Looking at your rightmost panel reminds me that we should include a function that allows the anisotropy parameter to be easily extracted as a function of the radial coordinate. See upcoming issue..)

@stggh
You are moving very fast. Excellent!
I did not expect that there are so many details to cope with. But you master them.
In my jupyter code there is a cell following cell 16 where the initial entries for the VMI_art1 test pictures are given. For quick reference I paste it here.
Data for test VMI: VMI_art1
radius|beta0|beta2|beta4
38  1.20 -0.40  0.00
70  1.50 -1.00  0.50
90  1.50  1.00  0.40
134 2.00  1.00  0.40
138 1.80  0.50  0.00
143 1.00  0.50  0.25
196 2.00  1.00 -0.50
230 2.00  1.00 -0.50
In your text evaluation the intensities beta0 are normed to 1 (ok), in the data above to 2 (arbitrary). The rightmost panel should be labelled Beta2 (normed the same as in the data.). In VMI_art1 there are no Beta1 anisotropies. Moreover the test example exhibits very well how difficult it is to unveil the real anisotropy parameters. The statistics are against us. In my VMI_art1 there are in the order of 200 000 counts in one Newton sphere. It seems that this is not sufficient for retrieving anisotropy parameters with an accuracy of a few percent. In most papers one should think that the stated parameters other than beta0 (speed, that is usually quite accurate) are within a 20 %.
The issue of quarter, half and full image is intriguing. I guess that in many inversion methods the quarter image evaluation was chosen to deal with smaller arrays saving RAM and calculating time.
However, in the case of evaluating quarter images you intrinsically assume "even" symmetry, or in the language of linbasex you assume that your Newton spheres are given by even Legendre polynomials P0,P2,P4,... (what is the case in VMI_art1). But there are applications where uneven polynomials play a role. (E.g. chirality. At the moment I am not aware whether BASEX allows uneven symmetry). In such cases uneven symmetries occur that are only contained in the half image, halved along the symmetry axis. In this cases you should work with a least half the image.  I consider the gain of a factor two using only half of the image not so tremendous that you could not work as well with the full image, especially as in linbasex you work only with 1 D projections greatly reducing the numerical effort.
As mentioned before, I am very busy the next time. So unfortunately I will hardly have chance to play with and test your code, as I should, honoring your work. Sorry.

@gerber-t, thank you for the interesting discussion of the anisotropy parameters!
Let me explain the "full-image/quarter-image" situation a little bit better. The idea is not to save RAM or improve processing speed, but to allow the user to select the symmetry of the problem and provide versatility for averaging. We are breaking up the image into 4 quadrants, transforming them all separately, and then recombining them to make the full image in a way that allows for completely asymmetric images (or, completely symmetric, as the user desires). In it's current state, PyAbel allows for abel transforms (including with the basex method) that are asymmetric around the horizontal axis, as well as asymmetric about the vertical axis. I believe that basex.exe assumed symmetry about the horizontal axis, and it certainly assumed symmetry around the vertical axis.
(Let me explicitly define the "vertical" and "horizontal" axes here. In PyAbel, we call the axis of cylindrical symmetry the "vertical", and the other axis the "horizontal". In a VMI experiment, the laser is polarized in the vertical direction and propagates in the horizontal direction.)
PyAbel allows the experimentalist a great deal of flexibility, since the quadrants used and the symmetry can be completely specified. For example, say the user has an image that is symmetric about both vertical and horizontal axes (most VMI images), but that the upper right and the lower left quadrants contain regions of a damaged detector and should be ignored. PyAbel can be instructed to just use the upper left and lower right quadrants, average them together, and return the full image corresponding to the symmetric transform.
You say that linbasex can handle "uneven" images through the use of off Legendre polynomials. By this, do you mean it can handle images that are asymmetric around the horizontal axis? This should take care of situations where the ionization is not symmetric along the laser polarization, for example, strong-field ionization using 400+800 nm fields can be asymmetric in this direction. Also, ionization with few-cycle pulses shows asymmetry in this direction with change in carrier-envelope phase.
For photoelectron circular dichroism with chiral molecules, the photoelectron angular distributions can become asymmetric in the other dimension: around the vertical axis. See this image:

In the strictest sense, this type of asymmetry violates the core assumption of the inverse Abel transform, which is that the object exhibits cylindrical symmetry! However, PyAbel can handle this case as well, by completing Abel transforms on each quadrant separately, the asymmetry can be preserved in the Abel Transform.
So, far from being a shortcut to save computational resources, the quadrant system that we are currently using allows us to offer maximum flexibility.
Of course, the splitting of images is not longer a simple choice when the transform methods offer to analytically calculate properties directly from the basis functions, as linbasex does for the anisotropy parameters. In this case, we are torn between offering the user to complete a completely asymmetric transform using the quadrants method and allowing linbasex to return a single set of anisotropy parameters.
@stggh's current implementation offers a nice compromise: an asymmetric linbasex transform is available using quadrants. However, if symmetry is assumed around the vertical axis, then linbasex is applied to the full image, providing direct access to the generated anisotropy parameters.
Does that clarify our approach?

@DanHickstein thanks for providing information about the diverse application of the Abel transform. It is great to hear about this stuff. I tend to think too much in terms of my own experiment.
@gerber-t you also provide great insight. My label Beta1 was simply the array Beta[1] ;-), you are correct that it should be Beta2. For the example_linbasex_hansenlaw.py also, naively limited the Legendre polynomial order to '[0, 2]', when it is now clear that 4 should be included.
I realise that you may be busy, so feel free to answer this query at your convenience:
Q: The angle used in the linbasex (or LIn_Basex) calculation appears to be the standard trigonometry definition (from PY_LIn_Basex_0.7_Jupyter\ 4.0.3_Python3.5.1.py):
an_rad=np.radians(an)               #Express angles in radians

for p in range(pol):
    for u in range(proj):
        B[p,u,:,:]=bas(un[p],np.cos(an_rad[u]),COS)/Norm

i.e. the projection e.g. an = [0, 90] angles are referenced to the horizontal direction, relative to the experimental vertical direction.
I would just like to clarify what the projection angle input, as a direction in space, represents.

@DanHickstein
Yes, linbasex  handles odd symmetry about the horizontal axis, if you allow evaluation including odd Legendre polynomials. In all your examples I recognize a cylinder symmetry either along the horizontal or vertical axis.
It is not intuitiv. But decomposition in spherical functions says that you can establish a symmetry along every axis by adding the appropriate functions with appropriate weight. Having a specific decomposition of a "body" into spherical harmonics produces after rotation another defined decomposition.
Linbasex projects those harmonics to a 1D function. As I showed in the Appendix of my paper all functions Y_lm with a "magnetic" component m unequal zero cancel out upon this 1D projection. (If you would like to explore those components you had to work with 2D projections. Doing so, you could also evaluate unsymmetrical bodies. The transform would then no longer be "Abel" but rather something like Hankel or Radon). So, linbasex assumes cylinder symmetry but allows for even and odd symmetry.
Astonishingly (as I pointed out in the Appendix of my paper) the polarization can point in every direction. The 1 D projections only depend on the angle between the polarization vector and the axis of the 1D projection. Special case: If the polarization is parallel to what you call horizontal axis, all 1D-projections are the same. Odd contributions degenerate to zero. But all even symmetries can still be retrieved. I admit: hard to imagine. (I myself, dealing extensively with it, get slowly mad about those spherical mimics)).
In linbasex you make a guess about the prevailing symmetry (a priori knowldege) by choosing the number and order  of appropriate spherical functions (i.e. Legendere polynomials). With this choice you can evaluate the full image. Choosing appropriate base vectors Basex could do the same.
But I know, experiments are not as gentle, there is always a blemish some where.
Of course you can look at quadrants. For application of linbasex as is, you can just make a full image out of one quadrant for calculating the projections, and then proceed with another quadrant, etc.
The nice thing at looking at projections along different axis in the camera plane tells you how round and how "odd" your actual VMI is.
@stggh
The easiest case is if the polarization vectors lies in the camera (the VMI)-plane. Projection along the polarization axis to the horizontal axis corresponds to a projection at angle 0. That is at least what I remember. I had to check carefully again. One can trace it by looking along what axes the read in VMI is summed up when the angle is 0 or 90 degree.
As I tested it, I guess the assignment at arbitrary angels in array an was correct.

I'm going to close this issue, because, thanks to the hard work by @stggh and @gerber-t, linbasex has been successfully incorporated into PyAbel!
If there are still questions and improvements that need to be made to the linbasex method, perhaps they can be posed in separate issues? (Or, feel free to re-open this Issue is that seems appropriate.)
Thanks again for the great work @stggh and @gerber-t!!

I have not been following in detail PyAbel for some time, and I just wanted to say that the examples look so much better now than the last time I checked! Thanks to everybody who worked on it!

Yes! We have improved the examples :)
I still think that it would be nice if there examples were automatically tested on Travis CI, since it seems like we frequency break them. (See https://github.com/PyAbel/PyAbel/issues/84)


apis: validate header and query param matches

What type of PR is this?
/kind feature
What this PR does / why we need it:
Adds webhook validation to ensure that no
HTTP header or query param is matched more
than once in a given route rule. Also updates
the godoc for HTTPQueryParamMatch to be
consistent with HTTPHeaderMatch.
Which issue(s) this PR fixes:

Fixes #1225
Does this PR introduce a user-facing change?:

Adds webhook validation to ensure that no HTTP header or query param is matched more than once in a given route rule.

/approve


MudTabs with Position set to Position.End has needless scrolling

Bug type
Component
Component name
MudTabs
What happened?
When I have a MudTab control with its position set to Position.End, if I have between 3 and 7 Tabs, even if I have the vertical space for all of the tab headers to display, I get tab header scrolling that I don't want.
If I click on the second tab, then the tab header scrolls and puts Tab 1 out of view unless I scroll back up to see it. Clicking on the third tab causes both the first and second one to scroll out of view, and so on.
If I have just two tabs, no scrolling occurs.
Expected behavior
If I have the vertical room for all of the tab headers to display, then there shouldn't be any scrolling.
Reproduction link
https://try.mudblazor.com/snippet/mYcIaoPQLZGdIneR
Reproduction steps

Create a  instance.
Set the Position to Position.End
Add more than 2 tabs, but less than 8

Relevant log output
No response
Version (bug)
6.15.0
Version (working)
No response
What browsers are you seeing the problem on?
Chrome
On which operating systems are you experiencing the issue?
Windows
Pull Request

[ ] I would like to do a Pull Request

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I don't see any weird behavior with the link you provided.

I don't see any weird behavior with the link you provided.

Here's screenshots for you.
On first load...

After clicking on Tab 2 in the upper tab control

This behaviour has been seen on multiple machines running both Windows 10 and Windows 11 with Google Chrome.

@Anu6is yes indeed they are. Took them just this morning. :)

I also can't reproduce the issue, Windows 11 Chrome. I tried browser zoom and 125% scaling in Windows and it works fine.


updates async-disk-cache to the latest version

may as well enable { compress: 'deflate' }

@stefanpenner updated.

cc @rwjblue now with even less bytes wasted!


Added support for @include/@skip directives.

Overview
Resolves #126
Added support for builtin @skip/@include directives. They only work with $if argument (!).
This supports syntax with inline variables and default ones.
It is very-very-very WIP PR. I'm really looking into ways of improving this code to actually bring in support for directives!
Default variables case
Query:
query($hero:Boolean = true){
  hero @include (if:$hero) {
    name
  }
}

Result:
{
  "data": {}
}

As expected.
Inline variables case
Query:
{
  hero @include (if:false) {
    name
  }
}

Result:
{
  "data": {}
}

As expected.
Passed variables with default values
Query:
query($hero:Boolean = false){
  hero @include (if:$hero) {
    name
  }
}

Variables:
{
"hero": true
}

Result:
{
  "data": {
    "hero": {
      "name": "R2-D2"
    }
  }
}

As expected.

Neat!
I think I'm happy to add a hacky solution temporarily until the new parser lands. Once thats in we will need a more generic user pluggable way to define directives, and there will need to be a bit of planning around that to support various use cases (authentication, api fetch resolvers, caching).

Thanks for addressing some comments. I will try to update PR ASAP.

@vektah could you please do review once again? Thanks in advance :}

@vektah thanks for comments! Will try to fix ASAP!

@vektah
I've moved populateVariables to block only if directive present. Regarding your comment about request ctx I'm not sure. Should I change gotpl to achieve this one? Because we call fields := graphql.CollectFields(ec.Doc, sel, implementors, ec.Variables) in there.
And If it's not necessary, maybe merge this PR and land CollectFieldsWithCtx in another PR ?

Yeah, the template changes are going to be required. A graphql query can have a few forms:

a single, unnamed query, eg { hello { world }.
a single, named query, eg query Hello { hello { world } }
multiple named queries, and the name of the one you want to run.

The only thing this works for is the first one, in order to get the operation name (you are using "", but thats 100% a bug). All this resolution has already been done though, when the validation happened, and it was put in context so the rest of the runtime knows whats going on.

Actually, as I understood, we are only taking one operation. And doesn't matter if it was named or not according to this code:
if operationName == "" {
		if len(d.Operations) > 1 {
			return nil, fmt.Errorf("more than one operation in query document and no operation name given")
		}
		for _, op := range d.Operations {
			return op, nil // return the one and only operation
		}
	}

Thats only if operationName == ""

@vektah
Updated. I've added OperationName to requestCtx. It wasn't in in previously. Now I'm calling
getOperationName(reqctx.OperationName)

The new parser is nearly done now, full directive support should only be a few weeks away.

Great. I'll close second one then. Thanks


add DatabaseAccount and migrate to MariaDBAccount

This moves ironic to fully use MariaDBAccount based on the dev work being done for mariadb-operator:
https://github.com/openstack-k8s-operators/mariadb-operator/pull/184/files
Lead Jira: OSPRH-4095

[x] controller calls EnsureMariaDBAccount up front to make sure MariaDBAccount is present
[x] error code from EnsureMariaDBAccount is handled, Conditions are amended when error is returned
[x] controller calls NewDatabaseForAccount instead of NewDatabase
[x]  GetAccountAndSecret is used to retrieve account /secret to populate template
[x]  GetDatabaseByName() , normally used for delete finalizers, replaced with GetDatabaseByNameAndAccount
[x]  CreateOrPatchAll() used to patch the Database, replacing CreateOrPatchDB / CreateOrPatchDBByName
[x]  controller calls DeleteUnusedMariaDBAccountFinalizers when launched pods are definitely running on a new MariaDBAccount, returns error code if present
[x]  PasswordSelectors that refer to database are removed
[x] all databaseUser replaced with databaseAccount inside of all XYZ_types.go added brand new databaseAccount fields
[x] all databaseUser replaced with databaseAccount inside of all kuttl/*.yaml
[x] all default databaseUser names become databaseAccount, replacing underscores with dashes inside of all XYZ_types.go
[x] all default databaseUser names become databaseAccount, replacing underscores with dashes inside of all kuttl/*.yaml
[x] MariaDBAccountSuiteTests are used in controller ginkgo tests if it has them
[x] Use configsecrets for database URLs; remove from job hash
[x] 184 is merged and replaces from go.mod are removed

recheck

/retest

recheck

recheck

recheck

recheck

retest

recheck

recheck

/retest

/recheck

/retest

recheck

/retest

This PR passes unit tests and kuttl tests locally.   the CI is having an issue with a galera node:
 openstackcontrolplane.core.openstack.org/openstack-galera created
+ sleep 60
+ oc kustomize /alabama/install_yamls/out/openstack/openstack/cr/
+ oc wait --for condition=Ready --timeout=1200s -f -
error: timed out waiting for the condition on openstackcontrolplanes/openstack-galera
{"component":"entrypoint","error":"wrapped process failed: exit status 1","file":"k8s.io/test-infra/prow/entrypoint/run.go:84","func":"k8s.io/test-infra/prow/entrypoint.Options.internalRun","level":"error","msg":"Error executing test process","severity":"error","time":"2024-03-06T02:05:41Z"}
error: failed to execute wrapped command: exit status 1
INFO[2024-03-06T02:05:41Z] Step ironic-operator-build-deploy-kuttl-openstack-k8s-operators-deploy failed after 27m52s.
INFO[2024-03-06T02:05:41Z] Step phase test failed after 1h2m2s.

recheck

/retest

recheck

/test ironic-operator-build-deploy-kuttl

recheck

recheck

@hjensas @stuggi this should be all set after finally passing CI

note this is another "big" one where I moved from configmaps to secrets.  tests passing / deployment succeeding is a good sign things are OK

/retest

/retest

OK it built, ready for approve @steveb @hjensas


Predicted hit often off by +1

I've noticed that the predicted hit is either correct, or +1 higher than the real value. I've started to track the XP values and situations which cause this and I cannot seem to figure out what the math is that would result in it. At first I assumed a rounding error but it doesn't seem like it.
Situation: Fire bolt, 42 xp drop, 11 predicted hit, actual hit 10. Since the spell gives a base XP of 22.5, that would leave 19.5 xp left for the damage which would be 9.75. Similarly, a 43 xp drop was predicted at 11 and actually hit a 10. But sometimes I'll see a 42 xp drop and it'll predict a 10 and actually hit a 10.
I'm guessing there's some rounding happening somewhere but it's unclear why it's inconsistent. It makes sense if it's not perfectly accurate, but it'd be ideal if the predicted hit was more conservative (opting for -1 of the real value) than overestimating. It's better to think you only hit a 10 and end up hitting 11 than thinking you hit 11 and only hit a 10.
Is it possible to round down and/or subtract one when there's ambiguity in the predicted hit amount?

Not much to fix here sadly. The problem you are seeing is because the plugin uses the HP xp gained to calculate hit round(HP xp * 3 / 4) assuming no xp bonuses for the target you're hitting. The HP xp isn't alway accurate since the server rounds it before sending it to the client leading to the off by 1 errors you've seen. No real way for the plugin to figure out the xp has been rounded either so to the plugin there aren't any ambiguities. It is just following the same formula.

I think the round part is the ambiguity I'm referring to. I started collecting data and running the numbers. Here's one example I found.

22 HP XP
Plugin estimated it as a hit of 17 (which is rounded up)
Actual hit was 16 (which is rounded down)

Similarly:

6 HP XP
Plugin estimated it as a hit of 5 (rounded up)
Actual hit was 4 (rounded down)

Or another example

2 HP XP
Plugin estimated it as a hit of 2 (rounded up)
Actual hit was 1 (rounded down)

It seems like a lot of the mistakes are when the hit calculation ends in a .5.
I see 3 different groups:

hit numbers that are whole numbers, these have a near perfect chance of being correct
hit numbers that are 0.25 or 0.75, these have a very high but not perfect chance of rounding to the nearest whole number
hit numbers that are 0.5, these seem to have a very high chance of rounding down rather than rounding up.

I think this can be reasonably solved and will result in a more accurate estimate as the middle values will be rounded down rather than up.

Your anecdotal evidence is not enough to prove the current formula wrong. Very simply; 2 HP xp can be and often is a 2 hit. I need a reputable source before I start testing a new formula. Also you have to accept it will never be perfect because of the server side rounding. The predicted hit functionality is just a small gimmick that tries to predict the actual hit with imperfect information hence why it is called predicted hit.
Also side note, in case you encounter very wrong predicted hits in the future, this is all assuming the target has 1.0 bonus xp multiplier and there is countless of npcs in the game that have a different multiplier. Some are included in a json file in the plugin but some are not.

Also you have to accept it will never be perfect because of the server side rounding.

I am not trying to make it perfect. I'm trying to make it so that whenever it's wrong it's always wrong with a number lower than the real, currently it's always wrong with a predicted number that is higher than the real. You have to admit that gameplay wise and human psychology wise, it's better to be surprised by something being wrong in your favor than something wrong not in your favor.
Let's take the 2 HP XP example, let's just assume that 50% of the time it's a 1 and 50% it's a 2. Currently it always says 2, it'd be better if it always said 1. It's still just as wrong, but it'll be a benefit to the player when it's wrong rather than a detriment.

For the record, I tested the theory with Wyrms which have a 2.5% xp boost. The predicted hit incorrect less often, but always incorrect by predicting lower than the actual hit by 1.
I think a very safe change to make would be the following:

If predictedHit % 0.5 == 0, floor(predictedHit), else round(predictedHit).

This would be very conservative and just improve the case which leads to predicting one too high without affecting anything else.

I'm not going to change to formula to miss predict hits just so it accidentally correctly predicts edge cases. If you want the behavior you're describing you can just set the xp multiplier in the config to something slightly below 1 such as 0.99. This should cause the value before rounding to shift slightly resulting in behavior you want.


Fail to make project

Hi, I am trying to make the project, but I get this error.
Could not GET 'https://dl.bintray.com/kotlin/kotlin-eap/androidx/core/core-ktx/maven-metadata.xml'. Received status code 502 from server: Bad Gateway
Disable Gradle 'offline mode' and sync project

fixed....


Unknown build problem for Android in UE4.11

Not much more to say about that, but maybe you can suggest where to look for a solution?

Wait a few weeks? I am working on an android port, and should have something out in a few weeks time.

Good to know, thanks @shanecolb
We have been working on a project for android and iOS for some time now, but haven't been able to fully test it yet. Does this already work for iOS?

Nope, Windows only at the moment.
I don't have any iOS devices, so it would make testing/dev hard.
Does anyone know if iOS emulation is friendly with speaker/microphone input?
If so, I could use that to allow me to work on an iOS port, shortly after Android.

Nope, Windows only at the moment.
I don't have any iOS devices, so it would make testing/dev hard.
Does anyone know if iOS emulation is friendly with speaker/microphone input?
If so, I could use that to allow me to work on an iOS port, shortly after Android.


Doesn't work on esm?

I get so many weird errors
import $C from "js-combinatorics"
export const version = '1.5.4';
^^^^^^

SyntaxError: Unexpected token 'export'
    at Object.compileFunction (node:vm:352:18)
    at wrapSafe (node:internal/modules/cjs/loader:1025:15)
    at Module._compile (node:internal/modules/cjs/loader:1059:27)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1147:10)
    at Module.load (node:internal/modules/cjs/loader:975:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:190:29)
    at ModuleJob.run (node:internal/modules/esm/module_job:185:25)
    at async Promise.all (index 0)
    at ESMLoader.import (node:internal/modules/esm/loader:281:24)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

If I use this:
import { Combination, Permutation } from "js-combinatorics";
It gives:

import { Combination, Permutation } from "js-combinatorics";
         ^^^^^^^^^^^
SyntaxError: Named export 'Combination' not found. The requested module 'js-combinatorics' is a CommonJS module, which may not support all module.exports as named exports.
CommonJS modules can always be imported via the default export, for example using:

This is fixed since version 1.5.5, suggest to close the issue


Restore Synthetic Data Processing Notebook

Somehow the synthetic data processing notebook got lost while addressing #337 , probably when organizing the notebooks in tutorials and examples.  It is still alive and well on the #333 branch.
I checked it out from there on fix_issue_337 branch.

Fixed and present on the joss branch


Add missing flags to FileAccess and improve documentation.

Add missing flags to FileAccess and improve documentation.
I have deprecated FileAccess.Reserved and replaced it with FileAccess.AccessSystemSecurity
Related to #117

I am honored to you offer to play a greater role in the project. :blush:
I can not say no to such a mission. :smile:

:heart: done !


Use FlutterFeatures to configure web and desktop devices

Description
A follow up to https://github.com/flutter/flutter/pull/36138, this PR enables the usage of flutter config for web, Linux, macOS, and Windows features. This removes the on-by default behavior of these features on master branch and instead replaces it with a configurable flag. For backwards compatibility, the old environment variables ENABLE_FLUTER_DESKTOP and FLUTTER_WEB still exist. I would like to deprecate and remove these at some point in the future, and encourage user to provide config values.
Example:
> flutter config
Usage: flutter config [arguments]
-h, --help                           Print this usage information.
    --[no-]analytics                 Enable or disable reporting anonymously tool usage statistics and crash reports.
    --clear-ios-signing-cert         Clear the saved development certificate choice used to sign apps for iOS device deployment.
    --gradle-dir                     The gradle install directory.
    --android-sdk                    The Android SDK directory.
    --android-studio-dir             The Android Studio install directory.
    --[no-]enable-web                Enable or disable Flutter Web on master, dev channels.
    --[no-]enable-linux-desktop      Enable or disable Flutter Desktop for Linux on the master channel.
    --[no-]enable-macos-desktop      Enable or disable Flutter Desktop for macOS on the master channel.
    --[no-]enable-windows-desktop    Enable or disable Flutter Desktop for Windows on the master channel.
    --clear-features                 Remove all configured features and restore them to the default values.

Run "flutter help" to see global options.

I've also removed the isExperimental command property, since I do not want to have two distinct ways to control feature visibility.


Monitor Alerts limit

Does the below limit also apply or going to apply to the new alert experience? If not will there be an upper limit like this
for it?
Metric Alerts	100 active alert rules per subscription

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1cd57b6d-3c6f-e546-b3ea-f83892f4c23d
Version Independent ID: 27aecc7b-f107-96d4-b044-ae026bca2727
Content: Azure subscription limits and quotas
Content Source: articles/azure-subscription-service-limits.md
Service: billing
GitHub Login: @rothja
Microsoft Alias: byvinyal

@waheedhussain Thanks for bringing this to our attention. Your feedback has been shared with the content owner for further review.

@waheedhussain Thanks for the question! I'm going to pull in a few other people on this to see if I can get an answer. @anirudhcavale @johnkemnetz @rboucher Do you know whether the old limits apply to the new alert experience? Do we have updates to make on this page? Thanks!!

Resolved in another thread.
Default limit is 100 per subscription. Maximum limit should be “Call support” like it is for older metric alerts.
Instead of near real-time metric alerts (Preview), it should be “Newer Metric Alerts” .
From: Rob Boucher
Sent: Friday, May 4, 2018 11:13 AM
To: 'MicrosoftDocs/azure-docs'<EMAIL_ADDRESS>MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: Mention<EMAIL_ADDRESS>Kiran Madnani<EMAIL_ADDRESS>Snehith Muvva<EMAIL_ADDRESS>Subject: RE: [MicrosoftDocs/azure-docs] Monitor Alerts limit (#7918)
Kiran or Snehith would know. Adding them.
From: Jason Roth <notifications<EMAIL_ADDRESS>Sent: Tuesday, May 1, 2018 1:32 PM
To: MicrosoftDocs/azure-docs <azure-docs<EMAIL_ADDRESS>Cc: Rob Boucher <robb<EMAIL_ADDRESS>Mention <mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] Monitor Alerts limit (#7918)
@waheedhussainhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fwaheedhussain&data=02|01|robb%40microsoft.com|1cac89172caf4f996a7e08d5afa28e51|72f988bf86f141af91ab2d7cd011db47|1|0|636608035064677563&sdata=lCTEQYPNWp0Fi9P0lyaLs4rqKTqd7SOTkEXIXNhCYEA%3D&reserved=0 Thanks for the question! I'm going to pull in a few other people on this to see if I can get an answer. @anirudhcavalehttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fanirudhcavale&data=02|01|robb%40microsoft.com|1cac89172caf4f996a7e08d5afa28e51|72f988bf86f141af91ab2d7cd011db47|1|0|636608035064677563&sdata=D7NNV1pASW2kOuE8RRbuaXJT1%2BzR2nWSo0sfUvSwh%2Bg%3D&reserved=0 @johnkemnetzhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjohnkemnetz&data=02|01|robb%40microsoft.com|1cac89172caf4f996a7e08d5afa28e51|72f988bf86f141af91ab2d7cd011db47|1|0|636608035064687568&sdata=PQrqwepfR9SvoFpdc5xwOIVjAkzHmrdT6KmlpOGqjDU%3D&reserved=0 @rboucherhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frboucher&data=02|01|robb%40microsoft.com|1cac89172caf4f996a7e08d5afa28e51|72f988bf86f141af91ab2d7cd011db47|1|0|636608035064687568&sdata=8XkXAlH0Nn13bfl8hUs%2BllGj%2FQV08vdPq3e2PQ45Mjo%3D&reserved=0 Do you know whether the old limits apply to the new alert experience? Do we have updates to make on this page? Thanks!!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fazure-docs%2Fissues%2F7918%23issuecomment-385781001&data=02|01|robb%40microsoft.com|1cac89172caf4f996a7e08d5afa28e51|72f988bf86f141af91ab2d7cd011db47|1|0|636608035064697578&sdata=WPClWVI5DjZRAriDMwvh%2BsADETqTR6CE1VcazVQotNs%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD6pebuF9EcQueX16e4j2tYnBWmzYqsPks5tuMYvgaJpZM4TtwIL&data=02|01|robb%40microsoft.com|1cac89172caf4f996a7e08d5afa28e51|72f988bf86f141af91ab2d7cd011db47|1|0|636608035064697578&sdata=p%2BINAMd1FRpYKJWRhmxDLBEl1%2BjjO9MgQ4qGTJ90kbw%3D&reserved=0.

@waheedhussain As Rob discovered, 100 is also the default limit for the new metrics. I'm going to update the topic thanks to your feedback. Appreciate it.


Allow relative $PUBLIC_URL in dev mode

As long as client side routing isn't used, a relative $PUBLIC_URL should
work correctly in both development and production.
There's no reason it should be limited to production.
Closes #8623
Also see https://github.com/cdr/code-server/issues/2565

Is there any update on this? I need to use relative paths in dev mode and it's being quite a nightmare (see https://github.com/webpack/webpack/discussions/16425)

I'm no longer interested in persuing this.


Release: design updates

✍ Description
Design updates and some minor fixes.

LGTM! 👍 You can include #121 as well!

Yup! I was waiting to have that one merged before deploying everything to prod. ☺ Thanks for the reminder anyway @beagleknight! 💙


Maduong/bugfix/282569 The CLI should look for a compose file anywhere in the folder (not just at the root)

Verifies docker-compose and docker file exist somewhere in the repo, not just the root

Hi @thegalah, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes. I promise there's no faxing. https://cla.azure.com.

TTYL, AZPRBOT;


full README license link

IDK how I missed this before during #96, but rustdoc gives a warning on the LICENSE link. The link does actually work! But I think rustdoc gets confused by "shorthand" link paths (LICENSE) because there could be a rust doc item by the same name.
rustdoc does not get confused by explicit URL paths tho, so this updates the LICENSE link to use the full URL

Coverage remained the same at 92.238% when pulling 113f95e1df4ee6553b36240bf24e631550835289 on license-link-lint into 8215f061433957712c5217917ab0aa3153d463f0 on main.


Webhook Improvements

Hey,
I've made some changes to the source code, we should discuss if it's so ok.
CHANGELOG:

Added Game & Extension Analytics Topic
I added $token to getWebhookSubscriptions() because it's required by this endpoint. @romanzipp Q: Should the token be passed as a parameter or rather with withToken? I don't know the best practice.
The urlencode() is now implemented directly in executeQuery.
I have updated the documentation because PhpStorm has shown me some bugs.

Concerning token usage with the getWebhookSubscriptions method:
You should add an optional token parameter just like getAuthedUser(string $token = null) if a method heavily rests on user authentication. This parameter should always be optional since you can set the token on the Twitch wrapper instance.

Hey, i removed the WIP flag. I think this PR will be a BC because of urlencode.


OpenUPM support. Unity Package Manager

Hi there, it would be great if RSG Promise could be apart of OpenUPM
It will also solve dependency issue mentioned in issue #76
However, it maybe need to be a new branch as that focuses on UPM structure while master branch still supporting plan .NET projects?

I have set this up already - https://openupm.com/packages/com.rsg.promise/
Published from my personal account because I no longer work for RSG and I'd prefer to keep the main repo set up as a regular Visual Studio project anyway. While many people use the library in Unity there are also people who use the Nuget package in regular .NET projects, so restructuring the main branch to be a package isn't ideal.
Migrating from my personal repo to a separate branch in the main repo would be a good idea but I wouldn't really like to do that without having some kid of process to automate it. Building, copying the DLL, checking out a different branch and committing it is a kind of complicated process to have to do manually all the time. Something like what gh-pages does where it can automatically copy build artifacts to a different branch for you would be ideal.

Brilliant! Don't why I didn't see that. My bad 😄

I'll update the readme with a link to the package so that it's easier to find in future :+1:


Parents field in Document model

Add a 'parent' field to Documents as an array of strings, stored in Qdrant payloads as well as the Core sql DB and GCS
In this PR, all document upserters (managed datasources sync workflows + standard DS doc upsert) provide an empty array
4  PRs will follow:

Make the field searchable in core (including a qdrant migration to index it)
Have the sync properly fill the field for Notion documents
Idem for Github
Idem for Slack

:pray: :pray:
Quick question: there's a build failure in core on CI that I don't have locally (using cargo build or cargo test --all); spend 20mn investigating and from afar it seems to be unrelated to our codebase? it seems to be during compilation of a dependency (v8), error E0080 on an assertion
I'll keep looking but if you already know what it is about I'm eager for any tips :bow:

Thanks for the help :pray:

The migration plan has a few caveats that I'm currently looking into

Regarding the second caveat (viability of pre-deployment code with newly altered DB), there is no issue in core: in postgres.rs the selects have explicit fields, and inserts are ok since the new column has a default value
Core, front, connectors should be deployed in the right order:

First connectors -- the result will be that front API gets requests with a 'parents' field in the payload, that it ignores
then front -- idem, core API receives a request with an additional 'parents' param that it ignores CF doc below
then regular deployment of core and it should do the trick

=> About the fact that core will ignore the parents field in the API call, this is what gave me confidence (after copilotGPT told me so):
#[serde(deny_unknown_fields)]
Always error during deserialization when encountering unknown fields. When this attribute is not present, by default unknown fields are ignored for self-describing formats like JSON.

https://serde.rs/container-attrs.html

So the migration plan is

Pull main
Migrate SQL DB
deploy connectors
deploy front
deploy core

@spolu  GFY?

This looks great! I feel like we'd want to add the migration that add an index to all existing QDrant collections in this PR. But that's not strongly held. I just worry we forget about it and get bitten by it later on?

I think the qdrant migration can be done with the /search filters in core since one is meant for the other
we limit the risks by making 2 deployments
You are right that we shouldn't forget this but I guarantee we won't:  it will be my next PR, I already started working on it (I actually have finished the migration script and will work on the search next)

Deploy procedure looks good but would be nice to check for real that core accepts extra field, that's pretty easy to do 👍

Ok will do

It works


Prevent truncation of cell method descriptions with nested brackets

🚀 Pull Request
Description

Improves the parsing of netcdf cell methods to not rely solely on regular expressions and thus fixes #4427. Initially created a broken test using the example from #4427 and then fixed it.
Questions for reviewer:

Is there a canonical list of what can make up cell methods (e.g. whether nested brackets are allowed) that could be used to check this code works on them all?
The previous code would fail silently on a cell method that didn't match the regex whereas now it's possible to identify strings that should have given a cell method but didn't match. Should these be warnings or exceptions?

To do:

[x] Warnings when a cell method can't be parsed
[x] What's new (do after merge)

Consult Iris pull request check list

Should these be warnings or exceptions?

I do not think an exception would be helpful here, as (as far as I know) there would be no way for the user to get around it within Iris.  All they could do is find some other tool to modify the NetCDF file before attempting to load it again.

Regarding your first point, on a canonical list, I think there is none. In CF 1.9 section 7.3.2 Recording the spacing of the original data and other information details are given how to combine standardized elements and non-standardized elements. But I did not find any guidance as to what can or can not go into the non-standardized part.
Will your PR handle nested brackets and other more or less difficult (in python sense) strings? If not it might be useful to have a check of the string before writing to file in the first case (cf. my comment in #4427).

Will your PR handle nested brackets and other more or less difficult (in python sense) strings?

It currently has to assume that you've closed all the brackets you've opened, otherwise there's no way to distinguish comment content from actual new methods.
The only way I can think of that Iris could check if it can interpret a cell method string is to try reading it once it's formed it, which is basically what happens when you attempt the load again, so I'm not sure warning then is any better than warning on load? I don't think Iris should do anything stronger than just warning because it can't read it back, because you might really want to write that for some other program.

It currently has to assume that you've closed all the brackets you've opened, otherwise there's no way to distinguish comment content from actual new methods.

Fair enough :+1:

The only way I can think of that Iris could check if it can interpret a cell method string is to try reading it once it's formed it, which is basically what happens when you attempt the load again, so I'm not sure warning then is any better than warning on load? I don't think Iris should do anything stronger than just warning because it can't read it back, because you might really want to write that for some other program.

Warning is perfect :+1:
Thanks for taking this on so quickly!

Any advice on better use of warnings, mocking and patching warmly welcomed

@rcomer apart from a whatsnew, which I'd pop in a separate PR to avoid conflicts, is there anything more to do here?

@wjbenfold I'm afraid I've not reviewed the bulk of this.  The re package is completely alien to me!

:+1: :+1: :+1:


A contract that only defines function owner() external returns (address) cannot correctly validate permissions when binding with an aspect.

System information
Artelad version: v0.4.9-rc9
Aspect-libs version: e.g<EMAIL_ADDRESS>Web3 version: e.g<EMAIL_ADDRESS>OS & Version: OSX
Commit hash : (if develop)
Expected behaviour
Deploy success.
Actual behaviour
If a contract only defines function owner() external returns (address), it encounters insufficient permission validation when binding with an aspect. The error message is as follows:
(node:66710) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
/Users/cppppppp/project/github/hyperlanehook/bridgeaspect/node_modules/web3-core-helpers/lib/errors.js:28
        var err = new Error('Returned error: ' + message);
                  ^

Error: Returned error: rpc error: code = Unknown desc = contract ownership validation failed
    at Object.ErrorResponse (/Users/cppppppp/project/github/hyperlanehook/bridgeaspect/node_modules/web3-core-helpers/lib/errors.js:28:19)
    at /Users/cppppppp/project/github/hyperlanehook/bridgeaspect/node_modules/web3-core-requestmanager/lib/index.js:300:36
    at /Users/cppppppp/project/github/hyperlanehook/bridgeaspect/node_modules/web3-providers-http/lib/index.js:127:13
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
  data: null
}

Steps to reproduce the behaviour
Backtrace
[backtrace]

When submitting logs: please submit them as text and not screenshots.

Fixed in: https://github.com/artela-network/aspect-core/pull/69


Auxiliary Capabilities Request

Hi，
I searched the source, found out that the "Auxiliary Capabilities Request" was not implemented , and I am confused about the protocol response describing "Number of Instances: Indicates the number of input/function instances where the described
Function attribute and Assigned attribute are the same". I'd like to contribute this part, can anyone explain how to get the number of instances?

It's a bit confusing to me as well... I can see if I can find out more. But, if you're wanting to contribute the client side of this, then a complete understanding might not be strictly needed I guess, as if I was going to implement sending of the request and receiving of the reply, I'd probably just give that information verbatim to the consumer of the library and let them use it as they see fit?
Like, I'd probably pass them (in a callback or something) the values listed under J.7.14 bytes 3-n repeating in a const class, maybe something like
class AuxiliaryCapabilities
{
public:
    class SetInformation
    {
        std::uint8_t numberOfInstances;
        std::uint8_t functionAttribute;
        std::uint8_t assignedAttribute;
    };
    NAME auxiliaryUnitNAME;
    std::vector<SetInformation> sets;
};

Could also use enums or something to clarify further, but I think that'd be the general idea?
Then the answer to "how to get the number of instances" is just "it's whatever the VT sends back to you" since the VT provides that based on the content of the other working sets' object pools.