hansenlaw.py code re-write/re-structure

This PR that follows on from PR #206, rewriting the hansenlaw method code! It is now 20-30% faster and uses less memory.
The original aim was to ensure the pixel grid indices are consistent with the definitions in the Hansen & Law paper:

in an effort to track down the source of the pixel alignment offset/shift of the transform
Outcomes

consistent index - mapped index arrays to use a single index, right-side --> left-side
transform direction - removed Gamma and Phi functions, separating the definition of the Gamma arrays according to the transform direction
smaller dimension arrays - Phi[cols-1, K, K] is diagonal for each pixel. Used numpy broadcasting to multiply a smaller 2D diagonal array. Similarly, Gamma is reduced to [cols-1, K]. Note, I examined storing a pre-calculated "basis" (Phi, Gamma), but there was little to gain in execution speed.
bad - image pixel shift still necessary to align O2- transform with the three_point method:

It is not clear what is going on here, I believe hansenlaw code/indexing is correct. In the end the hansenlaw inverse Abel transform does not produce as exact a result (from the TransformPairs()) as three_point, so the specifics of this minor pixel shift correction may not be important. I have made it an optional parameter in hansenlaw. It is less important for the forward transform which, from a TransformPair() comparison, is more accurate.
Some timings:
In [1]: import abel

In [2]: abel.benchmark.AbelTiming(n=[501])
Out[2]:
PyAbel benchmark run on x86_64

time in milliseconds

=========    Basis Abel implementations ==========

Implementation      n = 501
----------------------------------------
        basex_bs   19367.9
     linbasex_bs      13.0
onion_peeling_bs       1.6
  three_point_bs       6.2
    two_point_bs       2.1

=========  Forward Abel implementations ==========

Implementation      n = 501
----------------------------------------
        direct_C      19.9
   direct_Python     229.3
       hansenlaw       5.6

=========  Inverse Abel implementations ==========

Implementation      n = 501
----------------------------------------
           basex      19.4
        direct_C      21.7
   direct_Python     232.2
       hansenlaw       6.0
        linbasex      94.4
    onion_bordas     234.3
   onion_peeling       0.6
     three_point       0.4
       two_point       0.4

cf second column of timings in PR #209.    7.5ms -> 6ms == 20% improvement.
hansenlaw is slightly faster than three_point (when its basis calculation time is included)
and
[etalon:.+tg121/Test-Abel]$ ipython3
Python 3.6.4 |Anaconda custom (64-bit)| (default, Jan 16 2018, 18:10:19)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import abel

In [2]: whole = abel.tools.analytical.SampleImage(501).image

In [3]: half = whole[:, 250:]

In [4]: half.shape
Out[4]: (501, 251)

In [5]: %timeit abel.hansenlaw.hansenlaw_transform(half)
5.88 ms ± 170 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [6]: %timeit abel.dasch.three_point_transform(half)
1.43 ms ± 87.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [8]: (8.81-5.88)*100/8.81
Out[8]: 33.257661748013625    # % improvement

cf bottom result of the same link given above.
Benchmark figure ( i7-6700 CPU @ 3.40GHz, Ubuntu 17.10, Python  3.6.4 :: Anaconda)

benchmarks_4.13.0-25-genericx86_64Linux_2018-02-26_23h47m05s.csv

Nice work @stggh! Do I sense that there is still a competitive element, trying to get HL to be the fastest transform method? :) If you manage it, I'll bet @DhrubajyotiDas will recruit some of his Intel buddies to speedup the linear algebra functions in MKL so that he can reclaim the crown.
(An aside, but have you seen that Intel now has it's own Python distribution? I was playing with it the other day for some FFTs, but it wasn't any faster for my application.)
Does the 0.35 pixel shift still appear when you are comparing with the analytical result? It's possible that HL is providing an appropriate result and the other methods are shifted for some reason.
Also, how confident are we that the h and lam values are correct? It seems possible that a typo in the manuscript is causing the issue. Or, perhaps those values were calculated using a grid that was off-by-one. Also, the length of those arrays is arbitrary, right? We should get a more accurate transform if we used more values of h and lam, yes? Potentially we should shorten (or lengthen) those arrays and make sure that the transform result is converged with respect to this approximation.
Skimming quickly, your code looks pretty good. This might be a nice opportunity to add a few more comments to the transform method.

@DanHickstein I didn't even know we had a special python distribution - I guess I'll have to migrate to that now for professional ethics reasons :)
BTW, this Anaconda claims a 97X (!!) improvement in speed on the Intel distribution (https://www.continuum.io/sites/default/files/AnacondaIntelFAQFINAL.pdf). Maybe you need the right Intel hardware (i.e., latest greatest) to see a measurable difference between this and vanilla Anaconda?
I never quite managed to decipher the Hansen Law algorithm, so I'll trust yours and @stggh's judgment on the validity of the code. More comments in the code are mostly always good.

HL to be the fastest transform method?

Yes(!) and that the inverse Abel transform results for for some of the transform-pair functions are not a perfect match, as is it appears to be for three_point. This is important because the two/only  forward transform methods have what appear to be slightly inaccurate inverse transforms.
Curve A (profile 3) Fig 3: of Hansen & Law indicates that hansenlaw should do better:

basex speed would benefit from some python magic, similar to what was applied to the Dasch methods, without the need for fast linear algebra libraries, or chip based python implementations.

Does the 0.35 pixel shift still appear when you are comparing with the analytical result?

This is not clear cut. Let's say there is not a consistent shift for all the transform pairs.

HL is providing an appropriate result and the other methods are shifted for some reason

hansenlaw and direct have a similar transform behaviour, however, the Dasch methods do so well it is hard to argue that they could have a problem.

how confident are we that the h and lam values are correct

Good point, that I had also considered. I tried a least-square fit of the parameters, but the power terms e.g. Eq. (16a), make a fit possible for a variety of (extreme) parameter values. A single parameter adjustment, homes in to its own value. The h, and lam tabulated values are close.

should shorten (or lengthen) those arrays

Interesting point. Truncating the size to 5, behaves in a way that you expected:

add a few more comments to the transform method

You assume that I understand the method enough to make intelligent comments ... ;-)

I have dropped the pixel shift parameter from hansenlaw, as it may cause confusion, and I no longer believe in its universal applicabilty. The pixel image shift only appears necessary to align the methods for (my) O2-photoelectron spectra, and not for the transform-pairs (as I once thought). The correction may just as easily be applied to the radial profile, as a small radial correction (x1/1.0008) e.g see example_all_O2.py. This figure (not identical to the example) is for comparison of those shown above (just uses an energy scale):

I have added a few more comments to the hansenlaw doc string, but actually have little background in the "state-variable method approach" :-(   The coding is simply the equations given at the top of this PR.
I have also added the circularize_image J. Chem. Phys. reference to the function doc string and associated .rst file, as requested in #210
Not sure about the windows travis-ci build problem?

Thanks for this analysis @stggh!
Can you make the "Curve A, N=101" figure, so that we may make a direct comparison to Fig. 3 of the HL paper. Also, can you calculate the mean-squared error, so that we can compare to the 1.33-e6 value listed in the HL paper? It seems that we are somehow falling short of the performance shown in their manuscript, which points to some annoying bug.
I am curious if we can calculate our own h and lam values, since the paper says that they are only calclulated to an error of 0.001. Perhaps we can do better, and make a larger expansion. However, I found that paragraph of the paper (regarding the calculation of h and lamda) to be very confusing.

@DanHickstein  thanks for the guiding, it has helped identify issues that I thought I had visited.
N=101 and mse (good idea).
Interesting:
(1) with the hansenlaw.py code as above, poor result for hansenlaw (orange curve), three_point does very well, mse=10-9

(2) driving function (gradient or image) shifted by 1 pixel left. Better agreement between hansenlaw and direct:
    # 1-pixel shift better aligns 'curve A' transform pair
    drive[:, :-1] = drive[:, 1:]

(3) 2 pixels, drive[:, :-2] = drive[:, 2:], mse worsens, a shift too far

(4) -4/3 shift, using drive = scipy.ndimage.interpolation.shift(drive, (0, -4/3)), mse still not = H&L

No pixel shift can achieve a mse=1.33E-6 shown in H & L.
A 1-pixel offset may be due to relative indices for transform arrays cf the driving array, the n+1 index:
Hansen and Law (index 0 is right side of image):

and corresponding code:
    for col in n:  # right image edge to left edge
        X = Phi[col][:, None]*X + Gamma[col][:, None]*drive[:, col]
        AIM[:, col] = X.sum(axis=0)

Similarly, for Curve B (profile8):

It is clear that there should at least be a -1 pixel offset/shit. A fractional shift, I am not so sure.
I have temporarily returned the shift parameter to hansenlaw, and here is the test code  Curve A code

Ok, it turns out that the col index, for the transform code line,  was not aligned with the Phi and Gamma arrays. This gave a 1-pixel offset. Now curve A mse=5.77e-5. The inverse transform is very similar to the direct method result. However, (once again!) a -1/2 pixel shift gives mse=1.07e-6, which is better/close to the Hansen & Law value of 1.33E-6.
In the hansenlaw code I am now, using 'N' (for cols) in the hansenlaw code to better compare with the H&L variable names.
Note that the weightings  (N-n)/(N-n-1) need to be the numpy index order. This has always been the case, but points to an imperfect description in H & L.
   +----+----+                    +----+----+
   |  0 |  1  |                   | N-2 | N-1 |    numpy
   +----+---- +       ...         +----+----+
   | N-1 | N-2 |                  |  1 |  0 |     H & L
   +----+----+                    +----+----+
  (N-n)/(N-n-1), (N-n-1, N-n-2) ...(2/1)  (und.)

Curve A zoomed:
shift=0 tranfrom similar to direct

shift= -1/2, better agreement with transform pair

Curve B:

In conclusion, it is easy to make an index error. The -1/2 pixel shift may also apply to the direct method. This may have something to do with grid alignment to image pixels #39

Python weirdness
The "good" mse of 1.0e-7 for hansenlaw depended on whether a forward transform was calculated, anywhere in the code. All very odd!
This code produces a different result, 9.24e-07, from 1.52e-04 if the forward transform is called.
import numpy as np
import abel

shift = -1/2

f = abel.tools.analytical.TransformPair(101, 3)  # curve A

hl = abel.hansenlaw.hansenlaw_transform(f.abel, dr=f.dr, shift=shift)

# this line changes mse hl from 1.52e-04 to 9.24e-07
corrupt = abel.hansenlaw.hansenlaw_transform(f.func, dr=f.dr, direction='forward', shift=shift)

msehl = np.square(hl-f.func).mean()
print('hansenlaw mse = {:4.2e}'.format(msehl))

The forward transform has, line 143: drive = IM, which somehow changed/corrupted the input (f.func). I have set drive = IM.copy() to give a consistent result.
Unfortunately, the optimum, least-squares fit, is now no better than 1.25-04, for shift=-0.16, which has got me wondering why the "corruption" produced such a perfect result ...

This PR is turning into a blog of my incompetence.
(1) Python's pass by value will pass back the array change. Something that I "should" be familiar with :-(:
In [1]: import numpy as np

In [2]: def func(x):
   ...:     y = x
   ...:     y[2] = 10
   ...:     return y
   ...:

In [3]: x = np.arange(9)

In [4]: x
Out[4]: array([0, 1, 2, 3, 4, 5, 6, 7, 8])

In [5]: yy = func(x)

In [6]: x
Out[6]: array([ 0,  1, 10,  3,  4,  5,  6,  7,  8])

(2) Final hansenlaw.py code:
1-pixel offset of Phi and Gamma arrays relative to the input image (forward transform) or gradient (inverse transform), drive, 'AIM` (transform). Coded as:
    for col  in n:  # right image edge to left edge
        X = Phi[col][:, None]*X + Gamma[col][:, None]*drive[:, col+1]
        AIM[:, col+1] = X.sum(axis=0)

This form executes slightly faster:
    n1 = n + 1 # Phi, Gamma, 1-pixel offset relative to image
    for col, col1  in zip(n, n1):  # right image edge to left edge
        X = Phi[col][:, None]*X + Gamma[col][:, None]*drive[:, col1]
        AIM[:, col1] = X.sum(axis=0)

For curveA (profile3) and curveB (profile8) hansenlaw produces very similar results to direct:

[etalon:.+tg121/Test-Abel]$ python3 CurveA-mse.py

  forward method     mean-squared error
  --------------     ------------------
    hansenlaw             5.77e-05
    direct                2.99e-05

  inverse method     mean-squared error
  --------------     ------------------
    hansenlaw             5.77e-05
    direct                5.27e-05
    three_point           5.68e-09
[etalon:.+tg121/Test-Abel]$ python3 CurveB-mse.py

  forward method     mean-squared error
  --------------     ------------------
    hansenlaw             1.30e-04
    direct                6.52e-05

  inverse method     mean-squared error
  --------------     ------------------
    hansenlaw             9.07e-05
    direct                2.92e-05
    three_point           6.76e-08

additional -1/2 pixel offset
hansenlaw (and possibly direct) can do better if the input is shifted left by 1/2-pixel:

[etalon:.+tg121/Test-Abel]$ python3 CurveA-mse.py

shift = -1/2

  forward method     mean-squared error
  --------------     ------------------
    hansenlaw             1.55e-06
    direct                2.99e-05

  inverse method     mean-squared error
  --------------     ------------------
    hansenlaw             1.07e-06
    direct                5.27e-05
    three_point           5.68e-09
[etalon:.+tg121/Test-Abel]$ python3 CurveB-mse.py

shift = -1/2

  forward method     mean-squared error
  --------------     ------------------
    hansenlaw             5.63e-06
    direct                6.52e-05

  inverse method     mean-squared error
  --------------     ------------------
    hansenlaw             3.06e-06
    direct                2.92e-05
    three_point           6.76e-08

Here are the inverse transform pair % different plots  (= (method-f.func)*100):

and forward transform:

I conclude that the hansenlaw, and possibly direct, methods apply to an even width image, see the grid discussion in #39.  The additional
    drive = interpolation.shift(drive, (0, -1/2))

increases the transform time, although, it is still comparable three_point (including the basis calculation). It may be better to simply check whether the image is odd, and apply the grid shift if required.
I believe (hope!) that the hansenlaw code is correct, and the transform alignment issue is now solved.
My apologies for all the toing and froing.

Python's pass by value will pass back the array change.

I think that it's actually "pass-by-reference-like" behavior that is getting us into trouble. I actually think that Python is fairly confusing in this regard, and it sometimes seems to flip back and forth between passing by reference and passing by value. (This opinion is probably because I don't really understand what a "mutable" variable is, see here.)
This behavior has caused problems for me in the past. I think that we should modify the functions in PyAbel such that they don't modify the original image. Basically, I think that we need a np.copy() statement at the top of a lot of the transform functions. Could you open a new Issue about this, and we can maybe get @rth to weigh in, since he likely has an informed opinion on this.
It looks like you've made good progress on the sub-pixel shift though! Do I understand correctly that you have a MSE of 1.55e-6, which the H&L paper reports 1.33e-6? That seems like reasonably good agreement. That is annoying that the HL algorithm seems to operate on an even grid. Perhaps if we could figure out how to recalculate the h and lam arrays, then we could modify it to work on an odd grid.
Are you basically ready to merge? Should I have a final look through the code (when I get some time)?

"pass-by-reference-like"

You are right.

need a np.copy()

direct does this (of course @rth!), basex and linbasex create a full image, leaving only the methods that I had a hand in, that do not copy the input image ...
Not sure this needs discussion, the other methods simply should follow @rth's code: np.atleast_2d(image.copy()). I have made the change to hasenlaw, dasch methods (which oncludes three_point, two_point, and onion_peeling), and onion_borads in this PR.

MSE of 1.55e-6, which the H&L paper reports 1.33e-6

Actually, 1.07e-06 for curve A.

ready to merge

Hold off until after the W/E.

need a np.copy()

direct does this, basex and linbasex create a full image, leaving only the methods that I had a hand in, that do not copy the input image ...
Not sure this needs discussion, the other methods simply should follow @rth's code: np.atleast_2d(image.copy()).

From what I have seen, sometimes transformers have an option to copy or do inplace operations (e.g. sklearn.preprocessing.StandardScaler). Making a copy is safer but it does have some performance cost, which can be negligible or not depending on the application.
For instance fft does call np.asarray (np.atleast_2d would preclude 1D arrays) for validation but it doesn't make a copy. I imagine that the _raw_fft doesn't do inplace modifications of the input array.

Thanks @rth!
So, I think that there are two issues:

What do users expect?

What is the performance cost?

Regarding the first issue, I think that most people could expect our Abel transform method to work similarly to something like np.rot90 or scipy.ndimage.interpolation.rotate. These methods do not overwrite the input array, but simply return a new array with the result. (Some quick code to test this behavior.)
We might not need an actual "np.copy", so long as we are using some function that makes a copy of the array. But, perhaps making a copy is the safest way.
It's funny that you bring up the issue of the fft, because scipy.fftpack.fft does have an option overwrite_x, which is False by default, but can be set to True an allows the function to overwrite the input array with the fft, which can allow for higher performance in some situations, since a copy doesn't need to be made. In any case, since the default behavior is False, this communicates to me that the "expected" behavior is to make a copy of the array, and this should only be overridden if performance is an issue.
So, that brings us to the second issue: is performance an issue in this case? I prepare a quick gist to compare the speed of np.copy for arrays of various sizes. Unfortunately, my quick test shows that np.copy might be comparable to the speed of the fastest transform methods:

We should do a more careful test to see if copying the array actually causes a noticable slowdown when incorporated into the transform methods. If so, then I suppose we can do something similar to the scipy.fftpack.fft where we provide the "safe and slow" behavior (copying the array) by default, and allow users to select the "fast and crazy" (overwriting the array) if they require higher performance.

pass-by-reference-like

Note, I was a little too fast in requiring all algorithms to numpy.copy() the input image.
Input image destruction only applies to hansenlaw (forward), onion_bordas, and direct (probably, only forward too).

merge - hold off until after the W/E.

A long-weekend in Canberra (public holiday).
hansenlaw is definitely an even column  algorithm.  Here are the transform pair difference plots. Note, incorrect labeling of the y-axis for the pair plots above, the forward transform compares with the projection function G(R) (in hansenlaw labeling), and the inverse transform with the source function f(r).
Inverse Abel transform pairs, difference curves:

Note, the profile3 (== curveA) mse of 1.13e-6 (from 1.07e-6 above, is due to the 1/2-pixel shift being applied, externally to the image, rather than, within hansenlaw to the gradient (not image)).
Forward Abel transform pairs:

I have pulled the scipy.ndimage.interpolate.shift() code from hansenlaw.py since:

It is not the role of a method code to correct an image grid/centre, although, onion_bordas does attempt a correction. However, IMO, it cannot be universal, without considering how the image was centred.
In particular, the user would need to specify the whole-image-shape, and centre method, as separate parameters.
The 1/2-pixel difference between transform of an odd or even image is only noticeable on a spectroscopic scale, and in general the speed distribution scaling would "correct" the difference.

Hence, for spectroscopic accuracy, the input (whole) image should have an even column size. I should make changes to the docs to reflect this. Some, changes have been made.
The even image width for hansenlaw causes a few issues, as it now joins the onion_bordas method, with basex, three_point, two_point, onion_peeling, and direct being odd size. Again, the docs should be examined.
In summary, I think a more careful look at the method codes to check whether the input image is necessarily, mangled, and so whether a .copy() is required. I have reverted the dasch methods. The hansenlaw method applies to an even width (whole) image, grid axis on the left of the pixel, as per the diagram in the discussion #39. As long as the user knows, I don't think this is a big issue. Check the docs for any inconsistencies.

I think a more careful look at the method codes to check whether the input image is necessarily, mangled, and so whether a .copy() is required.

Yes, I think it might be an issue only in a few methods.

It's funny that you bring up the issue of the fft, because scipy.fftpack.fft does have an option overwrite_x,
Unfortunately, my quick test shows that np.copy might be comparable to the speed of the fastest transform methods:

For those where it is an issue, just having an optional 'copy=True' input parameter, might be the simplest way to get around this. This way it could be on by default to be safe, and carefully disabled when performance matters.

@stggh, thanks for the nice comparison!
So, there are two issues here:

As @rth suggests, the issue of transform methods mangling arrays is fairly easy to deal with: they shouldn't mangle the input array unless the user "gives permission" for this to happen via a boolean kwarg. I would probably call this kwarg something like "overwrite_input", or something like this.

The issue of which grid the transform methods use is much more difficult. We are calling this the "even/odd" issue, but it's perhaps better called the "y[0]=0 versus y[0]=0.5" issue (see #39 for nice diagrams by @rth). I was hoping that this issue was effectively resolved, because all of the method (except for Bordas) worked on odd images. But, if we now think that HL works for even images, then we need to think more carefully about how we want to deal with this. Fundamentally, it makes a bit more sense to me to use odd images, since the value at y=0 is specified.

I think that the direct method should be able to work on any grid, even a non-uniform grid, in principle. However, looking at the code, I'm fairly sure that is assumes an even grid. I opened a PR (#213) to change this to an odd grid.
@stggh, can you confirm that HL expects an even image? As you describe it, it sounds like you are just shifting the input by 0.5 pixels. But, to make a comparison with another method, I think that you would need to shift the input by 0.5 pixels, and then either shift the output back to the odd-grid, or make sure that you are plotting using an array of x-coordinates that is on the [0.5, 1.5, 2.5, 3.5 ...] grid.
If it does assume and even image, then we need to decide how to proceed. In general, it seems that PyAbel could provide the best comparisons if all of the methods adopted the same convention. So, one option would be to try to modify the method to use an odd image convention. It seems that Hansen is still a Prof. at Dartmouth. I might send him an e-mail and ask him if the algorithm could be modified in this way. Also, I am curious about the values in table 1, and if there is any incentive to re-calculate them with higher accuracy.

Input image mangling

A quick test reveals only hansenlaw (direction=forward) mangled the array. Now fixed.
No copy flag required, unless changes are made to direct and/or the other methods, to reduce any copy activity.

can you confirm that HL expects an even image
then either shift the output back to the odd-grid, or make sure that you are plotting using an array of x-coordinates that is on the [0.5, 1.5, 2.5, 3.5 ...] grid

This is a good point. I have removed the image ndimage.interpolation.shift(IM, (0, -1/2)) from hansenlaw.py, so that the method is parity independent. The grid adjustment is applied before the transform. The transform is compared directly with the unshifted-source function.

odd parity, image centre mid-pixel = no change to the transform pair function:

n = 100
f = abel.tools.analytical.TransformPair(n=n, profile=3)  # curve A
# inverse Abel transform
hl = abel.hansenlaw.hansenlaw_transform(f.abel, dr=f.dr)
tp = abel.dasch.three_point_transform(f.abel, dr=f.dr)
dr = abel.direct.direct_transform(f.abel, dr=f.dr)

msehl = np.square(hl-f.func).mean()
msetp = np.square(tp-f.func).mean()
msedr = np.square(dr-f.func).mean()

  inverse method     mean-squared error
  --------------     ------------------
    hansenlaw             5.88e-05
    direct                5.37e-05
    three_point           5.93e-09

Note  the hansenlaw transform is pretty similar to the result of direct.
However, see new discussion #213, which intimates that direct assumes an even grid if r is not supplied.

even-parity image = apply `shift(f.abel, -1/2)

fabel = interpolation.shift(f.abel, -1/2)

hl = abel.hansenlaw.hansenlaw_transform(fabel, dr=f.dr)
tp = abel.dasch.three_point_transform(f.abel, dr=f.dr)
dr = abel.direct.direct_transform(f.abel, dr=f.dr)

msehl = np.square(hl-f.func).mean()
msetp = np.square(tp-f.func).mean()
msedr = np.square(dr-f.func).mean()

  inverse method     mean-squared error
  --------------     ------------------
    hansenlaw             1.15e-06
    direct                5.37e-05
    three_point           5.93e-09

Here hansenlaw has much better agreement with the curveA source function:

The mse=1.15e-06 is similar to that stated for Fig. 2 of the H&L paper, 1.33E-6.
However, the figure caption also states 'N=101`!
Similar improvements, occur for even images (n=101) code:

Profile
mse(odd)
mse(even -1/2 shift)

1
5.68e-5
1.25e-6

2
5.48e-5
1.15e-6

3  (curve A)
5.77e-5
1.13e-6

4
(bad defn)
-

5
6.66e-3
4.03e-3

6
9.07e-5
3.07e-6

7
9.58e-5
3.28e-6

8  (curve B)
9.07e-5
3.07e-6

I'm pretty surprised by how sensitive these transforms are to an even/odd grid

Only in terms of a zoomed image, or mse.
The comparison is complicated by the fact that only hansenlaw and direct have forward transforms. I will examine hansenlaw in relation to an odd-direct before making a definitive statement on whether hansenlaw applies to an odd or even image width. i.e. I am still not 100% sure.

@DanHickstein grid bug fix for direct #213 now provides an accurate forward transform, to compare e.g. the Dribinski sample image transform (forward <-> inverse) with hansenlaw. The hansenlaw -1/2 pixel shift, applies only to each row. So, for a 2D image this would be scipy.ndimage.interpolate(IM, (0, -1/2)). This is easier to implement the shift in hansenlaw.py, so that abel.Transform(..., method='hansenlaw') includes the shift for each quadrant processing. I have added a new parameter align_grid=True which applied the shift. For even-width images (center pixel edge) align_grid=False will not apply the shift (intensity interpolation) and also provide a faster transform.

    method    mse-image(transform-source)   mse-speed(transform-source)
  hansenlaw               26.28              1.43e+10
  direct                1394.02              1.93e+10
  three_point             13.56              5.54e+09

code
Subject to the choice of variable name align_grid, and whether the docstring is sensible, I am happy with the compromise(*) that all methods should apply to odd-width (centre pixel grid) images.
(*) `scipy.ndimage.interpolate(), when applied, slows the algorithm

see #39 for nice diagrams by @rth

Hey, I think they may have been my diagrams   ;-)

Hey, I think they may have been my diagrams ;-)

Yes, I looks I took some of your for that issue :)

Yes, I looks I took some of your diagrams for that issue :)

Ha! Well, thanks to both of you for the nice diagrams - @stggh for making them, and @rth for reposting them :)
I wrote a quick test to investigate if HansenLaw suffers from an "even/odd" grid issue. I don't think that just shifting the input grid is quite the right way to go about this. Fortunately, with the analytical methods, we can simply generate a grid that is twice the density that we need, and pass either the even or the odd points to our transform function.
import numpy as np
import matplotlib.pyplot as plt
import abel

n     = 41
r_max = n
sigma = n*0.25

ref  = abel.tools.analytical.GaussianAnalytical(n, r_max, sigma, symmetric=False)
func = ref.func
proj = ref.abel

proj_even = proj[1::2]
proj_odd  = proj[0::2]
proj_shift = scipy.ndimage.interpolation.shift(proj, (-0.5))

r    = ref.r
dr   = ref.dr

r_even = r[1::2]
r_odd  = r[0::2]

fig, axs = plt.subplots(2, 1, figsize=(5,8), sharex=True, sharey=True)

axs[0].plot(r, func, label='Analytical', lw=1)
axs[1].plot(r, func, label='Analytical', lw=1)

hl_even  = abel.hansenlaw.hansenlaw_transform(np.copy(proj_even), dr=2, direction='inverse' , align_grid=False)
hl_odd   = abel.hansenlaw.hansenlaw_transform(np.copy(proj_odd ), dr=2, direction='inverse' , align_grid=False)
tp_odd   = abel.dasch.three_point_transform(proj_odd, dr=2, direction='inverse')
tp_even  = abel.dasch.three_point_transform(proj_even, dr=2, direction='inverse')

axs[0].plot(r_even, hl_even,  'o', label='HL with even grid')
axs[0].plot(r_odd,  hl_odd,   'o', label='HL with odd grid')

axs[1].plot(r_even, tp_even,  'o', label='Three-point with even grid')
axs[1].plot(r_odd,  tp_odd,   'o', label='Three-point with odd grid')

axs[0].legend(frameon=False)
axs[1].legend(frameon=False)

plt.show()

We see that Three-point has a clear preference for the "odd" grid. HL seems to be off for both grids. The fact that a 0.5 pixel shift to the input grid seems to improve the performance of the HL method, point to some "off by one error" somewhere in the code, but I'm not convinced that it's simply an issue with even/odd grids.

@DanHickstein - neat coding!  I will investigate.

Note, that for the odd-size image, align_grid=True, to apply the -1/2 pixel shift. In which case the transform is better:

I agree, even, with align_grid=False should achieve the same result - so, something wrong for the even case.

subtract 1 from r_even
axs[0].plot(r_even-1, hl_even,  'o', label='HL with even grid')`

I have grid confusion ...

Back to our nice PyAbel grid figure, even-column-image (centre is left pixel edge) vs odd-column-image (center is mid-pixel):

Here is my take on the -1/2 pixel shift (x = interpolated values):

It is not a matter of odd, or even sampling of the (Gaussian) function, or whether the image has an odd or even number of pixels. Just the location of the grid on which the image intensity is evaluated.

Hmmm, yeah, so I guess the grid can be defined in terms of the left side of the bin, the center, or the right side. But, the center seems like the most obvious choice.
I wonder if there is just some small error in the indexing in our HL algorithm, and we are just finding different ways to rationalize and compensate for the behavior. Looking again at the code, I had forgotten that the HL algorithm doesn't actually calculate he intensity at the center pixel, but instead just copies the value from the adjacent pixel.

I wonder if there is just some small error in the indexing in our HL algorithm

Agreed! I have been looking, over the past 2 months, but may be missing something obvious.

HL algorithm doesn't actually calculate he intensity at the center pixel

That's because (N-n)/(N-n-1) is ∞ for n=N-1 (index n = 0 at right side). This should apply for both the forward and inverse transform, yet the H&L paper initimates there is only an issue for the inverse transform:

Not sure whether this helps, but displacing the ratio matrix by -1/2, numerator and denominator:
    denom = N - n - 1 - 0.5 # N-n-1 in Hansen & Law
    ratio = (N - n - 0.5)/denom  # (N-n)/(N-n-1) in Hansen & Law

achieves a similar effect to shift(IM, (0, -1/2))

which points to @DanHickstein's "some small error in the indexing in our HL algorithm".

Oh, very interesting!
Is your HL code adapted from code written by H&L, or was it developed by someone else from the equations in the paper? I'm thinking about e-mailing Hansen to see if he has the original code sitting around on some old floppy disk (or tape drive? I was 1 year old in 1985 and don't know what technologies were in use. Hopefully not punch cards.)

HL code adapted from code written by H&L

No, it is my interpretation of what is written in their paper.
A couple of things are inconsistent/uncertain:

(N-n)/(N-n-1) vs n=0, ..., N-1   (forward transform), which would blow up for N-1.
xn+1 = Φn xn + Γn fn
 gn = C xn
The Python code works better with  gn = C  xn+1.
I use  x_0 = 0 for the initial value.

In the very distant past, I recall that we (ANU) did once contact Hansen and received a code (fax?) listing. However, it was not used, as I thought my version of the algorithm was working. :-(
I will be moving offices soon, and in the process, may find it.

1985

Chronology, I received my PhD in 1984!  We had mainframes, Commodore PET's, and HP calculators. ;-)
I will keep playing.

Here is a one-row version of hansenlaw:
import numpy as np
import abel
import matplotlib.pyplot as plt

# Hansen & Law parameters of exponential approximation, Table 1.
h = np.array([0.318, 0.19, 0.35, 0.82, 1.8, 3.9, 8.3, 19.6, 48.3])
lam = np.array([0.0, -2.1, -6.2, -22.4, -92.5, -414.5, -1889.4, -8990.9,
               -47391.1])

def hl(im, dr, shift=0):
    aim = np.zeros_like(im)
    N = len(im)

    deriv = np.gradient(im)

    K = len(h)
    gamma = np.zeros(K)
    x = np.zeros(K)

    for col in np.arange(N-1):

        ratio = (N-col-shift)/(N-col-1-shift)

        phi = ratio**lam

        gamma[0] = -h[0]*np.log(ratio)
        gamma[1:] = h[1:]*(1-phi[1:])/lam[1:]

        x = phi*x + gamma*deriv[N-col-1]
        aim[N-col-1] = x.sum()

    # missing 1st pixel
    aim[0] = aim[1]
    return aim/dr

# main -------------------------
f = abel.tools.analytical.TransformPair(101, 3)

shift = 0.45
him = hl(f.abel, f.dr, shift)

mse = np.square(f.func-him).mean()
print("shift={:.2f}, mse = {:4.2e}".format(shift, mse))

# plot
plt.plot(f.r, f.func, label=r'Curve A $f(r)$')
plt.plot(f.r, him, label='hansenlaw')
plt.legend()
plt.annotate(r'mse={:4.2e}'.format(mse), (0.7, 0.8))
plt.annotate(r'shift={:.2f}'.format(shift), (0.7, 0.75))
plt.xlabel(r'$r$')
plt.ylabel(r'$f(r)$')

plt.savefig('hansenlaw-1row.png', dpi=75)
plt.show()

I will be moving offices soon, and in the process, may find it.

Okay, let me know if you find it, otherwise I am tempted to e-mail Hansen sometime next week and see if we can get the original code. Your code does seem quite close, so I'm sure we can keep using it, we just need to figure out exactly where this heisenbug is located.

Here is a one-row version of hansenlaw

Awesome! For diagnosing this issue, it's probably best to work with the simplest example possible.

Full circle
The ratio correction (N-n-shift)/(N-n-1-shift) is a furphy, it does not produce the same amount of agreement as scipy.ndimage.interpolation.shift(IM, (0, -1/2)).  I have returned the (0, -1/2) pixel shift, if full_image_odd_cols=True (default).  I am open to a better choice for the variable name.
I conclude (again ;-)), hansenlaw is an even width full image transform.

test codes gist

This follow up June '85 paper by Hansen IEEE Trans. Acoust. Speech. Signal Proc. 33, 666 (1985) has a variation on the April '85 hansenlaw method:

we have found that a piecewise linear, or “first-order hold” approximation gives superior
results.

where, B0(β0) and B1(β1) replace Γ(γ) in the top of this discussion formula graphic.
In this June paper, Hansen states the test functions have an even column number:

III. Experiments ...  with both 128 and 512 samples

I am in the process of coding the "first-order hold" forward transform,  which may provide a useful comparison with the (accuracy of the) current method. However, the fact the hansenlaw method needed to be tweaked, may suggest that it will not meet the same level of accuracy we have attained using the Dasch three_point method.
The "first-order hold" approximation has been implement in the fast Hankel transform Matlab code, which lead  to the above paper and this alternative hansenlaw algorithm (for the forward transform).

Hanson published a later correction to the β expressions:

The algorithm slightly improves the forward Abel transform, and no '-1/2' pixel shift needed.

forward method
mean-squared error

hansenlaw
1.55e-06

hansenlaw2
7.25e-07

direct
2.66e-06

(note: the orginal hansenlaw applies -1/2 pixel shift)
NB AFAICT the new method only applies to the forward transform.
hansenlawv2.py

I should add it is possible map the same grid to each formula (see new push to my HansenLaw-fix fork), and so the -1/2-pixel shift  implies a even column width image for the hansenlaw method cf the hansen, above. i.e.
hansenlaw
    for indx,col in zip(n[::-1]-1, n[1:]):  #
        x  = phi[indx][:, None]*x + gamma[indx][:, None]*drive[:, col]
        AIM[:, col] = x.sum(axis=0)

hansen
    for indx, col in zip(n[::-1]-1, n[1:]):
        x  = phi[indx][:, None]*x + B0[indx][:, None]*drive[:, col+1]\
                                  + B1[indx][:, None]*drive[:, col]
        AIM[:, col] = x.sum(axis=0)

Oh, so we missed the "Correction" article! The title of the article says that it's a correction to the Fast Hankel Transform, but it also applies to the Abel transform as well?
And it appears that the correction allows the forward transform to achieve good agreement with the analytical results? Very exciting! Are you passing the hl2 method points on an even or odd grid?
And now you think that it might be possible to incorporate this correction into the inverse transform as well?

correction to the Fast Hankel Transform, but it also applies to the Abel transform as well?

Hansen's second paper defines a more accurate forward Abel transform,  independent of the evaluation of the Hankel transform(*).  The function f(ξ) previously assumed constant across each interval (pixel), in paper I, is now a linear approximation  f(ξ)=p + qξ. This changes the recursion expression from:

to

The subsequent paper III correction redefined β0 and β1 expressions used to evaluate B0, B1 of paper II.

Are you passing the hl2 method points on an even or odd grid?

The linear approximation improves the forward Abel transform, over the hansenlaw (paper I) method, and does not require the source function -1/2 pixel shift. The following  plots compares all the foward transform methods: new hansen (no shift, blue), hansenlaw with -1/2-pixel shift (orange dash),  and 'direct' (green dots), for each of the abel.tools.analaytical.TransformPairs(), mean-square-errors annotated:

All great, except the β0 and β1 expressions apply only for the forward Abel transform.

possible to incorporate this correction into the inverse transform as well?

No, unless we have expressions  β0 and β1 for the inverse transform. We could attempt to evaluate:

using sympy? or query Hansen, our best bet. I will send him something tomorrow, today is the Easter Monday holiday in Oz.
For testing, I have uploaded a new file hansen.py into my HansenLaw-fix fork, called in the usual manner, abel.hansen.hansen_transform(IM, dr)
Alternatively, this gist has a slightly more readable code, applicable for 1d images.
(*) The paper makes some interesting comments on sampling for the FFT

Eric Hansen has kindly outlined the algebra to obtain the gamma (γ) expressions Eq. (16c) and (18) of the Hansen & Law paper, from the state equation (Eq. (9) in the previous comment), so that we may obtain similar expressions for the inverse Abel transform in the first-order hold approximation, β0,1 Eq. (12), of the Fast Hankel paper.  It turns out that Φ in state equation integral evaluates to:

where a=0, 1, 2, depending on the transform direction, and the order of the hold approximation.  The zero-order hold approximation, where the source function is assumed constant across an interval is the hansenlaw transform, while the first-order hold approximation, assumes a iinear variation, "gives superior results" (as quoted from Hansen's Hankel paper).
The bottom line is that it is relatively straightforward to code the first-order hold approximation for both the forward and inverse Abel transforms. The results are excellent, for the Abel transform pairs, here plotted as % difference. Profile 3 is "curveA", blue hansen, yellow direct.
Forward:

Inverse, + green three_point difference:

The annotations are the mean-square error values.
three_point still wins for the speed and accuracy of the inverse Abel transform, but the Hansen algorithm is a winner for the forward transform (cf direct).
@DanHickstein Q: discussion concerning PyAbel structure:
Two new files have been added to this PR (my HansenLaw-fix fork, hansen1d.py, a single row or 1D-function Abel transform, and hansen.py the 2D vectorized version.  The 1d-version can be deleted before the merge.
hansen.py replaces hansenlaw.py, with the parameter hold_order=0. There is little reason to use this approximation, given superiority of the first-order hold (default hold_order=1).
I propose that hansenlaw.py and associated links be dropped, to be replace with hansen.py.  The transform call would be: abel.hansen.hasen_transform(image, dr=1, direction='inverse', hold_order=1). i.e. similar to the Dasch methods abel.dasch.three_point_transform(). Here, it would be the Hansen methods. Note, I realise that this would break the PyAbel API, but it is pointless to carry hansenlaw.py.  I also do not feel we should simply call the new algorithm hansenlaw, because Law was not part of the first-order hold approximation. If need be we could maintain a functional link hansenlaw.hansenlaw_transform() to hansen.hansen_transform() to maintain API compatibility.
The hansen.py code file at the moment is missing its doc string, which I will add back.  Execution speed is good, but there is always room for improvement. In particular, at the moment a global variable is used so that phi is not recomputed. I am open to suggestions.
[etalon:/home/stg121]$ ipython3
Python 3.6.5 |Anaconda custom (64-bit)| (default, Mar 29 2018, 18:21:58)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import abel

In [2]: im = abel.tools.analytical.SampleImage(1001)

In [3]: %timeit abel.hansen.hansen_transform(im.image)
81.4 ms ± 1.4 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [4]: %timeit abel.hansenlaw.hansenlaw_transform(im.image)
127 ms ± 949 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [5]: %timeit abel.dasch.three_point_transform(im.image)
14.7 ms ± 248 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

@DanHickstein if it is ok to swap hansen.py (with doc string) for hansenlaw.py, I will update all the calls, the examples, and readthedocs files.

Hmm, a 2D image is not correctly mapped in axis=0. I will look into my programming error ...

All now working. I forgot to add the sum axis aim[:, col] = x.sum(axis=0) in the conversion back to 2D.
BTW, hansen_transform(image, hold_order=0) == hansenlaw_transform(image, full_image_odd_cols=False). i.e. for no -1/2 pixel shift => hansen;aw coding was correct. The additional -1/2 pixel shift (for the zero-order hold approximation) improved the transform-pair fit, but not as much as an (unshifted) first-order hold.

snowballed into a fundamental better ... HL transform method

Yes, until the first-order hold approximation came along, I was becoming concerned that this time-consuming code-restructure process was gaining very little.

"half-pixel shift" issue has effectively been solved through the "first order hold" approach

Yes.

ask Eric

Will do.

fix the global variable issue

Work in progress.
@DanHickstein thanks for your great help in stepping through the intricacies of the hansen(law) method, with me, and in particular, initiating the contact with Eric Hansen, who kindly explained the detail of his 1985 algorithms to us.

Eric is a fast responder:

I am completely OK with keeping it as "hansenlaw" in the API, etc, and sharing the credit with Phaih-Lan.  Just cite both articles in your paper and in the comments for the code.
Thanks for thinking of it.
Eric

Awesome! That saves us some re-organizing :)

@stggh, Thanks again for all of your great work on this PR!
What is the current status? Are there outstanding issues? Are you waiting for code review?

Summary of this PR
This long PR and #206 started as the annoying difference between the hansenlaw method for the Curve A test case and what was published in the Hansen and Law paper:

It turned out that most of the difference was due to a grid indexing error.  However, the mean-square-error mse remained larger than stated on the Hansen and Law Figure:

  forward method     mean-squared error
  --------------     ------------------
    hansenlaw             5.89e-05
    direct                2.69e-06

  inverse method     mean-squared error
  --------------     ------------------
    hansenlaw             3.26e-06
    direct                2.59e-06
    three_point           5.93e-09

This difference could be ameliorated by a horizontal pixel shift of the driving function (gradient for an inverse transform, or the raw image, for the forward transform), which also shifts the transform curve. For CurveA (TransformPair(profile=3)) a pixel shift of -0.5 gives a good fit to the transform pair:

python CurveA-mse.py

  forward method     mean-squared error
  --------------     ------------------
    hansenlaw             1.57e-06
    direct                2.69e-06

  inverse method     mean-squared error
  --------------     ------------------
    hansenlaw             3.26e-06
    direct                2.59e-06
    three_point           5.93e-09

The -1/2 pixel shift hinted at a coding indexing error, and different sub-pixel shift values worked better for other test-curves, CurveB (TransformPair(profile=8)):

A sub_pixel_shift parameter was not very satisfying, because its value is not well determined.
In the mean time, a second Hansen paper was uncovered (discguised by the Hankel transform title), and Eric Hansen kindly helped us understand it.  The original hansenlaw algorithm assumed the driving function is constant across a grid interval (a pixel), called staircase or zero-order hold approximation.  The second paper, extended this to linear function, the first-order hold approximation.
The first-order algorithm does a great job for the transform pair test profiles:

mse = 7.24e-7

(not as good as the pixel shift case)
It turns out that the first-order hold algorithm does not replace the zero-order hold algorithm. For real image data, the zero-hold method, with sub-pixel-shift, the alignment is better:

This makes sense in that image data consists of pixels and so the zero-order hold for the driving function is correct. While for the transform pair curves the first-order hold is better.
The hansenlaw PyAbel method has been re-written to provide for the traditional zero-order method hold_order=0, with sub-pixel-shift sub_pixel_shift=-0.35 (default) and the first-order method hold_order=1, in this case no pixel shift is applied (sub_pixel_shift is ignored.
In conclusion, for the hansenlaw method with image data, we are stuck with the zero-order method and a sub-pixel-shift, to better align the transform data. For non pixelated data , the first-order method may be better. YMMV
In the end, the three_point provides the faster more accurate inverse Abel transform, however, for forward transforms one must use either hansenlaw or direct.

Great summary @stggh! And great work on this PR.
I merged #214, so you can now incorporate that into this PR.
I will proofread the code this weekend and then we can get this merged.

rebase, brought a great deal of code change baggage ...
Last and most useful Summary

A quick screenshot showing how the agreement of the HL transform with the analytical result is greatly improved by this PR:

BTW, did you want to include the reference to the Gascooke paper in the docstring for the circularize method? (discussed in https://github.com/PyAbel/PyAbel/issues/210) https://aip.scitation.org/doi/10.1063/1.4981024

Thanks @DanHickstein, all good suggestions.

include the reference to the Gascooke paper

Already present in this PR in hansenlaw.rst

HL transform with the analytical result

Is this for the case hold_order=1 ?

include the reference ... in the docstring for circularize_image

Sorry, for my confusion. The citation is already present in abel.tools.circularize.circularize_image?
In [1]: import abel

In [2]: abel.tools.circularize.circularize_image?

Signature: abel.tools.circularize.circularize_image(IM, method='lsq', center=Non
e, radial_range=None, dr=0.5, dt=0.5, smooth=0, ref_angle=None, inverse=False, r
eturn_correction=False)
Docstring:
Corrects image distortion on the basis that the structure should be
circular.

This is a simplified radial scaling version of the algorithm described in
`J. R. Gascooke and S. T. Gibson and W. D. Lawrance: 'A "circularisation"
method to repair deformations and determine the centre of velocity map
images' J. Chem. Phys. 147, 013924 (2017).
<https://dx.doi.org/10.1063/1.4981024>`_

default behavior, which is hold_order = 0

Yes, the deviation, for this analytical function, should/will be less with hold_order=1

Yes numpy.array(object, dtype=None, copy=True, order='K', subok=False, ndmin=0), the default is copy=True. array is slightly faster (although, I have misplaced the relevant web url).

Are we waiting on anything to merge this?

No, just the final nod! I am happy to press the green button.

Hit it! :)


demo: remove unnecessary chains secret

To push signatures and attestations to an OCI registry, Chains will use
whichever secret was used to push the corresponding image. It is not
necessary to create the same secret in the tekton-chains namespace.
Further proof of this is the fact that the demo script was linking the
secret to the pipeline ServiceAccount and the tekton-chains-controller
uses the tekton-chains-controller ServiceAccount.
Signed-off-by: Luiz Carvalho<EMAIL_ADDRESS>

I think the comment is off, but this is required by the hack/chains/release-pipeline-with-ec-demo.sh script:
https://github.com/redhat-appstudio/infra-deployments/blob/a1f782695bfe27a59c6a48adf81836aa2966338c/hack/chains/release-pipeline-with-ec-demo.sh#L8-L11

Though, incidentally, that pipeline ServiceAccount appears with a delay so first time the end-to-end-demo.sh is invoked that bit might fail. We might need to wait for it to be created by the Tekton operator before we proceed.

/approve


How to return PathTemplate with complex resource identifiers?

I see you added support for complex resource identifiers with #125. However, I can't seem to figure out how to correctly return the instantiated template with populated variable values. For example, let's say I have a template customers/{customer_id}/adGroupAds/{ad_group_id}~{ad_id}. I can create the PathTemplate, which recognizes the ~ character as a separator. However, when I call the instantiate(String... keysAndValues) method like so PATH_TEMPLATE.instantiate("customer_id", customerId, "ad_group_id", adGroupId, "ad_id", adId), the string returned does not have the ~ separator. The String that is returned is customers/1234/adGroupAds/5678/1011 instead of the expected customers/1234/adGroupAds/5678~1011.
Is there a way to do this so that the resulting string still contains the complex resource identifier? Or did #125 not address this?

Could you please link to the instantiate method mentioned above?

Here is the link.

This causes our the *Name.java classes to generate incorrect resource names when non / separators are in the URL pattern. It can parse them OK, but when you convert back to a String, the underlying PathTemplate doesn't remember the separator character; it assumes all separators are /s:
An example below:
String resourceNmae = "customers/123456789/adGroupAds/9876565423~082734345";
System.out.printf("input resource name: %s\n", resourceNmae);
AdGroupAdName n = AdGroupAdName.parse(resourceNmae);
System.out.printf("output resource name: %s\n", n.toString());

Produces:
input resource name: customers/123456789/adGroupAds/9876565423~082734345
output resource name: customers/123456789/adGroupAds/9876565423/082734345

The resource name gets mangled on the way out and is unusable now.

Hey folks, would it be possible to get this fixed over the next week please? We have another large body of work which depends on this feature, with a hard deadline in mid-July.

Please feel free to try out PR #159.

Just tested, and it looks like it works. Thanks for the quick fix!


Move to using github actions for all test runs and linting

This removes TravisCI from our pipeline and moves all the tests to Github, as a bonus we now check otp 21&22, with elixir 1.9 & 1.10.

Can we get an extra commit to test the dialyzer PLT caching, please and thanks?


Minecraft 1.12.1 server crash

hi,
my server crash since i used the new version.here is the crash report: https://pastebin.com/6rzQNFcx

use the issue template.


Register

@JuliaRegistrator register()

Error while trying to register: "Pre-release version not allowed"

@JuliaRegistrator register()

Registration pull request created: JuliaRegistries/General/99241
Tip: Release Notes
Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.
@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.
Tagging
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v1.0.0 -m "<description of version>" e140ad741442dd0dea642659c2b0e8027a1d9dc9
git push origin v1.0.0

Registration pull request created: JuliaRegistries/General/110460
Tip: Release Notes
Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.
@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.
Tagging
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v1.0.0 -m "<description of version>" 32810d859332ef4db011c63bf449e9371e1abd3c
git push origin v1.0.0

@JuliaRegistrator register()
Release notes:
The first release

The first release of the functions
More to come in the subsequent releases

Registration pull request created: JuliaRegistries/General/110461
Tagging
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v1.0.0 -m "<description of version>" 32810d859332ef4db011c63bf449e9371e1abd3c
git push origin v1.0.0

@JuliaRegistrator register()
Release notes:
The first release
The first release of the functions
More to come in the subsequent releases

Registration pull request updated: JuliaRegistries/General/110461
Tagging
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v1.0.0 -m "<description of version>" b694be29a8a5ebfce4aa9bd1a67d8aa1001aa43b
git push origin v1.0.0

Package registered.


Update DataService interfaces

Description
According to discussion, upodate the DataService interop APIs with 2 changes:

SendEnhancedResultData returns a unique int cookie value for each Sarif log sent.
Add a new CloseEnhancedResultData accept the cookie value as a parameter to cleanup tags/adornments and close Sarif explorer window.

cc @kylereedmsft, the interop API changes


Fix Ledger Disconnection Issue on OSX

This PR fixes a common issue where Ledger disconnects from lumd during signing, resulting in failure (often repeatedly).
This fix has been successful in increasing the stability of signing on Juno and other cosmos-sdk chains.

For contributor use:

[X] Targeted PR against correct branch (see CONTRIBUTING.md)
[X] Re-reviewed Files changed in the Github PR explorer

For admin use:

[ ] Added appropriate labels to PR (ex. WIP, R4R, docs, etc)
[ ] Reviewers assigned
[ ] Squashed all commits, uses message "Merge pull request #XYZ: [title]"

LGTM! Will merge this right after the other pending PR.


Click a link that doesn't have a class name ?

Is it possible to click a link that doesn't have a class name like this ?
The Link
I know you can do
bow.Click("a.new")
Like in the docs which will find any anchor with the class "new" but how do we click a link by just if it has a  anchor tag and some text like "The Link" ?
Click wants a selector string. Is there any way to get that from bow.Find ? I can get all the links with
bow.Find("a").Each(func(_ int, s *goquery.Selection) {
        if s.Text() == "The Link" {
            //how do i click on the link here ?
        }
}

But then bow.Click wants a selection string and I have a goquery.Selection. How do I click the link once I've found it ?

At the moment it does not click links, just extract the URL string from the href attribute and open it using private methods.
You can extract the URLs the same way
bow.Find("a").Each(func(I int, sel *goquery.Selection){
  var src string
  src, ok := sel.Attr("href")

  // open the link
  // look at the click method:
  //   bow.httpGet ...
})

https://github.com/headzoo/surf/blob/master/browser/browser.go#L346

This helped me out thanks .


Implement using a Composer

I don't like having to put in bits and bobs in Startup.cs - why not implement this using a composer? Whether or not to add the provider could be read from config.

Implemented in #27

Will be in v2.0.0


Dev server does not load on http://localhost:3000/hello-astro/

Hi, thanks for the creation of the astro template, I had a chance to try running this with yarn run dev, but the site does not load from http://localhost:3000/hello-astro/
The Issue
If I try the root path, it will return the expected 404 page:

However, if I navigate to http://localhost:3000/hello-astro/ it will just keep loading indefinitely with no response:

Other Findings

I tried run a different astro dev server with a different project under the same http://localhost:3000/ , seems to work perfectly. The issue seems to be specific to hello-astro.
In hello-astro, yarn build + yarn preview works flawlessly 🎉

Was there any other steps needed before running yarn run dev?
Any guidance would be much appreciated, thank you.

Hello - thanks for raising this issue. As I noted in the release notes for v2.1.0, I am also encountering this issue. Not quite sure why, I think it is a problem with Astro. I am still investigating, and will update if I find any solutions. If you have any ideas, let me know.

I've traced the problem to the following line:
const posts = await getCollection('blog')

This is just a vanilla call to grab a list of blog posts in Astro v2 and it works in build but not dev. There is no easy workaround, but I will investigate further.
For what it's worth, Release v1.1.0 does not have this problem, only the v2.x releases which use content collection.

I have now submitted a bug report to astro documenting this issue:
https://github.com/withastro/astro/issues/6173

Looks like we did encountered a real issue with circular dependencies when calling getCollection(), thanks for digging through this and raise it to the astro team.

I've traced the problem to the following line:
const posts = await getCollection('blog')

This is just a vanilla call to grab a list of blog posts in Astro v2 and it works in build but not dev. There is no easy workaround, but I will investigate further.
For what it's worth, Release v1.1.0 does not have this problem, only the v2.x releases which use content collection.

May I know how you manage to trace back to this line of code? Is there certain mode to run yarn run dev that can give us the pointers to help troubleshoot astro when it hangs like this? My browser wasn't loading anything so devTool wasn't helpful.

May I know how you manage to trace back to this line of code? Is there certain mode to run yarn run dev that can give us the pointers to help troubleshoot astro when it hangs like this? My browser wasn't loading anything so devTool wasn't helpful.

LOL I just did it the old school way, I kept commenting out blocks until I found the problem. I was suspecting getCollection from the start, so it didn't take long.

This issue is now resolved with Release v2.3.0 - astro dev works again.
Can you confirm this is the case?

This issue is now resolved with Release v2.3.0 - astro dev works again.
Can you confirm this is the case?

Confirmed release v2.3.0 with astro dev working again. Thanks!


[IZPACK-1280] UserInputPanel in console installers - not working dynamic evaluation of field conditions triggered by user input at one and the same panel

Provided the following example definitions:
install.xml:
<conditions>
    <condition id="RemoteInstall" type="variable">
      <name>install_method</name>
      <value>remote</value>
    </condition>
</conditions>

UserInputSpec.xml resource:
<panels>
  <panel id="panel.test">
    <field type="radio" variable="install_method">
      <spec>
        <choice txt="local" value="local" id="text.installmethod.local" />
        <choice txt="remote" value="remote" id="text.installmethod.remote" set="true" />
      </spec>
    </field>
    <field type="space"/>
    <field type="dir" variable="import_path" conditionid="!RemoteInstall">
      <spec txt="Import directory:" set="$INSTALL_PATH${FILE_SEPARATOR}server${FILE_SEPARATOR}import" id="text.import.path" size="20" />
    </field>
    <field type="text" variable="upload_host" conditionid="RemoteInstall">
      <spec txt="Host:" id="text.upload_host" set="localhost" size="30" />
    </field>
    <field type="space" conditionid="RemoteInstall" />
    <field type="text" variable="upload_port" conditionid="RemoteInstall">
      <spec txt="Port:" id="text.upload_port" set="8080" size="30" />
    </field>
  </panel>

  ...

</panels>

According to the definitions above, the condition RemoteInstall is set according to the radiobutton field for setting the variable "install_method" and the following fields on the same panel have to be displayed according to the RemoteInstall condition.
This works fine if the installer is started in GUI (Swing) mode, but the condition is not dynamically evaluated in console installation mode. There is just a random condition state used from the moment the installer has been started.

IzPack project » izpack #854 SUCCESS
This pull request looks good
(what's this?)


dont remove tracker on location stop too early

cc @joshblum if you have a spare second to confirm this is ok


Apostrophe breaks syntax highlighting for Ini files

VSCode Version: Code 1.10.1 (653f8733dd5a5c43d66d7168b4701f94d72b62e5, 2017-03-02T00:27:09.350Z)
OS Version: Darwin x64 15.6.0
Extensions:

Extension
Author
Version

python
donjayamanne
0.5.9

vim
vscodevim
0.5.3

Steps to Reproduce:

Create an ini file where a text value contains an apostrophe in the middle, e.g.:

[Sample]
SomeKey=Someone's Value
SomeOtherKey=42

Note that from the apostrophe onward, the rest of the file is considered a string.

Different ini implementations have their own interpretations of values. E.g., Windows does not consider the apostrophes of a quoted value part of that value. I'm not aware of the exact rules, but at least the current highlighting behavior is wrong. I think it would be better to stick to the universally valid rules – sections, keys and values – and not to consider apostrophes special at all.

This issue was moved to textmate/ini.tmbundle#3


Skip hanging tests in test_basic

This PR fixes #351. There are some difference between sdc.jit and numba.jit:

sdc.jit do not set parallel=True.

One test in old pipeline - test_str_split_filter - fails because #321.

Hello @PokhodenkoSA! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file sdc/tests/test_hiframes.py:

Line 47:36: E127 continuation line over-indented for visual indent

Hello @PokhodenkoSA! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file sdc/tests/test_hiframes.py:

Line 47:36: E127 continuation line over-indented for visual indent

Could you fix the indentations issue?

Could you fix the indentations issue?

Not in this PR. This code is inline with other lines before.
I think introducing new style is better with one big PR with autostyling tool like black. Otherwise it will confuse people.


feat: validate entity model against database

Adds validation of the entity model that is used against the actual database. The current validation only adds a check for the key that is used for each entity, but the setup could be used for additional validations in the future.
The check for the key that is set on an entity verifies that:

The key of the entity corresponds exactly with the columns of the primary key of the table
OR the key of the entity corresponds exactly with the columns of another unique not null-filtered index on the table.

This type of validation is not really supported by Entity Framework itself. The built-in model validation is intended to only validate the model, and not whether the model corresponds with the database. The support that is added in this PR is therefore limited:

The model validator relies on a singleton SpannerModelValidationConnectionProvider that provides a connection to the database.
If the application uses multiple different databases, the connection provider will detect this, and stop providing a connection, as there is no hard link between the moment a model validation occurs and when a connection is given to the connection provider through the UseSpanner method. This effectively disables model validation against the database for the remainder of the lifetime of the application. The normal model validation that is built into EF Core will still work.
Model validation against the database can also be manually disabled by calling SpannerModelValidationConnectionProvider.Instance.EnableDatabaseModelValidation(true).
Model validation against the database is automatically disabled when running migrations, as during a migration there might be differences between the entity model and the database.
Model validation against the database is a relatively heavy operation. It is however only executed once for each model during the lifetime of the application.

@laljikanjareeya The specific error that you were getting was logical (database does not exist), but specifying a valid database would also fail. I've added the design-time migrations operations to the list of methods that should skip validation of the context model against the actual database. PTAL.


Dialog Focus a different element on open?

Currently dialogs focus the first element on open. Is there a way to focus the second element instead for example?

Not out of the box, it just looks for the first focusable element. As a workaround I'm sure you can re-arrange the DOM, or or focus the element you want in the onOpen callback.


Deploy the stable-diffusion-webui On the Akash Network by Fogmeta Labs

Deploy the stable-diffusion-webui On the Akash Network by Fogmeta Labs

Hello, could you include the Dockerfile you used and rename the file "READNE.md" to "README.md"?

No problem, I will submit my docker file @zJuuu

@zJuuu Done

@brewsterdrinkwater
There is a video, a good description and necessary links.
Looks ready to merge.

@hengdingy Since you are using the custom image sxk1633/nvidia-cuda:11.4.0 in your Dockerfile, you also need to provide Dockerfile image  sxk1633/nvidia-cuda:11.4.0 , so that we can make sure it is secure.

@Dimokus88 I have change it to the default image, it also works.  The custom image is convenient to test in the local machine.
FROM library/ubuntu:20.04

@hengdingy Modify the root README.md ( https://github.com/akash-network/awesome-akash/blob/master/README.md ) of the awesome-akash repository to include a link to your deployment and you're ready to merge.

Ready to merge @brewsterdrinkwater !

@Dimokus88 I have updated the root readme, and it works.

@hengdingy ,you have duplicate data, fix it as soon as possible.

@Dimokus88 sorry for late, I have fix it. thanks again

Ready to merge @brewsterdrinkwater !


Fix workspace grammatical error

This pull request is about fixing a grammatical error on the Workspaces documentation page.

Thanks!


Calendar breaks under Angular 8 served in prod

I use the plugin in an Angular 8 application, and on building/serving the application with the --prod flag, the calendar breaks.
Steps to reproduce the behavior:

Make an Angular 8 application.
Add calendar to it in some way.
Serve using ng serve --prod true
See error

Expected behavior
This is what the calendar looks like without --prod:

This is what it looks like with --prod

I've been able to consistently replicate the issue over all versions of the library, including the ones that used moment.js instead of dayjs. I've tried different styles for importing:
import { Dayjs } from 'dayjs';
import * as dayjs from 'dayjs';
import dayjs from 'dayjs';

and I've added "allowSyntheticDefaultImports": true to my tsconfig.json as well. Nothing helps.

Hi @szala12
Thanks for finding out this issue. Can you please provide some example codesandbox also can you please confirm if this happens with ng build too? ng serve is not supposed to be ideally run with --prod true
Regarding older versions yes those are for angular-2 only. I doubt if they will work with angular 4 or not. So I dont recommend 1.x versions in angular 6 onwards.

My bad, I should've mentioned it. I came across it after running ng build in prod, that's how it came up in the first place. The ng serve --prod true is only to help replicate on local. I'm not entirely sure how to replicate a prod issue in sandbox, so if you could help me a little with that, that'd be great.
And yeah, we got the issue with<EMAIL_ADDRESS>as well. I'm not completely sure here, but it looks like whatever webpack does when we build or serve with --prod true, it breaks the import for your included dayjs library, the one in the node_modules directory inside your package.

Yeah may be it will be difficult to make production bundle in codesandbox. But condesandbox can be used to sync with your github repo. https://codesandbox.io/s/github.
If you can create a github repo backed sandbox. It will help me to debug and see it works in non-prod bundles.

@szala12 I was able to reproduce the problem.
Here are my findings
I guess is angulars build optimizer is removing dayjs extentions assuming they are not used. Seems like dayjs or my package needed to marked as not sideeffectfree
try setting "buildOptimizer": false in your angular.json
Here is the difference in bundles sizes
With build optimizer enabled

without build optimizer

As you can see the main bundle jumps from 451 to 586. Which is significant difference. I ll investigate more and let you know. Thanks for finding out the issue.

@technikhil314 Hey. I am terribly sorry about the delay. As weird as this sounds to hear, I've logged this bug on behalf of a colleague who doesn't happen to be on GitHub and is not keen on joining, so they decided to move ahead with another library. I was planning on replicating it this weekend, with it being a holiday proper.

No worries. As I guessed correctly I missed to mark my package to have side effects on dayjs since I am using some dayjs extentions. The next version I will release tomorrow will have it fixed. Thanks.

This is fixed in v2.1.1


Retry does not work for TimeoutException

I use Greenwich.release.
I have the following test:
    @Test
    public void shouldRetryTimeout() {
        wireMockServer.stubFor(get(urlEqualTo("/service-with-retry/request"))
                .willReturn(ok().withFixedDelay(110)));

        RestAssured
                .given()
                .when()
                .get("/service-with-retry/request")
                .prettyPeek()
                .then()
                .assertThat()
                .statusCode(GATEWAY_TIMEOUT.value())
        ;

        wireMockServer.verify(4, getRequestedFor(urlEqualTo("/service-with-retry/request")));
    }

and the following configuration:
service-with-retry:
  ribbon:
    NIWSServerListClassName: com.playtika.services.gateway.functional.configuration.WiremockServerList

spring:
  cloud:
    gateway:

      httpclient:
        connect-timeout: 100
        response-timeout: 100ms

      routes:
      - id: service-with-retry
        uri: lb://service-with-retry
        filters:
        - name: Retry
          args:
            retries: 3
            statuses: INTERNAL_SERVER_ERROR, BAD_GATEWAY, GATEWAY_TIMEOUT
        predicates:
        - Path=/service-with-retry/{segment}

Test fails because only one request to service-with-retry is sent, no retries are made.
In logs I see:
2019-02-06 17:56:39.590 TRACE 38270 --- [ctor-http-nio-2] o.s.c.g.f.LoadBalancerClientFilter       : LoadBalancerClientFilter url before: lb://service-with-retry/service-with-retry/request
2019-02-06 17:56:39.598 TRACE 38270 --- [ctor-http-nio-2] o.s.c.g.f.LoadBalancerClientFilter       : LoadBalancerClientFilter url chosen: http://localhost:8997/service-with-retry/request
2019-02-06 17:56:39.764 TRACE 38270 --- [     parallel-1] o.s.c.g.f.f.RetryGatewayFilterFactory    : setting new iteration in attr 0
2019-02-06 17:56:39.765 TRACE 38270 --- [     parallel-1] o.s.c.g.f.f.RetryGatewayFilterFactory    : exceedsMaxIterations false, iteration 0, configured retries 3
2019-02-06 17:56:39.765 TRACE 38270 --- [     parallel-1] o.s.c.g.f.f.RetryGatewayFilterFactory    : exception is not retryable org.springframework.web.server.ResponseStatusException, configured exceptions

I suppose the problem is in retryContextPredicate in RetryGatewayFilterFactory. TimeoutException is a cause of ResponseStatusException. As I understand for retry to work correctly in this case we need to traverse causes of exceptions as well.

Please also note that RetryGatewayFilterFactory has broken logging:
trace("exception is not retryable %s, configured exceptions",
context.exception().getClass().getName(), retryConfig.getExceptions());

that is why configured exceptions are not shown in logs.

Closed via f17cf9e


walrus operator :=

In new versions of python you can do while foo := some_function(). In C, this is very common, e.g. while ((c = getchar()) != EOF).
Jou should also have a := operator. This is even more important for Jou than for Python, because Jou aims to be similar to C.

Yes please! I like how python uses parentheses for it to show it is a complete and separate statement. One thing I'd make possible in Jou is allow it to set attributes (I.E. while ((self.foo := bar) == bar): ...).


[Bug] undefined method `spm_pkg'

What happened?
I installed: 'gem install cocoapods-spm' but spm_pkg undefined.
[!] Invalid Podfile file: undefined method `spm_pkg' for #Pod::Podfile:0x000000011d9e4198.
platform :ios, '13.0'

def capacitor_pods
  # Comment the next line if you're not using Swift and don't want to use dynamic frameworks
  use_frameworks!
  pod 'Capacitor', :path => '../node_modules/@capacitor/ios'
  pod 'CapacitorCordova', :path => '../node_modules/@capacitor/ios'
end

target 'Plugin' do
  capacitor_pods
  spm_pkg "NearbyConnections", :url => "https://github.com/google/nearby.git", :branch => "main"
end

target 'PluginTests' do
  capacitor_pods
end

CocoaPods environment
No response
Anything else?
No response

@Droppix Thanks for reaching out!
Basically, the error indicates that the plugin was not installed correctly.
May I know how you install and run CocoaPods with the plugin?
If you're using bundler (ie. having Gemfile in the project), then make sure:

gem "cocoapods-spm" is added to Gemfile
run bundle exec pod install instead of pod install

Thanks for your response.
Indeed, I had an old version of ruby (2.6.6) installed, the update to ruby 3.3.0 works well!
I tried to integrate the NearbyConnections SPM (google: https://github.com/google/nearby.git) which depends on "NearByCoreAdapter" (integrated in the NearbyConnections project), and I get the error:
Missing required module 'NearbyCoreAdapter'.
Whereas if I integrate the NearbyConnections SPM directly (excluding Cocoapods), it works fine.
If you have a suggestion, I'd love to hear it...

plugin "cocoapods-spm"

def capacitor_pods
  use_frameworks!
  pod 'Capacitor', :path => '../node_modules/@capacitor/ios'
  pod 'CapacitorCordova', :path => '../node_modules/@capacitor/ios'
end

target 'Plugin' do
  capacitor_pods
  spm_pkg "NearbyConnections", :git => "https://github.com/google/nearby.git", :branch => "main"
end

target 'PluginTests' do
  capacitor_pods
end

@Droppix I would suggest closing the current one and creating a follow-up issue (related to NearbyCoreAdapter), to keep it relevant. Is it ok?

Created https://github.com/trinhngocthuyen/cocoapods-spm/issues/39 for reference.


Backend conference attendees

To keep conference attendences consistent in case of a user closing the digital classroom tab, two scenarios come to mind:

The closing of the classroom-tab is immediately noted in the backend. It is possible to immediately remove the user from the datastructures holding information about the attending users. When the user reopens the tab it is not possible to reenter the rooms, since information about open tabs is not obtainable.
Pro: Easy to implement. Light on traffic.
Con: Not 100% accurat. User in conference is not visible to others on refresh, unless he rejoins the BBB conferences via new classroom tab. Better than leaving the users within the backends attendence list.

When a user connects / reconnects to the backend, the attendence of the conferences known to the backend is requested from the BBB server.
Pro: 100% Accurat.
Con: Possibly heavy traffic to BBB server & longer response times on connection. A little harder to implement.

Any thoughts on this?

I would like to add a third alternative:
3. Use BBB Webhook to get notified by the BBB-Server, when a user joins or leaves.
Pro: 100% Accurat and Light on traffic
Con: The BBB-Server needs to have bbb-webhooks installed

Unfortunately the PILOS BBB-Backend does not support the webhook API.
I will try to support it as an alternative.
Thanks

I will try to support it as an alternative.

Yes, that would be my suggestion as well. We use option 3 if possible and option 1 otherwise.
I find option 1 better than option 2 because the advantages that option 2 offers (100% Accurat) do not justify the disadvantages of this option.

PILOS contacts the BBB backend every minute. Mr. Kammer suggested to do that as well.
This could be combined with option 1:
We delete a leaving user from all conferences within the classroom and readd him if the BBB backend says, that he is still attending a conference. This is possible, since there is no strong consistency enforced and a user is only identified via its unique id, which is identical in gateway, classroom and BBB backend.


Mvs merge master to r2.0

Description of changes:
Merging master into R2.0
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

Merging #82 into r2.0 will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             r2.0      #82   +/-   ##
=======================================
  Coverage   97.72%   97.72%
=======================================
  Files          19       19
  Lines         659      659
  Branches       48       48
=======================================
  Hits          644      644
  Misses          9        9
  Partials        6        6

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5ce5a24...31d91fb. Read the comment docs.


[Bug] Undefined symbol: _SWIG_csharp_string_size, _SWIG_csharp_string_str, _SWIG_csharp_string_to_c

Description
I've upgraded my Unity(6000.0.25f1) Projects's Firebase SDK to 11.4.0
When I try to Archieve to distribute it it gives me error:

When I searched for:
- _SWIG_csharp_string_size
- _SWIG_csharp_string_str
- _SWIG_csharp_string_to_c
they are all used by Firebase products:

What I've tried:

Downgrade Xcode
pod install
update cocoapods
Add GameKit.framework

Reproducing the issue
No response
Firebase Unity SDK Version
12.4.0
Unity editor version
6000.0.25f1
Installation Method
.unitypackage
Problematic Firebase Component(s)
Analytics, Crashlytics, Messaging, Remote Config
Other Firebase Component(s) in use
No response
Additional SDKs you are using
AdMob
Targeted Platform(s)
Apple Platforms
Unity editor platform
Mac
Scripting Runtime
IL2CPP
Release Distribution Type
Pre-built SDK from https://firebase.google.com/download/unity
Relevant Log Output
Undefined symbols for architecture arm64:
  "_SWIG_csharp_string_size", referenced from:
      _SWIGStringWithLengthHelper_SWIG_csharp_string_size_m8F346145417F2C9491498D803A31784BD19013D7 in libGameAssembly.a[22](vu34smn0bqak.o)
      _SWIGStringWithLengthHelper_SWIG_c_to_csharp_string_mE1E23481DA0D5785F7A93B2E2198C097C57B3D5C in libGameAssembly.a[22](vu34smn0bqak.o)
      _SWIGStringWithLengthHelper_SWIG_csharp_string_size_m16FB33E0B820555284400210117E7A8A52B532FA in libGameAssembly.a[23](mz5zwj743dgj.o)
      _SWIGStringWithLengthHelper_SWIG_c_to_csharp_string_m84428208EED10043F30F6C512792615AE2FAC037 in libGameAssembly.a[23](mz5zwj743dgj.o)
      _SWIGStringWithLengthHelper_SWIG_csharp_string_size_m1948ACEBEE67E05211F90DE17590914433C663CE in libGameAssembly.a[25](q0ur51tkyt20.o)
      _SWIGStringWithLengthHelper_SWIG_c_to_csharp_string_mC21BFB1DCB3E5CD72C79A7899554A8182198C5CF in libGameAssembly.a[25](q0ur51tkyt20.o)
      _SWIGStringWithLengthHelper_SWIG_csharp_string_size_m424013BA371C1172D562D03620F1633DB2259C20 in libGameAssembly.a[26](xt9xt4swnlqe.o)
      ...
  "_SWIG_csharp_string_str", referenced from:
      _SWIGStringWithLengthHelper_SWIG_csharp_string_str_m2D07EC24FFFF0DDD94E1F45C362A788592713A3A in libGameAssembly.a[22](vu34smn0bqak.o)
      _SWIGStringWithLengthHelper_SWIG_c_to_csharp_string_mE1E23481DA0D5785F7A93B2E2198C097C57B3D5C in libGameAssembly.a[22](vu34smn0bqak.o)
      _SWIGStringWithLengthHelper_SWIG_csharp_string_str_m83F8D0FCAAA6F3D1AC73C859088E835E8D9184FF in libGameAssembly.a[23](mz5zwj743dgj.o)
      _SWIGStringWithLengthHelper_SWIG_c_to_csharp_string_m84428208EED10043F30F6C512792615AE2FAC037 in libGameAssembly.a[23](mz5zwj743dgj.o)
      _SWIGStringWithLengthHelper_SWIG_csharp_string_str_m6D6B1401557AB973D33F49254FCCB69A4C94065D in libGameAssembly.a[25](q0ur51tkyt20.o)
      _SWIGStringWithLengthHelper_SWIG_c_to_csharp_string_mC21BFB1DCB3E5CD72C79A7899554A8182198C5CF in libGameAssembly.a[25](q0ur51tkyt20.o)
      _SWIGStringWithLengthHelper_SWIG_csharp_string_str_mE8CC0F094D3082710D19C93A28A2C3CF8297B371 in libGameAssembly.a[26](xt9xt4swnlqe.o)
      ...
  "_SWIG_csharp_string_to_c", referenced from:
      _SWIGStringWithLengthHelper_SWIG_csharp_string_to_c0_mFE6E5675C210B8A012509DFC8D1733ED4BFF08B3 in libGameAssembly.a[22](vu34smn0bqak.o)
      _SWIGStringWithLengthHelper_SWIG_csharp_string_to_c_m8215BCAC3D160AF742FDBFAE9C3721158B1C1847 in libGameAssembly.a[22](vu34smn0bqak.o)
      _SWIGStringWithLengthHelper_SWIG_csharp_string_to_c0_m4748CBC22A07C4460FCE76182618154259A699AB in libGameAssembly.a[23](mz5zwj743dgj.o)
      _SWIGStringWithLengthHelper_SWIG_csharp_string_to_c_m987FC6537DCF628964F5623A836D12D2D153A61D in libGameAssembly.a[23](mz5zwj743dgj.o)
      _SWIGStringWithLengthHelper_SWIG_csharp_string_to_c0_mBFE7B39ECD66FBAC7F413F646B7537DF0736DBB7 in libGameAssembly.a[25](q0ur51tkyt20.o)
      _SWIGStringWithLengthHelper_SWIG_csharp_string_to_c_m04B714FEF399DC258EA140985995ED2B51EB5550 in libGameAssembly.a[25](q0ur51tkyt20.o)
      _SWIGStringWithLengthHelper_SWIG_csharp_string_to_c0_m0C29778A973671B9CA23102557968D5372D9B948 in libGameAssembly.a[26](xt9xt4swnlqe.o)
      ...
ld: symbol(s) not found for architecture arm64
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

If using CocoaPods for Apple platforms, the project's Podfile.lock

Expand Podfile.lock snippet

👀 Replace this line with the contents of your Podfile.lock!

Anybody?

Having the same issue with Unity 2022.3.52f1 and firebase 12.4.0

I can fixed of firebase version 12.0.0
Link: https://github.com/firebase/firebase-unity-sdk/releases?page=1

I can fixed bug of firebase version 12.0.0 Link: https://github.com/firebase/firebase-unity-sdk/releases?page=1

You can try version 12.0.0

Hey folks, thanks for bringing this up to our attention. While we investigate this further, could you help me with the steps to reproduce this behavior?
What I did was the following, however I didn't encounter any issues:

Create a new project in Unity
Use Assets > Import Package > Custom Package...
Add version 12.4.0 of Firebase Analytics, Crashlytics, Messaging, RemoteConfig
Build project with iOS platform
Open in Xcode then select Product > Archive

Am I missing anything? Does this issue only happen during archiving or does it also happen when building? I'm also testing with Unity 6000.0.23f1 if that makes any difference.

@argzdev I am using Unity 2022.3.26f1 version and I believe others are also using same Unity 2022.3.x version

I'm also getting the same erros while building.
Undefined symbol: _SWIG_csharp_string_size
Undefined symbol: _SWIG_csharp_string_str
Undefined symbol: _SWIG_csharp_string_to_c

Unity: 2021.3.17f1
Xcode: 16.1
Firebase SDK: 12.4.0

Same for Unity(6000.0.26f1) unfortunately.

Having same errors with Unity 6000.0.27f1 and Xcode 16.1.
Currently latest working version is 12.3.0

Hi all,
While we haven't been able to reproduce the error ourselves, using disassembly tools I have spotted some additional symbols that are present in the C# libraries about SWIGStringWithLengthHelper, which seems to be the root cause of the issue.  While I try to determine a proper fix for this, unfortunately the best idea I have is to use 12.3.0.
Sorry for the inconvenience.

same problem with Xcode 16.1. v12.3.0 works with Xcode 16.1

@hanilozmen where can we download 12.3.0 SDK? Can you paste the link here?

@argzdev thank you for the link

@hanilozmen where can we download 12.3.0 SDK? Can you paste the link here?

If you click Assets>Source Code links, it will start downloading specified Firebase version as a single bundle.
https://github.com/firebase/firebase-unity-sdk/releases

Alright, this should be resolved with the latest release https://github.com/firebase/firebase-unity-sdk/releases/tag/v12.4.1
I wasn't able to reproduce this error myself, but I confirmed with disassembly tools that the SWIGStringWithLengthHelper logic has been removed from the libraries, so hopefully it'll all be good to go.
Thanks for bringing this to our attention.

And thanks for your interest.
As soon as I've seen regarding release I've tried to rebuild again.. and good news it's working.
Thanks again..


Mortar and Pestle - FPS drop when using the mortar & pestle

Singleplayer
An FPS drop is noticeable when the mortar is being used. This happens whether a resource is in the mortar or not.

Fixed in V1.0.3


Issue: ListView w/ Nested StreamBuilder Janky on Scroll Up

I have a long list of posts that is using ListView.builder with a StreamBuilder nested inside of the list (to get users docs). The list scrolls perfectly while scrolling down. However, when scrolling up, it is very janky and typically scrolls all the way back to the top of the list. I have read several threads that also report this as an issue, but nothing seems to solve the problem. It does appear to be caused by the loading widget indicator and the StreamBuilder inside of the ListView while rebuilding the post on the scroll up after being disposed of on the scroll down, but if I set the height the same, it stops from scroll up (height of the posts is 400).
Is there anyway to 'rebuild' the widgets scrolling up like it does scrolling down? Here is some code on the ListView and the nested StreamBuilder.
//first query to get posts docs
StreamBuilder<List<PostsRecord>>(
                stream: queryPostsRecord(
                  queryBuilder: (postsRecord) =>
                      postsRecord.orderBy('created_time', descending: true),
                  limit: 200,
                ),
                builder: (context, snapshot) {
                  if (!snapshot.hasData) {
                    return Center(
                      child: SizedBox(
                        width: 40,
                        height: 40,
                        child: CircularProgressIndicator(
                          color: Color,
                        ),
                      ),
                    );
                  }
                  List<PostsRecord> listViewPostsRecordList = snapshot.data;
                  return ListView.builder(
                    padding: EdgeInsets.zero,
                    scrollDirection: Axis.vertical,
                    itemCount: listViewPostsRecordList.length,
                    itemBuilder: (context, listViewIndex) {
                      final listViewPostsRecord =
                          listViewPostsRecordList[listViewIndex];
                      return Visibility(
                        visible: !(vis.check)) ??
                            true,
//this is the StreamBuilder to get the users docs
                        child: StreamBuilder<UsersRecord>(
                          stream:
                              UsersRecord.getDocument(listViewPostsRecord.user),
                          builder: (context, snapshot) {
                            if (!snapshot.hasData) {
                              return Center(
                                child: SizedBox(
                                  width: 40,
//I tried setting this to 400 (size of media in post) but it prevented from scrolling up
                                  height: 40,
                                  child: CircularProgressIndicator(
                                    color: Color,
                                  ),
                                ),
                              );
                            }
                            final containerUsersRecord = snapshot.data;
                            return Posts()
//feed posts vary in height, but the media within the posts is 400 pixels

Here is a quick screen recording showing the issue:
https://youtube.com/shorts/ADz1hUtkMTQ?feature=share
Does anyone have any suggestions on how to fix this? Is there a way to rebuild widgets on the scroll up just like it does scrolling down? Seems like a major issue for flutter apps.
Expected results:
Scrolling back up should match smoothness and performance of scrolling down.
Actual results:
Scrolling up is very janky and typically jumps up several posts or to the beginning where scrolling down is very smooth.
--verbose results, although this exception does not solve the issue described above.
 ══╡ EXCEPTION CAUGHT BY WIDGETS LIBRARY ╞═══════════════════════════════════════════════════════════
                    The following NoSuchMethodError was thrown while finalizing the widget tree:
                    The method 'dispose' was called on null.
                    Receiver: null
                    Tried calling: dispose()

No errors on flutter analyze
Flutter doctor -v
[✓] Flutter (Channel stable, 2.10.5, on macOS 12.4 21F79 darwin-x64, locale en-US)
    • Flutter version 2.10.5 at /Users/dennisashford/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 5464c5bac7 (2 months ago), 2022-04-18 09:55:37 -0700
    • Engine revision 57d3bac3dd
    • Dart version 2.16.2
    • DevTools version 2.9.2

[✓] Android toolchain - develop for Android devices (Android SDK version 32.0.0)
    • Android SDK at /Users/dennisashford/Library/Android/sdk
    • Platform android-32, build-tools 32.0.0
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.11+0-b60-7590822)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 13.4.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • CocoaPods version 1.11.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.11+0-b60-7590822)

[✓] VS Code (version 1.59.1)
    • VS Code at /Users/dennisashford/Downloads/Visual Studio Code.app/Contents
    • Flutter extension version 3.25.0

[✓] Connected device (2 available)
    • iPhone 13 Pro (mobile) • FBFB4D51-EC31-47DF-8FE0-66B114806EA4 • ios            • com.apple.CoreSimulator.SimRuntime.iOS-15-5 (simulator)
    • Chrome (web)           • chrome                               • web-javascript • Google Chrome 102.0.5005.115
    ! Error: Dennis Ashford’s iPhone is not connected. Xcode will continue when Dennis Ashford’s iPhone is connected. (code -13)

[✓] HTTP Host Availability
    • All required HTTP hosts are available

Hello @exaby73, I just upgraded to 3.0.3 and can confirm the issue persists. Also, I added print(listViewIndex); to see which index was being built. I scrolled slowly to the 11th post and everything built fine. Then just scrolled up a little and it built every child above in rapid succession. I believe this is causing the issue. The output is below.
flutter: 2
flutter: 3
flutter: 4
flutter: 5
flutter: 6
flutter: 7
flutter: 8
flutter: 9
flutter: 10
flutter: 11
//stop scroll and down and barely scrolled up
flutter: 7
flutter: 6
flutter: 5
flutter: 4
flutter: 3
flutter: 2
flutter: 1
flutter: 0

Yes @exaby73 I will try. I am new to Flutter and still learning how to build a reprex. Need to figure out how to add the StreamBuilders into a minimal example.


Add Adafruit-Blinka to requirements.txt

https://github.com/adafruit/circuitpython/issues/1246

We don't actually need this because it doesn't import CircuitPython libraries: https://github.com/adafruit/adabot/issues/70


Auto-prepending with this. even for static methods in JS

In JS, in a class, type part of the name of a class static method. It will appear in the dropdown. Select it with Tab. It gets prepended with this. like other instance methods, but since it's a static method, this. doesn't work. Only <class name>.method will work.
VSC should recognize static methods and prepend them accordingly when auto-completing them.

Please share a small code example that shows this problem


Add title attributes to toolbar buttons

Not an issue but a proposal.
Is it possible to add title attributes to the buttons in the editor toolbar? So when you hover over each button a helpful hint shows. Much like the GitHub editor here:

Mock up using Chrome dev-tools:

Whilst many of us developers will recognise the editor icons, there are some clients (the ultimate the end user) who may not.

This is a good idea. This would have to be translatable (I can provide German and English). And maybe the regular "title" attribute is not enough, maybe a tooltip like you said here in Github would be nice.
Would you like to PR this?

I just saw you seem to be new on GitHub - with "PR" I mean write a pull request, and if that is not possible for you, no problem, I will add this when I get around to it.

@manogi I'm certainly new to contributing but I am happy to give this a shot. I'm sure I can spare some time this week or next to look into this 😄

This is done now (in English and German):