Microcredit Dashboard Page

Preview: https://web-git-feature-microcredit-dashboard-ipct.vercel.app/microcredit/dashboard
#308

@hugomarquesdev you can now finish it with https://impactmarket-api-staging.herokuapp.com/api-docs/#/microcredit/get_microcredit_global


⚠ NOWnews今日新聞 has degraded performance

In 921156b, NOWnews今日新聞 (https://www.nownews.com/) experienced degraded performance:

HTTP code: 200
Response time: 1735 ms

Resolved: NOWnews今日新聞 performance has improved in d95b0e4.


Bug: Issue with statemachine in v5

Description
It doesn't seem like waitFor is working and waiting for a transition.
I expect the second snap output to be light.  It stays green and the waitFor timesout

const lightMachine = createMachine(
  {
    /** @xstate-layout N4IgpgJg5mDOIC5gF8A0IB2B7CdGgBsBLKACwBd8QAHLWI8orDKgD0QFoA2dAT067I0IYmXIA6KACcwYFkhp0GTeaHYIALACY+iABwBGcQGYtWgJwGArOYDs5rQdsa9Q9KIriPlBbXqNmNkRtXQQDPS5xLg0DB1s9ewAGY0SNISEgA */
    id: 'light',
    initial: 'green',
    states: {
      green: {
        // action referenced via string
        entry: 'alertGreen',
        on: {
          // action referenced via object
          CLICK: 'light',
        },
      },
      light: {
        entry: ['alertGreen'],
        invoke: {
          src: myMachine,
        },
      },
    },
  },
  {
    actions: {},
  }
);

const actor = createActor(lightMachine);
const snap1 = actor.getSnapshot()
console.log('------------')
console.log('lets do this');
console.log('can', snap1.can({ type: 'CLICK' }));
console.log('can2', snap1.can({ type: 'CLICdK' }));
console.log('output', snap1.output);

actor.send({ 'type': 'CLICK' })
// console.log(answer);
await waitFor(actor, (snapshot)=> snapshot.matches('green'));
const snap2 = actor.getSnapshot();
console.log('snap2', snap2);

i'm using v5 beta 27
Expected result
I expect the second snap output to be light.
Actual result
it stays green or timesout because of waitFor
Reproduction
https://codesandbox.io/s/lively-silence-kc5f2t?file=/src/index.ts
Additional context

@davidkpiano any idea why this woudln't work?

@davidkpiano any idea why this woudln't work?

Yes, you didn't start the actor yet. Make sure to call actor.start()

doh 🤣  !  @davidkpiano  thank s so much!


Protocol for cleaning up after Velocity (to avoid memory leaks)

I'm using Velocity ("V") in an Aurelia project using the aurelia-animator-velocity plug-in, which, as you may know, is a very, very thin wrapper over V.
Everything works great, except that V doesn't clean up the DOM nodes involved in animation.  I ruled out the wrapper by removing it from my project and loading V in the usual way: a script tag in index.html.  I then replaced all calls to this.animator.animate(...) with Velocity(...).  Again, everything works perfectly.
But V is still leaking.  When I navigate away from a view, Aurelia invokes detached(), and all DOM nodes that can be removed are removed.  The V nodes remain.
With the plugin, I'm invoking V this way:
      this.animator.animate(document.getElementById('recoveryFieldSet'), { height: '80px' }, { duration: 100 });
In other words, I'm not using jQuery.  What do I need to do the prepare V nodes for removal from the DOM?
Thank you!
Eric

Unfortunately it's something that's on the todo list - currently you'd just have to call Velocity([el, ...], "stop", true); on them when removing (and even then I'm not sure if it frees resources properly).

@Rycochet Thanks for suggestion.
Unfortunately, resources are not freed properly.  But it did have an effect.  Apparently, stop with true does remove the node, but something is still hanging around.  I can comment and uncomment the animation calls and, at will, create/eliminate the memory leak.
Surely you guys would no how to release the very resources you're allocating?  Please give this top priority, as time permits, of course.
Thank you.

@estaylorco I'm the only coder maintaining Velocity, and I still don't know exactly how it does everything - if it was in Typescript it'd be simple to work out, but Javascript doesn't have interfaces or type tracking so it's non trivial trying to follow things in there - it's also one monolithic file that needs to be split up. I do what I can when I have time, but some months are busier than others ;-)

@Rycochet Oh, no problem.  That's why I indicated "...as time permits..."  Even if it's months and months from now, fixing this problem will be most welcome 👍 .
I did switch to Aurelia's CSS-based animator for now, but will be most happy to switch back to Velocity.

@Rycochet thanks for your efforts, it is much appreciated 👍

In Velocity 2 the per-Element data is stored in a WeakMap, so it will be freed when the Element is removed from the DOM and released. THe list of animations is also no longer a list so frees each one as they finish.

Changed to storing the data against the element itself, without any internal references (was slightly faster and made it easier to manage).

@estaylorco would be great If you could give this a Test drive and see If your issues are gone

@zewa666 @estaylorco Just remember that this is related to the develop branch - which is unlikely to be fully released until late December at the earliest.


Version 0.7 last call

I think it's about time that we released version 0.7! I should have time to do this on Sunday.
Anyone have more changes that they would like to make before then?
@PyAbel/developers

There are a few holes in the docs for hansenlaw, tools.vmi, that I will try to plug. doc making works on my (home) Macbook. livehtml is neat! Python virtualenv is also a good idea, which I will try.

I am not 100% happy with the quadrant combining logic in tools.symmetry.put_image_quadrants()
 if 0 in symmetry_axis:
     Q0 = Q1
     Q3 = Q2

 if 1 in symmetry_axis:
     Q2 = Q1
     Q3 = Q0

e.g. for symmetry_axis=0 In principle Q1 might be empty, which would then stomp Q0 etc

Extreme benchmark
(note my VMI images may be of size 8192x8192 pixels)

$lscpu
Architecture:          x86_64
    :                            :
Model name:            Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz

32GB RAM
SSD HD

$python benchmark.py
PyAbel benchmark run on x86_64

=========     Direct Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

Implementation |    n = 251          |    n = 501          |    n = 1001         |    n = 2001            |    n = 4001
-----------------------------------------------------------------------------------------------------------------------------------
 direct_Python |     68.0            |    516.0            |   5335.8            |  40789.8               | 1137469.9
     HansenLaw |     17.5            |     42.8            |    131.5            |    455.1               |   1197.6
      direct_C |      9.8            |     72.4            |    573.8            |   4944.3               |  36771.5

=========    Inverse Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

Implementation |    n = 251          |    n = 501          |    n = 1001         |    n = 2001            |    n = 4001
---------------------------------------------------------------------------------------------------------------------------
         BASEX |     20.3 (  4451.9) |    256.0 ( 22894.8) |   1967.5 (167616.5) |  13784.0 (1535687.8)   |  78991.8 (13295525.7)
   Three_point |     14.8 (   448.4) |    123.7 (  1839.2) |   1069.1 (  7465.4) |   8901.8 ( 32833.5)    |  66314.6 (118015.3)
      direct_C |     25.1            |    198.4            |   1698.1            |  13637.0               | 102893.4
     HansenLaw |     18.4            |     57.0            |    131.0            |    445.2               |   1397.3
 direct_Python |     88.1            |    634.7            |   6139.2            |  50126.2               | 1238070.6

I don't know why three_point improved for n=2001

@stggh The three_point  data point in your table is 8901.8, but it looks like it is around 1000 in the plot.

@DhrubajyotiDas thanks!  I made a typo in the data input file used for the plot 890.18 instead of 8901.8, too similar for me to spot!

Yes, at least one of the examples should be updated to demonstrate the combined centering, transform, and integration functionality in abel.transform.

My understanding was that put_image_quadrants() was intended to pick up the pieces from the get_image_quadrants() function. In this case, isn't Q1 never empty? But, I suppose it makes sense to indicate in the docstring that this put function only works perfectly in conjunction with our get function, and should not use a general purpose "quadrant-averager".

I think that this is the right way to go about it. It people want to do averaging of images on the re-stitching step, then they can always do the averaging themselves and pass 4 quadrants to put_image_quadrants().
Or perhaps I misunderstood your concern...

That benchmark look great! I was hoping that somebody would make the graph! I think the x and y labels are flipped. Also, perhaps you can go log-log so that we can draw a trend line! How will Hansen-Law do for Gigapixel images?

Opps! you may need to take back my physics degree. Corrected axes labels and log-log plot

2.The issue may be Q0 = Q1 when it is equally valid to use Q1 = Q0.  You may be right that an empty Q1 cannot occur.

Oh, plot looks beautiful!
It would be interesting to see how things change across a broad range of different sizes, and where the bottleneck is for each algorithm. I am a little surprised that BASEX and Three-point (not including the basis set generation) doesn't do better. They are just stright linear algebra, and that stuff should go straight to LAPACK and cruise along at the speed-of-fortran.
Actually, on this system (2012 iMac, Intel i5, 16 GB ram) BASEX, Three-point and HL are all about the same:

 =========    Inverse Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

 Implementation |    n = 251          |    n = 501          |    n = 1001
 ----------------------------------------------------------------------------
          BASEX |      5.9 (  3843.9) |     40.8 ( 18261.7) |    251.9 (140927.9)
    Three_point |      8.2 (   830.2) |     32.7 (  3230.3) |    425.6 ( 12957.5)
       direct_C |     13.8            |     94.6            |    930.5
      HansenLaw |     28.9            |     95.8            |    296.0
  direct_Python |     81.8            |    799.0            |  11321.9

Maybe your numpy somehow isn't compiled with BLAS/LAPACK. I think that even the default python installation on OSX is compiled with these, so perhaps you can re-run part of the benchmark on your macbook? You can check if Lapack is linked using this SO post (see second answer).
Now that I look into it, we should almost definitely be using scipy.linalg instead of numpy.linalg.

scipy.linalg contains all the functions in numpy.linalg. plus some other more advanced ones not contained in numpy.linalg
Another advantage of using scipy.linalg over numpy.linalg is that it is always compiled with BLAS/LAPACK support, while for numpy this is optional. Therefore, the scipy version might be faster depending on how numpy was installed.
Therefore, unless you don’t want to add scipy as a dependency to your numpy program, use scipy.linalg instead of numpy.linalg

from: http://docs.scipy.org/doc/scipy/reference/tutorial/linalg.html
(btw: earlier this week I learned taht scipy.fftpack beats numpy.fft. Poor numpy.)

Oh, and on the topic of speed: I ran a 1 gigapixel inverse Abel transform (with Hansen-law, of course): 569.4 seconds, so just under 10 mins. That was on a MacBook pro, i7, 16 GB ram. The ram usage was pretty high, and the CPU usage only in the middle, so a system with more ram might go faster.
Roman and I had an interesting discussion a few months ago about different "Direct" implementations and how different numbers of nested loops led to algorithms that performed optimally for different array image sizes. Basically, we found that we should stop vectorizing so much (and use more loops) as the array size grew. Otherwise, the ram usage became too high.

@DanHickstein fantastic information. You are right the Macbook pro (2.4GHz i5 8GB RAM OSX 10.11.3) performs better:
Implementation |    n = 251          |    n = 501          |    n = 1000
 ----------------------------------------------------------------------------
          BASEX |      5.9 (  4707.9) |     80.3 ( 22767.7) |      nan (     nan)
    Three_point |      6.0 (  1133.5) |     23.5 (  4619.3) |      nan (     nan)
       direct_C |     18.0            |    119.0            |   1032.0
      HansenLaw |     30.6            |     91.1            |    334.3
  direct_Python |    119.3            |   1093.2            |  10367.5

cf the Dell (3.4GHz i7 32GB, ubuntu 15.10)
 Implementation |    n = 251          |    n = 501          |    n = 1001
 ----------------------------------------------------------------------------
          BASEX |     20.5 (  4289.4) |    248.7 ( 20750.6) |      nan (     nan)
    Three_point |     15.0 (   492.6) |    125.7 (  1870.4) |      nan (     nan)
       direct_C |     26.5            |    195.6            |   1570.2
      HansenLaw |     17.8            |     45.3            |    120.9
  direct_Python |     83.8            |    624.8            |   5676.6

Did you install numpy from source on the Dell?
It is worth switching the linear algebra to scipy? @DhrubajyotiDas, could you look into this? The change appears pretty trivial. It probably won't affect many people, since few are probably installing numpy from scratch.

Installed direct from the ubuntu 15.10 image and apt-get.  You have provided very good information. A quick check:
In [2]: np.__config__.show()
blas_info:
    libraries = ['blas']
    library_dirs = ['/usr/lib']
    language = f77
lapack_info:
    libraries = ['lapack']
    library_dirs = ['/usr/lib']
    language = f77
atlas_threads_info:
  NOT AVAILABLE
blas_opt_info:
    libraries = ['blas']
    library_dirs = ['/usr/lib']
    language = f77
    define_macros = [('NO_ATLAS_INFO', 1)]
atlas_blas_threads_info:
  NOT AVAILABLE
openblas_info:
  NOT AVAILABLE
lapack_opt_info:
    libraries = ['lapack', 'blas']
    library_dirs = ['/usr/lib']
    language = f77
    define_macros = [('NO_ATLAS_INFO', 1)]
atlas_info:
  NOT AVAILABLE
lapack_mkl_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
mkl_info:
  NOT AVAILABLE

which looks more substantial than the Macbook:
In [2]: np.__config__.show()
mkl_info:
  NOT AVAILABLE
atlas_blas_threads_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
blas_opt_info:
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
    extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
atlas_3_10_blas_threads_info:
  NOT AVAILABLE
lapack_opt_info:
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
    extra_compile_args = ['-msse3']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
lapack_mkl_info:
  NOT AVAILABLE
openblas_lapack_info:
  NOT AVAILABLE
atlas_3_10_info:
  NOT AVAILABLE
openblas_info:
  NOT AVAILABLE
atlas_threads_info:
  NOT AVAILABLE
atlas_3_10_blas_info:
  NOT AVAILABLE
atlas_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
atlas_3_10_threads_info:
  NOT AVAILABLE

I will follow-up.

Item 2. (above). put_image_quadrants() is ok, because as @DanHickstein realised all quadrants have data. However, get_image_quadrants() requires use_quadrants to be set all True, following the first symmetry operation, as they have been redefined, and are all required, for any subsequent symmetry operation. This change has been made in PR #137.

@DanHickstein I see no issue in switching from numpy to scipy. My impression was that scipy was a superset of numpy. I wasn't aware of the differences in scipy.linalg vs numpy.linalg - speed is important, and this seems like a useful change.
Regarding speed benchmarks, on my Thinkpad,
PS C:\Users\Administrator\Documents\PyAbel\abel> python .\benchmark.py
PyAbel benchmark run on Intel64 Family 6 Model 58 Stepping 9, GenuineIntel

 =========     Direct Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

 Implementation |    n = 301          |    n = 501
 ----------------------------------------------------------
  direct_Python |    131.8            |    918.3
      HansenLaw |     30.8            |     77.6

 =========    Inverse Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

 Implementation |    n = 301          |    n = 501
 ----------------------------------------------------------
          BASEX |      8.0 (  6868.3) |     62.4 ( 20789.6)
    Three_point |      8.4 (  1110.3) |     28.2 (  2783.0)
      HansenLaw |     40.6            |     90.6
  direct_Python |    147.6            |    936.7

PS C:\Users\Administrator\Documents\PyAbel\abel> Get-WmiObject Win32_Processor

Caption           : Intel64 Family 6 Model 58 Stepping 9
DeviceID          : CPU0
Manufacturer      : GenuineIntel
MaxClockSpeed     : 2901
Name              : Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz
SocketDesignation : CPU Socket - U3E1

and on my work PC (Xeon)
PS E:\My Documents\PyAbel\abel> python .\benchmark.py
PyAbel benchmark run on Intel64 Family 6 Model 58 Stepping 9, GenuineIntel

 =========     Direct Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

 Implementation |    n = 301          |    n = 501
 ----------------------------------------------------------
  direct_Python |    134.6            |    849.5
      HansenLaw |     34.1            |     82.6

 =========    Inverse Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

 Implementation |    n = 301          |    n = 501
 ----------------------------------------------------------
          BASEX |      5.2 (  6587.6) |     31.2 ( 20323.4)
    Three_point |      7.9 (  1062.1) |     22.2 (  2826.4)
      HansenLaw |     41.9            |     94.0
  direct_Python |    137.0            |    907.2

PS E:\My Documents\PyAbel\abel> Get-WmiObject Win32_Processor

Caption           : Intel64 Family 6 Model 58 Stepping 9
DeviceID          : CPU0
Manufacturer      : GenuineIntel
MaxClockSpeed     : 3801
Name              : Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz
SocketDesignation : Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz

Interesting. So maybe it's just Ubuntu that has trouble with the linear algebra :).

I will wait until after we are done making changes to the get_image_quadrants() function before I make the 0.7 release. Hopefully we can do this in the next few days.

Perhaps the difference in performance is because of the back end of lapack and BLACS. (Anaconda and Enthought provide Intel MKL backend in their distribution)
test results
Anaconda (Intel MKL backend)
[gilbert@atomcool:abel 509]$ ~/anaconda2/bin/python benchmark.py

PyAbel benchmark run on x86_64

 =========     Direct Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

 Implementation |    n = 301          |    n = 501
 ----------------------------------------------------------
  direct_Python |    189.1            |    805.7
      HansenLaw |     44.8            |    111.2
       direct_C |     41.0            |    172.3

 =========    Inverse Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

 Implementation |    n = 301          |    n = 501
 ----------------------------------------------------------
          BASEX |     13.9 ( 16369.0) |    125.5 ( 45203.4)
    Three_point |     10.9 (  1677.3) |     24.0 (  4205.9)
       direct_C |     44.0            |    185.0
      HansenLaw |     49.0            |    122.8
  direct_Python |    195.2            |    919.5

In [2]: np.show_config()
lapack_opt_info:
    libraries = ['mkl_lapack95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'iomp5', 'pthread']
    library_dirs = ['/home/gilbert/anaconda2/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/home/gilbert/anaconda2/include']
blas_opt_info:
    libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'iomp5', 'pthread']
    library_dirs = ['/home/gilbert/anaconda2/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/home/gilbert/anaconda2/include']
openblas_lapack_info:
  NOT AVAILABLE
lapack_mkl_info:
    libraries = ['mkl_lapack95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'iomp5', 'pthread']
    library_dirs = ['/home/gilbert/anaconda2/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/home/gilbert/anaconda2/include']
blas_mkl_info:
    libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'iomp5', 'pthread']
    library_dirs = ['/home/gilbert/anaconda2/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/home/gilbert/anaconda2/include']
mkl_info:
    libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'iomp5', 'pthread']
    library_dirs = ['/home/gilbert/anaconda2/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/home/gilbert/anaconda2/include']

System Python
installing all the dependency for PyAbel...
In [2]: np.show_config()
blas_info:
    libraries = ['blas']
    library_dirs = ['/usr/lib']
    language = f77
lapack_info:
    libraries = ['lapack']
    library_dirs = ['/usr/lib']
    language = f77
atlas_threads_info:
  NOT AVAILABLE
blas_opt_info:
    libraries = ['blas']
    library_dirs = ['/usr/lib']
    language = f77
    define_macros = [('NO_ATLAS_INFO', 1)]
atlas_blas_threads_info:
  NOT AVAILABLE
lapack_opt_info:
    libraries = ['lapack', 'blas']
    library_dirs = ['/usr/lib']
    language = f77
    define_macros = [('NO_ATLAS_INFO', 1)]
atlas_info:
  NOT AVAILABLE
lapack_mkl_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
mkl_info:
  NOT AVAILABLE

Platform
[gilbert@atomcool:~ 502]$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 44
Stepping:              2
CPU MHz:               2393.981
BogoMIPS:              4787.90
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              12288K
NUMA node0 CPU(s):     0-7

The reason why PyAbel runs better in Mac than Ubuntu system python might raise from the fact that the BLAS and LAPACK are compiled with MAC accelerate framework.
https://developer.apple.com/library/prerelease/ios/documentation/Accelerate/Reference/AccelerateFWRef/index.html

Ah, okay, I like this explanation. Interesting how different things can be with different python distributions!
+1 for Anaconda I suppose!

Here is a benchmark of scipy versus numpy for a dot product on an NxN matrix. I tried with both /usr/bin/python (using apple's Accelerate framework) as well as Anaconda python using MKL. All results are nearly identical. I think that the only time the problem would arise is if somebody like @stggh has a python package that has not been compiled with optimized BLAS/LAPACK.

Still, while switching to scipy.linalg will probably not improve performance on most systems, it should guarantee optimal performance even when numpy is not installed with optimized math libraries.
From the Scipy site:

One of the design goals of NumPy was to make it buildable without a Fortran compiler, and if you don’t have LAPACK available NumPy will use its own implementation. SciPy requires a Fortran compiler to be built, and heavily depends on wrapped Fortran code.

A little more info here: https://pythonhosted.org/poppy/performance.html

OpenBLAS, MKL, and Accelerate all have very similar performance, so as long as your numpy is using one of those three you should be in good shape.

@stggh was using regular blas/lapack, so this likely explains the difference in performance.

@DanHickstein I'm totally agree to switch to scipy.linalg :D

I am not convinced that this is the full story. My Macbook pro runs enthought Python, not Anaconda. However, the linear algebra speed up may be apple's Accelerate framework. BLAS/LAPACK may be less optimized for Ubuntu, but this seems unlikely, given that the CPU is an Intel i7.
IMO a single timing run is inaccurate. How about we repeat our tests with transform_repeat=4, for n=251, 501:
[etalon:.+VMI/PyAbel/abel]$ !python
python benchmark.py
PyAbel benchmark run on x86_64

 =========     Direct Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

 Implementation |    n = 251          |    n = 501
 ----------------------------------------------------------
  direct_Python |     72.7            |    585.3
      HansenLaw |     16.4            |     46.8
       direct_C |      9.8            |     82.7

 =========    Inverse Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

 Implementation |    n = 251          |    n = 501
 ----------------------------------------------------------
          BASEX |      8.7 (  4346.1) |    129.0 ( 22278.0)
    Three_point |      4.9 (   454.2) |     66.0 (  1987.7)
       direct_C |     13.3            |    107.8
      HansenLaw |     17.8            |     56.0
  direct_Python |     74.3            |    558.6

The Mac run will have to wait until I am back home tonight.

@stggh, can you clone a version of my numpy-to-scipy branch and run the benchmarks with that? It should ensure consistent linear algebra performance across various systems.

Cloned, and I set transform_repeat=4, but left n=301, 501:
PyAbel benchmark run on x86_64

 =========     Direct Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

 Implementation |    n = 301          |    n = 501
 ----------------------------------------------------------
  direct_Python |    113.2            |    495.6
      HansenLaw |     21.7            |     42.1
       direct_C |     16.5            |     71.3

 =========    Inverse Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

 Implementation |    n = 301          |    n = 501
 ----------------------------------------------------------
          BASEX |     14.2 (  6673.2) |    120.0 ( 20881.8)
    Three_point |      7.8 (   645.0) |     64.3 (  1828.5)
       direct_C |     22.3            |     94.8
      HansenLaw |     23.9            |     46.0
  direct_Python |    119.2            |    514.9

Repeated:
 =========     Direct Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

 Implementation |    n = 301          |    n = 501
 ----------------------------------------------------------
  direct_Python |    110.7            |    504.7
      HansenLaw |     20.3            |     40.1
       direct_C |     16.5            |     80.3

 =========    Inverse Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

 Implementation |    n = 301          |    n = 501
 ----------------------------------------------------------
          BASEX |     14.1 (  6581.6) |    119.2 ( 20734.7)
    Three_point |      7.7 (   638.6) |     65.2 (  1806.6)
       direct_C |     21.3            |     93.9
      HansenLaw |     21.9            |     46.9
  direct_Python |    117.3            |    522.9

Note, also, my system logs the following warnings:
 python setup.py install --user
 :
In file included from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1761:0,
                 from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from abel/lib/direct.c:242:
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^
In file included from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:26:0,
                 from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from abel/lib/direct.c:242:
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/__multiarray_api.h:1629:1: warning: ‘_import_array’ defined but not used [-Wunused-function]
 _import_array(void)
 ^
In file included from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ufuncobject.h:327:0,
                 from abel/lib/direct.c:243:
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/__ufunc_api.h:241:1: warning: ‘_import_umath’ defined but not used [-Wunused-function]
 _import_umath(void)
:

I think that those warnings were there previously, caused by the Cython implementation of the Direct transform. They have been in place for a while. Not sure how to get rid of them.
I'm really surprised that switching from numpy to scipy didn't change the behavior on your system! Can you confirm that the basex.py file has the new scipy language, and that import abel; print(abel.__file__) is using the correct PyAbel?
After the basis sets are generated, the BASEX algorithm is just 4 dot products and a matrix transpose, so it should be really fast.
Of course, these benchmarks are all just a curiosity. The fact that HansenLaw does not need to generate basis sets gives it a huge advantage in most situations. It's only real competitor is the DirectC implementation.

@stggh Did you try to upgrade the numpy and cython to the latest version??

All looks correct:
basex.py line 15: from scipy.linalg import inv
In ipython, abel.__file__ just gives:
 In [4]: abel.__file__
Out[4]: 'abel/__init__.py'

Running this code:
import abel
import sys

#print(abel._file__)

print(sys.modules['abel.basex'])

Gives, the correct location:
[etalon:.+yAbel-DH/PyAbel]$ python ck.py
<module 'abel.basex' from '/home/stg121/Software/Python/VMI/PyAbel-DH/PyAbel/abel/basex.pyc'>

Extracting timings listed above:

n=501
basex /  hansenlaw (repeat=1)
repeat=4

@stggh
256/57 (x86)   80/91 (Mac)
119/47(x86)

@DanHickstein
41/96 (Mac)

@DhrubajyotiDas
62/91 (x86)   31/94 (Xenon)

@ldes89150
126/123 (x86)

Given the massive improvement for basex that @DhrubajyotiDas  sees for his Xenon, I think that @ldes89150 and @DanHickstein are right, the BLAS/LAPACK implementation plays a big part.
I find it amusing that the best execution times favour the code home computer origin.
As a final comment @DanHickstein, I would like to follow up on the statement  that you made, else where, to do with turning off the vectorization for certain image sizes (? can't locate it at the moment). How is this done? The hansenlaw code does fails to execute on my raspberry pi 2.

@ldes89150 good point. I have just retried:
pip -U numpy and pip -U cython and the commands have resulted in upgrades
(I am sure I tried this yesterday ...)
The new benchmarks simply improve hansenlaw!!!
 Implementation |    n = 301          |    n = 501
 ----------------------------------------------------------
          BASEX |     15.0 (  6261.4) |    120.1 ( 19876.7)
    Three_point |      7.8 (   436.8) |     67.1 (  1193.9)
       direct_C |     21.9            |     94.8
      HansenLaw |     13.4            |     30.4
  direct_Python |     73.9            |    316.9

@stggh,
I don't see much different with the number of repeats (though this should surely make the measurement more reliable). Here are the results with the scipy.linalg version:
abel.benchmark.AbelTiming(transform_repeat=1):
 Implementation |    n = 301          |    n = 501
 ----------------------------------------------------------
          BASEX |      3.6 (  5397.2) |     25.5 ( 16942.9)
    Three_point |      4.0 (  1085.1) |     13.5 (  2976.2)
       direct_C |     20.3            |     87.2
      HansenLaw |     36.4            |     89.9
  direct_Python |    127.1            |    581.6

abel.benchmark.AbelTiming(transform_repeat=4):
 Implementation |    n = 301          |    n = 501
 ----------------------------------------------------------
          BASEX |      3.5 (  5481.2) |     24.1 ( 16761.1)
    Three_point |      3.8 (  1065.9) |     14.0 (  3004.3)
       direct_C |     20.4            |     88.1
      HansenLaw |     34.4            |     85.6
  direct_Python |    124.5            |    565.8

abel.benchmark.AbelTiming(transform_repeat=100):
 Implementation |    n = 301          |    n = 501
 ----------------------------------------------------------
          BASEX |      3.3 (  5432.5) |     23.8 ( 17180.7)
    Three_point |      3.8 (  1041.4) |     13.9 (  2975.9)
       direct_C |     20.7            |     88.3
      HansenLaw |     31.8            |     83.3
  direct_Python |    124.8            |    561.4

I find it amusing that the best execution times favour the code home computer origin.

This accounts for the speed of BASEX and Hansen-Law on my machine, since I at least played some role in those. Three-point was all @DhrubajyotiDas, so it has no excuse for being so speedy! :)
@stggh, the previous discussion about the vectorization is here: https://github.com/PyAbel/PyAbel/pull/52#issuecomment-164516923. The basic idea is that the Abel transform can be written as three nested loops:

A loop over the rows on the final image.
A loop over the column in the final image.
A loop over each pixel in the original image. (Only need to loop over pixels outside of the pixel of interest.)
At each iteration, a contribution from each pixel in the original image is added to the final image.

However, it's possible to eliminate all of these loops by simply using numpy arrays. We trade the three loops for a three-dimensional matrix, one dimension for each: rows in final image, columns in final image, columns in the original image. Then, we can just sum the matrix.
Of course, it's also possible to use a combination of 1 loop and 2D matrices, or 2 loops and a 1D matrices.
Generally, when using Python, loops are slow and you want to vectorize as aggressively as possible. Dealing with vectors is faster because Numpy, scipy, etc ship these vectors to fast c-libraries that can loop over them really quickly. However, there is a pretty obvious limit to this: the three-loop abel only produces a NxN array (the end result), while the zero-loop abel transform needs to keep an NxNxN array in memory. As the array size becomes larger than the available RAM, then you lose the benefit that vectorizing brought.
HansenLaw operates in a very similar way, except that, for the third loop, it doesn't loop over the columns of the original image, but instead it loops over only 9 values of k. Although it still considers the gradient, the I assume that it's basically equivalent to looping over the column of the original image within some approximation.
Anyway, if I recall correctly, we saw a big speedup in the HL algorithms when we removed the loop over the rows and instead made X and AIM rows x 1 arrays.
Maybe this answer is too long and confusing. Can the raspberry pi handle small-sized transforms?

@DanHickstein a very clear explanation. Thank you!
I can access the pi2 (at home) from work, since it is left on 24/7. It turns out that the pi2 can handle n=361, and 1023 image sizes. I thought there was an issue, but there isn't. I haven't tested this in a while. Here is the benchmark, hansenlaw only!
[Rpi2:.+e/Python/PyAbel]$ lscpu
Architecture:          armv7l
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             1
Model name:            ARMv7 Processor rev 5 (v7l)
CPU max MHz:           1100.0000
CPU min MHz:           600.0000

PyAbel benchmark run on

 =========     Direct Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

 Implementation |    n = 301          |    n = 501
 ----------------------------------------------------------
      HansenLaw |    798.2            |   1551.9

 =========    Inverse Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

 Implementation |    n = 301          |    n = 501
 ----------------------------------------------------------
      HansenLaw |    863.9            |   1732.3

Who needs a "fancy" xenon processor ...

Ahem ;)
So, is the conclusion that @stggh had something weird going on with his numpy installation, and if we move all linear algebra to scipy, we won't have a potential future user face similar worse-than-expected performance on some of the transforms?

I think that:

future user face similar worse-than-expected performance on some of the transforms

is a real possibility.
My system is vanilla ubuntu 15.10 installation.  If the docs contain some benchmarks, then a user may be able to gauge whether a system upgrade might improve execution times.

Here is a benchmark from our 64-node Xeon cluster (PyAbel just runs mostly on one, I think) with Anaconda python:
 =========     Direct Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

 Implementation |    n = 301          |    n = 501          |    n = 1001
 ----------------------------------------------------------------------------
  direct_Python |    156.9            |    581.6            |   5153.4
      HansenLaw |     41.4            |    102.1            |    327.1
       direct_C |     33.2            |    144.2            |    958.3

 =========    Inverse Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

 Implementation |    n = 301          |    n = 501          |    n = 1001
 ----------------------------------------------------------------------------
          BASEX |      4.7 ( 12859.0) |     30.0 ( 40424.4) |      nan (     nan)
    Three_point |      7.2 (  1533.0) |     18.5 (  3920.4) |      nan (     nan)
       direct_C |     35.9            |    150.9            |   1037.2
      HansenLaw |     45.1            |    115.7            |    358.3
  direct_Python |    167.9            |    627.0            |   5189.9

very fast linear algebra, but everything else (especially basis set generation) is super slow.

Oh, and great work getting PyAbel running on the raspberry pi. That's so cool!

@stggh I had the similar issue while installing PyAbel for my system python :d

Those linear algebra gains, are pretty impressive. Anyone check the output results?
(I will take a look.)
For the other methods, I believe the execution times may scale with the CPU clock (ARM chip excluded).
Here's a local Xenon, run by systems admins, so definitely not kept package current:
$cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
$ lscpu
   :
Model name:            Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
$uname -a
Linux scucomp5.anu.edu.au 3.10.0-327.4.5.el7.x86_64 #1 SMP Mon Jan 25 22:07:14 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

 Implementation |    n = 301          |    n = 501          |    n = 1001
 ----------------------------------------------------------------------------
  direct_Python |    147.1            |    905.1            |   7265.1
      HansenLaw |     26.5            |    113.2            |    324.5

 =========    Inverse Abel implementations ==========
time to solution [milli-sec (=sec/1000)] -> transform (basis sets generation)

 Implementation |    n = 301          |    n = 501          |    n = 1001
 ----------------------------------------------------------------------------
          BASEX |     14.9 (  8780.9) |     70.6 ( 33669.8) |    466.7 (244754.9)
    Three_point |     28.7 (   968.8) |    378.9 (  4236.5) |    675.6 ( 14008.9)
      HansenLaw |     30.1            |    121.9            |    354.4
  direct_Python |    154.3            |   1230.6            |   7571.3

Version 0.7.1 is on PyPi!
https://pypi.python.org/pypi/PyAbel/0.7.1

@DanHickstein excellent work on the docs (and the whole PyAbel project, BTW). I am very impressed with the look that you have created. Thanks, for researching all the ins and outs.
Since, this topic was also hijacked (by me) on benchmarks, do you think that we should add a nice loglog-plot in the code, and perhaps, progress info, in case the user gets impatient?

do you think that we should add a nice loglog-plot in the code

Do you add a script to Pyabel/examples that makes a plot of the benchmarks? Yes, this would be cool. We could let it run for a long time and add the static image to the doc/transform_methods/comparison.rst and comment on the various speeds of the transforms.
We could loop over the various nxn images and then update the plot using plt.draw(). I was actually working on an example of this and will post shortly.
Note: I did some work to make the Benchmark.py output easier to parse: https://github.com/PyAbel/PyAbel/pull/144

Here is a script that automatically updates a log-log plot as the results come in. Warning: it's a little hypnotizing.
https://gist.github.com/DanHickstein/326d2df2d0a73c14cb1d
Here are the results so far with my 2012 iMac (intel i5, 16 GB ram):

@rth will be happy to see that his Cython implementation maintains an order-of-magnitude improvement over the pure Python Direct transform across a broad range of image sizes. In addition, it remains reasonably competitive with all other methods.
@DhrubajyotiDas will enjoy the fact that BASEX and Three-point roughly tie for 1st place in the "up to 1000x1000 weight class" (of course, after their basis sets are generated!)
@stggh will be delighted to see that the Hansen-Law algorithm has a favorable scaling with image size and starts outperforming all other algorithms at image sizes larger than about 2000x2000.
With all other horses taken, @DanHickstein is left rooting for the pure-Python implementation of the Direct Abel transform, in which he played a small part in developing. He will point out that code-readability and the ability to handle data that are not evenly spaced are much more important than raw speed. :)
@PyAbel/developers will be interested to see that, even on a log-log plot, some of the curves are not especially flat, suggesting a deviation from a simple n^x scaling. I speculate that this could be due to memory issues: as the arrays become larger, they must be moved to memory located further from the main computational region of the CPU (here I'm thinking L1 cache, L2 cache, RAM...).
In contrast, the basis set generation algorithms exhibit relatively flat scaling. I believe that these rely heavily on nested loops, and so they likely scale in a more predictable manner since they do not keep much information in memory. Similarly, the Hansen-Law algorithm used two nested loops and only works with a nx1 array at a time, so it also has a flat scaling.

While still on this crazy benchmarking tangent, I have one more thought:
A situation where the speed of the transform might really matters is if somebody wants to transform images collected from a CCD camera at video rates. This is an idea that has been tossed around in the photoelectron/photoion community for some time. For example, back in 2009 [Roberts, Verlet, et al.] (http://scitation.aip.org/content/aip/journal/rsi/80/5/10.1063/1.3126527) demonstrated the first "real time" reconstruction, transforming a 256x256 image in 0.6 seconds.
More recently, a paper from the Wells group demonstrated a <1 sec transform of a 1054x1054 pixel image and showed how this could be used for a feed-back scheme. This was with the onion peeling method that we have partially implemented.
In any case, I think that we have done a bit better with PyAbel. For a 1001x1001 image, @stggh showed that Hansen-Law takes about 0.13 seconds on a laptop, so almost an order of magnitude improvement over the Wells paper.
On my laptop, a 641x641 transform with the Three-Point method can run in 32 ms. This means that we can easily transform standard video resolution (640x480) images at 30 frames-per-second. So, perhaps this is the first demonstration of a "video rate" Abel transform. If we run in parallel on an 8-core, we might be able to claim "HD video" :).

@DanHickstein This looks really cool! I will report back with a plot of my own after I let it run overnight :)
I had given the "real-time" Abel transform some thought in the past, and I think it is definitely feasible. While I don't have any experience with multi-threaded programming, I imagine for transforms like three-point, it shouldn't be too difficult to split up an image into 8 pieces, ship off each piece to a core, do the transform, and restitch the image back together, leading to more speed gains. A similar approach can be utilized if we use a GPU to do the processing, but now I am definitely out of my element here.
For my applications, my data pipeline cannot handle uncompressed RAW video streams at the moment, so the utility of real-time transforms is restricted by rate-limiting steps upstream of PyAbel.

@DanHickstein, sorry to be the dummy here.
I cloned your fork: git clone https://github.com/DanHickstein/PyAbel.git -b benchmark-format
Where is the script to scape the timings and produce the plot?
I think it may be good idea to have benchmark cycle through the non-basis methods first (ordered dict), and print each timing as it becomes available, so that it is evident that the program is running, then followed by the table summary and a graphics at the end. Also, the bs calculations are timed, but now not provided?

Whoops, accidentally edited out the link to the gist while updating the figure. Here it is:
https://gist.github.com/DanHickstein/326d2df2d0a73c14cb1d

@DhrubajyotiDas, I think that the parallel processing could be even easier: each frame can simply be passed to a separate core and processed separately. You might run into problems where frames occasionally arrive out-of-order, but this shouldn't be a huge problem.

@DanHickstein I guess I was thinking of a situation where there isn't a proper distinction between each video frame, and we can think of the video as simply a running stream of rows that need to be inverted, coming in to be processed in batches of 1000 (or whatever the actual frame size is). That way, the cores (however many there are), can simply take however many they want at a time (dependent on the cache size, etc.), process them, and then wait for the next bunch. The situation where the average wait time of each core is close to 0 is the limit on how fast (hopefully "real time") we can process this.
I believe this sort of a row-by-row read is how many camera sensors capture data, so it would probably be very well suited to parallelization.

Yes, good point, I like this idea. There is probably an optimal number of rows that should be passed to each processor depending on the image size and the system architecture.
Too many fun things to try :)

Here it is:

Excellent!  Does all that I was whinging about.

Roman already though about this

I now sort of see where you where heading with the Abel class wrapper ...

In the context of trying to abel transform images at video rates, one way to think about the speeds of the various transforms would be in Megapixels/sec at the various image sizes. Of particular interest are values of ~10 Megapixels/sec, which corresponds to standard video rate (640x480 at 30 fps) and ~30 Mp/s, which corresponds to 30 fps at 720x1280 (HD video) (or, more practically for most experimentalists, 1000x1000 at 30 fps, which is what a lot of scientific cameras are capable of). "1080p" corresponds to 1080x19020x30 = ~60 Mp/s.
For reasonably sized images (in the 1000x1000 to 5000x5000), the Hansen-Law algorithm is fairly stable in terms of Mp/s, sticking around 6 Mp/s on my Macbook. I think that this is basically single-core performance. So, it would take only 5 cores to push this up to 30 Mp/s, which puts HD video rates in range. I'll bet that a suped-up quad-core desktop CPU could process images at HD video rates. I will need to see if we have a fancy CPU in the lab somewhere...

See PR #157 for an update on the benchmark timings.

Hi,
just wanted to say that the number of different implemented algorithms in PyAbel is getting really impressive. Great work everybody!
P.S: I know I still have 2 issues assigned to me. I will do my best to look into those in the next 2 weeks, but  it is probably safe to say that those will not be fixed in 0.7 but rather in 0.8.

@stggh, that looks great! It seems like some competitive spirit has driven all of the speeds faster :)
@rth, good to hear from you! Yes, any of those issues that you would like to take care of would be much appreciated. I hope you are doing well. Do you have a new job?

Updated benchmarks, that make use of fast linear algebra routines. See additional discussion in PR #155
Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz, Ubuntu 15.10

Selected transform methods (that will finish in a sensible time for the large 10k^2 image):
 In [4]: abel.benchmark.AbelTiming(n=[301, 501, 1001, 5001, 10001], select=['hansenlaw', 'onion_peeling', 'two_point', 'three_point'], n_max_bs=10005, n_max_slow=10005)
Out[4]:
PyAbel benchmark run on x86_64

time in milliseconds

=========    Basis Abel implementations ==========

Implementation      n = 301           n = 501           n = 1001          n = 5001          n = 10001
----------------------------------------------------------------------------------------------------------------
onion_peeling_bs       5.4              17.4              78.1            6852.1           49597.6
  three_point_bs      20.0              51.6             143.2            4360.8           18834.6
    two_point_bs       7.4              19.4              81.4            1471.9            6555.7

=========  Forward Abel implementations ==========

Implementation      n = 301           n = 501           n = 1001          n = 5001          n = 10001
----------------------------------------------------------------------------------------------------------------
       hansenlaw      12.5              19.8              59.9            1325.0            4754.0

=========  Inverse Abel implementations ==========

Implementation      n = 301           n = 501           n = 1001          n = 5001          n = 10001
----------------------------------------------------------------------------------------------------------------
       hansenlaw      12.1              27.7              89.4            1510.5            5591.1
   onion_peeling       1.2               5.3              42.0            5491.1           44699.8
     three_point       1.2               5.3              42.0            5544.3           45036.0

I think I'll close this thread just to get it off the main issues page. We can always keep working to make PyAbel faster :)


Can you make Donkey Konga 3 a patch?

I use linux, so I can't run the bat files, and running bat files through wine never works for me. Can you distribute it as a patch file like xdelta,bps, etc?

Donkey Konga 3 currently doesn't compile. However, I would recommend you just use the Linux (Ubuntu only for some reason?) version of bass and slightly modify the bat to an sh as it should be incredibly simple to convert.
Bass for linux: https://github.com/ARM9/bass/releases/download/v18/bass-ubuntu.zip
The next major update will have the proper delta patches in releases.

I'm lazy, so I guess I'll just wait. Thanks though.

Donkey Konga 3 currently doesn't compile.

What does this mean?
When patching I get
$ ./Donkey\ Konga\ 3.sh
error: malformed expression: ""File "fs/Resource/SongInfo.res" is bigger than Size 2132"\n" [unrecognized terminal]
games/Donkey Konga 3/Main.asm:37:1: "{MESSAGE}\n"
bass: assembly failed

BTW you should notice on the main page, that the hash values are from SHA1.

What I mean from what I said is that it doesn't compile, which is literally what you showed. I don't feel like I need to indicate that the hash is SHA1 because it's obvious.
You commented on a closed issue that is unrelated, please do not do this again including this issue.


[Gardening] Use apostrophes around symbols (but not file names) in diagnostic messages

Make almost all diagnostic messages use apostrophes around symbols. Most of them already do. This makes all the messages' (except those with file names) formatting consistent.

@swift-ci please test

@swift-ci please smoke test

Build failed
Swift Test OS X Platform
Git Sha - 9c35f808db20c2084c6f795a4e6bfe806b856b79

Build failed
Swift Test Linux Platform
Git Sha - 9c35f808db20c2084c6f795a4e6bfe806b856b79

@swift-ci please test

@swift-ci please smoke test

Build failed
Swift Test OS X Platform
Git Sha - 9c35f808db20c2084c6f795a4e6bfe806b856b79

@swift-ci please smoke test

@swift-ci please smoke test


How to set absolute path in "Specify Deployer File" (-f, --file) on Windows

dep -f"C:\Path\To\deploy.php" will be converted to C:\current\working-dir/C:/Path\To\deploy.php because it always prepends the current working directory, when the given path does not start with a slash. But this will fail on Windows, always, cause you need to prepend the drive letter.
Cygwin would work, but this should also work on native Windows CLI, imo.

Btw. when you change
$deployFile = ($userSpecifiedFile[0] === '/' ? '' : getcwd() . '/') . $userSpecifiedFile;

To
$deployFile = ($userSpecifiedFile[0] === '/' || preg_match('/^\w{1,2}\:.*/', $userSpecifiedFile) ? '' : getcwd() . '/') . $userSpecifiedFile;

It also works for absolute Windows paths. It checks not just for starting /, it also checks for "C:" or even "AB:" using a simple regular expression.

Done!


fix

add Convert To String ${step_rate} and  ${budget}

This change is

Reviewed 1 of 1 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


Karate netty: Multiple feature file -m (mocks) is rolled back

Karate netty no longer accepts multiple feature file. This feature was added and merged in 0.9.6 as part of https://github.com/intuit/karate/pull/1159.
In 1.0.0 FeatureServer was replaced MockServer in Main.java class and this https://github.com/intuit/karate/issues/1281 change rolled back the feature.
Please add the option of accepting multiple feature file backs.
Diff:
-- @Option(names = {"-m", "--mocks"}, description = "mock server file(s)")
--    List mocks;
++ @Option(names = {"-m", "--mock"}, description = "mock server file")
++   File mock;

this is a wontfix because it added more complexity and we re-wrote the mock from scratch. please contribute code if you want this back

@ptrthomas  Can you please provide pointer where should I start looking? I see that FeatureServer has been replaced by MockServer. If you can just list key files that I can start with to add this change back, I can work on it.

@dinesh19aug sure, here is an explanation on what went behind this change and thoughts on how it can be brought back: https://github.com/intuit/karate/issues/1338
and the MockHandler is the main file: https://github.com/intuit/karate/blob/4db11f0012aa89a6de008550ce50596e0457eb06/karate-core/src/main/java/com/intuit/karate/core/MockHandler.java
when you submit a PR you can use this issue as a reference

Hi,
@ptrthomas @dinesh19aug  we are also interested on multiple mock features, something simple, first feature in order that matches an scenario serves de request..
@dinesh19aug if you can help me test this https://github.com/ivangsa/karate/commits/features/mockHandler
other improvements we are interested also are configuring a prefix (or contextPath) which is already include above
and runtime-hooks for intercepting request/response which may involve a bit of work more

@dinesh19aug yes we have a similar use case, some endpoints are complex enough to go on a separate file, while others are simple and can be combined on one feature...
The refactoring is just in the MockHandler, instead of keeping just one feature/scenario-runtime variable I create a list of features/runtimes... but each are instantiated and executed just the same... everything else is just augmenting the API for passing several feature files..

Ok. I will test it and add comment by Friday Eod

@ivangsa  - I verified the solution fix and able to make use of multiple feature file after creating stand alone jar. Added some code refactoring for big methods in MockHandler. Please review PR - https://github.com/ivangsa/karate/pull/1

@ptrthomas / @ivangsa - There is another enhancement which I think would be very useful.
Summary:
Given a directory, iterate recursively and find all the feature file. Automatically add the multiple feature file.
EX:
/Users/dinesh/karate/featureDir

api1

ap1.feature
api1-feature1-response.json

api2

ap2.feature
api2-feature1-response.json

Assume that -F ==> Parent Feature directory
When I start java -jar karate.jar -p 3000 -F /Users/dinesh/karate/featureDir
The karate server start up would be equivalent to
java -jar karate.jar -p 3000 -m  /Users/dinesh/karate/featureDir/api1/api1.feature -m /Users/dinesh/karate/featureDir/api2/api2.feature
Let me if you see this as useful enhancement and I can create a new enhancement and start working on it. I used this feature for POC at work and it saves me tons of time instead of writing command line -m  multiple time

@dinesh19aug I'm not sure about this especially how to determine the order of finding files. I would prefer explicit and via the api.

@dinesh19aug if you can guaranty same order every time (i.e alphabetical) it should be ok as long a there is not "default" scenario..
Maybe you can reuse "--mocks" options as it is a list of  File and can hold a directory as well
For me it's ok... I will merge your changes now so you can manage this on top if @ptrthomas it's ok with that

@dinesh19aug if you can guaranty same order every time (i.e alphabetical) it should be ok as long a there is not "default" scenario..
Maybe you can reuse "--mocks" options as it is a list of File and can hold a directory as well
For me it's ok... I will merge your changes now so you can manage this on top if @ptrthomas it's ok with that

@ptrthomas / @ivangsa  - I am trying to understand why would ordering matter here? Ex - I have 10 feature files, why would  ordering matter if feature file 10 is loaded before feature file 1? The code is eventually, just creating a list of scenarios from all files.  If you can provide a criteria, I can provide an option load the file in order - by name(asc/desc), number of scenarios in each file or may be based on file timestamp.
@ptrthomas  - Yes. You are right, currently that is what I am doing, I am using --mocks options programmatically to load all the files in directory and then creating the list of feature files. However, for command line, if your 30+ files like I have, I would not want to write -m  so many times.

@dinesh19aug fine, fine, you all can do what you want - if you read the previous threads it may be clear why the previous PR submitter was very keen on having "precedence rules" - also people may use a blank scenario to "catch all", what happens if the same path appears in 2 files, and what is the order in which you recurse sub-directories
I'd like to stick to --mock if you don't mind unless you can figure a way to alias --mock and --mocks which may be possible with picocli

following back, we find this feature very useful: spliting mocks in several feature files (and also configuring a prefx/contextPath)
obviously paths can no be repeated in different files or first one will take precedence.. and default scenario would only make sense in last feature
there is already a draft PR #1570 already for this... if you guys agree I can update it to open

@ptrthomas / @ivangsa  - This is how I tested and it matches the the intended behavior:
Test 1: Single feature file with same paths
Given mock files test1.feature
When two scenarios with same path exist - /test exists
Then first scenario is picked up.
test1.feature
Feature: stateful mock server
The response file Bye.txt will be displayed
Scenario: pathMatches('/test')

def response = read('/example/Bye.txt')

Scenario: pathMatches('/test')

def response = read('/example/hi.txt')

Test 2: Multiple feature file with same path
Given mock files test1.feature and test2.feature start the server as  java -jar karate.jar -m test1.feature -m test2.feature -W
When two scenarios with same path exist - /test exists in test1.feature and test2.feature
Then first mock file's first scenario is picked up.
test1.feature
Feature: stateful mock server
The response file Bye.txt will be displayed
Scenario: pathMatches('/test')

def response = read('/example/Bye.txt')

test2.feature
Feature: stateful mock server
Scenario: pathMatches('/test')

def response = read('/example/hi.txt')

Started the server as java -jar karate.jar -m test1.feature -m test2.feature -W
Based on this I am getting the correct behavior.

@dinesh19aug thanks - re-opening

merged ! thanks all. would be nice if someone contributes documentation (just a few lines should do) we don't have a date yet in mind for release

@ptrthomas I will add some documentation

Updated documentation: https://github.com/intuit/karate/pull/1585

1.1.0 released


memory locking requested for elasticsearch process but memory is not locked

Hello guys!
My Host is Centos7. All components fully updated. I have commented out the single-node parameter in the elasticsearch.yml file. I am trying to cluster to an external host.  My .yml file is as follows:
## Default Elasticsearch configuration from elasticsearch-docker.
## from https://github.com/elastic/elasticsearch-docker/blob/master/build/elasticsearch/elasticsearch.yml
#
cluster.name: "docker-cluster"
network.host: <IP_ADDRESS>
network.publish_host: <IP_ADDRESS>
node.master: true
node.data: true
bootstrap.memory_lock: true
#discovery.type: single-node

In /etc/security/limits.conf I have set:
elasticsearch - nofile 65535
elasticsearch - memlock unlimited
root - memlock unlimited
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
docker - nofile 65535
docker - memlock unlimited
docker soft memlock unlimited
docker hard memlock unlimited
user hard memlock unlimited
user soft memlock unlimited
user - memlock unlimited

also set --
sysctl -w vm.swappiness=1;
sysctl -w vm.max_map_count=262144;

Any ideas why when trying to start the stack with "docker-compose up", I get elasticsearch failing with error - "memory locking requested for elasticsearch process but memory is not locked"
2/6/2018 11:16:58 PM	# allow user 'elasticsearch' mlockall
2/6/2018 11:16:58 PM	elasticsearch soft memlock unlimited
2/6/2018 11:16:58 PM	elasticsearch hard memlock unlimited

Thanks!!

You also have to set the ulimits settings properly on the container itself, and probably (can't remember for sure) run it with extra privileges.
Please note that setting user specific limits on the host is irrelevant here.
I've seen that before myself:
https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
https://github.com/pires/docker-elasticsearch/issues/7

Thank you!!  Wow, thats a good find....    testing this today.

Added the following and works perfectly!  Thank you...
version: '2'

services:

  elasticsearch:
    restart: always
    build:
      context: elasticsearch/
    volumes:
      - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx2048m -Xms2048m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - elk

Glad it helped :)

The ulimits options are ignored with swarm, what are the options to running elastic search in a docker stack as a swarm service?
Have tried these two options below for memory allocation but neither seem to work?
version: '3.4'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.3
container_name: elasticsearch
environment:
- cluster.name=docker-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "TZ=Europe/Helsinki"
deploy:
resources:
reservations:
memory: 20M
limits:
memory: 30M
ulimits:
memlock:
soft: -1
hard: -1
volumes:
  - elasticsearch:/usr/share/elasticsearch/data
ports:
  - 9200:9200
networks:
  - logging_nw

Error in the Elastic container in the service:
Logging_elasticsearch.1.m7q7t895wy6f@privxidaas    | [2018-03-30T20:26:03,547][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
Logging_elasticsearch.1.m7q7t895wy6f@privxidaas    | [2018-03-30T20:26:03,568][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
Logging_elasticsearch.1.m7q7t895wy6f@privxidaas    | [2018-03-30T20:26:03,568][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
Logging_elasticsearch.1.m7q7t895wy6f@privxidaas    | [2018-03-30T20:26:03,569][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example:
Logging_elasticsearch.1.m7q7t895wy6f@privxidaas    |    # allow user 'elasticsearch' mlockall
Logging_elasticsearch.1.m7q7t895wy6f@privxidaas    |    elasticsearch soft memlock unlimited
Logging_elasticsearch.1.m7q7t895wy6f@privxidaas    |    elasticsearch hard memlock unlimited
Logging_elasticsearch.1.m7q7t895wy6f@privxidaas    | [2018-03-30T20:26:03,569][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
Logging_elasticsearch.1.m7q7t895wy6f@privxidaas    | [2018-03-30T20:26:04,478][INFO ][o.e.n.Node               ] [] initializing ...
Logging_elasticsearch.1.m7q7t895wy6f@privxidaas    | [2018-03-30T20:26:04,735][INFO ][o.e.e.NodeEnvironment    ] [rdGPLKS] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vda1)]], net usable_space [105.5gb], net total_space [116.2gb], types [ext4]
Logging_elasticsearch.1.m7q7t895wy6f@privxidaas    | [2018-03-30T20:26:04,735][INFO ][o.e.e.NodeEnvironment    ] [rdGPLKS] heap size [495.3mb], compressed ordinary object pointers [true]
Logging_elasticsearch.1.m7q7t895wy6f@privxidaas    | [2018-03-30T20:26:04,737][INFO ][o.e.n.Node               ] node name [rdGPLKS] derived fro

It wored for me

Had the same issue while using swarm. Since it's not natively supported by the service definitions, you could set ulimits at the docker's service level.
Found a neat article while looking into this that details how to do this for distros using systemd or upstart:
http://tostr.pl/blog/setting-ulimits-for-docker-process-2
It boils down to editing the service configuration files to include the respective ulimits clauses.
One possibility of this being a thing with swarm is because you're not instrumenting a single docker instance.  These definitions will eventually become containers (aka tasks) running on different docker service instances (aka nodes). These nodes may differ wildly in terms of their relationship with their host machine, such as ulimits and other OS related features. It's up to you to provision the nodes accordingly unless Docker will implement some sort of cross-node daemon service configuration.


fix(vue-app): add fallback context for error

Types of changes

[X] Bug fix (a non-breaking change which fixes an issue)
[ ] New feature (a non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Description

Fixes https://github.com/nuxt/nuxt.js/issues/6517#issuecomment-573280207
Checklist:

[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly. (PR: #)
[ ] I have added tests to cover my changes (if not applicable, please state why)
[X] All new and existing tests are passing.

Codecov Report

Merging #6854 into dev will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##              dev    #6854   +/-   ##
=======================================
  Coverage   64.75%   64.75%
=======================================
  Files          78       78
  Lines        2721     2721
  Branches      708      708
=======================================
  Hits         1762     1762
  Misses        730      730
  Partials      229      229

Flag
Coverage Δ

#e2e
100% <ø> (ø)
:arrow_up:

#unit
64.75% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c9beac3...41f2e20. Read the comment docs.


Scroll to current playing song

Is your feature request related to a problem? Please describe.
When clicking the next button and looking at the next song in the playlist, I might need to scroll to find it.
Describe the solution you'd like
If the current playing song is below the current view, scroll down if clicking next.
Having a better UI/UX, if the current song is just at the bottom of the view and you click on next, it should follow it in the playlist view.

Counter argument: I may be browsing the playlist and also wish to skip the current track. If the playlist would then automatically scroll to the next treack, I'd lose my previous scroll position.
Instead, I believe doing what the official Spotify client does would be better: allow some user action to scroll to the currently playing song in its respective context (e.g. playlist, album, artist). Currently, this happens when you click on the album cover in the bottom left.


gandi.acme.molnett.net is forbidden

Out of a fresh installation closely following steps by the README, the plugin got stuck at this:
I0408 18:33:11.036144       1 dns.go:88] "presenting DNS01 challenge for domain" logger="cert-manager.challenges.Present" resource_name="wildcard-cert-1-2779278163-3683114198" resource_namespace="default" resource_kind="Challenge" resource_version="v1" dnsName="{REQUESTED_DOMAIN_HERE}" type="DNS-01" resource_name="wildcard-cert-1-2779278163-3683114198" resource_namespace="default" resource_kind="Challenge" resource_version="v1" domain="{REQUESTED_DOMAIN_HERE}"
E0408 18:33:11.037813       1 controller.go:167] "re-queuing item due to error processing" err="gandi.acme.molnett.net is forbidden: User \"system:serviceaccount:cert-manager:cert-manager\" cannot create resource \"gandi\" in API group \"acme.molnett.net\" at the cluster scope" logger="cert-manager.challenges" key="default/wildcard-cert-1-2779278163-3683114198"

More info:
kubectl describe challenge wildcard-cert-1-UUID
Name:         wildcard-cert-1-UUID
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  acme.cert-manager.io/v1
Kind:         Challenge
Metadata:
 {...}
Spec:
  Authorization URL:  https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/egefefefefefefef
  Dns Name:           {REQUESTED_DOMAIN_HERE}
  Issuer Ref:
    Kind:  Issuer
    Name:  letsencrypt-staging
  Key:     OJOEFJEOFJEOFJEOFJEOFJEOFJEOFJEF
  Solver:
    dns01:
      Cname Strategy:  Follow
      Webhook:
        Config:
          API Key Secret Ref:
            Key:        api-token
            Name:       gandi-credentials
          Root Domain:
        Group Name:     acme.molnett.net
        Solver Name:    gandi
  Token:                ZZZZZZZZZZ
  Type:                 DNS-01
  URL:                  https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/ZZZZZZZZ
  Wildcard:             true
Status:
  Presented:   false
  Processing:  true
  Reason:      gandi.acme.molnett.net is forbidden: User "system:serviceaccount:cert-manager:cert-manager" cannot create resource "gandi" in API group "acme.molnett.net" at the cluster scope
  State:       pending
Events:
  Type     Reason        Age                  From                     Message
  ----     ------        ----                 ----                     -------
  Normal   Started       4m5s                 cert-manager-challenges  Challenge scheduled for processing
  Warning  PresentError  100s (x6 over 4m5s)  cert-manager-challenges  Error presenting challenge: gandi.acme.molnett.net is forbidden: User "system:serviceaccount:cert-manager:cert-manager" cannot create resource "gandi" in API group "acme.molnett.net" at the cluster scope

In case you're still having this issue, make sure you're setting groupName in the chart values and the issuer.

In case you're still having this issue, make sure you're setting groupName in the chart values and the issuer.

This was the reason, maybe the values.yaml should be updated to reflect this change.

This should be fixed after merging #5. Thanks for notifying us!


[ALLUXIO-2709] Specify null variable in FileBlockInfo.java precondition check

ALLUXIO-2709
Specify null variable in FileBlockInfo.java precondition check(Located at Line 87 and 106)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/14381/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/14382/
Test PASSed.

alluxio-bot, check this please


[!] 8.3 File Name Removal Instructions

Running the script results in no actions, sadly no logs or anything in event viewer either.
https://schneegans.de/windows/no-8.3/#wim
I took install.wim, the ps1 script, and psexec, put them in the folder structure mentioned as well as E:\z\ for good measure.
Simply goes to next line if run in bundled PowerShell, or asks about security then goes to new line in newer PowerShell.
"
PS C:\iso\sources> .\Remove-WimShortNames -WimFile 'C:\iso\sources\install.wim';
PS C:\iso\sources> .\Remove-WimShortNames -WimFile 'C:\iso\sources\install.wim';
or
PS C:\iso\sources> .\Remove-WimShortNames -WimFile 'C:\iso\sources\install.wim';
Security warning
Run only scripts that you trust. While scripts from the internet can be useful, this script can potentially harm your
computer. If you trust this script, use the Unblock-File cmdlet to allow the script to run without this warning
message. Do you want to run C:\iso\sources\Remove-WimShortNames.ps1?
[D] Do not run  [R] Run once  [S] Suspend  [?] Help (default is "D"): R
PS C:\iso\sources>
"
I did also run unblock-file to get rid of the security warning in new PowerShell and it returns the same behavior as old PowerShell
EDIT: I also checked the script and changed the $PsExecPath to the same dir, the path in the script doesn't seem to exist as there is no installer for the tools, the windows store installer also doesn't put it there

PS C:\iso\sources> .\Remove-WimShortNames -WimFile 'C:\iso\sources\install.wim';

You first have to execute the function definition. The idea is to copy the entire function definition to the clipboard and paste it in a PowerShell window. Alternatively, you can download the .ps1 file with your browser and then type:

. "$env:USERPROFILE\Downloads\Remove-WimShortNames.ps1";

You then have to call the newly defined function by typing:

Remove-WimShortNames -WimFile 'C:\iso\sources\install.wim';

Running this function requires an elevated PowerShell session with administrative rights.

"You first have to execute the function definition. The idea is to copy the entire function definition to the clipboard and paste it in a PowerShell window. Alternatively, you can download the .ps1 file with your browser and then type:"
Correct, i had it downloaded from the site, PowerShell treats it no different putting .ps1 on the end or not, but I did try it both ways just-in-case
All done elevated as well which still did not work, will try that first one.

Closing for now, will circle back to this one with some edits later


fixes for CVE-2023-44487

require go 1.20
upgrade k8s and x/net mods
disable HTTP2 in webhook server

"[1/2] STEP 1/15: FROM golang:1.17-alpine3.16 AS builder" - dunno where it's pulling this from since release specifies "tag: golang-1.20"

/retest

/lgtm


Completion dropdown constantly reappears after selecting an option

With ale_completion_enabled and no other settings changed, the completion dropdown automatically appears as I'm typing, and selects the first option every time I press Enter. However, the completion dropdown appears again almost immediately afterwards, so it's impossible to actually insert a newline unless you press Enter twice in very rapid succession.
This makes it really hard to write code given that the autocomplete results appear automatically and I'm not necessarily anticipating them every time I need to insert a new line.
Here's a video - whenever you see the menu flash, I'm pressing enter: https://cloudup.com/c8TXp9UnAIm+
If this is intentional behavior, I'd love to know either (a) how to disable it or (b) how to only show autocomplete results when I press Tab.

ALEInfo

 Current Filetype: typescript
Available Linters: ['eslint', 'tslint', 'tsserver', 'typecheck']
  Enabled Linters: ['eslint', 'tslint', 'tsserver', 'typecheck']
 Linter Variables:
let g:ale_typescript_tslint_config_path = ''
let g:ale_typescript_tslint_executable = 'tslint'
let g:ale_typescript_tslint_ignore_empty_files = 0
let g:ale_typescript_tslint_rules_dir = ''
let g:ale_typescript_tslint_use_global = 0
let g:ale_typescript_tsserver_config_path = ''
let g:ale_typescript_tsserver_executable = 'tsserver'
let g:ale_typescript_tsserver_use_global = 0
Global Variables:
let g:ale_echo_cursor = 1
let g:ale_echo_msg_error_str = 'Error'
let g:ale_echo_msg_format = '%s'
let g:ale_echo_msg_warning_str = 'Warning'
let g:ale_enabled = 1
let g:ale_fix_on_save = 0
let g:ale_fixers = {}
let g:ale_keep_list_window_open = 0
let g:ale_lint_delay = 200
let g:ale_lint_on_enter = 1
let g:ale_lint_on_save = 1
let g:ale_lint_on_text_changed = 'always'
let g:ale_linter_aliases = {}
let g:ale_linters = {}
let g:ale_open_list = 0
let g:ale_set_highlights = 1
let g:ale_set_loclist = 1
let g:ale_set_quickfix = 0
let g:ale_set_signs = 1
let g:ale_sign_column_always = 0
let g:ale_sign_error = '>>'
let g:ale_sign_offset = 1000000
let g:ale_sign_warning = '--'
let g:ale_statusline_format = ['%d error(s)', '%d warning(s)', 'OK']
let g:ale_warn_about_trailing_whitespace = 1
Command History:
(started) ['/bin/zsh', '-c', 'cd ''/Users/dylan/dev/cala/studio/src/redux/options'' && /Users/dylan/dev/cala/studio/node_modules/.bin/tslint --format json -c ''/Users/dylan/dev/cala/studio/tslint.json'' ''/var/folders/1s/jy19st5d3cx8tzdz0zymzv4m0000gn/T/vGsnikX/72/reducer.ts''']
(executable check - failure) typecheck
(executable check - failure) eslint
(started) ['/bin/zsh', '-c', 'cd ''/Users/dylan/dev/cala/studio/src/redux/options'' && /Users/dylan/dev/cala/studio/node_modules/.bin/tslint --format json -c ''/Users/dylan/dev/cala/studio/tslint.json'' ''/var/folders/1s/jy19st5d3cx8tzdz0zymzv4m0000gn/T/vGsnikX/73/reducer.ts''']
(executable check - failure) typecheck
(executable check - failure) eslint
(started) ['/bin/zsh', '-c', 'cd ''/Users/dylan/dev/cala/studio/src/redux/options'' && /Users/dylan/dev/cala/studio/node_modules/.bin/tslint --format json -c ''/Users/dylan/dev/cala/studio/tslint.json'' ''/var/folders/1s/jy19st5d3cx8tzdz0zymzv4m0000gn/T/vGsnikX/74/reducer.ts''']
(executable check - failure) typecheck
(executable check - failure) eslint
(finished - exit code 0) ['/bin/zsh', '-c', 'cd ''/Users/dylan/dev/cala/studio/src/redux/options'' && /Users/dylan/dev/cala/studio/node_modules/.bin/tslint --format json -c ''/Users/dylan/dev/cala/studio/tslint.json'' ''/var/folders/1s/jy19st5d3cx8tzdz0zymzv4m0000gn/T/vGsnikX/75/reducer.ts''']
<<>>
[]
<<>>
(executable check - failure) typecheck
(executable check - failure) eslint
(finished - exit code 0) ['/bin/zsh', '-c', 'cd ''/Users/dylan/dev/cala/studio/src/redux/options'' && /Users/dylan/dev/cala/studio/node_modules/.bin/tslint --format json -c ''/Users/dylan/dev/cala/studio/tslint.json'' ''/var/folders/1s/jy19st5d3cx8tzdz0zymzv4m0000gn/T/vGsnikX/76/reducer.ts''']
<<>>
[]
<<>>
(executable check - failure) typecheck
(executable check - failure) eslint
(finished - exit code 0) ['/bin/zsh', '-c', 'cd ''/Users/dylan/dev/cala/studio/src/redux/options'' && /Users/dylan/dev/cala/studio/node_modules/.bin/tslint --format json -c ''/Users/dylan/dev/cala/studio/tslint.json'' ''/var/folders/1s/jy19st5d3cx8tzdz0zymzv4m0000gn/T/vGsnikX/77/reducer.ts''']
<<>>
[]
<<>>
(executable check - failure) typecheck
(executable check - failure) eslint
(finished - exit code 0) ['/bin/zsh', '-c', 'cd ''/Users/dylan/dev/cala/studio/src/redux/options'' && /Users/dylan/dev/cala/studio/node_modules/.bin/tslint --format json -c ''/Users/dylan/dev/cala/studio/tslint.json'' ''/var/folders/1s/jy19st5d3cx8tzdz0zymzv4m0000gn/T/vGsnikX/78/reducer.ts''']
<<>>
[]
<<>>
(executable check - failure) typecheck

Hey! Someone else actually tried the completion support I added.
I hadn't hit that bug myself, but I can repeat it. The problem occurs if you don't select anything for completion, and hit Return. If you select an item with Tab and then press Return, then the menu goes away.

Someone else actually tried the completion support I added

Yes! And very happy to have found something faster than tsuquyomi (in fact, your comment here let me to Ale). Thanks for your work on this!

This should be fixed now. Now ALE doesn't ask for completions shortly after CompleteDone.

I have updated the fix for this. Before it would wait 100ms before requesting completions again. Now it doesn't ask for completions again if the cursor hasn't moved. This makes it so you can type Boolean, hit return, then quickly type . to get the completions again, without the menu re-appearing in an annoying way.


Missing files.

Hey,
It appears that after 2015-01-07-13.json.gz there are no more files.

Thanks for the heads up, should be fixed now!


IS VAVR AFFECTED BY A SPRING-FRAMEWORK VUINERABILITY？（CVE-2022-22965）

IS VAVR AFFECTED BY A SPRING-FRAMEWORK VUINERABILITY？（CVE-2022-22965）

Vavr has zero dependencies, especially it has no dependency to Spring.


feat: test cases to check contract storage reverts for failed transactions

Currently, the Madara storage doesn't revert storage changes in case a smart contract transaction fails. I am referring to execution failures here, for example, you add a panic! inside the function call of a smart contract. Not other failures like missing fees etc. This should mostly be fixed in a few days. However, we need a good test case for this to ensure we don't miss this later on.

I would like to take this up


whether the results in tables are run on dev set?

I run your models, but can not achieve your results on test set.

The result are based on test set. I've done the greedy search of hyperparameter, for example random seed, lr_rate to get the best result. The hyperparameter written in main.py might not guarantee you to get best result.

Can you share your hyperparameter ?  Appreciate it!

Can you share your hyperparameter ? Appreciate it!

Sadly, I don't have the hyperparameter that I used:(
Is the difference if big? Maybe as I change the architecture of the repo recently, and it could make the difference of the result.

yes, the difference of the result big.
On snips, I can only achieve 91%  on sentence acc, 96% on slot f1.

Use bert

Can you share your hyperparameter ? Appreciate it!

As I remember, just changing the seed gives a big difference of the result.
I recommend you to refer this paper

yes, the difference of the result big.
On snips, I can only achieve 91% on sentence acc, 96% on slot f1.

I'll check that one and let you know

Thanks for your help!

yes, the difference of the result big.
On snips, I can only achieve 91% on sentence acc, 96% on slot f1.

I've just changed the seed to 771 and run it. Above is the test result I got. I'm sorry that I forgot the exact hyperparameter that I used to get the best result, but by controlling some hyperparameter (e.g. seed, lr), you could get the better result.

Oh, thanks a lot. It seems that the selection of the random seed is the key. I will follow your work and try more !

Great:) I'll close this issue. Feel free to reopen if I haven't fully answered your questions.


Serialization: Refine safety checks for extensions/protocols

In https://github.com/apple/swift/pull/65267 deserialization safety was made more conservative, allowing deserialization of any protocol and deserialization of any extension declaring an explicit conformance to any protocol. We can refine this to only allow deserialization of safe protocols and deserialization of extensions declaring conformances to safe protocols. Importantly, though, we must look up all conformances declared by the extension, not just the explicit ones.
Resolves rdar://114673761

@swift-ci please test

@swift-ci please test

@swift-ci please test


Live player additional bug fix

🧰 Ticket
Closes #61 and #62
🚀 Overview:
Add conditional for all roles template selections
🤔 Reason:
Adding new force with all roles selected, to a channel caused template to break
🔨Work carried out:

[x] Add conditional template selection for all roles
[x] Refactor template selection logic
[x] Tests pass

@zukilover - it looks like the e2e tests did their job.   The player UI crashes when logging in as a player.
While I can see the e2e results in Cypress, I don't know if you can.  I've made the Cypress project public. Hopefully you can click on the Cypress link, login via GitHub, and watch the video replay of the failed test.
Alternatively, you should be able to run yarn test:e2e - and see Cypress running locally.

Serge does have quite complex logic when it comes to forces. roles & templates.
When things quieten down, I guess it may be worth re-factoring the logic.  This may make it more reliable.  It may also make it easier to test - so we can use our Unit Tests to ensure the logic carries on working.
I look forward to that.  But, in the near term we could really do with this branch being fixed.  If you think the above refactoring should be part of the fix, I'm quite happy to sponsor it.

@zukilover  - here is the stack trace:

The e2e test is a walkthrough of this web-page:
https://sites.google.com/deepbluec.com/serge/introduction?authuser=0
But, I appreciate it's a lot of effort to get to that point.
I've just reproduced the problem locally by running yarn build in your branch, then yarn serge in one terminal, and yarn test:e2e in another.  The test fails, but it leaves an instance of the breaking wargame in the local server.
So, it's then possible to connect to the local server and go into the game. When I did that, any player (other than Game Control) caused the UI to fail.
Looking at the stack-trace, I think I've got it. Here is playerUI_Reducer:

If the method drops out at line 60 then there isn't a structure to decompose at line 75.


[IMPROVEMENT] fix contrast for blue text

This dark blue color is very hard to see on a dark-themed console. For example:
Mint-Y from Linux Mint os.

Darcula from JetBrains tools.

I had to select text in the console to read what was written there.

wow, it really is. i'll try and get to that soon. most important for dotenvx

@gander are you still using .env.vault files or have you moved to just dotenvx's encrypted .env files?

@motdotla I saw the mentioned blue color during the migration from Vault to a newer format, but I also saw these colors in other CLI commands.


Update enable-attack-surface-reduction.md

Added clarification for where to find the OMA-URI value to use for the example ASR rule.
This was not immediately clear that the reader needed to reference another part of the page for the specific value to use when following the example.

@JohanFreelancer9 - kindly do the copy editor review for this PR. Thanks


enhance property for CHyperlink

see https://github.com/whistyun/Markdown.Avalonia/issues/4

If it can be exposed the hyperlink mouse hover and leave after clicked effect could be more appreciate.

see #4

If it can be exposed the hyperlink mouse hover and leave after clicked effect could be more appreciate.

Thanks a lot for your attentions and patience!

In next version, I support pseudoclasses.
It can provide :hover :pointerhover, :pressed for CHyperlink.
after clicked (maybe visited?) is difficult, Because I don't know how to access hisotry of browser.
It is possible to provide some pseudoclasses meaning clicked while the application is running, However is it usefull?
<Style Selector="ctxt|CHyperlink">
    <Style.Setters>
        <Setter Property="IsUnderline" Value="true"/>
        <Setter Property="Foreground" Value="Green"/>

        <Setter Property="Padding" Value="2,1" />
        <Setter Property="BorderBrush" Value="Green"/>
        <Setter Property="BorderThickness" Value="2"/>
      </Style.Setters>
    </Style>

    <!--
           it is equls to :pointerover.
           Because Avalonia-team may change `:pointerover` to `:hover`.
     -->
    <Style Selector="ctxt|CHyperlink:hover">
      <Style.Setters>
        <Setter Property="IsUnderline" Value="true"/>
        <Setter Property="Foreground" Value="Red"/>

        <Setter Property="Padding" Value="5"/>
        <Setter Property="BorderBrush" Value="Red"/>
        <Setter Property="BorderThickness" Value="2"/>
      </Style.Setters>
  </Style>

  <Style Selector="ctxt|CHyperlink:pressed">
      <Style.Setters>
        <Setter Property="IsUnderline" Value="true"/>
        <Setter Property="Foreground" Value="White"/>
        <Setter Property="Background" Value="DarkGreen"/>

        <Setter Property="Padding" Value="2"/>
        <Setter Property="BorderBrush" Value="Gray"/>
        <Setter Property="BorderThickness" Value="2"/>
    </Style.Setters>
</Style>


Avoid intermediate JSON object allocations when generating JSON strings

Use jsontool library to avoid generating intermediate JSON objects when
generating JSON strings from messages.
Original proto3 code moved from #670. Generator for other JSON format is
updated to use jsontool as well.
Splits JSON parsers and generators into separate files.
AOT runtime

Before
After
Diff

ToJsonObject
10304 μs
11175 μs
+871 μs, +8.45%

ToJsonString
35030 μs
25946 μs
-9084 μs, -25.93%

ToProto3JsonObject
10812 μs
10824 μs
+12 μs, +0.11%

ToProto3JsonString
35906 μs
26758 μs
-9148 μs, -25.48%

JS runtime

Before
After
Diff

ToJsonObject
19637 μs
23313 μs
+3676 μs, +18.72%

ToJsonString
71535 μs
45477 μs
-26058 μs, -36.43%

ToProto3JsonObject
22318 μs
20323 μs
-1995 μs, -8.94%

ToProto3JsonString
71642 μs
50000 μs
-21642 μs, -30.21%

Benchmark binary sizes

Before
After
Diff

AOT
6,990,992 bytes
7,010,576 bytes
+19,584 bytes, +0.28%

JS
338,060 bytes
339,829 bytes
+1,769 bytes +0.52%

Internal
jsontool library added to internal in cl/466006106
This PR ported to internal in cl/473230561
A large JS release binary size reduced by 0.22%.

I benchmarked this PR on Golem: https://golem.corp.goog/Revision?repository=protobuf&revision=84839&patch=17197
This PR has no changes in binary serialization or deserialization (only changes JSON), but Golem reports 5% increase in a binary serialization benchmark (to_binary.dart in benchmarks/).
It seems like Golem results can also be quite noisy and cannot be trusted.

@osa1 LGTM

The jsontool package is a third-party package which is not supported by the Dart team.
Would you consider forking or vendoring it?

We should add GeneratedMessage members to generate encoded JSON, ready for writing to a socket.
Secondly, we can keep the current toJsonObject implementation as it's faster on both AOT and JS.


Translate README.md to pt-br ? [question]

I volunteer to translate the readme.md file to pt-br if that's wanted, I think the repo's description should be available for Brazilian who may not know English well enough. If that's needed I wish to be assigned to do it.

Please, refer to #87 ; )


Predefined type 'System.Object' is not defined or imported

I get this error message in C# (.cs) files within a project in Unity on a class name when I attempt to define a class.
I have the latest version of Mono installed, running on Windows 8.
Any ideas how I can overcome this basic error?!

Do you have dnvm installed? https://github.com/aspnet/Home
omnisharp-atom needs 1.0.0-beta4

I'll try that then post again, thank you.
I'm amazed neither the GitHub project homepage nor the main wiki page for this project mentions this requirement!

It's here https://github.com/OmniSharp/omnisharp-atom/wiki#runtime-prerequisites
Could be clearer though :)

Updated readme - https://github.com/OmniSharp/omnisharp-atom#runtime-prerequisites

Installing Visual Studio 2015 RC fixed the issue - thanks for the link :-)


CSRF Tokens - Submit API

Context
CSRF Tokens can be generated by calling ‘/api/auth/csrf’ and are valid for the lifetime of the session.
Issue

Can repeatedly call the ‘submit’ API with fake form submissions by sending requests with a CSRF Token.  The API will attempt to create the form submission in the Reliability Queue.

Impact

3rd Party service impact (AWS Lambdas, SQS, Dynambo DB)... flooding the Reliability Queue and Notify sending emails.
Hitting the maximum number of emails with Notify service.  This impacts GCForms users as no Form Responses, Testing Data Delivery, User Registrations, or passwords will be able to be completed.
Cleanup of Reliability Queue for undeliverable form submissions.
Attacker discovering both Draft and Published Form ID’s based on API error responses

Acceptance Criteria:

[x] A form submission is only created in the reliability queue if the form ID exists in the database.
[x] Request rate limiting set at 100 reqs / 5min.

Request rate set to 100 req/ 5 min in Production and Staging
Submission API verifies if forms exists before submitting to Reliability Queue.


Left/right view bob when walking should be user-configurable

Would appreciate an option by default to configure the viewbob that occurs when you walk / run, it's pretty intense by default!

This is already included (though not named that well I admit).
Try tweaking the Wiggle parameters on the player.gd script:


[Release] sdk/resourcemanager/compute/armcompute/6.0.0

https://github.com/Azure/sdk-release-request/issues/5299

The breaking changes are introduced and approved in this PR.

/azp run go - armcompute - ci

The breaking changes are introduced and approved in this PR.

@JeffreyRichter , please approve this major version release by adding label of Approved Major Version, thanks.


Prepend array length in FeltArray serialization

Describe your changes

prepend array length to the returned result in FeltArray.toCalldata
update felt array is convertible to calldata test

Linked issues

Closes #433
Breaking changes

[ ] This issue contains breaking changes

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 69.26%. Comparing base (d28b008) to head (357b8d1).
Report is 3 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #458      +/-   ##
==========================================
- Coverage   71.12%   69.26%   -1.86%
==========================================
  Files          75       75
  Lines        3338     3254      -84
  Branches      367      361       -6
==========================================
- Hits         2374     2254     -120
- Misses        798      843      +45
+ Partials      166      157       -9

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Supporting Proximity placement group and Availability zone properties in Azure Site Recovery for Vmware to Azure and Hyper V to Azure scenarios

Swagger PR : https://github.com/Azure/azure-rest-api-specs/pull/9688

@m-nash can you please review this PR ?

@m-nash can you please review this PR ?


Pin pip-tools to pip>=20.0

Refs #926.
Progress

[X] remove pip<20.0 compatible code
[X] remove obsolete tests
[X] remove do_import() from pip_compat module
[X] refactor codebase to import objects directly from pip
[X] remove pip<20.0 envs from tox.ini
[X] remove pip<=20.0 jobs from travis
[X] remove pip<=20.0 jobs from appveyor
[X] pin pip>=20 in setup.py

Changelog-friendly one-liner: Pin pip-tools to pip>=20.0.
Contributor checklist

Provided the tests for the changes.
[X] Gave a clear one-line description in the PR (that the maintainers can add to CHANGELOG.md on release).
[X] Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

FTR, rebased onto master and ready to review.

The CI now passes in ~7 minutes 😍 It was 1h 30min before.

@jazzband/pip-tools Hey, y'all. It seems this PR is ready to merge, thus I'm going to do it tomorrow if nobody minds. This means we'll bump version to 5.0.0 before pip==20.1 is being released.
@vphilippon @auvipy thank you so much for reviewing this! Much appreciated 🙏

🥇 🎉


Image not working

Hello, can you please tell me how to use public folder image? I tried "/image.jpg" , "public/image.jpg", "/public/image.jpg" but nothing works.

A few different ways to go about it, depending on where you want to import it.
Link: https://stackoverflow.com/questions/11804332/insert-an-image-in-chrome-extension
For this repo: Files in the public folder will be in the root directory of the dist folders. Your first "/image.jpg" try should work, given you don't import it as an asset in your react files. I.e. just use it as the src in the image tag.


Fix order

Motivation and context

Resolve #4025
How has this been tested?

Manually
Checklist

[x] I submit my changes into the develop branch
[x] I have added a description of my changes into CHANGELOG file
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
[x] I have linked related issues (read github docs)
- [ ] I have increased versions of npm packages if it is necessary (cvat-canvas,
cvat-core, cvat-data and cvat-ui)

License

[x] I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
[x] I have updated the license header for each file (see an example below)

# Copyright (C) 2021 Intel Corporation
#
# SPDX-License-Identifier: MIT

Could you check these lines? If the results are in the reversed order, it looks like the annotation_ir is reversed. I'm not sure that the order in frame_info is a valid (or reliable) fix.