Comparison of Abel Transform Methods

Let's start filling the “Comparison of Abel Transform Methods” section with useful information. I have made the plots for “Speed benchmarks” (see #280) and wrote some “general advice” about libraries and using “daz” (see #246).
This is a PR from the same repository. Let's see how this works....
If I understood this collaborative workflow correctly, everybody will need to do git fetch upstream and then checkout the comparison (or upstream/comparison?) branch. Then everybody can make commits to it, but will need to do git pull (or git fetch followed by git merge) to synchronize the local copy with other people's changes before pushing new changes.
The speed benchmarks data and plotting scripts are currently placed in doc/benchmarks. If we also want to compare transform results, maybe it would be better to move this stuff to doc/benchmarks/speed or doc/benchmarks/timings and create another subdirectory in doc/benchmarks for the transform results. Or maybe place them in a separate directory doc/comparison or something like that.
The help that I need for this part from @stggh is to add his script that generates the timings and to update the data files with his new timings (for i7-9700).
And @DanHickstein also wanted:

I would discuss the quality of the transform results first and then move on to the speed benchmarks. Probably we should simply adapt our manuscript to fit on this page and keep a "living version" of the comparison paper here.

so please go ahead. :–)
(I think, if we discuss proposed changes here and then commit them one by one instead of concurrently, this should minimize possible edit conflicts.)

Thanks for getting started on this Mikhail!
Is there any advantage of using commits to directly work on the same branch rather than making PRs to this branch?
I will get started working on including some more of the information from the paper into the comparison document.
I'm going to set up doc/comparison as a place where all of the scripts and data for making the figures for the comparison article can live.

Is there any advantage of using commits to directly work on the same branch rather than making PRs to this branch?

Only less work and cleaner history (no useless intermediate merge commits). PRs are intended for merging outside contributions (from people who cannot commit directly) or for merging “features” into master, but routine collaboration within a repository is more naturally done directly in “feature branches”.

I'm going to set up doc/comparison as a place where all of the scripts and data for making the figures for the comparison article can live.

OK, then no directory reorganization is needed. @stggh's script will go into doc/benchmarks, with the updated data.

Hello @MikhailRyazanov! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file doc/benchmarks/pyabel-plot.py:

Line 13:18: E203 whitespace before ','
Line 14:18: E203 whitespace before ','
Line 15:18: E203 whitespace before ','
Line 16:18: E203 whitespace before ','
Line 17:18: E203 whitespace before ','
Line 19:18: E203 whitespace before ','
Line 20:18: E203 whitespace before ','
Line 22:18: E203 whitespace before ','
Line 23:18: E203 whitespace before ','
Line 24:18: E203 whitespace before ','
Line 27:26: E231 missing whitespace after ','
Line 27:28: E231 missing whitespace after ','
Line 27:42: E231 missing whitespace after ','
Line 34:5: E722 do not use bare 'except'
Line 40:5: E303 too many blank lines (2)
Line 47:1: W293 blank line contains whitespace
Line 48:1: W293 blank line contains whitespace
Line 56:80: E501 line too long (99 > 79 characters)
Line 56:92: W605 invalid escape sequence ';'
Line 57:80: E501 line too long (99 > 79 characters)
Line 57:92: W605 invalid escape sequence ';'
Line 58:80: E501 line too long (99 > 79 characters)
Line 58:92: W605 invalid escape sequence ';'
Line 59:80: E501 line too long (99 > 79 characters)
Line 59:92: W605 invalid escape sequence ';'
Line 62:47: W291 trailing whitespace
Line 73:80: E501 line too long (102 > 79 characters)
Line 73:98: E231 missing whitespace after ','
Line 78:80: E501 line too long (110 > 79 characters)
Line 79:9: E231 missing whitespace after ','
Line 90:1: W293 blank line contains whitespace
Line 92:50: E231 missing whitespace after ','
Line 93:80: E501 line too long (122 > 79 characters)
Line 95:1: W293 blank line contains whitespace
Line 96:1: E305 expected 2 blank lines after class or function definition, found 1
Line 98:1: W293 blank line contains whitespace
Line 100:1: W293 blank line contains whitespace

In the file doc/benchmarks/pyabel-timing.py:

Line 68:28: W291 trailing whitespace
Line 70:14: W291 trailing whitespace
Line 102:22: W291 trailing whitespace
Line 137:8: E127 continuation line over-indented for visual indent

Okay, the comparison branch makes sense, apart from renegade @DanHickstein making his own PR #291 ;-) .
I have pushed some i7_9700 timings to the comparison branch. Note, my code hack to include rbasex and rbasex(None) into pyabel-timing.py may not be the best.  I will fix the pep8 issues.
Sorry, for the delay, I wanted to check that the old i7_6700 timings were similar to what we had in the past. Also, my wife has me painting rooms in our house ...

apart from renegade @DanHickstein making his own PR

I won't conform to your laws! :)

I have pushed some i7_9700 timings to the comparison branch.

Awesome!

 I just repainted one of our bedrooms a few weeks ago. Painting always takes a lot more time that I think.

The last point of rBasex (and rBasex (None)) seems like an outlier. Are we running into RAM limitations or something? I'm tempted to not plot it...

May well be, I am re-running.

Raspberry pi 4B benchmark:

Well, I have merged @DanHickstein's #291 here...
A question to everybody:
Do we really need the timings files for several different machines if we do not use them at all? Maybe they should be removed, leaving only the actual ones, as I had initially? Or, alternatively, they all can be plotted as well. In tat case, my suggestion would be to move the phrase “These benchmarks were completed using a personal computer equipped with a 3.0 GHz Intel i7-9700 processor and 32 GB RAM running GNU/Linux.” from Fig. 6 caption to the first paragraph of “Speed benchmarks”, since this remark is really about all the plots there, and after that sentence give links to subpages with the same plots for other machines. (Then I can also provide benchmarks for the cheap AMD Ryzen under Windows (and Linux), to cover a broader range of environments.)

Do we really need the timings files for several different machines if we do not use them at all? Maybe they should be removed, leaving only the actual ones, as I had initially?

I don't feel strongly about this. We can remove the old ones and just include the most recent ones if you like. Unless there is something really interesting about showing benchmarks from multiple different processors, I would suggest that we just show benchmarks from one processor. I think that the point is to just show the general scaling and speed differences between the methods, not to get too far into the nitty gritty details about different processors.

Thanks, @DanHickstein. I also thought initially that one set of benchmarks would be enough and asked @stggh to just replace the old files with his new results, but he decided to keep the old ones too, and even added more benchmarks.
These Raspberry Pi results are actually interesting in that rBasex becomes the most efficient method for all practical image sizes. :–) I guess, this is only due to the worse matrix-algebra performance, and perhaps nobody will run PyAbel on Raspberry Pi, but nonetheless...
My previous tests on AMD Ryzen also showed that speed differences between the methods are not always the same as in Steve's results (this depends on the number of cores and the balance between floating-point, integer, flow-control and memory-access operations); there were also differences between Linux and Windows on the same machine. Another consideration of more practical importance is that showing only the results from this expensive high-performance hardware can make an impression that people really need it to use PyAbel, although in reality it has good performance even on much cheaper hardware.
So, removing the unused data is the easiest option, but on the other hand, showing that data also has some advantages.

By the way, @stggh,

The last point of rBasex (and rBasex (None)) seems like an outlier. Are we running into RAM limitations or something? I'm tempted to not plot it...

May well be, I am re-running.

Did you get anything from that? I wouldn't be surprised if it starts swapping (which you should be able to see from RAM/swap usage) at such image sizes, unreasonably large for VMI data, since I did not try to minimize memory usage at the expense of speed for typical image sizes. If this is the case, maybe we indeed should not plot these last points for rBasex and Hansen–Law.

No, the results are the same (sorry, I thought that I posted this, before the Rpi results):
# rbasex          iabel(ms)           (None)(ms)          basis(ms)
    5             0.11261             0.06864             0.10807
    9             0.11310             0.06925             0.14842
   17             0.11511             0.06955             0.22544
   33             0.12354             0.07422             0.38970
   65             0.14372             0.08550             0.93613
  129             0.21301             0.12419             1.74566
  257             0.59298             0.33920             3.23757
  513             1.83382             0.97952             7.80618
 1025             8.30427             4.52088            20.75333
 2049            36.38921            18.02486            51.86263
 4097           175.62474            90.95619           221.65900
 8193           696.05620           344.29510          1017.67079
16385          2694.15025          1254.63244          5917.78351
32769<PHONE_NUMBER>0         43537.52580         37277.30449

# hansenlaw       iabel(ms)
    5             0.11626
    9             0.13581
   17             0.17827
   33             0.26336
   65             0.47840
  129             1.02095
  257             2.60069
  513             8.23860
 1025            33.69837
 2049           115.44571
 4097           489.75560
 8193          1557.08233
16385          6265.46319
32769         31533.56482

I think the issue is not memory, but thermal control. dmesg gives:

sensors does not detect any fan, so I am not sure how to control the cooling outside of the BIOS. I will take a look later.

I also thought initially that one set of benchmarks would be enough and asked @stggh to just replace the old files with his new results, but he decided to keep the old ones too, and even added more benchmarks.

Yes, but I did not like the idea of stomping over the old results. It is also "risky" to rely on one PC for the benchmarks, without verification from another similar system.

speed differences between the methods are not always the same

I favour benchmarks from various systems, but am also ambivalent.

Thanks, @stggh! I guess, the CPU fan has automatic control, so it does speed up when the temperature rises, thus it's unlikely that the observed throttling can be overcome...
If Dan has no objections, I'll try to create an auxiliary page with all the benchmarks from various systems, and if we don't like it, it can be removed.
But I will need the rbasex and rbasex(None) benchmarks from the old i7-6700 system. If you have them, please post them here (or replace my i5 data, which I'm not going to use anyway, in doc/transform_methods/comparison/fig_benchmarks/benchmarks_2019/).

Oddly, the fan speed/noise does not appear to increase, so something is not right with the fan control. I haven't yet rebooted to check the BIOS settings.

benchmarks from the old i7-6700 system

benchmarks_i7-6700_Linux_5.3.0-40-generic

no high temperature warning in dmesg.
 rbasex          iabel(ms)           basis(ms)
    5             0.10452             0.10355
    9             0.10721             0.14982
   17             0.11027             0.24400
   33             0.12008             0.42883
   65             0.14302             0.92872
  129             0.21891             1.72293
  257             0.59280             3.44470
  513             2.07769             8.50229
 1025             8.82631            24.83911
 2049            41.96303            66.59847
 4097           222.23797           292.11110
 8193           884.57199          1472.04311
16385          3275.78789          8700.22730
32769<PHONE_NUMBER>3         58130.10464

# rbasex(None)    iabel(ms)
    5             0.05711
    9             0.05883
   17             0.06006
   33             0.06565
   65             0.07777
  129             0.12134
  257             0.32868
  513             1.09392
 1025             4.80125
 2049            20.06163
 4097           118.22915
 8193           487.11763
16385          1653.54199
32769         45508.45924

Dell Optiplex 7070 BIOS only has the option to turn off fan control, and run the fan flat out. Sadly, the timing is similar to before:
# rbasex          iabel(ms)           basis(ms)
 2049            36.15992            50.07761
 4097           177.16946           214.18520
 8193           702.31928           999.62982
16385          2674.88479          5587.29921
32769         84370.10413         35879.46093

In this case there was no dmesg CPU throttling. Hence, it looks as though the issue is, after all, insufficient RAM.

If Dan has no objections, I'll try to create an auxiliary page with all the benchmarks from various systems, and if we don't like it, it can be removed.

Yeah, that sounds like fun. I doubt that it will be of too much interest to the general userbase, but it sounds like fun for us geeks. So why not? But, we shouldn't use up too much of our energy on benchmarks. I think that we tend to get a little carried away :). (That said, some of our big speed improvements have resulted from our tendency to run lots of comparative benchmarks.)

Thanks, @stggh! The i7-6700 rbasex data now completes that dataset.

the timing is similar to before:

Well, this:
# rbasex          iabel(ms)           basis(ms)
...
32769         84370.10413         ...

is noticeably better than before:
32769<PHONE_NUMBER>4         ...

What is the last line for rbasex(None)?

Odd that ... systems behave differently.

My AMD under Win10 behaves even more oddly differently!

See how strangely hansenlaw does not become quadratic (while rbasex remains quadratic)! And how onion_peeling basis-set generation behaves for small image sizes...
The onion_peeling results are reproducible (I have repeated the test 3 times, separately and with methods in different order). I'll redo the hansenlaw to make sure, but it look very systematic. I also want to do this under Linux to see whether it behaves differently (NumPy under Windows definitely has some memory-access issues).
@DanHickstein,

I doubt that it will be of too much interest to the general userbase,

The unexpected AMD/Win results were interesting to myself at least! :–)

But, we shouldn't use up too much of our energy on benchmarks.

I already have the benchmarks page, so I'll just need to wait for my benchmarks to finish (while I'm busy with other things).

(py38) [Gibbo:.+.4.0-29-generic]$ cat rbasex.dat
# rbasex          iabel(ms)           basis(ms)
    5             0.09292             0.09155
    9             0.09403             0.13238
   17             0.09617             0.21074
   33             0.10435             0.36328
   65             0.12438             0.87740
  129             0.19134             1.62683
  257             0.56183             3.24362
  513             1.74804             7.61616
 1025             8.01388            19.99236
 2049            36.15992            50.07761
 4097           177.16946           214.18520
 8193           702.31928           999.62982
16385          2652.63505          5717.84295
32769         81248.37484         36181.63510

(py38) [Gibbo:.+.4.0-29-generic]$ cat rbasex\(None\).dat
# rbasex(None)    iabel(ms)
    5             0.05015
    9             0.05055
   17             0.05127
   33             0.05584
   65             0.06696
  129             0.10483
  257             0.31532
  513             0.94748
 1025             4.52871
 2049            18.24100
 4097            95.64268
 8193           358.04784
16385          1247.35140
32769         24476.19363

I also find this study interesting.

Thanks, @stggh! I've updated the i7-9700 timings with your new data.
I also did some corrections to your timing script (it tried to add (None) to linbasex and thus was crashing) and made it always use a predictable working directory instead of cpuinfo output (which depends on the kernel revision version...). The timing-data directories are thus also renamed to shorter and more reasonable.
Then I've added my benchmarks on AMD Ryzen ran under Windows and Linux (not complete yet) and made an auxiliary page with all the speed benchmarks plotted. I think, it looks interesting. To complete it, I need to wait for my Linux benchmarks to finish, but I also need some information about @stggh's systems:

How much RAM this Raspberry Pi has?
What are the RAM types/speeds for the i7 systems (both)? This can be queried by sudo dmidecode --type 17.
Which Linux distribution and libraries were used on the i7 systems and the RPi? NumPy and SciPy versions can be found in their __version__ strings, and the BLAS type is probably known (I suppose, MKL on i7, but what on RPi?), however, finding its version depends on how it is installed.

made an auxiliary page with all the speed benchmarks plotted

How may I see this page?  Could you post the figure?
Rpi 4B
[Rpi4Bp:/home/stg121]$ sudo inxi --system --cpu --memory
X11 connection rejected because of wrong authentication.
System:
  Host: Rpi4Bp Kernel: 4.19.115-v7l+ armv7l bits: 32 Desktop: LXDE
  Distro: Raspbian GNU/Linux 10 (buster)
Memory:
  RAM: total: 3.89 GiB used: 710.3 MiB (17.8%) gpu: 256.0 MiB
  RAM Report: unknown-error: dmidecode was unable to generate data
Argument "Raspberry Pi 4 Model B Rev 1.1" isn't numeric in sprintf at /usr/bin/inxi line 6969.
CPU:
  Topology: Quad Core model: ARMv7 v7l variant: cortex-a72 bits: 32
  type: MCP
  Speed: 1500 MHz min/max: 600/1500 MHz Core speeds (MHz): 1: 1500 2: 1500
  3: 1500 4: 1500

[Rpi4Bp:/home/stg121]$ ipython3
Python 3.7.3 (default, Dec 20 2019, 18:57:59)

In [1]: import numpy, scipy

In [2]: numpy.__version__
Out[2]: '1.16.2'

In [3]: scipy.__version__
Out[3]: '1.1.0'

In [4]: numpy.show_config()
blas_mkl_info:
  NOT AVAILABLE
blis_info:
  NOT AVAILABLE
openblas_info:
  NOT AVAILABLE
atlas_3_10_blas_threads_info:
  NOT AVAILABLE
atlas_3_10_blas_info:
  NOT AVAILABLE
atlas_blas_threads_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
accelerate_info:
  NOT AVAILABLE
blas_info:
    libraries = ['blas', 'blas']
    library_dirs = ['/usr/lib/arm-linux-gnueabihf']
    include_dirs = ['/usr/local/include', '/usr/include']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
blas_opt_info:
    define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
    libraries = ['blas', 'blas']
    library_dirs = ['/usr/lib/arm-linux-gnueabihf']
    include_dirs = ['/usr/local/include', '/usr/include']
    language = c
lapack_mkl_info:
  NOT AVAILABLE
openblas_lapack_info:
  NOT AVAILABLE
openblas_clapack_info:
  NOT AVAILABLE
atlas_3_10_threads_info:
  NOT AVAILABLE
atlas_3_10_info:
  NOT AVAILABLE
atlas_threads_info:
  NOT AVAILABLE
atlas_info:
  NOT AVAILABLE
lapack_info:
    libraries = ['lapack', 'lapack']
    library_dirs = ['/usr/lib/arm-linux-gnueabihf']
    language = f77
lapack_opt_info:
    define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
    libraries = ['lapack', 'lapack', 'blas', 'blas']
    library_dirs = ['/usr/lib/arm-linux-gnueabihf']
    language = c
    include_dirs = ['/usr/local/include', '/usr/include']

i7-9700
(py38) [Gibbo:/home/stg121]$ sudo inxi --system --cpu --memory
System:
  Host: Gibbo Kernel: 5.4.0-29-generic x86_64 bits: 64 Desktop: Gnome 3.36.1
  Distro: Ubuntu 20.04 LTS (Focal Fossa)
Memory:
  RAM: total: 31.15 GiB used: 3.56 GiB (11.4%)
  Array-1: capacity: 64 GiB slots: 4 EC: None
  Device-1: DIMM1 size: 16 GiB speed: 2666 MT/s
  Device-2: DIMM2 size: 16 GiB speed: 2666 MT/s
  Device-3: DIMM3 size: No Module Installed
  Device-4: DIMM4 size: No Module Installed
CPU:
  Topology: 8-Core model: Intel Core i7-9700 bits: 64 type: MCP
  L2 cache: 12.0 MiB
  Speed: 2400 MHz min/max: 800/4700 MHz Core speeds (MHz): 1: 2400 2: 2400
  3: 2400 4: 2400 5: 2400 6: 2400 7: 2400 8: 2400

py38) [Gibbo:/home/stg121]$ ipython
Python 3.8.2 (default, Mar 26 2020, 15:53:00)

In [1]: import numpy, scipy

In [2]: numpy.__version__
Out[2]: '1.18.1'

In [3]: scipy.__version__
Out[3]: '1.4.1'

In [4]: numpy.show_config()
blas_mkl_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/home/stg121/Software/Python/anaconda3/envs/py38/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/home/stg121/Software/Python/anaconda3/envs/py38/include']
blas_opt_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/home/stg121/Software/Python/anaconda3/envs/py38/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/home/stg121/Software/Python/anaconda3/envs/py38/include']
lapack_mkl_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/home/stg121/Software/Python/anaconda3/envs/py38/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/home/stg121/Software/Python/anaconda3/envs/py38/include']
lapack_opt_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/home/stg121/Software/Python/anaconda3/envs/py38/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/home/stg121/Software/Python/anaconda3/envs/py38/include']

i7-6700
[etalon:/home/stg121]$ sudo inxi --system --cpu --memory
X11 connection rejected because of wrong authentication.
System:
  Host: etalon Kernel: 5.3.0-40-generic x86_64 bits: 64 Console: tty 6
  Distro: Ubuntu 19.10 (Eoan Ermine)
Memory:
  RAM: total: 31.23 GiB used: 5.46 GiB (17.5%)
  Array-1: capacity: 64 GiB slots: 4 EC: None
  Device-1: DIMM1 size: 8 GiB speed: 2133 MT/s
  Device-2: DIMM2 size: 8 GiB speed: 2133 MT/s
  Device-3: DIMM3 size: 8 GiB speed: 2133 MT/s
  Device-4: DIMM4 size: 8 GiB speed: 2133 MT/s
CPU:
  Topology: Quad Core model: Intel Core i7-6700 bits: 64 type: MT MCP
  L2 cache: 8192 KiB
  Speed: 800 MHz min/max: 800/4000 MHz Core speeds (MHz): 1: 800 2: 801
  3: 800 4: 800 5: 800 6: 800 7: 800 8: 800

[etalon:/home/stg121]$ ipython3
Python 3.8.2 | packaged by conda-forge | (default, Apr 16 2020, 18:04:51)

In [1]: import numpy, scipy

In [2]: numpy.__version__
Out[2]: '1.18.1'

In [3]: scipy.__version__
Out[3]: '1.4.1'

In [4]: numpy.show_config()
blas_mkl_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/home/stg121/Software/Python/anaconda3/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/home/stg121/Software/Python/anaconda3/include']
blas_opt_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/home/stg121/Software/Python/anaconda3/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/home/stg121/Software/Python/anaconda3/include']
lapack_mkl_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/home/stg121/Software/Python/anaconda3/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/home/stg121/Software/Python/anaconda3/include']
lapack_opt_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/home/stg121/Software/Python/anaconda3/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/home/stg121/Software/Python/anaconda3/include']

How may I see this page? Could you post the figure?

There are many figures...
The page itself is at https://pyabel--283.org.readthedocs.build/en/283/transform_methods/comparison/fig_benchmarks/benchmarks.html and is linked form https://pyabel--283.org.readthedocs.build/en/283/transform_methods/comparison.html#speed-benchmarks.

Rpi 4B

OK, I see: 4 GB RAM, Raspbian GNU/Linux 10, NumPy 1.16.2, SciPy 1.1.0.
Which BLAS implementation does it use? If you list
ls /usr/lib/arm-linux-gnueabihf/*blas*
and/or
apt list --installed "*blas*"
this will probably clarify.

i7-9700
i7-6700

OK, both apparently use DDR4 RAM; OS and NumPy/SciPy are clear, but which MKL versions? (Here I don't know hoe to check.)

Yes, difficult to find version information.

Rpi  BLAS

libblas3/stable,now 3.8.0-2 armhf [installed,automatic]

which MKL versions

9700
export MKL_VERBOSE=1

In [1]: import numpy as np
mkl-service + Intel(R) MKL: THREADING LAYER: (null)
mkl-service + Intel(R) MKL: setting Intel(R) MKL to use INTEL OpenMP runtime
mkl-service + Intel(R) MKL: preloading libiomp5.so runtime
MKL_VERBOSE Intel(R) MKL 2020.0 Product build 20191122 for Intel(R) 64 architecture Intel(R) Advanced Vector Extensions 2 (Intel(R) AVX2) enabled processors, Lnx 3.00GHz lp64 intel_thread
MKL_VERBOSE SDOT(2,0x55b58db96030,1,0x55b58db96030,1) 22.66us CNR:OFF Dyn:1 FastMM:1 TID:0  NThr:8

6700
In [1]: import numpy as np
mkl-service + Intel(R) MKL: THREADING LAYER: (null)
mkl-service + Intel(R) MKL: setting Intel(R) MKL to use INTEL OpenMP runtime
mkl-service + Intel(R) MKL: preloading libiomp5.so runtime
MKL_VERBOSE Intel(R) MKL 2019.0 Update 5 Product build 20190808 for Intel(R) 64 architecture Intel(R) Advanced Vector Extensions 2 (Intel(R) AVX2) enabled processors, Lnx 3.40GHz lp64 intel_thread
MKL_VERBOSE SDOT(2,0x55cef4ccd020,1,0x55cef4ccd020,1) 17.63us CNR:OFF Dyn:1 FastMM:1 TID:0  NThr:4`

OK, so “MKL 2019 Update 5” and “MKL 2020 Initial Release” is apparently how they are officially called...
Regarding RPi, libblas3 is the reference implementation, installed by default. It is not parallelized and does not use advanced instructions, so at least on amd64 it was very slow (see https://github.com/PyAbel/PyAbel/issues/230#issuecomment-433796009 and further discussion there with @rth's suggestions). If you really want, try libopenblas and/or libatlas3 instead — they are available for armhf, although I don't know about their performance. According to Debian wiki, they all can be installed together and switched using update-alternatives, but I was just uninstalling the default libblas3 and liblapack3 libraries to make sure.

If at all possible, it's really easy to switch BLAS backend when installing packages from the conda-forge conda channel,
https://conda-forge.org/docs/maintainer/knowledge_base.html#switching-blas-implementation
They have numpy, scipy builds for aarch64 and ppc64le, but not armhf I think.

I'm done with my benchmarks on AMD Ryzen 3, both Linux and Windows: https://pyabel--283.org.readthedocs.build/en/283/transform_methods/comparison/fig_benchmarks/benchmarks.html#amd-ryzen-3-2200g-linux
Interesting findings from these AMD results:

PyAbel under Linux is generally faster than under Windows. I think, mostly due to Python itself, but also due to poorly compiled NumPy for Windows (see https://github.com/numpy/numpy/issues/15513, for example).
hansenlaw, three_point and two_point failed at n = 32769 with MemoryError under Linux, but were able to complete (not very badly) under Windows. However, onion_peeling worked fine for n = 32769 in both cases.
hansenlaw under Windows behaves very strangely and poorly (I did rerun it, and these are the best results), but under Linux runs on the cheap Ryzen 3 2200G nearly as fast as on i7-9600 and definitely faster than on i7-6700.

@DanHickstein and @stggh, should we merge this now? If Steve wants to play with Raspbery Pi, but not right now, its benchmarks can be updated later.

I agree, good to merge. Interesting that an old AMD can keep up with modern systems.
On the Rpi I installed openblas, but it made little difference to the PyAbel benchmarks:

The Rpi ARM cpu is quite old, my phone (Xiaomi A2), running Andronix is faster (default blas):
Mobile/cell phone screen shot:

X11 screen via VNC:

Interesting that an old AMD can keep up with modern systems.

It's only about 2 years “old” (released in February 2018)... More surprising is the price — it was 99 USD (+125 for the motherboard) and now is even cheaper. At that time it had the best performance/price ratio, but I did not expect it to have this absolute single-core performance, especially in FP calculations (where AMD was always lagging).

my phone (Xiaomi A2), running Andronix is faster (default blas)

LOL! :–) “PyAbel: real-time Abel transforms in your pocket!”
Have you tried to use its camera for data acquisition? ;–)

@DanHickstein and @stggh, should we merge this now?

Yeah, I think that this is looking good and can be merged. We should just make sure that we haven't included any extra files. I think one of the plotting files above is now redundant.

Okay, now I'm fine with merging this.

Great! Many thanks to @DanHickstein for adapting the text and @stggh for providing the timings!


Android Unable to resolve dependency for ':app@debug/compileClasspath': Could not resolve project :react-native-japanese-tokenizer.

Can you provide steps to reproduce and your RN version?


WebDAV support?

Might be out of scope

48c88da9e3be96bafb3ceadf535701ae89399276 debugging was awkward, but the actual impl wasn't too bad

4936c677c985389e3ab274086c6d05a0da865c92
85308c808aec2891f8326c8b8d945a0c390822e7


Saved package config file seems to be corrupt

Following steps:
cabal clean
rm *.imports
cabal configure -O0 --disable-library-profiling
cabal build --ghc-option=-ddump-minimal-imports
packunused

yields the following error:
packunused: user error (Saved package config file seems to be corrupt. Try re-running the 'configure' command.)

Can be reproduced on https://github.com/snowdriftcoop/snowdrift
Using:
packunused version <IP_ADDRESS>
cabal-install version <IP_ADDRESS>
using version <IP_ADDRESS> of the Cabal library
The Glorious Glasgow Haskell Compilation System, version 7.8.4

Same problem here using cabal-install version <IP_ADDRESS> (compiled/installed from HEAD) and GHC 7.8.3


Bulk Update Multiple Tables and Columns via Single CSV File

We are currently using OMD v1.4.2 and testing a bulk update on Data Assets (description, Owner, Tier, Domain, etc.).
Since the OMD API does not support bulk importing of multiple tables with a single CSV file, we would like to know if it is possible to add this feature to the OMD API. Appreciate your feedback on this. Thanks!

you can do multiple tables against a database schema

Thanks for your input, @harshach. Do you have a step-by-step guide on how to do it? I'm new to working with APIs?
Is this documentation can help? https://docs.open-metadata.org/swagger.html#operation/importDatabaseSchema

@noelaum this is question in support channel and not for github issues. Please open a github issue once confirmed its a bug or a feature request

@harshach ,I already raised this concern, and according to the support team, the OMD API does not support bulk importing of multiple tables with a single CSV file. Here's the link to the Slack conversation: https://openmetadata.slack.com/archives/C02B6955S4S/p1718864234398369?thread_ts=1718680131.869139&cid=C02B6955S4S
I suggested adding this feature to the OMD API, and they advised me to file a feature request. Thanks!

you can do multiple tables against a database schema

Hi @harshach , I would like to confirm if it is feasible to perform updates on multiple tables within the database schema?Thanks!

Same opinion , I want to update the metadata for each table column using a CSV file without using the database schema because I don't have editing rights


Chore/upgrade mux [LESQ-822]

Description
Music year: 2011

Updates mux-node and mux-player-react to latest versions
update syntax for fetching signed urls
add required secrets to google secret manager

How to test

Go to lesson pages, eg https://deploy-preview-2417--oak-web-application.netlify.thenational.academy/teachers/programmes/chemistry-secondary-ks4-foundation-edexcel/units/atomic-structure-and-the-periodic-table/lessons/atomic-structure-negligible-electron-mass#video
The video should play as usual

:tada: This PR is included in version 1.433.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Fix HTML entities escaping for resources

Fix an issue preventing resources containing HTML entities from loading
when annotating a page.

When annotating a page like:
https://www.facebook.com/bbcnews/
Some resources (images/css/..) are not handled during the annotation because their URL includes html entities.
https://scontent.xx.fbcdn.net/v/t1.0-1/p160x160/1926656_10151955586072217_265500303_n.png?oh=65b239d3092a2de944f218fe7a8e991b**&**oe=58069918
is saved (in the Tenplate file) as:
https://scontent.xx.fbcdn.net/v/t1.0-1/p160x160/1926656_10151955586072217_265500303_n.png?oh=65b239d3092a2de944f218fe7a8e991b**&**oe=58069918
HTML entities within URLs should be interpeted automatically by the browser, however portia is not referencing the resources directly but through a proxy.
After encoding, the URL given to the browser will look like:
/proxy?url=https%3A%2F%2Fscontent.xx.fbcdn.net%2Fv%2Ft1.0-1%2Fp160x160%2F1926656_10151955586072217_265500303_n.png%3Foh%3D65b239d3092a2de944f218fe7a8e991b**%26amp%3B**oe%3D58069918
When it should be:
/proxy?url=https%3A%2F%2Fscontent.xx.fbcdn.net%2Fv%2Ft1.0-1%2Fp160x160%2F1926656_10151955586072217_265500303_n.png%3Foh%3D65b239d3092a2de944f218fe7a8e991b**%26**oe%3D58069918
The fix I applied was to change the wrap_url method in order to decode HTML entities before saving URLS.

Thanks for the fix


Add onProgress to browserHTTPRequest & getLoadHandlers

FEATURE
Feature requested in tensorflow/tfjs#336. This PR adds onProgress to browserHTTPRequest & getLoadHandlers, which supports onProgress feature in loadModel ( relative PR tensorflow/tfjs-layers#413 ), loadFrozenModel & loadTFHubModule ( relative PR tensorflow/tfjs-converter#261 ).
The name of onprogress is inspired by XMLHttpRequest.onprogress.  Through this feature, developer can add callback function, the progress callback function will be fired periodically before the load is completed. The onProgress callback can help tfjs developer build more user-friendly application.
To show the usage of onProgress, I built a local version of tf.min.js ( including change in tensorflow/tfjs-core#1485, tensorflow/tfjs-layers#413,  tensorflow/tfjs-converter#261 ), and wrote a simple demo:
Online demo link
Demo repository link
Towards https://github.com/tensorflow/tfjs/issues/336

This change is

@syt123450  cannot find the github link?


Error messages not being displayed

No error messages are being displayed when start competition is clicked when room doesn't meet requirements

As you can see in the image shown above.
There are two users in the room and both of them are in the same team though there must be minimum 1 user in the other team as well. So some error should be shown

@vuld0 please provide the room specification values you gave when you created the room, specifically "Maximum Members in a Team".

@Rec0iL99, the value of "Maximum Members in a team" was given as "1".


Having trouble in running Two-Tower built-in algorithm

Hello, so I was trying to run this code
I was trying to understand the Input Schema, this seems like an invalid JSON schema format.
I tried to run the code with this schema, and I got an error.
I am unsure as to what is the valid input schema format here.
Here is the error downloaded from the logs.
 {
    "textPayload": "The replica workerpool0-0 exited with a non-zero status of 1. To find out more about why your job exited please check the logs: https://console.cloud.google.com/logs/viewer?project=1004927726779&resource=ml_job%2Fjob_id%2F1464827664639459328&advancedFilter=resource.type%3D%22ml_job%22%0Aresource.labels.job_id%3D%221464827664639459328%22",
    "insertId": "1calpzlc593",
    "resource": {
      "type": "ml_job",
      "labels": {
        "task_name": "service",
        "project_id": "trell-staging",
        "job_id": "1464827664639459328"
      }
    },
    "timestamp": "2022-01-18T11:16:31.582429920Z",
    "severity": "ERROR",
    "labels": {
      "ml.googleapis.com/endpoint": ""
    },
    "logName": "projects/trell-staging/logs/ml.googleapis.com%2F1464827664639459328",
    "receiveTimestamp": "2022-01-18T11:16:32.678545724Z"
  },
  {
    "insertId": "t523mkflys0mw",
    "jsonPayload": {
      "message": "json.decoder.JSONDecodeError: Extra data: line 1 column 8 (char 7)\n",
      "levelname": "ERROR"
    },
    "resource": {
      "type": "ml_job",
      "labels": {
        "job_id": "1464827664639459328",
        "task_name": "workerpool0-0",
        "project_id": "trell-staging"
      }
    },
    "timestamp": "2022-01-18T11:16:10.582601640Z",
    "severity": "ERROR",
    "labels": {
      "ml.googleapis.com/trial_type": "",
      "compute.googleapis.com/resource_name": "cmle-training-5612951348054016528",
      "compute.googleapis.com/zone": "us-central1-c",
      "ml.googleapis.com/job_id/log_area": "root",
      "ml.googleapis.com/trial_id": "",
      "compute.googleapis.com/resource_id": "439373723071094359"
    },
    "logName": "projects/trell-staging/logs/workerpool0-0",
    "receiveTimestamp": "2022-01-18T11:16:42.042793190Z"
  },
  {
    "insertId": "t523mkflys0mv",
    "jsonPayload": {
      "levelname": "ERROR",
      "message": "    raise JSONDecodeError(\"Extra data\", s, end)\n"
    },
    "resource": {
      "type": "ml_job",
      "labels": {
        "task_name": "workerpool0-0",
        "project_id": "trell-staging",
        "job_id": "1464827664639459328"
      }
    },
    "timestamp": "2022-01-18T11:16:10.582595651Z",
    "severity": "ERROR",
    "labels": {
      "compute.googleapis.com/zone": "us-central1-c",
      "ml.googleapis.com/job_id/log_area": "root",
      "compute.googleapis.com/resource_name": "cmle-training-5612951348054016528",
      "ml.googleapis.com/trial_type": "",
      "compute.googleapis.com/resource_id": "439373723071094359",
      "ml.googleapis.com/trial_id": ""
    },
    "logName": "projects/trell-staging/logs/workerpool0-0",
    "receiveTimestamp": "2022-01-18T11:16:42.042793190Z"
  },
  {
    "insertId": "t523mkflys0mu",
    "jsonPayload": {
      "message": "  File \"/usr/lib/python3.8/json/decoder.py\", line 340, in decode\n",
      "levelname": "ERROR"
    },
    "resource": {
      "type": "ml_job",
      "labels": {
        "job_id": "1464827664639459328",
        "project_id": "trell-staging",
        "task_name": "workerpool0-0"
      }
    },
    "timestamp": "2022-01-18T11:16:10.582584385Z",
    "severity": "ERROR",
    "labels": {
      "compute.googleapis.com/resource_name": "cmle-training-5612951348054016528",
      "ml.googleapis.com/trial_id": "",
      "compute.googleapis.com/resource_id": "439373723071094359",
      "ml.googleapis.com/trial_type": "",
      "ml.googleapis.com/job_id/log_area": "root",
      "compute.googleapis.com/zone": "us-central1-c"
    },
    "logName": "projects/trell-staging/logs/workerpool0-0",
    "receiveTimestamp": "2022-01-18T11:16:42.042793190Z"
  },
  {
    "insertId": "t523mkflys0mt",
    "jsonPayload": {
      "levelname": "ERROR",
      "message": "    return _default_decoder.decode(s)\n"
    },
    "resource": {
      "type": "ml_job",
      "labels": {
        "job_id": "1464827664639459328",
        "project_id": "trell-staging",
        "task_name": "workerpool0-0"
      }
    },
    "timestamp": "2022-01-18T11:16:10.582578628Z",
    "severity": "ERROR",
    "labels": {
      "compute.googleapis.com/zone": "us-central1-c",
      "ml.googleapis.com/trial_type": "",
      "ml.googleapis.com/job_id/log_area": "root",
      "compute.googleapis.com/resource_id": "439373723071094359",
      "ml.googleapis.com/trial_id": "",
      "compute.googleapis.com/resource_name": "cmle-training-5612951348054016528"
    },
    "logName": "projects/trell-staging/logs/workerpool0-0",
    "receiveTimestamp": "2022-01-18T11:16:42.042793190Z"
  },
  {
    "insertId": "t523mkflys0ms",
    "jsonPayload": {
      "levelname": "ERROR",
      "message": "  File \"/usr/lib/python3.8/json/__init__.py\", line 357, in loads\n"
    },
    "resource": {
      "type": "ml_job",
      "labels": {
        "project_id": "trell-staging",
        "job_id": "1464827664639459328",
        "task_name": "workerpool0-0"
      }
    },
    "timestamp": "2022-01-18T11:16:10.582572352Z",
    "severity": "ERROR",
    "labels": {
      "compute.googleapis.com/resource_id": "439373723071094359",
      "compute.googleapis.com/zone": "us-central1-c",
      "ml.googleapis.com/job_id/log_area": "root",
      "ml.googleapis.com/trial_type": "",
      "ml.googleapis.com/trial_id": "",
      "compute.googleapis.com/resource_name": "cmle-training-5612951348054016528"
    },
    "logName": "projects/trell-staging/logs/workerpool0-0",
    "receiveTimestamp": "2022-01-18T11:16:42.042793190Z"
  },
  {
    "insertId": "t523mkflys0mr",
    "jsonPayload": {
      "message": "    return loads(fp.read(),\n",
      "levelname": "ERROR"
    },
    "resource": {
      "type": "ml_job",
      "labels": {
        "task_name": "workerpool0-0",
        "job_id": "1464827664639459328",
        "project_id": "trell-staging"
      }
    },
    "timestamp": "2022-01-18T11:16:10.582566745Z",
    "severity": "ERROR",
    "labels": {
      "compute.googleapis.com/zone": "us-central1-c",
      "ml.googleapis.com/job_id/log_area": "root",
      "compute.googleapis.com/resource_id": "439373723071094359",
      "compute.googleapis.com/resource_name": "cmle-training-5612951348054016528",
      "ml.googleapis.com/trial_type": "",
      "ml.googleapis.com/trial_id": ""
    },
    "logName": "projects/trell-staging/logs/workerpool0-0",
    "receiveTimestamp": "2022-01-18T11:16:42.042793190Z"
  },
  {
    "insertId": "t523mkflys0mq",
    "jsonPayload": {
      "levelname": "ERROR",
      "message": "  File \"/usr/lib/python3.8/json/__init__.py\", line 293, in load\n"
    },
    "resource": {
      "type": "ml_job",
      "labels": {
        "task_name": "workerpool0-0",
        "project_id": "trell-staging",
        "job_id": "1464827664639459328"
      }
    },
    "timestamp": "2022-01-18T11:16:10.582560508Z",
    "severity": "ERROR",
    "labels": {
      "compute.googleapis.com/resource_id": "439373723071094359",
      "compute.googleapis.com/resource_name": "cmle-training-5612951348054016528",
      "ml.googleapis.com/trial_id": "",
      "compute.googleapis.com/zone": "us-central1-c",
      "ml.googleapis.com/job_id/log_area": "root",
      "ml.googleapis.com/trial_type": ""
    },
    "logName": "projects/trell-staging/logs/workerpool0-0",
    "receiveTimestamp": "2022-01-18T11:16:42.042793190Z"
  },
  {
    "insertId": "t523mkflys0mp",
    "jsonPayload": {
      "message": "    input_schema = json.load(tf.io.gfile.GFile(args.input_schema_path))\n",
      "levelname": "ERROR"
    },
    "resource": {
      "type": "ml_job",
      "labels": {
        "job_id": "1464827664639459328",
        "task_name": "workerpool0-0",
        "project_id": "trell-staging"
      }
    },
    "timestamp": "2022-01-18T11:16:10.582554286Z",
    "severity": "ERROR",
    "labels": {
      "ml.googleapis.com/job_id/log_area": "root",
      "ml.googleapis.com/trial_id": "",
      "ml.googleapis.com/trial_type": "",
      "compute.googleapis.com/resource_id": "439373723071094359",
      "compute.googleapis.com/resource_name": "cmle-training-5612951348054016528",
      "compute.googleapis.com/zone": "us-central1-c"
    },
    "logName": "projects/trell-staging/logs/workerpool0-0",
    "receiveTimestamp": "2022-01-18T11:16:42.042793190Z"
  },
  {
    "insertId": "t523mkflys0mo",
    "jsonPayload": {
      "message": "  File \"/root/two_tower/task.py\", line 89, in main\n",
      "levelname": "ERROR"
    },
    "resource": {
      "type": "ml_job",
      "labels": {
        "job_id": "1464827664639459328",
        "task_name": "workerpool0-0",
        "project_id": "trell-staging"
      }
    },
    "timestamp": "2022-01-18T11:16:10.582548477Z",
    "severity": "ERROR",
    "labels": {
      "compute.googleapis.com/resource_id": "439373723071094359",
      "ml.googleapis.com/trial_id": "",
      "compute.googleapis.com/zone": "us-central1-c",
      "ml.googleapis.com/job_id/log_area": "root",
      "compute.googleapis.com/resource_name": "cmle-training-5612951348054016528",
      "ml.googleapis.com/trial_type": ""
    },
    "logName": "projects/trell-staging/logs/workerpool0-0",
    "receiveTimestamp": "2022-01-18T11:16:42.042793190Z"
  },
  {
    "insertId": "t523mkflys0mn",
    "jsonPayload": {
      "levelname": "ERROR",
      "message": "    main()\n"
    }

This is the wrong place for this question. Seems like a gcloud command.

Looks like it's a json schema issue. There are some issue in the doc that misleading the you. I will suggest you to use https://storage.googleapis.com/cloud-samples-data/vertex-ai/matching-engine/two-tower/movielens_100k/input_schema.json as an example

If this is a doc issue? Should it be reassigned to a TW?

Let's see if it solves @shim1998 's problem or not.

Hello, thank you for clarifying this to me!
I have changed the input schema to the sample you provided, and I have tried to include a Categorical feature.
My input schema looks like this:
{
   "query":{
      "movie_genre":{
         "feature_type":"Categorical",
         "config":{
            "vocab": [ "action", "comedy", "drama"]
         }
      }
   },
   "candidate":{
      "ratings":{
         "feature_type":"Numeric"
      }
   }
}

And my training data's sample row looks like this:
{
    "query":
    {
        "movie_genre": ["action"]
    },
    "candidate":
    {
        "ratings": [7.3, 6.0, 8.1]
    }
}

I am encountering this error statement.
"ValueError: Shapes (103, 64) and (91955, 64) are incompatible

I have used the list of various sizes and I am still getting this issue.
In the documentation it is written as:
Categorical : A list of strings, representing a categorical feature. The strings must be a subset of a vocabulary list specified in the input schema.

This is what I have followed in my dataset.
Can I get some clarification on this?

Hi @shim1998,
I guess it's due to checkpoint. You can try to use a new job dir to see if it can resolve your issue.


Sttp fails with EOFException when receiving empty gzip-encoded response

This is a test that is designed to reproduce a bug in the Sttp HTTP client library. Specifically, the bug occurs when a simple API call is made that responds with a Content-Encoding of gzip, and an empty response in combination with Created 201, and Accepted 202 codes.
When this occurs, the Sttp library fails with a java.io.EOFException (GZIPInputStream.java:268).
The exception is not observed in combination with NoContent 204 code.
To reproduce the error and see the bug in action, run the following command:
$ scala-cli test https://gist.github.com/baldram/cfcb9203baaf968107f1d3da44ae1150

The test demonstrates the issue using various backends. Both the client3 (<= 3.8.14) and the new client4 (4.0.0-M1) are affected. The problem is reproducible in JVM 11+ and also in JVM 8 when using the simple TryHttpURLConnectionBackend.
The exception occurs when trying to read the response from the server, and can be triggered by any backend.
The exception is not observed when using the ZIO 1 backend.
When commenting the line .withHeader("Content-Encoding", "gzip") in WireMock stub, all tests are green.
With the gzip header, the error responses are as follows:
For simple client:
Caused by: java.io.EOFException
	at java.base/java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:268)
	at java.base/java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:258)
	at java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:164)
	at java.base/java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:79)
	at java.base/java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:91)
	at sttp.client3.HttpClientSyncBackend.standardEncoding$$anonfun$1(HttpClientSyncBackend.scala:65)

For Cats:
Caused by: java.io.EOFException
	at java.base/java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:268)
	at java.base/java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:258)
	at java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:164)
	at java.base/java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:79)
	at java.base/java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:91)
	at sttp.client3.httpclient.cats.HttpClientCatsBackend.standardEncoding$$anonfun$1(HttpClientCatsBackend.scala:64)

The test snipped is available here: https://gist.github.com/baldram/cfcb9203baaf968107f1d3da44ae1150.
Are there any potential solutions to mitigate the problem? For example, I tried using quickRequest.post(apiUrl).response(IgnoreResponse).send(backend) but was unsuccessful.
If I receive any suggestions, I may attempt to work on a merge request to fix the issue, although my first attempt was unsuccessful.
Now, I consider using PushbackInputStream to read the first byte and determine if the stream is empty. It could be an effective solution if this approach works well with the HttpResponseInputStream currently in use. Please let me know if you find this suggestion suitable or if you have any alternative ideas in mind.
Next, the HttpClientBackend features a readResponse method, which we could potentially extend. This approach resembles existing checks, like the one for the HEAD method, that actively prevent decoding the response body.
PS: Similar is not an issue in case of JAX-RS/Jersey client.

Maybe we should check if the Content-Length header is not 0 before attempting to decode the body here: https://github.com/softwaremill/sttp/blob/e7ed7720a3ae4389c4b755b476e94bafb26ba694/core/src/main/scalajvm/sttp/client4/httpclient/HttpClientBackend.scala#L91 ? (and similarly in other backends)

But maybe a simpler solution would be replacing GZipInputStream with a custom implementation which correctly handles empty bodies, and delegates to the java one otherwise

That's some pathological response (without content-length) ;)

Yes, I agree :)

The problem here would be how you put the byte back.

As mentioned at the end of the ticket description, I am considering using the PushbackInputStream.

a custom implementation which correctly handles empty bodies

If the JVM standard is insufficient, that would be the next step.

Yes, I think we can try using such a thin-wrapper. The problem occurs deeper within, but if we could do only the necessary minimum and successfully delegate further processing to Java appropriately, it might work. I would try.


Creating of the product vision draft

From the guidelines:
Who is going to buy the product? Who is the target customer?
Which customer needs will the product address?
Which product attributes are crucial to satisfy the selected needs, and therefore to the success of the product?
How does the product compare against existing products, both from competitors and the same company? What are the product’s unique selling points?
What is the target timeframe and budget to develop and launch the product?
https://docs.google.com/document/d/19J8In7yXjVMQPCNsysOrMUZcK1h6n60V37eOJP-xMWE/pub

@mvanmeerten is editor / has the final responsibility
The way I view this:
This week

@mvanmeerten and @IwanHoogenboom create a draft
everyone checks to see whether they agree with that description of our vision
@mvanmeerten and @IwanHoogenboom finish the document
we hand in the draft

next week

we get feedback
someone (to be decided for next sprint) finishes the document


Images ?

Hi :)
I wanted to use your repository but it seems that there are no way to retrieve the images your data comes with. Could you pinpoint to where we could download them or could you add them to your repository ?
Thanks in advance :)

Hello,
The README file contains links to the websites from where the images can be downloaded.
Regards, Jean-Pierre.


Add an example using Xata

Add an example using Xata as database.
How to test
$ pnpm --filter example-with-xata run xata:init
$ pnpm --filter example-with-xata run dev

This is probably a better candidate for Xata's repository

Yeah no problem, this was more for discovery, since you had an example with prisma, but no hard feeling 😉


Document how to shrink Docker/WSL size on Windows

Purpose

Because the vdisks automatically grow, but do not automatically shrink.
Changes

Outline problem
How to shrink vdisk

Resources

https://github.com/microsoft/WSL/issues/4699#issuecomment-627133168

My comments on WSL2 experience with this issue may be informative. They're focused around Docker which has some additional gripes with storage management.

https://github.com/microsoft/WSL/issues/4699#issuecomment-1732526566 (sparse troubleshooting + Docker Desktop GUI purge method with screenshots)
https://github.com/microsoft/WSL/issues/4699#issuecomment-1736417123 (summary comment on the previous one regarding sparse size on disk)
https://github.com/microsoft/WSL/issues/4699#issuecomment-1738771703 (insights on resource usage + manual equivalent of wslcompact)
https://github.com/microsoft/WSL/issues/4699#issuecomment-1740681411 (advice + observations)

The third comment link responds to a user who used diskpart advice you linked which they said corrupted their disk.
wslcompact works well, but due to the pipe usage is not ideal if you're low on memory (some report corruption but lacking details), regardless you still need enough disk for the exported disk copy to replace the original.
Personally VMware guest seems easier to deal with than WSL2, but you can probably get away with configuring the disk to be sparse VHD (needs to be done early IIRC, otherwise the min size may not drop below the current size). For the most part, when data is removed from the vdisk then it should be reclaimed as free disk space again. It does seem to build up over time/use, so wslcompact or purging via Docker Desktop GUI (if lacking resources for wslcompact) seem to be eventually necessary to fully reclaim disk back.

Thanks @polarathene for the update!


Noise report using cloud providers

Hi! I'm using an EKS cluster version 1.24, with Krane, It was installed Krane v0.1.1 release, using the helm installation.
Using the krane report --incluster
In the report generated by krane is mapped as danger default RBAC resources from EKS, this cause noise in the report.
I would like to see your opinion about this topic, I suppose that will be the same for GKE, AKS or OpenShift. Thank you.
Result:
{
  "summary": {
    "danger": 9,
    "warning": 9,
    "info": 1,
    "success": 35
  },
  "results": [
    {
      "id": "risky-any-resource-list",
      "status": "danger",
      "group_title": "Risky Roles/ClusterRoles allowing list action on all resources",
      "info": "Roles/ClusterRoles allowing list action on all resources. This might be dangerous. Review listed
Roles!",
      "items": [
        "ClusterRole aws-node in * namespace(s)",
        "ClusterRole eks:addon-manager in * namespace(s)",
        "ClusterRole ks-sa-roles in * namespace(s)",
      ]
    },

This is certainly something that could be improved. As it stands the tool doesn't distinguish between vendor specific roles and custom roles. In the meantime you could perhaps look at whitelists. Bear in mind that currently helm doesn't support option for passing custom configuration files at install time. There is a PR opened around that though (#253) so the ability to pass these things in should land soon.


race: 'unsupported VMA range' on linux/arm64

I think it might related to
https://github.com/llvm-mirror/compiler-rt/blob/master/lib/dfsan/dfsan.cc#L85
What version of Go are you using (go version)?
master

$ go version devel +8c10ce164f Fri Jan 25 15:16:56 2019 +0000 linux/arm64

Does this issue reproduce with the latest release?
No
go test -v -race
go test: -race is only supported on linux/amd64, linux/ppc64le, freebsd/amd64, netbsd/amd64, darwin/amd64 and windows/amd64

What operating system and processor architecture are you using (go env)?

What did you expect to see?
ALL test passed.
What did you see instead?
## all non-race test passed

./all.bash -v
......
##### os/user with tag osusergo
ok      os/user 0.010s

##### GOMAXPROCS=2 runtime -cpu=1,2,4 -quick
ok      runtime 29.164s

##### cmd/go terminal test
PASS
ok      _/root/godev/src/cmd/go/testdata/testterminal18153      0.007s

##### sync -cpu=10
ok      sync    2.018s

##### Testing race detector
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48
FAIL    runtime/race    0.035s
2019/01/27 22:55:33 Failed: exit status 1
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48
FAIL    flag    0.041s
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48
FAIL    net     0.012s
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48
FAIL    os      0.015s
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48
FAIL    os/exec 0.009s
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48
FAIL    encoding/gob    0.012s
2019/01/27 22:55:37 Failed: exit status 1
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48
exit status 66
FAIL    _/root/godev/misc/cgo/test      0.007s
2019/01/27 22:56:00 Failed: exit status 1
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48
FAIL    flag    0.013s
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48
FAIL    os/exec 0.016s
2019/01/27 22:56:01 Failed: exit status 1

##### ../misc/cgo/stdio
skipped due to earlier error

##### ../misc/cgo/life
skipped due to earlier error

##### ../misc/cgo/test
skipped due to earlier error

##### ../misc/cgo/testgodefs
skipped due to earlier error
2019/01/27 22:56:07 FAILED

@cherrymui I took a quick look on disabling the test. But I don't have a good solution so far. I try to list what I found and the difficulty here:

Check VM bits

Reading kernel config can be difficult in some cases. Exporting kernel config is an optional feature.
Mapping some addresses may also be unreliable due to unknown memory layout during the execution.
llvm/compiler-rt get VMA size by checking the most significant bit of the stack. https://github.com/llvm-mirror/compiler-rt/blob/58d43607862096aeb32d72173911c9df244a30f1/lib/tsan/rtl/tsan_platform_linux.cc#L212

runtime check

Only g0 stack of the main thread is allocated by kernel. It is almost impossible to get VMA size by normal Go application. We will have to add some new interface in runtime to return VMA size.

existing checks for race detector support

build time check. All other architectures either support race detector or doesn't support. So they can do build time check: https://go.googlesource.com/go/+/master/src/cmd/go/internal/work/init.go#51 and https://go.googlesource.com/go/+/master/src/cmd/internal/sys/supported.go#9
dist/test will enable race detector if race detector is supported. https://go.googlesource.com/go/+/master/src/cmd/dist/test.go#1335 and https://go.googlesource.com/go/+/master/src/cmd/dist/test.go#1474
I don't have a good solution to check VMA size in dist/test because cmd/dist has to be buildable by Go 1.4. And I didn't have an easy way to check VMA size, except adding some new interface in runtime.

Possible solution

no build time check for arm64. We are always able to build with race. This is reasonable, because it can be cross-compiled.
execution will fail if the binary is build with race and kernel is not configured to use 48-bit VMA.
Always disable "-race" for arm64 in https://go.googlesource.com/go/+/master/src/cmd/dist/test.go#1474

Does this make sense?

Thanks for the information.
As you said, we should not do anything at build time, exactly because of cross compilation.
Always disabling -race test in cmd/dist is also not a good idea, as we'll not know if it breaks.
Maybe in cmd/dist, we can check the stderr and if it is the unsupported VMA failure, we turn it into SKIP instead of FAIL?
Users who use the race detector can still encounter this in their own programs. So we still need to document this.

If I understand this correctly, the race detector simply doesn't work reliably on arm64 systems.  And the problem can only be fixed in race.syso, not in Go code.
That's too bad, but I think that for 1.12 we should remove arm64 from raceDetectorSupported in cmd/dist/test.go and cmd/internal/sys/supported.go.  It wasn't supported in 1.11, so this is not a regression.  We can try again to enable it in 1.13.

@ianlancetaylor Actually  for 44-bit VMA , ppc64le has same problem.  Only 46-bit VMA and 47-bit VMA are currently supported for powercp64 on Go.

@ianlancetaylor, what do you want to do here (if anything) for Go 1.12? I see that https://go-review.googlesource.com/c/go/+/160919/ stalled and hasn't been updated with review feedback.

@cherrymui @mengzhuo Should we keep or disable arm64 race detector for the 1.12 release?  I don't fully understand the circumstances under which it fails.

@ianlancetaylor I think keep linux/arm64 race detector Linux/arm64 is fine. This issue only happens when you try to run race builds on unsupported kernel

OK, thanks.

If I understand correctly, the ARM64 race detector support works fine under certain kernel configurations (probably the popular one?), but doesn't work at all under some other kernel configurations. So I was suggesting we skip the tests under unsupported kernel.

@cherrymui, skipping tests is fine (even for Go 1.12, or a later point release) if you want. But I don't think this is a release blocker at this point. That is, all.bash passing for everybody's distro/version isn't a blocker for shipping a release.

I agree that this is not a release-blocker but more generally I don't think skipping the tests is really a fix here.  If we can skip the tests, we can and should also report the problem when running a binary built with -race.

@cherrymui @ianlancetaylor Currently data race works fine on the kernel configured with 48-bits VMA which is popular.  Data race doesn't support the kernel configured with 39-bit VMA and 42-bit VMA because the shadow memory calculated in TSAN runtime will be out of the address space.  I will investigate how to support 39-bit VMA in TSAN runtime.

@bradfitz I also agree that this does not block release.

@ianlancetaylor TSAN runtime will report the error message when running a binary built with -race on unsupported VMA.
FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48

OK, thanks.


My bot has stopped receiving notifications chat_member, my_chat_member, chat_join_reuest.

Question

Hello!
This is not the first time that my bot has stopped receiving notifications chat_member, my_chat_member, chat_join_reuest. All this is written in getWebhookInfo. Telegram simply does not send this data. It sends messages, sends data about the buttons pressed.
I tried deleting and re-specifying the webhook. Turned it on via polling. But nothing helps.
Last time a few hours passed and everything worked by itself.
But this is a very long time.

There was a second site that could duplicate the connection. Reloading the token helped. But disabling the second site helped more :)

Hi @hailovigor
I've an issue that is maybe close to yours.
Since you solved it and no support is finding why new_chat_members is not working properly, have you maybe any insight about this?
Did telegram change something, making events from library unable to work as intended ?
Thanks

Check getWebhookInfo.  Is this event added there?

If I insert logs into processing webhook requests, I didn’t have any data there either.
This is not a library

@hailovigor Hi Igor thanks for your help. Thanks for your test! Does this mean it did not work for you either to catch new members? So node-telegram-bot is not updated ?

i am also receiving this issue actually via other package (grammyJS)

This library works fine.  But sometimes I get a 429 error for bot authorization and a block for 30 minutes.  I have thousands of users, but the bot does not work for so long!  I wish I knew the limits and rules so as not to get blocked.

Regarding the topic.  Check the data in webhook.  If the data does not arrive there.  Then check GetWebhookInfo.  Add the desired event there.  If all else fails, create a new token.

Regarding the topic. Check the data in webhook. If the data does not arrive there. Then check GetWebhookInfo. Add the desired event there. If all else fails, create a new token.

Actually, i did both - made sure event is listened to via this method + tried to generate a new token (and even a completely new bot). unfortunately it is still inconsistantly sending me these events.
It does feel like some rate limit/rule broken by the bot and also i wish that it was public! i sent a message to Telegram support hopefully they will reply

Regarding the topic. Check the data in webhook. If the data does not arrive there. Then check GetWebhookInfo. Add the desired event there. If all else fails, create a new token.

Actually, i did both - made sure event is listened to via this method + tried to generate a new token (and even a completely new bot). unfortunately it is still inconsistantly sending me these events.
It does feel like some rate limit/rule broken by the bot and also i wish that it was public! i sent a message to Telegram support hopefully they will reply

Hi @OfirYC @hailovigor
With your answers I realize I'm not the only one having problem to get this event!
Great you sent a message to telegram support but why devs are not doing this themselves, by fixing those issues? Why are they not listening to the community?
Do you know any library that works to get this simple new members event ? Cheers

Do you have this error with different bots?

Do you have this error with different bots?

Yeap. tried even creating new bot.

Regarding the topic. Check the data in webhook. If the data does not arrive there. Then check GetWebhookInfo. Add the desired event there. If all else fails, create a new token.

Actually, i did both - made sure event is listened to via this method + tried to generate a new token (and even a completely new bot). unfortunately it is still inconsistantly sending me these events.
It does feel like some rate limit/rule broken by the bot and also i wish that it was public! i sent a message to Telegram support hopefully they will reply

Hi @OfirYC @hailovigor With your answers I realize I'm not the only one having problem to get this event! Great you sent a message to telegram support but why devs are not doing this themselves, by fixing those issues? Why are they not listening to the community?
Do you know any library that works to get this simple new members event ? Cheers

i hope they answer quickyl and shed some light on it, for now it seems like my users will have to suffer for a bit :(
A workaround i am expermiting with right now is accessing update.message.new_chat_members or .left_chat_member.
But i was told that this is wont work in all cases because in super group it wont send it.
do you have telegram? msg me on @perpiepapa maybe we can debug together, because i wanna test on supergroup but having hard time to do it.

Regarding the topic. Check the data in webhook. If the data does not arrive there. Then check GetWebhookInfo. Add the desired event there. If all else fails, create a new token.

Actually, i did both - made sure event is listened to via this method + tried to generate a new token (and even a completely new bot). unfortunately it is still inconsistantly sending me these events.
It does feel like some rate limit/rule broken by the bot and also i wish that it was public! i sent a message to Telegram support hopefully they will reply

Hi @OfirYC @hailovigor With your answers I realize I'm not the only one having problem to get this event! Great you sent a message to telegram support but why devs are not doing this themselves, by fixing those issues? Why are they not listening to the community?
Do you know any library that works to get this simple new members event ? Cheers

i hope they answer quickyl and shed some light on it, for now it seems like my users will have to suffer for a bit :(
A workaround i am expermiting with right now is accessing update.message.new_chat_members or .left_chat_member.
But i was told that this is wont work in all cases because in super group it wont send it.
do you have telegram? msg me on @perpiepapa maybe we can debug together, because i wanna test on supergroup but having hard time to do it.

Hey 😊👍
Since I'm really beginner, I'm OK to help testing by entering the group, to help you check if event triggers. And when you find the solution please share it then and I'll try to put it in my script 😊
OK I'll write to you. Let's hope devs will fix this asap ....

I talked to them.  I was asked "is there only one instance running?"

I talked to them. I was asked "is there only one instance running?"

why did they ask that? whats the underlying logic? what are they checking exactly? that's too vague to figure out whats going on

I had a “transition” from the old bot code to the new one.  These are two different sites.  I turned on the new site, and threw telegram webhook events on the old one.  And it seems that this knocked everything out so that the new token helped. ( Of course, I tried different order of launching the site.)

what do you mean by site?
thats not my case anyway, and also Telegram doesnt let u to set up 2 different instances of a single bot afaik

Regarding the topic. Check the data in webhook. If the data does not arrive there. Then check GetWebhookInfo. Add the desired event there. If all else fails, create a new token.

Actually, i did both - made sure event is listened to via this method + tried to generate a new token (and even a completely new bot). unfortunately it is still inconsistantly sending me these events.
It does feel like some rate limit/rule broken by the bot and also i wish that it was public! i sent a message to Telegram support hopefully they will reply

Hi @OfirYC @hailovigor With your answers I realize I'm not the only one having problem to get this event! Great you sent a message to telegram support but why devs are not doing this themselves, by fixing those issues? Why are they not listening to the community?
Do you know any library that works to get this simple new members event ? Cheers

i hope they answer quickyl and shed some light on it, for now it seems like my users will have to suffer for a bit :(
A workaround i am expermiting with right now is accessing update.message.new_chat_members or .left_chat_member.
But i was told that this is wont work in all cases because in super group it wont send it.
do you have telegram? msg me on @perpiepapa maybe we can debug together, because i wanna test on supergroup but having hard time to do it.

There is still no follow up by the main devs.
I'm checking telegraf since this telegram node api is not properly updated and maintained

Hi! seems that node-telegram-bot-api does not automatically add telegram optional updates, i tryed to add it to getUpdates opts in telegramPolling.js (line 137-141) but for some reason does not works, i hope that someone is going to fix this
Currently a workaround is asking telegram again to receive this updates:
await bot.setWebHook("",{allowed_updates: JSON.stringify(["message", "edited_channel_post", "callback_query", "message_reaction", "message_reaction_count", "chat_member"])})

Hi! seems that node-telegram-bot-api does not automatically add telegram optional updates, i tryed to add it to getUpdates opts in telegramPolling.js (line 137-141) but for some reason does not works, i hope that someone is going to fix this
Currently a workaround is asking telegram again to receive this updates: await bot.setWebHook("",{allowed_updates: JSON.stringify(["message", "edited_channel_post", "callback_query", "message_reaction", "message_reaction_count", "chat_member"])})

Thank you so much! I've tried a bunch of options, and if I just call getUpdates and pass allowed_updates to it, they are successfully updated and displayed in the:
/getWebhookInfo
But if I try to pass in polling: {} - it is not updated

Some update...
export const bot = new TelegramApi(
process.env.TG_KEY,
{
polling: {
params: {
allowed_updates: JSON.stringify(allowedUpdates),
},
},
});

Worked for me!
just add JSON.stringify(your_array_params)

Question

Hello!
This is not the first time that my bot has stopped receiving notifications chat_member, my_chat_member, chat_join_reuest. All this is written in getWebhookInfo. Telegram simply does not send this data. It sends messages, sends data about the buttons pressed.
I tried deleting and re-specifying the webhook. Turned it on via polling. But nothing helps. Last time a few hours passed and everything worked by itself. But this is a very long time.

Have you send res.sendStatus(200); right after getting data from webhook, when setting up a Telegram webhook, it's recommended to send res.sendStatus(200); immediately upon receiving a message or any data from Telegram.
it actually helps keep your Telegram response time fast and prevents delays. Here’s why:
1.Immediate Acknowledgment: Responding with 200 OK right away tells Telegram that your server received the update successfully, so Telegram can quickly move on. If you wait to respond (e.g., by processing data first), Telegram may interpret this delay as a failure, causing retries and potentially slowing down your system with redundant updates.
2.Async Processing: After sending 200 OK, you can handle message processing asynchronously, without holding up the initial response. This lets you perform actions—like saving data to a database or triggering other processes—without blocking or slowing down the interaction with Telegram.
3.Reduced Risk of Overloading: Not responding quickly can result in multiple retry attempts from Telegram. This could lead to a buildup of queued requests, which might temporarily slow down your server and make it feel unresponsive.


install uses chown root.root instead of chown root:root

https://github.com/SequentMicrosystems/megaind-rpi/blob/870731343ecac88c825bbb376f2b39a40fc9b595/Makefile#L38
The man page does not list this syntax
https://linux.die.net/man/1/chown

The syntax has been fixed. Thank you again.
Alex.


how to find percentage of false positive and false negative in confusion matrix

sir
i i have 4 categories in my class variable hence the confusion matrix is a 4X4 matrix so how do i find the percentage of false negative and false positive

@sudeeshsahu  the answer here by makis explains the answer to your question beautifully with diagrams
https://stackoverflow.com/questions/50666091/true-positive-rate-and-false-positive-rate-tpr-fpr-for-multi-class-data-in-py
Do go through it and if you still have questions, I will be happy to answer.

Thank you for the support sir, i will look into it .
Have a good day sir.
On Nov 12, 2019 11:49 AM, "Anush Sankaran"<EMAIL_ADDRESS>wrote:

@sudeeshsahu https://github.com/sudeeshsahu the answer here by makis
explains the answer to your question beautifully with diagrams
https://stackoverflow.com/questions/50666091/true-positive-rate-and-false-
positive-rate-tpr-fpr-for-multi-class-data-in-py
Do go through it and if you still have questions, I will be happy to
answer.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/goodboyanush/isme-bangalore-Oct-Nov-2019/issues/2?email_source=notifications&email_token=AMH5NA52AGL45W5X4T64AX3QTJDIFA5CNFSM4JLWL7FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDZFJNI#issuecomment-552752309,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AMH5NA7KDKTPZO7GAE45DDTQTJDIFANCNFSM4JLWL7FA
.

sir i actually found a method to convert a multi-class confusion matrix into a binary class confusion matrix. so can i use this way to find out the percentage of false positive and false negative. this is the code i used:
import mlxtend
from mlxtend.evaluate import confusion_matrix
confusion_matrix(y_test,nn_pred,binary=True)
can i move forward with this method?

@sudeeshsahu That will not exactly work in this case. Consider the example:
y_actual=    [1, 1, 1, 0, 0, 2, 0, 3]
y_predicted = [1, 0, 1, 0, 0, 2, 1, 3]

where there are 4 possible class labels - [0,1,2,3]
The actual confusion matrix, will look something like:

However, when we want to convert them into binary class label based confusion matrix, we will do the following:
cm = confusion_matrix(y_target=y_actual,
                      y_predicted=y_predicted,
                      binary=True,
                      positive_label=1)

where, along with the other inputs we should also give which is that one class label that we consider as positive class - positive_label=1. Thus, all the other three class labels become negative class and made as 0 . In the above example all the three classes [0,2,3] becomes 0.
We cannot do that because, all four class labels are correct. Thus, I strongly recommend that we go by the method as proposed in:
https://stackoverflow.com/questions/50666091/true-positive-rate-and-false-positive-rate-tpr-fpr-for-multi-class-data-in-py


Resourcekey vs Key bug

How should I understand Key and resourceKey on tthe localizationrecords db?
#My case:
i want to add Two records:
insert into dbo.localizationrecords
([Key],localizationculture,resourcekey,text,updatedTimestamp)
values  ('bank transfer','en','Iwencik.Host.Views.Registration.DetailsIwencik.Host','Bank transfer',getdate())
, ('bank transfer','en','Iwencik.Host.Controllers.RegistrationController','Bank transfer',getdate()))

Both records has the same key but another resourceKey. I would assume, that there is posibility to add Two records with this same key and another resorce key, but when i try to add this records i get duplicate key constraint exception:
Msg 2627, Level 14, State 1, Line 49
Violation of UNIQUE KEY constraint 'AK_LocalizationRecords_Key_LocalizationCulture'. Cannot insert duplicate key in object 'dbo.LocalizationRecords'. The duplicate key value is (bank transfer, en).
The statement has been terminated.

Is it bug or my bad understanding of using this model?

Hi @garfieldos I use the Key and the localizationculture as a constraint, because I assumed that it makes no sense to repeat this translation, the bank transfer is the same, just used form 2 different controllers, views.
I was thinking this would make more sense as a shared resource.
May it would be better if I used the resourcekey also in the constraint, then the user of the package can decide.
If you want, I can add this, I also need to update to 1.1.
Opinion?
Greetings Damien

One key for all resource would be good, if there will be some possibility to reach this translation withot resourcekey context (without view/controller context).
Now we are using Localizer object with view/controller context and in all that contextes it should be a possibility of using this same translation keys (because their meaning could differ across different contextes).
I think, that it should be composite key on columns: Key and ResourceKey (of course sense of column name: "key" will be lost after this change)

@garfieldos makes sense, I'll make the constraint using all 3 properties. The key still makes sense for me, as this is the key without the resource.
Will update this ASAP. (also only with .NET 1.1) I want to move on from 1.0. Is this ok, or do you require the fix for .NET Core 1.0?
Greetings Damien

1.1 will be ok for me.
Thx

@garfieldos
Updated NuGet:
https://www.nuget.org/packages/Localization.SqlLocalizer/1.0.4
Code:
https://github.com/damienbod/AspNet5Localization/tree/master/AspNet5Localization/src/Localization.SqlLocalizer

Thx a lot!


Automated cherry pick of #5919: fix: 非私有云同步region name

Cherry pick of #5919 on release/3.1.
#5919: fix: 非私有云同步region name

Codecov Report

Merging #5922 into release/3.1 will increase coverage by 0.00%.
The diff coverage is 0.00%.

@@             Coverage Diff             @@
##           release/3.1   #5922   +/-   ##
===========================================
  Coverage         7.45%   7.45%
===========================================
  Files              584     584
  Lines            98591   98593    +2
===========================================
+ Hits              7350    7352    +2
- Misses           90668   90669    +1
+ Partials           573     572    -1

Flag
Coverage Δ

#aFlag
7.45% <0.00%> (+<0.01%)
:arrow_up:

Impacted Files
Coverage Δ

pkg/compute/models/cloudregions.go
1.83% <0.00%> (-0.01%)
:arrow_down:

pkg/appsrv/response.go
70.58% <0.00%> (+2.94%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8497569...5d2d5f2. Read the comment docs.


Roll Engine from 2b6c71c4d3ab to 88cc61e7ec65 (3 revisions)

https://github.com/flutter/engine/compare/2b6c71c4d3ab...88cc61e7ec65
2020-06-23<EMAIL_ADDRESS>Roll Skia from 7f9aa5a2c609 to 3b6b7478421b (7 revisions) (flutter/engine#19246)
2020-06-23<EMAIL_ADDRESS>Roll Fuchsia Linux SDK from RGOLO... to 3lfit... (flutter/engine#19243)
2020-06-23<EMAIL_ADDRESS>Roll Dart SDK from 5a6c1e515849 to 6b630c3719bd (7 revisions) (flutter/engine#19247)
Also rolling transitive DEPS:
fuchsia/sdk/core/linux-amd64 from RGOLO8d3F4MP to 3lfitkpybVNA
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/master/autoroll/README.md

Commit queue failed; closing this roll.


detect_swipe not working for me

Hi there,
I'm attempting to use detect_swipe with the featherlight gallery plugin, and it doesn't seem to be working for me. No matter what I do, I'm still seeing the prev/next arrows on my phone, and swiping isn't working. I've read up on loading the detect_swipe plugin before featherlight, and I'm pretty sure this is all in place, however it doesn't seem to be working. Here's my WordPress enqueuing:
/**
 * jQuery Detect Swipe plugin.
 *
 * @see    https://github.com/marcandre/detect_swipe
 */
wp_enqueue_script(
	'detect_swipe',
	'//cdn.rawgit.com/marcandre/detect_swipe/v2.1.1/jquery.detect_swipe.js',
	array( 'jquery' ),
	null,
	false
);

/**
 * Featherlight JS.
 *
 * @see    https://github.com/noelboss/featherlight
 */
wp_enqueue_script(
	'featherlight',
	'//cdn.rawgit.com/noelboss/featherlight/1.3.2/release/featherlight.min.js',
	array( 'jquery' ),
	null,
	true
);
wp_enqueue_script(
	'featherlight-gallery',
	'//cdn.rawgit.com/noelboss/featherlight/1.3.2/release/featherlight.gallery.min.js',
	array( 'jquery' ),
	null,
	true
);

And I'm loading up my custom featherlight call on document.ready. I've tested to see that detect_swipe is enabled at the time featherlight is initialized with the following:
console.log( $.detectSwipe );

. . . and it's definitely loaded up. Any idea what I'm doing wrong?
Thanks!

This probably should be on Stack Overflow, but it looks like you're not loading the dependencies correctly. The cascade should be jQuery->detect swipe->featherlight->featherlight gallery. Something like this should get you up and running.

:+1: to all of what @robneu said :smile:

Annnnnnnd it was a caching issue :) Yay. Those are so fun... thanks for the help though.

:+1:


Use a stable sort in TextReporter

Right now the order for tests in TextReporter is undefined. It'd be nice to sort the test results so they're easier to scan.

Oops.
The change I made for #6 only affected the order of modules we're loading. In hindsight that wasn't a very solid solution.


Issue 723

Wiring the clustered cache API to the voltron client code.

Pretty sure that the empty file clustered/client/private/cache/retriever/catalog.xml should not be part of this


Credentials stored in clear text in minemeld-web.log

In the AUDIT logs in minemeld-web.log, the passwords for the users are stored in cleartext.
I believe the issue resides in:
https://github.com/PaloAltoNetworks/minemeld-core/blob/409896211a3d9cb67ffdc743ed9102b3dbcc2257/minemeld/flask/aaa.py#L91-L95
I apologize in advance if this is expected functionality.

Any update on this?
I understand this can be a lower priority issue.
Thanks for your time!


Auto-generated encoder model config

resolve #902

@Slyne , please review.

I think I don't get your point.
Tritonserver supports Auto-Generated Model Configuration. So I just remove the output in the template.
Do you mean that we must define output shape explicitly?

I think I don't get your point.
Tritonserver supports Auto-Generated Model Configuration. So I just remove the output in the template.
Do you mean that we must define output shape explicitly?

It's not a must.
But it will be easier for users to debug.
For example, the wenetspeech offline model exported by onnx has an encoder of shape [B, T, -1], while for aishell2 u2++ model , the output shape is [B, T, 256]. After we generate the onnx model, users may encounter some issues in server part and if the config.pbtxt can provide more info, it will be more helpful.

Why not use HTTP API?
curl localhost:8000/v2/models/<model name>/config

Then we can completely avoid processing the model details.

Why not use HTTP API?
curl localhost:8000/v2/models/<model name>/config

Then we can completely avoid processing the model details.

This suggestion is great!
We may call other model's configuration instead of reading onnx_config.json in triton. But, for example, the feature_size in this line can be inferred from encoder_out.   However, it's still due to the previous issue I mentioned, this feature size might be '-1', and it will fail from this line.
I'm working on updating this example to the latest triton, 22.01. The issue you mentioned will also be fixed. But I will fix this issue by reading the generated onnx model. I still believe a config file with full details will be better.

Great!
This PR can close If you're working on that.


Nwchem fixes

Hello!
I incorporated parsing of  the new requested metadata objects from https://github.com/cclib/cclib/pull/1143 into your branch. Let me know what how this looks to you.
Once https://github.com/OpenChemistry/chemicaljson/pull/21 is approved/merged, you will have access to the requested objects in the chemical json output in the inputParameters field, while the current metadata storage within cclib remains consistent.
The only additional object we aren't incorporating yet is the keywords_line. Although there is value to having this information, it is going to take some planning and will benefit from being it's own pull request.

Thanks Amanda! Just realized this PR existed

They slip under the radar so easily, no worries!


Changes to hook API to make it more Observable-friendly

Addresses #682. See that for detail on new API
Since this affects both HystrixCommand and HystrixObservableCommand and had a large impact on unit tests, I took some time to share code between the two.  This partially addresses #581

NetflixOSS » Hystrix » Hystrix-pull-requests #45 SUCCESS
This pull request looks good


Unable to build after migrating to react native 0.60

As the title says, after I have upgraded React Native to 0.60, I can't compile react-native-shake.
I get the following error in RNShakeEventModule
error: package android.support.annotation does not exist
I figured it had something to do with Android X, so I replaced the package with:
import androidx.annotation.Nullable;
I cleaned the project (removed the build folder) and rebuild the project, but then i get the following compiler errors:
Caused by: com.android.tools.r8.CompilationFailedException: Compilation failed to complete
Caused by: com.android.tools.r8.utils.AbortException: Error: null, Cannot fit requested classes in a single dex file (# fields: 65702 > 65536)
Is anyone getting similar errors or have a working react-native 0.60 with react-native-shake?

I have pushed a new npm package version which address this issue. However, your second issue is not related to this lib specifically. It should be solved by this one:
https://stackoverflow.com/questions/53336676/androidx-multidex-not-found

Thank you for the quick resolution.
For future readers, who might encounter the same problem with the dex file, I posted my fix on Stack Overflow


added Darkmode Toggle using Provider

The file ./lib/models/theme_data.dart is the place to change the appearance of the app.
Please avoid using properties like color, background-color in the widgets.
Rather update the ./lib/models/theme_data.dart file so that the toggle-darkmode functionality works without any hassle.

@arbazdiwan Let me know when you update this PR

I have added the safety checks for in ternary operators where boolean values are to be evaluated.
@himanshusharma89, Please review the commit.

@all-contributors please add @arbazdiwan for code


Backport #2006 to citadel: Fix finding new versions of protobuf

Part of https://github.com/osrf/homebrew-simulation/issues/2274. Use rebase and merge.

I believe test failures on macOS are from https://github.com/gazebosim/gz-physics/issues/442


test(module:cdk): added missing tests for resize observer

PR Checklist
Please check if your PR fulfills the following requirements:

[✔] The commit message follows our guidelines: https://github.com/NG-ZORRO/ng-zorro-antd/blob/master/CONTRIBUTING.md#commit
[✔] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

PR Type
What kind of change does this PR introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Documentation content changes
[ ] Application (the showcase website) / infrastructure changes
[✔] Other... Please describe: missing unit tests for the resize observer

What is the current behavior?
What is the new behavior?

Before
After

Statements
66.03%
100%

Branches
53.33%
100%

Functions
52.94%
100%

Lines
66%
100%

Does this PR introduce a breaking change?

[ ] Yes
[✔] No

Other information

@Laffery
Thank you for checking my PR
Your requested change is made 👍


Need help with libffi arm build

I currently trying to add support of libffi for arm.
I've made port that only lacks building of arm asm files (at least I suppose so).
armasm is unable to build resulting .S files. As I understand armasm have it's own dialect not compatible with libffi asm files.
So my question is: what could be used as replacement of arm assembler?

We do have nasm and yasm wired up in vcpkg_find_acquire_program() -- these might be able to target arm.

They're both not support arm:
http://www.tortall.net/projects/yasm/manual/html/running-arch.html
and Nasm also claims absence of ARM

Maybe this will help:
https://groups.google.com/forum/#!topic/discuss-webrtc/VuM_GkoB_fo

This issue hasn’t been updated in a year, if it is still an issue, please reopen.


ConnectBot Can´t find key Files

Where I must put the Key Files?
ConnectBot can´t find them anywhere.
I had puted them in Downloads.
Thanks,

What version of Android are you using? What version of ConnectBot?

Android 5.1.1
ConnectBot 1.9.2

When you select Manage Pubkeys it should take you to the pubkeys that you currently have loaded.  In your case it will probably be empty, then in the upper right hand corner there should be a folder icon, click that and it will give you a menu with different options to load a key from.  Select Downloads if that's where you've put your key, then click on your key.  It should load it into the previous screen and it can now be used in your connections.

Am 04.03.2017 um 01:36 schrieb
  f00bar12:

  When you select Manage Pubkeys it should take you to the
    pubkeys that you currently have loaded. In your case it will
    probably be empty, then in the upper right hand corner there
    should be a folder icon, click that and it will give you a menu
    with different options to load a key from. Select Downloads if
    that's where you've put your key,

To this Point everything works but the Keyfiles won´t shgow up there
like in all other filemanagers and d other ssh-clients find them
just fine only connectBot will find nothing.

   then click on your key. It should load it into the previous
    screen and it can now be used in your connections.

I can´t because he won´t show up.

  —
    You are receiving this because you authored the thread.
    Reply to this email directly, view
      it on GitHub, or mute
      the thread.

  {"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/connectbot/connectbot","title":"connectbot/connectbot","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/connectbot/connectbot"}},"updates":{"snippets":[{"icon":"PERSON","message":"@f00bar12 in #484: When you select Manage Pubkeys it should take you to the pubkeys that you currently have loaded.  In your case it will probably be empty, then in the upper right hand corner there should be a folder icon, click that and it will give you a menu with different options to load a key from.  Select Downloads if that's where you've put your key, then click on your key.  It should load it into the previous screen and it can now be used in your connections."}],"action":{"name":"View Issue","url":"https://github.com/connectbot/connectbot/issues/484#issuecomment-284108888"}}}

Thanks,
--
  -=============================== Jan-Peter Rühmann & Kuma ===============================-

Gubkower Str.7               [  Tel.:  +49 (38205) 65484  ]<EMAIL_ADDRESS>18195 Prangendorf            [  FAX:   +49 (38205) 65212  ]      http://www.ruehmann.name
[  Tel.:  +49 (38205) 65215  ]
[  Mobil: +49<PHONE_NUMBER>  ]           IT-Servicetechniker
Skype: jan-peter_ruehmann / ICQ: 288192920 / WhatsApp:<PHONE_NUMBER>54 / Twitter: @JPRuehmann
-========================================================================================-
Die Verwendung der Daten zu Werbezwecken ist verboten.

Can you take a screenshot of the pubkeys screen and then the screen you get when you click on the folder icon and post it here?

Here
Hope that helps.
Thanks

I'm having the same problem. The file dialog which connectbot brings up only sees documents, images, videos, and music. It doesn't see any other types of files (in fact it can't even browse anywhere, I only see 'Recents', some Google Drive links, and surprisingly a termux link...

Looks like it is impossible to import the key unless you download it via a browser, but that's a pretty hard task, because keys are simple text files and most browsers will just show their contents instead of downloading them as file.

I reported my workaround here: https://github.com/connectbot/connectbot/issues/541
tl;dr: use Chrome to "Download link", works with local files
Copying local id_rsa to the "Download" folder does not work because it does not appear in the "Downloads" virtual folder, but I found a workaround: I used Chrome to navigate to file:///storage/emulated/0/foldername (and once deep enough I can follow links), long-pressed on id_rsa and selected "Download link". id_rsa is copied to the  "Download" but now appears in the "Downloads" virtual folder and can be added in "Manage pubkeys" in ConnectBot. Horrible, but it works. I am using Marshmallow.

Hello please does anyone know where connectbot saves downloaded files?.
I tried to scrap a website so I can get the source code and connectbot says it's saved as index.html but I don't know where to find index.html


Correct urllib3 compatibility comment

So that it's in line with the code.
I saw the warning raised recently and had to dive into the code for exact compatibility, but I stopped after reading the comment rather than the code.
Maybe a better solution would be to surface the supported versions in the warning? However, I see it was changed recently in https://github.com/requests/requests/pull/4144/, any specific reason for removing that?

The primary reason is just trying to reduce the maintenance area of this code: it's painful to need to update in lots of places. Ideally we'd have a nice refactor that will work here to minimise the number of places we need to write our version constraints: if someone wants to do that, it might be a worthwhile project.


Test provided dependency scope in the ibm.ibmmq connector

Description:

$subject

This functionality was tested and send a draft PR for the IBM MQ module [1]. The feature is working as expected.
[1] - https://github.com/ballerina-platform/module-ballerinax-ibm.ibmmq/pull/41


Add missing code for parser entry template and fix build-script.py

In #1346 code was modified on the file and not in the template

@swift-ci please test

The error was properly introduced in #1149 where we refactored the code-gen into a single executable.
Not sure why the CI didn't catch that before

@swift-ci plese test

@swift-ci please test

@kimdv looks like things don't build without this, is it good to merge?

@kimdv looks like things don't build without this, is it good to merge?

For me yea. But we need approval from @ahoppen I think 🤔

@swift-ci please test and merge

Let's merge and do post-commit review if needed to unblock folks

@Keith this broke the build: https://ci.swift.org/job/oss-swift-incremental-RA-macos/2331/. I am going to try reverting it.

dang, sorry, thanks! looks like we need to run the integration tests before merging again

Not sure why the CI didn't catch that before

We don’t verify that the CodeGenerated files match the templates because running code generation requires us to check out an older version of SwiftSyntax (the one specified in the Package.swift of CodeGeneration). But CI doesn’t have internet access at that point to perform that checkout… So it’s a little tricky to do.


exclude bin and obj from workspace scan ( defaults )

This PR improves workspace scan speed especially when working with c# which produces bin and obj folder by excluding these two in the default settings "commentAnchors.workspace.excludeFiles".

Thanks for your PR!