Sign_in custom method: how?

Hi all.
I'm having trouble implementing a custom method in a controller to login an user.
In the User model I have, as requested in the doc, added include DeviseTokenAuth::Concerns::User, which should allow me to use methods like create_new_auth_token.
But, in my custom login method - which handles user which logged in via Facebook SDK in a mobile client, passing me back the access_token - :
def login_user_facebook
    @graph = Koala::Facebook::API.new(facebook_params[:access_token])
    @user_profile = @graph.get_object("me?fields=id,email")
    if @user = User.where(provider: "facebook", uid: @user_profile["id"])
        sign_in(:user, @user)
        new_auth_headers = @user.create_new_auth_token
        //posting the answer there
    else
        //posting the error there
    end
end

I get the following error: undefined method 'create_new_auth_token' for #<User::ActiveRecord_Relation:0x00000005451f50>.
Is it a bug causing this? Am I implementing the sign_in wrongly?

Ok I think I figured it out.
@lynndylanhurley feel free to close this issue, or leave it open, as it may help some people using your gem is a scoped route.
Basically, I was making two mistakes here:

by using User.where I was not getting back an user, so that's why create_new_auth_token wasn't working. Needed to use User.find

Since my user resource is scoped into API::V1:: in the routes, I needed to call sign_in by scoping it too.

So the working code is:
   @user = User.find_by(provider: "facebook", uid: @user_profile["id"])
    if @user != nil
        sign_in(:api_v1_user, @user)

        new_auth_header = @user.create_new_auth_token()

        # update response with the header that will be required by the next request
        response.headers.merge!(new_auth_header)

        render json: {
            user: @user,
        }, status: 200
    else
        render json: {
            error: "There is no user via facebook with this name"
        }, status: 400
    end

`


cmd/link: SIGBUS error when linking (because disk space consumed)

What version of Go are you using (go version)?

$ go version go1.13 linux/amd64

What operating system and processor architecture are you using (go env)?
go env Output
$ go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jasper/.cache/go-build"
GOENV="/home/jasper/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"

What did you do?
I tried to run make to build my golang project

What did you expect to see?
Normal binaries being built
What did you see instead?
unexpected fault address 0x7fad5e500000
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x7fad5e500000 pc=0x45b96a]

goroutine 1 [running]:
runtime.throw(0x690d25, 0x5)
        /usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc00085cef0 sp=0xc00085cec0 pc=0x42da22
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:391 +0x455 fp=0xc00085cf20 sp=0xc00085cef0 pc=0x4429c5
runtime.memmove(0x7fad5e1fa520, 0xc00d9ac000, 0x632a42)
        /usr/local/go/src/runtime/memmove_amd64.s:422 +0x50a fp=0xc00085cf28 sp=0xc00085cf20 pc=0x45b96a
cmd/link/internal/ld.(*OutBuf).Write(0xc0006ca000, 0xc00d9ac000, 0x632a42, 0x950144, 0x200, 0x10, 0x0)
        /usr/local/go/src/cmd/link/internal/ld/outbuf.go:65 +0xa0 fp=0xc00085cf78 sp=0xc00085cf28 pc=0x5ac2e0
cmd/link/internal/ld.(*OutBuf).WriteSym(0xc0006ca000, 0xc00aec2080)
        /usr/local/go/src/cmd/link/internal/ld/outbuf.go:159 +0x6c fp=0xc00085cfc8 sp=0xc00085cf78 pc=0x5acaac
cmd/link/internal/ld.blk(0xc0006ca000, 0xc00f314000, 0x169eb, 0x1c400, 0x135f520, 0xb8bf62, 0x8771c0, 0x200, 0x200)
        /usr/local/go/src/cmd/link/internal/ld/data.go:786 +0x10f fp=0xc00085d098 sp=0xc00085cfc8 pc=0x55972f
cmd/link/internal/ld.writeDatblkToOutBuf(0xc0006c8000, 0xc0006ca000, 0xe06000, 0xb8bf62)
        /usr/local/go/src/cmd/link/internal/ld/data.go:825 +0xaf fp=0xc00085d260 sp=0xc00085d098 pc=0x559d7f
cmd/link/internal/ld.Datblk(...)
        /usr/local/go/src/cmd/link/internal/ld/data.go:808
cmd/link/internal/amd64.asmb(0xc0006c8000)
        /usr/local/go/src/cmd/link/internal/amd64/asm.go:688 +0x1fe fp=0xc00085d2d0 sp=0xc00085d260 pc=0x5d1ade
cmd/link/internal/ld.Main(0x84bdc0, 0x10, 0x20, 0x1, 0x7, 0x10, 0x69aa6a, 0x1b, 0x697551, 0x14, ...)
        /usr/local/go/src/cmd/link/internal/ld/main.go:262 +0xd5d fp=0xc00085d428 sp=0xc00085d2d0 pc=0x5ab32d
main.main()
        /usr/local/go/src/cmd/link/main.go:65 +0x1d6 fp=0xc00085df60 sp=0xc00085d428 pc=0x614396
runtime.main()
        /usr/local/go/src/runtime/proc.go:203 +0x21e fp=0xc00085dfe0 sp=0xc00085df60 pc=0x42f3be
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00085dfe8 sp=0xc00085dfe0 pc=0x45a2b1

Resolution
The problem was that the "/" directory was 100% consumed and there was not enough space to build the project. The error message from build should be a bit more clear

Hi. Thanks for raising this issue. Can you please upgrade to go 1.15.2 and confirm if the problem is present there.

This looks like a real bug. I believe the new linker is memory mapping the output, but given the trace it appears not be using runtime/debug.SetPanicOnFault and reporting handling memory faults from I/O errors gracefully.
/cc @cherrymui @thanm @jeremyfaller

Looks like dup of #37310, which is fixed on Linux and darwin in Go 1.15 (by preallocating file space), but not on other OSes.
Go 1.13 is no longer supported.
Closing as a dup.
If we do #41155, we could use SetPanicOnFault to make it fail gracefully on OSes we don't preallocate.


Make filter sidebar slide animation restricted to div

Describe the bug
The slide animation when closing and opening the filter sidebar comes from the edge of the screen regardless of where the component is implemented. It should just slide out of the side of the component.
To Reproduce
Steps to reproduce the behavior:

npm install @open-rpc/logs-react
import it into a project (inspector works well)
pass it test logs (as in the readme)
open and close filter
observe

:tada: This issue has been resolved in version 1.0.5 :tada:
The release is available on:

GitHub release
1.0.5
npm package (@latest dist-tag)

Your semantic-release bot :package::rocket:


Why not provide requirement.txt file

I‘m used to virtualenv, and use "pip install -r xxx", why can't provide it, hhh, just think it's easy to quickly start.

I‘m used to virtualenv, and use "pip install -r xxx"

Just like I'm used to poetry. I use it in most of my python projects because it helps control dependencies in a very convenient way (IMO). Its installation is quite simple, you can see an example in the GA workflows.
BTW. poetry can generate a requirements.txt file if you need it.


Line break inside bold or italic markup leaves first part marked up

Run the example Mac app.
Place the caret in the middle of the word bold.
Hit return.
Observe that the first part of the word (bo) is still bold while the part on the new line (ld) is regular weight. It seems that both parts should either be bold or not be bold.

Same type of error for italics.

Hi, any interest in submitting a pull request with a fix?


KeywordCompletionProvider: added Nette 3 keys

@matej21 Is it ok?

Can you please check it? @matej21 @juzna

Ping @matej21

merged 74265f8ee3f7fa47a76abb42b8a8d07001edefb7

🎆

also, update is available in plugin marketplace


Faster 2D convolution approaches

I think of different 2D convolution approach for signigically speed-up this plugin.
One possible speed-up of 2D-convolution with typical 8 or even 10 bit unsigned integer input data: To make not kernel_x_input+output_sum but LUT addition to output sum.
But I do not know is there significant difference with todays CPUs in Mul+Add operation in compare with Add only. It looks only implementation and testing required.
Because for 2D convolution we need to multiply each kernel sample with each input sample but all possible 8bit input samples are only 256 numbers limited count - we can make pre-multiplied 256_x_kernel_size LUT and just read-index this LUT instead or Mul. For even 10 taps 2D-kernel we have 20x20x4byte_float_x_256=about 400 kbyte LUT that is good cacheable on most CPUs in season of 201x years and may be later.
So the main computational line of convolution (from C-routine)
result += src_ptr[lx] * coeff_ptr[lx];
may be replaced with something like
result += LUT[src_ptr[lx]]; //- no multiplication - just cache read and addition
The LUT start pointer is valid for all line of kernel so it may be calculated once per summing of full kernel line if using SIMD ASM processing.
Addition:
I think there may be 2 significally different approaches for 2D convolution. They give same output result but may be very different in speed on different platforms:

Each output sample got kernel-weighted and input-area covered by kernel_size (filter size/support size) sum.
Each input sample 'casts' (add) kernel weighted by input sample to output buffer.

The 1. needs significant memory-read traffic (to both input buffer and kernel buffer) and produces very small output write traffic to memory (write once - may be uncached). Kernel buffer is read-only and can be easily shared between all cores in multi-core processing. Input memory buffer is also read-only and cached memory image may be shared by many cores proceses neibour input samples. For LUT using I still not understand if it can be used in this approach.
The 2. produces very small read memory traffic for input buffer and produces read_(mul+)add_write traffic for output buffer. If this traffic is good cashed - the actual memory writes depends on CPU memory manager. This approach allows to use LUT for weighting by small number of input variants kernel buffer. But for multi core processing it mostly require each core process far enough input and output memory arrays because for read_(mul+)add_write memory access to output buffer may reqiuire many resources to keep cache coherence between cores. Also this approach allows for easy skip zero input samples processing with simple compare_and_continue. Because zero input sample makes all-zero kernel addition to output buffer and do not changes it.
May it good to test both approaches on todays hardware platforms to compare its processing speed.
As I see from C-resampler subroutine it uses 1. approach.

Can you create pull request?

This is just an ideas to the future and requires a lot of coding (for me) and debugging to attempt to make a version to arbitrary ratio resize as with current JincResize project. I will try to do it later. At first I need to test it at my small separate project with integer ratio resizing (i.e. x2, x3 etc). May be it will be at first separate branch of processing code - i.e. current resampler for any float resize ratio and separate resampler for integer ratio,

Here is current work in progress on additive 2d convolution with Jinc (EWA Lz) kernel available - https://github.com/DTL2020/2D_resampler . I still poor in Avisynth plugin interface for input from Avisynth and output to Avisynth buffers so it uses libtiff and tiff files for input and output. It is only short demo on the additive LUT 2D convolution approach and it do not process edges of buffer correctly. So the test images must be placed closer the center.
It now can use Jinc (EWA Lz) kernel and shows results close to this plugin resizer. Compare is in attach.

Test script:
[code]
Loadplugin("ResampleMT.dll")
LoadPlugin("JincResize.dll")
function Ast2(clip c, int isize)
{
return Subtitle(c, "7",font="Arial",size=isize,x=10,y=20,halo_color=$FF000000, text_color=$00e0e0e0)
}
BlankClip(100,200,180,"RGB24",25,color=$00202020)
Animate(last, 0,100,"Ast2",  45,  180)
AddBorders(150,140,100,104,color=$00202020)
ConvertToYV12()
SinPowResizeMT(last,width/4,height/4,p=2.7)
#return last
#SincLin2ResizeMT(width4,height4,taps=20)
Jinc256Resize(width4,height4)
[/code]

Well - finally some pull-request created. It have still many things to go: 1. Found out why OpenMP of addition loop fails with x64 build. 2. Add ASM summation functions for different SIMD instructions sets. 3. Make kernel half-height for less memory requirement so larger kernels will fit in CPU cache. 4. May be make kernel's lines sort of run-lengh encoded to possibly save about 27% of memory space and ADD operations because about 27% or round kernel are zeroes and not required for keeping in memory and addition. 5. Add downscaling function.

Well - finally some pull-request created. It have still many things to go: 1. Found out why OpenMP of addition loop fails with x64 build. 2. Add ASM summation functions for different SIMD instructions sets. 3. Make kernel half-height for less memory requirement so larger kernels will fit in CPU cache. 4. May be make kernel's lines sort of run-lengh encoded to possibly save about 27% of memory space and ADD operations because about 27% or round kernel are zeroes and not required for keeping in memory and addition. 5. Add downscaling function.

The pull request merged to branch master-1.
I made some changes so now building with OpenMP is possible, the leftover floats (ASM) are processed too.
Quick test with this image - https://i.imgaa.com/2021/01/04/5ff26063d2a7f928883289.png

jincresize(width*10, height*10, tap=10, opt=0, threads=1):

v2.0.1 - 0.707 fps, long initial time, 3 GB RAM;
new - 0.193 fps, 280 MB RAM.

jincresize(width*10, height*10, opt=0, threads=1):

v2.0.1 - 3.47 fps, 260 MB RAM;
new - 5.29 fps, 90 MB RAM.

jincresize(width*10, height*10, tap=10, opt=2, threads=0):

v2.0.1 - 22.45 fps, long initial time, 3 GB RAM;
new - 2.27 fps, 205 MB RAM.

jincresize(width*10, height*10, opt=2, threads=0):

v2.0.1 - 182.65 fps, 260 MB;
new - 52.17 fps, 90 MB.

The pull request merged to branch master-1.
I made some changes so now building with OpenMP is possible, the leftover floats (ASM) are processed too.
Quick test with this image - https://i.imgaa.com/2021/01/04/5ff26063d2a7f928883289.png

jincresize(width*10, height*10, tap=10, opt=0, threads=1):

v2.0.1 - 0.707 fps, long initial time, 3 GB RAM;
new - 0.193 fps, 280 MB RAM.

jincresize(width*10, height*10, opt=0, threads=1):

v2.0.1 - 3.47 fps, 260 MB RAM;
new - 5.29 fps, 90 MB RAM.

jincresize(width*10, height*10, tap=10, opt=2, threads=0):

v2.0.1 - 22.45 fps, long initial time, 3 GB RAM;
new - 2.27 fps, 205 MB RAM.

jincresize(width*10, height*10, opt=2, threads=0):

v2.0.1 - 182.65 fps, 260 MB;
new - 52.17 fps, 90 MB.

It is sad even new 2020 year C compilers looks like have too few AI to good put to ASM so simple addition task. The one line intrinsic looks like compiled into processing in 1 SIMD register only too. So we have about 7 of 8 (or may be 15 of 16 at some platforms) SIMD registers free and unused. I think of accumulating more sums for output row in SIMD registers from neibough input samples before writing to output buffer. The shift between start points of summation = iMul float32 and with small enough practical multiplication values like upsampling HD to UHD = 2 of SD to HD/UHD = 2 to 4 the shift is 8 to 16 bytes.  So we can decrease number of reads and writes to output buffer. It going to work fast enough untill kernel LUT array fits in part of CPU cache. For practical using of upsampler I think typically iMul < iTaps and looks like iMul 2..4 or may be 8 and Taps are 4 to 10 and more. I will try to make test ASM subroutine of row addition for 2 input sample kernels at single processing loop and iMul=2 to look at its performance.

It is sad even new 2020 year C compilers looks like have too few AI to good put to ASM so simple addition task. The one line intrinsic looks like compiled into processing in 1 SIMD register only too. So we have about 7 of 8 (or may be 15 of 16 at some platforms) SIMD registers free and unused. I think of accumulating more sums for output row in SIMD registers from neibough input samples before writing to output buffer. The shift between start points of summation = iMul float32 and with small enough practical multiplication values like upsampling HD to UHD = 2 of SD to HD/UHD = 2 to 4 the shift is 8 to 16 bytes.  So we can decrease number of reads and writes to output buffer. It going to work fast enough untill kernel LUT array fits in part of CPU cache. For practical using of upsampler I think typically iMul < iTaps and looks like iMul 2..4 or may be 8 and Taps are 4 to 10 and more. I will try to make test ASM subroutine of row addition for 2 input sample kernels at single processing loop and iMul=2 to look at its performance.

Some work in progress with working demo with fixed size of kernel - https://github.com/DTL2020/AviSynth-JincResize/tree/master-1 . The c-function also works, but without significant improving. Main is AVX2 version now. It is not finished debug even current realization of 2-samples convolution at one horizontal pass. With upsize of 1920x1080 frame to 2x with taps=8 or 4x with taps=4 it shows performance improves about 50% over old large-ram kernel processing.
Current investigation results: Attempt of decreasing workutins for threads do not works as expected. The fastest is MT at different and far parts of input and output buffers. The reading 256-images LUT works a bit slower in compare with 1 kernel multiplied to input sample at processing time. The most performance boot is increasing number of input samples processing without writing to memory - in the number of SIMD registers. The simple 'unrolling' summation loop with just loading all SIMD registers available adds very few to performance like 5%. But transfer from 1 to 2 input samples processind adds about +50%. The increasing number of input samples gives less addition to performance but still do - transfer from 2 to 3 samples adds about 25..20% performance. Current version shows how to use 2 samples. I think the using of AVX512 registers will also make good addition to performance, but I still have not access to such CPU for checking design and without debug it may still contain errors. The complex convolution work of loading most available SIMD registers and process 3 and more samples in 1 pass is much more complex in compare with 1 line of SIMD intrinsics of previous versions. Attempt to decrease round kernel processing add a bit to C-subroutine but looks like not good for SIMD subroutines because of any intervention in highly-loaded SIMD processing loop greatly decreases performance. This is also hard task for coder because it looks the fastest possible SIMD subroutines may be made only for fixed sizes of iKernelSize. May be approach to process with decreasing workload for each loop will work a bit acceptable. Like at first process workunit of largest loadable to available SIMD registers, later 1/2 and later 1/4 etc. The more we can load in large AVX512 registers and the greater difference with 'last' part of workload being processed now with 'non-SIMD' code. I tried to make 'function-call' looks like inline with different workunits processed by SIMD functions - it is void(JincResize::* AVX2Row)(int64_t k_col_x, float* pfProc, float* pfCurrKernel_pos, float *pfSample); function. But looks it is not inlined but called as function and it greatly degrade performance. So at today only hardcoded ASM into main processing loop works fast enough. Will try to fix AVX2 and other ASM subroutines in the next releases so it can work with different kernel-sizes.

Some work in progress with working demo with fixed size of kernel - https://github.com/DTL2020/AviSynth-JincResize/tree/master-1 . The c-function also works, but without significant improving. Main is AVX2 version now. It is not finished debug even current realization of 2-samples convolution at one horizontal pass. With upsize of 1920x1080 frame to 2x with taps=8 or 4x with taps=4 it shows performance improves about 50% over old large-ram kernel processing.
Current investigation results: Attempt of decreasing workutins for threads do not works as expected. The fastest is MT at different and far parts of input and output buffers. The reading 256-images LUT works a bit slower in compare with 1 kernel multiplied to input sample at processing time. The most performance boot is increasing number of input samples processing without writing to memory - in the number of SIMD registers. The simple 'unrolling' summation loop with just loading all SIMD registers available adds very few to performance like 5%. But transfer from 1 to 2 input samples processind adds about +50%. The increasing number of input samples gives less addition to performance but still do - transfer from 2 to 3 samples adds about 25..20% performance. Current version shows how to use 2 samples. I think the using of AVX512 registers will also make good addition to performance, but I still have not access to such CPU for checking design and without debug it may still contain errors. The complex convolution work of loading most available SIMD registers and process 3 and more samples in 1 pass is much more complex in compare with 1 line of SIMD intrinsics of previous versions. Attempt to decrease round kernel processing add a bit to C-subroutine but looks like not good for SIMD subroutines because of any intervention in highly-loaded SIMD processing loop greatly decreases performance. This is also hard task for coder because it looks the fastest possible SIMD subroutines may be made only for fixed sizes of iKernelSize. May be approach to process with decreasing workload for each loop will work a bit acceptable. Like at first process workunit of largest loadable to available SIMD registers, later 1/2 and later 1/4 etc. The more we can load in large AVX512 registers and the greater difference with 'last' part of workload being processed now with 'non-SIMD' code. I tried to make 'function-call' looks like inline with different workunits processed by SIMD functions - it is void(JincResize::* AVX2Row)(int64_t k_col_x, float* pfProc, float* pfCurrKernel_pos, float *pfSample); function. But looks it is not inlined but called as function and it greatly degrade performance. So at today only hardcoded ASM into main processing loop works fast enough. Will try to fix AVX2 and other ASM subroutines in the next releases so it can work with different kernel-sizes.

I have uploaded some more work in-progress to masrter-1 branch. Tested FMA instruction with memory operand. It looks to get the highest possible Gflops from cpu we need to make FMA only with registers arguments. Because even L1 data access takes about 5 cycles and FMA engines (2 per typical core in the current CPUs) can supply 2x256 bits FMA results each cycle. So in theory typical 3 GHz CPU with 16 fmaops per cycle and 4 cores can provide 192 Gfmaops. For 4x upsize with 4taps (32x32 floats kernel) and 1920x1080 frame size in YV12 we need about 3 Gfmapos. So 190 Gfmaops CPU have to run at about up to 63 fps. With /2 loss on multithreading and /2 loss on other processing - about 15 fps. As for AVX512 I not sure if it can provide 2x512 FMA per cycle - may be limited to 2x256 or 1x512. To make processing in-between registers of long kernels and >1 input samples we need some way of shifting data in a number of SIMD registers while shifting in floats and shifted data must go in and from edges of register. I not found still how to do it fast enough. One easy way was to use memory-operand for any iMul value - but it need to touch L1-cache frequently that looks like lowers performance. The only integer 'shift' with simply register renaming is iMul=8. But I think 8x upsizing is not very practical processing. There is another approach of 'shift' via store-load operation. Because it may be performed less rare in compare with input samples processing it looks adds to performance significally. Also Intel C compiler looks like understands and put some further optimisation to it. Unfortunately MSVC compiler do not. May we can looking at Intel C compiler output move some more intrinsics to C code and make this technique available for MSVC compiler too and may be even remove some intermediate store commands too. In current sources (master-1 branch at fork) the function JincResize::KernelRow_avx2_mul4_taps4 shows this approach. It process 8 input samples at 1 loop pass and process 4 odd samples with 8-step ymm registers renaming and then store(addr)-load(addr+4float) and 4 even samples. It is still not good debugged for correct output math but shows about 4+x speed up in compare with 'large-kernel resampler from resize_plane_avx2.cpp at processing images about 1920x1080 or 960x540 sized (upsize 4x with 4taps). Will try to debug it and make close to pull request later. As highest performance subroutines looks like may be made only for exact combination of upsize ratio and taps I think of simple adding some named resize functions like 'JincUpsize64_4x()' . The using of AVX512 zmm 512bits registers will allow to use larger kernels and/or more samples in 1 pass procesing. Current bad thing - Intel C compiler really builds slower resize_plane_avx2.cpp binary but much more optimized KernelRow_avx2.cpp binary and MSVC vice-versa. So the solution may be to build .lib with different compilers and link.  Though with OpenMP it may cause additional troubles. Also it looks using FMA instead of add+mul instruction cause some bug with OpenMP at the borders of processed parts of buffer - where output result of threads overlaps - will try to looks at it later. It is very strange.

I have uploaded some more work in-progress to masrter-1 branch. Tested FMA instruction with memory operand. It looks to get the highest possible Gflops from cpu we need to make FMA only with registers arguments. Because even L1 data access takes about 5 cycles and FMA engines (2 per typical core in the current CPUs) can supply 2x256 bits FMA results each cycle. So in theory typical 3 GHz CPU with 16 fmaops per cycle and 4 cores can provide 192 Gfmaops. For 4x upsize with 4taps (32x32 floats kernel) and 1920x1080 frame size in YV12 we need about 3 Gfmapos. So 190 Gfmaops CPU have to run at about up to 63 fps. With /2 loss on multithreading and /2 loss on other processing - about 15 fps. As for AVX512 I not sure if it can provide 2x512 FMA per cycle - may be limited to 2x256 or 1x512. To make processing in-between registers of long kernels and >1 input samples we need some way of shifting data in a number of SIMD registers while shifting in floats and shifted data must go in and from edges of register. I not found still how to do it fast enough. One easy way was to use memory-operand for any iMul value - but it need to touch L1-cache frequently that looks like lowers performance. The only integer 'shift' with simply register renaming is iMul=8. But I think 8x upsizing is not very practical processing. There is another approach of 'shift' via store-load operation. Because it may be performed less rare in compare with input samples processing it looks adds to performance significally. Also Intel C compiler looks like understands and put some further optimisation to it. Unfortunately MSVC compiler do not. May we can looking at Intel C compiler output move some more intrinsics to C code and make this technique available for MSVC compiler too and may be even remove some intermediate store commands too. In current sources (master-1 branch at fork) the function JincResize::KernelRow_avx2_mul4_taps4 shows this approach. It process 8 input samples at 1 loop pass and process 4 odd samples with 8-step ymm registers renaming and then store(addr)-load(addr+4float) and 4 even samples. It is still not good debugged for correct output math but shows about 4+x speed up in compare with 'large-kernel resampler from resize_plane_avx2.cpp at processing images about 1920x1080 or 960x540 sized (upsize 4x with 4taps). Will try to debug it and make close to pull request later. As highest performance subroutines looks like may be made only for exact combination of upsize ratio and taps I think of simple adding some named resize functions like 'JincUpsize64_4x()' . The using of AVX512 zmm 512bits registers will allow to use larger kernels and/or more samples in 1 pass procesing. Current bad thing - Intel C compiler really builds slower resize_plane_avx2.cpp binary but much more optimized KernelRow_avx2.cpp binary and MSVC vice-versa. So the solution may be to build .lib with different compilers and link.  Though with OpenMP it may cause additional troubles. Also it looks using FMA instead of add+mul instruction cause some bug with OpenMP at the borders of processed parts of buffer - where output result of threads overlaps - will try to looks at it later. It is very strange.

Finally some new pull-request available. It looks like good compiled with MSVS2019 and produces close speed binary with Intel C Compiler. It have one main subroutine for mul=4 taps=4 more optimized AVX2+FMA processing. At tests with 960x540 and 1920x1080 YV12 inputs it shows about 2 to 4 times faster processing in compare with old large-kernel way. Unfortunately AVX2 instructions do not provide easy way to add mul=2,3,5, and other integer shift between ymm registers fast enough. Or it need to be tested in the future. The much better is AVX512 instructions even in most common AVX512F set and many more better performing combinations of mul+taps may be made with AVX512 in the future. I do not have such CPU for now but will try to upgrade. Next versions I will try to add mul=2 version with SSE instructions becase shufps allow to exchange between halfs of different registers. As testing shows with high mul value like 8 and higher it works slower in compare with old large-kernel variant. So it good to try to make new subroutines for small enough mul values like 2 to may be 6. May be AVX512 will give better performance in the future.

Finally some new pull-request available. It looks like good compiled with MSVS2019 and produces close speed binary with Intel C Compiler. It have one main subroutine for mul=4 taps=4 more optimized AVX2+FMA processing. At tests with 960x540 and 1920x1080 YV12 inputs it shows about 2 to 4 times faster processing in compare with old large-kernel way. Unfortunately AVX2 instructions do not provide easy way to add mul=2,3,5, and other integer shift between ymm registers fast enough. Or it need to be tested in the future. The much better is AVX512 instructions even in most common AVX512F set and many more better performing combinations of mul+taps may be made with AVX512 in the future. I do not have such CPU for now but will try to upgrade. Next versions I will try to add mul=2 version with SSE instructions becase shufps allow to exchange between halfs of different registers. As testing shows with high mul value like 8 and higher it works slower in compare with old large-kernel variant. So it good to try to make new subroutines for small enough mul values like 2 to may be 6. May be AVX512 will give better performance in the future.

Some more strategic idea: Currently we have some tactic speed increases but come close to the limit of FMA ops by host CPU per second. The currently required FMA ops is about (iMuliTaps2)^2 per each input sample. It may be decreased a bit (like 10% in SIMD to may be about 30% in C-code with current optimization of kernel_row_useful_range table) skipping outside zeroes around round kernel inside square buffer but it is not very significant.
When I look at current full row-walking processing it seems like applying FIR filter in 1D space. With size of kernel iTapsiMul2. But there may be possible to calculate equal or acceptably closer looking IIR filter kernel for the same or acceptably same output result. But IIR filter kernel will have less number of coefficients so it may will require less FMA ops to compute. At least for rows 1D processing. So required number of FMA ops to compute full 2D resize will be decreased to about (iMuliTaps2C_IIR), where C_IIR is fixed number of IIR filter coefficients for rows processing and C_IIR < iTapsiMul*2. May be it also possible to perform 2D/planar IIR filtering/processing with further reducing of FMA ops required per each input sample instead of current 2D FIR processing, but it may be slow performance because of image scanning placement in memory. Will try to start test this idea from calculating IIR filter coefficients from current kernel rows and to look if errors will be with small enough number of coefficients.

Some more strategic idea: Currently we have some tactic speed increases but come close to the limit of FMA ops by host CPU per second. The currently required FMA ops is about (iMuliTaps2)^2 per each input sample. It may be decreased a bit (like 10% in SIMD to may be about 30% in C-code with current optimization of kernel_row_useful_range table) skipping outside zeroes around round kernel inside square buffer but it is not very significant.
When I look at current full row-walking processing it seems like applying FIR filter in 1D space. With size of kernel iTapsiMul2. But there may be possible to calculate equal or acceptably closer looking IIR filter kernel for the same or acceptably same output result. But IIR filter kernel will have less number of coefficients so it may will require less FMA ops to compute. At least for rows 1D processing. So required number of FMA ops to compute full 2D resize will be decreased to about (iMuliTaps2C_IIR), where C_IIR is fixed number of IIR filter coefficients for rows processing and C_IIR < iTapsiMul*2. May be it also possible to perform 2D/planar IIR filtering/processing with further reducing of FMA ops required per each input sample instead of current 2D FIR processing, but it may be slow performance because of image scanning placement in memory. Will try to start test this idea from calculating IIR filter coefficients from current kernel rows and to look if errors will be with small enough number of coefficients.

Uploaded some supplementary repository https://github.com/DTL2020/FIR_to_IIR . Unfortunately with number of IIR coefficients about 1/2 of FIR the quality of 'non-covered' by IIR samples inpulse response is still poor and error is at some samples significally > 100%. And if attempt to use more coefficients - the used solver fails for unknown reason. May be because of limited precision of float calculations in solver though it uses double-pr. May be need to use another way of converting FIR to IIR or may be add some refining algorithm with acceptable speed for refining coefficients of IIR impulse response at the required edges. Anyway will try to do version of resampler with IIR processing to look at actual image processing results even with current level of errors from required kernel samples. And to test its speed.

Uploaded some supplementary repository https://github.com/DTL2020/FIR_to_IIR . Unfortunately with number of IIR coefficients about 1/2 of FIR the quality of 'non-covered' by IIR samples inpulse response is still poor and error is at some samples significally > 100%. And if attempt to use more coefficients - the used solver fails for unknown reason. May be because of limited precision of float calculations in solver though it uses double-pr. May be need to use another way of converting FIR to IIR or may be add some refining algorithm with acceptable speed for refining coefficients of IIR impulse response at the required edges. Anyway will try to do version of resampler with IIR processing to look at actual image processing results even with current level of errors from required kernel samples. And to test its speed.

Strategic announce: Commit to https://github.com/DTL2020/2D_resampler a version of resampler with significally decreased memory use. It now use only Output_width_x_iKernelSize temp buf (per each thread) so the whole processing will fit into CPU cache  at least L2/L3. This approach may have some difficulties for multi-threading processing at the edge rows for each thread - but I think it may be easy enough workarounded. Will try to add to Avisynth master-1 branch when I will have access to its build enviroment a few days later. With SIMD versions I will test for non-cached load and non-cached store of input and output data if it will add more speed because of preventing cache pollution of read-once and write-once data.
Also this version uses output of finally processed rows from main processing loop - so the additional pass to main memory for conversion from float to output uint8 also removed.

Strategic announce: Commit to https://github.com/DTL2020/2D_resampler a version of resampler with significally decreased memory use. It now use only Output_width_x_iKernelSize temp buf (per each thread) so the whole processing will fit into CPU cache  at least L2/L3. This approach may have some difficulties for multi-threading processing at the edge rows for each thread - but I think it may be easy enough workarounded. Will try to add to Avisynth master-1 branch when I will have access to its build enviroment a few days later. With SIMD versions I will test for non-cached load and non-cached store of input and output data if it will add more speed because of preventing cache pollution of read-once and write-once data.
Also this version uses output of finally processed rows from main processing loop - so the additional pass to main memory for conversion from float to output uint8 also removed.

Well - finally some milesone in performance: Function KernelRowAll_avx2_mul4_taps4_cb running as single thread and with multithreading frame-based in Avisynth+ reach about 33% of theoretical G-FMAs performance of CPU like i3-9100T and i5-9600K. It allow to upsize FullHD 1920x1080 YV12 frame to 8K with about 45fps at i5-9600K running at 4.3 GHz in 6 threads. The most performance boost looks like reached because of smaller memory using with 'circulating buf' of size only out_width_x_KernelSize instead of full output frame size in floats. With C-code upsample using of circulating buf makes very small difference may be because of too slow CPU processing without good AVX FMA engine. I think of makiing multithreaded of the '_cb' functions in the future - I still poor in C++ so need to found how to make array of vectors for each thread. Also some work need for aligning edges of processed frame stripes by each thread. Also some more ideas for future optimizations may add to performance. I made some attempt to release binaries at https://github.com/DTL2020/AviSynth-JincResize/releases/tag/v0.3-alpha for testing.

Some note: Looks like found way of faster enough shift float32 with a sequence of SIMD registers on any number of floats. To implement multi-input-samples per pass convolution for multiplication with 2,3,5 ratio etc. For AVX2 it is sequence of 2 instructions - _mm256_permutevar_8x32_ps for rotating floats inside ymm for required number of floats and _m256_blend_ps for transfer shifted-out floats to another ymm. And for AVX512F+AVX512VL there is one instruction for permute any exact float32s between 2 zmm registers - _m512_permutex2var_ps . I hope there will be many 'client' CPUs with AVX512F+AVX512VL instruction set available soon. It looks we need to add more 'opt' parameter values to separate opt=3 AVX512F from AVX512F+AVX512VL CPU features required. May be opt=4 or 5.

I did quick speed test with 1920x1080 YV12 clip, CPU 7900x, VM with 16 threads + 16 GB RAM:

JincResize(7680, 4320, tap=4, opt=1, ap=1):

Frames processed:                   120 (0 - 119)
FPS (min | max | average):          5.454 | 12.46 | 11.90
Process memory usage (max):         394 MiB
Thread count:                       66
CPU usage (average):                70.0%

JincResize(7680, 4320, tap=4, opt=1, ap=0):

Frames processed:                   101 (0 - 100)
FPS (min | max | average):          6.673 | 10.52 | 9.977
Process memory usage (max):         1245 MiB
Thread count:                       66
CPU usage (average):                83.5%

JincResize(7680, 4320, tap=4, opt=1, ap=2) + prefetch(8):

Frames processed:                   121 (0 - 120)
FPS (min | max | average):          1.108 | 263158 | 12.10
Process memory usage (max):         1475 MiB
Thread count:                       44
CPU usage (average):                51.9%

The test was done with -timelimit=10 (every test run lasts 10s).
Every ap=x was tested twice and the run with the higher fps was picked.
opt=1 was used because opt=2/3 + ap=1 gives wrong output.
Here the same frame with different opt/ap options (top left subtitle):
https://0x0.st/-H7U.png
https://0x0.st/-H7G.png
https://0x0.st/-H7D.png
https://0x0.st/-H7k.png
https://0x0.st/-H7d.png
https://0x0.st/-H7n.png

Heh - the message for e-mail was for unknown reason sorted to spam-box by gmail.com. The currently fastest functions works only with AVX2 for mul4 and mul2 (in latest commits to master-1 branch). Also I usually use multipliers in arguments for less probability of error in width and height calculations.
So the better test strings for testing currently fastest versions is JincResize(width2, height2, opt=2, ap=2) and JincResize(width4, height4, opt=2, ap=2) . I not test with exact 7680, 4320 numbers as input. Yes - the using of ap=1 gives distortions at some combinations of other args - need to fix this bug or if ap=2 being converted to internal multithreading gives every time better results in speed so ap=1 will be removed from future versions.
As a developer it better to use fresh builds of maser-1 branch and test via debugger if code really uses the currently fastest functions KernelRowAll_avx2_mul4_taps4_cb() for iMul=4 and iTaps=4 and KernelRowAll_avx2_mul2_taps4_cb() for iMul=2 and iTaps=4. The start of resizer in JincResize::JincResize() have large selector-code for use many of designed resizers so it may fails to select proper function if some bug in detection/selection logic still exist.
Also it looks auto-multithreading by OpenMP sometime fails with correct processing of rows stripes and gives errors (green lines) at the start or end of stripe. So it is better to make hand-writed buffers managment for multithreading. Also using OpenMP ofcourse for smaller codesize. Simply use #pragma parallel (num_threads.. etc) {} on block of parallel code instead of using parallel for {} with auto-management of loop variable. And instead of parallel by loop-variable use omp_get_thread_num() to get thread_id and make hand-writing striping of processed buffer. It anyway required for the smallest memory using _cb functions because inability of auto-aligning of striped edges. Will try to do it later a bit.

Finally added internal multithreading to _cb family of functions. It actually uses a bit more FMA for correct aligning of rows stripes in compare with 1 threaded so it is currenlty if threads=1 used old 1-threaded and if threads > 1 used new multithreaded versions. AVX512 build also available using intel C compiler. I add new build release at https://github.com/DTL2020/AviSynth-JincResize/releases/tag/v0.4-alpha . I made some tests at different CPUs with ImageReader source and ConvertToYV12() before upsize. Also last tests with faster source - BlankClip of 1920x1080. Looks ImageReader and ConvertToTV12() running with 1 thread avisynth+ is too slow for even 1920x1080 source feeding. Results file attached.
res.txt

It currently uses vector of vectors for storing and rotating memory pointers for rows temp buf but it looks uses vectors copy at function calls. If someone will help to rewrite for using pointers to vectors instead of vectors copying I think it will be a bit faster. I still too poor in C++ to convert it to vectors pointers at JincResize::ConvertiMulRowsToInt_avx2(std::vector<float*>Vector, int iInpWidth, int iOutStartRow, unsigned char* dst, int iDstStride) call.

You're probably looking for JincResize::ConvertiMulRowsToInt_avx2(std::vector<float*>const& Vector, int iInpWidth, int iOutStartRow, unsigned char* dst, int iDstStride).

Also I not sure if it needed or not delete internal vectors defined (created as new) at https://github.com/DTL2020/AviSynth-JincResize/blob/19c636dfefe7971722c99beefd54ddb21fd0c67e/src/JincResize.cpp#L691 at class delete at https://github.com/DTL2020/AviSynth-JincResize/blob/19c636dfefe7971722c99beefd54ddb21fd0c67e/src/JincResize.cpp#L1499 (if uncomented it creates error on exit somhere in vector class). Currently it may cause some memory leak if not deleted correctly.

That vector ( https://github.com/DTL2020/AviSynth-JincResize/blob/19c636dfefe7971722c99beefd54ddb21fd0c67e/src/JincResize.cpp#L691 ) should be deleted after https://github.com/DTL2020/AviSynth-JincResize/blob/19c636dfefe7971722c99beefd54ddb21fd0c67e/src/JincResize.cpp#L698 in the same scope.
Does this variant have performance penalty:
std::vector<float*> ThreadRowPointers;
ThreadRowPointers.reserve(iKernelSize);

for (int i = 0; i < iKernelSize; i++)
{
    ThreadRowPointers.push_back((float*)_mm_malloc(iWidthEl * (iMul + 1)  * sizeof(float), 32)); // +1 looks like buf overrun somwhere
}

vpvThreadsVectors.push_back(ThreadRowPointers);

Thank you. It looks to work ok without visible performance penalty. I found some wierd feature - as well as intel C compiler suggests to put most used 'global' class members 'iMul/iTaps/iKernelSize' to the beginning because it improves data locality - I found making local copy of these constants for processing or even replacing to hardcoded constants significally improves performance (like about +20%). So I started long process of re-write most or all of KernelRow* functions to have these control values function arguments (where needed). So it will take some time for next updates. Also found the way of improving balance of 'scatter/gather' processing - as it done with 'horizontal multi-samples' processing it may partially be done with 'multi-rows vertical' data gathering without additional memory read-write transactions. Though it significally increases code size and complexity. I currently made some _r2 functions (2 rows per 1 loop pass processing and per registers load-store operation). It adds about +20% performance. So may be will try to make _r4 (4 rows) later to see if it will add some more. Already it reach about 50% of G-FMA performance or 4 core FMA2x256 CPU. So it unlikely can reach many more. Currently unfinished debug for aligning multithreaded versions of _r2 functions uploaded to my fork repository. Single threaded works.


admin: Define issue triaging process

We need to define key players/stakeholders on triaging issues of different types and what is the overall process of triaging to be in async contact/update and create a transparent place for every community member

Grooming 12/07/2022:

Moving to mainnet as we need to control(have a process)the stages of the bug dashboard
If Incentivize overflow us with bugs, then move this one more to it

Grooming 01/11/2022:
@MSevey Can we also discuss across teams how we tackle triaging

Grooming 01/11/2022:
@MSevey Can we also discuss across teams how we tackle triaging

yes, @YazzyYaz should be involved in this discussion as well. And moving forward what our general triage process is for inbound community issues.


[Task]: Add fiat price data for shimmer

Task description
When desktop merges the fiat price, we can include it in our balance also, tasks.
Depends on https://github.com/iotaledger/firefly/issues/4817
Requirements

[ ] Add fiat conversion to asset / balance of asset
[ ] Add polling of coin gecko api for shimmer balance
[ ] Add attribution to currency selector

Creation checklist

[ ] I have assigned this task to the correct people
[X] I have added the most appropriate labels
[X] I have linked the correct milestone and/or project

fiat is already included


KAFKA-12648: fix IllegalStateException in ClientState after removing topologies

Fix for one of the causes of failure in the NamedTopologyIntegrationTest: org.apache.kafka.streams.errors.StreamsException: java.lang.IllegalStateException: Must initialize prevActiveTasks from ownedPartitions before initializing remaining tasks.
This exception could occur if a member sent in a subscription where all of its ownedPartitions were from a named topology that is no longer recognized by the group leader, eg because it was just removed from the client. We should filter each ClientState based on the current topology only so the assignor only processes the partitions/tasks it can identify. The member with the out-of-date tasks will eventually clean them up when the #removeNamedTopology API is invoked on them

The check before was essentially about two things, (a) sanity check that the ClientState methods are executed in the correct order since the task initialization ended up becoming spread out over the assignment method, and (b) verifying that the application did not try to do the exact thing that the NamedTopology feature is intended to enable, ie that the user did't try to modify the topology in an incompatible way. The check in (a) is still preserved, while the check in (b) isn't valid for NamedTopologies

All tests failures unrelated: ReplicaManagerTest.[1] usesTopicIds=true

Two unrelated test failures in .LeaderElectionCommandTest.[1] Type=Raft, Name=testAllTopicPartition, Security=PLAINTEXT, one failure in related NamedTopologyIntegrationTest.shouldRemoveNamedTopologyToRunningApplicationWithMultipleNodesAndResetsOffsets but this is only part 1 of the upcoming fixes for that test and the failure in this build is from a different root cause.
Merging


Cannot install binary using haste-cabal

I cannot install text because its dependency binary fails to install with the following message
Resolving dependencies...
In order, the following will be installed:
binary-<IP_ADDRESS> (reinstall) (changes: array-<IP_ADDRESS> added, base-<IP_ADDRESS> added, bytestring-<IP_ADDRESS> added, containers-<IP_ADDRESS> -> <IP_ADDRESS>)
Warning: Note that reinstalls are always dangerous. Continuing anyway...
Configuring binary-<IP_ADDRESS>...
Building binary-<IP_ADDRESS>...
Failed to install binary-<IP_ADDRESS>
Build log ( /home/kranich/.cabal/logs/binary-<IP_ADDRESS>.log ):
Configuring binary-<IP_ADDRESS>...
Building binary-<IP_ADDRESS>...
Preprocessing library binary-<IP_ADDRESS>...

src/Data/Binary/Builder/Base.hs:192:37: Warning:
    In the use of ‘inlinePerformIO’
    (imported from Data.ByteString.Internal):
    Deprecated: "If you think you know what you are doing, use 'unsafePerformIO'. If you are sure you know what you are doing, use 'unsafeDupablePerformIO'. If you enjoy sharing an address space with a malevolent agent of chaos, try 'accursedUnutterablePerformIO'."

src/Data/Binary/Get/Internal.hs:402:21: Warning:
    In the use of ‘B.inlinePerformIO’
    (imported from Data.ByteString.Internal):
    Deprecated: "If you think you know what you are doing, use 'unsafePerformIO'. If you are sure you know what you are doing, use 'unsafeDupablePerformIO'. If you enjoy sharing an address space with a malevolent agent of chaos, try 'accursedUnutterablePerformIO'."

src/Data/Binary/Get/Internal.hs:369:1: Warning:
    Rule "<$> to <*>" may never fire because ‘<$>’ might inline first
    Probable fix: add an INLINE[n] or NOINLINE[n] pragma on ‘<$>’

src/Data/Binary/Get.hs:434:1: Warning:
    Rule "getWord8/readN" may never fire
      because ‘getWord8’ might inline first
    Probable fix: add an INLINE[n] or NOINLINE[n] pragma on ‘getWord8’

src/Data/Binary/Get.hs:435:1: Warning:
    Rule "getWord16be/readN" may never fire
      because ‘getWord16be’ might inline first
    Probable fix: add an INLINE[n] or NOINLINE[n] pragma on ‘getWord16be’

src/Data/Binary/Get.hs:436:1: Warning:
    Rule "getWord16le/readN" may never fire
      because ‘getWord16le’ might inline first
    Probable fix: add an INLINE[n] or NOINLINE[n] pragma on ‘getWord16le’

src/Data/Binary/Get.hs:437:1: Warning:
    Rule "getWord32be/readN" may never fire
      because ‘getWord32be’ might inline first
    Probable fix: add an INLINE[n] or NOINLINE[n] pragma on ‘getWord32be’

src/Data/Binary/Get.hs:438:1: Warning:
    Rule "getWord32le/readN" may never fire
      because ‘getWord32le’ might inline first
    Probable fix: add an INLINE[n] or NOINLINE[n] pragma on ‘getWord32le’

src/Data/Binary/Get.hs:439:1: Warning:
    Rule "getWord64be/readN" may never fire
      because ‘getWord64be’ might inline first
    Probable fix: add an INLINE[n] or NOINLINE[n] pragma on ‘getWord64be’

src/Data/Binary/Get.hs:440:1: Warning:
    Rule "getWord64le/readN" may never fire
      because ‘getWord64le’ might inline first
    Probable fix: add an INLINE[n] or NOINLINE[n] pragma on ‘getWord64le’
hastec: hastec: panic! (the 'impossible' happened)
  (GHC version 7.8.3 for x86_64-unknown-linux):
	cat_evals
<<details unavailable>>

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

cabal: Error: some packages failed to install:
binary-<IP_ADDRESS> failed during the building phase. The exception was:
ExitFailure 1

#316 appears to be related to this issue.

binary-<IP_ADDRESS> is the latest version that can currently be installed, binary-<IP_ADDRESS> is the first failing version.

This is a problem for me also: for my current project I want to use uuid, which depends on binary.

Fixed on master with GHC 7.10.


Error when running GetStatistics for Shares with size over 2GB

Which version of the SDK was used?
v0.8.0
Which platform are you using? (ex: Windows, Linux, Debian)
Windows, Linux
What problem was encountered?
ShareURL.GetStatistics(ctx) will fail to unmarshal if share size > 2,147,483,647 bytes returning an error instead.
How can we reproduce the problem in the simplest way?
I have written out some code that will reproduce the issue, see here. (Note to run this make sure to fill out values for variables at the start of function main)
Have you found a mitigation/solution?
The issue stems from the structure which holds ShareStates using int32 instead of int64. Fixing the issue amounts to changing this value's type to int64 in zz_generated_models.go:2958 as follows:
// ShareStats - Stats for the share.
type ShareStats struct {
	rawResponse *http.Response
	// ShareUsageBytes - The approximate size of the data stored in bytes. Note that this value may not include all recently created or recently resized files.
	ShareUsageBytes int64 `xml:"ShareUsageBytes"`
}

I assume to actually fix this this of course amounts to a swagger change and regenerating the sdk. The swagger change should just specify the correct integer format. Not sure how these swaggers are generated so just posting what the change should probably look like here :
           "properties": {
                "ShareUsageBytes": {
                    "description": "The approximate size of the data stored in bytes. Note that this value may not include all recently created or recently resized files.",
                    "type": "integer",
                    "format": "int64"
                }
            }

Looks like a dupe of https://github.com/Azure/azure-storage-file-go/issues/28
@adreed-msft @mohsha-msft


Huge variety in specifying functions in method signatures

Here's the results of my survey of the docs. This is how the docs specify that a term accepts a function in the signature spec at the top:

Term
says it accepts a function with

eq_join, do
function

filter, inner_join, outer_join, offsets_of
predicate

map, concatMap
mapping_function

update, replace
`object

merge
`object

contains
`value

count
value_or_predicate

reduce
reduction_function

for_each
write_query

group, min, max, sum
field_or_function

order_by
no mention it accepts a function in the spec, only in examples

http
no mention in spec, page optarg accepts function

I suggest we use the following specs:

Term
calls it:

update, replace, merge, group, min, max, sum, order_by, concatMap
function/1

for_each
write_function/1

filter, eq_join, inner_join, outer_join, offsets_of, contains, count
predicate_function/1

do, map
function/1+

reduce
function/2

http
should specify function/1 in the page optarg description

I'm going with erlang-style arity declarations since that seems about right for a dynamic language. It would become too verbose I think doing full type declarations for the functions (predicate_function and write_function are a nod in this direction without getting too far into the weeds).
@mlucy @danielmewes thoughts?

While I like the simplification, I'm not sure I like the Erlang arity bit; it's a fairly unusual notation we'd need to explain, and nearly all of our cases only take one argument. I'd consider just going with function, write_function and predicate_function, and specifically say "one or more arguments" or "two arguments" in text when necessary.

That seems fine too


Finalize Models to minimze requiring Migrations

With more users onboarding soon, we should aim to minimize the need to migrate. Since transactions model is pending completion in its refactoring, we should proritize that and reviewing the others to prevent any major migration requirements early on.

Closing this.
We will open a new issue to address migrations and a strategy for it.


How to get an element by a random number?

Hi,
How can I get an element by a random number?
Update: I think the nth method is it, right?
Thanks!

Hi,
Yes if you want an element at a random index in the hash-map/set, generate a number between [0;size) and use nth.


Add banner on landing page linking to idseq public site

Description
Adds a banner at the top of the page linking to https://public.idseq.net/ to redirect people who may be looking for our response to the nCov-2019 virus.

I'll point out that the current video bulletin is still active, so it'll look a little different

Going to do this as a "release fix" instead


[AutoRelease] t2-security-2022-09-13-.3023(Do not merge)

https://github.com/Azure/sdk-release-request/issues/3155
Live test success
https://dev.azure.com/azure-sdk/internal/_build?definitionId=1028

issue link:https://github.com/Azure/sdk-release-request/issues/3155

/azp run

/azp run python - securityinsight


Java Runtime version 55.0 is not compatible with Android SDK (class file version 61.0)

The environment building of make build_toolbox hits Java Runtime and Android SDK compatibility issue as the error:
yes y | /opt/android/sdk/cmdline-tools/latest/bin/sdkmanager "platform-tools"
Error: LinkageError occurred while loading main class com.android.sdklib.tool.sdkmanager.SdkManagerCli
	java.lang.UnsupportedClassVersionError: com/android/sdklib/tool/sdkmanager/SdkManagerCli has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0

This error makes CI build failed https://ci.endlessos.org/job/kolibri-installer-android-pr/484/console

According to SDK Build Tools release notes, I guess because of supporting Android 14 preview APIs (since Revision 34.0.0)?  If so, then according to Which Java APIs can I use in my Java or Kotlin source code?, Android 14 (API 34) should go with Java 17.
And, we are not alone Class file has wrong version 61.0, should be 55.0

sdkmanager is actually in the cmdline-tools, and sadly they seem to not be doing release notes anymore. Locally I upgraded from 9.0 to 11.0 and was able to reproduce this bug. The git history is here, although it's mostly not useful.

Thanks!  Verified with https://github.com/endlessm/kolibri-installer-android/pull/170#issuecomment-1667239382


Registry tool: should accept user-specified limits on parallelism

Currently the registry tool performs several bulk operations in parallel, with a default setting of 64 concurrent goroutines. This seemed like a good default for workstation environments, but in lighter-weight environments (certain Kubernetes deployments), this can be too much. It's also occasionally useful to limit operations to a single goroutine for observation.
We should be able to support a user-specified limit on parallelism with a top-level option, either at the level of the registry tool or at some other appropriate level in the command hierarchy. This could continue to default to 64 but allow users to set arbitrary values.

Recently this was partially addressed in https://github.com/apigee/registry/pull/406, which adds a --jobs flag to bulk uploads commands. In that update, it seemed better to address this on a command-by-command basis than to add a global flag that would be irrelevant to many subcommands. We expect to continue to add --jobs flag as needed to other subcommands that execute tasks concurrently.


output a single FeatureCollection (and trap auth errors)

resolves #2

group output features in a single FeatureCollection
return a more informative error when an invalid token is provided.

what's thrown currently for access_token=foo is pretty cryptic.
/Users/john6251/github/vt2geojson/node_modules/pbf/index.js:195
        else throw new Error('Unimplemented type: ' + type);
             ^

Error: Unimplemented type: 3

switch to single quotes and tweak output geometry by 1/1,000,000th of a gnat's eyelash to get tests passing in Node 6.

Thanks @jgravois - looks great.


Add Automatic-Module-Name in manifest so Java9 modular applications c…

…an depend on this library
This is essentially a duplication of https://github.com/apache/httpcomponents-core/pull/65 but now for the 5.x versions.
As indicated here https://github.com/apache/httpcomponents-client/pull/104#issuecomment-403436258 I have made sure the module names to not contain a hyphen ('-').
Looking at what was done for the 4.x version I have chosen to stick as close as possible to essentially groupdId.artifactId as possible.
Please indicate if this is the right choice or if you want something different.
I see this as a very important choice to get right at this point as it is something I expect will be hard to change in the future.
I added this to ALL maven artifacts (also the -testing), please asses if this is correct.
Please commit this into both master and the 5.1.x branch if you agree with these changes.

Note that I have created the same change for both httpcomponents-core and httpcomponents-client.

Folks, any objection to the proposed changes?

Isn't there a way to do this once? Say in the parent POM and each module POM can define a property used from the parent.

Same opinion here as for httpclient5.

Isn't there a way to do this once? Say in the parent POM and each module POM can define a property used from the parent.

@nielsbasjes Is there a chance to collapse multiple plugin declarations in module POMs into a single one in the top project POM?

@ok2c Perhaps this can be done. I'll experiment.

@ok2c I was able to simplify the code change as you suggested. I also made the same change for the other PR.

@michael-o @garydgregory If I hear no objections I will merge this PR.


[BUG] Gemini Model fails to invoke any tools.

Is there an existing issue for this?

[X] I have searched the existing issues

Vercel Runtime Logs

[ ] I have checked the Vercel Runtime Logs for errors (if applicable)

Current Behavior
I switched from OpenAI to Google Gemini Flash 1.5. It now directly provides answers using the LLM. However, after two hours, I noticed the 'toolChoice: required' setting in the Vercel AI SDK. As a result, the output continuously triggered search cards without stopping the process or generating responses based on the search results.
Here is some snaps:

Expected Behavior
It should function the same way as OpenAI’s GPT-4 version.
Steps To Reproduce

I just added the Generative API key to the .env file.
After that, I received answers directly from the LLM without any tools being executed.
Later, I found the toolChoice: 'required' option and added it to the researcher.tsx file.

Environment
- OS:Mac
- Browser:Chrome

Anything else?
"@ai-sdk/google": "0.0.23",
"@google/generative-ai": "^0.3.1",
"ai": "^3.2.10",

This error is an issue with the Gemini model. If toolChoice is set to auto (default), the model decides whether to call the tool. If toolChoice is set to required, the model will call the tool as needed. This also depends on the model.
In the latest code, maxSteps is set to 5, so the tool should not continue to be called: https://github.com/miurla/morphic/blob/main/lib/agents/researcher.tsx#L30
https://sdk.vercel.ai/docs/ai-sdk-core/tools-and-tool-calling#multi-step-calls
Please try with the latest code.
https://github.com/miurla/morphic/blob/main/package.json#L15

it's still the same error. After I updated the packages

Still Don't call any tools:

it's still the same error. After I updated the packages

No, I explained that in the latest version, the issue of continuously calling tools has been resolved.

I found that there are limitations in the schema of the Google Generative API: https://sdk.vercel.ai/providers/ai-sdk-providers/google-generative-ai#troubleshooting-schema-limitations
PR: #358
It started calling tools. The model I confirmed was gemini-1.5-pro-002. gemini-1.5-flash-002 did not call tools. This should be dependent on the model.

Thanks Let me check.

Followed these steps:
git clone https://github.com/miurla/morphic
cd morphic
bun install
nano .env added GOOGLE_GENERATIVE_AI_API_KEY and other necessary environment variables
bun run dev

Did you tried some query?  The model decides whether to call the tool

How can we make forceable to call tool everytime before generating the answer?


Android Application Crash when using Camera with NativeScript 6

After taking photo and clicking accept in the preview screen then android application crash or restarted. Also during build process there is a warn about deprecated methods in NativeScript 6
Camera_PreviewCallback.java uses or overrides a deprecated API
To Reproduce
-Upgrade to NativeScript 6

Try to pick a photo using the camera on Android

Expected behavior
Pick selected after accepting preview
NativeScript Version:
6.0.3

Didn't follow the template

I don't think this plugin have ported to NS6.0 just yet so that might be why it is throwing an error like that.


Add suds installation for accessing some sites

This will install suds during installation so that data queries to radioreference.com will no longer fail.

Just a ping to see if anyone is out there.


feat: Update rector recipe

add Symfony 6.4 setlist, php 8.3 setlist, remove NameImportingPostRector since rector::rule method expects a RectorInterface

Thanks


[AutoPR resources/resource-manager] Add list_by_scope

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/5757

This PR has been merged into https://github.com/Azure/azure-sdk-for-go/pull/4724


generic: transfer: check if links are empty

Otherwise we might run into IndexError: list index out of range if none link types are supported and we use on_error.

The error is a separate topic from https://github.com/iterative/dvc-objects/pull/224


Add jq tool

Just a reminder / FYI, you need to merge this, I can't. And 'jq brings back some memories :)


circular imports

currently we're getting lucky that things work nicely despite abusing imports. for example, when trying to use MonkeyType:
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/user/code/capa-pub/capa/features/freeze.py", line 58, in <module>
    import capa.features.insn
  File "/home/user/code/capa-pub/capa/features/insn.py", line 9, in <module>
    import capa.render.utils
  File "/home/user/code/capa-pub/capa/render/__init__.py", line 11, in <module>
    import capa.rules
  File "/home/user/code/capa-pub/capa/rules.py", line 88, in <module>
    capa.features.insn.API,
AttributeError: module 'capa.features' has no attribute 'insn'

I get a similar error in VS Code from pylance:

and can reproduce an import issue locally:

in summary, we have a web of code and dependencies spread among __init__.py files and *.py files. some of the *.py files import from their parents through their dependencies. things somehow work, but it doesn't seem correct, and is preventing me from using MonkeyType easily.
i propose that we refactor our use of __init__.py to keep those bare and move code into shared/sub modules. unfortunately, this will be a breaking change, as the location of some of these functions/classes changes. so, we should assess if we can/want to do this before 2.0 and get it in.

here are the offending files:
> fd __init__.py | while read -r LINE; do stat -c "%s %n" "$LINE"; done                                                  feature-447!?
0       capa/__init__.py
9177    capa/features/__init__.py
11467   capa/features/extractors/__init__.py
3549    capa/features/extractors/ida/__init__.py
5878    capa/features/extractors/pefile/__init__.py
1706    capa/features/extractors/smda/__init__.py
2795    capa/features/extractors/viv/__init__.py
0       capa/ida/__init__.py
3467    capa/ida/helpers/__init__.py
4084    capa/ida/plugin/__init__.py
11708   capa/render/__init__.py

proposed changes:
capa/__init__.py: none
its already empty.
capa/features/__init__.py

move constants and functions to capa.features.common
move classes to capa.features.common, too

capa/features/extractors/__init__.py
move logic to capa/features/extractors/common.py
or maybe pick a better name, like feature_extractor.py?
capa/features/extractors/ida/__init__.py
move logic to capa/features/extractors/ida/extractor.py
capa/features/extractors/pefile/__init__.py
rename to capa/features/extractors/pefile.py
capa/features/extractors/smda/__init__.py
move logic to capa/features/extractors/smda/extractor.py
capa/features/extractors/viv/__init__.py
move logic to capa/features/extractors/viv/extractor.py
capa/ida/__init__.py: none
its already empty
capa/ida/helpers/__init__.py
rename to capa/ida/plugin/helpers.py
capa/ida/plugin/__init__.py
im not sure if maybe IDA relies on this location and we cannot change it
capa/render/__init__.py

move "convert" logic to capa.render.result_document or similar
move render_vverbose to capa.render.vverbose.render, also verbose, default, json

the circular import was an issue of:
features.insn -> common.Feature -> engine.Result -> rules.index_rules_by_namespace -> features.file -> common.Feature

fixed in https://github.com/fireeye/capa/pull/624/commits/527e993bb446686fefb2fb528017ae13710b5dc9 by moving topologically_order_rules from engine to rules.

@mike-hunhoff do you know if IDA requires capa/ida/plugin/__init__.py or if we can move the code?


Update Central DB signal handling to default to fast shutdown

Description
Default to fast shutdown, which kills all in flight txns and rolls them back. This will allow for faster shutdown and less likely sigkills in the middle of termination
Checklist

[ ] Investigated and inspected CI test results
[ ] Unit test and regression tests added
[ ] Evaluated and added CHANGELOG entry if required
[ ] Determined and documented upgrade steps
[ ] Documented user facing changes (create PR based on openshift/openshift-docs and merge into rhacs-docs)

If any of these don't apply, please comment below.
Testing Performed
Ran the equivalent in a chaos test for several days

With one minor comment


Incorrectly range vertices/edges when using barrier

Bug Type (问题类型)
gremlin (结果不合预期)
Before submit

[X] 我已经确认现有的 Issues 与 FAQ 中没有相同 / 重复问题

Environment (环境信息)

Server Version: v0.12.0
Backend: in memory
OS: xx CPUs, xx G RAM, Centos 7.x
Data Size:  xx vertices, xx edges

Expected & Actual behavior (期望与实际表现)
g.V().range(2,3).range(0,1)   ==> v[1]
g.V().range(2,3).barrier().range(0,1)  ==>{Invalid range: [2, 1)}

We create three vertices (e.g., v1, v2, and v3) and range them with range(2,3) and range(0,1). We expect one vertex (e.g., v1) can be returned.
However, when we execute the second query where a barrier() is added between these two range operations, an exception is thrown instead of returning one vertex.
Vertex/Edge example (问题点 / 边数据举例)
No response
Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)
No response

I executed this test case in the latest version 1.0.0 and can get the expected behavior.
Maybe it's already fixed?


aae-export is not working in blender 4.2

When I try to use aae-export in blender 4.2, it returns error as below:
Python: Traceback (most recent call last):
  File "C:\Users\[redacted]\AppData\Roaming\Blender Foundation\Blender\4.2\scripts\addons\aae-export.py", line 4934, in execute
    clip_settings = context.edit_movieclip.AAEExportSettingsClip
                    ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Context' object has no attribute 'edit_movieclip'

fwiw basic plane and tracker export worked for me with

aae-export as of dec178a46eb16f6f7807d854a2d98156d7d5564c
Blender 4.2.0
Windows 10 10.0.19044-SP0
Python 3.11.7


Prevent Popup from propagating touch_up/move events outside itself

This is related to the post I send on the kivy-user list.
https://groups.google.com/forum/?hl=fr#!topic/kivy-users/kR9WHcGfVyk
As the change are super simple I'm trying to send this request.
However

I'm completely new to git/github
the request might be non-sens
in one of thes cases please appologize for the noise !

Almost everything was ok, just comments that are not necessary :) thanks for the contribution!


Gracefully handle [null, null] for GeoPointField value

In case you need to set coordinates to null this would be useful

Thanks for PR, Igor! Not sure, if this is desired behavior, but if you find this change useful, will trust you on that.


Month is moved backward in new version of Metabase + latest driver.

In Driver 0.4 and Metabase 0.32.8 August is shown like July.
Take a look at it please.
In previous versions everything is OK.

Hi @dzarlax sorry about that. Could you please add some more detail to your description? What does your data look like in native ClickHouse client? What is your query in Metabase? Which timezones do your servers run at? Do you have access to ClickHouse query log? Perhaps you can see the "bad" statement. Or is the statement correct and the month names are 1 off in the Metabase result table?

This was fixed indirectly.


Connecting in Non play mode

Hello, and sorry for so many issues, I just like it and would like to use it in a project :D
I need the plugin to work in editor mode in c++, and it doesn't seem to behave.
I literally call the same function on this button and on begin play.
In BeginPlay, i get a server connection, when not in play mode, I do not get a connection.
Basically I have this situation I want to acomplish. I have already done it with vanilla websockets, so I know it can work

When in play mode, everything works fine and as imagine.
The main problem is I don't get any error, it simply does not connect.
I suspect this has something to do with the const UObject* WorldContextObject,˛ passed to USCClient::Create( but I am not sure about this. I have tried following the code, and break it intentionally and I get an error "no world found"
I have tried to put GetOwner(), GetWorld(),  this and all kinds of other stuff but to no avail.
Is this behaviour something that is known?
Thank you in advance

Where are you creating the client from?
It has to be an object that is located in the world, the reason is Timers are connected to the world, at least timers that can be located at a later date are, socketcluster requires this for ack timer out (no response from server)
If there is no world the client can't be created.

From a component attached to an actor(cinecamera), i think that has a valid world even if it's not in play mode?

Oke but when you are saying not in play mode, means you are not starting the game at all, your not starting a session ?

Im in editor mode, and the game is stopped, if I click play it works, if not it doesn't.
So is it designed not to work that way? I don't really use the timers even, i just need to send and receive messages to a server.
Can you atleast point me to the piece of code which fails? Since there is actually no error.
The use case is setting properties with a external control panel, which seems like a reasonably straight forward use of sockets :)

Yes that's the problem you are in Editor Mode, World is created in game not in the editor, sockcluster is not designed for that, the timers are required for ACK, which means server timeout when the server shutsdown or crashes, RPC call etc.. it's part of the framework.
I understand that it's a reasonably straight forward use of sockets, but socketcluster is not websockets, it's a framework build to be used within the framework, it has functionality that standard sockets don't have. that said.
You would have to customize the whole framework.

You would have to remove this code where it checks if the World exists
https://github.com/ziicreater/socketcluster-client-ue4/blob/2591c31e1c06fdec7e3e216985f7c8dc5e978a5e/SocketCluster/Source/SCClient/Private/SCClientSocket.cpp#L145

Remove every timer and timeout function

Remove every Timer Manager function
GetWorld()->GetTimerManager().SetTimer

It's going to be a lot of customization and not really where this framework is meant for if you simply only want to set properties you are better off coding something simpler, you could use the source code of socketcluster, but only bare-bones, like the websocket part written on top of LibWebsockets it's separated module
https://github.com/ziicreater/socketcluster-client-ue4/tree/2591c31e1c06fdec7e3e216985f7c8dc5e978a5e/SocketCluster/Source/SCSocket

Hmm, ok.
The main thing that puzzles me here is that it doesn't throw this error if I'm in editor mode - 		USCErrors::InvalidActionError("Unable to access current game world.");
I goes through, it just doesn't connect.
Are you sure there isn't a world in editor?
A fast check
    UWorld* world = GetWorld();
    UE_LOG(LogTemp, Warning, TEXT("Is there a world?  %d"),  world != nullptr);

outputs
LogTemp: Warning: Is there a world?  1
so its not null i guess.
and the docs say " In the editor many Worlds exist: The level being edited, each PIE instance, each editor tool which has an interactive rendered viewport, and many more." https://docs.unrealengine.com/en-US/API/Runtime/Engine/Engine/UWorld/index.html
I already have a vanilla WebSockets version, but I was hoping I could use SocketCluster since it seems robust and has all of the surrounding features already implemented.
In my other version I have made a fairly simple ondisconnect which triggers when a server goes down, don't really see why timers would be essential here, but I understand. The property changing is just a simple explanation of a basic feature to give an example, the main feature being toggling play mode on n engines in the local network, which is obviously impossible at the moment with socketcluster.
Thank you for the explanation

If there is a world it should just work,
Maybe it needs some more checking, if you don't get the error, then it the world is not the error,


Mathematical programs related to geometrical shapes

This will contain programs to calculate
Area and Perimeter for 2D objects like:

square
rectangle
triangle
circle

Total surface area, Curved surface area and Volume for 3D objects like:

Cube
Cuboid
Cylinder
Cone
Sphere
Hemisphere

I will do this in Python, @Kumar-laxmi sir. Please assign this work to me.

Please assign the task. I can perform the above task in c++.

Kindly raise issue pertaining to standard algorithms.
Even self made algorithms are welcomed!


Add #[async_test] to futures-test crate?

Hi!
I'm interested what you're thinking about adding an #[async_test] proc-macro to the futures-test crate.
There were previous discussions about it at https://github.com/rust-lang-nursery/futures-rs/issues/1169#issuecomment-416301993 and there are also some implementations existing already.

I have a kind of ugly (sry but I'm not good at proc-macros) but working implementation at futures-await-test-preview which uses futures_executor::LocalPool::new().run_until(), I'm not sure if this is better or worse then using block_on, for my use-cases it seems to work perfectly.
There is one created for tokio: tokio-async-await-test

Examples for intended use-case:
#[async_test]
async fn normal_test() {
    assert!(create_future().await == 4);
}

#[async_test]
#[should_panic]
async fn panic_test() {
    assert!(create_future().await == 5);
}

#[async_test]
async fn result_test() -> Result<(), ()> {
    assert!(create_result_future().await? == 4);
    Ok(())
}

Currently I use my implementation to write tests for a Rust wrapper around an asynchronous C++ library, which uses only futures-0.3, not tokio. I find it really useful, perhaps others would too even if it uses futures-executor only at first. Later it could be somehow extended to support other executors as well..
Would you be interested in adding something like this to futures-test?

@ngg tokio currently has one https://docs.rs/tokio/0.2.0-alpha.5/tokio/attr.test.html


Re-enable rustls-tls feature

The rustls-tls feature can be re-enabled with the release of hyper-rustls =0.18.0-alpha.2

As an update, I want to add this feature into master, I've hesitated because previously I couldn't control hyper-rustls updating quickly, and so didn't want to be blocked on it when upgrading to a new hyper or tokio alpha.

yeah, i noticed the time gap between alpha.1 and alpha.2.
we could keep the feature disabled but the code path fixed.


A bunch of improvements for the classification skill

Thank you for an excellent library. I ran it against our dataset of 10k social media posts to detect their sentiment and classify it into a set of topics. This set of patches is what it took for me to get it working - from minor fixes to improvements in the learning process.
I have a bunch of ideas on how to improve learning performance with more advanced learning strategies - I'd be happy to discuss this if there is interest in implementing advanced learning strategies.
PS Let me know if you want me to break this into smaller PRs.

Hi, @chemeris ! thanks for your great contribution!

I'd be happy to discuss this if there is interest in implementing advanced learning strategies.

Absolutely, it would be very helpful to have different learning strategies and reasoning path, and give user the options. Feel free to open the github issue where we can discuss the solutions and reference it in PR hereafter


It takes around 3-5 hours depending on your tasks.


func _input(event): dose not detect any inputs in v4.0.dev.custom_build.09a2b77b2

Godot version:
 v4.0.dev.custom_build.09a2b77b2
OS/device including version:
windows and linux
Issue description:
if i try to use an input event under func _input(event): nothing will happen when i press the key
Steps to reproduce:
make a project add a script to a node then add func _input(event):  and add a input event then make it print something then after that play it. when you hit that key nothing will happen
Minimal reproduction project:

godot 4.0 input bug.zip

This is fixed by #40770


React Native Emarsys does not build on latest react native version

Hello there,
we recently planned to upgrade react native to 0.68.2.
The build works but we get runtime erros on android.
On iOS a lot of patching is needed, but i managed to build everything, sadly we get the same runtime errors there too.
The following issue mentiones the same error, but here it is not caused by metro
React Native issue
Error text (or similar)
...\app\node_modules\react-native\Libraries\Image\ImageBackground.js: helperSkipTransparentExpressionWrappers.skipTransparentExprWrapperNodes is not a function
As far as i can tell the problem is the version of react native used in this lib.
the emarsys wrapper uses 0.60.0, a version of react native that is over 3 years old.
Here is the output of npm ls tracing the error.
├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>extraneous
├──<EMAIL_ADDRESS>extraneous
├──<EMAIL_ADDRESS>extraneous
├──<EMAIL_ADDRESS>extraneous
├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>invalid<EMAIL_ADDRESS>from the root project
├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>└──<EMAIL_ADDRESS>
npm ERR! code ELSPROBLEMS
npm ERR! extraneous<EMAIL_ADDRESS>.../node_modules/@babel/plugin-transform-block-scoped-functions
npm ERR! extraneous<EMAIL_ADDRESS>.../node_modules/@babel/plugin-transform-member-expression-literals
npm ERR! extraneous<EMAIL_ADDRESS>.../node_modules/@babel/plugin-transform-object-super
npm ERR! extraneous<EMAIL_ADDRESS>.../node_modules/@babel/plugin-transform-property-literals
npm ERR! invalid<EMAIL_ADDRESS>.../node_modules/react-native-emarsys-wrapper

as the documentation of npm ls states extraneous warnings will not result in an error.
Can you look into this?
If you need any more information let me know.
Thanks a lot!
Greetings
Dennis

Hi @DennisOstertag,
We updated react-native to 0.67.3 in the latest release.
Please, next time open a ticket in the Emarsys help portal.
Best,
Edu


Send email via SES

Overview
Creates SES verification and DKIM records in AWS and configures the application to use SES for email transport.
Notes
I've opened up an AWS support case to have SES taken out of sandbox mode so that we can send registration emails to arbitrary recipients. I'll indicate when the ticket goes through in a comment.
The verification link in the registration email will not be valid. I set up #163 to track this bug.
Testing Instructions

Run docker-compose -f docker-compose.ci.yml run --rm terraform.
Run GIT_COMMIT=41f9c48 ./scripts/infra plan. Confirm that there are no new changes to be applied.
Log in to the DistrictBuilder AWS Console.
Go to SES and verify the email you plan to use to test registration.
Go to https://staging.districtbuilder.azavea.com/register and register a new user using the email that you previously verified.
Confirm that you receive a verification email from<EMAIL_ADDRESS>Start up a local development server with scripts/setup and scripts/server.
Go to the local server running on http://localhost:3005.
Perform the same registration process with the same email you had previously verified in the SES console.
Confirm that you receive a verification email from<EMAIL_ADDRESS>
Closes #54

The SES sandbox restrictions have been lifted and our limit has been increased to 50,000 emails per day (14 per second). We can now send emails to unverified addresses. I'll simplify the testing instructions to account for this 📬

Awesome, thank you for helping me polish this one!
Regarding the activation link not matching an internal URL, I hadn't thought about that case. Currently that can be configured by setting the CLIENT_URL environment variable which is used to generate the emailed URLs. I'm leaning towards handling this sort of thing in application config, e.g. with Terraform on AWS, vs. dynamically for sake of predictability.
If there's a useful case I'm not thinking of though, I'd be happy to set up an issue to track it!


fix: undefined method

This fixes the error:
Uncaught TypeError: Cannot read property 'open' of undefined
    at Constructor.handleOnClick (eval at evalScript (http://localhost:3000/_next/next-dev.bundle.js:35040:4), <anonymous>:18030:24)

Thanks! 😀


define => export

Initially chose define and require to match requirejs naming conventions but import/export makes more sense given that require is already a ruby global.

Fixed by bb9e7dbf434c7fa45e986b3e75c7b5fd405f669b


update to shakespeare 2.0 and remove shakespeare-text

shakespeare-text has been deprecated in favor of shakespeare
https://hackage.haskell.org/package/shakespeare-text

this was merged just now, closing


refactor: remove unused dependency afero

Removed unused dependency afero

Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation change
[X] Refactor

Checklist

[ ] Documentation added for the feature
[ ] CI and all relevant tests are passing
[ ] Code Review completed
[ ] Verified independently by reviewer

This would be a chore - it is not a code change.


BlobClient.SyncCopyFromUriAsync fails with "The lease ID specified does not match the lease ID for the blob" when the source URI is from Azure blob storage

Describe the bug
Using the Azure.Storage.Blobs client library, when trying to overwrite an existing blob with data that resides in blob storage, using a lease to lock the file, the Azure.Storage.Blobs.BlobClient.SyncCopyFromUriAsync operation fails. It receives a 412 error from the REST API and an ErrorCode of LeaseIdMismatchWithBlobOperation and the text The lease ID specified did not match the lease ID for the blob.
Expected behavior
Azure.Storage.Blobs.BlobClient.SyncCopyFromUriAsync called with a lease ID will succeed when the source URI is from blob storage itself, and the lease ID is valid.
Actual behavior (include Exception or Stack Trace)
It receives the following error in an exception message:
       The lease ID specified did not match the lease ID for the blob.
       RequestId:9173351a-501e-00a4-57cf-0df414000000
       Time:2021-02-28T12:45:01.0139914Z
       Status: 412 (The lease ID specified did not match the lease ID for the blob.)
       ErrorCode: LeaseIdMismatchWithBlobOperation

       Headers:
       Server: Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
       x-ms-request-id: 9173351a-501e-00a4-57cf-0df414000000
       x-ms-client-request-id: 0069def0-b853-4389-81b4-b362488634c3
       x-ms-version: 2020-02-10
       x-ms-error-code: LeaseIdMismatchWithBlobOperation
       Date: Sun, 28 Feb 2021 12:45:00 GMT
       Content-Length: 264
       Content-Type: application/xml

To Reproduce
Below is the full code from Program.cs in a .NET 5 console app to reproduce the problem. It has been run on Windows with the following package references:
Azure.Identity: 1.3.0
Azure.Storage.Blobs: 12.8.0
The code is being run from within Visual Studio in which there is an attached account that has membership of the "Owner" and "Storage Blob Data Owner" roles on the destination storage account. The destination storage account has disabled public access, and the destination container is thus private.
The code firstly creates a blob by specifying a source URI which is an image from the public internet.
Then it attempts to overwrite that blob with the same content, but this time specifying a lease that it acquires.
Finally it attempts to overwrite the blob using the same lease but where the source is a blob from a different storage account where the blob has been configured for public access.
The full received error information is in a comment in the code.
Whilst this code doesn't demonstrate it, the same problem occurs when the source URI is a SAS URI to a private blob.
using System;
using Azure.Identity;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
using Azure.Storage.Blobs.Specialized;

var publicInternetSourceUri = new Uri(
    "https://purepng.com/public/uploads/large/purepng.com-microsoft-logo-iconlogobrand-logoiconslogos-251519939091wmudn.png");
var publicBlobSourceUri = new Uri("https://intentionallypublicjpg.blob.core.windows.net/pub/ms-windows-logo.png");

const string destinationStorageAccount = "destinationstorageaccount"; // TODO: Change
const string destinationContainerName = "destinationcontainername"; // TODO: Change

// Brand new blob name - doesn't currently exist
var destinationBlobName = $"destination{DateTime.UtcNow.Ticks}.png";
var destinationContainerClient = new BlobContainerClient(
    new Uri($"https://{destinationStorageAccount}.blob.core.windows.net/{destinationContainerName}"),
    new DefaultAzureCredential());
var destinationBlob = destinationContainerClient.GetBlobClient(destinationBlobName);

BlobLeaseClient destinationBlobLease = null;
try
{
    // Setup code - copying something from a public URL to this new blob name:
    await destinationBlob.SyncCopyFromUriAsync(publicInternetSourceUri);

    // Acquire a lease before trying to overwrite the blob.
    destinationBlobLease = destinationBlob.GetBlobLeaseClient();
    await destinationBlobLease.AcquireAsync(
        BlobLeaseClient.InfiniteLeaseDuration,
        new BlobLeaseRequestConditions());

    // Set up the structure for providing the lease ID to the SyncCopyFromUriAsync operations
    var blobOptions = new BlobCopyFromUriOptions
    {
        DestinationConditions = new BlobRequestConditions
        {
            LeaseId = destinationBlobLease.LeaseId
        }
    };

    // Works: Overwriting blob using public internet source data
    await destinationBlob.SyncCopyFromUriAsync(
        publicInternetSourceUri,
        blobOptions);

    // Doesn't work: Overwriting blob using public Azure blob as source daata
    await destinationBlob.SyncCopyFromUriAsync(
        publicBlobSourceUri,
        blobOptions);

    /* It receives the following error in the exception message:

       The lease ID specified did not match the lease ID for the blob.
       RequestId:9173351a-501e-00a4-57cf-0df414000000
       Time:2021-02-28T12:45:01.0139914Z
       Status: 412 (The lease ID specified did not match the lease ID for the blob.)
       ErrorCode: LeaseIdMismatchWithBlobOperation

       Headers:
       Server: Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
       x-ms-request-id: 9173351a-501e-00a4-57cf-0df414000000
       x-ms-client-request-id: 0069def0-b853-4389-81b4-b362488634c3
       x-ms-version: 2020-02-10
       x-ms-error-code: LeaseIdMismatchWithBlobOperation
       Date: Sun, 28 Feb 2021 12:45:00 GMT
       Content-Length: 264
       Content-Type: application/xml
    */
}
catch (Exception ex)
{
    Console.WriteLine(ex.Message);
}
finally
{
    if (destinationBlobLease != null)
        await destinationBlobLease.ReleaseAsync();
}

Environment:
Windows 10
.NET 5 Console App
Package references:

Azure.Identity: 1.3.0
Azure.Storage.Blobs: 12.8.0
Source URI is in Azure blob storage where that blob is either publicly accessible or the URI includes a SAS token

Thank you for your feedback.  Tagging and routing to the team best able to assist.

Hi,
So I was able to reproduce using the code snippet you provided. However I did my own reproduce with my own storage account and source blob Url and I wasn't able to get the same problem and exception you are facing. So far I've only been able to reproduce with the source blob you've provided.
I will update the thread once I figure out what's happening, I'm inclined to think this is not a SDK issue as the request is sending as it should.

Hi,
After talking to some folks from the service, looks like this is a service bug. The blob you have was uploaded or created using CommitBlockList or the Rest API call Put Block List. This is causing the error because once it goes to copy from the source to the destination blob which has a lease on it, it looks like its failing to apply the lease to the rest of the blocks in the blob.
That's why I wasn't able to reproduce your issue with my own blobs, because I wasn't creating my blobs using CommitBlockList.
I will update this thread if there's a better plan of mitigation, however looks like there needs to be a fix on the service in order to resolve this issue.

Thanks for looking into this, Amanda. Is there an issue number to track with the service team? Or is that internal to Microsoft?
Just for information and interest, I uploaded the test blob from file using the Microsoft Azure Storage Explorer application. I'm not quite sure what operation it used under the hood.

Thanks for looking into this, Amanda. Is there an issue number to track with the service team? Or is that internal to Microsoft?

It is internal to Microsoft.  Service features and bugs take some time, ideally it could be fixed in the next few months.

Since this is a service bug and not an SDK bug we have to close this issue.


Persistence setting not working

I have been trying to deploy pyroscope with persistence but as the  container is running as non-root (101), its not able to create folder in the mounted volume. This even with the securityContext settings.
persistence:
  enabled: true
  size: 16Gi
  storageClassName: ssd
  accessModes:
    - ReadWriteOnce
  finalizers:
    - kubernetes.io/pvc-protection

I have managed to resolved the issue by adding the below initContainer to the deployment
      initContainers:
      - name: volume-mount-fix
        command:
        - sh
        - -c
        - chmod 777 /var/lib/pyroscope && chown -R 101:101 /var/lib/pyroscope
        image: busybox
        volumeMounts:
        - name: storage
          mountPath: /var/lib/pyroscope

Hope this can be added in the official chart or at least the option of customizing the initContainers section.

No problem, thanks for looking.

When I remove the chown it stop working
I don't have any specific SecurityContextConstraints or PodSecurityPolicy its a default RKE cluster

I am mainly using the default chart settings, so it should be reproducible

Thank you @syepes for the contribution! It's an interesting issue, because I can't reproduce this using AWS EKS. I suspect that this might be specific to RKE and the volume type/CSI provisioner you use: for example, I suppose fsGroup may not work properly with hostPath volumes or if the CSI driver does not support this.
I think it makes sense to add this init container to the helm chart as an option. However, I'd like to figure out the exact root cause in this particular case. May I ask you to post details of the storage class you use?
kubectl describe storageclass ssd

Oh, this is interesting. Could it be related to https://github.com/kubernetes/cloud-provider-openstack/issues/1362?

Ya lock like that :-)
The initContainers option could also be helpful for other stuff

We should add it definitely. If you want to help, contributions are always greatly appreciated :)

Submitted PR: https://github.com/pyroscope-io/helm-chart/pull/93