Unaligned stores/loads

ubsan detects a lot of undefined stores/loads:
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x00000206ff52 for type 'density_byte', which requires 4 byte alignment
0x00000206ff52: note: pointer points here
 69 70  0e 3f 39 90 98 7f 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  01 00 00 00 06 00
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x00000206ff56 for type 'density_byte', which requires 4 byte alignment
0x00000206ff56: note: pointer points here
 70 72 69 6d 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  01 00 00 00 06 00 00 00  04 00
             ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x00000206ff5a for type 'density_byte', which requires 4 byte alignment
0x00000206ff5a: note: pointer points here
 69 73  20 69 00 00 00 00 00 00  00 00 00 00 00 00 00 00  01 00 00 00 06 00 00 00  04 00 00 00 04 00
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x00000206ff5e for type 'density_byte', which requires 4 byte alignment
0x00000206ff5e: note: pointer points here
 6e 20 66 61 00 00  00 00 00 00 00 00 00 00  01 00 00 00 06 00 00 00  04 00 00 00 04 00 00 00  00 00
             ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x00000206ff62 for type 'density_byte', which requires 4 byte alignment
0x00000206ff62: note: pointer points here
 75 63  69 62 00 00 00 00 00 00  01 00 00 00 06 00 00 00  04 00 00 00 04 00 00 00  00 00 00 00 b7 73
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x00000206ff66 for type 'density_byte', which requires 4 byte alignment
0x00000206ff66: note: pointer points here
 75 73 20 6f 00 00  01 00 00 00 06 00 00 00  04 00 00 00 04 00 00 00  00 00 00 00 b7 73 e4 b2  00 00
             ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x00000206ff6a for type 'density_byte', which requires 4 byte alignment
0x00000206ff6a: note: pointer points here
 72 63  69 20 00 00 06 00 00 00  04 00 00 00 04 00 00 00  00 00 00 00 b7 73 e4 b2  00 00 00 00 00 00
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x00000206ff6e for type 'density_byte', which requires 4 byte alignment
0x00000206ff6e: note: pointer points here
 6c 75 63 74 00 00  04 00 00 00 04 00 00 00  00 00 00 00 b7 73 e4 b2  00 00 00 00 00 00 00 00  00 00
             ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x00000206ff72 for type 'density_byte', which requires 4 byte alignment
0x00000206ff72: note: pointer points here
 75 73  20 65 00 00 04 00 00 00  00 00 00 00 b7 73 e4 b2  00 00 00 00 00 00 00 00  00 00 00 00 00 00
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x00000207007a for type 'density_byte', which requires 4 byte alignment
0x00000207007a: note: pointer points here
 76 65  72 72 00 00 00 00 00 00  c8 41 73 90 98 7f 00 00  c8 41 73 90 98 7f 00 00  70 00 07 02 00 00
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x00000207007e for type 'density_byte', which requires 4 byte alignment
0x00000207007e: note: pointer points here
 61 2e 20 43 00 00  c8 41 73 90 98 7f 00 00  c8 41 73 90 98 7f 00 00  70 00 07 02 00 00 00 00  70 00
             ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x000002070082 for type 'density_byte', which requires 4 byte alignment
0x000002070082: note: pointer points here
 72 61  73 20 73 90 98 7f 00 00  c8 41 73 90 98 7f 00 00  70 00 07 02 00 00 00 00  70 00 07 02 00 00
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x000002070086 for type 'density_byte', which requires 4 byte alignment
0x000002070086: note: pointer points here
 69 6e 74 65 00 00  c8 41 73 90 98 7f 00 00  70 00 07 02 00 00 00 00  70 00 07 02 00 00 00 00  79 e9
             ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x00000207008a for type 'density_byte', which requires 4 byte alignment
0x00000207008a: note: pointer points here
 72 64  75 6d 73 90 98 7f 00 00  70 00 07 02 00 00 00 00  70 00 07 02 00 00 00 00  79 e9 0d a3 75 c0
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x00000207008e for type 'density_byte', which requires 4 byte alignment
0x00000207008e: note: pointer points here
 20 76 65 6c 00 00  70 00 07 02 00 00 00 00  70 00 07 02 00 00 00 00  79 e9 0d a3 75 c0 8d d4  84 f2
             ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x000002070092 for type 'density_byte', which requires 4 byte alignment
0x000002070092: note: pointer points here
 20 6e  69 73 07 02 00 00 00 00  70 00 07 02 00 00 00 00  79 e9 0d a3 75 c0 8d d4  84 f2 35 bf 79 df
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x000002070096 for type 'density_byte', which requires 4 byte alignment
0x000002070096: note: pointer points here
 6c 20 69 6e 00 00  70 00 07 02 00 00 00 00  79 e9 0d a3 75 c0 8d d4  84 f2 35 bf 79 df a2 f6  3d 5e
             ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x00000207009a for type 'density_byte', which requires 4 byte alignment
0x00000207009a: note: pointer points here
 20 66  61 63 07 02 00 00 00 00  79 e9 0d a3 75 c0 8d d4  84 f2 35 bf 79 df a2 f6  3d 5e 6a cc 51 da
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x00000207009e for type 'density_byte', which requires 4 byte alignment
0x00000207009e: note: pointer points here
 69 6c 69 73 00 00  79 e9 0d a3 75 c0 8d d4  84 f2 35 bf 79 df a2 f6  3d 5e 6a cc 51 da 51 c3  9f 56
             ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x0000020700a2 for type 'density_byte', which requires 4 byte alignment
0x0000020700a2: note: pointer points here
 69 73  2e 20 0d a3 75 c0 8d d4  84 f2 35 bf 79 df a2 f6  3d 5e 6a cc 51 da 51 c3  9f 56 07 7d 4b 00
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x0000020700a6 for type 'density_byte', which requires 4 byte alignment
0x0000020700a6: note: pointer points here
 43 75 72 61 8d d4  84 f2 35 bf 79 df a2 f6  3d 5e 6a cc 51 da 51 c3  9f 56 07 7d 4b 00 2d 06  b6 78
             ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x0000020700aa for type 'density_byte', which requires 4 byte alignment
0x0000020700aa: note: pointer points here
 62 69  74 75 35 bf 79 df a2 f6  3d 5e 6a cc 51 da 51 c3  9f 56 07 7d 4b 00 2d 06  b6 78 64 8b d2 e2
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x0000020700ae for type 'density_byte', which requires 4 byte alignment
0x0000020700ae: note: pointer points here
 72 20 73 6f a2 f6  3d 5e 6a cc 51 da 51 c3  9f 56 07 7d 4b 00 2d 06  b6 78 64 8b d2 e2 c2 d9  a4 9b
             ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x0000020700b2 for type 'density_byte', which requires 4 byte alignment
0x0000020700b2: note: pointer points here
 6c 6c  69 63 6a cc 51 da 51 c3  9f 56 07 7d 4b 00 2d 06  b6 78 64 8b d2 e2 c2 d9  a4 9b e8 ee 63 94
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x0000020700b6 for type 'density_byte', which requires 4 byte alignment
0x0000020700b6: note: pointer points here
 69 74 75 64 51 c3  9f 56 07 7d 4b 00 2d 06  b6 78 64 8b d2 e2 c2 d9  a4 9b e8 ee 63 94 2a a5  33 a6
             ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x0000020700ba for type 'density_byte', which requires 4 byte alignment
0x0000020700ba: note: pointer points here
 69 6e  20 74 07 7d 4b 00 2d 06  b6 78 64 8b d2 e2 c2 d9  a4 9b e8 ee 63 94 2a a5  33 a6 63 ad d1 2f
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x0000020700be for type 'density_byte', which requires 4 byte alignment
0x0000020700be: note: pointer points here
 6f 72 74 6f 2d 06  b6 78 64 8b d2 e2 c2 d9  a4 9b e8 ee 63 94 2a a5  33 a6 63 ad d1 2f 8b 2e  e5 f0
             ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x0000020700c2 for type 'density_byte', which requires 4 byte alignment
0x0000020700c2: note: pointer points here
 72 20  76 65 64 8b d2 e2 c2 d9  a4 9b e8 ee 63 94 2a a5  33 a6 63 ad d1 2f 8b 2e  e5 f0 1c ef d4 ac
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x0000020700c6 for type 'density_byte', which requires 4 byte alignment
0x0000020700c6: note: pointer points here
 6c 20 63 6f c2 d9  a4 9b e8 ee 63 94 2a a5  33 a6 63 ad d1 2f 8b 2e  e5 f0 1c ef d4 ac 41 87  54 c2
             ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:109:33: runtime error: store to misaligned address 0x00000206ffc4 for type 'density_chameleon_signature', which requires 8 byte alignment
0x00000206ffc4: note: pointer points here
  20 73 65 64 00 00 00 00  00 00 00 00 20 74 65 6d  70 6f 72 20 70 75 72 75  73 20 63 75 72 73 75 73
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x0000020700d2 for type 'density_byte', which requires 4 byte alignment
0x0000020700d2: note: pointer points here
 2a a5  33 a6 63 ad d1 2f 8b 2e  e5 f0 1c ef d4 ac 41 87  54 c2 46 e2 4f 8c db 44  84 3c 1a 13 f1 4f
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x0000020700d6 for type 'density_byte', which requires 4 byte alignment
0x0000020700d6: note: pointer points here
 20 61 75 63 8b 2e  e5 f0 1c ef d4 ac 41 87  54 c2 46 e2 4f 8c db 44  84 3c 1a 13 f1 4f 5f 8e  eb b8
             ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x0000020700da for type 'density_byte', which requires 4 byte alignment
0x0000020700da: note: pointer points here
 74 6f  72 2e 1c ef d4 ac 41 87  54 c2 46 e2 4f 8c db 44  84 3c 1a 13 f1 4f 5f 8e  eb b8 07 d2 07 b6
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode.c:126:38: runtime error: store to misaligned address 0x00000207089a for type 'density_byte', which requires 4 byte alignment
0x00000207089a: note: pointer points here
 8b 39  f0 21 20 65 75 69 73 6d  6f 64 2c 20 6e 6f 6e 20  76 61 72 69 75 73 20 66  65 6c 69 73 20 64
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_encode_template.h:117:25: runtime error: store to misaligned address 0x00000207082a for type 'density_chameleon_signature', which requires 8 byte alignment
0x00000207082a: note: pointer points here
 75 73  20 66 75 65 74 20 65 73  74 20 e3 f9 20 64 69 63  74 75 6d 2e 65 f6 29 f5  6b 9a 70 75 73 20
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_decode.c:97:14: runtime error: load of misaligned address 0x00000206ff52 for type 'density_byte', which requires 4 byte alignment
0x00000206ff52: note: pointer points here
 69 70  0e 3f 70 72 69 6d 69 73  20 69 6e 20 66 61 75 63  69 62 75 73 20 6f 72 63  69 20 6c 75 63 74
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_decode.c:97:14: runtime error: load of misaligned address 0x00000206ff56 for type 'density_byte', which requires 4 byte alignment
0x00000206ff56: note: pointer points here
 70 72 69 6d 69 73  20 69 6e 20 66 61 75 63  69 62 75 73 20 6f 72 63  69 20 6c 75 63 74 75 73  20 65
             ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_decode.c:97:14: runtime error: load of misaligned address 0x00000206ff5a for type 'density_byte', which requires 4 byte alignment
0x00000206ff5a: note: pointer points here
 69 73  20 69 6e 20 66 61 75 63  69 62 75 73 20 6f 72 63  69 20 6c 75 63 74 75 73  20 65 74 20 75 6c
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_decode.c:97:14: runtime error: load of misaligned address 0x00000206ff5e for type 'density_byte', which requires 4 byte alignment
0x00000206ff5e: note: pointer points here
 6e 20 66 61 75 63  69 62 75 73 20 6f 72 63  69 20 6c 75 63 74 75 73  20 65 74 20 75 6c 3a 6d  65 73
             ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_decode.c:97:14: runtime error: load of misaligned address 0x00000206ff62 for type 'density_byte', which requires 4 byte alignment
0x00000206ff62: note: pointer points here
 75 63  69 62 75 73 20 6f 72 63  69 20 6c 75 63 74 75 73  20 65 74 20 75 6c 3a 6d  65 73 20 70 6f 73
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_decode.c:97:14: runtime error: load of misaligned address 0x00000206ff66 for type 'density_byte', which requires 4 byte alignment
0x00000206ff66: note: pointer points here
 75 73 20 6f 72 63  69 20 6c 75 63 74 75 73  20 65 74 20 75 6c 3a 6d  65 73 20 70 6f 73 75 65  72 65
             ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_decode.c:97:14: runtime error: load of misaligned address 0x00000206ff6a for type 'density_byte', which requires 4 byte alignment
0x00000206ff6a: note: pointer points here
 72 63  69 20 6c 75 63 74 75 73  20 65 74 20 75 6c 3a 6d  65 73 20 70 6f 73 75 65  72 65 20 63 75 62
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_decode.c:97:14: runtime error: load of misaligned address 0x00000206ff6e for type 'density_byte', which requires 4 byte alignment
0x00000206ff6e: note: pointer points here
 6c 75 63 74 75 73  20 65 74 20 75 6c 3a 6d  65 73 20 70 6f 73 75 65  72 65 20 63 75 62 69 6c  69 61
             ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_decode.c:85:24: runtime error: load of misaligned address 0x00000206ffc4 for type 'density_byte', which requires 8 byte alignment
0x00000206ffc4: note: pointer points here
  20 73 65 64 00 00 00 00  80 00 00 00 20 74 65 6d  70 6f 72 20 70 75 72 75  73 20 63 75 72 73 75 73
              ^
/home/nemequ/local/src/squash/plugins/density/density/src/kernel_chameleon_decode.c:97:14: runtime error: load of misaligned address 0x000002071cda for type 'density_byte', which requires 4 byte alignment
0x000002071cda: note: pointer points here
 00 00  e3 f9 20 64 69 63 74 75  6d 2e 65 f6 29 f5 6b 9a  70 75 73 20 13 23 72 a7  b4 b9 65 20 61 20
              ^

I only tested chameleon there, but it's probably a good bet that cheetah and lion have similar issues.

I saw that this pointer alignment requirement is actually well-hidden in the C reference...
It's quite concerning because fixing it will require a good amount of memcpy I'm afraid, checking if the address is aligned would make things even slower so I guess memcpy is the best option.
That's going to generate a performance hit for sure. Do you have any idea on your side about a fast workaround ?

It seems like there really should be a way to get the C compiler to basically do what it is doing now if the hardward supports unaligned loads/stores, but fall back on loading/storing individual bytes on hardware which doesn't support it (MIPS?).  I haven't found anything, though.  I was thinking about asking on SO…  I'll try to do that later today, I'll let you know how it goes.
If there is somewhere you basically want a fast memcpy so you use int64_t or something instead of uint8_t, I was thinking it might be possible to use the new OpenMP SIMD support.  Something like
void not_memcpy (uint8_t* dest, uint8_t src, size_t size) {
  #pragma omp simd safelen(???)
  for (size_t i = 0 ; i < size ; i++)
    dest[i] = src[i];
}

I haven't tried it to see if there is any speedup, though, and obviously it would require OpenMP 4.0 for there to be one.  It would probably be okay to omit safelen for a memcpy replacement, but for an memmove replacement it would take a bit of thought… if you're replacing a loop on uint64_t it could obviously be at least sizeof(uint64_t), but I expect it would be much better if it were at least 16, though 32 or 64 would be much better.
FWIW, my current understanding is that unaligned store/load is basically free on modern x86/x86_64 CPUs, the main danger is that the CC will auto-vectorize the code and unaligned access will trap on vectors.  On ARM the situation is similar, except I believe there is a significant penalty for unaligned access.  MIPS doesn't support unaigned access… the CPU will trap them, and by default Linux will currently catch that and emulate the request using safe instructions—cost of the safer instructions aside (basically loading uint8_ts and shifting/oring together a value), it's very expensive because of the whole trap/catch/retry thing.

http://fastcompression.blogspot.com/2014/11/portability-woes-endianess-and.html has some good ideas.

Thanks for your replies, I checked the link which is very informative.
I like the SIMD idea for memcpy, and apparently it is already implemented on some platforms (osx). It does however require openmp 4 as you say ... Probably too restrictive for now.
After further thinking, I think I might be able to find a very fast workaround for chameleon and cheetah encoding/decoding, but for lion things are different and I don't see a fast enough solution just yet.

I like the SIMD idea for memcpy, and apparently it is already implemented on some platforms (osx). It does however require openmp 4 as you say ... Probably too restrictive for now.

I'm not sure what you mean here—if you're talking about memcpy using SIMD, all platforms should be doing that.  However, the memcpy library function has some overhead as it will take some time to determine what method to use (depending on things like alignment).  It's great for larger operations, but for smaller ones it is pretty expensive.  That said, most compilers will actually inline many memcpy calls, especially for smaller buffers with sizes that are know at compile-time, so if you can use fixed sizes memcpy would probably be fairly snappy.  GCC has a __builtin_memcpy, but AFAIK it's unnecessary unless you compile with -fno-builtins.
If you're talking about OS X (i.e. clang) supporting OpenMP 4.0, it doesn't—hopefully the next version of clang will.  GCC does since 4.9.  That said, you don't even have to put an ifdef around it… if the compiler doesn't support OpenMP 4 it will still work, it just will not use SIMD (unless the C compiler does it).  If you want to take a vastly different approach when OpenMP 4.0 isn't available you can always use #if defined(_OPENMP) && (_OPENMP >= 201307)

I'm not sure what you mean here—if you're talking about memcpy using SIMD, all platforms should be doing that. However, the memcpy library function has some overhead as it will take some time to determine what method to use (depending on things like alignment). It's great for larger operations, but for smaller ones it is pretty expensive. That said, most compilers will actually inline many memcpy calls, especially for smaller buffers with sizes that are know at compile-time, so if you can use fixed sizes memcpy would probably be fairly snappy. GCC has a __builtin_memcpy, but AFAIK it's unnecessary unless you compile with -fno-builtins.

I'll perform a few tests later on : memcpy vs direct copy by using uint types (unsafe due to alignment issues) vs openmp copies, on OS X.

If you're talking about OS X (i.e. clang) supporting OpenMP 4.0, it doesn't—hopefully the next version of clang will. GCC does since 4.9. That said, you don't even have to put an ifdef around it… if the compiler doesn't support OpenMP 4 it will still work, it just will not use SIMD (unless the C compiler does it). If you want to take a vastly different approach when OpenMP 4.0 isn't available you can always use #if defined(_OPENMP) && (_OPENMP >= 201307)

I was talking about this project :
https://github.com/clang-omp/clang
Very simple to deploy on OS X and it offers the omp simd pragma for clang. I'll check it out.

I was talking about this project :
https://github.com/clang-omp/clang
Very simple to deploy on OS X and it offers the omp simd pragma for clang. I'll check it out.

AFAIK that is the project they're trying to merge into clang.  Unfortunately this has been going on for several years.

Okay, I created a small code snip to test all of this :
#include <omp.h>
#include <stdio.h>
#include <strings.h>
#include <sys/resource.h>

#define MAX_SIZE	(1 << 24)
#define MICROSECONDS	1000000.0

void method_memcpy(const unsigned char* input, unsigned char* output, const unsigned int size, const unsigned int iterations) {
    for(unsigned int j = 0; j < iterations; j ++) {
        memcpy(output, input, size);
        *(output + j % size) = 123;
    }
}

void method_byte_to_byte_copy(const unsigned char* input, unsigned char* output, const unsigned int size, const unsigned int iterations) {
    for(unsigned int j = 0; j < iterations; j ++) {
        for(unsigned int i = 0; i < size; i ++)
            *(output + i) = *(input + i);
        *(output + j % size) = 123;
    }
}

void method_simd_copy(const unsigned char* input, unsigned char* output, const unsigned int size, const unsigned int iterations) {
    for(unsigned int j = 0; j < iterations; j ++) {
#pragma omp simd
        for(unsigned int i = 0; i < size; i ++)
            *(output + i) = *(input + i);
        *(output + j % size) = 123;
    }
}

void method_unsafe_8byte_to_8byte_copy(const unsigned char* input, unsigned char* output, const unsigned int size, const unsigned int iterations) {
    for(unsigned int j = 0; j < iterations; j ++) {
        for(unsigned int i = 0; i < (size / sizeof(uint64_t)); i ++)
            *((uint64_t*)output + i) = *((uint64_t*)input + i);
        output[j % size] = 123;
    }
}

void method_unsafe_4byte_to_4byte_copy(const unsigned char* input, unsigned char* output, const unsigned int size, const unsigned int iterations) {
    for(unsigned int j = 0; j < iterations; j ++) {
        for(unsigned int i = 0; i < (size / sizeof(uint32_t)); i ++)
            *((uint32_t*)output + i) = *((uint32_t*)input + i);
        output[j % size] = 123;
    }
}

void method_unsafe_2byte_to_2byte_copy(const unsigned char* input, unsigned char* output, const unsigned int size, const unsigned int iterations) {
    for(unsigned int j = 0; j < iterations; j ++) {
        for(unsigned int i = 0; i < (size / sizeof(uint16_t)); i ++)
            *((uint16_t*)output + i) = *((uint16_t*)input + i);
        output[j % size] = 123;
    }
}

void output_result(const char* title, const struct timeval* start, const struct timeval* stop, const unsigned int size, const unsigned int iterations, const unsigned int bogus) {
    double elapsed = ((stop->tv_sec * MICROSECONDS + stop->tv_usec) - (start->tv_sec * MICROSECONDS + start->tv_usec)) / MICROSECONDS;
    printf("%s\tsize = %d, iterations = %d, time = %3lfs, bogus = %i\n", title, size, iterations, elapsed, bogus);
}

int main() {

    unsigned char* input = malloc(MAX_SIZE * sizeof(unsigned char));
    unsigned char* output = malloc(MAX_SIZE * sizeof(unsigned char));
    struct rusage usage;

    for(unsigned int i = 0; i < MAX_SIZE; i ++)
        *(input + i) = (unsigned char)i;

    unsigned int iterations = (1 << 30);
    for(unsigned int size = 2; size <= MAX_SIZE; size = size << 1, iterations = iterations >> 1) {
        // Memcpy
        getrusage(RUSAGE_SELF, &usage);
        struct timeval start = usage.ru_utime;
        method_memcpy((const unsigned char*)input, output, size, iterations);
        getrusage(RUSAGE_SELF, &usage);
        struct timeval stop = usage.ru_utime;
        output_result("MEMCPY\t\t", &start, &stop, size, iterations, (unsigned int)output[123 % size]);

        // Byte to byte
        getrusage(RUSAGE_SELF, &usage);
        start = usage.ru_utime;
        method_byte_to_byte_copy((const unsigned char*)input, output, size, iterations);
        getrusage(RUSAGE_SELF, &usage);
        stop = usage.ru_utime;
        output_result("BYTE TO BYTE\t", &start, &stop, size, iterations, (unsigned int)output[123 % size]);

        // SIMD copy
        getrusage(RUSAGE_SELF, &usage);
        start = usage.ru_utime;
        method_simd_copy((const unsigned char*)input, output, size, iterations);
        getrusage(RUSAGE_SELF, &usage);
        stop = usage.ru_utime;
        output_result("BYTE TO BYTE SIMD", &start, &stop, size, iterations, (unsigned int)output[123 % size]);

        // 8 byte to 8 byte
        getrusage(RUSAGE_SELF, &usage);
        start = usage.ru_utime;
        method_unsafe_8byte_to_8byte_copy((const unsigned char*)input, output, size, iterations);
        getrusage(RUSAGE_SELF, &usage);
        stop = usage.ru_utime;
        output_result("8 BYTES TO 8 BYTES", &start, &stop, size, iterations, (unsigned int)output[123 % size]);

        // 4 byte to 4 byte
        getrusage(RUSAGE_SELF, &usage);
        start = usage.ru_utime;
        method_unsafe_4byte_to_4byte_copy((const unsigned char*)input, output, size, iterations);
        getrusage(RUSAGE_SELF, &usage);
        stop = usage.ru_utime;
        output_result("4 BYTES TO 4 BYTES", &start, &stop, size, iterations, (unsigned int)output[123 % size]);

        // 2 byte to 2 byte
        getrusage(RUSAGE_SELF, &usage);
        start = usage.ru_utime;
        method_unsafe_2byte_to_2byte_copy((const unsigned char*)input, output, size, iterations);
        getrusage(RUSAGE_SELF, &usage);
        stop = usage.ru_utime;
        output_result("2 BYTES TO 2 BYTES", &start, &stop, size, iterations, (unsigned int)output[123 % size]);
    }

    free(input);
    free(output);
}

To launch it, I installed the latest clang-omp :

$ /usr/local/bin/clang-omp --version
clang version 3.5.0
Target: x86_64-apple-darwin14.3.0
Thread model: posix

Compilation was done with the following command :

$ /usr/local/bin/clang-omp -I/usr/local/include/libiomp/ -fopenmp -Ofast -fomit-frame-pointer -flto copy_study.c -o copy

And here is the result after some data sorting :

A few things clearly stand out :

no difference between basic byte copy and the simd version : auto vectorization is well done by Clang
memcpy is almost always the slowest (it's faster to copy bytes the basic way)
there is no difference in timing, for sizes > 8 bytes, between all methods apart from memcpy

And a few "strange" things are seen :

8 bytes to 8 bytes copy is extremely fast for sizes < 8 bytes (instruction optimizing or problem with the program ?)
*4 bytes to 4 bytes copy is alqo extremely fast for sizes < 4 bytes (same thing ?)

Here is the data in a table form for reference :

Fixed in 240088c110180afc921aeac9e795b2297cf62fcb


Question: GroupJoin issue

Hi,
I'm trying to group join two collections (employees and paychecks) with join key "EmployeeId". With this method (System.Linq) everythings works fine:
var resultGroupJoin1 = employees.GroupJoin(paychecks, x => x.EmployeeId, y => y.EmployeeId,
(x, y) => new { x.FirstName, x.LastName, NumberOfPaychecks = y.Count() });

but with this method (System.Linq.Dynamic.Core):
var resultGroupJoin2 = employees.GroupJoin(paychecks, "EmployeeId", "EmployeeId",
"new ( outer.FirstName, outer.LastName, inner.Count() as NumberOfPaychecks)");

I get error "The string argument ItemName cannot be empty."
Is there any solution to this?

Can you try this:
var resultGroupJoin2 = employees.AsQueryable().GroupJoin(
                paychecks,
                "it.EmployeeId",
                "EmployeeId",
"new ( outer.FirstName, outer.LastName, inner.Count() as NumberOfPaychecks)");

I tried your code and exactly same error ocurred.

See https://github.com/StefH/System.Linq.Dynamic.Core/blob/master/test/System.Linq.Dynamic.Core.Tests/QueryableTests.GroupJoin.cs
Tests GroupJoin_1() and GroupJoin_2().

I looked at these tests and try with this:
var realQuery = employees.AsQueryable().GroupJoin(
    paychecks,
    employee => employee.EmployeeId,
    paycheck => paycheck.EmployeeId,
    (emp, paycheckList) => new { Name = emp.FirstName + " " + emp.LastName, NumberOfPaychecks = paycheckList.Count() });

var dynamicQuery = employees.AsQueryable().GroupJoin(
    paychecks,
    "it.EmployeeId",
    "EmployeeId",
    "new(outer.FirstName + \" \" + outer.LastName as Name, inner.Count() as NumberOfPaychecks)");

Real query works but dynamic doesn't. My entity classes are:
public class Employee
{
    [Key]
    public int EmployeeId { get; set; }
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public int Age { get; set; }

    public ICollection<Paycheck> Paychecks { get; set; } = new List<Paycheck>();
}

public class Paycheck
{
    [Key]
    public int PaycheckId { get; set; }
    public decimal HourlyWage { get; set; }
    public int HoursWorked { get; set; }
    public DateTimeOffset DateCreated { get; set; }

    [ForeignKey("EmployeeId")]
    public Employee Employee { get; set; }
    public int EmployeeId { get; set; }
}

It works fine for me, I've added example GroupJoin_3().
Are you sure you are using correct NuGet?
System.Linq.Dynamic.Core
version="<IP_ADDRESS>"

I looked at example GroupJoin_3(), it does not work again. I get same error "The string argument ItemName cannot be empty". I use correct NuGet package:
<PackageReference Include="System.Linq.Dynamic.Core" Version="<IP_ADDRESS>" />

with .NET Core 2.0 and EF Core 2.0.

I've created some example console-apps, see https://github.com/StefH/System.Linq.Dynamic.Core.TestApps
However these apps do not connect to real DB. Maybe that differs from your application?

I've also updated my example app (https://github.com/StefH/System.Linq.Dynamic.Core.TestApps/tree/master/ConsoleApp.NetCore2) to use a real DB with EF 2.01
And this works fine.
So there should be a difference?
Maybe about [Key] and [ForeignKey("EmployeeId")] ?

Hi Stef,
finally it's working now. The problem was that my paycheck list is IQueryable<Paycheck> type, but when I convert that list to List<Paycheck> type, results are yielded.
Before:
var dynamicQuery = employees.AsQueryable().GroupJoin(
    paychecks,
    "it.EmployeeId",
    "EmployeeId",
    "new(outer.FirstName + \" \" + outer.LastName as Name, inner.Count() as NumberOfPaychecks)");

After:
var dynamicQuery = employees.AsQueryable().GroupJoin(
    paychecks.ToList(),
    "it.EmployeeId",
    "EmployeeId",
    "new(outer.FirstName + \" \" + outer.LastName as Name, inner.Count() as NumberOfPaychecks)");

Ok. Closing this issue.


CZSWAP and CXSWAP gate

Is your feature request related to a use case or problem? Please describe.
stim allow the use of gates like CZSWAP and CXSWAP but cirq doesn't have these gates.
stim define a stimcirq gate currently with all the parameters: stimcirq.CZSwapGate()
Describe the solution you'd like
I would like for cirq to have a CZSWAP gate and a CXSWAP gate for easy moving back and forth between stim and cirq.
[optional] Describe alternatives/workarounds you've considered
Using the FSIM gate is a possibility. However since it's a parametrized gate, it's not always practical to use.

Hi @AlMrvnm, @NoureldinYosri and @verult, I am interested in this issue. Please assign me this issue.

I know it, I was already registered in Unitary Hack.

@NoureldinYosri I've submitted a PR that implements the CXSWAP and CZSWAP gates for the Unitary Hack challenge.


Deploy fails because of missing dependencies of new mysql driver

See eg https://github.com/SURFscz/SRAM-deploy/actions/runs/3468509411/jobs/5794401176

Among others https://github.com/SURFscz/SRAM-deploy/commit/74e183cc8a80f407d6bda9bf9a8b6fc56cca973b

werkt.


Output of automl.fit() with verbose=False is too verbose

The output of automl.fit(X_train, y_train) is too verbose even when using verbose=False. Some of the output seems to be coming from primitives that we have no control over (see e.g. the output below taken from the semi_supervised_classification_example notebook). The flag verbose=False should imply that alpha-automl only shows human-readable progress output produced by alpha-automl. Maybe consider only showing a progress bar using tqdm-notebook.
automl = AutoMLSemiSupervisedClassifier(output_path, time_bound=10, verbose=False,
                                        split_strategy_kwargs={'test_size':.2})
automl.fit(X_train, y_train)

INFO:alpha_automl.automl_api:Found pipeline, time=0:00:04, scoring...
INFO:alpha_automl.automl_api:Scored pipeline, score=0.9136242208370436
INFO:gluonts.mx.context:Using CPU
INFO:gluonts.mx.context:Using CPU
INFO:alpha_automl.automl_api:Found pipeline, time=0:00:11, scoring...
INFO:alpha_automl.automl_api:Scored pipeline, score=0.778272484416741
INFO:alpha_automl.automl_api:Found pipeline, time=0:00:11, scoring...
INFO:alpha_automl.automl_api:Scored pipeline, score=0.7800534283170079
INFO:alpha_automl.automl_api:Found pipeline, time=0:00:19, scoring...
INFO:alpha_automl.automl_api:Scored pipeline, score=0.8076580587711487
INFO:alpha_automl.automl_api:Found pipeline, time=0:00:40, scoring...
INFO:alpha_automl.automl_api:Scored pipeline, score=0.9082813891362422
INFO:alpha_automl.automl_api:Found pipeline, time=0:01:02, scoring...
INFO:alpha_automl.automl_api:Scored pipeline, score=0.8806767586821014
INFO:alpha_automl.automl_api:Found pipeline, time=0:01:03, scoring...
INFO:alpha_automl.automl_api:Scored pipeline, score=0.9091718610863758
INFO:alpha_automl.automl_api:Found pipeline, time=0:01:03, scoring...
INFO:alpha_automl.automl_api:Scored pipeline, score=0.8263579697239537
INFO:alpha_automl.automl_api:Found pipeline, time=0:01:06, scoring...
INFO:alpha_automl.automl_api:Scored pipeline, score=1.0
INFO:alpha_automl.automl_api:Found pipeline, time=0:01:08, scoring...
INFO:alpha_automl.automl_api:Scored pipeline, score=0.9839715048975958
INFO:alpha_automl.automl_api:Found pipeline, time=0:01:08, scoring...
INFO:alpha_automl.automl_api:Scored pipeline, score=0.7880676758682101
INFO:alpha_automl.automl_api:Found pipeline, time=0:01:11, scoring...
INFO:alpha_automl.automl_api:Scored pipeline, score=1.0
INFO:alpha_automl.automl_api:Found pipeline, time=0:01:35, scoring...
INFO:alpha_automl.automl_api:Scored pipeline, score=0.9982190560997328
INFO:alpha_automl.automl_api:Found pipeline, time=0:01:37, scoring...
INFO:alpha_automl.automl_api:Scored pipeline, score=0.918967052537845
INFO:alpha_automl.automl_api:Found pipeline, time=0:01:37, scoring...
INFO:alpha_automl.automl_api:Scored pipeline, score=0.8076580587711487
INFO:alpha_automl.automl_api:Found pipeline, time=0:01:41, scoring...
INFO:alpha_automl.automl_api:Scored pipeline, score=0.7880676758682101
INFO:alpha_automl.automl_api:Found pipeline, time=0:01:41, scoring...
INFO:alpha_automl.automl_api:Scored pipeline, score=0.9973285841495992
INFO:alpha_automl.automl_api:Found pipeline, time=0:01:41, scoring...
[LightGBM] [Info] Number of positive: 102, number of negative: 393
[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.094171 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1634
[LightGBM] [Info] Number of data points in the train set: 495, number of used features: 22
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.206061 -> initscore=-1.348837
INFO:alpha_automl.automl_api:Scored pipeline, score=1.0
INFO:alpha_automl.automl_api:Found pipeline, time=0:01:50, scoring...
[LightGBM] [Info] Start training from score -1.348837
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
INFO:alpha_automl.automl_api:Scored pipeline, score=0.9848619768477294
INFO:alpha_automl.automl_api:Found pipeline, time=0:01:50, scoring...
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf

[LightGBM] [Warning] No further splits with positive gain, best gain: -inf[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[...]

As agreed, we will use 3 levels of logs: 1) all logs, 2) find and score of pipelines, and 3) no logs + a progress bar


Minor cleanup for numbers2

2 line cleanup change for numbers2.  Will merge once CI catches back up.

I am going try to make a pull request to replace ant.   It is being a major problem in the CI.


Fix failing unit test coverage areas

Jest is reporting several areas are failing proper unit test coverage.  We need to improve coverage so we have a higher confidence level in the quality of the code.

Unit tests are fixed and increasing code coverage will be tackled with #37


Added support for Kafka 0.9, Kerberos, SASL_PLAINTEXT

kafka-offset-monitor now supports Kafka brokers >= v0.9.  This includes the added support for Kerberos / SASL_PLAINTEXT!

Upgraded Kafka bindings to <IP_ADDRESS> and changed kafka client to new Java-based client - supporting Kafka brokers >= v0.9 and also supporting authentication/authorization.

This will no longer work with versions of kafka pre-<IP_ADDRESS>, but has been tested to work on <IP_ADDRESS> and 0.10.
Re-factored usage of ZkUtils due to a significant API change from 0.8 to
0.9.
Added two new command-line parameters: kafkaBrokers, and kafkaSecurityProtocol -- allowing users to choose SASL_PLAINTEXT for the security protocol when using kafka as the offset storage location, but defaulting to the original PLAINTEXT protocol when no option is chosen.
Replaced copy-pasted wire-protocol code that retrieves metadata from the kafka brokers with an implementation that uses the kafka provided AdminClient.
Replaced usage of the kafka Scala client from pre-0.9 with the new kafka java-based client.
NOTE: While I have tried to make all of the necessary changes, zookeeper and storm offset storage paths have not been tested outside of the existing unit-tests.

Refactored ZkUtilsWrapper due to underlying API change of ZkUtils.
Changed the headings on the group include html to be more representative of the actual data.

Offset becomes CommittedOffset.
logSize becomes LogEndOffset.

Upgraded Scala to 2.11.8
Upgraded SBT to 13.13

@rcasey212 ,after applying your new patches for the  KafkaOffsetMonitor,I can see the consumer group using offsets.storage as "kafka",and fetch the group offset normally. but I found that the offsetapp.db was alway empty with ddl but without any data.Is it something configured unproperly?Thanks

@copoo I didn't actually change the code which reads/write to the db, so it should not be affected by this change.  My own personal build writes to the db without any issue.  Until this gets patched, would you try cloning from here instead: https://github.com/Morningstar/kafka-offset-monitor.  This is the fork from which I made the PR.  It isn't 100% in sync with my personal repo, which I will delete.

@mortimerp9 What is the process to get this merged?  Is there something I need to do?
Also I'll have a new PR for this in a few days that adds some robustness.

Why hasn't this been merged?

I get the impression that Quantifind does not care about this project anymore. :(

Definitely a bummer that there is 0 movement on this PR.

Thanks to @Mortimerp9 for approving.
Unfortunately it appears this PR will not get merged.  I provided the agreement which Quantifind requests before submitting the PR, but have never heard anything back from them, via email or otherwise.
Regardless, I'm back with the promised stability/robustness updates.
You can find the new release here: https://github.com/Morningstar/kafka-offset-monitor/releases/tag/0.4.1
You can also follow me on github here: https://github.com/rcasey212
Here are the release notes:
v0.4.1 - Stability improvements
Lots of stability improvements in this release.  Here are the changes:
Functional changes:

Graphs could sometimes be difficult to read because you would see non-whole numbers on the y-axis legend, when facts being drawn are always whole numbers. Set y-axis to draw only whole numbers in the legend to make the graph easier to interpret.
Respect command-line arg kafkaOffsetForceFromStart, starting consumer offset listener clients from the beginning of the log by implementin a ConsumerRebalanceListener.

Stability improvements:

Created function tryParseOffsetMessage to attempt to parse a kafka offset message retrieved from the internal committed offset topic:

Handles messages of other types and questionable correctness.
Added 100% unit-test coverage for this new function.

Add robustness to the log-end-offset getter thread:

No longer shutting down the application on error. Instead, closing and destroying the client and re-creating it.
Sleeping on error before re-creating client and continuing to process

Deal with thread-safety issues on shared memory between threads that retrieve data from Kafka.
Stopped polluting consumer groups in zookeeper by not creating a unique consumer group name for the consumer-offset and log-end-offset listener at each client instantiation.
Improved createNewAdminClient code, simplifying the error paths and property calling close on error.
Re-factored some of the error handling paths, simplifying them.
Closing all kafka clients on error so connections do not leak.

General improvements:

Begin to reduce usage of Zookeeper when using offsetStorage = kafka:

Override getTopics() in KafkaOffsetGetter to retrieve topics directly from the Kafka broker
Override getClusterVis() in KafkaOffsetGetter to retrieve cluster information directly from the Kafka broker.

Use constants for all property in createNewKafkaConsumer().
Fixed some bad grammar in error messages.
Fixed silly com.twitter.util-core dependency in build.sbt.

Thanks so much for this! I'll let you know if it's working well for me.

Unfortunately I'm still having issues. One of our devs tells me there might be a breaking consumer API change in 0.10.0 which is what we are using.

Hi @aatchison, you might go here, which is where I am tracking this project since this one is defunct.  We are using kafka-offset-monitor v0.4.1 with Kafka v0.10.0 without issue.

Awesome, thanks so much! I will check it out and let you know sometime here
soonish.
On Mon, Apr 24, 2017 at 12:16 PM, Robert Casey<EMAIL_ADDRESS>wrote:

Hi @aatchison https://github.com/aatchison, you might go here
https://github.com/Morningstar/kafka-offset-monitor, which is where I
am tracking this project since this one is defunct. We are using
kafka-offset-monitor v0.4.1 with Kafka v0.10.0 without issue.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/quantifind/KafkaOffsetMonitor/pull/107#issuecomment-296794601,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AIJOO8IA30vihz97mxGFfIpnl5AntHbLks5rzPUdgaJpZM4LmP2t
.

I think there is no reason to keep this open any longer.  Closing.


Fix storage overcommit default not applied to Longhorn

Problem:
Harvester storage overcommit-config is set to 200% while the value not applied to Longhorn  StorageOverProvisioningPercentage which is still 100%.
Solution:
Take care Default value in  https://github.com/harvester/harvester/blob/6b79086f42d005c19070b760a877fe27ccb4767a/pkg/controller/master/setting/overcommit_config.go#L17
And sync storage overcommit-config during harvester bootstrap.
Related Issue:
https://github.com/harvester/harvester/issues/5695
Test plan:

Setup harvester with one node (easy for testing), and go to longhorn dashboard > setting tab, check Storage Over Provisioning Percentage is 200 instead of 100.
Setup harvester 1.3.1 with one node (easy for testing), upgrade to the harvester version contains this fix, and go to longhorn dashboard > setting tab, check Storage Over Provisioning Percentage is 200 instead of 100.

@mergifyio backport v1.2 v1.3

(1) the validator only checks if the overcommit < 100, but no up limit check; if user sets an invalid high value, Harvester validator does not deny it.

Sounds good !
This could be handled in separate issue, but how do we define "invalid high value" ? Any suggestions ?

(2) for upgrade path, can the default value also be synced to LH?
thanks.

Yes, during the upgrade, the controller will restart and trigger syncOvercommitConfig to update Longhorn storage overcommit setting. And thanks, you remind me I need to add the upgrade part in test plan.

(1) the validator only checks if the overcommit < 100, but no up limit check; if user sets an invalid high value, Harvester validator does not deny it.

Sounds good ! This could be handled in separate issue, but how do we define "invalid high value" ? Any suggestions ?

Good question. I thought to check the longhorn source, but that only specifies a minimum, not a maximum (and the minimum there is zero which seems weird to me): https://github.com/longhorn/longhorn-manager/blob/4d90cc71fb5fbb33008bb9de870f2e8cb018af58/types/setting.go#L524-L535
Anyway, as you both already said, that's really a separate issue :-)

@w13915984028 Feel free to open a new issue for the highest value.

@w13915984028 Feel free to open a new issue for the highest value.

We can do this after https://github.com/harvester/harvester/pull/6088.


Don't allow NuGet tasks to call vulnerability APIs

When doing NuGet dependency detection, we essentially call dotnet restore and part of that involves NuGet checking for known vulnerabilities.  We don't use this directly.  There's likely an environment variable or MSBuild property that we can set to avoid this.
Relevant error numbers are NU1903 and NU3028

This might be solved by setting the property $(NuGetAudit) to false when evaluating anything.


Renewable - refactor Renewable as Interface

Renewable - refactor as Interface
UserCapabilityJunction - cleanup Created/LastModified Aware Interface
Refactored crunch Renewable to Interface so that Capability could implement it and replace all the fields I added the other day.  Also ended tidying up the User Created/LastModified interfaces as well as used by the UserCapabilityJunction. We had legacy code no longer required with our changes to Interface - Joel

LGTM


Close staging repositories after the release

Hey @belaban !
Please take at some additional changes to the release process.
I tested it using my own instance with the following command mvn deploy -Dnexus.server.url=http://<IP_ADDRESS>:8081/nexus -DskipTests -Dmaven.test.skip=true. It worked as expected.
Thanks
Sebastian

Thanks, this works now ! I had to change my settings.xml though to include a jboss-releases-repository though...
Cheers,

Great news! Thanks :)


Use match statments in PVI code

Depends on https://github.com/bluesky/ophyd-async/pull/171
Match statements would greatly simplify the type parsing logic in the pvi code introduced in https://github.com/bluesky/ophyd-async/pull/142

https://github.com/bluesky/ophyd-async/blob/f2ea8c418ec40777ebfe7646671d0f128d263e67/src/ophyd_async/epics/pvi/pvi.py#L161-L162

Probably best to wait on https://github.com/bluesky/ophyd-async/pull/185 for this.


Setup issue

its very difficult to follow setup  as we are unable to know where to place or test config file and how to run it kindly share a sample tutorial or a clear instructions read.md file as on readme its written to run node test but no such file is created and if its a config script code share in readme that where to made it?

i am facing server has closed issue


rxjs setTimeout error. Causing after successful login

After login. When I am performing any task.
It is giving this error and crashing the server.
2020-06-14T06:54:42.726346+00:00 app[worker.1]: /app/node_modules/rxjs/internal/ util/hostReportError.js:4 2020-06-14T06:54:42.726362+00:00 app[worker.1]: setTimeout(function () { throw e rr; }, 0); 2020-06-14T06:54:42.726363+00:00 app[worker.1]: ^ 2020-06-14T06:54:42.726364+00:00 app[worker.1]: 2020-06-14T06:54:42.726364+00:00 app[worker.1]: ClientDisconnectedError: MQTToTC lient got disconnected. 2020-06-14T06:54:42.726365+00:00 app[worker.1]: at SafeSubscriber.client.$discon nect.subscribe [as _next] (/app/node_modules/instagram_mqtt/dist/realtime/realti me.client.js:152:30) 2020-06-14T06:54:42.726366+00:00 app[worker.1]: at SafeSubscriber.__tryOrUnsub ( /app/node_modules/rxjs/internal/Subscriber.js:205:16) 2020-06-14T06:54:42.726367+00:00 app[worker.1]: at SafeSubscriber.next (/app/nod e_modules/rxjs/internal/Subscriber.js:143:22) 2020-06-14T06:54:42.726367+00:00 app[worker.1]: at Subscriber._next (/app/node_m odules/rxjs/internal/Subscriber.js:89:26) 2020-06-14T06:54:42.726368+00:00 app[worker.1]: at Subscriber.next (/app/node_mo dules/rxjs/internal/Subscriber.js:66:18) 2020-06-14T06:54:42.726368+00:00 app[worker.1]: at Subject.next (/app/node_modul es/rxjs/internal/Subject.js:60:25) 2020-06-14T06:54:42.726369+00:00 app[worker.1]: at MQTToTClient.setDisconnected (/app/node_modules/mqtts/dist/mqtt.client.js:329:30) 2020-06-14T06:54:42.726369+00:00 app[worker.1]: at Object.disconnect (/app/node_ modules/mqtts/dist/mqtt.client.js:97:26) 2020-06-14T06:54:42.726370+00:00 app[worker.1]: at TLSSocket.socket.on (/app/nod e_modules/mqtts/dist/transport/tls.transport.js:20:52) 2020-06-14T06:54:42.726370+00:00 app[worker.1]: at TLSSocket.emit (events.js:194 :15) 2020-06-14T06:54:42.726371+00:00 app[worker.1]: at endReadableNT (_stream_readab le.js:1103:12) 2020-06-14T06:54:42.726371+00:00 app[worker.1]: at process._tickCallback (intern al/process/next_tick.js:63:19) 2020-06-14T06:54:42.726372+00:00 app[worker.1]: Emitted 'error' event at: 2020-06-14T06:54:42.726372+00:00 app[worker.1]: at RealtimeClient.emitError (/ap p/node_modules/instagram_mqtt/dist/realtime/realtime.client.js:23:38) 2020-06-14T06:54:42.726373+00:00 app[worker.1]: at SafeSubscriber.client.$discon nect.subscribe [as _next] (/app/node_modules/instagram_mqtt/dist/realtime/realti me.client.js:152:20) 2020-06-14T06:54:42.726373+00:00 app[worker.1]: at SafeSubscriber.__tryOrUnsub ( /app/node_modules/rxjs/internal/Subscriber.js:205:16) 2020-06-14T06:54:42.726374+00:00 app[worker.1]: [... lines matching original sta ck trace ...] 2020-06-14T06:54:42.726374+00:00 app[worker.1]: at process._tickCallback (intern al/process/next_tick.js:63:19)
I think problem with request module.

Please add DEBUG=mqtt:*,ig:* as an environment variable.

@Nerixyz it's the same issue with regeneration sessions, I assume they are using the anitban module, it also regenerates sessions, @linkfy is working on it. @mitali-gupta remove setAntiBan if it's in your script

Please add DEBUG=mqtt:*,ig:* as an environment variable.

Not working.

@Nerixyz it's the same issue with regenerating sessions, I assume they are using the anitban module, it also regenerates sessions, @linkfy is working on it. @mitali-gupta remove setAntiBan if it's in your script

It worked.

Yes, another way is to set online mode to false on the env file, in this way you can use antiban at the moment.

I don't use antiBanMode, but I still get the same error when ONLINE_MODE is set to true


Search not responding at all

As of now, Campexplorer- regardless of what tag is searched- is not responding at all. Have tried different browsers and even IP's.

Thanks for bringing this up, the service had ended up in a faulty state. Should be fixed now and I've set up a tool to get me notified quickly if it happens again.


Fixed dex

I'm pretty pissed as to how dex does not work so here is a fixed one. (Same one as the broken one just an actually working one)

Though this isn't actually going to make it work correctly, as that requires adding stuff server-side in a way that's secure tho I have ideas

Right. I can probably think up something

right this adds a tiny bit of security and actually works mostly

Going to merge this but dex as a whole and everything related to it is going to be moved off to its own plugin.

Annnddd apparently this is kaborked so going to yoink the dark dex and make it work I guess


Recommendation for boost::ptree modification

Hey @KindDragon ! Thanks for maintaining this project, it was a big help for me in root causing an issue I was having. The ptree visualizations didn't work for me though (I was trying on vs 2019, and it doesn't allow function calls -- I'm not sure if the ptree natvis that's in the repo is a placeholder or it somehow works for you) so I made some modifications which I figured I'd post in case you find them useful:
    <!--Common ptree view-->
    <Type Name="boost::property_tree::basic_ptree<*>">
        <DisplayString> {m_data} : {static_cast<subs::base_container*>(m_children),na} </DisplayString>
        <Expand HideRawView="true">
            <Item Name="m_data">m_data</Item>
            <Item Name="m_children">static_cast<subs::base_container*>(m_children),na</Item>
        </Expand>
    </Type>

    <!-- ptree with boost::any as the value type (doesn't show the node value if it's empty) -->
    <Type Name="boost::property_tree::basic_ptree<*,boost::any,*>">
        <DisplayString Condition="m_data.content != 0"> {m_data} : {static_cast<subs::base_container*>(m_children),na} </DisplayString>
        <DisplayString Condition="m_data.content == 0"> {*static_cast<subs::base_container*>(m_children)} </DisplayString>
        <Expand HideRawView="true">
            <Item Condition="m_data.content != 0" Name="m_data">m_data</Item>
            <ExpandedItem>static_cast<subs::base_container*>(m_children)</ExpandedItem>
        </Expand>
    </Type>

    <!-- std::pair representation. Makes it easier to drill down the nodes in the tree -->
    <Type Name="std::pair<*,boost::property_tree::basic_ptree<*>>">
        <DisplayString> {{ {first} : {second} }} </DisplayString>
        <Expand HideRawView="true">
            <ExpandedItem>second</ExpandedItem>
        </Expand>
    </Type>

I didn't want to make a PR because I wrote this kind of specifically for the type of ptree I was looking at, so I'm not sure how exactly it behaves with other trees, but just in case it's useful to you.
Feel free to close this once you've seen it.

Can you show minimal repro and add error log? (https://docs.microsoft.com/en-us/visualstudio/debugger/create-custom-views-of-native-objects?view=vs-2019#BKMK_Diagnosing_Natvis_errors)
For which versions of Boost and VS2019 you have problem?


Revert "[Rust] upgrade to v1.48.0 (#79)"

This reverts commits 3f67e2fcff0d0c0c93366ae8fa8678b018be2d31 and
dd4e51969bcb970e3399c215b279fe9bbd605cc2 until issues with Rust on 32-bit
Windows are fixed.

Close in favour of #86


Avoiding name conflicts

Simple function names like name are likely to collide with variable names or other methods in the user's scope. We may want to consider turning all function names into verbs. e.g. by prepending with get (or get_). e.g.

model -> getmodel or get_model
info -> getinfo or get_info
name -> getname or get_name

I've decided to employ the get_ prefix and will update the API accordingly.

Mmm, I really didn't like all the get_s. I'll hold off on this change until a user tells me it's a problem.


Support Python 3.11

What
Ensure openassetio wheels can be built for Python 3.11
Why
Ensures we don't get too far out of sync with Python as it evolves
Tasks

Update pybind to 2.10.0
Update tests to cover re-worded exceptions [should we be validating the actual message when it originates from outside of our code?].

There isn't yet a ConanCenter package for Python 3.11 and I can't find any open issues/PRs on the ConanCenter recipe GitHub, so it may be a while.
Currently we only ctest against 3.9 anyway, so perhaps no big deal if Conan doesn't have 3.11 yet.  Though the fact that it's not a drop-in replacement gives me pause.


abort_multi_part_stream has been moved and renamed into couch_httpd_mutl...

...ipart:abort_multipart_stream

commit message was shortened to address @kxepal comment

+1


Build time optimization

Summary
Closes #425

What kind of change does this PR introduce? (check at least one)

[ ] Bugfix
[ ] Feature
[ ] Code style update
[ ] Refactor
[ ] Docs
[x] Build-related changes
[ ] Other, please describe:

If changing the UI/CSS related code, please provide the before/after screenshot:
Does this PR introduce a breaking change? (check one)

[ ] Yes
[x] No

If yes, please describe the impact and migration path for existing applications:
The PR fulfills these requirements:

[x] Briefly describing what this PR does in the title in Angular Commit Message Conventions, e.g. fix: rebuild when a page is added

To avoid wasting your time, it's best to open a feature request issue first and wait for approval before working on it.
Other information:

Hmm I didn't notice any build time improvement when this plugin is enabled 😅

@egoist
Client
from this https://github.com/saberland/saber/issues/425#issuecomment-530463126 and this https://github.com/saberland/saber/issues/425#issuecomment-530755868 we get Compiled client successfully in 36.5s! before and Compiled client successfully in 28.6s! after but this is not the most important part the most important is that before we have modules with no loaders took 29.16 secs and after we have modules with no loaders took 8.001 secs
So why we get a big improvement in  modules with no loaders but the general output time is nearly the same? The answer if I am not mistaking is multithreading and from the first and second comment we can see that css-loader with postcss-loader are the one that took the majority of the time
Server
For the server, I got success Compiled server successfully in 31.7s! before and Compiled server successfully in 17.1s! after
PS: For the states before using autodll there are from the 3rd run (yarn dev because the first run is always slower) as for the states after autodll they are from 2nd (+3 from before so it's 5th) run because in the first run there is no cache and so it's slower because the cache is created in it

with cache-loader (the latest saber version) and autodll I get this
[success] Compiled server successfully in 12.5s!
[success] Compiled client successfully in 21.8s!

@egoist should I update this PR or close it?

@egoist I think that there is no need to merge this if webpack 5 release is soon

I think that the main reason for why this should be merged is HMR build time for me it goes from 2s-3s to around 300ms


Main images in background all have a grid of black dots through them.

Awesome website template and I'm trying to use it to showcase some of my photography work but the large images that load are all perforated with black pixels, like the function to display the image isn't quite right?

I've attached a portion of one of the photos that shows the effect I'm talking about.  Have you any ideas how to fix this please?
Much appreciated, Damon

You can fix this by going to the style.css and changing the opacity on line 134 #overlay.
ex.
#overlay {
width: 100%;
height: 100%;
position: fixed;
top: 0px;
left: 0px;
background: url(images/dotted.png);
opacity: 0.0;

Cheers mate that worked a treat!
Thanks for your help and for providing this template, good on you!
Damon
From: fcimp42<EMAIL_ADDRESS>Sent: Monday, 11 April 2016 9:40 AM
To: priteshgupta/vivid-photo
Cc: Damon Thompson
Subject: Re: [priteshgupta/vivid-photo] Main images in background all have a grid of black dots through them. (#2)
You can fix this by going to the style.css and changing the opacity on line 134 #overlay.
ex.
#overlay {
width: 100%;
height: 100%;
position: fixed;
top: 0px;
left: 0px;
background: url(images/dotted.png);
opacity: 0.0;
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub https://github.com/priteshgupta/vivid-photo/issues/2#issuecomment-208092693 https://github.com/notifications/beacon/AQfyiiCt_TVXKFclUpTd4eI_RkzJP6VSks5p2Yo7gaJpZM4HcQku.gif


Peer Review - Non-abstract FR

Artifact Under Review
Functional Requirement FR2 from the project requirements specification.
Team Number for Team Doing the Review
Team 27
Description of Issue
It seems like the current wording of FR2 might be leaning a bit into the how rather than the what by specifying the use of a "diffusion model" for processing CXR images.

A functional requirement typically focuses on the outcome — what the system needs to do — without getting into specific implementation details. Maybe rewording FR2 to focus on the goal of detecting disease-specific patterns could make it more abstract and aligned with the role of functional requirements.

@Tan630


Modal Animation IOS Safari

Modal Animation (with fade={true}) doesn't work on IOS Safari.

components: Modal
reactstrap version: latest
react version: >16.4
bootstrap version: 4.1.1

What is happening?
in Cordova / IOS there is no slide from top to bottom and fade effect. On Chrome (MAC) I see all animation on your presentation website (Modals)
When I browse the same website from IOS Safari (11.4.1) there are no animations.
IOS Simulator works ok as well as Android or web browser (desktop)
What should be happening?
See modal animation in IOS Safari device
Steps to reproduce issue
Browse the modal examples from an IOS device. (I used an Iphone 6)

Did you activate "reduced motions" in your iphone? That is affecting web browsers too.

I had the same Issue, solved it by providing a zIndex value to the Modal component from reactStrap. You can supply an empty string


update org.activiti.dependencies:activiti-dependencies to 7.0.50

UpdateBot pushed maven dependency: org.activiti.dependencies:activiti-dependencies to: 7.0.50

UpdateBot commands:
updatebot push-version --kind maven org.activiti.dependencies:activiti-dependencies 7.0.50 --merge false


issues with relational data in jwt token and req.user

Description
This is a WIP pr that contains tests that currently fails, but demonstrates issues listed in #1361

[ x] I have read and understand the CONTRIBUTING.md document in this repository

Type of change

[ x] Bug fix (non-breaking change which fixes an issue)

Checklist:

[ x] I have added tests that prove my fix is effective or that my feature works
[ ] Existing test suite passes locally with my changes

Related issue fixed in #1361


update rails helper

add in single quote escape

Fix already accepted - closing request from 'patch' branch to master.
For future reference, a valid submission must be made from your fork master branch directly to the 418sec:master branch!


Multiple SWA can't be on the same domain due since root endpoint can't be changed

Multiple SWA can't be on the same domain due since root endpoint can't be changed
TLDR: No possibility to serve application-x.azurestaticapps.net/.auth/* under application-x.azurestaticapps.net/<app-x>/.auth/* which leads to only one SWA being able to be served under one domain.
The easiest way to describe this is with an example:
Assume you have a domain example.com where we serve two applications: example.com/x and example.com/y. The proxy is done with an Application Gateway, so /x* goes to application-x.azurestaticapps.net and /y* goes to application-y.azurestaticapps.net, with header X-FORWARDED-FOR and X-ORIGINAL-HOST set by the proxy.
One solution is to serve the static files from a subdirectory x and y for each app. This would be fine, if you don't need authentication.
The authencation routes are served under application-x.azurestaticapps.net/.auth/*, and when behind the proxy this url becomes example.com/.auth which does not match any of our defined routes. Therefore the need to move the default /.auth files to /x/.auth becomes apparent.
To Reproduce
Steps to reproduce the behavior:

Create a SWA where the root is on a different page than /, e.g. application-x.azurestaticapps.net/app-x/
Have a proxy for example.com that answers on the route example.com/app-x and forwards to application-x.azurestaticapps.net/app-x/ with rewritten host.
Try to login: example.com/app-x/login

{
  "trailingSlash": "auto",
  "routes": [
     {
      "route": "/app-x/login",
      "rewrite": "/.auth/login/github"
    },
    {
      "route": "/app-x/profile*",
      "allowedRoles": ["authenticated"]
    }
  ],
  "responseOverrides": {
    "401": {
      "redirect": "/app-x/login",
      "statusCode": 302
    }
  }
}

Expected behavior
Expected behavior is that more than one SWA can be created on the same domain, while supporting authentication.

Hi @dforslof, I understand your issue, though I'm not sure if allowing .auth at a different path would work, since the auth credentials for multiple SWAs would be found within the memory of the same site and would conflict. What do you expect the behavior to be for this case? The architecture you explain seems to indicate that for each path, a user would need to sign in again

Yes you have understood the problem correctly.
Each application is published on its on route and requires the user to login again. However, today this login is done silently if the user is already logged in, and is handled by the OIDC protocol.
Out auth layer is today handled by an Apache proxy for each application, which publish its own auth endpoint (under the application route, so example.com/application_x/auth), since every application has its own OIDC CLIENT ID.
Is there no possible way to move the /.auth catalog from the root of the application to an arbitrary route on the application? I.e. from example.com/.auth to example.com/new_root/.auth

I will leave this here as a better explanation. If static web apps is based on azurewebsites this should be very easy to implement: https://microsoft.github.io/AzureTipsAndTricks/blog/tip64.html

Also interested in something like this. We are using frontdoor with path routing and running into issues with ./auth/* always being referenced at the root
ourdomain.com/* -> routed to Azure Static Web App A (uses auth0)
ourdomain.com/storybook/* -> routed to Azure Static Web App B hosting our storybook.
Because the ./auth routes are always at the root, our traffic is getting redirected to Azure Static Web A after the user logs in. We want that traffic to go to App B

I have the same setup and issue as @jsheetzmt just that we don't have anything under the root, all SWA is under different paths. My idea was to have one Azure Front Door and domain and then multiple SWA where people can deploy their docs to without having to bother with basically anything else than just setting up their SWA and they can't mess anything up for someone else (allowing them to experiment all they want). Using my Bicep template I can just add another path to an array and a few minutes later Azure Front Door and the new SWA is all setup and just waiting for the first deployment.
So I have this setup atm:
ourdomain.com/* => redirect to /AAA
ourdomain.com/AAA => rewrite to my AAA-swa
ourdomain.com/BBB => rewrite to my BBB-swa
Both AAA and BBB uses AAD (or Microsoft Entra, said the old man who yelled at clouds) but none can actually be accessed since all auth sooner or later ends up on ourdomain/.auth/*, resulting in an infinite loop or bad gateway, depending if I turn off the redirect or not. If I put one of the SWA directly at the root everything works fine (for that SWA).

I have the exact same use case as @OskarKlintrot. I tried doing a similar setup last week and ran into the exact same issues regarding SWA auth. Ideally, we would let each of our internal teams publish their documentation to their own SWA. Then, we would make it accessible through a single Front Door domain that uses path-based routing rules to route a request to the appropriate doc site. The kicker is that we want to ensure only authenticated users can view the documentation. At that point, the aforementioned problems occur where auth flows end up at a fixed domain.com/.auth/* location, which won't work for any of the doc sites that are hosted (ideally) at domain.com/site-a/, domain.com/site-b/, etc.

Did I understand it correctly @dforslof that you're not trying to use any of the built in providers but rather your own OIDC server?

@thomasgauvin is this a limitation in SWA or AAD?

Also interested - using App Gateway to rewrite "virtual apps" which alongside old-school IIS virtual directories under the same root domain - works fine until we need swa auth and then it breaks owing to rooted "/.auth" - sadly may need to use hosted Asp.net apps instead since this gives enough control to modify redirects/cookie paths etc. - would prefer to use swa+functions as this seems to be more Azure-friendly.


Revert "TBDGen: pass-down target variant when generating textual interface stubs"

Reverts apple/swift#30217

@swift-ci Please test

Build failed
Swift Test OS X Platform
Git Sha - 58d4868e422ac2ce678665bcd9d186fc63bbdca0


Pin tox packages for sphinx images

When I rebuilt the sphinx images to test all SWE-Bench Verified instances all newly built images started to fail. Seems to be because of a new version of  tox. This PR is to pin the version in the previously built images. Which seems to work.
Compare this benchmark run with failing sphinx instances:
https://eval.moatless.ai/evaluations/fcbb473f957149e7a5a45baeaa11800c
With this fix it looks like this:
https://eval.moatless.ai/evaluations/ab7a61b78e334a759fff3b344d30bd70

I reproduced this and your changes work!
Thanks so much!

Thanks Albert!


Property ordering in Vaadin

Java Beans always order properties alphabetically. The order should be customizable for auto-generated UIs.

Vaadin supported discontinued.


Verify SHA-512 checksums on Alpine source downloads

We can probably trust HTTPS but the others are open to being badly-downloaded. The APKBUILD files have sha512 sums, so let's use them.

Fixed in master.


Research CDN Cache Invalidation

Description

Do some initial research into CDN cache invalidation. Cache invalidation on the CDN after document modification (modify or delete) would allow us to greatly increase the cache timeout (currently 5 minutes) and consequently the cache hit ratio and HTML load time of the document pages. The scope of this research is limited to cache invalidation of document pages after document edits and deletes, not after any possible modification event (e.g., updating a document's Jinja2 template or CSS).
NOTE: This task is lower priority than the Django 1.11 upgrade work, so can be shelved if needed. That's the only reason why the confidence level is at medium (Cf:Med).
Some questions:

[ ] Could cache invalidation be simplified by not caching the wiki.children or other endpoints?

Acceptance Criteria

Create a list of tasks and related estimates that would be required to accomplish cache invalidation of the document pages on the current CloudFront CDN.

This is no longer relevant, closing.


White line above start button

In the latest release by Ivo, there is a white line above my modified start orb:

I am unsure if this is only an issue on my machine (if so, why?) or if it is widespread.
I am using a textured taskbar and custom start orb, with the Windows 7 style start menu skin.

Look if somewhere in your configuration there is not a white color used

Okay...
Taskbar text color is 'FFFFFF'. Changing that to 0000FF did remove the white line, however now my clock is red, which I don't want.
I tried then to change the color to FFFFF0 so it was close enough to white that I couldn't see a difference, however the line reappeared.

And with "FEFEFE"?

Still a white line.

So it's really a bug, as an alternative I think you can hide this line with your orb by creating a line on the white one

I'm not to fussed about it, I only bought it up in case someone with a bit more skill than myself could find the bug and fix it in hopeful future releases

Future issue submission: https://github.com/passionate-coder/Classic-Start/issues
Any future development will be made at the above address.

I can confirm that the white line is there.
To clarify, it's not there as a result of/because of the image itself...
It's a result of/because of the Classic Menu Start Button image bleedover (overlapping) the top edge of the bar itself.
I hope that I had explained this okay via text... :-)
~Ibuprophen


changed CAGR default preiods to 365 from 252

When calculating CAGR, you are using calendar year of 365 days (see https://www.investopedia.com/terms/c/cagr.asp#:~:text=To calculate the CAGR of,one from the subsequent result.)
Using 252 periods is wrong and will produce misleading CAGR.
For a simple example:
Say the between 2022-01-01 and 2022-12-31, I turned $100 to $200.
My return is 100% and my CAGR is 100%
Using 252 periods, my CAGR would be 61.83%, which is totally wrong.

this is a duplicate with PR 281
unfortunately i haven't had an answer from the maintainers. I'm currently installing quantstats from my fork
pip install git+https://github.com/gnzsnz/quantstats-cagr.git@cagr

not ideal.....

I agree. Use 365 days right the wrong.

A quick patch, whilst we wait for the mainteners to merge the PR, can be (better than installing from some forks):
just insert the following lines after you import statements (incl. "import quantstats as qs")
qs_cagr = qs.stats.cagr
def temp_cagr(returns, rf=0.0, compounded=True, periods=365):
return qs_cagr(returns, rf, compounded, periods)
qs.stats.cagr = temp_cagr

Since this library no longer seems to be maintained, we have created a duplicate of it that we plan to maintain and improve over time. If you'd like to use our version check it out here: https://github.com/Lumiwealth/quanstats_lumi
Please feel free to fork and send your pull request to this new library, we will be actively monitoring the library and approving PRs


ModuleNotFoundError: No module named 'ldm.models'; 'ldm' is not a package & Is Conitive Face API Required?

I'm having an issue running this using what's available here. I've gotten this far but I'm just not sure how to proceed at this point. Is cognitive face API required? can it be removed? I'd personally rather not have every single sample going to microsoft if I could personally help it.
   ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
 NVIDIA GeForce RTX 2080 Ti

Python: 3.10.4

you installed the wrong package: ldm from pypi is not the right package.

I'm trying to follow your directions of cloning the repo and pip installing like so: pip install git+https://github.com/CompVis/latent-diffusion.git@main#egg=ldm but I get an error:
WARNING: Discarding git+https://github.com/CompVis/latent-diffusion.git@main#egg=ldm. Requested latent-diffusion from git+https://github.com/CompVis/latent-diffusion.git@main#egg=ldm has inconsistent name: filename has 'ldm', but metadata has 'latent-diffusion'
ERROR: Could not find a version that satisfies the requirement ldm (unavailable) (from versions: 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.6, 0.0.7, 0.0.8, 0.0.9, 0.1.0, 0.1.1, 0.1.3)
ERROR: No matching distribution found for ldm (unavailable)

Ideas?

I'd advice you to clone the repository and then install locally. Is that what you already did?

@jdluzen he's saying clone then do pip install . also https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/index-identity if you want the face api.
@ben-selas @limiteinductive I'm getting the same error sorry I don't fully understand do we have to install: https://github.com/CompVis/latent-diffusion/tree/main/models/ldm from the main model then run the sample on glide?
 When you open your python console and type import ldm does it throw an error?

@limiteinductive It's not installed. As the setup in the latent diffusion doesn't contain the install for the package: https://github.com/CompVis/latent-diffusion/blob/main/setup.py

@limiteinductive It's not installed. As the setup in the latent diffusion doesn't contain the install for the package: https://github.com/CompVis/latent-diffusion/blob/main/setup.py
"ldm from pypi is not the right package" Where do we get the correct ldm?

Did you try installing it using conda and the environment.yaml file like advertised in the Readme of the CompVis package?

:shame: I installed Conda but forgot to actually run the yaml... Thank you for your contribution and time!

@limiteinductive PROMISE I will leave you alone after this. I installed ld with conda and I can run the ld sample, but now trying to run glide:
(ldm) PathTo\glid-3-xl>python sample.py --model_path finetune.pt --batch_size 6 --num_batches 6 --text "a cyberpunk girl with a scifi neuralink device on her head"
Traceback (most recent call last):
  File "sample.py", line 19, in <module>
    from dalle_pytorch import DiscreteVAE, VQGanVAE
ModuleNotFoundError: No module named 'dalle_pytorch'```

Do I run this in the ldm env?

@wes-kay ha ha, don't worry. Yes sure you can install it. If it bugs it means you probably need to install a newer version of pytorch in your ldm env. We should probably refactor the glid-3-xl to have a better install script

Thank you for all your help it's working now :D Had to update pytorch conda install pytorch torchvision -c pytorch

Okay fixed that, with a re install and update of torch, now getting
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 4.00 GiB total capacity; 3.42 GiB already allocated; 0 bytes free; 3.49 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

:D

@wes-kay bad news: your GPU is doesn't have enough VRAM. You should use a service like google colab, or kaggle notebooks if you wanna run this package

@limiteinductive Thought as much, final question, I spun up a deep learning EC2

but getting:
/glid-3-xl$ python sample.py --clip_guidance --model_path finetune.pt --batch_size 1 --num_batches 12 --text "a cyberpunk girl with a scifi neuralink device on her head | trending on artstation" Using device: cpu Killed
Do I need higher specs?

yeah it seems so. You probably won't get a GPU for less than 0.25/hour

you installed the wrong package: ldm from pypi is not the right package. You should clone this repo https://github.com/CompVis/latent-diffusion and pip install .

I still have the same problem and I have done pip install . locally. This is the error:
eprint(line:60) :: Error when calling Cognitive Face API:
status_code: 401
code: 401
message: Access denied due to invalid subscription key or wrong API endpoint. Make sure to provide a valid key for an active subscription and use a correct regional API endpoint for your resource.
eprint(line:60) :: img_url:https://raw.githubusercontent.com/Microsoft/Cognitive-Face-Windows/master/Data/detection1.jpg


Ecommerce Traking

Hey, i could not found the ecommerce traking in docs so kindly include it as soon as possible because without it, our goal is incomplete according to the current running project.
Thanks

Hi @Hamidkhanswb, i will look into it. Can you link me some Matomo docs concerning that ecommerce tracking so that I can work with it? Thanks.

Hi @Hamidkhanswb, can you please elaborate on this one?


RenameVariableToCamelCase misses names

fixed by https://github.com/openrewrite/rewrite-kotlin/pull/185


Support html in output

I wanted to write an extension that converts urls into clickable links. But it looks like there's no way to render html.

Related to https://github.com/zeit/hyperterm/issues/101

Maybe it's a bad idea to modify the output. We can also convert the links on hover or CMD+hover.
Like iTerm is doing it.

+1

See this: https://github.com/zeit/hyperlinks

Having some sort of special API for rendering HTML would be (like rendering colors) would be super awesome, e.g. I really really want to be able to link to github pull requests from #123 in my github pull requests lists (custom script I wrote).

I think we can close it.
@Dorian I think it can be possible to do this with some ANSI escape code. Xterm uses this to let scripts change window/term title. A plugin can intercept this sequences and use meta data inside (like URL)


fix release src cannot compile bug

fix release src cannot compile bug

Seems I can't still compile from the src.tgz made from this branch?

maybe could compile on my mac.

@EvanLjp see my update 👆

@kezhenxu94 @EvanLjp I think this is why we use a shell rather than a simple make file to build src release.
@kezhenxu94 didn't do git submodule init/update, but run make release-src, then still src is not complete.

https://github.com/apache/skywalking/blob/master/tools/releasing/create_source_release.sh Like I asked, take a look at the main repo, the whole process matters.

@kezhenxu94 @EvanLjp I think this is why we use a shell rather than a simple make file to build src release.
@kezhenxu94 didn't do git submodule init/update, but run make release-src, then still src is not complete.

protocol files have been already added to src release.

protocol files have been already added to src release.

Try this, use an empty folder, do git clone, then make. Your tar ball is broken.

You are feeling good, because you use your dev env.


Update dependencies

I notice there is no Cargo.lock tracked in this repo; I recommend that it be done, so as to allow people to keep installing and using the binaries through the --locked flag on cargo install, should some dependency incorrectly introduce a breaking change within a semver-compatible version change (which, whilst rare, does happen).
And, in order to guarantee the Cargo.lock file is in sync with the Cargo.toml, I recommend that a
cargo update -w --locked step be added to CI.

Thanks for this suggestion, @danielhenrymantilla. I'm going to merge this PR and open an issue so that I can think about it some more.


Current maintenance state

Hi!
I see that for a long time there is no activity in this repo. Is the library still maintained? If not maybe will be a good idea to share access rights to the repo with some volunteers and keep this crate maintained.
Thanks in advance!
Related issue: https://github.com/dotenv-rs/dotenv/issues/60
CC @ZoeyR

It would be nice if someone looked at #72 to fix the IDE/Rust Analyzer error
unresolved macro proc_macro_call!

I have written @ZoeyR a mail a few weeks ago, though I have not received any response. I would love to help out though

It looks like up to 11 people have fixes/improvements ready:
https://github.com/dotenv-rs/dotenv/network
I also wrote an email to @ZoeyR, today.

I've created a fork called dotenvy. It's published to crates.io as:

dotenvy
dotenvy_codegen
dotenvy_codegen_impl

I've merged in many of the PRs from this repo.
This fork is intended to serve as the development home for the dotenv implementation in Rust.
Legend has it that the Lost Maintainer will return, merging changes from dotenvy into dotenv with such thrust that all Cargo.tomls will lose one keystroke. Only then shall the Rust dotenv crateverse be united in true harmony.
Until then, this repo dutifully carries on the dotenv torch. It will be actively maintained. Contributions and PRs are very welcome!
[0.15.1] - 2022-02-28
Added

dotenv crate forked as dotenvy
dotenv_codegen forked as dotenvy_codgen
dotenv_codegen_implementation forked as dotenvy_codegen_impl
Crate description for dotenvy_codegen
Crate description for dotenvy_codgen_impl
New language in README
MIT license badge in README
Generate helpful errors from dotenv! macro (full merge of dotenv-rs/dotenv #58)

Changed

replaced deprecated std::env_home:dir() with dirs:home_dir
Better handling of home_dir (merge of dotenv-rs/dotenv #62)
assertions dealing with Result (based on dotenv-rs/dotenv #57)
upgraded clap in dotenvy bin from v2 to v3.1 (covers dotenv-rs/dotenv #76)

Removed

example folder. The simple example has been moved to the README.
extern
unnecessary use statements in doc examples

ehh wow, cool @allan2 , thank you! :-)
would you be interested/willing in additional maintainers (e.g. @somehowchris)? not that it's necessary .. I am happy with there being an active fork already.. thanks for mentioning it here!

ouh and .. maybe also mention it in all the PRs in this repo which you already merged, to flock more crowd over to your fork.

I did that now. :-)
Did you already add  @somehowchris as backup maintainer on your repo?

Thanks for the support @hoijui.
I'm definitely willing to add more maintainers. I don't think there's a need right now though.
If the fork picks up, I'll open a Discussion on the new repo to ask who's interested!


mix.json

Hello, when I want to train the model, it has a error that I don't have a mix.json. How can I create the mix.json?

https://github.com/JusperLee/CTCNet/blob/main/local/preprocess_lrs2.py
use this code


High-level Architecture diagrams

We should add architecture diagrams that clarifies at least the following relations:

LazyLedger layer-1 (including block producers/validators)
application specific sidechain dealing with state (and maybe block producers of that chain)
end-user client submitting Tx

The first version of this can be a very simple draft. Point 3) could also be skipped in a first diagram if that simplifies things. Additionally, more detailed diagrams explaining all touching points and interactions between all three would be great.

I think this is still useful but these diagrams are not necessary to live in the specs repository but for developer focused documentation.


add reverse option for leaderboard

Some game like racing, sort leaderboard in reverse order. The reverse option may come in handy.

Thank you for the contribution! I merged it into master via patch instead of the button. I will release a new version of the gem on 2012-02-22.


advance song skips 1 song

hi - i set my hotkey up as Alt+Shift+. to advance a spotify track. it defintely advances past the track that's supposed to follow. so it skips past a song right after using the key combo.

Hey,
I just tried it out and the binding successfully advances to the next track, without "skip over" the next track. Please check you're not sending it twice accidentally!

you're right. i had at multiple tabs of Spotify open. i've closed a few so one remains and it's fine. TY


[FEAT] Support Yarn 2 (specifically Plug n Play)

Is your feature request related to a problem? Please describe.
I see in the documentation that currently only Yarn 1 is supported by Refine:

I tested this a bit myself, and was in fact able to get a Refine project to run using Yarn Modern, but where it breaks down is in attempting to use the Plug 'n' Play. Specifically the first issue is here, where various .bin/ scripts that aren't listed as dependencies of the project are imported.
Are there any plans on the roadmap to add support for PNP?
Describe alternatives you've considered
No response
Additional context
No response
Describe the thing to improve
I'd like to be able to use Refine in a project that's using yarn Plug n Play!

Hello @s1anden unfortunately, I don't have enough context atm, but would like to do some research and get back to you in couple of days. In the meantime, if you would like to contribute, feel free to do it! We are more than happy to assist you in the process.

@achalagarwal, what's the error you're facing here? Can you give a bit more context about the problems you're having? I'm not sure if this related with the plug-n-play feature of yarn v2 or not 🤔

Seems to be an issue with resolving the .bin dependency. Maybe I have to yarn dlx patch:add pathToRefineBin. Don't know. Will continue using npm for now
Error: @refinedev/cli tried to access .bin, but it isn't declared in its dependencies; this makes the require call ambiguous and unsound.
Required package: .bin (via ".bin/vite")

Traces:
/home/B/A/.pnp.cjs:28668
Error.captureStackTrace(firstError);
^
Error: @refinedev/cli tried to access .bin, but it isn't declared in its dependencies; this makes the require call ambiguous and unsound.
Required package: .bin (via ".bin/vite")
Required by: @refinedev/cli@npm:2.16.16 (via /home/B/A/.yarn/cache/@refinedev-cli-npm-2.16.16-6fa370b561-afaaa2ab71.zip/node_modules/@refinedev/cli/dist/)
Require stack:

/home/B/A/.yarn/cache/<EMAIL_ADDRESS>at require$$0.Module._resolveFilename (/home/B/A/.pnp.cjs:28668:13)
at Function.resolve (node:internal/modules/cjs/helpers:127:19)
at Object.getBin (/home/B/A/.yarn/cache/@refinedev-cli-npm-2.16.16-6fa370b561-afaaa2ab71.zip/node_modules/@refinedev/cli/dist/cli.js:4:953)
at Command.mo (/home/B/A/.yarn/cache/@refinedev-cli-npm-2.16.16-6fa370b561-afaaa2ab71.zip/node_modules/@refinedev/cli/dist/cli.js:17:1347)
at Command.listener [as _actionHandler] (/home/B/A/.yarn/cache/commander-npm-9.4.1-9deffb2749-bfb18e325a.zip/node_modules/commander/lib/command.js:482:17)
at /home/B/A/.yarn/cache/commander-npm-9.4.1-9deffb2749-bfb18e325a.zip/node_modules/commander/lib/command.js:1264:65
at /home/B/A/.yarn/cache/commander-npm-9.4.1-9deffb2749-bfb18e325a.zip/node_modules/commander/lib/command.js:1155:33
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Hello @achalagarwal thanks for the inspection! Would you like to work on this issue with your findings?

Had I known the solution, I would have pushed a fix 😔


List secondary value

Closes #3180
Summary

Support secondaryValue prop an object with:

      {
         value: PropTypes.func,
         label: PropTypes.string,
      }

Change List (commits, features, bugs, etc)

fix(listitem): support secondary value obj with custom label when func
chore(test): snapshots update

Acceptance Test (how to verify the PR)

Go to ListItem with object and render should keep working

Regression Test (how to make sure this PR doesn't break old functionality)

N/A no changes to the current behaviour

Things to look for during review

[ ] Make sure all references to iot or bx class prefix is using the prefix variable
[ ] (React) All major areas have a data-testid attribute. New test ids should have test written to ensure they are not changed or removed.
[ ] UI should be checked in RTL mode to ensure the proper handling of layout and text.
[ ] All strings should be translatable.
[ ] The code should pass a11y scans (The storybook a11y knob should show no violations). New components should have a11y test written.
[ ] Unit test should be written and should have a coverage of 90% or higher in all areas.
[ ] All components should be passing visual regression test. For new styles or components either a visual regression test should be written for all permutations or the base image updated.
[ ] Changes or new components should either write new or update existing documentation.
[ ] PR should link and close out an existing issue

Looks like the unit tests are failing

@bjornalm bump


tinygo: disable build on hydra

Description of changes

disable build on hydra

Since update to 0.31.1 tinygo does not build on hydra because of "Output limit exceeded" (src is too big):
https://hydra.nixos.org/job/nixos/trunk-combined/nixpkgs.tinygo.x86_64-linux
https://hydra.nixos.org/build/268960641
Most likely this change to lib/cmsis-svd drastically increased submodule size between 0.30.0 and 0.31.1:
https://github.com/tinygo-org/tinygo/pull/4007
https://github.com/tinygo-org/tinygo/pull/4030
Current src size on unstable:
6.3G    /nix/store/0qszq6dmvg1vd8gzjlmax73k5qynhpsb-source
6.3G    /nix/store/0qszq6dmvg1vd8gzjlmax73k5qynhpsb-source/lib
5.6G    /nix/store/0qszq6dmvg1vd8gzjlmax73k5qynhpsb-source/lib/cmsis-svd
5.6G    /nix/store/0qszq6dmvg1vd8gzjlmax73k5qynhpsb-source/lib/cmsis-svd/data
280M    /nix/store/0qszq6dmvg1vd8gzjlmax73k5qynhpsb-source/lib/CMSIS
274M    /nix/store/0qszq6dmvg1vd8gzjlmax73k5qynhpsb-source/lib/CMSIS/CMSIS
179M    /nix/store/0qszq6dmvg1vd8gzjlmax73k5qynhpsb-source/lib/stm32-svd
165M    /nix/store/0qszq6dmvg1vd8gzjlmax73k5qynhpsb-source/lib/stm32-svd/svd
98M     /nix/store/0qszq6dmvg1vd8gzjlmax73k5qynhpsb-source/lib/mingw-w64
71M     /nix/store/0qszq6dmvg1vd8gzjlmax73k5qynhpsb-source/lib/mingw-w64/mingw-w64-headers

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

tinygo listed maintainers:
@Madouura
@muscaln

I have an alternative approach to this by repairing the hydra build here:

#336882

I have an alternative approach to this by repairing the hydra build here:
* [tinygo: repair hydra build #336882](https://github.com/NixOS/nixpkgs/pull/336882)

If tinygo builds/works fine then it seems to be a much better way.

Closed in favor of #336882


Shedlock Support for Spring Cloud Function @PollableBean

Is your feature request related to a problem? Please describe.
The Spring cloud function was released while ago, and with it, many code bases are shifting to a more functional-style methods, this entailed some changes to the normal scheduled job, for example, a normal Scheduled job that emits elements to a Kafka topic for example would look like this
@Scheduled(cron=..)
public void runScheduledJob(){
    boolean result = subscribableChannel.send("hello world");
    log.info("The message sending operation result was: "+result);
}

Now, using Spring cloud function, a bean with return type Supplier and annotated as @Pollable can replace this function as follows:
@PolllableBean
public Supplier<String> getScheduledJob(){
    return () -> return "hello world";
}

Please note that the cron for this can be configured within the application.properties file using spring.cloud.stream.poller.cron=0 0/10 * * * *
The issue here as probably notices is that Shedlock annotation would not work here, and although you are free to add it, however it would not only be useless, but shall in case of there is a lock on the method prevent the bean for being registered (I have already tried that https://stackoverflow.com/questions/70623688/an-error-creating-bean-with-name-functionbindingregistrar-exception-every-ti/70671570#70671570 and asked if there is a solution here)
Describe the solution you'd like
It would always be easy to say: Could you please add support for Spring Cloud function suppliers? I don't have any solution that I can think of
Describe alternatives you've considered
The only way around is to revert back to the good old normal method and dump spring cloud function Supplier implementation all together
Additional context
None

Hi, this is really interesting. I see two ways of solving this.

The most straigntforward is to use AOP on another method

@PolllableBean
public Supplier<Void> getScheduledJob(){
    return otherBean::doIt;
}

class OtherBean {
  @SchedulerLock(...)
  public void doIt() {
    ....
  }
}

please note that the method is implemented on different class to allow AOP to work. Also I had to change the return type to Void as we would not know what to return in case the lock is locked.

The other option is to use locking without framework

Thanks for you quick turn around Lukas, I have tried the first approach, however, it seems that the lock is not activated, please find a running sample of the code here https://github.com/Mahm0ud/ShedlockWithSpringCloudFunction.

By default ShedLock is using Spring AOP. In order for it to work, the ShedulerLock annotation has to be on a bean (a class instantiated by Spring). Your ScheduledTask class is not a bean.