Segmentation fault on Apple Silicon / arm64

Building mimalloc fails in the TLS initialization with a segmentation fault:
% lldb ./mimalloc-test-api
(lldb) target create "./mimalloc-test-api"
Current executable set to '/Users/uwe/Development/mimalloc/build/mimalloc-test-api' (arm64).
(lldb) run
Process 17703 launched: '/Users/uwe/Development/mimalloc/build/mimalloc-test-api' (arm64)
mimalloc-test-api was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 17703 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x000000010001f264 mimalloc-test-api`_mi_process_init [inlined] mi_tls_slot(slot=0) at mimalloc-internal.h:711:9 [opt]
   708 	#elif defined(__aarch64__)
   709 	  void** tcb; UNUSED(ofs);
   710 	  __asm__ volatile ("mrs %0, tpidr_el0" : "=r" (tcb));
-> 711 	  res = tcb[slot];
   712 	#endif
   713 	  return res;
   714 	}
Target 0: (mimalloc-test-api) stopped.

Out of curiosity, would the 'dev' branch still fails ?

Fails with exactly the same issue:
Process 18154 launched: '/Users/uwe/Development/mimalloc/build/mimalloc-test-api' (arm64)
mimalloc-test-api was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 18154 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x000000010001efa0 mimalloc-test-api`_mi_process_init [inlined] mi_tls_slot(slot=0) at mimalloc-internal.h:712:9 [opt]
   709 	#elif defined(__aarch64__)
   710 	  void** tcb; UNUSED(ofs);
   711 	  __asm__ volatile ("mrs %0, tpidr_el0" : "=r" (tcb));
-> 712 	  res = tcb[slot];
   713 	#endif
   714 	  return res;
   715 	}
Target 0: (mimalloc-test-api) stopped.
(lldb) p slot
(size_t) $0 = 0
(lldb) p tcb
(void **) $1 = 0x0000000000000000

weird because the assembly is supposed to be different.

weird because the assembly is supposed to be different.

As far as I see in the diff to master, some of the x86 assembly has changed but the aarch64 assembly stayed the same. As the x86_64 code has special casings for __MACH__, I guess we need that for __MACH__ && aarch64`, too.

Right ... what happens if you use this change :
uintptr_t tcb; UNUSED(ofs);UNUSED(slot);   __asm__ volatile ("mrs %0, tpidr_el0" : "=r" (tcb));   res = tcb;
Just theorical, I wish I had a M1 to test for real :-)

Seems to pass and now faults in mi_tls_slot_set where we have the near exact same code path.

Yes it would be similar-ish solution if it works like :
uintptr_t tcb; UNUSED(ofs);UNUSED(slot)   __asm__ volatile ("mrs %0, tpidr_el0" : "=r" (tcb));   tcb = (uintptr_t)value;
Thanks for your patience :-)

With MI_MALLOC_OVERRIDE=OFF, ./mimalloc-test-api now passes.
With MI_MALLOC_OVERRIDE=ON, we now get to:
(lldb) run
Process 27086 launched: '/Users/uwe/Development/mimalloc/build/mimalloc-test-api' (arm64)
Process 27086 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xb0)
    frame #0: 0x0000000180cbc6cc libsystem_platform.dylib`_platform_memset + 108
libsystem_platform.dylib`_platform_memset:
->  0x180cbc6cc <+108>: stp    q0, q0, [x0]
    0x180cbc6d0 <+112>: stp    q0, q0, [x0, #0x20]
    0x180cbc6d4 <+116>: add    x3, x0, #0x40             ; =0x40
    0x180cbc6d8 <+120>: and    x3, x3, #0xffffffffffffffc0
Target 0: (mimalloc-test-api) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xb0)
  * frame #0: 0x0000000180cbc6cc libsystem_platform.dylib`_platform_memset + 108
    frame #1: 0x0000000100008038 mimalloc-test-api`mi_stats_reset at stats.c:359:35 [opt]
    frame #2: 0x000000010001ec2c mimalloc-test-api`mi_process_init at init.c:476:3 [opt]
    frame #3: 0x000000010007377c dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 468
    frame #4: 0x0000000100073b94 dyld`ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 56
    frame #5: 0x000000010006d84c dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 620
    frame #6: 0x000000010006b300 dyld`ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 192
    frame #7: 0x000000010006b3cc dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 96
    frame #8: 0x000000010005684c dyld`dyld::initializeMainExecutable() + 220
    frame #9: 0x000000010005cb98 dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) + 7388
    frame #10: 0x0000000100055258 dyld`dyldbootstrap::start(dyld3::MachOLoaded const*, int, char const**, dyld3::MachOLoaded const*, unsigned long*) + 476
    frame #11: 0x0000000100055038 dyld`_dyld_start + 56

Too bad the changes work on Linux arm64 maybe the M1 gear needs a specific code path in this case.
Just an idea, what happens if you comment line 298 in include/mimalloc-internal.h
Just wanted to see while mimalloc uses the thread slot maybe that s not such good thing for the M1 case. Maybe I m wrong tough, lot of maybe at the moment :-)

When I comment line 298, the mimalloc-api-test passes but the mimalloc-test-stress still fails which seems to be related to the commenting of line 298.
(lldb) run
Process 30829 launched: '/Users/uwe/Development/mimalloc/build/mimalloc-test-stress' (arm64)
Process 30829 stopped
* thread #1, stop reason = signal SIGABRT
    frame #0: 0x0000000180c43cec libsystem_kernel.dylib`__pthread_kill + 8
libsystem_kernel.dylib`__pthread_kill:
->  0x180c43cec <+8>:  b.lo   0x180c43d0c               ; <+40>
    0x180c43cf0 <+12>: pacibsp
    0x180c43cf4 <+16>: stp    x29, x30, [sp, #-0x10]!
    0x180c43cf8 <+20>: mov    x29, sp
Target 0: (mimalloc-test-stress) stopped.
(lldb) bt
* thread #1, stop reason = signal SIGABRT
  * frame #0: 0x0000000180c43cec libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x0000000180c74c24 libsystem_pthread.dylib`pthread_kill + 292
    frame #2: 0x0000000180bbc864 libsystem_c.dylib`abort + 104
    frame #3: 0x0000000180c93d58 libdyld.dylib`_tlv_bootstrap + 16
    frame #4: 0x000000010012b208 libmimalloc-debug.1.6.dylib`mi_calloc [inlined] mi_get_default_heap at mimalloc-internal.h:346:10 [opt]
    frame #5: 0x000000010012b1f8 libmimalloc-debug.1.6.dylib`mi_calloc(count=<unavailable>, size=80) at alloc.c:584 [opt]
    frame #6: 0x000000018a2b5a7c libkeymgr.dylib`dwarf2_unwind_dyld_add_image_hook + 40
    frame #7: 0x0000000100018d94 dyld`dyld::registerAddCallback(void (*)(mach_header const*, long)) + 400
    frame #8: 0x0000000180c7b1c8 libdyld.dylib`_dyld_register_func_for_add_image + 132
    frame #9: 0x000000018a2b5798 libkeymgr.dylib`__keymgr_initializer + 32
    frame #10: 0x000000018a2c67d0 libSystem.B.dylib`libSystem_initializer + 196
    frame #11: 0x000000010003390c dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 868
    frame #12: 0x0000000100033b94 dyld`ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 56
    frame #13: 0x000000010002d84c dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 620
    frame #14: 0x000000010002d794 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 436
    frame #15: 0x000000010002d794 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 436
    frame #16: 0x000000010002b300 dyld`ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 192
    frame #17: 0x000000010002b3cc dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 96
    frame #18: 0x000000010001684c dyld`dyld::initializeMainExecutable() + 220
    frame #19: 0x000000010001cb98 dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) + 7388
    frame #20: 0x0000000100015258 dyld`dyldbootstrap::start(dyld3::MachOLoaded const*, int, char const**, dyld3::MachOLoaded const*, unsigned long*) + 476
    frame #21: 0x0000000100015038 dyld`_dyld_start + 56

Right so maybe M1 needs a specific code path (specific register like x86 case ?). I see on other projects issues popping up about M1 here and there seems to be really a particular animal.

Hi @xhochy , lucky you with an M1 :-)  Thanks @devnexen for looking into this.
Unfortunately, there have been various issues on macOS and it may be related to that and not directly to the M1; I currently have a mac mini to develop on so I will first go through previous issues and then get back to this. The main trouble on macOS is that the loader itself (dyldImageLoader) calls mallocto allocate memory for thread local storage -- causing an infinite loop if any thread-local storage is accessed. Currently, we avoid this by using a dedicated thread slot but that is not guaranteed to work on future OS's and may be part of the problem -- maybe you can try another slot number? a few good ones are in the comments inmimalloc-internal.h`.
If you get to it, can you submit the assembly changes together with the correct pre-processor test for the M1?
Hope we can get this working!
(I tested mimalloc extensively on aarch64 on Linux, including long TSAN tests and there was recent work with model-checking the atomics so pretty sure that part is working ok)

@xhochy, I just pushed an update for macOS ; can you try again with the latest dev ?   (may not make a difference but then we know at least it is not a bug in the malloc zones overriding)

@xhochy, I just pushed an update for macOS ; can you try again with the latest dev ? (may not make a difference but then we know at least it is not a bug in the malloc zones overriding)

Latest dev didn't make a difference. Exactly the same behaviour as before.

Currently, we avoid this by using a dedicated thread slot but that is not guaranteed to work on future OS's and may be part of the problem -- maybe you can try another slot number?

Also didn't make any difference, with or without the suggestions by @devnexen.
In general this looks like the same issue as people are facing on iOS (see https://github.com/microsoft/mimalloc/issues/262 and I tried to use the fixes suggested there.
Thus I defined MI_TLS_RECURSE_GUARD in my C(XX)FLAGS and changed the tcb acess to happen when tcb is not-NULL. Also I commented out MI_TLS_SLOT.
This then leads to a slight different error:
(lldb) run
Process 61782 launched: '/Users/uwe/Development/mimalloc/build/mimalloc-test-stress' (arm64)
objc[61782]: realized class 0x1e8e712c8 has corrupt data pointer 0x40000040090
Process 61782 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
    frame #0: 0x0000000180c64130 libsystem_kernel.dylib`__abort_with_payload + 8
libsystem_kernel.dylib`__abort_with_payload:
->  0x180c64130 <+8>:  b.lo   0x180c64150               ; <+40>
    0x180c64134 <+12>: pacibsp
    0x180c64138 <+16>: stp    x29, x30, [sp, #-0x10]!
    0x180c6413c <+20>: mov    x29, sp
Target 0: (mimalloc-test-stress) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x0000000180c64130 libsystem_kernel.dylib`__abort_with_payload + 8
    frame #1: 0x0000000180c66a20 libsystem_kernel.dylib`abort_with_payload_wrapper_internal + 104
    frame #2: 0x0000000180c669b8 libsystem_kernel.dylib`abort_with_reason + 32
    frame #3: 0x0000000180b311dc libobjc.A.dylib`_objc_fatalv(unsigned long long, unsigned long long, char const*, char*) + 120
    frame #4: 0x0000000180b31164 libobjc.A.dylib`_objc_fatal(char const*, ...) + 44
    frame #5: 0x0000000180b20f7c libobjc.A.dylib`realizeClassWithoutSwift(objc_class*, objc_class*) + 112
    frame #6: 0x0000000180b21048 libobjc.A.dylib`realizeClassWithoutSwift(objc_class*, objc_class*) + 316
    frame #7: 0x0000000180b21060 libobjc.A.dylib`realizeClassWithoutSwift(objc_class*, objc_class*) + 340
    frame #8: 0x0000000180b21048 libobjc.A.dylib`realizeClassWithoutSwift(objc_class*, objc_class*) + 316
    frame #9: 0x0000000180b0b564 libobjc.A.dylib`_read_images + 2684
    frame #10: 0x0000000180b0a44c libobjc.A.dylib`map_images_nolock + 2468
    frame #11: 0x0000000180b1b770 libobjc.A.dylib`map_images + 92
    frame #12: 0x000000010001a16c dyld`dyld::notifyBatchPartial(dyld_image_states, bool, char const* (*)(dyld_image_states, unsigned int, dyld_image_info const*), bool, bool) + 1672
    frame #13: 0x000000010001a358 dyld`dyld::registerObjCNotifiers(void (*)(unsigned int, char const* const*, mach_header const* const*), void (*)(char const*, mach_header const*), void (*)(char const*, mach_header const*)) + 80
    frame #14: 0x0000000180c7bd88 libdyld.dylib`_dyld_objc_notify_register + 164
    frame #15: 0x0000000180b09a08 libobjc.A.dylib`_objc_init + 1384
    frame #16: 0x0000000180ac7768 libdispatch.dylib`_os_object_init + 24
    frame #17: 0x0000000180ad632c libdispatch.dylib`libdispatch_init + 476
    frame #18: 0x000000018a2c67e8 libSystem.B.dylib`libSystem_initializer + 220
    frame #19: 0x000000010003390c dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 868
    frame #20: 0x0000000100033b94 dyld`ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 56
    frame #21: 0x000000010002d84c dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 620
    frame #22: 0x000000010002d794 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 436
    frame #23: 0x000000010002d794 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 436
    frame #24: 0x000000010002b300 dyld`ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 192
    frame #25: 0x000000010002b3cc dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 96
    frame #26: 0x000000010001684c dyld`dyld::initializeMainExecutable() + 220
    frame #27: 0x000000010001cb98 dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) + 7388
    frame #28: 0x0000000100015258 dyld`dyldbootstrap::start(dyld3::MachOLoaded const*, int, char const**, dyld3::MachOLoaded const*, unsigned long*) + 476
    frame #29: 0x0000000100015038 dyld`_dyld_start + 56

I guess that the fix for the M1 and iOS will be the exact same as they both use similar execution models and the same chip design ;)

I just did a run with an iPhone 6 :

with the actual dev but with these little changes already mentioned

Yes, that also works me for. Only mimalloc-test-stress is failing for me.

Alright little thing I found out, the assembly instruction for getting the tls value et setting it are likely incorrect for darwin devices, I debugged and returns always 0 however below seems workable :
void** tcb; UNUSED(ofs);   __asm__ volatile ("mrs %0, tpidrro_el0" : "=r" (tcb));   res = tcb[slot]; ... void** tcb; UNUSED(ofs);   __asm__ volatile ("mrs %0, tpidrro_el0" : "=r" (tcb));     tcb[slot] = value;
tcb not null in my side thus the guard later is likely unnecessary. However that does not fix everything but at least that might highlight why tls did not work in the first place.

Was wondering if, only in the case of darwin ARM devices, we would use, tpidrr0_el0 TID register instead ? tpidr_el0 always return 0 otherwise.

Hmm, I wish I had an M1 to test with; It is always tricky on the mac as the loader calls malloc to initialize thread local state but with the right tcb slot it should be ok. Perhaps the tpidrr0_el0 register should be used as @devnexen suggests?
This is tricky to debug but it would be great if you can find the culprit :-)  Without an M1 I available it is hard for me to guess what is going wrong :-(

tpidrro_el0 seems to work fine on macOS, Linux only seems to set tpidr_el0 so I special cased that.

They're completely unrelated registers that are both available. Linux and FreeBSD leave it entirely up to userspace to manage TLS how it sees fit, and so userspace needs a writable register not a read-only one. On macOS/Mach-O TLS used to be more bolted on the side (and came extremely late) unlike the implementation for ELF, and it appears it still is. But aside from that, my guess is that for dubious "security" reasons they decided to have the kernel play a larger part in controlling userspace's TLS and, whilst userspace can read it quickly, writing it needs to go through the kernel (which isn't exactly a hot path, but also there's not much good reason for it, it's just unnecessary overhead, especially since you still need to context switch tpidr_el0 given it's writable, yet if you have userspace use that one you can avoid needing to context switch tpidrro_el0). The existence of both in the Arm architecture is a bit of a wart that should never have happened (and especially not with so similar names), but here we are.
So, both choose which register to use based on their requirements and design. Neither is right or wrong, nor is either a special case, they're just different. Having said that, macOS is the odd one out for the Unixes.

They're completely unrelated registers that are both available. Linux and FreeBSD leave it entirely up to userspace to manage TLS how it sees fit, and so userspace needs a writable register not a read-only one. On macOS/Mach-O TLS used to be more bolted on the side (and came extremely late) unlike the implementation for ELF, and it appears it still is. But aside from that, my guess is that for dubious "security" reasons they decided to have the kernel play a larger part in controlling userspace's TLS and, whilst userspace can read it quickly, writing it needs to go through the kernel (which isn't exactly a hot path, but also there's not much good reason for it, it's just unnecessary overhead, especially since you still need to context switch tpidr_el0 given it's writable, yet if you have userspace use that one you can avoid needing to context switch tpidrro_el0). The existence of both in the Arm architecture is a bit of a wart that should never have happened (and especially not with so similar names), but here we are.
So, both choose which register to use based on their requirements and design. Neither is right or wrong, nor is either a special case, they're just different. Having said that, macOS is the odd one out for the Unixes.

I believe we figured out a way to make this work -- thanks so much! Really nice to see how this could be patched and merged without me having an M1 available -- the power of the internet :-)  Thanks again!

I believe we figured out a way to make this work -- thanks so much! Really nice to see how this could be patched and merged without me having an M1 available -- the power of the internet :-)  Thanks again!

@daanx,
I bought an M1 for testing because I contribute to Open Source projects. You are welcomed to use it. Send over your authorized_keys for SSH access (sorry, public key only). My email is noloader, gmail account.
The GCC compile farm also has a M1 for testing. You can get an account at https://cfarm.tetaneutral.net/. See the "Request and Account" link at the top of the page.

Hi @xhochy  -- this has taken awhile but I think I fixed the issue in the latest releases (v1.7.2, v2.0.2).
The issue was two-fold; we need indeed the tldrro_el0 register but it seems we need to clear the bottom 3 bits; moreover, we need to not use interpose anymore; i think other fixes made the zones overriding now work reliably and on the M1 interpose causes some sort of problems now (not sure what).  With interpose off, everything seems to work :-)
I did some quick testing and mimalloc is often twice as fast (up to 10x in xmalloc-test) than the system allocator:
# --------------------------------------------------
# benchmark allocator elapsed rss user sys page-faults page-reclaims

cfrac sys 06.13 2208 6.12 0.00 4 304
cfrac xmi 03.61 2464 3.61 0.00 0 334
cfrac je 04.23 2864 4.22 0.00 0 364

espresso sys 05.26 2208 5.25 0.01 12 296
espresso xmi 03.73 4480 3.71 0.01 0 460
espresso je 04.06 3968 4.05 0.01 0 433

xmalloc-testN sys 7.854 49760 29.27 10.52 3 5632
xmalloc-testN xmi 0.601 116096 39.53 0.15 0 7452
xmalloc-testN je 0.794 70272 32.77 1.78 0 4596

mstressN sys 03.24 1061712 4.39 2.10 0 657980
mstressN xmi 01.65 717056 3.28 0.18 0 68190
mstressN je 04.05 732992 3.37 3.49 0 1327397

I tested the latest dev branch (commit 076f81) with -DMI_OSX_ZONE=ON -DMI_INTERPOSE=OFF, and both Release and Debug build  worked well.

any progress on it?

I also got it working on my machine by setting -DMI_OSX_ZONE=ON -DMI_INTERPOSE=OFF.
@daanx is this expected?


Is this project dead ?

Hi,
What is the sattus of your project ? I can see last commit happened 3 years ago. Can we still hope for some improvements ?
Thanks ! :)

unfortunately install no longer works. links dead etc. pity. Hope it gets refreshed


The theme color reference is a bit of a mess

I'm maintaining a theme with the stated goal that it themes every single themeable color. While developing and maintaining it I'm finding a few issues with the whole color reference thing.
Colors missing from the reference
My theme is currently theming the following colors:

breadcrumb.background
editorGroup.background

They are not even mentioned in the reference, I guess I got them from a release note, who knows.
Missing colors from release notes (maybe?)
In my theme's first release I've themed every single color, and I've always paid attention to the release notes for any added colors. Still, quite surprisingly to me, all these colors are now present in the reference but my theme is not customizing them:

diffEditor.border
diffEditor.insertedTextBorder
diffEditor.removedTextBorder
editor.findRangeHighlightBorder
editorOverviewRuler.commonContentForeground
editorOverviewRuler.rangeHighlightForeground
editorOverviewRuler.selectionHighlightForeground
editorRuler.foreground
editorUnnecessaryCode.border
editorUnnecessaryCode.opacity
editorWidget.resizeBorder
gitDecoration.addedResourceForeground
gitDecoration.submoduleResourceForeground
list.inactiveFocusBackground
list.invalidItemForeground
merge.border
merge.commonContentBackground
merge.commonHeaderBackground
notification.background
notification.buttonBackground
notification.buttonForeground
notification.buttonHoverBackground
notification.errorBackground
notification.errorForeground
notification.foreground
notification.infoBackground
notification.infoForeground
notification.warningBackground
notification.warningForeground
panel.dropBackground
peekViewEditor.matchHighlightBorder
statusBar.debuggingBorder
statusBar.noFolderBorder
tab.hoverBackground
tab.hoverBorder
tab.unfocusedHoverBackground
tab.unfocusedHoverBorder

Maybe I did a very poor job on my first release and left out all of them, but I think it's more likely that they have been added over time but they have not been mentioned in the release notes. If that's the case that's a problem, we can't be expected to keep monitoring the reference for updates.
Difficult to theme properties
Most properties are obvious, but some of them I didn't immediately understand what they were theming, like:

editor.focusedStackFrameHighlightBackground
editor.stackFrameHighlightBackground

Some more screenshots would be appreciated.
Also in order to theme some colors, like:

notification.errorBackground
notification.errorForeground
terminal.ansiBlack
terminal.ansiBlue

We need to see how they are actually being used, i.e. we need to trigger all kinds of custom notifications and have a script that uses all colors in the terminal. It might be good to develop some "official" scripts for this or at least link to third-party's scripts we could use.
Incomplete colors support
The whole set of provided themeable colors seems incomplete to me. It seems to me almost like unless a particular color in a particular state is not themed explicitly by VSCode itself then that color isn't themeable at all by themes.
One quick example, tabs now have a top border, and they can be in one of these states: active, inactive, unfocused-active, unfocused-inactive.
If we want to customize the top border for active tabs we have tab.activeBorderTop, for unfocused-active tabs we have tab.unfocusedActiveBorderTop, and the other states just aren't themeable (or I guess those properties aren't mentioned in the reference?).
Poorly searchable/manipulable properties
Coming up with these lists of properties was a bit painful, it would be better if there was a JSON file or something that listed all of them, so that I could diff my theme's settings with it. Also seeing what values are assigned to those properties by default would be useful. Maybe this file exists already somewhere in the repo, if it does it should be mentioned in the reference as well.

Colors missing from the reference

Should probably go there https://github.com/Microsoft/vscode-docs/issues

it would be better if there was a JSON file or something

https://github.com/Microsoft/vscode-docs/issues/1588
TLDR: Developer: Generate Color Theme From Current Settings should return JSON with all colors

Also in order to theme some colors, like:
notification.errorBackground
notification.errorForeground

https://marketplace.visualstudio.com/items?itemName=svipben.notification-tester

Another missing color: tab.hoverForeground, there doesn't seem to be any good reason to me for not adding this one, the whole thing is just very incomplete.

Another property difficult to customize: tab.hoverBorder. This one should be easy, but I'm not seeing any changes at all 🤷♂.

Another weird one: tab.hoverBackground (there might be a problem when all properties I try to customize have issues...). If one assigns a color to it then all tabs, even the active one will use that color when they are hovered. Now, one may think that by default this is assigned to a specific color, like (all?) the other colors. But not, if this property is not customized then tabs just don't change their background color on hover, totally undocumented, totally incomplete, there should be analogous properties for when the tab is also active etc.

totally incomplete, there should be analogous properties for when the tab is also active etc.

Weirdly there are properties for customizing unfocused tabs (unfocusedHoverBackground), which I would guess most themes won't care about using, but an activeHoverBackground, which actually makes sense, is not present.

Weirdness #∞+1: I tried setting editorUnnecessaryCode.opacity to 1, but the schema expects a string. Fine, I wronte "1" then, but the schema is actually excepting a color string, which doesn't make sense for the opacity.

editorUnnecessaryCode.opacity

I'd also like to know that the default value for this is. Unfortunately autocompleting it inserts the value "#ff0000"...

editorUnnecessaryCode.opacity

This doesn't seem to be working at all, whatever I set it to the opacity becomes 1, maybe the buggy schema thinks I'm setting an invalid value (I tried setting 0.5, .5 and askjdn) and defaults to "1"? 🤷♂

list.inactiveFocusBackground

The reference says "An active list has keyboard focus, an inactive does not. Currently only supported in lists." I haven't been able to see how this color is being used... I selected a list item via the keyboard, then switched the focus to an editor, nothing.

list.invalidItemForeground

The reference mentions "for example an unresolved root in explorer.", so I added a folder to the workspace and then deleted it, without removing it from the workspace, nothing happened. Apparently one has to first save the workspace, then reopen it again, weird.

Setting a transparent (#00000000) border as the value of editor.findRangeHighlightBorder strangely causes kind of a border to appear, I'm not quite sure why:

When setting the following properties:

diffEditor.insertedTextBorder
diffEditor.removedTextBorder

The added border is not added properly, since the whole highlight becomes misaligned. Here there's a before/after comparison:


Add relative bars into match columns

resolves #86

[ ] Add relative bars to columns and tables that should use them @howardchung
[ ] Consolidate current displayFn: TablePercent columns

[ ] Support for percent (i.e val / 100 currently we only do val / total)
[ ] Support for displaying total field

I'll work on this later tonight or tomorrow.
@howardchung can you figure out what columns are we adding this to?

Performances table
Farm (unit kills, last hits, gold sources, xp sources)
casts (maybe?)
objectives (objective damage, runes)
vision (bottom table, maybe)
actions table
teamfights tables (maybe)

Here's a wild idea. What if we make them appear only when the user hovers over column, this way we avoid clutter and can add them to all numeric fields. It also emphasizes the column the users hovers on.
BTW I'm not gonna be able to work on this till Friday 💯. Let me know if you wanna take over or something.

Rebasing in a bit

I just wanna put it out there, now sortFn provides the value to the field, if it doesn't exist then we looks at row[column.field]. so displayFn essentially becomes displayFn(row, column, sortFn(row)).
No breaking changes, but at some point we should rename it to valueFn or something.
Just noticed the conflict. I'm on it


Grant admin as host

The host should be able to grant admin to other players. These admin players would then be able to run console commands on the host.

While builtin mod support has been removed this is still relevant for allowing select players to perform administrative tasks when without physical access to the server machine

Still applicable, the new command "system" that has been around for a while now makes this much easier and safer. I need to abstract how commands print so that a command being run on the server can print to the console of the admin client which executed the command. Beyond tracking who is and isn't an admin this should be trivial.


[8051] Can't specify Carry flag as function return

Describe the bug
A common 8051 code pattern is to use the carry flag to return a success-or-failure result. Often this is the function's only return value.
It doesn't seem to be possible to specify this in the function definition ("Edit Function" window).
Code sample
CODE:01ae  D3    SETB CY   ; Set carry
CODE:01af  22    RET

CODE:018d  C3    CLR CY  ; Clear carry
CODE:018e  22    RET

It is possible to specify that the return value is stored in BITS:D7 (PSW.CY, the carry flag) but this doesn't show the setting and clearing of the carry flag -- only a volatile read of it, followed by a 'return'.
To Reproduce
Steps to reproduce the behavior:

Decompile the above code
Define a function at the SETB CY / CLR CY.
Specify the return value as BITS:D7, boolean, size 1.
Note that the set/clear of the carry flag is ignored by the decompiler but

Expected behavior
'return false' or 'return true' where the carry flag is set or cleared prior to a return.
Environment (please complete the following information):

OS: Ubuntu 18.04.3 LTS
Java Version: 11.0.5
Ghidra Version: 9.1

Have you tried ticking the custom storage checkbox? As long as the register doesn't have the hidden attribute you should be able to set it as the return with custom storage.

Have you tried ticking the custom storage checkbox? As long as the register doesn't have the hidden attribute you should be able to set it as the return with custom storage.

I have indeed -- sorry, I forgot that step from the details above.
If I try to specify a Register, CY isn't in the list I can choose from. PSW (of which CY is a constituent bit) is, so I tried that:
                             bool __stdcall CSYNC_READ(void)
             undefined         PSW:1          <RETURN>
                             CSYNC_READ                                  XREF[6]:     (xrefs redacted)
       CODE:0277 d2 80           SETB       P0.0_CARD_DAT                                    = ??
       CODE:0279 a2 80           MOV        CY,P0.0_CARD_DAT                                 = ??
       CODE:027b 92 a7           MOV        P2.7,CY                                          = ??
       CODE:027d a2 80           MOV        CY,P0.0_CARD_DAT                                 = ??
       CODE:027f 22              RET

And the decompiled output:
bool CSYNC_READ(void)

{
  byte bVar1;
  char cVar2;
  byte in_PSW;

  write_volatile_1(P0.0_CARD_DAT,1);
  bVar1 = read_volatile_1(P0.0_CARD_DAT);
  write_volatile_1(P2.7,bVar1 & 1);
  cVar2 = read_volatile_1(P0.0_CARD_DAT);
  return (bool)(in_PSW & 0x7f | cVar2 << 7);
}

Looking at the Atmel 8051 Microcontrollers Hardware Manual this is technically correct, but the shift-and-AND obfuscates the meaning somewhat.
If I specify the PSW.CY bit (BITS:d7) which should be equivalent to the byte-addressed form, I get this output:
bool CSYNC_READ(void)

{
  byte bVar1;
  bool bVar2;

  write_volatile_1(P0.0_CARD_DAT,1);
  bVar1 = read_volatile_1(P0.0_CARD_DAT);
  write_volatile_1(P2.7,bVar1 & 1);
  read_volatile(P0.0_CARD_DAT);
  bVar2 = (bool)read_volatile_1(C);
  return bVar2;
}

Which is completely incorrect. The result of the read_volatile appears to be thrown away, but in reality it's the return value.

Have you tried ticking the custom storage checkbox? As long as the register doesn't have the hidden attribute you should be able to set it as the return with custom storage.

I have indeed -- sorry, I forgot that step from the details above.
If I try to specify a Register, CY isn't in the list I can choose from. PSW (of which CY is a constituent bit) is... -snip- ...
bool CSYNC_READ(void)

{
  byte bVar1;
  bool bVar2;

  write_volatile_1(P0.0_CARD_DAT,1);
  bVar1 = read_volatile_1(P0.0_CARD_DAT);
  write_volatile_1(P2.7,bVar1 & 1);
  read_volatile(P0.0_CARD_DAT);
  bVar2 = (bool)read_volatile_1(C);
  return bVar2;
}

Which is completely incorrect. The result of the read_volatile appears to be thrown away, but in reality it's the return value.

I snipped out part of the large quote. It appears that CY is used as a macro in the sleigh files so specifying PSW.CY is likely what should be done.
The result there isn't being thrown away, it is reading one volatile bit and casting it to a bool. When PSW.CY is specified the decompiler is only taking what is stored in bit 7 to be the return value; true or false isn't being shown because the decompiler doesn't appear to know the value is a constant.
From the following:
https://github.com/NationalSecurityAgency/ghidra/blob/02df944b0e62c6d94f55f93b6682d70a444c705c/Ghidra/Processors/8051/data/languages/8051_main.sinc#L876-L885
The only two which would produce a constant the decompiler would utilize would be line 876 or 880 and I'm not certain which one SETB       P0.0_CARD_DAT would be.

I snipped out part of the large quote. It appears that CY is used as a macro in the sleigh files so specifying PSW.CY is likely what should be done.

Unfortunately Ghidra doesn't allow this. "PSW.CY" isn't available in the dropdown on the Storage Address Editor (Edit Function -> double-click Storage) screen.

The result there isn't being thrown away, it is reading one volatile bit and casting it to a bool.

The decompiled code suggests otherwise.
  read_volatile(P0.0_CARD_DAT);
  bVar2 = (bool)read_volatile_1(C);
  return bVar2;

This reads P0.0_CARD_DAT (a bit variable in the BITS segment -- actually a port pin), but discards the value. In reality, the value is in the carry flag but the decompiler output doesn't show this.
It then reads the carry flag and returns it -- which isn't inaccurate, but probably not too intuitive.
I'd have expected something more like this:
  bVar2 = (bool)read_volatile(P0.0_CARD_DAT);
  return bVar2;

The only two which would produce a constant the decompiler would utilize would be line 876 or 880 and I'm not certain which one SETB P0.0_CARD_DAT would be.

If it was a SETB instruction, it would be. Instead it's a move from a bit register to the carry bit, which is
https://github.com/NationalSecurityAgency/ghidra/blob/02df944b0e62c6d94f55f93b6682d70a444c705c/Ghidra/Processors/8051/data/languages/8051_main.sinc#L801-L806
BIT_OPS == "BIT_ADDRS" is set at the top of the file, so this is the only chunk in that block which matters.


xlsx vulnerability CVE-2023-30533

Hi,
The version of xlsx used in the latest version of alasql is 0.18.0 which has a vulnerability and it is recommended to use 0.19.3+ on sheetjs' CDN as described here: https://github.com/SheetJS/sheetjs/issues/2822
They've abandoned putting their latest versions on npm. While it is possible to address this with an override in our package.json, is there a plan to do this directly in alasql?
Thanks!

Thank you so much for pointing this out!

Released as part of v4.0.2

Thank you so much for the quick turnaround!


0x80070490 Winget source it's not working

Brief description of your issue
Error 0x80070490  I can't use winget because the source is not working.

Steps to reproduce
Using any comand to get info at source
Expected behavior
Normal winget functions
Actual behavior
Winget doesnt work becaus the source is not working.

Environment
PowerShell 7.4.5
PS C:\Users\Ramshu> winget --info
Administrador de paquetes de Windows v1.8.1911
Copyright (c) Microsoft Corporation. Todos los derechos reservados.

Windows: Windows.Desktop v10.0.19045.4842
Arquitectura del sistema: X64
Paquete: Microsoft.DesktopAppInstaller v1.23.1911.0

Directorios de WinGet
-----------------------------------------------------------------------------------------------------------------------
Registros                                   %LOCALAPPDATA%\Packages\Microsoft.DesktopAppInstaller_8wekyb3d8bbwe\LocalS…
Configuración de usuario                    %LOCALAPPDATA%\Packages\Microsoft.DesktopAppInstaller_8wekyb3d8bbwe\LocalS…
Directorio de vínculos portátiles (usuario) %LOCALAPPDATA%\Microsoft\WinGet\Links
Directorio de vínculos portátiles (equipo)  C:\Program Files\WinGet\Links
Raíz de paquete portátil (usuario)          %LOCALAPPDATA%\Microsoft\WinGet\Packages
Raíz de paquete portátil                    C:\Program Files\WinGet\Packages
Raíz de paquete portátil (x86)              C:\Program Files (x86)\WinGet\Packages
Descargas del instalador                    F:\Ramshu\Downloads

Vínculos
-----------------------------------------------------------------------------------
Declaración de privacidad   https://aka.ms/winget-privacy
Contrato de licencia        https://aka.ms/winget-license
Avisos de terceros          https://aka.ms/winget-3rdPartyNotice
Página principal            https://aka.ms/winget
Términos de Microsoft Store https://www.microsoft.com/en-us/storedocs/terms-of-sale

Configuración de administrador            Estado
-------------------------------------------------------
LocalManifestFiles                        Deshabilitado
BypassCertificatePinningForMicrosoftStore Deshabilitado
InstallerHashOverride                     Deshabilitado
LocalArchiveMalwareScanOverride           Deshabilitado
ProxyCommandLineOptions                   Deshabilitado
DefaultProxy                              Deshabilitado
PS C:\Users\Ramshu>

same issue. Maybe in the next few hours the server will be back up

I don't believe the service was down. The URL in the source export in the screenshot above isn't the path to the Pre-Indexed package.
The correct path is:
https://cdn.winget.microsoft.com/cache/source2.msix

Does winget source update -n winget also fail?

Does also fail?winget source update -n winget

Yes, it also fails.

Can you run the command and share verbose logs?
winget source update -n winget --verbose-logs --logs

Can you run the command and share verbose logs? winget source update -n winget --verbose-logs --logs

Sure, forgive the delay in answering you.
WinGet-2024-09-08-21-12-24.011.log
WinGetCOM-2024-09-08-11-54-55.658.log
WinGetCOM-2024-09-08-01-05-29.895.log

+1
same problem

Winget has never worked properly, especially on clean installations, even though it comes pre-installed. The installation is often broken, preventing it from running or updating. The best solution is to install it directly.
Start-BitsTransfer -Source "https://github.com/microsoft/winget-cli/releases/download/v1.8.1911/Microsoft.DesktopAppInstaller_8wekyb3d8bbwe.msixbundle" -Destination "$env:USERPROFILE\Downloads\Microsoft.DesktopAppInstaller.msixbundle"
Add-AppxPackage -Path "$env:USERPROFILE\Downloads\Microsoft.DesktopAppInstaller.msixbundle"

Winget has never worked properly, especially on clean installations, even though it comes pre-installed. The installation is often broken, preventing it from running or updating. The best solution is to install it directly.
Start-BitsTransfer -Source "https://github.com/microsoft/winget-cli/releases/download/v1.8.1911/Microsoft.DesktopAppInstaller_8wekyb3d8bbwe.msixbundle" -Destination "$env:USERPROFILE\Downloads\Microsoft.DesktopAppInstaller.msixbundle"
Add-AppxPackage -Path "$env:USERPROFILE\Downloads\Microsoft.DesktopAppInstaller.msixbundle"

I'm also having problems with appx...
When the problems with winget started, I tried uninstalling it with appx, getting the same error codes as now. I have checked and the services (AppXSVC) and (ClipSVC) are up and running, but for some reason that I don't know appx doesn't work.

Ok, I suggest first repairing the Microsoft Store by running wsreset -i. After a while, wait for a notification that confirms the Store has been installed.

Same here.
Windows-Paket-Manager v1.8.1911
Copyright (c) Microsoft Corporation. Alle Rechte vorbehalten.

Windows: Windows.Desktop v10.0.22631.4112
Systemarchitektur: X64
Paket: Microsoft.DesktopAppInstaller v1.23.1911.0

WinGet-Verzeichnisse
-----------------------------------------------------------------------------------------------------------------------------------------
Protokolle                                   %LOCALAPPDATA%\Packages\Microsoft.DesktopAppInstaller_8wekyb3d8bbwe\LocalState\DiagOutputDir
Benutzereinstellungen                        %LOCALAPPDATA%\Packages\Microsoft.DesktopAppInstaller_8wekyb3d8bbwe\LocalState\settings.json
Verzeichnis für portierbare Links (Benutzer) %LOCALAPPDATA%\Microsoft\WinGet\Links
Portables Linkverzeichnis (Computer)         C:\Program Files\WinGet\Links
Portierbarer Paketstamm (Benutzer)           %LOCALAPPDATA%\Microsoft\WinGet\Packages
Portierbarer Paketstamm                      C:\Program Files\WinGet\Packages
Portierbares Paketstamm (x86)                C:\Program Files (x86)\WinGet\Packages
Installationsprogrammdownloads               %USERPROFILE%\Downloads

Links
-----------------------------------------------------------------------------------------
Datenschutzerklärung              https://aka.ms/winget-privacy
Lizenzvereinbarung                https://aka.ms/winget-license
Hinweise von Drittanbietern       https://aka.ms/winget-3rdPartyNotice
Startseite                        https://aka.ms/winget
Windows Store-Nutzungsbedingungen https://www.microsoft.com/en-us/storedocs/terms-of-sale

Administratoreinstellung                  Status
-----------------------------------------------------
LocalManifestFiles                        Deaktiviert
BypassCertificatePinningForMicrosoftStore Deaktiviert
InstallerHashOverride                     Deaktiviert
LocalArchiveMalwareScanOverride           Deaktiviert
ProxyCommandLineOptions                   Deaktiviert
DefaultProxy                              Deaktiviert

Here's a log from a "winget source update --verbose-logs".
WinGet-2024-09-11-10-42-10.128.log

Ok, I suggest first repairing the Microsoft Store by running wsreset -i. After a while, wait for a notification that confirms the Store has been installed.

Done but the problems persist.

I fixed it by granting my user admin perssions, followed by a
winget source reset --force
and
winget source update
Then I removed the admin permissions again and winget still works.

I fixed it by granting my user admin perssions, followed by a
winget source reset --force
and
winget source update
Then I removed the admin permissions again and winget still works.

That's literally what I do in the first comment, obviously it didn't work, so they're not the same problem. If you can give more details of the problem you were having, in case it helps to find a solution that would be great.

Did you granted your normal office account the admin priviledges? Because as Administrator, winget never showed an issue, only with my normal, unpriviledge office account.

Did you granted your normal office account the admin priviledges? Because as Administrator, winget never showed an issue, only with my normal, unpriviledge office account.
Yes, my account has the privileges.

I don't know if it would help or not, but there is a "Repair-WinGetPackageManager" cmdlet in the Microsoft.WinGet.Client PowerShell module. It's designed to "fix" issues with WinGet from an install/bootstrapping perspective.
You could try running Repair-WinGetPackageManager -Latest -Force to see if that addresses the problem.
If that works, it's clearly a work around and not the solution to the problem, but I'd love to help unblock the issue until we can get a solid set of instructions to reproduce the situation so we can figure out how to fix the actual root cause.

Repair-WinGetPackageManager -Latest -Force
Doesnt work for me...

I don't know if it would help or not, but there is a "Repair-WinGetPackageManager" cmdlet in the Microsoft.WinGet.Client PowerShell module. It's designed to "fix" issues with WinGet from an install/bootstrapping perspective.
You could try running Repair-WinGetPackageManager -Latest -Force to see if that addresses the problem.
If that works, it's clearly a work around and not the solution to the problem, but I'd love to help unblock the issue until we can get a solid set of instructions to reproduce the situation so we can figure out how to fix the actual root cause.

Doesn't works for me...

Winget has never worked properly, especially on clean installations, even though it comes pre-installed. The installation is often broken, preventing it from running or updating. The best solution is to install it directly.
Start-BitsTransfer -Source "https://github.com/microsoft/winget-cli/releases/download/v1.8.1911/Microsoft.DesktopAppInstaller_8wekyb3d8bbwe.msixbundle" -Destination "$env:USERPROFILE\Downloads\Microsoft.DesktopAppInstaller.msixbundle"
Add-AppxPackage -Path "$env:USERPROFILE\Downloads\Microsoft.DesktopAppInstaller.msixbundle"

this completely solved it for me. appreciate so much.

@Ramshu it looks like you may not have "installed" the module before trying to run the cmdlet.
Try running Install-Module Microsoft.WinGet.Client before trying to run Repair-WinGetPackageManager -Force -Latest.
This did something, but it didn't fix the problem.

Winget has never worked properly, especially on clean installations, even though it comes pre-installed. The installation is often broken, preventing it from running or updating. The best solution is to install it directly.
Start-BitsTransfer -Source "https://github.com/microsoft/winget-cli/releases/download/v1.8.1911/Microsoft.DesktopAppInstaller_8wekyb3d8bbwe.msixbundle" -Destination "$env:USERPROFILE\Downloads\Microsoft.DesktopAppInstaller.msixbundle"
Add-AppxPackage -Path "$env:USERPROFILE\Downloads\Microsoft.DesktopAppInstaller.msixbundle"

I managed to get the commands to work using Windows PowerShell instead of the latest version of Powershell, but unfortunately it didn't solve the problem.
I have also tried to uninstall winget completely and installing the latest version, but it did not solve anything either.
Get-AppxPackage Microsoft.DesktopAppInstaller | Remove-AppxPackage
Add-AppxPackage -Path "$env:USERPROFILE\Downloads\Microsoft.DesktopAppInstaller_8wekyb3d8bbwe.msixbundle"

I have been able to solve it in the most unusual way.
I remembered that the problems with winget started after deleting a Guest Account... So I tried to create a new administrator account, I tried to use WinGet List on the new account, it asked me to accept the terms and it worked.
I have returned to my usual user account and winget worked magically, it is curious that it also asked me to accept the terms as if it were a new installation.
I really don't understand what happened or why, I just know that it has worked.

I don't believe the service was down. The URL in the source export in the screenshot above isn't the path to the Pre-Indexed package.
The correct path is: https://cdn.winget.microsoft.com/cache/source2.msix

This helped me, had this winget error since 3-4 Months and couldn't solve it. Thanks so much! @denelon


Bug 2127931: Change top consumer metrics default to 30 minutes

Signed-off-by: Matan Schatzman<EMAIL_ADDRESS>
📝 Description
Change top consumer metrics default time to 30mins, to match observe dashboard.
🎥 Demo

/retest


Diagnostic for calculating Lamb Weathertypes

Hello everyone!
At the Wegener Center in Graz, Austria, we have been doing some research with Lamb Weathertypes for quite some while now, and recently started developing a diagnostic for ESMValTool to make things easier, and potentially be of use for others!
The features are:
Calculate the 27 Lamb Weathertypes over a region specified in the corresponding recipe
Plot means, anomalies as well as standard deviations for those weathertypes for psl, tas and prcp
Combining the weathertypes based on precipitation patterns over an area specified in the corresponding recipe as done here: https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2020JD032824 (Table 2)
Create the same plots for those combined weathertypes

The weathertypes are calculated for ERA5 and model data, and the precipitation data for correlation calculations are taken from ERA5 and E-OBS.
The branch we are working on is called weathertyping_wegc.
We are happy to discuss this project with the community, get some feedback and features you would like to see!
Thank you very much and with kind regards,
Thomas Kroißenbrunner and Martin Jury
Before you get started

[ ] ☝ Create an issue to discuss what you are going to do

Checklist
It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

[x] 🛠 This pull request has a descriptive title
[x] 🛠 Code is written according to the code quality guidelines
[x] 🛠 Documentation is available
[ ] 🛠 Tests run successfully
[x] 🛠 The list of authors is up to date
[ ] 🛠 Any changed dependencies have been added or removed correctly
[ ] 🛠 All checks below this pull request were successful

New or updated recipe/diagnostic

[x] 🧪 Recipe runs successfully
[x] 🧪 Recipe is well documented
[x] 🧪 Figure(s) and data look as expected from literature
[x] 🛠 Provenance information has been added

New or updated data reformatting script

[x] 🛠 Documentation is available
[ ] 🛠 The dataset has been added to the CMOR check recipe
[ ] 🛠 The dataset has been added to the shared data pools of DKRZ and Jasmin by the @ESMValGroup/OBS-maintainers team
[x] 🧪 Numbers and units of the data look physically meaningful

@thomaskroi1996 glad to see this in PR form, well done :clap: :beer: I can perform a technical review of it, but for a scientific review, I'd suggest we think who's the closest expert amongst us to perform the review. Just a couple questions before the review, please:

are you happy to have this reviewed now, or you still working on it? If work is still to be performed, then I'd recommend you convert this PR to a Draft PR, and when it's ready, make it back ready for review;
to find the best scientific reviewers, it'd be good if you could suggest one or more folks that already have experience with this type of diagnostic, if not maybe we can ask the list (scientific reviewers, which I can invoke via a GitHub tag)

Cheers :beer:

@valeriupredoi Hey there! I am still doing some work, and definitely have to do all the tests, therfore I will convert it to a draft now, thank you so much!
As for the 2nd point, I can talk to my supervisor next week, and then maybe we have some suggestions for scientific reviewers, in which case I will reply to you here again!

@valeriupredoi Hello! :) I have now finished all the main work now, and implemented everything we wanted to! Now I think it is time for the technical review, correct? Thank you so much for all the help!

many thanks @thomaskroi1996 - and my apologies, I was on summer holidays until a few days ago, let me do a tech review now, cheers :beer:

@ESMValGroup/science-reviewers any of you good folk want to have a look at this new diagnostic, please? It looks very cool, but unfortunately, I am no expert in weather types, all I know is "rainy" in Britain :gb: :grin:

I'll look at it tomorrow, not an expert in weather types but can still see if each part is scientifically sound, esp if there's a linked paper to compare it to!

I'll look at it tomorrow, not an expert in weather types but can still see if each part is scientifically sound, esp if there's a linked paper to compare it to!

wonderful, many thanks, Tina :beer:

@valeriupredoi @bettina-gier Hello both of you and thank you so much for the replies! Not sure how I missed that, so sorry for the late reply!
Thank you so much for the input, I will correct everything as soon as i have some time on my hands! Unfourtunately I have switched to a different project, but it should be fine to allocate some hours throughout the next weeks to work on the mistakes/improvements!

@valeriupredoi @bettina-gier Hello! Currently the Circle CI test seems to fail because I use "from wt_utils import *", just thought i would comment that here!
Also the minor things are dealt with now, leaving only including the possibility for more observational datasets (for which I will talk to @mwjury), and compacting the recipes!
I also need to rerun the recipes so that I can update the documentation with the new plots etc., however since I currently need our computational resources for another project, hopefully next week I can tend to it!
Thanks again for the great input! :)

@thomaskroi1996 take your time, no rush at all! I can help sometime next week too, knock off some of those Flake8 issues. You are absolutely correct - and my bad - Flake8 flags star imports - always discouraged people do that, but looks like Flake8 is now the Punisher :grin:

@valeriupredoi Alright, over the next couple of days I will import only the necessary functions, and I only just discovered that there were more issues with the code, even though flake8 didn't output any anymore, I should have also run pytest!
One issue is "too many local variables". Those started popping up because I tried to fix "too many arguments" by passing Dicts and then reassigning them to local variables in the functions! Those are all marked as minor issues, should I fix those issues? In my opinion I think the way I have it now makes it a bit more readable and less clustered, and if I remember correctly I once saw a PR where it was said that a few of these issues are okay!

One issue is "too many local variables". Those started popping up because I tried to fix "too many arguments" by passing Dicts and then reassigning them to local variables in the functions! Those are all marked as minor issues, should I fix those issues? In my opinion I think the way I have it now makes it a bit more readable and less clustered, and if I remember correctly I once saw a PR where it was said that a few of these issues are okay!

That's great you tried to clean up, but don't worry too much about the "too many variables" linter message, scientific computing for ESMs is bound to have lots of variables anyway :beer:

One issue is "too many local variables". Those started popping up because I tried to fix "too many arguments" by passing Dicts and then reassigning them to local variables in the functions! Those are all marked as minor issues, should I fix those issues? In my opinion I think the way I have it now makes it a bit more readable and less clustered, and if I remember correctly I once saw a PR where it was said that a few of these issues are okay!

That's great you tried to clean up, but don't worry too much about the "too many variables" linter message, scientific computing for ESMs is bound to have lots of variables anyway 🍺

@valeriupredoi Hello again! Sorry for the quite long hiatus, but unfourtunately i had to juggle some projects! However, I have gotten to redo a lot of the things for this contribution! One thing that we did not manage to do for now, is the inclusion of other observational datasets! Would it be possible to continue without doing so?
And as for the checks that still fail, in the Codacy Static Code Analysis there are things like "too many local variables, variable name does not conform ..." i.e. minor issues! Should we see to fix all of those? As for the readthedocs fail, I am really not quite sure what is going wrong there, do you maybe have a pointer?
Thank you so much in advance!


running against a class results in "Element must be a table" error

I am initialising the plugin with
new Tablesort(document.getElementsByClassName('trackTable'), {
    descending: true
});

on a table that starts:
<table id="mySearchesTable" class="trackTable noSelect" cellpadding="0" cellspacing="0">
Yet I get the following error:
Uncaught Error: Element must be a table

Document.getElementById(), which is the usual way of picking the table element, returns a scalar (an Element), while Document.getElementsByClassName() returns a vector (an HTMLCollection).
If you're sure you have just one element with that class, try something like:
document.getElementsByClassName('trackTable')[0]

But it's much better if you pick the element by its ID!
(Please confirm that's the issue you're having, so I can close this :)

I personally dislike using classes to enable JS. I add a data attribute to my tables
data-table-sortable or data-table-sortable="desc"
And then use a function I can call to init all tables
export function sortAllTables() {
  const sortableTables = document.querySelectorAll('[data-table-sortable]')
  if (sortableTables.length > 0) {
    sortableTables.forEach((table) => {
      // eslint-disable-next-line no-new
      let sortAsc = true
      if (table.dataset.tableSortable) {
        sortAsc = table.dataset.tableSortable === 'asc' ? false : true
      }
      new Tablesort(table, { descending: sortAsc })
    })
  }
}

(@dombarnes, a suggestion to make your code a bit more succinct)

export function sortAllTables() {
  const sortableTables = document.querySelectorAll('[data-table-sortable]')
  sortableTables.forEach(table => {
    // eslint-disable-next-line no-new
    let sortAsc = true
    if (table.dataset.tableSortable)
      sortAsc = table.dataset.tableSortable != 'asc'
    new Tablesort(table, { descending: sortAsc })
  })
}

Closing this as it's not an issue with Tablesort itself


Use http and check hash after download for swig.

Signed-off-by: luckychess<EMAIL_ADDRESS>

Description of the Change

Https can be not supported by cmake (it depends of does libcurl support ssl or not). This pr replaces https for http for swig package download and checks its hash after to prevent from downloading something malicious.
Benefits

More universal way to download swig with comparable level of security.
Possible Drawbacks

None.

Doesn't solve a problem since we http url redirects to another http which redirects to https.


[Linux] PCIMapper pci.ids file location bug

I found a bug using the GPU module.
When compiled as user PCIMapper looks for the id files at  /home/${user}/.hwinfo/pci.ids
However if the program is ran as sudo or root the getenv("HOME"); function will return /root/.hwinfo/pci.ids
I propose this line be changed to store the pci.ids file to the same location as the applications binary.
https://github.com/lfreist/hwinfo/blob/b4adfd795aa2aa25b8dff6dd8116725f72b187dd/CMakeLists.txt#L39C67-L39C67
I'm trying to get into some open-source coding so if you would let me know your opinions I would be happy to implement them.

Hi, Thanks for reporting this Bug!
The pci.ids file brings some more issues: HWinfo is a library that can be build shared and static. For a shared library, it is relatively easy to place the pci.ids file along the binaries. For static builds it becomes more difficult. I am currently working on a PCIMapper that does not parse the pci.ids file but builds a dictionary like structure in compile time from the file. This way, static and shared builds become much easier.


Utilization of capabilities and requirements

This PR introduces some utilization of the capabilities/requirements functionality added in WildFly Core.
The first commit is the most significant one, as it introduces use of capabilities/requirements to mediate the interaction between the iiop and transactions subsystems.
The other commits are simpler, replacing how subsystems needing access to the MBeanServer get it from the JMX subsystem.

@tadamski  FYI
@jmesnil I can drop the messaging subsystem commit if you intend to remove the runtime stuff anyway. The artemis integration probably needs something similar.

Linux Build 6663 is now running using a merge of 295f31e53ab0c13aaf5f417607a96386f5ef9a0e

Linux with security manager Build 1564 is now running using a merge of 295f31e53ab0c13aaf5f417607a96386f5ef9a0e

Windows Build 1737 is now running using a merge of 295f31e53ab0c13aaf5f417607a96386f5ef9a0e

Linux with security manager Build 1564 outcome was SUCCESS using a merge of 295f31e53ab0c13aaf5f417607a96386f5ef9a0e
Summary: Tests passed: 832, ignored: 320 Build time: 0:05:59

Windows Build 1737 outcome was SUCCESS using a merge of 295f31e53ab0c13aaf5f417607a96386f5ef9a0e
Summary: Tests passed: 2877, ignored: 375 Build time: 0:47:22

Linux Build 6663 outcome was SUCCESS using a merge of 295f31e53ab0c13aaf5f417607a96386f5ef9a0e
Summary: Tests passed: 2875, ignored: 375, muted: 2 Build time: 0:50:00

@bstansberry you can keep the messaging commit: there may be some time before the new messaging-activemq subsystem is integrated and the runtime removed from the legacy one. Until then, we can still build on top of this messaging capability if anyone requires it.

@jmesnil the names are about namespacing.
org.wildfly namespace is owned by the WildFly and WildFly Core projects
org.wildfly.extension namespace is for capabilities not provided by or used by the "kernel"
The org.wildfly part I think is critical.
It's a good discussion for the mailing list whether separating kernel stuff from extension stuff adds any real value. I'll send up a thread.

The discussion re: names was at http://lists.jboss.org/pipermail/wildfly-dev/2015-June/004084.html
Consensus there was to get rid of the 'extension' bit, so I'm going to close this and redo it. Also https://github.com/wildfly/wildfly-core/pull/824 gets rid of the 'extension' part in core capabilities so this will need to be updated to reflect that.


Added ability to wrap output in a banner to make it easier to find

When you set the use_banners option to true, the output from the debug call will print a new line, followed by 80 # symbols, and another newline either side of the output.
Specs are all passing, and readme etc has been updated.

preparing to archive...


Prices older than 6 months and hourly prices are missing in the DB

Query to check the prices:
SELECT timestamp, COUNT(*)
FROM "PrismaTokenPrice"
GROUP BY 1
ORDER BY 1 ASC;

I'm not sure why we are missing so much data. I triple checked the SQL pruning and no problem there. Prices aren't getting deleted anywhere else. Any updates could cause missing hourly pricing?

Where do you see that? PROD db has all that data

The daily prices in the test DB I recall that I have accidentally removed them at some point, hence we only have them till Sept 30th. We still have them in the prod db all the way back.
The big question is, why are hourly prices removed after two days. It seems this was introduced between Feb 9 and March 1st, as I have two DB backups from those dates. Feb 9 backup still has hourly prices for 100 days, March 1st backup only has two days of hourly prices

Found it https://github.com/balancer/backend/commit/6c158f5f300bb73d891026236385d6367f7990b8
Somehow that number is not converted correctly, also doesnt work if it is a string

might not be fixed after all

worked on test

currently I see daily prices back to April 1st for both prod and test. Let's check tomorrow again

Still see hourly prices back to April 1st for both prod and test. Let's check back tomorrow

April 1st prices still there, fixed


stl

About the PR
Someone reminded me to add this.
Screenshots

Changelog

This PR is going to change the landscape of SS14 throughly, a neverbeforeseen masterpiece of a craft, a great deed that will forever leave a print in the history of mankind, God's gift to SS14 as a whole.
This, without a doubt, will be the best PR of this month, year, the entire lifespan of SS14, and beyond, not just due to it's sheer elegancy, yet also due to the concept of this magnificent work of art, perhaps so good you could rival it with some of the greatest pieces known to date, Mona Lisa, The Last Supper, God knows what else you can compare this to, yet the result is clear, in every comparison this glorious advancement outscales and thwarts such piece of art due to it's sheer glory and brilliance.

@Myctai i'm sure you'll love this one :)

stl


The example at async/unstable_throttle.ts$38-57.ts is flaky

ref https://github.com/denoland/std/actions/runs/11493365063/job/31988931062

The example of throttle API seems fairly flaky these days, but I could understand how that could be flaky (func.lastExecution seems getting NaN value in such cases). https://github.com/denoland/std/blob/ff6037adf78392039fe508c242c5f3d6432b4744/async/unstable_throttle.ts#L38
@lowlighter Do you have any idea about how this can be flaky?

Looked like it failed because sometimes the request didn't reach the server in 100ms.


Allow services to be injected into extension object constructors

Creation of extension objects is inconsistent with other places where model objects are constructed, in that the plugin code that creates the extension objects is responsible for finding and injecting the services required by the extension. In other places, such as plugin application, task creation or using ObjectFactory, the created doesn't need to care and Gradle takes care of injecting the services.
This has an impact on the native plugins, as dependencies of the extensions leak into the public API of the plugins. This also adds complexity to the plugins. Instead, change ExtensionContainer.create() to inject services into extensions. It will need to do this in a backwards compatible way wrt the @Inject annotation. Use this capability to simplify the native plugins.

@adammurdoch Was this fixed with all the recent changes you did around the injector?

yes, it was fixed in 5.2


<EMAIL_ADDRESS>hash check failed

Updating 'usbtreeview' (3.8.5 -> 3.8.6)
Downloading new version
UsbTreeView_x64.zip (387.2 KB) [==============================================================================] 100%
Checking hash of UsbTreeView_x64.zip ... ERROR Hash check failed!
App:         404NetworkError/usbtreeview
URL:         https://www.uwe-sieber.de/files/UsbTreeView_x64.zip
First bytes: 50 4B 03 04 14 00 00 00
Expected:    757b5038b622d18d138d2a537a534e546050704401d9f7ffa2d5c84c68681966
Actual:      598fc376af073a3da39c590359120ea1cd5278d818346d48956d55654e33caf9
Please try again or create a new issue by using the following link and paste your console output:
https://github.com/404NetworkError/scoop-bucket/issues/new?title=usbtreeview%403.8.6%3A+hash+check+failed

The ease of reporting is really what made me want to do it, just copying and pasting it can't get any simpler, so I appreciate the simplification of the work flow for reporting such issues

Honestly, the praise should really go to the devs over at ScoopInstaller/BucketTemplate for setting up a great template that handles all of the GitHub Actions (including updating buckets). Feel free to create your own bucket that is more specialized to your needs using this template if you like.


Update "Don't see your source/destination listed here?" section on taps/targets pages

Migrated from GitLab: https://gitlab.com/meltano/hub/-/issues/80
Originally created by @DouweM on 2021-06-09 22:01:58

@tayloramurphy The "Don't see your source/destination listed here?" section at the very bottom of https://hub.meltano.com/singer/taps/ and https://hub.meltano.com/singer/targets/ links to https://gitlab.com/meltano/hub/-/issues/11, but that issue is already closed.
Can we link somewhere else to inform users of the effort to get all taps and targets listed?

View 1 previous comment from the original issue on GitLab