S3D freeze at large node counts

As suspected by Mike in our meeting today, we are seeing freezes in S3D at large node counts. This is a preliminary report because I just encountered the issue and am still getting data.
What I know so far:

My most recent job on the 24.03.0 rc branch froze and 2048 and 8192 nodes, but ran at 4096 nodes
A previous job with my hacked up branch mentioned in https://github.com/StanfordLegion/legion/issues/1653#issuecomment-2005732308 froze at 8192 nodes but worked at all other node counts. Edit: I have now seen freezes at 2048 and 4096 nodes as well.
To the best of my knowledge, we have never been able to run a job past 8192 nodes. When Seshu tried (I can't remember if this was February 2024 or November 2023), he hit symptoms that looked like a freeze. It's possible these are all the same symptoms, but we didn't investigate at the time so it's hard to be sure

I have backtraces from the 2048 node run I just did. Don't blame me for not having -ll:force_kthreads; I was not expecting the freeze and wasn't prepared to collect backtraces, so they are what they are. From an initial scan I haven't see anything interesting, so likely I will need to rerun with -ll:force_kthreads.
http://sapling.stanford.edu/~eslaught/s3d_freeze_debug2/bt2048/

Do these runs have the same bug that @syamajala found with incomplete partitions being marked complete? If so, that can lead to all sorts of undefined behavior in the runtime including hangs so I would make sure we don't have that bug in these runs at all before going about doing anymore debugging.

The branch @elliottslaughter is running does not have those partitions. I have fixed the issue with partitions incorrectly being marked as complete, it has not solved my problem though. Will open a separate issue for that soon.

For this issue I am running the same code as in #1653. In the initial investigation into that issue I ran a number of checks, including -lg:partcheck and -lg:safe_ctrlrepl 1. I do not have any reason to believe those results would have changed since the application code is unmodified.
My freeze appears to be sensitive to -ll:force_kthreads. When I run with that flag, I have not been able to replicate a freeze at lower than 8192 nodes. I set up a new job attempting to get backtraces at that node count. I'll reply to this issue again if I get them.

I got a run to freeze at 4096 nodes with -ll:force_kthreads. I collected three sets of backtraces, about 10 minutes apart. Because the collection process is finicky they don't all have the same number of nodes, but this should allow a direct comparison of at least the earlier node numbers:

initial backtraces: http://sapling.stanford.edu/~eslaught/s3d_freeze_debug2/bt4096/
several minutes later: http://sapling.stanford.edu/~eslaught/s3d_freeze_debug2/bt4096-2/
ten minutes after that: http://sapling.stanford.edu/~eslaught/s3d_freeze_debug2/bt4096-3/

One pattern I see in the backtraces is threads like this:
Thread 3 (Thread 0x7ffe5d20f740 (LWP 81080) "s3d.x"):
#0  0x00007fffed8f19d5 in pthread_spin_lock () from /lib64/libpthread.so.0
#1  0x00007fffdd874207 in ofi_genlock_lock (lock=<optimized out>) at ./include/ofi_lock.h:359
#2  cxip_send_common (txc=0x6754838, tclass=512, buf=0x7ffe75a1f278, len=912, desc=<optimized out>, data=data@entry=0, dest_addr=74, tag=0, context=0x7ffe75a1f250, flags=83886080, tagged=false, triggered=false, trig_thresh=0, trig_cntr=0x0, comp_cntr=0x0) at prov/cxi/src/cxip_msg.c:5510
#3  0x00007fffdd875001 in cxip_send (fid_ep=<optimized out>, buf=<optimized out>, len=<optimized out>, desc=<optimized out>, dest_addr=<optimized out>, context=<optimized out>) at prov/cxi/src/cxip_msg.c:5874
#4  0x00007fffe0f72136 in gasnetc_ofi_am_send_medium () from /ccs/home/eslaught/frontier/gb2024-subrank3/legion/language/build/lib/librealm.so.1
#5  0x00007fffe0fefd91 in gasnetc_AM_CommitRequestMediumM () from /ccs/home/eslaught/frontier/gb2024-subrank3/legion/language/build/lib/librealm.so.1
#6  0x00007fffe00d7628 in Realm::XmitSrcDestPair::push_packets (this=0x7ffd04031d50, immediate_mode=<optimized out>, work_until=...) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2092
#7  0x00007fffe00d925e in Realm::GASNetEXInjector::do_work (this=<optimized out>, work_until=...) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2725
#8  0x00007fffdff6d778 in Realm::BackgroundWorkManager::Worker::do_work (this=0x7ffe5d20d520, max_time_in_ns=<optimized out>, interrupt_flag=0x0) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/bgwork.cc:599
#9  0x00007fffdff6e051 in Realm::BackgroundWorkThread::main_loop (this=0x1fb2cdb0) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/bgwork.cc:103
#10 0x00007fffe007e541 in Realm::KernelThread::pthread_entry (data=0x1fdad8b0) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/threads.cc:831
#11 0x00007fffed8e96ea in start_thread () from /lib64/libpthread.so.0
#12 0x00007fffe912150f in clone () from /lib64/libc.so.6

Otherwise I don't see much going on.

This looks to me like the network locked up. There's multiple threads on pretty much every single node trying to push active messages into the network and they are all spinning on some lock in OFI inside a call to gex_AM_CommitRequestMedium2. If we can't send active messages it's not terribly surprising that things are going to hang. Now part of what might be going wrong here is that the code before frame 6 in Realm is using gex_AM_MaxRequestMedium to figure out how many medium active messages we can commit into the network without blocking and if we can't send them then the background worker threads will go off and do something else important (e.g. like handle incoming active messages). However, what it looks like is happening here is we got an indication from GASNet that we're going to be able to send these messages right away as the request call has succeeded, but then when we go to commit the messages, the actual commit code is blocking internally. Now I don't think this is necessarily GASNet's fault directly since it looks like OFI is actually the one that is blocking and not GASNet, but I think we might need to have GASNet modify the call to gex_AMRequestMedium to make sure it handles all the cases where OFI might block in the commit call. @PHHargrove @bonachea do you see something else going on here that I'm missing in these backtraces?

Since we're discussing a potential network hang, I will just include the network-related environment variables here for posterity:
export FI_MR_CACHE_MONITOR=memhooks
export FI_CXI_RX_MATCH_MODE=software
export GASNET_OFI_DEVICE_0=cxi2
export GASNET_OFI_DEVICE_1=cxi1
export GASNET_OFI_DEVICE_2=cxi3
export GASNET_OFI_DEVICE_3=cxi0
export GASNET_OFI_DEVICE_TYPE=Node
export GASNET_OFI_RECEIVE_BUFF_SIZE=8M
export MPICH_OFI_NIC_POLICY=BLOCK

(I think that GASNET_OFI_DEVICE_0 et al. are actually superfluous here because we are running 1 rank/node.)
And for @PHHargrove @bonachea's benefit, this is GASNet 2023.9.0 without memory kinds enabled.

FWIW, I spot checked a few more of the threads in the same processes across the backtraces at different times and the threads are not making any progress. The this pointers of the objects at all the frames in each thread remain the same across the several minutes of checkpoints, so they are very much stuck and not going anywhere, happily spin waiting on their OFI locks.

Please try with GASNET_OFI_RECEIVE_BUFF_SIZE=recv (instead of 8M).
The MetaHipMer team has pretty clear evidence that the issue with FI_MULTI_RECV was not fixed in the SlingShot 2.1.0 release.  So, this work-around to disable it may still be necessary.

Is this going to exacerbate the issue with the number of receive buffers and "unexpected" messages like we saw with the GASNET_OFI_RECEIVE_BUFF_SIZE=single workaround we had previously tested, or do you expect recv to avoid those problems?

@elliottslaughter wrote:

Is this going to exacerbate the issue with the number of receive buffers and "unexpected" messages

With use of CXI's buggy multi-recv feature disabled via GASNET_OFI_RECEIVE_BUFF_SIZE=recv, the conduit needs to post orders of magnitude more recv buffers in order to soak the same approximate volume of incoming AM messages before libfabric tips over into unexpected messages. The conduit adjusts the default value of GASNET_OFI_NUM_RECEIVE_BUFFS to help accomplish this (currently, the default changes from 8 to 450) and you also have the option to manually override GASNET_OFI_NUM_RECEIVE_BUFFS.
If the application is currently hitting the known bugs in multi-recv that arise under heavy load, then turning off that feature is probably the only surgical alternative from a correctness perspective. The only other (more intrusive) possibility that comes to mind is changing the communication pattern in the hopes of lowering the chances of breaking multi-recv.

like we saw with the GASNET_OFI_RECEIVE_BUFF_SIZE=single workaround we had previously tested, or do you expect recv to avoid those problems?

The GASNET_OFI_RECEIVE_BUFF_SIZE=recv setting activates a slightly improved version of the now-deprecated GASNET_OFI_RECEIVE_BUFF_SIZE=single setting (details here). Both seek to avoid the known bugs in the CXI provider implementation of FI_MULTI_RECV.  The main difference is =recv entirely removes FI_MULTI_RECV from the arguments GASNet passes to libfabric, whereas =single took a less direct approach of leaving multi-recv enabled but tuning the buffer parameters to induce every buffer to be "single shot".

@lightsighter wrote:

the code before frame 6 in Realm is using gex_AM_MaxRequestMedium to figure out how many medium active messages we can commit into the network without blocking and if we can't send them then the background worker threads will go off and do something else important (e.g. like handle incoming active messages). However, what it looks like is happening here is we got an indication from GASNet that we're going to be able to send these messages right away as the request call has succeeded, but then when we go to commit the messages, the actual commit code is blocking internally. Now I don't think this is necessarily GASNet's fault directly since it looks like OFI is actually the one that is blocking and not GASNet, but I think we might need to have GASNet modify the call to gex_AMRequestMedium to make sure it handles all the cases where OFI might block in the commit call.

First a clarification: gex_AM_MaxRequestMedium() is a query of maximum message size limit (not count) and in particular is independent of current network state. Attempts to inject messages with payloads over that size are simply erroneous; gex_AM_MaxRequestMedium() is not in any way relevant to whether or not that injection might block.
Setting that aside, you are correct that Realm::XmitSrcDestPair::push_packets() might be asking GASNet to avoid blocking at injection. That request is controlled by GEX_FLAG_IMMEDIATE, and that flag is  conditionally set by Realm::XmitSrcDestPair::push_packets() based on a caller argument (which is optimized away in the backtrace above, but appears to be hard-coded to true at this particular call-site).
Regardless of whether or not GEX_FLAG_IMMEDIATE is passed to gex_AM_PrepareRequestMedium() (or FPAM injection), this is a "best effort" GASNet feature, and does not (cannot) provide a hard guarantee that the underlying network library will not block during injection. Currently I'd characterize GASNet's "best effort" here as "pretty good" on ibv-conduit and "not so great" on ofi-conduit. My understanding of the current state of IMMEDIATE injection in ofi-conduit is that FPAM injection should faithfully detect cases where GASNet's conduit-level metadata (buffer descriptors) are running scarce, and should correctly return failure IFF the underlying fi_inject/fi_send call to the libfabric provider returns -FI_EAGAIN. However ofi-conduit does not attempt to model/predict the resource state of the underlying provider, who might choose to block without returning -FI_EAGAIN.
Moreover, it's worth noting that ofi-conduit does not currently have a native implementation of NPAM at all (GASNET_NATIVE_NP_ALLOC_REQ_MEDIUM==undef). Instead it currently uses a "reference implementation" of NPAM over FPAM, such that the Prepare/Commit calls in ofi-conduit currently just invoke FPAM underneath (as partially revealed in the stack trace). One important consequence of this "reference implementation" that is particularly critical here is that this reference implementation always strips off any GEX_FLAG_IMMEDIATE flags given to gex_AM_PrepareRequestMedium() (there are fundamental reasons for this arising from the NPAM vs FPAM interface difference). But the upshot here is that  even if Realm is passing GEX_FLAG_IMMEDIATE to gex_AM_PrepareRequestMedium(), ofi-conduit is currently always ignoring that flag to NPAM injection.
Coincidentlaly we've just been awarded funding that we hope can be used to improve both of these sub-optimal behaviors in ofi-conduit, but that work is still early stages and won't be ready for users for some time.

Regardless of whether or not GEX_FLAG_IMMEDIATE is passed to gex_AM_PrepareRequestMedium() (or FPAM injection), this is a "best effort" GASNet feature, and does not (cannot) provide a hard guarantee that the underlying network library will not block during injection.

Ok, I see where Realm is running a separate polling thread that always pulls active messages off the wire and that should be sufficient to ensure forward progress even if all the background worker threads get stuck in a commit call. I thought we had gotten rid of the polling thread in the GASNetEX module, but apparently not.

The MetaHipMer team has pretty clear evidence that the issue with FI_MULTI_RECV was not fixed in the SlingShot 2.1.0 release. So, this work-around to disable it may still be necessary.

So to make sure I'm clear: this bug is sufficient to prevent forward progress? I would kind of hope that even if we were running short of buffers for sending and blocking in the commit calls, that eventually the active message polling thread can eventually pull a bunch of messages off the wire, drain the network, and thereby free up resources for doing the sends again and that would ensure forward progress. That is not possible here and something else is going wrong in OFI?

But the upshot here is that even if Realm is passing GEX_FLAG_IMMEDIATE to gex_AM_PrepareRequestMedium(), ofi-conduit is currently always ignoring that flag to NPAM injection.

How is that an upshot? 😅

Coincidentlaly we've just been awarded funding that we hope can be used to improve both of these sub-optimal behaviors in ofi-conduit, but that work is still early stages and won't be ready for users for some time.

That is promising to hear!

Thread partially continued in email...

Several people have been asking me to run experiments, so I am going to report the results of those experiments here.
Experiment 1: GASNET_OFI_RECEIVE_BUFF_SIZE=recv
My initial experiment with the default value of GASNET_OFI_NUM_RECEIVE_BUFFS (450, per https://github.com/StanfordLegion/legion/issues/1657#issuecomment-2015812989) worked up to 2048 nodes and then froze at 4096 nodes. I tried doubling GASNET_OFI_NUM_RECEIVE_BUFFS to 900, and that rerun also froze at 4096 nodes.
It might help to get some guidance from @bonachea as to how much space I am actually allocating. For posterity, here is the most recent set of environment variables that I ran with:
export FI_MR_CACHE_MONITOR=memhooks
export FI_CXI_RX_MATCH_MODE=software
export GASNET_OFI_DEVICE_0=cxi2
export GASNET_OFI_DEVICE_1=cxi1
export GASNET_OFI_DEVICE_2=cxi3
export GASNET_OFI_DEVICE_3=cxi0
export GASNET_OFI_DEVICE_TYPE=Node
export GASNET_OFI_RECEIVE_BUFF_SIZE=recv
export GASNET_OFI_NUM_RECEIVE_BUFFS=$(( 450 * 2 )) # default is 450 when in recv mode
export MPICH_OFI_NIC_POLICY=BLOCK

If I'm allocating (say) the maximum size of a medium active message, I could be quite a bit more aggressive about this setting, if that's what we think we need to do. I'll probably jump to 450 * 10 for my next experiment unless someone jumps in to tell me otherwise.
Experiment 2: GASNet 2023.3.0
This is the version that @syamajala previously ran to 8192 nodes, as mentioned in the top comment on this issue. I am not aware of any technical reason why GASNet 2023.3.0 would work, but just to cover all our bases, I am running it. So far, it has been successful up to 4096 nodes. (This does not necessarily mean anything because the freezes are non-deterministic anyway, so we won't really know if it makes any meaningful difference until I can get a new 8192 node job through.)
Experiment 3: Tracing 1 Timestep
The code in the top comment of this issue traces 10 timesteps at a time. This is a shot in the dark, but to the extent that the tracing code (especially when first capturing a trace) induces network communication, my thought was that reducing the number of timesteps per trace might reduce that. (At the very least, fences between different replays of the trace would slow things down.)
So far it's working up to 4096 nodes, which again doesn't mean much in a nondeterministic world. But so far so good.

I should also point out that I am hitting the following warning in my runs:
*** WARNING: ofi-conduit failed to configure FI_CXI_* environment variables due to prior conflicting settings. This may lead to unstable behavior and/or degraded performance.  If you did not intentionally set FI_CXI_RX_MATCH_MODE='software', FI_CXI_RDZV_THRESHOLD='16384', or FI_CXI_RDZV_GET_MIN then this condition may have resulted from initializing MPI prior to initialization of GASNet. For more information on that scenario, please see "Limits to MPI interoperability" in the ofi-conduit README.
You can see my variables here: https://github.com/StanfordLegion/legion/issues/1657#issuecomment-2014441856 . I am indeed setting FI_CXI_RX_MATCH_MODE, but not the other two. I do not know if this matters.
(S3D is a hybrid code and I believe that we do initialize MPI first.)

I think it's worth noting that Frontier was upgraded to Slingshot 2.1.2 earlier this week  (Mar 18 2024 if directory timestamps are to be believed). This upgrade included replacing the installed libfabric CXI provider, portions of Cray MPI and other components of the system stack. If you've observed changes in network behavior relative to runs preceding Mar 18 2024, then the new Slingshot stack should be high on the list of suspects (not that we can do much about it).
Additionally, I'd like to ensure we are not "chasing phantoms" here. Because we are encountering problems, Please ensure you've rebuilt all objects and executables from scratch since Mar 18, especially GASNet and anything making MPI calls.

I should also point out that I am hitting the following warning in my runs:
*** WARNING: ofi-conduit failed to configure FI_CXI_* environment variables due to prior conflicting settings. This may lead to unstable behavior and/or degraded performance. If you did not intentionally set FI_CXI_RX_MATCH_MODE='software', FI_CXI_RDZV_THRESHOLD='16384', or FI_CXI_RDZV_GET_MIN then this condition may have resulted from initializing MPI prior to initialization of GASNet. For more information on that scenario, please see "Limits to MPI interoperability" in the ofi-conduit README.
You can see my variables here: #1657 (comment) . I am indeed setting FI_CXI_RX_MATCH_MODE, but not the other two. I do not know if this matters.

First, I should note for the record that all FI_* envvar settings control the (closed-source) HPE libfabric CXI provider that sits underneath GASNet (and Cray MPI) on HPE Cray EX. Our knowledge of how these settings operate is mostly constrained to system documentation and closed-box experimentation. That being said, in the absence of conflicting FI_ settings, ofi-conduit over CXI defaults to setting:
FI_CXI_RX_MATCH_MODE=hybrid
FI_CXI_RDZV_THRESHOLD=256
FI_CXI_RDZV_GET_MIN=256

This is NOT an endorsement by GASNet of guaranteed stability for this combination of CXI provider settings or suitability for all use cases, IIUC it's just a "better than nothing" default that we've found to be reasonable (especially for "pure GASNet" use cases where Cray MPI is not in use). ofi-conduit issues this warning if it finds one of these settings is already set, partially because they globally affect CXI provider behavior and MPI is known to set some of these even if the user did not (so the warning alerts you to the current settings in use).
So in your case the appearance of the warning is expected (because you are explicitly setting export FI_CXI_RX_MATCH_MODE=software), but the content reveals that FI_CXI_RDZV_THRESHOLD='16384' has additionally been set by someone (presumably by MPI or some other part of your software stack, given you did not set it explicitly). This "combination of stealthy settings from disparate sources" is the other reason ofi-conduit prints this warning, to help clarify what's actually being used by CXI provider.
Here is some documentation from the fi_cxi man page (emphasis added):

FI_CXI_RDZV_THRESHOLD
Message size threshold for rendezvous protocol.

FI_CXI_RDZV_GET_MIN
Minimum rendezvous Get payload size.  A Send with length less than or equal to FI_CXI_RDZV_THRESHOLD plus FI_CXI_RDZV_GET_MIN will be performed using the eager protocol.  Larger Sends will be performed using the rendezvous protocol with FI_CXI_RDZV_THRESHOLD bytes of payload sent eagerly and  the  remainder  of  the  payload  read  from  the  source  using  a  Get.               FI_CXI_RDZV_THRESHOLD plus FI_CXI_RDZV_GET_MIN must be less than or equal to FI_CXI_OFLOW_BUF_SIZE.

FI_CXI_RX_MATCH_MODE
Specify the receive message matching mode to be utilized.  FI_CXI_RX_MATCH_MODE=hardware | software | hybrid

hardware  -  Message  matching  is  fully  offloaded, if resources become exhausted flow control will be performed and existing unexpected message headers will be onloaded to free re-               sources.
software - Message matching is fully onloaded.
hybrid - Message matching begins fully offloaded, if resources become exhuasted hardware will transition message matching to a hybrid of hardware and software matching.

For both "hybrid" and "software" modes and care should be taken to minimize the threshold for rendezvous processing (i.e. FI_CXI_RDZV_THRESHOLD + FI_CXI_RDZV_GET_MIN).   When  running               in  software  endpoint  mode the environment variables FI_CXI_REQ_BUF_SIZE and FI_CXI_REQ_BUF_MIN_POSTED are used to control the size and number of the eager request buffers posted to handle incoming unmatched messages.

The docs above go out of their way to emphasize that the value (FI_CXI_RDZV_THRESHOLD + FI_CXI_RDZV_GET_MIN) should be "minimized" in FI_CXI_RX_MATCH_MODE=software mode. The rationale is presumably that larger eager messages greatly exacerbate the data volume of any unexpected messages, thus consuming the available overflow buffers even faster. I'm not sure that FI_CXI_RDZV_THRESHOLD=16384 constitutes "minimized", and FI_CXI_RDZV_GET_MIN is using an undocumented CXI default value of "who knows". So in this case the warning may be alerting you to a genuine mis-configuration problem (at least a configuration counter-indicated by HPE documentation). I think it's worth at least one small-scale run where you manually set all three "related" values together, eg:
export FI_CXI_RX_MATCH_MODE=software
export FI_CXI_RDZV_THRESHOLD=256
export FI_CXI_RDZV_GET_MIN=256
// plus your other settings

and consult the generated warning message to confirm all three values "survived" to configure CXI. Once that's confirmed, then perhaps a larger run to see if it impacts the deadlocks.

I tried doubling GASNET_OFI_NUM_RECEIVE_BUFFS to 900, and that rerun also froze at 4096 nodes. It might help to get some guidance from @bonachea as to how much space I am actually allocating.

The reason we currently default GASNET_OFI_NUM_RECEIVE_BUFFS=450 in  GASNET_OFI_RECEIVE_BUFF_SIZE=recv mode is documented here, specifically to workaround what we believe to be a not-yet-mentioned bug in CXI provider at high PPN (eg 128 ppn) that results in hang or crash very early in GASNet startup, even at small node scales. Assuming S3D is running less than 8ish PPN, it's probably safe to greatly increase GASNET_OFI_NUM_RECEIVE_BUFFS. However in order to avoid wasting allocation I'd start with a smaller-scale run to validate your chosen GASNET_OFI_NUM_RECEIVE_BUFFS value makes it past Realm initialization.
Based on reading the code and GASNet trace outputs, the per-process memory impact of GASNET_OFI_NUM_RECEIVE_BUFFS in GASNET_OFI_RECEIVE_BUFF_SIZE=recv mode should very roughly be GASNET_OFI_RECEIVE_BUFF_SIZE * (GASNET_OFI_MAX_MEDIUM + am_header_sz + metadata_sz) bytes, where GASNET_OFI_MAX_MEDIUM defaults to 8KiB and (am_header_sz + metadata_sz) is under 100 bytes. So the defaults:
GASNET_OFI_MAX_MEDIUM=8192
GASNET_OFI_RECEIVE_BUFF_SIZE=recv
GASNET_OFI_NUM_RECEIVE_BUFFS=450

entail a total of about 3.5 MiB of GASNet-level receive buffers at each process.  This excludes other sources of ofi-conduit buffer memory consumption (notably send buffers, which are controlled independently).
Using GASNet-only hello world (no MPI) and GASNET_OFI_RECEIVE_BUFF_SIZE=recv GASNET_OFI_MAX_MEDIUM=8192 I just was able to safely raise GASNET_OFI_NUM_RECEIVE_BUFFS=7700 with 8PPN on Frontier, for 60MiB of buffers per process, but GASNET_OFI_NUM_RECEIVE_BUFFS=7800 was "too much" at 8PPN and resulted in fi_mr_enable failed for aux_seg: -28(No space left on device). MPI is believed to consume some of the precious device resources that are the limiting factor here, so YMMV.

Thanks. I'll run these tests and get back to you. I do need to rebuild my software.
I'll just note that we have given up on the 8 PPN configuration of S3D and run exclusively in a 1 PPN configuration, which we intend to do for the foreseeable future (as we have other, unrelated issues at 8 PPN). Therefore, I think we could push these values even higher (if I understand correctly).

Amendment to previous post:
I believe the bug4478 h/w recv resource limitations I mentoined only apply in FI_CXI_RX_MATCH_MODE=hardware and FI_CXI_RX_MATCH_MODE=hybrid, which makes sense because the entire point of FI_CXI_RX_MATCH_MODE=software is to avoid relying on h/w offloaded recv. So that limit may be totally irrelevant in this use case.

we have given up on the 8 PPN configuration of S3D and run exclusively in a 1 PPN configuration, which we intend to do for the foreseeable future (as we have other, unrelated issues at 8 PPN). Therefore, I think we could push these values even higher (if I understand correctly).

In FI_CXI_RX_MATCH_MODE=software mode with 1PPN and the following settings:
FI_CXI_RX_MATCH_MODE=software
FI_CXI_RDZV_THRESHOLD=256
FI_CXI_RDZV_GET_MIN=256
GASNET_OFI_RECEIVE_BUFF_SIZE=recv
GASNET_OFI_MAX_MEDIUM=8192

I'm able to crank Frontier's GASNET_OFI_NUM_RECEIVE_BUFFS up over 1000000 for a whopping 7.7 GiB of recv buffer space per process (i.e. per node) without encountering the init-time problem. (Again in a GASNet-only program with no MPI resource consumption. I'm not recommending that much buffer space in production (because it's basically guaranteed to incur heavy TLB performance penalties), but the point is in FI_CXI_RX_MATCH_MODE=software mode it seems you can freely crank up GASNET_OFI_NUM_RECEIVE_BUFFS until you exhaust main memory.

Here is my status report as of tonight. I rebuilt the code from an entirely fresh checkout. This is the version that traces 1 timestep at a time (similar to Experiment 3 in https://github.com/StanfordLegion/legion/issues/1657#issuecomment-2016708414, but note this is a fresh build now), because that version had the shortest startup time.
Experiment 4: GASNET_OFI_RECEIVE_BUFF_SIZE=recv and GASNET_OFI_NUM_RECEIVE_BUFFS=10000

Expand for full variables
export FI_MR_CACHE_MONITOR=memhooks
export FI_CXI_RX_MATCH_MODE=software
export FI_CXI_RDZV_THRESHOLD=256
export FI_CXI_RDZV_GET_MIN=256
export GASNET_OFI_DEVICE_0=cxi2
export GASNET_OFI_DEVICE_1=cxi1
export GASNET_OFI_DEVICE_2=cxi3
export GASNET_OFI_DEVICE_3=cxi0
export GASNET_OFI_DEVICE_TYPE=Node
export GASNET_OFI_RECEIVE_BUFF_SIZE=recv
export GASNET_OFI_NUM_RECEIVE_BUFFS=10000
export MPICH_OFI_NIC_POLICY=BLOCK

Results:

From 1 to 4096 nodes everything worked perfectly
At 8192 nodes, the job timed out without ever finishing a time step

For posterity, the contents of the GASNet warning printed in these runs confirms that our configuration is getting through:
*** WARNING: ofi-conduit failed to configure FI_CXI_* environment variables due to prior conflicting settings. This may lead to unstable behavior and/or degraded performance.  If you did not intentionally set FI_CXI_RX_MATCH_MODE='software', FI_CXI_RDZV_THRESHOLD='256', or FI_CXI_RDZV_GET_MIN='256' then this condition may have resulted from initializing MPI prior to initialization of GASNet. For more information on that scenario, please see "Limits to MPI interoperability" in the ofi-conduit README.
Experiment 5: GASNET_OFI_RECEIVE_BUFF_SIZE=recv and GASNET_OFI_NUM_RECEIVE_BUFFS=100000
Increasing by another order of magnitude just to see what would happen.

Expand for full variables
export FI_MR_CACHE_MONITOR=memhooks
export FI_CXI_RX_MATCH_MODE=software
export FI_CXI_RDZV_THRESHOLD=256 # suggested: https://github.com/StanfordLegion/legion/issues/1657#issuecomment-2016929847
export FI_CXI_RDZV_GET_MIN=256 # suggested: https://github.com/StanfordLegion/legion/issues/1657#issuecomment-2016929847
export GASNET_OFI_DEVICE_0=cxi2
export GASNET_OFI_DEVICE_1=cxi1
export GASNET_OFI_DEVICE_2=cxi3
export GASNET_OFI_DEVICE_3=cxi0
export GASNET_OFI_DEVICE_TYPE=Node
export GASNET_OFI_RECEIVE_BUFF_SIZE=recv
export GASNET_OFI_NUM_RECEIVE_BUFFS=100000
export MPICH_OFI_NIC_POLICY=BLOCK

Results:

From 1 to 4096 nodes worked perfectly
Pending results for 8192 nodes

Experiment 6: GASNET_OFI_RECEIVE_BUFF_SIZE=8M
This was my original configuration for Experiment 3, but I decided to rerun it in the new build.

Expand for full variables
export FI_MR_CACHE_MONITOR=memhooks
export FI_CXI_RX_MATCH_MODE=software
export GASNET_OFI_DEVICE_0=cxi2
export GASNET_OFI_DEVICE_1=cxi1
export GASNET_OFI_DEVICE_2=cxi3
export GASNET_OFI_DEVICE_3=cxi0
export GASNET_OFI_DEVICE_TYPE=Node
export GASNET_OFI_RECEIVE_BUFF_SIZE=8M
export MPICH_OFI_NIC_POLICY=BLOCK

Results:

From 1 to 4096 nodes everything worked perfectly
At 8192 nodes, the job timed out without ever finishing a time step

Experiment 7: GASNET_OFI_RECEIVE_BUFF_SIZE=64M
I figured if memory is the issue, why not go up? So I ran a test at 64 MB.

Expand for full variables
export FI_MR_CACHE_MONITOR=memhooks
export FI_CXI_RX_MATCH_MODE=software
export GASNET_OFI_DEVICE_0=cxi2
export GASNET_OFI_DEVICE_1=cxi1
export GASNET_OFI_DEVICE_2=cxi3
export GASNET_OFI_DEVICE_3=cxi0
export GASNET_OFI_DEVICE_TYPE=Node
export GASNET_OFI_RECEIVE_BUFF_SIZE=64M
export MPICH_OFI_NIC_POLICY=BLOCK

Results:

From 1 to 4096 nodes everything worked perfectly
At 8192 nodes, I hit the error below:

srun: error: Task launch for StepId=1794522.0 failed on node frontier04727: Socket timed out on send/recv operation
srun: error: Application launch failed: Socket timed out on send/recv operation
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.

I think I'll rerun this one. In my experience this has been a transient error.
Experiment 8: GASNET_OFI_RECEIVE_BUFF_SIZE=64M and RDZV variables to 256
It wasn't obvious to me if @bonachea's advice above would apply to this mode, so I figured I'd try.

Expand for full variables
export FI_MR_CACHE_MONITOR=memhooks
export FI_CXI_RX_MATCH_MODE=software
export FI_CXI_RDZV_THRESHOLD=256
export FI_CXI_RDZV_GET_MIN=256
export GASNET_OFI_DEVICE_0=cxi2
export GASNET_OFI_DEVICE_1=cxi1
export GASNET_OFI_DEVICE_2=cxi3
export GASNET_OFI_DEVICE_3=cxi0
export GASNET_OFI_DEVICE_TYPE=Node
export GASNET_OFI_RECEIVE_BUFF_SIZE=64M
export MPICH_OFI_NIC_POLICY=BLOCK

Results:

From 1 to 4 nodes worked fine
8 nodes timed out
16 nodes crashed with the error below:

s3d.x: /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank5-trace1/legion/runtime/realm/utils.cc:118: uint32_t Realm::crc32c_accumulate(uint32_t, const void*, size_t): Assertion `accum == accum_sw' failed.

Experiment 9: GASNET_OFI_RECEIVE_BUFF_SIZE=8M with -lg:inorder -ll:force_kthreads
All this had me wondering if we were on the right track. So I figured I'd go back and try to capture a backtrace that tells us exactly what Legion is doing at the point of the freeze. Or maybe -lg:inorder will slow things down enough that it doesn't happen. It seems crazy, but the application does actually run at scale with -lg:inorder.

Expand for full variables
export FI_MR_CACHE_MONITOR=memhooks
export FI_CXI_RX_MATCH_MODE=software
export GASNET_OFI_DEVICE_0=cxi2
export GASNET_OFI_DEVICE_1=cxi1
export GASNET_OFI_DEVICE_2=cxi3
export GASNET_OFI_DEVICE_3=cxi0
export GASNET_OFI_DEVICE_TYPE=Node
export GASNET_OFI_RECEIVE_BUFF_SIZE=8M
export MPICH_OFI_NIC_POLICY=BLOCK

Results:

From 1 to 2048 nodes worked, although the application is about 60% slower in this configuration. Fortunately, the slowdown holds steady as I scale out, so it seems plausible I could take this out all the way.
4096 and 8192 nodes hit the socket error from Experiment 7. I am rerunning them.

Discussion
Obviously, the runs haven't worked yet and I am not sure I have validated our hypotheses. I am waiting some runs that would help firm this up. But perhaps it's worth going back to the drawing board to consider what else could be going on.

I was asked to do runs with libfabric logging enabled. The results are below.
This is still the same application configuration as https://github.com/StanfordLegion/legion/issues/1657#issuecomment-2017320621
Experiment 10: GASNET_OFI_RECEIVE_BUFF_SIZE=8M (with CXI logging)

Expand for full variables
export LD_LIBRARY_PATH=$DEV/build/$MECHANISM:$KERNEL_PATH:$DEV/legion/language/build/lib:$LD_LIBRARY_PATH
export FI_MR_CACHE_MONITOR=memhooks
export FI_CXI_RX_MATCH_MODE=software
export FI_LOG_LEVEL=warn
export FI_LOG_PROV=cxi
export GASNET_OFI_DEVICE_0=cxi2
export GASNET_OFI_DEVICE_1=cxi1
export GASNET_OFI_DEVICE_2=cxi3
export GASNET_OFI_DEVICE_3=cxi0
export GASNET_OFI_DEVICE_TYPE=Node
export GASNET_OFI_RECEIVE_BUFF_SIZE=8M
export MPICH_OFI_NIC_POLICY=BLOCK

Logs: http://sapling.stanford.edu/~eslaught/ammonia.run5-trace1-logs/pwave_x_8192_hept/run/
Application output is under the respective out_*.txt files. Legion logging is under run_*.log.
Note that like before, 1 to 4096 nodes ran to completion while 8192 nodes froze.
Experiment 11: GASNET_OFI_RECEIVE_BUFF_SIZE=recv and GASNET_OFI_NUM_RECEIVE_BUFFS=100000 (with CXI logging)

Expand for full variables
export LD_LIBRARY_PATH=$DEV/build/$MECHANISM:$KERNEL_PATH:$DEV/legion/language/build/lib:$LD_LIBRARY_PATH
export FI_MR_CACHE_MONITOR=memhooks
export FI_CXI_RX_MATCH_MODE=software
export FI_LOG_LEVEL=warn
export FI_LOG_PROV=cxi
export FI_CXI_RDZV_THRESHOLD=256
export FI_CXI_RDZV_GET_MIN=256
export GASNET_OFI_DEVICE_0=cxi2
export GASNET_OFI_DEVICE_1=cxi1
export GASNET_OFI_DEVICE_2=cxi3
export GASNET_OFI_DEVICE_3=cxi0
export GASNET_OFI_DEVICE_TYPE=Node
export GASNET_OFI_RECEIVE_BUFF_SIZE=recv
export GASNET_OFI_NUM_RECEIVE_BUFFS=100000
export MPICH_OFI_NIC_POLICY=BLOCK

Logs: http://sapling.stanford.edu/~eslaught/ammonia.run5-trace1-logs-buff-size-recv-100k/pwave_x_8192_hept/run/
Application output is under the respective out_*.txt files. Legion logging is under run_*.log.
Note that like before, 1 to 4096 nodes ran to completion while 8192 nodes froze.

There are too many experiments to track at this point, so I have moved the tracking into a spreadsheet here:
https://docs.google.com/spreadsheets/d/1OoDW9ie4uewUWaGHKQTOC8bIavGzQfxPNHE2EwJIY5Q/edit?usp=sharing

To follow up on this, we did eventually reach a set of variables that allow S3D to run up to 8,192 nodes on Frontier. Having said that, it's not entirely obvious that these variables are actually necessary or sufficient.
They may not be necessary because Seshu has been running recently without them, and got up to 8,192 nodes.
They may not be sufficient because we have still seen issues: Seshu in #1683 and myself in #1696
I'm not sure what else to say. There is more to do to figure out what actually is necessary and sufficient for running on these networks.

In the Realm meeting today I think we did identify a potential forward progress issue in the GASNetEX module. My investigation of the Realm code earlier asserting that Realm has an independent progress thread for pulling messages off of the wire is not actually correct:

Ok, I see where Realm is running a separate polling thread that always pulls active messages off the wire and that should be sufficient to ensure forward progress even if all the background worker threads get stuck in a commit call.

Instead, my first assertion that we had gotten rid of this progress thread and instead relied on the background worker threads to service incoming active messages is actually right:

I thought we had gotten rid of the polling thread in the GASNetEX module

Realm's GASNet1 networking module does have such a progress thread, but the GASNetEX module has indeed removed this polling progress thread.
Unfortunately this model of forward progress is only sound if calls to gex_AM_PrepareRequest* with the GEX_FLAG_IMMEDIATE flag will precisely predict whether the subsequent call to gex_AM_CommitRequest* will block or not. As we've previously established in this issue, GASNet cannot guarantee precision in this regard for reasons having to do with the details of the networking stack:

Regardless of whether or not GEX_FLAG_IMMEDIATE is passed to gex_AM_PrepareRequestMedium() (or FPAM injection), this is a "best effort" GASNet feature, and does not (cannot) provide a hard guarantee that the underlying network library will not block during injection. Currently I'd characterize GASNet's "best effort" here as "pretty good" on ibv-conduit and "not so great" on ofi-conduit. My understanding of the current state of IMMEDIATE injection in ofi-conduit is that FPAM injection should faithfully detect cases where GASNet's conduit-level metadata (buffer descriptors) are running scarce, and should correctly return failure IFF the underlying fi_inject/fi_send call to the libfabric provider returns -FI_EAGAIN. However ofi-conduit does not attempt to model/predict the resource state of the underlying provider, who might choose to block without returning -FI_EAGAIN.

What this means is that it is possible for all the background worker threads in Realm to get stuck trying to send active messages and not have any remaining threads available for polling incoming active messages to drain the network. The fact that we see this much more commonly on Slingshot than Infiniband aligns with the assertion that GASNet is much more precise in its responses about when blocking active messages are going to occur for Infiniband than Slingshot.
I suspect that we'll need to alter the architecture of the Realm GASNetEX module to look more like the GASNet1 module if this is the case in order to guarantee forward progress.
@manopapad for determining priorities for addressing this issue
@SeyedMir to check on similar properties for the UCX network module
@JiakunYan @streichler for general interest

Checked with the UCC team and we should be fine with UCX. ucp_worker_progress will not block the calling thread. Of course, this is not applicable/related to Slingshot in any way as UCX does not have support for Slingshot.

However ofi-conduit does not attempt to model/predict the resource state of the underlying provider, who might choose to block without returning -FI_EAGAIN.

@bonachea  Do you suspect that ofi communication calls (fi_send, fi_recv, etc) can be blocked indefinitely if users does not call the progress function (fi_cq_read*)?

@JiakunYan Together, @bonachea and I have looked previously at the libfabric documentation and we determined that it allows for the (unfortunate) behavior you describe.  However, we do not know the behavior of any particular providers (such as cxi) nor would there be any guarantee that the behavior would not change in a future release of libfabric.

@PHHargrove @bonachea  Thanks! To be honest, I would be very surprised if any provider actually behaves in this way in practice. If this is the case, it seems to me the only way to avoid this issue is to always have dedicated progress threads. It would put most MPI programs at the risk of deadlock.

@JiakunYan
I believe that at least one class of problematic conditions we identified are ones in which a sending process/thread was blocked in fi_send*() because the remote process was not processing completion records as needed to complete receives and/or was not providing the necessary buffer resources.  A table in the fi_domain manpage actually appears to require "retry" (and prohibit FI_EAGAIN) for the lack of a receive buffer at the target (for RDM endpoints and RM enabled).

@PHHargrove Thanks! I see what you mean. I don't understand why they have this weird "retry" requirement. It would still be good to make sure it is actually stuck inside the libfabric, instead of the inifinite gasnetex send-retry-progress loop.
FWIW, I did see libfabric progress is being called from @elliottslaughter freeze backtrace.
Thread 3 (Thread 0x7ffe5d20f740 (LWP 81080) "s3d.x"):
#0  0x00007fffdd8877df in cxip_evtq_progress (evtq=evtq@entry=0x67885c0) at prov/cxi/src/cxip_evtq.c:388
#1  0x00007fffdd8591c9 in cxip_ep_progress (fid=<optimized out>) at prov/cxi/src/cxip_ep.c:184
#2  0x00007fffdd85e969 in cxip_util_cq_progress (util_cq=0x40b6810) at prov/cxi/src/cxip_cq.c:112
#3  0x00007fffdd83a301 in ofi_cq_readfrom (cq_fid=0x40b6810, buf=<optimized out>, count=64, src_addr=0x0) at prov/util/src/util_cq.c:232
#4  0x00007fffe0f6f62c in gasnetc_ofi_tx_poll () from /ccs/home/eslaught/frontier/gb2024-subrank3/legion/language/build/lib/librealm.so.1
#5  0x00007fffe0f7211d in gasnetc_ofi_am_send_medium () from /ccs/home/eslaught/frontier/gb2024-subrank3/legion/language/build/lib/librealm.so.1
#6  0x00007fffe0fefd91 in gasnetc_AM_CommitRequestMediumM () from /ccs/home/eslaught/frontier/gb2024-subrank3/legion/language/build/lib/librealm.so.1
#7  0x00007fffe00d7628 in Realm::XmitSrcDestPair::push_packets (this=0x7ffd04031d50, immediate_mode=<optimized out>, work_until=...) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2092
#8  0x00007fffe00d925e in Realm::GASNetEXInjector::do_work (this=<optimized out>, work_until=...) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2725
#9  0x00007fffdff6d778 in Realm::BackgroundWorkManager::Worker::do_work (this=0x7ffe5d20d520, max_time_in_ns=<optimized out>, interrupt_flag=0x0) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/bgwork.cc:599
#10 0x00007fffdff6e051 in Realm::BackgroundWorkThread::main_loop (this=0x1fb2cdb0) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/bgwork.cc:103
#11 0x00007fffe007e541 in Realm::KernelThread::pthread_entry (data=0x1fdad8b0) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/threads.cc:831
#12 0x00007fffed8e96ea in start_thread () from /lib64/libpthread.so.0
#13 0x00007fffe912150f in clone () from /lib64/libc.so.6

(http://sapling.stanford.edu/~eslaught/s3d_freeze_debug2/bt4096-3/bt_frontier00001_81054.log line 322)

Here are the two earlier backtraces for the same thread on the same node:
Thread 3 (Thread 0x7ffe5d20f740 (LWP 81080) "s3d.x"):
#0  0x00007fffed8f19d5 in pthread_spin_lock () from /lib64/libpthread.so.0
#1  0x00007fffdd874207 in ofi_genlock_lock (lock=<optimized out>) at ./include/ofi_lock.h:359
#2  cxip_send_common (txc=0x6754838, tclass=512, buf=0x7ffe75a1f278, len=912, desc=<optimized out>, data=data@entry=0, dest_addr=74, tag=0, context=0x7ffe75a1f250, flags=83886080, tagged=false, triggered=false, trig_thresh=0, trig_cntr=0x0, comp_cntr=0x0) at prov/cxi/src/cxip_msg.c:5510
#3  0x00007fffdd875001 in cxip_send (fid_ep=<optimized out>, buf=<optimized out>, len=<optimized out>, desc=<optimized out>, dest_addr=<optimized out>, context=<optimized out>) at prov/cxi/src/cxip_msg.c:5874
#4  0x00007fffe0f72136 in gasnetc_ofi_am_send_medium () from /ccs/home/eslaught/frontier/gb2024-subrank3/legion/language/build/lib/librealm.so.1
#5  0x00007fffe0fefd91 in gasnetc_AM_CommitRequestMediumM () from /ccs/home/eslaught/frontier/gb2024-subrank3/legion/language/build/lib/librealm.so.1
#6  0x00007fffe00d7628 in Realm::XmitSrcDestPair::push_packets (this=0x7ffd04031d50, immediate_mode=<optimized out>, work_until=...) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2092
#7  0x00007fffe00d925e in Realm::GASNetEXInjector::do_work (this=<optimized out>, work_until=...) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2725
#8  0x00007fffdff6d778 in Realm::BackgroundWorkManager::Worker::do_work (this=0x7ffe5d20d520, max_time_in_ns=<optimized out>, interrupt_flag=0x0) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/bgwork.cc:599
#9  0x00007fffdff6e051 in Realm::BackgroundWorkThread::main_loop (this=0x1fb2cdb0) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/bgwork.cc:103
#10 0x00007fffe007e541 in Realm::KernelThread::pthread_entry (data=0x1fdad8b0) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/threads.cc:831
#11 0x00007fffed8e96ea in start_thread () from /lib64/libpthread.so.0
#12 0x00007fffe912150f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7ffe5d20f740 (LWP 81080) "s3d.x"):
#0  0x00007fffed8f19d5 in pthread_spin_lock () from /lib64/libpthread.so.0
#1  0x00007fffdd85e954 in ofi_genlock_lock (lock=0x40b6928) at ./include/ofi_lock.h:359
#2  cxip_util_cq_progress (util_cq=0x40b6810) at prov/cxi/src/cxip_cq.c:109
#3  0x00007fffdd83a301 in ofi_cq_readfrom (cq_fid=0x40b6810, buf=<optimized out>, count=64, src_addr=0x0) at prov/util/src/util_cq.c:232
#4  0x00007fffe0f6f62c in gasnetc_ofi_tx_poll () from /ccs/home/eslaught/frontier/gb2024-subrank3/legion/language/build/lib/librealm.so.1
#5  0x00007fffe0f7211d in gasnetc_ofi_am_send_medium () from /ccs/home/eslaught/frontier/gb2024-subrank3/legion/language/build/lib/librealm.so.1
#6  0x00007fffe0fefd91 in gasnetc_AM_CommitRequestMediumM () from /ccs/home/eslaught/frontier/gb2024-subrank3/legion/language/build/lib/librealm.so.1
#7  0x00007fffe00d7628 in Realm::XmitSrcDestPair::push_packets (this=0x7ffd04031d50, immediate_mode=<optimized out>, work_until=...) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2092
#8  0x00007fffe00d925e in Realm::GASNetEXInjector::do_work (this=<optimized out>, work_until=...) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2725
#9  0x00007fffdff6d778 in Realm::BackgroundWorkManager::Worker::do_work (this=0x7ffe5d20d520, max_time_in_ns=<optimized out>, interrupt_flag=0x0) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/bgwork.cc:599
#10 0x00007fffdff6e051 in Realm::BackgroundWorkThread::main_loop (this=0x1fb2cdb0) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/bgwork.cc:103
#11 0x00007fffe007e541 in Realm::KernelThread::pthread_entry (data=0x1fdad8b0) at /autofs/nccs-svm1_home1/eslaught/frontier/gb2024-subrank3/legion/runtime/realm/threads.cc:831
#12 0x00007fffed8e96ea in start_thread () from /lib64/libpthread.so.0
#13 0x00007fffe912150f in clone () from /lib64/libc.so.6

It's going around a loop here somewhere. Not sure if that tells us anything or not. I wouldn't necessarily rule out part of the problem also being in the Slingshot driver/hardware not behaving as we would expect either.

What the backtraces tell me is that Realm is not stuck in ofi send but in the gasnetex try-send-progress loop.
#define OFI_INJECT_RETRY_IMM(lock, fxn, poll_type, imm, label)\
    do {\
        GASNETC_OFI_LOCK_EXPR(lock, fxn);\
        if (ret == -FI_EAGAIN) {\
          if (imm) goto label; \
          GASNETI_SPIN_DOWHILE(ret == -FI_EAGAIN, {\
            GASNETC_OFI_POLL_SELECTIVE(poll_type);\
            GASNETC_OFI_LOCK_EXPR(lock, fxn);\
          });\
        } \
    }while(0)

#define GASNETC_OFI_POLL_SELECTIVE(type) do {\
    if (type == OFI_POLL_ALL) {\
        gasnetc_ofi_am_recv_poll_cold(1);\
        gasnetc_AMPSHMPoll(0);\
    }\
    else {\
        gasnetc_AMPSHMPoll(1);\
    }\
    gasnetc_ofi_am_recv_poll_cold(0);\
    gasnetc_ofi_tx_poll();\
}while(0)

The GASNETC_OFI_POLL_SELECTIVE here may not be comprehensive enough to make progress on all resources that ofi/cxi needs to issue the send request (e.g. poll the right completion queue, post buffers to the right endpoint).

@JiakunYan
Tracing down the code connected to your previous posting:
The backtraces show a call to gasnetc_AM_CommitRequestMediumM().
So, as a "request" call we know isreq==1 in gasnetc_medium_commit() (called from gasnetc_ofi_am_send_medium() but not show in the backtrace, presumably due to inlining).
Quoting from gasnetc_medium_commit():
    if (isreq) {
        ep = gasnetc_ofi_request_epfd;
        am_dest = gasnetc_fabric_addr(REQ, dest);
        poll_type = OFI_POLL_ALL;
    }

So, type == OFI_POLL_ALL in the code you've quoted.
There is no more "comprehensive" option than polling "ALL".

@lightsighter wrote (in part)

I wouldn't necessarily rule out part of the problem also being in the Slingshot driver/hardware not behaving as we would expect either

I agree that the "flow-control gets stuck" behavior we've seen and/or speculated about when speaking w/ HPE would likely lead to the observed lack of progress.

@PHHargrove Thanks for the explanation! I don't want to trouble you with too many questions. I will just poke around one last possibility.
The progress of a sending process also depends on its target process having receive buffers posted/completion queue polled, so only the sending process doing "comprehensive" progress is not enough. I see gasnetex is using two sets of endpoints and completion queues (one for request and one for reply). If all threads in process A sending requests to process B (so it only polls the request ep/cq) but process B is sending replies to process A (so it only polls the reply ep/cq), could this lead to a deadlock?

I agree that the "flow-control gets stuck" behavior we've seen and/or speculated about when speaking w/ HPE would likely lead to the observed lack of progress.

@PHHargrove Do you think that if Realm had a separate progress thread here that did nothing but poll for incoming active messages and pull them off the wire would it make a difference or would it not matter?

I agree that the "flow-control gets stuck" behavior we've seen and/or speculated about when speaking w/ HPE would likely lead to the observed lack of progress.

@PHHargrove Do you think that if Realm had a separate progress thread here that did nothing but poll for incoming active messages and pull them off the wire would it make a difference or would it not matter?

I think it would, at best, reduce the probability of the Slingshot network stack getting into  "bad state".  Same is true of using various environment variables to increase the buffering in the network stack. Would not care to speculate if that is "good enough" or not.

If the concern is that all the threads get stuck in a pattern such as the one shown in the stack traces (and assuming that a Slingshot bug is not the root cause), then perhaps there is a better approach than dedicating a progress/polling thread.
Imagine you maintain a counting semaphore with initial value one less than the number of threads.  Before an operation which might block, you "try down" the semaphore.  It that succeeds, then there must be at least one thread not attempting potentially blocking operations.  So, you proceed with the (potentially) blocking operation and "up" the semaphore on completion.
In the event you fail the "try down", you know that the caller is the "last" thread not to be in a potentially blocking operation.  In this case that thread could just begin alternating poll/try-down until obtaining the semaphore indicates at least one other thread is now not in a potentially blocking operation.  Alternatively, a failure to obtain the semaphore might be handled in the same manner as "immediate failure" (a return of GEX_AM_SRCDESC_NO_OP) of gex_AM_Prepare*(), though I don't actually know exactly when Realm does in that case.

Alternatively, a failure to obtain the semaphore might be handled in the same manner as "immediate failure" (a return of GEX_AM_SRCDESC_NO_OP) of gex_AM_Prepare*(), though I don't actually know exactly what Realm does in that case.

Realm should come back around and call gasnet_AMPoll here in that case:
https://gitlab.com/StanfordLegion/legion/-/blob/master/runtime/realm/gasnetex/gasnetex_internal.cc?ref_type=heads#L2955
It's not clear to me if that is equivalent to the polling loop already happening in gasnetc_AM_CommitRequestMediumM() (e.g. the loop referenced above) or whether gasnet_AMPoll does some additional kind of polling that might be necessary for forward progress.

Alternatively, a failure to obtain the semaphore might be handled in the same manner as "immediate failure" (a return of GEX_AM_SRCDESC_NO_OP) of gex_AM_Prepare*(), though I don't actually know exactly what Realm does in that case.

Realm should come back around and call gasnet_AMPoll here in that case: https://gitlab.com/StanfordLegion/legion/-/blob/master/runtime/realm/gasnetex/gasnetex_internal.cc?ref_type=heads#L2955
It's not clear to me if that is equivalent to the polling loop already happening in gasnetc_AM_CommitRequestMediumM() (e.g. the loop referenced above) or whether gasnet_AMPoll does some additional kind of polling that might be necessary for forward progress.

I don't believe there should be a difference, based on the dive I took into the code on Wed.

That is what I would expect as well. That suggests that the fact that Realm does not have an independent progress thread is not actually a forward progress issue. It might cause some performance hiccups but we should not hang as a result since GASNet is going to do polling inside the active message commit to ensure forward progress as necessary.

FWIW, I'm reasonably confident the failure to return from fi_cq_readfrom that @elliottslaughter  observed on Frontier is a Slingshot software bug. I found enough evidence in the backtraces to indicate that some progress calls were neither returning nor actually making progress. This is on top of the FI_MULTI_RECV issues that @PHHargrove has already isolated and reported.
We were never able to isolate a reproducer we could report for the progress hang (which I'd still like us to do) but I'm aware of at least one internal ticket with a lot of similar characteristics with an MPI reproducer that was very recently fixed. I have no way to predict the timeline for that actually making it to the DoE systems though, and without a reproducer no way to promise that it actually is the same issue.
I don't think we ever validated that processing the receives actually would have allowed progress to resume. It seems to me like figuring out how to test that would be a good first step.
I'm a little worried that we're trying to architect a solution based on reasoning about a software bug that is not functioning as expected or in a reasonable way. I'd be more interested in Realm or GASNet detecting when progress has stopped or slowed to a crawl and giving as much information as they can about what they were doing. Even once the software bugs are fixed, the Legion communication patterns tend to be different enough from standard MPI applications that users might need to change some of the default CXI provider settings based on their workload to get reasonable performance.


NO Mandatory Sign

Path: Setup>All Module
Issue: There is no Mandatory Sign  given for the fields.

Module can't be created anymore in the system.

@jaglans The name module is mandatory, over there no sign is given. but you can not save a record keeping that field empty. You must give a mandatory sign beside each field.

@jaglans this issue is not closed yet...


fix infinite loop in session test

fixes #598.
The mockReceivedPacketHandler returned the same ACK frame over and over again, so that the loop in session.sendPacket() would send packets containing this packet indefinitely.
Note that while this fixes the flaky test, there's still a race condition. I'll create a separate issue for this.

Codecov Report

Merging #599 into master will decrease coverage by 0.15%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #599      +/-   ##
==========================================
- Coverage   90.96%   90.81%   -0.15%
==========================================
  Files          79       79
  Lines        5432     5432
==========================================
- Hits         4941     4933       -8
- Misses        315      319       +4
- Partials      176      180       +4

Impacted Files
Coverage Δ

packet_packer.go
88.32% <0%> (-1.46%)
:arrow_down:

session.go
86.82% <0%> (-1.41%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cc2dc2a...ab88e94. Read the comment docs.

The Codecov diff is interesting. It shows that this test must have been triggering some race conditions quite frequently.


Remove the theming-plugin

Feature Request
Description
Remove theming-plugin: https://github.com/cerner/terra-dev-site/blob/master/src/theming-plugin.js
This is no longer leveraged in our wepback and the preferred theme aggregation is the terra-toolkit aggregate-themes pre-build script.
Should be able to remove the postcss dependency as well:
https://github.com/cerner/terra-dev-site/blob/master/package.json#L89

Resolved in v5


Impersonate user - privilege error messages presents the wrong id

In the case where user A is impersonating user B, and user A does not have permission to the action performed, the following error message is returned:

Principal user (Id=<guid>, type=8) is missing prvReadOpportunity privilege (Id=f7767f74-628d-48b3-a79f-9c6e24fedb69)'

The principal user id is always the systemuserid of the impersonated user (User B), not User A. So when the impersonated user has the necessary permissions the error message is misleading.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e217f1ee-48c8-6ac4-f51f-52057470ad7c
Version Independent ID: 6ff8aca3-a5f9-da05-395f-792637a9fc52
Content: Impersonate another user (Developer Guide for Dynamics 365 Customer Engagement)
Content Source: ce/developer/org-service/impersonate-another-user.md
Service: crm-online
GitHub Login: @JimDaly
Microsoft Alias: jdaly

Sorry for the delay in responding.
The scope of the issues tracked for this repository is feedback on the documentation.  What you are reporting here looks like a product issue, and it should be reported using the product feedback process.


shortest_Job_First.cpp

Shortest Job First (SJF) is a CPU scheduling algorithm used in operating systems. In SJF, the process with the smallest execution time (burst time) is given the highest priority and is executed first. It minimizes waiting time and ensures that shorter jobs are completed before longer ones, making it an efficient algorithm for reducing turnaround time and improving system responsiveness.
"SJF is like a cafe that serves the shortest line of customers first—because even CPUs appreciate efficiency and shorter wait times!"

Added into the folder mam.


Update README.md

Added dangerously brief description.
Changes proposed in this merge request:
-1,6 +1,6

@nnichols I'm going to just merge this - reverse it if you hate it.


H264 Playback/Sync issues with AML-grabber

Configuration:
Vero 4K with OSMC, Debian 9.2, Kodi 17.6 (also on previous versions).
Hyperion: latest "classic" version and latest NG version.
Audio DTS/AC3 in passthrough mode or AAC as PCM
h264 encoded files
Issue: after some time playback seems to "slow down" while audio continues at normal speed. When I skip back 10s the video/audio re-syncs and playback continues as normal (for some time).
Issue does not occur when aml-grabber is disabled.

when aml capturing is used (kernel side) hw acceleration is off, we cant fo anything from user space (hyperion). amlogic must implement a stable hw accelerated capturing api...
this is not a hyperion issue, it is the aml platform itself.

thanks. will also report this back to the OSMC forum.

can this be documented with the aml-grabber. I know at least of 1 other user who has the same issue. I can bypass this as I also have a v4L-grabber. In it's current state the aml-grabber is not very usefull.


Prevent withVariantReselection from doing anything useful

This is a hack

@bot-gradle test QF

I've triggered the following builds for you:

QuickFeedback build


Support the Checkpoint API

K8s 1.25 introduced the Checkpoint API https://betterprogramming.pub/backup-and-restore-of-containers-with-kubernetes-checkpointing-api-e310cf29cd4a. It would be great to have support in kubeletctl

Hi @christophetd
We added support to the /checkpoint API.

Great to hear, thanks!


[simple] remove redundant value merge.

as titled, provide a default dictionary. that’s it. make code simpler and also make type check simpler.

Yep, makes sense. Let me close this.

Appreciate the cleanup though!


Add Romanian language support

This PR adds Romanian (code: ro) localization to the Social app.

I found that another Romanian localization PR #6456
Would you mind integrating your contributions into that PR? I think having just one PR would help speed up the Romanian localization effort.

Thanks, I'll have a look. I haven't noticed it.


set HikariCP log level to ERROR

What type of PR is this：

[ ] bug
[ ] feature
[ ] enhancement
[ ] refactor
[ ] others

Which issues of this PR fixes ：

Fixes #
Problem Summary(Required) ：

@mergifyio backport branch-2.3


How best to keep updates and modify project schema?

Hey @AXeL-dev !
Hope you are doing well!
I have been able to modify this project for my schema but I want to understand how can I leverage updates to the project. I saw quite a few changes since I forked this repo.
Do you have any suggestions?

Hey,
I'm great, i hope you're doing well, as well!

I saw quite a few changes since I forked this repo.

As you said, there was no significant changes to the project lately, which is due to my occupation on other projects & mostly on my full time job, so for now, all i can handle is few changes from time to time.

how can I leverage updates to the project

I don't know if you are talking about updating your fork, or pushing your updates to the main project, but in the first case, since you're using a custom fork of the project, you have to update it manually, using git for example, you can simply create an upstream branch which will be always updated from the main repo., then merge it with your development branch & handle conflicts manually. A plugins system can be implemented in the future to separate core features from custom ones & give other developers the freedom of customising the CMS without headaches.

Thanks for the reply. I have added an upstream and I think that should solve the problem.
I had another doubt - how do I deploy this to firebase hosting? Will this work on firebase hosting? I tried the steps here https://angular.io/start/start-deployment but it does not work. I can only see the default index.html that firebase has created not this project.
I did build the project by running npm run-script build

You're welcome :wink:

how do I deploy this to firebase hosting? Will this work on firebase hosting?

Yes it should, since the demo is working on github pages, it should also work on any PHP or nodejs powered hosting.

I did build the project by running npm run-script build

In fact after checking the scripts on the package.json:
https://github.com/FirebaseGoodies/FireAdmin/blob/485ea74260cbcaf37ca508b932525f175bdd9356/package.json#L4-L15
I figured out or remembered that the project is in fact a workspace, containing both demo & library projects, so when you run npm run-script build it will in fact build the library not the demo project, so in your case, to build the demo project, you have to type the following command: ng build demo --prod
Deploy steps
First, make sure you have the firebase-tools package installed by running:
npm install -g firebase-tools

Next, if you didn't init firebase yet on your project you should do it:
firebase init hosting

If you're asked to choose the deploy folder you have to type: dist/demo, if you missed it, you can change that manually on the firebase.json file
Also, you have to login to your firebase account if you're not logged in:
firebase login

Finally, run the deploy command (don't forget to build the demo project first) :
firebase deploy --only hosting

P.S.: i hope i didn't forget a step :laughing:

This was the culprit - ng build demo --prod
Thanks a lot! Can I work as an intern for you?! 😆 There's so much to learn from you!

Haha, honestly i hate leader roles, i prefer to be a silent ghost :eyes:, i'm just sick of perfection, so even when it comes to help someone, i always try to provide the best explanation & the maximum details lool :laughing: (need a perfection therapy :roll_eyes: )


Useful error descriptions

"unknown error" is mildly rage inducing.

Note: The CI fail is unrelated.

Thank you for improving error handling. The PR is now added to master. Do you think it is worth it to add variants for all other error types to the cbor type, or is it ok to map them all to custom error?

It would be nice but I'd say that's pretty low priority. IMO, one shouldn't even try to automatically recover from (de)serialization errors. The only errors one needs to be able to distinguish between are IO/non-IO errors.
I only made this pull request to make debugging easier.


Update manage-application-permissions.md

Typo on line 106
Connect-MgGraph -Scopes "Application.ReadWrite.All", "Directory.ReadWrite.All", "DelegatedPermissionGrant.ReadWrite.All" "AppRoleAssignment.ReadWrite.All" -> Connect-MgGraph -Scopes "Application.ReadWrite.All", "Directory.ReadWrite.All", "DelegatedPermissionGrant.ReadWrite.All", "AppRoleAssignment.ReadWrite.All"
Incorrect command on line 122
$spApplicationPermissions = Get-MgServicePrincipalAppRoleAssignedTo -ServicePrincipalId $Sp.Id -All | Where-Object { $.PrincipalType -eq "ServicePrincipal" } -> $spApplicationPermissions = Get-MgServicePrincipalAppRoleAssignment -ServicePrincipalId $Sp.Id -All | Where-Object { $.PrincipalType -eq "ServicePrincipal" }

@Jackson-Woods
Can you review the proposed changes?
When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

@Jackson-Woods
Thanks for approving the PR. For future PRs, please note that a #sign-off comment is the best way to signal to the review team that a PR is ready to merge.


docs: publicize web component based design systems

Adds list of design systems to the FAQ page's "Deep Dives" section.

uh Ionic should be there as well :+1:

Very nice :) Can we add something like this:
https://github.com/open-wc/open-wc/pull/351/files#diff-9a50e31fb068c43b65d32ae46a490d5fR7
"Want to add your design system to our list? Make an issue or a pull request on our github"

another one
https://component.kitchen/elix

Great contribution, very useful for people I think!
Is design systems the best name for this? Judging from the description the intention is to direct people to component/element sets, so something along those lines might be more eye catching. Under 'design systems' I'd expect more a showcase of UX concepts like https://material.io/

Good point @LarsDenBakker! Does “Element Libraries” communicate more clearly? I plan to add one lines on each of these on Monday and tweaking the title/URL before this is fully merged makes a lot of sense.

@Westbrook maybe "Web component collections" ?

Component Libraries?

remembered another one...
https://www.htmlelements.com
really good we are writing this down now... keeping that all in my head didn't work out as well as I thought 😅🙈😂

Updated the listing to include some editorialization. Mainly copping the content from the individual maintainers, but sometimes putting my own thoughts on to their early documentation. Please help to correct what are likely to be many avoidable grammatical infractions.

thaaankk you for your contribution :heart:


Validate the copy from clipboard input

We do not want to paste from the clipboard if the pasted value is not hex.
This would make the ux better.
Also reconsider if the paste should be behind a button and the copying should be by double-click.

As the paste from clipboard is not natively implemented on all user agents this will not be worked on


Implement GPT3/GPT-4

We want to use the most powerful model on the planet via the Openai API
https://github.com/openai/openai-python

General OpenAI implementation requirements:

implement retrying requests with exponential backlog (more info)

can use Backoff or Tenacity libraries

batching
Guidance on how to finetune

Free rate limits:

20 requests /min
150, 000 tokens /min

Implementation:

Use the OpenAI or Azure OpenAI model to generate completions and prompts
Can transcribe the videos for ease of finding videos and to increase accessibility

Benefits:

will learn quickly

Limitations:

rate limited as outlined above
need to apply for access to Azure OpenAI

@dianzrong any progress on this?


#600: Fix deprecated storybook

-
-

Issue
https://github.com/ripe-tech/ripe-white/issues/600

Dependencies

Decisions
--

Animated GIF
--

@BeeMargarida please split this and the package fixing into 2 separate PRs. I know it's a pain but we really need to be able to look at them on their own, revert just one if necessary, etc.


@bentley/imodeljs-native 3.3.3

Locked down ts-node to 10.8.0 because this bug surfaced in core-common cover script failing windows builds with path not found and causing upload of coverage report to fail on macOS.
Locked down superagent to 7.1.3 because 7.1.6 produced following backend-integration-test failures in electron only:
  1) Background map (#integration)
       obscures model based on settings:

      AssertionError: expected true to equal false
      + expected - actual

      -true
      +false

      at expectColor (home/vsts/work/1/s/full-stack-tests/core/src/frontend/map/BackgroundMap.test.ts:69:72)
      at expectPixelTypes (home/vsts/work/1/s/full-stack-tests/core/src/frontend/map/BackgroundMap.test.ts:73:7)
      at home/vsts/work/1/s/full-stack-tests/core/src/frontend/map/BackgroundMap.test.ts:117:9
      at testOnScreenViewport (home/vsts/work/1/s/full-stack-tests/core/src/frontend/TestViewport.ts:336:5)
      at Context.<anonymous> (home/vsts/work/1/s/full-stack-tests/core/src/frontend/map/BackgroundMap.test.ts:112:5)

  2) RealityDataAccess (#integration)
       should be able to call getPublisherProductInfo on RealityDataSource for all supported displayable reality data in itwin project:
     Error: Browser-only version of superagent could not find XHR
      at parse (home/vsts/work/1/s/core/frontend/src/request/Request.ts:169:19)
      at request (home/vsts/work/1/s/core/frontend/src/request/Request.ts:424:25)
      at RealityDataSourceContextShareImpl.getRealityDataTileJson (home/vsts/work/1/s/core/frontend/src/RealityDataSourceContextShareImpl.ts:119:18)
      at RealityDataSourceContextShareImpl.getPublisherProductInfo (home/vsts/work/1/s/core/frontend/src/RealityDataSourceContextShareImpl.ts:248:28)
      at Context.<anonymous> (home/vsts/work/1/s/full-stack-tests/core/src/frontend/standalone/RealityDataAccess.test.ts:192:23)

  3) RealityDataAccess (#integration)
       should be able to call getFileInfo when RealityDataSource is a 3dTile reality data:
     Error: Browser-only version of superagent could not find XHR
      at parse (home/vsts/work/1/s/core/frontend/src/request/Request.ts:169:19)
      at request (home/vsts/work/1/s/core/frontend/src/request/Request.ts:424:25)
      at RealityDataSourceContextShareImpl.getRealityDataTileJson (home/vsts/work/1/s/core/frontend/src/RealityDataSourceContextShareImpl.ts:119:18)
      at Context.<anonymous> (home/vsts/work/1/s/full-stack-tests/core/src/frontend/standalone/RealityDataAccess.test.ts:211:32)

  4) RealityDataAccess (#integration)
       should be able to call getSpatialLocationAndExtents on RealityDataSource for all supported displayable reality data in itwin project:
     Error: Browser-only version of superagent could not find XHR
      at parse (home/vsts/work/1/s/core/frontend/src/request/Request.ts:169:19)
      at request (home/vsts/work/1/s/core/frontend/src/request/Request.ts:424:25)
      at RealityDataSourceContextShareImpl.getRealityDataTileJson (home/vsts/work/1/s/core/frontend/src/RealityDataSourceContextShareImpl.ts:119:18)
      at RealityDataSourceContextShareImpl.getSpatialLocationAndExtents (home/vsts/work/1/s/core/frontend/src/RealityDataSourceContextShareImpl.ts:221:28)
      at Context.<anonymous> (home/vsts/work/1/s/full-stack-tests/core/src/frontend/standalone/RealityDataAccess.test.ts:228:33)

@kabentley @MichaelBelousov the churn around HubAccess, HubMock, LocalHub, CloudSqlite, etc between master and imodel02 produced quite the constellation of merge conflicts. Please review.

@kabentley I am stuck on core-backend due to conflicts in TileCache.test.ts. Please resolve.

@grigasp @saskliutas @roluk presentation test failure please investigate.
  1) PresentationManager
       addon results conversion to Presentation objects
         getContentDescriptor
           returns content descriptor:
     MockException: expected invocation of Function(It.isAny(),It.is((serializedParam) => {
                const param = JSON.parse(serializedParam);
                expectedParams = JSON.parse(JSON.stringify(expectedParams));
                return deepEqual(param, expectedParams);
            })) exactly 1 times, invoked 0 times
 Configured setups:
 Function(It.is(function (x) { return _.isEqual(x, a); })), at least 1 times
 Function(It.isAny(),It.isAnyString())
 Function(It.isAny(),It.is((serializedParam) => {
                const param = JSON.parse(serializedParam);
                expectedParams = JSON.parse(JSON.stringify(expectedParams));
                return deepEqual(param, expectedParams);
            }))

 Performed invocations:
 Function(["/home/paul/c/j2/s/itwinjs/presentation/backend/lib/cjs/assets/supplemental-presentation-rules"])
 Function(["/home/paul/c/j2/s/itwinjs/presentation/backend/lib/cjs/assets/primary-presentation-rules"])
 Function("Function")
 Function("undefined","{\"requestId\":\"GetContentDescriptor\",\"params\":{\"rulesetId\":\"laborum reprehenderit-4a4a69da79af8b42259f314fbfc5a77d\",\"displayType\":\"withdrawal\",\"keys\":{\"instanceKeys\":[[\"SQL\",[\"0x124220000015528\"]]],\"nodeKeys\":[{\"version\":2,\"type\":\"ECInstancesNode\",\"pathFromRoot\":[\"631315c6-1a54-4503-826f-ad8ef3dc5d71\",\"47883321-d07f-45f3-8cfe-89fc00a4a3db\"],\"instanceKeys\":[{\"className\":\"calculating\",\"id\":\"0x78730000005996\"},{\"className\":\"Wooden\",\"id\":\"0x10d54000000e805\"}]}]},\"selection\":{\"providerName\":\"COM\",\"level\":32444},\"contentFlags\":512}}")

      at MockException.Exception [as constructor] (/home/paul/c/j2/s/itwinjs/common/temp/node_modules/.pnpm/typemoq@2.1.0/node_modules/typemoq/dist/Error/Error/Error/Exception.ts:3:9)
      at new MockException (/home/paul/c/j2/s/itwinjs/common/temp/node_modules/.pnpm/typemoq@2.1.0/node_modules/typemoq/dist/Error/Error/Error/MockException.ts:20:9)
      at InterceptorExecute.throwVerifyCallCountException (/home/paul/c/j2/s/itwinjs/common/temp/node_modules/.pnpm/typemoq@2.1.0/node_modules/typemoq/dist/InterceptorExecute.ts:75:17)
      at InterceptorExecute.verifyCallCount (/home/paul/c/j2/s/itwinjs/common/temp/node_modules/.pnpm/typemoq@2.1.0/node_modules/typemoq/dist/InterceptorExecute.ts:64:18)
      at DynamicMock.verify (/home/paul/c/j2/s/itwinjs/common/temp/node_modules/.pnpm/typemoq@2.1.0/node_modules/typemoq/dist/DynamicMock.ts:44:31)
      at verifyMockRequest (src/test/PresentationManager.test.ts:688:26)
      at verifyWithSnapshot (src/test/PresentationManager.test.ts:696:7)
      at Context.<anonymous> (src/test/PresentationManager.test.ts:1354:9)

@kabentley I will attempt to infer intended release tags.
Warning: You have changed the public API signature for this project. Updating /home/paul/c/j2/s/itwinjs/common/api/core-backend.api.md
Error: src/IModelDb.ts:2093:1 - (ae-missing-release-tag) "CloudContainerArgs" is exported by the package, but it is missing a release tag (@alpha, @beta, @public, or @internal)
Error: src/IModelDb.ts:2095:1 - (ae-missing-release-tag) "SnapshotDbOpenArgs" is exported by the package, but it is missing a release tag (@alpha, @beta, @public, or @internal)
Error: src/LocalHub.ts:49:1 - (ae-missing-release-tag) "LockStatusExclusive" is exported by the package, but it is missing a release tag (@alpha, @beta, @public, or @internal)
Error: src/LocalHub.ts:55:1 - (ae-missing-release-tag) "LockStatusShared" is exported by the package, but it is missing a release tag (@alpha, @beta, @public, or @internal)
Error: src/workspace/Workspace.ts:35:1 - (ae-missing-release-tag) "WorkspaceAccount" is exported by the package, but it is missing a release tag (@alpha, @beta, @public, or @internal)
Error: src/workspace/Workspace.ts:51:1 - (ae-different-release-tags) This symbol has another declaration with a different release tag
Error: src/workspace/Workspace.ts:51:1 - (ae-missing-release-tag) "WorkspaceContainer" is exported by the package, but it is missing a release tag (@alpha, @beta, @public, or @internal)
Error: src/workspace/Workspace.ts:84:1 - (ae-different-release-tags) This symbol has another declaration with a different release tag
Error: src/workspace/Workspace.ts:84:1 - (ae-missing-release-tag) "WorkspaceDb" is exported by the package, but it is missing a release tag (@alpha, @beta, @public, or @internal)
Error: src/workspace/Workspace.ts:133:1 - (ae-missing-release-tag) "WorkspaceResource" is exported by the package, but it is missing a release tag (@alpha, @beta, @public, or @internal)
Error: src/workspace/Workspace.ts:202:1 - (ae-missing-release-tag) "WorkspaceCloudCacheProps" is exported by the package, but it is missing a release tag (@alpha, @beta, @public, or @internal)

@kabentley SnapshotOpenOptions has been superseded in @public methods like SnapshotDb.openFile/Checkpoint by SnapshotDbOpenArgs which mixes in CloudContainerArgs which has a property of type IModelJsNative.CloudContainer which is part of the internal API of @bentley/imodeljs-native. What is your intent here?

@MichaelBelousov @kabentley the following two failures occur only on Windows.
--[ FAILURE: @itwin/core-common ]------------------[ 6 minutes 40.7 seconds ]--

Path contains invalid characters: D:\vsts_b\15\s\core\common\lib\cjs\test\coverage\lcov-report\src\file:\D:\vsts_b\15\s\core\common\src

--[ FAILURE: @itwin/core-transformer ]-------------[ 6 minutes 40.7 seconds ]--

The filename, directory name, or volume label syntax is incorrect.

@kabentley docs errors.
Invalid Link: "[SettingSchemas]($backend)" in file "learning/backend/workspace"
Invalid Link: "[IModelHostConfiguration.workspace.settingsFiles]($backend)" in file "learning/backend/workspace"
Invalid Link: "[IModelHost.appWorkspace.settings]($backend)" in file "learning/backend/workspace"
Invalid Link: "[CloudSqlite]($backend)" in file "learning/backend/workspace"

Backend integration tests have one new failure followed by a crash.
2022-06-19T21:52:50.7234596Z   BriefcaseManager
2022-06-19T21:52:51.0963291Z Error | GeoCoord | Error: attach error: HTTP/1.1 403 Server failed to authenticate the request. Make sure the value of the Authorization header is formed correctly including the signature., account=http://<IP_ADDRESS>:10000/devstoreaccount1
2022-06-19T21:52:51.1155121Z     1) should open and close an iModel from the Hub
2022-06-19T21:52:53.2764764Z     ✔ should reuse checkpoints (2159ms)
2022-06-19T21:52:55.1187114Z Stadium Dataset 1 Download Progress ... 21.79%Stadium Dataset 1 Download Progress ... 42.31%Stadium Dataset 1 Download Progress ... 60.26%Stadium Dataset 1 Download Progress ... 79.49%Stadium Dataset 1 Download Progress ... 100.00%
2022-06-19T21:52:56.4634335Z download took 3.064 seconds
2022-06-19T21:52:56.5579253Z     ✔ should be able to show progress when downloading a briefcase (#integration) (3281ms)
2022-06-19T21:52:57.7677294Z     ✔ Should be able to cancel an in progress download (#integration) (1209ms)
2022-06-19T21:52:57.7682223Z
2022-06-19T21:52:57.7683401Z   ChangedElements
2022-06-19T21:52:58.4116054Z Error | DgnCore.VersionCompare | ProcessChangesets: Db filename does not exist on disk
2022-06-19T21:53:06.3065324Z Segmentation fault (core dumped)
2022-06-19T21:53:06.3298310Z Azurite Blob service is closing...
2022-06-19T21:53:06.3304811Z Azurite Blob service successfully closed
2022-06-19T21:53:06.3397095Z ERROR: "test:integration:core" exited with 139.
2022-06-19T21:53:06.3514275Z ERROR: "test:integration:azurite" exited with 1.
2022-06-19T21:53:06.3687643Z ##[error]Bash exited with code '1'.
2022-06-19T21:53:06.3704094Z ##[section]Finishing: Run Backend Tests

core-full-stack integration tests have 4 new failures (electron only).
  1) Background map (#integration)
       obscures model based on settings:

      AssertionError: expected true to equal false
      + expected - actual

      -true
      +false

      at expectColor (home/vsts/work/1/s/full-stack-tests/core/src/frontend/map/BackgroundMap.test.ts:69:72)
      at expectPixelTypes (home/vsts/work/1/s/full-stack-tests/core/src/frontend/map/BackgroundMap.test.ts:73:7)
      at home/vsts/work/1/s/full-stack-tests/core/src/frontend/map/BackgroundMap.test.ts:117:9
      at testOnScreenViewport (home/vsts/work/1/s/full-stack-tests/core/src/frontend/TestViewport.ts:336:5)
      at Context.<anonymous> (home/vsts/work/1/s/full-stack-tests/core/src/frontend/map/BackgroundMap.test.ts:112:5)

  2) RealityDataAccess (#integration)
       should be able to call getPublisherProductInfo on RealityDataSource for all supported displayable reality data in itwin project:
     Error: Browser-only version of superagent could not find XHR
      at parse (home/vsts/work/1/s/core/frontend/src/request/Request.ts:169:19)
      at request (home/vsts/work/1/s/core/frontend/src/request/Request.ts:424:25)
      at RealityDataSourceContextShareImpl.getRealityDataTileJson (home/vsts/work/1/s/core/frontend/src/RealityDataSourceContextShareImpl.ts:119:18)
      at RealityDataSourceContextShareImpl.getPublisherProductInfo (home/vsts/work/1/s/core/frontend/src/RealityDataSourceContextShareImpl.ts:248:28)
      at Context.<anonymous> (home/vsts/work/1/s/full-stack-tests/core/src/frontend/standalone/RealityDataAccess.test.ts:192:23)

  3) RealityDataAccess (#integration)
       should be able to call getFileInfo when RealityDataSource is a 3dTile reality data:
     Error: Browser-only version of superagent could not find XHR
      at parse (home/vsts/work/1/s/core/frontend/src/request/Request.ts:169:19)
      at request (home/vsts/work/1/s/core/frontend/src/request/Request.ts:424:25)
      at RealityDataSourceContextShareImpl.getRealityDataTileJson (home/vsts/work/1/s/core/frontend/src/RealityDataSourceContextShareImpl.ts:119:18)
      at Context.<anonymous> (home/vsts/work/1/s/full-stack-tests/core/src/frontend/standalone/RealityDataAccess.test.ts:211:32)

  4) RealityDataAccess (#integration)
       should be able to call getSpatialLocationAndExtents on RealityDataSource for all supported displayable reality data in itwin project:
     Error: Browser-only version of superagent could not find XHR
      at parse (home/vsts/work/1/s/core/frontend/src/request/Request.ts:169:19)
      at request (home/vsts/work/1/s/core/frontend/src/request/Request.ts:424:25)
      at RealityDataSourceContextShareImpl.getRealityDataTileJson (home/vsts/work/1/s/core/frontend/src/RealityDataSourceContextShareImpl.ts:119:18)
      at RealityDataSourceContextShareImpl.getSpatialLocationAndExtents (home/vsts/work/1/s/core/frontend/src/RealityDataSourceContextShareImpl.ts:221:28)
      at Context.<anonymous> (home/vsts/work/1/s/full-stack-tests/core/src/frontend/standalone/RealityDataAccess.test.ts:228:33)

Why so many dependencies were added to map-layers-auth? It shouldn't require react at all?

You tell me

Why so many dependencies were added to map-layers-auth? It shouldn't require react at all?

You tell me

I did fix this right before 3.2, see

I did fix this right before 3.2, see

Thanks. This is why running rush update on imodel02 branch is bad idea.

Thanks. This is why running rush update on imodel02 branch is bad idea.

Looks more like a bad merge into the imodel02/native branch; not sure how that happened. But agreed rush updates in that branch is a bad idea.

Please let me know if you want me to push a fix for this bad merge

Yes please.

@calebmshafer please review pinned dependencies.

@calebmshafer please review pinned dependencies.

@pmconne reviewing now.

Looks like the merge reintroduced a pretest script from #2430 that isn't in master for the transformer build process.
This appears to trigger a windows-specific bug in nyc which I've reported here.
The pretest script isn't necessary and I'm going to push a commit removing it shortly.

@kabentley may the GeoCoord errors be causing some tests to time out after 60 seconds?
2022-06-20T22:48:45.9724507Z     Using variables in rules
2022-06-20T22:48:45.9725629Z Error | GeoCoord | Error: attach error: Timeout was reached, account=https://gcs-data.itwin.org/ {"ActivityId":"1d80902b-1d74-43c6-ae72-2f08088914dd","SessionId":"4739dc65-a4ee-43b7-8f93-92cefbbc63ae","ApplicationId":"2686","ApplicationVersion":"1.0.0","rpcMethod":"openFile"}
2022-06-20T22:48:45.9726514Z       1) "before each" hook for "can specify lots of ids with Id64[] overload"

2022-06-20T23:00:47.2140261Z   1) Ruleset Variables
2022-06-20T23:00:47.2140787Z        Using variables in rules
2022-06-20T23:00:47.2141334Z          "before each" hook for "can specify lots of ids with Id64[] overload":
2022-06-20T23:00:47.2143162Z      Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/agent/_work/1/s/full-stack-tests/presentation/lib/frontend/RulesetVariables.test.js)
2022-06-20T23:00:47.2144143Z
2022-06-20T23:00:47.2144497Z
2022-06-20T23:00:47.2144972Z   2) Learning Snippets
2022-06-20T23:00:47.2145467Z        "before each" hook for "using for customization":
2022-06-20T23:00:47.2146931Z      Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/agent/_work/1/s/full-stack-tests/presentation/lib/learning-snippets/common/RelatedInstanceSpecification.test.js)
2022-06-20T23:00:47.2148045Z       at processTicksAndRejections (node:internal/process/task_queues:96:5)
2022-06-20T23:00:47.2148435Z
2022-06-20T23:00:47.2148806Z   3) Learning Snippets
2022-06-20T23:00:47.2149328Z        "before each" hook for "uses `handleTargetClassPolymorphically` attribute":
2022-06-20T23:00:47.2150764Z      Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/agent/_work/1/s/full-stack-tests/presentation/lib/learning-snippets/content/customization/RelatedPropertiesSpecification.test.js)
2022-06-20T23:00:47.2151740Z       at runMicrotasks (<anonymous>)
2022-06-20T23:00:47.2152324Z       at processTicksAndRejections (node:internal/process/task_queues:96:5)
2022-06-20T23:00:47.2152726Z
2022-06-20T23:00:47.2153089Z   4) Learning Snippets
2022-06-20T23:00:47.2153489Z        Content
2022-06-20T23:00:47.2153937Z          "before each" hook for "uses `label` attribute":
2022-06-20T23:00:47.2155201Z      Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/agent/_work/1/s/full-stack-tests/presentation/lib/learning-snippets/ContentCustomization.test.js)
2022-06-20T23:00:47.2156091Z       at runMicrotasks (<anonymous>)
2022-06-20T23:00:47.2156671Z       at processTicksAndRejections (node:internal/process/task_queues:96:5)
2022-06-20T23:00:47.2157055Z
2022-06-20T23:00:47.2157444Z   5) Learning Snippets
2022-06-20T23:00:47.2157946Z        "before each" hook for "uses `class` attribute":
2022-06-20T23:00:47.2159252Z      Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/agent/_work/1/s/full-stack-tests/presentation/lib/learning-snippets/customization/DisabledSortingRule.test.js)
2022-06-20T23:00:47.2160164Z       at runMicrotasks (<anonymous>)
2022-06-20T23:00:47.2160731Z       at processTicksAndRejections (node:internal/process/task_queues:96:5)
2022-06-20T23:00:47.2161125Z
2022-06-20T23:00:47.2161488Z   6) Learning Snippets
2022-06-20T23:00:47.2161979Z        "before each" hook for "uses `hideIfNoChildren` attribute":
2022-06-20T23:00:47.2163311Z      Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/agent/_work/1/s/full-stack-tests/presentation/lib/learning-snippets/hierarchies/specifications/Shared.test.js)
2022-06-20T23:00:47.2164470Z       at runMicrotasks (<anonymous>)
2022-06-20T23:00:47.2165040Z       at processTicksAndRejections (node:internal/process/task_queues:96:5)

backend-integration-tests are failing.
  Checkpoints
INFO(e) [2022/06/20 22:35:19.632]: listening on localhost port 2030
    1) should use fallback directory when blockcache_dir has no daemon in it
    2) "after each" hook for "should use fallback directory when blockcache_dir has no daemon in it"
    3) "after all" hook for "should be able to open and read V2 checkpoint with daemon running"

  Cloud workspace containers
    4) cloud workspace

  IModelOpen
    ✔ Unauthorized requests should cause an obvious error
    5) should be able to handle simultaneous open calls
terminate called after throwing an instance of 'Napi::Error'
  what():  Error in native callback
Aborted (core dumped)
Azurite Blob service is closing...
Azurite Blob service successfully closed
ERROR: "test:integration:core" exited with 134.
ERROR: "test:integration:azurite" exited with 1.

backend-integration-tests are failing then terminating abnormally. I tried pinning azurite to the version (3.14.0) used on master to no effect.

I will have to talk to @nick4598. Something about starting/stopping the daemon isn't right. I skipped those tests for now.


mapserver: fix build with libxml2 2.12

Description of changes
Fix mapserver build with libxml2 2.12.
This PR is based on https://github.com/MapServer/MapServer/pull/6975 ,
which can't be applied without other patches.
Required for https://github.com/NixOS/nixpkgs/pull/275620

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Can't you use this URL instead: https://github.com/MapServer/MapServer/commit/2c1d4f928b9657dbb4517ecbfc8cac30cc73ae9f.patch ?

Can't you use this URL instead: https://github.com/MapServer/MapServer/commit/2c1d4f928b9657dbb4517ecbfc8cac30cc73ae9f.patch ?

This was my first attempt, but
    (fetchpatch {
      url = "https://github.com/MapServer/MapServer/commit/2c1d4f928b9657dbb4517ecbfc8cac30cc73ae9f.patch";
      hash = "sha256-Muar7NU7sQ/bhYF+Sg/JXOCNjpvOfIZw4YYgrjBb1N4=";
    })

failed on
Running phase: patchPhase
applying patch /nix/store/jhr0spcjk9y25jysrkxb460if8abadjn-2c1d4f928b9657dbb4517ecbfc8cac30cc73ae9f.patch
patching file mapows.c
Hunk #1 FAILED at 162.
1 out of 1 hunk FAILED -- saving rejects to file mapows.c.rej
patching file mapwcs.cpp
Hunk #1 FAILED at 383.
1 out of 1 hunk FAILED -- saving rejects to file mapwcs.cpp.rej
patching file mapwcs20.cpp
Hunk #1 FAILED at 1459.
1 out of 1 hunk FAILED -- saving rejects to file mapwcs20.cpp.rej

I am actually not sure why.

Most probably that patches has been added in the meantime on the development branch.
Anyway, your PR LGTM.

Cherry-picking this onto staging-next and replacing stdenv with clangStdenv, I get the following failure:
/build/source/mapshape.c:1755:3: error: call to undeclared function 'strlcpy'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  strlcpy(shpfile->source, filename, sizeof(shpfile->source));
  ^
/build/source/mapshape.c:1841:3: error: call to undeclared function 'strlcat'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  strlcat(dbfFilename, ".dbf", bufferSize);
  ^
/build/source/mapshape.c:1978:3: error: call to undeclared function 'strlcpy'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  strlcpy(tiFileAbsDir, tiFileAbsDirTmp, MS_MAXPATHLEN);
  ^

@jtojnar , is it failing only on Darwin ? Bot Linux a Darwin builds of this PR are successful and the error message doesn't look related to this PR.

IIRC, the prerequisite is Clang 16. You can try replacing stdenv with llvmPackages_16.stdenv.

Do you mean keep using clang 15 by changing
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -27559,7 +27559,9 @@ with pkgs;

   mapproxy = callPackage ../servers/geospatial/mapproxy { };

-  mapserver = callPackage ../servers/geospatial/mapserver { };
+  mapserver = callPackage ../servers/geospatial/mapserver {
+    stdenv = if stdenv.isDarwin then overrideLibcxx llvmPackages_16.stdenv else stdenv;
+  };

Sorry, I am not Darwin user.

Actually, it might be that mapserver just has bad platform detection or something. I can reproduce the same error with clangStdenv on master. I retract my objection.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/week-in-geospatial-team/37035/2

Result of nixpkgs-review pr 279418 run on x86_64-darwin 1

1 package built:

mapserver

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/week-in-geospatial-team/37035/3


task/subtask name issues in some score files

Fix subtask naming issues in some score*json files. Additionally, this includes some fixes that seems to have happened to the upstream json files (e.g. duplicate subtask scores).

Related Q: I've noticed that some "subtask_description" fields are of the form [task name]:[subtask name], and some are of the form [subtask name]. I just want to confirm that both of these are OK, otherwise I can modify the score files so that they are consistent in this respect. (probably a Q for @ajslone ?)

Related Q: I've noticed that some "subtask_description" fields are of the form [task name]:[subtask name], and some are of the form [subtask name]. I just want to confirm that both of these are OK, otherwise I can modify the score files so that they are consistent in this respect. (probably a Q for @ajslone ?)
For json tasks, all subtask scores should have the form subtask_description=task:subtask, and the aggregate scores over subtasks should just have subtask_description=task. Ideally this would also be the case for programmatic tasks, but this is difficult to enforce/standardize.


Rename Code Inspector to Codiga

Code Inspector is rebranding as Codiga (and now supports more languages). We therefore changes the data accordingly.

Ah that's perfect. Congratulations on the new homepage and the rebranding. 🚀