Introduce a heuristic to amortize the per-window overhead in MaxScoreBulkScorer.

It is sometimes possible for MaxScoreBulkScorer to compute windows that don't contain many candidate matches, resulting in more time spent evaluating maximum scores per window than evaluating candidate matches on this window.
This PR introduces a heuristic that tries to require at least 32 candidate matches per clause per window to amortize the per-window overhead. This results in a speedup for the OrMany task.
                            TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
                       OrHighLow      830.99      (2.8%)      821.55      (2.0%)   -1.1% (  -5% -    3%) 0.236
                 CountAndHighMed      149.53      (3.2%)      148.06      (1.8%)   -1.0% (  -5% -    4%) 0.335
                CountAndHighHigh       49.23      (3.3%)       48.85      (2.1%)   -0.8% (  -6% -    4%) 0.483
                      OrHighRare      277.29      (5.9%)      275.20      (5.1%)   -0.8% ( -11% -   10%) 0.728
                         LowTerm     1006.28      (2.7%)      999.28      (2.7%)   -0.7% (  -5% -    4%) 0.512
                    OrHighNotMed      461.91      (2.0%)      459.09      (3.1%)   -0.6% (  -5% -    4%) 0.556
                      AndHighMed      205.48      (2.0%)      204.44      (2.2%)   -0.5% (  -4% -    3%) 0.547
            HighTermTitleBDVSort       20.30      (4.4%)       20.22      (4.0%)   -0.4% (  -8% -    8%) 0.798
                    OrHighNotLow      483.66      (2.2%)      481.97      (4.3%)   -0.3% (  -6% -    6%) 0.794
                   OrNotHighHigh      283.34      (2.3%)      282.47      (2.0%)   -0.3% (  -4% -    4%) 0.714
                    OrNotHighLow     1058.78      (3.5%)     1055.94      (2.6%)   -0.3% (  -6% -    6%) 0.826
                     AndHighHigh       78.53      (1.8%)       78.33      (1.9%)   -0.3% (  -3% -    3%) 0.721
                      OrHighHigh       77.35      (1.6%)       77.23      (1.6%)   -0.2% (  -3% -    3%) 0.812
                    OrNotHighMed      314.20      (2.9%)      313.96      (2.7%)   -0.1% (  -5% -    5%) 0.944
             And2Terms2StopWords      155.15      (2.9%)      155.07      (1.8%)   -0.0% (  -4% -    4%) 0.961
                   OrHighNotHigh      285.50      (2.5%)      285.63      (1.8%)    0.0% (  -4% -    4%) 0.958
                  CountOrHighMed      104.73      (1.6%)      104.95      (1.6%)    0.2% (  -2% -    3%) 0.744
                       And3Terms      167.95      (3.2%)      168.63      (2.6%)    0.4% (  -5% -    6%) 0.729
                          IntNRQ       90.83      (4.7%)       91.26     (14.9%)    0.5% ( -18% -   21%) 0.913
                       OrHighMed      200.80      (2.1%)      201.78      (1.7%)    0.5% (  -3% -    4%) 0.511
               HighTermTitleSort      149.37      (2.5%)      150.20      (2.0%)    0.6% (  -3% -    5%) 0.528
                 CountOrHighHigh       49.93      (1.4%)       50.24      (1.5%)    0.6% (  -2% -    3%) 0.270
                      AndHighLow     1079.98      (2.6%)     1086.73      (3.6%)    0.6% (  -5% -    7%) 0.613
              Or2Terms2StopWords      158.09      (4.1%)      159.09      (2.4%)    0.6% (  -5% -    7%) 0.630
                        HighTerm      515.68      (2.2%)      519.07      (2.6%)    0.7% (  -4% -    5%) 0.490
               HighTermMonthSort     3222.57      (3.4%)     3244.84      (2.9%)    0.7% (  -5% -    7%) 0.576
                         MedTerm      582.99      (2.5%)      587.15      (2.5%)    0.7% (  -4% -    5%) 0.468
                        Wildcard       82.76      (4.3%)       83.45      (3.8%)    0.8% (  -6% -    9%) 0.599
                    AndStopWords       30.49      (4.7%)       30.77      (2.4%)    0.9% (  -5% -    8%) 0.537
           HighTermDayOfYearSort      813.54      (3.4%)      821.97      (2.1%)    1.0% (  -4% -    6%) 0.355
                        PKLookup      272.42      (2.7%)      275.38      (2.5%)    1.1% (  -4% -    6%) 0.288
                        Or3Terms      166.90      (4.3%)      168.77      (2.7%)    1.1% (  -5% -    8%) 0.424
                     OrStopWords       33.64      (6.5%)       34.29      (3.2%)    1.9% (  -7% -   12%) 0.335
                      TermDTSort      344.04      (6.6%)      351.30      (5.3%)    2.1% (  -9% -   15%) 0.371
                         Prefix3      123.31      (3.5%)      126.03      (6.6%)    2.2% (  -7% -   12%) 0.286
                       CountTerm     8267.89      (4.4%)     8628.08      (4.7%)    4.4% (  -4% -   14%) 0.014
                          OrMany       13.25      (3.7%)       18.87      (3.7%)   42.4% (  33% -   51%) 0.000

There is a good speedup on OrMany as expected: https://benchmarks.mikemccandless.com/OrMany.html. I'll push an annotation.


Theus: prometheus logging across bittensor components

Prometheus logging:
Bittensor and miner classes can enable a local Prometheus server that logs statistics. These can be picked up by a global Prometheus server for use in network monitoring.
Features:
Global Prometheus switch
--prometheus.level <DEBUG, INFO, OFF>
Object level Prometheus switches:
--config.promehteus.level <DEBUG, INFO, OFF>
--axon.promehteus.level <DEBUG, INFO, OFF>
--dendrite.promehteus.level <DEBUG, INFO, OFF>
Turning off the global Prometheus switch ensures the main Prometheus server is not created.
Additional changes:
-- Removes defunct wandb statistics logging from Axon and Dendrite and from Validator and Miner classes.

Closing this for new PR that replace this one: #928


Window mutability, event loop, serious lifetime issues

Window:  It's a clusterf k.
So far, the methods in Window only take shared &self references. I considered the possibility of changing the setters to &mut self, but there are deep-rooted problems...
SDL can change just about everything about a Window from the event loop. If the user resizes or drags the window, the window will get mutated at the event loop and our exclusive &mut self method call will do us no good.
So at that point, I gathered that the "setters" such as set_title() or set_size() could stay as &self and could be seen more as "event sending". It's very race-condition-y (but not quite a race condition), but that's how SDL decided to do things. Fine. We can work with that.
But then there's Window::get_surface(), or SDL_GetWindowSurface(). Hoh boy.
SDL_GetWindowSurface() obtains a surface handle owned by the window. It's useful for when we want to draw directly to the window without using a renderer.
Here's a rather large caveat: every time the window is resized, the Surface is destroyed and reallocated.

Proof: https://github.com/spurious/SDL-mirror/blob/c2c793b74c2c9478814e4d49ea44577a6829c154/src/video/SDL_video.c#L2161 https://github.com/spurious/SDL-mirror/blob/c2c793b74c2c9478814e4d49ea44577a6829c154/src/video/SDL_video.c#L1962

loop {
   let surface = window.get_surface();
   run_sdl2_event_loop();
   surface.get_size();         // CRASH (sometimes)
   // ...
}

We really need some way to enforce the lifetime of a Surface obtained from a window.
Any time that the window could be resized (such as at the event loop or Window::set_size()), the lifetime must end.
We could add &mut self to window setters and add a lifetime parameter to Surface--which would partially solve the problem--but we would also need to ensure that no Surface<'window> values live through the event loop.
And don't forget about Renderer, which can consume a Window.
I'm currently unsure what will happen if you:

Create a software renderer and draw a shape
Resize the window with Window::set_size()
Draw another shape and present

It's probably not good.

The most ideal implementation is that we add &mut self to the window setters, and guarantee a lifetime when the event loop won't intervene with the Window. It would solve the Surface issue, and mutable setters are generally clearer for intent. The question is how to safely pull it off...
:confounded: :confounded: :confounded:

My head just exploded.

Could this maybe be solved w/ a Mutex?
My thought is the Window could own a Mutex<()> that all the setters #lock() on.
Then instead of providing access to the Surface directly you could let the user apply a closure to the Surface, that is run while the mutex is locked. -- That or perhaps return a RAII guard allowing access to the surface that unlocks the mutex when dropped?
This way the Window setters cannot be used while the Surface is being accessed?

I don't think mutexes will work, unfortunately.
The surface handle will get destroyed if SDL_PumpEvents() is called inside the closure you described, and if SDL_PumpEvents() resizes the window.
window.get_surface(|surface| {
    run_sdl2_pump_events();
    surface.get_size();         // CRASH (sometimes)
});

If we acquire the lock before calling SDL_PumpEvents() as well as before calling Window's getters/setters, we'll be acquiring the lock twice. A second lock on StaticMutex will deadlock. :(

The above issue is a corner case, though. If it happens, we could simply panic.
Even then, I am hesitent to using a Mutex, or any other locking mechanism; it introduces additional overhead that doesn't exist in SDL2.
But... if we decide to go that route, we'll need to know if it's unsafe to call the SDL_SetWindow...() or SDL_GetWindow...() methods from non-main threads (see https://github.com/AngryLawyer/rust-sdl2/issues/318#issuecomment-73410292).
With all of that said, I think a statically checkable solution is most ideal.

Here's a proof-of-concept solution that uses borrows and lifetimes. The syntax is a bit verbose, but it could work: https://gist.github.com/nukep/0d66d1278e84d1011738
Rust playpen's share button won't work right now, so just paste the code I guess.
The one major problem I have with the proof-of-concept solution is that window.mutable(&event_pump)` could be very confusing.
"Why does the window need an event pump?", or "This is too much code!".


add support for opentofu for binary plans

At the moment trying to use tf-summarize with a binary plan that was generated by opentofu does not work.
It will raise the error (version refs will depend on your terraform version, latest opentofu version is 1.6.1)
bsh ❯ tf-summarize .tfplan
error when running 'terraform show -json .tfplan':
╷
│ Error: Couldn't show local plan
│
│ Plan read error: plan file was created by Terraform 1.6.1, but this is
│ 1.6.6; plan files cannot be transferred between different Terraform
│ versions
╵

Make sure you are running in terraform directory and terraform init is done

Workaround
Under the covers, tf-summarize just calls terraform show -json when presented with a binary plan; so just turn it into json via tofu.
bsh ❯ tofu plan -out .tfplan
bsh ❯ tofu show -json .tfplan >.tfplan.json
bsh ❯ tf-summarize .tfplan.json
+--------+----------+
| CHANGE | RESOURCE |
+--------+----------+
+--------+----------+

At this moment if you need to have a support of Terraform as well as OpenTofu (and Terragrunt :) ) in one tool you can use https://github.com/tofuutils/tenv which my team wrote some months ago. A lot of users switched to that tool to unify version management in the world of Terraform.
You're welcome to open any issues or contribute to tenv.


--scala-version 3 returns 2.13 artefacts

Using Coursier 2.1.0-M5-18-gfebf9838c :
% cs fetch --scala-version 3 'org.scalacheck::scalacheck:1.16.0'
/home/smarter/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scalacheck/scalacheck_2.13/1.16.0/scalacheck_2.13-1.16.0.jar
/home/smarter/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.8/scala-library-2.13.8.jar
/home/smarter/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-sbt/test-interface/1.0/test-interface-1.0.jar

(Also IMO it's time to change the default to scala 3 from scala 2.13 now that cs launch scala starts Scala 3).

I think some more work is required for 3.x to work fine when passed via --scala-version. Relatedly, I think cs can't infer a Scala 3 version when passed Scala dependencies like org::name:ver.


Remove häte label, check the status of issues tagged with it

https://github.com/ozkriff/zemeroth/labels/häte

Closed #68, #128
Updated #178, #111, #221, #26
Replaced #181 with #251 as it contains too much outdated information


openrc typo

was in exercise.sh
NOTE: Using the 2.0 auth api does mean that compute api is 2.0.
Did you mean "Using the 2.0 auth api does NOT mean that compute api is 2.0."?

yep - thanks!