    Elapsed time       [s]      6.942                          7.271                          +4.7%
        CPU time       [s]     12.746                         12.985                          +1.9%
 CPU utilisation       [%]       23.0                           22.3                          -2.7%
           Calls      [op]    1000000                        1000000                          +0.0%
          Errors      [op]          0                              0                          +0.0%
              └─       [%]        0.0                            0.0                          +0.0%
        Requests     [req]    1000000                        1000000                          +0.0%
              └─  [req/op]        1.0                            1.0                          +0.0%
            Rows     [row]     500425                         499581                          -0.2%
              └─ [row/req]        0.5                            0.5                          -0.2%
         Samples                    7                              8                         +14.3%
Mean sample size      [op]     142857                         125000                         -12.5%
              └─     [req]     142857                         125000                         -12.5%
     Concurrency     [req]      297.8                          360.0                         +20.9%
              └─       [%]       77.5                           93.8                         +20.9%
      Throughput    [op/s]     144068 ± 2123                  137543 ± 3159                   -4.5%     0.00011  **
              ├─   [req/s]     144068 ± 2123                  137543 ± 3159                   -4.5%     0.00011  **
              └─   [row/s]      72095 ± 1229                   68714 ± 1363                   -4.7%     0.00004  ***
  Mean call time      [ms]      4.627 ± 0.103                  5.333 ± 0.120                 +15.2%     0.00000  *****
 Mean resp. time      [ms]      4.622 ± 0.103                  5.330 ± 0.120                 +15.3%     0.00000  *****

RESPONSE TIMES [ms] ════════════════════════════════════════════════════════════════════════════════════════════════════════
                            ───────────── A ─────────────  ────────────── B ────────────     Change     P-value  Signif.
          Min                   0.344 ± 0.108                  0.728 ± 0.286                +111.7%     0.00260  *
           25                   2.498 ± 0.090                  3.820 ± 0.079                 +52.9%     0.00000  *****
           50                   3.781 ± 0.107                  4.832 ± 0.117                 +27.8%     0.00000  *****
           75                   5.621 ± 0.189                  6.267 ± 0.175                 +11.5%     0.00000  ****
           90                   8.105 ± 0.369                  8.021 ± 0.258                  -1.0%     0.55184
           95                  10.405 ± 0.543                  9.359 ± 0.405                 -10.1%     0.00031  **
           98                  14.847 ± 1.472                 11.281 ± 0.747                 -24.0%     0.00006  ***
           99                  19.435 ± 3.130                 13.376 ± 2.164                 -31.2%     0.00028  **
           99.9                28.216 ± 6.287                 22.519 ± 6.067                 -20.2%     0.05167
           99.99               30.291 ± 3.934                 25.367 ± 6.235                 -16.3%     0.04922
          Max                  34.476 ± 5.021                 33.786 ± 8.295                  -2.0%     0.81925

So I ported current master of latte to scylladb driver, and I noticed a weird issue - minimum and average latencies are much higher than cassandra_cpp, and throughput is still a bit behind in this test (even though this test is short, this phenomenon exist also when running it for longer). I've run it a few times, also in reversed order and it is very repeatable. The difference is larger on single thread.
Are there any tuning options for the buffering thing you've added recently?
BTW: not sure why the number of returned rows is only half the number of queries, but that looks like a bug in the workload generation / definition; that's exactly same code on both sides, so shouldn't change the validity of this test.

Interestingly, when I decrease the amount of concurrent requests from 384 to only 64, scylla_db now outperforms cassandra_cpp on almost every metric (except min latency). I just wonder, maybe the internal driver queue / buffer is too small?

BTW: I'm not saying these latencies from scylla are worse - actually in REAL app I'd take lower tail latency over larger min latency every time. However, I'm just curious why I'm getting those numbers. Maybe there is an opportunity to improve something even more.

Looking perf stats, one thing I notice is that scylla branch spends a lot more time in the system, and makes 2x more context switches:
master:

 Performance counter stats for './target/release/latte run workloads/basic/read.rn -d 1000000':

         13 215,82 msec task-clock                #    1,362 CPUs utilized
            51 499      context-switches          #    3,897 K/sec
             1 432      cpu-migrations            #  108,355 /sec
             7 094      page-faults               #  536,781 /sec
    39 277 452 851      cycles                    #    2,972 GHz
    32 477 310 140      instructions              #    0,83  insn per cycle
     6 607 178 073      branches                  #  499,945 M/sec
        70 631 696      branch-misses             #    1,07% of all branches

       9,701747340 seconds time elapsed

      11,554137000 seconds user
       1,807549000 seconds sys

scylla-driver-rebase:

 Performance counter stats for './target/release/latte run workloads/basic/read.rn -d 1000000':

         13 170,58 msec task-clock                #    1,226 CPUs utilized
           101 559      context-switches          #    7,711 K/sec
             4 966      cpu-migrations            #  377,052 /sec
             1 592      page-faults               #  120,875 /sec
    38 972 668 976      cycles                    #    2,959 GHz
    36 413 811 976      instructions              #    0,93  insn per cycle
     6 182 693 111      branches                  #  469,432 M/sec
        53 918 503      branch-misses             #    0,87% of all branches

      10,740003985 seconds time elapsed

      10,124568000 seconds user
       3,358842000 seconds sys

Both runs 1000000 iterations of read.rn benchmark:
scylla-driver-rebase:
           127 260      syscalls:sys_enter_recvfrom
            55 854      syscalls:sys_enter_sendto
            75 392      syscalls:sys_enter_epoll_wait

master:
            62 662      syscalls:sys_enter_read
            10 492      syscalls:sys_enter_write
            16 730      syscalls:sys_enter_writev
            45 434      syscalls:sys_enter_epoll_wait

Althouth there is a huge improvement since 0.3.0, it looks like scylla driver still makes significantly more syscalls than cassandra_cpp. Looks like it splits data into smaller chunks and the system overhead is higher.
It would be good if I could configure scylla driver to make bigger batches when high throughput is more important than latency?
BTW: On a fixed rate test I'm getting a lot better latencies from scylla driver, in particular p75-p99.9.

Indeed, increasing buffer sizes will increase throughput at the cost of latency, up to a point. The whole idea is to find a sweet spot for your workload and hardware, and you're right that it's not currently possible in scylla-rust-driver, because the sizes are hardcoded. I was planning to push a patch that makes the sizes configurable - I'll put it higher on my priority list and push a pull request early next week.

Cool, let me quickly fork it then and check if changing that buffer size makes any difference first ;)

Sure. Btw, no need to fork the repo - once I send a pull request to a branch, you can specify a git branch as dependency in Cargo.toml: scylla = {git = "https://github.com/scylladb/scylla-rust-driver", branch="some-branch"}

I increased the buffers size to 32k and can't see any change in the numbers of recvfrom nor sendto calls. Huh.

What is the policy of flushing those buffers? Also what controls the frequency of reading from socket into the read buffer? Maybe it reads so frequently that there isn't enough data in the socket to read?

So I guess this is this is the main loop:
      while let Some(mut task) = task_receiver.recv().await {
            let mut num_requests = 0;
            let mut total_sent = 0;
            while let Some(stream_id) = Self::alloc_stream_id(handler_map, task.response_handler) {
                let mut req = task.serialized_request;
                req.set_stream(stream_id);
                let req_data: &[u8] = req.get_data();
                total_sent += req_data.len();
                num_requests += 1;
                write_half.write_all(req_data).await?;
                task = match task_receiver.try_recv() {
                    Ok(t) => t,
                    Err(_) => break,
                }
            }
            trace!("Sending {} requests; {} bytes", num_requests, total_sent);
            write_half.flush().await?;
        }

After enabling logging, I noticed this only initially sends a full buffer with requests, and then it quickly drops down to sending whatever it received.
The client app uses buffer_unordered to limit the number of pending request futures. So, once it hits that limit (384 futures by default), it stops sending more requests to the driver and waits for some of the pending ones to finish. Now after some time Cassandra responds and a query result arrives. Because all of this is so low-latency here on the Rust side, the driver picks the result near instantly and the number of pending request futures drops by just 1 or a few, but not by 384 (this is what I see in the logs) and the writer loops now gets to spin another cycle. But it only gets a few tasks to process, so it runs out of tasks long before the buffer fills up, and flushes whatever was buffered.
Hence, the buffer size does not matter that much - the driver is very "eager" - whatever comes it jumps to process it and doesn't let things accumulate.
BTW: This problem was realized by the creators of DataStax C++ driver and there is a configurable delay (default 200 µs), where the driver waits for more requests in order to fill up the buffer - so if the buffer doesn't fill up, it adds at worst those 200 µs to the latency, but saves many syscalls by potentially batching more queries at once.
If you wish I can try to implement such configurable delay if time permits this weekend (could be turned off by default so it wouldn't cause any regression for latency sensitive workloads).

Of course, all contributors are welcome. We'll probably want to preserve current behavior as default, but allowing the driver to explicitly accumulate the requests for X microseconds sounds like a very nice tuning option.

I've spent a bit of time today on this, and there is something weird going on, but I haven't figured that out yet. :(
Because there I couldn't find anything to do async sleep with submillisecond timing in tokio, my first attempt was to just spin on tokio::task::yield_now().await for ~200 µs or until I get the buffer filled up to at least 8k - well, that did reduce the number of write syscalls, but increased the number of epoll calls, and overall performance and cpu utilisation was beasically the same.
while let Some(mut task) = task_receiver.recv().await {
            let start_time = Instant::now();
            let mut num_requests = 0;
            let mut total_sent = 0;
            'outer: while let Some(stream_id) =
                Self::alloc_stream_id(handler_map, task.response_handler)
            {
                let mut req = task.serialized_request;
                req.set_stream(stream_id);
                let req_data: &[u8] = req.get_data();
                total_sent += req_data.len();
                num_requests += 1;
                write_half.write_all(req_data).await?;
                task = loop {
                    match task_receiver.try_recv() {
                        Ok(t) => break t,
                        Err(_) => {
                            if Instant::now() - start_time < tokio::time::Duration::from_micros(200) && total_sent < 8196
                            {
                                tokio::task::yield_now().await;
                            } else {
                                break 'outer;
                            }
                        }
                    }
                }
            }

Then I tried blocking with std::time::sleep, just for an experiment (I know it is totally not acceptable to do so - but I'm just breaking things to learn better how they work ;)), and this time it reduced the amount of syscalls significantly, it reduced a bit CPU usage, but the overall throughput stayed the same. Still couldn't match the throughput I am getting with cassandra_cpp.
So I conclude, maybe those syscalls are a red herring and the real issue is somewhere else (or there is no problem and I'm nitpicking and 100k max throughput vs 120k with 1 thread is not such a big difference, and with 4 threads it's only a tad slower).
However, I tried one more thing - what happens if I increase the number of concurrent queries to a crazy value, on the latte side. It is easy to do so with -p. And there I've found something big:
 ./target/release/latte run workloads/basic/read.rn -d 1000000 -p 4096

     Concurrency     [req]     4071.6
              └─       [%]       99.4
      Throughput    [op/s]      23979 ± 1508
              ├─   [req/s]      23979 ± 1508
              └─   [row/s]      23979 ± 1508

Wait, what?! 24k req/s? 5x lower?
Interestingly, during this test my local cassandra node sits half idle, top reports Cassandra using about 160% of a core (out of 4 physical cores), and total CPU is about 60-70% idle.
Let's try that with master branch using cassandra_cpp:
./target/release/latte run workloads/basic/read.rn -d 1000000 -p 4096

     Concurrency     [req]     1742.2
              └─       [%]       42.5
      Throughput    [op/s]     131797 ± 6877
              ├─   [req/s]     131797 ± 6877
              └─   [row/s]     131797 ± 6877

Now Cassandra is also a lot more loaded, gets about 450-500% of a core, total idle is about 15-20% (still weird why, but it was always like that, so that might be a problem in Cassandra).
So overall - it looks like there is a bottleneck somewhere when a crazy number of futures is outstanding. Can you try with -p 4096 (or even hogher values) and check if you also reproduce that?

Seems to be inversely linear. Going from 4096 to 8152 concurrent queries halves the throughput near exactly. I'm using single threaded tokio runtime here so it is likely not a classic thread contention.

I made a flamegraph and it is hugely confusing to me.
flamegraph.zip
First it doesn't show that scylla request takes a huge amount of time, at least not directly.
It shows lot of time being spent in.... Future.
Also when I change the workload to empty.rn (on the same branch, no recompilation needed), which does not call into the driver at all at all, yet it invokes the Rune scripting engine and basically runs the same code path on the latte side – I'm getting 1.5 M calls per second, regardless of -p setting. 1.5 million vs 12k. Also, with cassandra_cpp there is no such behaviour.
So it kinda works as if polling scylla future made everything slow, but not scylla driver code itself.
I wonder, could the future returned form scylla execute be deeply nested - e.g. aren't we running into the problem described here: https://internals.rust-lang.org/t/questioning-the-design-of-the-future-trait-and-the-cost-implied-by-it/14217 ?

Ok, so I think I got it. Indeedt the driver must be running into a recursive Future issue somewhere.
I wrapped the call to session.execute into its own task and throughput is now a steady 115k req/s at -p 384, and it drops to only 105k req/s at -p 8152.
Before (session.rs: 206):
        let rs = self.inner.execute(statement, params).await;

After:
        let session = self.inner.clone();
        let statement = statement.clone();
        let rs =
            tokio::task::spawn(async move { session.execute(statement.as_ref(), params).await })
                .await
                .unwrap();

Very impressive detective work! I just ran a simple experiment and can confirm that increasing the parallelism causes reduced throughput after a sweet spot is reached. I wonder if this problem (https://internals.rust-lang.org/t/questioning-the-design-of-the-future-trait-and-the-cost-implied-by-it/14217) is even solvable without explicitly spawning a task, we'll definitely need to educate ourselves more with Rust and Tokio internals to try and bypass this. /cc @piodul

Very impressive detective work! I just ran a simple experiment and can confirm that increasing the parallelism causes reduced throughput after a sweet spot is reached. I wonder if this problem (https://internals.rust-lang.org/t/questioning-the-design-of-the-future-trait-and-the-cost-implied-by-it/14217) is even solvable without explicitly spawning a task

I mean, it obviously is solvable, but it would be great to find a way that doesn't involve rewriting the whole core of the driver.

It's still quite interesting that adding a spawn in latte already leads to such reduction - it would mean that the depth of futures in the driver itself is not problematic on its own, because otherwise a spawn wouldn't help (tribute for this observation goes to @piodul), but once it's used from within an application that adds more depth - like latte, the problem gets amplified. I wonder if the cassandra-cpp backend wouldn't eventually experience the same issue, just much later, because the bindings make the depth of the resulting state machine much shorter.
In any case, I did some quick experiments and it's evident that for large parallelism values, simplifying the state machine causes vast improvements - e.g. I just prepared a branch which ignores retry policy and speculative retries, and it immediately increases throughput by around 20% for parallelism ~2048 (I picked 2048, because my local machine already experiences a huge drop in throughput compared to lower values).

Obviously I can't for sure rule out the O(n) complexity happens on latte side. However, I double checked the only place where multiple queries are processed is the main loop, which is pretty simple and standard - it uses buffer_unordered from crate futures to run multiple futures, and I don't do any manual future chaining by myself. The documentation of buffer_unordered says that they optimized that in such a way that it should poll only one future per wakeup, so if they are right, this should not cause the problem:
https://docs.rs/futures/0.3.18/futures/stream/futures_unordered/struct.FuturesUnordered.html
And the main loop is here:
https://github.com/pkolaczk/latte/blob/8680cfe90d982bc315c11d796b8d1c1c88f4c288/src/main.rs#L90-L125

simplifying the state machine causes vast improvements

The main question is though, are those machines fixed size, or do they somehow grow with the number of queries submitted to the driver? According to the Rust forums thread you may run into an issue if there are recursive .await calls or if there is and_then chaining of futures.

simplifying the state machine causes vast improvements

The main question is though, are those machines fixed size, or do they somehow grow with the number of queries submitted to the driver? According to the Rust forums thread you may run into an issue if there are recursive .await calls or if there is and_then chaining of futures.

I tried to inspect the code rather thoroughly and didn't find any obvious recursive calls - retry policies are candidates, but dropping this path did not mitigate the issue. Also, if recursive awaits in the driver would be the root cause, I would assume that with sufficiently high parallelism you'd still hit the issue even after spawning a new task for each execute - since the driver side machines would still eventually grow big enough to cause noticeable overhead.

Switched from buffer_unordered to just buffered which is simpler.
The actual parallelism level went down, because that introduced a head-of-line blocking problem. For -p 8152 I'm getting:
     Concurrency     [req]     3534.5
              └─       [%]       43.4
      Throughput    [op/s]      30584 ± 1284
              ├─   [req/s]      30584 ± 1284

Which is obviously far below expectations.
If I run it with buffer_unordered and -p 3500 I'm getting roughly the same throughput, so I conclude the same problem exists for buffered and buffer_unordered.

I haven't verified if my hastily crafted patch is correct, but I'll present it below anyway.
I dropped the idea of using buffer_unordered entirely (and as a result also buffered, since it's basically a worse flavor of buffer_unordered due to head-of-line blocking), and instead used another approach:

Run in a loop until the shared count of all operations is decremented to 0
Inside a loop, take a fixed amount of tasks to be queued (== parallelism), and await them by collecting them to FuturesUnordered

Unless I made some grievous design mistake, this approach works quite similarly to yours original code, with an exception that it makes the parallelism come in waves - a first batch must finish before the next one is scheduled.
Still, this patch vastly speeds up latte - it easily reaches >150kops on my machine for parallelism of 2048. One can still observe regression when the number goes up (4096 is still ok, but 8192 starts to get slower), but I guess that after some point Tokio is not going to behave well anyway with so many tasks being in the loop.
I don't have any hard proof except for the observation above, but it kind of looks like buffer_unordered causes the state machine size to explode.
diff --git a/src/count_down.rs b/src/count_down.rs
index 392325d..0ae836c 100644
--- a/src/count_down.rs
+++ b/src/count_down.rs
@@ -41,6 +41,10 @@ impl CountDown {
             }
         }
     }
+
+    pub fn load(&self) -> u64 {
+        self.value.load(Ordering::Relaxed)
+    }
 }

 pub struct BatchedCountDown {
@@ -72,4 +76,8 @@ impl BatchedCountDown {
             true
         }
     }
+
+    pub fn shared_count(&self) -> u64 {
+        self.shared.load()
+    }
 }
diff --git a/src/main.rs b/src/main.rs
index 5b6f9bb..26aefcd 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -7,6 +7,7 @@ use std::time::{Duration, Instant};

 use clap::Clap;
 use futures::future::ready;
+use futures::stream::futures_unordered::FuturesUnordered;
 use futures::{Future, SinkExt, Stream, StreamExt};
 use scylla::Session;
 use tokio::runtime::Builder;
@@ -92,34 +93,40 @@ where
         let action = &action;
         let mut remaining_count = BatchedCountDown::new(count, 64);
         let pending_count = AtomicUsize::new(0);
-        let mut req_stats = IntervalStream::new(interval(rate))
-            .take_while(|_| ready(remaining_count.dec()))
-            .enumerate()
-            .map(|(i, _)| {
-                let context = context.clone();
-                let pending_count = &pending_count;
-                async move {
-                    let queue_len = pending_count.fetch_add(1, Ordering::Relaxed);
-                    let start = Instant::now();
-                    let result = action(context, i as u64).await;
-                    let end = Instant::now();
-                    pending_count.fetch_sub(1, Ordering::Relaxed);
-                    RequestStats::from_result(result, end - start, queue_len)
-                }
-            })
-            .buffer_unordered(parallelism);

         let mut sample = Sample::new(Instant::now());
-        while let Some(req) = req_stats.next().await {
-            sample.record(req);
-            let now = Instant::now();
-            if now - sample.start_time > sampling_period {
-                let start_time = sample.start_time;
-                let elapsed_rounded = round(now - start_time, sampling_period);
-                let end_time = start_time + elapsed_rounded;
-                sample.finish(end_time);
-                tx.send(sample).await.unwrap();
-                sample = Sample::new(end_time);
+
+        while remaining_count.shared_count() > 0 {
+            let mut req_stats = IntervalStream::new(interval(rate))
+                .take_while(|_| ready(remaining_count.dec()))
+                .enumerate()
+                .map(|(i, _)| {
+                    let context = context.clone();
+                    let pending_count = &pending_count;
+                    async move {
+                        let queue_len = pending_count.fetch_add(1, Ordering::Relaxed);
+                        let start = Instant::now();
+                        let result = action(context, i as u64).await;
+                        let end = Instant::now();
+                        pending_count.fetch_sub(1, Ordering::Relaxed);
+                        RequestStats::from_result(result, end - start, queue_len)
+                    }
+                })
+                .take(parallelism)
+                .collect::<FuturesUnordered<_>>()
+                .await;
+
+            while let Some(req) = req_stats.next().await {
+                sample.record(req);
+                let now = Instant::now();
+                if now - sample.start_time > sampling_period {
+                    let start_time = sample.start_time;
+                    let elapsed_rounded = round(now - start_time, sampling_period);
+                    let end_time = start_time + elapsed_rounded;
+                    sample.finish(end_time);
+                    tx.send(sample).await.unwrap();
+                    sample = Sample::new(end_time);
+                }
             }
         }
         if !sample.is_empty() {

I'll test it tomorrow, but beware that your patch lowers the average concurrency level, because the number of pending futures will decrease down to zero in each wave. So obviously it would improve throughput.
However, if you're seeing a huge improvement from that, it is even more weird, because buffer_unordered itself is very simple and it mostly delegates all the real work to FuturesUnordered which you've also used here. The only difference is that buffer_unordered keeps adding new futures as soon as possible so it keeps the number of pending futures mostly constant. In your solution that adding is deferred.

I'll test it tomorrow, but beware that your patch lowers the average concurrency level, because the number of pending futures will decrease down to zero in each wave. So obviously it would improve throughput even if the original problem remains.

It does lower average concurrency, but it comes in bursts, so if the problem is that high concurrency doesn't mix well with large state machines produced by compiling the driver, I think it would get amplified after the patch. The reasoning is that each time a burst of concurrency comes, it would suffer from reduced throughput, and the breaks between bursts only lower the throughput further, because nothing gets executed. But that's still just a guess. In any case, I keep pushing this theory because it's so confusing that adding a tokio::spawn in latte fixes the issue... which would suggest that even though the complexity of futures in the driver might be largish, it's ultimately something in latte that induces the regression, and buffer_unordered sounds like the only sensible culprit. And by the way - the main loop in scylla-rust-driver doesn't really contain any recursive calls, it pretty much uses a pair of channels to queue requests, and a pair of regular loops to send and receive responses.

The documentation of buffer_unordered says that they optimized that in such a way that it should poll only one future per wakeup, so if they are right, this should not cause the problem:

I did a test and it looks like FuturesUnordered doesn't scale well as number of futures increases. It looks as if it has to go through all of the futures on each poll().
The test puts n futures in a single FuturesUnordered and then waits for it to finish. All futures finish one by one by locking a mutex.
n increases 2x, but time increases 4x, looks quadratic:
n: 10000, time: 23ms
n: 20000, time: 87ms
n: 40000, time: 331ms
n: 80000, time: 1230ms
n: 160000, time: 5100ms

Here's the code
main.rs:
use futures::stream::futures_unordered::FuturesUnordered;
use futures::stream::StreamExt;
use tokio::sync::Mutex;
use std::sync::Arc;

async fn work(mutex: Arc<Mutex<()>>) {
    mutex.lock().await;
}

async fn do_test(n: usize) {
    let start: std::time::Instant = std::time::Instant::now();

    let mutex: Arc<Mutex<()>> = Arc::new(Mutex::new(()));
    let mutex_guard = mutex.lock().await;

    let mut futs = Vec::new();
    for _ in 0..n {
        futs.push(work(mutex.clone()));
    }
    let mut unorder: FuturesUnordered<_> = futs.into_iter().collect();

    std::mem::drop(mutex_guard);

    for _ in 0..n {
        unorder.select_next_some().await;
    }

    let end: std::time::Instant = std::time::Instant::now();
    println!("n: {}, time: {}ms", n, end.duration_since(start).as_millis());
}

#[tokio::main]
async fn main() {
    for n in [10_000, 20_000, 40_000, 80_000, 160_000] {
        do_test(n).await;
    }
}

Cargo.toml:
[dependencies]
tokio = { version = "1.12", features = ["full"] }
futures = "0.3.6"

Doing it using tokio::spawn gives linear time:
n: 10000, time: 5ms
n: 20000, time: 11ms
n: 40000, time: 23ms
n: 80000, time: 47ms
n: 160000, time: 86ms

use tokio::sync::Mutex;
use std::sync::Arc;

async fn work(mutex: Arc<Mutex<()>>) {
    mutex.lock().await;
}

async fn do_test(n: usize) {
    let start: std::time::Instant = std::time::Instant::now();

    let mutex: Arc<Mutex<()>> = Arc::new(Mutex::new(()));
    let mutex_guard = mutex.lock().await;

    let mut futs = Vec::new();
    for _ in 0..n {
        futs.push(tokio::spawn(work(mutex.clone())));
    }

    std::mem::drop(mutex_guard);

    for f in futs {
        f.await.unwrap();
    }

    let end: std::time::Instant = std::time::Instant::now();
    println!("n: {}, time: {}ms", n, end.duration_since(start).as_millis());
}

#[tokio::main]
async fn main() {
    for n in [10_000, 20_000, 40_000, 80_000, 160_000] {
        do_test(n).await;
    }
}

Sad. We need to double-check our internal uses of FuturesUnordered, but it seems that we only use it in the connection pool and speculative execution. For speculative execution the number will probably be single digit, so quadratic behavior doesn't matter, and for the connection pool there already exists a fixme that the usage of FuturesUnordered should probably be replaced with a channel. I still don't expect it to be a problem, because we won't likely be establishing tens of thousands of connections, but it's better to be safe than sorry.

I wonder what's going on here, the implementation of FuturesUnordered is smart and tries hard to be linear. This could be a bug, I might open an issue.

I wonder what's going on here, the implementation of FuturesUnordered is smart and tries hard to be linear. This could be a bug, I might open an issue.

Does it though? It has mechanisms for not polling its futures more than once, but I don't see any mechanisms that prevent polling O(n) futures when FuturesOrdered are awakened, until it finds the one that's actually ready

It looks like it has two queues:

for futures that need to be polled
fur futures that don't need to be polled

When polling a managed future it gives it a custom waker, which moves the future to the first queue and then calls the waker of FuturesUnordered
This way it calls poll only on futures that actuall called their waker, all other pending futures are left alone on the other queue.
That should ensure that performance is fine o_0

It looks like it has two queues:

for futures that need to be polled
fur futures that don't need to be polled

When polling a managed future it gives it a custom waker, which moves the future to the first queue and then calls the waker of FuturesUnordered
This way it calls poll only on futures that actuall called their waker, all other pending futures are left alone on the other queue.
That should ensure that performance is fine o_0

Alright, I was looking at the source code for some ancient 0.1.x version... I keep doing that when browsing Rust docs.

@cvybhu in any case, please post your reproducer and results on future's bug tracker: https://github.com/rust-lang/futures-rs/issues and link to this issue, perhaps it's a known thing

Oh, we had a race here. Never mind :)

@pkolaczk @cvybhu https://github.com/rust-lang/futures-rs/issues/2526#issuecomment-982587160, the patch below indeed not only makes the problem disappear completely, but also significantly speeds up the execution even for lower concurrency values for me!
diff --git a/src/main.rs b/src/main.rs
index 5b6f9bb..cd80643 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -101,7 +101,7 @@ where
                 async move {
                     let queue_len = pending_count.fetch_add(1, Ordering::Relaxed);
                     let start = Instant::now();
-                    let result = action(context, i as u64).await;
+                    let result = tokio::task::unconstrained(action(context, i as u64)).await;
                     let end = Instant::now();
                     pending_count.fetch_sub(1, Ordering::Relaxed);
                     RequestStats::from_result(result, end - start, queue_len)

Thanks for investigation. Wow, I'd never think about a trap like that!
@psarna  I think you're looking at an older latte version, but anyway, I'll port the patch to the new one.

Wait, adding unconstrained will disable blocking protection and we will block the runtime, no?
We could instead add tokio::task::yield_now().await after reading next result from FuturesUnordered.

It will only block if the driver never yields, i.e. sockets are always ready and so on. For a benchmarking tool, turning off yielding might make sense, depending how it influences latency

I'm reading that differently - it doesn't yield only if a future is ready, but it still yields if future is not ready.
https://docs.rs/tokio/1.5.0/tokio/task/index.html#unconstrained
However, I fail to see how it is connected to O(n^2) complexity for buffer_unordered.

On Wed, Dec 1, 2021 at 9:25 AM Piotr Kołaczkowski @.***>
wrote:

I added unconstrained and gave it a test. The results are different but
still unsatisfactory.
The quadratic complexity issue is indeed gone. Upping the parallelism
level does not lower the throughput.
The max throughput went up by about 10-15%.
However, my max throughput is still lower than the one I can get from
cassandra_cpp (another 15% needed).
What I noticed is that now regardless of the -p setting, the actual
average number of pending futures is about 150-170, and hits some kind of a
ceiling.
-p 256: actual concurrency 116.3, throughput 114k
-p 512: actual concurrency 131.9, throughput 114k
-p 1000: actual concurrency 166.3, throughput 112k
cassandra_cpp is doing about 125-130k on the same server, with 1 thread,
and the actual parallelism level is much higher - easily goes above 350.
Note that these are extremely tiny statements on a link with extremely low
latency, so parallelism must be high in order to fully load C* on the other
side.
If I increase the number of threads to 2, then I can almost match
cassandra_cpp on 1 thread.
I've got 2 hypotheses what's causing this:

now that I've added unconstrained, the producer gets starved and
simply can't deliver enough concurrent requests for execution; all power
goes into the driver to get them processed as fast as possible
2.
because cassandra_cpp has a separate libuv executor, it is not truly
single threaded; so it is not a true apple-to-apple comparison, as the
producer runs on a different thread than the I/O thread sending the
requests. So it might be simply the it is too much work to do for a single
thread, and scylla is using most time processing the request and not much
of the CPU core is left for producing the requests. However, CPU time seems
to disprove that - CPU time is not higher for cassandra_cpp at 130k req/s.
Interesting. You can pin the process to a single core and see if the
result changes

BTW: if I remove the actual session.execute call and leave all the other
things in place (running the script, including parameter conversion), it
reaches over 610k req/s. So the producer is definitely fast enough for this
test. Obviously I can work on that part more but first I'd like to rule out
problem 1.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/scylladb/scylla-rust-driver/issues/362#issuecomment-983363109,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AANHURKQCE6GHSSEPSZXXFDUOXEYHANCNFSM5I25XU3Q
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

That's a great idea!
 taskset -c 0 ./target/release/latte run workloads/basic/read.rn -d 1000000 -p 128
cassandra_cpp:
    Elapsed time       [s]      9.886
        CPU time       [s]      9.592
 CPU utilisation       [%]       97.0       // looks correct, got 1 core only
     Concurrency     [req]       95.2
              └─       [%]       74.4
      Throughput    [op/s]     101158 ± 1350
              ├─   [req/s]     101158 ± 1350
              └─   [row/s]     101158 ± 1350
  Mean call time      [ms]      1.075 ± 0.018
 Mean resp. time      [ms]      1.071 ± 0.018

scylla:
    Elapsed time       [s]     19.756
        CPU time       [s]     17.771
 CPU utilisation       [%]       90.0

  Concurrency     [req]       79.2
              └─       [%]       61.9
      Throughput    [op/s]      50620 ± 414
              ├─   [req/s]      50620 ± 414
              └─   [row/s]      50620 ± 414
  Mean call time      [ms]      1.520 ± 0.022
 Mean resp. time      [ms]      1.516 ± 0.022

And I verified CPU frequency was properly locked at 3 GHz during both runs.

It looks like it sends one request at a time:
Sending 1 requests; 46 bytes
Sending 1 requests; 46 bytes
Sending 1 requests; 46 bytes
Sending 1 requests; 46 bytes
Sending 1 requests; 46 bytes
Sending 1 requests; 46 bytes
Sending 1 requests; 46 bytes
Sending 1 requests; 46 bytes
Sending 1 requests; 46 bytes
Sending 1 requests; 46 bytes
Sending 1 requests; 46 bytes
Sending 1 requests; 46 bytes
Sending 1 requests; 46 bytes
Sending 1 requests; 46 bytes
Sending 1 requests; 46 bytes

I added yield_now() before the task_receiver.try_recv() but it didn't help. :(

Oh, man, another trap here:
This function may not yield all the way up to the executor if there are any special combinators above it in the call stack. For example, if a tokio::select! has another branch complete during the same poll as the yield_now(), then the yield is not propagated all the way up to the runtime.
It is generally not guaranteed that the runtime behaves like you expect it to when deciding which task to schedule next after a call to yield_now()

So I guess the way to go is to wait for futures_rs to backport their fix to 0.3 and remove unconstrained.

I managed to test that by temporarily dropping IntervalStream, but that did not help  - still sending mostly 1 request at a time.

Well, one more thing I tried - wrapping the workload.run in its own task, spawned by tokio::spawn_local (I have to spawn local, because it is !Send). Surprisingly also didn't help, although it changed the concurrency level reported by latte - now it runs the number of futures close to the limit, but the driver still sends out queries one per buffer.

I tried to debug how buffer_unordered actually polls scylla futures and got a weird observation:

when I just launch a simple buffer_unordered in main, it all works as desired and buffering works (polls multiple futures, they return PENDING and afterwards they send all requests in one go).
when I added the same debugging to the futures inside latte main loop, it sends one request at a time....

Something fishy must be going on on the latte side then, stay tuned.

Looks like there is a huge difference just stemming from launching on another thread:
    let rt = Builder::new_current_thread()
        .enable_all()
        .build()
        .unwrap();
    std::thread::spawn(move || {
        rt.block_on(async move {
            println!("Stream started");
            let mut s = futures::stream::iter(0..10)
                .map(|i| DebugFuture::new(format!("q {}", i), loader.run(i)) )
                .buffer_unordered(5);

            while let Some(result) = s.next().await {
                println!("Got query result: {:?}", result);
            }
            println!("Stream ended");
        });
    });
    tokio::time::sleep(Duration::from_secs(100)).await;

Output:
Stream started
Polling future q 0
Future q 0 returned PENDING
Polling future q 1
Sending 1 requests; 46 bytes     /// <<<----- oh, why are you already sending?
Future q 1 returned PENDING
Polling future q 2
Sending 1 requests; 46 bytes
Future q 2 returned PENDING
Polling future q 3
Sending 1 requests; 46 bytes
Future q 3 returned PENDING
Polling future q 4
Sending 1 requests; 46 bytes
Future q 4 returned PENDING
Sending 1 requests; 46 bytes
Polling future q 1
Future q 1 returned READY
Got query result: Ok(())
Polling future q 0
Future q 0 returned READY
Got query result: Ok(())
Polling future q 2
Future q 2 returned READY
Got query result: Ok(())
Polling future q 5
Future q 5 returned PENDING
Polling future q 6
Sending 1 requests; 46 bytes
Future q 6 returned PENDING
Polling future q 3
Sending 1 requests; 46 bytes
Future q 3 returned READY
Got query result: Ok(())
Polling future q 4
Future q 4 returned READY
Got query result: Ok(())
Polling future q 7
Future q 7 returned PENDING
Polling future q 8
Sending 1 requests; 46 bytes
Future q 8 returned PENDING
Polling future q 9
Sending 1 requests; 46 bytes
Future q 9 returned PENDING
Polling future q 5
Sending 1 requests; 46 bytes
Future q 5 returned READY
Got query result: Ok(())
Polling future q 6
Future q 6 returned READY
Got query result: Ok(())
Polling future q 7
Future q 7 returned READY
Got query result: Ok(())
Polling future q 8
Future q 8 returned READY
Got query result: Ok(())
Polling future q 9
Future q 9 returned READY
Got query result: Ok(())
Stream ended

But the same code executed on the main (also singlethreaded) runtime, without spawining additional thread:
 println!("Stream started");
let mut s = futures::stream::iter(0..10)
    .map(|i| DebugFuture::new(format!("q {}", i), loader.run(i)) )
    .buffer_unordered(5);

while let Some(result) = s.next().await {
    println!("Got query result: {:?}", result);
}
println!("Stream ended");

Stream started
Polling future q 0
Future q 0 returned PENDING
Polling future q 1
Future q 1 returned PENDING
Polling future q 2
Future q 2 returned PENDING
Polling future q 3
Future q 3 returned PENDING
Polling future q 4
Future q 4 returned PENDING
Sending 5 requests; 230 bytes    // <<<<<<<<<<<<<<<<<<< SEE THIS
Polling future q 4
Future q 4 returned READY
Got query result: Ok(())
Polling future q 3
Future q 3 returned READY
Got query result: Ok(())
Polling future q 0
Future q 0 returned READY
Got query result: Ok(())
Polling future q 2
Future q 2 returned READY
Got query result: Ok(())
Polling future q 1
Future q 1 returned READY
Got query result: Ok(())
Polling future q 5
Future q 5 returned PENDING
Polling future q 6
Future q 6 returned PENDING
Polling future q 7
Future q 7 returned PENDING
Polling future q 8
Future q 8 returned PENDING
Polling future q 9
Future q 9 returned PENDING
Sending 5 requests; 230 bytes   // <<<<<<<<<<<<<<<<<<< SEE THIS
Polling future q 7
Future q 7 returned READY
Got query result: Ok(())
Polling future q 5
Future q 5 returned READY
Got query result: Ok(())
Polling future q 6
Future q 6 returned READY
Got query result: Ok(())
Polling future q 8
Future q 8 returned READY
Got query result: Ok(())
Polling future q 9
Future q 9 returned READY
Got query result: Ok(())
Stream ended

Note sure if this is the right issue to comment on, but I was quite surprised to see that cdrs_tokio is quicker than this crate in most scenario's according to the benchmarks provided in the readme: https://github.com/krojew/cdrs-tokio. Maybe interesting to see how the maintainer managed to make it perform so quick.

@Jasperav the benchmarks presented in the readme do not mention any of the following:

cluster setup: number of nodes, network, type of machines
was it run against Cassandra or Scylla
client setup: on what machine was the benchmark itself run - was it remote or local to the cluster?
drivers setup: which non-default configuration options were used for respective drivers, if any?
in general: the source code of the benchmarks, to make it reviewable and reproducible (or maybe it is somewhere, I just didn't find any mention of it)

so it's hard to deduce anything from the output. For any reproducible benchmarks we'd be happy to rerun and verify them, and do our best to match and/or outperform other drivers, if they prove to be faster.

After I switched to a single tokio runtime I'm getting way better results now and buffering on Scylla 0.3.1 works properly in that scenario, even with tokio::task::unconstrained.
Running on a single executor has also this nice property that now when I specify -t 1 (one thread), then it really doesn't use more than 1 CPU core. As I've said before, cassandra_cpp is running its own libuv executor, so it may achieve better results by simply using more CPU.
I have also found that adding lto = true; panic = "abort" to the release profile improves speed significantly.
To summarize:

futures::buffer_unordered has a scalability issue when running many futures with bigger state machines; there is an easy workaround by wrapping futures in unconstrained, and there is a fix ready in futures-rs project.
scylla driver seems to not like when the session is established on one executor but queries are spawned on a different executor - something weird happens and buffering doesn't work properly.

@Jasperav there are a lot of different things that can affect benchmark results.

FYI, FuturesUnordered is likely to be able to detect if a future wants to cooperatively yield soon: https://github.com/rust-lang/futures-rs/pull/2551 . The nice trick is checking if it's the task that wants to wake itself, vs when something else wants to wake it.

@pkolaczk if the problem persists in the newest version of the driver, please we'll reopen this.
For now, closing as stale.


[mysql-k8s] Create public documentation for data backups and restoration

Context
We need to create public documentation for users that want to backup their data and restore from it.
What needs to get done
Create a section in this discourse post with information about data backup and restoration for DB (mysql-k8s) charms used in CKF (kfp-db and katib-db).
Definition of Done
A section in the discourse post that exposes information about backup and restoration for DBs used in CKF.

Completed in https://charmed-kubeflow.io/docs/backup and https://charmed-kubeflow.io/docs/restore


fix confusing function def

This should be a easy merge. Just a small fix to remove the confusion.
renamed function run() to main() as it collides with Pipelines run() method.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

All the other examples use run(), did you want to change them all?

R: @udim

Yes that would be nice. I proposed changes which occurred to me. Thx.

Some tests are broken: https://ci-beam.apache.org/job/beam_PreCommit_Python_Commit/19231/

Can someone look into why this precommit CI is failing ? Also not sure what codecov check here is.

Can someone look into why this precommit CI is failing ?

This was a flaky test. It should fixed now.

Also not sure what codecov check here is.

It is checking that at least majority of the new code is covered by unit tests.

Run Python PreCommit"

Run Python PreCommit

Run Python PreCommit


Please simplify APCSP Students section of README

You have APCSP Students - 1st/2nd Quarter, and APCSP Students - 4th Quarter in the README file.  This is confusing for the 9th graders reading this section, and not needed.  We don't use node anymore.  Could you please combine the two section into one APCSP Students section that only has Python installation info.  Thanks!

Updated!


WIP: Add Django 3 and Python 3.8 support

Description
This pull request adds Django 3.0 and Python 3.8 support.
Checklist

[x] I have opened this pull request against develop
[x] I have updated the CHANGELOG.rst
[x] I have added or modified the tests when changing logic

@yakky would you be able to have a look at this


Tracing for Subgraphs in DGS

I am trying to get the tracing information for my subgraph in Apollo Gateway with subgraphs implemented in the DGS framework.
I referred this example to get the tracing information in the response. I am able to get the data in the required format.
The problem is Apollo expects tracing information in this - (How tracing data is exposed from a subgraph) format for the subgraphs. Can you please help me to understand how can I get the response to have this ftv1 field having the tracing information for the requested subgraph.

@rhino88 - Thanks for your response to this post. I was not aware of this implementation. I will update our documentation to reflect this as well.


Update GSF logo across docs

Story
As a GSF contributor I want the branding to be consistent across all our assets.
Rationale
The GSF logo has been updated since we launched the docs website, so we need to replace the original anywhere it is used.
Implementation details
Expect this is a case of updating the image file in public/imgs - all renders should then automatically update across site.
Priority
4/5
Scope
Docs only, no text changes, just swapping out file.
Size
1 hour
What does "done" look like?
File updated, website checked for correct rendering
Does this require updates to documentation or other materials??
no
What testing is required?
none
Is this a known/expected update?
expected

https://github.com/Green-Software-Foundation/if-docs/pull/14


add 'notnull_equals' as compat option for merging non-conflicting data

This solves #742 and partially #835 (does not address a combine_first option yet). It essentially adds a notnull_equals method to Variable (and currently DataArray/set) that merge can use to compare objects, subsequently combining them with 'fillna'. Used as such:
>>> import xarray as xr
>>> ds1 = xr.Dataset(data_vars={'x': ('a', [1, 2])})
>>> ds2 = xr.Dataset(data_vars={'x': ('a', [2, 3])}, coords={'a': [1, 2]})
>>> xr.merge([ds1, ds2], compat='notnull_equals')
<xarray.Dataset>
Dimensions:  (a: 3)
Coordinates:
  * a        (a) int64 0 1 2
Data variables:
    x        (a) float64 1.0 2.0 3.0

Should be very easy to add other merging options such as overwriting left to write and vice versa.
TODO

docs
tests for various type combinations
some of the error excepting is mirrored from equals and might not be necessary (e.g. for numpy structured arrays).

ISSUES

It seemed natural to add notnull_equals as a method to Dataset, but it might be unnecessary/not useful. Current version is conservative in that it still requires aligned Datasets with all the same variables.
Due to the float nature of NaN, type is sometimes not preserved, e.g. merging two int arrays yields a float array, even when the final array has no NaN values itself.
float rounding errors can cause unintended failures, so a notnull_allclose option might be nice.

Currently all tests passing (6 new), and no flake8 errors on the diff...

This is quite clever! I'm impressed this was so easy using fillna.
One potential concern here is that performance is not going to be so great if you attempt to combine a bunch of variables with lazy data loaded with dask, because each comparison will trigger a separate computation. To that end, it would be nice to do the safety check in a single dask operation.
To be fair, this is already an issue if you're trying to merge lots of datasets together. But I expect this will become more of an issue when there are useful ways to merge a bunch of datasets, which is what this PR enables.

Current version is conservative in that it still requires aligned Datasets with all the same variables.

:+1: it's much less obvious to treat missing variables as "missing data."

Due to the float nature of NaN, type is sometimes not preserved, e.g. merging two int arrays yields a float array, even when the final array has no NaN values itself.

Yes, this is consistent with the xarray/pandas type system. (Which is not ideal, but that's another bigger issue.)

compat='no_conflicts' is possibly a better keyword argument.

One of the commits in this PR is not yours. Take a look here -- you want to rebase instead of merge (yes git is confusing):
http://stackoverflow.com/questions/5968964/avoid-unwanted-merge-commits-and-other-commits-when-doing-pull-request-on-github
I think you can still safely rebase now.

Ah sorry - yes rebased locally then mistakenly merged the remote fork...

compat='no_conflicts' is possibly a better keyword argument.

Yes I thought that might be better also, the advantages of 'notnull_equals', (slightly more precise operational description and slightly better 'grammar' when using as a method: if ds1.notnull_equals(ds2): ... vs if ds1.no_conflicts(ds2): ...) are probably negligible since 'no_conflicts' is more intuitive for merge - and this is likely the main usage.

One potential concern here is that performance is not going to be so great if you attempt to combine a bunch of variables with lazy data loaded with dask, because each comparison will trigger a separate computation. To that end, it would be nice to do the safety check in a single dask operation.

I will have a look into how to do this, but am not that familiar with dask.

I think this is nearly ready. We just need to decide on the final keyword name (e.g., notnull_equals or no_conflicts) and add mention of this in the documentation / what's new.
I'm not sure we really need this new method for Dataset/DataArray (for the current implementation, adding it on Variable would be enough). Given that the user can catch MergeError, it seems straightforward enough just call merge to see if merging is safe.

OK, I have stripped the Dataset/Array methods which I agree were largely redundant. Since this sets this type of comparison/merge slightly apart, 'no_conflicts' seems the more intuitive wording when used only as a compat option so I've changed it to that.
And I've done a first pass at updating the docs.


Adding Ziklight language to fallbackConfig

Attempting to import the creature Clockwork Horror from the new source Monstrous Compendium Vol 1: Spelljammer Creatures resulted in an error due to one of its languages being missing. This PR simply adds in the language (I have tested locally that the error goes away and the creature is successfully imported after this change).
See also https://www.dndbeyond.com/api/config/json to check that it matches there.

Thankyou!


Update tool plugins to new structure. Pin flake8<5 and pycodestyle<2.9.0.

I'm not sure what to do here, the checks depend on a new version of statick-action, but the new version of statick-action depends on a new version of this package.

Codecov Report
Base: 100.00% // Head: 100.00% // No change to project coverage :thumbsup:

Coverage data is based on head (beb2b8c) compared to base (7b9c4ec).
Patch coverage: 100.00% of modified lines in pull request are covered.

:exclamation: Current head beb2b8c differs from pull request most recent head 1679a4b. Consider uploading reports for the commit 1679a4b to get more accurate results

Additional details and impacted files
@@            Coverage Diff            @@
##              main       #69   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            3         3
  Lines          137       131    -6
=========================================
- Hits           137       131    -6

Impacted Files
Coverage Δ

src/statick_tex/plugins/tool/chktex_tool_plugin.py
100.00% <100.00%> (ø)

...rc/statick_tex/plugins/tool/lacheck_tool_plugin.py
100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

This made it into #72.


docs_refactor_proposal

This is the proposal for the new documentation refactor.
There are some things to change, I went to the first chapter more in deep adding the dynamic links, and other rst things. Did not spend a lot of time in the second chapter, no need in case we do not want to go this way.  Anyway you can comment, and see if it is a good idea or not.
Please review and comment @FoamyGuy @kmatch98

Thanks, This is in accordance with the design guide, I will translate this PR to the CircuitPython Graphics sections once libraries are moved. Thanks @kmatch for reviewing :)


[ADD] RETAS STUDIO

I add the app name.

See the test error
[retas-studio] Cask audit must be error free.
["at least one activatable artifact stanza is required"]

Could you please squash your commits?

Closed in favour of #10596.


panic runtime error index out of range [1] with length 1

Background information:
Since there are no releases, I'm am basing this issue on having tried to follow the build instructions. The result of the docker method and the result of the go method produced different sized binaries, but both seem to have the same issue.
Issue
When running pgremapper with different commands (pgremapper cancel-backfill --verbose, pgremapper export-mappings --verbose 12, pgremapper drain 12 --verbose), the binary fails with very similar output:
** executing: ceph osd dump -f json
** executing: ceph pg dump pgs_brief -f json
** executing: ceph osd dump -f json
panic: runtime error: index out of range [1] with length 1

goroutine 1 [running]:
main.computeBackfillSrcsTgts(0xc00277b10) (export mapings) - or (0xc0025dc58) (cancel-backfill), (0xc00277c28) (drain)
< trace into different modules >

I know very little about Go so I don't where to start troubleshooting the issue.

Hi! I was hoping to avoid that since the ceph cluster is on an offline, secure system.. But I have initiated the process to transfer the files with the outputs and should have them available tomorrow. I did try running export-mappings on a second ceph cluster (with no backfilling operations running) and it seemed to work, so I'm sure you're right about the messy cluster being in an unusual state.
Thanks!

Huh, this is quite strange. Both OSD 64 and 38 are up and in, 34 is up and out (being drained).
Wait, the issue is that 64 is both up_primary and acting_primary. Or, that 64 is up but not 38 and 24 (which are also up, even if 34 is out).
Is there anything I can do to get you more information?
And is there anything I can do about that pg?!

The issue is that the up set should have 3 members, but it has 1.
Looking deeper, that's the only PG in that pool (pool 1). Should that pool even exist?


[feat] Support test fixtures

This PR introduces support for ReFrame test fixtures. In this context, a fixture is just another test that prepares some resources for the parent test. Test fixtures may contain other fixtures, and so on, where the parent fixture (or test) will only execute after the required fixtures have finished. Fixtures are implemented using the dependencies framework, so the resources generated by a given fixture are not cleared (i.e. its cleanup pipeline stage is not executed) until the last test that requires from that fixture completes.
Resource sharing
Fixtures enable multiple tests to share resources. The level of resource sharing can be controlled by the fixture scope. The valid scope values are session, partition, environment and test, where test is the scope default value:

session: The resources are shared by the whole ReFrame session. This must not depend on the partition nor the programming environment, so these fixtures must be RunOnlyRegressionTests.
partition: Fixtures with this scope will only share resources with tests within the same partition. This fixture will run as many times as the number of partitions required by the parent tests. This fixtures must not depend on the programming environment either, which means that they must also be RunOnlyRegressionTests.
environment: The resources are only shared across the tests in a given partition-environment combination. The fixture class can be of any kind from the derived classes from RegressionTest.
test: The resources are private to the parent test.

class MyFixture(rfm.RunOnlyRegressionTest):
   '''Manage a resource'''

@rfm.simple_test
class TestA(rfm.RegressionTest):
    # These resources are shared across the full session
    foo = fixture(MyFixture, scope='session')

    # This will trigger another private evaluation of `MyFixture`
    bar = fixture(MyFixture, scope='test')

Fork-Join model
A fixture may be parameterised using the parameter builtin. By default, a parameterised fixture will parameterise its parent tests. This behaviour is referred to as fork and can be controlled with the action argument. Alternatively, the parent test may want to gain access to all the variants from a given fixture, which in that case, the action argument must be set to join:
class MyFixture(rfm.RegerssionTest):
   p = parameter(range(2))

@rfm.simple_test
class MyForkedTest(rfm.RegressionTest):
    '''This test will run once per variant of it's fixtures.'''
    f = fixture(MyFixture, action='fork')

    @run_after('setup')
    def access_fixtures(self):
        '''Print the parameter value from the fixture f.'''
        print(self.f.p)

@rfm.simple_test
class MyReductionTest(rfm.RegressionTest):
    '''This test gathers all the variants from ``MyFixture``'''
    f = fixture(MyFixture, action='join')

    @run_after('setup')
    def access_fixtures(self):
        '''Print the parameter value from each fixture variant.

        Here f is a list containing all the variants from MyFixture.
        '''
        for fixture in self.f:
            print(fixture.p)

Fixture parameter space sub-selection
When a given fixture has a complex  or large parameter space, the parent test may filter out undesired points in the parameter space. This can be controlled through the variants argument:
class MyComplexFixture(rfm.RegressionTest):
    # A fixture with 400 different variants.
    p0 = parameter(range(100))
    p1 = parameter(['a', 'b', 'c', 'd'])

@rfm.simple_test
class MyTest(rfm.RegressionTest):
    # Use only 10 variants from MyComplexFixture
    foo = fixture(MyComplexFixture, variants={'p0': lambda x: x<10, 'p1': lambda x: x=='d'})

    # Select the fixture variants by index
    bar = fixture(MyComplexFixture, variants=range(300,310))

Setting fixture variables
A parent test may set the default value of variables from the fixture class through the variables argument. Fixtures that set default variables will be interpreted as different from other fixtures that use the same class with its default values.
class MyFixtyre(rfm.RunOnlyRegressionTest):
    '''Manage a resource'''
    my_variable = variable(int, value=10)
    ...

@rfm.simple_test
class MyTest(rfm.RegressionTest):
    # Set my_var in the fixture foo
    foo = fixture(MyFixture, scope='session', variables={'my_variable': 5})

    # Here bar is a different fixture and will also run despite that foo has a session scope.
    bar = fixture(MyComplexFixture, scope='session')

Other code improvements
Each variant of a class derived from the RegressionTest class has a unique index that points to a location in both the parameter and fixture spaces. The variant of a class is selected during class instantiation, where the variant number is passed as the variant_num argument to the class constructor.
The total number of variants of a class can be queried through the cls.num_variants property, and the full name of a given class variant can be computed with the class method cls.fullname(variant_num=<the_given_variant_num>). The current variant number of an instance is available through the property self.variant_num, along the variant sub-components of the parameter and fixture spaces (self.param_variant and self.fixture_variant, respectively).
The above makes serialisation of large parameter spaces trivial:
@rfm.simple_test
class MyTest(rfm.RegressionTest):
    p0 = parameter(range(1000))
    p1 = parameter(['a', 'b', 'c', 'd'])

    @run_after('init')
    def serialise_param_space_execution(self):
        if self.variant_num > 0:
            self.depend_on(type(self).fullname(self.variant_num-1)

Also, sub-selecting custom portions of the full variant space based on the test parameters is straightforward with the class method cls.get_variant_nums:
# Get a list of variant numbers for the variants with p0 < 500 and p1 == 'a'
variants = MyTest.get_variant_nums(p0=lambda x: x < 500, p1=lambda x: x == 'a')

Notes
Docs and unit tests to follow after we settle on the implementation. The current state of the different docstrings is just a draft. Those will be revisited alongside the documentation.
Closes #1577
Closes #1793

I like this PR a lot. It gives a whole new dimension on how to run and combine tests! The addition of a join action for the fixtures is also very powerful and goes a step forward compared to pytest fixtures for example. I have just a couple of comments:
We should mention at the documentation how the standard concepts of "setup" and "tear-down" can be implemented with ReFrame fixtures. The "setup" I see it as a series of post-init hooks, whereas the "tear-down" is a set of cleanup hooks. There is also an implicit "tear-down", since ReFrame cleans up automatically the resources of the fixtures if their dependents have finished successfully.
My second comment is about the indexing and how it is exposed to the users. I like that the parametrised tests are now indexed consistently, but as a user it has little value to address "test 123" if I don't know which exact test in the parametrisation space this corresponds to. The same goes for the example of serialisation, the index - 1 test could be anything. If we want to allow users to use raw indexes, I think that we should make the test indexing transparent, i.e., we should document the way the indexes are generated.
I will later have a look into the implementation.

Regarding the indexing, users can query the actual parameter values doing cls.param_space[<variant_num>], and this will return a dictionary with the value for each parameter for this given variant number. The only issue is that, with fixtures, the "global" variant number can be larger than the size of the parameter space. That mapping from "global" (or collapsed) index into the parameter+fixture components is done by the cls._map_variant_num function. I think it may make sense to make this one public.
Also, another thing to take into account is that the conditions passed to filter the parameter space are always inclusive, so users would need to call the cls.get_variant_nums function multiple times to build more complex sub-sets
param_set = set()
param_set.update(MyTest.get_variant_nums(<a_set_of_conditions>))
param_set.update(MyTest.get_variant_nums(<another_set_of_conditions>))

Let me think a bit about it also from the test naming perspective, what would be a reasonable user API. What you have now, it totally makes sense, but it feels a bit too low level.

I'm playing with the example fixtures on a system with multiple partitions and multiple environments and I'm getting could not resolve dependency errors when trying to list a the test example with the fixtures. Here is my file:
import reframe as rfm

class MyFixture(rfm.RunOnlyRegressionTest):
    '''Manage a resource'''

@rfm.simple_test
class TestA(rfm.RegressionTest):
    # These resources are shared across the full session
#    foo = fixture(MyFixture, scope='session')

    # This will trigger another private evaluation of `MyFixture`
    bar = fixture(MyFixture, scope='environment')

    valid_systems = ['*']
    valid_prog_environs = ['*']

And here is the command I'm trying with:
./bin/reframe -C config/cscs.py -c ../misc/reframe-tests/fixtures/example.py -l

And here's the errors and the output:
[ReFrame Setup]
  version:           3.9.0-dev.0+99a0b5f6
  command:           './bin/reframe -C config/cscs.py -c ../misc/reframe-tests/fixtures/example.py -l'
  launched by:       karakasv@dom101
  working directory: '/users/karakasv/Devel/reframe'
  settings file:     'config/cscs.py'
  check search path: (R) '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py'
  stage directory:   '/users/karakasv/Devel/reframe/stage'
  output directory:  '/users/karakasv/Devel/reframe/output'

./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:login', 'builtin') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:login', 'PrgEnv-cray') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:login', 'PrgEnv-gnu') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:login', 'PrgEnv-intel') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:login', 'PrgEnv-pgi') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:login', 'PrgEnv-nvidia') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:gpu', 'builtin') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:gpu', 'PrgEnv-cray') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:gpu', 'PrgEnv-gnu') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:gpu', 'PrgEnv-intel') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:gpu', 'PrgEnv-pgi') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:gpu', 'PrgEnv-nvidia') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:mc', 'builtin') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:mc', 'PrgEnv-cray') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:mc', 'PrgEnv-gnu') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:mc', 'PrgEnv-intel') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:mc', 'PrgEnv-pgi') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:mc', 'PrgEnv-nvidia') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:jupyter_gpu', 'builtin') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: could not resolve dependency: ('TestA_MyFixture', 'dom:jupyter_mc', 'builtin') -> 'MyFixture_dom:jupyter_gpu_PrgEnv-intel'
./bin/reframe: skipping all dependent test cases
  - ('TestA_MyFixture', 'dom:jupyter_mc', 'builtin')
  - ('TestA_MyFixture', 'dom:gpu', 'builtin')
  - ('TestA_MyFixture', 'dom:gpu', 'PrgEnv-cray')
  - ('TestA_MyFixture', 'dom:login', 'PrgEnv-nvidia')
  - ('TestA_MyFixture', 'dom:mc', 'builtin')
  - ('TestA_MyFixture', 'dom:login', 'PrgEnv-intel')
  - ('TestA_MyFixture', 'dom:login', 'builtin')
  - ('TestA_MyFixture', 'dom:mc', 'PrgEnv-cray')
  - ('TestA_MyFixture', 'dom:login', 'PrgEnv-pgi')
  - ('TestA_MyFixture', 'dom:login', 'PrgEnv-gnu')
  - ('TestA_MyFixture', 'dom:mc', 'PrgEnv-nvidia')
  - ('TestA_MyFixture', 'dom:gpu', 'PrgEnv-nvidia')
  - ('TestA_MyFixture', 'dom:mc', 'PrgEnv-intel')
  - ('TestA_MyFixture', 'dom:gpu', 'PrgEnv-intel')
  - ('TestA_MyFixture', 'dom:jupyter_gpu', 'builtin')
  - ('TestA_MyFixture', 'dom:login', 'PrgEnv-cray')
  - ('TestA_MyFixture', 'dom:gpu', 'PrgEnv-pgi')
  - ('TestA_MyFixture', 'dom:mc', 'PrgEnv-pgi')
  - ('TestA_MyFixture', 'dom:mc', 'PrgEnv-gnu')
  - ('TestA_MyFixture', 'dom:gpu', 'PrgEnv-gnu')

[List of matched checks]
- MyFixture_dom:gpu_PrgEnv-nvidia (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
- MyFixture_dom:mc_PrgEnv-intel (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
- MyFixture_dom:mc_PrgEnv-cray (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
- MyFixture_dom:login_PrgEnv-intel (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
- MyFixture_dom:mc_PrgEnv-gnu (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
- MyFixture_dom:login_PrgEnv-gnu (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
- MyFixture_dom:login_PrgEnv-cray (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
- MyFixture_dom:gpu_PrgEnv-intel (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
- MyFixture_dom:gpu_builtin (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
- MyFixture_dom:mc_builtin (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
- MyFixture_dom:gpu_PrgEnv-gnu (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
- MyFixture_dom:jupyter_mc_builtin (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
- MyFixture_dom:gpu_PrgEnv-cray (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
- MyFixture_dom:gpu_PrgEnv-pgi (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
- MyFixture_dom:mc_PrgEnv-pgi (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
- MyFixture_dom:jupyter_gpu_builtin (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
- MyFixture_dom:login_builtin (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
- MyFixture_dom:mc_PrgEnv-nvidia (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
- MyFixture_dom:login_PrgEnv-nvidia (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
- MyFixture_dom:login_PrgEnv-pgi (found in '/users/karakasv/Devel/misc/reframe-tests/fixtures/example.py')
Found 20 check(s)

Log file(s) saved in '/users/karakasv/Devel/reframe/reframe.log', '/users/karakasv/Devel/reframe/reframe.out'

On another note regarding this PR, I think that we should not list fixtures as separate tests when passing the -l option. We could still show them, though, in the detailed listing with the -L option.

This happens only when scope is either partition or environment.

Yes, it turns out the bug I fixed yesterday was something else. I'll push the fix for your reproducer shortly.

bomb-proof

Caution, I have a big 💣
This works, but it fails with if you try to scope it with the -p option. It seems that fixtures are built for environments that are not filtered:
--system=daint:login -p builtin

./bin/reframe: could not resolve dependency: ('TestA_MyFixture_MyFixture_MyFixture_MyFixture', 'daint:login', 'builtin') -> 'MyFixture_daint:login_gnu'
./bin/reframe: skipping all dependent test cases
  - ('TestA_MyFixture_MyFixture_MyFixture_MyFixture', 'daint:login', 'builtin')

In my defence, this is not a bug on the fixtures side per-se (🥲) The above error is a side effect of the sys/env filtering happening too late. I can see a workaround, a hack, and a proper fix to address this:

The workaround: Simply document that filtering with --system and -p from the command line is disallowed when using fixtures (not ideal).
The hack: Move the filtering into the RegressionTestRegistry. Then, we can simply set valid_partitions and valid_prog_environs to the filtered sys/env pair if the test supports it.
The LTS fix: run the whole valid_partitions-valid_prog_environs matrix through the parameter space. As discussed, this might be the right path to also address #1987. This option would also simplify the FixtureRegistry implementation.

"The bomb has been defused ..."
The main issue with the above 💣is that --system and -p are treated differently by the framework. The runtime would only load the partition into runtime.runtime().system.partitions as specified by the --system option. However, regardless what we pass in -p, this runtime structure will contain all the environments per partition as specified in the config file. This means that there is no way for the fixture registry to check whether a given environment is a valid one or not, because that filtering is done much later by the framework.
Since those environments that do not match the patterns provided by -p are never used in the session, I have moved that "filtering" way ahead to when this runtime structure is constructed. Then each partition in the runtime structure will only have the environments that match the patters provided by -p. This makes this information available for the FixtureRegistry, which now always selects a valid partition and a valid environment.

I have added the new class method cls.get_variant_info that allows to access the "raw" variant information. See the gist below to play around with it. Also, the naming scheme of the fixtures has been improved a bit.
https://gist.github.com/jjotero/cf6a4051ce77d9d8cfb912aebee28e4d

I don't understand why fullname() is part of the metaclass which makes it eventually a test class method. The final name of the fixtures uses information from the object at the injection time (the valid prog. envs and the partitions). This practically restricts the possibilities we have with the fullname and also leads to inconsistent names: at the test level the name uses currently the variant id, whereas the fixture name uses the prog. env and partition from the scope (wherever relevant). Not having access to the fixture scope information (i.e., the exact environment and partition that is instantiated for) makes it difficult to support a richer naming syntax as the one I proposed in #1794. Whereas I could reproduce the test name proposed there, I got stuck when I tried to reproduce the fixture name that uses scope information. I believe that the fullname() must be an instance method that has access to all the information of either the test or the fixture.
Another comment is why do you index by name the fixtures in the FixtureRegistry and not by the variant number? That practically duplicates the naming logic.

I have already a version with the naming scheme in #1794, but I came across the following, which is present also in your branch. Suppose you have this test:
class MyFixture(rfm.RunOnlyRegressionTest):
    '''Manage a resource'''
    p = parameter([1, 2])

@rfm.simple_test
class TestA(rfm.RunOnlyRegressionTest):
    f = fixture(MyFixture, scope='test', action='join')
    t = fixture(MyFixture, scope='test')

    valid_systems = ['*']
    valid_prog_environs = ['*']

The produced dependencies for one variant of TestA (e.g.TestA^MyFixture@0+MyFixture^) are the following:
      - ('TestA^MyFixture@0+MyFixture^', 'generic:default', 'builtin') -> ('MyFixture_1~TestA^MyFixture@0+MyFixture^', 'generic:default', 'builtin')
      - ('TestA^MyFixture@0+MyFixture^', 'generic:default', 'builtin') -> ('MyFixture_1~TestA^MyFixture@0+MyFixture^', 'generic:default', 'builtin')
      - ('TestA^MyFixture@0+MyFixture^', 'generic:default', 'builtin') -> ('MyFixture_2~TestA^MyFixture@0+MyFixture^', 'generic:default', 'builtin')

Which means that one of the fixtures of f and t is shared. Is this intended? I saw the same with my naming scheme, but got back to your original branch to check if I have messed up with something. This is how it looks with my naming scheme:
    Dependencies (conceptual):
      TestA%t.p=2
        ^MyFixture%p=2~TestA%t.p=2
        ^MyFixture%p=1~TestA%t.p=2
        ^MyFixture%p=2~TestA%t.p=2

    Dependencies (actual):
      - ('TestA%t.p=2', 'generic:default', 'builtin') -> ('MyFixture%p=2~TestA%t.p=2', 'generic:default', 'builtin')
      - ('TestA%t.p=2', 'generic:default', 'builtin') -> ('MyFixture%p=1~TestA%t.p=2', 'generic:default', 'builtin')
      - ('TestA%t.p=2', 'generic:default', 'builtin') -> ('MyFixture%p=2~TestA%t.p=2', 'generic:default', 'builtin')

The first MyFixture%p=2~TestA%t.p=2 is coming from t and the second one from f. Checking in the registry's add() method it seems that indeed both of these fixtures are registered with the same name.

Yes, that's correct. Both fixtures are the same because the final fixture name does not take into account the action argument. However, the scope set here is 'tests' and this is the same test that is simply using the fixture twice. So from the scope point of view, this behaviour is correct.

Another comment is why do you index by name the fixtures in the FixtureRegistry and not by the variant number? That practically duplicates the naming logic.

One is allowed to register the same fixture multiple times with different scopes, which in this case they make their way in to the name. Also, a fixture can be registered multiple times setting variables from the fixture directive, and this will also modify their name. All these examples would have the same variant, but different name. Perhaps referring to this as the 'name' is not very accurate of what it represents. This is just a unique string that I can use to build and fetch the dependencies later on.

Yes, that's correct. Both fixtures are the same because the final fixture name does not take into account the action argument. However, the scope set here is 'tests' and this is the same test that is simply using the fixture twice. So from the scope point of view, this behaviour is correct.

I'm fine with this behaviour, but I think that this is worth pointing it out in the documentation, because I was expecting that f and t are different instances of MyFixture that would be executed separately. This means that MyFixture%p=2~TestA%t.p=2 will be executed only once and reused two times.


Merge UI Code

Mark, can you erase the contents of your bin directory and then try rebuilding?
If that works, can you add the /src/GenomeExplorer/GenomeExplorer/bin directory to the /.gitignore file and git forget all those files?  Including the bin directory doubles the project's size and shouldn't be necessary if we can just rebuild the files.  We don't really want to track binary files that get rebuilt with every source change.

That won't work. SQLite is in that bin directory, which is a big driver of the size. I could probably remove some of the contents of that folder if you still wanted me too, but the sqlite dlls need to be there.


Plex won't play music on local network if internet is offline

What version of Music Assistant has the issue?
2.0.7
What version of the Home Assistant Integration have you got installed?
No response
Have you tried everything in the Troubleshooting FAQ and reviewed the Open and Closed Issues and Discussions to resolve this yourself?

[X] Yes

The problem
When my internet is offline, I unable to play files on a plex server inside my local network.
Getting a 443 error, while trying to connect to plex.tv.
Ideally I would like it to use the local plex API once it has been set up, as internet issues really shouldn't effect local playback 🤞
How to reproduce
Set up MASS, plex.
Take network offline.
Try playing music from plex, no bueno.
Music Providers
Plex
Player Providers
Unable to get this far, files don't even sync. Both browser and other speakers having similar issue.
Full log output
No response
Additional information
No response
What version of Home Assistant Core are your running
2024.7.1
What type of installation are you running?
Home Assistant Container
On what type of hardware are you running?
Generic x86-64 (e.g. Intel NUC)

Sorry, just realised this is the integration, not the server. Should probably move it there?


Update index.md

Added a line to the detailed commit messages section on the importance of making sure your commit messages explain what changed fro ease of roll-back

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
Closes #XXXXX

Thank you for your contribution to the page! 👍
We're happy to accept these changes, and look forward to future contributions. 🎉