SQLite's StatementWorker consumes 100% inside its busy loop

Hi :-)
First of all, the code:
[dependencies]
sqlx = { version = "0.4.0-beta.1", default-features = false, features = ["runtime-tokio", "sqlite"] }
tokio = { version = "0.2", features = ["full"] }

use sqlx::Connection;

#[tokio::main]
async fn main() {
    let mut db = sqlx::SqliteConnection::connect(":memory:")
        .await
        .unwrap();

    let value: Option<(i64,)> = sqlx::query_as("SELECT 1")
        .fetch_optional(&mut db)
        .await
        .unwrap();

    // Keep the application alive
    loop {
        tokio::time::delay_for(tokio::time::Duration::from_secs(1)).await;
    }
}

This code makes the application instantaneously use 100% CPU.
Curious about it, I've conducted an investigation that led me here first:
impl<'c> Executor<'c> for &'c mut SqliteConnection {
    /* ... */

    // sqlx-core/src/sqlite/connection/executor.rs:157
    fn fetch_optional<'e, 'q: 'e, E: 'q>(
        self,
        query: E,
    ) -> BoxFuture<'e, Result<Option<SqliteRow>, Error>>
    where
        'c: 'e,
        E: Execute<'q, Self::Database>,
    {
        let mut s = self.fetch_many(query);

        Box::pin(async move {
            while let Some(v) = s.try_next().await? {
                if let Either::Right(r) = v {
                    return Ok(Some(r));
                }
            }

            Ok(None)
        })
    }

    /* ... */
}

... and later there:
impl StatementWorker {
    pub(crate) fn new() -> Self {
        /* ... */

        let handle = spawn({
            /* ... */

            move || {
                /* ... */

                // sqlx-core/src/sqlite/statement/worker.rs:49
                'run: while status.load(Ordering::Acquire) >= 0 {
                    'statement: loop {
                        match status.load(Ordering::Acquire) {
                            STATE_CLOSE => {
                                /* ... */
                            }

                            STATE_READY => {
                                /* ... */
                            }

                            _ => {
                                // waits for the receiving end to be ready to receive the rows
                                // this should take less than 1 microsecond under most conditions
                                spin_loop_hint();
                            }
                        }
                    }
                }
            }
        });

        /* ... */
    }

    /* ... */
}

The issue is that the current implementation of fetch_optional() retrieves only the first row from the stream and then silently discards the stream, without letting StatementWorker know about it. The worker then starts entering spin_loop_hint() millions times per second, waiting for response from a stream that doesn't exist anymore.
This forever-busy-loop can be reproduced even without referring to fetch_optional(), like so:
async fn main() {
    /* ... */

    let mut stream = sqlx::query_as::<_, (Option<i64>,)>("SELECT 1")
        .fetch_many(&mut db);

    // Retrieve just the first row and then discard rest of the stream
    while let Some(v) = s.try_next().await.unwrap() {
        break;
    }

    // Keep the application alive
    loop {
        tokio::time::delay_for(tokio::time::Duration::from_secs(1)).await;
    }
}

As for a possible solution, I'd suggest introducing a new type, called ExecutionToken:
pub(crate) struct ExecutionToken {
    status: Arc<AtomicI32>,
}

impl Drop for ExecutionToken {
    fn drop(&mut self) {
        self.status.store(STATE_READY, Ordering::Release);
    }
}

/* ... */

impl StatementWorker {
    pub(crate) fn execute(&self, statement: &StatementHandle) -> ExecutionToken {
        let token = ExecutionToken {
            status: Arc::clone(&self.status),
        };

        self.statement
            .store(statement.0.as_ptr(), Ordering::Release);

        token
    }
}

/* ... */

// and then later just:
// let _token = worker.execute(handle);

Since I'm proficient in neither sqlx's nor SQLite's internals, I'm not sure if that's the correct approach (although it works on my machine ™); if you think that's alright, I could prepare a merge request :-)
Thank you for your great work on this library,
Cheers.

I'm having second thoughts though: while using ExecutionToken does fix fetch_optional(), following code still causes 100% CPU usage, for the same reason:
async fn main() {
    /* ... */

    let mut stream = sqlx::query_as::<_, (i64,)>("SELECT 1")
        .fetch_many(&mut db);

    // Fetch first item to initialize the stream
    stream.try_next().await.unwrap();

    loop {
        tokio::time::delay_for(tokio::time::Duration::from_secs(10)).await;
    }

    drop(stream);
}

Maybe instead of a busy-wait loop with shared state we could use an async-aware mpsc channel?

Okie, so I've managed to get a correct proof-of-concept working:
pub(crate) struct StatementWorker {
    tx: mpsc::Sender<StatementWorkerCommand>,
}

enum StatementWorkerCommand {
    Step {
        statement: StatementHandle,
        sent: Arc<AtomicBool>,
        tx: mpsc::Sender<Result<Either<u64, ()>, Error>>,
    },
}

impl StatementWorker {
    pub(crate) fn new() -> Self {
        let (tx, rx) = mpsc::channel();

        thread::spawn(move || {
            for cmd in rx.into_iter() {
                match cmd {
                    StatementWorkerCommand::Step {
                        statement,
                        sent,
                        tx,
                    } => {
                        let status = unsafe { sqlite3_step(statement.0.as_ptr()) };

                        let resp = match status {
                            SQLITE_ROW => Ok(Either::Right(())),
                            SQLITE_DONE => Ok(Either::Left(statement.changes())),
                            _ => Err(statement.last_error().into()),
                        };

                        let _ = tx.send(resp);

                        sent.store(true, Ordering::Release);
                    }
                }
            }
        });

        Self { tx }
    }

    pub(crate) async fn step(
        &mut self,
        statement: &StatementHandle,
    ) -> Result<Either<u64, ()>, Error> {
        let (tx, rx) = mpsc::channel();
        let sent = Arc::new(AtomicBool::new(false));

        self.tx
            .send(StatementWorkerCommand::Step {
                statement: *statement,
                sent: sent.clone(),
                tx,
            })
            .unwrap();

        while !sent.load(Ordering::Acquire) {
            yield_now().await;
        }

        rx.recv().unwrap()
    }
}

Sadly, it's about 1/4 slower than the original code: it manages to launch ~30k queries per second, while the original one gets around ~40k.
I'm just gonna drop this patch in case someone needs a quick fix for their codebase, and get back to figuring stuff out :-P

Okie, I've managed to get a correct proof-of-concept working:
// worker.rs

use crate::error::Error;
use crate::sqlite::statement::StatementHandle;
use crossbeam_channel::{bounded, unbounded, Sender};
use either::Either;
use libsqlite3_sys::{sqlite3_step, SQLITE_DONE, SQLITE_ROW};
use sqlx_rt::yield_now;
use std::thread;

// Each SQLite connection has a dedicated thread.

// TODO: Tweak this so that we can use a thread pool per pool of SQLite3
// connections to reduce       OS resource usage. Low priority because a high
// concurrent load for SQLite3 is very       unlikely.

pub(crate) struct StatementWorker {
    tx: Sender<StatementWorkerCommand>,
}

enum StatementWorkerCommand {
    Step {
        statement: StatementHandle,
        tx: Sender<Result<Either<u64, ()>, Error>>,
    },
}

impl StatementWorker {
    pub(crate) fn new() -> Self {
        let (tx, rx) = unbounded();

        thread::spawn(move || {
            for cmd in rx.into_iter() {
                match cmd {
                    StatementWorkerCommand::Step { statement, tx } => {
                        let status = unsafe { sqlite3_step(statement.0.as_ptr()) };

                        let resp = match status {
                            SQLITE_ROW => Ok(Either::Right(())),
                            SQLITE_DONE => Ok(Either::Left(statement.changes())),
                            _ => Err(statement.last_error().into()),
                        };

                        let _ = tx.send(resp);
                    }
                }
            }
        });

        Self { tx }
    }

    pub(crate) async fn step(
        &mut self,
        statement: &StatementHandle,
    ) -> Result<Either<u64, ()>, Error> {
        let (tx, rx) = bounded(1);

        self.tx
            .send(StatementWorkerCommand::Step {
                statement: *statement,
                tx,
            })
            .unwrap();

        while rx.is_empty() {
            yield_now().await;
        }

        rx.recv().unwrap()
    }
}

In terms of performance, this version is slightly slower than the original one, but - thanks to crossbeam's channels - not much: I've done a simple benchmark that launches the same query 100k times, and the original code took 350 ms to run, while this one took 420 ms, so I think it's in the ballpark of "alright" when it comes to SQLite.
Whaddya think?

I'm being bit by this, too, and definitely wouldn't mind a 0.7 microsecond overhead per query.  (BTW, thanks for investigating this, @Patryk27)

Thanks a lot for digging into this. StatementWorker is definitely pretty experimental and if we can keep the perf similar I'm totally open to re-arranging it. What you suggest seems pretty good at first glance.
Are you open to sending in a PR with the changes?

Yeah, totally - I'll try to submit the patch today :-)


make opendolphin and integration tests more reliable

Currently the opendolphin tests run often in time-out exceptions with no reason. Most of the time retriggering solves the problem.
Regarding the integration tests here we run in endless long-poll with no reason as well.
Both should be solved to get better results from the build

Integration test problem is fixed by https://github.com/canoo/dolphin-platform/pull/770


ci: Enable caching

Now that most jobs are on the stable channel, enabling caching should be more effective than it would have been on nightly.
Checklist

[ ] Sensible git history (for example, squash "typo" or "fix" commits). See the Rewriting History guide for help.
[ ] Update the changelog (if necessary)

I think we should use an approach as in https://github.com/rust-osdev/multiboot2/blob/main/.github/workflows/_build-rust.yml#L52 instead.
The solution you picked for caching only caches dependencies and only works reliably with a Cargo.lock file, as the README says. https://github.com/Swatinem/rust-cache
Hence, I'm in favor of a solution as in the Multiboot2 crate.

Could you please check the times with and without cache with the cargo.lock now being part of the tree?
If it works, sure, why not. A "pretty good" solution with low effort is probably better than an over-engineered solution that takes us much time. :)

Very similar timings to before: https://github.com/rust-osdev/uefi-rs/actions/runs/4519831136
I think the addition of Cargo.lock doesn't make an immediate difference, it's more of an over-time thing as dependent crates release semver-compat updates that invalidate the cache.

Looks to me as the cache doesn't work as efficient as it should.. There are way to many cache regenerations although the dependencies didn't change.

The rust-build-feature_permutations seems expected since that's nightly Rust. It'll be invalidated every day.
I'm unsure why the rust-test one was invalidated more recently than the rest, that's odd.
Overall it seems like the cache is working well, our PR runs are faster (except often for the Windows job, which seems stubbornly slow.) Are there specific runs of the action you can link to that seem slower than expected?


Black screen on iOS 11

@muaz-khan
Desktop Chrome new version

iOs 11 Safari:

How can i fix it! Thank you!

Hi,
I have same issue

did you solve it?

Same issue.  Is it possible for this to work using mobile Safari or mobile Chrome on iOS 11?

Simply set this for dynamically created video elements:
connection.onstream = function(event) {
    event.mediaElement.setAttributeNode(document.createAttribute('autoplay'));
    event.mediaElement.setAttributeNode(document.createAttribute('playsinline'));
    event.mediaElement.setAttributeNode(document.createAttribute('controls'));

    document.body.appendChild(event.mediaElement);
};

Where event.mediaElement is HTMLVideoElement.

@muaz-khan i follow
I followed this tutorial http://www.rtcmulticonnection.org/docs/Write-iOS-Apps/
At first the app worked perfectly, after updating the MAC OS, the code did not work anymore. The remote video turned black
The remote video was black, I upgraded the version of RTCMultiConnection to v3.4.4
The video player looks better, but the remote video does not show up any more
My index.js

I have the same issues on Ubuntu


Weird etcdHighNumberOfFailedGRPCRequests prometheus rule in v2.5 rancher-monitoring

I just was trying out the v2.5 monitoring chart and noticed that I was getting a large number of alerts for
etcdHighNumberOfFailedGRPCRequests for kube-etcd (alertname="etcdHighNumberOfFailedGRPCRequests", grpc_method="Watch", grpc_service="etcdserverpb.Watch", prometheus="monitoring/monitoring-rancher-monitor-prometheus", severity="critical")
I took a look at the rules and something struck me as odd. This is the expression used to determine this alert: 100 * sum(rate(grpc_server_handled_total{job=~".*etcd.*", grpc_code!="OK"}[5m])) BY (job, instance, grpc_service, grpc_method) / sum(rate(grpc_server_handled_total{job=~".*etcd.*"}[5m])) BY (job, instance, grpc_service, grpc_method)
Which seems to result in garbage data:

However changing BY (job, instance, grpc_service, grpc_method) with simply BY (instance) seems to produce a much more meaningful metric (% grpc failed per node)

From my perspective, it seems like it should be changed to BY (instance) as this seems to make more sense but not 100% sure. I am very new to Prometheus in general.
gz#14464
gz#14015
gz#14746
gz#15416
gz#16124
gz#16211

@gmintoco thanks for calling this out! Based on https://github.com/etcd-io/etcd/pull/12122/files#diff-051b19bef7add661d9678608040a89efaead1aa9d3063323effdfedae11c9978R76-R100, it seems like you are correct, although the etcd team seems to replace BY (job, instance, grpc_service, grpc_method) with without (grpc_type, grpc_code) instead. Can you see if that also works for you?
With respect to bringing this change into Rancher, Rancher pulls these rules from the upstream Prometheus community Helm chart kube-prometheus-stack, which has an open issue out w.r.t. trying to update the etcd alerts in https://github.com/prometheus-community/helm-charts/issues/225. Once that is resolved, we can pull in the latest etcd alerts to get those changes in.

Hi @aiyengar2 yeah I realised that after digging into things a little deeper. I actually opened that issue :) but yes just waiting to see what happens there. Thanks for the response

This alert started to rain down on me today after a node memory pressure event, and it is firing constantly even though all the nodes were rebooted. Latest RKE with latest Rancher 2.5 and Monitoring V2 stack. Can it be supressed somehow until it is fixed in upstream?
alerts:
- annotations:
    message: 'etcd cluster "kube-etcd": 100% of requests for Watch failed on etcd
      instance <IP_ADDRESS>:2379.'
  endsAt: '0001-01-01T00:00:00Z'
  fingerprint: bce5342bd6fd6152
  generatorURL: http://rancher-monitoring-prometheus.cattle-monitoring-system:9090/graph?g0.expr=100+%2A+sum+by%28job%2C+instance%2C+grpc_service%2C+grpc_method%29+%28rate%28grpc_server_handled_total%7Bgrpc_code%21%3D%22OK%22%2Cjob%3D~%22.%2Aetcd.%2A%22%7D%5B5m%5D%29%29+%2F+sum+by%28job%2C+instance%2C+grpc_service%2C+grpc_method%29+%28rate%28grpc_server_handled_total%7Bjob%3D~%22.%2Aetcd.%2A%22%7D%5B5m%5D%29%29+%3E+5&g0.tab=1
  labels:
    alertname: etcdHighNumberOfFailedGRPCRequests
    grpc_method: Watch
    grpc_service: etcdserverpb.Watch
    instance: <IP_ADDRESS>:2379
    job: kube-etcd
    prometheus: cattle-monitoring-system/rancher-monitoring-prometheus
    severity: critical
  startsAt: '2020-12-12T18:45:10.979Z'
  status: firing
commonAnnotations:
  message: 'etcd cluster "kube-etcd": 100% of requests for Watch failed on etcd instance
    <IP_ADDRESS>:2379.'
commonLabels:
  alertname: etcdHighNumberOfFailedGRPCRequests
  grpc_method: Watch
  grpc_service: etcdserverpb.Watch
  instance: <IP_ADDRESS>:2379
  job: kube-etcd
  prometheus: cattle-monitoring-system/rancher-monitoring-prometheus
  severity: critical
externalURL: http://rancher-monitoring-alertmanager.cattle-monitoring-system:9093
groupKey: '{}:{job="kube-etcd"}'
groupLabels:
  job: kube-etcd
receiver: matrix
status: firing
truncatedAlerts: 0
version: '4'

After another 109 alerts over the night, I added a silence for etcdHighNumberOfFailedGRPCRequests in AlertManager.

Can confirm we also have the same issue in our v2.5 rancher-monitoring rke setup..
Are we all waiting on upstream fixes or any updates to this issue?

Still happens after the 2.5.5 update.
Rancher | v2.5.5
User Interface | v2.5.4
Helm | v2.16.8-rancher1
Machine | v0.15.0-rancher49

Still happens after the 2.5.5 update.
Rancher | v2.5.5
User Interface | v2.5.4
Helm | v2.16.8-rancher1
Machine | v0.15.0-rancher49

Hi, I set up a new instance of Rancher 2.5.5 and a new RKE cluster and am getting these alerts. Do I need to worry about the health of the cluster? I literally just started setting things up and there are plenty of resources / low load on the servers. Thanks

Hi, I set up a new instance of Rancher 2.5.5 and a new RKE cluster and am getting these alerts. Do I need to worry about the health of the cluster? I literally just started setting things up and there are plenty of resources / low load on the servers. Thanks

My cluster is healthy and fairly sable, and I also get these constantly. I eventually silenced them through the alertmanager secret (under alertmanager.yaml):
route:
...
  # temporarily silence this alert until it's fixed upstream
  - match:
      alertname: etcdHighNumberOfFailedGRPCRequests
    receiver: "null"
...

My cluster is healthy and fairly sable, and I also get these constantly. I eventually silenced them through the alertmanager secret (under alertmanager.yaml):
route:
...
  # temporarily silence this alert until it's fixed upstream
  - match:
      alertname: etcdHighNumberOfFailedGRPCRequests
    receiver: "null"
...

My cluster is healthy and fairly sable, and I also get these constantly. I eventually silenced them through the alertmanager secret (under alertmanager.yaml):
route:
...
  # temporarily silence this alert until it's fixed upstream
  - match:
      alertname: etcdHighNumberOfFailedGRPCRequests
    receiver: "null"
...

Is it safe to silence/ignore them? I also don't see any problems at all with the cluster itself.

My cluster is healthy and fairly sable, and I also get these constantly. I eventually silenced them through the alertmanager secret (under alertmanager.yaml):
route:
...
  # temporarily silence this alert until it's fixed upstream
  - match:
      alertname: etcdHighNumberOfFailedGRPCRequests
    receiver: "null"
...

Is it safe to silence/ignore them? I also don't see any problems at all with the cluster itself.

If safe is equivalent to sleeping well while the cluster purring like a cat and no late night calls while this alert is somehow involved with a crashing service, yes, it's fairly safe 😃

If safe is equivalent to sleeping well while the cluster purring like a cat and no late night calls while this alert is somehow involved with a crashing service, yes, it's fairly safe 😃

If safe is equivalent to sleeping well while the cluster purring like a cat and no late night calls while this alert is somehow involved with a crashing service, yes, it's fairly safe 😃

I've silenced it for now :)

If safe is equivalent to sleeping well while the cluster purring like a cat and no late night calls while this alert is somehow involved with a crashing service, yes, it's fairly safe 😃

I've silenced it for now :)

The cause of the frequent etcdHighNumberOfFailedGRPCRequests alarms is not the metric expression itself.
The output of the existing expression with by:
100 * sum by(job, instance, grpc_service, grpc_method) (rate(grpc_server_handled_total{grpc_code!="OK",job=~".*etcd.*"}[5m])) / sum by(job, instance, grpc_service, grpc_method) (rate(grpc_server_handled_total{job=~".*etcd.*"}[5m]))
does not differ from the new expression using without:
100 * sum(rate(grpc_server_handled_total{job=~".*etcd.*", grpc_code!="OK"}[5m])) without (grpc_type, grpc_code) / sum(rate(grpc_server_handled_total{job=~".*etcd.*"}[5m])) without (grpc_type, grpc_code)
The high number of alerts is caused by the upstream etcd issue tracked in https://github.com/etcd-io/etcd/issues/10289, with PR https://github.com/etcd-io/etcd/pull/12196 open to address this. Until this is fixed, silencing or removing this specific alert will address the alert noise.

The cause of the frequent etcdHighNumberOfFailedGRPCRequests alarms is not the metric expression itself.
The output of the existing expression with by:
100 * sum by(job, instance, grpc_service, grpc_method) (rate(grpc_server_handled_total{grpc_code!="OK",job=~".*etcd.*"}[5m])) / sum by(job, instance, grpc_service, grpc_method) (rate(grpc_server_handled_total{job=~".*etcd.*"}[5m]))
does not differ from the new expression using without:
100 * sum(rate(grpc_server_handled_total{job=~".*etcd.*", grpc_code!="OK"}[5m])) without (grpc_type, grpc_code) / sum(rate(grpc_server_handled_total{job=~".*etcd.*"}[5m])) without (grpc_type, grpc_code)
The high number of alerts is caused by the upstream etcd issue tracked in https://github.com/etcd-io/etcd/issues/10289, with PR https://github.com/etcd-io/etcd/pull/12196 open to address this. Until this is fixed, silencing or removing this specific alert will address the alert noise.

Can we get an update on when this merge will happen please? It looks like the upstream changes are closed out?

Hi @ccittadino-ctic, maybe I misunderstand something about the issue or you linked the wrong issue? I'm not sure how prometheus-community/helm-charts/225 is related.  I do however see that the PR that Alex linked, which is the root cause of this issue, has been merged for etcd v3.5.0-alpha and may be backported to v3.4.x
When this is done, we should see these alerts clear up.  In the meantime, a good workaround would be to silence or remove this alert and make sure you have other alerts set up to track etcd health.

@ccittadino-ctic the issue was closed out in upstream without resolution. As @Tejeev mentioned, the upstream chart is still out of date at etcd v3.4.x: https://github.com/prometheus-community/helm-charts/blob/8318ce1d92e8c3dd487d89066d0f7b2405c65653/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/etcd.yaml#L2

@aiyengar2 So are we just waiting for 3.5 of prometheus to release before Rancher can take it in?

We are waiting on kube-prometheus-stack to absorb the latest dashboards from etcd; once kube-prometheus-stack has made the changes, we'll rebase on the latest upstream chart to get in the fix.

Update: upstream still has not updated their dashboards primarily because etcd has not updated the dashboards published on their website.
To get this change from the upstream chart:

A user must submit a PR to etcd-io/etcd at contrib/mixin/mixin.libsonnet - DONE
The contents of etcd-io/website at content/en/docs/v3.4/op-guide/etcd3_alert.rules.yml must be manually synced based on the above merged contents - NOT DONE
kube-prometheus-stack must run a script to update etcd dashboards - BLOCKED
Rancher must rebase to the latest upstream chart - BLOCKED

cc: @Jono-SUSE-Rancher @snasovich it may make sense to prioritize this issue as a patch on Monitoring directly since it's been over a year since the original PR was merged in.

Has this been resolved? I'm seeing errors in my environment but the system seems fine.

Has this been resolved? I'm seeing errors in my environment but the system seems fine.

I don't think so. I'm running Rancher 2.5.9, fresh install of the monitoring solution, and I'm seeing this alert on clusters running 3 etcd instances. I'm silencing it for now, hoping we don't need to in the future.

I'm not running rancher but I was looking. Seems the k8s version starting at 1.22 will upgrade to a version of etcd that fixes this bug.

Let me try to summarize the issue:

The problem of the expression comes from the grpc_code!="OK" part: it treats other non-error codes (canceled, invalidArgument, notFound, and etc) as errors so the expression reports a high rate of errors  (see GRPC standard here) thus triggers too many alerts.
The above problem in the expression has been fixed in the upstream etcd repo: https://github.com/etcd-io/etcd/pull/13127
But the above fix is not synced to the kube-prometheus-stack repo: notice that the wrong expression is still in the main branch

For rancher-monitoring charts, I think we can patch the expression instead of waiting for the upstream’s fix.

So this means in the current versions these are expected "issues"...
So "best option until fix" is to..silence alerts coming from etcd?

seeing the same issue , we adjust alerting to ignore this until fixed

@geethub97 , please check if this is reproducible in monitoring v2.
SURE-2721

@MKlimuszka Could you please link the commit where this is fixed?

same issue here, we silenced alerting to ignore this until fixed

Its the same situation with  rancher-monitoring 100.1.2+up19.0.3 .

For kube-prometheus-stack user that are coming along this issue.
I put this in the helm values.yml file for a temporary fix:
defaultRules:
  disabled:
    # the results of this rule are not accurate: https://github.com/rancher/rancher/issues/29939
    etcdHighNumberOfFailedGRPCRequests: true

additionalPrometheusRulesMap:
  additional-rules:
    groups:
      # fix etcdHighNumberOfFailedGRPCRequests alerting rules from above, adapted from: https://github.com/etcd-io/etcd/pull/13127
      - name: etcd-additional
        rules:
          - alert: etcdHighNumberOfFailedGRPCRequestsWarning
            annotations:
              message: >-
                etcd cluster "{{ $labels.job }}": {{ $value }}% of requests for {{
                $labels.grpc_method }} failed on etcd instance {{ $labels.instance
                }}.
            expr: >-
              100 * sum(rate(grpc_server_handled_total{job=~".*etcd.*",
                grpc_code=~"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded"}[5m]))
              /
              sum(rate(grpc_server_handled_total{job=~".*etcd.*"}[5m]))
              > 1
            for: 10m
            labels:
              severity: warning
          - alert: etcdHighNumberOfFailedGRPCRequestsCritical
            annotations:
              message: >-
                etcd cluster "{{ $labels.job }}": {{ $value }}% of requests for {{
                $labels.grpc_method }} failed on etcd instance {{ $labels.instance
                }}.
            expr: >-
              100 * sum(rate(grpc_server_handled_total{job=~".*etcd.*",
                grpc_code=~"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded"}[5m]))
              /
              sum(rate(grpc_server_handled_total{job=~".*etcd.*"}[5m]))
              > 5
            for: 5m
            labels:
              severity: critical

The queries do not include the 'without' clauses, but the results are looking good to me:

QA, please check if this is fixed in 2.7.0, monitoring v2.

Pass Verified in 2.7.0-rc8

created downstream rke1 cluster
installed rancher-monitoring
checked prometheus UI for etcdHighNumberOfFailedGRPCRequests - 0 active
manually ran query for etcdHighNumberOfFailedGRPCRequests alert, decreased check to 1 second: 0 results

no longer receiving high number of alerts for etcdHighNumberOfFailedGRPCRequests

same issue here with RKE rancher-monitoring 100.1.3+up19.0.3


traditional/Australian scrolling in ubuntu opposite to chromeos

In the middle of installing crouton my trackpad's scrolling switched to "Australian" scrolling style, however once I started unity I found that Ubuntu still used traditional scrolling.  When I changed back to traditional scrolling in ChromeOS it switched Ubuntu to natural scrolling and vice versa.  Whatever direction I set ChromeOS to scroll, Unity will scroll in the opposite direction.
I have tried changing the scrolling style from Ubuntu, but there is not an option under settings.
I also tried changing the scroll direction in ChromeOS while logged out of Unity, but upon starting the chroot again it is switched just the same.
I even tried powerwashing and reinstalling my whole crouton set up this time changing the ChromeOS setting back before crouton was done installing, but again when starting unity the scrolling direction was opposite to ChromeOS
Sorry if I'm missing something, but I haven't been able to find anything about this so I figured I'd post it here.
Samsung Chromebook XE303C12 (arm)
~$ croutonversion
crouton: version 1-20140826230534~master:85cae7b9
release: precise
architecture: armhf
targets: unity
host: version 5841.98.0 (Official Build) stable-channel daisy

This may be the same root cause as crbug.com/460548

@drinkcat or @dnschneid Have you guys been able to make any progress? 😄
I installed linux yesterday, and today I noticed that no matter what I tried, the scrolling was inverted from ChromeOS to Linux.
Specs:
I realize I might be giving you more info than you need, but better safe than sorry.
Chromebook: Samsung Chromebook 3
Model: 500C13-K01
ChromeOS Version: 57.0.2987.146 (64-bit)
Platform: 9202.64.0 (Official Build)
ARC Version: 3880239
Firmware Google_Celes.7287.92.74
Crouton version 1-20170315143304~master:95589555
Linux is Xenial Xerus but when I run uname -r it gives me 3.18.0-13759-g98e7e0e so I'm not sure on the version number for my linux install.


add query parameters

A bit naive, but it won't do anything unless you supply parameters, so the existing behaviour should be untouched. The alternative would be a fully fledged query parser, but I couldn't find one that would understand the Influx syntax (things like fill(...) and group by time(...))

I like it, but I don't know if this fits into the scope of this Gem. I would rather build a query generator on top of influxdb and distribute is as seperate Gem (much like influxer, but without the dependency to ActiveModel).
PS. Don't get me wrong: I'll still merge it, given you provide test cases and fix the Rubocop issues ;-)

Would love to. A few questions:

Do you know where the query generator lives in influxer? The code seems intermingled with the activerecord stuff, and I really don't want the built-in ORM.
Any idea why the JRuby test specifically fails?
Can I just add the test to https://github.com/influxdata/influxdb-ruby/blob/master/spec/influxdb/client_spec.rb ?
How do I run these tests on Travis? I enabled Travis for my fork and pushed a change, but it's not doing anything
One of the rubocop complaints is that the file now is longer than 100 lines. Should I split this change off just to avoid this?

WRT query building: would you prefer something like this?

I really don't know how to lower the cyclomatic complexity of the query builder without splitting it up into separate methods just to avoid that complaint. It doesn't appear to me it would make the code more readable.

OK, I'm fairly close here, but

Rubocop demands that a module is less than 100 lines, I'm currently at 108
I'm trying to figure out the spec syntax (used to cucumber), but I can't find where a select is fired through the client.

Sorry for the delay. I'm currently at work and will have a look when I'm home.
Don't worry too much about the module length warning from Rubocop. I'm OK with raising the limit in the .rubocop.yml (to, say, 120 lines).
JRuby usually fails over the Async specs, I usually hit the "try again" button in Travis.

So, again, sorry for the delay. Here are my remarks.
I hope this wall of text doesn't discourage you :-)
—Dominik

Do you know where the query generator lives in influxer? The code seems intermingled with the activerecord stuff, and I really don't want the built-in ORM.

No, but it's relatively easy to start from scratch (been there, done that, but I've build the code on my employers time, so I need to check whether I'm allowed to open source it...)

Any idea why the JRuby test specifically fails?

It looks like the JRuby config for Travis doesn't set $TEST_TASK and thus executes bundle exec rake (I recently made a change which adds the Rubocop task to the dependency of the default task, see f2797d0656c550e53a9c3add8690d9aee9e25356)

Can I just add the test to https://github.com/influxdata/influxdb-ruby/blob/master/spec/influxdb/client_spec.rb ?

I'd prefer a seperate file, i.e. spec/influxdb/cases/query_with_params_spec.rb or similar.

How do I run these tests on Travis? I enabled Travis for my fork and pushed a change, but it's not doing anything

That should be it... Simply push to your repo and check whether https://travis-ci.org/retorquere/influxdb-ruby/builds or https://travis-ci.org/influxdata/influxdb-ruby/pull_requests catches the commit. If neither does, drop a line to<EMAIL_ADDRESS>they responds reasonably quickly.

One of the rubocop complaints is that the file now is longer than 100 lines. Should I split this change off just to avoid this?

As I said before,  I'm OK with raising the limit.

WRT query building: would you prefer something like this?

Absolutely, yes. The responsibilty of the #query method is to pass a query to the server and receive the result. Externalizing the query building code in a seperate method really helps to understand what's going on (you can test #query_method in isolation).
On that note, I don't like this pattern:
params = Hash[* params.collect.with_index { |p, i| [(i + 1).to_s.to_sym, p] }.flatten]

because it creates and immediately destroys a lot of objects (the map creates n arrays of length 2, which flatten copies into a new array of length 2n, which then gets destructed by the *splat operator); it also iterates three times over the collection (once for map, flatten and hash construction).
In my eyes (and performance wise) a better solution would be a more functional approach:
params = params.each_with_object({}).with_index do |(param, hash), i|
  key = (i+1).to_s # no need to symbolize
  hash[key] = param
end

This might help understand this bit:
[:a,:b,:c].each_with_object({}).with_index {|(e,l),i| l[e] = i }
#=> {:a=>0, :b=>1, :c=>2}

Is there one of the existing test case files I could use to copy & modify? I'm unfamiliar with the spec syntax, and none of them immediately strike me as "ah, that one uses client.query(...)"

rubocop runs green, so I just need to add the test. I've made the changes to the query_builder.

You could use spec/influxdb/cases/query_core_spec.rb as a starting point.

How do I run tests locally? Going through Travis each time is time consuming, I bet that's not what you're doing.

I just toss bundle exec rake into my terminal, see Testing section in the Readme ;-)
To just run Rubucop either use bundle exec rubocop or bundle exec rake rubocop.

Drat, sorry about that. I found rubocop, but missed spec
Good thing Github allows squash-while-merge these days. Spec & Rubocop run green.

Are you OK with the {query => params} style? It's the least verbose syntax I could think of that was still unambiguous.

Are you OK with the {query => params} style?

I would actually prefer an ActiveRecord style:
query("string :with :placeholders", with: "value", placeholder: 42)
query("string ? ?", "value", 42) # maybe not now, but later/in a seperate PR?

Do you think, that's feasible?

I would actually prefer an ActiveRecord style:

query("string :with :placeholders", with: "value", placeholder: 42)

Hm. Looking at the current signature of `#query`, I now tend more to

```ruby
query("string :with :placeholders", params: { with: "value", placeholder: 42 })

which isn't much an improvement over
query("string :with :placeholders", => { with: "value", placeholder: 42 })

with the exception that "string :with :placeholders" is potentially large (and strings as keys for hashes are copied)...

It's why I initially used symbols as keys, I was led to believe those were more economical to use.
Question marks is not harder than the current replacer, it's just that the replacer is incredibly naive, and will blithely replace whatever it finds that matches. I thought the :...: was simply less likely to occur than simply ?. I initially did have :..., but that would make something like where time > :start:s (not the seconds designator) impossible, as I wouldn't be able to tell whether :starts means start seconds or starts at whatever default precision.

where time > :start:s (note the seconds designator)

Oh, I forgot about that. Ruby's string formatting has this feature:
"foo is %{bar}s" % { bar: 42 }
#=> "foo is 42s"

Maybe this allows for some refactoring?
On the #query method signature, let's go with
query("string :with :placeholders", params: { with: "value", placeholder: 42 })

OK, so params move to the opts, correct? That's easy enough. The whole placeholder syntax is to be replaced with %{...} then? I'm OK with that.

Note that this makes the placeholder syntax (%{...}) sort of ruby-specific. Once this PR crystallizes, I intend to propose the same syntax to at least influxdb-java.

In Ruby, we can get some perf out of %{...}, and in Java you'll need regexes anyway. In this case, there is not much difference in /:[_\w]+:/ and /%\{[_\w]+\}/ (although, I must admit, I haven't had the joy to program in Java for some 7 years I think :-)).

Btw. I've double checked whether \w in a Regexp includes _... and it does (both in Ruby as well as in Java's java.util.regex.Pattern). So /%\{[_\w]+\}/ simplifies to /%\{\w+\}/.

I'm not fond of Java, but I am sometimes subjected to it.
I have things working now, with your proposed tests added. But now rubocop has an opinion about the matter again. 121 lines in the module, 18 lines in the query builder. I can pacify robocop on the latter by splitting off the params quoting into something like query_builder_build_params_hash, but at that point I'm only doing it because the tests don't run green, I honestly don't think that adds anything to legibility.

Maybe it's time to extract #query_builder (and maybe #quote) in a seperate class, e.g.
# lib/influxdb/query/builder.rb
class Builder < Struct.new(:query)
  def build(params)
    # former Client#query_builder code
  end

  def quote
    # former Client#quote code
  end
end

and in #query just call query = Builder.new(query).build(opts[:params]). Thoughts?

It'd still run into the same rubocop complaint.

This could satisfy Rubocop:
params = case params
         when Array    then convert_from_array(params)
         when Hash     then convert_from_hash(params)
         when NilClass then {}
         else raise ArgumentError
         end

(where convert_from_* are private)

Also, class construction has significant overhead compared to method calling (if you're concerned with performance):
require 'benchmark'

class X
  def calc(n)
    3 * n
  end
end

def calc(n)
  3 * n
end

x = X.new
puts Benchmark.measure {
  10000.times {
    x.calc(10)
  }
}
puts Benchmark.measure {
  10000.times {
    X.new.calc(10)
  }
}

Your call. I'd suggest a private builder that could be reused, the class wouldn't have any state.

Yeah, I like the idea of a private method returning a stateless builder instance, go for it.
In the end this PR (including my comment earlier) might be an exemplary case of over-engineeing: If I'm honest, converting some method arguments shouldn't be as much a bottleneck as, say the network request and or processing the returned result. On the other hand, given a seperate QueryBuilder class provides room for improvement and expansion.
Btw. I'll need to get up in about 6 hours again, so I'll better lay down for today. Thanks for your commitment so far!

The case...when doesn't satisfy rubocop, it still comes to 16 lines.

Except Core isn't a class but a module, so I can't give it a private var.
Perhaps more tomorrow. Early to rise for me too. But I think the builder is pretty lean as it is after the latest changes -- I personally don't see much value in spinning it off.

The modules in lib/influxdb/query are mixed into InfluxDB::Client. They are (historically) thightly coupled and not intended to be used outside of InfluxDB::Client (I wouldn't suggest changing this in this PR ;-))
So, continueing with that anti-pattern, I'd add this to the Client
def query_builder
  @query_builder ||= QueryBuilder.new
end

and reference it from Core.

Final remark for today:  # rubocop:disable Metrics/MethodLength is a valid option. Has already happened ;-)

Tests run green, but at the cost of this. I know I could split them off to params_from_array and params_from_hash, but then in each of those I'd have to check the type of the params object again to be safe and it'd just be code duplication.

Wait what, you did that globally? Let me fetch from upstream.

Ah, no, the rubocop directive you pointed to doesn't affect core.rb. So my "cost" comment stands. I still think this is a clean implementation -- the splitting off doesn't add to the legibility from my POV.

I know I could split them off to params_from_array and params_from_hash

Please do.

but then in each of those I'd have to check the type of the params object again to be safe and it'd just be code duplication.

No. Make these methods private. You have the type assertion in the case expression, and that's the only place where the converter methods are called.

Also, please move the Builder class into a seperate file. lib/query_builder.rb is a prime candidate.
(You may need to add a require in lib/influxdb.rb as well, I'm not sure).

OK, tests are running.

:thumbsup: Thanks again for your effort!
I'm going to merge this and release a new version when I'm home. Do you mind if I squash your commits a bit?

Oh jeez please do.

The jruby tests (.8 and .9) alternately run red on the async test BTW, although restarting them sometimes helps.

(oh I see now you mentioned before the jruby tests do that)

I've squashed your commits into 8917edfaa3cbdec403c4ca56c4dc9f337ebcbb93 and added some minor refactorings.
Gem v0.3.8 is now available. :sparkles:


No longer in Hacs

Hi,
the repo is no longer available in HACS
Could you make it available again?
Thanks
Michael

Let me check that

To be honest, I think I've never officially submitted this to HACS into the default repo 😅
You can still add it as a custom repo to HACS and install it that way!
https://hacs.xyz/docs/faq/custom_repositories

Done!


context menu overlay

Description
when an icon is at the edge of the screen and when right clicking to open with, the submenu is brought up to the main menu. and sometimes the submenu did not disappear.
Steps To Reproduce
when an icon is at the edge of the screen and when right clicking to open with, the submenu is brought up to the main menu. and sometimes the submenu did not disappear.
Expected behavior
open submenu on the side
Files Version
2.3
Windows Version
w10 21h2 home
Relevant Assets/Logs
debug.log
debug_fulltrust.log

Do you have the "show overflow items in submenu" option enabled in settings > Appearance?

yes it is similar to these two cases
https://github.com/files-community/Files/issues/7346
https://github.com/files-community/Files/issues/6922

and the menus/panels become opaque

Merging with #6922


dota_unit_allow_moveto_direction has been removed.

From the patchnotes: Replaced dota_unit_allow_moveto_direction with +dota_unit_movetodirection which can now be bound to a key. I am working on a patch as we speak.

so it basically becomes "Press a key to move at direction?" or "Have this key pressed, and move commands go in a straight line"

The second. It's a mode that can be toggled. The old behavior got restored, though, so I just replaced the name of the command in the end. No hacks necessary!


Comments parsed instead of ignored

Cross reference;
https://github.com/lingua-pupuli/puppet-vscode/issues/283
Original issue from: @kaplun
What Versions are you running?
OS Version: Ubuntu 18.04
VSCode Version: 1.23.1
Puppet Extension Version: 0.10.0
Puppet Agent Version: not installed
PDK Version (if applicable):
What You Are Seeing?
An single quote within a comment is considered as the start of a new string:

What is Expected?
When no single quote is in the comment everything behaves nicely:

Thanks for the bug report @kaplun.  Also +100 for the pictures.  Makes it so much easier to debug.

I suspect the array brackets are causing the confusing for the parser.


x/vuln/client: implement sanity check on the return value of httpReadJSON

From "TODO" in https://github.com/golang/vuln/blob/41df7bda74f30fac0cab688927dab16df2f938eb/client/client.go#L314-L315:
The basic check is desirable since json.Unmarshal is too generous and types like osv.Entry allows omitempty for many fields. Assume that the client is built with an old version of golang.org/x/vuln/client, and vuln.go.dev decided to serve a slightly different osv.Entry (e.g. https://go-review.googlesource.com/c/vuln/+/424375) for some unforeseen reasons. The cache may be polluted with incomplete data.

Obsolete with re-design of client


Manage project complexity through the use of Static Analysis tooling

      Static analysis tools operate on our source code (static) vs. running program (dynamic). They help us maintain the quality of our source code by fixing formatting issues, spotting suspicious coding constructs, or alerting us to common errors.
      I plan to apply the following new features to Magic-SSG.

[x] 1. Create a CONTRIBUTING.md File.
[x]  2. Add Python Black code formatter.
[x]  3. Add a Linter - Python Flake8
[x]  4. Provide a way to integrate the above 2 tools into  my IDE
[x]  5. dd a Git Pre-Commit Hook

Merged the PR d7cb1f4,  so closed this issue.


Does not find autoclosing tag (ie: )

Soup.find('img') fails to find  tags
To Reproduce

s=Soup('

')
print(s.find('img'))
>None

Expected behavior
It should return img tag.
As a matter of fact

s=Soup(

')

works
Environment:

OS: Linux

Hey @mallegrini!
Thanks for flagging this bug. Unfortunately, this is caused by malformed HTML...
 tags aren't supposed to end with /> (source)
Meaning that gazpacho is looking for:
s=Soup('

')
print(s.find('img'))

But, like you, I have encountered some websites that contain this malformed img tag syntax (and it sucks).
I'll have to meditate on a fix... unfortunately I don't think I have a silver bullet right now :(

Fixed in Version 1.0!


prisma2 needs a more security conscious installation method for binaries

related #931 https://github.com/prisma/prisma2/issues/834
There are several concerns with the way binaries are being installed for the Query Engine, Migration Engine, and prisma-fmt utility.
First, they require use of --unsafe-perm when running npm install. In today's IT platforms, apps are explicitly disallowed the ability to switch users, or run as root.
Secondly, these are being distributed as tar.gz files from an amazon s3 bucket, instead of a legitimate package repository, which includes gpg key checking.
GPG key checking is a must. . I need to have some way to verify the binary I'm downloading is compiled from the source code I expect it to be compiled from.
Third, there needs to be a way to configure a proxy/mirror in front of these binaries, since many environment this technology will be used in is also disallowed from connecting directly to the internet, and tends to only have connections to package repositories that are lazy proxy-mirrors that can scan software as it comes in.  A good open source example of such technology, making it easy to use in testing scenarios for this feature requirement, is Sonatypes Nexus Repository Manager.
At the moment, users have no control over what prisma binaries are being added to their system, and that's a huge security risk, and in many cases makes this tool impossible to use due to platform or machine actively managed policy tools.

Also, I'd be happy to contribute a PR for this, just want to have an agreement from prisma team on what acceptance criteria for this are.
I'm super excited about trying out prisma2, but right now this issue is keeping me from even running hello-world type of exercises.

The postinstall script for prisma2 could first check for a PRISMA_QUERY_ENGINE_PATH, (same for migration and fmt), and that environment var could either be a system path or a URI.  If that exists it would either skip download (if it's a system path), or use the given URL to download.
Secondly, if the above variable isn't found, it could check for a PRISMA_BINARY_MIRROR_URL, and if that is found, then it would inject that into the host of the download binary, and the real host would be the first fragment in the uri path after the string passed by the PRISMA_BINARY_MIRROR_URL.
Finally, regardless of whether any of those variables are set (either explicitly, or as NPM_CONFIG_PRISMA_...), the postinstall script should check connection for the resolved url host, before attempting to download the binary, and should throw a hard failure causing npm install to fail and exit if the endpoint can't be reached.
An potentiakl alternative solution would be to package these binaries and host them in a package repository so the native OS package managers for linux can download them, which would provide built-in gpg key checking. Obviously that doesn't work with Windows and Mac, but for that we could likely use github releases api, which would also allow for verifying gpg signatures (See: Creating Signed GitHub Releases
I'm also working as we speak on a Pull Request that will take each tag of node and compile the binaries from scratch and then skip the post-install step of npm install prisma2. This will provide a dockerized prisma2 cli that is compiled directly from source code.
This image could easily be injected with the above described env variables as build args, and easily consume NPM_CONFIG_REGISTRY, making it compatible with VPN's that don't allow open-ended internet connections and use repository managers to manager external dependencies.

@pantharshit00 @dpetrick The current docs in docs/core/generators/photon.md tell us that in order to compile our own binaries we should check out the prisma/prisma project and checkout the alpha branch and run cargo on it.
However, the most recent commit on that branch https://github.com/prisma/prisma/commit/1d95a48eaa8bfd088c988895b9711e85216caf29 removed all the rust code, and all it said was that it was moved to another repository. the commit message didn't link an issue with more information, so I don't know where this code was moved to.
Was this code redone in go-lang and needs to be compiled with go now, or did it move to another repository? I have had any luck in finding it, so I'm stuck on the plans mentioned above until I can answer this question.
Can either you answer this one for me?

N/m, I think i found it: https://github.com/prisma/prisma-engine. That look right to you?

another thought:  I recently learned that visual studio code extensions are built with a globally installed npm package called vsce. That cli tool looks at the package.json engines field of an extension for a version dependency of the vscode platform that the extension is built for.
I wonder if we can do something similar with the prisma2 cli, specifying either individual binaries, or the engine as a whole.
example...
//photon's package.json

{
  "private": true,
  "engines": {
    "prisma" : "< 1.0.0",
    "prisma-fmt": "< 1.0.0",
    "migration-engine": "< 1.0.0",
    "introspection-engine": "< 1.0.0"
  }
}

...and I just realized i learned how vsce uses the engine field because I needed to compile the vscode-prisma extension from source :man_facepalming:

Thanks for all the research and posting your thoughts on this @LongLiveCHIEF!
I'll answer in reverse chronological order :)

Regarding pinning the version in the package.json, we're doing that here already:
https://github.com/prisma/prisma2/blob/master/cli/prisma2/package.json#L21
As of now we want all engine versions to be the same. The hash you see there will soon be replaced with a version hash instead.

Yes, prisma/prisma-engine is the correct repo for the Rust code. We should update our docs.

Moving to Github signed releases is an interesting idea. We just recently revamped our build setup using S3, so this won't happen in the coming weeks, we first need to consider all advantages/disadvantages and possible solutions using S3.

The PRISMA_QUERY_ENGINE_PATH should for sure be taken into account when downloading the binary. If it's provided, there is indeed no need to download it. No matter when we will download the binary, this is a nice optimization we can perform.

What exactly are you working on in the Pull Request? Be aware, that this involves multiple different parts of the system and first needs to be approved by at least 3 Prisma engineers involved with this part of the system. Before you implement something that we won't merge, let's first get in sync, what solution we want.
We don't want you to waste your time.
As this is very involved logic that has many angles to it (not an isolated standalone PR), we, in general, are careful with this, even if we achieved consensus, I hope you understand that.
But I'm sure we will find a great way how you can contribute to this topic and it's very exciting to see your engagement around this!  🚀

We don't want you to waste your time.

Totally understand, that's why I've been chatty with my posts.
All I've done at the moment is work on a Dockerfile that compiles the binaries from the docker-engine project in a way that accepts build args for common firewall/VPN scenarios, so I can get back to learning prisma2.
I wasn't planning on doing anything more than that until I got responses from your team on the various issues I've commented on (which you've done!) I was just digging around in the issues and repos to compare what I'm seeing in builds in different distros.
Here's a link to my branch: https://github.com/LongLiveCHIEF/prisma2/tree/add-docker-prisma2-cli-images

So I've got my dockerfile "finished", and the end result is dissapointing.
This whole thing is extremely overcomplicated. I've defined binary path via env variables, and using lift and generate commands completely ignores those paths.
At this point, I can't tell if that's a bug, or if I'm just doing something wrong, because the advanced docs for the binaries say the env vars don't change the download behaviors... just the path.
What exactly is that supposed to mean?  The docs say the env vars for that can be relative paths, but if download behavior isn't changed, then that means its going to try to fetch over https binaries that are already local?
I'm also confused about the combination of providing a path to the binaries and the name of the binary to be provided.

CLI binaries can only be overridden by a path to a custom binary. It does not alter download behavior, it just overrides the binary path provided for respective commands. This means that using known binaries is not possible.

So here again I compiled my own binaries, and overrode the path, but it is still looking for the native binary for my platform instead of the one specified by the override. (to see what I mean, clone my branch and then run make buster)
It kinda feels like my problem is I'm just naming the query engine binary incorrectly.
Am I right that I need to name it something very specific, based on the glibc-ssl installed on my platform?  I can sorta glean that from the available binaries table
Part of the "overcomplication" feeling might just be because you guys are making really quick progress, and docs might lag from changes by a week or two, and they sometimes have information that should all be in one place distributed across 3 different repos.
My end goal is to build a binary from self-defined mirrors of the upstream sources, and prevent the need to use npm i --unsafe-perms and prevent the cli from attempting to download new binaries.
I think I'm basically there, with the dockerfile i've built, aliasing the run of that image to prisma2 in my dev environment.  I think I'm just missing something simple (like needing a specific binary name) and it will work and solve the challenge of using behind a VPN (as described in #834)

The individual concerns in this issue should be split into separate issues.

I believe the 3rd issue has been resolved with supported env vars to specify a prisma download host mirror.

Uh pretty old issue that is still hanging around - let's fix that by one by one responding to the concerns raised and how we actually have the tools in place to make them not a problem:

First, they require use of --unsafe-perm when running npm install. In today's IT platforms, apps are explicitly disallowed the ability to switch users, or run as root.

Fortunately npm did away with that, and it is not required any more for Prisma in any way that I am aware of.

Secondly, these are being distributed as tar.gz files from an amazon s3 bucket, instead of a legitimate package repository, which includes gpg key checking.
GPG key checking is a must. . I need to have some way to verify the binary I'm downloading is compiled from the source code I expect it to be compiled from.

Prisma downloads its own checksum files with the engines, and then checks the archive and unzipped file against these.
Currently there is no way to provide ones own checksums, but that is just because no one has ever asked for that.
If there is some need to add more capabilities to this functionality, please create an issue.

Third, there needs to be a way to configure a proxy/mirror in front of these binaries, since many environment this technology will be used in is also disallowed from connecting directly to the internet, and tends to only have connections to package repositories that are lazy proxy-mirrors that can scan software as it comes in. A good open source example of such technology, making it easy to use in testing scenarios for this feature requirement, is Sonatypes Nexus Repository Manager.

PRISMA_ENGINES_MIRROR can set the domain at which Prisma CLI send the requests to download: https://www.prisma.io/docs/reference/api-reference/environment-variables-reference#prisma_engines_mirror This can be used with a custom repository, or something that caches what we provide under binaries.prisma.sh.

The postinstall script for prisma2 could first check for a PRISMA_QUERY_ENGINE_PATH, (same for migration and fmt), and that environment var could either be a system path or a URI. If that exists it would either skip download (if it's a system path), or use the given URL to download.

All the different engine file paths can be overridden with local environment variables, which leads to us also skip the downloads: https://www.prisma.io/docs/reference/api-reference/environment-variables-reference#custom-engine-file-locations (PRISMA_QUERY_ENGINE_LIBRARY and similar).

Secondly, if the above variable isn't found, it could check for a PRISMA_BINARY_MIRROR_URL, and if that is found, then it would inject that into the host of the download binary, and the real host would be the first fragment in the uri path after the string passed by the PRISMA_BINARY_MIRROR_URL.

Don't think our solution works exactly like that as we just replace the host with the provided mirror URL, but usually that should be fine. If not, please open a new issue.

Finally, regardless of whether any of those variables are set (either explicitly, or as NPM_CONFIG_PRISMA_...), the postinstall script should check connection for the resolved url host, before attempting to download the binary, and should throw a hard failure causing npm install to fail and exit if the endpoint can't be reached.

If the download fails, the postinstall script should fail. We don't need a separate check for that, as downloading the file and then failing has the same result as doing an additional check that fails - just without the overhead of doing an additonal check for all users (which 99.999% will not have any connection problems).

An potentiakl alternative solution would be to package these binaries and host them in a package repository so the native OS package managers for linux can download them, which would provide built-in gpg key checking. Obviously that doesn't work with Windows and Mac, but for that we could likely use github releases api, which would also allow for verifying gpg signatures (See: Creating Signed GitHub Releases

We did not get further feedback from anyone that this owuld be useful or needed. If that is indeed the case, please open a new issue and we are happy to take a look.

another thought: I recently learned that visual studio code extensions are built with a globally installed npm package called vsce. That cli tool looks at the package.json engines field of an extension for a version dependency of the vscode platform that the extension is built for.
I wonder if we can do something similar with the prisma2 cli, specifying either individual binaries, or the engine as a whole.

Each Prisma CLI version expects exactly one version of the engine files. The files themselves differ depending on the platform, OpenSSL and so on - but generally we pin the engines that are used now. There is no dynamic component in Prisma CLI that chooses between different versions. We can only pin the CLI to a new version of the engines by releasing a new version of the CLI.

This whole thing is extremely overcomplicated. I've defined binary path via env variables, and using lift and generate commands completely ignores those paths.
At this point, I can't tell if that's a bug, or if I'm just doing something wrong, because the advanced docs for the binaries say the env vars don't change the download behaviors... just the path.

That should not be the case, and if that was a bug some time ago I am pretty sure it has been fixed. If not, please open a new issue.

My end goal is to build a binary from self-defined mirrors of the upstream sources, and prevent the need to use npm i --unsafe-perms and prevent the cli from attempting to download new binaries.

I think with the information provided above this should today fully be possible.


Unification of docker-maven-plugin and fabric8-maven-plugin

The fabric8-maven-plugin is quite similar to this plugin. This plugins manage all facets for integrating Docker in the Maven build cycle. fabric8-maven-plugin does the same for Kubernetes and OpenShift and relies on docker-maven-plugin for generating the images.
There is quite some overlap in configuration (guess it's around 80%) and since I'm working with both, I believe its a good idea to combine both. Unfortunately it is not possible to share configuration between plugins. A unification will widen the scope of this combined plugin, yes, but it will make it also much more powerful. For example, if we add finally docker-compose support we could easily map docker compose to Kubernetes configuration. For people using Kubernetes with Maven this would be a big, big win.
But before I start, I'd like to collect some opinions on that. Do you see any drawbacks when combining these plugins ? Do you also think its a good idea ?
Fire at will :)

can you give an example of exactly where the overlap is and what structure you see this taking?
will it be possible to put kubernetes code in a separate jar and pull that dependency in if it's actually needed or is the plugin potentially about to grow a lot in size? :)
can we make it require java8? lambdas are nice :-)

@jgangemi I'm just in the beginning, but the overlap is probably more for the run aspect, not so much for the build aspect.
E..g. this configuration is shared for creating Kubernetes templates for Services and ReplicationControllers:

The image name (of course ;-)
Ports exposes
Environment variables when running
Labels

....  (but see also this kubernetes reference)
The other idea is to externalize the Kubernetes-config generation stuff external to the plugin, based on a Docker docker-compose.yml, which also hold this information. In order to share this the d-m-p only must be able to use a compose config, which is on the roadmap anyway.
@jstrachan is there already an issue for the docker-compose story for the kubernetes plugin ? If not I could create one.
The main use case indeed is to reuse configuration and avoid config duplication (and if possible to reuse code, too, but this is secondary. UX has the highest priority). I think we are better off with this 'external' config approach because the user audience for d-m-p is quite different than for fabric8 (although the fabric8 users will need the d-m-p, too, for building images).
/wrt to Java 8: Yeah lambda would be nice, but I really believe we should be conservative and support Java 7 for using the plugin. I've not experience yet, but is it possible to use a Java-8 based maven plugin with mvn running under Java 7 ? If so, we could consider the switch, otherwise I wouldn't do it.

@rhuss don't think there's an issue yet.
I was thinking of experimenting a little bit (e.g. taking compose, add pods somehow then extending the format to include stuff for services/SAs/ReplicationControllers/DeploymentControllers et al.
From a Java perspective I wonder if we need the option to have mvn post-process it and implement some property resolving (so the pom.xml could have some versions in it maybe?) Not sure how useful that would be yet ;)
I was wondering if we should bring this up in the kubernetes community as a whole as having a single compose like yml that can help developers define what they need in a simple concise way would be a big win really (with sensible smart defaults for most things; so generally things just work with little effort)

Hi, what's the current state of docker-maven-plugin vs fabric8-maven-plugin?
Am I supposed to use both? I know the DMP, starting with FMP. From the first look it seems I could replace DMP with FMP entirely, right?

Also what about Jenkins X. Will it be integrated with either of these? We are just building a pipeline in our company and I wonder whether it would be worth waiting a bit and starting with JenkinsX rather than rewriting later.

I  have found: https://maven.fabric8.io/#generators

Since fabric8-maven-plugin includes docker-maven-plugin the way by which images are built is identical.
Does that mean that swapping DMP with FMP is supposed to work without any further effort?

This issue is quite old and a lot of things have happened in the meantime.
The current status is:

I have no ideas about Jenkins X and as it uses an orthogonal approach there will be no overlap
d-m-p already went into f-m-p and is used for all plain Docker builds (but not s2i builds)
The generator framework is still f-m-p only but it is planned to adopt it for d-m-p, too.
The general vision is outlined in https://github.com/fabric8io/fabric8-build for a possible future of d-m-p and f-m-p

Thanks @rhuss . Still not clear to me.
Does the FMP cover all functionality of DMP?
In other words, does DMP do something that FMP can't?

Yes, there is a difference:

f-m-p only exposes the "build" goal from d-m-p, nothing about "run" or any other goals.
f-m-p adds OpenShift s2i capabilitites to the build, which are not available in d-m-p


Texture API rework

Texture APIs are very clumbsy for the moment. Here is what I suggest in term of functionnalities.
impl TextureSomething {
    // returns the number of mipmaps, minimum 1
    fn get_mipmap_levels(&self) -> u32;

    // with `new`, glium chooses which format and whether you have mipmaps
    // non-array textures
    fn new<I>(data: I) -> Self where I: Texture*dData;
    // array textures
    fn new<I>(data: Vec<I>) -> Self where I: Texture*dData;

    // builds the texture with a specific format
    // `mipmaps` set to true does not necessarly mean you'll have mipmaps, but when it's
    // set to false you are sure not to get any
    // non-array textures
    fn with_format<I>(data: I, format: RelevantFormatType, mipmaps: bool)
                      -> Result<Self, TextureCreationError> where I: Texture*dData;
    // array textures
    fn with_format<I>(data: Vec<I>, format: RelevantFormatType, mipmaps: bool)
                      -> Result<Self, TextureCreationError> where I: Texture*dData;

    // builds the texture and specify whether you want mipmaps or not
    // `mipmaps` set to true does not necessarly mean you'll have mipmaps, but when it's
    // set to false you are sure not to get any
    // non-array textures
    fn with_mipmaps<I>(data: I, mipmaps: bool)
                       -> Self where I: Texture*dData;
    // array textures
    fn with_mipmaps<I>(data: Vec<I>, mipmaps: bool)
                       -> Self where I: Texture*dData;

    // all textures
    // tries to get the "best" format, with 4 components
    // replaces `new_empty`
    fn empty(dimensions...: RelevantDimensions...)
             -> Self;

    // all textures
    fn empty_with_format(format: RelevantFormatType, dimensions...: RelevantDimensions...)
                  -> Result<Self, TextureCreationError>;

    // non-array and non-cube
    // equivalent to `main_level().read()`
    fn read<I>(&self) -> I where I: Texture*dData;
    // array and cube textures
    fn read<I>(&self) -> Vec<I> where I: Texture*dData;

    // the `write` functions regenerate all mipmaps when it is called
    fn write<I>(&mut self, Rect, data: I) -> I where I: Texture*dData;

    // non-array and non-cube textures only
    // depending on the texture, the return type is different
    // call be called with level `0`
    fn mipmap<'a>(&'a self, level: u32) -> Option<MipmapSomething<'a>>;

    // non-array and non-cube textures only
    // depending on the texture, the return type is different
    // equivalent to `mipmap(0).unwrap()` (avoids an `unwrap`)
    fn main_level<'a>(&'a self) -> MipmapSomething<'a>;

    // array textures
    fn layer<'a>(&'a self, layer: u32) -> Option<LayerSomething<'a>>;
    // cube textures
    fn layer<'a>(&'a self, layer: CubeLayer) -> LayerSomething<'a>;

    // for float-and-non-compressed 2d textures only
    fn as_surface<'a>(&'a self) -> TextureSurface<'a>;
}

impl ToSomethingAttachment for TextureSomething {}
impl UniformValue for TextureSomething {}

// the exact name of the `MipmapSomething` struct depends on the texture type
impl MipmapSomething {
    fn read<I>(&self) -> I where I: Texture*dData;

    // does *not* regenerate all mipmaps
    fn write<I>(&mut self, Rect, data: I) -> I where I: Texture*dData;
}

// the exact name of the `LayerSomething` struct depends on the texture type
impl LayerSomething {
    // gives the same result as the main texture
    fn get_mipmap_levels(&self) -> u32;

    fn as_surface<'a>(&'a self) -> TextureSurface<'a>;

    fn main_level(&self) -> MipmapSomething;
    fn mipmap(&self, level: u32) -> MipmapSomething;
}

impl ToSomethingAttachment for LayerSomething {}

enum TextureCreationError {
    UnsupportedFormat
}

For example:

Drawing on a texture of an array: texture.layer(2).unwrap().as_surface().draw(...).
Reading a specific mipmap: texture.mipmap(4).unwrap().read().

For the moment this more or less ignores cube textures, buffer textures and multisample textures.
The adjustements should be minor though.

After #475, remains:

All read and write functions
Trait implementations on layers
Writing tests


Night mode not working

I have a Google Pixel 9 Pro XL with GrapheneOS<PHONE_NUMBER> (the newest version at the time of writing) and Camera version 74  (the newest version at the time of writing). I'm very happy overall, except that the night mode does not seem to work in the Camera app. Switching to the night mode shows me a beautiful preview image, but pressing the button to actually take the photo never actually saves it. In other words, taking the photo never finishes. I see the following error when trying to open the Gallery from the Camera app:

Please wait for the image to get captured before trying to open the gallery.

Pressing the button for switching the camera causes the app to show the following error message:

Unable to capture image, error code: 3

Pressing Show details in that window shows me the following call stack:
osVersion: google/komodo/komodo:15/AP3A.241105.008/2024111800:user/release-keys
package: app.grapheneos.camera:74

x.S: Camera is closed.
	at y.r.a(Unknown Source:9)
	at x.Q.x(Unknown Source:12)
	at x.q0.B(Unknown Source:0)
	at E.h.C(Unknown Source:197)
	at E.h.A(Unknown Source:14)
	at androidx.camera.lifecycle.d.d(Unknown Source:53)
	at d1.p.t(Unknown Source:179)
	at k1.l.onClick(Unknown Source:542)
	at android.view.View.performClick(View.java:8036)
	at d1.e.onTouch(Unknown Source:129)
	at android.view.View.performOnTouchCallback(View.java:16445)
	at android.view.View.dispatchTouchEvent(View.java:16412)
	at android.view.ViewGroup.dispatchTransformedTouchEvent(ViewGroup.java:3116)
	at android.view.ViewGroup.dispatchTouchEvent(ViewGroup.java:2789)
	at android.view.ViewGroup.dispatchTransformedTouchEvent(ViewGroup.java:3144)
	at android.view.ViewGroup.dispatchTouchEvent(ViewGroup.java:2803)
	at android.view.ViewGroup.dispatchTransformedTouchEvent(ViewGroup.java:3122)
	at android.view.ViewGroup.dispatchTouchEvent(ViewGroup.java:2803)
	at android.view.ViewGroup.dispatchTransformedTouchEvent(ViewGroup.java:3122)
	at android.view.ViewGroup.dispatchTouchEvent(ViewGroup.java:2803)
	at android.view.ViewGroup.dispatchTransformedTouchEvent(ViewGroup.java:3122)
	at android.view.ViewGroup.dispatchTouchEvent(ViewGroup.java:2803)
	at android.view.ViewGroup.dispatchTransformedTouchEvent(ViewGroup.java:3122)
	at android.view.ViewGroup.dispatchTouchEvent(ViewGroup.java:2803)
	at android.view.ViewGroup.dispatchTransformedTouchEvent(ViewGroup.java:3122)
	at android.view.ViewGroup.dispatchTouchEvent(ViewGroup.java:2803)
	at android.view.ViewGroup.dispatchTransformedTouchEvent(ViewGroup.java:3122)
	at android.view.ViewGroup.dispatchTouchEvent(ViewGroup.java:2803)
	at android.view.ViewGroup.dispatchTransformedTouchEvent(ViewGroup.java:3122)
	at android.view.ViewGroup.dispatchTouchEvent(ViewGroup.java:2803)
	at com.android.internal.policy.DecorView.superDispatchTouchEvent(DecorView.java:458)
	at com.android.internal.policy.PhoneWindow.superDispatchTouchEvent(PhoneWindow.java:1981)
	at android.app.Activity.dispatchTouchEvent(Activity.java:4541)
	at e.t.dispatchTouchEvent(Unknown Source:2)
	at com.android.internal.policy.DecorView.dispatchTouchEvent(DecorView.java:416)
	at android.view.View.dispatchPointerEvent(View.java:16737)
	at android.view.ViewRootImpl$ViewPostImeInputStage.processPointerEvent(ViewRootImpl.java:7947)
	at android.view.ViewRootImpl$ViewPostImeInputStage.onProcess(ViewRootImpl.java:7710)
	at android.view.ViewRootImpl$InputStage.deliver(ViewRootImpl.java:7106)
	at android.view.ViewRootImpl$InputStage.onDeliverToNext(ViewRootImpl.java:7163)
	at android.view.ViewRootImpl$InputStage.forward(ViewRootImpl.java:7129)
	at android.view.ViewRootImpl$AsyncInputStage.forward(ViewRootImpl.java:7295)
	at android.view.ViewRootImpl$InputStage.apply(ViewRootImpl.java:7137)
	at android.view.ViewRootImpl$AsyncInputStage.apply(ViewRootImpl.java:7352)
	at android.view.ViewRootImpl$InputStage.deliver(ViewRootImpl.java:7110)
	at android.view.ViewRootImpl$InputStage.onDeliverToNext(ViewRootImpl.java:7163)
	at android.view.ViewRootImpl$InputStage.forward(ViewRootImpl.java:7129)
	at android.view.ViewRootImpl$InputStage.apply(ViewRootImpl.java:7137)
	at android.view.ViewRootImpl$InputStage.deliver(ViewRootImpl.java:7110)
	at android.view.ViewRootImpl.deliverInputEvent(ViewRootImpl.java:10214)
	at android.view.ViewRootImpl.doProcessInputEvents(ViewRootImpl.java:10165)
	at android.view.ViewRootImpl.enqueueInputEvent(ViewRootImpl.java:10134)
	at android.view.ViewRootImpl$WindowInputEventReceiver.onInputEvent(ViewRootImpl.java:10356)
	at android.view.InputEventReceiver.dispatchInputEvent(InputEventReceiver.java:295)
	at android.os.MessageQueue.nativePollOnce(Native Method)
	at android.os.MessageQueue.next(MessageQueue.java:346)
	at android.os.Looper.loopOnce(Looper.java:189)
	at android.os.Looper.loop(Looper.java:317)
	at android.app.ActivityThread.main(ActivityThread.java:8744)
	at java.lang.reflect.Method.invoke(Native Method)
	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:580)
	at com.android.internal.os.ExecInit.main(ExecInit.java:50)
	at com.android.internal.os.RuntimeInit.nativeFinishInit(Native Method)
	at com.android.internal.os.RuntimeInit.main(RuntimeInit.java:369)

Thank you in advance for your help, and thank you for the great work on GrapheneOS and the Camera app! :smile:

Hi @doomlaur,
Thanks a lot for filing this issue and your kind words and appreciation!
How long do you wait until night mode finishes taking the shot? Does the issue occur even after waiting for 10-15 seconds? (considering the worst case scenario)

Hi @MHShetty and thank you for your quick answer! :slightly_smiling_face: The issue does occur even after waiting for some time. I just tried it again and waited for 5 minutes (confirmed by the stopwatch) and it still did not finish.

Perfect, thank you so much! :smile:

@doomlaur Hi, should be fixed with the new camera version 75.

I can confirm that it works with version 75. Thank you! :smile:

I'm using the Pixel 8 and with version 75 night mode does not take any pictures anymore.
Is there something I can do about it?
Or at least provide logs here?
Thanks in advance!

Can confirm that a simple reboot made the issue I was facing go away. Thanks!


Add access modifier sort rewrite rule

I have other changes I'd like to do wrt Scalafmt, but I'm waiting with those until a certain closed source project gets opensources
https://github.com/bitcoin-s/bitcoin-s-core/pull/462#discussion_r282488952
@Christewart

Coverage increased (+0.02%) to 90.475% when pulling 524074c816fa9d49e4272e335cb2dcd4243cbcf1 on torkelrogstad:2019-05-13-scalafmt into 88122fc30ae040a2a5ac7a85bec92c321448ed74 on bitcoin-s:master.


Fix az commands used in Service Endpoint Manager

While debugging this issue, found a few bugs in the code here. This PR addresses the ones that I ran into.

@Hazhzeng Could you review this?


Removed how to use thread pool topics

Per discussion in #6180 and since PRs dotnet/samples#206 and dotnet/dotnet-api-docs#389 are merged, this PR removes "How to use a thread pool" topics from the conceptual docs and redirects to the ThreadPool.QueueUserWorkItem API page.
Contributes to #6180

@rpetrusha thank you for review; I've addressed both of your comments.


Improve AsinhNumericalEncoder

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (1e0d2b6) to head (acadd44).
Report is 4 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #353   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files          101       101
  Lines         3196      3193    -3
  Branches       280       278    -2
=========================================
- Hits          3196      3193    -3

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Update schema validation

What this PR does / why we need it:

Add not null validation for all generated columns
Add default value for INT, BOOL typed columns (except kind since ApplicationKind type must be specified on create)

Which issue(s) this PR fixes:
Fixes #
Does this PR introduce a user-facing change?:

NONE

Nice.
/lgtm

/approve


Enable Mypy

See title

Nvm it's done already. I couldnt find the mypy config, but it's in setup.cfg


Fix GLASGOW_HASKELL ifdefs

These were demanding GHC 8.20.xx, thus disabling some of the Hoed backend features in GHC 8.2 by accident.

Thanks!


setxattr timout (s3fs-0.0.9)

S3 timout on setting the attribute
s3.setxattr('the.bucket/' + path1, a='b')
attribute was set on inspection however:
s3.metadata('the.bucket/' + path1)
{'a': 'b'}
file size is 128MB
stack trace
TypeError                                 Traceback (most recent call last)
/home/brian/py_env2/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, **httplib_request_kw)
371             try:  # Python 2.7, use buffering of HTTP responses
--> 372                 httplib_response = conn.getresponse(buffering=True)
373             except TypeError:  # Python 2.6 and older
TypeError: getresponse() got an unexpected keyword argument 'buffering'
During handling of the above exception, another exception occurred:
timeout                                   Traceback (most recent call last)
/home/brian/py_env2/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, **httplib_request_kw)
373             except TypeError:  # Python 2.6 and older
--> 374                 httplib_response = conn.getresponse()
375         except (SocketTimeout, BaseSSLError, SocketError) as e:
/usr/lib/python3.5/http/client.py in getresponse(self)
1196             try:
-> 1197                 response.begin()
1198             except ConnectionError:
/usr/lib/python3.5/http/client.py in begin(self)
296         while True:
--> 297             version, status, reason = self._read_status()
298             if status != CONTINUE:
/usr/lib/python3.5/http/client.py in _read_status(self)
257     def _read_status(self):
--> 258         line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
259         if len(line) > _MAXLINE:
/usr/lib/python3.5/socket.py in readinto(self, b)
574             try:
--> 575                 return self._sock.recv_into(b)
576             except timeout:
timeout: timed out
During handling of the above exception, another exception occurred:
ReadTimeoutError                          Traceback (most recent call last)
/home/brian/py_env2/lib/python3.5/site-packages/botocore/vendored/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
369                     retries=self.max_retries,
--> 370                     timeout=timeout
371                 )
/home/brian/py_env2/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, **response_kw)
596             retries = retries.increment(method, url, error=e, _pool=self,
--> 597                                         _stacktrace=sys.exc_info()[2])
598             retries.sleep()
/home/brian/py_env2/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
244             if read is False:
--> 245                 raise six.reraise(type(error), error, _stacktrace)
246             elif read is not None:
/home/brian/py_env2/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/packages/six.py in reraise(tp, value, tb)
309             raise value.with_traceback(tb)
--> 310         raise value
311
/home/brian/py_env2/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, **response_kw)
543                                                   timeout=timeout_obj,
--> 544                                                   body=body, headers=headers)
545
/home/brian/py_env2/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, **httplib_request_kw)
375         except (SocketTimeout, BaseSSLError, SocketError) as e:
--> 376             self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
377             raise
/home/brian/py_env2/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py in _raise_timeout(self, err, url, timeout_value)
303         if isinstance(err, SocketTimeout):
--> 304             raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
305
ReadTimeoutError: HTTPConnectionPool(host='<IP_ADDRESS>', port=9000): Read timed out. (read timeout=15)
During handling of the above exception, another exception occurred:
ReadTimeout                               Traceback (most recent call last)
 in ()
----> 1 s3.setxattr('the.bucket/' + path1, a='b')
/home/brian/py_env2/lib/python3.5/site-packages/s3fs/core.py in setxattr(self, path, **kw_args)
431                             Key=key,
432                             Metadata=metadata,
--> 433                             MetadataDirective='REPLACE')
434
435         # refresh metadata
/home/brian/py_env2/lib/python3.5/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
251                     "%s() only accepts keyword arguments." % py_operation_name)
252             # The "self" in this scope is referring to the BaseClient.
--> 253             return self._make_api_call(operation_name, kwargs)
254
255         _api_call.name = str(py_operation_name)
/home/brian/py_env2/lib/python3.5/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
528         else:
529             http, parsed_response = self._endpoint.make_request(
--> 530                 operation_model, request_dict)
531
532         self.meta.events.emit(
/home/brian/py_env2/lib/python3.5/site-packages/botocore/endpoint.py in make_request(self, operation_model, request_dict)
139         logger.debug("Making request for %s (verify_ssl=%s) with params: %s",
140                      operation_model, self.verify, request_dict)
--> 141         return self._send_request(request_dict, operation_model)
142
143     def create_request(self, params, operation_model=None):
/home/brian/py_env2/lib/python3.5/site-packages/botocore/endpoint.py in _send_request(self, request_dict, operation_model)
168             request, operation_model, attempts)
169         while self._needs_retry(attempts, operation_model, request_dict,
--> 170                                 success_response, exception):
171             attempts += 1
172             # If there is a stream associated with the request, we need
/home/brian/py_env2/lib/python3.5/site-packages/botocore/endpoint.py in _needs_retry(self, attempts, operation_model, request_dict, response, caught_exception)
247             event_name, response=response, endpoint=self,
248             operation=operation_model, attempts=attempts,
--> 249             caught_exception=caught_exception, request_dict=request_dict)
250         handler_response = first_non_none_response(responses)
251         if handler_response is None:
/home/brian/py_env2/lib/python3.5/site-packages/botocore/hooks.py in emit(self, event_name, **kwargs)
225                  handlers.
226         """
--> 227         return self._emit(event_name, kwargs)
228
229     def emit_until_response(self, event_name, **kwargs):
/home/brian/py_env2/lib/python3.5/site-packages/botocore/hooks.py in _emit(self, event_name, kwargs, stop_on_response)
208         for handler in handlers_to_call:
209             logger.debug('Event %s: calling handler %s', event_name, handler)
--> 210             response = handler(**kwargs)
211             responses.append((handler, response))
212             if stop_on_response and response is not None:
/home/brian/py_env2/lib/python3.5/site-packages/botocore/retryhandler.py in call(self, attempts, response, caught_exception, **kwargs)
181
182         """
--> 183         if self._checker(attempts, response, caught_exception):
184             result = self._action(attempts=attempts)
185             logger.debug("Retry needed, action of: %s", result)
/home/brian/py_env2/lib/python3.5/site-packages/botocore/retryhandler.py in call(self, attempt_number, response, caught_exception)
249     def call(self, attempt_number, response, caught_exception):
250         should_retry = self._should_retry(attempt_number, response,
--> 251                                           caught_exception)
252         if should_retry:
253             if attempt_number >= self._max_attempts:
/home/brian/py_env2/lib/python3.5/site-packages/botocore/retryhandler.py in _should_retry(self, attempt_number, response, caught_exception)
275             # If we've exceeded the max attempts we just let the exception
276             # propogate if one has occurred.
--> 277             return self._checker(attempt_number, response, caught_exception)
278
279
/home/brian/py_env2/lib/python3.5/site-packages/botocore/retryhandler.py in call(self, attempt_number, response, caught_exception)
315         for checker in self._checkers:
316             checker_response = checker(attempt_number, response,
--> 317                                        caught_exception)
318             if checker_response:
319                 return checker_response
/home/brian/py_env2/lib/python3.5/site-packages/botocore/retryhandler.py in call(self, attempt_number, response, caught_exception)
221         elif caught_exception is not None:
222             return self._check_caught_exception(
--> 223                 attempt_number, caught_exception)
224         else:
225             raise ValueError("Both response and caught_exception are None.")
/home/brian/py_env2/lib/python3.5/site-packages/botocore/retryhandler.py in _check_caught_exception(self, attempt_number, caught_exception)
357         # the MaxAttemptsDecorator is not interested in retrying the exception
358         # then this exception just propogates out past the retry code.
--> 359         raise caught_exception
/home/brian/py_env2/lib/python3.5/site-packages/botocore/endpoint.py in _get_response(self, request, operation_model, attempts)
202                 request, verify=self.verify,
203                 stream=operation_model.has_streaming_output,
--> 204                 proxies=self.proxies, timeout=self.timeout)
205         except ConnectionError as e:
206             # For a connection error, if it looks like it's a DNS
/home/brian/py_env2/lib/python3.5/site-packages/botocore/vendored/requests/sessions.py in send(self, request, **kwargs)
571
572         # Send the request
--> 573         r = adapter.send(request, **kwargs)
574
575         # Total elapsed time of the request (approximately)
/home/brian/py_env2/lib/python3.5/site-packages/botocore/vendored/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
431                 raise SSLError(e, request=request)
432             elif isinstance(e, ReadTimeoutError):
--> 433                 raise ReadTimeout(e, request=request)
434             else:
435                 raise
ReadTimeout: HTTPConnectionPool(host='<IP_ADDRESS>', port=9000): Read timed out. (read timeout=15)

Does this issue still persist for you? Timeouts are more often caused by network problems than by the code.

I’m closing this issue because it has been inactive for a few years.
Please reopen if you still encounter this issue with the latest stable version(v0.5.2).


feat(storage): reduce size of compaction task

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
I found that there may be several large compaction task which larger than 2GB. To use parallel compact we will not use fast compact algorithm for those task larger than 2GB. When I seek the log file I find that some file in input of compaction is very large (similar to 512MB). I think that is because we do not cut file for those vnode with too much data.
A large compaction task will cost a long time to finish and it may also bring larger write amplification.
Checklist

[ ] I have written necessary rustdoc comments
[ ] I have added necessary unit tests and integration tests
[ ] I have added test labels as necessary. See details.
[ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
[ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
[ ] All checks passed in ./risedev check (or alias, ./risedev c)
[ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

[ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

[ ] My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

Main branch:

This branch:

We can see that the max size of task decrease a lot. But compaction throughput only decrease a little (Almost not different)

The changes in this PR are on the critical path:

reduce target_file_size (both base l0 and high level).
reduce the max select size of MinOverlappingPicker.
remove the vnode_changed switch condiction of SstableBuilder.

maybe we need more detailed testing and conclusions

Nexmark Benchmark like Q20 (I found that the performance of the branch test has slightly regressed, and the number of pending bytes has increased.)
Backfill test (Since we are also doing SST alignment during the backfill stage, I hope to test this path.)

Here is backfill benchmark result:
main branch:

current branch:

Current branch could reduce the size of compact task. But in this case, the compactor read throughput of current branch seems to be a little higher.