Failing with "there is no reactor running"

I'm trying to integrate this crate instead of my custom build worker pool, but I receive a panic of "there is no reactor running, must be called from the context of a Tokio 1.x runtime". I'm 100% using Tokio, since my entire app is built around it and async.
I'm not really sure what to change here. I've tried spawn, spawn_await, and complete, and they all panic.
Here's the commit where I switched to the pool:
https://github.com/moonrepo/moon/pull/518/commits/31d0a6583bd3e54863a58dfaa0789ef37dfc3297#diff-fa55688d36973c7de638f454b143b99e29a7748d86c753d291f8a2960f75e148R81

This doesn't sound like a panic thrown by this crate directly, can you provide a full backtrace using RUST_BACKTRACE=1?
One thing that sounds like it might be relevant here is that this pool doesn't spawn any threads until you submit a task to it. Depending on where the panic comes from it may be useful to use start_core_threads in case something expects threads to be running. Of course functions like spawn_await spin up threads if necessary themselves.
Another possibility is that the pool gets dropped and its channel broken before the submitted work is finished. But briefly looking at your code it seems you join all work before dropping the pool so that's probably not it. In fact I don't even think that's possible because each AsyncTask keeps a clone of the pool, so if there are live AsyncTasks the pool's channel should not get closed.
These are just shots in the dark, can't say much more without knowing where the panic comes from. To me it sounds like the submitted tasks are expected to run within the tokio runtime, or a tokio version mismatch somewhere else (this crate doesn't depend on tokio, it only uses crossbeam-channel and the futures lib).

Ok, I created a new PR with only rust changes: https://github.com/moonrepo/moon/pull/527 I also tried start_core_threads, same issue.
Here's the full backtrace.
thread 'rusty_pool_1_thread_4' panicked at 'there is no reactor running, must be called from the context of a Tokio 1.x runtime', /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.19/src/client/connect/dns.rs:121:24
stack backtrace:
   0: rust_begin_unwind
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/panicking.rs:65:14
   2: core::panicking::panic_display
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/panicking.rs:139:5
   3: tokio::runtime::context::current
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/runtime/context.rs:22:19
   4: tokio::runtime::blocking::pool::spawn_blocking
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/runtime/blocking/pool.rs:134:14
   5: tokio::task::blocking::spawn_blocking
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.1/src/task/blocking.rs:208:9
   6: <hyper::client::connect::dns::GaiResolver as tower_service::Service<hyper::client::connect::dns::Name>>::call
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.19/src/client/connect/dns.rs:121:24
   7: <S as hyper::client::connect::dns::sealed::Resolve>::resolve
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.19/src/client/connect/dns.rs:350:13
   8: hyper::client::connect::dns::resolve::{{closure}}
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.19/src/client/connect/dns.rs:360:5
   9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  10: hyper::client::connect::http::HttpConnector<R>::call_async::{{closure}}
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.19/src/client/connect/http.rs:338:17
  11: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  12: <hyper::client::connect::http::HttpConnector<R> as tower_service::Service<http::uri::Uri>>::call::{{closure}}
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.19/src/client/connect/http.rs:271:61
  13: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  14: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/future.rs:124:9
  15: <hyper::client::connect::http::HttpConnecting<R> as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.19/src/client/connect/http.rs:406:9
  16: <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.25/src/future/either.rs:89:32
  17: <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.25/src/future/either.rs:89:32
  18: <hyper_tls::client::HttpsConnector<T> as tower_service::Service<http::uri::Uri>>::call::{{closure}}
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-tls-0.5.0/src/client.rs:136:33
  19: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  20: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/future.rs:124:9
  21: <hyper_tls::client::HttpsConnecting<T> as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-tls-0.5.0/src/client.rs:162:9
  22: reqwest::connect::Connector::connect_with_maybe_proxy::{{closure}}
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.12/src/connect.rs:378:40
  23: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  24: reqwest::connect::with_timeout::{{closure}}
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.12/src/connect.rs:546:10
  25: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  26: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/future.rs:124:9
  27: <hyper::service::oneshot::Oneshot<S,Req> as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.19/src/service/oneshot.rs:60:28
  28: <F as futures_core::future::TryFuture>::try_poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.25/src/future.rs:82:9
  29: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.25/src/future/try_future/into_future.rs:34:9
  30: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.25/src/future/future/map.rs:55:37
  31: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.25/src/lib.rs:91:13
  32: <futures_util::future::try_future::MapErr<Fut,F> as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.25/src/lib.rs:91:13
  33: <F as futures_core::future::TryFuture>::try_poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.25/src/future.rs:82:9
  34: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.25/src/future/try_future/into_future.rs:34:9
  35: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.25/src/future/future/map.rs:55:37
  36: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.25/src/lib.rs:91:13
  37: <futures_util::future::try_future::MapOk<Fut,F> as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.25/src/lib.rs:91:13
  38: <F as futures_core::future::TryFuture>::try_poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.25/src/future.rs:82:9
  39: <futures_util::future::try_future::try_flatten::TryFlatten<Fut,<Fut as futures_core::future::TryFuture>::Ok> as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.25/src/future/try_future/try_flatten.rs:49:61
  40: <futures_util::future::try_future::TryFlatten<Fut1,Fut2> as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.25/src/lib.rs:91:13
  41: <futures_util::future::try_future::AndThen<Fut1,Fut2,F> as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.25/src/lib.rs:91:13
  42: <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.25/src/future/either.rs:89:32
  43: <hyper::common::lazy::Lazy<F,R> as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.19/src/common/lazy.rs:69:28
  44: futures_util::future::future::FutureExt::poll_unpin
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.25/src/future/future/mod.rs:562:9
  45: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.25/src/future/select.rs:108:35
  46: hyper::client::client::Client<C,B>::connection_for::{{closure}}
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.19/src/client/client.rs:358:48
  47: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  48: hyper::client::client::Client<C,B>::send_request::{{closure}}
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.19/src/client/client.rs:230:61
  49: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  50: hyper::client::client::Client<C,B>::retryably_send_request::{{closure}}
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.19/src/client/client.rs:200:65
  51: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  52: <hyper::client::client::ResponseFuture as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.19/src/client/client.rs:611:9
  53: <reqwest::async_impl::client::PendingRequest as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.12/src/async_impl/client.rs:1894:29
  54: <reqwest::async_impl::client::Pending as core::future::future::Future>::poll
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.12/src/async_impl/client.rs:1873:51
  55: reqwest::get::{{closure}}
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.12/src/lib.rs:263:47
  56: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  57: proto_core::resolver::load_versions_manifest::{{closure}}
             at ./crates/proto/core/src/resolver.rs:125:37
  58: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  59: proto_node::depman::resolve::<impl proto_core::resolver::Resolvable for proto_node::depman::NodeDependencyManager>::load_manifest::{{closure}}
             at ./crates/proto/node/src/depman/resolve.rs:54:9
  60: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  61: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/future.rs:124:9
  62: proto_node::depman::resolve::<impl proto_core::resolver::Resolvable for proto_node::depman::NodeDependencyManager>::resolve_version::{{closure}}
             at ./crates/proto/node/src/depman/resolve.rs:98:44
  63: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  64: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/future.rs:124:9
  65: proto_core::Tool::is_setup::{{closure}}
             at ./crates/proto/core/src/lib.rs:88:46
  66: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  67: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/future.rs:124:9
  68: <moon_node_tool::yarn_tool::YarnTool as moon_tool::tool::Tool>::setup::{{closure}}
             at ./crates/node/tool/src/yarn_tool.rs:96:52
  69: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  70: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/future.rs:124:9
  71: <moon_node_tool::node_tool::NodeTool as moon_tool::tool::Tool>::setup::{{closure}}
             at ./crates/node/tool/src/node_tool.rs:186:51
  72: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  73: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/future.rs:124:9
  74: moon_tool::manager::ToolManager<T>::setup::{{closure}}
             at ./crates/core/tool/src/manager.rs:60:57
  75: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  76: <moon_node_platform::platform::NodePlatform as moon_platform::platform::Platform>::setup_tool::{{closure}}
             at ./crates/node/platform/src/platform.rs:339:57
  77: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  78: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/future.rs:124:9
  79: moon_action_pipeline::actions::setup_tool::setup_tool::{{closure}}
             at ./crates/core/action-pipeline/src/actions/setup_tool.rs:38:9
  80: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  81: moon_action_pipeline::processor::process_action::{{closure}}
             at ./crates/core/action-pipeline/src/processor.rs:59:84
  82: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  83: moon_action_pipeline::pipeline::Pipeline::run::{{closure}}::{{closure}}
             at ./crates/core/action-pipeline/src/pipeline.rs:142:25
  84: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  85: rusty_pool::ThreadPool::try_spawn_await::{{closure}}
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/rusty_pool-0.7.0/src/lib.rs:668:32
  86: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/future/mod.rs:91:19
  87: <alloc::sync::Arc<rusty_pool::AsyncTask> as rusty_pool::Task<()>>::run
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/rusty_pool-0.7.0/src/lib.rs:120:16
  88: rusty_pool::ThreadPool::try_execute::{{closure}}
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/rusty_pool-0.7.0/src/lib.rs:488:17
  89: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ops/function.rs:251:5
  90: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/alloc/src/boxed.rs:1987:9
  91: rusty_pool::Worker::exec_task_and_notify
             at /Users/milesj/.cargo/registry/src/github.com-1ecc6299db9ec823/rusty_pool-0.7.0/src/lib.rs:1119:9
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

 ERROR  Unknown error!

It looks like rusty is running, and a lot of my code is running as well, and then it fails on hyper trying to do something (via the reqwest crate. I'm using reqwest with default features, so I'm really not sure what to dive into next.

I tried enabling the hyper runtime feature. No luck. https://docs.rs/hyper/0.14.23/hyper/#optional-features

I don't think it's possible to use async reqwest without calling it from a thread belonging to a tokio runtime.
This produces the same error you're getting:
#[tokio::main]
async fn main() {
    let pool = rusty_pool::ThreadPool::default();
    let handle: rusty_pool::JoinHandle<Result<(), ()>> = pool.spawn_await(async {
        let resp = reqwest::get("https://httpbin.org/ip")
            .await
            .map_err(|_| ())?
            .json::<HashMap<String, String>>()
            .await
            .map_err(|_| ())?;
        println!("{:#?}", resp);
        Ok(())
    });

    let _ = handle.receiver.await;
}

But setting up a runtime for the function you call inside the pool works:
#[tokio::main]
async fn main() {
    let pool = rusty_pool::ThreadPool::default();
    let handle: rusty_pool::JoinHandle<Result<(), ()>> = pool.spawn_await(async { inner() });

    let _ = handle.receiver.await;
}

#[tokio::main]
async fn inner() -> Result<(), ()> {
    let resp = reqwest::get("https://httpbin.org/ip")
        .await
        .map_err(|_| ())?
        .json::<HashMap<String, String>>()
        .await
        .map_err(|_| ())?;
    println!("{:#?}", resp);
    Ok(())
}

I'm honestly a bit surprised at this myself, I'm fairly sure I've done the former in the past myself, not sure if that's a new requirement.
The second code snippet would set up a multithreaded tokio runtime for each thread in the pool on first call, so it's not ideal. Using #[tokio::main(flavor = "current_thread")] would improve that a bit. It's probably best to set up a tokio runtime for the current rusty_pool thread using runtime::Builder::new_current_thread(). Either that, or you could pass a handle to your main tokio runtime using Handle::current() and run the reqwest in there, that would mean that reqwest's task don't run in rusty_pool but since that's mostly IO waiting for requests to complete that shouldn't require a lot of threads anyway. The latter would be easiest, the former might not be as easy as all that and might require to change the pool to optionally wrap the worker look in a block_on call.
Something like this should work though:
#[tokio::main]
async fn main() {
    let pool = rusty_pool::ThreadPool::default();
    let tokio_handle = tokio::runtime::Handle::current();
    let handle: rusty_pool::JoinHandle<Result<(), reqwest::Error>> = pool.spawn_await(async move {
        let resp = tokio_handle
            .spawn(async {
                reqwest::get("https://httpbin.org/ip")
                    .await?
                    .json::<HashMap<String, String>>()
                    .await
            })
            .await
            .unwrap()?;
        println!("{:#?}", resp);
        /* other non-io stuff */
        Ok(())
    });

    handle.await_complete().unwrap();
}

Ok that makes a lot more sense, thanks for diving into this a bit. Glad I wasn't going crazy.
Also, pulling in the current tokio runtime is super clever and totally worked. I never would of thought of trying that. https://github.com/moonrepo/moon/pull/527
Thanks a bunch, will close this.

Nice, glad it's working now.
I looked at your PR a bit and I have two notes:
I was wondering why you switched from spawn_await to complete. spawn_await would be more elegant here because complete simply blocks a worker until the future is complete whereas spawn_await works more like a proper futures executor that polls futures step by step and awakens them by resubmitting them to the pool when progress can be made / when the future is awakened.
Also, I noticed that you wrap the entire process_action call inside a tokio handler call. This works, but means that this pools is only really used to wait for tokio to do all the work. You'll get more out of this pool if you pass the Handle to the process_action function and use it there where necessary. Then you can use tokio to run reqwest's IO-bound task and use this pool for the cpu-bound or blocking work (if you have any) and get the best of both worlds.

@robinfriedli I switched to complete because visually (in the terminal) it acts more like how a user expects it to work. For example, if we pass --concurrency 1, we'd only use 1 core/thread, and the terminal will process a task 1 by 1. This is very apparent as we log a message when the task is starting and finishing.
For spawn_await, it would process every task in the batch and visually it looks like it's running X tasks in parallel as it logs a ton of starting messages. Although under the hood it's only using 1 thread, to the end user, it kind of looks like a bug and that it's not working as intended.
As for the process_action handle, the entirety of that function and all of its branching logic are async and require Tokio. I'm not sure how I could break it up, but I'll give it a shot.

@milesj Ah I see. If it's just tokio stuff and you don't do any cpu intensive or blocking work in there it doesn't matter much, you'd just get some more effective use out of the threads you spawn. If it could be relevant for your users that tasks are executed sequentially then using complete does make sense.


Updated integration test to align to new tenant setup

What about test_workflow_client.test_get_all_credentials_with_limit_and_offset that was failing because there's only one credential in the tenant?
Is this for FT-825?

I’m not aware of the failing test and FT-825 ticket: test_get_all_credentials_with_limit_and_offset 😅
I was only fixing the integration tests that failed yesterday during the release. I think we can remove this test as it seems to be a duplicate (cc: @vaibhavatlan what do you think?). Since there’s already an integration test (test_get_all_credentials_with_filter_limit_offset) that covers the same scenario with snowflake credentials].

What about test_workflow_client.test_get_all_credentials_with_limit_and_offset that was failing because there's only one credential in the tenant?
Is this for FT-825?

I’m not aware of the failing test and FT-825 ticket: test_get_all_credentials_with_limit_and_offset 😅 I was only fixing the integration tests that failed yesterday during the release. I think we can remove this test as it seems to be a duplicate (cc: @vaibhavatlan what do you think?). Since there’s already an integration test (test_get_all_credentials_with_filter_limit_offset) that covers the same scenario with snowflake credentials].

Yes, you’re absolutely right. test_get_all_credentials_with_limit_and_offset addresses the same scenario as test_get_all_credentials_with_filter_limit_offset for Snowflake credentials. The failure occurs because, as Ernest pointed out, the new tenant fieldtec33 only has one credential available. Since the integration test test_get_all_credentials_with_filter_limit_offset already covers this case, we can safely remove the failing test.


Add internal useForm hook

Summary

Internal change, should not affect the input/output of any of these components

As of now, every component handles forms in its own way. This means a lot of duplication of code.
This PR introduces a custom hook useForm, which extracts common form logic to a common place. This should remove the need of repeating multiple states per component (valid, errors, data). It also handles events and validation.
Highlights of useForm
Supported options:

schema: Every form has a schema (that can change via setSchema, if needed). This is used to keep track of the fields of a form, and to prefill other state objects.
defaultData: Used to prefill data in the state.
rules: Set of rules used in the Validator class.
formatters: Set of functions that will run for each field, formatting its value before validating it and storing it in state.

State/methods:

data: Holds the actual values of each form field defined in the schema.
valid/errors: Hold the validation state of each form field in the schema.
handleChangeFor(fieldName): Will run validations and set the data for a specific form field.
triggerValidation: Revalidates every form field.

Example
    const { handleChangeFor, triggerValidation, data, valid, errors, isValid } = useForm({
        schema: ['field1', 'field2'],
        defaultData: { field1: 'Field 1' },
        rules: { field1: (value) => { /* ... */ } },
    });

Roadmap

[x] IssuerLists migrated to useForm
[x] Blik migrated to useForm
[x] Boleto Bancario migrated to useForm
[x] Dragonpay migrated to useForm

Does useForm always generate errors on 'blur'? Does this mean that once a field is clicked into - clicking out of it will always generate an error, even if the field is empty?

Does useForm always generate errors on 'blur'? Does this mean that once a field is clicked into - clicking out of it will always generate an error, even if the field is empty?

Hey @sponglord, thanks for your comments.

Does useForm always generate errors on 'blur'? Does this mean that once a field is clicked into - clicking out of it will always generate an error, even if the field is empty?

The useForm hook doesn't enforce any of this, it all depends on how changes are emitted from the component using it (handleChangeFor), and the validation rules for each of the modes (input/blur). In the components we've converted on this PR, the blur validation will happen only when the field has been edited.

I see (and like) that you can format a value as you validate it (in processField) - but there was that addition that the Dev from Singapore made about whether return an errorMessge with the validation (I'm not sure what the use case was, exactly, so I don't know if this is something we might still need? Just wonderin'...)

That's a very good point. For now, the errors state is just a simple object with boolean values, but this could be extended further as we convert the relevant payment methods (the ones in #202). Will keep this in mind 👍

Hey @sponglord, thanks for your comments.

Does useForm always generate errors on 'blur'? Does this mean that once a field is clicked into - clicking out of it will always generate an error, even if the field is empty?

The useForm hook doesn't enforce any of this, it all depends on how changes are emitted from the component using it (handleChangeFor), and the validation rules for each of the modes (input/blur). In the components we've converted on this PR, the blur validation will happen only when the field has been edited.

I see (and like) that you can format a value as you validate it (in processField) - but there was that addition that the Dev from Singapore made about whether return an errorMessge with the validation (I'm not sure what the use case was, exactly, so I don't know if this is something we might still need? Just wonderin'...)

That's a very good point. For now, the errors state is just a simple object with boolean values, but this could be extended further as we convert the relevant payment methods (the ones in #202). Will keep this in mind 👍

From this code in useForm. updateFieldData :
setErrors(prevErrors => ({ ...prevErrors, [key]: mode === 'blur' && !isFieldValid }));

It looks like errors are only set on "blur".
Am I reading this correctly? What about if you want to generate on "input" for example when the shopper exceeds the expected number of characters in a field

From this code in useForm. updateFieldData :
setErrors(prevErrors => ({ ...prevErrors, [key]: mode === 'blur' && !isFieldValid }));

It looks like errors are only set on "blur".
Am I reading this correctly? What about if you want to generate on "input" for example when the shopper exceeds the expected number of characters in a field


Automated cherry pick of #11564: feat(host): add openEuler rootFs driver

Cherry pick of #11564 on release/3.6.
#11564: feat(host): add openEuler rootFs driver

/lgtm

/approve


StorageClass with the maximum IOPS and throughput parameters cannot be used normally

Events:
  Type     Reason                  Age                  From                     Message
  ----     ------                  ----                 ----                     -------
  Normal   Scheduled               19m                  hwameistor-scheduler     Successfully assigned default/nginx-local-storage-lvm-7cbc7fd9db-xdbjt to k8s-node1
  Normal   SuccessfulAttachVolume  19m                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-67c11c4b-0e87-4541-b340-8085956053f9"
  Warning  FailedMount             17m                  kubelet                  MountVolume.SetUp failed for volume "pvc-67c11c4b-0e87-4541-b340-8085956053f9" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Warning  FailedMount             3m35s (x7 over 17m)  kubelet                  Unable to attach or mount volumes: unmounted volumes=[html-root], unattached volumes=[html-root kube-api-access-g5bcv]: timed out waiting for the condition
  Warning  FailedMount             78s                  kubelet                  Unable to attach or mount volumes: unmounted volumes=[html-root], unattached volumes=[kube-api-access-g5bcv html-root]: timed out waiting for the condition
  Warning  FailedMount             47s (x15 over 17m)   kubelet                  MountVolume.SetUp failed for volume "pvc-67c11c4b-0e87-4541-b340-8085956053f9" : rpc error: code = Unknown desc = format of disk "/dev/LocalStorage_PoolHDD/pvc-67c11c4b-0e87-4541-b340-8085956053f9" failed: type:("xfs") target:("/var/lib/kubelet/pods/1c450e2f-faa9-49e9-a2e0-27772d1dd96f/volumes/kubernetes.io~csi/pvc-67c11c4b-0e87-4541-b340-8085956053f9/mount") options:("defaults") errcode:(exit status 1) output:(mkfs.xfs: cannot open /dev/LocalStorage_PoolHDD/pvc-67c11c4b-0e87-4541-b340-8085956053f9: Device or resource busy

More info:
previous  mkfs.xfs call hangs.
[root@k8s-node1 673]# lsof /dev/LocalStorage_PoolHDD/pvc-76acaab6-749a-4dba-8ac0-2179194cb3ba
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
mkfs.xfs 673 root    4u   BLK  253,4      0t0 198967 /dev/LocalStorage_PoolHDD/../dm-4
[root@k8s-node1 673]# ps -ef | grep 673
root       673 21446  0 11:31 ?        00:00:00 mkfs.xfs /dev/LocalStorage_PoolHDD/pvc-76acaab6-749a-4dba-8ac0-2179194cb3ba
[root@k8s-node1 673]# pwd
/proc/673/task/673
[root@k8s-node1 673]# cat stack
[<ffffffff84b5db4c>] blkdev_issue_discard+0x2ac/0x2d0
[<ffffffff84b67375>] blk_ioctl_discard+0xe5/0x130
[<ffffffff84b67ec2>] blkdev_ioctl+0x632/0xa20
[<ffffffff84a8e7f1>] block_ioctl+0x41/0x50
[<ffffffff84a632e0>] do_vfs_ioctl+0x3a0/0x5b0
[<ffffffff84a63591>] SyS_ioctl+0xa1/0xc0
[<ffffffff84f93f92>] system_call_fastpath+0x25/0x2a
[<ffffffffffffffff>] 0xffffffffffffffff

It's a kernel bug in 3.10.0-1160.el7.x86_64, and I can reproduce it by the following steps:
# Step 1: Create lv named "demo1" and "demo2" from the LocalStorage_PoolHDD vg
[root@k8s-node1 ~]# lvcreate -L 1G -n demo1 LocalStorage_PoolHDD
  Logical volume "demo1" created.
[root@k8s-node1 ~]# lvcreate -L 1G -n demo2 LocalStorage_PoolHDD
  Logical volume "demo2" created.

# Step 2: Look up the device major:minor of the demo2 lv
[root@k8s-node1 ~]# lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0   160G  0 disk
├─sda1                         8:1    0     1G  0 part /boot
└─sda2                         8:2    0   159G  0 part
  ├─centos-root              253:0    0    50G  0 lvm  /
  ├─centos-swap              253:1    0   7.9G  0 lvm
  └─centos-home              253:2    0 101.1G  0 lvm  /home
sdb                            8:16   0   200G  0 disk
├─LocalStorage_PoolHDD-pvc--67c11c4b--0e87--4541--b340--8085956053f9
                             253:3    0     1G  0 lvm
├─LocalStorage_PoolHDD-pvc--76acaab6--749a--4dba--8ac0--2179194cb3ba
                             253:4    0     1G  0 lvm
├─LocalStorage_PoolHDD-pvc--281344a6--471c--4fca--b25c--4f65362b3f24
                             253:5    0     1G  0 lvm  /var/lib/kubelet/pods/0773d06b-f40c-4516-ae90-42857a8f496c/volumes/kubernetes.io~
├─LocalStorage_PoolHDD-demo1 253:6    0     1G  0 lvm
└─LocalStorage_PoolHDD-demo2 253:7    0     1G  0 lvm

# Step 3: write blkio settings for demo2
echo "253:7   1048576" > /sys/fs/cgroup/blkio/blkio.throttle.write_bps_device

# Step 4: make filesystem on demo1
[root@k8s-node1 ~]# mkfs.xfs /dev/mapper/LocalStorage_PoolHDD-demo1
meta-data=/dev/LocalStorage_PoolHDD/demo1 isize=512    agcount=4, agsize=65536 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

# Step 5: make filesystem on demo2 and you will see mkfs.xfs hangs
[root@k8s-node1 ~]# mkfs.xfs /dev/mapper/LocalStorage_PoolHDD-demo2

# Step 6: open another terminal and run `lsof` to see the process of mkfs.xfs
[root@k8s-node1 ~]# lsof /dev/mapper/LocalStorage_PoolHDD-demo2
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
mkfs.xfs 12171 root    4u   BLK  253,7      0t0 529067 /dev/mapper/../dm-7

# Step 7: See the stack and wchan of the mkfs.xfs process
[root@k8s-node1 ~]# cat /proc/12171/stack
[<ffffffff84b5db4c>] blkdev_issue_discard+0x2ac/0x2d0
[<ffffffff84b67375>] blk_ioctl_discard+0xe5/0x130
[<ffffffff84b67ec2>] blkdev_ioctl+0x632/0xa20
[<ffffffff84a8e7f1>] block_ioctl+0x41/0x50
[<ffffffff84a632e0>] do_vfs_ioctl+0x3a0/0x5b0
[<ffffffff84a63591>] SyS_ioctl+0xa1/0xc0
[<ffffffff84f93f92>] system_call_fastpath+0x25/0x2a
[<ffffffffffffffff>] 0xffffffffffffffff

[root@k8s-node1 ~]# cat /proc/12171/wchan
blkdev_issue_discard

In the above steps, the mkfs.xfs process hangs in blkdev_issue_discard function, and the wchan is blkdev_issue_discard. It seems that the blkdev_issue_discard function hangs. However, it works well in 3.10.0-1160.90.1.el7.x86_64 when I repeat the above steps.
According to the gathered information, I think it's a kernel bug in 3.10.0-1160.el7.x86_64 and it has been fixed in 3.10.0-1160.90.1.el7.x86_64. And I also find a workaround for this issue, but is requires to modify the NodePublishVolume function in pkg/local-storage/member/csi/node.go file of the hwameistor source code.
To validate the workaround, I open another terminal on the same node as before, and repeat the above steps like demo2, but it works well this time.
[root@k8s-node1 ~]# lvcreate -L 1G -n demo3 LocalStorage_PoolHDD
  Logical volume "demo3" created.
[root@k8s-node1 ~]# lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0   160G  0 disk
├─sda1                         8:1    0     1G  0 part /boot
└─sda2                         8:2    0   159G  0 part
  ├─centos-root              253:0    0    50G  0 lvm  /
  ├─centos-swap              253:1    0   7.9G  0 lvm
  └─centos-home              253:2    0 101.1G  0 lvm  /home
sdb                            8:16   0   200G  0 disk
├─LocalStorage_PoolHDD-pvc--67c11c4b--0e87--4541--b340--8085956053f9
                             253:3    0     1G  0 lvm
├─LocalStorage_PoolHDD-pvc--76acaab6--749a--4dba--8ac0--2179194cb3ba
                             253:4    0     1G  0 lvm
├─LocalStorage_PoolHDD-pvc--281344a6--471c--4fca--b25c--4f65362b3f24
                             253:5    0     1G  0 lvm  /var/lib/kubelet/pods/0773d06b-f40c-4516-ae90-42857a8f496c/volumes/kubernetes.io~
├─LocalStorage_PoolHDD-demo  253:6    0     1G  0 lvm
├─LocalStorage_PoolHDD-demo2 253:7    0     1G  0 lvm
└─LocalStorage_PoolHDD-demo3 253:8    0     1G  0 lvm

[root@k8s-node1 ~]# echo "253:8 1048576" >> /sys/fs/cgroup/blkio/blkio.throttle.write_bps_device

[root@k8s-node1 ~]# mkfs.xfs -K /dev/LocalStorage_PoolHDD/demo3
meta-data=/dev/LocalStorage_PoolHDD/demo3 isize=512    agcount=4, agsize=65536 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

The major difference is that I add the -K option for mkfs.xfs command, and it works well.
I'm not sure that it is enough reasonable if we add the -K flag for mkfs.xfs command when the FormatAndMount function is called by the NodePublishVolume RPC.
By the way, We need a document to clarify which range of kernel version is supported by the hwameistor project. Unfortunately, I look up the kernel document of cgroups v1, but not find a recommended kernel version when the blkio cgroup is used.
If someone has any idea about this issue, please let me know. Thanks.
cc @FloatXD @SSmallMonster @sun7927

I'm not sure that it is enough reasonable if we add the -K flag for mkfs.xfs command when the FormatAndMount function is called by the NodePublishVolume RPC.

-K flag is used to discard blocks when format FileSystem. This flag may not need to change in FormatAndMount  function.
If we can determine that kernel version 3.10.0-1160.90.1 is working properly, then we can indicate in the documentation that the recommended kernel version is > 3.10.0-1160.90.1.
@carlory

The root cause of the mkfs process hangs is that the blkdev_issue_discard is affected by the blkio subsystem. we set 1Mi/s to /sys/fs/cgroup/blkio/blkio.throttle.write_bps_device, so blkdev_issue_discard is limited.

We need consider to re-desgin the feature.

2 possible solutions as below:

Format the volume before apply qos policy
Format the volume without discard.


Added CONTRIBUTING.md

#27

Added installation instructions for developers. As mentioned In the documentation link (https://surfactant--63.org.readthedocs.build/en/63/contribute.html#installation-for-developers).

Thanks for the PR! It looks like someone else also opened a PR yesterday adding a contributing file, though it also seems to be just a direct copy of a template and needs a number of changes. I would like to give the other person a day or so to see if they make requested changes.
Otherwise, my thinking is this one looks like it covers the essentials and has been tailored to this project — maybe some info could be added to the section on submitting a bug report to mention https://github.com/LLNL/Surfactant/security for reporting security issues/vulnerabilities, and a link to the code of conduct, then it will be ready to merge?

I would work on that and submit another other PR.

I have added the CODE OF CONDUCT link of contributor covenant which has widely used and adopted by many firms.
(https://www.contributor-covenant.org).
Also mentioned the link to report the security issues and vulnerabilities in the CONTRIBUTING.md file.

Can you make it Hackoctober fest label?

I would work on all those and submit another PR.
Can you make it Hackoctober label?

It should be possible to go to https://github.com/LLNL/Surfactant/pull/63/files and click "add suggestion to batch" to commit them as a group -- probably the easiest way to make the changes.
The repository has the hacktoberfest label, so when the PR is approved/merged you'll get credit (may take a little bit for the hacktoberfest website to update its status).

If needed yes -- from their FAQ (and previous years experience with Hacktoberfest) any of the three conditions are sufficient for it to count ("Your PR/MRs must be merged, have the "hacktoberfest-accepted", or have an overall approving review.`)

Done with the changes.Can you review it and merge it, if there are any more changes to be made I would work on that.

Regarding other changes, currently the CODE_OF_CONDUCT.md file we have in the repository (https://github.com/LLNL/Surfactant/blob/main/.github/CODE_OF_CONDUCT.md) is for version 2.0 of the contributor covenant -- that could get updated to the 2.1 version.

Ok I will create an issue and will make the changes.I guess there is no need for hackoctober  label because yesterday for my pull request which is merged it was shown in my profile in hackoctober.


chore: bdd-spring-1582849405 to 0.0.1

chore: Promote bdd-spring-1582849405 to version 0.0.1

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


vector-<IP_ADDRESS> release violates PVP

vector-<IP_ADDRESS> added a new export
@@@ Data.Vector.FFT
++ crossCorrelation :: Vector (Complex Double) -> Vector (Complex Double) -> Vector (Complex Double)

Yet, PVP requires:

Non-breaking change. Otherwise, if only new bindings, types, classes, non-orphan instances or modules (but see below) were added to the interface, then A.B MAY remain the same but the new C MUST be greater than the old C.

Please be careful with versioning. I don't think you need to do anything now. Feel free to close this issue when you see it.

Thanks for letting me know!


runs.2017_05_01_pages table in BigQuery is 0 bytes

https://bigquery.cloud.google.com/table/httparchive:runs.2017_05_01_pages?tab=details
The rest of the 5/1 runs tables are missing (pages_mobile, requests, requests_mobile).

Do you know if it imports from the csv or from the sql dump?  If it imports from the csv that would explain it since they were all 0 length.

AFAICT it syncs the CSV with BQ: https://github.com/HTTPArchive/bigquery/blob/master/sync_csv.sh

So this and #105 have the same root cause, but to resolve this issue we also need to manually rerun sync_csv.

I was able to restore the desktop pages table.
Getting errors related to the requests table:
$ ./sync_csv.sh May_1_2017
Processing May_1_2017, mobile: 0, archive: May_1_2017
Downloading data for May_1_2017
Pages data already converted, skipping.
Request data already converted, skipping.
Syncing data to Google Storage
Skipping existing item: gs://httparchive/May_1_2017/pages.csv.gz
Skipping existing item: gs://httparchive/May_1_2017/requests_aa.gz
Skipping existing item: gs://httparchive/May_1_2017/requests_ab.gz
Skipping existing item: gs://httparchive/May_1_2017/requests_ac.gz

==> NOTE: You are performing a sequence of gsutil operations that may
run significantly faster if you instead use gsutil -m -o ... Please
see the -m section under "gsutil help options" for further information
about when gsutil -m can be advantageous.

Skipping existing item: gs://httparchive/May_1_2017/requests_ad.gz
Skipping existing item: gs://httparchive/May_1_2017/requests_ae.gz
Skipping existing item: gs://httparchive/May_1_2017/requests_af.gz
Skipping existing item: gs://httparchive/May_1_2017/requests_ag.gz
Submitting new pages import runs.2017_05_01_pages to BigQuery
Successfully started load httparchive:bqjob_r3456bf622d766e07_0000015c3200c898_1
Submitting new requests import runs.2017_05_01_requests to BigQuery: requests_ag.gz
Waiting on bqjob_r712c4447b3bd521e_0000015c3200cc03_1 ... (0s) Current status: DONE
BigQuery error in load operation: Error processing job
'httparchive:bqjob_r712c4447b3bd521e_0000015c3200cc03_1': Too many errors encountered.
Failure details:
- /gzip/subrange//bigstore/httparchive/May_1_2017/requests_ag.gz: CSV
table references column position 37, but line starting at
position:25771 contains only 37 columns.
Submitting append requests import runs.2017_05_01_requests to BigQuery: requests_af.gz
BigQuery error in load operation: Error processing job
'httparchive:bqjob_r620147c7d3001225_0000015c3200d545_1': No schema specified on job or table.
Submitting append requests import runs.2017_05_01_requests to BigQuery: requests_ae.gz
BigQuery error in load operation: Error processing job
'httparchive:bqjob_r3036f05044055d45_0000015c3200d7c9_1': No schema specified on job or table.
Submitting append requests import runs.2017_05_01_requests to BigQuery: requests_ad.gz
BigQuery error in load operation: Error processing job
'httparchive:bqjob_r5e19510967299ee6_0000015c3200da3a_1': No schema specified on job or table.
Submitting append requests import runs.2017_05_01_requests to BigQuery: requests_ac.gz
BigQuery error in load operation: Error processing job
'httparchive:bqjob_r47d531755a51f544_0000015c3200dcfd_1': No schema specified on job or table.
Submitting append requests import runs.2017_05_01_requests to BigQuery: requests_ab.gz
BigQuery error in load operation: Error processing job
'httparchive:bqjob_rfa53cd086e10e39_0000015c3200df7c_1': No schema specified on job or table.
Submitting append requests import runs.2017_05_01_requests to BigQuery: requests_aa.gz
BigQuery error in load operation: Error processing job
'httparchive:bqjob_r6a4c1f96f2f86a02_0000015c3200e1fe_1': No schema specified on job or table.

Failure details:
- /gzip/subrange//bigstore/httparchive/May_1_2017/requests_aa.gz: CSV
table references column position 53, but line starting at
position:121444 contains only 53 columns.

Looking on, before, and after line 25771 in requests_ag.csv, I don't see anything out of the ordinary.

As a rule of thumb, it's almost always something to do with escaping.. which field is ~37 in above string?

"max-age=2592000, public"
The comma is in quotes, but that could have something to do with it.

Another clue is that I truncated the file at line 25770, 1 before the supposed error.
$ wc -l archive/processed/May_1_2017/requests_aga.csv
25770 archive/processed/May_1_2017/requests_aga.csv

This shows that it really only has 25770 lines.
I gzipped and copied over to GS then attempted to load it into BQ.
$ bq load runs.2017_05_01_requests gs://httparchive/May_1_2017/requests_aga.csv.gz code/schema/requests.json
Waiting on bqjob_r19519200981b9369_0000015c35d93739_1 ... (1s) Current status: DONE
BigQuery error in load operation: Error processing job 'httparchive:bqjob_r19519200981b9369_0000015c35d93739_1': Too many errors encountered.
Failure details:
- /gzip/subrange//bigstore/httparchive/May_1_2017/requests_aga.csv.gz
: CSV table references column position 37, but line starting at
position:25771 contains only 37 columns.

Got the same error at line 25771, which shouldn't even exist. So maybe there's an unescaped newline somewhere, but not sure how wc -l wouldn't catch that.
FWIW also tried the same with a raw CSV (no gzip) and got the same result.

My approach for debugging these kinds of errors before is to bisect the datafile until you can isolate the error..

Binary searched for the line causing the error. Found this on line 33:
"348612505","76995462","10","472","GET","https://code.jquery.com/jquery-1.12.4.min.js","https://code.jquery.com/jquery-1.12.4.min.js","","0","0","ori","331","","0","","200","","375","","0","0","315360000","application/javascript; charset=utf-8","status = 200, x-cache = HIT, access-control-allow-origin = *","*/*","","gzip, deflate, sdch, br","en-US,en;q=0.8","","","","","https://yts2.com/","Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 PTST/1","","","max-age=315360000\
public","","gzip","","","","application/javascript; charset=utf-8","Thu, 11 May 2017 18:58:57 GMT","W/""573f46fe-17b8b""","Thu, 31 Dec 2037 23:55:55 GMT","","Fri, 20 May 2016 17:18:54 GMT","","","NetDNA-cache/2.2","","Accept-Encoding","","","NetDNA","0","487","script","js",""

So it was exactly column 37 (cache-control) as described.
$ curl -I https://code.jquery.com/jquery-1.12.4.min.js
HTTP/1.1 200 OK
Date: Tue, 23 May 2017 15:35:27 GMT
Content-Type: application/javascript; charset=utf-8
Content-Length: 97163
Connection: keep-alive
Last-Modified: Fri, 20 May 2016 17:18:54 GMT
Vary: Accept-Encoding
ETag: "573f46fe-17b8b"
Expires: Thu, 31 Dec 2037 23:55:55 GMT
Cache-Control: max-age=315360000
Cache-Control: public
Access-Control-Allow-Origin: *
Server: NetDNA-cache/2.2
X-Cache: HIT
Accept-Ranges: bytes

Maybe because there are two Cache-Control response headers, there was a bug joining the two values together. Rather than joining with a comma like Cache-Control: max-age=2592000, public, they were joined with a newline.
So where does that response header joining happen?

Check the mysql table.. My guess is that the newline is in the value --> maybe Linux agents are doing something funny? /cc @pmeenan

mysql> select resp_cache_control from requests where requestid='348612505';
+--------------------------+
| resp_cache_control       |
+--------------------------+
| max-age=315360000
public |
+--------------------------+
1 row in set (0.00 sec)

I'm passing the value straight through from dev tools from the looks of it but my guess is that Rick is right and for multiple instances of a header they are putting it as multiple lines in the value (note, a value with a comma should not have the same problem, this is probably multiple actual headers).
Any preference for what to do?  I can pick the first or join them with a safe character.

btw, Cache-Control: max-age=2592000, public is a valid header by itself and "joining" isn't usually done.

Here's the corresponding request on WPT: http://httparchive.webpagetest.org/result/170501_30_EPBH/1/details/#step1_request19
It looks ok in the UI cache-control: max-age=315360000 public but JSON formatted the value is "cache-control: max-age=315360000\npublic".
I queried previous crawls for this URL to see how we're storing the header value:
SELECT
  resp_cache_control,
  COUNT(0)
FROM
  [httparchive:runs.2017_04_15_requests]
WHERE
  url='https://code.jquery.com/jquery-1.12.4.min.js'
GROUP BY
  resp_cache_control

max-age=315360000 public	716
max-age=315360000		10

Interestingly, if I go back one more crawl to 04_01, all 679 instances of the request have a value of max-age=315360000 public. So something is changing where the "public" part is dropped.
Should we be consistent and join on a space or be valid and join on a comma?

Just pushed a change to merge them with commas (for the various headers that are pulled out into individual fields for reporting).  It won't help until the June 1 crawl though since it's an agent change.
Can we do a string substitution for the cache-control, expires, content-encoding, content-type and content-length columns (if we have them) to strip out any newlines and re-export the CSV?  Maybe wait an hour until the agents have picked up the fix.

I updated the requests and requestsmobile tables to replace instances of \n with ,  for the columns Pat mentioned above.
FYI here's a breakdown of how many requests contained newlines in the 5/1 crawl:
			Desktop	Mobile
resp_cache_control	128610	113247
resp_expires		3570	3267
resp_content_encoding	0	0
resp_content_type	386	292
resp_content_length	2	2

After sanitizing the tables, I reran the bulktest/update.php script for desktop and mobile. This takes about 2 hours each :(. With the new CSV dumps, I reran update_csv.sh on desktop. Unfortunately, it's still failing - this time on a different column: respOtherHeaders. For example:
mysql> select respOtherHeaders from requests where requestid=348743067;
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| respOtherHeaders                                                                                |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| status = 200
200 OK, x-response-time = 5, x-xss-protection = 1; mode=block, x-transaction = 000985ee00592a83, x-content-type-options = nosniff, x-twitter-response-tags = BouncerCompliant, x-connection-hash = 0805f2cea60e9841cb3f271bb48d039e, strict-transport-security = max-age=631138519, x-frame-options = SAMEORIGIN, p3p = CP="CAO DSP LAW CURa ADMa DEVa TAIa PSAa PSDa IVAa IVDa OUR BUS IND UNI COM NAV INT" |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

So now I'm doing the whole thing over again to sanitize respOtherHeaders. Hopefully that's the last one.

I'm also looking into the sed expressions in sync_csv to do this replacement automatically. However, since we're dealing with newline characters, I'm not sure if sed is the right tool.
I did come up with a proof of concept for an expression that can do what we want:
printf '"1"\n"2\n3"\n"4"\n' | sed -e ':a;N;$!ba;s/\([^"]\)\n/\1, /g'
"1"
"2, 3"
"4"

but I'm pretty sure that this loads the entire CSV into memory first, which is a no-go given that our files are many GB.
Curious if anyone more familiar with this stuff than me has other ideas besides replacing newlines directly in the DB.

sed doesn't load files into memory so is well suited to working with very large files, assuming you can get the hang of the of the syntax!

@Themanwithoutaplan that's normally true but AFAIK in this case the entire file is loaded so that we can operate on the newline delimiters. I'll run a test.

Ran this script on a 2.9GB CSV:
sed -e ':a;N;$!ba;s/\([^"]\)\n/\1, /g' requests_ag.csv > /dev/null

Looking at top in another shell clearly showed free memory drop by a comparable amount.
My understanding is that $! in the expression reads until EOF, so all newlines would be included. Otherwise sed usually just reads until EOL, excluding the newlines.
That said, I wrote a Python script that just reads line by line including the newline char and doesn't load the entire file into memory. So if needed we could just execute that script from sync_csv.sh.

Ok now we're getting somewhere. Here's the Python script:
import fileinput
for line in fileinput.input():
	if line.endswith('\\\n'):
		line = line[:-2] + ','
	print line,

Used as such:
~/archive/processed/May_1_2017$ gunzip -c requests_aa.gz | python fixcsv.py | gzip > requests_aa.gz

Or something. I did that in more steps than shown above. Also for each requests_[aa-ag].gz.
Then reuploaded to gs://httparchive and resynced with BQ. 8M rows have been successfully imported in the first file, so I don't anticipate any surprises with the remaining 6 files. See the table status. Each one takes about 15 minutes to import, so give it 90 minutes until it should be back to normal.

Now that the 5/15 crawl is complete I need to do the same thing to fix the requests table in BQ. Hopefully Pat's fix on the WPT side halfway through the crawl means this is the last time it needs to be done.

Requests tables for 5/01 mobile, 5/15 mobile, and 5/15 desktop have been regenerated. BQ should be all caught up now.

One nice thing I found is that gsutil and bq commands support wildcards. For example:
bq load runs.2017_05_01_requests_mobile gs://httparchive/mobile_May_1_2017/requests_* ../../../code/schema/requests.json

This loads all of the requests for the crawl into BQ from storage. sync_csv.sh currently runs bq load for each request file and has logic to only include the schema on the first run.
Not only does the wildcard make the command much simpler but it's also much faster than if run one at a time.

Are you sure that results in append? I remember earlier I had to submit separate jobs because we needed to run at least one with the schema, and then we could kick-off all the rest in parallel.

Yeah it looks ok:
		04/15		05/01
Table Size	41.1 GB		42.3 GB
Number of Rows	44,489,371	45,257,278

Table size/rows are comparable before and after wildcarding.

cool!


Update composer.json

That's actually very wrong - the whole sense of this plugin is to run it on Composer 1 and you want to make it impossible.
Anyways: https://github.com/hirak/prestissimo/pull/223#issuecomment-717711913


make UI usable without mouse (mostly for gamepad support)

It just would need to be fully usable with the standard UI controls. Then it should be usable with a gamepad as well.
The dragging of the items might be the biggest challenge. I guess while dragging it should probably just follow the UI focus.

Looked a bit into the code and it seems like a much bigger task than I would have imagined. But if it isn't addressed, many wont be able to use the plugin if they want controller support in the game.
I would love to help out but for me not really knowing the code base, it might not be the best task to start with..

Well I couldnt stop myself and I actually got it figured out pretty fast with very few code changes. :) This would be my first commit to a public repo, so I gotta figure that out first.

Your contribution would be great, but it is worth remembering that the plugin is divided into two parts in general, the core and the demos, the input part is just part of the demonstration of how to use the inventory system in an elegant way.
Anyway, your contribution is useful, because few users will recreate the architecture that is in the demo, but use what is already ready.


User passwords can't contain '&' character

If user password contains & character, user can no longer log in - not sure if the issue is at save time or at login time (hashing the supplied password).
I checked that data is not accidentally sent as a GET request but didn't investigate the core issue ATM.

This issue was fixed for 3.6.0.


WIP: Temporarily disable review request requirement

Signed-off-by: Chris Evich<EMAIL_ADDRESS>

@Mergifyio refresh


[4.x] Add base path to $request and use RouteContext to read

Another attempt for #2837. This is an alternative solution to #2859.
It uses the route runner to bring the base path to the $request object and then the route context helper class to read that attribute from the $request.

Coverage remained the same at 100.0% when pulling de91cf01c1e86da366772510806aaa7c2535c497 on adriansuter:patch-route-context-base-path into 80a52f78f5c3731913168d73e4796c46a6a95b16 on slimphp:4.x.

Thank you!

As the basePath attribute is set in RouteRunner, request passed to middlewares and indirectly to error handlers doesn't have have it set.
Only place is the Route callable.

@piotr-cz Should we add the basePath attribute directly in the App->run() method? If someone is overriding this method, then it is important to make sure that this attribute will be written to the request. Otherwise we can't access it.
@l0gicgate Do you think we should/could undo the RouteRunner modification?

A middleware run before the routing middleware would not have access to any route context objects, would it? So this solution would only work in a route callback.

After this PR attributes  route, routingResults and basePath are set on the request by the RouteRunner which is available in route callable.
As I understand it, if you want to get route attributes in the middleware, you have to add RoutingMiddleware (see docs). But this sets only route and routingResults attributes.
Another option could be to set the basePath request attribute in App::handle method.
To be honest, I'm not sure what is the best way out of this.
I've decided to set base path on the view renderer during dependecy initialization.

@piotr-cz Injecting the base path into the middleware, error handlers and route callables is the preferred way to go. At least if you use dependency injection. The solution to use request attributes should IMO only be used in small projects in case of absence of dependency injection.

okay, thanks for explanation


⚠ Bofrost.at has degraded performance

In 9ffcece, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2199 ms

Resolved: Bofrost.at performance has improved in bd3ebd9.


Problem with ember-bootstrap and<EMAIL_ADDRESS>

I've created a brand new ember 2.15 app, added ember-bootstrap with bootstrap 4 support and discovered that navbar is always being showed as the mobile version.
Messing around I've discovered that when installed ember-bootstrap put bootstrap version ^4.0.0-alpha.6 on package.json. The issue is that since 4.0.0-beta has been released it is being used instead of 4.0.0-alpha.6 which seems to have changed the way navbar should work: https://github.com/twbs/bootstrap/issues/23788
I've put an app example with this issue at https://github.com/michelboaventura/ember_bootstrap_test
A work around is just fix the correct version on package.json
Output from ember bootstrap:info (available since 1.0.0-alpha.9):
Npm packages:
ember-bootstrap: ^1.0.0-rc.2 -> 1.0.0-rc.2
ember-cli: ~2.15.0 -> 2.15.0
bootstrap: ^4.0.0-alpha.6 -> 4.0.0-beta
bootstrap-sass: n/a
ember-cli-sass: n/a
ember-cli-less: n/a
Bower packages:
bootstrap: n/a
bootstrap-sass: n/a
ember-bootstrap configuration:
bootstrapVersion: 4
importBootstrapFont: false
importBootstrapCSS: true

Hi, @michelboaventura. Thank you for the report. The BS4 beta work is happening on the bs4-beta branch. There were quite a few changes.

Hi, @srvance. it took me a while to figure this out because ember-bootstrap was installing an incompatible BS version, so I've created this issue to help others with the same issue.
Cheers,

Closing this. #415 makes sure that for now only alpha is installed by default. And #462 is a tracking issue for supporting beta...


How to restart cow?

I run cow in background on Linux, by " sudo nohup cow". But I cannot find the process, by "ps" command. When I want to restart cow, the error comes up "the address has been in use."
How to restart cow?

Maybe you can try
ps aux | grep  cow
Works for me

Also sending SIGUSR1 do a reload on configs (actually restart cow)

try init.d or systemd script, then you can use sudo service cow start | stop | restart


feat: file based naming resolver

Description
Provide a file based naming resolver to replace mDNS so that dapr can address each other either on the same host or via shared file system, which would be more stable and friendly for development.
The default directory to keep naming informations is: $HOME/.dapr/naming, and it can be customized in Dapr's config.yaml:
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: appconfig
spec:
  nameResolution:
    component: "file"
    configuration:
      dir: "/var/tmp/dapr-naming"

Pro and con of this solution:

Pro: Since the name resolving is file based, it means that it can not only work in slim mode, but also in standalone mode, since the directory to keep the naming informations can be anywhere as long as it's accessible to Dapr's instances.
Con: Since there has no lifecycle mechanism for components, this solution doesn't have chance to remove (unregister) its naming info from the file when Dapr instance goes away. It may not a big deal in development stage, but in the worst case user needs to know where the directory locates and delete the files manually.

Issue reference
This PR will close: dapr/dapr#3256
Checklist
Please make sure you've completed the relevant tasks for this PR, out of the following list:

[x] Code compiles correctly
[x] Created/updated tests
[ ] Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

@daixiang0 can you please review this?

@yaron2 sure.

ping @beiwei30

ping @beiwei30

sorry for late response, will follow up the review comments today.

@daixiang0 @artursouza I have updated this PR, pls. take a look again.

@beiwei30 Please resolve the conflicts.

keepalive

@yaron2 would you mind to review it again? I resolved the conflict today.

@beiwei30 can you please run go mod tidy and push the changes? Thanks!

@beiwei30 - When the app becomes unhealthy, there is lot of manual activity involved -

User should manually delete the name resolution file and lock file.
All the app instances should be restarted to create a new file with updated name-resolution information for all app instances.[Since we have a common name resolution information for all apps with same id]

Also having 777 file permissions allows anyone to write to the nameresolution file which is a security issue, since the consuming app can be misguided to connect to any endpoint.
To complete this solution, there should be a cleanup of the nameresolution information. when file is chosen as name resolution - dapr cli can periodically check if all the apps as running and clean up the nameresolution information to be accurate - thoughts? @mukundansundar @berndverst

I think we need someone to create a design document for the overall design / experience including the CLI, and the various edge cases before we should proceed here.
This feature is useful, but there seem to be plenty of gaps.

@berndverst and I synced yesterday, and I agree a file-based nameresolver should happen in 1.13, but IMHO this should be based on SQLIte.
Using a single SQLite database will already solve problems such as concurrent access by multiple sidecars (locking, etc). A single file is also easier to handle and to garbage-collect. Lastly, SQLite is also generally faster than the filesystem for these kinds of things.


Don't Merge! PR real time test for source branch full build

Docs Build status updates of commit fd3872f:
:clock10: Full build: the average full build time is 5 min(s) 50 sec(s), this is based on the last 30 build(s) for this repository.

Docs Build status updates of commit fd3872f:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docfx.json
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Redirects array in afterBuild hook always empty

Description
I'm trying to access the redirects array listed at https://gridsome.org/docs/dynamic-routing/#generating-rewrite-rules. Whenever I log it out though it's empty even on a brand new install of Gridsome.
Steps to reproduce
In gridsome.server.js add the following...
module.exports = function (api, options) {
  api.afterBuild(({ redirects }) => {
    console.log(redirects);
  });
}

Expected result
The redirects array in the afterBuild hook should container all necessary rewrite rules that should be generated.
Actual result
The array is empty []
Environment
System:
    OS: macOS 10.15.1
    CPU: (8) x64 Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
  Binaries:
    Node: 8.10.0 - ~/.nvm/versions/node/v8.10.0/bin/node
    Yarn: 1.9.4 - /usr/local/bin/yarn
    npm: 6.13.1 - ~/.nvm/versions/node/v8.10.0/bin/npm
  Browsers:
    Chrome: 78.0.3904.108
    Safari: 13.0.3
  npmPackages:
    @gridsome/plugin-sitemap: ^0.2.3 => 0.2.3
    @gridsome/source-datocms: ^0.2.1 => 0.2.1
    gridsome: ^0.7.11 => 0.7.11
    gridsome-plugin-gtm: ^0.1.1 => 0.1.1
    gridsome-plugin-tailwindcss: ^2.2.27 => 2.2.27
  npmGlobalPackages:
    @gridsome/cli: 0.3.1

Only paths for dynamic pages will be added to the redirects array since static assets usually don't require any special server config :) Or do you create dynamic pages in your project that aren't included?

Hey @hjvedvik,
Thanks for the response.
Yes we're creating dynamic pages and the redirects array is empty. Well I assume we are as we're using the Dato plugin and creating the following templates...
templates: {
    DatoLandingPage: "/:slug",
    DatoArticle: "/article/:slug",
    DatoService: "/services/:slug"
  },

My assumption was the above created dynamic pages, is that correct?
We're only trying to use the redirects array because we can't get the 404 page to work in production due to https://github.com/gridsome/gridsome/issues/834 and we thought using the redirects array might be a work around to that issue e.g we write all the redirects to the _redirects file on Netlify and anything else is treated as a 404.  Because the 404 pages aren't working correctly we are starting to be penalised by Google so we are trying to find a solution.

Template routes are dynamic routes, but not treated like dynamic pages :) I'll do some debugging and see if I can figure out why it doesn't show a 404 page. You can fix the problem temporarily by setting the path for the landing page with a function instead, like this:
DatoLandingPage: node => `/${node.slug}`

Ahh ok that makes sense. Thanks @hjvedvik setting the path with a function works 👌I've set it on all of the templates because incorrect child routes on a service for example weren't working either e.g /services/sddsfsd would try and return a service instead of a 404. But setting it with a function instead works! Thank you!
templates: {
    DatoLandingPage: node => `/${node.slug}`,
    DatoArticle: node => `/article/${node.slug}`,
    DatoService: node => `/services/${node.slug}`
}

Would you like me to close this issue and https://github.com/gridsome/gridsome/issues/834 or leave them open if you're still working on them?

Actually I spoke too soon @hjvedvik - the 404s still don't work properly in production. I wonder if it's an issue with Netlify. I was testing them locally before by running gridsome build then serving them using Superstatic. They even work locally with Netlify Dev so I'm not sure why they wouldn't be working in production. It feels like it's an issue with the router not recognising the route and serving up the index route. We're seeing some of the 404 page components load and the rest is the homepage, very strange. You can see the behaviour here: current vs expected. Thanks

I'm not sure why the /index.html file is opened for missing paths. It should have been the /404.html file. Do you have any rewrite rules in a netlify.toml or _redirects file?

Hey @hjvedvik yes we have the following redirects rule applied in our Netlify _redirects file:
/* /index.html 200
Netlify recommends you do this in their redirect docs.
I've tested without it and the homepage works but all other routes don't load.

Hey @hjvedvik /* /index.html 200 was the issue. I've removed it and everything is working fine now. I still have to use the node => /${node.slug}` fix for the LandingPage route but that's fine.

I've tested without it and the homepage works but all other routes don't load.

To clarify I tested without the index.html so /* / 200. That loads the root route but nothing else.
Is it worth adding something to https://gridsome.org/docs/deploy-to-netlify/ to avoid any confusion for other people in the future? I'm happy to submit a PR for this.
Thanks for your help and sorry for the confusion on this end.

Great @gilesbutler :) Yes, it would probably be good to mention it in the docs. Usually, a SPA has only one /index.html file, so paths must be rewritten to open that file. But Gridsome generates HTML files for every path on the site, so rewrite rules aren't necessary (except for dynamic pages).

Thanks @hjvedvik - yep makes perfect sense now. I think my mind was in SPA mode because Gridsome uses Vue under the hood ha. I'll get this added to the docs and send through a PR :)

Done @hjvedvik https://github.com/gridsome/gridsome.org/pull/319

Hey, just happen to came across this same issue. Does it make sense to use the string template syntax as the docs explain if 404 will not work properly? path as a function works property with a custom 404 page.

Just ran into this, just wanted to verify that, since this is closed, is the function way for templates:
node => `/${node.slug}`

the official way to handle this?

Just ran into this, just wanted to verify that, since this is closed, is the function way for templates:
node => `/${node.slug}`

the official way to handle this?

Hey, I was facing with the same issue. I'm not using Netlify I serve one of my websites from my own server.
So I just set the following in nginx end:
error_page 404 /404/index.html;
location / {
  index index.html;
  try_files $uri $uri/ =404;
}

Maybe this will help someone in the future.


Question: Version jump

Why did the library jump from v3 to v7?

Found the answer

For reference: https://github.com/uuidjs/uuid#upgrading-from-uuid3


Add-in crashes when multiple sessions are ran

Thinking about how the user might want to jump back  and forth from LLOS to RLOS a bug has been found that causes ArcGIS Pro to fail. This happens also if you try to run RLOS more than once in ArcGIS Pro 1.2.
Currently the possible fix is to have the add-in write  a new feature classes each time the tool is being ran.

@topowright is this still an issue or has it been fixed at 1.4?

@nfeuerstein This issue has been fixed.


Font that was once rendering correctly, now shows empty boxes

I am on Windows 10 Pro 1803 (17134.228) on a Dell XPS 15 9570 with the 4K screen (not sure if this matters).
I have been using "Ubuntu mono derivative Powerline" without issues on WSL. Yesterday I noticed that my zsh prompt no longer renders the font correctly; it now shows empty boxes instead of >>> (three arrows).

Here is what it used to look like:

I'm really not sure what triggered this; here is my Windows Update log:

Hey @burhan - in the erroneous case, what font is your Console using?

The same font, I haven't changed the console settings.
On Thu, Aug 30, 2018, 7:07 PM Rich Turner<EMAIL_ADDRESS>wrote:

Hey @burhan https://github.com/burhan - in the erroneous case, what
font is your Console using?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/console/issues/243#issuecomment-417374775,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAkz6NeU-xqShgtZjZOl1GmeVvQlVKwNks5uWA3BgaJpZM4WTCRS
.

--
--
Burhan Khalid
Sent from a mobile device

@Burhan: And do you have a link to the font you're using?

The font is installed as per the properties window.

The font installed is the same one as in your link @bitcrazed
I am not sure if this is related, but in VS code, the font renders correctly

Could you share your zsh configuration declaring the color chevrons?

${SSH_TTY:+"%F{9}%n%f%F{7}@%f%F{3}%m%f "}%F{4}${_prompt_sorin_pwd}%(!. %B%F{1}#%f%b.)${editor_info[keymap]}

And what does your .~/zshrc look like?

@burhan, The fonts in provided VS Code and cmd.exe screenshots look different to the naked eye. While the one in cmd.exe definitely looks like Ubuntu Mono, the VS Code one most certainly doesn't.

I wonder if this is the same problem as #3257? There's a lot more discussion there and a potential workaround, selecting "Install for all users" on the font may get it working properly.

I wonder if this is the same problem as #3257? There's a lot more discussion there and a potential workaround, selecting "Install for all users" on the font may get it working properly.

Hey so this was filed like, four years ago, and obviously there's been a lot of work done since then. Is this still happening? I think we added some better support for unsupported glyphs in the console in the latest Win11 bits. Then there's also the Terminal, which may have never had this issue at all. We never did nail down a specific repro for this one, so maybe it's gone/?


inkscape/textext: init at 1.8.1

Motivation for this change
Introduce the 'textext' extension for including LaTeX graphics in Inkscape
Test it with nix-shell -p "inkscape-with-extensions.override { inkscapeExtensions = [ inkscape-extensions.textext ]; }" --run inkscape
Alternative approach to #100565
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 121711 at d3fce869 run on aarch64-linux 1

1 package built successfully:

inkscape-extensions.textext

2 suggestions:

warning: python-include-tests
Consider adding a checkPhase for tests, or if not feasible, pythonImportsCheck.
Near pkgs/applications/graphics/inkscape/extensions.nix:104:0:
    |
104 |          description = "Re-editable LaTeX graphics for Inkscape";
    | ^

warning: missing-patch-comment
Please add a comment on the line above, explaining the purpose of this patch.
Near pkgs/applications/graphics/inkscape/extensions.nix:61:11:
   |
61 |           ./textext-placeholders.patch
   |           ^

Please rebase

Since this is multiple files, it would be nice to move them into a subdirectory.
Also, I tried switching to the pygobject but we will need to add bash to Inkscape’s interpreter list.
diff --git a/pkgs/applications/graphics/inkscape/extensions/textext.nix b/pkgs/applications/graphics/inkscape/extensions/textext.nix
index 29e001bfca8..4da59dba952 100644
--- a/pkgs/applications/graphics/inkscape/extensions/textext.nix
+++ b/pkgs/applications/graphics/inkscape/extensions/textext.nix
@@ -5,9 +5,12 @@
 , pdflatex
 , lualatex
 , python3
+, wrapGAppsHook
+, gobject-introspection
+, gtk3
 }:

-python3.pkgs.buildPythonPackage rec {
+python3.pkgs.buildPythonApplication rec {
   pname = "textext";
   version = "1.3.1";

@@ -25,17 +28,36 @@ python3.pkgs.buildPythonPackage rec {
       src = ./textext-paths.patch;
       inherit pdflatex lualatex;
     })
+
+    # Since we are wrapping the extension, we need to change the interpreter
+    # from Python to Bash.
+    ./textext/interpreter.patch
+  ];
+
+  nativeBuildInputs = [
+    wrapGAppsHook
+    gobject-introspection
+  ];
+
+  buildInputs = [
+    gtk3
   ];

   propagatedBuildInputs = [
     # GTK is recommended, but I haven't looked into pkgs.pygobject3 yet
     # https://nixos.org/manual/nixpkgs/stable/#ssec-gnome-hooks-gobject-introspection
-    python3.pkgs.tkinter
+    python3.pkgs.pygobject3
   ];

+  # strictDeps do not play nicely with introspection setup hooks.
+  strictDeps = false;
+
   # textext doesn't have a 'bdist_wheel' target
   dontUseSetuptoolsBuild = true;

+  # Avoid wrapping two times by just using Python’s wrapping.
+  dontWrapGApps = true;
+
   buildPhase = ''
     runHook preBuild

@@ -62,6 +84,17 @@ python3.pkgs.buildPythonPackage rec {
     runHook postInstall
   '';

+  preFixup = ''
+    # Prepare for wrapping
+    chmod +x "$out/share/inkscape/extensions/textext/__main__.py"
+    sed -i '1i#!/usr/bin/env python3' "$out/share/inkscape/extensions/textext/__main__.py"
+    makeWrapperArgs+=("''${gappsWrapperArgs[@]}")
+  '';
+
+  postFixup = ''
+    wrapPythonProgramsIn "$out/share/inkscape/extensions/textext" "$out $pythonPath"
+  '';
+
   meta = with lib; {
     description = "Re-editable LaTeX graphics for Inkscape";
     homepage = "https://textext.github.io/textext/";
diff --git a/pkgs/applications/graphics/inkscape/extensions/textext/interpreter.patch b/pkgs/applications/graphics/inkscape/extensions/textext/interpreter.patch
new file mode 100644
index<PHONE_NUMBER>0..39dcdb76eff
--- /dev/null
+++ b/pkgs/applications/graphics/inkscape/extensions/textext/interpreter.patch
@@ -0,0 +1,12 @@
+diff --git a/textext/textext.inx b/textext/textext.inx
+index b1c8835..1cdf7b3 100644
+--- a/textext/textext.inx
++++ b/textext/textext.inx
+@@ -8,6 +8,6 @@
+     </effects-menu>
+   </effect>
+   <script>
+-    <command location="inx" interpreter="python">__main__.py</command>
++    <command location="inx" interpreter="bash">__main__.py</command>
+   </script>
+ </inkscape-extension>

There is actually a shell interpreter so trying that now.

This works great:
diff --git a/pkgs/applications/graphics/inkscape/extensions/textext.nix b/pkgs/applications/graphics/inkscape/extensions/textext.nix
index 29e001bfca8..0e7c4299a60 100644
--- a/pkgs/applications/graphics/inkscape/extensions/textext.nix
+++ b/pkgs/applications/graphics/inkscape/extensions/textext.nix
@@ -5,9 +5,13 @@
 , pdflatex
 , lualatex
 , python3
+, wrapGAppsHook
+, gobject-introspection
+, gtk3
+, gtksourceview3
 }:

-python3.pkgs.buildPythonPackage rec {
+python3.pkgs.buildPythonApplication rec {
   pname = "textext";
   version = "1.3.1";

@@ -25,17 +29,37 @@ python3.pkgs.buildPythonPackage rec {
       src = ./textext-paths.patch;
       inherit pdflatex lualatex;
     })
+
+    # Since we are wrapping the extension, we need to change the interpreter
+    # from Python to Bash.
+    ./textext/interpreter.patch
+  ];
+
+  nativeBuildInputs = [
+    wrapGAppsHook
+    gobject-introspection
+  ];
+
+  buildInputs = [
+    gtk3
+    gtksourceview3
   ];

   propagatedBuildInputs = [
-    # GTK is recommended, but I haven't looked into pkgs.pygobject3 yet
-    # https://nixos.org/manual/nixpkgs/stable/#ssec-gnome-hooks-gobject-introspection
-    python3.pkgs.tkinter
+    python3.pkgs.pygobject3
+    # TODO: Required by inkex but is not inherited from inkscape when we use custom Python interpreter.
+    python3.pkgs.lxml
   ];

+  # strictDeps do not play nicely with introspection setup hooks.
+  strictDeps = false;
+
   # textext doesn't have a 'bdist_wheel' target
   dontUseSetuptoolsBuild = true;

+  # Avoid wrapping two times by just using Python’s wrapping.
+  dontWrapGApps = true;
+
   buildPhase = ''
     runHook preBuild

@@ -62,6 +86,17 @@ python3.pkgs.buildPythonPackage rec {
     runHook postInstall
   '';

+  preFixup = ''
+    # Prepare for wrapping
+    chmod +x "$out/share/inkscape/extensions/textext/__main__.py"
+    sed -i '1i#!/usr/bin/env python3' "$out/share/inkscape/extensions/textext/__main__.py"
+    makeWrapperArgs+=("''${gappsWrapperArgs[@]}")
+  '';
+
+  postFixup = ''
+    wrapPythonProgramsIn "$out/share/inkscape/extensions/textext" "$out $pythonPath"
+  '';
+
   meta = with lib; {
     description = "Re-editable LaTeX graphics for Inkscape";
     homepage = "https://textext.github.io/textext/";
diff --git a/pkgs/applications/graphics/inkscape/extensions/textext/interpreter.patch b/pkgs/applications/graphics/inkscape/extensions/textext/interpreter.patch
new file mode 100644
index<PHONE_NUMBER>0..64b39b61760
--- /dev/null
+++ b/pkgs/applications/graphics/inkscape/extensions/textext/interpreter.patch
@@ -0,0 +1,12 @@
+diff --git a/textext/textext.inx b/textext/textext.inx
+index b1c8835..1cdf7b3 100644
+--- a/textext/textext.inx
++++ b/textext/textext.inx
+@@ -8,6 +8,6 @@
+     </effects-menu>
+   </effect>
+   <script>
+-    <command location="inx" interpreter="python">__main__.py</command>
++    <command location="inx" interpreter="shell">__main__.py</command>
+   </script>
+ </inkscape-extension>

Cool! Do you want to make a new PR to supersede this one? (you're also welcome to force-push to this branch, but I'm not sure github will let us)

I can force push.

I can confirm this works for me as well! :+1: nice

Done.
Another worrying aspect is that both the config and the produced SVGs contain absolute store path of the preamble:
  "preamble": "/nix/store/0g42vvrpxanilivh4292ky90scvs80z7-inkscape-with-extensions-1.1/share/inkscape/extensions/textext/default_packages.tex",

       ns5:preamble="/nix/store/0g42vvrpxanilivh4292ky90scvs80z7-inkscape-with-extensions-1.1/share/inkscape/extensions/textext/default_packages.tex"

Maybe we will need some code that resolves it to default_packages.tex on export and to the absolute path on import.

Done.

Thanks!

Another worrying aspect is that both the config and the produced SVGs contain absolute store path of the preamble:
  "preamble": "/nix/store/0g42vvrpxanilivh4292ky90scvs80z7-inkscape-with-extensions-1.1/share/inkscape/extensions/textext/default_packages.tex",

       ns5:preamble="/nix/store/0g42vvrpxanilivh4292ky90scvs80z7-inkscape-with-extensions-1.1/share/inkscape/extensions/textext/default_packages.tex"

Maybe we will need some code that resolves it to default_packages.tex on export and to the absolute path on import. And possibly upstream that.

Yeah, that can't really be portable... I haven't looked at the code, but did a quick experiment breaking the preamble in a saved file, and that didn't cause problems: I was still able to edit the file (also the tex part).

I peeked at the code a bit and we would need to use basename on the path here when it is the default one:
https://github.com/textext/textext/blob/2f1e11c6671756d4693ad497d9f5b28d0e51e214/textext/base.py#L367
We probably would not even need to re-add it, thanks to:
https://github.com/textext/textext/blob/2f1e11c6671756d4693ad497d9f5b28d0e51e214/textext/base.py#L235-L236
But we can merge this as is and return to this later if you do not feel like addressing that now.

We probably would not even need to re-add it, thanks to:
https://github.com/textext/textext/blob/2f1e11c6671756d4693ad497d9f5b28d0e51e214/textext/base.py#L235-L236

Aah, that explains why it worked indeed ;)

But we can merge this as is and return to this later if you do not feel like addressing that now.

I think that'd make sense for now.

Aah, that explains why it worked indeed ;)

I guess it will only break when the preamble in the config.json does not exist (after update and garbage collection).
Also forgot to mention this line where the config is being updated:
https://github.com/textext/textext/blob/2f1e11c6671756d4693ad497d9f5b28d0e51e214/textext/base.py#L417-L420
The file existence check might rescue it but only after the next edit and the empty string in config would have to make the program use the preamble from the default config. We should probably test that.

I've tested this out from your branch, and I get an error that the GUI library is not found.

I fixed the problem, but I don't know how to send a PR to a PR... Here are my changes anyways: https://github.com/olynch/nixpkgs/tree/inkscape-textext-init-at-1.3.1

I am unable to reproduce that issue. And your patch should not really fix the issue – if the check is unable to find the GUI library, how could the extension?

Just tried rebasing this and the issue with hardcoded paths happens even without opening any files – if the path in settings (~/.config/inkscape/extensions/textext/config.json) is GC’d, the extension will complain:
Command /nix/store/cyw4szxxaibdd3jayjf5r18cagjm4kb2-texlive-combined-basic-2021.20210408/bin/lualatex tmp.tex -interaction=nonstopmode -halt-on-error failed: [Errno 2] No such file or directory: '/nix/store/cyw4szxxaibdd3jayjf5r18cagjm4kb2-texlive-combined-basic-2021.20210408/bin/lualatex'

Just tried rebasing this and the issue with hardcoded paths happens even without opening any files – if the path in settings (~/.config/inkscape/extensions/textext/config.json) is GC’d, the extension will complain:
Command /nix/store/cyw4szxxaibdd3jayjf5r18cagjm4kb2-texlive-combined-basic-2021.20210408/bin/lualatex tmp.tex -interaction=nonstopmode -halt-on-error failed: [Errno 2] No such file or directory: '/nix/store/cyw4szxxaibdd3jayjf5r18cagjm4kb2-texlive-combined-basic-2021.20210408/bin/lualatex'

I could reproduce this problem on textext 1.4.0, but then I updated this PR to 1.8.1 and now I don't see it anymore \o/

I tried rebasing onto nixos-unstable this but it no longer runs:
__main__.py: __main__.py: cannot execute binary file

possibly due to introduction of binary wrappers.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/how-to-define-an-inkscape-extension-with-python-dependencies/29384/3

rebased and added a 'start script' to make sure the extension is invoked again.
No errors anymore, and the extension python code is indeed executed, but it seems to hang(?) without opening an edit window.

When running inkscape with textext from the nixpkgs of the pr, the textext window does not open for me and no error message is logged into the terminal :(

When running inkscape with textext from the nixpkgs of the pr, the textext window does not open for me and no error message is logged into the terminal :(

Yes, that confirms what I saw. I looked into it slightly but saw nothing obvious. Let's first update to 1.9.0, which requires #245431

Thanks for your efforts. I hacked together a solution with tkinter that kinda works but is super ugly and a pain in the neck as it builds a large number of python packages every time...

okay, fixed two problems. This PR should be usable now (even without 1.3), if you can ignore a <frozen importlib._bootstrap>:671: ImportWarning: DynamicImporter.exec_module() not found; falling back to load_module() warning.

Can confirm: it works )
thank you so much

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/inkscape-extensions/33882/4

@jtojnar
Very sorry for the ping, but could you look into merging this as it seems ready? Thanks in advance!

@jtojnar I know that life can get busy some times. Could you maybe nominate somebody to have a look whether this can be merged?

This has been ready for a while, I don't see any obvious problems and additions are low risk. I'm sure @raboof or any of the other contributors want to help out if anything turns out can still be improved.


Investigate replaceWithText function

Describe the bug
There is not the first time, when public LeafElement replaceWithText() from LeafElement.java causes the exceptions in diKTat
Seems, that it can break the structure of AST tree in some cases, therefore we can met exceptions.
But this behavior should be investigated, and probably, we must refuse to use this function in diKTat.
Steps to Reproduce
E.g.: In function insertNewlinesBetweenBlocks from FileStructureRule
change line with replaceChild to the
(this as LeafPsiElement).replaceWithText("\n\n${text.replace("\n", "")}")

Another evidence: when diktat is executed with ktlint like ktlint -R diktat-0.6.1.jar -F smoke/src/main/kotlin/Example1Test.kt we also get this error, because ktlint mutates the tree:

[ERROR] 2021-06-23 11:52:49 Internal error has occurred in block-structure. Please make an issue on this bug at https://github.com/cqfn/diKTat/.
java.lang.NullPointerException
        at org.jetbrains.kotlin.com.intellij.psi.impl.source.codeStyle.CodeEditUtil.saveWhitespacesInfo(CodeEditUtil.java:122)
        at org.jetbrains.kotlin.com.intellij.psi.impl.source.tree.ChangeUtil.saveIndentationToCopy(ChangeUtil.java:106)
        at org.jetbrains.kotlin.com.intellij.psi.impl.source.tree.ChangeUtil.copyLeafWithText(ChangeUtil.java:82)
        at org.jetbrains.kotlin.com.intellij.psi.impl.source.tree.LeafElement.replaceWithText(LeafElement.java:154)
        at org.cqfn.diktat.ruleset.rules.chapter3.BlockStructureBraces$checkCloseBrace$2.invoke(BlockStructureBraces.kt:248)
        at org.cqfn.diktat.ruleset.rules.chapter3.BlockStructureBraces$checkCloseBrace$2.invoke(BlockStructureBraces.kt:244)

The issue was in the replaceWithText method.
It requires a special Indenting Service that saves original indenting in the node (useful in cases when you want just to change the functional part of the node without thinking about newlines and spaces).
But! if you will not provide this service during parsing (in ktlint for example), you will get an NPE (during the loading of the Service):
    public static void saveWhitespacesInfo(ASTNode first) {
        // if indentation was changed HERE <<<<
        if (first != null && !isNodeGenerated(first) && getOldIndentation(first) < 0) {
            PsiElement psiElement = first.getPsi();
            if (psiElement != null) {
                PsiFile file = psiElement.getContainingFile();
                // IndentHelper.getInstance() will be null here if we will not pass this service to kotlin compiler
                // and we will get NPE
                setOldIndentation((TreeElement)first, IndentHelper.getInstance().getIndent(file, first));
            }
        }
    }

Ktlint has already faced it in:
https://github.com/pinterest/ktlint/commit/7fb5885c551ceea0329983dc1c15ce0993f1f6f9

already done, we use rawReplaceWithText