re-joining after simulated node crash panics on trying to re-add an already existing node.

Describe the bug
Re-joining a leader to a cluster crashes with the attempt to re-add already known node to the progress state.
Oct 31 15:49:52.769 ERRO e: The node 2 already exists in the voters set., raft_id: 1
   0: backtrace::backtrace::libunwind::trace
             at /Users/vsts/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.34/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /Users/vsts/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.34/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:47
   3: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:36
   4: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:200
   5: std::panicking::default_hook
             at src/libstd/panicking.rs:214
   6: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:477
   7: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:384
   8: rust_begin_unwind
             at src/libstd/panicking.rs:311
   9: core::panicking::panic_fmt
             at src/libcore/panicking.rs:85
  10: core::result::unwrap_failed
             at src/libcore/result.rs:1084
  11: core::result::Result<T,E>::unwrap
             at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/libcore/result.rs:852
  12: uring::raft_node::RaftNode::on_ready
             at src/raft_node.rs:325
  13: uring::loopy_thing
             at src/main.rs:635
  14: uring::main::{{closure}}
             at src/main.rs:693

To Reproduce
I set up a mini demo to replicate:

clone https://github.com/wayfair-incubator/uring/tree/2416031ac34759f002a9a1539b5a2a54bbd84946
start node 1: cargo run -- -e <IP_ADDRESS>:8081 -i 1
wait for it to elect itself leader
start node 2: cargo run -- -e <IP_ADDRESS>:8082 -i 2 -p <IP_ADDRESS>:8081
wait for it to join the cluster
start node 3: cargo run -- -e <IP_ADDRESS>:8083 -i 3 -p <IP_ADDRESS>:8081
wait for it to join the cluster.
Terminate node 1 (leader) CTRL+C
node 2 or 3 will become leader
restart node 1 and let it re-join the cluster cargo run -- -e <IP_ADDRESS>:8081 -i 01 -p <IP_ADDRESS>:8082

Expected behavior
Stopping and starting nodes in a cluster should be handled gracefully
System information
(probably not relevant)

CPU architecture: x86
Distribution and kernel version: OS X 10.14.6
SELinux on?: No
Any other system details we should know?: no

Additional context
The shared repo is a minimal demo app trying to put raft-rs into a re-state for a reft cluster.

/cc @darach

The node 2 and node 3 both panicked directly when I try to reproduce the scenario in step 8:
Nov 01 00:24:56.636 ERRO e: The node 3 already exists in the voters set., raft_id: 2

Nov 01 00:24:56.648 ERRO e: The node 3 already exists in the voters set., raft_id: 3

Maybe there is  something wrong in the state machine 🤔.

After some digging I noticed that logs are not replicated to joining nodes. So joining 2 and the 3 they do share the same snapshot but they don't share the same logs - could that be where the error originates from?

@Hoverbear PTAL

Hmm...
This is typically caused by one of these:
https://github.com/tikv/raft-rs/blob/d34f6936e737563cc930bb8f03970a18f0cba7c9/src/progress/progress_set.rs#L301-L341
You can see when a node is removed from the ProgressSet we remove it from the voter set:
https://github.com/tikv/raft-rs/blob/d34f6936e737563cc930bb8f03970a18f0cba7c9/src/progress/progress_set.rs#L347-L355
That's called by remove_node here:
https://github.com/tikv/raft-rs/blob/d34f6936e737563cc930bb8f03970a18f0cba7c9/src/raft.rs#L2104-L2123
Which is called via apply_conf_change here:
https://github.com/tikv/raft-rs/blob/d34f6936e737563cc930bb8f03970a18f0cba7c9/src/raw_node.rs#L324-L340
I see you're updating it:
https://github.com/wayfair-incubator/uring/blob/a0a5ffa2990e2e8c91f9849cd7eefdd082501543/src/raft_node.rs#L333-L345
I need to play with this some :)

Thanks taking the time to provide  all the context!
The assumption made is that add and remove node via a conf change event is for orderly growing or shrinking the raft group not intermittent failure. So currently there is no code path in which we send a remove-node conf change event.
So basically in the code we call add_node only when a new node joining the cluster (on the simplest possible algorithm of pre-defined/shared node id's and if they're known or not), we didn't got to removing nodes again yet since we're still battling the restart case.

So some preliminary tests:

Online 1, Election (L1), Online 2 & 3, Pause, Down 2 or 3, no issues. Rejoin is fine.
Online 1, Election (L1), Online 2 & 3, Pause, Down 1, panic.

trace:
Nov 06 10:53:55.571 DEBG Sending from 2 to 1, msg: msg_type: MsgHeartbeatResponse to: 1, to: 1, from: 2, raft_id: 2
[WS Onramp] Connection terminated.
system stopped
Error: Failed to connect to host: Connection refused (os error 111)
Nov 06 10:53:55.745 WARN down(local), id: 1
Nov 06 10:53:57.321 INFO starting a new election, term: 2, raft_id: 2
Nov 06 10:53:57.321 INFO became pre-candidate at term 2, term: 2, raft_id: 2
send raft message to 1 fail, let Raft retry it
Nov 06 10:53:57.322 INFO 2 received message from 2, term: 2, msg: MsgRequestPreVote, from: 2, id: 2, raft_id: 2
Nov 06 10:53:57.323 INFO [logterm: 2, index: 5] sent request to 1, msg: MsgRequestPreVote, term: 2, id: 1, log_index: 5, log_term: 2, raft_id: 2
Nov 06 10:53:57.324 DEBG Sending from 2 to 1, msg: msg_type: MsgRequestPreVote to: 1 term: 3 log_term: 2 index: 5, to: 1, from: 2, raft_id: 2
Nov 06 10:53:57.325 INFO [logterm: 2, index: 5] sent request to 3, msg: MsgRequestPreVote, term: 2, id: 3, log_index: 5, log_term: 2, raft_id: 2
Nov 06 10:53:57.326 DEBG Sending from 2 to 3, msg: msg_type: MsgRequestPreVote to: 3 term: 3 log_term: 2 index: 5, to: 3, from: 2, raft_id: 2
Nov 06 10:53:57.327 DEBG State transition, next-state: PreCandidate, last-state: Follower
Nov 06 10:53:57.332 INFO received from 3, term: 2, msg type: MsgRequestPreVoteResponse, from: 3, raft_id: 2
Nov 06 10:53:57.333 DEBG reset election timeout 18 -> 16 at 0, election_elapsed: 0, timeout: 16, prev_timeout: 18, raft_id: 2
Nov 06 10:53:57.333 INFO became candidate at term 3, term: 3, raft_idsend raft message to 1 fail, let Raft retry it
: 2
Nov 06 10:53:57.334 INFO 2 received message from 2, term: 3, msg: MsgRequestVote, from: 2, id: 2, raft_id: 2
Nov 06 10:53:57.335 INFO [logterm: 2, index: 5] sent request to 1, msg: MsgRequestVote, term: 3, id: 1, log_index: 5, log_term: 2, raft_id: 2
Nov 06 10:53:57.336 DEBG Sending from 2 to 1, msg: msg_type: MsgRequestVote to: 1 term: 3 log_term: 2 index: 5, to: 1, from: 2, raft_id: 2
Nov 06 10:53:57.337 INFO [logterm: 2, index: 5] sent request to 3, msg: MsgRequestVote, term: 3, id: 3, log_index: 5, log_term: 2, raft_id: 2
Nov 06 10:53:57.338 DEBG Sending from 2 to 3, msg: msg_type: MsgRequestVote to: 3 term: 3 log_term: 2 index: 5, to: 3, from: 2, raft_id: 2
Nov 06 10:53:57.339 DEBG State transition, next-state: Candidate, last-state: PreCandidate
Nov 06 10:53:57.344 INFO received from 3, term: 3, msg type: MsgRequestVoteResponse, from: 3, raft_id: 2
Nov 06 10:53:57.345 DEBG reset election timeout 16 -> 19 at 0, election_elapsed: 0, timeout: 19, prev_timeout: 16, raft_id: 2
Nov 06 10:53:57.345 INFO became leader at term 3, [src/raft_node.rs:434] raft_group.raft.raft_log.last_index() + 1 = 7
term: 3, raft_id: 2
Nov 06 10:53:57.345 DEBG Sending from 2 to 1, msg: msg_type: MsgAppend to: 1 log_term: 2 index: 5 entries {term: 3 index: 6} commit: 5, to: 1, from: 2, raft_id: 2
Nov 06 10:53:57.346 DEBG Sending from 2 to 3, msg: msg_type: MsgAppend to: 3 log_term: 2 index: 5 entries {term: 3 index: 6} commit: 5, to: 3, from: 2, raft_id: 2
Nov 06 10:53:57.347 DEBG State transition, next-state: Leader, last-state: Candidate
send raft message to 1 fail, let Raft retry it
Nov 06 10:53:57.409 DEBG committing index 6, index: 6, raft_id: 2
Nov 06 10:53:57.409 DEBG Sending from 2 to 3, msg: msg_type: MsgAppend to: 3 log_term: 3 index: 6 entries {entry_type: EntryConfChange term: 3 index: 7 data: "\030\003"} commit: 6, to: 3, from: 2, raft_id: 2
Nov 06 10:53:57.458 DEBG committing index 7, index: 7, raft_id: 2
Nov 06 10:53:57.458 DEBG Sending from 2 to 3[src/raft_node.rs:343] &cc = node_id: 3

, msg: msg_type: MsgAppend to: 3 log_term: 3 index: 7 commit: 7, tothread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Exists(3, "voters")', src/libcore/result.rs:1165:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
: 3, from: 2, raft_id: 2
Nov 06 10:53:57.459 DEBG adding node (learner: false) with ID 3 to peers., id: 3, learner: false, raft_id: 2
Nov 06 10:53:57.460 DEBG Inserting voter with id 3, id: 3, raft_id: 2
Nov 06 10:53:57.461 ERRO e: The node 3 already exists in the voters set., raft_id: 2
thread 'actix-rt:worker:1' panicked at 'called `Result::unwrap()` on an `Err` value: "SendError(..)"', src/libcore/result.rs:1165:5
thread 'actix-rt:worker:1' panicked at 'called `Result::unwrap()` on an `Err` value: "SendError(..)"', src/libcore/result.rs:1165:5
stack backtrace:
   0:     0x56294b85ced4 - backtrace::backtrace::libunwind::trace::hf9ad636648efb226
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1:     0x56294b85ced4 - backtrace::backtrace::trace_unsynchronized::hed853f204024d151
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2:     0x56294b85ced4 - std::sys_common::backtrace::_print_fmt::h540be03650997d7f
                               at src/libstd/sys_common/backtrace.rs:77
   3:     0x56294b85ced4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h859895b5f65feccf
                               at src/libstd/sys_common/backtrace.rs:61
   4:     0x56294b8805ac - core::fmt::write::h55a2a37a9ae04dad
                               at src/libcore/fmt/mod.rs:1028
   5:     0x56294b857d87 - std::io::Write::write_fmt::h55b418b8331f602e
                               at src/libstd/io/mod.rs:1412
   6:     0x56294b85f44e - std::sys_common::backtrace::_print::h01cf1aa97e3bff6e
                               at src/libstd/sys_common/backtrace.rs:65
   7:     0x56294b85f44e - std::sys_common::backtrace::print::h992feaba39b9b6be
                               at src/libstd/sys_common/backtrace.rs:50
   8:     0x56294b85f44e - std::panicking::default_hook::{{closure}}::hf519e5920434d96d
                               at src/libstd/panicking.rs:188
   9:     0x56294b85f141 - std::panicking::default_hook::ha5560763526d8f2a
                               at src/libstd/panicking.rs:205
  10:     0x56294b85fb4b - std::panicking::rust_panic_with_hook::h23e3b045936d3e0a
                               at src/libstd/panicking.rs:464
  11:     0x56294b85f6ee - std::panicking::continue_panic_fmt::h22cad8a6b4a9a888
                               at src/libstd/panicking.rs:373
  12:     0x56294b85f5d6 - rust_begin_unwind
                               at src/libstd/panicking.rs:302
  13:     0x56294b87c4ce - core::panicking::panic_fmt::h9f376b306ccc647c
                               at src/libcore/panicking.rs:139
  14:     0x56294b87c5c7 - core::result::unwrap_failed::ha857e2e239846901
                               at src/libcore/result.rs:1165
  15:     0x56294a38d6d4 - core::result::Result<T,E>::unwrap::h2b62ba69aa6e5d84
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/result.rs:933
  16:     0x56294a3e2a1a - <uring::UrSocket as actix::actor::Actor>::stopped::h39bc887b77fa9f3d
                               at src/main.rs:129
  17:     0x56294a31dec8 - <actix::contextimpl::ContextFut<A,C> as futures::future::Future>::poll::h20f0330cfe0a782f
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-0.8.3/src/contextimpl.rs:438
  18:     0x56294a31c705 - <actix::contextimpl::ContextFut<A,C> as core::ops::drop::Drop>::drop::h7575ff147dd421c0
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-0.8.3/src/contextimpl.rs:220
  19:     0x56294a36ad01 - core::ptr::real_drop_in_place::h0f7834c8d060f1e5
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  20:     0x56294a36b521 - core::ptr::real_drop_in_place::h15157f2273174a4a
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  21:     0x56294a382eb1 - core::ptr::real_drop_in_place::hfd3912dcfbeb53e5
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  22:     0x56294a379348 - core::ptr::real_drop_in_place::h9bdb1342952ae961
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  23:     0x56294a3715bb - core::ptr::real_drop_in_place::h4cd96d3c8d8904ab
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  24:     0x56294a377cd5 - core::ptr::real_drop_in_place::h89d944f4395f2954
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  25:     0x56294a380383 - core::ptr::real_drop_in_place::he465ebf8b5d81fc4
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  26:     0x56294a36e565 - core::ptr::real_drop_in_place::h303c9666062b1852
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  27:     0x56294a373439 - core::ptr::real_drop_in_place::h639d6c80397d3994
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  28:     0x56294a37c476 - core::ptr::real_drop_in_place::hbd286707c9379e36
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  29:     0x56294a369842 - core::ptr::real_drop_in_place::h050601c6fa9adbb3
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  30:     0x56294a382e91 - core::ptr::real_drop_in_place::hfd1b973e3f3405b7
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  31:     0x56294a37d47f - core::ptr::real_drop_in_place::hc6257ba62dd3ee4e
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  32:     0x56294a3800a1 - core::ptr::real_drop_in_place::he21e27ce352065f6
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  33:     0x56294a37ca48 - core::ptr::real_drop_in_place::hbfdd5e57c1ea6a8a
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  34:     0x56294a36fd81 - core::ptr::real_drop_in_place::h3e1b516237dc7899
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  35:     0x56294a36f1f4 - core::ptr::real_drop_in_place::h384c3f60fbd1cfd3
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  36:     0x56294aeb48b8 - core::ptr::real_drop_in_place::h25dd0baed7f0f375
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  37:     0x56294aeb4fe1 - core::ptr::real_drop_in_place::hdd6aac9bbf1b25da
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  38:     0x56294aeb4851 - core::ptr::real_drop_in_place::h1bc710e61ac51dd8
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  39:     0x56294aeb488f - core::ptr::real_drop_in_place::h20559ea5933991c4
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  40:     0x56294aeb6d47 - core::mem::drop::h7804235b62a558b2
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/mem/mod.rs:704
  41:     0x56294ae10035 - tokio_current_thread::scheduler::release_node::h72a3b1970158c2e8
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/scheduler.rs:382
  42:     0x56294ae0ff4f - <tokio_current_thread::scheduler::Scheduler<U>::tick::Bomb<U> as core::ops::drop::Drop>::drop::{{closure}}::hf313f7599952abe0
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/scheduler.rs:271
  43:     0x56294ae0cbe4 - tokio_current_thread::Borrow<U>::enter::{{closure}}::{{closure}}::h93152cf040b0cfc6
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:788
  44:     0x56294ae0c593 - tokio_current_thread::CurrentRunner::set_spawn::hb299958eebe1449b
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:825
  45:     0x56294ae0ca85 - tokio_current_thread::Borrow<U>::enter::{{closure}}::h8bb87fc431b3369e
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:788
  46:     0x56294adf4b1a - std::thread::local::LocalKey<T>::try_with::h4b4da45cb0953780
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libstd/thread/local.rs:262
  47:     0x56294adf3568 - std::thread::local::LocalKey<T>::with::hbe6f12a20964b981
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libstd/thread/local.rs:239
  48:     0x56294ae0c61b - tokio_current_thread::Borrow<U>::enter::h0d6f514f2853732f
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:786
  49:     0x56294ae0feb7 - <tokio_current_thread::scheduler::Scheduler<U>::tick::Bomb<U> as core::ops::drop::Drop>::drop::h60e904c13e6346e3
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/scheduler.rs:271
  50:     0x56294ae066a5 - core::ptr::real_drop_in_place::h87d3db269008cc8c
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ptr/mod.rs:175
  51:     0x56294ae11879 - tokio_current_thread::scheduler::Scheduler<U>::tick::h8ba9f0e91fe9beb0
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/scheduler.rs:342
  52:     0x56294ae0ce2d - tokio_current_thread::Entered<P>::tick::h015233dbf859aa87
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:612
  53:     0x56294ae0cff0 - tokio_current_thread::Entered<P>::block_on::h5c966374bfc009d6
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-current-thread-0.1.6/src/lib.rs:502
  54:     0x56294ae0168e - actix_rt::runtime::Runtime::block_on::{{closure}}::ha255279c9e9e0678
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-0.2.5/src/runtime.rs:128
  55:     0x56294ae01d61 - actix_rt::runtime::Runtime::enter::{{closure}}::{{closure}}::{{closure}}::{{closure}}::h71c41cadaf16c741
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-0.2.5/src/runtime.rs:168
  56:     0x56294ae2851d - tokio_executor::global::with_default::{{closure}}::hf2f1bdd7887fd578
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.8/src/global.rs:209
  57:     0x56294adf6448 - std::thread::local::LocalKey<T>::try_with::heda6599f2047da06
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libstd/thread/local.rs:262
  58:     0x56294adf348c - std::thread::local::LocalKey<T>::with::hb8098ea981ebb349
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libstd/thread/local.rs:239
  59:     0x56294ae280a2 - tokio_executor::global::with_default::hc1252736b4e426c1
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.8/src/global.rs:178
  60:     0x56294ae01f7e - actix_rt::runtime::Runtime::enter::{{closure}}::{{closure}}::{{closure}}::h9de40e2003eada88
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-0.2.5/src/runtime.rs:166
  61:     0x56294ae19dc2 - tokio_timer::timer::handle::with_default::{{closure}}::h6e3dd5bd04bbe4cc
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.11/src/timer/handle.rs:101
  62:     0x56294adf3a28 - std::thread::local::LocalKey<T>::try_with::h04334be1dd5a6501
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libstd/thread/local.rs:262
  63:     0x56294adf2c1c - std::thread::local::LocalKey<T>::with::h1110cb2dd7a5e41a
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libstd/thread/local.rs:239
  64:     0x56294ae19955 - tokio_timer::timer::handle::with_default::h6e7191e9227a3395
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.11/src/timer/handle.rs:84
  65:     0x56294ae020a9 - actix_rt::runtime::Runtime::enter::{{closure}}::{{closure}}::hcb4ffcbdedde3879
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-0.2.5/src/runtime.rs:159
  66:     0x56294adf0df8 - tokio_timer::clock::clock::with_default::{{closure}}::h4d2baf6a8640417e
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.11/src/clock/clock.rs:137
  67:     0x56294adf4844 - std::thread::local::LocalKey<T>::try_with::h44e90ccbce8ba7a0
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libstd/thread/local.rs:262
  68:     0x56294adf38ad - std::thread::local::LocalKey<T>::with::hf8efabeded619c12
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libstd/thread/local.rs:239
  69:     0x56294adf0a7e - tokio_timer::clock::clock::with_default::h744d246f9ee1c3c7
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.11/src/clock/clock.rs:117
  70:     0x56294ae020fd - actix_rt::runtime::Runtime::enter::{{closure}}::h25beeae16b0aab2b
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-0.2.5/src/runtime.rs:158
  71:     0x56294ae1a53f - tokio_reactor::with_default::{{closure}}::h047eea81de149d26
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.10/src/lib.rs:237
  72:     0x56294adf5544 - std::thread::local::LocalKey<T>::try_with::h983b47c211a53258
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libstd/thread/local.rs:262
  73:     0x56294adf2b9d - std::thread::local::LocalKey<T>::with::h0ddb474724c356fd
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libstd/thread/local.rs:239
  74:     0x56294ae1a318 - tokio_reactor::with_default::hedeceb1f1204f41e
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.10/src/lib.rs:217
  75:     0x56294ae0196c - actix_rt::runtime::Runtime::enter::he0a8a92e295c18e4
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-0.2.5/src/runtime.rs:157
  76:     0x56294ae01478 - actix_rt::runtime::Runtime::block_on::h5a965072a66504f6
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-0.2.5/src/runtime.rs:126
  77:     0x56294ae26303 - actix_rt::arbiter::Arbiter::new::{{closure}}::h03e49a8780bdb49d
                               at /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-0.2.5/src/arbiter.rs:113
  78:     0x56294ae1f0f5 - std::sys_common::backtrace::__rust_begin_short_backtrace::h11b303417e3a8564
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libstd/sys_common/backtrace.rs:129
  79:     0x56294ae040c1 - std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}::he7652596c7722712
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libstd/thread/mod.rs:469
  80:     0x56294adf6664 - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h8613466756385bec
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libstd/panic.rs:317
  81:     0x56294adf1911 - std::panicking::try::do_call::h711ea6f48f6c4718
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libstd/panicking.rs:287
  82:     0x56294b86414a - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:78
  83:     0x56294adf1768 - std::panicking::try::h7cbe878e6ead05a2
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libstd/panicking.rs:265
  84:     0x56294adf69d6 - std::panic::catch_unwind::hf74332f300029515
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libstd/panic.rs:396
  85:     0x56294ae03eb2 - std::thread::Builder::spawn_unchecked::{{closure}}::h7970385913689ac2
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libstd/thread/mod.rs:468
  86:     0x56294ae04434 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hf90741fee7c7944d
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/libcore/ops/function.rs:227
  87:     0x56294b85100f - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::hab0345739862a0e3
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/liballoc/boxed.rs:942
  88:     0x56294b863210 - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h97269b0dcf5f4700
                               at /rustc/b520af6fd515b186caed436d75162a42aa183d95/src/liballoc/boxed.rs:942
  89:     0x56294b863210 - std::sys_common::thread::start_thread::h6d1cb01c00583451
                               at src/libstd/sys_common/thread.rs:13
  90:     0x56294b863210 - std::sys::unix::thread::Thread::new::thread_start::h2c73c5e5c4fa31f9
                               at src/libstd/sys/unix/thread.rs:79
  91:     0x7f4cd68f4669 - start_thread
  92:     0x7f4cd6800323 - clone
  93:                0x0 - <unknown>
thread panicked while panicking. aborting.
Illegal instruction

Seems this bit is what's important:
Nov 06 10:53:57.459 DEBG adding node (learner: false) with ID 3 to peers., id: 3, learner: false, raft_id: 2
Nov 06 10:53:57.460 DEBG Inserting voter with id 3, id: 3, raft_id: 2
Nov 06 10:53:57.461 ERRO e: The node 3 already exists in the voters set., raft_id: 2

Ja, which is odd, the only time we fire a AddNode event if the raft state says it's an unknown node: https://github.com/wayfair-incubator/uring/blob/master/src/main.rs#L557 since node 1 did see 2 and 3 before and persists the state to disk it should never get there - I don't see any add-node messages fire either.

Hmm, so ConfChange should normally come from Raft log entries. This is the replicated log, so 'injecting' an entry is inherently unsafe. I suggest you remove that code. (https://github.com/wayfair-incubator/uring/blob/a0a5ffa2990e2e8c91f9849cd7eefdd082501543/src/main.rs#L557-L584) :)
If you want to add a node you can try in a way like:
https://github.com/tikv/raft-rs/blob/d34f6936e737563cc930bb8f03970a18f0cba7c9/examples/five_mem_node/main.rs#L405-L419
This is the correct way to add something to the log.
But I'm not sure you want to try to do that in this situation. :) Just commenting out that code seems to remove the panic...
It does seem like the remaining followers fail to correctly hold a new election though. I'm going to let you take a look and let me know if this helps. :)

Also: I'm sorry the docs aren't so clear. :( We are trying to improve it and would definitely love any concrete issues/prs with improvements.

No reason to be sorry, I think everyone who has ever worked on a project of any kind of complexity can understand, it’s hard to document what you know since everything is obvious at that point ;)
To the code, we started with a direct copy but had to add the second change even or the other nodes didn’t learn about the first node. But the method is even after that still the same - it gets published over a proposal and agreed on not injected in the logs. But I’ll try to remove / replace the code later perhaps the issue it was added for originally was a different one. That said it’s a bit different from the memory nodes since they all exist in their own world and are networked do we can’t just join 5 nudes at boot time, they got to grow as they connect

You should definitely be able to grow/shrink your Raft cluster online. :)

So I tried the suggestion of commenting out the code, but that just doesn't crash because it never creates a group. Node 1 knows itself, as a voter, but neither node 2 or three know of any other node
Node1
Nov 06 21:26:57.192 DEBG NODE STATE, remote-mailboxes: [2, 3], local-mailboxes: [], randomized-election-timeout: 15, election-elapsed: 4, pass-election-timeout: false, promotable: true, voters: {1}, votes: {}, vote: 1, last-index: 2, first-index: 2, term: 2, leader-id: 1, role: Leader, node-id: 1

Node2
Nov 06 21:27:13.903 DEBG NODE STATE, remote-mailboxes: [3], local-mailboxes: [1], randomized-election-timeout: 12, election-elapsed: 565, pass-election-timeout: true, promotable: false, voters: {}, votes: {}, vote: 0, last-index: 0, first-index: 1, term: 0, leader-id: 0, role: Follower, node-id: 2

Node3
Nov 06 21:26:24.032 DEBG NODE STATE, remote-mailboxes: [], local-mailboxes: [2, 1], randomized-election-timeout: 14, election-elapsed: 93, pass-election-timeout: true, promotable: false, voters: {}, votes: {}, vote: 0, last-index: 0, first-index: 1, term: 0, leader-id: 0, role: Follower, node-id: 3

I'll try to make an explicit 'add node' action that isn't connected to the join on the clsuter perhaps that works better

So I changed how chance events are send and made it explicit.
That shifts the problem slightly, it now goes to (after restarring node 1 (initial leade), and node 3 (new leader) - on node 1):
Nov 06 21:43:09.803 WARN down(local), id: 3
thread '<unnamed>' panicked at 'slice[6,6] out of bound[2,4], raft_id: 1', /Users/heinz/.cargo/git/checkouts/raft-rs-42b8049ef2e3af07/2ce67a0/src/raft_log.rs:420:13

which is a bit odd since none of the nodes has 6 as a index.
1 (before it crashed)
Nov 06 21:43:01.487 DEBG NODE STATE, remote-mailboxes: [], local-mailboxes: [3, 2], randomized-election-timeout: 10, election-elapsed: 2, pass-election-timeout: false, promotable: true, voters: {1, 2, 3}, votes: {}, vote: 0, last-index: 4, first-index: 2, term: 3, leader-id: 3, role: Follower, node-id: 1

2
Nov 06 21:44:58.602 DEBG NODE STATE, remote-mailboxes: [], local-mailboxes: [], randomized-election-timeout: 10, election-elapsed: 6, pass-election-timeout: false, promotable: true, voters: {1, 2, 3}, votes: {2: true}, vote: 3, last-index: 5, first-index: 4, term: 3, leader-id: 0, role: PreCandidate, node-id: 2

3
Nov 06 21:43:02.790 DEBG NODE STATE, remote-mailboxes: [1], local-mailboxes: [2], randomized-election-timeout: 12, election-elapsed: 3, pass-election-timeout: false, promotable: true, voters: {1, 2, 3}, votes: {}, vote: 3, last-index: 5, first-index: 5, term: 3, leader-id: 3, role: Leader, node-id: 3

Hmmm, that's a bit of a puzzle isn''t it!
I see that index 5 is done:
Nov 07 12:17:02.582 INFO [commit: 5, term: 3] restored snapshot [index: 5, term: 3], snapshot_term: 3, snapshot_index: 5, commit: 5, term: 3, raft_id: 1

Yup, but only on node 2 & 3, node 2 is still at 4 - I suspect 5 was the election of the new leader when 1 was killed so it makes sense that it doesn't have it yet. I'm still unsure where it gets 6 from

@Licenser It looks like once you start the nodes and call the curl -X requests you still end up with 3 followers:

Since the cluster has otherwise been left at a steady state for several seconds, I expected at least one node to be the leader.

When I read the logs right, node 1 claims leadership, all 3 nodes agree with leader-id: 1 and on the first node it claims role: Leader

Yup I misread and deleted the comment :-P

:D ah sorry, I just saw the mail.

@Licenser If you don't call curl -X POST <IP_ADDRESS>:8081/node/1 in your reproduction case it seems to work without panicking... At least raft is trying to send the correct messages but it seems to be unable to.

Thanks! That's an interesting observation. I didn't try that yet - it's a bit late here (close to midnight) but I'll give that a try tomorrow and keep digging.
The failing to send message might be because the node was booted without knowing it's peers that's a easy to fix thing  (famous last words) at least easy to work around for the same of getting the raft-rs use case working enough to have a minimal functioning kv store in it by just passing all peers to all nodes with more -p in the command line (obviously not a production grade solution 😂)

Yeah, let's get this working! :) Would love to see more raft-rs derivatives and this is informing how we can better document things.
(Un)Fortunately we're updating our membership change mechanisms (hello Joint Consensus) hopefully soon once we resolve some lingering safety issues with the etcd folks.
If you need help learning to use Raft @luciofranco made a simple KV here: https://github.com/LucioFranco/kv, and the TiKV Raftstore is a maximal example: https://github.com/tikv/tikv/tree/master/src/raftstore

Learning raft isn't really my problem I'm not unfamiliar with it. Working through the assumptions in raft-rs is where I am stuck :(.
I looked at @LucioFranco and as far as I can tell it suffers from the same problems.
bootstrap 1, join 2 1, join 3 1, kill 1, join 1 2 ends up in "The node 1 already exists in the voters set."

So I've come full circle with this. The original 'node already known problem' back where I am. The solution to that was to not load the ConfState from disk when restarting a node. That on the other hand causes nodes to forget their peers so restarting 1 it only comes up with voter 2, 3 (but not itself), restarting 2, and 2 only has 3 as voter but neither 1 or itself.
More general it seems that nodes only know about nodes added after them, so node 1 after a restart knows about 2/3, node 2 after a restart knows about 3, node 3 after a restart knows none.

bootstrap 1, join 2 1, join 3 1, kill 1, join 1 2 ends up in "The node 1 already exists in the voters set."

At the join 1 step, why are you doing this? 1 was already added? Once a node is added to the set, it should only be removed via an explicit Remove. As you know from Raft the leader (2 or 3) will continually try to reconnect to the lost node (1), once they manage to hear back, the node is recovered. It never needs to be removed unless you've actually explicitly given up on that node.
Here's how the process should go:

Create & start Raft node 1, it becomes leader.
Create Raft node 2 & 3 and idle them.
Inform Leader (1) of node 2 & 3 through an AddNode proposal. Let it run and hold an election.
The cluster is lively.
Kill 1
2/3 hold an election, one wins.
1 recovers, begins listening and ticking it's election timeout.
Leader (2/3) either contacts it or hears from it and captures it. (the other follower would just redirect it to the current leader)
The cluster is lively again.

So you'd do remove/add when you were decommisioning a host, and if you were just taking a host down for maintenance or a reboot, you can just leave it in the membership pool. :)

bootstrap 1, join 2 1, join 3 1, kill 1, join 1 2 ends up in "The node 1 already exists in the voters set."

At the join 1 step, why are you doing this? 1 was already added? Once a node is added to the set, it should only be removed via an explicit Remove. As you know from Raft the leader (2 or 3) will continually try to reconnect to the lost node (1), once they manage to hear back, the node is recovered. It never needs to be removed unless you've actually explicitly given up on that node.

kv needs to be called with either join or bootstrap there is no other way to start it, I'd rather not call either but that isn't an option.

Here's how the process should go:

Create & start Raft node 1, it becomes leader.
Create Raft node 2 & 3 and idle them.
Inform Leader (1) of node 2 & 3 through an AddNode proposal. Let it run and hold an election.
The cluster is lively.
Kill 1
2/3 hold an election, one wins.
1 recovers, begins listening and ticking it's election timeout.

This is where it gets problematic, after recovery the nodes either run into a "node already known" problem when the ConfState is persisted to disk and re-loaded or it forgets about the nodes. That's where I'm stuck at the moment.

Leader (2/3) either contacts it or hears from it and captures it. (the other follower would just redirect it to the current leader)
The cluster is lively again.

So you'd do remove/add when you were decommisioning a host, and if you were just taking a host down for maintenance or a reboot, you can just leave it in the membership pool. :)

I know, there are no calls to add outside of the PUT node/X and no calls to remove at all. That's the mystery.

This is where it gets problematic, after recovery the nodes either run into a "node already known" problem when the ConfState is persisted to disk and re-loaded or it forgets about the nodes. That's where I'm stuck at the moment.

So when 1 recovers and it loads from the hard state, it should already know about 1,2,3 already no? Why is it adding nodes?

That's where I'm stuck. There is no code path in uring that would add the nodes again after a reboot the call to node/<id> is the only way a node change event is issuesd.
Where it explodes is here:
https://github.com/wayfair-incubator/uring/blob/master/src/storage.rs#L223 - the code as it is won't load the nodes from the disk ever, which seems wrong if we never load them why persist but the comment in the  docs suggest this is how raft-rs expects this function to behave (https://github.com/tikv/raft-rs/blob/d34f6936e737563cc930bb8f03970a18f0cba7c9/src/storage.rs#L74). Now following that, the node will forget other nodes after its restarted. Loading the nodes and ignoring the doc linked will lead to a "node already exists" error. So it's just trading one issue for another.

Hmm.... I wrote a test at one point that simulated power cycles with persisted state:
https://github.com/tikv/raft-rs/blob/faa4384d2ed2458279403de1dfef5e160e9ed9f9/harness/tests/integration_cases/test_membership_changes.rs#L1534-L1566
That test suite was for Joint Consensus, but the poweroff/on behavior should be similar.
Can you help me find where you're loading the storage from your save state? When node 1 reboots the RawNode should be passed the old storage at creation.
I see https://github.com/wayfair-incubator/uring/search?q=RawNode%3A%3Anew&unscoped_q=RawNode%3A%3Anew shows loading via UsedStorage which is URRocksStorage. It opens it here: https://github.com/wayfair-incubator/uring/blob/bd7dac7dd986bf087e6c3216c2277ffc06e6d253/src/storage.rs#L126-L132
I see in https://github.com/wayfair-incubator/uring/blob/bd7dac7dd986bf087e6c3216c2277ffc06e6d253/src/storage.rs#L223-L244, perhaps this could be the problem?
According to https://docs.rs/raft/0.6.0-alpha/raft/storage/trait.Storage.html#tymethod.initial_state if the initial state returned in intial_state is initialized this means the storage was created with a configuration. Here's the doc for that function: https://docs.rs/raft/0.6.0-alpha/raft/storage/struct.RaftState.html#method.initialized
Let's look at TiKV for an example: https://github.com/tikv/tikv/blob/dcde0919aec6ea6b2acf4ed4e73c1634fe5e7412/src/raftstore/store/peer_storage.rs#L451-L454
Here is the initial_state there:
https://github.com/tikv/tikv/blob/dcde0919aec6ea6b2acf4ed4e73c1634fe5e7412/src/raftstore/store/peer_storage.rs#L530-L547

So the 'load from disk' is currently commented out: https://github.com/wayfair-incubator/uring/blob/master/src/storage.rs#L230 since it caused crashes for "node already exists".
There are 3 options I tried in there:

the current one (which aside of on the bootstrap node) does always return an ConfState state. This has the effect that restarts do not crash but nodes forget peers which existed before them. (https://github.com/wayfair-incubator/uring/blob/master/src/storage.rs#L224-L228)
the one I tried initially, that persisted and loaded both the ConfState and HardState from disk (https://github.com/wayfair-incubator/uring/blob/master/src/storage.rs#L230-L234). This leads to the "Voter already known ..." on restarts
And one that is as close as I could get to how TiKV does it minus the region code - effectively only loading conf state from disk when the hard state isn't default (https://github.com/wayfair-incubator/uring/blob/master/src/storage.rs#L237-L243) that shares the same issue as #231

The TiKV code and #2/#3 look the same to just that instead of storing voters and learners in a seperate structure it just persists the ConfState state to disk (I tried only taking voters and learners out there is no change) it still causes the "Voter already known..." crash
If we look at where AddNode is called in the codebase

There is only one place (617-633 is a comment: https://github.com/wayfair-incubator/uring/blob/master/src/main.rs#L612-L640)
The place it's called the (Badly named) PutNode message is https://github.com/wayfair-incubator/uring/blob/master/src/main.rs#L488-L504  that is called from the (also badly named, sorry) post_node handler (https://github.com/wayfair-incubator/uring/blob/master/src/main.rs#L128-L141) for
POST /node/<id>.
So I think the request to re-add a node that is already know has to come somewhere from within raft-rs unless I'm missing something.

Sorry I forgot to add context. get_conf_state always reads the conf state from disk https://github.com/wayfair-incubator/uring/blob/bd7dac7dd986bf087e6c3216c2277ffc06e6d253/src/storage.rs#L141-L147 it's not the most performant way to do it removes some caching needs.

So I could see AddNode and RemoveNode commands being replayed if the leader was trying to 'catch up' a node. Is it possible that when the Leader 1 comes back up it's not properly restoring itself?
I see a restore() that we use in several tests that might be useful if you find it's not.

I'd be a bit worried to use restore() directly. It looks like a internal method that should be trigged by the FSM or am I wrong there? And node 1 for example never receives a snapshot so there is nothing to initialise it with, unless I'd store the proposed snapshot somewhere, and that feels just wrong as it would bypass the consensus on it .

restore() also isn't called anywhere in tikv that I can find :(

I think I've solved the mystery. https://github.com/wayfair-incubator/uring/commit/347c3a1b5e010d98ca7879bccd553fa4330e2d06 is the relevant commit.
The issue was caused by raft-rs replaying logs that were already applied to the raft state machine. I would suggest the following:

change the initialisation to not replay old logs
call out in set_hardstate docs that this should truncate the log at 'commit'
Include truncation in the exanples
update the examples to include transferring state along with a snapshot (since that will be needed with log truncation)
make the truncate a callback and part of the Storage logic.

The reason for 1. is that persisting hard state and truncating logs can't be done in an atomic operation (at least not in any storage I can think of). So the 'safe' way of persisting hard state would be: First persist hardstate, then truncate log. That case if the system crashes between the two steps there are logs prior to hard state but they do no harm as they'll be ignored and truncated on the next hard state update.
The reason for 2. is that it is not obvious form any of the examples that this needs to happen, I only found this by going side by side of code & raft-paper to check on all logic and the sentence "The leader decides when it is safe to apply a log entry to the state machine; such an entry is called committed." ticked me off that either logs are not treated as idempotent (which they don't seem to be) and have to be manually truncated.
The reason for 3. this is complicated enough that someone using raft-rs doesn't easily  come to the conclusion of this being required, even with prior knowledge of raft. So having it in the examples, even so the memory nodes can't be restarted, will help adoption.
The reason for 4. the snapshot in the examples doesn't seem to include the 'application' data, with logs being truncated application state can't be recovered via log replay, so it needs to be transfered via snapshot (which is sensible!) but the example relies on log replay to active this which obfuscates that it doesn't work in a real environment.
The reason for 5. without idempotent logs, log truncation is essential part of the raft logic, so it should be handled by the library and not forced onto the user, the implementation of truncation will differ on use case so callbacks seem sensible.
I'm happy to open separate tickets and (for some of the issue PR's)  but wanted to discuss it first.

On a second thought, it feels more correct to treat all raft related logs as idempotent, see section 5.5 of the raft paper - at the point of a restart a node is a follower, and replaying the raft log is effectively replaying the RPC calls to it. Making the assumption ( I suspect raft-rs enforces this) that hard state is only ever committed to disk after the logs are written replaying all logs up to hard state should resolve in the same config state as before (i.e. re-adding a node does no harm during replay, as if it was deleted later that deletion will be replayed as well). The only issue would be if the hard state commit is not larger then the last log (at which point it feels like a good time to hard crash and call that node bricked)

Sleeping over it I get the feeling that truncating in commit might be wasteful. It forces a complete snapshot for every time a leader needs to catch up a follower. While that will work for small working sets it will probably break down quickly. So I think it is back to raft-rs ignoring logs it already committed (or treating them as idempotent) and not replaying them and moving truncation logic into raft-rs as the Leader should eventually get to know the last log committee to all followers

Hi @Licenser , sorry for the radio silence, holiday weekend. :(
I'm glad you seem to have it resolved! I wonder if there is still some lingering problem though. @hicqu do you think you could have a look? Perhaps we missed something?
Specific issue tickets are always welcome, @Licenser, I think that's an easier way to have a discussion about multiple issues. :)


Input/Buffer: Add efficient give/send variants for iterators

Signed-off-by: Moritz Hoffmann<EMAIL_ADDRESS>

Thanks! At first glance, this looks like code specialization that we would hope LLVM is doing for us already. Is there something in the iterator methods that they are doing that LLVM cannot (or will not)?

One would assume so, but it might not be the case:
➜  timely-dataflow git:(input_iterator) cargo run --example exchange --release --<PHONE_NUMBER> -w 2
    Finished release [optimized + debuginfo] target(s) in 0.02s
     Running `target/release/examples/exchange<PHONE_NUMBER> -w 2`
1.874163359s    worker 1 complete; rate: 218550856.08710042
1.87416988s     worker 0 complete; rate: 218550096.82580316
➜  timely-dataflow-master git:(master) ✗ cargo run --example exchange --release --<PHONE_NUMBER> -w 2
    Finished release [optimized + debuginfo] target(s) in 0.02s
     Running `target/release/examples/exchange<PHONE_NUMBER> -w 2`
3.091625675s    worker 1 complete; rate: 132486933.10020721
3.091640853s    worker 0 complete; rate: 132486282.6732056

I can dig into the LLVM IR if I find the time.

So, after setting codegen-units = 1 on master and this branch, the results look different. See for yourself:
➜  timely-dataflow-master git:(master) ✗ perf stat target/release/examples/exchange<PHONE_NUMBER> -w 2
1.807798519s    worker 0 complete; rate: 226573925.90966797
1.807808194s    worker 1 complete; rate: 226572713.3348043

 Performance counter stats for 'target/release/examples/exchange<PHONE_NUMBER> -w 2':

          3,616.52 msec task-clock                #    1.998 CPUs utilized
               129      context-switches          #    0.036 K/sec
                 5      cpu-migrations            #    0.001 K/sec
               965      page-faults               #    0.267 K/sec
    13,426,881,756      cycles                    #    3.713 GHz                      (74.98%)
    36,138,212,227      instructions              #    2.69  insn per cycle           (75.01%)
     7,178,555,006      branches                  # 1984.937 M/sec                    (75.01%)
         5,675,041      branch-misses             #    0.08% of all branches          (75.02%)

       1.810087356 seconds time elapsed

       3.565536000 seconds user
       0.052022000 seconds sys

➜  timely-dataflow git:(input_iterator) ✗ perf stat target/release/examples/exchange<PHONE_NUMBER> -w 2
1.880033636s    worker 0 complete; rate: 217868446.25882998
1.880041397s    worker 1 complete; rate: 217867547.33949047

 Performance counter stats for 'target/release/examples/exchange<PHONE_NUMBER> -w 2':

          3,760.53 msec task-clock                #    1.998 CPUs utilized
               133      context-switches          #    0.035 K/sec
                 3      cpu-migrations            #    0.001 K/sec
               970      page-faults               #    0.258 K/sec
    13,993,357,817      cycles                    #    3.721 GHz                      (74.91%)
    37,953,449,547      instructions              #    2.71  insn per cycle           (74.90%)
     8,854,479,881      branches                  # 2354.580 M/sec                    (75.07%)
         6,138,204      branch-misses             #    0.07% of all branches          (75.12%)

       1.881790208 seconds time elapsed

       3.673465000 seconds user
       0.088035000 seconds sys

This would justify not proceeding with this change as it doesn't cause a (positive) change in performance. From the limited data, I couldn't tell them apart, with the number of instructions indicating that master is better.


OpenEXR split

status:

needs doing IImagePipeline for handling metadata
needs to improve the example

IImageMetadata : public IAssetMetadata

moving to https://github.com/Crisspl/IrrlichtBAW/pull/30


Feat: fromPromise support promise cancelable?

https://github.com/ReactiveX/rxjs/blob/0d78d21643d9d9ce8b1778696b23cb17a2bc2879/src/internal/observable/from.ts#L191-L205
I saw the soure code of fromPromise to Observable, Why not support promise cancelable?
 function fromPromise<T>(promise: PromiseLike<T>) {
   return new Observable((subscriber: Subscriber<T>) => {
     promise
       .then(
         (value) => {
           if (!subscriber.closed) {
             subscriber.next(value);
             subscriber.complete();
           }
         },
         (err: any) => subscriber.error(err)
       )
       .then(null, reportUnhandledError);
   });
   return () => {
     // See https://github.com/sindresorhus/p-cancelable/blob/master/index.js
     if (promise && typeof promise.cacel === 'function') {
       if (!promise.isCanceled) {
            promise.cacel();
       }
     }
   };
 }

There is discussion around to support spec-compliant abort signal (https://github.com/ReactiveX/rxjs/issues/5863). Rxjs don't support non-spec compliant cancel from 3rd party libraries.

There is discussion around to support spec-compliant abort signal (https://github.com/ReactiveX/rxjs/issues/5863). Rxjs don't support non-spec compliant cancel from 3rd party libraries.


Destructo officer.squad update

Changed officer.squad to contain the voice.channel object instead. Reference squad name with
Officer.squad.name

@Hroi can you merge this PR? We discussed this change weeks ago, just haven't gotten around to reviewing it


Added tests for Get-Module over remoting

Some additional additional tests that are remoting specific based on code coverage
Part of https://github.com/PowerShell/PowerShell/issues/4161

@SteveL-MSFT can you please add [feature] tag and trigger the feature test run? The full build triggered by the first commit failed.

@daxian-dbw done

@SteveL-MSFT there is one failure in the new tests. Please fix the test.
Describing Remote module tests
VERBOSE: Using Global AppVeyor Credential
VERBOSE: Using Global AppVeyor Credential
 [-] Error occurred in Describe block 425ms
   Method not found: 'Void System.Security.SecureStringMarshal.ZeroFreeCoTaskMemUnicode(IntPtr)'.
   At C:\projects\powershell\test\tools\Modules\HelpersRemoting\HelpersRemoting.psm1:66 char:20

Please also include [Feature] tag when submitting new commits.

@daxian-dbw that api doesn't exist in our sources.  I suspect it's in MI.dll.

@SteveL-MSFT the error seems to be caused by $cimsession = New-RemoteSession -CimSession. It's thrown from HelpersRemoting.psm1. It possibly reveals a code bug.

I'm pretty sure it's a product issue.  However, it's likely an issue with mi.dll.

Can you please open an issue with a simple repro step? The tests that require $CimSession need to be marked pending.

Created https://github.com/PowerShell/PowerShell/issues/4819

There are 2 test failurs in AppVeryor test run. Can you please fix them?


how to Resume Camera to start scanning again after stopping

starting from v2.0.1, there is pause() and resume() for this.
Also, there is barcodeCapture.refresh(boolean forceRefresh) in case you want to force some new configurations.

Great...Thanks for great library

Can u please release this version to use these latest methods 2.0.1

@kprathap23 it is already available and release.


Service unavailable.

Issue Title
Be as descriptive as you can with your title.
General
Provide details on the problem you are experiencing including the .NET Core version, operating system version and anything else that is relevant.
Please provide a code sample for your issue if it is relevant, either inline, link to a gist (or similar) or add it as a zipped attachment.
For some issues, you will get a quicker and better response if you file it at a more specific .NET repo. For example, if the problem is with ASP.NET Core, you are encouraged to use the aspnet/home repo.

We'll need some specific information on what you're experiencing in order to help. Feel free to re-open this issue with additional details.

I’m really not sure how to use this but I’m dying to learn. What exactly is a repository and where are these commands and/or repos activated? In the url ??

@anlarini are you familiar with GitHub and its workflows? What exactly is the problem you encountered?


fix: always use absolute positioning to prevent jumping Orbit slider

Before and after a slide is animted its position attribute is changed between absolute and relative which can produce a jump of the element after a slide change.
Closes https://github.com/zurb/foundation-sites/issues/11082

@DanielRuf Tell me when you done your tests or if I should take care of them 😉

or if I should take care of them

Some help is really appreciated and would be great.

Yeah, and remove 'position': 'relative' from line 153. But make some tests (with and without motionUI / mui classes), I'm not sure.

I found a bug before figuring out that 'position': 'relative' was not removed. I took care of this.

7daaba5 fix: remove position reset of Orbit slides preventing the "full-absolute" new behavior
All Orbit slides should always have an absolute positioning to prevent jumping Orbit slider (see #11107). The 'position': 'relative' reset is no longer necessary here and break the Orbit behavior as inline styles overrides .orbit-slide class styles.


Kika changed its live stream, update address

Thank you...

Thank you for your report!
The new stream url seems to be<EMAIL_ADDRESS>The url can be updated at https://github.com/cemrich/zapp/blob/master/app/src/main/res/raw/channels.json
This is a nice issue for beginners in Android development 👍  Please claim the issue before you start working on it.

I'm on it

The next update will contain the fixed url.

I can not wait :) ! Thank you!

The updated version 1.8.1 will be released soon on F-Droid.


how should i configure load balancing in docker swarm for aws ?

Closing this ticket due to its age, and the impending refactor. If you think this is in error, feel free to reopen. Thanks!


Teach markdownlint-cli2 to ignore explainaboard/third_party

Currently, the Markdown linter CLI, markdownlint-cli2 is not properly configured to ignore explainaboard/third_party. This is because we used a different type of the configuration file format .markdownlint.yaml, which is a configuration file for the library, markdownlint, not for the CLI. The configuration file doesn't support ignores key. We need to use a configuration file format for the CLI, .markdownlint-cli2.yaml which supports ignoreskey. This PR replaces the .markdownlint.yaml with .markdownlint-cli2.yaml, and fixes the syntax to control rules. (I'll apply similar change in explainaboard_web as well to avoid confusions caused by different config file formats across related repositories)

@neubig This issue blocks #532. Would you review this PR? If this looks good, would you merge this PR for me, and then merge main into third_party_code_for_text_to_sql_evaluation?

It looks this change is somewhat urgent: approved it on behalf of assigned reviewers.

@odashi Thanks!


Stale hard-coded runtime branding

There are stale properties with "2.1.0 - Preview" in them in Branding.props.
I don't think they are displayed anymore, but I did not check the Mac installer.
D:\src\cli\build> findstr /s /c:BrandName *
Branding.props:    <SdkBrandName>Microsoft .NET Core SDK $(CliBrandingVersion)</SdkBrandName>
Branding.props:    <MSBuildExtensionsBrandName>.NET Standard Support for Visual Studio 2015</MSBuildExtensionsBrandName>
Branding.props:    <SharedFrameworkBrandName>Microsoft .NET Core Runtime 2.1.0 - Preview</SharedFrameworkBrandName>
Branding.props:    <SharedHostBrandName>Microsoft .NET Core Host 2.1.0 - Preview</SharedHostBrandName>
Branding.props:    <HostFxrBrandName>Microsoft .NET Core Host FX Resolver 2.1.0 - Preview</HostFxrBrandName>
package\Installer.DEB.targets:        <ReplacementString>$(SdkBrandName)</ReplacementString>
package\Installer.MSI.targets:                      '$(SdkBrandName)'
package\Installer.MSI.targets:                      '$(MSBuildExtensionsBrandName)'
package\Installer.MSI.targets:                      '$(SdkBrandName)'
package\Installer.PKG.targets:        <DistributionTemplateReplacement Include="{CLISdkBrandName}">
package\Installer.PKG.targets:          <ReplacementString>$(SdkBrandName)</ReplacementString>
package\Installer.PKG.targets:        <DistributionTemplateReplacement Include="{SharedFxBrandName}">
package\Installer.PKG.targets:          <ReplacementString>$(SharedFrameworkBrandName)</ReplacementString>
package\Installer.PKG.targets:        <DistributionTemplateReplacement Include="{SharedHostBrandName}">
package\Installer.PKG.targets:          <ReplacementString>$(SharedHostBrandName)</ReplacementString>
package\Installer.PKG.targets:        <DistributionTemplateReplacement Include="{HostFxrBrandName}">
package\Installer.PKG.targets:          <ReplacementString>$(HostFxrBrandName)</ReplacementString>
package\Installer.RPM.targets:        <ReplacementString>$(SdkBrandName)</ReplacementString>

D:\src\cli\packaging> findstr /s /c:BrandName *
osx\clisdk\Distribution-Template:    <title>{CLISdkBrandName} (x64)</title>
osx\clisdk\Distribution-Template:    <choice id="{SharedFxComponentId}.pkg" visible="true" title="{SharedFxBrandName} (x64)" description="The .NET Core Shared Framework">
osx\clisdk\Distribution-Template:    <choice id="{HostFxrComponentId}.pkg" visible="true" title="{HostFxrBrandName} (x64)" description="The .NET Core Host FX Resolver">
osx\clisdk\Distribution-Template:    <choice id="{SharedHostComponentId}.pkg" visible="true" title="{SharedHostBrandName} (x64)" description="The .NET Core Shared Host.">
osx\clisdk\Distribution-Template:    <choice id="{CLISdkComponentId}.pkg" visible="true" title="{CLISdkBrandName} (x64)" description="The .NET Core SDK">

This was found in release/2.1.4xx. If we want to fix this in 3.0 only, then this issue should be moved to dotnet/core-sdk

Sorry, wrong repo already.

This issue was moved to dotnet/cli#9786


Allow indentation in node bodies

At this moment, the editor, on save/load strips all white-space from node bodies. This is annoying when trying to format more complex statement including if-then-else bodies, etc...

Whitespaces are now preserved.


i need help in gridview

in grid view all things overlapping

can anyone give some example in gridview style


fix: click offset issue (#56)

Fixed scrollbar offset when clicking and dragging left mouse cursor.
Closes #56
Tested with Neovim v0.10.1 in Kitty 0.36.1

This makes the comment incorrect.

Right. Removed comment as it no longer holds truth.


LRA-678-lsa-ride-share-reservation-show-page-buttons-design

I set up the reservation show page buttons (LRA-678)
I used nested grids and created the layout described by Maria L's design. I did not use the class buttons that were pre-designed since these buttons required special treatment.

I see that you changed the labels' and buttons' styles and instead of creating CSS classes and using them in the reservations show page you put styles into the page code.
Could you please create new classes for buttons and labels? I tried to merge your branch into mine with more buttons (updating all recurring reservations, etc.)  and had difficulty resolving the show page conflict.

@rsmoke Please try to decrease the screen size - the buttons are not responding very well.

I set the break point to be 1 grid col at lg and less screen sizes

@rsmoke this is what I see on a phone:

Yup, looks good

Yup, looks good

What do you mean 'looks good'? What about the word Update between buttons?

I propose sending the whole button set to be designed better. Then send it
back to me for implementing.

Rick Smoke, Application Architect
For assistance please submit a request at:
https://teamdynamix.umich.edu/TDClient/47/LSAPortal/Requests/TicketRequests/NewForm?ID=lMiqIZLcXJ0_&RequestorType=ServiceOffering
College of Literature, Science, and the Arts | University of Michigan
LSA Technology Services | 301 East Liberty Street, Ann Arbor, MI 48109
On Thu, Oct 5, 2023, 22:22 Margarita Barvinok @.***>
wrote:

Yup, looks good
What do you mean 'looks good'? What about the word Update between buttons?
[image: Screenshot 2023-10-05 at 6 36 05 PM]
https://user-images.githubusercontent.com/56089936/273087215-05eebf69-11c5-4943-96aa-9d229caf4cf9.png
—
Reply to this email directly, view it on GitHub
https://github.com/lsa-mis/lsa_ride_share/pull/339#issuecomment-1749896697,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABMW6NN5X7XA7EROSPE2TLDX55TNRAVCNFSM6AAAAAA5INPIO6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBZHA4TMNRZG4
.
You are receiving this because you were mentioned.Message ID:
@.***>

I fixed the text overflow

@rsmoke I think that on the mobile device buttons should be on the top of the screen not after the reservation fields. Right now they are on the top.
I think we have to ask the users' opinions about buttons. For me, the front is too small on new buttons. Also, it's easier to select the correct button when that are different sizes. When all of them are the same, I have to pay extra attention to select the correct one.


mysqld, mysql

To reproduce
Paste a command that includes the attribute that needs attention.
mysqld

Describe the problem with what is currently being rendered
[x]Missing
[ ]Incorrect
[ ]Poorly worded
[ ]Other: Please describe
Expected description
If known, what is the definition of this command attribute.
Language
[x] English
[ ] Spanish

Definition changed and subcommands added.


Pointgrey(Grassjopper3'sCamera) View Error Occured

Recently, I've succeeded to draw pointgrey's USB-Camera's View on PC.
But Flycapture's configuration screen fall into unstable.
(GUI viewer is very slowly to come on)
It's an message sometimes displayed on Terminal when in above situation.
“libglade-WARNING **: unknown attribute `swapped' for .”
What does it mean? No Swap-territory in PC? Or others?
In case of this error coming from swap-setting, please show me how many territory I have to set swap?
If anyone fall into same torouble, please show me your PC's structure, or your problem solution.
Regs

Thanks so much to view this thread.
I've been to clear above problem.


Customer cannot be attached in orders

-After login to app, go to burger menu>Orders.
-click edit on any order.
-Click select Customer.
-Select any customer from choices offered.
Observed. It does not get added to the order. The field remains empty.

I think this might be a duplicate of #80

The field is assigned but the user has not visual indication

I think this is a case of customer case not shown on the order list page.

Thanks. looking forward to getting this fixed in newer version


Price List doesn't populate on android

On App version 1.2.2
My buddy on IOS is not having the issue, while my other buddy on android is.
Describe the bug
Price List doesn't populate on android
To Reproduce
Steps to reproduce the behavior:

goto price listing it just spins and never returns any data.

Expected behavior
Prices
Screenshots
N/A
Smartphone (please complete the following information):

Device: Pixel 3XL
OS: Android 11
Browser N/A (using app)
Version 1.2.2

Additional context
My buddy has the same issue Android 10,  on Galaxy Note 9, using the latest app.

I must have missed it, when do you expect to release a new build to Google play store?

Thanks for the quick fix btw <3

I must have missed it, when do you expect to release a new build to Google play store?

The new release (v1.2.3) is already live for over 20 hours.


Missing List Object

ksonnet includes a generic list object in k.libsonnet that can be used to combine multiple objects into a single object to send to a kubernetes api server. The implementation for this list object is here.
The List type can be found here for kubernetes.

Hi! Thanks for raising this issue.
While designing the generator of this project, we omitted List types on purpose, as there were quite a lot of them which were all the same but had different names.
We also experienced List to be sort-of an antipattern, because it makes analyzing the resources inside the list harder, without adding any benefits afaict.
From a API and UX perspective, it might still make sense to have a single generic List type, which could be defined in custom/list.libsonnet.
Could you give an example of a use-case where List is required / much better than alternatives?

My setup is a little bit different, as I don't use the ksonnet or tanka binaries. I use bazel for all my apps, so naturally I use bazel to bridge the gap between jsonnet and kubernetes. I use a bazel library called rules_k8s to interface with kubernetes and bazel's rules_jsonnet to transform k.libsonnet into json kubernetes can understand. In rules_k8s, there is a function called k8s_object, the normal way this would work is that you would say add a deployment object as a json / yaml manifest and then you can use bazel commands to apply / delete the object in kubernetes. k8s_object looks for a valid object and it turns out there is a valid k8s meta object -- List.
So long story short, in a jsonnet file I create a local variable using the normal import and merge mechanisms in jsonnet. Then I use a list object to select which manifest objects I would like to use in my k8s_object. Here is an example of the jsonnet:
local k = import 'k.libsonnet';

local m = (import 'memcached.libsonnet') + (import 'configs.libsonnet') +
  {
    _config+: { var: true },
  };

k.core.v1.list.new([
  m.service,
  m.deployment,
]),

It makes sense... I don't think people would use a list of roleBindings in a jsonnet / libsonnet to combine multiple roleBindings :shrug:. But I do think there is merit to having the core List object.
I only use the core List type (so far). I use list.new() and list.items().

Right, that sounds very reasonable to me.
Do you want to contribute this feature yourself? You'd need to create a file called custom/list.libsonnet and add the custom type in there as such:
{
  core+: {
    v1+: {
      list+: {
        new(items):: { /* ... */}
      }
    }
  }
}

That works for me... I'll put in a PR. :smile_cat: Thank you


Add support for channels to docker compose testnet

@mpowaga Could you please add a line in the docs mentioning this new endpoint.
I'll be also grateful if you mention the websocket endpoint in the docs too.

Thanks

Good catch @lucafavatella! Fixed it.


schedule.rb:37-42: New scheduling types like 'at', 'once'...

The puzzle DEV-e234446b from #DEV has to be resolved:
https://github.com/dgroup/lazylead/blob/37c35874b93f82ac20f4c96ec8add5bd15f9c593/lib/lazylead/schedule.rb#L37-L42
The puzzle was created by Yurii Dubinka on 06-Jun-20.
role: DEV.
If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

That's a key feature for app. skeleton during unit testing

The puzzle DEV-e234446b has disappeared from the source code, that's why I closed this issue.


fix: #1826 Changed text color according to mode

What

fix: #1826 Changed text color according to mode

Screenshot

Fixes bug(s)

Fixes: #1826

Part of

#513
#684

Codecov Report

Merging #1845 (40e0c42) into develop (2ea0da3) will decrease coverage by 1.77%.
The diff coverage is 9.64%.

@@            Coverage Diff             @@
##           develop   #1845      +/-   ##
==========================================
- Coverage     8.86%   7.09%   -1.78%
==========================================
  Files          161     174      +13
  Lines         6623    7868    +1245
==========================================
- Hits           587     558      -29
- Misses        6036    7310    +1274

Impacted Files
Coverage Δ

...h_app/lib/cards/category_cards/abstract_cache.dart
0.00% <0.00%> (ø)

...p/lib/cards/category_cards/asset_cache_helper.dart
0.00% <0.00%> (ø)

...p/lib/cards/category_cards/raster_async_asset.dart
0.00% <0.00%> (ø)

...oth_app/lib/cards/category_cards/raster_cache.dart
0.00% <0.00%> (ø)

..._app/lib/cards/category_cards/svg_async_asset.dart
0.00% <0.00%> (ø)

...smooth_app/lib/cards/category_cards/svg_cache.dart
0.00% <0.00%> (ø)

...th_app/lib/cards/data_cards/image_upload_card.dart
0.00% <0.00%> (ø)

...es/smooth_app/lib/cards/data_cards/score_card.dart
4.54% <0.00%> (-0.22%)
:arrow_down:

...t_cards/knowledge_panels/knowledge_panel_card.dart
0.00% <0.00%> (ø)

...knowledge_panels/knowledge_panel_element_card.dart
0.00% <0.00%> (ø)

... and 119 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b8f97a...40e0c42. Read the comment docs.


User who created meet goes to lobby

if user initially creates a meeting ,user is being redirected to lobby even if he is the one who first created the meeting. and it say you will join the meeting as soon as someone accepts your request and asks for password.
turning off lobby , turning off password , or turning off security is not working.

yes i am also facing same problem in flutter

@DP-NOTHING i can fix that

@DP-NOTHING i can fix that

Can you please share solution

solution :
if you don't provide server url then it will use jitsi default server meet.jit.si which require login to be moderator . now there are some community run servers jitsi.github.io/handbook/docs/community/community-instances
which doesn't require login to be moderator . just add serverurl in jitsi configuration with one of the community servers and it will run fine

@MihirPatel1432 solution : if you don't provide server url then it will use jitsi default server meet.jit.si which require login to be moderator . now there are some community run servers jitsi.github.io/handbook/docs/community/community-instances
which doesn't require login to be moderator . just add serverurl in jitsi configuration with one of the community servers and it will run fine

yes this is the exact answer


bug report on version 0.10.2

version: V0.10.2 (STABLE)
operation system:  centos 6.5,   3.14.48-33.39.amzn1.x86_64
unexpected fault address 0x7f8846c4c627
fatal error: fault
[signal 0xb code=0x1 addr=0x7f8846c4c627 pc=0x82eda1]

goroutine 2658085 [running]:
runtime.gothrow(0xbddb60, 0x5)
        /root/.gvm/gos/go1.4.3/src/runtime/panic.go:503 +0x8e fp=0xc217fa0d00 sp=0xc217fa0ce8
runtime.sigpanic()
        /root/.gvm/gos/go1.4.3/src/runtime/sigpanic_unix.go:29 +0x265 fp=0xc217fa0d50 sp=0xc217fa0d00
github.com/influxdb/influxdb/tsdb/engine/tsm1.btou16(0x7f8846c4c626, 0x2, 0x2fcab, 0xc2149a9f40)
        /tmp/influxdb/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/writer.go:631 +0x31 fp=0xc217fa0d58 sp=0xc217fa0d50
github.com/influxdb/influxdb/tsdb/engine/tsm1.func·010(0x48e, 0xb0000c20814009c)
        /tmp/influxdb/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/reader.go:422 +0xe0 fp=0xc217fa0de0 sp=0xc217fa0d58
sort.Search(0x91c, 0xc217fa0f00, 0xc210df4000)
        /root/.gvm/gos/go1.4.3/src/sort/search.go:66 +0x59 fp=0xc217fa0e10 sp=0xc217fa0de0
github.com/influxdb/influxdb/tsdb/engine/tsm1.(*indirectIndex).search(0xc2140dea00, 0xc210df4000, 0x9b, 0xa0, 0xa0)
        /tmp/influxdb/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/reader.go:426 +0x69 fp=0xc217fa0f20 sp=0xc217fa0e10
github.com/influxdb/influxdb/tsdb/engine/tsm1.(*indirectIndex).Entries(0xc2140dea00, 0xc21c7495e0, 0x9b, 0x0, 0x0, 0x0)
        /tmp/influxdb/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/reader.go:458 +0xf8 fp=0xc217fa1020 sp=0xc217fa0f20
github.com/influxdb/influxdb/tsdb/engine/tsm1.(*TSMReader).Entries(0xc21751cd20, 0xc21c7495e0, 0x9b, 0x0, 0x0, 0x0)
        /tmp/influxdb/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/reader.go:290 +0x70 fp=0xc217fa1068 sp=0xc217fa1020
github.com/influxdb/influxdb/tsdb/engine/tsm1.(*FileStore).locations(0xc20a7eaf50, 0xc21c7495e0, 0x9b, 0xece7063fc, 0x8873d75, 0x1035bc0, 0x1, 0x0, 0x0, 0x0)
        /tmp/influxdb/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/file_store.go:470 +0x3e9 fp=0xc217fa1290 sp=0xc217fa1068
github.com/influxdb/influxdb/tsdb/engine/tsm1.(*KeyCursor).init(0xc215622c40, 0xece7063fc, 0x8873d75, 0x1035bc0, 0x1)
        /tmp/influxdb/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/file_store.go:558 +0xa5 fp=0xc217fa1300 sp=0xc217fa1290
github.com/influxdb/influxdb/tsdb/engine/tsm1.(*KeyCursor).SeekTo(0xc215622c40, 0xece7063fc, 0xe08873d75, 0x1035bc0, 0x1035b01, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/influxdb/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/file_store.go:582 +0xa3 fp=0xc217fa1428 sp=0xc217fa1300
github.com/influxdb/influxdb/tsdb/engine/tsm1.(*devCursor).SeekTo(0xc21b15fa20, 0x1439c9772e1e1576, 0xffffffffffffffff, 0x0, 0x0)
        /tmp/influxdb/src/github.com/influxdb/influxdb/tsdb/engine/tsm1/engine.go:737 +0x173 fp=0xc217fa1530 sp=0xc217fa1428
github.com/influxdb/influxdb/tsdb.(*TagsCursor).SeekTo(0xc2160694a0, 0x1439c9772e1e1576, 0x0, 0x0, 0x0)
        /tmp/influxdb/src/github.com/influxdb/influxdb/tsdb/cursor.go:368 +0x61 fp=0xc217fa1570 sp=0xc217fa1530
github.com/influxdb/influxdb/tsdb.readMapInputs(0xc217bac000, 0x7b1f, 0x9800, 0xc213f4cf00, 0x1, 0x1, 0x1439c9772e1e1576, 0x1439c9772e1e1576, 0x1439c9779ccc6c00, 0x0, ...)
        /tmp/influxdb/src/github.com/influxdb/influxdb/tsdb/aggregate.go:835 +0x18e fp=0xc217fa1798 sp=0xc217fa1570
github.com/influxdb/influxdb/tsdb.(*AggregateMapper).NextChunk(0xc2080fa0f0, 0x0, 0x0, 0x0, 0x0)

@andybrier Can you give us more info? How did this happen? Do you get the same error when you try to bring the service back up?
It looks like this might have been related to a query - if so, could you provide the query that triggered this?
Additionally, would you be able to share your data with us for further diagnosis?

@toddboom  I restart, and get the same error now.  The detail file is here:
influxd.log.txt

@andybrier Can you provide us some more information on what kind of data you're writing to the server and how you are submitting it? How many series/tags/fields are you using, and what types of values are you storing the in the fields?
To reiterate @toddboom's question: would you be able to share with us a snapshot of your database?

@joelegasse  I would like to share the snapshot of my database. but how can i get the snapshot?

It happened about 2 or 3 days after i restarted the influxdb.

@andybrier You can just create an archive (tar or zip) of the directories in the database directory (data is the most important, but the others can be included, too)

This is likely fixed in 0.11 via #6082.  The file causing the panic will need to be removed from your data directory since #6082 only fixes writing partial TSM files..


Docs: explain how to use Hapi.js response.status for multiple status codes

Not sure what version Hapi.js introduced the response.status but after trying it out with hapi-swagger today it turns out that Swagger-UI does recognize this option, however the Docs didn't demonstrate this so I decided to write about it.
If there is some limitation with this option, please let me know.
I made sure to document the differences between the response.status and the plugin.hapi-swagger.responses options for full clarification.

Coverage remained the same at 100.0% when pulling 533f3026ba0e34905ff0443d30f269b74e829d89 on feelextra:master into 206a1f9a587c78e1f6b9dff5a4645e80c6ae9482 on glennjones:master.

Coverage remained the same at 100.0% when pulling 533f3026ba0e34905ff0443d30f269b74e829d89 on feelextra:master into 206a1f9a587c78e1f6b9dff5a4645e80c6ae9482 on glennjones:master.


added test JSON server for frontend

created a test JSON test server and script for help of frontend while backend is built out. Feel free to change the schema of JSON while the backend/database is filled out.


Feat: Jumanji register manager

What?
Implement a registry manager class for Jumanji environments. In this case, we check the existence of the given scenario and create the corresponding attributes to initialise the environment class.
Why?
This manager class and the accompanied functions help avoid creating numerous configuration files for each scenario (e.g. for RWARE and LBF).
How?
For the selected environment-scenario pair:

we first check the existence of the JSON file jumanji_scenarios.json and then the existence of env_name with the scenario name.
Create a limited number of scenarios and store them if env_name doesn't exist in the file keys. Otherwise, check if the scenario is valid and extract its attributes if it's already created or create them and add them to the JSON file.
This JSON file will be automatically initialised for the given environment, making it easier to extract attribute names and values to create the env object.

Extra
Teaser: This also helps merge the gym and Jumanji configs for the two common environments used (this will be the focus of a separate PR).
Close #975.

The problem is that for each Jumanji environment we add, we'll have to add a specific function to extract its attributes, which will increase the number of functions added. So we've decided to close this PR for now and use YAML configurations and treat this as an environment specification that hopefully can be handled in Jumanji itself in the future.
Changes will remain on the feat/jumanji_env_register branch for a while 🤞


Language barrier

[ ] Can you make it to support the English Language?

for the roast ?
you can change it on config.json here "TEXT_ROAST" to something like "Please Roast This Instagram Account Politely"
{
"API_KEY_GEMINI" : "xxxxxxxxx",
"TEXT_ROAST" : "Please Roast This Instagram Account Politely"
}


test/cmd/services_spec: fix test.

Was broken by https://github.com/Homebrew/homebrew-services/pull/443

Review period skipped due to critical label.


[hive]Scala code format alarm clear #2835

What is the purpose of the change
Scala code format alarm clear in linkis-engineconn-plugins/hive
Related issues/PRs
Related issues: #2835
Brief change log

Fix some Scala code formats and eliminate warnings
change JavaConversions to JavaConverters

Checklist

[x] I have read the Contributing Guidelines on pull requests.
[x] I have explained the need for this PR and the problem it solves
[x] I have explained the changes or the new features added to this PR
[ ] I have added tests corresponding to this change
[ ] I have updated the documentation to reflect this change
[ ] I have verified that this change is backward compatible (If not, please discuss on the Linkis mailing list first)
[ ] If this is a code change: I have written unit tests to fully verify the new behavior.

Code format check failed
Run 'mvn spotless:apply' to fix these violations.

Code format check failed Run 'mvn spotless:apply' to fix these violations.

It has been fixed, please help to review it again, thank you very much! @casionone

Code format check failed Run 'mvn spotless:apply' to fix these violations.

It has been fixed, please help to review it again, thank you very much! @casionone

Can I add your wechat friends? My wechat account:casinoxia

Code format check failed Run 'mvn spotless:apply' to fix these violations.

It has been fixed, please help to review it again, thank you very much! @casionone

Can I add your wechat friends? My wechat account:casinoxia

sure


How to run this project？

Thank you very much for your project， but I find there is no package.json. Could you please provide this file or explain how to run it？Best wishes to you!

Thank you for recognising my work, but unfortunately this project is quite old and all the code and content has been removed. I'm sorry I can't provide more details.


Define DSL

WHY
Whenever DSL does not accept timezone. That's not useful for us...
WHAT
Define new DSL which is enhanced Whenever DSL. Timezone format is based on tz database.
defaults do
  timezone "Asia/Tokyo"
  cron_timezone "UTC"
end

every :day, at: "10:30 am" do
  name "Update Elasticsearch indices"
  rake "update_elasticsearch_indices"
end

every :day, at: "10:30 am", timezone: "Asia/Singapore" do
  name "Send e-mails for Singapore users"
  rake "send_emails[SG]"
end

every :wednesday, at: "11:00 am" do
  name "Send notifications"
  rake "send_notifications"
end

every "0 12 10,20 * *" do
  name "Send reports"
  rake "send_reports"
end

will be converted to:
30 1 * * * /bin/bash -l -c 'bundle exec rake "update_elasticsearch_indices"'
30 2 * * * /bin/bash -l -c 'bundle exec rake "send_emails[SG]"'
0 11 * * 3 /bin/bash -l -c 'bundle exec rake "send_notifications"'
0 12 10,20 * * /bin/bash -l -c 'bundle exec rake "send_reports"'

Implemented at #3 .


How do I test my contract's constructor?

I want to test that state variables are changed after creating a new contract with constructor arguments and setting state variables inside the constructor. I want to do something like openzeppelin's ERC20Mock.new(args..). (https://github.com/OpenZeppelin/openzeppelin-contracts/blob/master/test/token/ERC20/ERC20.test.js#L23) inside of a test file. However, deployments.fixture doesn't accept deployment arguments I can pass in. What should I do?

Hi!
I don't know why you want to use that specific method... I would suggest you do it like this:

add chai's assert into your test file:
const { assert } = require("chai")

add a constructor describe and a getter function in your contract to get the variables you want to check --> get the variables --> assert the variables are equal to a certain value:
describe("constructor", async function(){
it("sets the state variables correctly, async function() {
const yourStateVariable = await contractName.getterFunctionWithStateVarName();
assert.equal(yourStateVariable, whatYouWantItToBe)
})
})

you can create a specific fixture that deploy a specifically configured contract instance.
the deploy fixture are meant to be used to test the deployment itself, so you would not want to configure it

Also stated in https://github.com/wighawag/hardhat-deploy/issues/289, it would be really useful to be able to pass constructor arguments when loading fixtures, sometime we may need to test the contract with specific arguments.
await deployments.fixture()
await deployments.fixture({ 'myContract': [...args] })

Imagine you have a addresses Leaderboard contract and to test it you need real signers as players. It would be preferable to deploy it with a length of 2 for the tests so that you need 2 signers, and with a length of 100 in production.
for now we have to write a deploy script for each testing case when it comes to arguments

@Sceat for the case you mention you could read the number of signers from a different file for test network,
You can also add tags to network so you can differentiate between test network and real network programmatically.


Project: AVD Session Manager

Project name
AVD Session Manager
Description
AVD Session Manager is a Blazor server application built on Mudblazor and the Azure .NET SDK to more easily control user sessions in Azure Virtual Desktop. This is a sanitized version of my in house production tooling for Microsoft Hack Together 2023.
This app is designed to be CSP/MSP friendly, meaning it authenticates the user to the CSP tenant, and then uses an app registration in the target tenant to interact with AVD. Eventually, this app will allow multitenant management of AVD for MSPs from one central app instead of market alternatives like Nerdio that require an appservice to be stood up in each tenant you manage.
Repo URL
https://github.com/cscaminaci/AVD-Session-Manager
Team members
cscaminaci, mspautomator (me but public repo)

Thank you for submitting your hack @cscaminaci. It doesn't seem though like it's using Microsoft Graph and the Microsoft Graph .NET SDK, which are both prerequisites for this hackathon. Are you planning to extend it with Microsoft Graph?


Optimization 3

#10 16.51 > Build failed because of webpack errors
#10 ERROR: process "/bin/sh -c npm run build" did not complete successfully: exit code: 1
------
> [6/6] RUN npm run build:
16.49 ./src/components/Partners.jsx
16.49 Module not found: Can't resolve<EMAIL_ADDRESS>16.49
16.49 https://nextjs.org/docs/messages/module-not-found
16.49
16.49 Import trace for requested module:
16.49 ./src/app/page.tsx
16.49
16.51
16.51 > Build failed because of webpack errors
------
Dockerfile:17
--------------------
15 |
16 | # Build the Next.js app
17 | >>> RUN npm run build
18 |
19 | # Expose the port the app will run on
--------------------
ERROR: failed to solve: process "/bin/sh -c npm run build" did not complete successfully: exit code: 1
[2024-Jul-12 18:34:11.199090] Deployment failed. Removing the new version of your application.


[Bug]: Disable crucible relentless card when it is unavailable (during iron banner and trials)

Describe the bug
No response
To Reproduce
No response
Expected behavior
No response
Screenshots
No response
Device
No response
Browser
No response

Relentless is available during Iron Banner.

trialsOfOsiris sheet was checked more than 30 times per second and reading from undefined sometimes


Dark mode option.

Hi there
I wondered if you had any plans to incorporate the upstream dark mode, or perhaps those changes are incompatible with your front end updates?
Thanks in advance.

Fair enough, :) thanks for the clarification.

Work on this will be going on at #41 :)

Looking great and super smooth, many thanks.


Add keys step metrics

rerun tests

rerun tests

rerun tests

run skydive-cdd-overview-tests

run skydive-compile-tests


qutebrowser fails to start with webkit backend

The current version of qutebrowser (1.13.0, nixpkgs svn-revision: 234831.e97fdce4e1b) packaged in unstable fails to start with webkit backend.
Forcing the backend in the resulting error dialog does nothing.
To Reproduce
Steps to reproduce the behavior:

set c.backend = "webkit" in ~/.config/qutebrowser/config.py
start qutebrowser

Screenshots

Notify maintainers
@jagajaga @rnhmjoj @ebzzry

The qtwebkit backend was disabled in 8df467ebd6f50eec35f42209e2282df33f32441d. Apparently at the time there was talk about dropping the backend but eventually they decided to keep it for a while. In any case here's what the current README says about it

The latest QtWebKit release is based on old WebKit revision with known unpatched vulnerabilities. Please use it carefully and avoid visiting untrusted websites and using it for transmission of sensitive data.

Soo, we could revert the commit but it seems it's going to be dropped in a couple years anyway.

Forcing the backend in the resulting error dialog does nothing.

This is the reason, it looks like a variable that points to the qutebrowser executable is not actually resolved:
Traceback (most recent call last):
  File "/nix/store/nr58vw4zcr5sfw9xn7zfqxq5w2vpr9zq-qutebrowser-1.11.1/lib/python3.7/site-packages/qutebrowser/misc/quitter.py", line 193, in restart
    subprocess.Popen(args)
  File "/nix/store/vhnb9dk5py14by8rfrb45a0pi1cy4lk9-python3-3.7.6/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/nix/store/vhnb9dk5py14by8rfrb45a0pi1cy4lk9-python3-3.7.6/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '@qutebrowser@': '@qutebrowser@'

I'm pretty new to nix. Let me now if this would be a viable solution: https://github.com/con-f-use/nixpkgs/commit/0e0960f5c30898448a402e6d3ffe1f86ca9a9ffe and I'll make the PR

You can just revert the commit I mentioned: that should take care of also enabling qtwebkit by default, which is probably what you want. To make a PR you should target the master branch, also take a look at the contributing guide.

Hmm... I don't really know, what to make of this part:
  postFixup = lib.optionalString (! withWebEngineDefault) ''
    wrapProgram $out/bin/qutebrowser --add-flags "--backend webkit"
  '';

in the current version it reads:
   postFixup = ''
     wrapProgram $out/bin/qutebrowser \
      "''${gappsWrapperArgs[@]}" \
      "''${qtWrapperArgs[@]}"
   '';

and reverting just the other changes gives me build-errors.

Uhm, try something like this:
   postFixup = ''
     wrapProgram $out/bin/qutebrowser \
      "''${gappsWrapperArgs[@]}" \
      "''${qtWrapperArgs[@]}" \
      ${optionalString (! withWebEngineDefault) "--add-flags '--backend webkit'"}
   '';

Now I get the original import error on starting qutebrowser, when I have withWebEngineDefault set to false.  Maybe https://github.com/con-f-use/nixpkgs/commit/0e0960f5c30898448a402e6d3ffe1f86ca9a9ffe was the right solution after all, because that at least worked.


Strategies for making the Tester more user-friendly

Currently, for using the tester, a user must have to work directly on the code. Some features may make the Tester more user-friendly and could be considered.

We reported one of these strategies in Issue #2. Once the feature there reported is implemented, the user does not need to care about the source data location.

For Test 2, the user could give the following parameters as arguments or the tester could require the user to give this values after the test function is called (the parameters are hard-coded in the current version):

MINIMUM_ALLOWED_NUMBER_CLASSES
PERCENTAGE_INITIAL
PERCENTAGE_FINAL
PERCENTAGE_RATE
NUMBER_OF_EXECUTIONS_PER_DATASET_PER_PERCENTAGE

Two different tests are implemented in the tester as of now: test1 and test2. The parameter -r calls all test functions, but we could separate it in -r1 and -r2 for calling the different tests.
the is_automatic and is_complete OntCatOWL argument is hard coded in the tester. We could keep the is_automatic as it is, but we should provide is_complete as an argument for running the tests (using -c or -n, as in OntCatOWL).

The constants are defined in .env file, the env.example is also provided. Libs that are used for this are typing (for Final) and python-decouple (for config).

3. Two different tests are implemented in the tester as of now: test1 and test2. The parameter -r calls all test functions, but we could separate it in -r1 and -r2 for calling the different tests.

Implemented

Item 4 must be implemented. I set it as necessary in the documentation. Please check the Issue #15.

@mozzherina, please solve this issue only after the re-generation of the dataset.

Done.
2 options:
AUTOMATIC=True
COMPLETE=True
are specified in .env, but the user can also run with -n/-c and -i/-a parameters for both tests.


Memory leak when using EJB Singleton

I am trying to subscribe inside a singleton as defined in SchedulerEvents.java in this gist: https://gist.github.com/chongma/2a3ab451f2aeabc98340a9b897394cfe
It works fine but during development if I change the code then the context will be destroyed and it causes the stack trace log.txt in the gist.  Is there any way to ensure the threads get destroyed?

Seems to work ok if i create a producer like this
import java.io.IOException;
import java.util.concurrent.TimeoutException;

import javax.annotation.Resource;
import javax.enterprise.context.ApplicationScoped;
import javax.enterprise.inject.Disposes;
import javax.enterprise.inject.Produces;

import io.nats.streaming.Options;
import io.nats.streaming.StreamingConnection;
import io.nats.streaming.StreamingConnectionFactory;

@ApplicationScoped
public class NatsConnectionProducer {

	@Resource(name = "baseAddressNats")
	private String baseAddressNats;

	@Produces
	@ApplicationScoped
	public StreamingConnection instance() throws IOException, InterruptedException {
		StreamingConnectionFactory cf = new StreamingConnectionFactory(new Options.Builder().natsUrl(baseAddressNats)
				.clusterId("my-cluster").clientId("my-client-id").build());
		return cf.createConnection();
	}

	public void destroy(@Disposes final StreamingConnection instance)
			throws IOException, TimeoutException, InterruptedException {
		instance.close();
	}
}

and then inject it using CDI into managed class
@Singleton
@Startup
public class SchedulerEvents {
        @Inject
	private StreamingConnection sc;

	@PostConstruct
	private void construct() {
		try {
			sc.subscribe("scheduler:notify", new MessageHandler() {
				@Override
				public void onMessage(Message m) {
					try {
						System.out.printf("Received a message: %s\n", m.getData());
						m.ack();
					} catch (Exception e) {
						e.printStackTrace();
					}
				}
			}, new SubscriptionOptions.Builder().startWithLastReceived().manualAcks().ackWait(Duration.ofSeconds(5))
					.durableName("scheduler-service").build());
		} catch (IOException | InterruptedException | TimeoutException e) {
			e.printStackTrace();
		}
	}
}

There are threading issues with this approach and it seems better to use JCA for this kind of sub pub.  Jon Gallimore at the TomEE team is helping me with an adapter https://github.com/jgallimore/tomee-chatterbox/tree/nats/chatterbox-nats


Query Open - Cases for RunTime Error

Could you please specify which runtime errors can occur on metho Open() of a query?
I can't think of any reason.
The follwowing was mentioned on a PR for the business central baseapp:
https://github.com/microsoft/BusinessCentralApps/pull/239
	public bool ALOpen(DataError errorLevel)
	{
		try
		{
			Open();
			return true;
		}
		catch (NavNCLDialogException) when (errorLevel == DataError.TrapError)
		{
			return false;
		}
		catch (NavALException) when (errorLevel == DataError.TrapError)
		{
			return false;
		}
	}

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: c7c57086-19bb-c29b-c1ee-d6142971947b
Version Independent ID: 7a0ab16a-1940-6220-ee05-a25eb83f95dc
Content: Query.Open() Method - Business Central
Content Source: dev-itpro/developer/methods-auto/query/queryinstance-open-method.md
Service: dynamics365-business-central
GitHub Login: @SusanneWindfeldPedersen
Microsoft Alias: solsen

Thanks for the feedback, but this is out of scope for this reference article - we don't document runtime errors as it is right now. Thanks.


Project Details New Page

New Project page to create a project
Wireframe screenshots
AgMT Components Breakup Figma

Show popup after adding project to add books to it after addding project

@RevantCI , @Joel-C-Johnson  on creating a project , select a source and target language and save it.
Now create another project name and try selecting the same source and target language multiple times, the project gets created. Is this valid.  Can we discuss ?

ii )

The project name  allows a string to have only special characters

-Can allowing invalid characters such as only === in the project name be restricted ? Should we allow only readable characters.

-Update Source Language select to  'Source List' and in the placeholder ' Select Language'  update it to Select  'Source '

Works as expected.
The Source Language List now displays as  SourceList.
The criteria to be followed while creating a Project such as allowing only special characters etc can be taken up for later.