Question: No New Net (2019)

Hi!
My question is about your paper No New Net which cites your previous paper, corresponding to this repository.
Are you using Concatenation to supply feature maps from encoder pathway to decoder pathway of your UNet (as in this repository), or simple Addition which is maybe more memory efficient?

Hi,
I use concatenation. Addition could work just as well, though. I have not compared the two.
Best,
Fabian

Thank you so much for the prompt response. Just have a few more to clear out:

It seems that there are no skips connections/residual blocks or pre-activation blocks in No New Net. Also there is use of Max-Pooling in contrast to Strided Convolution in this repository. Am I on track?
In No New Net, the term Trilinear Up-Sampling is used. Does this refer to Up-Sampling by repetition (as in this repository) or Up-Sampling by linear interpolation?
If it was recorded during implementation, in both cases, what was the memory footprint of your architectures on GPU?

Best Regards,
Talha


macOS NNW 6.0a1 - strange position of setting tabs

I just update NNW to the new alpha version (NNW 6.a1) and I’ve just noticed that in settings, the tabs are in a weird position.
Is this intended?

Same, looks like yes


GitHub Actions tests are failling

https://github.com/microsoft/vscode-python/actions/runs/117232234

@kimadeline @karthiknadig I believe this has been resolved in the PR. Please check and close.

They seem to pass now: https://github.com/microsoft/vscode-python/actions/runs/118242974
Closing, thank you @DonJayamanne 🥳


{libqalculate, qalculate-gtk, qalculate-qt}: 4.8.1 -> 4.9.0

Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Priorities
Add a :+1: reaction to pull requests you find important.

@alyaeanyx also gtk and qt derivations are missing meta.mainProgram

Done!

obforg eval error happens to other PRs

@ofborg eval

@GrahamcOfBorg eval

OK I'll take the almost empty risk of merging without waiting for ofborg - changes seem trivial after all and I can't trigger it.


Control key incorrectly detected on macOS

The function checking for the control key is actually checking for the Command (⌘) key on macOS:  https://github.com/dvargas92495/roam-js-extensions/blob/9d6cff39d3ef46468c0a044ceaa2596c3a718840/src/entry-helpers.ts#L754-L755
The documentation for the mouseless and todont extension say that the keyboard extension uses the Control key.
Either the function should check for the Control key (by just removing the Apple-specific logic in the above function and check for e.ctrlKey) or the documentation should reflect the different behavior on macOS.
My preference would be to use the Control key. I'm happy to submit a PR to fix it.

I think I want to update the documentation - I was told by previous mac users to use cmd which is where this change came from. Particularly for todont, I want it to mirror the TODO/DONE functionality.
Thanks for pointing it out!


Attempt to negate with overflow in ld_reg_str of disassembler

The disassembler would panic in ld_reg_str when it tries to negate the 0x8000i16 value in -insn.off
https://github.com/qmonnet/rbpf/blob/4812c52fe1009ee1e3c2307662d175bdd07b5c71/src/disassembler.rs#L37-L42
The PoC program to reproduce:
disassembler::disassemble(&[113, 1, 0, 128, 0, 0, 31, 145])

This would encounter the following panic:
thread '<unnamed>' panicked at 'attempt to negate with overflow'

Noted that, this is issue is also encountered in st_reg_str, jmp_imm_str, jmp_reg_str functions.
To enhance the robustness of the disassembler, the negation logic of i16 could be restructured in those function.

Addressed in https://github.com/qmonnet/rbpf/pull/93.


Conformance Tests Kokoro job is failing

Looks like PRs have been blocked since March: https://github.com/googleapis/conformance-tests/pull/43
Failure:
.kokoro/validator.sh
/tmp ~/Documents/conformance-tests
Cloning into 'googleapis'...
remote: Enumerating objects: 5917, done.
remote: Counting objects: 100% (5917/5917), done.
remote: Compressing objects: 100% (2198/2198), done.
remote: Total 5917 (delta 4385), reused 4175 (delta 3616), pack-reused 0
Receiving objects: 100% (5917/5917), 4.13 MiB | 3.88 MiB/s, done.
Resolving deltas: 100% (4385/4385), done.
~/Documents/conformance-tests
~/Documents/conformance-tests/storage/v1 ~/Documents/conformance-tests
/go/include: warning: directory does not exist.
protoc-gen-go: unable to determine Go import path for "google/cloud/conformance/storage/v1/tests.proto"

Please specify either:
        • a "go_package" option in the .proto source file, or
        • a "M" argument on the command line.

See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.

Hi @tbpg and @codyoss,
@tritone recommended I pull y'all in on what we could potentially do to unblock the kokoro failures. (I'm going on leave so Chris will be taking my place, just wanted to to start the thread).
Cheers!

The protoc invocation is here: https://github.com/googleapis/conformance-tests/blob/e128d775153634f7ae3abfdd5aa6eeb74710086f/.kokoro/validator.sh#L42-L46
The proto gets imported here: https://github.com/googleapis/conformance-tests/blob/e128d775153634f7ae3abfdd5aa6eeb74710086f/storage/v1/validator.go#L25
We shouldn't be using a relative import path. We should probably put a go.mod in the root of the directory and use a real import path (github.com/googleapis/conformance-tests plus whatever the proto is).
Then we can pass the import path to protoc as described in the error.
I don't know why the proto paths are the way they are (https://github.com/googleapis/conformance-tests/blob/master/storage/v1/proto/google/cloud/conformance/storage/v1/tests.proto). Why do we need the proto/google/cloud/conformance/storage/v1 part of the path? Is it relevant somehow? Would be fine to keep, it's just surprisingly long. It matches the proto package declared in the .proto file: https://github.com/googleapis/conformance-tests/blob/e128d775153634f7ae3abfdd5aa6eeb74710086f/storage/v1/proto/google/cloud/conformance/storage/v1/tests.proto#L17

I sent #47 to hopefully get everything running again.


Sequence number caching in Hermes is unreliable

Crate
Hermes
Summary
Query the state for every chain in config every X sec for the relayer account's seq number, and update cached seq number with it. Make X a configurable param.
Problem Definition
At some point (around height 1008 on Cosmos hub), for some reason, the cached seq number on our relayer instance got incremented, but the actual on-chain seq number wasn't. From there, our relayer hasn't been able to relay, producing the following error:
Aug 06 15:51:41.681  INFO [irishub-1:transfer/channel-12 -> cosmoshub-4] result events:
    ChainErrorEv(deliver_tx on chain cosmoshub-4 for Tx hash EC50AD139769F2585F351CDE69636B0C82F522A5B29ADDD1C279737088F4E098 reports error: code=Err(32), log=Log("account sequence mismatch, expected 1008, got 1358: incorrect account sequence"))

While we don't know what caused this mismatch to happen, we couldn't detect it and thought our relayer was running properly. It would be good to figure out why the mismatch happened, but it would also be good to have a failsafe that updates the cached seq number with the actual on-chain seq number periodically so that the relayer can recover on its own.

For Admin Use

[ ] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate milestone (priority) applied
[ ] Appropriate contributors tagged
[ ] Contributor assigned/self-assigned

I've done a preliminary investigation. Here's some thoughts on what might be happening
The sequence is incremented at the of send_tx here https://github.com/informalsystems/ibc-rs/blob/142c7dc29c7636d1c9ece51ce8fd6f7c50ca3477/relayer/src/chain/cosmos.rs#L384
then the response from the broadcast_tx_sync is returned from send_tx but there are no checks or validation on the response at this stage, so I'm assuming if the response has an "unsuccessful" or "unexpected" response the internal sequence increments anyways so this might potentially cause some situations that the account sequence will mismatch because the internal sequence is ahead of the sequence on chain.
I believe the sequence is only retrieved once and cached here https://github.com/informalsystems/ibc-rs/blob/142c7dc29c7636d1c9ece51ce8fd6f7c50ca3477/relayer/src/chain/cosmos.rs#L521
An account is returned in the logic below and the sequence is retrieved from this account
https://github.com/informalsystems/ibc-rs/blob/142c7dc29c7636d1c9ece51ce8fd6f7c50ca3477/relayer/src/chain/cosmos.rs#L544
I believe this logic to cache the sequence was implemented to improve performance since for every transaction you'd need to make an additional query to get the sequence and could slow things down.
Maybe another approach is to have an internal retry counter or logic to "refresh" the sequence before all retries are exhausted and Hermes gives up on that transaction.
Looks like that if you re-start Hermes then it's able to relay the transaction because the sequence might be retrieved from the chain.

I think what me might try to do in Hermes is to parse the exact error message and update the cached sequence number on-the-fly, only when an error occurs. The message looks as follows, and should be easy for Hermes to interpret and update the cached value:
	ChainErrorEv(deliver_tx on chain test-1 for Tx hash 21C1871B90E00523BD6E4A42E4F5AD747714D760EAB77595946FE31BAC24F4F3 reports error: code=Err(32), log=Log("account sequence mismatch, expected 8, got 7: incorrect account sequence"))

I managed to reproduce this issue using a slightly more complex scenario, with help from Greg and Mircea. The goal we're trying to reach in the scenario I'll describe below, which mimics production issues, is as follows:

Hermes is connected to a full node a, say rpc_addr = 'a', on network X
Suppose this node accepts a transaction tx1 from Hermes, and this transaction enters the mempool of node a
This transaction tx1 will not be deliver-ed (for reasons we'll detail below), and will remain in the mempool of node a instead
Now if Hermes attempts to submit another transaction tx2 to this node, the node will reject this transaction on grounds of having incorrect sequence number; the log will report:

Sep 20 12:36:11.334 DEBUG [a] send_tx: retrieved account sequence=8 number=1
Sep 20 12:36:11.334 DEBUG [a] send_tx: sending 2 messages using nonce 8
Sep 20 12:36:11.351 TRACE [a] send_tx: based on the estimated gas, adjusting fee from Fee { amount: [Coin { denom: "stake", amount: "500" }], gas_limit: 500000, payer: "", granter: "" } to Fee { amount: [Coin { denom: "stake", amount: "500" }], gas_limit: 500000, payer: "", granter: "" }
Sep 20 12:36:11.360 DEBUG [a] send_tx: broadcast_tx_sync: Response { code: Err(32), data: Data([]), log: Log("account sequence mismatch, expected 9, got 8: incorrect account sequence"), hash: transaction::Hash(7BF91E0641594119F1E29BF5E9DBACD7F3152603F94A13727D924188716AD080) }
Sep 20 12:36:11.361  INFO [Async~>a] response(s): 1; Err(32):7BF91E0641594119F1E29BF5E9DBACD7F3152603F94A13727D924188716AD080

Sep 20 12:36:11.361  INFO [b:transfer/channel-0 -> a] success
Sep 20 12:36:11.361 TRACE [a:transfer/channel-0 -> b] putting error response in error event queue: Response { code: Err(32), data: Data([]), log: Log("account sequence mismatch, expected 9, got 8: incorrect account sequence"), hash: transaction::Hash(7BF91E0641594119F1E29BF5E9DBACD7F3152603F94A13727D924188716AD080) }
Sep 20 12:36:11.361 TRACE Packet worker produced relay summary: RelaySummary { events: [ChainError("deliver_tx on chain a for Tx hash 7BF91E0641594119F1E29BF5E9DBACD7F3152603F94A13727D924188716AD080 reports error: code=Err(32), log=Log(\"account sequence mismatch, expected 9, got 8: incorrect account sequence\")")] }

Notice that the sequence number fetched is 8, but then the node rejects the submitted tx reporting that it should use s.n. 9. The explanation is that s.n. 8 is already used by an earlier transaction, which is stuck in the mempool. At this point, Hermes is unable to make further progress, because it cannot rely on the fetched sequence number 8.
The only solution we know to solve this is to flush the mempool of this node. Even this solution will partially fix it, because this same node might have broadcasted the problematic tx (stuck with s.n. 8) to other nodes, which may broadcast it back to our node.
To reproduce this problem, the steps are as follows, using gm.

use this config file as your gm.toml: https://gist.github.com/adizere/fe9542b764ceed891e7a311f40de7c44
gm start and then gm stop to generate all the necessary config and genesis files
set minimum-gas-prices = "0.5stake" in file .gm/a/config/app.toml

by default, this should be minimum-gas-prices = ""

gm reset
gm start
gm hermes config
gm hermes keys
hermes create client a b

this should fail with "Failed Tx: no confirmation"
the explanation is that the full node (called node-1 in the gm.toml) which receives the CreateClient transaction from Hermes will accept this transaction, but the network will reject the transaction
the network is configured with a certain minimum gas price (see step 3 above) which the node is not respecting
the tx will remain stuck in the node's mempool

hermes create client a b

now we should see the sequence number problem occurring:

Sep 20 16:41:04.138  WARN [a] waiting for commit of tx hashes(s) 431E4AF496A1DB8280E745F33EC73BCA032DDA8B978C0501A7A6DC8D7E7B844E
Success: ChainError(
"deliver_tx on chain a for Tx hash 431E4AF496A1DB8280E745F33EC73BCA032DDA8B978C0501A7A6DC8D7E7B844E reports error: code=Err(32), log=Log("account sequence mismatch, expected 1, got 0: incorrect account sequence")",
)

Would it be useful if Tendermint full-nodes communicated a cryptographic commitment/hash of their (consensus-related) config to other nodes and warned users if there was a mismatch?

I think the problem is that there are multiple levels of configurations. The tendermint-specific configuration is not at fault here, but the app.toml -- the application-specific configuration -- is causing transactions hanging in the mempool in the scenario outlined above. Not sure if a commitment proof would help there. But note that there may be different reasons why tx-es could remain stuck in the mempool (connectivity issues, misbehavior).

Looks like this goes a bit deeper. I've just discovered that the sequence number issue comes up even with a fully synced node. Not sure why. I've attached a couple of log snippets. This log shows the timeline from a successful osmosis node TX to the next TX which now throws a sequence number error.
2021-12-03T16:15:44.623338Z DEBUG ThreadId(60) [osmosis-1] send_tx: sending 2 messages using account sequence 166792
 2021-12-03T16:15:44.623860Z DEBUG ThreadId(60) [osmosis-1] send_tx: max fee, for use in tx simulation: Fee { amount: "501uosmo", gas_limit: 5000000 }
 2021-12-03T16:15:44.966066Z DEBUG ThreadId(60) [osmosis-1] estimate_gas: tx simulation successful, gas amount used: 1358253
 2021-12-03T16:15:44.966130Z DEBUG ThreadId(60) [osmosis-1] send_tx: using 1358253 gas, fee Fee { amount: "150uosmo", gas_limit: 1494079 }
 2021-12-03T16:15:45.155148Z DEBUG ThreadId(60) [osmosis-1] send_tx: broadcast_tx_sync: Response { code: Ok, data: Data([]), log: Log("[]"), hash: transaction::Hash(C819745777EA7921B9E45FB7463C4F8DFEF08F4583E0933EBCB7FB4DC22E5D6C) }
 2021-12-03T16:15:45.155241Z  INFO ThreadId(332) [Async~>osmosis-1] response(s): 1; Ok:C819745777EA7921B9E45FB7463C4F8DFEF08F4583E0933EBCB7FB4DC22E5D6C

 2021-12-03T16:15:45.155259Z  INFO ThreadId(332) [sentinelhub-2:transfer/channel-0 -> osmosis-1] success
 2021-12-03T16:15:47.961640Z  INFO ThreadId(68) [core-1:transfer/channel-6 -> osmosis-1] clearing pending packets height=Some(Height { revision: 1, height: 3752199 })
 2021-12-03T16:15:49.326799Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] generate messages from batch with 1 events
 2021-12-03T16:15:49.328207Z DEBUG ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] juno-1 => SendPacketEv(SendPacket - h:1-888329, seq:27166, path:channel-0/transfer->channel-42/transfer, toh:1-2241526, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:15:49.729998Z DEBUG ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] osmosis-1 <= /ibc.core.channel.v1.MsgRecvPacket from SendPacketEv(SendPacket - h:1-888329, seq:27166, path:channel-0/transfer->channel-42/transfer, toh:1-2241526, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:15:49.730049Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] scheduling op. data with 1 msg(s) for Destination (height 1-888330)
 2021-12-03T16:15:49.764730Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] relay op. data of 1 msgs(s) to Destination (height 1-888330), delayed by: 34.671901ms [try 1/5]
 2021-12-03T16:15:49.764773Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] prepending Destination client update @ height 1-888330
 2021-12-03T16:15:49.881922Z  INFO ThreadId(68) [core-1:transfer/channel-6 -> osmosis-1] finished scheduling pending packets clearing height=Some(Height { revision: 1, height: 3752199 })
 2021-12-03T16:15:49.956622Z  WARN ThreadId(84) [juno-1 -> osmosis-1:07-tendermint-1457] src header Timestamp(2021-12-03T16:15:42.368880883+00:00) is after dst latest header Timestamp(2021-12-03T16:14:05.968261873+00:00) + client state drift 5s, wait for next height on osmosis-1
 2021-12-03T16:15:50.258558Z ERROR ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] worker: schedule execution encountered error: failed during a client operation: update header from juno-1 with height 1-888330 and time Timestamp(2021-12-03T16:15:42.368880883+00:00)
is in the future compared with latest header on osmosis-1 with height 1-2241409 and time Timestamp(2021-12-03T16:14:12.379102539+00:00), adjusted with drift 5s
 2021-12-03T16:15:51.258694Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] generate messages from batch with 1 events
 2021-12-03T16:15:51.260064Z DEBUG ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] juno-1 => SendPacketEv(SendPacket - h:1-888329, seq:27166, path:channel-0/transfer->channel-42/transfer, toh:1-2241526, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:15:51.332890Z DEBUG ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] osmosis-1 <= /ibc.core.channel.v1.MsgRecvPacket from SendPacketEv(SendPacket - h:1-888329, seq:27166, path:channel-0/transfer->channel-42/transfer, toh:1-2241526, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:15:51.332928Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] scheduling op. data with 1 msg(s) for Destination (height 1-888330)
 2021-12-03T16:15:51.380005Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] relay op. data of 1 msgs(s) to Destination (height 1-888330), delayed by: 47.068267ms [try 1/5]
 2021-12-03T16:15:51.380046Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] prepending Destination client update @ height 1-888330
 2021-12-03T16:15:51.446319Z  INFO ThreadId(85) [osmosis-1:transfer/channel-42 -> juno-1] generate messages from batch with 1 events
 2021-12-03T16:15:51.494865Z DEBUG ThreadId(85) [osmosis-1:transfer/channel-42 -> juno-1] osmosis-1 => WriteAcknowledgementEv(WriteAcknowledgement - h:1-2241408, seq:27164, path:channel-0/transfer->channel-42/transfer, toh:1-2241550, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:15:51.495907Z  WARN ThreadId(84) [juno-1 -> osmosis-1:07-tendermint-1457] src header Timestamp(2021-12-03T16:15:42.368880883+00:00) is after dst latest header Timestamp(2021-12-03T16:14:12.379102539+00:00) + client state drift 5s, wait for next height on osmosis-1
 2021-12-03T16:15:51.497732Z DEBUG ThreadId(85) [osmosis-1:transfer/channel-42 -> juno-1] WriteAcknowledgement - h:1-2241408, seq:27164, path:channel-0/transfer->channel-42/transfer, toh:1-2241550, tos:Timestamp(NoTimestamp)) already handled
 2021-12-03T16:15:51.797838Z ERROR ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] worker: schedule execution encountered error: failed during a client operation: update header from juno-1 with height 1-888330 and time Timestamp(2021-12-03T16:15:42.368880883+00:00)
is in the future compared with latest header on osmosis-1 with height 1-2241410 and time Timestamp(2021-12-03T16:14:18.398713323+00:00), adjusted with drift 5s
 2021-12-03T16:15:52.807991Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] generate messages from batch with 1 events
 2021-12-03T16:15:52.809328Z DEBUG ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] juno-1 => SendPacketEv(SendPacket - h:1-888329, seq:27166, path:channel-0/transfer->channel-42/transfer, toh:1-2241526, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:15:53.367801Z DEBUG ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] osmosis-1 <= /ibc.core.channel.v1.MsgRecvPacket from SendPacketEv(SendPacket - h:1-888329, seq:27166, path:channel-0/transfer->channel-42/transfer, toh:1-2241526, tos:Timestamp(NoTime
stamp)))
 2021-12-03T16:15:53.367846Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] scheduling op. data with 1 msg(s) for Destination (height 1-888330)
 2021-12-03T16:15:53.374377Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] relay op. data of 1 msgs(s) to Destination (height 1-888330), delayed by: 6.520016ms [try 1/5]
 2021-12-03T16:15:53.374429Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] prepending Destination client update @ height 1-888330
 2021-12-03T16:15:53.427802Z  INFO ThreadId(85) [osmosis-1:transfer/channel-42 -> juno-1] generate messages from batch with 1 events
 2021-12-03T16:15:53.466756Z DEBUG ThreadId(85) [osmosis-1:transfer/channel-42 -> juno-1] osmosis-1 => SendPacketEv(SendPacket - h:1-2241409, seq:34145, path:channel-42/transfer->channel-0/transfer, toh:1-888805, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:15:53.467068Z  WARN ThreadId(84) [juno-1 -> osmosis-1:07-tendermint-1457] src header Timestamp(2021-12-03T16:15:42.368880883+00:00) is after dst latest header Timestamp(2021-12-03T16:14:18.398713323+00:00) + client state drift 5s, wait for next height on osmosis-1
 2021-12-03T16:15:53.468493Z DEBUG ThreadId(85) [osmosis-1:transfer/channel-42 -> juno-1] SendPacket - h:1-2241409, seq:34145, path:channel-42/transfer->channel-0/transfer, toh:1-888805, tos:Timestamp(NoTimestamp)) already handled
 2021-12-03T16:15:53.768628Z ERROR ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] worker: schedule execution encountered error: failed during a client operation: update header from juno-1 with height 1-888330 and time Timestamp(2021-12-03T16:15:42.368880883+00:00) is in the future compared with latest header on osmosis-1 with height 1-2241411 and time Timestamp(2021-12-03T16:14:25.221697141+00:00), adjusted with drift 5s
 2021-12-03T16:15:54.788764Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] generate messages from batch with 1 events
 2021-12-03T16:15:54.790183Z DEBUG ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] juno-1 => SendPacketEv(SendPacket - h:1-888329, seq:27166, path:channel-0/transfer->channel-42/transfer, toh:1-2241526, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:15:54.890601Z DEBUG ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] osmosis-1 <= /ibc.core.channel.v1.MsgRecvPacket from SendPacketEv(SendPacket - h:1-888329, seq:27166, path:channel-0/transfer->channel-42/transfer, toh:1-2241526, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:15:54.890679Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] scheduling op. data with 1 msg(s) for Destination (height 1-888330)
 2021-12-03T16:15:55.132114Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] relay op. data of 1 msgs(s) to Destination (height 1-888330), delayed by: 241.423197ms [try 1/5]
 2021-12-03T16:15:55.132155Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] prepending Destination client update @ height 1-888330
 2021-12-03T16:15:55.228690Z  WARN ThreadId(84) [juno-1 -> osmosis-1:07-tendermint-1457] src header Timestamp(2021-12-03T16:15:42.368880883+00:00) is after dst latest header Timestamp(2021-12-03T16:14:25.221697141+00:00) + client state drift 5s, wait for next height on osmosis-1
 2021-12-03T16:15:55.530461Z ERROR ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] worker: schedule execution encountered error: failed during a client operation: update header from juno-1 with height 1-888330 and time Timestamp(2021-12-03T16:15:42.368880883+00:00) is in the future compared with latest header on osmosis-1 with height 1-2241412 and time Timestamp(2021-12-03T16:14:31.319009583+00:00), adjusted with drift 5s
 2021-12-03T16:15:56.560599Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] generate messages from batch with 1 events
 2021-12-03T16:15:56.561915Z DEBUG ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] juno-1 => SendPacketEv(SendPacket - h:1-888329, seq:27166, path:channel-0/transfer->channel-42/transfer, toh:1-2241526, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:15:56.568240Z DEBUG ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] osmosis-1 <= /ibc.core.channel.v1.MsgRecvPacket from SendPacketEv(SendPacket - h:1-888329, seq:27166, path:channel-0/transfer->channel-42/transfer, toh:1-2241526, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:15:56.568281Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] scheduling op. data with 1 msg(s) for Destination (height 1-888330)
 2021-12-03T16:15:56.573545Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] relay op. data of 1 msgs(s) to Destination (height 1-888330), delayed by: 5.255016ms [try 1/5]
 2021-12-03T16:15:56.573583Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] prepending Destination client update @ height 1-888330
 2021-12-03T16:15:56.675096Z  WARN ThreadId(84) [juno-1 -> osmosis-1:07-tendermint-1457] src header Timestamp(2021-12-03T16:15:42.368880883+00:00) is after dst latest header Timestamp(2021-12-03T16:14:37.552683596+00:00) + client state drift 5s, wait for next height on osmosis-1
 2021-12-03T16:15:58.486070Z ERROR ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] worker: schedule execution encountered error: failed during a client operation: update header from juno-1 with height 1-888330 and time Timestamp(2021-12-03T16:15:42.368880883+00:00) is in the future compared with latest header on osmosis-1 with height 1-2241414 and time Timestamp(2021-12-03T16:14:48.558922254+00:00), adjusted with drift 5s
 2021-12-03T16:15:59.526240Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] generate messages from batch with 1 events
 2021-12-03T16:15:59.527821Z DEBUG ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] juno-1 => SendPacketEv(SendPacket - h:1-888329, seq:27166, path:channel-0/transfer->channel-42/transfer, toh:1-2241526, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:15:59.654127Z DEBUG ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] osmosis-1 <= /ibc.core.channel.v1.MsgRecvPacket from SendPacketEv(SendPacket - h:1-888329, seq:27166, path:channel-0/transfer->channel-42/transfer, toh:1-2241526, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:15:59.654176Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] scheduling op. data with 1 msg(s) for Destination (height 1-888330)
 2021-12-03T16:15:59.712119Z  INFO ThreadId(68) [core-1:transfer/channel-6 -> osmosis-1] generate messages from batch with 1 events
 2021-12-03T16:15:59.761218Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] relay op. data of 1 msgs(s) to Destination (height 1-888330), delayed by: 107.031714ms [try 1/5]
 2021-12-03T16:15:59.761251Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] prepending Destination client update @ height 1-888330
 2021-12-03T16:15:59.762621Z DEBUG ThreadId(68) [core-1:transfer/channel-6 -> osmosis-1] core-1 => WriteAcknowledgementEv(WriteAcknowledgement - h:1-3752202, seq:16110, path:channel-4/transfer->channel-6/transfer, toh:1-3752345, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:15:59.795814Z DEBUG ThreadId(68) [core-1:transfer/channel-6 -> osmosis-1] WriteAcknowledgement - h:1-3752202, seq:16110, path:channel-4/transfer->channel-6/transfer, toh:1-3752345, tos:Timestamp(NoTimestamp)) already handled
 2021-12-03T16:15:59.892870Z  WARN ThreadId(84) [juno-1 -> osmosis-1:07-tendermint-1457] src header Timestamp(2021-12-03T16:15:42.368880883+00:00) is after dst latest header Timestamp(2021-12-03T16:14:48.558922254+00:00) + client state drift 5s, wait for next height on osmosis-1
 2021-12-03T16:16:00.797795Z ERROR ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] worker: schedule execution encountered error: failed during a client operation: update header from juno-1 with height 1-888330 and time Timestamp(2021-12-03T16:15:42.368880883+00:00) is in the future compared with latest header on osmosis-1 with height 1-2241415 and time Timestamp(2021-12-03T16:14:54.759623082+00:00), adjusted with drift 5s
 2021-12-03T16:16:01.636967Z  INFO ThreadId(85) [osmosis-1:transfer/channel-42 -> juno-1] generate messages from batch with 1 events
 2021-12-03T16:16:01.638348Z DEBUG ThreadId(85) [osmosis-1:transfer/channel-42 -> juno-1] osmosis-1 => SendPacketEv(SendPacket - h:1-2241414, seq:34146, path:channel-42/transfer->channel-0/transfer, toh:1-888469, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:16:01.640235Z DEBUG ThreadId(85) [osmosis-1:transfer/channel-42 -> juno-1] SendPacket - h:1-2241414, seq:34146, path:channel-42/transfer->channel-0/transfer, toh:1-888469, tos:Timestamp(NoTimestamp)) already handled
 2021-12-03T16:16:01.847953Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] generate messages from batch with 1 events
 2021-12-03T16:16:01.849410Z DEBUG ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] juno-1 => SendPacketEv(SendPacket - h:1-888329, seq:27166, path:channel-0/transfer->channel-42/transfer, toh:1-2241526, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:16:02.532325Z DEBUG ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] SendPacket - h:1-888329, seq:27166, path:channel-0/transfer->channel-42/transfer, toh:1-2241526, tos:Timestamp(NoTimestamp)) already handled
 2021-12-03T16:16:02.643907Z  INFO ThreadId(85) [osmosis-1:transfer/channel-42 -> juno-1] generate messages from batch with 1 events
 2021-12-03T16:16:02.645690Z DEBUG ThreadId(85) [osmosis-1:transfer/channel-42 -> juno-1] osmosis-1 => WriteAcknowledgementEv(WriteAcknowledgement - h:1-2241415, seq:27165, path:channel-0/transfer->channel-42/transfer, toh:1-2241553, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:16:02.648494Z DEBUG ThreadId(85) [osmosis-1:transfer/channel-42 -> juno-1] WriteAcknowledgement - h:1-2241415, seq:27165, path:channel-0/transfer->channel-42/transfer, toh:1-2241553, tos:Timestamp(NoTimestamp)) already handled
 2021-12-03T16:16:07.729850Z  INFO ThreadId(957) [osmosis-1:transfer/channel-4 -> core-1] generate messages from batch with 1 events
 2021-12-03T16:16:07.731401Z DEBUG ThreadId(957) [osmosis-1:transfer/channel-4 -> core-1] osmosis-1 => SendPacketEv(SendPacket - h:1-2241419, seq:16110, path:channel-4/transfer->channel-6/transfer, toh:1-3752345, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:16:07.733539Z DEBUG ThreadId(957) [osmosis-1:transfer/channel-4 -> core-1] SendPacket - h:1-2241419, seq:16110, path:channel-4/transfer->channel-6/transfer, toh:1-3752345, tos:Timestamp(NoTimestamp)) already handled
 2021-12-03T16:16:09.589043Z  INFO ThreadId(322) [osmosis-1:transfer/channel-2 -> sentinelhub-2] generate messages from batch with 1 events
 2021-12-03T16:16:09.591219Z DEBUG ThreadId(322) [osmosis-1:transfer/channel-2 -> sentinelhub-2] osmosis-1 => WriteAcknowledgementEv(WriteAcknowledgement - h:1-2241420, seq:25241, path:channel-0/transfer->channel-2/transfer, toh:1-2241879, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:16:09.593793Z DEBUG ThreadId(322) [osmosis-1:transfer/channel-2 -> sentinelhub-2] WriteAcknowledgement - h:1-2241420, seq:25241, path:channel-0/transfer->channel-2/transfer, toh:1-2241879, tos:Timestamp(NoTimestamp)) already handled
 2021-12-03T16:16:13.968157Z  INFO ThreadId(85) [osmosis-1:transfer/channel-42 -> juno-1] generate messages from batch with 1 events
 2021-12-03T16:16:13.969958Z DEBUG ThreadId(85) [osmosis-1:transfer/channel-42 -> juno-1] osmosis-1 => WriteAcknowledgementEv(WriteAcknowledgement - h:1-2241423, seq:27166, path:channel-0/transfer->channel-42/transfer, toh:1-2241526, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:16:13.973030Z DEBUG ThreadId(85) [osmosis-1:transfer/channel-42 -> juno-1] WriteAcknowledgement - h:1-2241423, seq:27166, path:channel-0/transfer->channel-42/transfer, toh:1-2241526, tos:Timestamp(NoTimestamp)) already handled
 2021-12-03T16:16:37.723996Z  INFO ThreadId(1696) [osmosis-1:transfer/channel-6 -> irishub-1] generate messages from batch with 1 events
 2021-12-03T16:16:37.725994Z DEBUG ThreadId(1696) [osmosis-1:transfer/channel-6 -> irishub-1] osmosis-1 => SendPacketEv(SendPacket - h:1-2241429, seq:10340, path:channel-6/transfer->channel-3/transfer, toh:1-12775266, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:16:37.732677Z DEBUG ThreadId(1696) [osmosis-1:transfer/channel-6 -> irishub-1] irishub-1 <= /ibc.core.channel.v1.MsgRecvPacket from SendPacketEv(SendPacket - h:1-2241429, seq:10340, path:channel-6/transfer->channel-3/transfer, toh:1-12775266, tos:Timestamp(NoTimestamp)))

 2021-12-03T16:16:37.732714Z  INFO ThreadId(1696) [osmosis-1:transfer/channel-6 -> irishub-1] scheduling op. data with 1 msg(s) for Destination (height 1-2241430)
 2021-12-03T16:16:37.738312Z  INFO ThreadId(1696) [osmosis-1:transfer/channel-6 -> irishub-1] relay op. data of 1 msgs(s) to Destination (height 1-2241430), delayed by: 5.590259ms [try 1/5]
 2021-12-03T16:16:37.738356Z  INFO ThreadId(1696) [osmosis-1:transfer/channel-6 -> irishub-1] prepending Destination client update @ height 1-2241430
 2021-12-03T16:16:37.805367Z DEBUG ThreadId(1696) [osmosis-1 -> irishub-1:07-tendermint-5] MsgUpdateAnyClient from trusted height 1-2241384 to target height 1-2241430
 2021-12-03T16:16:37.805697Z  INFO ThreadId(1696) [osmosis-1:transfer/channel-6 -> irishub-1] assembled batch of 2 message(s)
 2021-12-03T16:16:37.805751Z DEBUG ThreadId(52) send_messages_and_wait_check_tx with 2 messages
 2021-12-03T16:16:37.805771Z DEBUG ThreadId(52) [irishub-1] send_tx: sending 2 messages using account sequence 3308
 2021-12-03T16:16:37.806303Z DEBUG ThreadId(52) [irishub-1] send_tx: max fee, for use in tx simulation: Fee { amount: "2001uiris", gas_limit: 2000000 }
 2021-12-03T16:16:37.839079Z DEBUG ThreadId(52) [irishub-1] estimate_gas: tx simulation successful, gas amount used: 333649
 2021-12-03T16:16:37.839135Z DEBUG ThreadId(52) [irishub-1] send_tx: using 333649 gas, fee Fee { amount: "368uiris", gas_limit: 367014 }
 2021-12-03T16:16:37.844667Z DEBUG ThreadId(52) [irishub-1] send_tx: broadcast_tx_sync: Response { code: Ok, data: Data([]), log: Log("[]"), hash: transaction::Hash(3ABF3F68CE738E940BC7DC75A47753FF83E22A635B74C70E4B26B8F9DB995CD2) }
 2021-12-03T16:16:37.844753Z  INFO ThreadId(1696) [Async~>irishub-1] response(s): 1; Ok:3ABF3F68CE738E940BC7DC75A47753FF83E22A635B74C70E4B26B8F9DB995CD2

 2021-12-03T16:16:37.844772Z  INFO ThreadId(1696) [osmosis-1:transfer/channel-6 -> irishub-1] success
 2021-12-03T16:16:47.104938Z  INFO ThreadId(332) [sentinelhub-2:transfer/channel-0 -> osmosis-1] clearing pending packets height=Some(Height { revision: 2, height: 3541699 })
 2021-12-03T16:16:48.847104Z  INFO ThreadId(332) [sentinelhub-2:transfer/channel-0 -> osmosis-1] finished scheduling pending packets clearing height=Some(Height { revision: 2, height: 3541699 })
 2021-12-03T16:17:01.120178Z  INFO ThreadId(1959) [impacthub-3:transfer/channel-4 -> osmosis-1] clearing pending packets height=Some(Height { revision: 3, height: 1601699 })
 2021-12-03T16:17:01.147441Z  INFO ThreadId(1959) [impacthub-3:transfer/channel-4 -> osmosis-1] finished scheduling pending packets clearing height=Some(Height { revision: 3, height: 1601699 })
 2021-12-03T16:17:03.634916Z  INFO ThreadId(1064) [regen-1:transfer/channel-1 -> osmosis-1] generate messages from batch with 1 events
 2021-12-03T16:17:03.636219Z DEBUG ThreadId(1064) [regen-1:transfer/channel-1 -> osmosis-1] regen-1 => SendPacketEv(SendPacket - h:1-3223686, seq:8378, path:channel-1/transfer->channel-8/transfer, toh:1-2241581, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:17:03.851079Z DEBUG ThreadId(1064) [regen-1:transfer/channel-1 -> osmosis-1] osmosis-1 <= /ibc.core.channel.v1.MsgRecvPacket from SendPacketEv(SendPacket - h:1-3223686, seq:8378, path:channel-1/transfer->channel-8/transfer, toh:1-2241581, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:17:03.851145Z  INFO ThreadId(1064) [regen-1:transfer/channel-1 -> osmosis-1] scheduling op. data with 1 msg(s) for Destination (height 1-3223687)
 2021-12-03T16:17:03.859993Z  INFO ThreadId(1064) [regen-1:transfer/channel-1 -> osmosis-1] relay op. data of 1 msgs(s) to Destination (height 1-3223687), delayed by: 8.834422ms [try 1/5]
 2021-12-03T16:17:03.860036Z  INFO ThreadId(1064) [regen-1:transfer/channel-1 -> osmosis-1] prepending Destination client update @ height 1-3223687
 2021-12-03T16:17:03.930378Z DEBUG ThreadId(1064) [regen-1 -> osmosis-1:07-tendermint-6] MsgUpdateAnyClient from trusted height 1-3223626 to target height 1-3223687
 2021-12-03T16:17:03.930603Z  INFO ThreadId(1064) [regen-1:transfer/channel-1 -> osmosis-1] assembled batch of 2 message(s)
 2021-12-03T16:17:03.930661Z DEBUG ThreadId(60) send_messages_and_wait_check_tx with 2 messages
 2021-12-03T16:17:03.930678Z DEBUG ThreadId(60) [osmosis-1] send_tx: sending 2 messages using account sequence 166793
 2021-12-03T16:17:03.931200Z DEBUG ThreadId(60) [osmosis-1] send_tx: max fee, for use in tx simulation: Fee { amount: "501uosmo", gas_limit: 5000000 }
 2021-12-03T16:17:03.934970Z ERROR ThreadId(60) [osmosis-1] estimate_gas: failed to simulate tx with non-recoverable error: gRPC call failed with status: status: InvalidArgument, message: "account sequence mismatch, expected 166791, got 166793: incorrect account sequence: invalid request", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }
 2021-12-03T16:17:03.935058Z ERROR ThreadId(1064) [regen-1:transfer/channel-1 -> osmosis-1] worker: schedule execution encountered error: failed with underlying error: gRPC call failed with status: status: InvalidArgument, message: "account sequence mismatch, expected 166791, got 166793: incorrect account sequence: invalid request", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }
 2021-12-03T16:17:04.935224Z  INFO ThreadId(1064) [regen-1:transfer/channel-1 -> osmosis-1] generate messages from batch with 1 events
 2021-12-03T16:17:04.936779Z DEBUG ThreadId(1064) [regen-1:transfer/channel-1 -> osmosis-1] regen-1 => SendPacketEv(SendPacket - h:1-3223686, seq:8378, path:channel-1/transfer->channel-8/transfer, toh:1-2241581, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:17:04.942811Z DEBUG ThreadId(1064) [regen-1:transfer/channel-1 -> osmosis-1] osmosis-1 <= /ibc.core.channel.v1.MsgRecvPacket from SendPacketEv(SendPacket - h:1-3223686, seq:8378, path:channel-1/transfer->channel-8/transfer, toh:1-2241581, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:17:04.942849Z  INFO ThreadId(1064) [regen-1:transfer/channel-1 -> osmosis-1] scheduling op. data with 1 msg(s) for Destination (height 1-3223687)
 2021-12-03T16:17:04.948341Z  INFO ThreadId(1064) [regen-1:transfer/channel-1 -> osmosis-1] relay op. data of 1 msgs(s) to Destination (height 1-3223687), delayed by: 5.485438ms [try 1/5]
 2021-12-03T16:17:04.948375Z  INFO ThreadId(1064) [regen-1:transfer/channel-1 -> osmosis-1] prepending Destination client update @ height 1-3223687
 2021-12-03T16:17:05.014692Z DEBUG ThreadId(1064) [regen-1 -> osmosis-1:07-tendermint-6] MsgUpdateAnyClient from trusted height 1-3223626 to target height 1-3223687
 2021-12-03T16:17:05.014876Z  INFO ThreadId(1064) [regen-1:transfer/channel-1 -> osmosis-1] assembled batch of 2 message(s)
 2021-12-03T16:17:05.014934Z DEBUG ThreadId(60) send_messages_and_wait_check_tx with 2 messages
 2021-12-03T16:17:05.014958Z DEBUG ThreadId(60) [osmosis-1] send_tx: sending 2 messages using account sequence 166793
 2021-12-03T16:17:05.015632Z DEBUG ThreadId(60) [osmosis-1] send_tx: max fee, for use in tx simulation: Fee { amount: "501uosmo", gas_limit: 5000000 }
 2021-12-03T16:17:05.020174Z ERROR ThreadId(60) [osmosis-1] estimate_gas: failed to simulate tx with non-recoverable error: gRPC call failed with status: status: InvalidArgument, message: "account sequence mismatch, expected 166791, got 166793: incorrect account sequence: invalid request", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }
 2021-12-03T16:17:05.020264Z ERROR ThreadId(1064) [regen-1:transfer/channel-1 -> osmosis-1] worker: schedule execution encountered error: failed with underlying error: gRPC call failed with status: status: InvalidArgument, message: "account sequence mismatch, expected 166791, got 166793: incorrect account sequence: invalid request", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }

The second one shows that, even after 35 minutes, Hermes was still getting the same error which in turn was causing all Osmosis tx submissions to fail and relaying to stop. Only after a Hermes restart did the issue clear.
2021-12-03T16:51:20.405416Z ERROR ThreadId(60) [osmosis-1] estimate_gas: failed to simulate tx with non-recoverable error: gRPC call failed with status: status: InvalidArgument, message: "account sequence mismatch, expected 166791, got 166793: incorrect account sequence: invalid request", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }
 2021-12-03T16:51:20.405517Z ERROR ThreadId(68) [core-1:transfer/channel-6 -> osmosis-1] worker: schedule execution encountered error: failed with underlying error: gRPC call failed with status: status: InvalidArgument, message: "account sequence mismatch, expected 166791, got 166793: incorrect account sequence: invalid request", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }
 2021-12-03T16:51:20.412916Z DEBUG ThreadId(65) [crypto-org-chain-mainnet-1:transfer/channel-10 -> osmosis-1] osmosis-1 <= /ibc.core.channel.v1.MsgAcknowledgement from WriteAcknowledgementEv(WriteAcknowledgement - h:1-3479364, seq:30344, path:channel-5/transfer->channel-10/transfer, toh:1-3479510, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:51:20.412968Z  INFO ThreadId(65) [crypto-org-chain-mainnet-1:transfer/channel-10 -> osmosis-1] scheduling op. data with 1 msg(s) for Destination (height 1-3479365)
 2021-12-03T16:51:20.413386Z DEBUG ThreadId(1686) [irishub-1:transfer/channel-3 -> osmosis-1] osmosis-1 <= /ibc.core.channel.v1.MsgAcknowledgement from WriteAcknowledgementEv(WriteAcknowledgement - h:1-12775128, seq:10340, path:channel-6/transfer->channel-3/transfer, toh:1-12775266, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:51:20.413415Z  INFO ThreadId(1686) [irishub-1:transfer/channel-3 -> osmosis-1] scheduling op. data with 1 msg(s) for Destination (height 1-12775129)
 2021-12-03T16:51:20.413912Z DEBUG ThreadId(221) [iov-mainnet-ibc:transfer/channel-2 -> osmosis-1] osmosis-1 <= /ibc.core.channel.v1.MsgRecvPacket from SendPacketEv(SendPacket - h:0-6555675, seq:1567, path:channel-2/transfer->channel-15/transfer, toh:1-2241783, tos:Timestamp(NoTimestamp)))
 2021-12-03T16:51:20.413958Z  INFO ThreadId(221) [iov-mainnet-ibc:transfer/channel-2 -> osmosis-1] scheduling op. data with 1 msg(s) for Destination (height 0-6555676)
 2021-12-03T16:51:20.418821Z  INFO ThreadId(65) [crypto-org-chain-mainnet-1:transfer/channel-10 -> osmosis-1] relay op. data of 1 msgs(s) to Destination (height 1-3479365), delayed by: 5.842251ms [try 1/5]
 2021-12-03T16:51:20.418868Z  INFO ThreadId(65) [crypto-org-chain-mainnet-1:transfer/channel-10 -> osmosis-1] prepending Destination client update @ height 1-3479365
 2021-12-03T16:51:20.420975Z  INFO ThreadId(1686) [irishub-1:transfer/channel-3 -> osmosis-1] relay op. data of 1 msgs(s) to Destination (height 1-12775129), delayed by: 7.553282ms [try 1/5]
 2021-12-03T16:51:20.421053Z  INFO ThreadId(1686) [irishub-1:transfer/channel-3 -> osmosis-1] prepending Destination client update @ height 1-12775129
 2021-12-03T16:51:20.424912Z  WARN ThreadId(65) [crypto-org-chain-mainnet-1 -> osmosis-1:07-tendermint-4] resolving trusted height from the full list of consensus state heights for target height 1-3479365; this may take a while
 2021-12-03T16:51:20.430265Z  WARN ThreadId(1686) [irishub-1 -> osmosis-1:07-tendermint-5] resolving trusted height from the full list of consensus state heights for target height 1-12775129; this may take a while
 2021-12-03T16:51:21.431067Z  INFO ThreadId(62) [akashnet-2:transfer/channel-9 -> osmosis-1] generate messages from batch with 1 events
 2021-12-03T16:51:22.215780Z  INFO ThreadId(68) [core-1:transfer/channel-6 -> osmosis-1] clearing pending packets height=Some(Height { revision: 1, height: 3752299 })
 2021-12-03T16:51:24.531656Z  INFO ThreadId(84) [juno-1:transfer/channel-0 -> osmosis-1] generate messages from batch with 1 events
 2021-12-03T16:51:25.963152Z  INFO ThreadId(332) [sentinelhub-2:transfer/channel-0 -> osmosis-1] clearing pending packets height=Some(Height { revision: 2, height: 3541999 })
 2021-12-03T16:51:31.636600Z  INFO ThreadId(1064) [regen-1:transfer/channel-1 -> osmosis-1] clearing pending packets height=Some(Height { revision: 1, height: 3223999 })
 2021-12-03T16:51:39.661945Z  INFO ThreadId(01) using default configuration from '/home/hermes/.hermes/config.toml'
 2021-12-03T16:51:39.663034Z  INFO ThreadId(01) starting REST API server listening at http://<IP_ADDRESS>:3000
 2021-12-03T16:51:39.668173Z  INFO ThreadId(01) Hermes has started
 2021-12-03T16:51:39.680474Z DEBUG ThreadId(41) [akashnet-2] starting event monitor

The second one shows that, even after 35 minutes, Hermes was still getting the same error which in turn was causing all Osmosis tx submissions to fail and relaying to stop. Only after a Hermes restart did the issue clear.

You did not try restarting the Osmosis node, right (i.e., the mempool flushing approach)? Only Hermes was restarted.
Looking at the logs, it's possible that the error message is trustworthy:

2021-12-03T16:17:03.934970Z ERROR ThreadId(60) [osmosis-1] estimate_gas: failed to simulate tx with non-recoverable error: gRPC call failed with status: status: InvalidArgument, message: "account sequence mismatch, expected 166791, got 166793: incorrect account sequence: invalid request", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }

At the top of log you shared shows a transaction that has s.n. 166792. This transaction has hash C819745777EA7921B9E45FB7463C4F8DFEF08F4583E0933EBCB7FB4DC22E5D6C which does not appear on mintscan, hinting that this tx was unsuccessful -- it never left the mempool. So whatever was the underlying problem, it manifested in the full node around the top of the log when this tx was submitted to the mempool, or even earlier.
We might consider reviving Soares' PR https://github.com/informalsystems/ibc-rs/pull/1349 which had a account refresh method (ref). Though it's not clear from these logs if that would fix it.
But this is all speculation. We need to test if we can trust either the query_account gRPC (the solution Soares proposed with this PR), or the error message we receive as response as seen in the snippet above.

I see several ways:

shift the work of counting to grpc and update account sequence before send each txn  - transaction in broadcast mode not be faster than new block come.
process the event as an error and update the self.account before next txn.

At the top of log you shared shows a transaction that has s.n. 166792. This transaction has hash C819745777EA7921B9E45FB7463C4F8DFEF08F4583E0933EBCB7FB4DC22E5D6C which does not appear on mintscan, hinting that this tx was unsuccessful -- it never left the mempool. So whatever was the underlying problem, it manifested in the full node around the top of the log when this tx was submitted to the mempool, or even earlier.

The node was not restarted. Only Hermes was restarted and resumed operations normally.

https://github.com/cosmos/cosmos-sdk/issues/5731

sync: Wait for the tx to pass/fail CheckTx
async: Don't wait for pass/fail CheckTx; send and return tx immediately
block: Wait for the tx to pass/fail CheckTx, DeliverTx, and be committed in a block (not-recommended)
It's best to always use sync.

Why block mode not recomended ?

Why block mode not recomended ?

I don't have a complete answer to this question. We simply noticed it was not reliable, so we switched away from block to sync some time ago (in #986). This comment provides some rationale for that change https://github.com/informalsystems/ibc-rs/issues/977#issuecomment-847144450.

I still think use increment account sequence is not fully correct with broadcast-mode sync.

Indeed. Part of the reason is because Hermes can be quite aggressive with submitting transactions, and as this comment hints https://github.com/cosmos/cosmos-sdk/issues/4186#issuecomment-592994069, it is prone to failure, we just haven't found a more robust option yet.

The node was not restarted. Only Hermes was restarted and resumed operations normally.
process the event as an error and update the self.account before next txn - refresh on error

I think we'll resurrect the refresh-on-error approach and give that a go.

A bit more context here.
If the node is slow in any way to gossip transactions, Hermes will keep on submitting new ones and increasing the counter in the cache. Even after the mempool is clear, Hermes will still now have an incorrect sequence number and keeps trying to submit txs using it. No way to recover from this unless Hermes is restarted.

Thanks Mircea! We would like to move forward with a potential solution and we'll prepare a branch for testing that. We'd appreciate if you can help us test it!

Seems that upgrading to IBC 2.0 is causing a lot more sequence number issues. All networks that have been upgraded to the latest version of the SDK now generate these errors where they were not before.

PR https://github.com/informalsystems/ibc-rs/pull/1349 is ready for testing. Would you consider trying to run from that dev branch Mircea?
We're also waiting on feedback from another relayer operator who was willing to try this branch. Would be great to have you also on it!

When restart osmosis node I get a situation when timeout txn do infinite loop for hermes instance:
2021-12-17T12:13:45.430305Z ERROR ThreadId(1510) [osmosis-1:transfer/channel-6 -> irishub-1] timed out while confirming TxHashes: count=1; CAC57E3637C51B34CC371A7E385A532E845A692ABA00D30E24D72A130F32C985
2021-12-17T12:13:45.430380Z  INFO ThreadId(1510) [irishub-1:transfer/channel-3 -> osmosis-1] relay op. data of 1 msgs(s) to Destination (height 1-12918776), delayed by:<PHONE_NUMBER>17579s [try 1/5]
2021-12-17T12:13:45.430403Z  INFO ThreadId(1510) [irishub-1:transfer/channel-3 -> osmosis-1] prepending Destination client update @ height 1-12918776
2021-12-17T12:13:45.433564Z  WARN ThreadId(1510) [irishub-1 -> osmosis-1:07-tendermint-5] resolving trusted height from the full list of consensus state heights for target height 1-12918776; this may take a while
2021-12-17T12:13:45.604013Z  INFO ThreadId(1510) [irishub-1:transfer/channel-3 -> osmosis-1] assembled batch of 2 message(s)
2021-12-17T12:13:45.616997Z ERROR ThreadId(22) [osmosis-1] estimate_gas: failed to simulate tx with non-recoverable error: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 590, got 591: incorrect account sequence", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "2422098"} }
2021-12-17T12:13:45.617127Z ERROR ThreadId(1510) error processing pending events in destination chain: failed with underlying error: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 590, got 591: incorrect account sequence", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "2422098"} }
2021-12-17T12:13:45.818738Z ERROR ThreadId(1510) [osmosis-1:transfer/channel-6 -> irishub-1] timed out while confirming TxHashes: count=1; CAC57E3637C51B34CC371A7E385A532E845A692ABA00D30E24D72A130F32C985
2021-12-17T12:13:45.818825Z  INFO ThreadId(1510) [irishub-1:transfer/channel-3 -> osmosis-1] relay op. data of 1 msgs(s) to Destination (height 1-12918776), delayed by:<PHONE_NUMBER>61734s [try 1/5]
2021-12-17T12:13:45.818864Z  INFO ThreadId(1510) [irishub-1:transfer/channel-3 -> osmosis-1] prepending Destination client update @ height 1-12918776
2021-12-17T12:13:45.822413Z  WARN ThreadId(1510) [irishub-1 -> osmosis-1:07-tendermint-5] resolving trusted height from the full list of consensus state heights for target height 1-12918776; this may take a while
2021-12-17T12:13:45.988462Z  INFO ThreadId(1510) [irishub-1:transfer/channel-3 -> osmosis-1] assembled batch of 2 message(s)
2021-12-17T12:13:45.998275Z ERROR ThreadId(22) [osmosis-1] estimate_gas: failed to simulate tx with non-recoverable error: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 590, got 591: incorrect account sequence", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "2422098"} }
2021-12-17T12:13:45.998678Z ERROR ThreadId(1510) error processing pending events in destination chain: failed with underlying error: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 590, got 591: incorrect account sequence", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "2422098"} }
2021-12-17T12:13:46.200827Z ERROR ThreadId(1510) [osmosis-1:transfer/channel-6 -> irishub-1] timed out while confirming TxHashes: count=1; CAC57E3637C51B34CC371A7E385A532E845A692ABA00D30E24D72A130F32C985
2021-12-17T12:13:46.200884Z  INFO ThreadId(1510) [irishub-1:transfer/channel-3 -> osmosis-1] relay op. data of 1 msgs(s) to Destination (height 1-12918776), delayed by:<PHONE_NUMBER>22239s [try 1/5]
2021-12-17T12:13:46.200906Z  INFO ThreadId(1510) [irishub-1:transfer/channel-3 -> osmosis-1] prepending Destination client update @ height 1-12918776
2021-12-17T12:13:46.203637Z  WARN ThreadId(1510) [irishub-1 -> osmosis-1:07-tendermint-5] resolving trusted height from the full list of consensus state heights for target height 1-12918776; this may take a while
2021-12-17T12:13:46.376525Z  INFO ThreadId(1510) [irishub-1:transfer/channel-3 -> osmosis-1] assembled batch of 2 message(s)
2021-12-17T12:13:46.389308Z ERROR ThreadId(22) [osmosis-1] estimate_gas: failed to simulate tx with non-recoverable error: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 590, got 591: incorrect account sequence", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "2422098"} }
2021-12-17T12:13:46.390305Z ERROR ThreadId(1510) error processing pending events in destination chain: failed with underlying error: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 590, got 591: incorrect account sequence", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "2422098"} }
2021-12-17T12:13:46.591871Z ERROR ThreadId(1510) [osmosis-1:transfer/channel-6 -> irishub-1] timed out while confirming TxHashes: count=1; CAC57E3637C51B34CC371A7E385A532E845A692ABA00D30E24D72A130F32C985
2021-12-17T12:13:46.591917Z  INFO ThreadId(1510) [irishub-1:transfer/channel-3 -> osmosis-1] relay op. data of 1 msgs(s) to Destination (height 1-12918776), delayed by:<PHONE_NUMBER>55254s [try 1/5]
2021-12-17T12:13:46.591936Z  INFO ThreadId(1510) [irishub-1:transfer/channel-3 -> osmosis-1] prepending Destination client update @ height 1-12918776
2021-12-17T12:13:46.594990Z  WARN ThreadId(1510) [irishub-1 -> osmosis-1:07-tendermint-5] resolving trusted height from the full list of consensus state heights for target height 1-12918776; this may take a while
2021-12-17T12:13:46.760441Z  INFO ThreadId(1510) [irishub-1:transfer/channel-3 -> osmosis-1] assembled batch of 2 message(s)
2021-12-17T12:13:46.778062Z ERROR ThreadId(22) [osmosis-1] estimate_gas: failed to simulate tx with non-recoverable error: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 590, got 591: incorrect account sequence", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "2422098"} }
2021-12-17T12:13:46.778370Z ERROR ThreadId(1510) error processing pending events in destination chain: failed with underlying error: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 590, got 591: incorrect account sequence", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "2422098"} }
2021-12-17T12:13:46.980044Z ERROR ThreadId(1510) [osmosis-1:transfer/channel-6 -> irishub-1] timed out while confirming TxHashes: count=1; CAC57E3637C51B34CC371A7E385A532E845A692ABA00D30E24D72A130F32C985
2021-12-17T12:13:46.980118Z  INFO ThreadId(1510) [irishub-1:transfer/channel-3 -> osmosis-1] relay op. data of 1 msgs(s) to Destination (height 1-12918776), delayed by:<PHONE_NUMBER>55123s [try 1/5]
2021-12-17T12:13:46.980153Z  INFO ThreadId(1510) [irishub-1:transfer/channel-3 -> osmosis-1] prepending Destination client update @ height 1-12918776
2021-12-17T12:13:46.983278Z  WARN ThreadId(1510) [irishub-1 -> osmosis-1:07-tendermint-5] resolving trusted height from the full list of consensus state heights for target height 1-12918776; this may take a while
2021-12-17T12:13:47.146116Z  INFO ThreadId(1510) [irishub-1:transfer/channel-3 -> osmosis-1] assembled batch of 2 message(s)
2021-12-17T12:13:47.158125Z ERROR ThreadId(22) [osmosis-1] estimate_gas: failed to simulate tx with non-recoverable error: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 590, got 591: incorrect account sequence", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "2422098"} }
2021-12-17T12:13:47.158215Z ERROR ThreadId(1510) error processing pending events in destination chain: failed with underlying error: gRPC call failed with status: status: Unknown, message: "account sequence mismatch, expected 590, got 591: incorrect account sequence", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "2422098"} }
2021-12-17T12:13:47.359842Z ERROR ThreadId(1510) [osmosis-1:transfer/channel-6 -> irishub-1] timed out while confirming TxHashes: count=1; CAC57E3637C51B34CC371A7E385A532E845A692ABA00D30E24D72A130F32C985
2021-12-17T12:13:47.359899Z  INFO ThreadId(1510) [irishub-1:transfer/channel-3 -> osmosis-1] relay op. data of 1 msgs(s) to Destination (height 1-12918776), delayed by:<PHONE_NUMBER>36785s [try 1/5]
2021-12-17T12:13:47.359920Z  INFO ThreadId(1510) [irishub-1:transfer/channel-3 -> osmosis-1] prepending Destination client update @ height 1-12918776
2021-12-17T12:13:47.364098Z  WARN ThreadId(1510) [irishub-1 -> osmosis-1:07-tendermint-5] resolving trusted height from the full list of consensus state heights for target height 1-12918776; this may take a while

Why block mode not recomended ?

I don't have a complete answer to this question. We simply noticed it was not reliable, so we switched away from block to sync some time ago (in #986). This comment provides some rationale for that change #977 (comment).

I still think use increment account sequence is not fully correct with broadcast-mode sync.

Indeed. Part of the reason is because Hermes can be quite aggressive with submitting transactions, and as this comment hints cosmos/cosmos-sdk#4186 (comment), it is prone to failure, we just haven't found a more robust option yet.

The node was not restarted. Only Hermes was restarted and resumed operations normally.
process the event as an error and update the self.account before next txn - refresh on error

I think we'll resurrect the refresh-on-error approach and give that a go.

Thanks for the detailed answer. The reasons for the problems with the sequence number are now more clear.

Thanks for the detailed answer. The reasons for the problems with the sequence number are now more clear.

Since you are noticing the sequence number caching problem (according to your logs above), consider running Hermes from this dev branch https://github.com/informalsystems/ibc-rs/pull/1349, and let us know if it resolves the problem. Appreciate any feedback!


appmesh deployementType related changes

PR is being drafted

Checked with Eli. Connoisseur was never used. Can bypass this check and merge.


Installation currently broken due to crates-index SemVer violation

cargo install cargo-dl currently fails due to cargo-dl using http 0.2, but crates-index migrating to http 1 (a breaking change under SemVer) in a non-breaking SemVer release. See https://github.com/frewsxcv/rust-crates-index/issues/172 for details.
Possible solutions:

Wait for crates-index to fix its SemVer violation, and then either do nothing (stay on http 0.2), or migrate to http 1 and crates-index 3.
Migrate to http 1 and crates-index >= 2.10. This however runs a risk that crates-index decides to fix its SemVer violation, which would break cargo-dl again.

As a temporary workaround, cargo install --locked cargo-dl will install with the Cargo.lock that was present at the time the latest cargo-dl version was published, i.e. with an older crates-index.

Fun, this is not the first time a dependency did a breaking http update https://github.com/algesten/ureq/issues/683. I'll wait and see what happens to avoid any breakage from fixes.
I'm disappointed I didn't catch this earlier with my nightly CI job as it got auto-disabled. I'll have to work out some way to ensure all my repos nightly jobs remain running even when they don't need continuous maintenance :thinking:.

Looks like it's been fixed upstream now. Thanks for letting me know and opening the upstream issue!


Generate event handler code action not working in Blazor Web App

Type: Bug

Create a Blazor Web App
In Counter.razor remove the IncrementCount method
Put the cursor on IncrementCount in the @onclick attribute and type ctrl-.

 Most likely the latter. The code action will only be suggested when the diagnostic is present.

This doesn't repro for me either, @danroth27 . I suggest we close this unless we nail down a repro. What do you think?

This doesn't repro for me either, @danroth27 . I suggest we close this unless we nail down a repro. What do you think?

@alexgav Sure, that's fine. If I see this again, I'll be sure to let you know.

Closing per discussion above as I can't repro. Happy to investigate if we have a at least a semi-reliable repro for this.

The same issue was confirmed in VM 2022, v 4.8.09032.

The same issue was confirmed in VM 2022, v 4.8.09032. Similar setup: Standard Blazor templated app with counter and weather. Create any class with an event and try to subscribe to that event in the Blazor component. After typing "+=" and pressing the tab, nothing happened. But the explicit manual assignment of event handler works properly.

That is a slightly different problem and tracked by https://github.com/dotnet/razor/issues/8862


"Ceiling Sum" problem of codechef

Aim
I want to add the code for "Ceiling Sum" problem of codechef (Starters 7)
Details
Problem name : Ceiling Sum
Problem Code : CEILSUM
Problem link : https://www.codechef.com/START7C/problems/CEILSUM
Please assign me this issue
LGM-SOC Participant
Programming language
C++

[ ] C
[x] C++
[ ] Java
[ ] Python

@ravikr126  Kindly assign me this issue under LGMSOC'21

I would like to solve this in java language.


[FIXED] assignment copies lock value for crypto/tls.Config

Running go vet ./... with go 1.7.3 would report the following:
server/route.go:342: assignment copies lock value to tlsConfig: crypto/tls.Config contains sync.Once contains sync.Mutex
server/server.go:479: assignment copies lock value to config: crypto/tls.Config contains sync.Once contains sync.Mutex

Add a “clone” function while waiting for this to be addressed
by the language itself (https://go-review.googlesource.com/#/c/28075/)

@derekcollison The use of reflect is because there are extra fields in 1.7 that are not in previous release. Also, should we drop 1.5.4 from travis?

Coverage increased (+0.1%) to 89.725% when pulling fe22e0f523b71bfb9af384b5937ada5c877a0286 on fix_tls_config_copy into 10cd7aa16dee1346be2735ec5337d5b321268dfe on master.

Yes would drop 1.5.4.

Coverage remained the same at 89.595% when pulling<PHONE_NUMBER>c4a8ee83ca78801c0832ceb96833a2 on fix_tls_config_copy into 10cd7aa16dee1346be2735ec5337d5b321268dfe on master.


Sinatra tracer gem

This uses the Rack::Tracer gem to handle route tracing.
View rendering is traced by patching the render method.
There's probably room for more detailed tracing if desired, but routes and rendering seemed like the two most important operations.

Add test for spans generated when render is called.

@keitwb I'll rename the repos per your suggestion after merging these PRs

@max Agreed, in practice it should ideally only be small snippets. Do we want to enforce an arbitrary length limit on the tag?

No it's fine we already enforce a limit on span size in the gateway to protect ourselves.


[Search files] expand entire token

Evolved from #109. Now you can use variables in the token as well. Of course, tilde (~) is expanded.

@kidonng Mind I'm ready to merge in the main change but not the bug fix. Mind if I or you revert the unnecessary changes and merge them in later?

Mind I'm ready to merge in the main change but not the bug fix. Mind if I or you revert the unnecessary changes and merge them in later?

That's ok if you want to, but I'm not in favor of the idea because:

This feature is affected by the bug
The new PR still needs to refer this one
The fix is not a lot of code (unless you have to include the test changes)

That's ok if you want to, but I'm not in favor of the idea because:

This feature is affected by the bug
The new PR still needs to refer this one
The fix is only 3 LOC (unless you have to include the test changes)

True true we'll keep it in this PR. Let me reset it for you.
I don't know what you meant by

Actually I just realized there's a implicit prerequisite: you have to set fzf_preview_dir_cmd or the code won't run eval $fzf_preview_dir_cmd "$file_path" but just regular command ls -A -F "$file_path".

Can you fix that by taking over this PR again?

Can you fix that by taking over this PR again?

Perhaps I'm not expressing myself clearly, I was saying that probably you didn't set fzf_preview_dir_cmd so can't reproduce.
Maybe eval can be kept, we just need to quote the arguments properly.

@kidonng Finally merged #137. Can you fix the merge conflicts and we may finally merge this today? Thanks! Sorry for the long wait.

Should be covered by existing test for tilde expanding but we can also add tests for variable expanding if you want.

Yes please, could you add a test for the TMPDIR? An idea I have is to execute mktmp and then check for it in TMPDIR. Up to you.

Nice work there 👍 sorry for always letting you finishing the tests.

Nice work there 👍 sorry for always letting you finishing the tests.

No problem, it's teamwork! You've also brought a lot of good ideas and expanded my perspective on what a good plugin is.


Service implementation catches all exceptions (python)

Using gRPC python I had something like:
class MyServer(pb.MyService):
  def MyMethod(self, request, context):
    some_method_that_raised_an_exception()
    return pb.MyReply()

The output of the server is completely blank, the client reports a remote error.
I'm not able to provide exact output right now, but it should be easy enough to reproduce.

I believe this is fixed as of the integration of pull request 6787 - please reopen if I'm mistaken?


failed to open the html file using broswer-plus in atom

[Enter steps to reproduce below:]

...
...


feat(): Terminate other Customer Environment Versions

Changes:

Added a new parameter/option to the "New-Environment" PSCmdlet and "deploy" command to allow the user to terminate the existing Customer Environments of the current one that he is creating. This only happens if the user is creating a new version. This process will be triggered and waited for after the new version is created BUT BEFORE it is deployed.
Version bumped to 1.1.7.

On-hold until next CP production release is completed!

As a general comment, maybe we should try not to mix spaces with tabs


Screenshots
I think it would be cool to also have that applied for icon::
Reason is that you can make your page icons and favorites more specific than with default emojis.
Example:

Attach files
Link you provided on Discord:
https://github.com/ryanoasis/nerd-fonts/tree/master/patched-fonts/FiraCode#browser-support
Links to main sites:
https://www.nerdfonts.com/
https://fontawesome.com/

good idea...
but most (my and others) themes using sans fonts, not monospace for classic (non-code) text (((
would be cool to find FiraSans nerd, or patch it with icons by myself....

test latest release
usage - https://github.com/yoyurec/logseq-solarized-extended-theme#-custom-page-icons


with openssl 3.0.1 have problem

make[1]: Entering directory '/root/KeyDB/src'
CC Makefile.dep
CC release.o
CC tls.o
tls.cpp: In function ‘void tlsInit()’:
tls.cpp:158:26: error: too few arguments to function ‘const char* ERR_lib_error_string(long unsigned int)’
ERR_lib_error_string();
^
In file included from tls.cpp:41:0:
/usr/include/openssl/err.h:445:13: note: declared here
const char ERR_lib_error_string(unsigned long e);
^~~~~~~~~~~~~~~~~~~~
tls.cpp:159:5: error: ‘SSL_int_init’ was not declared in this scope
SSL_int_init();
^~~~~~~~~~~~
tls.cpp:159:5: note: suggested alternative: ‘SSL_in_init’
SSL_int_init();
^~~~~~~~~~~~
SSL_in_init
tls.cpp: In function ‘int tlsConfigure(redisTLSContextConfig)’:
tls.cpp:359:24: error: cannot convert ‘DH* {aka dh_st*}’ to ‘BIGNUM* {aka bignum_st*}’ for argument ‘1’ to ‘void BN_free(BIGNUM*)’
BN_free(dh);
^
In file included from /usr/include/openssl/crypto.h:36:0,
from /usr/include/openssl/bio.h:30,
from /usr/include/openssl/conf.h:24,
from tls.cpp:39:
/usr/include/openssl/types.h:134:16: note: class type ‘DH {aka dh_st}’ is incomplete
typedef struct dh_st DH;
^~~~~
tls.cpp:363:20: error: cannot convert ‘DH* {aka dh_st*}’ to ‘BIGNUM* {aka bignum_st*}’ for argument ‘1’ to ‘void BN_free(BIGNUM*)’
BN_free(dh);
^
In file included from /usr/include/openssl/crypto.h:36:0,
from /usr/include/openssl/bio.h:30,
from /usr/include/openssl/conf.h:24,
from tls.cpp:39:
/usr/include/openssl/types.h:134:16: note: class type ‘DH {aka dh_st}’ is incomplete
typedef struct dh_st DH;
^~~~~
At global scope:
cc1plus: warning: unrecognized command line option ‘-Wno-atomic-alignment’
cc1plus: warning: unrecognized command line option ‘-Wno-address-of-packed-member’
make[1]: *** [Makefile:451: tls.o] Error 1
make[1]: Leaving directory '/root/KeyDB/src'
make: *** [Makefile:9: install] Error 2

OPENSSL_CFLAGS=-I/usr/include/openssl OPENSSL_LDFLAGS=-L/lib64 LDFLAGS=-L/lib64 LDLIBS=-lssl -lcrypto make USE_SYSTEMD=yes BUILD_TLS=yes install

I am getting the same problem. I'm currently unable to build on Raspberry Pi 3 using Debian stretch.
The steps I followed was:
sudo apt-get install build-essential nasm autotools-dev autoconf libjemalloc-dev tcl tcl-dev uuid-dev libcurl4-openssl-dev git
git clone https://github.com/EQ-Alpha/KeyDB.git
cd KeyDB
make
sudo make install```

And the error is:

```pi@Pi3:~/Builds/KeyDB $ make
cd src && make all
make[1]: Entering directory '/home/pi/Builds/KeyDB/src'
    CC Makefile.dep
    CC object.o
object.cpp: In function ‘char* serializeStoredStringObject(sds, robj_roptr)’:
object.cpp:1585:5: error: static assertion failed
     static_assert((sizeof(robj) + sizeof(mvcc)) == sizeof(redisObjectStack), "");
     ^~~~~~~~~~~~~
At global scope:
cc1plus: warning: unrecognized command line option ‘-Wno-atomic-alignment’
cc1plus: warning: unrecognized command line option ‘-Wno-address-of-packed-member’
Makefile:482: recipe for target 'object.o' failed
make[1]: *** [object.o] Error 1
make[1]: Leaving directory '/home/pi/Builds/KeyDB/src'
Makefile:6: recipe for target 'all' failed
make: *** [all] Error 2
pi@Pi3:~/Builds/KeyDB $ openssl version -a
OpenSSL 1.1.0l  10 Sep 2019
built on: reproducible build, date unspecified
platform: debian-armhf
options:  bn(64,32) rc4(char) des(long) blowfish(ptr)
compiler: gcc -DDSO_DLFCN -DHAVE_DLFCN_H -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DOPENSSLDIR="\"/usr/lib/ssl\"" -DENGINESDIR="\"/usr/lib/arm-linux-gnueabihf/engines-1.1\""
OPENSSLDIR: "/usr/lib/ssl"
ENGINESDIR: "/usr/lib/arm-linux-gnueabihf/engines-1.1"```

The interesting thing is it looks like i'm getting the same problem with OpenSSL 1.1 if I'm not mistaken. Assuming that the build uses the system Openssl.

Looking at the code in object.cpp:
sds serializeStoredStringObject(sds str, robj_roptr o)
{
    uint64_t mvcc;
    mvcc = mvccFromObj(o);
    str = sdscatlen(str, &mvcc, sizeof(mvcc));
    str = sdscatlen(str, &(*o), sizeof(robj));
    static_assert((sizeof(robj) + sizeof(mvcc)) == sizeof(redisObjectStack), "");
    switch (o->encoding)
    {
    case OBJ_ENCODING_EMBSTR:
    case OBJ_ENCODING_RAW:
        str = sdscatsds(str, (sds)szFromObj(o));
        break;

    case OBJ_ENCODING_INT:
        break;  //nop
    }

    return str;
}

Not sure this has anything to do with OpenSSL at all in fact.

The only reference to "-Wno-atomic-alignment" I can find in the entire repository is in the makefile:
# Default settings
STD=-pedantic -DREDIS_STATIC=''
CXX_STD=-std=c++14 -pedantic -fno-rtti -D__STDC_FORMAT_MACROS
ifneq (,$(findstring clang,$(CC)))
  STD+=-Wno-c11-extensions
else
ifneq (,$(findstring FreeBSD,$(uname_S)))
  STD+=-Wno-c11-extensions
endif
endif
WARN=-Wall -W -Wno-missing-field-initializers -Wno-address-of-packed-member -Wno-atomic-alignment
OPT=$(OPTIMIZATION)

Fixed in 6.3.2!


[QUESTION]: Post Upgrade of Dell CSI Operator on Openshift from 1.8 to 1.10, its not working anymore

How can the Team help you today?
Details: ?
Post Upgrade of Dell CSI Operator on Openshift from 1.8 to 1.10, its not working anymore and below are the errors from the pod logs. Can you please advice on the upgrade path?
There seems to be new configs introduced, and this has broken the operator
W0207 05:26:51.893435       1 feature_gate.go:237] Setting GA feature gate Topology=true. It will be removed in a future release.
I0207 05:26:51.893502       1 feature_gate.go:245] feature gates: &{map[Topology:true]}
I0207 05:26:51.893523       1 csi-provisioner.go:150] Version: v3.3.0
I0207 05:26:51.893530       1 csi-provisioner.go:173] Building kube configs for running in cluster...
 This is a blocker for us

@soudamsugit , we work in IST timezone and we can get into a call after half an hour which is 12:00 PM IST. Please let us know if that works for you

@rensyct, 12 PM IST will be great.

Yes @soudamsugit, we will be able to join the teams call

@rensyct https://teams.microsoft.com/l/meetup-join/19%3Ameeting_NjM0Y2UxODMtNDg1MC00ZTExLTkyZWEtYzg3Njg3NGNkMjg4%40thread.v2/0?context={"Tid"%3A"1f4f7eda-6e51-425e-a0f9-4c2fcef58a52"%2C"Oid"%3A"63a774a0-2e52-48bd-b0db-edde453cc553"}

Hello @rensyct,
Thanks for yourtime yesterday. I had ran this with storage team, the credentails look fine.
Also, I have tested them from inside pod and they seem to work fine
[soudamv@qocpa001l ~/powermax/2.3] $ oc rsh powermax-controller-5ccddb4b7-mn2hf
sh-4.4$ curl -kv -u username:password https://<IP_ADDRESS>:8443/univmax/restapi/version

Trying <IP_ADDRESS>...
TCP_NODELAY set
Connected to <IP_ADDRESS> (<IP_ADDRESS>) port 8443 (#0)
ALPN, offering h2
ALPN, offering http/1.1
successfully set certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
TLSv1.3 (OUT), TLS handshake, Client hello (1):
TLSv1.3 (IN), TLS handshake, Server hello (2):
TLSv1.2 (IN), TLS handshake, Certificate (11):
TLSv1.2 (IN), TLS handshake, Server key exchange (12):
TLSv1.2 (IN), TLS handshake, Server finished (14):
TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
TLSv1.2 (OUT), TLS handshake, Finished (20):
TLSv1.2 (IN), TLS handshake, Finished (20):
SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
ALPN, server did not agree to a protocol
Server certificate:
subject: C=US; ST=Massachusetts; L=Hopkinton; O=EMC Corporation; OU=Storage Platform Enablers and Applications; CN=VMAX-000497600159
start date: Aug  3 23:42:18 2021 GMT
expire date: Aug  1 23:42:18 2031 GMT
issuer: C=US; ST=Massachusetts; L=Hopkinton; O=EMC Corporation; OU=Storage Platform Enablers and Applications; CN=VMAX-000497600159
SSL certificate verify result: self signed certificate (18), continuing anyway.

TCP_NODELAY set
Connected to <IP_ADDRESS> (<IP_ADDRESS>) port 8443 (#0)
ALPN, offering h2
ALPN, offering http/1.1
successfully set certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
TLSv1.3 (OUT), TLS handshake, Client hello (1):
TLSv1.3 (IN), TLS handshake, Server hello (2):
TLSv1.2 (IN), TLS handshake, Certificate (11):
TLSv1.2 (IN), TLS handshake, Server key exchange (12):
TLSv1.2 (IN), TLS handshake, Server finished (14):
TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
TLSv1.2 (OUT), TLS handshake, Finished (20):
TLSv1.2 (IN), TLS handshake, Finished (20):
SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
ALPN, server did not agree to a protocol
Server certificate:
subject: C=US; ST=Massachusetts; L=Hopkinton; O=EMC Corporation; OU=Storage Platform Enablers and Applications; CN=VMAX-000497600159
start date: Aug  3 23:42:18 2021 GMT
expire date: Aug  1 23:42:18 2031 GMT
issuer: C=US; ST=Massachusetts; L=Hopkinton; O=EMC Corporation; OU=Storage Platform Enablers and Applications; CN=VMAX-000497600159
SSL certificate verify result: self signed certificate (18), continuing anyway.
 Can we please connect at 10 AM IST today?

Hi @soudamsugit, thank you for your response, Since the operator part is working as expected now and the current issue is related to PowerMax, we will need to have the Powermax team on the call. I will reach out to them and let you know the time that works for the PowerMax team

Thanks a lot @rensyct. Sorry for chasing, appreciate if you can prioritise this issue

Np @soudamsugit. I can understand. This issue is a priority for us. Will keep you posted on the time when we can connect today

@soudamsugit, team is available at 11:00 AM IST. Does that time work for you?

sure, we can have a chat at 11 AM

https://teams.microsoft.com/l/meetup-join/19%3Ameeting_NjM0Y2UxODMtNDg1MC00ZTExLTkyZWEtYzg3Njg3NGNkMjg4%40thread.v2/0?context={"Tid"%3A"1f4f7eda-6e51-425e-a0f9-4c2fcef58a52"%2C"Oid"%3A"63a774a0-2e52-48bd-b0db-edde453cc553"}

@rensyct  any update on this case ?

@rensyct , sorry for bugging. I still see few pods crashloopbacking
[soudamv@qocpa001l ~/powermax/2.3] $ oc get pods
NAME                                  READY   STATUS             RESTARTS        AGE
powermax-controller-b8f87cdb6-jnmv5   5/5     Running            0               21m
powermax-controller-b8f87cdb6-qq5tf   5/5     Running            0               21m
powermax-node-2gc4d                   2/2     Running            0               21m
powermax-node-6vn8f                   1/2     CrashLoopBackOff   8 (2m7s ago)    21m
powermax-node-97b8j                   1/2     CrashLoopBackOff   8 (77s ago)     21m
powermax-node-fldjp                   1/2     CrashLoopBackOff   8 (2m31s ago)   21m
powermax-node-g4krm                   2/2     Running            0               21m
powermax-node-j9n86                   2/2     Running            0               21m
powermax-node-nr6n5                   1/2     CrashLoopBackOff   8 (4m9s ago)    21m
powermax-node-nwlzk                   2/2     Running            0               21m
powermax-node-qxt4v                   1/2     CrashLoopBackOff   8 (71s ago)     21m
powermax-node-r6rl7                   1/2     CrashLoopBackOff   8 (95s ago)     21m
powermax-node-x9h8l                   2/2     Running            0               21m
powermax-node-xjsw5                   2/2     Running            0               21m
powermax-node-zblfk                   1/2     CrashLoopBackOff   8 (2m42s ago)   21m
powermax-node-zhcjb                   1/2     CrashLoopBackOff   8 (3m32s ago)   21m
error:
{"level":"error","msg":"No topology keys could be generated","time":"2023-02-08T07:19:51.188623463Z"}
{"level":"info","msg":"/csi.v1.Node/NodeGetInfo: REP 0016: rpc error: code = FailedPrecondition desc = no topology keys could be generate","time":"2023-02-08T07:19:51.188642894Z"}
{"level":"info","msg":"/csi.v1.Identity/GetPluginInfo: REQ 0017: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2023-02-08T07:24:57.360533138Z"}
{"level":"info","msg":"/csi.v1.Identity/GetPluginInfo: REP 0017: Name=csi-powermax.dellemc.com, VendorVersion=2.3.0, Manifest=map[commit:9ee4c6afaedb58725f6518239fc527e076c3e0a9 formed:Tue, 14 Jun 202
{"level":"info","msg":"/csi.v1.Node/NodeGetInfo: REQ 0018: XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2023-02-08T07:24:59.336730277Z"}
{"level":"error","msg":"Couldn't find any ip interfaces on any of the port-groups","time":"2023-02-08T07:24:59.336778092Z"}
{"level":"error","msg":"No topology keys could be generated","time":"2023-02-08T07:24:59.336785694Z"}
{"level":"info","msg":"/csi.v1.Node/NodeGetInfo: REP 0018: rpc error: code = FailedPrecondition desc = no topology keys could be generate","time":"2023-02-08T07:24:59.336800696Z"}

hi @soudamsugit , the nodes which are failing would be having FC connection issues hence it is not creating any topology keys.
could you please check FC/iscsi connection for nodes .

Thanks @boyamurthy, Can you pls let me know if any documentation on checking the FC/iscsi connection for nodes

hi @soudamsugit  , please find the documentation  for prerequisites .
https://dell.github.io/csm-docs/docs/csidriver/installation/helm/powermax/#prerequisites

sh-4.4# rpm -qa | grep iscsi
iscsi-initiator-utils-iscsiuio-<IP_ADDRESS>-1.gita8fcb37.el8.x86_64
iscsi-initiator-utils-<IP_ADDRESS>-1.gita8fcb37.el8.x86_64
sh-4.4# storage
sh: storage: command not found
sh-4.4# cat /etc/iscsi/initiatorname.iscsi;
InitiatorName=iqn.1994-05.com.redhat:cfed26c7f7aa
sh-4.4# curl -v telnet://10.204.xxx.xx:8443

Rebuilt URL to: telnet://10.204.xxx.xx:8443/
Trying 10.204.xxx.xx...
TCP_NODELAY set
Connected to 10.204.xxx.xx (10.204.xxx.xx) port 8443 (#0)

@soudamsugit , could we connect over a call so that the Powermax team can take a look at the environment to check the issue

Thanks @rensyct @delldubey
Can we please connect tomorrow 9 AM IST on this issue?

Hi @soudamsugit,
We can connect at 10:30 AM tomorrow. Please let us know if that time works for you

Thanks @rensyct, 10:30 AM works fine

is the invite. Can you guys pls join
https://teams.microsoft.com/l/meetup-join/19%3Ameeting_NjM0Y2UxODMtNDg1MC00ZTExLTkyZWEtYzg3Njg3NGNkMjg4%40thread.v2/0?context={"Tid"%3A"1f4f7eda-6e51-425e-a0f9-4c2fcef58a52"%2C"Oid"%3A"63a774a0-2e52-48bd-b0db-edde453cc553"}

Sure @soudamsugit

Hi @soudamsugit
Posting the updates after today's call
@boyamurthy figured out that FC HBA's(WWN's ) are not having connectivity/zoning to the respective storage array . Hence it is failing to create initiator group and topology keys on the nodes where driver is in crashloopback state. We dont see a failure on the nodes where the zoning is done properly
Requested @soudamsugit to reach out to the storage team from their end and ensure that the zoning is done properly between the array and the host.

hi @soudamsugit  ,
if it is working fine, could you close this .

Hello @rensyct,
Thank you for the support you provided, I am closing this case.
Appreciate the support you guys project
Regards,
Suresh

Thank you @soudamsugit for the confirmation and for closing the case.
Thank you @boyamurthy and @delldubey for helping with the queries and looking into the configuration issues related to Powermax driver.


Option to generate "(555)" phone numbers

In movies they use the area code 555 for US phone numbers to avoid generating phone numbers for real people / organisations. See https://en.wikipedia.org/wiki/555_(telephone_number)
I'd love an option to generate numbers with area code 555, like:
chance.phone({ country: 'us', exampleNumber: true }); // -><PHONE_NUMBER>
PS: Fantastic library.

I'll look into this if maintainer are willing to merge it. :smiley:

Apologies, I missed this but yes of course I’d happily accept a PR with this addition!

@victorquinn I've opened a PR to address this. #480


Fix: Seems like account not consuming note ?

Since in the plaintext version the account is supposed to consume only one notes right (the receive card one) right ?

commit: 45f19bc26b4f12af5f0aa321bdbc13ba5a2f3a02 in branch: v-plaintext fixes this


What server badges are not here

I don't know how to get their pictures, but I know for sure that there are no such icons here:

Non community server (1 boost)
Non community server (1 level)
Non community server (2 level)
Non community server (3 level)

I don't think there is a badge for those tbh. Guilds for example just the the icon from the clan tag.

Hm... Okay, I'll fix it.

As said I dont think the other servers have badges too, unless you find them somewhere

For example:

The colors of the icons can be found through a network search in the dev tools.
For example, you can take "iconTierTwo"

Sadly, I can't give all 4 icons.

Also, if you type the contents of the  tag of my icon into the search, then you can access other icons in this way.
M10.95 1.4a1.59 1.59 0 0 1 2.1 0l4.9 4A3.37 3.37 0 0 1 19 8v8c0 1.07-.4 2.06-1.05 2.6l-4.9 4c-.65.53-1.45.53-2.1 0l-4.9-4A3.37 3.37 0 0 1 5 16V8c0-1.07.4-2.06 1.05-2.6l4.9-4Z

Oh interesting I might add them next weekend!

@mezotv  Cheers! I found all icons! :basecampy:
I hope you didn't start working in parallel with me.

Icon
Title

No Level

Level 1

Level 2

Level 3

https://github.com/mezotv/discord-badges/assets/109104246/2b99f84f-0e75-4b22-ae63-618793a1b31d
https://github.com/mezotv/discord-badges/assets/109104246/ad8ecdc2-4645-45e9-bc97-d10adb7cca78
https://github.com/mezotv/discord-badges/assets/109104246/f7d96617-a126-4d36-8db1-a376e481b39b
https://github.com/mezotv/discord-badges/assets/109104246/2d0eeeef-fb3d-43bf-9e6a-0790350c2327
https://github.com/mezotv/discord-badges/assets/109104246/20d5f25a-22aa-4cfb-b11d-f23f39dd7620
https://github.com/mezotv/discord-badges/assets/109104246/5503dad8-d6f9-4e76-9fa1-2b49c03984da

Oh thats awesome, gonna be honest I havent had the time yet so this helps!

Not sure how to add them to the repo in the best way tho. Might be a bit of a challenge.

Maybe I'll try to make a pull request?


fix: typos in ai-quota

fix typos in ai-quota document

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


[Feature/extensions] CreateDetectorAction extends BaseExtensionRestHandler

NOTE: Requires merging SDK #246 first.
Description
Simplifies CreateDetectorAction by extending BaseExtensionRestHandler:

changes routes() to routeHandlers() which auto-checks the method and path
removes the unnecessary check for the method and path
renames the handler method to avoid name conflict/override

Issues Resolved
Fixes SDK #245
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@dbwiddis rebase with feature/extensions?


Fix for EXPLAIN ANALYZE DECLARE CURSOR

This request fixes the behavior of EXPLAIN ANALYZE DECLARE CURSOR. When building a query tree, the tree looks like this:
stmt (T_ExplainStmt) -> query (CMD_SELECT) -> utilityStmt (T_DeclareCursorStmt)
stmt (T_ExplainStmt) is processed at the coordinator, and then the query (CMD_SELECT) plan node is wrapped in stmt (T_PlannedStmt) and sent to the segments for execution.
The segments receive the query plan tree, which looks like this:
stmt (T_PlannedStmt) -> Query (CMD_SELECT) -> utilityStmt (T_DeclareCursor)
When we pass the plan to the segment, we choose the strategy for executing the plan. The strategy is determined in PortalStart() -> ChoosePortalStrategy().
In the ChoosePortalStrategy() function, there is a condition that if the query has type CMD_SELECT and the query has no utilityStmt, PORTAL_ONE_SELECT will be chosen (a strategy when the portal contains only one SELECT query). Our query does not meet this condition, so the PORTAL_MULTI_QUERY strategy is chosen (for all other cases).
Next, in function PortalRun() with PORTAL_ONE_SELECT, the query is executed, but with PORTAL_MULTI_QUERY, the backend will attempt to open a cursor (in our case) in function standard_ProcessUtility(), which it shouldn't do because the query contains EXPLAIN ANALYZE.
We can't fix the error while building the query tree, but we can do it later. It would be more reasonable to remove utilityStmt (T_DeclareCursorStmt) at the coordinator, so that the backends receive the query (CMD_SELECT) without it and choose the correct strategy for execution.

This solution does not take into account that plan for cursor (fetching first tuples) and plan for SELECT (fetching all tuples) can be different. See the tuple_fraction variable and cursor_tuple_fraction GUC. A plan for cursor is created with Postgres query optimizer.
https://github.com/arenadata/gpdb/blob/6ed03a1d78ef8ab6560dc9a5b40a47cd75faaa75/src/backend/optimizer/plan/planner.c#L270 Here utilityStmt is used to get options (CURSOR_OPT_FAST_PLAN).

@chicago-industry, are you going to provide this solution to the community by yourself?


Concurrent request very slow

I am testing trtllm backend v0.6.0 for llama2-7b with below setup.  The code snippets is as below. If I set one request with length being about 1000, it took about 2-3 seconds to finish. And if I send 16 requests at the same time, it took about 30 seconds, so it feels like there is no inflight batching at all.
        with multiprocessing.Manager() as manager:
            def send(client, input_id, input_length, i, shared_list):
                inputs = [
                    _input("input_ids", np.array(input_id, dtype=np.int32).reshape(1, -1)),
                    _input("input_lengths", np.array([input_length], dtype=np.int32).reshape(1, -1)),
                    _input("request_output_len", np.array([max_new_tokens], dtype=np.uint32).reshape(1, -1)),
                    _input("end_id", np.array([2], dtype=np.uint32).reshape(1, -1)),
                ]
                with grpcclient.InferenceServerClient("localhost:8001", verbose=False) as client:
                    print('send')
                    shared_list[i] = client.infer('tensorrt_llm', inputs).as_numpy('sequence_length').reshape(-1)[0]
            processes = []
            shared_list = manager.list([""] * batch_size)
            start = time.time()
            for i in range(batch_size):
                process = multiprocessing.Process(target=send, args=(None, input_id[i], input_lengths[i], i, shared_list))
                processes.append(process)
                process.start()

            for process in processes:
                process.join()
            latency = time.time() - start

Trtllm setup as below
name: "tensorrt_llm"
backend: "tensorrtllm"
max_batch_size: 1024

model_transaction_policy {
  decoupled: false
}

input [
  {
    name: "input_ids"
    data_type: TYPE_INT32
    dims: [ -1 ]
    allow_ragged_batch: true
  },
  {
    name: "input_lengths"
    data_type: TYPE_INT32
    dims: [ 1 ]
    reshape: { shape: [ ] }
  },
  {
    name: "request_output_len"
    data_type: TYPE_UINT32
    dims: [ 1 ]
  },
  {
    name: "end_id"
    data_type: TYPE_UINT32
    dims: [ 1 ]
    reshape: { shape: [ ] }
    optional: true
  },
  {
    name: "pad_id"
    data_type: TYPE_UINT32
    dims: [ 1 ]
    reshape: { shape: [ ] }
    optional: true
  }
]
output [
  {
    name: "output_ids"
    data_type: TYPE_INT32
    dims: [ -1, -1 ]
  },
  {
    name: "sequence_length"
    data_type: TYPE_INT32
    dims: [ -1 ]
  }
]
instance_group [
  {
    count: 1
    kind : KIND_CPU
  }
]
parameters: {
  key: "max_beam_width"
  value: {
    string_value: "1"
  }
}
parameters: {
  key: "FORCE_CPU_ONLY_INPUT_TENSORS"
  value: {
    string_value: "no"
  }
}
parameters: {
  key: "gpt_model_type"
  value: {
    string_value: "inflight_batching"
  }
}
parameters: {
  key: "gpt_model_path"
  value: {
    string_value: "/models/trt_engines/llama-2-7b-chat-hf/1-gpu/"
  }
}
parameters: {
  key: "max_tokens_in_paged_kv_cache"
  value: {
    string_value: "8192"
  }
}
parameters: {
  key: "max_kv_cache_length"
  value: {
    string_value: "4096"
  }
}
parameters: {
  key: "batch_scheduler_policy"
  value: {
    string_value: "max_utilization"
  }
}
parameters: {
  key: "max_num_sequences"
  value: {
    string_value: "96"
  }
}
parameters: {
  key: "exclude_input_in_output"
  value: {
    string_value: "true"
  }
}

Below is how we generate the engine
python build.py --model_dir /models/llama-2-7b-chat-hf/ \
                --dtype float16 \
                --remove_input_padding \
                --use_gpt_attention_plugin float16 \
	 --use_inflight_batching \
                --enable_context_fmha \
        --paged_kv_cache \
        --max_input_len 3072 \
        --max_output_len 1024 \
                --use_gemm_plugin float16 \
                 --output_dir /models/trt_engines/llama-2-7b-chat-hf/1-gpu/ \
        --world_size 1 \
        --max_batch_size 96 \
 --tp_size 1

A more broad question, what is the recommended way of running batched inference through triton+trtllm - asking since the server only accepts one prompt at a time.

I think the issue is from max_tokens_in_paged_kv_cache, but as long as i set it for larger than 16k, it will OOM. It seems weird on A100(40GiB) machine for a llama2-7b fp16 model.

Same question, it feels like there is no inflight batching or dynamic batching. Were you able to resolve this?

Same issue. It seams that inflight batching does not work

@kaiyux should I set --max_num_tokens to be equal to --max_input_len?
I've tried both inflight_fused_batching and inflight_batching -- the results were similar

should I set --max_num_tokens to be equal to --max_input_len?

@fursovia --max_num_tokens could be roughly estimated by max_batch_size * ratio * max_input_len + max_batch_size * (1 - ratio), ratio stands for the ratio of the context phase requests on each forward calculation in the inference, it's usually around 5~20%, and depends on the actual scenario.
Please also note that you may also want to increase --max_batch_size, so that larger batches are allowed to feed into the engine.

@kaiyux you mean max_batch_size * ratio * max_input_len + max_batch_size * (1 - ratio) * max_output_len?

@sfc-gh-zhwang No, I meant max_batch_size * ratio * max_input_len + max_batch_size * (1 - ratio).
Let me explain more on this: --max_num_tokens means the max number of input tokens, which equals to max_batch_size * max_input_len by default. When remove input padding is enabled, since the tokens from different sequences are packed together, a max_num_tokens lower than default is allowed, and engines can be built with higher max_batch_size, which helps getting higher throughput because larger batch size could be reached.
Let's say if the percentage of context phase requests among all requests is ratio, since the max input length of context phase is max_input_len, and that of generation phase is 1, the rough estimation would be max_batch_size * ratio * max_input_len + max_batch_size * (1 - ratio) * 1.

Hi, I can reproduce the problem stably.
python benchmark_core_model.py  --num-requests 200 --max-input-len 1024 -i grpc token-norm-dist --input-mean 256 --input-stdev 0 --output-mean 256 --output-stdev 0
python benchmark_core_model.py  --num-requests 200 --max-input-len 1024 -i grpc token-norm-dist --input-mean 512 --input-stdev 0 --output-mean 512 --output-stdev 0
python benchmark_core_model.py  --num-requests 200 --max-input-len 1024 -i grpc token-norm-dist --input-mean 256 --input-stdev 0 --output-mean 256 --output-stdev 0

You will find the command in L3 gets a 5x slow-down, even if the workload is the same as in L1.

using v0.6.1 fixed the issue.


Import all datatables npm packages and build with vite

Hi,
im using laravel 9 app.
I want build all datatables npm packages with Vite.
How can build ?? where I Can put all imports ?

I experimented on it I think last week, see the links below:
https://github.com/yajra/laravel-datatables-vite
https://yajrabox.com/docs/laravel-datatables/10.0/quick-starter
Still a work in progress and needs more improvement but should be enough to get started I think on a fresh app.


Update netcdf references to recent LDT paper

Included the LDT description paper citation within
routines that generate any netcdf files that contain
references for LDT and LIS as global attributes.

Code compiles w/o issue.  Since this just changes metadata in the netCDF output, I don't see the need to manually run this.  I'll merge the changes now.


ActiveMQ plugin - Browse Queue doesn't load messages at initial access

We need to push the refresh button to view messages in a queue. It should show messages immediately instead.

I fixed this some time ago in https://github.com/hawtio/hawtio-integration/commit/1d779e3f18d00ce43e0d6be2bdb1073a2789176a.