Implement uart_err_check for cdc_acm

Is your enhancement proposal related to a problem? Please describe.
I am using the cdc_acm driver a device, to sync the communication I am waiting for a UART_BREAK.
I would expect to use this snippet:
static void usb_uart_interrupt_handler_new(struct device* dev)
{
	int ret = 0;
	char c;

	while (uart_irq_update(dev) && uart_irq_is_pending(dev)) {
		while(0 == uart_irq_tx_complete(dev)){
			continue;
		};
		ret = uart_err_check(dev);
		if(ret == UART_BREAK){
			/* do something */
		}
	}

}

Describe the solution you'd like
Implement the missing functions of the cdc_acm driver.
static_int cdc_acm_err_check(struct device *dev);
static void cdc_acm_err_irq_err_enable(struct device *dev);
Describe alternatives you've considered
I have consider to send a special command as a break signal.
Additional context
Add any other context or graphics (drag-and-drop an image) about the feature request here.

static void usb_uart_interrupt_handler_new(struct device* dev)
{
	int ret = 0;
	char c;

	while (uart_irq_update(dev) && uart_irq_is_pending(dev)) {
		while(0 == uart_irq_tx_complete(dev)){
			continue;
		};
		ret = uart_err_check(dev);
		if(ret == UART_BREAK){
			/* do something */
		}
	}

}

From the UART driver API, it is not clear if uart_err_check() is intended or can be used from interrupt handler context and is not part of IRQ API.

Describe the solution you'd like
Implement the missing functions of the cdc_acm driver.

No plans to implement it.


Content-Type matching is too strict / doesn't follow matcher specification

Given the pact JSON with term matcher on Content-Type header:
"response": {
  "status": 200,
  "headers": {
    "Content-Type": "text/plain"
  },
  "body": "fake-header.fake-token.fake-signature",
  "matchingRules": {
    "$.headers.Content-Type": {
      "match": "regex",
      "regex": "^text\\/plain"
    },
    "$.body": {
      "match": "regex",
      "regex": "^[^.]+.[^.]+.[^.]+$"
    }
  }
}

Complete JSON: demo-loader-auth.json
and the provider responding with
HTTP/1.1 200 OK
Connection: close
Content-Type: text/plain; charset=utf-8

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE3MTE2NTEyMjEsImlhdCI6MTcxMTY0NDAyMSwidG9waWNQZXJtaXNzaW9ucyI6ImVKeUtqZ1VFQUFELy93RVZBTGs9IiwidXNlcklkIjoic2FnYW5BZG1pbiIsIm9yZ0lkIjoic2FnYW4iLCJyb2xlSWRzIjpbInNhZ2FuLWFkbWluIl0sInNhZ2FuVXNlciI6InNhZ2FuLW9wZXJhdG9yIiwic2Vzc2lvbiI6eyJpZCI6IiJ9fQ.GaDrAz1132zOptIyElUp8-vSUkTZn47lFTGrzBKSycw

which matches Content-Type regex: https://regex101.com/r/eaiKgC/2
the corresponding pactffi trace:
RequestResponseInteraction { id: None, description: "create admin token for valid operator token", provider_states: [ProviderState { name: "operator has valid token", params: {} }], request: Request { method: "POST", path: "/api/v1/admin_tokens", query: None, headers: Some({"Content-Type": ["application/json"]}), body: Present(b"{\"token\":\"1234567890abcdef1234567890abcdef12345678\",\"username\":\"sagan-operator\"}", None, None), matching_rules: MatchingRules { rules: {} }, generators: Generators { categories: {} } }, response: Response { status: 200, headers: Some({"Content-Type": ["text/plain"]}), body: Present(b"fake-header.fake-token.fake-signature", Some(ContentType { main_type: "text", sub_type: "plain", attributes: {}, suffix: None }), None), matching_rules: MatchingRules { rules: {HEADER: MatchingRuleCategory { name: HEADER, rules: {DocPath { path_tokens: [Root, Field("Content-Type")], expr: "Content-Type" }: RuleList { rules: [Regex("^text\\/plain")], rule_logic: And, cascaded: false }} }, BODY: MatchingRuleCategory { name: BODY, rules: {DocPath { path_tokens: [Root], expr: "$" }: RuleList { rules: [Regex("^[^.]+.[^.]+.[^.]+$")], rule_logic: And, cascaded: false }} }} }, generators: Generators { categories: {} } } }

the verification fails with:
5) Verifying a pact between demo-loader and auth Given operator has valid token - create admin token for valid operator token
    5.1) includes header 'Content-Type' with value 'text/plain'
           Expected header 'Content-Type' to have value 'text/plain' but was 'text/plain; charset=utf-8'

This issue is present in libpact_ffi 0.4.16.

Just change your regex to ^text/plain(;.*)? and it will match

@rholshausen TYVM for the tip. I confirm that it works.
Still arguably the behavior is unexpected because normally ^text/plain regex should match all strings that start with that prefix.


Feature: Improve design of dashboard page for student

Is your feature request related to a problem? Please describe.

Current design seems to be very basic and not appealing to users.

Describe the solution you'd like

Use bootstrap or other CSS libs to improve UI/UX

Additional context

Will give levels after opening of the PR


Resolve import issues #89

Resolves import issues in #89. Please check while reviewing that importing the package works for you in any context (notebooks/terminals/scripts etc) in any directory so that we can close this issue once and for all.

I just found this line find_packages(include=["peony*"]) here. I think this does exactly what your lambda function does just looks a little more nice.

I just found this line find_packages(include=["peony*"]) here. I think this does exactly what your lambda function does just looks a little more nice.

Nice find. I checked it out and it does the same thing so I swapped it with this version for better readability.


Split spring cloud actuator module into a standalone module

Parent task: https://github.com/Azure/azure-sdk-for-java/issues/23925
Design doc: https://github.com/Azure/azure-sdk-for-java/wiki/Spring-Cloud-Azure-design#222-spring-cloud-azure-modules-relationship

Hi, @saragluna .

FYI: I created this issuse.
Is it OK to assign this issue to you?

Resolved in #24309.


Hipify cublas API

This PR has changes to hipify cublas api. The cublas dependent tests are tested with these changes on rocm 5.7 and rocm 6.0

This PR is ready for review, I've attached logs below for cusolver and cublas tests on rocm 5.7 and 6.0.
cusolver_cublas_tests_rocm5.7.log
cusolver_cublas_tests_rocm6.0.log


Update rails to <IP_ADDRESS> for CVEs

https://rubyonrails.org/2024/10/15/Rails-Versions-7-0-8-5-7-1-4-1-and-7-2-1-1-have-been-released
@bdunne Please review.

Backported to radjabov in commit b0b3aac611779e30972ea0287250d863bf1ffbf7.
commit b0b3aac611779e30972ea0287250d863bf1ffbf7
Author: Brandon Dunne<EMAIL_ADDRESS>Date:   Wed Oct 16 16:52:22 2024 -0400

    Merge pull request #23236 from Fryguy/update_rails

    Update rails to <IP_ADDRESS> for CVEs

    (cherry picked from commit 08eeee28ab567d55fdec11e43b60389155884687)


Externalities not allowed to fail within runtime: "Trie lookup error: Database missing expected key

Is there an existing issue?

[X] I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

[X] This is not a support question.

Description of bug
Hi, I'm from Phala team, previously, our all collators (based on Polkadot 0.9.34) sudden looping panic with this error
here's a log
khala_collator_1.log.zip
At that time, I tried many ways but our collators still can't back to work, luckily I'm testing our new one based on Polkadot v0.9.39 node, so I try to use that binary, and they finally work again, so I was wondering the bug has been fixed.
But recently, a user reported this error with our latest node based on Polkadot v0.9.40, so I believe there still has a bug
I ask him for backup DB, but it's too big (~1T)
Thread 'tokio-runtime-worker' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Database missing expected key: 0x4806be757d6e6f3be9fed0cdf4d5b03f8563e10beef7cc64994d6ddc3ce19448"', /root/.cargo/git/checkouts/substrate-7e08433d4c370a21/76fed9b/primitives/state-machine/src/ext.rs:192

====================

Version: 0.1.23-dev-ae1f5251e7e

   0: sp_panic_handler::set::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/f3126500f25114ba4e0ac3e76694dd45a22de56d/library/alloc/src/boxed.rs:2002:9
      std::panicking::rust_panic_with_hook
             at /rustc/f3126500f25114ba4e0ac3e76694dd45a22de56d/library/std/src/panicking.rs:692:13
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/f3126500f25114ba4e0ac3e76694dd45a22de56d/library/std/src/panicking.rs:579:13
   3: std::sys_common::backtrace::rust_end_short_backtrace
             at /rustc/f3126500f25114ba4e0ac3e76694dd45a22de56d/library/std/src/sys_common/backtrace.rs:137:18
   4: rust_begin_unwind
             at /rustc/f3126500f25114ba4e0ac3e76694dd45a22de56d/library/std/src/panicking.rs:575:5
   5: core::panicking::panic_fmt
             at /rustc/f3126500f25114ba4e0ac3e76694dd45a22de56d/library/core/src/panicking.rs:64:14
   6: core::result::unwrap_failed
             at /rustc/f3126500f25114ba4e0ac3e76694dd45a22de56d/library/core/src/result.rs:1790:5
   7: <sp_state_machine::ext::Ext<H,B> as sp_externalities::Externalities>::storage
   8: sp_io::storage::get_version_1
   9: sp_io::storage::ExtStorageGetVersion1::call
  10: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1),R>>::into_func::wasm_to_host_shim
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: <unknown>
  15: <unknown>
  16: <unknown>
  17: <unknown>
  18: <unknown>
  19: <unknown>
  20: wasmtime_runtime::traphandlers::catch_traps::call_closure
  21: wasmtime_setjmp
  22: sc_executor_wasmtime::runtime::perform_call
  23: <sc_executor_wasmtime::runtime::WasmtimeInstance as sc_executor_common::wasm_runtime::WasmInstance>::call_with_allocation_stats
  24: sc_executor_common::wasm_runtime::WasmInstance::call_export
  25: sc_executor::native_executor::WasmExecutor<H>::with_instance::{{closure}}
  26: <sc_executor::native_executor::WasmExecutor<H> as sp_core::traits::CodeExecutor>::call
  27: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_aux
  28: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_using_consensus_failure_handler
  29: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
  30: <khala_parachain_runtime::RuntimeApiImpl<__SrApiBlock,RuntimeApiImplCall> as sp_api::Core<SrApiBlock>>::__runtime_api_internal_call_api_at
  31: <&sc_service::client::client::Client<B,E,Block,RA> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
  32: <alloc::sync::Arc<T> as sc_consensus::block_import::BlockImport<B>>::import_block::{{closure}}
  33: <cumulus_client_consensus_common::ParachainBlockImport<Block,BI,BE> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
  34: <alloc::boxed::Box<dyn sc_consensus::block_import::BlockImport<B>+Error = sp_consensus::error::Error+Transaction = Transaction+core::marker::Sync+core::marker::Send> as sc_consensus::block_import::BlockImport<B>>::import_block::{{closure}}
  35: sc_consensus::import_queue::basic_queue::BlockImportWorker<B>::new::{{closure}}
  36: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
  37: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
  38: tokio::runtime::task::raw::poll
  39: std::sys_common::backtrace::__rust_begin_short_backtrace
  40: core::ops::function::FnOnce::call_once{{vtable.shim}}
  41: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/f3126500f25114ba4e0ac3e76694dd45a22de56d/library/alloc/src/boxed.rs:1988:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/f3126500f25114ba4e0ac3e76694dd45a22de56d/library/alloc/src/boxed.rs:1988:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/f3126500f25114ba4e0ac3e76694dd45a22de56d/library/std/src/sys/unix/thread.rs:108:17
  42: <unknown>
  43: <unknown>

Steps to reproduce
No reproduce step, it's occurred randomly and very rare

Do you use paritydb?

Do you use paritydb?

Yes

Latest patch release?

Latest patch release?

Yes, 0.4.6, I forgot the previous collators' used version
For 0.4.6, our users reported another issue https://github.com/paritytech/parity-db/issues/198 but I think it is unrelated to this one...

Can you provide the db?
CC @arkpar

I need to ask the user, left message to him

Can you provide the db?
CC @arkpar

Ah sorry, that user says he doesn't back up the DB that's too big to upload
Now I have dozens of nodes using paritydb, not sure I can meet them again

Without a db it will probably be not possible to find out what is wrong.

Without a db it will probably be not possible to find out what is wrong.

I understand... sorry...
I shall ensure the reporter keeps the broken DB in next time

Thank you!

@bkchr In your knowledge, does this error only affect ParityDB?

I can not say for sure, but I think so.

@bkchr @arkpar Hi, could you provide your SSH pub key?
we success trigger this again when we sync Polkadot, and my colleague catch a weird case that finalized > best, and https://github.com/paritytech/parity-db/issues/198
We upload these bad databases to a GCP server and they're > 100G

Sure, I'd take a look
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCZ0p00e9LEqvwLYAqRwDd258JweLKoVP+3dwHIUtO6Dn7eUntlHJ4sVTwuc/0cgCc3ol/ZYpWZEpY2V9Wa7B76+XQThWgOWeUQiD/0GUOWtfVybtBBeGwb7ZoZNxgVpgiyedJQ1VGv3KvmxCCrY2ajDM7Rc++6vVyV2V7+M7gbVivK7/DpwMAdhHPsDQYfwJ0Gb6yCfgzGSY9UAQNhm+rXHQHyZU2/ij44603fNweU2JN90reEYbdOimf0wYjOjmd6PtJWGNYeEcwCDX3sy660sol6RGc22sZB968GhoXR+vHdpK1IRdoSPsx7b8jffR6R6RZV9yoRW4Gy2yQdZyJXajomVdMxXs1rirWzaWUFPt2fyo8acklZjLEdZ+S7f9kZJ+CMUqL/cqL3VXaKYpf5n/SUAsqpxK1AbAPffCfUPzHZ7PpvMW+iuh0zFwvLuabS56uERWVhCOcnH4Ei0KpOOtEQ9EeJ8tX2m2mn5prbpZlb51pgEyA4+BqXHLprtc= arkadiy@localhost

Please also specify a how to run a node that uses this database

Sure, I'd take a look
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCZ0p00e9LEqvwLYAqRwDd258JweLKoVP+3dwHIUtO6Dn7eUntlHJ4sVTwuc/0cgCc3ol/ZYpWZEpY2V9Wa7B76+XQThWgOWeUQiD/0GUOWtfVybtBBeGwb7ZoZNxgVpgiyedJQ1VGv3KvmxCCrY2ajDM7Rc++6vVyV2V7+M7gbVivK7/DpwMAdhHPsDQYfwJ0Gb6yCfgzGSY9UAQNhm+rXHQHyZU2/ij44603fNweU2JN90reEYbdOimf0wYjOjmd6PtJWGNYeEcwCDX3sy660sol6RGc22sZB968GhoXR+vHdpK1IRdoSPsx7b8jffR6R6RZV9yoRW4Gy2yQdZyJXajomVdMxXs1rirWzaWUFPt2fyo8acklZjLEdZ+S7f9kZJ+CMUqL/cqL3VXaKYpf5n/SUAsqpxK1AbAPffCfUPzHZ7PpvMW+iuh0zFwvLuabS56uERWVhCOcnH4Ei0KpOOtEQ9EeJ8tX2m2mn5prbpZlb51pgEyA4+BqXHLprtc= arkadiy@localhost

Please also specify a how to run a node that uses this database

Sorry for late response.
SSH key added, please try ssh<EMAIL_ADDRESS>you can do anything you want, the VM is used for debugging.
I just validated all uploaded databases can reproduce problem again...
In home /home/ubuntu. there are 3 cases:
Case 1:
broken1.sh shall start a polkadot v0.9.41 with database polkadot.broken1, this case shows
Syncing, target=#15104383 (12 peers), best: #662528 (0xdc93…5db5), finalized #699904 (0xbb2d…84cd), ⬇ 904.6kiB/s ⬆ 27.0kiB/s
Finalized higher than best
Case 2:
broken2.sh shall start a polkadot v0.9.41 with database polkadot.broken2, this case shows
Thread 'tokio-runtime-worker' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Database missing expected key: 0x2ef32d4e98fc876924394ebec5841a0d49d08d236a9e6a65eb21249fa5ba0a03"'
But in the server, it stuck on booting for at least an hour, so I can't validate it
Case 3:
cd stuck-on-booting && docker compose up, this case is https://github.com/paritytech/parity-db/issues/198 the DB provided by our user, the node is based on polkadot-v0.9.41 branch, https://github.com/Phala-Network/khala-parachain/commit/c6a82f2f49bd9c5c23e27ecdf726d34506685edb this commit
I hope these sample could help.
I also backup original uploaded artifects in ~/tmp to avoid validating process broken the case.

Case 1:

This one is broken indeeed. Somehow an older commit was written on top of the new state. I'm going to add some changes to the database that will prevent this.

Case 2:

Opens ans sync fine for me. This is probably the issue that was fixed in parity-db 0.4.6.
It also has about 5Gb of queued commits so it takes a while to open on a slow disk. The commit queue may only grow this large if the disk is slow to perform fsync. I'd like to know what kind of hardware and OS this was originally running on? Was that an SSD at least?

Case 3:

Thils looks more of ancumulus issue.
2023-04-16 10:44:16.939 DEBUG main db: [Parachain] Opened blockchain db, fetched best = 0x49d94deb5f43067ba6783ef33102aaa58d42d561412f693a3c559f51d9f9b501 (2290332)
2023-04-16 10:44:16.939 DEBUG main db: [Parachain] Opened blockchain db, fetched final = 0x74a8f4665c44ed0aa9f16352673d778ac989221ae63cbb04b2d479f5da271a39 (1697672)

There are ~600000 blocks on top of the last finalized block, so cumulus tries to query a huge tree route on startup here:
https://github.com/paritytech/cumulus/blame/master/client/consensus/common/src/level_monitor.rs#L114
and gets stuck.
@bkchr I think we've ran into similar issues before. tree_route must not be called when the difference in block number for the start and end block in the route is too high.

@bkchr I think we've ran into similar issues before. tree_route must not be called when the difference in block number for the start and end block in the route is too high.

Yeah we had this issue in the transaction pool.
But yeah we should handle this, especially as this could happen after syncing a node and the relay chain hasn't finished syncing yet.
@davxy can you please handle this?

Opens ans sync fine for me. This is probably the issue that was fixed in parity-db 0.4.6.
It also has about 5Gb of queued commits so it takes a while to open on a slow disk. The commit queue may only grow this large if the disk is slow to perform fsync. I'd like to know what kind of hardware and OS this was originally running on? Was that an SSD at least?

This is my colleague's home server
hardware:
2x 9300 LSI 16 port controller
32x 1TB in 4x 8 RAIDZ2
Os TrueNAS (with 2 VM;s on it running the nodes)
so it's ZFS

For case 3, AFAIK they use SATA SSD too, our users already know HDD can't run a node well, no matter RocksDB or ParityDB.

There are ~600000 blocks on top of the last finalized block

I just report we might facing https://github.com/paritytech/substrate/issues/9360 this issue, not sure if they're the same thing

My colleague won't give up, he is uploading a new case which he just confirm has the "Trie lookup error"
once he uploaded it, I shall validate myself to ensure it can reproduce the error

IIRC ZFS is configured to use 128k IO pages by default, which is terrible for DB performance. Cosider using much smaller record size or a different FS.

IIRC ZFS is configured to use 128k IO pages by default, which is terrible for DB performance. Cosider using much smaller record size or a different FS.

Ok, I'm letting him discuss here, he is an experienced sysops not developer

@bkchr For this issue, because we use ink, our off-chain node uses the same Runtime infra with Substrate,
last week we fixed a bug in our off-chain computing node which also panicked with
state-machine/src/ext.rs:Externalities not allowed to fail within runtime: "Trie lookup error: Decoding failed for hash 0x2a2e4115bccf59eb99e5b7400da19d57a2e7ec0b8a7ebcf34db228a2bccc8027; err: Decode(Error { cause: None, desc: "out of data" })"', /root/.cargo/git/checkouts/substrate-7e08433d4c370a21/ba87188/primitives/state-machine/src/ext.rs:237:237:61
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

The fix is simple https://github.com/Phala-Network/phala-blockchain/pull/1246
The trie framework will pass an empty value when the rc is negative. So, the value would be mis-cleared when decreasing the rc.
This PR fixes it by setting the value only when rc > 0.

I'm thinking if there's a bug inside Substrate that not well maintain the RC?

Please for reference, @jasl send you issues designated as problem1 and problem2. After sending them to him, i removed them and started two new syncs:
So lets recap:

I start 2 nodes in a VM (Qemu) (which is ext4 btw) one with polkadot the other kusuma to sync. The VM's directly run on the TrueNAS system, which yes runs ZFS, but as block devices served to the VM, as such block sizes shouldnt matter. I also looked up trough documentation and did not find any tuning/specs on this area btw. But lets circle back on this issue.
they sync for 2/3 days and reaches 7202158 and 7676973 resp.
Then 2 different trie errors appear.  Both errors seems to resolve about a missing key, but their behavior is different, one just keeps spitting out errors and stop sync (designation problem3) and the other crashes and spits out  the trie error (designation problem4)

problem3 partial log (as the same error keeps looping):
2023-04-15 21:45:38 ⚙  Syncing  0.0 bps, target=#15106642 (100 peers), best: #7202158 (0x751b…aa0c), finalized #7201792 (0x01b8…90a5), ⬇ 309.7kiB/s ⬆ 311.7kiB/s 2023-04-15 21:45:39 Failed to write to trie: Database missing expected key: 0xec8604a2b16587a2fe61d76c4dfd079f866b5ff262fec5fef19505442050bae5 2023-04-15 21:45:39 panicked at 'Storage root must match that calculated.', /cargo-home/git/checkouts/substrate-7e08433d4c370a21/91061a7/frame/executive/src/lib.rs:473:9 2023-04-15 21:45:39 Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm unreachableinstruction executed WASM backtrace: error while executing at wasm backtrace:     0: 0x235d - <unknown>!rust_begin_unwind     1: 0x20aa - <unknown>!core::panicking::panic_fmt::h3ab5417155b7ba3b     2: 0x173f - <unknown>!core::panicking::panic::h5bfdfaa3db9a4b4a     3: 0x1a81ec - <unknown>!Core_execute_block 2023-04-15 21:45:44 💔 Error importing block 0xb845970efd3e40564de3a1bcadd16b209f2a5743372e51784d0e1f50c5045e0e: consensus error: Import failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasmunreachableinstruction executed WASM backtrace: error while executing at wasm backtrace:     0: 0x235d - <unknown>!rust_begin_unwind     1: 0x20aa - <unknown>!core::panicking::panic_fmt::h3ab5417155b7ba3b     2: 0x173f - <unknown>!core::panicking::panic::h5bfdfaa3db9a4b4a     3: 0x1a81ec - <unknown>!Core_execute_block 2023-04-15 21:45:44 💔 Error importing block 0xfbde0a5e93072626a7b0d5f80d80026d6c450b5d34ca4538aba2ede1d51cd7d3: block has an unknown parent 2023-04-15 21:45:44 💔 Error importing block 0x2df9d1bead7bafc3bbe673510202f27e53868faaa689803d3c72c6790859eaa7: block has an unknown parent 2023-04-15 21:45:44 💔 Error importing block 0xa0d56ab5624d36a028d861879178a7cba041ba075e726046c1acd1f2304f0002: block has an unknown parent 2023-04-15 21:45:44 Failed to write to trie: Database missing expected key: 0xec8604a2b16587a2fe61d76c4dfd079f866b5ff262fec5fef19505442050bae5 2023-04-15 21:45:44 panicked at 'Storage root must match that calculated.', /cargo-home/git/checkouts/substrate-7e08433d4c370a21/91061a7/frame/executive/src/lib.rs:473:9 2023-04-15 21:45:44 Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasmunreachableinstruction executed WASM backtrace: error while executing at wasm backtrace:     0: 0x235d - <unknown>!rust_begin_unwind     1: 0x20aa - <unknown>!core::panicking::panic_fmt::h3ab5417155b7ba3b     2: 0x173f - <unknown>!core::panicking::panic::h5bfdfaa3db9a4b4a     3: 0x1a81ec - <unknown>!Core_execute_block 2023-04-15 21:45:44 💔 Error importing block 0xb845970efd3e40564de3a1bcadd16b209f2a5743372e51784d0e1f50c5045e0e: consensus error: Import failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasmunreachableinstruction executed WASM backtrace: error while executing at wasm backtrace:     0: 0x235d - <unknown>!rust_begin_unwind     1: 0x20aa - <unknown>!core::panicking::panic_fmt::h3ab5417155b7ba3b     2: 0x173f - <unknown>!core::panicking::panic::h5bfdfaa3db9a4b4a     3: 0x1a81ec - <unknown>!Core_execute_block 2023-04-15 21:45:44 Failed to write to trie: Database missing expected key: 0xec8604a2b16587a2fe61d76c4dfd079f866b5ff262fec5fef19505442050bae5 2023-04-15 21:45:44 panicked at 'Storage root must match that calculated.', /cargo-home/git/checkouts/substrate-7e08433d4c370a21/91061a7/frame/executive/src/lib.rs:473:9 2023-04-15 21:45:44 Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasmunreachableinstruction executed WASM backtrace: error while executing at wasm backtrace:     0: 0x235d - <unknown>!rust_begin_unwind     1: 0x20aa - <unknown>!core::panicking::panic_fmt::h3ab5417155b7ba3b     2: 0x173f - <unknown>!core::panicking::panic::h5bfdfaa3db9a4b4a     3: 0x1a81ec - <unknown>!Core_execute_block 2023-04-15 21:45:44 💔 Error importing block 0xb845970efd3e40564de3a1bcadd16b209f2a5743372e51784d0e1f50c5045e0e: consensus error: Import failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasmunreachableinstruction executed WASM backtrace: error while executing at wasm backtrace:     0: 0x235d - <unknown>!rust_begin_unwind     1: 0x20aa - <unknown>!core::panicking::panic_fmt::h3ab5417155b7ba3b     2: 0x173f - <unknown>!core::panicking::panic::h5bfdfaa3db9a4b4a     3: 0x1a81ec - <unknown>!Core_execute_block
and for problem4:
`2023-04-15 22:02:38 ⚙  Syncing  0.0 bps, target=#17498449 (75 peers), best: #7676973 (0x726c…2a34), finalized #7676928 (0xf6c1…c683), ⬇ 19.3kiB/s ⬆ 14.7kiB/s
====================
Version: 0.9.41-e203bfb396e
0: sp_panic_handler::set::{{closure}}
1: <alloc::boxed::Box<F,A> as core::ops::function::Fn>::call
at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:2002:9
std::panicking::rust_panic_with_hook
at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:692:13
2: std::panicking::begin_panic_handler::{{closure}}
at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:579:13
3: std::sys_common::backtrace::__rust_end_short_backtrace
at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys_common/backtrace.rs:137:18
4: rust_begin_unwind
at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:575:5
5: core::panicking::panic_fmt
at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panicking.rs:64:14
6: core::result::unwrap_failed
at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/result.rs:1790:5
7: <sp_state_machine::ext::Ext<H,B> as sp_externalities::Externalities>::storage
8: sp_io::storage::get_version_1
9: sp_io::storage::ExtStorageGetVersion1::call
10: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller,A1),R>>::into_func::wasm_to_host_shim
11:
12:
13:
14:
15:
16:
17:
18:
19: wasmtime_runtime::traphandlers::catch_traps::call_closure
20: wasmtime_setjmp
21: sc_executor_wasmtime::runtime::perform_call
22: <sc_executor_wasmtime::runtime::WasmtimeInstance as sc_executor_common::wasm_runtime::WasmInstance>::call_with_allocation_stats
23: sc_executor_common::wasm_runtime::WasmInstance::call_export
24: sc_executor::native_executor::WasmExecutor::with_instance::{{closure}}
25: <sc_executor::native_executor::NativeElseWasmExecutor as sp_core::traits::CodeExecutor>::call
26: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_aux
27: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_using_consensus_failure_handler
28: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt>::call_api_at
29: <kusama_runtime::RuntimeApiImpl<SrApiBlock,RuntimeApiImplCall> as sp_api::Core<SrApiBlock>>::__runtime_api_internal_call_api_at
30: <&sc_service::client::client::Client<B,E,Block,RA> as sc_consensus::block_import::BlockImport>::import_block::{{closure}}
31: <sc_consensus_grandpa::import::GrandpaBlockImport<BE,Block,Client,SC> as sc_consensus::block_import::BlockImport>::import_block::{{closure}}
32: <sc_consensus_beefy::import::BeefyBlockImport<Block,BE,Runtime,I> as sc_consensus::block_import::BlockImport>::import_block::{{closure}}
33: <sc_consensus_babe::BabeBlockImport<Block,Client,Inner> as sc_consensus::block_import::BlockImport>::import_block::{{closure}}
34: <alloc::boxed::Box<dyn sc_consensus::block_import::BlockImport+Transaction = Transaction+Error = sp_consensus::error::Error+core::marker::Sync+core::marker::Send> as sc_consensus::block_import::BlockImport>::import_block::{{closure}}
35: sc_consensus::import_queue::basic_queue::BlockImportWorker::new::{{closure}}
36: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
37: <tracing_futures::Instrumented as core::future::future::Future>::poll
38: tokio::runtime::task::raw::poll
39: std::sys_common::backtrace::__rust_begin_short_backtrace
40: core::ops::function::FnOnce::call_once{{vtable.shim}}
41: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:1988:9
<alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:1988:9
std::sys::unix::thread::Thread::new::thread_start
at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys/unix/thread.rs:108:17
42:
43:
Thread 'tokio-runtime-worker' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Database missing expected key: 0x8d911c369321efa0dc5d5079d12430281450c42bfbb7658febaf1c761444e71f"', /usr/local/cargo/git/checkouts/substrate-7e08433d4c370a21/980eb16/primitives/state-machine/src/ext.rs:192
This is a bug. Please report it at:
    https://github.com/paritytech/polkadot/issues/new

`
Until the crash it starts and works just fine and the speed looks fine too. Although i am seeing extreme I/O expecially when using khala-node (syncing phala + polkadot together) which looks like a whole different issue (looks a lot like https://github.com/paritytech/substrate/issues/9360), but lets skip that for now and focus on the DB issue.
I have 6 systems, 2 ZFS systems and 4 HW-RAID Dells (2x R450 and 2x R730xd) all showing the same issue. And its not that i am doing weird magic with startup:
`#!/bin/sh
WORK_PATH=$(dirname $(readlink -f "$0"))
NODE_NAME="${NODE_NAME:-"polkadot-node"}"
DATA_PATH="/opt/phala-node/polkadot/"
$WORK_PATH/polkadot
--chain polkadot
--base-path $DATA_PATH
--database paritydb
--name $NODE_NAME
--out-peers 50
--in-peers 50
--no-mdns
--blocks-pruning archive-canonical
--state-pruning archive-canonical
--max-runtime-instances 64
--runtime-cache-size 64
--no-hardware-benchmarks
--no-telemetry
`
I am now retrying the sync with @jasl original script (max-run 16, cache 8, out-peers 4, inpeers 20), but i doubt this should impact the DB at this level.
I also noticed nodes sync attempts sometimes just stopped syncing (no errors, nothing, just no block movement) where the finalized block is beyond the best block, which would be kinda weird as well? But unsure if that is related to the trie errors
So for me, i can't get beyond the point where it would sync to 7202158  and 7676973 (lets call this 7-8M range, cause all attempts so far crash in this area) and is pretty fast about it. If i run benchmarks with e.g. Elastic or MySQL the system (also on long term looped querying) is just fast and responsive. If the underlying storage is at fault, it would not sync for such a long time and the issues would start much sooner. @jasl has also nodes on a totally different site and hardware (also Dell with 12x SSD's) running VMware that through this error, totally different hardware and setup.
Ill be happy to switch block sizes in ZFS if you feel that helps ... however i just started 2 new attempts of syncing and they are each syncing in about :
`
polkadot:
2023-04-16 09:57:50 ⚙  Syncing 439.0 bps, target=#15113639 (24 peers), best: #681423 (0x479b…d39a), finalized #680960 (0x53d1…c240), ⬇ 58.3kiB/s ⬆ 2.7kiB/s
kusuma:
2023-04-16 09:56:53 ⚙  Syncing 550.4 bps, target=#17505252 (24 peers), best: #814400 (0x3ea8…7e1f), finalized #814080 (0x3267…bd12), ⬇ 189.2kiB/s ⬆ 7.2kiB/s
`
In the stats i do not get any messages at the I/O is full, there are moments where it is more busy than others, mainly with a lot of random read I/O. During that time the block sync does go to 0 bps for a few lines, then returns back to the original speeds and starts to write with 83MB/s and 107MB/s (funny enough the kusuma one always seems faster). but i would hardly say this I/O would be slow. We have always known node-software is poor on I/O when syncing (which is why HDD's are no-go's), but if i look on how much I/O it does on SSD's it is soo much (e.g. when i sync 200GB, the writes on the SSD increased with 3839GB) even factoring the ZFS parity, having more then 20x the write than data seems weird for me, and this does not happen when i copy/rsync or use other applications with heavy write (e.g. during the MySQL/Elastic testing).
That all said, i am uploading the problem3 and problem4 folders to @jasl at this moment, which are about 200GB each so will take some time to be send over. I'll be happy to see what i can aid in debugging this, but i am not a developer.

@arkpar I've tried to locally replicate the issue 3

There are ~600000 blocks on top of the last finalized block, so cumulus tries to query a huge tree route on startup here:
https://github.com/paritytech/cumulus/blame/master/client/consensus/common/src/level_monitor.rs#L114
and gets stuck.

Unfortunately I'm not able to locally replicate the issue. E.g. the level monitor initialization is smooth even with 3_000_000 blocks between the finalized and the leaf block.
Before putting there some hacky code with a completely arbitrary bound I would like to:

be sure that is a performance degradation and what is the order (and is not some kind of bug)
(if 1 is true) find out what is the bound and the reason

Is this scenario replicated using paritydb as well?
Can I have access to the db ?

@davxy
I've checked again, and it does get stuck in tree_route. I've put some traces there, and it looks like a single call to tree_route only takes a couple of seconds indeed. But it is called for every leaf. And there a lot of leaves because finality is long behind. Each small fork created since final block adds a leaf. So it go to a few hundred thousand leaves. Which is an issue of its own, I guess.

Weird, my colleague upload a new case polkadot.broken3 to that server, and he claims he reproduced Externalities not allowed to fail within runtime: "Trie lookup error: Database missing expected key: 0x294d588cf096b9a001ef2d5817558b007fb67577c41bf3094b6578e93560a941"
but when I validate his upload database, I see the error is
====================

Version: 0.9.41-e203bfb396e

   0: sp_panic_handler::set::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:2002:9
      std::panicking::rust_panic_with_hook
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:692:13
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:577:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys_common/backtrace.rs:137:18
   4: rust_begin_unwind
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:575:5
   5: core::panicking::panic_fmt
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panicking.rs:64:14
   6: core::panicking::panic
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panicking.rs:114:5
   7: parity_db::file::TableFile::read_at
   8: parity_db::column::Column::get_value
   9: parity_db::column::HashColumn::get_in_index
  10: parity_db::column::HashColumn::get
  11: parity_db::db::Db::get
  12: <sc_client_db::parity_db::DbAdapter as sp_database::Database<H>>::get
  13: sc_service::builder::new_db_backend
  14: polkadot_service::new_full
  15: polkadot_cli::command::run_node_inner::{{closure}}::{{closure}}
  16: sc_cli::runner::Runner<C>::run_node_until_exit
  17: polkadot_cli::command::run
  18: polkadot::main
  19: std::sys_common::backtrace::__rust_begin_short_backtrace
  20: main
  21: <unknown>
  22: __libc_start_main
  23: _start

Thread 'main' panicked at 'called Option::unwrap() on a None value', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/parity-db-0.4.4/src/file.rs:116

====================

Version: 0.9.41-e203bfb396e

   0: sp_panic_handler::set::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:2002:9
      std::panicking::rust_panic_with_hook
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:692:13
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:577:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys_common/backtrace.rs:137:18
   4: rust_begin_unwind
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:575:5
   5: core::panicking::panic_fmt
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panicking.rs:64:14
   6: core::panicking::panic
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panicking.rs:114:5
   7: parity_db::file::TableFile::read_at
   8: parity_db::column::Column::get_value
   9: parity_db::column::HashColumn::get_in_index
  10: parity_db::column::HashColumn::get
  11: parity_db::db::Db::get
  12: <sc_client_db::parity_db::DbAdapter as sp_database::Database<H>>::get
  13: sc_service::builder::new_db_backend
  14: polkadot_service::new_full
  15: polkadot_cli::command::run_node_inner::{{closure}}::{{closure}}
  16: sc_cli::runner::Runner<C>::run_node_until_exit
  17: polkadot_cli::command::run
  18: polkadot::main
  19: std::sys_common::backtrace::__rust_begin_short_backtrace
  20: main
  21: <unknown>
  22: __libc_start_main
  23: _start

Thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/parity-db-0.4.4/src/file.rs:116

Hi,
So far the problem keeps popping up, between different hardware systems and configurations. I also retested two Dell R450 with DC SSD's in RAID0 which also gave this error.
So far it was not possible to sync any of time till the current height. That said these are all high-end systems, i also have a Dell R730XS which is not low-end but it only has 10K drives in two RAID6 sets, so I/O wise pretty slow. Interesting enough this node just passed block 10M which is the furthest i have gotten any of the nodes. The only difference is that this machine is very slowly syncing (like barely 1 bps). Which started me to think if this might be triggered because it is syncing too fast.
We always had problems with node syncing, specifically I/O bound (which i see some fixes might be in that area now by @davxy ) which led to me thinking in last night about old notes i have on this topic. My idea so far:

We know old nodes have some bad blocks, in the past the node would 'just step over it' and even that broken block is served out to clients (which refused them obviously), and people switched their node to another, sync over it, then switch back. This happened on random blocks and never found a cause.
The trie error seems to be similar to this behavior, but it handles this error bricking the node and crashing it. Although it is logical behavior there is no way (for me?) to recover the database. I would think that up to a certain block the database is still fine. Having some kind of recovery would be a nice-to-have. There is a fsck tool that can scan a running node for bad block/chains, but only report IF there is an issue, not fix it.
We know that over time not only I/O affects syncing (gets worse) but also the virtual memory usage. This stops once the node is syncs and returns to 'normal' after the first restart of a synced node. (which might be solved with @davxy fix) ... either way, there is a build-up. If @davxy has a build for the patched version, i'd be happy to try this.

During my testing time a few DB's got lost and removed before fully sending to @jasl, which is totally my fault and caused some delays. Currently i have 3 broken DB's with Trie errors (2 crashing type, 1 looping type) for which the first (problem6)  is currently being uploaded to @jasl to double check this. The other two are larger (140 problem7 and 180GB problem8) which take more time to upload, but i can provide a reverse shell to the system if you want to take a look.
So far i have tested them with either TrueNAS on ZFS directly, or inside a VM with ext4. My next step is to reinstall the machines (after uploading all the data ofc) with Ubuntu 22.04 LTS and MDADM raid and retry syncing. I am also retrying a sync but with RocksDB instead of ParityDB on the same setup where the problem6 happened.

FINALLY! Trie lookup error triggered, and can reproduce on that VM
this is database polkadot.broken6

@bkchr @arkpar My new Polkadot node in a new VM triggered this too, it has the same backtrace just like broken6 on that server

@jasl Would it be possible to share the whole VM?

@jasl Would it be possible to share the whole VM?

Sure!
added your ssh pub key, ssh<EMAIL_ADDRESS>it's still running Polkadot v0.9.41, start script ~/bin/start_polkadot_node.sh, data folder ~/data/polkadot-node
ubuntu@gr-phat-cluster-main:~$ bin/start_polkadot_node.sh
2023-04-22 09:39:04 Parity Polkadot
2023-04-22 09:39:04 ✌  version 0.9.41-e203bfb396e
2023-04-22 09:39:04 ❤  by Parity Technologies<EMAIL_ADDRESS>2017-2023
2023-04-22 09:39:04 📋 Chain specification: Polkadot
2023-04-22 09:39:04 🏷  Node name: polkadot-node
2023-04-22 09:39:04 👤 Role: FULL
2023-04-22 09:39:04 💾 Database: ParityDb at //home/ubuntu/data/polkadot-node/chains/polkadot/paritydb/full
2023-04-22 09:39:04 ⛓  Native runtime: polkadot-9401 (parity-polkadot-0.tx22.au0)
2023-04-22 09:39:07 🏷  Local node identity is: 12D3KooWGBLzN3g6VbpcV1fp9RidjWBH7cqUEeJqpmnEP3iPpnub
2023-04-22 09:39:07 💻 Operating system: linux
2023-04-22 09:39:07 💻 CPU architecture: x86_64
2023-04-22 09:39:07 💻 Target environment: gnu
2023-04-22 09:39:07 💻 CPU: Intel(R) Xeon(R) E-2386G CPU @ 3.50GHz
2023-04-22 09:39:07 💻 CPU cores: 6
2023-04-22 09:39:07 💻 Memory: 31492MB
2023-04-22 09:39:07 💻 Kernel: 5.15.0-70-generic
2023-04-22 09:39:07 💻 Linux distribution: Ubuntu 22.04.2 LTS
2023-04-22 09:39:07 💻 Virtual machine: no
2023-04-22 09:39:07 📦 Highest known block at #7347283
2023-04-22 09:39:07 〽 Prometheus exporter started at <IP_ADDRESS>:9615
2023-04-22 09:39:07 Running JSON-RPC HTTP server: addr=<IP_ADDRESS>:9933, allowed origins=["http://localhost:*", "http://<IP_ADDRESS>:*", "https://localhost:*", "https://<IP_ADDRESS>:*", "https://polkadot.js.org"]
2023-04-22 09:39:07 Running JSON-RPC WS server: addr=<IP_ADDRESS>:9944, allowed origins=["http://localhost:*", "http://<IP_ADDRESS>:*", "https://localhost:*", "https://<IP_ADDRESS>:*", "https://polkadot.js.org"]

====================

Version: 0.9.41-e203bfb396e

   0: sp_panic_handler::set::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:2002:9
      std::panicking::rust_panic_with_hook
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:692:13
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:579:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys_common/backtrace.rs:137:18
   4: rust_begin_unwind
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:575:5
   5: core::panicking::panic_fmt
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panicking.rs:64:14
   6: core::result::unwrap_failed
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/result.rs:1790:5
   7: <sp_state_machine::ext::Ext<H,B> as sp_externalities::Externalities>::storage
   8: sp_io::storage::get_version_1
   9: sp_io::storage::ExtStorageGetVersion1::call
  10: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller<T>,A1),R>>::into_func::wasm_to_host_shim
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: <unknown>
  15: <unknown>
  16: wasmtime_runtime::traphandlers::catch_traps::call_closure
  17: wasmtime_setjmp
  18: sc_executor_wasmtime::runtime::perform_call
  19: <sc_executor_wasmtime::runtime::WasmtimeInstance as sc_executor_common::wasm_runtime::WasmInstance>::call_with_allocation_stats
  20: sc_executor_common::wasm_runtime::WasmInstance::call_export
  21: sc_executor::native_executor::WasmExecutor<H>::with_instance::{{closure}}
  22: <sc_executor::native_executor::NativeElseWasmExecutor<D> as sp_core::traits::CodeExecutor>::call
  23: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_aux
  24: sp_state_machine::execution::StateMachine<B,H,Exec>::execute_using_consensus_failure_handler
  25: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
  26: <polkadot_runtime::RuntimeApiImpl<__SrApiBlock__,RuntimeApiImplCall> as sp_api::Core<__SrApiBlock__>>::__runtime_api_internal_call_api_at
  27: <&sc_service::client::client::Client<B,E,Block,RA> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
  28: <sc_consensus_grandpa::import::GrandpaBlockImport<BE,Block,Client,SC> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
  29: <sc_consensus_beefy::import::BeefyBlockImport<Block,BE,Runtime,I> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
  30: <sc_consensus_babe::BabeBlockImport<Block,Client,Inner> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}}
  31: <alloc::boxed::Box<dyn sc_consensus::block_import::BlockImport<B>+Transaction = Transaction+Error = sp_consensus::error::Error+core::marker::Sync+core::marker::Send> as sc_consensus::block_import::BlockImport<B>>::import_block::{{closure}}
  32: sc_consensus::import_queue::basic_queue::BlockImportWorker<B>::new::{{closure}}
  33: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
  34: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
  35: tokio::runtime::task::raw::poll
  36: std::sys_common::backtrace::__rust_begin_short_backtrace
  37: core::ops::function::FnOnce::call_once{{vtable.shim}}
  38: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:1988:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:1988:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys/unix/thread.rs:108:17
  39: <unknown>
  40: <unknown>

Thread 'tokio-runtime-worker' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Database missing expected key: 0x14f3857fd6a01a2bdb49617734950a21e47804875c81d3953d26c0fe69a6a99e"', /usr/local/cargo/git/checkouts/substrate-7e08433d4c370a21/980eb16/primitives/state-machine/src/ext.rs:192

This is a bug. Please report it at:

        https://github.com/paritytech/polkadot/issues/new

@arkpar i think @jasl gave SSH access to the machine, it is in Azure also the disk is like 1TB to receive multiple DB's so its quite large to download i think. I can add SSH keys if yours is not added

I am nearly completed syncing 4 different nodes with rocksdb, 2 of them are nearly in the end-zone ... but the DB speed is still terrible after block 8M.

Sure!
added your ssh pub key, ssh<EMAIL_ADDRESS>it's still running Polkadot v0.9.41, start script ~/bin/start_polkadot_node.sh, data folder ~/data/polkadot-node

I've ran a latest polkadot master on this machine and it synced with no issues (/home/ubuntu/tempdb/).

Sure!
added your ssh pub key, ssh<EMAIL_ADDRESS>it's still running Polkadot v0.9.41, start script ~/bin/start_polkadot_node.sh, data folder ~/data/polkadot-node

I've ran a latest polkadot master on this machine and it synced with no issues (/home/ubuntu/tempdb).

It's not stable to reproduce, and unpredictable which block will fail, @PHA-SYSOPS makes a 6 machines cluster with different hardware to sync round and round and can reproduce a few times.
Do you find something in the broken DB?

Polkadot master branch can also continue syncing the database in /home/ubuntu/data/polkadot-node. So it is not really broken. As it was mentioned before in this thread, this issue is fixed in  parity-db 0.4.6 and polkadot master.

Polkadot master branch can also continue syncing the database in /home/ubuntu/data/polkadot-node. So it is not really broken. As it was mentioned before in this thread, this issue is fixed in parity-db 0.4.6 and polkadot master.

OK, thank you for your time, I'll close this issue now


Cost Monitoring

This is a large issue that we see come up a lot. Would like to have some way to assess costs of specific users and groups.
Currently the most promising one is kubecost.

A prototype is in the kubecost branch, ideally merge #591 first since kubecost branches off that.


Breadcrumbs in Raven

Hi,
I am using Raven in my openERP project to log to Sentry. During this, I have observed that the breadcrumbs limit cannot be set by the client through the API. I can only enable_breadcrumbs or disable them. I would like to log breadcrumbs for ERROR only.
To summarise, I am looking forward to:

Set the limit for number of breadcrumbs from client.
To log breadcrumbs for specific log levels.

Below is additional information:
I have gone through the Raven package and understood that we are getting breadcrumbs if below statement is true-
insert breadcrumbs
if self.enable_breadcrumbs:
Also breadcrumbs limit is set to 100, in breadcrumbs.py that can't be changed by client.
Please let me know if we have a workflow to get these items through the API else, can these changes be provided in your upcoming releases.
Thanks,
Prasantarmaye

This is the repository for the Node.js SDK; it sounds like you're looking for https://github.com/getsentry/raven-python.


FancyZonesEditor not working anymore

Microsoft PowerToys version
0.60.0
Running as admin

[X] Yes

Area(s) with issue?
FancyZones, FancyZones Editor
Steps to reproduce
I connect via remote desktop to a remote laptop running windows 11 with powertoys 0.60.0. I have two monitors and fancy zones won't do anything anymore. Starting the fancy zones editor a message appears to report the bug but no further information what went wrong.
✔ Expected Behavior
Fancy zones work or the editory opens.
❌ Actual Behavior
activating fancy zones, nothing happens and opening the editor throws an error.
Other Software
No response

I have same issue

https://github.com/microsoft/PowerToys/issues/19298

PowerToysReport_2022-07-11-14-31-12.zip
of course. see attached zip file.

Thank you!

/dup of https://github.com/microsoft/PowerToys/issues/19223


(layers) Added support for LCM protocol

This patch features support for LCM layers. The code is based on the LCM
protocol definition (https://lcm-proj.github.io/).
LCM is a lightweight communications protocol that is used e.g. in the
automotive industry for testing and developing purposes.
The submitted patch allows for decoding LCM headers plus the registration
of underlying decoders (which should be used in combination with an
incoming patch towards the LCM GitHub repository that will emit gopacket
code based on LCM defintions).

I signed it!

Hi @TacoVox . Great to have a new contrib.
I am not a LCM expert, so hard to review the code on the dissection part.
A small comment : looks like you did not used gofmt ?
https://github.com/google/gopacket/blob/master/CONTRIBUTING.md specifies a few rules, we try to enforce before merging PR. Would you please go through them ?
Thanks again !

Hi @lhausermann!
Yes, everthing is formatted using gofmt. It has to, due to .travis.gofmt.sh.
So which lines are you referring to?

I was thinking first that the way the main struct fields are misaligned was a non formatting issue.. but I realize that is due to the comments in the middle.
Just ignore my initial remark.

Hi @gconnell!
Thanks for your nice comments. I improved the code based on that.
The particularly liked the call for returning gopacket.LayerTypeFragment in case of a fragmented message. I actually haven\t told you that I open another PR after this one is in for a defragmenter of LCM messages that will then do the defragmentation :sunglasses:
@lhausermann Nooo problem :)

Thanks for the help!


image-sets-view: fixes image-sets view org-id selection.

Fixes # (issue)
Type of change
What is it?

[X] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Documentation update
[ ] Tests update
[ ] Refactor

Checklist:

[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[X] New and existing unit tests pass locally with my changes
[X] I run make pre-commit to check fmt/vet/lint/test-no-fdo

Codecov Report
Base: 45.43% // Head: 45.46% // Increases project coverage by +0.03% :tada:

Coverage data is based on head (1b45995) compared to base (507febf).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1774      +/-   ##
==========================================
+ Coverage   45.43%   45.46%   +0.03%
==========================================
  Files          70       70
  Lines        9816     9816
==========================================
+ Hits         4460     4463       +3
+ Misses       4921     4919       -2
+ Partials      435      434       -1

Impacted Files
Coverage Δ

pkg/services/imagesets.go
52.90% <100.00%> (ø)

pkg/services/devices.go
50.43% <0.00%> (+0.32%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Fix. Typo Angular LCI to Angular CLI

Changing link title from Angular LCI to Angular CLI

@aelmaghrabi : Thanks for your contribution! The author(s) have been notified to review your proposed change.


When you add a datatable as anything but the last step within background, all steps after the datatable get deleted

So with the change I made in https://github.com/olexale/bdd_widget_test/pull/60 I have found an issue I'm having difficulty fixing.
An example:
Background:
    Given the following songs
    | artist           | title                |
    | 'The Beatles'    | 'Let It Be'          |
    | 'Camel'          | 'Slow yourself down' |
    And I wait

  Scenario: Testing scenario
    Given the following users exist <name> <twitter>
    | name            | twitter       |
    | 'Oleksandr'     | '@olexale'    |
    | 'Flutter'       | '@FlutterDev' |

should give:
// GENERATED CODE - DO NOT MODIFY BY HAND
// ignore_for_file: unused_import, directives_ordering

import 'package:bdd_widget_test/data_table.dart' as bdd;
import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';

import './step/the_following_songs.dart';
import './step/i_wait.dart';
import './step/the_following_users_exist.dart';

void main() {
  group(\'\'\'Testing feature\'\'\', () {
    Future<void> bddSetUp(WidgetTester tester) async {
      await theFollowingSongs(tester, const bdd.DataTable([[artist, title], ['The Beatles', 'Let It Be'], ['Camel', 'Slow yourself down']]));
      await iWait(tester);
    }
    testWidgets(\'\'\'Testing scenario\'\'\', (tester) async {
      await bddSetUp(tester);
      await theFollowingUsersExist(tester, 'Oleksandr', '@olexale');
      await theFollowingUsersExist(tester, 'Flutter', '@FlutterDev');
    });
  });
}

but currently gives:
// GENERATED CODE - DO NOT MODIFY BY HAND
// ignore_for_file: unused_import, directives_ordering

import 'package:bdd_widget_test/data_table.dart' as bdd;
import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';

import './step/the_following_songs.dart';
import './step/the_following_users_exist.dart';

void main() {
  group(\'\'\'Testing feature\'\'\', () {
    Future<void> bddSetUp(WidgetTester tester) async {
      await theFollowingSongs(tester, const bdd.DataTable([[artist, title], ['The Beatles', 'Let It Be'], ['Camel', 'Slow yourself down']]));
    }
    testWidgets(\'\'\'Testing scenario\'\'\', (tester) async {
      await bddSetUp(tester);
      await theFollowingUsersExist(tester, 'Oleksandr', '@olexale');
      await theFollowingUsersExist(tester, 'Flutter', '@FlutterDev');
    });
  });
}

With this issue I would also like to ask how you can debug a project like this? If It try to run tests from file instead of the overarching flutter test command, I get the following error: Dart library 'dart:ui' is not available on this platform

Thanks for raising the issue! I fixed in https://github.com/olexale/bdd_widget_test/commit/7f3a38d6fa3f2ef92cc33f08aec82e1a1005cec4
When I need to debug the package, I write a failing test.


Do virtual lists make sense in Lightning?

Virtual lists for DOM have shown they can be much more performant than simply rendering large lists of items.

I was wondering though if this technique is going to matter in the realm of Lightning. My approach would be to simply always set a fixed size of children for an element (say 20, based on the viewport size) and then simply remove/add  these items from children as I'm scrolling.
Will this work or is Lightning smart enough to do this behind these scenes when setting children?

@lmatteis-wm
Lighting will instantiate the elements outside the screen but it will not render them. If your issue is a long loading time, cutting down initial number of elements should help a lot. If your problem is rendering performance (e.g. during actions like scrolling), I wouldn't expect as much of an improvement since Lightning has many optimisations in that area.
While Lightning has many optimisations by default, it cannot account for performance problems specific to your application. Any optimisation tailored specifically to your problem is likely to be better than Lightning's default.
Best regards

Hi @lmatteis-wm, it has been over a year since this issue was opened, and we appreciate the question you raised. As @g-zachar mentioned in their response, Lightning has built-in optimisations for rendering performance, but it might not account for all specific use cases.
Since there hasn't been any further discussion or follow-up questions, I will be closing this issue. If you have any additional questions or concerns related to this topic, feel free to reopen the issue or create a new one.


Change author

closes #14

There seems to be some problem with the BoundaryCondition. Did we introduce breaking changes in Oceananigans? I thought the changes were not breaking.

Hm... We call a change breaking if it's breaking for the user-facing API we export...

Hm... We call a change breaking if it's breaking for the user-facing API we export...

I take it back! Things shouldn't break in a patch release!

Do we need to change the license?


[DO NOT MERGE] Upgrade Grafana via Docker

GOV.UK have recently undertaken work to remove the old ELK stack that was running Elasticsearch 1.4. As I understand it, running the older version caused problems when trying to upgrade Grafana as we use Elasticsearch as a data source. We're now using Logit, and I wish to test the latest version of Grafana against the Logit backend (to see if this is possible).
I will probably try to test this change in AWS to begin with, and then if successful, move on to our current live environments.
Upgrade Grafana by running it in a container rather than installing as a package.
This is officially documented here:
http://docs.grafana.org/installation/docker/
All configuration options can be passed in as environment variables, which makes it straight forward to pass in any additional arguments in the future.
To upgrade we would need to update the image tag we use, which should make life a little easier (in theory).
I considered moving all the dashboards across and removing the grafana module, but decided it was clearer to leave them where they were. I moved across the grafana.ini and vhost config files to be with the container.

Going to close this off for the time-being.


Typo: Missing double -- for full parameter name

The command given is
az ml job stream -name <job_name>

Notice it is using single - but full parameter name
It should either be -n or --name
az ml job stream -n <job_name>
az ml job stream --name <job_name>

Given the subsequent commands use --name that is likely what it should match for clarity

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: b2daa5dd-be4d-2da5-2fcf-472c8975ce43
Version Independent ID: 9014e67d-b34c-a7b3-85af-8ffb2e01dc4c
Content: Troubleshooting batch endpoints - Azure Machine Learning
Content Source: articles/machine-learning/how-to-troubleshoot-batch-endpoints.md
Service: machine-learning
Sub-service: core
GitHub Login: @santiagxf
Microsoft Alias: fasantia

@mattmazzola
Thanks for your feedback! We will investigate and update as appropriate.

@mattmazzola
Thanks for reporting this! We have created a PR for this issue and the changes should go live soon.


[GUI] - Error when pass the APPLICATION_URL flag

I'm submitting a ...

[x] bug report
[ ] feature request
[ ] support request

Do you want to request a feature or report a bug?
Bug

What is the current behavior?

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem
When we pass the flag APPLICATION_URL none of the labels are loaded. This appears to be a i18n error

What is the expected behavior?

What is the motivation / use case for changing the behavior?

Please tell us about your environment:

Version: 0.5.2
Environment: docker-compose
Operating system: [Ubuntu 18.04]

Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, gitter, etc)

To change the URL of dojot GUI for navegation i tried to set APPLICATION_URL enviroment variable.
Also change the kong.config.sh like above

When i access the gui interface the labels don't appears correctly.


HNC: Add critical condition to both culprit namespaces when they cause a cycle.

Right now if setting the parent of a namespace causes a cycle, only the parent will get CritParentInvalid condition. When restarting HNC, it is nondeterministic which namespace in the cycle will have CritParentInvalid condition. For details, see discussions in #660.
Ideally, if we set parent of a to b and it causes a cycle, the condition should be on both a and b and not on other namespaces in the cycle. Furthermore, the locations of the condition should not change when restarting HNC.
Since it is possible that b might be a descendant of a but not a child of a, we will change the condition name to exclude the word "parent", e.g., CritCycle.

cc:
@adrianludwin
@adrianchung
@yiqigao217


Facebook Bots Not Working

Hey Hello,
Me agar.io bot  in facebookManager.js using but maximum 5 facebook bots to me ?
I have 500 facebook cookie available help me.
Good Work!

maybe ur facebook cookies dont work

@LostCodesy
Welcome to the army of facebook peoples
https://github.com/pulviscriptor/agario-client/search?p=1&q=fb&type=Issues&utf8=✓
https://github.com/pulviscriptor/agario-client/search?utf8=✓&q=facebook&type=Issues
Your problem is not fixable by agario-client. You 're in the wrong place.


Update DeviceUID.m

Description
Fixed issue #

Added yourNewMethodName() that allows ...
update.DeciceUID
add get appleIFA
Compatibility

OS
Implemented

iOS
✅

Checklist

[ x] I have tested this on a device/simulator for each compatible OS
[ ] I added the documentation in README.md.
[ x] I mentionned this change in CHANGELOG.md.

What's appleIFA ?


add teamviewer 15.10.5

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?

Microsoft Reviewers: Open in CodeFlow

@KevinLaMS it works on my end:

I uninstalled teamviewer before doing what is shown in the screenshot

Kevin... Why do you always have issues with Teamviewer?..

@KevinLaMS Depending on your UAC configuration, the installer would trigger a dialog box. Make sure to allow the process if you want the installation to start.

@KevinLaMS It looks like the download failed for you, yet the installer was validated in the Validation URLs in manifest files pipeline.
Could you try once more?

Duplicate of #4150

I would say #4150 is the duplicate in this case and the user ignored the "Have you checked that there aren't other open pull requests for the same manifest update/change?" requirement but you merged the duplicated PR anyway.
This could encourage people to copy the changes of a pending PR since the latest PR has a bigger chance of getting merged.


Use PUDL metadata sub package for dbf2sqlite function

Currently the dbf2sqlite function:

Drops existing tables from the ferc1.sqlite db using pudl.helpers.drop_tables()
Gathers ferc1 table names using pudl.extract.ferc1.get_dbc_map()
define_sqlite_db() Creates sqlalchemy models for each table.
Extracts the raw data and writes it to the database.
Reads the f1_respondent_id tables from the db, does some pandas processing to insert missing ids, writes the changes back to the database.

Is it possible to use PUDL's metadata sub package to handle the schema creation? This would allow us to re use the SQLite IO Manager #1972 we will create for the PUDL ETL, output and analysis table. If this is too big a lift, I think we could just wrap the dbf2sqlite function in an op and create an io manager for _etl_ferc1() that can read tables from the ferc1.sqlite database.

Is it possible to use PUDL's metadata sub package to handle the schema creation?

This isn't currently possible. The only VisualFoxPro / DBF schema information we have is their actual database files (I think it's mostly stored in the DBC files). There are 116 tables and more than 3000 columns in the original FERC 1 DBF databases, so actually specifying all that structure would be a lot of work, and the database doesn't seem to enforce types or foreign key relationships, and we have no descriptions (other than the f1_row_lit_tbl entries) of what the various columns mean.
However, given a programmatically generated SQL Alchemy schema it seems like it should be possible to transform that structural information into other forms. We were going to have to do some kind of generalization / extension of that kind to get the FERC Form 2 (and any other FERC DBF data) loaded into SQLite anyway.  It would be nice to be able to publish schemas (even if they're mostly machine generated) for the FERC SQLite DBs that we create, alongside the ones that are being created for the XBRL data and the PUDL DB.

Got it. This doesn't seem super feasible or useful right now so I'm going to close this issue.


[BUG] Infinite Paging & Duplication (Settings GUI)

Plugin Version: 1.6.5
Minecraft Version:  (Paper Spigot) 1.15.2-R0.1-SNAPSHOT
Summary:
If you change the groups.yml file and then do /ep reload this will cause the settings GUI to break (however still functional). See the images below for examples. This will also cause the user to be able to page infinitely forwards or backward.
Normal

After /ep reload

Thank you!


id="runtime" appears twice in the HTML

id="runtime" appears twice in the https://github.com/golang/go/blob/master/doc/go1.14.html
This is invalid because the id must be unique throughout the HTML document.
(https://github.com/golang/go/blob/master/doc/go1.13.html has the same problem. ref: https://go-review.googlesource.com/c/go/+/192977)

related: #36878

@dmitshur

Thanks for letting us know! I'll send a fix.


[FIX] Update environment variable name and link to query tool README

Closes #179

Changes proposed in this pull request:

API_QUERY_URL replaced by NB_API_QUERY_URL
Using mkdocs-include-markdown-plugin, query tool documentation page contents replaced by link to query tool README sections where appropriate for easier syncing

Checklist
Please leave checkboxes empty for PR reviewers

[ ] PR has an interpretable title with a prefix ([ENH], [FIX], [REF], [TST], [CI], [MNT], [INF]) see our Contributing Guidelines for more info)
[ ] PR links to GitHub issue with mention Closes #XXXX
[ ] Checks pass

Hey @rmanaem, thanks for catching that!
I think since the deployment recipes and docs have been already modified anyways related to this variable, I would say we can go ahead and change the port to the one used by the container to the Vite default for now. This way, at least our current instructions will work in the interim before we switch to the single-Compose recipe, and we can decide when we refine that new single-Compose file whether we want to keep this port or not.
Will request your review on a PR for this port change in the recipes repo !


Nest notes under their relevant highlight

It would be ideal if notes were nested under the highlight they are applied to. This would create a more contextual experience, and reflect better how most readers actually take notes, which is attached to specific clippings of text.
Current behavior:

Desired:

Thanks!
Despite this small issue, the plugin has massively improved my workflow (and life!)

I'm happy to play around with the source and get a working demo up if that would be helpful, or if this is not a change that the maintainer wants to implement, I'd be happy to fork and do it myself!


Update README.md

Update Code Inspector public dashboard URL

Codecov Report

Merging #103 into master will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #103      +/-   ##
==========================================
- Coverage   74.11%   74.09%   -0.02%
==========================================
  Files         215      218       +3
  Lines        6552     6574      +22
  Branches     3412     3425      +13
==========================================
+ Hits         4856     4871      +15
- Misses        813      815       +2
- Partials      883      888       +5

Impacted Files
Coverage Δ

core/libp2p/upgrader/connection_upgrader.hpp
100% <0%> (ø)

.../core/libp2p/upgrader/connection_upgrader_test.cpp
50% <0%> (ø)

.../libp2p/upgrader/impl/connection_upgrader_impl.cpp
100% <0%> (ø)

core/libp2p/muxer/yamux/yamux.cpp
63.05% <0%> (+0.4%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b2202f9...02e8b9e. Read the comment docs.


Embeddable attribute override support

This allows for overriding of embeddable mapping definitions from the parent entity/embeddable.
Nested embeddables are supported via dot notation along with the ability to drop column(s) from the embedded object if necessary.
The syntax is equivalent to attribute overrides on a subclass with the addition of passing a null value for a overridden column to drop it from the parent's field mappings.
Via annotations:
/** @Embeddable */
class ValueObject {

    /** @Column(type="string") */
    public $value;

    /** @Column(type="integer") */
    public $count;

}

/** @Embeddable */
class NestedValueObject {

    /** @Embedded(class="ValueObject", columnPrefix=false) */
    public $nested;

}

/** @Entity */
class FooEntity {

    /** @Id @Column(type="integer") */
    private $id;

    /**
     * @Embedded(class="ValueObject", columnPrefix=false)
     * @AttributeOverride(name="value", column=@Column(type="string", name="single_value_override"))
     */
    public $valueObjectWithSingleOverride;

    /**
     * @Embedded(class="ValueObject", columnPrefix=false)
     * @AttributeOverrides({
     *     @AttributeOverride(name="value", column=@Column(type="string", name="multiple_value_override")),
     *     @AttributeOverride(name="count", column=null)
     * })
     */
    public $valueObjectWithMultipleOverrides;

    /**
     * @Embedded(class="NestedValueObject", columnPrefix=false)
     * @AttributeOverrides({
     *     @AttributeOverride(name="nested.value", column=@Column(type="string", name="nested_value_override")),
     *     @AttributeOverride(name="nested.count", column=null)
     * })
     */
    public $valueObjectWithNestedOverrides;
}

Via XML:
<?xml version="1.0" encoding="UTF-8"?>
<doctrine-mapping xmlns="http://doctrine-project.org/schemas/orm/doctrine-mapping"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://doctrine-project.org/schemas/orm/doctrine-mapping
                            https://raw.github.com/doctrine/doctrine2/master/doctrine-mapping.xsd">
    <embeddable name="ValueObject">
        <field name="value" type="string" />
        <field name="count" type="integer" />
    </embeddable>
</doctrine-mapping>

<?xml version="1.0" encoding="UTF-8"?>
<doctrine-mapping xmlns="http://doctrine-project.org/schemas/orm/doctrine-mapping"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://doctrine-project.org/schemas/orm/doctrine-mapping
                            https://raw.github.com/doctrine/doctrine2/master/doctrine-mapping.xsd">
    <embeddable name="NestedValueObject">
        <embedded name="nested"
                class="ValueObject"
                use-column-prefix="false"/>
    </embeddable>
</doctrine-mapping>

<?xml version="1.0" encoding="UTF-8"?>
<doctrine-mapping xmlns="http://doctrine-project.org/schemas/orm/doctrine-mapping"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://doctrine-project.org/schemas/orm/doctrine-mapping
                            https://raw.github.com/doctrine/doctrine2/master/doctrine-mapping.xsd">
    <entity name="FooEntity" table="entity">
        <id name="id" column="id"/>
        <embedded name="valueObjectWithSingleOverride"
                class="ValueObject"
                use-column-prefix="false">
            <attribute-override name="value">
                <field column="single_value_override" type="string"/>
            </attribute-override>
        </embedded>
        <embedded name="valueObjectWithMultipleOverrides"
                class="ValueObject"
                use-column-prefix="false">
            <attribute-overrides>
                <attribute-override name="value">
                    <field column="multiple_value_override" type="string"/>
                </attribute-override>
                <attribute-override name="count"/>
            </attribute-overrides>
        </embedded>
        <embedded name="valueObjectWithNestedOverrides"
                class="NestedValueObject"
                use-column-prefix="false">
            <attribute-overrides>
                <attribute-override name="nested.value">
                    <field column="nested_value_override" type="string"/>
                </attribute-override>
                <attribute-override name="nested.count"/>
            </attribute-overrides>
        </embedded>
    </entity>
</doctrine-mapping>

Via YML:
ValueObject:
  type: embeddable
  fields:
    value:
      type: string
    count:
      type: integer

NestedValueObject:
  type: embeddable
  embedded:
    nested:
      class: ValueObject
      columnPrefix: false

FooEntity:
  type: entity
  id:
    id:
      type: integer
  embedded:
    valueObjectWithSingleOverride:
      class: ValueObject
      columnPrefix: false
      attributeOverrides:
        value:
          column: single_value_override
          type: string
    valueObjectWithMultipleOverrides:
      class: ValueObject
      columnPrefix: false
      attributeOverrides:
        value:
          column: multiple_value_override
          type: string
        count:
          column: null
    valueObjectWithNestedOverrides:
      class: NestedValueObject
      columnPrefix: false
      attributeOverrides:
        nested.value:
          column: nested_value_override
          type: string
        nested.count:
          column: null

A further improvement could be to merge the column override over the top of the original column instead of replacing it.  Doing so would remove the need to redefine original column attributes even if they have not been changed.  This change is outside the scope of this pull request.
References #6047

It would be awesome to have this functionality. Can you tell me how to applly these changes using composer? (to not wait for merge)

@Arkemlar yes, instructions are on the composer website: https://getcomposer.org/doc/05-repositories.md#vcs

@andy-shea Thank you! Installation successful. Working with Embeddables much more easier now.
Btw is it really needs php ^5.6 version? My dev system is ok with it but production runs on 5.6 so it needs upgrade.

I found bug.
Setting up overridings in one entity automaticly reflects on other entities:
I have embeddable with two fields:
<MyEmbeddable>
  type: embeddable
  fields:
    valueDefault:
      column: default
      type: string
      length: 255
      nullable: false   # This should be NOT NULL by default
    valueTranslated:
      column: translated
      type: string
      length: 255
      nullable: true

When I set up overridings in Product mapping, mapping in other tables changes:
<MyProductEntity>
  embedded:
    name:
      class: <MyEmbeddable>
      attributeOverrides:
        valueDefault:
          column: default
          nullable: true
    manufacturer:
      class: <MyEmbeddable>
      attributeOverrides:
        valueDefault:
          column: default
          nullable: true

When overriding is on:
$ bin/console doctrine:schema:update --dump-sql
ALTER TABLE product CHANGE name_default name_default VARCHAR(255) DEFAULT NULL, CHANGE name_translated name_translated VARCHAR(255) DEFAULT NULL, CHANGE manufacturer_default manufacturer_default VARCHAR(255) DEFAULT NULL, CHANGE manufacturer_translated manufacturer_translated VARCHAR(255) DEFAULT NULL;
(Down below is unexpected)
ALTER TABLE product_category CHANGE name_default name_default VARCHAR(255) DEFAULT NULL;
ALTER TABLE product_mark CHANGE name_default name_default VARCHAR(255) DEFAULT NULL, CHANGE description_default description_default VARCHAR(255) DEFAULT NULL;

When overriding is off (commented whole 'embedded' section in Product mapping):
$ bin/console doctrine:schema:update --dump-sql
ALTER TABLE product CHANGE name_default name_default VARCHAR(255) NOT NULL, CHANGE name_translated name_translated VARCHAR(255) DEFAULT NULL, CHANGE manufacturer_default manufacturer_default VARCHAR(255) NOT NULL, CHANGE manufacturer_translated manufacturer_translated VARCHAR(255) DEFAULT NULL;

Hi,
that will be useful for me. When will this be accepted?

I think this feature is mandatory when working with value objects. Until the merge is done, is there any alternative to map orm value objects with multiple fields columns names?

Looking at this, and I don't see the feature as necessary.
Overrides on embeddables are indeed useful, but for the few edge cases where they apply, a metadata listener would suffice.
That's my take on it at least, so I won't merge and will pass the token to the next maintainer looking at this.

@Ocramius
Then what's the way to map the orm on a table that has "phone1" and "phone2", and you want the same value object "phone" to be used?

@ovidals you just change the columns to be phone1_phone and phone2_phone. Then you can use your Phone embeddable for both the $phone1 and the $phone2 properties of your object.

@ovidals you just change the columns to be phone1_phone and phone2_phone. Then you can use your Phone embeddable for both the $phone1 and the $phone2 properties of your object.

@Ocramius Can you show us an example, please? Do you mean to change de db column names? it's a legacy code so that's not possible.
Reference: http://stackoverflow.com/questions/41246632/doctrine-orm-value-object-for-mutiple-database-fields

Do you mean to change de db column names? it's a legacy code so that's not possible.

Don't expect doctrine to always 1:1 map on legacy stuff. A bit of flexibility here is required, like a r/w view for the legacy app, and moving the table to a newer structure. Expecting Doctrine to fix your legacy problems is out of question.

What about my bug report I provided above? It needs to be fixed I suppose...

@Ocramius my use case for this feauture is overriding some column settings:

'Product' entity has embeddable 'Name' with $name and $nameTranslated fields
'ProductCategory' has embeddable 'Name' too
for 'Product' names I want allow nullable values for both variables (nullable=true) due to business needs
for 'ProductCategory' I want the $name to be set as nullable=false and $nameTranslated to be set as nullable=true

Generally:

I use this embeddable in other entities too and may need to change varchar length or nullable flag for some entities so it will differ from default implementation
I want to use one embeddable class everywhere (to avoid a lot of descendants with one/two rewritten property)

I don't see the way to make this without this feature.

This is a no go for me too...
Defining the attribute overrides in the class level should be sufficient IMO (and according to @schmittjoh embeddables were designed to be overridden in that way).
I've just debugged the ClassMetadataFactory and the reason that prevents it is because we're calling ClassMetadataInfo#setAttributeOverride() while loading the class metadata which happens before of inlining the class' embeddables.
With that said, I think that we don't need a new way to override attributes but rather fix the one we have now, the only thing is that it can be a bit tricky to fix it properly (without introducing a lot of duplication but is doable IMO).

Closing as per @lcobucci's feedback. Sorry @andy-shea, marking as won't fix :-\

I couldn't make work embeddable overriding and I try to find another way. The way I found is using doctrine events. I build an EntityListener which is subscribed to postLoad event and where I build my ValueObject and inject it to my Entity.
What do you think about it?

@danitome24 as said on https://github.com/doctrine/doctrine2/pull/6151#issuecomment-271432401 the proper solution would be fixing the order of the methods' calls (to make sure that the metadata state is right).


Proof of concept for find failing on ambiguous matches

Just so we have some code to test with our real-life apps. :)
An "unambiguous" match is one with exactly one matching node or exactly
one matching visible node.
Error messages are not updated, it just says "not found" if there is
more than one match.
The test suite of course does not pass with this.

(Closing immediately -- this is not actually for merging into master.) Repo is here if you want to try it: https://github.com/joliss/capybara/tree/ambiguous-matches


About OOM problem

I'm trying to reproduce your result on MuST-C en-de but I run into the problem of OOM. I use 4 RTX 3090 (24G) gpus, wav2vec 2.0 base model , mbart one-to-many model and mbart 250,000 dictionary. The total number of parameters is 817M (344M to train), and I find that the maximum number of "max-tokens" I can set is 40,000, while yours is 440,000. I think cutting the dictionary may help but I notice that you just use the original dictionary.
Do you have any thoughts on this? Or can you reveal the total parameters of your model? Thank you a lot!

Hi!
We didn't modify the dictionary. Considering that we are freezing the embeddings and the final linear projection, its size should have little impact in the memory.
When we were working on this project, we used smaller GPUs than yours (2080), when we started using 3090 GPUs for the next IWSLT participation, we could even increase max_tokens to 1,150,000.
Can you confirm that you are using the exact same config than us?

Thank you! Perhaps I need to examine my settings again. Maybe there's something wrong with my training script.
btw, I guess your total number of parameters is about 1B？I use wav2vec 2.0 base model and my total number of parameters is 817M.

Hi! I checked the training log of the lna_ed architecture and this is what I found:
[2021-04-21 08:06:23,627][fairseq_cli.train][INFO] - num. shared model params: 781,977,216 (num. trained: 159,188,992)

I don't understand why you have more parameters than us, considering that you are using wav2vec base instead of the large architecture.
Apart from this, maybe you removed batch_size from the config? We had OOMs when just setting max_tokens, because there were some batches containing a lot of small samples. That's why we limit the size of the batch with both max_tokens: 480_000  and batch_size: 18.
Are you using fp16: True and memory_efficient_fp16: True, right?

Thank you! I use fp16 so it won't be the problem. But I did not set bacth_size and I think that's the exact reason for my problem. I've never noticed this.
About the mismatch of parameters, I'll check my model structure.
Thank you for your help!

Perfect, I'm happy to hear that! Tell us if you need anything else 👍


ERR_REQUIRE_ESM error when running node16 actions

Hello,
I am trying to implement wretry.action into our workflow to sync our files into an r2 bucket using ryand56's r2-upload-action since syncing to an r2 bucket is quite flaky. Though seems like I got a roadblock ahead with ERR_REQUIRE_ESM error that I got (details below).
For disclaimer, I am not quite familiar with Node or JavaScript in general but feel free to ask something that you might need to investigate this issue.
GitHub workflow file: https://github.com/neon-nyan/CollapseLauncher-ReleaseRepo/actions/runs/6687813790/workflow
Failed Workflow log: https://github.com/neon-nyan/CollapseLauncher-ReleaseRepo/actions/runs/6687813790/job/18168980627
Actions used: https://github.com/ryand56/r2-upload-action
Workflow command example:
      - name: Sync repair_indexes
        uses: Wandalen/wretry.action@master
        if: always()
        with:
          attempt_limit: 3
          attempt_delay: 30000
          action: ryand56/r2-upload-action@latest
          with: |
            r2-account-id: ${{ secrets.R2_ACCOUNT_ID }}
            r2-access-key-id: ${{ secrets.R2_ACCESS_KEY_ID }}
            r2-secret-access-key: ${{ secrets.R2_SECRET_ACCESS_KEY }}
            r2-bucket: ${{ secrets.R2_BUCKET }}
            source-dir: cl-cdn/metadata/repair_indexes
            destination-dir: cl-cdn/metadata/repair_indexes
            output-file-url: false

Error log snippet:
/home/runner/work/_actions/Wandalen/wretry.action/v1.3.0_js_action/src/Runner.js:21
  require( process.argv[ 2 ] )
  ^

Error [ERR_REQUIRE_ESM]: require() of ES Module /home/runner/work/_actions/Wandalen/r2-upload-action/dist/index.js from /home/runner/work/_actions/Wandalen/wretry.action/v1.3.0_js_action/src/Runner.js not supported.
Instead change the require of index.js in /home/runner/work/_actions/Wandalen/wretry.action/v1.3.0_js_action/src/Runner.js to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (/home/runner/work/_actions/Wandalen/wretry.action/v1.3.0_js_action/src/Runner.js:21:3) {
  code: 'ERR_REQUIRE_ESM'
}
Error: Process returned exit code 1
Launched as "/home/runner/runners/2.[311](https://github.com/neon-nyan/CollapseLauncher-ReleaseRepo/actions/runs/6687813790/job/18168980627#step:6:313).0/externals/node16/bin/node /home/runner/work/_actions/Wandalen/wretry.action/v1.3.0_js_action/src/Runner.js /home/runner/work/_actions/Wandalen/r2-upload-action/dist/index.js"
Launched at "/home/runner/work/CollapseLauncher-ReleaseRepo/CollapseLauncher-ReleaseRepo"

Hello @bagusnl
I'll look at the issue when I have time.

same problem here

Hello @bagusnl and @eric2788
Can you try the new version of action and give me a feedback? I made a lot of changes and it can solve the issue.

Hi, sorry it takes awhile for me to try it again. But unfortunately it did not fixed it and outputs pretty much the same error
Workflow example:
      - name: Sync Squirrel\Preview folder
        uses: Wandalen/wretry.action@master
        with:
          attempt_limit: 3
          attempt_delay: 10000
          action: ryand56/r2-upload-action@latest
          with: |
            r2-account-id: ${{ secrets.R2_ACCOUNT_ID }}
            r2-access-key-id: ${{ secrets.R2_ACCESS_KEY_ID }}
            r2-secret-access-key: ${{ secrets.R2_SECRET_ACCESS_KEY }}
            r2-bucket: ${{ secrets.R2_BUCKET }}
            source-dir: squirrel-preview/cl-cdn/squirrel/preview
            destination-dir: cl-cdn/squirrel/preview
            output-file-url: false

Error output:
/home/runner/work/_actions/Wandalen/wretry.action/v3.0.0_js_action/src/Runner.js:21
  require( process.argv[ 2 ] )
  ^
Error [ERR_REQUIRE_ESM]: require() of ES Module /home/runner/work/_actions/Wandalen/r2-upload-action/dist/index.js from /home/runner/work/_actions/Wandalen/wretry.action/v3.0.0_js_action/src/Runner.js not supported.
Instead change the require of index.js in /home/runner/work/_actions/Wandalen/wretry.action/v3.0.0_js_action/src/Runner.js to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (/home/runner/work/_actions/Wandalen/wretry.action/v3.0.0_js_action/src/Runner.js:21:3) {
  code: 'ERR_REQUIRE_ESM'
}
Node.js v20.8.1

Hello @bagusnl
I fixed the issue in the new release v3.0.1.


Compatibility with python3 and airflow 1.9

Hi,
First - thanks for this excellent library. We're using it to do some stuff that's otherwise very hard in Airflow.
We're going to be upgrading our codebase to airflow 1.9 and python 3.6.1, so I've made some changes to allow compatibility with both.
Changes as follows:

Make imports explicit throughout modules
Pass a lambda: None to the init kwargs of the PythonOperator to satisfy its check for a callable
Pop data_dependencies from the super().init kwargs of DiveOperatorto stop the deprecation warning that airflow emits.

I've made various edits to the unit tests since implicit bytes == str comparisons will fail under python3. Mostly this has just been passing explicit encoding='utf-8' arguments to the various as_string methods. I notice that the core library is careful to do this anyway.
All tests are passing in python2.7.13 and python3.6.1.

@jamesrmccallum awesome, thanks for the PR! I will look it over this holiday weekend :)

@lauralorenz - well it was working well when i did the PR. Let me know if you want me to make any tweaks! I've since moved on from where i was using it, but we ran it in production there with no issues.


[HELP] OAuth issue : same credentials on two AWS instances, but error 400 on one of them

Hello !
We are having trouble with SentinelHub authentication. We are using sentinelhub-py on two AWS instances running Jupyter Notebook servers. On the first one there is no problem using it ; on the other and using the exact same credentials, the request returns a 400 error as follows.
(During execution of task SentinelHubInputTask: ({'status': 400, 'reason': 'Bad Request', 'message': 'Illegal secret!', 'code': 'OAUTH_ERROR'}) )
Do you have any idea where we could look at in order to troubleshoot this ?
Thanks in advance,
Best,
J.

Error was caused by a typo identified in the secret string. Closing this topic :)


FEATURE REQUEST: Change default columns (uncategorised and done)

Hello again,
If possible, could we have a setting added that allows us to change the name of the default columns? It would be great to be able to have all new tasks just dumped in a "backlog" column, instead of "uncategorised".
Cheers!

Hey @source-toad we can look at doing this for you! To confirm it's also being able to name the column for all untagged tasks?

Also quick note, saw your pull request - thanks so much for doing that!!! We'll have a look ASAP and get back to you on that too 😁 @source-toad


Close open-ended range

An open-ended range has been created, how can it later be closed by using another moment() time?
It doesn't seem to work by assigning it to .end.

This should work:
const r = moment.range(moment('2017-01-01T13:30:00'));
console.log(r.toString());  // 2017-01-01T13:30:00+00:00/275760-09-13T01:00:00+01:00
r.end = moment();
console.log(r.toString());  // 2017-01-01T13:30:00+00:00/2019-10-07T10:58:58+01:00

Otherwise, feel free to use .copy() and pass in the old range's start and your new end.


[INLONG-1673] use the valid links  after office-website refactored

Fixes #1673
Motivation
Explain here the context, and why you're making that change. What is the problem you're trying to solve.
Modifications
Describe the modifications you've done.
Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

Codecov Report

Merging #1675 (b57dc29) into master (736a28e) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #1675      +/-   ##
============================================
- Coverage     12.21%   12.19%   -0.02%
- Complexity     1048     1049       +1
============================================
  Files           392      392
  Lines         32755    32755
  Branches       5159     5159
============================================
- Hits           4001     3995       -6
- Misses        27989    27995       +6
  Partials        765      765

Impacted Files
Coverage Δ

.../java/org/apache/flume/sink/tubemq/TubemqSink.java
51.42% <0.00%> (-4.00%)
:arrow_down:

.../producer/qltystats/DefaultBrokerRcvQltyStats.java
45.70% <0.00%> (+0.39%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 736a28e...b57dc29. Read the comment docs.