Feature Request: Support librdkafka from add_subdirectory

Since modern-cpp-kafka only works with recent versions of librdkafka, I see myself forced to not use the system-provided version of librdkafka.
I considered vcpkg, but it did not seem feasible to set up vcpkg just for a single package, since modern-cpp-kafka is not available on vcpkg.
So in my current setup, I have both librdkafka and modern-cpp-kafka next to each other in different subdirectories.
I would expect the following CMake snippet to work:
set(RDKAFKA_BUILD_STATIC ON CACHE BOOL "")
add_subdirectory(libs/librdkafka EXCLUDE_FROM_ALL)
add_subdirectory(libs/modern-cpp-kafka)

Unfortunately, there are 2 issues:

librdkafka makes headers available without librdkafka prefix, so #include <rdkafka.h> is needed instead of #include <librdkafka/rdkafka.h>. Maybe this issue should be fixed in librdkafka, but CMake support there is kind of unofficial.
modern-cpp-kafka requires environment variables to be set to find headers and libraries. I would propose to implement a FindRdKafka CMake module and check for the existence of the rdkafka target.

Are there best practices on how to set up librdkafka with modern-cpp-kafka with compatible versions?

Alternatively, provide a bundled version of librdkafka.

Not sure whether add_subdirectory (for these 2 libraries) would work since one is depending on another.
While, if you're interested, I'd suggest to have a try with conan -- in short, you just need to call conan install .. --build=missing before the cmake command, and all dependencies would be built for you. E.g. https://github.com/morganstanley/modern-cpp-kafka/blob/main/.github/workflows/kafka_api_demo_conan_build.yml#L37
I'm not quite expert for conan, but once I tried to add a recipe for modern-cpp-kafka, https://github.com/conan-io/conan-center-index/blob/master/recipes/modern-cpp-kafka  , and the compatible version could be assigned there, e.g.
https://github.com/conan-io/conan-center-index/blob/master/recipes/modern-cpp-kafka/all/conanfile.py#L18
A demo project for conan build, https://github.com/morganstanley/modern-cpp-kafka/blob/main/demo_projects_for_build/conan_build/

Not sure whether add_subdirectory (for these 2 libraries) would work since one is depending on another.

This usually works. It is possible to depend on a target from subdirectory A while being in subdirectory B if the subdirectories are added in a valid order.

While, if you're interested, I'd suggest to have a try with conan -- in short, you just need to call conan install .. --build=missing before the cmake command, and all dependencies would be built for you. E.g. https://github.com/morganstanley/modern-cpp-kafka/blob/main/.github/workflows/kafka_api_demo_conan_build.yml#L37
I'm not quite expert for conan, but once I tried to add a recipe for modern-cpp-kafka, https://github.com/conan-io/conan-center-index/blob/master/recipes/modern-cpp-kafka , and the compatible version could be assigned there, e.g. https://github.com/conan-io/conan-center-index/blob/master/recipes/modern-cpp-kafka/all/conanfile.py#L18
A demo project for conan build, https://github.com/morganstanley/modern-cpp-kafka/blob/main/demo_projects_for_build/conan_build/

I could use vcpkg in a similar way as conan, but currently I try to avoid depending on such dependency managers.

Hi, @stertingen I like your idea but I'm not sure whether it could be achieved with a normal (with no "ugly hack") approach.
E.g. the project might contain 2 sub-projects, A and B (which depends on A)
add_subdirectory(A)
add_subdirectory(B)

You might want the sequence of build A->install A->build B->install B, while within the same parent project, it could only be build A->build B ->install A->install B. While building the library for B, it would search the path of A library, and it makes more sense if it's the installation path.

Hi, @stertingen I like your idea but I'm not sure whether it could be achieved with a normal (with no "ugly hack") approach. E.g. the project might contain 2 sub-projects, A and B (which depends on A)
add_subdirectory(A)
add_subdirectory(B)

You might want the sequence of build A->install A->build B->install B, while within the same parent project,

That's the only way the build currently works. And that will continue to work if done right (see below). Most CMake projects allow multiple kinds of 'being included' by other projects. Fully building and installing a sub-project before configuring another is not always useful, especially when cross-compiling.

it could only be build A->build B ->install A->install B. While building the library for B, it would search the path of A library, and it makes more sense if it's the installation path.

I would expect the following behavior from this project's CMakeLists.txt:

Check if library 'A' was already declared as CMake target.
If not, search library 'A', possibly using find_package().
Link against 'A', either using the previously declared CMake target or the library which was found in 2.

If needed, an option can be added to 'B's CMakeLists.txt to force finding 'A' on it's install path.
The point I'm trying to make is that I would like to get rid of the need to install 'A' before being able to use it.

About Check if library 'A' was already declared as CMake target, any option suggested?
I could only think of one workaround, -- if you could predict the librdkafka build path, env var LIBRDKAFKA_INCLUDE_DIR/LIBRDKAFKA_LIBRARY_DIR could be used,
https://github.com/morganstanley/modern-cpp-kafka/blob/main/CMakeLists.txt#L42
https://github.com/morganstanley/modern-cpp-kafka/blob/main/CMakeLists.txt#L47
But please make sure the librdkafka library would be static linked.


Can't create transform job in Lambda function

Lambda execution creates this Cloudwatch log:

[ERROR] ClientError: An error occurred (ValidationException) when calling the CreateTransformJob operation: Could not assume role arn:aws:iam::XXX:role/PredictiveMaintenance-NotebookInstanceExecutionRol-YVBYTBKPP92H. Please ensure that the role exists and allows principal 'sagemaker.amazonaws.com' to assume the role.
Traceback (most recent call last):
File "/var/task/index.py", line 30, in lambda_handler
batch_transform_response = run_batch_transform(transform_input)
File "/var/task/index.py", line 76, in run_batch_transform
sm.create_transform_job(**payload)
File "/var/runtime/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 661, in _make_api_call
raise error_class(parsed_response, operation_name)

This role does not exist, but another one (PredMaint-SolutionBuilder-NotebookInstanceExecutio-EP4DFXR5ZUO6) was created by the Cloudformation template.

Hi @mstfldmr I haven't been able to recreate this issue. Can you confirm if it's still occurring and the steps you've taken to reach this? Thanks!

Hi @mstfldmr, we were able to re-create the issue. A fix has been checked in and is being deployed to the official soln. You can git clone within your workspace to pull the latest code.
Commit 52b85fe**

This fix is now released on the official solution page on AWS solutions so you can redeploy from there: https://aws.amazon.com/solutions/implementations/predictive-maintenance-using-machine-learning/. Alternatively you can use the quick-launch button in the README of this repository to get the latest version via github.


Store path template in config file (WIP)

I'd like to improve error filtering in server.js, but that need exists outside of this feature, so I'll treat it as a refactor branch separately.
As I mentioned in slack, GitSheets does not currently support multiple datasets in a repo, so this feature doesn't either. For that reason it uses .gitsheets/config. Should be a relatively easy fix to support multiple datasets (albeit breaking changes to the API) but is a distinct feature from this one.
EDIT: Don't merge yet; found an issue

I've just discovered that the current import functionality will create a new tree that leaves out the .gitsheets subtree from the previous commit. Need to fix that before merging.

Sorted

@timwis looks good, this ready to merge now?

yessir


Purge page cache on query string pages not working

Issue with 0.9.5
I enabled the “Cache URIs with query string variables” option to enable page caching on query string pages. The cache method I am using is disk (basic).
Page caching is working properly but when I try to purge the cache (on query string pages) it is not working. Purging the caching on pages without query strings is working fine.
It would be really great if someone can look into this matter
Best regards,
Tim

Hey @timmen22,
It seems getting this done with W3TC is a bit of a pain.
Instead, I have added a page cache flusher widget in the admin dashboard (Performance => Dashboard).
You can paste any URL you want into the textbox, and it will be purged from the cache.
I have pushed this to my branch in here: https://github.com/charlesLF/fix-w3tc/commits/w3tc-0-9-5
Can you try it out and let me know if this works for you?

Thanks for heling me Charles. It is really much appreciated. After pasting a url the cache is purged so it's working! Is it also possible to make a function that works like this:
When I put the following URL into the textbox:  www.example.com/?utm=* --> purge al pages that start with ?utm

@timmen22, no problem! Glad to be of help.
I will merge this feature to the 0.9.5 branch on this repo too.
As for the wildcard purging, unfortunately I don't think this is feasible with the W3TC code as it is right now.
I believe this would require adding a separate key somewhere that will track all cached URL's and then use them for purging.
Otherwise, W3TC currently does not "know" which URL's are cached and which ones aren't before they are actually requested, making it unable to interpret wildcards in this context.


Add a CI pipeline using github actions

It would be nice to have a CI pipeline using Github actions that at least builds the extension on every commit & PR. Eventually it would be also nice to have some tests, but I am not sure how feasible that is.
It should be possible to steal most of the CI pipeline from the open build service connector: https://github.com/SUSE/open-build-service-connector

@D4N Can we use Travis CI?

I'd prefer to avoid Travis, as their recent pricing changes made the usage as a non-paying customer quite inconvenient. Plus github actions involve a lot less magic.

@m1-key are you still working on this issue?


WebGPU: Add support for compute shaders

I thought I should make a first PR even if the support is not complete yet.
What's missing:

support for storage buffers as input types. Currently, there is support for textures, storage textures and ubos Added
full serialization / parsing. This one is a bit involved because:

empty created raw textures are currently not serialized => need to add proper serialization to them
ubos are not serializable
when storage buffers will be in, serialization will have to be added to them too

Here are two samples:

https://playground.babylonjs.com#3URR7V#59. Use a CS to clear the texture applied to the sphere and another CS to copy a texture into another one, applied on the ground. Note that it's for demonstration only, I set the workgroup size to be (1,1,1) which is really bad for performances! Also creates a CS to compute the multiplication of two matrices, to demonstrate usage of storage buffers
https://playground.babylonjs.com#3URR7V#46. It is a direct port of the WebGPU sample imageBlur

Regarding the implementation:

The implementation is modeled after Effect / PipelineContext, so there's a ComputeEffect and a ComputePipelineContext classes (but much simpler than their rendering counter part!). The main user land class is ComputeShader which is modeled a bit like ShaderMaterial.
To support storage textures, I needed to know that a texture should be created as a storage texture or not, so I have added a creationFlag parameter which is a bit field. Only bit 0 is currently used and when set means "storage texture" (there's a constant for it in Constants.ts)

I have added two new methods CreateRGBAStorageTexture and CreateRStorageTexture to ease creating storage textures that automatically set the bit when creating the corresponding raw textures

Note that WGSL does not support preprocessing, so the #define that we normally generate in the source shader are prefixed by //.
The browsers don't currently support reflection from WGSL code, so it's not possible to retrieve the group + binding values of a given variable by name: the user must directly provide the group and binding numbers.  The interface (see setTexture, setStorageTexture, setUniformBuffer) is designed to also be able to take a string instead of two numbers for when the browsers will support reflection.

All good for me!! I feel like we should write the page doc asap :)

Yes, I plan to do it after I have added the support for the storage buffers.

Lovely!! Let's merge and prepare a tweet ;)

oh just a lint issue and you can merge :)

Can you add this to breaking changes (is a break if using imports from @babylonjs/core
-import { DataBuffer } from "../Meshes/dataBuffer";
-import { Buffer, VertexBuffer } from "../Meshes/buffer";
+import { DataBuffer } from "../Buffers/dataBuffer";
+import { Buffer, VertexBuffer } from "../Buffers/buffer";

Excellent catch!
@Popov72


Can you release v0.3.4?

The commit I'm interested in is https://github.com/cakeinpanic/jira-description-action/commit/eb1e91b20ac768c5dcb2c1845d5110b1d69eef19 because github actions no longer support node 12. I could use the main branch of this action in my workflow but I prefer to use a pinned version. Thanks!

Plus one on that request please

easy!
0.4.0 is released
https://github.com/cakeinpanic/jira-description-action/releases/tag/v0.4.0

Thanks!


refactor tabs

Description
reafctor tabs

What is the purpose of this pull request?

[x] Bug fix
[ ] New Feature
[ ] Documentation update
[ ] Other

Before submitting the PR, please make sure you do the following

[x] Read the Contributing Guidelines.
[ ] Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
[ ] Provide a description in this PR that addresses what the PR is solving, or reference the issue that it solves (e.g. fixes #123).
[ ] Ideally, include relevant tests that fail without this PR but pass with it.

Codecov Report

Merging #192 (f512753) into master (402eaa3) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #192   +/-   ##
=======================================
  Coverage   79.95%   79.95%
=======================================
  Files         126      126
  Lines       11663    11663
  Branches     1217     1217
=======================================
  Hits         9325     9325
  Misses       2338     2338

Flag
Coverage Δ

unittests
79.95% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Process group scopes

Our process groups don't have "scopes" like the Erlang ones (https://www.erlang.org/doc/man/pg.html)
As we move towards distributed actors, scopes will probably help limit which actors are being referenced in process groups and when.
This is a tracking issue to add scopes to our pg module.

My questions before I might give it a try:

Is there anything discouraging implementation at this moment?
Is my understanding correct?

Scopes are possibly non-disjoint groups of process groups.
If no scope is explicitly set, all process groups are part of the default scope.
To join a non-default scope, processes have to be joined with process groups in the explicit (non-default) scope.
Processes can join any number of process groups in any number of scopes.

As Rust doesn't support variadic functions: Does ractor have preferences regarding the interface, i.e.,

splitting the functions into, e.g., a join and a join_with_scope(increasing the number of functions), or
setting an explicit Default function parameter for the scope (increasing noise when only using default scope), or
using macros for variadic interfaces?

Is there a technical reason the pg module's comments for the monitor/demonitor function mention 'scope or groups' already?

Can't promise anything but am curious so far.

Thanks for the answers and the welcome! Let's see how it goes ;)
Dug a bit deeper into Erlang's pg module and got a bit surprised with the disjointness of scopes.
My updated understanding is:

a process group is always in exactly one scope, and
there can be multiple process groups with the same name, but in different scopes, so
I must think of (scope, process_group) as an identifying tuple.

Is that correct?

Ya I find their docs lacking in some of the PG designs. But if we diverge here for a more logical solution I think that's fine too. I'm happy to be based on what they have but it doesn't have to be exactly Erlang's implementation.
You're summary sounds correct to my knowledge but I have not dug deeply through PG mainly for lack of time.

Thanks for the clarification and agreed. Will try to follow their design first.


Added some code for RSA keygen

oi kardo merge.

oi kardo merge.

ill do it myself then.

ill do it myself then.


Implement minimal viable eth_sendRawTransaction

What was wrong?
Trintiy doesn't yet allow sending transactions via the eth_sendRawTransaction JSON-RPC API. (See #29)
How was it fixed?

This refactors the DefaultTransactionValidator to be reusable for the eth_sendRawTransaction API and moves it into a new namespace trinity.transactions (not sure about the namespace, open to suggestions)
Transactions are validated in the Eth API module and either an error is presented to the user or the transactions is send over the event bus to be picked up by the transaction pool
Tests where added to ensure local transactions are relayed to other peers and certain errors are presented to the user
The transaction pool picks up the transaction and processes it among all other transactions
As a bonus, this fixes a bug where we currently relay transactions even if they are for the wrong chain

Known issues

This doesn't yet implement any of the more sophisticated things that #29 mentions such as ensuring locally transmitted transactions survive across restarts. I'd like to keep this PR small and move these things over to a new issue.

To-Do

[x] Clean up commit history
[x] Some manual testing
[x] Fix broken existing peer pool tests (probably fail because we are trying to send invalid tx which are now properly caught)

[x] Add entry to the release notes

Cute Animal Picture

eek...manually testing revealed that we can't validate the transaction when we don't have the state for the account..obviously. I think that the reason why I got this is because I wasn't beam syncing yet (happened during header sync) because I think this should be taken care of (fetch state from peers) if we are beam syncing.
Tracking it hear and doing some more manual testing.
    INFO  2020-07-10 11:02:22,675                  root  RPC method caused exception
Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/batch.py", line 69, in __getitem__
    value = self._track_diff[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/diff.py", line 80, in __getitem__
    raise DiffMissingError(key, result)  # type: ignore # ignore over cast for perf reasons
eth.db.diff.DiffMissingError: (b'\xcb\xff\x15\xc0{\x16\x83\xc6r\xde\xfc\x82-\xab;\xd1\xc8\xb3\xce\x8e3\x84s\x95\x90N<\x84^\x18\x88-', 'Key is missing because it was never inserted')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 203, in _traverse
    root_node = self.get_node(root_hash)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 570, in get_node
    encoded_node = self.db[node_hash]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/batch.py", line 74, in __getitem__
    return self.wrapped_db[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/accesslog.py", line 89, in __getitem__
    result = self.wrapped_db.__getitem__(key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/db/manager.py", line 291, in __getitem__
    raise KeyError(key)
KeyError: b'\xcb\xff\x15\xc0{\x16\x83\xc6r\xde\xfc\x82-\xab;\xd1\xc8\xb3\xce\x8e3\x84s\x95\x90N<\x84^\x18\x88-'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 133, in get
    return self._get(root_hash, trie_key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 143, in _get
    node, remaining_key = self._traverse(root_hash, trie_key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 205, in _traverse
    raise MissingTraversalNode(root_hash, ())
trie.exceptions.MissingTraversalNode: Trie database is missing hash HexBytes('0xcbff15c07b1683c672defc822dab3bd1c8b3ce8e33847395904e3c845e18882d'), found when traversing down ().

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 341, in _get_encoded_account
    return lookup_trie[address]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/journal.py", line 336, in __getitem__
    return self._wrapped_db[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/cache.py", line 22, in __getitem__
    self._cached_values[key] = self._db[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/accesslog.py", line 45, in __getitem__
    result = self.wrapped_db.__getitem__(key)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/keymap.py", line 28, in __getitem__
    return self._db[mapped_key]
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 778, in __getitem__
    return self.get(key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 140, in get
    ) from traverse_exc
trie.exceptions.MissingTrieNode: Trie database is missing hash HexBytes('0xcbff15c07b1683c672defc822dab3bd1c8b3ce8e33847395904e3c845e18882d') needed to look up node at prefix (), when searching for key HexBytes('0xd541d86395704616f46d3be8cebb4d49c84d4aada60e7c794401d4f8cd567f6f') at root hash HexBytes('0xcbff15c07b1683c672defc822dab3bd1c8b3ce8e33847395904e3c845e18882d')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/main.py", line 121, in _get_result
    self.event_bus, method, params, self.chain,
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/retry.py", line 108, in execute_with_retries
    return await func(*params)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/format.py", line 213, in formatted_func
    return await func(self, *formatted)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/modules/eth.py", line 286, in getTransactionCount
    nonce = state.get_nonce(address)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/vm/state.py", line 129, in get_nonce
    return self._account_db.get_nonce(address)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 247, in get_nonce
    account = self._get_account(address)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 352, in _get_account
    rlp_account = self._get_encoded_account(address, from_journal)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 343, in _get_encoded_account
    raise MissingAccountTrieNode(*exc.args) from exc
eth.vm.interrupt.MissingAccountTrieNode: State trie database is missing node for hash 0xcbff15c07b1683c672defc822dab3bd1c8b3ce8e33847395904e3c845e18882d, which is needed to look up account with address hash 0xd541d86395704616f46d3be8cebb4d49c84d4aada60e7c794401d4f8cd567f6f at root hash 0xcbff15c07b1683c672defc822dab3bd1c8b3ce8e33847395904e3c845e18882d -- (HexBytes('0xcbff15c07b1683c672defc822dab3bd1c8b3ce8e33847395904e3c845e18882d'), HexBytes('0xcbff15c07b1683c672defc822dab3bd1c8b3ce8e33847395904e3c845e18882d'), HexBytes('0xd541d86395704616f46d3be8cebb4d49c84d4aada60e7c794401d4f8cd567f6f'), ())

Ok, same thing during beam sync.
INFO  2020-07-10 14:45:20,113     BeamBlockImporter  Beam importing <BlockHeader #3021933 99d34fc5> (4 txns) ...
    INFO  2020-07-10 14:45:24,776   BaseBodyChainSyncer  Imported block 3021933 (4 txs) in 4.66 seconds, lagging 0 blocks | 11s
    INFO  2020-07-10 14:45:25,880                  root  RPC method caused exception
Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/batch.py", line 69, in __getitem__
    value = self._track_diff[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/diff.py", line 80, in __getitem__
    raise DiffMissingError(key, result)  # type: ignore # ignore over cast for perf reasons
eth.db.diff.DiffMissingError: (b'\x1f\x05\x93?z\xdft 5\xb5\\\xf3\xd2\xb7,\xe4,\xe3{\x0f\x05\x8a\xbe/\xe9\\\xb5\x87Oc\xaa\x18', 'Key is missing because it was never inserted')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 272, in _traverse_from
    node = self.get_node(next_node_pointer)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 570, in get_node
    encoded_node = self.db[node_hash]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/batch.py", line 74, in __getitem__
    return self.wrapped_db[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/accesslog.py", line 89, in __getitem__
    result = self.wrapped_db.__getitem__(key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/db/manager.py", line 291, in __getitem__
    raise KeyError(key)
KeyError: b'\x1f\x05\x93?z\xdft 5\xb5\\\xf3\xd2\xb7,\xe4,\xe3{\x0f\x05\x8a\xbe/\xe9\\\xb5\x87Oc\xaa\x18'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 133, in get
    return self._get(root_hash, trie_key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 143, in _get
    node, remaining_key = self._traverse(root_hash, trie_key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 207, in _traverse
    return self._traverse_from(root_node, trie_key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 276, in _traverse_from
    raise MissingTraversalNode(exc.args[0], used_key)
trie.exceptions.MissingTraversalNode: Trie database is missing hash HexBytes('0x1f05933f7adf742035b55cf3d2b72ce42ce37b0f058abe2fe95cb5874f63aa18'), found when traversing down (0xd, 0x5, 0x4).

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 341, in _get_encoded_account
    return lookup_trie[address]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/journal.py", line 336, in __getitem__
    return self._wrapped_db[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/cache.py", line 22, in __getitem__
    self._cached_values[key] = self._db[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/accesslog.py", line 45, in __getitem__
    result = self.wrapped_db.__getitem__(key)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/keymap.py", line 28, in __getitem__
    return self._db[mapped_key]
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 778, in __getitem__
    return self.get(key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 140, in get
    ) from traverse_exc
trie.exceptions.MissingTrieNode: Trie database is missing hash HexBytes('0x1f05933f7adf742035b55cf3d2b72ce42ce37b0f058abe2fe95cb5874f63aa18') needed to look up node at prefix (0xd, 0x5, 0x4), when searching for key HexBytes('0xd541d86395704616f46d3be8cebb4d49c84d4aada60e7c794401d4f8cd567f6f') at root hash HexBytes('0xf07b2e276b71d4424e92d6132027792f608a326de8d8c50f077ccc374487342c')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/main.py", line 121, in _get_result
    self.event_bus, method, params, self.chain,
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/retry.py", line 108, in execute_with_retries
    return await func(*params)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/format.py", line 213, in formatted_func
    return await func(self, *formatted)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/modules/eth.py", line 286, in getTransactionCount
    nonce = state.get_nonce(address)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/vm/state.py", line 129, in get_nonce
    return self._account_db.get_nonce(address)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 247, in get_nonce
    account = self._get_account(address)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 352, in _get_account
    rlp_account = self._get_encoded_account(address, from_journal)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 343, in _get_encoded_account
    raise MissingAccountTrieNode(*exc.args) from exc
eth.vm.interrupt.MissingAccountTrieNode: State trie database is missing node for hash 0x1f05933f7adf742035b55cf3d2b72ce42ce37b0f058abe2fe95cb5874f63aa18, which is needed to look up account with address hash 0xd541d86395704616f46d3be8cebb4d49c84d4aada60e7c794401d4f8cd567f6f at root hash 0xf07b2e276b71d4424e92d6132027792f608a326de8d8c50f077ccc374487342c -- (HexBytes('0x1f05933f7adf742035b55cf3d2b72ce42ce37b0f058abe2fe95cb5874f63aa18'), HexBytes('0xf07b2e276b71d4424e92d6132027792f608a326de8d8c50f077ccc374487342c'), HexBytes('0xd541d86395704616f46d3be8cebb4d49c84d4aada60e7c794401d4f8cd567f6f'), (0xd, 0x5, 0x4))
    INFO  2020-07-10 14:45:36,625                  root  RPC method caused exception
Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/batch.py", line 69, in __getitem__
    value = self._track_diff[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/diff.py", line 80, in __getitem__
    raise DiffMissingError(key, result)  # type: ignore # ignore over cast for perf reasons
eth.db.diff.DiffMissingError: (b'\x1f\x05\x93?z\xdft 5\xb5\\\xf3\xd2\xb7,\xe4,\xe3{\x0f\x05\x8a\xbe/\xe9\\\xb5\x87Oc\xaa\x18', 'Key is missing because it was never inserted')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 272, in _traverse_from
    node = self.get_node(next_node_pointer)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 570, in get_node
    encoded_node = self.db[node_hash]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/batch.py", line 74, in __getitem__
    return self.wrapped_db[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/accesslog.py", line 89, in __getitem__
    result = self.wrapped_db.__getitem__(key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/db/manager.py", line 291, in __getitem__
    raise KeyError(key)
KeyError: b'\x1f\x05\x93?z\xdft 5\xb5\\\xf3\xd2\xb7,\xe4,\xe3{\x0f\x05\x8a\xbe/\xe9\\\xb5\x87Oc\xaa\x18'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 133, in get
    return self._get(root_hash, trie_key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 143, in _get
    node, remaining_key = self._traverse(root_hash, trie_key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 207, in _traverse
    return self._traverse_from(root_node, trie_key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 276, in _traverse_from
    raise MissingTraversalNode(exc.args[0], used_key)
trie.exceptions.MissingTraversalNode: Trie database is missing hash HexBytes('0x1f05933f7adf742035b55cf3d2b72ce42ce37b0f058abe2fe95cb5874f63aa18'), found when traversing down (0xd, 0x5, 0x4).

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 341, in _get_encoded_account
    return lookup_trie[address]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/journal.py", line 336, in __getitem__
    return self._wrapped_db[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/cache.py", line 22, in __getitem__
    self._cached_values[key] = self._db[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/accesslog.py", line 45, in __getitem__
    result = self.wrapped_db.__getitem__(key)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/keymap.py", line 28, in __getitem__
    return self._db[mapped_key]
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 778, in __getitem__
    return self.get(key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 140, in get
    ) from traverse_exc
trie.exceptions.MissingTrieNode: Trie database is missing hash HexBytes('0x1f05933f7adf742035b55cf3d2b72ce42ce37b0f058abe2fe95cb5874f63aa18') needed to look up node at prefix (0xd, 0x5, 0x4), when searching for key HexBytes('0xd541d86395704616f46d3be8cebb4d49c84d4aada60e7c794401d4f8cd567f6f') at root hash HexBytes('0xf07b2e276b71d4424e92d6132027792f608a326de8d8c50f077ccc374487342c')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/main.py", line 121, in _get_result
    self.event_bus, method, params, self.chain,
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/retry.py", line 108, in execute_with_retries
    return await func(*params)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/format.py", line 213, in formatted_func
    return await func(self, *formatted)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/modules/eth.py", line 286, in getTransactionCount
    nonce = state.get_nonce(address)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/vm/state.py", line 129, in get_nonce
    return self._account_db.get_nonce(address)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 247, in get_nonce
    account = self._get_account(address)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 352, in _get_account
    rlp_account = self._get_encoded_account(address, from_journal)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 343, in _get_encoded_account
    raise MissingAccountTrieNode(*exc.args) from exc
eth.vm.interrupt.MissingAccountTrieNode: State trie database is missing node for hash 0x1f05933f7adf742035b55cf3d2b72ce42ce37b0f058abe2fe95cb5874f63aa18, which is needed to look up account with address hash 0xd541d86395704616f46d3be8cebb4d49c84d4aada60e7c794401d4f8cd567f6f at root hash 0xf07b2e276b71d4424e92d6132027792f608a326de8d8c50f077ccc374487342c -- (HexBytes('0x1f05933f7adf742035b55cf3d2b72ce42ce37b0f058abe2fe95cb5874f63aa18'), HexBytes('0xf07b2e276b71d4424e92d6132027792f608a326de8d8c50f077ccc374487342c'), HexBytes('0xd541d86395704616f46d3be8cebb4d49c84d4aada60e7c794401d4f8cd567f6f'), (0xd, 0x5, 0x4))
    INFO  2020-07-10 14:45:37,669                  root  RPC method caused exception
Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/batch.py", line 69, in __getitem__
    value = self._track_diff[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/diff.py", line 80, in __getitem__
    raise DiffMissingError(key, result)  # type: ignore # ignore over cast for perf reasons
eth.db.diff.DiffMissingError: (b'\x1f\x05\x93?z\xdft 5\xb5\\\xf3\xd2\xb7,\xe4,\xe3{\x0f\x05\x8a\xbe/\xe9\\\xb5\x87Oc\xaa\x18', 'Key is missing because it was never inserted')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 272, in _traverse_from
    node = self.get_node(next_node_pointer)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 570, in get_node
    encoded_node = self.db[node_hash]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/batch.py", line 74, in __getitem__
    return self.wrapped_db[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/accesslog.py", line 89, in __getitem__
    result = self.wrapped_db.__getitem__(key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/db/manager.py", line 291, in __getitem__
    raise KeyError(key)
KeyError: b'\x1f\x05\x93?z\xdft 5\xb5\\\xf3\xd2\xb7,\xe4,\xe3{\x0f\x05\x8a\xbe/\xe9\\\xb5\x87Oc\xaa\x18'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 133, in get
    return self._get(root_hash, trie_key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 143, in _get
    node, remaining_key = self._traverse(root_hash, trie_key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 207, in _traverse
    return self._traverse_from(root_node, trie_key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 276, in _traverse_from
    raise MissingTraversalNode(exc.args[0], used_key)
trie.exceptions.MissingTraversalNode: Trie database is missing hash HexBytes('0x1f05933f7adf742035b55cf3d2b72ce42ce37b0f058abe2fe95cb5874f63aa18'), found when traversing down (0xd, 0x5, 0x4).

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 341, in _get_encoded_account
    return lookup_trie[address]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/journal.py", line 336, in __getitem__
    return self._wrapped_db[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/cache.py", line 22, in __getitem__
    self._cached_values[key] = self._db[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/accesslog.py", line 45, in __getitem__
    result = self.wrapped_db.__getitem__(key)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/keymap.py", line 28, in __getitem__
    return self._db[mapped_key]
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 778, in __getitem__
    return self.get(key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 140, in get
    ) from traverse_exc
trie.exceptions.MissingTrieNode: Trie database is missing hash HexBytes('0x1f05933f7adf742035b55cf3d2b72ce42ce37b0f058abe2fe95cb5874f63aa18') needed to look up node at prefix (0xd, 0x5, 0x4), when searching for key HexBytes('0xd541d86395704616f46d3be8cebb4d49c84d4aada60e7c794401d4f8cd567f6f') at root hash HexBytes('0xf07b2e276b71d4424e92d6132027792f608a326de8d8c50f077ccc374487342c')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/main.py", line 121, in _get_result
    self.event_bus, method, params, self.chain,
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/retry.py", line 108, in execute_with_retries
    return await func(*params)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/format.py", line 213, in formatted_func
    return await func(self, *formatted)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/modules/eth.py", line 286, in getTransactionCount
    nonce = state.get_nonce(address)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/vm/state.py", line 129, in get_nonce
    return self._account_db.get_nonce(address)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 247, in get_nonce
    account = self._get_account(address)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 352, in _get_account
    rlp_account = self._get_encoded_account(address, from_journal)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 343, in _get_encoded_account
    raise MissingAccountTrieNode(*exc.args) from exc
eth.vm.interrupt.MissingAccountTrieNode: State trie database is missing node for hash 0x1f05933f7adf742035b55cf3d2b72ce42ce37b0f058abe2fe95cb5874f63aa18, which is needed to look up account with address hash 0xd541d86395704616f46d3be8cebb4d49c84d4aada60e7c794401d4f8cd567f6f at root hash 0xf07b2e276b71d4424e92d6132027792f608a326de8d8c50f077ccc374487342c -- (HexBytes('0x1f05933f7adf742035b55cf3d2b72ce42ce37b0f058abe2fe95cb5874f63aa18'), HexBytes('0xf07b2e276b71d4424e92d6132027792f608a326de8d8c50f077ccc374487342c'), HexBytes('0xd541d86395704616f46d3be8cebb4d49c84d4aada60e7c794401d4f8cd567f6f'), (0xd, 0x5, 0x4))
    INFO  2020-07-10 14:45:38,084                  root  RPC method caused exception
Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/batch.py", line 69, in __getitem__
    value = self._track_diff[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/diff.py", line 80, in __getitem__
    raise DiffMissingError(key, result)  # type: ignore # ignore over cast for perf reasons
eth.db.diff.DiffMissingError: (b'\x1f\x05\x93?z\xdft 5\xb5\\\xf3\xd2\xb7,\xe4,\xe3{\x0f\x05\x8a\xbe/\xe9\\\xb5\x87Oc\xaa\x18', 'Key is missing because it was never inserted')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 272, in _traverse_from
    node = self.get_node(next_node_pointer)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 570, in get_node
    encoded_node = self.db[node_hash]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/batch.py", line 74, in __getitem__
    return self.wrapped_db[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/accesslog.py", line 89, in __getitem__
    result = self.wrapped_db.__getitem__(key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/db/manager.py", line 291, in __getitem__
    raise KeyError(key)
KeyError: b'\x1f\x05\x93?z\xdft 5\xb5\\\xf3\xd2\xb7,\xe4,\xe3{\x0f\x05\x8a\xbe/\xe9\\\xb5\x87Oc\xaa\x18'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 133, in get
    return self._get(root_hash, trie_key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 143, in _get
    node, remaining_key = self._traverse(root_hash, trie_key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 207, in _traverse
    return self._traverse_from(root_node, trie_key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 276, in _traverse_from
    raise MissingTraversalNode(exc.args[0], used_key)
trie.exceptions.MissingTraversalNode: Trie database is missing hash HexBytes('0x1f05933f7adf742035b55cf3d2b72ce42ce37b0f058abe2fe95cb5874f63aa18'), found when traversing down (0xd, 0x5, 0x4).

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 341, in _get_encoded_account
    return lookup_trie[address]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/journal.py", line 336, in __getitem__
    return self._wrapped_db[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/cache.py", line 22, in __getitem__
    self._cached_values[key] = self._db[key]
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/accesslog.py", line 45, in __getitem__
    result = self.wrapped_db.__getitem__(key)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/keymap.py", line 28, in __getitem__
    return self._db[mapped_key]
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 778, in __getitem__
    return self.get(key)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/trie/hexary.py", line 140, in get
    ) from traverse_exc
trie.exceptions.MissingTrieNode: Trie database is missing hash HexBytes('0x1f05933f7adf742035b55cf3d2b72ce42ce37b0f058abe2fe95cb5874f63aa18') needed to look up node at prefix (0xd, 0x5, 0x4), when searching for key HexBytes('0xd541d86395704616f46d3be8cebb4d49c84d4aada60e7c794401d4f8cd567f6f') at root hash HexBytes('0xf07b2e276b71d4424e92d6132027792f608a326de8d8c50f077ccc374487342c')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/main.py", line 121, in _get_result
    self.event_bus, method, params, self.chain,
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/retry.py", line 108, in execute_with_retries
    return await func(*params)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/format.py", line 213, in formatted_func
    return await func(self, *formatted)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/rpc/modules/eth.py", line 286, in getTransactionCount
    nonce = state.get_nonce(address)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/vm/state.py", line 129, in get_nonce
    return self._account_db.get_nonce(address)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 247, in get_nonce
    account = self._get_account(address)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 352, in _get_account
    rlp_account = self._get_encoded_account(address, from_journal)
  File "/home/cburgdorf/Documents/hacking/ef/py-evm/eth/db/account.py", line 343, in _get_encoded_account
    raise MissingAccountTrieNode(*exc.args) from exc
eth.vm.interrupt.MissingAccountTrieNode: State trie database is missing node for hash 0x1f05933f7adf742035b55cf3d2b72ce42ce37b0f058abe2fe95cb5874f63aa18, which is needed to look up account with address hash 0xd541d86395704616f46d3be8cebb4d49c84d4aada60e7c794401d4f8cd567f6f at root hash 0xf07b2e276b71d4424e92d6132027792f608a326de8d8c50f077ccc374487342c -- (HexBytes('0x1f05933f7adf742035b55cf3d2b72ce42ce37b0f058abe2fe95cb5874f63aa18'), HexBytes('0xf07b2e276b71d4424e92d6132027792f608a326de8d8c50f077ccc374487342c'), HexBytes('0xd541d86395704616f46d3be8cebb4d49c84d4aada60e7c794401d4f8cd567f6f'), (0xd, 0x5, 0x4))
    INFO  2020-07-10 14:45:45,854   BaseBodyChainSyncer  Waited 21.1s for <BlockHeader #3021934 db62b585> body
    INFO  2020-07-10 14:45:45,854     BeamBlockImporter  Beam importing <BlockHeader #3021934 db62b585> (1 txns) ...
    INFO  2020-07-10 14:45:48,476   BaseBodyChainSyncer  Imported block 3021934 (1 txs) in 2.62 seconds, lagging 1 blocks | 20s
    INFO  2020-07-10 14:45:48,476     BeamBlockImporter  Beam importing <BlockHeader #3021935 0d0eb919> (2 txns) ...

I thought that this would automatically fetch the missing state but apparently it doesn't...investigating further.

Ah, found it, the method needs to have the @retryable decorator for it to automatically fetch missing state :)

Tada :tada: This is the first transaction ever send from metamask via a local Trinity node (Görli)
https://goerli.etherscan.io/tx/0x19afa0f2c8a1ac19bf40789ba18adee22937c901740d639cae4fa0a502ee0844
Still based on ugly unpushed code though

Just to finish my own monologue, the retryable isn't actually needed on the eth_sendRawTransaction since it doesn't depend on state. What I was seeing was an error produced by eth_getTransactionCount (which metamask fires off to get the nonce) and which does depend on state but that error is fixed by #1873

This doesn't yet implement any of the more sophisticated things that #29 mentions such as ensuring locally transmitted transactions survive across restarts. I'd like to keep this PR small and move these things over to a new issue.

Totally :+1: for reducing the scope of #29 and being able to mark it as complete with this PR. Would you link the new issue with the local treatment of transactions, and put it in the beta milestone?

What happens when you try to getTransactionReceipt or getTransaction on the pending transaction?

You will receive get returned a user friendly error telling that that we do not know about this transaction. As I type it out, I realize we could probably do better and take pending transactions into account. But since we don't have a concept of pending transactions yet, I'll move that into a follow up issue to #29

Was this supposed to remove the trinity/components/builtin/tx_pool/validators.py files?

Yep, good catch.

Created new issues: #1887 #1888 #1889

Totally +1 for reducing the scope of #29 and being able to mark it as complete with this PR. Would you link the new issue with the local treatment of transactions, and put it in the beta milestone?

Created #1887 for this. I wonder if we should create a new milestone for a post-beta and try to keep the beta reduced for the most pressing issues (whichever that are depending on our goals :sweat_smile: )

I wonder if we should create a new milestone for a post-beta and try to keep the beta reduced for the most pressing issues (whichever that are depending on our goals  )

Might be worth talking about at standup. My definition of beta is: no new features before going stable. So if it doesn't go into the beta milestone, then it would go into a v0.2 milestone. (I'm not decided on whether we can release v0.1 without support for looking up pending transactions... maybe it's fine)

My definition of beta is: no new features before going stable

Yep, I share that definition.

I'm not decided on whether we can release v0.1 without support for looking up pending transactions... maybe it's fine)

My personal opinion is that this would be fine but I say this under the assumption that we would have regular follow up releases (like at least every 3 months or so, ideally even more frequent) and so it wouldn't be a big deal to not ship this with the first version.

Might be worth talking about at standup.

Yep, let's do that.


Cannot clear "all" caches

[x] I am on the latest Poetry version.
[x] I have searched the issues of this repo and believe that this is not a duplicate.
[x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: Ubuntu 18.04
Poetry version: 1.0.0a3

Issue

poetry cache clear --all still expects a cache name (and there seems to be no commands for listing cache names).
$ poetry cache clear --all

[CannotParseArgsException]
Not enough arguments (missing: "cache").

Judging from this comment, there is a cache named pypi that has to be given. Perhaps the help message could be improved instead?

Is it possible to explain to people using a code example ??
for example:
poetry cache clear PyPi --all
Delete 86 entries? (yes/no) [yes] yes
yes.


added output_dir and verbose as 1st class params

closes #5

Codecov Report

Merging #10 into master will decrease coverage by 3.73%.
The diff coverage is 95%.

@@            Coverage Diff             @@
##           master      #10      +/-   ##
==========================================
- Coverage   92.85%   89.12%   -3.74%
==========================================
  Files           3        3
  Lines         238      239       +1
==========================================
- Hits          221      213       -8
- Misses         17       26       +9

Impacted Files
Coverage Δ

tuningdeap/main.py
89.2% <95%> (-4.2%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 276fadb...ea51b1c. Read the comment docs.

Will add an issue to create a test for checking the population size


crash on box select or use of rotation tool

as said on the tin, I cant use some of the better features of abnormal because of it and I have no idea what is going on. Logs say to clear out 2d/3d view handler. Google gives no advice as to what that means or how to do it. On 3.2 using version 1.2. Tried a fresh download and re install. Seems to be a TypeError: mathutil.Vector(). Zero idea what I could be doing wrong.
error is Python: Traceback (most recent call last):
File "C:\Users\Deryian\AppData\Roaming\Blender Foundation\Blender\3.2\scripts\addons\Abnormal\operators_modal.py", line 64, in modal
refresh_batches(self, context)
File "C:\Users\Deryian\AppData\Roaming\Blender Foundation\Blender\3.2\scripts\addons\Abnormal\functions_drawing.py", line 13, in refresh_batches
create_selection_drawing_lists(modal)
File "C:\Users\Deryian\AppData\Roaming\Blender Foundation\Blender\3.2\scripts\addons\Abnormal\functions_drawing.py", line 231, in create_selection_drawing_lists
init_loc = Vector(modal._mode_cache[0][0]).xy
TypeError: mathutils.Vector()

I just pushed a fix for this. Please redownload and give it another try to see if it is fixed for you

hadn't expected such a quick reply. Uninstalled the last version, re-installed the update, restarted blender, and the box select worked. So to did the rotate gizmo as well! Thank you for the quick reply and fix. Im glad I can now continue to use this add on in my work flow.


Feature request: List ports of selected object

In addition to #3, if an object is selected, it would be nice to have a list of all its ports. This could be placed in another tab in the side panel (where world view etc. is located). Or is this already implemented and I did not find It?

Thats a good idea imo 👍


Turbowarp error for AttributeError: 'TwCloudRequests' object has no attribute 'respond_in_thread'

I'm using repl, and it keeps coming up with that error for some reason.
Full error:
Traceback (most recent call last):
File "main.py", line 569, in
client.run()
File "/home/runner/TechnologicalLiveProcesses/venv/lib/python3.8/site-packages/scratchattach/_cloud_requests.py", line 334, in run
self._run(events, data_from_websocket=True)
File "/home/runner/TechnologicalLiveProcesses/venv/lib/python3.8/site-packages/scratchattach/_cloud_requests.py", line 295, in _run
self.call_request(request_id, req_obj, arguments)
File "/home/runner/TechnologicalLiveProcesses/venv/lib/python3.8/site-packages/scratchattach/_cloud_requests.py", line 55, in call_request
if not self.respond_in_thread:
AttributeError: 'TwCloudRequests' object has no attribute 'respond_in_thread'
(this was a test project)

nvm fixed

nvm its broken again :/

whenever it recieves a request it does the error

whenever it recieves a request it does it

hmmmm very strange. I see you are using replit. it could have something to do with that

Update scratchattach. There was a bug in v1.0, it is fixed now


raised JSONDecodeError

Bug description

Executing python code：python startup.py "写个类似今日头条的推荐系统"
Error reported as follows：raised JSONDecodeError

So, the error caused by incorrect json format. At the same time, I find the json format is different elsewhere, as follows:

Now, there are two questions, one is how to fix the bug, the other is why json format annotations are different? Is it random?
Bug solved method

The first question can fix by setting "REPAIR_LLM_OUTPUT: true" in key.yaml. The program runs normally after modification.

The Second question, why json format annotations are different? Is it random? I don't know the answer. Please tell me if you know anything.
Environment information

LLM type and model name: gpt-4-1106-preview
System version: macOS Sonoma Version 14.2.1
Python version: Python 3.11

json is from LLM, so it is random. We have to handle it to make it more standardized.


Script initialize() can be triggered twice

If an Entity is disabled at runtime but is enabled in the initialize() method of a script that runs before it in the hierarchy. It will be initialized twice. Once in onEnable, and once in the normal initialize phase.

Duplicate of #359

It also happens when entity is disabled in Editor.


Update the hashbrown to use the same version

Signed-off-by: koushiro<EMAIL_ADDRESS>

Thanks!


Dockerfile: Consolidate steps

Moving up workdir will make mkdir call redundant
Consolidate into a single RUN instruction
Prefix etc/prometheus with a slash (it only worked because WORKDIR was
/ )

Thanks!


profile-cleaner: 2.36 -> 2.37

Semi-automatic update generated by https://github.com/ryantm/nix-update tools. These checks were done:

built on NixOS
ran /nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37/bin/profile-cleaner -h got 0 exit code
ran /nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37/bin/profile-cleaner --help got 0 exit code
ran /nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37/bin/profile-cleaner help got 0 exit code
ran /nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37/bin/profile-cleaner -V and found version 2.37
ran /nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37/bin/profile-cleaner -v and found version 2.37
ran /nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37/bin/profile-cleaner --version and found version 2.37
ran /nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37/bin/profile-cleaner version and found version 2.37
ran /nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37/bin/profile-cleaner -h and found version 2.37
ran /nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37/bin/profile-cleaner --help and found version 2.37
ran /nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37/bin/profile-cleaner help and found version 2.37
found 2.37 with grep in /nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37
found 2.37 in filename of file in /nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37

cc @devhell for review

No attempt on x86_64-darwin
The following builds were skipped because they don't evaluate on x86_64-darwin: profile-cleaner
No log is available.

Success on x86_64-linux (full log)
Attempted: profile-cleaner
Partial log (click to expand)
ln -s pc.1.gz "/nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37""/share/man/man1/profile-cleaner.1.gz"
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37
gzipping man pages under /nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37/share/man/
strip is /nix/store/b0zlxla7dmy1iwc3g459rjznx59797xy-binutils-2.28.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37/bin
patching script interpreter paths in /nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37
/nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37/bin/.profile-cleaner-wrapped: interpreter directive changed from "/bin/bash" to "/nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/bash"
checking for references to /tmp/nix-build-profile-cleaner-2.37.drv-0 in /nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37...
/nix/store/b8vjlszy2ksb98qnmyyc49p2yhinvcr1-profile-cleaner-2.37

Success on aarch64-linux (full log)
Attempted: profile-cleaner
Partial log (click to expand)
ln -s pc.1.gz "/nix/store/g23wq2x8l7krz1jl3qgsm4qdjp1sfbk6-profile-cleaner-2.37""/share/man/man1/profile-cleaner.1.gz"
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/g23wq2x8l7krz1jl3qgsm4qdjp1sfbk6-profile-cleaner-2.37
gzipping man pages under /nix/store/g23wq2x8l7krz1jl3qgsm4qdjp1sfbk6-profile-cleaner-2.37/share/man/
strip is /nix/store/lvx1acn1ig1j2km8jds5x3ggh3f2wa8v-binutils-2.28.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/g23wq2x8l7krz1jl3qgsm4qdjp1sfbk6-profile-cleaner-2.37/bin
patching script interpreter paths in /nix/store/g23wq2x8l7krz1jl3qgsm4qdjp1sfbk6-profile-cleaner-2.37
/nix/store/g23wq2x8l7krz1jl3qgsm4qdjp1sfbk6-profile-cleaner-2.37/bin/.profile-cleaner-wrapped: interpreter directive changed from "/bin/bash" to "/nix/store/lnws52qanbi5vz8qg24v4ym57nznfndg-bash-4.4-p12/bin/bash"
checking for references to /build in /nix/store/g23wq2x8l7krz1jl3qgsm4qdjp1sfbk6-profile-cleaner-2.37...
/nix/store/g23wq2x8l7krz1jl3qgsm4qdjp1sfbk6-profile-cleaner-2.37


About the initialization of the pose for tracking optimization on Replica dataset

Hi! Thanks for your nice work!
I wonder that the reason why you choose multi-scale RGBD odometry to initialize the pose on Replica rather than using constant speed assumption which is used in other datasets. Could you please share some brief idea behind this choice?

Hi @pengchongH. Constant speed assumption works nicely also on Replica dataset. The odometry initialization here gives slightly better performance.

Thanks for your reply. I have tried to use constant speed on the Replica dataset. However, it looks like the constant speed cannot work well in room0 and room1, but it works well in office0.

I confirm the const speed initialization alone does lead to large pose error on room0 and room1. The is due to that a few times along the sequence there are much higher re-rendering error after initialization.
A mechanism is there to switch to odometer initialization in this case, please also enable help_camera_initialization in configs/Replica/replica.yaml when using const speed initialization. This helps get small pose error.

Thanks for your reply. I have tried this and it works well. I think this issue only occurs in room scenes, including room0-room2, in Replica, since you use const speed assumption for other datasets, and the const speed assumption also works well for office scenes in Replica. Based on this, I would like to know whether you have some brief idea why this kind of issue occurs.

Pose optimization is quite sensitive for initialization. We think that in the rooms since they fall to the local minima and can't recover, while odom initialization helps avoid them.

@pengchongH After investigation, we figured out that there was a big in the release branch. This function works, but not entirely correct - we need to initialize rotation quaternions differently. We fixed it and now testing it on all the datasets. We will push the fix to the main branch.

Pushed to the main. Feel free to reopen if you have any further questions!

Thanks for your reply. I will try your updated codes. Really appreciate!


roles/nv_gpu/tasks/ci_checks.yml: always capture Pod images and other info after the tests

@ArangoGutierrez can you have a look at this PR? I'd like to push it to release-4.* after it's merged

@ArangoGutierrez can you have a look at this PR? I'd like to push it to release-4.* after it's merged


Homepage/Grid Directory, Cards and Mock Data

Directory, Cards and Mock Data
📇 Changes proposed
This is a collection of improvements to the card styling, grid directory, a simple skeleton for a useBreedsDirectory hook and some mock data to show how the project might progress.
✅ Check List (Mark all the applicable boxes)
[x] - Correct; marked as done
[ ] - Not correct; marked as not done
-->

[x] My code follows the code style of this project.

[ ] This PR contains changes in the documentation

[ x The title of my pull request is a short description of the requested changes.

📄 Note to reviewers (optional)
Just an iteration on the work complete by @Abhishek7Tech, @yaredow, @musa108, @nwhator , @KarlsSoftware , @skoly1 and more!
📷 Screenshots (optional)

this pull request helps with #42


1.1 maven artifacts not being published (to artifactory)

Browsing https://odpi.jfrog.io/odpi/egeria-snapshot/org/odpi/egeria/
it appears that 1.1 artifacts are not being published despite the fact that the main project version is 1.1
This means it likely we won't be able to promote to staging either
cc: @bramwelt
Needs investigating

@planetf1 1.1 builds shouldn't be showing up in the snapshot repo as they're not snapshots. In order for the staging artifacts to be deployed into the egeria-staging repository, you'll want to start the 1.1 release process in AZP by clicking 'Create Release' and selecting the commit to stage from the egeria-release-1.1 branch.

Thanks for the explanation, we're in the stage of addressing some 1.1 issues prior to being ready to release/stage.. so I hadn't realise the build artifacts wouldn't go anywhere.
This probably means next time it makes sense to defer to version renaming until nearly ready for staging - if that's automated that's less of a problem in any case.
Closing, but we should capture this in the docs. Thanks

No worries, @planetf1. I'll have an PR to the docs up shortly where we can collaborate on the release steps.


Differential Rate Limiting Based on White Lists

I am operating nostream at nostr.drss.io. I would like to put specific pubkeys on a whitelist to grant them a different rate limit than the default for the relay. I don't want there to be no limit at all, just a different one. I have a personal use case for this, that of publishing events pulled from RSS feeds. I created this issue at @Cameri's request in note19kw9a07dw2qdpxwm23m022vfed2acfaclejsexwpaev5a2lp0k7sluh76w.

This can be useful also for IPs in a relay sync use case.


Add support for unknown fields

See https://docs.google.com/document/d/1KMRX-G91Aa-Y2FkEaHeeviLRRNblgIahbsk4wA14gRk/view and https://github.com/google/protobuf/issues/272.

I'm interested in this.
In order to make this work, unknown fields have to be stored in the structs somehow, something along the lines of adding an UnknownFieldSet to every generated message struct. It could be made to be just a null pointer and nothing more unless there are unknown fields.
This would be an API break, since existing struct instantiations would break because of not setting this field. A seemingly sensible way to make this sane is to recommend everyone to instantiate structs like this:
syntax = "proto3";

message Person {
  string name = 1;
}

Person { name: "Per".to_string(), ..Default::default() };

This would work if all generated structs would #[derive(Default)] (or equivalent), which seems like a sensible thing to do in general because Protobufs in general assume that no code breaks from just adding a field. (Not sure how this would interact with proto2 default values?) This also seems compatible with https://github.com/danburkert/prost/issues/43 (#[non_exhaustive]).
For completeness, the parsing code would probably have to be updated to at least not discard groups, even if it's not made capable of actually parsing them.
I don't know if known fields with unexpected type should go into the unknown fields set or not.
Does this make sense?

I have started hacking on this.
There are some ugly corner cases that I'm not entirely sure how to deal with. For example the well-known types such as BoolValue when represented as their Rust native types obviously won't be able to keep unknown values. The same applies to maps.

@per-gron thanks for the PR.  Could you explain the motivation behind the unknown fields feature?  I've read through the upstream docs linked in the first comment, but I don't really understand why it's become such a priority for Google to support this feature.  My take is that unknown fields are full of subtle footguns[1], and the described usecases are better either not a good fit for protobuf (RMW) or better done through preserving the original encoded message (intermediary servers).

@danburkert I honestly don't know exactly why either, but I can speculate a bit: It seems like the fact that proto3 is different from proto2 in this regard has made it difficult for a lot of internal projects to adopt proto3. It's an API break that simply makes it scary to upgrade when you have a huge code base, and the difference is not important enough to justify a change of behavior.
I agree that this feature is footgun-prone; yet I think it is better to behave as similarly as possible to the other protobuf libraries than to not have it at all. (It is of course possible to provide some kind of Prost setting to disable the feature for projects that don't care about having the same behavior across languages.)
For me personally, I don't really care much about this particular feature or lack of it, the reason I wanted to work on this is that it seems like Prost is a very nice Rust Protobuf library, and I think it would be good for Rust to have a more "officially endorsed" version that people can trust will work as expected, including across languages. When working with large projects and across different languages, uniformity really helps a lot, even with really small details.
(Note: Even though I work at Google I don't have any special power to "officialize" any particular Rust Protobuf library, but I hope that fixing details like this one so that its behavior is very close to the main Protobuf libraries along with having people from the Rust community agree that Prost's struct-based API is nice will go a long way.)

Just to set expectations, it's never been a goal of mine to have prost officially endorsed by Google or the upstream Protobuf authors or community.  Nor has it been a goal to have feature compatibility with other Protobuf libraries, since there are so many Protobuf features (at least in the upstream C++ implementation).  My goals with prost are  roughly (and probably omitting some important ones):

Idiomatic generated code
performance
straightforward build system integration
Compatibility with other Protobuf libraries (wouldn't really be protobuf otherwise)
Support the core proto2/proto3 features

The upstream project has done a great job publishing conformance tests which prost uses to ensure compatibility.
As far as adding features for the sake of parity, there's no way I can feasibly do that as a single maintainer, nor do I think I could even maintain such a library if the features were contributed by others.  prost is currently 'feature-complete' for the purposes of the application for which I wrote it, so isn't a burning need for me to keep adding features.  I'm especially disinclined to add features which I think are harmful to creating flexible applications, and while I haven't completely made up my mind, I tend to think unknown fields fall in that category.  I've yet to come across a solid usecase for them that isn't error prone or better solved in a different way.

Thanks for the information about your priorities with Prost. (I've been on vacation hence my slow reply.)
The reasons that I wrote this PR are 1) there was an open issue about it written by you, 2) unlike JSON/groups/reflection this is something that is or will soon be required by the spec so this is not quite the same as those other features.
Given help with maintenance, would you be interested in expanding the scope of this library to include features that makes it possible to get broader adoption? If not, would you support a fork with that goal?

I don't really understand why it's become such a priority for Google to support this feature. My take is that unknown fields are full of subtle footguns1, and the described usecases are better either not a good fit for protobuf (RMW) or better done through preserving the original encoded message (intermediary servers).

I happen to work at Google (not on protobufs!) and know folks who really pushed for this (reversing the proto 2->3 decision to drop unknown field handling). Those two use cases are the reason why.
I'm not sure why you say protobuf is not a good fit for RMW; it's overwhelmingly common for databases to be full of protobuf values which servers do RMW on. It generally works well...but most teams are still using proto 2 (with no plans to adopt proto 3). Some that started using proto 3 got nasty surprises; thus the push.
I see your point intermediary servers could preserve the original encoded message, but there are also reasons it's easier to work with a message field embedded in a message field rather than a bytes field embedded in a message field. Think ease of writing an ASCII message (to check in as test data or to send an RPC by hand on the commandline while debugging), ease of understanding debug representations (inverse of the above), and not having to explicitly have code do additional serialization/deserialization steps / possibly mess up the type of the contained proto.
The oneof footgun doesn't concern me too much, fwiw. It's wrong to have a message that has more than one of the oneof present; inconsistent behavior in which is used should probably be expected. There are bigger footguns involved in proto compatibility (reusing field numbers / changing types of fields is a huge common way to screw up; there are a lot of variations of how to make this mistake and how it can bite), but dealing with backward and forward compatibility can't be avoided in many applications.

Given help with maintenance, would you be interested in expanding the scope of this library to include features that makes it possible to get broader adoption? If not, would you support a fork with that goal?

I'm also interested in the answer here (though I can't commit to significant help myself). I appreciate the hard work you've done on this project. Nonetheless, it's a bit frustrating that there are at least three apparently-commonly-used Rust proto implementations, with none being a superset of the others (much less matching Google's official C++ protobuf implementation). I'd love a path to resolve that, and that probably starts with knowing if the base should be prost or one of the others.

Hi, what's the current status for supporting unknown fields? I see multiple attempts over the past couple of years but no progress.
It would be great to rm this file :) https://github.com/tokio-rs/prost/blob/master/conformance/failing_tests.txt

Unknown fields still need to be supported in edition 2023 onwards.
I can think of two reasonable ways to support this:
In-message state
Add a private field to every generated message to be exposed via an .unknown_fields() accessor in Message:
#[derive(prost::Message)]
pub struct MyMessage {
    __prost_unknown_fields: Vec<prost::UknownField>,

    #[prost(string, tag = "1")]
    pub a_field: String,
}

This will effectively make every message non-exhaustive for matching and impossible to construct with struct expressions, so it will need builders like in #901. These can be considered good features with regard to protobuf versioning semantics. This change can be introduced as a build option to preserve backward compatibility.
One disadvantage is an increase in struct size for every message, regardless of need.
Out-of-band data for encode/decode
Add methods to Message to enable round-tripping with unknown fields for users who need them:
pub trait Message {
    // ...

    fn decode_with_unknown(
        mut buf: impl Buf,
    ) -> Result<(Self, Vec<UnknownField>)>, DecodeError>
    where
        Self: Default,
    {
        // ...
    }

    fn encode_with_unknown(
        &self,
        unknown_fields: &[UnknownField],
        buf: &mut impl BufMut
    ) -> Result<(), EncodeError>
    where
        Self: Sized,
    {
         // ...
    }
}

I think this could be added in a backward-compatible way by providing a kind of DecodeError that transports unknown fields, and handling it in the in-crate and generated implementations.

I can think of two reasonable ways to support this:
In-message state

I like this approach. It guarantees that the unknown fields are part of the message and it doesn't have to break existing APIs.

Add a private field to every generated message to be exposed via an .unknown_fields() accessor in Message:

It is not necessary to hide this field. We can make it a publicly accessible field as long as it doesn't have a conflicting name. The public field also prevents the struct from becoming non-exhaustive.

One disadvantage is an increase in struct size for every message, regardless of need.

I think it should be an optional feature to save unknown fields. When people are interested, they can enable it in the code generator with a self chosen name.

In-message state

I like this approach. It guarantees that the unknown fields are part of the message and it doesn't have to break existing APIs.

Only if it is opted in by matcher settings in Config, otherwise it will break generated APIs, which for prost-build is tantamount to breaking its own API.

Add a private field to every generated message to be exposed via an .unknown_fields() accessor in Message:

It is not necessary to hide this field. We can make it a publicly accessible field as long as it doesn't have a conflicting name.

OK, with a configurable name they can avoid conflicts.

The public field also prevents the struct from becoming non-exhaustive.

A developer who wants their generated Rust bindings to adhere to Protobuf semver semantics would actually want the message structs to be non-exhaustive, so that future field additions do not become breaking changes. But this can be done by other means, such as the #[non_exhaustive] attribute.
I expect the added field to be a nuisance to any user who doesn't care about unknown fields, so they'll match it with .. anyway.

One disadvantage is an increase in struct size for every message, regardless of need.

I think it should be an optional feature to save unknown fields. When people are interested, they can enable it in the code generator with a self chosen name.

One problem with this is, it's yet another setting affecting the generated API that users might disagree on. In an ideal world, a set of proto packages is represented by a single Rust library crate that everybody else can depend on, rather than generating sets of incompatible types for the same thing (a-la prost-types vs. pbjson-types vs...). If the in-struct baggage for unknown fields will prove as unpopular as I expect, developers of such "designated binding" crates would be discouraged from enabling it, meaning that it won't be generally available for binding consumers. A feature gate guarding the struct members is not usually a good solution, because API should not be breakable by enabling feature gates.

I'm leaning towards the out-of-band approach. The general philosophy of Prost, as I see it, has been to make simple things easy and hard things sometimes possible. Most protobuf applications do not care about unknown fields and would not know what to do with them because, well, if they'd know the format of a field, they'd use a proto revision that describes it!
Support for unknown fields is necessary for components that inspect messages, possibly set or modify some known fields, then re-encode the messages to pass them along with any fields they don't recognize. With the parallel, recursive UnknownFields data structure as I sketched it, it would be easy to handle the unknown fields of the top-level message, but modifications in nested message fields would need some painstaking effort to keep the parallel data structure up to date.

API should not be breakable by enabling feature gates.

Instead, prost-build could generate separate struct variants augmented with uknown fields, in an added-on module, and provide conversions between the two:
#[derive(prost::Message)]
pub struct Foo {
    #[prost(message, optional, tag = "1")]
    pub bar: Option<Bar>,
}

#[derive(prost::Message)]
pub struct Bar {}

#[cfg(feature = "unknown_fields")]
pub mod with_unknown_fields {
    #[derive(prost::Message, prost::UnknownFields)]
    pub struct Foo {
        #[prost(unknown_fields)]
        __prost_unknown_fields: prost::UnknownFields,
        #[prost(message, optional, tag = "1")]
        pub bar: Option<Bar>,
    }

    #[derive(prost::Message, prost::UnknownFields)]
    pub struct Bar {
        #[prost(unknown_fields)]
        __prost_unknown_fields: prost::UnknownFields,
    }

    // Convert to a `Foo` message struct not augmented with unknown field
    // information.
    impl From<Foo> for super::Foo {
        fn from(full_message: Foo) -> Self {
            todo!()
        }
    }

    // Convert from a `Foo` message struct carrying no unknown fields.
    impl From<super::Foo> for Foo {
        fn from(foo: super::Foo) -> Self {
            todo!()
        }
    }

    impl prost::WithUnknownFields for super::Foo {
        type FullMessage = Foo;

        fn with_unknown_fields(
            self,
            unknown_fields: prost::UnknownFields,
        ) -> Result<Foo, prost::UnknownFieldError> {
            todo!("check if the unknown field tags are not found among the known fields")
        }
    }
}


"knife bootstrap" throws an exception when "--policy-name" and "--policy-group" flags provided

Description
knife bootstrap's --policy-name and --policy-group flags do not appear to be functional; instead they cause knife to raise exceptions.
Chef Version
Encountered issue in 14.2.0 and 14.11.21.  Have not tried any other versions yet.
Platform Version
Arch Linux
Replication Case
[birdsnest ~/Work/git/chef_metadata](master|✔)[N]% knife bootstrap -i ~/.ssh/localization_rsa -N unix-766 --policy-name UNIX-766 --policy-group testing -x root --bootstrap-version=14.2.0 -VVV localhost
INFO: Using configuration from /home/jay/.chef/knife.rb
Doing old-style registration with the validation key at /home/jay/.chef/validation.pem...
Delete your validation key in order to use your user credentials instead

Connecting to localhost
TRACE: Looking for bootstrap template in /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/knife/bootstrap/templates
TRACE: Found bootstrap template in /opt/chefdk/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/knife/bootstrap/templates
ERROR: knife encountered an unexpected error
This may be a bug in the 'bootstrap' knife command or plugin
Please collect the output of this command with the `-VVV` option before filing a bug report.
Exception: NoMethodError: undefined method `empty?' for nil:NilClass

However, knife does not complain when the policy name and policy group are provided via --json-attribute-file, and bootstrap completes successfully:
[birdsnest ~/Work/git/chef_metadata](master|✔)[I]% cat nodes/unix-766.json
{
  "name": "unix-766",
  "policy_name": "UNIX-766",
  "policy_group": "testing"
}
[birdsnest ~/Work/git/chef_metadata](master|✔)[N]% knife bootstrap -i ~/.ssh/localization_rsa -N 'unix-766' -x root --json-attribute-file nodes/unix-766.json --bootstrap-version=14.2.0 localhost
Doing old-style registration with the validation key at /home/jay/.chef/validation.pem...
Delete your validation key in order to use your user credentials instead

Connecting to localhost
Enter passphrase for /home/jay/.ssh/localization_rsa:
localhost -----> Existing Chef installation detected
localhost Starting the first Chef Client run...
localhost Starting Chef Client, version 14.2.0
localhost Creating a new client identity for unix-766 using the validator key.
localhost [2019-03-21T17:19:17-06:00] WARN: Using deprecated positional arguments for sign(), please update to keyword arguments (from /opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/http/authenticator.rb:114)
localhost [2019-03-21T17:19:18-06:00] WARN: Using deprecated positional arguments for sign(), please update to keyword arguments (from /opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/http/authenticator.rb:114)
localhost [2019-03-21T17:19:18-06:00] WARN: Using deprecated positional arguments for sign(), please update to keyword arguments (from /opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/http/authenticator.rb:114)
localhost [2019-03-21T17:19:18-06:00] WARN: Using deprecated positional arguments for sign(), please update to keyword arguments (from /opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/http/authenticator.rb:114)
localhost Using policy 'UNIX-766' at revision '2937f320a7f0bf998efd13f501b1f2d4f0dcbc2745a9fb580540d5ff8fc2e1c9'
localhost [2019-03-21T17:19:18-06:00] WARN: Using deprecated positional arguments for sign(), please update to keyword arguments (from /opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/http/authenticator.rb:114)
localhost [2019-03-21T17:19:18-06:00] WARN: Using deprecated positional arguments for sign(), please update to keyword arguments (from /opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/http/authenticator.rb:114)
localhost resolving cookbooks for run list<EMAIL_ADDRESS>(7f1ba72)"]
localhost [2019-03-21T17:19:18-06:00] WARN: Using deprecated positional arguments for sign(), please update to keyword arguments (from /opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/http/authenticator.rb:114)
localhost Synchronizing Cookbooks:
localhost   - UNIX-766 (0.1.0)
localhost Installing Cookbook Gems:
localhost Compiling Cookbooks...
localhost [2019-03-21T17:19:18-06:00] WARN: Using deprecated positional arguments for sign(), please update to keyword arguments (from /opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/http/authenticator.rb:114)
localhost Converging 1 resources
localhost Recipe: UNIX-766::default
localhost   * file[/tmp/foo] action create
localhost     - create new file /tmp/foo
localhost     - update content in file /tmp/foo from none to fcde2b
localhost     --- /tmp/foo	2019-03-21 17:19:18.494413841 -0600
localhost     +++ /tmp/.chef-foo20190321-7171-1y53quo	2019-03-21 17:19:18.494413841 -0600
localhost     @@ -1 +1,2 @@
localhost     +bar
localhost     - restore selinux security context
localhost [2019-03-21T17:19:18-06:00] WARN: Using deprecated positional arguments for sign(), please update to keyword arguments (from /opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/http/authenticator.rb:114)
localhost
localhost Running handlers:
localhost Running handlers complete
localhost Chef Client finished, 1/1 resources updated in 02 seconds

Client Output
see above
Stacktrace
n/a

Nevermind.  I didn't realize I still had the workaround for https://github.com/chef/chef/issues/5353 in place, which, as another user commented:

As a small side-note the monkey-patch above fixes the reported problem but seems to then break --policy-group / --policy-name options for knife bootstrap.

Since #5353 has been fixed in recent versions of Chef, I removed the workaround, which solved the problem.


Clean up in NOTICE.md

Description
Removed duplicated statement

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Daniel seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.