feat: Fast storage optimization for queries and iterations

Background
Link to the original spec
Link to the original PR in Osmosis
Background
Historically IAVL has had a very slow performance during state machine execution, and for responding to queries to live state. This release speeds up these routines by an order of magnitude, alleviating large amounts of pressure from all users of the IAVL database.
Details
This PR introduces an auxiliary fast storage system to IAVL, which represents a copy of the latest state much more amenable to efficient querying and iteration.
Prior to this PR, all data gets & iterations suffered two significant performance drawdowns:

Every get/iteration is forced to walk the tree structure
Every node (including leaves) is stored on disk, with its key being a SHA256 hash of something.

All nodes were indexed by their Merkle tree inner node hash. This breaks data locality and makes every Get() that should be in RAM / CPU caches instead be a random leveldb file open.
The fast storage nodes are instead indexed by the logical key on the disk. This allows us to preserve data locality for the latest state, significantly improving iterations and queries. (Depending on the particular benchmark, between 5-30x improvements) This implementation introduces a negligible overhead for writes.
Downgrade-re-upgrade protection
We introduced a downgrade and re-upgrade protection where we guard for potential downgrades of iavl and the subsequent enablement of the fast storage again. This is done so by storing the metadata about the current version of the storage and the latest live state stored.
Summary of Changes
IAVL is divided into two trees, mutable_tree and immutable_tree. Sets only happen on the mutable tree.
Things that need to change and be investigated for getting and setting, and the fast node:

mutable tree

GetVersioned

[x] Change the logic to check the FastNode cache first, then do the GetImmutable logic

Set

[x] Cache unsaved fast nodes in memory, avoid persisting them to disk right away
[x] Update unsaved removals if needed

Remove

[x] Cache unsaved removals in memory, avoid persisting changes to disk
[x] Update unsaved additions if needed

SaveVersion

[x] Persist unsaved fast node additions and removals to disk
[x] Sort by key before saving to db to ensure data locality
[x] Potential optimizations:

[x] Look into if removing before writing to db is faster
[x] Look into if combining sorted removals with sorted additions is faster than doing them one after the other
[x] Conclusion is documented here: #13

Iterate

[x]  Now that we have some unsaved changes cached in the mutable tree, we cannot use the Iterate's implementation from the immutable tree. Introduce this new method.
[x] Ensure that we iterate in the most efficient manner in sorted order by having 2 pointers:

to the next element on disk
to the next unsaved change in the mutable tree

[x] Compare the values of the 2 pointers on each iteration and choose the next one

Iterator

[x] Immutable tree is embedded into mutable. From the way composition works in Go, the mutable tree can access all methods and fields of the immutable. So we need to overwrite the implementation of the immutable tree and return an invalid iterator by default.
[x] The iterator in the mutable tree is invalid because we cannot support updates and delayed iterations at the same time.

Get

[x] For the same reason as in Iterate, we must check unsaved additions first before attempting to use the strategy employed by the immutable tree

enableFastStorageAndCommit and its variations

[x] These are the methods that are used to perform automatic upgrades to fast storage if the system detects that fast cache is not used. This detection happens by checking the value of a special metadata key called mstorage_version where m is a new prefix. If the version is lower than the fastStorageVersionValue threshold - migration is triggered.
[x] LoadVersion, LazyLoadVersion

added upgrade logic and downgrade + re-upgrade protection. It works by storing the storage version in the database.
if storage version is low -> upgrade
if the latest fast node version on disk does not match latest version in ndb -> upgrade

[x] isUpgradeable - determines if the upgrade sis going to be performed. This method can be called on the SDK side to determine if we should log a warning message

immutable_tree

Get and (GetWithIndex

[x] renamed Get to GetWithIndex. GetWithIndex always uses the default live state traversal strategy
[x] introduced Get method. Get attempts to use the fast cache first. Only fallbacks to regular tree traversal strategy if the fast cache is disabled or tree is not of the latest version

Iterator

returns either regular or updated fast iterator (see more below) depending on if fast storage is enabled and migration is complete

nodedb

[x] updated and tested the underlying storage management logic

fast_iterator

[x] introduced and tested a new iterator that binds directly to the database iterator by searching for keys that begin with the prefix f which stands for fast. Basically, all fast nodes are sorted on disk by key in ascending order so we can simply traverse the disk ensuring efficient hardware access.

unsaved_fast_iterator

[x] iterates over the latest state via fast nodes, taking into account unsaved changes to fast nodes that could have been made within a mutable tree

testing

[x] unit tests for mutable tree Get, Set, Save, Delete
[x] unit tests for immutable - Get, Iterator - fast and slow
[x] some unit tests for  nodedb
[x] updated randomized tests that function like integration tests
[x] updated bench tests

Old Benchmark
Date: 2022-01-22 12:33 AM PST
Branch: dev/iavl_data_locality with some modifications to the bench tests
go version go1.17.6 linux/amd64

Init Tree took 78.75 MB
goos: linux
goarch: amd64
pkg: github.com/cosmos/iavl/benchmarks
cpu: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz
BenchmarkMedium/memdb-100000-100-16-40/query-no-in-tree-guarantee-slow-8         	   95841	     12315 ns/op	     592 B/op	      12 allocs/op
BenchmarkMedium/memdb-100000-100-16-40/query-hits-slow-8                         	   85990	     15533 ns/op	     760 B/op	      15 allocs/op
BenchmarkMedium/memdb-100000-100-16-40/iteration-slow-8                          	       2	 838458850 ns/op	88841084 B/op	 1600092 allocs/op
--- BENCH: BenchmarkMedium/memdb-100000-100-16-40/iteration-slow-8
    bench_test.go:109: completed 100000 iterations
    bench_test.go:109: completed 100000 iterations
    bench_test.go:109: completed 100000 iterations
BenchmarkMedium/memdb-100000-100-16-40/update-8                                  	   10000	    148915 ns/op	   27312 B/op	     335 allocs/op
BenchmarkMedium/memdb-100000-100-16-40/block-8                                   	      76	  16865728 ns/op	 2910568 B/op	   35668 allocs/op
Init Tree took 46.71 MB
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-slow-8     	   55309	     22354 ns/op	    1550 B/op	      30 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-slow-8                     	   43566	     27137 ns/op	    2093 B/op	      39 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-8                      	       1<PHONE_NUMBER> ns/op	225813440 B/op	 3857215 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-8
    bench_test.go:109: completed 100000 iterations
BenchmarkMedium/goleveldb-100000-100-16-40/update-8                              	    6194	    307266 ns/op	   40138 B/op	     406 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/block-8                               	      28	  40663600 ns/op	 5150422 B/op	   53771 allocs/op
PASS
ok  	github.com/cosmos/iavl/benchmarks	25.797s

Latest Benchmark
Date: 2022-01-22 10:15 AM PST
Branch: roman/fast-node-get-set
go version go1.17.6 linux/amd64

Init Tree took 114.29 MB
goos: linux
goarch: amd64
pkg: github.com/cosmos/iavl/benchmarks
cpu: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz
BenchmarkMedium/memdb-100000-100-16-40/query-no-in-tree-guarantee-fast-8         	  672999	      1887 ns/op	     112 B/op	       4 allocs/op
BenchmarkMedium/memdb-100000-100-16-40/query-no-in-tree-guarantee-slow-8         	   95888	     11884 ns/op	     440 B/op	       8 allocs/op
BenchmarkMedium/memdb-100000-100-16-40/query-hits-fast-8                         	  891831	      1208 ns/op	      16 B/op	       0 allocs/op
BenchmarkMedium/memdb-100000-100-16-40/query-hits-slow-8                         	   79842	     15644 ns/op	     607 B/op	      11 allocs/op
BenchmarkMedium/memdb-100000-100-16-40/iteration-fast-8                          	      20	  63956090 ns/op	18400254 B/op	  300000 allocs/op
--- BENCH: BenchmarkMedium/memdb-100000-100-16-40/iteration-fast-8
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
	... [output truncated]
BenchmarkMedium/memdb-100000-100-16-40/iteration-slow-8                          	       2	 947611750 ns/op	88841044 B/op	 1600092 allocs/op
--- BENCH: BenchmarkMedium/memdb-100000-100-16-40/iteration-slow-8
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
BenchmarkMedium/memdb-100000-100-16-40/update-8                                  	    9198	    174306 ns/op	   27524 B/op	     342 allocs/op
BenchmarkMedium/memdb-100000-100-16-40/block-8                                   	      58	  19855266 ns/op	 2948779 B/op	   36495 allocs/op
Init Tree took 66.82 MB
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-fast-8     	  228343	      4938 ns/op	     814 B/op	      16 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-no-in-tree-guarantee-slow-8     	   59304	     18046 ns/op	    1420 B/op	      24 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-fast-8                     	  611349	      1684 ns/op	      93 B/op	       1 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/query-hits-slow-8                     	   50778	     23126 ns/op	    2005 B/op	      34 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-8                      	      12	  94702442 ns/op	29327220 B/op	  522988 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-fast-8
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
    bench_test.go:117: completed 100000 iterations
	... [output truncated]
BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-8                      	       1<PHONE_NUMBER> ns/op	235504072 B/op	 3998006 allocs/op
--- BENCH: BenchmarkMedium/goleveldb-100000-100-16-40/iteration-slow-8
    bench_test.go:117: completed 100000 iterations
BenchmarkMedium/goleveldb-100000-100-16-40/update-8                              	    8994	    257683 ns/op	   44702 B/op	     447 allocs/op
BenchmarkMedium/goleveldb-100000-100-16-40/block-8                               	      31	  44907345 ns/op	 6973362 B/op	   72924 allocs/op
PASS
ok  	github.com/cosmos/iavl/benchmarks	43.513s

Benchmarks Interpretation
Highlighting the difference in performance from the latest benchmarks:
Old branch is dev/iavl_data_locality
New branch is roman/fast-node-get-set
Initial size: 100,000 key-val pairs
Block size: 100 keys
Key length: 16 bytes
Value length: 40 bytes
Query with no guarantee of the key being in the tree:

Old: 22354 ns/op
New on regular logic: 18046 ns/op
New on fast logic: 4938 ns/op
New fast logic  shows a 77% decrease in time relative to the old branch

Query with the key guaranteed to be in the latest tree:

Old:  27137 ns/op
New on regular logic: 23126 ns/op
New on fast logic: 1684 ns/op
New fast logic  shows a 93% decrease in time relative to the old branch

Iteration:

Old:<PHONE_NUMBER> ns/op
New on old logic:<PHONE_NUMBER> ns/op
New on fast logic: 94702442 ns/op
New fast logic shows a 96% decrease in time relative to the old branch

Update:
run Set, if this is a try that is divisible by blockSize, attempt to SaveVersion and if the latest saved version number history exceeds 20, delete the oldest version

Old: 307266 ns/op
New: 257683 ns/op
New logic shows a 16% decrease in time relative to the old branch

Block:
for block size, run Get and Set. At the end of the block, SaveVersion and if the latest saved version number history exceeds 20, delete the oldest version

Old: 40663600 ns/op
New: 44907345 ns/op
New logic shows a 9% increase in time relative to the old branch

@odeke-em Thanks for the feedback. I will address all comments and fix the benchmarks soon

@odeke-em Thanks so much for such a great and detailed review.
I addressed all your comments. The current TODOs left for me in this PR are the following:

[ ] resolve all linter issues
[ ] better benchstat summary
[ ] Look into the failing bench test - related to cleveldb and might be due to a recent tendermint upgrade

Please note that some of the benchmarks that are worse-off are expected. Optimizing for reads forces us to introduce some additional overhead for writes. I will get back with a more detailed report on every negative change:

[ ] document all performance regressions

Better benchstat summary:
~/projects/iavl (master)$ benchstat bench_old_large.log bench_fast_large.log
name                                                             old time/op    new time/op    delta
Large/memdb-1000000-100-16-40/query-no-in-tree-guarantee-16        11.6µs ±18%     2.1µs ± 6%  -81.86%  (p=0.008 n=5+5)
Large/memdb-1000000-100-16-40/query-hits-16                        13.0µs ± 3%     1.5µs ±21%  -88.06%  (p=0.008 n=5+5)
Large/memdb-1000000-100-16-40/iteration-16                          7.05s ± 9%     0.52s ± 2%  -92.67%  (p=0.008 n=5+5)
Large/memdb-1000000-100-16-40/update-16                             220µs ±10%     218µs ± 9%     ~     (p=0.841 n=5+5)
Large/memdb-1000000-100-16-40/block-16                             23.0ms ± 3%    24.3ms ± 2%   +5.28%  (p=0.016 n=5+5)
Large/goleveldb-1000000-100-16-40/query-no-in-tree-guarantee-16    21.5µs ±11%     5.7µs ± 2%  -73.60%  (p=0.008 n=5+5)
Large/goleveldb-1000000-100-16-40/query-hits-16                    21.8µs ± 5%     2.7µs ±28%  -87.75%  (p=0.008 n=5+5)
Large/goleveldb-1000000-100-16-40/iteration-16                      16.6s ±11%      0.5s ± 1%  -96.81%  (p=0.008 n=5+5)
Large/goleveldb-1000000-100-16-40/update-16                         245µs ±11%     269µs ±10%     ~     (p=0.222 n=5+5)
Large/goleveldb-1000000-100-16-40/block-16                         28.8ms ± 9%    31.8ms ±19%     ~     (p=0.151 n=5+5)

name                                                             old alloc/op   new alloc/op   delta
Large/memdb-1000000-100-16-40/query-no-in-tree-guarantee-16          566B ±37%       88B ± 0%  -84.45%  (p=0.008 n=5+5)
Large/memdb-1000000-100-16-40/query-hits-16                          606B ± 0%       67B ±46%  -88.91%  (p=0.008 n=5+5)
Large/memdb-1000000-100-16-40/iteration-16                          888MB ± 0%     184MB ± 0%  -79.28%  (p=0.016 n=4+5)
Large/memdb-1000000-100-16-40/update-16                            36.7kB ± 3%    37.2kB ± 3%     ~     (p=0.222 n=5+5)
Large/memdb-1000000-100-16-40/block-16                             3.80MB ± 1%    3.86MB ± 1%   +1.62%  (p=0.008 n=5+5)
Large/goleveldb-1000000-100-16-40/query-no-in-tree-guarantee-16    2.72kB ±29%    0.99kB ± 0%  -63.51%  (p=0.008 n=5+5)
Large/goleveldb-1000000-100-16-40/query-hits-16                    2.70kB ± 1%    0.34kB ±46%  -87.34%  (p=0.008 n=5+5)
Large/goleveldb-1000000-100-16-40/iteration-16                     3.71GB ± 0%    0.30GB ± 0%  -92.00%  (p=0.016 n=4+5)
Large/goleveldb-1000000-100-16-40/update-16                        79.8kB ± 6%    84.1kB ±10%     ~     (p=0.421 n=5+5)
Large/goleveldb-1000000-100-16-40/block-16                         9.05MB ± 9%    9.82MB ±16%     ~     (p=0.095 n=5+5)

name                                                             old allocs/op  new allocs/op  delta
Large/memdb-1000000-100-16-40/query-no-in-tree-guarantee-16          10.8 ±39%       3.0 ± 0%  -72.22%  (p=0.008 n=5+5)
Large/memdb-1000000-100-16-40/query-hits-16                          11.0 ± 0%       1.0 ± 0%  -90.91%  (p=0.000 n=5+4)
Large/memdb-1000000-100-16-40/iteration-16                          16.0M ± 0%      3.0M ± 0%  -81.25%  (p=0.016 n=4+5)
Large/memdb-1000000-100-16-40/update-16                               466 ± 8%       459 ± 8%     ~     (p=1.000 n=5+5)
Large/memdb-1000000-100-16-40/block-16                              52.0k ± 0%     53.2k ± 0%   +2.29%  (p=0.008 n=5+5)
Large/goleveldb-1000000-100-16-40/query-no-in-tree-guarantee-16      37.2 ±29%      19.0 ± 0%  -48.92%  (p=0.008 n=5+5)
Large/goleveldb-1000000-100-16-40/query-hits-16                      36.0 ± 0%       6.0 ±50%  -83.33%  (p=0.008 n=5+5)
Large/goleveldb-1000000-100-16-40/iteration-16                      49.8M ± 0%      5.3M ± 0%  -89.34%  (p=0.016 n=4+5)
Large/goleveldb-1000000-100-16-40/update-16                           684 ± 6%       729 ±12%     ~     (p=0.310 n=5+5)
Large/goleveldb-1000000-100-16-40/block-16                          81.5k ± 7%     84.8k ±12%     ~     (p=0.421 n=5+5)

Since the API has changes and bench tests were rewritten between master and the current branch, I had to create a custom version of bench_test.go that works for both branches.

The bencher was failing from an earlier commit and is now broken on master.
From initial investigation, it only happens for databases other than goleveldb and memdb so I commented them out.
I'm going to let bencher pass for goleveldb and memdb.
For all others, #469 is created. To be addressed in another PR.

An update from me: Osmosis validators reported an increased RAM usage while upgrading to fast storage. I've been digging into that. So far, I don't see any issues besides just naturally requiring more RAM to create a copy of the live state. I have some optimization ideas and will document them as soon as the investigation is done.
I've been mostly doing heap profiles. Now, I want to make sure that there is no memory leak due to a blocked thread somewhere. However, this isn't likely since validators only reported the increased RAM during the upgrade. Eventually, it goes down. Again, the increased RAM usage during the upgrade is expected though it can be optimized if that's too bad of an issue.
I heard reports of getting OOM killed with 16GB RAM node in Osmosis, still need to try testing that myself.
TODOs left in this PR:

[ ] look into gobencher timing out with large data
[ ] document all performance regressions

everything I observed so far is expected, just need some spare cycles to document it for the reviewers

Is the FastNode DB only in the RAM, or you store it in DM? I think we need the latter, and clean the cache in RAM every x blocks. Otherwise it will dramatically change the validator requirements.

No, it is primarily in the DM. There is some configurable cache in nodedb. Also, some additions and removals are persisted in memory until we SaveVersion them to DM.
We have observed some RAM increase. However, nothing notable so far.
There were issues with RAM increasing during the upgrade of the storage to the point where it would get OOM'ed on 16 Gb nodes.
However, I have a spike to address that here: https://github.com/osmosis-labs/iavl/pull/19
It does the following:

force GC every second
avoid using the nodedb cache during the upgrade

Added several data race fixes that have been identified by running nodes in Osmosis.
This PR is updated with the latest changes and caught up with the master branch.
My local tests and linter are passing. Could someone run the CI please? I don't have the permissions.
Also, gobencher is failing for reasons outside of my control: https://github.com/cosmos/iavl/pull/468/checks?check_run_id=5726930260

thanks for the update, sorry for the delay I was off last week. We will get to this asap

Better benchstat summary:
~/projects/iavl (master)$ benchstat bench_old_large.log bench_fast_large.log
name                                                             old time/op    new time/op    delta
Large/memdb-1000000-100-16-40/query-no-in-tree-guarantee-16        11.6µs ±18%     2.1µs ± 6%  -81.86%  (p=0.008 n=5+5)
Large/memdb-1000000-100-16-40/query-hits-16                        13.0µs ± 3%     1.5µs ±21%  -88.06%  (p=0.008 n=5+5)
Large/memdb-1000000-100-16-40/iteration-16                          7.05s ± 9%     0.52s ± 2%  -92.67%  (p=0.008 n=5+5)
Large/memdb-1000000-100-16-40/update-16                             220µs ±10%     218µs ± 9%     ~     (p=0.841 n=5+5)
Large/memdb-1000000-100-16-40/block-16                             23.0ms ± 3%    24.3ms ± 2%   +5.28%  (p=0.016 n=5+5)
Large/goleveldb-1000000-100-16-40/query-no-in-tree-guarantee-16    21.5µs ±11%     5.7µs ± 2%  -73.60%  (p=0.008 n=5+5)
Large/goleveldb-1000000-100-16-40/query-hits-16                    21.8µs ± 5%     2.7µs ±28%  -87.75%  (p=0.008 n=5+5)
Large/goleveldb-1000000-100-16-40/iteration-16                      16.6s ±11%      0.5s ± 1%  -96.81%  (p=0.008 n=5+5)
Large/goleveldb-1000000-100-16-40/update-16                         245µs ±11%     269µs ±10%     ~     (p=0.222 n=5+5)
Large/goleveldb-1000000-100-16-40/block-16                         28.8ms ± 9%    31.8ms ±19%     ~     (p=0.151 n=5+5)

name                                                             old alloc/op   new alloc/op   delta
Large/memdb-1000000-100-16-40/query-no-in-tree-guarantee-16          566B ±37%       88B ± 0%  -84.45%  (p=0.008 n=5+5)
Large/memdb-1000000-100-16-40/query-hits-16                          606B ± 0%       67B ±46%  -88.91%  (p=0.008 n=5+5)
Large/memdb-1000000-100-16-40/iteration-16                          888MB ± 0%     184MB ± 0%  -79.28%  (p=0.016 n=4+5)
Large/memdb-1000000-100-16-40/update-16                            36.7kB ± 3%    37.2kB ± 3%     ~     (p=0.222 n=5+5)
Large/memdb-1000000-100-16-40/block-16                             3.80MB ± 1%    3.86MB ± 1%   +1.62%  (p=0.008 n=5+5)
Large/goleveldb-1000000-100-16-40/query-no-in-tree-guarantee-16    2.72kB ±29%    0.99kB ± 0%  -63.51%  (p=0.008 n=5+5)
Large/goleveldb-1000000-100-16-40/query-hits-16                    2.70kB ± 1%    0.34kB ±46%  -87.34%  (p=0.008 n=5+5)
Large/goleveldb-1000000-100-16-40/iteration-16                     3.71GB ± 0%    0.30GB ± 0%  -92.00%  (p=0.016 n=4+5)
Large/goleveldb-1000000-100-16-40/update-16                        79.8kB ± 6%    84.1kB ±10%     ~     (p=0.421 n=5+5)
Large/goleveldb-1000000-100-16-40/block-16                         9.05MB ± 9%    9.82MB ±16%     ~     (p=0.095 n=5+5)

name                                                             old allocs/op  new allocs/op  delta
Large/memdb-1000000-100-16-40/query-no-in-tree-guarantee-16          10.8 ±39%       3.0 ± 0%  -72.22%  (p=0.008 n=5+5)
Large/memdb-1000000-100-16-40/query-hits-16                          11.0 ± 0%       1.0 ± 0%  -90.91%  (p=0.000 n=5+4)
Large/memdb-1000000-100-16-40/iteration-16                          16.0M ± 0%      3.0M ± 0%  -81.25%  (p=0.016 n=4+5)
Large/memdb-1000000-100-16-40/update-16                               466 ± 8%       459 ± 8%     ~     (p=1.000 n=5+5)
Large/memdb-1000000-100-16-40/block-16                              52.0k ± 0%     53.2k ± 0%   +2.29%  (p=0.008 n=5+5)
Large/goleveldb-1000000-100-16-40/query-no-in-tree-guarantee-16      37.2 ±29%      19.0 ± 0%  -48.92%  (p=0.008 n=5+5)
Large/goleveldb-1000000-100-16-40/query-hits-16                      36.0 ± 0%       6.0 ±50%  -83.33%  (p=0.008 n=5+5)
Large/goleveldb-1000000-100-16-40/iteration-16                      49.8M ± 0%      5.3M ± 0%  -89.34%  (p=0.016 n=4+5)
Large/goleveldb-1000000-100-16-40/update-16                           684 ± 6%       729 ±12%     ~     (p=0.310 n=5+5)
Large/goleveldb-1000000-100-16-40/block-16                          81.5k ± 7%     84.8k ±12%     ~     (p=0.421 n=5+5)

Since the API has changes and bench tests were rewritten between master and the current branch, I had to create a custom version of bench_test.go that works for both branches.

@robert-zaremba I will do a new one in the next few days, sorry for the delay. However, here's the latest one.

hi, why modify  mtx from sync.RWMutex to sync.Mutex type . I think the sync.RWMutex should also work

@lyh169 there was a benchmark suggesting that. (normal Mutex was faster)

wow, I've just realized that I had not submitted comments.


[Bug]: Accepting any number of rows, columns and players.

Give a brief about the bug ✍
When attempting to set the number of rows, columns, and players on the homepage, inputs outside the acceptable range are currently being accepted.

What is the expected behavior? 🤔
Users should receive a validation message when attempting to input values for rows, columns, and number of players outside the acceptable range. This message should prompt them to provide values within the specified range.
Provide step by step information reproduce the bug 📄

Open the application and navigate to the homepage.
Locate the input fields for setting the number of rows, columns, and players.
Input values for rows, columns, and players that are outside the specified acceptable range.
Submit the input by clicking the appropriate button or triggering the submission action.
Observe the application's response. Note that there is no validation message displayed despite the inputs being outside the acceptable range.
Verify that the submitted values are accepted by the application without any indication of error or constraint violation.
Attempt to proceed with the application flow or perform actions dependent on these inputs to observe any unexpected behavior or errors that may occur as a result.

Select program in which you are contributing
GSSoC24
Code of Conduct

[X] I follow CONTRIBUTING GUIDELINE of this project.

I want to solve this Issue Under GSSOC, Kindly Assign me this Issue.

Hey @Lovelyfin00 could you assign me this. I want to fix it under GSSoC'24.

Hi @adityakishore25
Can you give a title to your issue?
I'll assign it to you now

Hey, can you please assign me this issue if it is not solved? This is my first time making an open-source contribution, and I really want to showcase my skills. I am very familiar with languages like Java, HTML, CSS, and JS. Apart from coding, I'm an ux/ui freelancer as well.

Hey @adityakishore25 can you assign this issue to  me , I'm a beginner in this field and i would love to explore and work on this issue

Hi @adityakishore25 Can you give a title to your issue? I'll assign it to you now
I have just created a PR, could you review it.

Hey @Lovelyfin00 , I have just created a PR, could you review it.

If you have any doubt you can directly send message on our discord (https://discord.gg/rZb46cCMmK), we have created separate channels for all projects...
Thank you :)

Hi, I am a beginner and would love to work on this issue. Please assign this issue to me

Hi, I am a beginner and would love to work on this issue. Please assign this issue to me

This issue has already been solved.


Allow schema renaming for internal state tables

Problem
We have a logic app with SQL trigger and a function with a SQL trigger running against the same database. Both use the internal state tables. The problem is that they seem to be having different requirements regarding the schema. The SQL trigger on the function adds a not nullable LastAccessTime column which is not used by the SQL trigger on the logic app, making it fail. We now manually changed the column to be nullable. However that's not something that you'd want to do.
Solution
A solution to this problem would be to allow for a configurable schema name on the SQL trigger for functions.
Describe alternatives you've considered
We mitigated the problem by making the LastAccessTime column nullable.

Thanks @sylvaingirardbe. We've put this request to add configuration for schema names on our backlog.
As for why there was a mismatch for the LastAccessTime column between SQL Trigger in Logic Apps vs. SQL Trigger in Function Apps, that's because the version of the SQL Trigger used in Logic Apps is slightly older, and an update to the latest version of the SQL Trigger is in progress by the Logic Apps team. I don't have an ETA available at this time other than to say that it's in progress - deployments are done by the Logic Apps folks themselves.


HADOOP_HOME is deprecated.

Hello!
I was trying out snakebite on my hadoop cluster and it seems that snakebite still uses HADOOP_HOME environment variable to check things. To my knowledge, this had been deprecated lately and most new users will not have this in their config files.
I can write a patch and submit a PR that can fix this to check on HADOOP_INSTALL or HADOOP_PREFIX (both in most cases point to the same path, usually /usr/local/hadoop)

Would be great if you write a patch (+tests) for this!

HADOOP_HOME is no longer deprecated..

Thanks for pointing that out @ehiggs


Make logic in the calendarHelper less convoluted

Return a flat array instead of a pre-generated grid of dates, and use limitTo and a custom offset filter to handle chunking up dates into rows for the month and year view. This would also be a good time to add unit tests.

Is there a specific part that you'd like to refactor?  Most of the pieces are so straight-forward that they don't even need comments.
  They're just doing a bit much for one function, I would think.  Maybe some common elements could be broken out, or perhaps part of it could be moved to controller logic for these views?
What did you have in mind?

This is now done and the code base has become a lot simpler because of it :)


Add ability to populate fields on media

Thanks to @dannylamb's patience on IRC, I got this working:
curl -v -u admin:islandora -H "Content-Type: application/json" -X PATCH --data-binary @my.json "http://localhost:8000/media/17?_format=json"
where my.json is
{"bundle": [{"target_id":"web_content","target_type":"media_bundle"}], "field_premis_event": [{"value":"Hi there."}]}

The syntax for this has probably been obsoleted.


remove bash start

What this PR does / why we need it:
remove $ from all bash commands so that the copy function can be used.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Release note:

@mandelsoft , I got the intention 😊. But it leads to not being able to copy and paste the commands easily without modifying them. That the commands are in a shell should be visible when using github or an advanced reader. For the cases we had input and output in one shell, I separated those into to fields.


[Editor] Change the cursor icons

/botio-linux preview

Given that the new ink-icon looks like a definite improvement, and should also be the "most" affected one here, do we perhaps want to land that one right away so that it makes the next release (this weekend)?

/botio-linux preview

Unfortunately the cursor-editorFreeText.svg icon looks quite "fuzzy" on a non-HiDPI screen.

/botio-linux preview


How to connect two hiveserver？

Now I hive two hiveserver to connect hive, and How to connect two hiveserver ？

I use the vip to resolve this issue


Initial implementation of mjpeg camera

Uh oh! Will figure out whats going on here...

Hmmm, Ill try to see if I can figure out how to get the raw stream out; tried with stream=True and looping over .iter_lines, but it doesn't seem to be getting the right data.

Don't worry about the test coverage, that's my fault and is something in the dev branch.

This is the initial documentation for this platform. Feel free to add more details, comments, or hints to simplify the usage for the users. Thanks.


Issue #166 : Change Posting Prompts to be a Clearer Step by Step

There are inline styles in the
 tag, those can be removed if required. Also, text for step 1 can be changed if any of the devs think its needed.

I removed the
 tag as the markup wasn't w3c compliant. Instead, I used styling to separate text from link.


Optimize the round-robin load balance strategy and use it as the default

Motivation

Currently, the computation complexity of round-robin partition placement strategy is O(n), we can optimize it to O(1).
Currently, the default partition placement strategy is random whose load balance effect is not stable sometimes, especially when the number of data partitions is small.

Changes
To achieve the goal, we can make PartitionPlacementStrategy aware of the shuffle worker list and do the selection accordingly.
Test

Unit test.
Test manually on a cluster.

Resolved.


parameters.py call to txfunc.py?

Rather than reading in a pickle, should parameters.py call txfunc.py?  The benefit is that we won't be messing around with pickled data. The cost is that we might sometimes want to save the time it takes to estimate the tax functions.
Perhaps we need a flag that decides to call the function or read a pickle, depending upon how this model run differs from the baseline.

We know do this, including the flag.


Add Python Version badge

Because why not?

I think we can add a few more: https://forthebadge.com/

I still need to merge that other PR. Y'know, the one that actually does something. Maybe I'll do that tonight.


docs: adding more documenation about the project

Codecov Report

Merging #18 (6088b62) into main (d616794) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main      #18   +/-   ##
=======================================
  Coverage   76.37%   76.37%
=======================================
  Files           3        3
  Lines         127      127
=======================================
  Hits           97       97
  Misses         19       19
  Partials       11       11

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d616794...6088b62. Read the comment docs.


No exception raised when assign_partitions! fails

deliver_messages silently (only logs an error) fails when assign_partitions! doesn't succeed.
https://github.com/zendesk/ruby-kafka/blob/cd406829bad3b3d8f5e9bff0bb87eb3c72c67e0d/lib/kafka/producer.rb#L334-L337
https://github.com/zendesk/ruby-kafka/blob/cd406829bad3b3d8f5e9bff0bb87eb3c72c67e0d/lib/kafka/producer.rb#L310-L314
Messages still in @pending_message_queue might get lost after deliver_messages returns.

Can you reproduce message loss? @pending_message_queue was built to be resistant to message loss: https://github.com/zendesk/ruby-kafka/blob/master/lib/kafka/pending_message_queue.rb
Basically, we need to accept that exceptions will happen as part of the normal operation – nodes will come and go. The producer should handle most of this: it's not just a thin wrapper around a Kafka produce request, it's a stateful buffer of messages.


[DOC] Explain common WASM errors

Is your feature request related to a problem? Please describe.
Common WASM errors are not always clear, we should have a list of them with possible explanations.
Describe the solution you'd like
For example for:
Failure(ActionError(ActionError { index: Some(0), kind: FunctionCallError(HostError(MemoryAccessViolation)) }))

Possible explanaitions:

index out of bounds for memory access.
also happens when stuff does not fit into contract memory.
also happens when we are trying to fit stuff which is too large into the register. It can be thrown in many places so it would be helpful to know which line of code actually throws it.

Another:
 { FunctionCallError: { WasmTrap: { msg: 'unknown' } } }

see: https://github.com/nearprotocol/nearcore/issues/2529

we have a list of errors but no accessible explanation that ties things back to the dApp developer perspective so this would be really helpful
https://docs.nearprotocol.com/docs/roles/integrator/errors/introduction


Uncaught (in promise) cancelled

Seems if I destroy a worker after scheduling a render task, something in rendering promise triggers a cancelled error which I can't catch. (I've tried to intercept render(...).promise.catch, so I get cancelled, but error still occurs).
cc @yurydelendik

Can you provide a test case? I'm thinking it's possible to reproduce that with just calling cancel on the render task object.

I think you can still use my previous example, I've updated it to throw this error (added .then() to render function). Take a big PDF and wait pdf.js to start processing it, then cancel.

Hmm, I was not able to reproduce that so far -- probably I'm using wrong PDF or timing is bad. (Ability to be able to reproduce will help quickly to fix the issue)

You can use this one: http://www.nasa.gov/pdf/750614main_NASA_FY_2014_Budget_Estimates-508.pdf to reproduce the issue. Right after you select it, press cancel and check developer console

I've sent you a PDF in the email, @yurydelendik

Just tested PDF I've sent you in FF, got this:

In FF I have this error message with a few seconds timeout

I see that when using your document, but it's expected. The render promise must be resolved to something (and it shall not be a success)
To avoid breaking in the debugger tools or console message, please use catch e.g.:
       page.render({
          canvasContext: canvas.getContext('2d'),
          viewport: page.getViewport(scale),
        }).promise.then(function() {
          console.log('finished');
        }, function (reason) {
          console.log('stopped ' + reason);
        });

or
       page.render({
          canvasContext: canvas.getContext('2d'),
          viewport: page.getViewport(scale),
        }).promise.catch(function (reason) {
          console.log('stopped ' + reason);
        });

I hope this helps.

Thanks a lot!


Octopath Traveler 1.0.4 [NS]

The current script seems not working with ver. 1.0.4 (OCTOPATH TRAVELER [01000E200DC58000] [v131072] (1G+1U)).
Please check it.
Thanks.

Pick the correct version the script is made for (1.0.0)


dev: move chain_id into the config rather than the storage

The chain_id is currently stored in the storage
    #[pallet::storage]
    #[pallet::getter(fn chain_id)]
    pub(super) type ChainId<T: Config> = StorageValue<_, Felt252Wrapper, ValueQuery>;

Meaning that, in theory, this value could be changed at any time. It would be more coherent to have it as a const declared in the pallet Config.

i can work on this

@lambda-0x please go ahead!


Adding GZIP support #74

Addresses: https://github.com/edjafarov/node-webkit-updater/issues/74

I'm sorry no one has gotten back to you on this by now :/. Do you mind resolving the conflicts? I'll need to do some careful manual tests then 😃


Added new region southcentralusstg to Azure staticRegionToZones

Change Overview
Added new region southcentralusstg to Azure staticRegionToZones
Pull request type
Please check the type of change your PR introduces:

[ ] :construction: Work in Progress
[ ] :rainbow: Refactoring (no functional changes, no api changes)
[x] :hamster: Trivial/Minor
[ ] :bug: Bugfix
[ ] :sunflower: Feature
[ ] :world_map: Documentation
[ ] :robot: Test

Issues
Test Plan

[ ] :muscle: Manual
[ ] :zap: Unit test
[ ] :green_heart: E2E

@TribalNightOwl - Is this ready to be kueued ?


DM: Home page - where you're signed in - compatibility sessions

Depends on #1499
compatibility sessions are displayed under browser sessions on the home page
AC:

Subheader for compatibility sessions
Latest 5 sessions are displayed
A 'View all' button follows the list and navigates to /account/compat-sessions

Superseded by https://github.com/matrix-org/matrix-authentication-service/issues/1546


Added list of false positiitives

Issue: (cpp) Keywords followed by parens are treated as built_in

Resolves #3980
Changes

Added list of false positives to FUNCTION_DISPATCH
The List:
      /(?!decltype)/,
      /(?!if)/,
      /(?!for)/,
      /(?!switch)/,
      /(?!while)/,
      /(?!alignas)/,
      /(?!alignof)/,
      /(?!asm)/,
      /(?!catch)/,
      /(?!const_cast)/,
      /(?!dynamic_cast)/,
      /(?!noexcept)/,
      /(?!reinterpret_cast)/,
      /(?!sizeof)/,
      /(?!static_assert)/,
      /(?!static_cast)/,
      /(?!typeid)/,
      /(?!requires)/,
      /(?!explicit)/,
      /(?!case)/,
      /(?!delete)/,

      // compound operators
      /(?!bitand_eq)/,
      /(?!bitor_eq)/,
      /(?!xor_eq)/,
      /(?!not_eq)/,
      /(?!or_eq)/,
      /(?!and_eq)/,

      // operators
      /(?!and)/,
      /(?!or)/,
      /(?!bitand)/,
      /(?!bitor)/,
      /(?!xor)/,

      // unary operators
      /(?!not)/,
      /(?!compl)/,
      /(?!co_await)/,
      /(?!co_return)/,
      /(?!co_yield)/,

      // Reserved types
      /(?!int)/,
      /(?!char)/,
      /(?!double)/,
      /(?!float)/,
      /(?!bool)/,
      /(?!auto)/,

Before:

After:

Checklist

[x] Added markup tests, or they don't apply here because...
[x] Updated the changelog at CHANGES.md

Keywords followed by parens are treated as built_in

Could we build the exclude list dynamically from our keywords list then instead of repeating ourselves?

Keywords followed by parens are treated as built_in

Could we build the exclude list dynamically from our keywords list then instead of repeating ourselves?

Sure

Its working fine now.

hi @joshgoebel, can you please tell me if our engine has a priority based highlighting or it is just limited to regex. if no such feature exists, should we add this feature to our engine for better control over it?


newlib: constrain to non-hosted environments

The build failure was exposed by commit cf1b7c4d5 "newlib: fix build of nano variant on non-ARM architectures" where an assertion was added when $out is an empty directory. On x86_64-linux (hosted) newlib attribute failed as: https://hydra.nixos.org/log/nv0q296sc06achvd7ljlrsn8x3qh8fg1-newlib-<IP_ADDRESS>30120.drv
Let's mark newlib as broken on hosted (non-embedded, no-bare-metal) targets.
ZHF: https://github.com/NixOS/nixpkgs/issues/265948
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

failed as: https://hydra.nixos.org/log/nv0q296sc06achvd7ljlrsn8x3qh8fg1-newlib-<IP_ADDRESS>30120.drv

I can't view this right now because

Looks like Hydra is having some problems. Sorry about that!

but if it's

an assertion was added when $out is an empty directory

can't we just remove the assertion?
Marking newlib as broken for almost-every platform seems like quite a sledgehammer.
When hydra is back online I'll fix whatever platforms it's broken on.

https://github.com/NixOS/nixpkgs/pull/266364


Calibration matrix from Carla´s RGB camera

Hello,
I am working with Carla's RGB camera and processing the data with Matlab. But, in order to calibrate the RGB camera in Matlab I need to complete the intrinsic matrix and the distortion matrix.
For the intrinsic matrix or calibration matrix I would need to obtain the focal lengths (fx,fy) of the camera in the x and y axis (parameters that are measured in pixels) and the displacement of the centre (cx,cy) of the image in x and y axis.
In the distortion matrix, both the radial and the tangential distortions appear, so for both cases, it would be necessary to have 2 of the distortion coefficients (K1,k2 for the radial distortion and p1,p2 for the tangential distortion) and the coordinates of the distorted pixel in x and y (xd,yd for the radial distortion and xd,yd for the tangential distortion)
The data regarding Carla's RGB camera can be found in the following link:
https://carla.readthedocs.io/en/0.9.9/ref_sensors/#rgb-camera
Then, is anyone able to tell me how I can obtain the above mentioned data from the camera datasheet of the previous link?
Thank you very much in advance.

Hi,
As far as I remember, any lens distortion is achieved by setting 'enable_postprocess_effects' to True, i.e there is no distortion by default.
All the other intrinsic parameters are just those of an ideal camera, that is (cx, cy) are the middle of the image and fx=fy=f where f can be calculated from the hfov and image width and there is no skew.
Hope that helps,
Gabi

Thank you very much for your contribution.

Hi Gabi,
I'm processing the images with the data you told me about, that is, with the null distortion matrix (parameters that I don't know if they are right or should be different from zero). But the data I'm getting is wrong or skips errors. Then, you could tell me if the data that appears in the section "Camera lens distortion attributes" has any meaning for the distortion matrix? I have read by some discussion that the values that appear in that section could be K1 (lens_k) or K2 (lens_kcube) which would be the values for the radial distortion, but for the tangential distortion I have not found anything about it. Although I am not sure about it, so if you knew or someone knew about it, it would be very helpful.
Thank you very much in advance.

Hi,
Are you sure that you are actually seeing distortion, rather than facing a different issue?
Do straight lines look curved in your images, for example?
Gabi

@DSantosO can you follow up on this please?

Hello @indartsu,
If you have not included distortion, the camera matrix should be just the one computed with fx,fy,cx,cy. Be careful with the frame transformation between Unreal's XYZ and the camera one that can be tricky. You can check the lidar_to_camera.py script in which this matrix is used to project points from 3D space to the camera image.


use fastdom to avoid unnecessary document reflows

CodeMirror and codemirror.next can use fastdom to avoid unnecessary document reflows when possible, like below function (updateGutterSpace)

After changing updateGutterSpace to use fastdom, the performance is far more better:

We have our own, similar (but synchronous) solution for this (see operations, specifically the endOperation code)


ACM Paper - CCSXML section

The doc issue
There is a section called CCSXML in the paper template:
%%
%% The code below is generated by the tool at http://dl.acm.org/ccs.cfm.
%% Please copy and paste the code instead of the example below.
%%
\begin{CCSXML}
...

It wants us to classify our paper using an online tool. I don't know how to classify the paper exatly.
Suggest a potential alternative/fix
I am going to search for word embeddings in the ACM digital library and look at the classifications there. I will classify our paper with similar topics.

I think this issue is completed.


Flutter 3.22 support

PR Description:

Adds the necessary changes to support plugin for Flutter 3.22.

Hi @gtalha07 thank you!


Context Menus going off the edge of the screen

Context menus, if used by a UI element near the bottom or right edge of the screen, can go off the edge and leave some of its selections either partially unreadable, or not visible or selectable on the whole, context menus should go upwards and be offset horizontally from the click so as to not go off the edge of the screen

This actually needs some tech to figure out the UI-relative screen size
(Starbound doesn't provide a way to get at it, so we need to do some
anchor-and-canvas trickery to check)
note to self: do this on pane opening!
On Sat, Jun 11, 2022, 1:50 AM WasabiRaptor @.***> wrote:

Context menus, if used by a UI element near the bottom or right edge of
the screen, can go off the edge and leave some of its selections either
partially unreadable, or not visible or selectable on the whole, context
menus should go upwards and be offset horizontally from the click so as to
not go off the edge of the screen
—
Reply to this email directly, view it on GitHub
https://github.com/zetaPRIME/sb.StardustSuite/issues/28, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABS5V6VASYPN7EJE5PA26TTVOQSKHANCNFSM5YPRO22A
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

as this is mainly a problem with UIs that are anchored to the bottom and right corner of the screen, for those ones I don't believe it should be too hard to do some math based off of the canvas click position, knowing the window size and anchor offsets and all


Fix swift-tools specification

The new line is necessary, otherwise SPM doesn't seem to be able to read it correctly

Thanks


Displaying Taxonomies in frontend

Fixes #255
Todo:

[x] Make links to taxonomy listings
[x] Make taxonomy listings work
[x] Add tests
[x] Make fixtures add taxonomies

Should be GTG now! 🍺

Finally green lights!

Add unit test/spec for taxonomy filter

I still need to make time someday to get started with PhpSpec, someday. :-/

I've added an issue for this: #371

Please don't add tests in separate PRs/issues. I can't accept PR without it being tested.

OK I will accept this PR by way of exception but please don't forget about tests.


Add elimination and roundrobin shortcuts

elim and rr require less typing and I get lazy some days.
Only reason really. Can refuse.

We have /mt [tier] as a shortcut to making elimination tournaments, and we hardly do rr tours, so I don't think this is needed.


Fix datastore: unprefixed ids in client

If datastore id comes back prefixed from the API, and Client.datastore_id is not prefixed, the 'Keys do not match dataset ID' exception is raised.
Fixed to use _dataset_ids_equal for the comparison instead, which compares ignoring valid prefixes and seems to be the preferred comparison method used elsewhere in the code.

@mpeg Thanks for bringing this up. I'm trying to understand what this fixes.
Can you provide a stacktrace when that ValueError was raised previously?

@dhermes Sure ! It's my first patch so wasn't sure how much detail to give
For the dataset Client class, I've got the dataset_id set from an env variable to mpeg-personal (my real dataset name)
Here's a REPL and stack trace highlighting the way I encountered this issue:
>>> from gcloud import datastore
>>> gcd = datastore.Client(namespace='namespace')
>>> existing_key = gcd.get(gcd.key('Test',<PHONE_NUMBER>213120)).key
>>> existing_key
<Key[{'id':<PHONE_NUMBER>213120, 'kind': 'Test'}], dataset=e~mpeg-personal>
>>> gcd.get(existing_key)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/usr/local/lib/python3.4/dist-packages/gcloud/datastore/client.py", line 263, in get
 deferred=deferred)
 File "/usr/local/lib/python3.4/dist-packages/gcloud/datastore/client.py", line 293, in get_multi
 raise ValueError('Keys do not match dataset ID')
ValueError: Keys do not match dataset ID

It all comes down to that string comparison done in the lines my patch changes, which does a comparison ignoring dataset prefixes (as the API will accept an unprefixed dataset_id).
The unexpected behaviour for me was that I couldn't store the key that came back and reuse it.

Really great stacktrace, reproduced on my end! Thanks. Hopefully this will all be mitigated with v1beta3 but for now this fix is needed. Will comment on PR.

So the only real issue with this is that we don't have a unit test to make sure the failure doesn't occur again. I can help with that if you like or you can give it a spin yourself?
I really appreciate the contribution.

Sure, I can add the unit test, I'm thinking to the gcloud.datastore.test_client tests, add a test that gets a mock entity with a prefixed dataset_id ?

You could style it after
https://github.com/GoogleCloudPlatform/gcloud-python/blob/6692df26f262ee6298629decb278fd02eeae727e/gcloud/datastore/test_client.py#L342-L351

Added a test, styled after the test_get_multi_hit_multiple_keys_same_dataset and test_get_multi_hit_multiple_keys_different_dataset tests, as it's an extension of those, makes sure:

dataset ids that are prefixed with "e~" or "s~" don't raise the ValueError exception
prefixed dataset ids actually get the correct entity values (mostly in case of the mock classes changing in the future)

Really great test thanks! I made some comments but overall it is about perfect (which is not something you expect from a first contrib).

It just occurred to me that it'd be smart to have another similar test that does fail when the IDs disagree, e.g.
dataset_id1 = 'e~foo'
dataset_id2 = 's~bar'

Yeah I agree with your style comments (sorry about the pep8 failure-to-indent), didn't add a failing test like that since it's largely covered by test_get_multi_hit_multiple_keys_different_dataset but perhaps would be good in case it all changes in the future and the issue comes back.

Should all be good now, added the extra test case for e~foo and s~bar too.
Thanks for the help, hopefully first of many

LGTM
Thanks again!


OK，i run the command minifab up, it meets error with the step of cc install,

my env is Linux Ubuntu20.04.1

i think the error is about go proxy, but i can  access to google site, so what i should do to solve this problem?

i run this command go env -w GOPROXY=https://goproxy.cn to change go proxy ,but it still doesn't work.

@zwnode look at this file,https://github.com/hyperledger-labs/minifabric/blob/main/spec.yaml, pay attention to the line 22, to line 25 specifically for this kind of issue. I think you need to make a copy of this file and place it in your current directory, then the settings in the spec.yaml file will allow you to use different proxy, then you should be fine.

@zwnode look at this file,https://github.com/hyperledger-labs/minifabric/blob/main/spec.yaml, pay attention to the line 22, to line 25 specifically for this kind of issue. I think you need to make a copy of this file and place it in your current directory, then the settings in the spec.yaml file will allow you to use different proxy, then you should be fine.

it still doesn't work ,the file is already in the directory when i colne minifab, and i run the command go env ,the proxy is https://goproxy.io,direct ,    Is there anything else I need to modify?

@zwnode look at this file,https://github.com/hyperledger-labs/minifabric/blob/main/spec.yaml, pay attention to the line 22, to line 25 specifically for this kind of issue. I think you need to make a copy of this file and place it in your current directory, then the settings in the spec.yaml file will allow you to use different proxy, then you should be fine.

it still doesn't work ,the file is already in the directory when i colne minifab, and i run the command go env ,the proxy is https://goproxy.io,direct , Is there anything else I need to modify?

@zwnode as I said you need to pay attention to the lines that I said above. you are not using the right proxy. Try to use this one goproxy.cn, not goproxy.io

whatever go env you set up in your own shell does not matter, minifabric does not use any of these. Please change settings  in spec.yaml file. That is the only place that you can change settings. Please also read docs in the docs directory, it will be a great help for you.

whatever go env you set up in your own shell does not matter, minifabric does not use any of these. Please change settings in spec.yaml file. That is the only place that you can change settings. Please also read docs in the docs directory, it will be a great help for you.

can you tell me how to change settings, i trid but failed, i'm a new hand.

whatever go env you set up in your own shell does not matter, minifabric does not use any of these. Please change settings in spec.yaml file. That is the only place that you can change settings. Please also read docs in the docs directory, it will be a great help for you.

can you tell me how to change settings, i trid but failed, i'm a new hand.

Is there any problems?

@zwnode try place the following content in spec.yaml file and place the file in your working directory, then make sure your env is clean. then try minifab up
fabric:
  cas:
  - "ca1.org0.example.com"
  - "ca1.org1.example.com"
  peers:
  - "peer1.org0.example.com"
  - "peer2.org0.example.com"
  - "peer1.org1.example.com"
  - "peer2.org1.example.com"
  orderers:
  - "orderer1.example.com"
  - "orderer2.example.com"
  - "orderer3.example.com"
  settings:
    ca:
      FABRIC_LOGGING_SPEC: DEBUG
    peer:
      FABRIC_LOGGING_SPEC: DEBUG
    orderer:
      FABRIC_LOGGING_SPEC: DEBUG
  goproxy: "https://goproxy.cn,direct"

@zwnode try place the following content in spec.yaml file and place the file in your working directory, then make sure your env is clean. then try minifab up
fabric:
  cas:
  - "ca1.org0.example.com"
  - "ca1.org1.example.com"
  peers:
  - "peer1.org0.example.com"
  - "peer2.org0.example.com"
  - "peer1.org1.example.com"
  - "peer2.org1.example.com"
  orderers:
  - "orderer1.example.com"
  - "orderer2.example.com"
  - "orderer3.example.com"
  goproxy: "https://goproxy.cn,direct

I do as you said ,it run sucessgul Thank you very much ,it mean a lot to me

@zwnode try place the following content in spec.yaml file and place the file in your working directory, then make sure your env is clean. then try minifab up
fabric:
  cas:
  - "ca1.org0.example.com"
  - "ca1.org1.example.com"
  peers:
  - "peer1.org0.example.com"
  - "peer2.org0.example.com"
  - "peer1.org1.example.com"
  - "peer2.org1.example.com"
  orderers:
  - "orderer1.example.com"
  - "orderer2.example.com"
  - "orderer3.example.com"
  goproxy: "https://goproxy.cn,direct"

i do as you said and it run successfully. Thank you for your help,it mean a lot to me .


Enable retries for api calls

requests (via urllib3) allows setting retries in cases of connection errors.  We have recently had a problem with a ConnectionError while sending a large message body that on later attempts worked, so it would be helpful to make retries accessible to render-python on some level.

Now that I look into it more, it seems like the ability to specify a session (which I think all of the api calls have) kind of handles this.  while it might be nice to make it easier to access, this should not be high priority if this fixes the issues.

This can be implemented for any render webservice call with:
s = requests.Session()
s.mount('http://', requests.adapters.HTTPAdapter(max_retries=5))

render_func(myinput, session=s)

Closing this without change -- session is exposed enough for now that it's not a big barrier to running things.


fix typo on getting_started.rst

Previous sentence: "The example hereunder shows a position-controlled robot ..."
New sentence: "The example here shows a position-controlled robot ..."

I'm not a native speaker, but this word exists and makes sense in the context?

I'm not a native speaker, but this word exists and makes sense in the context?

Actually exists. I was not aware of its existence till you ask. It is less common, though. I'm not a native english speaker also.


adding example terraform config for aws iam resources

This is an example configuration of terraforming IAM resources in AWS.
Overview:

📁 terraform

📁  aws-custom-policies - JSON configurations for customer-managed policies (AWS-managed policies are referenced by ARN and not needed here). Permission levels 2, 3, & 4 need policies created according to this discussion

📄 level-2-project-admin-policy.json
📄 level-3-ops-mentor-policy.json
📄 level-4-iam-services-admin-policy.json
📁 existing-policies - a few of our current policy configurations for reference

📁 modules - reusable Terraform configurations
📄 aws-custom-policies.tf - maintain custom policies here
📄 aws-groups.tf - maintain groups here
📄 aws-users.tf - maintain users here

Note: if you are new to Terraform, the three .tf files above are where the resources are maintained and should be the only files that need changes for maintaining policies, groups, and users. You add a group in aws-groups.tf, a user in aws-users.tf, etc. These files essentially use preconfigured modules to create resources given provided input variables.
Feel free to ask questions in the comments. If you're new to pull requests, you can open the files changed tab, navigate to the section you have a question about, and comment directly on the line(s) of code or add comments on the conversation tab. 😺
View the branch

This is not a working solution but should only need minor adjustments/additions to get it working. There are a few limitations right now with this being a private repo. We can't designate rules or branch protections without upgrading from the free plan...so that is a discussion we will need to have. The main reason it's currently private is because it exposes account names and users' first and last names. There are differing views on whether or not first and last names alone constitute PII. Under the GDPR (we fall under w/ global membership), one would be able to search something like 'first name last name hack for la' and identify a person on LinkedIn...so it would be classified as personal data

Overall, a really nice skeleton. I'd love to see policies filled in and this started to be applied ASAP.
As a separate issue, modernizing existing config should get recorded, so that the manual references to policies or users that already exist can be removed eventually. Not difficult, just a tedious process of finding users and terraform importing them.

Completely agree on all points. I was thinking that importing existing IAM resources would make good small first issues for members to work on after getting Terraform set up


feat: add Idle Portofino to BASE_OPPORTUNITIES_BY_ID

Description

Pull Request Type

[ ] :bug: Bug fix (Non-breaking Change: Fixes an issue)
[ ] :hammer_and_wrench: Chore (Non-breaking Change: Doc updates, pkg upgrades, typos, etc..)
[ ] :nail_care: New Feature (Breaking/Non-breaking Change)

Issue (if applicable)

closes #
Risk

Testing

Engineering

Operations

Screenshots (if applicable)

Current dependencies on/for this PR:

develop

PR #4347   👈

This comment was auto-generated by Graphite.


After continuously streaming for several days, the memory keeps increasing and cannot be reclaimed.

Note: Please read FAQ before file an issue, see #2716

Description
Please description your issue here

SRS Version: windows 5.0.170

SRS Log:

The log is normal

SRS Config:

listen 1935;
max_connections 1000;
daemon off;
srs_log_tank console;
stream_caster {
enabled on;
caster gb28181;
output rtmp://<IP_ADDRESS>/live/[stream];
listen 9000;
sip {
enabled on;
listen 5060;
# @see https://ossrs.net/lts/zh-cn/docs/v5/doc/gb28181#config-candidate
candidate $CANDIDATE;
}
}
http_server {
enabled on;
listen 8099;
dir ./objs/nginx/html;
}
http_api {
enabled on;
listen 1985;
}
rtc_server {
enabled on;
listen 8000; # UDP port
# @see https://ossrs.net/lts/zh-cn/docs/v4/doc/webrtc#config-candidate
candidate $CANDIDATE;
}
srt_server {
enabled on;
listen 10080; # UDP port
maxbw<PHONE_NUMBER>;
connect_timeout 4000;
latency 20;
peerlatency 20;
recvlatency 20;
}
vhost defaultVhost {
rtc {
enabled on;
# @see https://ossrs.net/lts/zh-cn/docs/v4/doc/webrtc#rtmp-to-rtc
rtmp_to_rtc on;
# @see https://ossrs.net/lts/zh-cn/docs/v4/doc/webrtc#rtc-to-rtmp
rtc_to_rtmp on;
}
http_remux {
enabled on;
mount [vhost]/[app]/[stream].flv;
}
hls {
enabled on;
hls_dispose     60;
hls_fragment 2;
hls_window 1800;
hls_ts_file [app]/[stream]/[seq]-[timestamp].ts;
hls_m3u8_file [app]/[stream]/[stream].m3u8;
}
http_hooks {
enabled         on;
on_publish      http://<IP_ADDRESS>:7071/controller/srs/onPublish;
on_unpublish      http://<IP_ADDRESS>:7071/controller/srs/onUnPublish;
on_hls             http://<IP_ADDRESS>:7071/controller/srs/onHls;
}
}
Replay
Please describe how to replay the bug?
Step 1: xxxxxx
xxxxxxxxxxxx
xxxxxxxxxxxx
xxxxxxxxxxxx

Step 2: xxxxxx
xxxxxxxxxxxx
xxxxxxxxxxxx
xxxxxxxxxxxx

Step 3: xxxxxx
xxxxxxxxxxxx
xxxxxxxxxxxx
xxxxxxxxxxxx

Expect
Please describe your expectation.
TRANS_BY_GPT4

这个出问题只出现在window系统，linux系统不会

测试了一周时间，好像是gb28181的问题

Seems memory leak on GB28181, which should be reproduced by a long time running.


reorganize the device packages by manufacturers

As we are adding more devices, it's getting hard to find them. For instance I organized the seedstudio devices under grove which was probably a mistake.
My suggestion is to use the manufacturer name such as seeedstudio, adafruit, pimorini etc..
Thoughts?

@rakyll @zankich what do you think?

Do you have any idea what to do with cases like dotstar's? Even though they are named by adafruit as dotstar, APA-102 is a common protocol and there are many manufacturers that sells a similar strip and you can use the dotstar package to run them.

That's really the core issue. GoBot chose to name the packages based on the chipset and it's a total PITA IMHO. You should be able to google "device you bought" + go and find the package. Most users don't know nor care what the microcontroller or chipset is called.
So I see a few options:

create a package per low level protocol and implement "meta packages" on top (or document what devices implement the protocol).
use popular package names: (folder adafruit, package dotstar) and document the compatibility/create alias packages.

I don't think there is a perfect solution, but I think we should first consider the user experience, especially the first time user experience. Let say I went to adafruit or seeedstudio and bought a bunch of devices, how do I find and start writing Go code for those. GoBot chose a certain pattern and I can NEVER remember the package to use for the LCD screen even though I wrote it: https://github.com/hybridgroup/gobot/blob/dev/platforms/i2c/jhd1313m1_driver.go
But at the same time, if I buy another APA-102 led strip that comes straight from China, I should be able to figure out how to use the package.

The second option seems to be the best. Anyone who understands/knows about the low-level protocol can already find the popular variant and use that package.
We can also make the READMEs more Google-able by mentioning the underlying protocol.

I think I agree about #2, we still need to agree on a structure or organize these packages and probably extend the template to make it easier for user to google underlying protocols etc..
@zankich @deadprogram what do you think?

Many of the devices sold by Adafruit, Seeedstudio, DFRobot, etc are the same chipsets in different packages/breakout boards. Hence separating them by manufacturer would not be best using these designations, you would need to use the OEM's designation.
In Gobot, we chose to organize by first protocol of device (GPIO, I2C), and then the actual OEM designation for the hardware chipset, to be able to accommodate the many different version sold by vendors. Example: the "BMP180" breakout boards sold by Adafruit and those sold by various other vendors all use the same Bosch BMP180.

Anything that is branded by a specific company should be placed in a sub folder under that manufactures name.
In situations where it's a "generic" board, which different manufactures mount the same exact sensor and have no specific branding around it, those could possibly go in a generic package and create meta packages under specific manufactures as we need?
Creating meta packages which exist under different manufactures sounds like it could work, but where does the real implementation live? Does that exist in the manufacturers package that we chose to implement first? I don't really like the idea of having each of these packages reach out into separate packages and then return types from a separate package. For instance a seedstudio.NewAccel would actually return an adafruit.Accel or whatever.
At least if there is some central package they all pull from, we won't have implementations scattered throughout the sub packages.

I don't really like the idea of having each of these packages reach out into separate packages and then return types from a separate package. For instance a seedstudio.NewAccel would actually return an adafruit.Accel or whatever.

We are actually never going to make brand names the proper packager names. They will be just there for the purpose of hierarchy and organization of the actual packages.
As far as I understand, dotstar will keep being its own package, just underneath a brand.

At least if there is some central package they all pull from, we won't have implementations scattered throughout the sub packages.

If there is a generic implementation required, we can put the implementation in an internal package and provide packages on top of the internal packages.

Hence separating them by manufacturer would not be best using these designations, you would need to use the OEM's designation.

This makes me why we are not going with a flat hierarchy. We don't have to organize by OEM. Let's only have one-level with user facing names. If two devices need to share the same implementation, we can make the implementation an internal package.

As far as I understand, dotstar will keep being its own package, just underneath a brand.

Agreed

If there is a generic implementation required, we can put the implementation in an internal package and provide packages on top of the internal packages.

I like this approach, we can also deal with it when it will present itself

why are we not going with a flat hierarchy?

This is a very good point :p It's my fault I wanted to nicely organize the packages so they are easier to find, I was also influenced by GoBot that organizes the adapters by device protocol. I think I really like having a flat hierarchy, it will allow for a better godoc experience.
So my vote is for:

flat hierarchy (don't group packages by brands, refer brands in the README.md files)
name packages after their marketing name (i.e: dotstar, piglow)
use internal packages when we need to share implementations for different devices sharing the same interface.
deal with conflicting package names when they will happen not before (i.e: accel3xdigital could potentially collide in the future)

I am in favor of what @mattetti has voted for. Dealing with conflicting names is rather hard though without injecting protocol or brand names into the package name. Do we have a better strategy?

Yeah for instance the accel3xdigital package if very much associated with grove branded naming

Just a reminder from Effective Go:

It's helpful if everyone using the package can use the same name to refer to its contents, which implies that the package name should be good: short, concise, evocative. By convention, packages are given lower case, single-word names; there should be no need for underscores or mixedCaps. Err on the side of brevity, since everyone using your package will be typing that name. And don't worry about collisions a priori. The package name is only the default name for imports; it need not be unique across all source code, and in the rare case of a collision the importing package can choose a different name to use locally. In any case, confusion is rare because the file name in the import determines just which package is being used.

Two seems things to be pertinent:

keep names concise
don't worry about collisions a priori

Adding the name of the OEM or some sort of abbreviation seems wrong to me. That's also why I had opted for some sort of folders structure initially. In practice, I'm not sure how often this issue will come up but I think it's worth dealing with it when it will come (feel free to blame them then ;))

So then branded things like grove could have its own sub folder and everything else can be flat?

No sub folders, everything flat and if we find conflicts we can deal with them then


lldpd service is broken (NixOS 18.09)

Issue description
On the nixpkgs master, the lldpd service will not run since upstream commit https://github.com/vincentbernat/lldpd/commit/b0ca585f627aad476604a11bd356b842c664fb84. It still works in NixOS 18.03.
Steps to reproduce
The option "services.lldpd.enable = true;" will result in the lldpd service failing to start due to "ExecStartPre=/bin/mkdir -p /var/run/lldpd (code=exit, status=203/EXEC)".
Technical details

system: "x86_64-linux"
host os: Linux 4.14.59, NixOS, 18.09.git.cc1d821 (Jellyfish)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.0.4

Pinging maintainer @fpletz.

This will be fixed upstream in the next lldpd release (after 1.0.1).

fix: https://github.com/vincentbernat/lldpd/commit/90a50860ebdcdeb5b7dc85790b18bed23c97ec32


[Feature request] Add a tab that allows you to navigate the runescape wiki

I think adding the runescape wiki to the client would allow many players to have a much more pleasant experience, for example, if they were stuck on a quest, they can simply click a single button to be directed to that quests guide on the wiki. This would also allow players to search the wikis great features like the high alch calculator, or the detailed skill training guides. I don't know why this feature wasn't implemented from the start!

This would have to open links in your default browser instead of having a separate 'tab' in order to keep the client lightweight. I do like the idea of being able to open a quest guide from the quest list, though!

A menu in the sidebar that is able to load the mobile version of the osrs wiki would be pretty cool.

There is PR that adds wiki lookup options to various things in game. I feel like having wiki straight in the sidebar is moderately useless and that lookup PR is a lot better approach.

You can checkout the PR Deathbeam mentioned here.

Adding browser to sidebar is bad idea, closing as wontfix


sxzz

So fast! Let me try this out locally before merging.

Could we introduce Netlify preview in each PR

Ready to merge now (please use rebase merge, instead of squash)


The README.md states that there are recompiled bitstreams available in the bin directory but the bin directory does not exist

The README.md states: "You can also find two pre-compiled example bitstreams in the bin/ directory of this repository." but the bin directory does not exist in this repository or in the submodule.

We updated the repository to include Bootloader, Signaloid Core, and a blink bitstream in the bin/ directory.


No response when user speaks for more than around 20s

currently building ai tutors using the new conversational ai react SDK. Been great so far until I found that if the user speaks continuously for more than around like 20-20s, the “listening” state will be stuck forever.  Any ideas for fix? Thanks.

Hey @louisjoecodes , any update here?
Perhaps it should be temporarily documented?


Empty tool calls leading to done condition

This PR brings PaperQAEnvironment to support empty tool calls leading to the environment being done. This creates a back door for the agent to trigger a rollout to conclude.

I ran some local testing and it seems if anything performance got better


fix(text input): add support for prefixes

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Julia Hempel seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

:tada: This PR is included in version 1.2.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


[WIP][DO NOT MERGE]add aks 20200101 api-version

Latest improvements:
MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Contribution checklist:

[x] I have reviewed the documentation for the workflow.
[ ] Validation tools were run on swagger spec(s) and have all been fixed in this PR.
[ ] The OpenAPI Hub was used for checking validation status and next steps.

ARM API Review Checklist

[ ] Service team MUST add the "WaitForARMFeedback" label if the management plane API changes fall into one of the below categories.
adding/removing APIs.
adding/removing properties.
adding/removing API-version.
adding a new service in Azure.

Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs.

[ ] If you are blocked on ARM review and want to get the PR merged urgently, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.
Please follow the link to find more details on API review process.

This PR added new AKS api-version v20200101, which only added a new field diskEncryptionSetID in managed cluster struct. (compared to v20191101)
@andliu @weinong @yangl900 @xizhamsft  for AKS related code reivew
cc @zikalino

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

Can one of the admins verify this patch?

/azp run automation - sdk

/azp run automation - sdk

You also need to make similar tag changes to readme.go, readme.python etc.

/azp run automation - sdk

You also need to make similar tag changes to readme.go, readme.python etc.

done, pls take a look @xizhamsft

/azp run automation - sdk

@jhendrixMSFT

compared the changes, seems the difference between apis is minor (diskEncryptionSetId), so i decided no arm review shall be necessary


While recording depth align to color using align-adv sample code, depth image is zoomed out compared to color image.

Required Info |
-- | --
Camera Model | D435i
Firmware Version | 5.12.11
Operating System & Version | Linux (Ubuntu 18)
Kernel Version (Linux Only) | 5.4.70
Platform | VM
SDK Version | 2.42.0
Language | C++
Segment | {Robot/Smartphone/VR/AR/others }

### Issue Description
I am doing a recording of depth alignment (to color) and storing it in the rosbag, when I look at the aligned depth and color image I see the depth image is zoomed out compared to the color image, I use the standard align-adv sample code to do the recording.

Hi @adharma1   Bag files do not store aligned images.  They store the streams separately.  When the bag file is loaded into memory, an align operation can then be performed on the data in real-time.

@MartyG-RealSense Do have sample code using which I can do this?

Thanks a lot for quick response!

Intel have a Jupyer Notebook tutorial for Python called distance_to_object that demonstrates aligning depth and color from a bag.
https://github.com/IntelRealSense/librealsense/blob/jupyter/notebooks/distance_to_object.ipynb

I see you are using C++.  You could use the SDK's rs-align sample program for live-camera alignment and adapt it to use a recorded bag file as the data source instead.
https://github.com/IntelRealSense/librealsense/tree/master/examples/align
The first paragraph of the link below explains how to adapt the code of a live-camera example program to read a bag instead.
https://github.com/IntelRealSense/librealsense/blob/master/doc/sample-data.md

Hi @adharma1  After consultation with the Intel RealSense team, the response was a confirmation of my earlier advice.
The RealSense rosbag is a recording of raw data from the sensors.  Alignment data is not saved into the bag and alignment and post-processing are performed in real-time after the bag is loaded.
The advice also stated the following points:

There is no tool to convert a RealSense rosbag to a ROS rosbag.

The rs-convert SDK tool can extract frames and metadatas from a RealSense rosbag, but it may not be of help because they still perform the align frame and convert them back to rosbag.

The most straightforward way to capture aligned images would be to use RealSense ROS to enable and publish the aligned_depth topics and record the topics using the generic ROS recorder.

https://github.com/IntelRealSense/realsense-ros#aligned-depth-frames

@MartyG-RealSense would you be aware of implementation which integrates the generic ROS with c++ for real sense recording?

I do not know of a ROS rosbag recording solution within the SDK in C++, though a RealSense user did write a rosbag parser for reading rosbags.
https://github.com/IntelRealSense/librealsense/issues/2215

Hi @adharma1  Do you require further assistance with this case, please?  Thanks!

Case closed due to no further comments received.


Why is a tokenizer required if running in ollama mode?

I am using privateGPT in ollama mode and found out that this parameter is still used here .
Is that correct? I would have expected that with ollama all tokenization happens in ollama itself.
Also it looks like privateGPT still relies somehow on this tokenizer. Because after removing it something tries to pull the gpt3.5 tokenizer from the web here.

I also want to know why is tokenizer needed.
In fact it seems it accesses an external site that needs authentication
I thought the ollama mode was totally local
I removed it and it still run

In order to calculate the exact number of tokens, we need to have the tokenizer downloaded and accessible. Since ollama does not allow to calculate tokens via API, that is the reason.
There have been a lot of problems related to that. We've thought that the best idea is to make it optional, and use the llama-index one if you can't download it, avoiding problems.


fmu-v5: ERROR [pwm] can't open /dev/pwm_output0

Describe the bug
All the ESCs start beeping on startup and never stop if the battery is connected.
In the QGroundControl Mavlink Console I type pwm info and get:
ERROR [pwm] can't open /dev/pwm_output0
My board is px4_fmu-v5. Compiling and uploading from the current master https://github.com/PX4/Firmware/commit/e69049ec4183d820c6f716be1577df2ad07194b8
I can say that it worked well around two weeks ago, and it works without this issue on the v1.8.2 release.

Drone (please complete the following information):

Pixhawk 4 on DJI F450 quad_x airframe

@JMshare this should be fixed in https://github.com/PX4/Firmware/pull/10983
Thanks for reporting.

Fixed in https://github.com/PX4/Firmware/pull/10983


Import Assertion Error

I'm gettin import assertion errors when trying to use this plugin.
Steps to reproduce:

Create a new vanilla vite project with typescript
In vite.confit.ts use the plugin as described in the readme
Observe the node log

The following error gets thrown:
failed to load config from /Users/simeonoff/Projects/tmp/svg-sitemap-test/vite.config.ts
error during build:
TypeError [ERR_IMPORT_ASSERTION_TYPE_MISSING]: Module "file:///Users/simeonoff/Projects/tmp/svg-sitemap-test/node_modules/svg-element-attributes/index.json" needs an import assertion of type "
json"
    at new NodeError (node:internal/errors:400:5)
    at validateAssertions (node:internal/modules/esm/assert:82:15)
    at defaultLoad (node:internal/modules/esm/load:84:3)
    at nextLoad (node:internal/modules/esm/loader:163:28)
    at ESMLoader.load (node:internal/modules/esm/loader:605:26)
    at ESMLoader.moduleProvider (node:internal/modules/esm/loader:457:22)
    at new ModuleJob (node:internal/modules/esm/module_job:64:26)
    at #createModuleJob (node:internal/modules/esm/loader:480:17)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:434:34)
    at async ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:79:21)

Thanks for the issue, I have succeed to reproduce the error. I have checked your PR, it will work but it will broke the compatibility with Node 14 (it is the current version supported by Vite 4). I will check to fix this issue in the following days.


terraform plan/apply always provision a replacement because of secret_id

Since the terraform state'ssecret_id uses arn, assigning the name to it causes a replacement. Checked using the arn and or id for secret and the replacement is not happening anymore.

@jmonte-sph thanks for sharing this. To understand better, can you share your secrets definitions?

@jmonte-sph also check the examples, in particular the one for defining rotation secrets https://github.com/lgallard/terraform-aws-secrets-manager/tree/master/examples/rotation

👋 there! Looks like I have the same issue with the secret I'm rotating with lambda.
Terraform = 1.3.7
Terragrunt = 0.42.5
AWS Provider = 4.48.0
Rotation secret definition:
...
inputs = {
  unmanaged = true
  rotate_secrets = {
    "/${include.common.locals.env_short}/rds/credentials" = {
      description = "This is a secret to be rotated by a lambda"
      secret_key_value = {
        password = dependency.rds.outputs.db_instance_password
        username = dependency.rds.outputs.db_instance_username
        host     = dependency.rds.outputs.db_instance_address
        db_name  = dependency.rds.outputs.db_instance_name
        port     = dependency.rds.outputs.db_instance_port
        engine   = "postgres"
      }
      ...
      rotation_lambda_arn     = dependency.lambda.outputs.secret_rotator_lambda_arn
      recovery_window_in_days = 0
    }
  }
}
...

First run works as expected, the module creates secrets.
At the second run it recreates the secret with the same secret_id and rotates it.
  # aws_secretsmanager_secret_rotation.rsm-sr["/dev/rds/credentials"] must be replaced
-/+ resource "aws_secretsmanager_secret_rotation" "rsm-sr" {
      ~ id                  = "arn:aws:secretsmanager:eu-west-1:0000:secret:/dev/rds/credentials-fCuWja" -> (known after apply)
      ~ rotation_enabled    = true -> (known after apply)
      ~ secret_id           = "arn:aws:secretsmanager:eu-west-1:0000:secret:/dev/rds/credentials-fCuWja" -> "/dev/rds/credentials" # forces replacement
        # (1 unchanged attribute hidden)

        # (1 unchanged block hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

I managed to fix it in the fork previously by adding lifesycle to rsm-sr resource, but not sure if it's right solution.

@lgallard , I have the same declaration as @alexrygalov.
@alexrygalov, looks like your lifecycle implementation fixed mine as well.


chore(deps): update node.js to v18.17.1

This PR contains the following updates:

Package
Type
Update
Change

node
engines
patch
>=18.17.0 -> >=v18.17.1

Release Notes

nodejs/node (node)
v18.17.1: 2023-08-09, Version 18.17.1 'Hydrogen' (LTS), @RafaelGSS
Compare Source
This is a security release.
Notable Changes
The following CVEs are fixed in this release:

CVE-2023-32002:  Policies can be bypassed via Module._load (High)
CVE-2023-32006: Policies can be bypassed by module.constructor.createRequire (Medium)
CVE-2023-32559: Policies can be bypassed via process.binding (Medium)
OpenSSL Security Releases

OpenSSL security advisory 14th July.
OpenSSL security advisory 19th July.
OpenSSL security advisory 31st July

More detailed information on each of the vulnerabilities can be found in August 2023 Security Releases blog post.
Commits

[fe3abdf82e] - deps: update archs files for openssl-3.0.10+quic1 (Node.js GitHub Bot) #49036
[2c5a522d9c] - deps: upgrade openssl sources to quictls/openssl-3.0.10+quic1 (Node.js GitHub Bot) #49036
[15bced0bde] - policy: handle Module.constructor and main.extensions bypass (RafaelGSS) nodejs-private/node-private#417
[d4570fae35] - policy: disable process.binding() when enabled (Tobias Nießen) nodejs-private/node-private#460

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.

This PR has been generated by Renovate Bot.

Renovate Ignore Notification
Because you closed this PR without merging, Renovate will ignore this update (>=v18.17.1). You will get a PR once a newer version is released. To ignore this dependency forever, add it to the ignoreDeps array of your Renovate config.
If you accidentally closed this PR, or if you changed your mind: rename this PR to get a fresh replacement PR.


KOGITO-7302: Patch PR for KOGITO-7301 with doc improvements in Persistence with PostgresSQL databases guide

JIRA: https://issues.redhat.com/browse/KOGITO-7302

Description: Patch PR for KOGITO-7301 with doc improvements in Persistence with PostgresSQL databases guide

https://github.com/kiegroup/kogito-docs/pull/75
Please make sure that your PR meets the following requirements:

[x] You have read the contributions doc
[x] Pull Request title is properly formatted: KOGITO-XYZ Subject
[x] Pull Request title contains the target branch if not targeting main: [0.9.x] KOGITO-XYZ Subject
[x] The nav.adoc file has a link to this guide in the proper category
[x] The index.adoc file has a card to this guide in the proper category, with a meaningful description

@cristianonicolai Can you please review the changes applied in this PR and share your comments?


First Example is not properly formatted

First example looks like a nested code box inside another code box.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7d24dee1-929c-ba3b-e555-8451d458ff97
Version Independent ID: a3cde3a3-d23f-07e8-7249-82024f46e235
Content: VisualElement.Resources Property (Xamarin.Forms)
Content Source: docs/Xamarin.Forms/VisualElement.xml
Product: xamarin
Technology: xamarin-forms
GitHub Login: @dotnet-bot
Microsoft Alias: dotnetcontent

Thanks for the feedback, we'll fix it up!

Fixed in bb0d2bd0


pindexer: swap_volume_over_window contains null values

Describe the bug
While testing recent pindexer bug fixes by rerunning pindexer from block zero, I encountered a failure on the penumbra-1 chain:
2024-11-18T23:27:34.974561Z DEBUG sqlx::query: summary="INSERT INTO insights_supply(height, total, …" db.statement="\n\nINSERT INTO\n  insights_supply(\n    height,\n    total,\n    staked,\n    price,\n    price_numeraire_asset_id\n  )\nVALUES\n  ($1, $2, $3, $5, $4) ON CONFLICT (height) DO\nUPDATE\nSET\n  total = excluded.total,\n  staked = excluded.staked,\n  price = excluded.price,\n  price_numeraire_asset_id = excluded.price_numeraire_asset_id\n" rows_affected=0 rows_returned=0 elapsed=3.095199ms
2024-11-18T23:27:34.974774Z DEBUG sqlx::query: summary="SELECT events.rowid, events.type, events.height, …" db.statement="\n\nSELECT\n  events.rowid,\n  events.type,\n  events.height,\n  tx_results.tx_hash,\n  events.attrs\nFROM\n  (\n    SELECT\n      (\n        SELECT\n          height\n        FROM\n          blocks\n        WHERE\n   blocks.rowid = block_id\n      ) as height,\n      rowid,\n      type,\n      block_id,\n      tx_id,\n      jsonb_object_agg(attributes.key, attributes.value) AS attrs\n    FROM\n      events\n      LEFT JOIN attributes ON rowid = attributes.event_id\n    WHERE\n      block_id >= (\n        SELECT\n          rowid\n        FROM\n blocks\n        where\n          height = $1\n      )\n      AND block_id <= (\n        SELECT\n          rowid\n        FROM\n          blocks\n        where\n          height = $2\n      )\n    GROUP BY\n      rowid,\n      type,\n      block_id,\n      tx_id\n    ORDER BY\n      rowid ASC\n  ) events\n  LEFT JOIN LATERAL (\n    SELECT\n    *\n    FROM\n      tx_results\n    WHERE\n      tx_results.rowid = events.tx_id\n    LIMIT\n      1\n  ) tx_results ON TRUE\nORDER BY\n  events.rowid ASC\n" rows_affected=0 rows_returned=0 elapsed=830.672533ms
Error: error returned from database: null value in column "swap_volume_over_window" of relation "dex_ex_pairs_summary" violates not-null constraint

To Reproduce
Steps to reproduce the behavior:

Dump a cometbft event db from penumbra-1 chain (i used a dump from height 2129921, available here: https://artifacts.plinfra.net/penumbra-1/)
Import cometbft event db locally
Check out main branch in this repo, at commit 0b413ac79ad15df14b256f828f30ab9717a2e614
Run pindexer against that cometbft db writing to a newly created dest db
Wait for pindexer to reach "current" height on that process (~25m)
Dump the pindexer db, restore it to an environment with with a live cometbft db
Observe error shortly after pindexer start

Expected behavior
Pindexer syncs up to current height and is happy.
Additional context
I'm surprised that when dealing with dumps, pindexer works fine. Only when I restore that dump to a db and hook up pindexer ingesting a live source db does the error occur.

Filed this early: I'm going to try again to reproduce to confirm the inputs and outputs were correct.

Ah-ha, container image was stale, so I wasn't testing against latest code. Image builds broke a few hours ago, with merge of #4930. Investigating...

Testing with #4934 shows that the problem was indeed not using the latest code. That's resolved now, and when using the latest pindexer code, I no longer see this specific error. I do see a new error, however, but filed a new report in #4935 for that, for clarity.


fix typo on configuration.md

どうもすみません, but the term prolyfill has been chosen intentionally. ;)

@tgfjt One question though: Are you actively using svg stacks as described in that part of the documentation?

Oh I see. sorry, my bad!
I just use symbols now.
I wonder if I should use defs or symbols or stacks... Is it stacks good??

@tgfjt I'd stick to symbols as well! :)
Personally I don't think stacks are the way to go. I was just wondering if anybody is using them at all. I'm planning a new major release and am thinking about dropping support completely ...

I agree with dropping support.
We want simple and stable way!


IPFS API

Implement WASM supported IPFS API.

A true fix would be some kind of browser ipfs api instead of http, only a dream at this point...


NTR:increased protein localization to centromere central core

increased protein localization to centromere central core

PMID:17677001

increased protein localization to centromere central core during vegetative growth FYPO:0007344
edit file: edc1833f265156f8ac09560413df1d4a56a6a2a8
release: f66d90f3ae622e1b9fa51878bf403d9b115adac1


Fixed issue where using path parameters but no custom domain breaks

I just redeployed an api to api gateway without a custom domain, but with path parameters in the url. This broke the parsing. I think this was an edge cased that I missed last time.
This change fixes this issue. It checks if both path and resource are set. If so, it checks if they are the same. If so, it uses resource, if not. It uses path, which will contain all the values for the path parameters.

support for python 3.3 removed and all tox removed in
https://github.com/trustpilot/python-lambdarest/pull/35

can you rebase on master?  @svdgraaf then it should work

@sloev awesome, checks out :)

Are you gonna fix the code so the test is happy ? 😊 @svdgraaf

@sloev yes, but will be somewhere later this week :)

@sloev done ;)

thanks

on it

I bumped the version to 5.0.1


Split the travis Build into Verification and Build steps

Split Travis CI to different stages so that its easy for users to understand what went wrong.

@duglin Thanks for your review, I have fixed all the suggestions

@asifdxtreme since we can't really make good use of the "stage" feature I'd like to close this PR and we can possibly reopen it later on if its a better fit (e.g. if travis supports re-using containers across stages).  ok?

@duglin Okay you can close this PR but I still think it will be a good if we add the stages as it is useful for new users to understand at what point CI failed.


package path import bug

when run the sample.py on python, there's a bug for import error.
need to fix.

82bd52c2180f01c3da017647bf57d7ef9d14823c


Compress dictionary before writing an index entry to the index file.

This can reduce the size of the index by around 60%.

Done


ansible.extra_vars = { ansible_python_interpreter:"/usr/bin/python3" }

Coverage remained the same at 86.329% when pulling fcf0fc603b1e73a2832a80a66512c9a9106b2430 on ansible-vagrant into 5b8a0e2ae3b9d06fca20912eb8ae0767b706cf7d on master.


Page appears to be out of date

It refers to VS2017 and UWP, neither of which are supported for use with Xbox anymore.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 408b2110-fede-206d-23dd-59229351e82a
Version Independent ID: 709af139-79f2-0ab5-cdea-8f65a3618cfc
Content: Quickstart C++ for Xbox - PlayFab
Content Source: playfab-docs/sdks/playfab-cpp/quickstart-xbox.md
Service: playfab
GitHub Login: @DanBehrendt
Microsoft Alias: joanlee

Page is in fact outdated and now orphaned. We will delete it. Thanks for letting us know!
Current Xbox/GDK QuickStart for this legacy C++ SDK is here: https://learn.microsoft.com/en-us/gaming/playfab/sdks/playfab-cpp/quickstart-gdk
And for our more recent C/C++ SDK here: https://learn.microsoft.com/en-us/gaming/playfab/sdks/c/quickstart-gdk