closes #2185: exclude pattern for the access log

What this PR does:
Excludes /stargate and /metrics from the access log by default. When access log is activated, the logs gets filled by those as those are accessed frequently.
Which issue(s) this PR fixes:
Fixes #2185
Checklist

[ ] Rebase after #2216 is merged

@tatu-at-datastax @jeffreyscarpenter I am not sure if this is actually needed. Maybe it's good to see in the access logs that metrics are getting scraped and health endpoints are checked?

@ivansenic I can see it both ways. But if we have Prometheus metrics for call counts, separate from access logs, that would fulfill same strategy? If we don't, then I can see why we would want entries in access logs.

@tatu-at-datastax I think we also exclude some patterns in the metrics.. So for example metrics, and health points are not included. What do you think, then we completely lose a view on those?

@tatu-at-datastax I think we also exclude some patterns in the metrics.. So for example metrics, and health points are not included. What do you think, then we completely lose a view on those?

Ah. Yeah then we'd lose all visibility which seems unfortunate. I think I'd leave them on access logs for now, esp. since access logs are not enabled by default themselves.
But should we enable scraping of healthcheck, metrics in Prometheus then? And if that was done we could exclude them here.

Agree here with @tatu-at-datastax , will close this for now.


Coercing NSNull value to nil in shouldSetValue:atKeyPath: -- should be fixed.

I get this log message when mapping JSON payload such as:
{
    'image': null
}

In CoreData the image property is defined as Transformable.

I use swift and restkit and my solution is make null properties to NSObject instead of string ...
http://baiduhix.blogspot.co.uk/2015/03/restkit-swift-handle-null-value.html


[missingtype]: element 'com.example.xxxx.xxxxx.xxxRoomDatabase' references a type that is not present

I used the same steps to create a project of my own, but it's giving the above error and I am unable to build it. Please let me know if anyone know where the issue might be or the fix for the same

Any solution?

I has the save problem

@yinxiucheng @WazirxDax
Remove this part of code from your App level Gradle, followed by invalidating caches and restarting your studio, and you'll be good to go

configurations { cleanedAnnotations compile.exclude group: 'org.jetbrains', module: 'annotations' }


Version 0.4.0 crashes with "invalid memory address"

Version 0.3.4 ran without issue however version 0.4.0 is having a nil pointer error.  I experience this issue with the provided binary or building from source.
I'm running go version 1.13.6.  I am using the linux-armv7 architecture.
Here is the error text:
2021-01-31 23:43:15.362 [INF] LITD: Handling gRPC web request: /poolrpc.Trader/NodeRatings
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x84be3c]
goroutine 555 [running]:
github.com/lightningnetwork/lnd/macaroons.(*Service).ValidateMacaroon(0x0, 0x1604150, 0x4302c00, 0x400f310, 0x1, 0x1, 0xf57278, 0x1b, 0xe7ba28, 0x42ef614)
/home/#######/gocode/pkg/mod/github.com/lightningnetwork/lnd@v0.12.0-beta/macaroons/service.go:271 +0x198
github.com/lightninglabs/pool.(*Server).ValidateMacaroon(...)
/home/#######/gocode/pkg/mod/github.com/lightninglabs/pool@v0.4.3-alpha/server.go:359
github.com/lightninglabs/lightning-terminal.(*LightningTerminal).ValidateMacaroon(0x41d1cc0, 0x1604150, 0x4302c00, 0x400f310, 0x1, 0x1, 0xf57278, 0x1b, 0x0, 0x5f3a100)
/home/#######/lightning-terminal-source-v0.4.0-alpha/terminal.go:412 +0x248
github.com/lightninglabs/lightning-terminal.(*rpcProxy).UnaryServerInterceptor.func1(0x1604150, 0x4302ac0, 0xec6190, 0x4302ae0, 0x5f425a0, 0x5f425b0, 0x75c630, 0xe7ba28, 0x4302ac0, 0xe75610)
/home/#######/lightning-terminal-source-v0.4.0-alpha/rpc_proxy.go:233 +0x170
github.com/lightninglabs/pool/poolrpc._Trader_NodeRatings_Handler(0xf23ef8, 0x4442de0, 0x1604150, 0x4302ac0, 0x54638f0, 0x41659d0, 0x1604150, 0x4302ac0, 0x5f3a180, 0x23)
/home/#######/gocode/pkg/mod/github.com/lightninglabs/pool@v0.4.3-alpha/poolrpc/trader.pb.go:4144 +0x108
google.golang.org/grpc.(*Server).processUnaryRPC(0x4172500, 0x160d6b0, 0x44a4a00, 0x4495950, 0x42ccbe0, 0x1fc0be8, 0x0, 0x0, 0x0)
/home/#######/gocode/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1082 +0x3f8
google.golang.org/grpc.(*Server).handleStream(0x4172500, 0x160d6b0, 0x44a4a00, 0x4495950, 0x0)
/home/#######/gocode/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1405 +0xaa0
google.golang.org/grpc.(*Server).serveStreams.func1.1(0x4476c00, 0x4172500, 0x160d6b0, 0x44a4a00, 0x4495950)
/home/#######/gocode/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:746 +0x84
created by google.golang.org/grpc.(*Server).serveStreams.func1
/home/#######/gocode/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:744 +0x7c

A few questions to narrow this down:

are you running remote mode or integrated mode?
is your lnd unlocked and fully started when you try to access the GUI?

I installed LITD separately but used my existing binaries for LND (0.12.0), LOOP, and POOL.  Pool must be disabled for LITD to start as it must contain it's own pool daemon.  I am running LND and LOOP through systemd.  However LITD is running on the same machine.
Yes, LND is unlocked and running prior to launching LITD.
One bit of extra info.  LITD did execute the demo educational walk through on first startup.  As soon as I completed the demo and switched to regular operating mode LITD crashed.
On Mon, Feb 1, 2021, at 2:16 AM, Oliver Gugger wrote:

A few questions to narrow this down:

are you running remote mode or integrated mode?
is your lnd unlocked and fully started when you try to access the GUI?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/lightninglabs/lightning-terminal/issues/177#issuecomment-770663614, or unsubscribe https://github.com/notifications/unsubscribe-auth/AR4GSCUHBUNG3H4Y53OEZHTS4ZPPJANCNFSM4W4FL73Q.

LiT also contains an instance of Loop that'll collide with an already running one. Does the problem still occur if you stop the Loop that's running through systemd and then restart LiT?

You are correct.  After I disabled Loop in systemd it started working correctly.  Thanks for your assistance.  I'm closing this issue.


0.7.0 Live: Make mouse work earlier

Possibly devmatch needs to be started earlier
/etc/rc.d/devmatch

# PROVIDE: devmatch
# REQUIRE: kldxref
# BEFORE: netif
# KEYWORD: nojail

/etc/rc.d/kldxref

# PROVIDE: kldxref
# REQUIRE: FILESYSTEMS
# BEFORE:  netif
# KEYWORD: nojail

The language menu has been entered in live, but the mouse cannot be used "temporarily" because the USB is still loading. People will think that it cannot be used and be deterred.

Yes, one needs to wait until the mouse becomes active. This will be solved when someone finds a way to fix this issue.


klDivergence on mixed distributions

Closes #505

[x] Get one test case to work with integral for comparing continuous part normal to continuous part uniform.
[ ] add more test cases

[x] continuous part uniform against continuous part uniform is a clear candidate

Codecov Report

Merging #512 (0b8da03) into develop (396bf5b) will increase coverage by 13.90%.
The diff coverage is 83.33%.

@@             Coverage Diff              @@
##           develop     #512       +/-   ##
============================================
+ Coverage    43.27%   57.17%   +13.90%
============================================
  Files           23       45       +22
  Lines         2066     2758      +692
============================================
+ Hits           894     1577      +683
- Misses        1172     1181        +9

Impacted Files
Coverage Δ

...c/rescript/Distributions/DistributionOperation.res
52.59% <ø> (ø)

...le-lang/src/rescript/Distributions/GenericDist.res
65.03% <ø> (ø)

...Interface/ReducerInterface_GenericDistribution.res
53.65% <50.00%> (ø)

.../src/rescript/Distributions/PointSetDist/Mixed.res
11.62% <100.00%> (+11.62%)
:arrow_up:

...script/Distributions/PointSetDist/PointSetDist.res
71.28% <100.00%> (+13.28%)
:arrow_up:

...rc/rescript/Reducer/Reducer_Js/Reducer_Js_Gate.res
100.00% <0.00%> (ø)

...r/Reducer_Expression/Reducer_Expression_Lambda.res
91.30% <0.00%> (ø)

.../Reducer/Reducer_Expression/Reducer_Expression.res
82.85% <0.00%> (ø)

...xpression/Reducer_Expression_ExpressionBuilder.res
90.00% <0.00%> (ø)

...ges/squiggle-lang/src/rescript/Reducer/Reducer.res
0.00% <0.00%> (ø)

... and 30 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4c68b1d...0b8da03. Read the comment docs.

@Hazelfire some git happened, please make sure I resolved MC's correctly. (I do not know where the MC's came from; Nuño and I did not touch the files you touched).

The math is pretty confusing so is stuff we need to be extra careful with.

I'd like to

merge this in because I think my next story may build on it
file a story to make tests stronger
file a story to document the integrals better

I approved it, go ahead with the strategy you prefer


aia_requests.build_aia_urls struggles with querying short time intervals.

Problem
Sometimes, aia_requests.parse_file_ids keys a NoneType object (resulting in an exception) when the time interval provided to build_aia_urls is relatively short (< 5 minutes). I do not know what is causing this, and I do not know why it only occurs for short time intervals. However, I have noticed that this issue is not consistent, meaning it's likely not a problem with the relevant functions in aia_requests and rather is an issue with the server being queried.
Solution
Since this is not a consistent problem, it can be mitigated by attempting the query again if it fails. It will eventually succeed by adding a while loop to attempt the query again when it fails.
Will commit a solution later today.

thanks reedOn May 4, 2023, at 09:25, Reed B. Masek @.***> wrote:
Problem
Sometimes, aia_requests.parse_file_ids keys a NoneType object (resulting in an exception) when the time interval provided to build_aia_urls is relatively short (< 5 minutes). I do not know what is causing this, and I do not know why it only occurs for short time intervals. However, I have noticed that this issue is not consistent, meaning it's likely not a problem with the relevant functions in aia_requests and rather is an issue with the server being queried.
Solution
Since this is not a consistent problem, it can be mitigated by attempting the query again if it fails. It will eventually succeed by adding a while loop to attempt the query again when it fails.
Will commit a solution later today.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

Stop checking your email - you're on vacation.

:(On May 4, 2023, at 09:43, Reed B. Masek @.***> wrote:
Stop checking your email - you're on vacation.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

Fixed with commit, although the implementation of the fix could maybe be improved.


MultibodyPlant vs Pinocchio Performance

This is an issue to track Drake::MBP vs alternatives (currently Pinocchio). This example is on a Cassie URDF. Some comments on the plant:

Does not include loop closure
Does not include contact
Fixed base model (pelvis)

Test code is https://github.com/DAIRLab/dairlib/blob/pinocchio/examples/Cassie/test/benchmark_dynamics.cc
To replicate:

Clone dairlib
Checkout pinocchio branch of dairlib
Install Pinocchio (https://github.com/stack-of-tasks/pinocchio). Install directions at https://stack-of-tasks.github.io/pinocchio/download.html. You will not need to configure environment variables
Build and run examples/Cassie:benchmark_dynamics. For non-Penn users, recommend

bazel run --define WITH_SNOPT=OFF examples/Cassie:benchmark_dynamics

Status, July 18, 2020
As run on Michael's laptop
(multibody_plant) 100x inertia autodiff calculations took 182 miliseconds. 1820 microseconds per.
(multibody_plant) 10000x inverse dynamics calculations took 125 miliseconds. 12 microseconds per.
(multibody_plant) 100xautodiff inverse dynamics calculations took 240 miliseconds. 2400 microseconds per.
(multibody_plant) 10000x forward dynamics calculations took 304 miliseconds. 30 microseconds per.
(multibody_plant) 100xautodiff forward dynamics calculations took 348 miliseconds. 3480 microseconds per.
(pinocchio) 10000x rnea inverse dynamics calculations took 20 miliseconds. 2 microseconds per.
(pinocchio) 10000x rnea inverse dynamics derivatives calculations took 61 miliseconds. 6 microseconds per.
(pinocchio) 10000x aba forward dynamics calculations took 32 miliseconds. 3 microseconds per.
(pinocchio) 10000x aba forward dynamics derivative calculations took 120 miliseconds. 12 microseconds per.
vdot_mbp, vdot_pin, diff

Summary (Drake, Pinocchio)
For numerical derivatives, approximating by double x 42 (counting fixed-base coordinates nq=nv=16 and nu=10 )

Forward dynamics: (30, 3)
Inverse dynamics: (12, 2)
Forward dynamics derivatives , Drake-Autodiff: (3480, 12)
Forward dynamics derivatives , Drake-Numerical: (1260, 12)
Inverse dynamics derivatives , Drake-Autodiff: (2400, 6)
Inverse dynamics derivatives , Drake-Numerical: (504, 6)

I'm making some progress on this -- found some inadvertent inner-loop heap allocations in MBP. Not sure yet how much those affect performance but they can't be good. Will fix shortly.

FYI @mposa there appear to be some bugs in the test cases for autodiff of inverse & forward dynamics. Try setting the iteration counts to 1 in the test and run a Debug configuration (so that all the Eigen asserts are enabled). Looks like inconsistent use if drake::math::initializeAutoDiff() producing different numbers of derivatives for (e.g.) x and u in forward dynamics. Possible fixes mentioned here.

Yes, you're right. I caught that when I added the comparison between gradient matrices, but did not bother fixing the for loops. The bug probably makes AutoDiff actually look better than it really is.

A few benchmark regard the center of mass:
All units below are [seconds per 1,000,000 calls].
Without resetting drake context every loop:
r_com time (Pinocchio):0.833135
r_com time (Drake):0.63641
v_com time (Pinocchio):1.38537
v_com time (Drake):1.50729
When resetting context in every loop:
r_com time (Pinocchio):1.09758
r_com time (Drake):3.59607
v_com time (Pinocchio):1.63794
v_com time (Drake):6.92368
J_com time (Pinocchio):1.29014
J_com time (Drake):13.6425

Thanks @yminchen! I see that the Pinocchio times changed in the second set also -- is there an equivalent of resetting the Context in Pinocchio? Also, can you say what you did specifically to "reset" the Context in Drake?

cc @amcastro-tri

@sherm1 Thanks for the question! I should've mentioned this in the previous post. The benchmark was done not directly on Pinocchio but PinocchioPlant (we derived this class from Drake's MultibodyPlant by overriding center of mass related methods). This is why there is a difference between with and without resetting context for Pinocchio. The extra runtime is from calling plant.SetPositions(context.get(), q), for example.


citation from 23.0.1 contains a duplicate paragraph with one difference

What happened?
I noticed when trying to copy the citation from version 23.0.1 that the paragraph labeled "Functional data preprocessing" is duplicated with one small difference: Paragraph 1 contains "BOLD runs were slice-time corrected to 0.721s (0.5 of slice acquisition range 0s-1.44s)" and paragraph 2 contains "BOLD runs were slice-time corrected to 0.722s (0.5 of slice acquisition range 0s-1.45s)" -- which is the correct paragraph? And am I allowed to delete a paragraph when I incorporate the whole citation into my paper, since I would be technically modifying what you request to copy as a whole? Thanks so much!
What command did you use?
inspecting html reports after running fmriprep 23.0.1

What version of fMRIPrep are you running?
23.0.1
How are you running fMRIPrep?
Singularity
Is your data BIDS valid?
Yes
Are you reusing any previously computed results?
No
Please copy and paste any relevant log output.
No response
Additional information / screenshots
No response

which is the correct paragraph?

I would guess that there was a very slight difference in the RepetitionTime metadata across runs. Because the text was not identical, it wasn't deduplicated.
I would select the one that matches your understanding of the data and delete the other. A 1ms difference should not substantially impact the results of processing, so I would not worry about adding qualifying text.

And am I allowed to delete a paragraph when I incorporate the whole citation into my paper, since I would be technically modifying what you request to copy as a whole?

The point of the suggestion that people post it verbatim is because we have had people unnecessarily change text in order to avoid plagiarism detection software, so we wanted to stand behind users and make it clear that was our intention. (See Copy-pasting your methods section is good, actually.)
FWIW I think this is a very good reason for modifying the text, but I can't stress enough that you do not need our permission for anything. The text is released into the public domain and can be used verbatim or modified to any extent and for any reason.

Sounds great, thanks!


Move from discord to matrix

As this is an open source project, it would be nice to move it from discord to matrix.

We have evaluated a number of options and Discord is the best one for us at this point in time.


#1 Add install generator

Adds rails generator to create a template "config/application.yml" file.

Don't mind me. Just testing something out.


firefox: 82.0.3 -> 83.0, firefox-esr: 78.4.1esr -> 78.5.0esr, nss: 3.56 -> 3.58, cacert refactor

Motivation for this change
This updates the firefox source packages to the latest upstream version.
This is the continued work of #104051 & #100765.
In #100765 @vcunat pointed out that we could decouple cacert from the NSS package to make it more rebuild friendly. Just rebuilding packages that depend on NSS seems to be about ~100. Rebuilding all the packages that depend on cacert is >9k as of this writing.This makes it much more feasible to upgrade high-profile packages that are (rightfully) pedantic on their NSS version like firefox and thunderbird.
Things done

[x] firefox NixOS test executed
[x] firefox-esr NixOS test executed
[ ] thunderbird pinned to older NSS version (FIXME!)
[ ] verified that pipewire still works

Paging @vcunat and @ajs124 for the NSS & cacert changes, @worldofpeace & @mweinelt for the pipewire functionality and finally @stigtsp and @taku0 for the actual firefox change. :)

Shouldn't we go with NSS 3.53 instead of 3.57 for ESR? 3.53 is an NSS ESR release, whereas I don't think 3.57 is.

Shouldn't we go with NSS 3.53 instead of 3.57 for ESR? 3.53 is an NSS ESR release, whereas I don't think 3.57 is.

Sounds reasonable. I'll update the PR.

NOTE: Whenever you update this version also update the cacert package
with a PR to staging. cacert is decoupled from this to reduce the
amount of unnecessary rebuilds while both a built from the same
source project.

Maybe we could use a script to compare whether this makes a difference and thus avoid generating meaningless rebuilds even on staging.  For example, I compared the current difference (3.57 vs 3.59) and the cert bundles are binary equal.

NOTE: Whenever you update this version also update the cacert package
with a PR to staging. cacert is decoupled from this to reduce the
amount of unnecessary rebuilds while both a built from the same
source project.

Maybe we could use a script to compare whether this makes a difference and thus avoid generating meaningless rebuilds even on staging. For example, I compared the current difference (3.57 vs 3.59) and the cert bundles are binary equal.
EDIT: it could be simple... like cacert.overrideAttrs (a_:{ inherit (nss) src version; }). (after switching to pname to avoid confusion)

Yeah that might work. I've been looking at the script that we run on the source tree and did wonder if we couldn't have a fetcher that extracts just the data that we care about but that would duplicate a lot of the logic that is in the script :/

We could make it a fixed-output derivation, but that wouldn't really remove the necessity to check if something's changed.

We could make it a fixed-output derivation, but that wouldn't really remove the necessity to check if something's changed. (and it wouldn't be trivial, due to us wanting two outputs)

Lets scratch the fixed output idea for now.
Where would you put that check script?
This would probably be sufficient to check if we have to update the sources of cacerts:
#!/usr/bin/env nix-shell
#!nix-shell -i bash -p nix

# Build both the cacert package and an overriden version where we use the source attribute of NSS.
# Cacert and NSS are both from the same upstream sources. They are decoupled as
# the cacert output only cares about a few infrequently changing files in the
# sources while the NSS source code changes frequently.
#
# By having cacert on a older source revision that produces the same
# certificate output as a newer version we can avoid large amounts of
# unnecessary rebuilds.
#
# As of this writing there are a few magnitudes more packages depending on
# cacert than on nss.

set -ex

BASEDIR="$(dirname "$0")/../../../.."

CURRENT_PATH=$(nix-build --no-out-link -A cacert.out)
PATCHED_PATH=$(nix-build --no-out-link -E 'with import ./. {}; (cacert.overrideAttrs (_: { inherit (nss) src version; })).out')

# Check the hash of the etc subfolder
# We can't check the entire output as that contains the nix-support folder
# which contains the output path itself.
CURRENT_HASH=$(nix-hash "$CURRENT_PATH/etc")
PATCHED_HASH=$(nix-hash "$PATCHED_PATH/etc")

if [[ "$CURRENT_HASH" !=  "$PATCHED_HASH" ]]; then
    echo "mismatch, consider updating"
    exit 1
fi

Maybe into cacert's passthru.updateScript?  Though I don't really know any of the auto-updating styles so far.

Maybe into cacert's passthru.updateScript? Though I don't really know any of the auto-updating styles so far.

Done. I also didn't know much about those before today.. It wasn't that hard to implement. I skipped the pname migration of cacerts as that would trigger a mass rebuild.


1201 failure aware scheduling

#1201

Lol wow this is a lot of commits. Is this ready for review? Can you add an explanation of what you're doing here?
Welcome to swarmkit new friends! Feel free to reach out to me directly on the Docker community slack or through any other channel if you'd like help with contributing.

@dperny  Hi!  everything in this pull request is ready for review with the exception of possibly adding new tests (the new code is mostly being hit by existing unit tests).
The goal of this pull request is to implement an optional randomized exponential backoff for the restart delay on task failure as per the discussion in the original issue #1201.  The new delay policy will be the default behavior, but the old constant delay functionality will still be available.  All other functionality of the restart policy will remain the same.
-Updated RestartPolicy to contain a new BackoffPolicy with a base delay, delay factor, and max delay.
-Updated CLI to allow for configuration of the backoff policy.
-Updated restart.go to use the new restart policy if applicable.  Old constant behavior will be activated if Restart.Delay != nil and Restart.Backoff == nil
-Added a new failure count to Restart.go which is reset when a task's state becomes RUNNING.  Please let us know if there is a better way to reset the failure count.

Please sign your commits following these rules:
https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "1201-failure-aware-scheduling" git@github.com:zapper59/swarmkit.git somewhere
$ cd somewhere
$ git rebase -i HEAD~842354607912
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

You'll need to rebase this and squash it into 1 commit before we merge. If you need help doing so, let me know, I'll give you the instructions.
Also, with this change, I think maybe it would be good to eliminate the Delay field on the service, and instead do fixed delay  by just making base = desired delay and factor = 0? WDYT?

@dperny It seems reasonable to consolidate the two delay values.  My only concern is that removing restart.Delay would require removing the existing command line flags and refactoring all tests that reference that delay.  We designed our logic around making sure that existing taskspec's that don't reference the backoff would not have their behavior changed.  I'm not familiar enough with swarm to know how you guys handle depreciation.  I'd be open to removing the base delay and using restart.Delay instead.

I think using Delay instead of Base is a good compromise, actually. You then just have to check if a factor and a max are set to compute a backoff. It would simplify your code quite a bit, too, because then you wouldn't need two separate code paths for Delay and Backoff.

@zapper59 are you still working on this PR?

@nishanttotla @stevvooe Sorry for the radio silence.  I think that where this pull request stands is that all code review comments have been addressed via code changes or responses to those reviews.

No big deal on the radio silence from your end. We haven't really been keeping up well on our end either. I'll take another look at this tomorrow and I'll either get it merged for you or tell you what needs to change to get it through.


calibration info needed to support reco and muon processing

right now there are a number of hard-coded values in code like reco.ImPACTReconstructor and image.muon for things like pedestal values and single-PE widths.  We should come up with a common way to store and access these in the ctapipe.instrument module.   This is not so different from other instrumental parameters like the atmosphere profile.
At the very least, the values should be stored in a file that is accessable through ctapipe.utils.datasets.get_dataset (which searches for files in ctapipe_resources (defaults) or any directory listed in $CTAPIPE_SVC_PATH (user-specified).
It could make sense that all analyses should assume that a set of required files are in a directory accessible by get_dataset, so we can build analysis configurations that way.
My suggestion for a quick way to do this would be to

define a common file-name and format for each piece of required info (this could be a FITS file, text file of some kind, or even a DB file like sqlite)
put examples/defaults in ctapipe_resources
provide a Tool that extracts the info from MC files and creates the needed files, if possible (perhaps modifying ctapipe-dump-instrument), to make it easy to build an analysis config directory with all needed inputs

eventually, we want the interface to work regardless of whether the data are in files or databases, but that could be a 2-step process where the user first runs a Tool to extract the necessary info from a DB and write it to a set of files, sets $CTAPIPE_SVC_PATH, and then runs the analysis (this would be ideal for batch processing where we don't want to access a DB if possible, since there could be many connections)
Any other ideas?

This is basically solved by using TelescopeParameters for simple things and is a duplicate of monitoring data handling for more complex  things.
Also, most of this is now available in the mon data containers at least for simtel files.


Cluster unknown faces alongside known faces

This is a significant enhancement to the clustering algorithm. Now, it clusters unknown faces too when you select the Train Faces button. It even estimates the probability that each unknown faces corresponds to an unknown cluster.

Works well and the code looks good! Very awesome :+1:
You only forgot to use the pre commit hook, I think, and I encountered one issue:
How to recreate:
Scan and cluster with your first user. Create another account, scan some other images, and then try to cluster them. The job will immediately stop. If you go to the admin panel, you can see that the job failed.
You will also find this error message within the logs:
2022-07-18 19:53:29,674 : api_util.py : get_search_term_examples : 195 : INFO : Found 100 photos
2022-07-18 19:53:37,833 : face_classify.py : cluster_all_faces : 98 : ERROR : An error occurred
Traceback (most recent call last):
  File "/code/api/face_classify.py", line 83, in cluster_all_faces
    delete_clusters()
  File "/code/api/face_classify.py", line 159, in delete_clusters
    ClusterManager.delete_cluster(cluster)
  File "/code/api/cluster_manager.py", line 29, in delete_cluster
    unknown_cluster: Cluster = ClusterManager.get_unknown_cluster()
  File "/code/api/cluster_manager.py", line 132, in get_unknown_cluster
    unknown_cluster: Cluster = Cluster.get_or_create_cluster_by_id(UNKNOWN_CLUSTER_ID)
  File "/code/api/models/cluster.py", line 30, in get_or_create_cluster_by_id
    return Cluster.objects.get_or_create(cluster_id=cluster_id)[0]
  File "/usr/local/lib/python3.9/dist-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/django/db/models/query.py", line 573, in get_or_create
    return self.get(**kwargs), False
  File "/usr/local/lib/python3.9/dist-packages/django/db/models/query.py", line 433, in get
    raise self.model.MultipleObjectsReturned(
api.models.cluster.Cluster.MultipleObjectsReturned: get() returned more than one Cluster -- it returned 2!
2022-07-18 19:54:58,264 : face_classify.py : cluster_all_faces : 98 : ERROR : An error occurred
Traceback (most recent call last):
  File "/code/api/face_classify.py", line 83, in cluster_all_faces
    delete_clusters()
  File "/code/api/face_classify.py", line 159, in delete_clusters
    ClusterManager.delete_cluster(cluster)
  File "/code/api/cluster_manager.py", line 29, in delete_cluster
    unknown_cluster: Cluster = ClusterManager.get_unknown_cluster()
  File "/code/api/cluster_manager.py", line 132, in get_unknown_cluster
    unknown_cluster: Cluster = Cluster.get_or_create_cluster_by_id(UNKNOWN_CLUSTER_ID)
  File "/code/api/models/cluster.py", line 30, in get_or_create_cluster_by_id
    return Cluster.objects.get_or_create(cluster_id=cluster_id)[0]
  File "/usr/local/lib/python3.9/dist-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/django/db/models/query.py", line 573, in get_or_create
    return self.get(**kwargs), False
  File "/usr/local/lib/python3.9/dist-packages/django/db/models/query.py", line 433, in get
    raise self.model.MultipleObjectsReturned(
api.models.cluster.Cluster.MultipleObjectsReturned: get() returned more than one Cluster -- it returned 2!
2022-07-18 19:55:12,692 : api_util.py : get_search_term_examples : 181 : INFO : 100 possible ids

Probable cause:
I think it has something to do with clusters. You wrote in the issue, that an issue could be that clusters are not associated with users. I think specifically "Unknown - Other" or the numbering of the unknown clusters is off.

I believe I've addressed this - I cannot reproduce what you're describing, and I added a small sanity check when deleting the unknown cluster (to prevent it, actually).
Thanks for bearing with me; this is my first time using Python or pre-commit (we have other languages and processes where I work). I'm hoping this latest round of changes is good.
I am still not making clusters user-specific. This does cause one user's face training to "overwrite" another's. It doesn't seem like a massive deal, but I can add it now (or in a subsequent pull request), your choice.

I just went ahead and made clusters user-specific now anyway. It feels more usable now that different users can't accidentally trample each other's clusterings.

Now it works without the try except block! Awesome. 🚀
I have encountered out of memory issues and long run times with a large dataset (>50k faces). It runs fine and is fast with 2k faces. I will run it again with the user specific clusters to see if I can reproduce it.
Usually, large queries without pagination can lead to OOM errors. Long run times, that only occur after a specific number of items, are usually related to quadratic time complexity somewhere in the code.
How large is your testing database?

The size depends on what I'm testing. I use 16-20 (very small) for full database rebuilds (so I can test migrations very quickly, for example), and only about 200 for larger clustering tasks.
I can try a larger dataset and see what happens. Sadly, my crappy old Macbook Air may not have room for a truly enormous data set, so I'll have to do my best.

The runtime of DBSCAN according is in the worst case O(n^2) and in the best case O(n log(n)) link. I think this inefficiency gets introduced somewhere else, but that could be one reason :smile:
You could use this dataset to get more faces: http://vis-www.cs.umass.edu/lfw/#download

Alright, I tested it with a very large dataset (170k faces) and did not find any OOM issues. It also finished in 20 minutes, which is in line with your findings, and it looks like it scales linearly.
The frontend only shows 50k images anyway, so it should be future-proof! 😄
Awesome work!


Prometheus scraper server side implementation

PR created for initial review.

all the required changes incorporated

@c-bik, #533 checkpoints are met.


Build darwin

Code changes to successfully build on OSX, these may not be suitable for the linux build.

Main repo has this working now, this branch is no longer needed.


Add alerts and their macOS implementation.

API
This PR adds a cross-platform alert API that will enable a good user experience on Windows/macOS/Linux.
The API is designed in a way that a single AlertOptions configuration will result in a native feeling alert on all these platforms. The buttons are ordered automatically, so none of this windows-app-on-mac or vice versa feeling when you encounter an alert with a weird layout.
The API also favors a good user experience with contextualized (and translated) button labels. This is something that all the platform guideline documents agree on. Yes you can find plenty of legacy apps with basic labels like Confirm or Yes but neither of those are actually recommended.
macOS implementation
This PR also contains the macOS implementation of this API to get things started and show that the API works.
I am aware of at least two macOS druid-shell issues that show up in relation to the alerts work. These issues aren't directly related to the alerts API though so I am tackling them separately. The first one is timers not firing when modal dialogs are up, which is resolved in #1028. When that fix is applied another issue surfaces - Application::quit doesn't properly end the runloop when an app modal dialog is up - it just ends the modal runloop. This is an issue I'm still investigating but will also solve separately as it's more of a bug in the quit method.
Alert example
I also added a new alert example. It shows a few different ways to use alerts, although some of it can feel needlessly complex for an example. I built it more as a testing tool to see how edge cases are handled when multiple alerts show up in various ways. This made me think that perhaps we should have an examples directory but also some sort of testing directory of example-like apps that help us confirm whether various implementations are working properly. For now, I think the alert example can live as an example, but this distinction is something to think about for the future.
Future work
I actually started this alerts work with the Windows implementation, which I've had 80% done for months now. This helped me really understand the cross-platform requirements. Moving forward I'll finish up the Windows implementation and post that as a separate PR.

Fixes #383.

Tested the example on MacOS and everything worked very nicely. Looked very "native".
Tested the example on Manjaro Linux with i3 desktop environment and:

"New window" worked.
"Quit in style" quits the app after ~3 second without showing any alerts.
Nothing else worked.

When that fix is applied another issue surfaces - Application::quit doesn't properly end the runloop when an app modal dialog is up - it just ends the modal runloop. This is an issue I'm still investigating but will also solve separately as it's more of a bug in the quit method.

Interesting... It looks like this is because the stop method only applies to the current runloop https://github.com/xi-editor/druid/blob/6a429144309cef53c5d8c772b9e9c25416c50cee/druid-shell/src/platform/mac/application.rs#L96
It's not clear to me why this isn't just using terminate:, so perhaps that or calling stop via performSelectorOnMainThread:withObject:waitUntilDone:

It's not clear to me why this isn't just using terminate:,

Because terminate kills the process and cleanup code won't run.

so perhaps that or calling stop via performSelectorOnMainThread:withObject:waitUntilDone:

That's not enough either. I've found a way to stop two levels of runloops with abortModal + stop, however not three levels yet. I'll try some sort of handle based targeting soon probably.

An additional thought I missed in my original review: I'll try and use this in runebender shortly and see how it feels, that will hopefully give me better perspective.

Just had a thought here: what if you passed a Command along with each alert option? then we could not worry about tokens etc, and just handle the command as usual?

An update: I implemented a "you have unsaved changes" dialog when trying to quit in runebender, and I'm fairly happy with the result. I do think this idea of just associating a command with each button could be helpful. If we wanted to be very rigorous, we might also want to use a token to identify a specific modal; I'm not sure how necessary this is in practice.

What's the status of this? Are there any major changes to the API that need to be made? If not, I'd like to resolve the conflicts and get this merged. (My main limitation is that I can't test on mac, so I don't want to make non-trivial changes to the mac part)

Given that the original author has gone radio silent (I hope you're ok, @xStrom, and just taking a break; in any case we appreciate your work on this), I'd be very grateful for you to champion getting this merged, especially as it's a very often requested feature. I'm willing to take on mac testing if nobody else steps up.

closing this in favor of #1256

I am now back and will update this PR shortly. I have some API improvement ideas that weren't pushed that I'll also review - and I'll read through the feedback and address that as well.

How is progress on this coming along? Is there anything I can do to help? Basically alerts and PasswordInput are the only two things blocking me from use Druid in real world applications.

Yea, I need to pick this up and get it finished.


Fixes in closure wrappers

Basically:

the previous test to verify whether we had a owning closure or not was to check is the storing type was a pointer or not. This prevented to have closures on pointer types.
the get method should have returned references on the wrapped type, so that we can swap things.

Thanks to @wolfv for catching the requirement for the add_const_t.


flutter-devicelab-mac-8 has not run tasks for more than a month

The last task ran on 4/13/22.  https://chromium-swarm.appspot.com/bot?id=flutter-devicelab-mac-8
I'm guessing nothing is being scheduled on this bot because its device is 15.2.1 and .ci.yaml is specifying iOS-15.1 but that may not be the reason.
https://github.com/flutter/flutter/blob/dfda76da544e4014084880ca2d54ab490a46b101/.ci.yaml#L89
An additional devicelab Mac would help with framework testing capacity.

@jmagman ios version is potentially the problem here.
@christopherfujino has a request for the ios devices to be upgraded.
Do we want to restrict tests to a specific ios minor version?

Do we want to restrict tests to a specific ios minor version?

I think major version is fine, we can revisit if it causes issues.

This bot is now running jobs fine.


communication

[ ] article for https://www.salesforceben.com/write-for-us/
[ ] article on https://www.apexhours.com/contact-us/
[ ] youtube video one for each feature
[ ] https://salesforcegeek.in/contact-us/
[ ] https://gettectonic.com/contact-us/
[ ] reddit https://www.reddit.com/r/salesforce/
[ ] https://unofficialsf.com/feedback-and-comments/
[ ] trailblazer groups: https://trailhead.salesforce.com/fr/trailblazer-community/groups/0F9300000001rzcCAA

video:
screenshots => image improoved with MyHeritage + hightlight some parts & avatar with Synthesia + video editing Clipchamp

Export :Inline edit, compare sosl, download csv
Apex runner : logs, jobs, profiler
Streaming


Add pre-commit config and bring in LMNT ruff configs.

Note: I'm going to rework this (and the other PR) in the morning to just use ruff, I think what autopep8 adds on top of ruff may not be material enough to be worthwhile.


Constexprfy all methods

Can we add tests to ensure that these functions can meaningfully be constexpr?

I hope I understand your question correctly: You can have a look at the test-suit of MSVC STL. They write all their test-functions as constexpr. And then in their main function they call the function both from "runtime-context" and "compile-time-context":
int main() {
    test();
    static_assert(test());
}

Unfortunately I am not sure yet how to combine this approach with Google Test.

#65 should replace this. Tests pass on AppleClang with bazel but not with GCC.
Will work on a local (Docker) repro.
@Ukilele input welcome if you're interested/able.


support --untar flag for dep update

What this PR does / why we need it:
support --untar flag for dep update,and use it  what untar *.tgz to ./charts/. At the same time, the local tgz package will be deleted
fix issue:https://github.com/helm/helm/issues/8479
Special notes for your reviewer:
If applicable:

[ ] this PR contains documentation
[x] this PR contains unit tests
[ ] this PR has been tested for backwards compatibility

Thanks for the PR. In this case, the tests are failing due to the style checker. You removed a new line in the imports. Full details are in the details of the full test run. It's a minor change to fix.

Thanks for the PR. In this case, the tests are failing due to the style checker. You removed a new line in the imports. Full details are in the details of the full test run. It's a minor change to fix.

Done.Thx for your note, looking forward you review. 0.0.

@Cweiping This PR was discussed at the dev weekly call last Thursday. @adamreese asked to take a look at the PR and will provide feedback shortly.

@Cweiping This PR was discussed at the dev weekly call last Thursday. @adamreese asked to take a look at the PR and will provide feedback shortly.

@adamreese  Hi,looking forward to your suggestions for improvement.

I don't know if this will solve the problem I was having in https://github.com/helm/helm/issues/8120, but it seems like it could be related.

@hickeyma any news?

@hickeyma any news?

Waiting for any update.

Waiting for any update.

Ping @adamreese. You mentioned in the past that you wanted to look at this.

Ping @adamreese. You mentioned in the past that you wanted to look at this.

any updates

@Cweiping maybe you can rebase this PR to check if CI will still fail?

@Cweiping can you get this to pass CI?


Publish Palenight Theme

This submit the Palenight Theme to the gallery for use by costumers.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: joserocha sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

I'll publish this extension with our other April release extension updates planned for tomorrow morning.  Thanks!


Constrain SQLAlchemy to <=1.4.35

SQLAlchemy changes in versions greater than 1.4.35 break relationships and break even the examples in the docs, the latest working version seems to be 1.4.35.
This has been the cause of at least two issues (https://github.com/tiangolo/sqlmodel/issues/255, https://github.com/tiangolo/sqlmodel/issues/315) but probably more.
The breaking change wasn't picked up by dependabot because the current constraint of <1.5.0 does not register any releases under 1.5.0 as 'new' since the version constraint does not have to be bumped up for it. By limiting it to <= dependabot will start bumping up the upper end of the constraints.
PR https://github.com/tiangolo/sqlmodel/pull/322 fixes this with a simple code change but hasn't been merged yet, the constraint range change should still be made so that dependabot will automatically create PRs and run tests on newer versions of SQLAlchemy.

Since sqlmodel has been broken since, I guess, May (?) with this PR unmerged since May, should I assume sqlmodel is unmaintained? Not a gripe at all, just trying it out and wondering if the project still active.

This has been the cause of at least two issues (https://github.com/tiangolo/sqlmodel/issues/255, https://github.com/tiangolo/sqlmodel/issues/315) but probably more.

Note that https://github.com/tiangolo/sqlmodel/issues/255 was opened on Feb 28, while SQLAlchemy 1.4.36 was released on Apr 6, 2022.

Since sqlmodel has been broken since, I guess, May (?) with this PR unmerged since May, should I assume sqlmodel is unmaintained? Not a gripe at all, just trying it out and wondering if the project still active.

I think @tiangolo is just pretty busy right now. The rate of new issues and PRs is pretty high so it's easy to miss something like this, especially if you're relying on dependabot to test and flag these kind of issues. Changing the constraint from < to <= should help a bit since it'll bump up the version and run the tests for new SQLAlchemy versions in the future.

This has been the cause of at least two issues (#255, #315) but probably more.

Note that #255 was opened on Feb 28, while SQLAlchemy 1.4.36 was released on Apr 6, 2022.

Aha, interesting, to be honest I was lazy and just searched for 1.4.35 and assumed that any issue where people said installing that version fixed the problem was related to this. This comment https://github.com/tiangolo/sqlmodel/issues/255#issuecomment-1151481402 mentioned that pinning the version to <=1.4.35 fixed the problem so it seemed related, but yeah based on the dates that doesn't make sense :thinking:

Thanks @RobertRosca! Good to know it's still active.

+1 for this PR so that, as @RobertRosca observes, dependabot can more easily track this issue.

Awesome, thank you @RobertRosca! :rocket:
I upgraded the max version to the current latest, as everything is passing with it. :nerd_face:
And thanks everyone for the discussion. The underlying issue was solved in https://github.com/tiangolo/sqlmodel/pull/322, and this version pin will prevent it from happening inadvertently.
This will be available in the next version, in the next hours. SQLModel 0.0.7 :rocket:


Guidance on aliased GraphQL operations in a single request

Note: If your question is regarding the AWS Amplify Console service, please log it in the
official AWS Amplify Console forum

Which Category is your question related to?
Aliased GraphQL operations  in a single request - Many-To-Many - AppSync
Amplify CLI Version
3.17.0
What AWS Services are you utilizing?
AppSync
Expected behavior
Most efficient way to add many editors to a post, using only one request.
Provide additional details e.g. code snippets
Regarding the example with many-to-many provided here: https://github.com/aws-amplify/docs/pull/900/files#diff-a8516a884b9597a490fdec8e8c96ce9a , I'm looking for guidance on aliased GraphQL operations in a single request.
The documentation gives the following schema:
type Post @model {
   id: ID!
   title: String!
   editors: [PostEditor] @connection(keyName: "byPost", fields: ["id"])
}
# Create a join model and disable queries as you don't need them
# and can query through Post.editors and User.posts
type PostEditor
   @model(queries: null)
   @key(name: "byPost", fields: ["postID", "editorID"])
   @key(name: "byEditor", fields: ["editorID", "postID"])
{
   id: ID!
   postID: ID!
   editorID: ID!
   post: Post! @connection(fields: ["postID"])
   editor: User! @connection(fields: ["editorID"])
}
type User @model {
   id: ID!
   username: String!
   posts: [PostEditor] @connection(keyName: "byEditor", fields: ["id"])
}

And the following mutations to create users and posts:
mutation CreateData {
    p1: createPost(input: { id: "P1", title: "Post 1" }) {
        id
    }
    p2: createPost(input: { id: "P2", title: "Post 2" }) {
        id
    }
    u1: createUser(input: { id: "U1", username: "user1" }) {
        id
    }
    u2: createUser(input: { id: "U2", username: "user2" }) {
        id
    }
}

After users and posts are created, we can create a connection between them creating a PostEditor object:
mutation CreateLinks {
    p1u1: createPostEditor(input: { id: "P1U1", postID: "P1", editorID: "U1" }) {
        id
    }
    p1u2: createPostEditor(input: { id: "P1U2", postID: "P1", editorID: "U2" }) {
        id
    }
    p2u1: createPostEditor(input: { id: "P2U1", postID: "P2", editorID: "U1" }) {
        id
    }
}

So far, I was able to create this connection in my app, but with a mutation for every user:
const users = [{id: "U1", username: "user1"}, {id: "U2", username: "user2"}];
        const post = { id: "P1", title: "Post 1" };
        /*
        After creating two users and a post using the approriate mutations
        Using the CreatePost join below to make user1 and user2 editor on Post 1
        */
        function graphqlCreatePostEditor(editorID) {
            return new Promise((resolve, reject) => {
              resolve(
                API.graphql(graphqlOperation(createPostEditor, {
                  input: {
                      postID: post.id,
                      editorID
                    }
                }))
              )
            })
          }

        let promises = users.map(user=> {
            return graphqlCreatePostEditor(user.id)
            .then(e => {
                console.log(e)
            return e;
            })
        });

        Promise.all(promises)
            .then(results => {
                console.log(results)
            })
            .catch(e => {
                console.error(e);
            })

My use case involves to be able to add many editors to a post in one single request. For example, the post creator will be able to select 50 users, then add them as editor of the post newly created. Is there a better way to handle this use case than creating a promise and doing a mutation for every connection (it seems not efficient for 50 users)?
I can provide more code if needed.

@devnantes44 I think as it is a custom use case perhaps using a custom mutation with custom resolvers is the best solution, utilizing Batch operations for DynamoDB.
@janhesters has a blog article about batch operations<EMAIL_ADDRESS>


Notification stacking function

What problem does this feature solve?
The notification will be automatically closed
threshold
What does the proposed API look like?
nzThreshold

Ant Design has supported to fold the notifications when there are too many:


Add gopkg.in versioned releases information.

Also made the markdown headings consistently use prefixed #'s, no suffix.

:+1: Looks good @GeertJohan

Great


refactor: implements client registration

:white_check_mark: Build TheIdServer 1.0.0-dynamic-client1-0094 completed (commit https://github.com/Aguafrommars/TheIdServer/commit/8a8d979bca by @aguacongas)

:white_check_mark: Build TheIdServer 1.0.0-dynamic-client1-0095 completed (commit https://github.com/Aguafrommars/TheIdServer/commit/d6f2fb7934 by @aguacongas)

:white_check_mark: Build TheIdServer 1.0.0-dynamic-client1-0096 completed (commit https://github.com/Aguafrommars/TheIdServer/commit/3b5b9ef21d by @aguacongas)


Packaging Request: xemu

Project description
Original Xbox Emulator.
Metadata

homepage URL: https://xemu.app
source URL: https://github.com/xemu-project/xemu
license: gpl2
platforms: unix

I've tried to get this working, but I'm getting a link error. Here are both of my attempts:
{ stdenv
, fetchFromGitHub
, lib
, qemu

, libpcap
, libsamplerate
, python3
}:

(qemu.override (_: {
  guestAgentSupport = false;
  numaSupport = false;
  seccompSupport = false;
  alsaSupport = false;
  pulseSupport = false;
  sdlSupport = true;
  jackSupport = false;
  gtkSupport = true;
  vncSupport = false;
  smartcardSupport = false;
  spiceSupport = false;
  ncursesSupport = false;
  libiscsiSupport = false;
  tpmSupport = false;
  uringSupport = false;
  hostCpuTargets = [ "i386-softmmu" ];
})).overrideAttrs (new: old: rec {
  name = "xemu";
  version = "0.7.83";

  src = fetchFromGitHub {
    owner = "xemu-project";
    repo = name;
    rev = "d0d3e7b";
    fetchSubmodules = true;
    hash = "sha256-ryL7u2sKR5nBkKNUseit8iuep0Ga6R7TwVHh1F5OQoE=";
  };

  patches = [];

  buildInputs = old.buildInputs ++ [ libpcap libsamplerate python3.pkgs.pyyaml ];

  setOutputFlags = false;

  configureFlags = old.configureFlags ++ [
    "--enable-lto"
    "--disable-werror"
  ];

  preConfigure = ''
    chmod +x ./scripts/shaderinclude.pl
    patchShebangs . >/dev/null
    configureFlagsArray+=("--extra-cflags=-DXBOX=1 -Wno-error=redundant-decls")
  '';

  meta = with lib; {
    homepage = "https://xemu.app/";
    description = "Original xbox emulator";
    maintainers = with maintainers; [ ];
    license = qemu.meta.license;
  };
})

{ stdenv
, fetchFromGitHub
, lib
, qemu

, pkg-config
, python3
, ninja
, meson
, which
, perl

, glib
, gtk3
, libpcap
, openssl
, libepoxy
, libsamplerate
, SDL2
, SDL2_image
, mesa
, libdrm
, libGLU
, gettext
, vte
}:

stdenv.mkDerivation rec {
  name = "xemu";
  version = "0.7.83";

  src = fetchFromGitHub {
    owner = "xemu-project";
    repo = name;
    rev = "v${version}";
    fetchSubmodules = true;
    hash = "sha256-L6ax7u2PL1QpTmDFM8Os0Wwj3zFWXQXqM7AGqZql1F0=";
  };

  nativeBuildInputs = [ pkg-config python3 python3.pkgs.pyyaml ninja which meson perl ];

  buildInputs = [ glib gtk3 openssl mesa libepoxy libdrm libpcap libsamplerate SDL2 libGLU SDL2_image gettext vte ];

  separateDebugInfo = true;

  dontUseMesonConfigure = true;

  setOutputFlags = false;

  configureFlags = [
    "--disable-strip"
    "--meson=meson"
    "--extra-ldflags="
    "--target-list=i386-softmmu"
    "--enable-lto"
    "--disable-werror"
    "--enable-opengl"
  ];

  buildFlags = [ "qemu-system-i386" ];

  preConfigure = ''
    patchShebangs . >/dev/null
    configureFlagsArray+=("--extra-cflags=-DXBOX=1 -Wno-error=redundant-decls")
  '';

  preBuild = ''
    cd build
    substituteInPlace ./build.ninja --replace /usr/bin/env $(which env)
  '';

  postBuild = ''
    rm -rf dist
    mkdir -p dist
    cp build/qemu-system-i386 dist/xemu
    python3 ./scripts/gen-license.py > dist/LICENSE.txt
  '';

  installPhase = ''
    runHook preInstall

    cp -r dist $out

    runHook postInstall
  '';

  meta = with lib; {
    homepage = "https://xemu.app/";
    description = "Original xbox emulator";
    maintainers = with maintainers; [ ];
    license = qemu.meta.license;
  };
}

I haven't tried anything. I will test this. I wanted emulators (i.e. Vita3k, xenia, etc) and emulator-related tools to be part of NixOS ecosystem too.
Yes, it worked for me. I'm seeing xemu just right now. I will need to test this a bit more after setting things up. Will let you know the end results after poking around and opening everything.

Got an issue - xemu: GLib-GIO: No GSettings schemas are installed on the system.

Push what I hope is a fix, can you try it again?

You got that one fixed. I'm now able to select files. Now, I'm stuck at Please insert an Xbox disc....  Weird thing is that the titlebar is xemu | v.  Not sure if that's intentional or if something got stripped.
If I restarted (after renaming ISO), I get a missing file dialog so I know it's working (i.e. xemu recognizes the file is missing), but it still throw me Please insert an Xbox disc... message.

The input screen works. It recognizes my controller and the buttons seems to be responding correctly. Taking a screenshot works too. I have finished adding necessary files. I just can't get it to load a disc.

Oh wait. I think I know what the problem is... I forget to do a step here. Hold on.

Yes! The game loads! Thank you!

As for xemu | v titlebar, the version is missing. I looked at few xemu screenshot Google images.
Same with version, branch, commit.

the version/commit info depends on the .git directory being present, which is possible but makes it really hard to be reproducible

Honestly, this is great for initial package. Version fixing can come later. I recall citra and cemu have similar issue with versioning too. citra reports lack of version and cemu reports vague version. Dolphin does it perfectly. Something for experienced coders to look into. Thank you. Please merge it. :-)

Got the version stuff to work, also added a desktop item


⚠ Bofrost.at has degraded performance

In daef663, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2801 ms

Resolved: Bofrost.at performance has improved in ccb8efc after 313 days, 1 hour, 48 minutes.


Added support for electra tokenizer as a special case of BERT tokenizer

Added support for the ELECTRA tokenizer. The ELECTRA models is waiting for the ELECTRA PR on curated transformers.
Checklist

[x] I confirm that I have the right to submit this contribution under the project's MIT license.
[x] I ran the tests, and all new and existing tests passed.
[ ] My changes don't require a change to the documentation, or if they do, I've added all required information.

I have not added the documentation required for ELECTRA as it required the PR of curated transformers

Ah yes thanks for the suggestions (getting too used to ruff cleaning up after me). I also formatted it to black (saw that the tests failed).

Hi @shadeMe seems like this PR is ready to merge?


feat: upgrade default app version to v2.55.0

:warning: BREAKING CHANGE
The default ZITADEL version is now v2.55.0.
This requires Cockroach DB to be at >= v23.2
If you are using an older version of Cockroach DB, please upgrade it before upgrading ZITADEL.
Note that in order to upgrade cockroach, you should not jump minor versions.
For example:
# install Cockroach DB v23.1.14
helm upgrade db cockroachdb/cockroachdb --version 11.2.4 --reuse-values
# install Cockroach DB v23.2.5
helm upgrade db cockroachdb/cockroachdb --version 12.0.5 --reuse-values
# install Cockroach DB v24.1.1
helm upgrade db cockroachdb/cockroachdb --version 13.0.1 --reuse-values
# install ZITADEL v2.55.0
helm upgrade my-zitadel zitadel/zitadel --version 8.0.0 --reuse-values

Please refer to the docs by Cockroach Labs. The ZITADEL tests run against the official CockroachDB chart.
(Credits to @panapol-p and @kleberbaum :pray:)
Replaces #210
Definition of Ready

[x] I am happy with the code
[x] Short description of the feature/issue is added in the pr description
[x] PR is linked to the corresponding user story
[x] Acceptance criteria are met
[x] All open todos and follow ups are defined in a new ticket and justified
[x] Deviations from the acceptance criteria and design are agreed with the PO and documented.
[x] No debug or dead code
[x] My code has no repetitions
[x] Documentation/examples are up-to-date
[x] All non-functional requirements are met
[x] If possible, the test configuration is adjusted so acceptance tests cover my changes

@panapol-p @kleberbaum is this helpful?

Yeppp , thanks bro  🎉🥳


Show how to set invalid pointer targets with Zinnia Rules

Tutorial showing how to set targets to invalid using zinnia rules.

https://www.youtube.com/watch?v=GILtPiHJKOo


EPIC: migrate plugins to use Cloud Access Policies

API keys are being deprecated in favour of Access Policies: https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/access-policies/.
Currently we advise and require the use of API keys to sign plugins, so we need to figure out the impact of this change and start moving developers to use the new mechanism.
### Tasks
- [x] 🤞 try a Access Policy token and see if everything still works with no further changes...
- [ ] Validate the required access [scope(s)](https://github.com/grafana/auth/blob/main/pkg/api/models/scope.go) for replacing an API key token
- [ ] Update documentation for signing a plugin
- [ ] If changes are required in `sign-plugin`, determine whether we can support both until API keys are fully removed
- [ ] Add validator warning on use of API key with advice to move to API policy
- [ ] Update provided GitHub actions for sign/releasing of plugins
- [ ] Communicate the changes to the community

Tried it out
plugins:write is the scope that has to be used.

No changes are required to sign plugin:
This "If changes are required in sign-plugin, determine whether we can support both until API keys are fully removed" can be removed / checked as done.


Docs: Implements Documentation to install Cilium on k3s

Adds documentation to install Cilium on K3s (https://k3s.io) - a certified distribution of Kubernetes. Previously there was no documentation to create a working K3s sandbox environment running Cilium.
Fixes: #10471
Signed-off-by: Sean Winn<EMAIL_ADDRESS>

This change is

Coverage increased (+0.02%) to 45.638% when pulling ae16f16f4d6620d6b6340cb8e03a0c8c65c8c9e5 on seanmwinn:k3s-install into 9c346306a329895e4507d377dd43167005bfcbaf on cilium:master.

test-docs-please

test-docs-please

test-docs-please

@seanmwinn noticed that in your commit msg you had written needs-backport/1.7. We usually add it to the labels when PR is created.


Disable remark for non-governance use

We should disable calling remarks without either going through governance or other pallets to prevent spam.

mby just artificially increase cost of the remark

It looks like it is not possible to use CallFilter for this. The Call parameter used is the filter doesn't carry the information about origin. https://docs.substrate.io/rustdocs/latest/frame_system/pallet/enum.Call.html


Build out mini MVP

take in some input mols, define how we want to pack/arage them, apply ff, run md, profit

container that takes in a toml file

cli that also takes in a toml file

evaluate if interchange will work for us (including CG)

See #29


ZSH completion is broken due to bad quoting in commands.go

Summary
I assume commands.go is responsible for automatically generating ZSH completion. Here:
https://github.com/gopasspw/gopass/blob/00de84a305286f6d5983261372d8cd0b6a2fdea9/commands.go#L538
The word FILE is quoted with ` and when ZSH parses the completion function it tries to evaluate the command FILE which (usually) doesn't exist.
Steps To Reproduce
$ gopass otp -<tab>
zsh: command not found: FILE
zsh: command not found: FILE
zsh: command not found: FILE

Expected behavior
$ gopass otp -<tab>
Completing no arguments
Completing option
Completing corrections
--clip  -- Copy the time based token into the clipboard
--qr    -- Write QR code to FILE

Environment

OS: Arch Linux
OS version: Linux NUC 4.18.16-arch1-1-ARCH #1 SMP PREEMPT Sat Oct 20 22:06:45 UTC 2018 x86_64 GNU/Linux
gopass Version:

gopass 1.8.2 (2018-08-03 20:31:47) go1.10.3 linux amd64
<root>     - gpg 2.2.10 - git 2.19.1 -   fs 0.1.0

Installation method: Arch Linux community repository.

Additional context
This issue is completely unrelated to #585 which should be closed IMO since it is obvious the user who opened it is not familiar with ZSH's completion system.
It's very easy to work around this issue by creating a file name _gopass in your $fpath where FILE is not quoted there.

Thanks. I can reproduce this on my systems. Would you mind creating a PR for this?

Kindly, but I still think this should be reported upstream to where the library that configures the command line options - github.com/urfave/cli?

I think we switched to a self maintained version at some point:
https://github.com/gopasspw/gopass/blob/master/pkg/completion/zsh/template.go

BTW just out of curiosity in Go development, why don't you use submodules with links to your own forks?

This should give you a few more details:
https://github.com/golang/dep/issues/1240
Go is actually moving away from vendor/ and dep again. Go modules is the thing for the future. But lets not get too much off topic here. Happy to help elsewhere. :wink:

IMHO git submodules are somewhat broken and the useability and the tool support is really bad. Also Go Modules might be the way to go in the future.


Uber Facebook login redirect doesn't work on mobile devices

To reproduce:

start playground
connect to playground in iOS simulator (remember to enable popups in iOS safari settings)
tap to connect uber
press on facebook icon
log in to facebook
redirects to uber but doesn't go further

Also seems to happen on Android. Works fine from a desktop browser.

Simpler reproducible case:

console.log the authorizationCodeRequestUrl L65 in OAuth2Manager.js
tab to connect uber
copy link
open link using Chrome desktop in mobile mode (Galaxy S5) using debugger tools
click on facebook -> no redirect after facebook login
open link using Chrome without mobile mode -> works

After some digging I found that this error occurs:

which comes from a token verification code:

I found that decodeURIComponent(n) yields {"query":"?breeze_local_zone=dca1, which obviously is a truncated JSON containing incomplete info.
n is coming from t.fbstate -- would that mean the facebook state returned is invalid?
..that's just weird variable o is not even used.


Contract source code validator

Contract source code validator for contract creators to upload their source code. If source code matches the byte code of the contract, store that source code and abi. With the ABI, we are able to parse the execution input and output, and show them in formatted way (say JSON).
Similar functionality can be found on etherscan: https://etherscan.io/address/0xa5025faba6e70b84f74e9b1113e5f7f4e7f4859f#code

This breakdown to the following steps:

let someone upload the solidity src code, and specifies the compiler version.
server verifies it by compiling and comparing it to the bytecode in the tx (CAPTCHA needed to mitigate DDOS)
server accepts and displays contract src code, and never accept another copy of the src code.


simpler float4 direct store and locals support

follow-up on #3564
Since the UOpGraph holds a single copy of every uop, if we update that UOp directly it auto-updates vins too.
This simplifies things a lot, and supports locals out of the box

I don't understand this. What changed about the args to the STORE op?
We really need docs for the uops.

change: The STORE's children are now [the float4 acc, scalar PHIs]
It should've always been there, because the childless pass was skipped it wasn't caught.
But if you skip the CAST, there should still be something linking the PHIs to the STORE.
We can't store the PHIs directly because the access pattern is indexed.
@geohot example: TestOps.test_matvec
example: TestOps.test_matvec
kernel void r_8_4_8_16_4(device float* data0, const device float* data1, const device float* data2, uint3 gid [[threadgroup_position_in_grid]], uint3 lid [[thread_position_in_threadgroup]]) {
  // DEFINE_ACC dtypes.float4
  float4 acc0 = float4(0.0f,0.0f,0.0f,0.0f);

  for (int ridx0 = 0; ridx0 < 16; ridx0++) {
    // PHI dtypes.float vin=[GEP,...]
    (acc0).x = ((val0*val1)+(acc0).x);
    // PHI dtypes.float
    (acc0).y = ((val0*val2)+(acc0).y);
    // PHI dtypes.float
    (acc0).z = ((val0*val3)+(acc0).z);
    // PHI dtypes.float
    (acc0).w = ((val0*val4)+(acc0).w);
  }
  // --- before, CAST was PHI's child
  // CAST dtypes.float4 [PHI, PHI, PHI, PHI]
  // then, we stored the cast:
  //  STORE vin=[cast]
  *((threadgroup float4*)(temp+alu1+(lidx2*4))) = float4(acc0.x, acc0.y, acc0.z, acc0.w);

  // -- now, the CAST is skipped
  // The PHI should be directly linked to the STORE
  // STORE vin=[acc0, *PHI values]
  *((threadgroup float4*)(temp+alu1+(lidx2*4))) = acc0;
}

otherwise the PHIs are childless.

Oh...this is a different issue. I don't think the PHIs should be in the store like that, but I see that they are childless after you make the change. It's almost a rendering issue...

yeah it's rendering dependant tbh. I gave this a try in cstyle, can bring it back tho
https://github.com/tinygrad/tinygrad/pull/3592/commits/bf25e935f8ecef78bbd07ed9fa10edc5331ef0a5#diff-9694d15708a36a57a7e1485db60b1b4b1d2fc1e7278bc87127a73efd0bfa7dc0R123

then you'd just pass the CAST's vins to store as a Tuple[UOp]:

I'm fine with leaving it in UOps, just after the child removal and it breaks the graph (what it is now). Adding UOps.END has a similar feel. It's fine to do the old cast removal manually


Manual Reset button for Flexispot E5B

Add a manual reset button that simulates a 5-second press of the down button. I am able to reset the 'ASR' error on Flexispot E5B using this method.
This is related to #24
All credit to @KrX3D for the idea.

Great idea! I will have a look on this when I return from holidays.

@jeakob would it make more sense to display this as a button? (single action). Or is there a reason you made it a switch?
We could make it a requirement that the desk is fully down, in order to execute this command?

@iMicknl it makes more sense for it be displayed as a button instead.
My desk sometimes requires a factory reset to fix the ASR error when it is fully up, so that requirement would prevent the reset.


Programmer

Description

Why

Possible Implementation & Open Questions

Is this something you're interested in working on?

[x] YES
[ ] NO

Please assign me as hacktoberfest contributor


add config option to allow sand dupe

This PR addresses #98 and adds a config option to toggle Paper's fix for duping sand with and end portal.
This is my first patch I've truly made myself, please tell me if I could've done something differently!

https://github.com/pl3xgaming/Purpur/issues/98#issuecomment-733149782


GraalVM 22.3.0 with Java 19

Hello, I hope ur good!
I was trying to get spring boot native working with the java-native-image buildpacks.
The actual problem is that the repo is missing the native-image-svm for java 19 like java 17 has:
Java 17 native
Is the support soon planning? Thanks you!
Here some more info on the problems:
Paketo Buildpack for BellSoft Liberica 9.10.0     [creator]       unable to find dependency     [creator]       no valid dependencies for native-image-svm, 19.0.1, and io.paketo.stacks.tiny in [(jdk, 8.0.352, [io.buildpacks.stacks.bionic io.paketo.stacks.tiny *]) (jre, 8.0.352, [io.buildpacks.stacks.bionic io.paketo.stacks.tiny *]) (jdk, 11.0.17, [io.buildpacks.stacks.bionic io.paketo.stacks.tiny *]) (jre, 11.0.17, [io.buildpacks.stacks.bionic io.paketo.stacks.tiny *]) (native-image-svm, 11.0.17, [io.buildpacks.stacks.bionic io.paketo.stacks.tiny *]) (jdk, 17.0.5, [io.buildpacks.stacks.bionic io.paketo.stacks.tiny *]) (jre, 17.0.5, [io.buildpacks.stacks.bionic io.paketo.stacks.tiny *]) (native-image-svm, 17.0.5, [io.buildpacks.stacks.bionic io.paketo.stacks.tiny *]) (jdk, 19.0.1, [io.buildpacks.stacks.bionic io.paketo.stacks.tiny *]) (jre, 19.0.1, [io.buildpacks.stacks.bionic io.paketo.stacks.tiny *])]     [creator]     ERROR: failed to build: exit status 1

This is an upstream limitation, Bellsoft doesn't have NIK available for Java 19 at the moment. If/when they make it available, we'll add it. For the time being, you'd need to stick with Java 11 or 17 for building native image apps.

I have the same issue.
the jdk 19 is really important due to the new features such as project loom (which able us to use virtual threads), so please consider adding support for it

Again, we cannot add this until it's released by Bellsoft. They don't have Java 19 support yet, so we cannot add it. As soon as they do, we'll add it. We should be able to add this for GraalVM Buildpack though. Give us a few days to get that updated and released, you can then use these instructions to use GraalVM instead of Bellsoft NIK.

A couple of notes here:

According to our vendor Bellsoft, they are not going to support Java 19 in NIK. It goes out of support in Jan 2023, so it's not really enough time. They are contemplating support for Java 20, and will definitely support Java 21 which is the next LTS. Their general strategy is to support the LTS versions, but they may support some non-LTS versions as well.
GraalVM has experimental support for Java 19, so I added this into the GraalVM buildpack. You can use that if you want to try out Java 19.

To build with GraalVM, see these instructions.
It's basically like this though:
pb my-cool-app -b paketo-buildpacks/graalvm -b urn:cnb:builder:paketo-buildpacks/java -e BP_NATIVE_IMAGE=true -e BP_JVM_VERSION=19

Or set the buildpack list in build.gradle or pom.xml if you're using the Spring Boot Gradle/Maven build tools.

@dmikusa
thanks for all of ur effort in this matter.
could u please represent a simple pom.xml or build.gradle in gist or somewhere else? it would be very very helpful

See the maven docs, the <buildpacks> tag is what you want. It takes a list of buildpacks. You would use the same two buildpacks I mentioned in my previous note. Example here.


How to add rotation function correctly?

I'm trying to add a rotation function to inspect. I tracked the source code until I saw a projection variable (ortho_projection) in the shader, I think this should be what I'm looking for.
https://github.com/andyborrell/imgui_tex_inspect/blob/1c9c0faa0cc0c62b565523a05ea4622bc0f0a307/backends/tex_inspect_opengl.cpp#L262-L268
https://github.com/andyborrell/imgui_tex_inspect/blob/1c9c0faa0cc0c62b565523a05ea4622bc0f0a307/backends/tex_inspect_opengl.cpp#L348-L360
But after I modified this matrix, I found that the image position will change when I drag the image window, and the image will also deviate when I drag the image content and when I scroll. I added a 30° rotation and the result is as shown in the figure.
https://user-images.githubusercontent.com/25359546/136394805-7e821700-dd58-4723-bc23-e1680234c06b.mp4
Rotation matrix is:
 sin(30°),  cos(60°),  0.00,  0.00,
-cos(60°),  sin(30°),  0.00,  0.00,
 0.00,      0.00,      1.00,  0.00,
 0.00,      0.00,      0.00,  1.00,

We have some other features such as "stylize image", "adjust color range", "merge pixel scaling". We plan to use it as a part of a gene viewing tool, would you accept if I add a shader chain or something to achieve these functions? Are these features suitable to add to inspect?

Thanks for your reply, I do feel now that there are some things that are not suitable to add and might be more suitable for another as a new project. I can talk about these features casually.
We often use a method called "bin scaling" to look at genes with very low expression in the wide range, where bin N means summing up the gene expression in the NxN region and presenting it as a pixel, which I call "merge pixel scaling" above.
Since gene expressions do not always fall in the regular color values (UInt8 or UInt16), we often need to normalize by the appropriate gamma value, and sometimes we need to ignore data with expressions greater than certain values. I call this "adjust color range" in the above.
At other times, we also need to generate a pseudo-color image of the expression, the process involves several separate steps and therefore may require a chain of shaders, which I call "stylize image" above.


make PWA icons maskable

Pull Request Template
Description
changed icon "purpose" to "any maskable" to make the PWA installable while enabling masking of the icon
Fixes # (issue)
Type of change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
I tested this on replit.com. the link to the preview is https://nunustudio.glitch128.repl.co/docs/editor/index.html. The install prompt will appear in the address bar automatically.
Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] Any dependent changes have been merged and published in downstream modules
[x] I have checked my code and corrected any misspellings

hello, is this going to get merged?

Yes, merged look okay to me.
Tahnks a lot once again!


glitched

woops, made pr on the wrong project lol, meant to do this on my fork - sorry for the spam


Increase cache duration for unversioned files to 7 days

Unversioned js files like https://cdn.ampproject.org/v0.js are cached for 5 minutes and deployed every seven days. We keep this cache age low so that we can quickly deploy a fix if we happen to ship any.
But in AMP-service worker we serve these files with a staleWhileRevalidate strategy which means if a user lands on a buggy version, he will still get the same buggy version next time but will be fixed on the third time.
We wanted to keep this number low initially so that we don't get stuck to this situation for a long time.
But this might not be the best UX as many web apps are visited after a gap of one day and might have to download the scripts before serving to the page even if the scripts did not change.
With current defaults this can happen upto ~6 times a week for the user. So a better UX would be  to the AMPs release schedule instead and let the scripts update in background.

cc: @kristoferbaxter


HKCategoryTypeIdentifier.mindfulSession not supported

It's not in the enum CategoryType and I think It's also not supported in the cocoapod.
I saw that you are updating both synchronously, maybe you can add some missing types when you get a chance.
Thanks
Alfred

Hi @schilken ,
Thank you very much that you spotted it. For sure I will add this in a next version

Hi @schilken ,
Thank you very much that you spotted it. For sure I will add this in a next version

Hi @schilken ,
I have expanded the original CategoryType Enum with cases which were not yet covered. It is in original CcocaPod https://github.com/VictorKachalov/HealthKitReporter
In order to use it in the Flutter plugin, you need to update the Podfile in your Flutter project.
You should see that the plugin has updated the dependency HealthKitReporter to version 1.3.3

Hi @schilken ,
I have expanded the original CategoryType Enum with cases which were not yet covered. It is in original CcocaPod https://github.com/VictorKachalov/HealthKitReporter
In order to use it in the Flutter plugin, you need to update the Podfile in your Flutter project.
You should see that the plugin has updated the dependency HealthKitReporter to version 1.3.3

Hi Victor,
thanks for your quick reaction. I saw there are now a lot of added entries in the pod. Even 'runnyNose' is supported by HealthKit - funny.
But the flutter plugin also needs a change, or do I get that wrong? Will you do that in next days?
Regards
Alfred

Hi Victor,
thanks for your quick reaction. I saw there are now a lot of added entries in the pod. Even 'runnyNose' is supported by HealthKit - funny.
But the flutter plugin also needs a change, or do I get that wrong? Will you do that in next days?
Regards
Alfred

Hi @schilken,
Yes true, you have the point, I totally forgot about this, made a change in Flutter code as well, please check the version 1.0.8
Regards,
Victor

Hi @schilken,
Yes true, you have the point, I totally forgot about this, made a change in Flutter code as well, please check the version 1.0.8
Regards,
Victor


generic proxy: add additional protocol/request/response options to control the behavior of proxy

Commit Message: generic proxy: add additional protocol/request/response options to control the behavior of proxy
Additional Description:
Add a interface to the codec factory to return custom ProtocolOptions. By this way, the custom codec could enable the stateful connection support in some special scenarios.
Refactor the ResponseDecoderCallback/RequestDecoderCallback to allow the custom codec to provide ExtendedOptions to the generic proxy. The ExtendedOptions could contains some metadata or control fields of request/response, such as request id, expect response or not, is heartbeat or normal request/response, shoud drain close or not, etc.
Extended ResponseDecoderCallback/RequestDecoderCallback to add writeToConnection interface. By this interface, the custom codec could write some raw binary as reply to the downstream/upstream connection when some specific data is received.
Risk Level: low. Contrib only update.
Testing: unit, integration.
Docs Changes: n/a.
Release Notes: n/a.
Platform Specific Features: n/a.

/retest

will wait for initial @soulxu review.
/wait

friendly ping @mattklein123

cc @mattklein123 could you give another stamp? I have merged the main and resolved the conflicts.


batch messages can be too large in new raft client implementation

Bug Report
New raft client implementation tends to batch more messages than before and we found it exceeded the max grpc messages in internal tests.
[2021/02/25 17:42:42.658 +08:00] [ERROR] [raft_client.rs:431] ["connection aborted"] [addr=] [receiver_err="Some(RpcFailure(RpcStatus { stat
us: 8-RESOURCE_EXHAUSTED, details: Some(\"Sent message larger than max (10705556 vs. 10485760)\") }))"] [sink_error=Some(RemoteStopped)] [store_id=7]

Calculating the message size using Message::compute_size should make it correct at the cost with extra CPU usage. We may also check what fields can be missed as even max batch size (128) is reached, there was about 1K extra size per message.

How about remove the size limit but only keep the batch count limit? So the total size of a batch won't be too exorbitant.

Perhaps we should bench what size is a reasonable good size.

I believe https://github.com/tikv/tikv/pull/8926 introduce the regression.

I prints the whole messages in a test case that can reproduce the issue frequently. It turns out it's because there are too many entries batch into one messages. A batch with 30 messages can contains about 45199. Each entry has about 11 extra bytes overhead, which means there are at least 497189 extra bytes, which is very close to the default extra buffer 524288.
Given now we have collected all bytes length from the pb message, now I feel confident to disable the size hard limit on the connection level to solve the issue for all. We still need a way to monitor the abnormal message size though.