Making docker local volume driver support all standard FS mount type

Making docker local volume driver support all FS mount type that standard /bin/mount support, such as glusterfs, cephfs, fused-based fs (sshfs/curlftpfs/..) and so on.
Options for each kind of fs-type are configurable via '--volume-opt'.
This fix is essential for docker-swarm cluster whose services run on different machines at different time.
Besides, to build a swarm cluster with persistent and secure volume storage, most of unstable 3rd-party plugin drivers are no longer needed.
Signed-off-by: CUI Wei<EMAIL_ADDRESS>

Any better ideas or suggestions?

A full example of sftp volume:
docker service create --name test --network ingress --replicas 1 --publish 5901:5901 --mount type=volume,volume-opt=o=workaround=all:o=reconnect:o=IdentityFile=${HOME}/.ssh/id_rsa:o=StrictHostKeyChecking=no,volume-opt=device=REMOTE-SSH-SERVER:/home,volume-opt=type=fuse.sshfs,source=sftp-test,target=/mount ghostplant/flashback

/cc @cpuguy83

The format from linux standard mount to swarm volume mount using local volume driver:
mount $DEVICE $TARGET -t $TYPE -o $OPT-1 -o $OPT-2 .. -o OPT-N
=====>
--mount type=volume,volume-opt=device=$DEVICE,volume-opt=type=$TYPE,volume-opt=o=$OPT-1:o=$OPT-2:o=..:o=$OPT-N,source=volume-1,target=/mount
Originally, Docker local volume driver supports NFS with the following format:
--mount type=volume,volume-opt=device=:/nfs-exports,volume-opt=type=nfs,volume-opt=o=addr=<IP_ADDRESS>,source=volume-1,target=/mount
With this fix, the following format also works:
--mount type=volume,volume-opt=device=<IP_ADDRESS>:/nfs-exports,volume-opt=type=nfs,source=volume-1,target=/mount
since we usually mount nfs using "mount -t nfs <IP_ADDRESS>:/nfs-exports /mount"

This is cool. My only concern is having different behavior on different hosts and that being relatively difficult to debug.
What do other people think?

I have no knowledge/background on this particular functionality.
Is there a way to better structure the code as not an error condition? This format kind of smells and I think people should have to explicitly opt-in to it with a different flag.

We discussed this in the maintainers meeting, and we see the feature in itself being useful, but share @cpuguy83's concerns about portability. Especially when in a Swarm; containers may end up on a node that supports (or not) a specific type.
Is a driver better? Or are we okay with adding this to the local volume driver?
ping @stevvooe for thoughts

@thaJeztah This shouldn't have anything to do with volumes. The design seems to confound mounts and volumes in an unfortunate manner.
This just seems like a raw mount but most of this can be accomplished with existing volume-opts or a volume driver.

I agree that options can be accomplished by volume-opts but the fact is that current source code use syscall.Mount which only supports very basic fs type to mount while mount command doesn't. So this fix is to cover all other standart mount type such as fuse and gluaterfs.
Volume driver can manage stardard fs type definitely but not necessary in most cases, and it won't simplify to deploy a machine since its HA and cross-platform is a concern.

I think the requirement is indeed needed. As for implementation, we can continue to discuss while I think the current solution makes the least change and effect.

@ghostplant You can already do this using o= (see https://docs.docker.com/engine/reference/commandline/volume_create/#/driver-specific-options). We may need to update the --mount docs to show this.

Yes, but syscall.mount only support tmpfs, insecure, bind and so on, for other types such as sftp, dockerd will report fatal error

In addition, this solution has been testes to fix both swarm mount and volumes created by docker local volume driver and both of the root cause is using syscall.mount

For the concerns from thaJeztah and cpuguy83, this fix only let more failture mounts to succeed and never let secceeded mounts to fail.
For those who don't want to use local driver but a self-designed one, they never get any differences with or without this fix

@MHBauer
Will extending a new flag like "--try-external-mount" to dockerd is better?

Please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "master" git@github.com:ghostplant/docker.git somewhere
$ cd somewhere
$ git rebase -i HEAD~842353822632
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

I proposed a second commit that is better than the first one.
The first one applies for fs-mount retry with global effect, while the second commit only make it work for local volume driver.

Please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "master" git@github.com:ghostplant/docker.git somewhere
$ cd somewhere
$ git rebase -i HEAD~842354417608
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

sudo apt-get install openssh-server
ssh-keygen -t rsa -P ''
mv ~/.ssh/id_rsa ~/.ssh/authorized_keys
sudo docker service create --name test --mount type=volume,volume-opt=type=fuse.sshfs,volume-opt=device=localhost:/home/,volume-opt=o=workaround=all:o=reconnect:o=StrictHostKeyChecking=no,source=test-home,target=/sftp-home

Any other concerns?

I don't think a flag to enable or disable this is a good idea.
I would sort of expect to do either 1 or the other... e.g. on package init do exec.LookPath("mount") if it exists we use mount, if not use the syscall. From what I've seen, the mount CLI can take the syscall forms just fine.
Still not 100% sold on adding this just due to portability and debug-ability... and I've actually experienced the mount CLI behaving differently on different distros even.... so all in all, really trying to justify this because I want this (simplifies a lot for users) but definitely some concerns.
Alternative would be to implement our own handlers in code for different types but I could see this getting unwieldy.
Stable types such as nfs may work, but others may be more difficult to maintain.
@stevvooe The main issue with using just the syscall is, for instance, when using something like mount -t glusterfs, this calls a helper mount.glusterfs, which in turn does a bunch of setup (kind of ridiculous how much gluster does specifically), and then does a syscall.... doing the syscall directly w/o the setup does not work.

I think it's nice for local driver to support NFS mount which is also the unique choice to support remote mount. I want to use it, but unfortunately NFS + swarm has security problem, so the privacy is a big  concern for me.
NFS have bad UAC control but using IP whitelist for access from client. If a swarm host can connect an external NFS, then applications inside swarm containers is also able to connect it, since the eth-docker0 packets would be NAT-ed. So in other words, if swam machine choose to work with NFS, it might suffer from successful connection from applications inside any of the containers.

I suggest supporting at least one kind of SECURE remote FS type for built-in (local) driver.

My concerns are still;

overloading the "local" driver with too much functionality. (Perhaps even some the current options already cross that line, but let's not add more without thinking through the consequences)
portability, especially in a swarm. your example already shows that software needs to be installed on the host for it to work correctly. In a swarm setup, I'd like to limit access to the host as much as possible. Differences between distros is similar; this option seems to have no abstraction whatsoever, so things can break. A container may run perfectly well on one host, then fail on another (when rescheduled)
volume driver plugins do have some similar issues, but are on our roadmap to be supported in swarm setups, i.e., swarm being able to install plugins on nodes in the swarm.

Even if we consider adding functionality, we should not do this without thinking

I see the point. But I think current local driver also have these problems:

As for local volume driver, the current behavior is already different among different OS such linux and windows;
I saw the implementation of syscall.Mount is also to invoke the mount API, and this might also behave different among different kernel and distros;
If we keep the current design of local driver, it will never be used in real production because it is either storage-unsafe, or without any sense (local fs-type, tmpfs, ..) from swarm. So is it better to drop this driver since its useless but only for build-test?

In other words, just as my point 3) has explained:
If this fix is not applied, local driver will never be used by anyone definitely,
while with this fix, partial users are potential use it for reducing costs to maintain 3rd-party driver.

I‘ll show another example, if docker swarm is installed to a linux os which doesn't containers nfsd.ko. It will also result in that a container may run perfectly well on one host, then fail on another (when rescheduled)

@ghostplant Just because you want to use networked storage doesn't mean that everyone wants to, or even that it is the only way, or the "right" way.
I get the use-case, I'm the one that added the support for mounts in the local driver to begin with... but glossing over compatability issues just because you want something is not the way to go.
I've seen the mount CLI behave differently between debian and non-debian before (in ways that broke our tests, no less).
Differences between platforms (windows, linux) are expected and that's fine. Differences between distros, not so much.... worse differences between versions of a distro.
You've also completely missed the point of my mention of NFS. It was not a suggestion to you to use NFS. Please re-read the comment.
No one has said "no" here (and that's why this is still open), but we need time to discuss and think through failure scenarios.
Thanks!

Also keep in mind, we do have plugins for this kind of thing.
Plugins in 1.13 have a great UX and distribution model, and can potentially even be scheduled onto a node as required by swarm.

Thank you @cpuguy83 , I am just trying my best to prove my option to be reasonable, and I also keep to think and want to find better solutions, since 2nd-commit is an example to further reduce the bad effect for external mount. Maybe I have more ideas on this issue afterwards.

I have a question on 3rd-party plugin driver like: calavera/docker-volume-glusterfs.
When volume-driver crashed, it seems to lose the value of reference count for each volume, thus resulting in further errors, unless persistent storage is needed and trasction is also needed (to keep the mounting counts absolutely equal to real mounting times). Anyone who know whether I am true about this? I guess it to be true so I am not willing to use plugin driver.

What about this solution?
If users want to mount advanced fs that only external mount command supports, the type field should start with a special charactor like '%' to indicate he doesn't want to use built-in mount. Thus, the usage would be like this:
--mount type=volume,volume-opt=type=%glusterfs,volume-opt=device=localhost:/,source=test,target=/mount
This won't have any effect on built-in mount such as:
--mount type=volume,volume-opt=type=tmpfs, .., source=test,target=/mount
So, the source code could get away with twice time mount retry:
if string.HasPrefix(v.opts.MountType, "@") {     cmd := []string{"-t", v.opts.MountType, v.opts.MountDevice, v.path}     .. concat options ..     return exec.Command("mount", cmd...).Run() } // using original way err := mount.Mount(v.opts.MountDevice, v.path, v.opts.MountType, mountOpts) return errors.Wrapf(err, "error while mounting volume with options: %s", v.opts)

Sorry, I'll use a branch to pull this request later.


Add script to setup end-user runtime environment (Trac #14)

Certain modifications to the user's environment may need to be made to run microdrop properly.  This may include installing packages, setting environment variables, etc.  On Linux, this should at least run the appropriate package manager commands to install Boost and GTK libraries.
This will likely need to be tied into the installer actions from #9.
Migrated from http://microfluidics.utoronto.ca/ticket/14
{
    "status": "closed",
    "changetime": "2014-04-17T19:39:01",
    "description": "Certain modifications to the user's environment may need to be made to run microdrop properly.  This may include installing packages, setting environment variables, etc.  On Linux, this should at least run the appropriate package manager commands to install Boost and GTK libraries.\n\nThis will likely need to be tied into the installer actions from #5.",
    "reporter": "cfobel",
    "cc": "",
    "resolution": "fixed",
    "_ts": "1397763541728826",
    "component": "microdrop",
    "summary": "Add script to setup end-user runtime environment",
    "priority": "major",
    "keywords": "",
    "version": "0.1",
    "time": "2011-10-09T16:23:36",
    "milestone": "Microdrop 1.0",
    "owner": "cfobel",
    "type": "enhancement"
}

Trac update at 20111009T16:25:08:

cfobel changed owner from "ryan" to "cfobel"
cfobel changed status from "new" to "assigned"

Trac update at 20111010T23:34:33: ryan changed summary from "Add script to end-user runtime environment" to "Add script to setup end-user runtime environment"

Trac update at 20111127T22:18:30:

cfobel commented:

As of f8383a9, scripts are included to generate either a Windows installer MSI and a Ubuntu .deb package.
For Windows, run 'scons' from within the root directory.
For Ubuntu, run 'python '.  Note: this requires [http://www.manatlan.com/page/py2deb py2deb].

cfobel changed resolution from "" to "fixed"
cfobel changed status from "assigned" to "closed"

Trac update at 20140417T19:39:01: ryan changed component from "core" to "microdrop"


turned off running of run_knmi_bm.sh benchmark with make check (for --enable-benchmark builds)

Fixes #1812
turned off running of run_knmi_bm.sh benchmark with make check (for --enable-benchmark builds). THis benchmarking script gets some files from Unitdata's ftp site, which presents problems and is slow. So this benchmark will no longer be run by make check.
@WardF it would be great if this could make it into the release...

Merging into v4.8.1-wellspring.wif.


feature: blink.cmp support for git-cmp

Did you check the docs?

[x] I have read all the LazyVim docs
[x] This is not a request for a new or existing extra (see above)

Is your feature request related to a problem? Please describe.
Since LazyVim is moving to blink.cmp, I would like it if possible for git-cmp (which is part of the git extra) support in blink.cmp
Describe the solution you'd like
{
        "saghen/blink.cmp",
        optional = true,
        dependencies = {
            { "petertriho/cmp-git", opts = {} },
            "saghen/blink.compat",
        },
        opts = {
            sources = {
                compat = { "git" },
                providers = {
                    git = {
                        score_offset = -1,
                    },
                },
            },
        },
},

ps. I've implemented the above but git-cmp seems to override the friendly snippets for ft=gitcommit.
For example, when I type "build" the snippet does not trigger. When I type ":build", the snippet triggers
Describe alternatives you've considered
.
Additional context
No response

I've tried it in my personal configuration but the completion is quite slow. I wouldn't recommend it even as an Extra. Only if someone wants to have it in his personal configuration.
And even without this, you still get completion from friendly-snippets. I stopped using it and I can use snippets just fine. Only thing is I don't get access to issues/prs. But that's something I can live with compared to the slowness it causes when typing.
This is just my personal opinion as a simple user.

try adding async=true and see if that makes it faster, or at least no longer blocks other completions.

I did add it, but it made a slight difference as to the menu not being blocked. The lag I'm noticing is the kind of when I press letters they don't appear immediately. I'm guessing it's because of some combination of cmp-git and blink-compat, because if I switch to nvim-cmp it behaves as expected.
So, I'm not saying to ditch cmp-git completely from LazyVim. I'm just saying that based on my experience it's a bad candidate for blink.cmp. It should be kept only with nvim-cmp as a default and users who want it can add it in their personal configuration.
Or if you don't notice this degradation in performance (which as I said it might be because of my old laptop) you can of course go ahead and I'll just disable it myself.

i've tried with async=true, git-cmp still overrides friendly-snippets. I build, feat, chore etc, only trigger with git-cmp trigger characters (:, #)

Not for me. I just type build and it shows the snippets immediately. However, I'm on blink main branch and there have been some changes/fixes with regard to snippets, so maybe that's the reason.


Kwxm/cost modelling update

Cost model update (SCP-3363)
This updates the cost model for the current version of the evaluator and adds some improvements to the modelling code.  Recall that the cost model has two components:

A costing function for each built-in function
A cost for each CEK machine step.

The costing functions for the built-in functions are obtained by running benchmarks and then using R to fit suitable models to the benchmark results.  The CEK machine step cost is obtained by running some other (very simple) benchmarks, subtracting the time spent executing builtins according to the costing functions obtained earlier, then fitting a linear model for the resulting times against the number of CEK compute steps, giving an average time per step. There's quite a lot of overhead in calling builtins, both in terms of work done in the machine and work carried out in the builtin machinery, including converting Plutus values into Haskell values to be fed to the underlying Haskell function, and also converting the result back to a Plutus value. Instead of trying to account for all this precisely we try to trade it off between the step cost and the costs for the builtins.   I'd been having some difficulty coming up with a sensible value for the CEK step costs, but then reading one of @effectfully's documents I realised that I'd failed to take account of the fact that arguments can be handled in several different ways, with varying costs (see #4456).  Changing the measure that we use for overhead of calling builtins gives us much better results, and that's done in this PR.
Comparison with existing cost model
I updated the cost model by generating new models for builtins using the current evaluator and the new measure of overhead, and then I ran the thing that tells us what the CEK step cost should be, getting a value of 19655.  This file contains figures comparing actual execution times of the validation benchmarks to predictions obtained using the new cost model.  The predictions are between 16% too low and 4.8% too high;  this is under-costing things a little so I bumped the CEK step cost up to 22000 and got the results in this file. This gives us predictions between about 10% under the actual execution time and 12% over, which seems pretty reasonable.  I've committed the model with the step cost of 22000 in this PR, but we could increase it a bit more if we want to be more conservative.  The benchmark that I use to infer the step cost is underestimating things a bit, but it's at least in the right ballpark. I think it needs more work to make the process fully automated, but it's not too bad at the moment; I had to do a little bit of manual adjustment (from 19655 to 20000), but only by about 10%.
I also checked predicted execution times for the nofib examples and got these figures for a step cost of 19655 and these ones for a step cost of 20000. We're overpredicting by 13-40% in the first case and 24-50% in the second; this is good because these things are much more expensive (by a factor of 15 to 500) than anything that we're likely to want to run on the chain, so it's better to err on the side of penalising them.
Changes in costs since Alonzo
We've improved the evaluator quite a bit since the Alonzo release, so how do the costs with the current evaluator compare?  I checked the costs of each of the validation scripts with the Anlonzo evaluator and both versions of the new cost model:  the results are here.  With the 22000 step cost we're between 10% and 20% cheaper in CPU costs and with the 19655 step cost we're between about 16% and 27% cheaper.  The memory costs are unchanged. Bear in mind that (a) the compiler's producing smaller code most of the time now, and (b) these figures only reflect execution time, not deserialisation time.  Deserialisation time has decreased a lot for a number of reasons recently, so total validation time should also have decreased.  Remember also that actual Ada validation charges are obtained from these costs via some conversion factor that's a ledger parameter and hence outside our control, so it's hard to know what effect this will have on real-life costs.
The forthcoming node release
A new node release is due soon and there's a Plutus branch here which will be used for that. That's still using the Alonzo cost model, and I haven't tried to update it yet.  However it doesn't contain some of our most recent changes: the current evaluator in master is about 10-15% faster than that (see these figures), I think due to using De Bruijn indices for variables, along with binary random access lists to store variable bindings. I benchmarked the validation scripts with the evaluator in that branch and compared them with the predictions given by the Alonzo cost model and got these results.  These aren't too bad: the predictions are between 7% below and 25% above the actual execution times.  I could try to regenerate the cost model for that branch using the new cost-model-generation code, but haven't done so yet (and it would require backporting the stuff in this branch, which might be a little tricky).

Bear in mind that (a) the compiler's producing smaller code most of the time now

That shouldn't matter, since the validation scripts are fixed, right?

Bear in mind that (a) the compiler's producing smaller code most of the time now

That shouldn't matter, since the validation scripts are fixed, right?

Yes.  That paragraph was trying to explain how things had improved (mostly for Andrew), so I was pointing out that there were further improvements that wouldn't show up in those figures.

Does reverting the change for data mean that the model for equalsData might be wrong now? not that it matters much, happy to just ignore it

Does reverting the change for data mean that the model for equalsData might be wrong now? not that it matters much, happy to just ignore it

I think that'll be a very small difference, so it should be OK. Anyway, I think I collected the data before making that change in the first place.


Prepare changelog for 1.36.0 release

What does this PR do?
Prepare changelog for 1.36.0 release

:green_heart: Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-01-27T08:21:23.562+0000

Duration: 1 min 11 sec

:grey_exclamation: Flaky test report
No test was executed to be analysed.
:robot: GitHub comments
Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

run benchmark tests : Run the benchmark tests.

run jdk compatibility tests : Run the JDK Compatibility tests.

run integration tests : Run the Agent Integration tests.

run end-to-end tests : Run the APM-ITs.

run windows tests : Build & tests on windows.

run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)


chore: Separate algebraic multiset lemmas

Resplit Algebra.BigOperators.Multiset.Basic and Algebra.BigOperators.Multiset.Lemmas into two new files:

Algebra.BigOperators.Group.Multiset for lemmas that require group-like structures (Monoid, Group, ...)
Algebra.BigOperators.Ring.Multiset for lemmas that require ring-like structures (MonoidWithZero, Ring, ...)

Add assert_not_exists Ring in the former. Once #12974 lands, it will be strenghtenable to assert_not_exists MonoidWithZero.

[x] depends on: #12836

According to a rough "auto-diff", no declarations were lost and the only added one is natAbs_of_isUnit.

Thanks :tada:
bors merge


Something went wrong activating IntelliCode support for Python

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version: 1.59.0
OS Version: Windows_NT x64 10.0.19043

Steps to Reproduce:

Open VS Code

it show the dialog

VS IntelliCode log:
Language server is set to "Default". Pylance is used as current language server.
Acquiring model
Acquiring model 'intellisense-members-lstm-pylance' for python
Querying IntelliCode service for available models.
vs-intellicode-python was passed a model: {}.
Cached model is up to date.
Activating Python extension
Language server extension command failed. Exception: Error: command 'python.intellicode.loadLanguageServerExtension' not found
at g._tryExecuteCommand (vscode-file://vscode-app/d:/programs/Microsoft%20VS%20Code/resources/app/out/vs/workbench/workbench.desktop.main.js:1525:3364)
at vscode-file://vscode-app/d:/programs/Microsoft%20VS%20Code/resources/app/out/vs/workbench/workbench.desktop.main.js:1525:3245
Error while activating Python: {}

If you check the Python extension in the extensions viewlet, does it say something like "reload required"?
I'm not sure whether the extension is outdated or disabled in your case.


tools-deps-alpha out of date

381 is the latest version, this brings auth support which is crucial for us.

381 is now available


Environment  setting for ventus

Hi Ventus GPGPGU team:
I'm setting development environment for Ventus.
I'm flowing this link: https://zhuanlan.zhihu.com/p/586445036
When i execute below command:
make init
I got below Errors:
git submodule update --init --recursive
Cloning into '/home/leonz/hw/ventus-gpgpu/dependencies/berkeley-hardfloat'...
<EMAIL_ADDRESS>Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:ucb-bar/berkeley-hardfloat.git' into submodule path '/home/leonz/hw/ventus-gpgpu/dependencies/berkeley-hardfloat' failed
Failed to clone 'dependencies/berkeley-hardfloat'. Retry scheduled
Cloning into '/home/leonz/hw/ventus-gpgpu/dependencies/cde'...
<EMAIL_ADDRESS>Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:chipsalliance/cde.git' into submodule path '/home/leonz/hw/ventus-gpgpu/dependencies/cde' failed
Failed to clone 'dependencies/cde'. Retry scheduled
Cloning into '/home/leonz/hw/ventus-gpgpu/dependencies/fpuv2'...
<EMAIL_ADDRESS>Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:liuxd17thu/fpuv2.git' into submodule path '/home/leonz/hw/ventus-gpgpu/dependencies/fpuv2' failed
Failed to clone 'dependencies/fpuv2'. Retry scheduled
Cloning into '/home/leonz/hw/ventus-gpgpu/dependencies/rocket-chip'...
<EMAIL_ADDRESS>Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:chipsalliance/rocket-chip.git' into submodule path '/home/leonz/hw/ventus-gpgpu/dependencies/rocket-chip' failed
Failed to clone 'dependencies/rocket-chip'. Retry scheduled
Cloning into '/home/leonz/hw/ventus-gpgpu/dependencies/rocket-chip-inclusive-cache'...
<EMAIL_ADDRESS>Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:chipsalliance/rocket-chip-inclusive-cache.git' into submodule path '/home/leonz/hw/ventus-gpgpu/dependencies/rocket-chip-inclusive-cache' failed
Failed to clone 'dependencies/rocket-chip-inclusive-cache'. Retry scheduled
Cloning into '/home/leonz/hw/ventus-gpgpu/dependencies/berkeley-hardfloat'...
<EMAIL_ADDRESS>Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:ucb-bar/berkeley-hardfloat.git' into submodule path '/home/leonz/hw/ventus-gpgpu/dependencies/berkeley-hardfloat' failed
Failed to clone 'dependencies/berkeley-hardfloat' a second time, aborting
make: *** [Makefile:2: init] Error 1

Do i need set some other things so i can clone above code?

Should set id_rsa.pub.
Closed by myself.

@zhangliang1997 ，sir, could you please tell me how to solve this problem, thanks in advance


Improve the warning of CLI

https://github.com/ruby/ruby/actions/runs/7885458586/job/21516643427?pr=9777#step:10:83
When executing the launchable record build command, we don't know what happens. We need to improve it.
WARNING: Failed to process a change to a file
java.io.UncheckedIOException: org.eclipse.jgit.errors.MissingObjectException: Missing blob c7f57385b32005caa5512d52f72135a5dc1df4de
	at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:38)
	at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transform(CommitGraphCollector.java:410)
	at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transfer(CommitGraphCollector.java:343)
	at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:195)
	at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:129)
	at com.launchableinc.ingest.commits.CommitIngester.run(CommitIngester.java:149)
	at com.launchableinc.ingest.commits.CommitIngester.main(CommitIngester.java:77)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing blob c7f57385b32005caa5512d52f72135a5dc1df4de
	at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:137)
	at org.eclipse.jgit.diff.ContentSource$ObjectReaderSource.open(ContentSource.java:112)
	at org.eclipse.jgit.diff.ContentSource$Pair.open(ContentSource.java:257)
	at org.eclipse.jgit.diff.DiffFormatter.open(DiffFormatter.java:1074)
	at org.eclipse.jgit.diff.DiffFormatter.createFormatResult(DiffFormatter.java:1001)
	at org.eclipse.jgit.diff.DiffFormatter.format(DiffFormatter.java:704)
	at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:36)
	... 6 more

Feb 13, 2024 11:05:02 AM com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository transform
WARNING: Failed to process a change to a file
java.io.UncheckedIOException: org.eclipse.jgit.errors.MissingObjectException: Missing blob 64332a8f53ac12e252641da2dac966e5c5344e1e
	at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:38)
	at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transform(CommitGraphCollector.java:410)
	at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transfer(CommitGraphCollector.java:343)
	at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:195)
	at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:129)
	at com.launchableinc.ingest.commits.CommitIngester.run(CommitIngester.java:149)
	at com.launchableinc.ingest.commits.CommitIngester.main(CommitIngester.java:77)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing blob 64332a8f53ac12e252641da2dac966e5c5344e1e
	at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:137)
	at org.eclipse.jgit.diff.ContentSource$ObjectReaderSource.open(ContentSource.java:112)
	at org.eclipse.jgit.diff.ContentSource$Pair.open(ContentSource.java:257)
	at org.eclipse.jgit.diff.DiffFormatter.open(DiffFormatter.java:1074)
	at org.eclipse.jgit.diff.DiffFormatter.createFormatResult(DiffFormatter.java:1001)
	at org.eclipse.jgit.diff.DiffFormatter.format(DiffFormatter.java:704)
	at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:36)
	... 6 more

Feb 13, 2024 11:05:02 AM com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository transform
WARNING: Failed to process a change to a file
java.io.UncheckedIOException: org.eclipse.jgit.errors.MissingObjectException: Missing blob f2244ec20aa38532faa50243abe821486ca07244
	at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:38)
	at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transform(CommitGraphCollector.java:410)
	at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transfer(CommitGraphCollector.java:343)
	at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:195)
	at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:129)
	at com.launchableinc.ingest.commits.CommitIngester.run(CommitIngester.java:149)
	at com.launchableinc.ingest.commits.CommitIngester.main(CommitIngester.java:77)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing blob f2244ec20aa38532faa50243abe821486ca07244
	at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:137)
	at org.eclipse.jgit.diff.ContentSource$ObjectReaderSource.open(ContentSource.java:112)
	at org.eclipse.jgit.diff.ContentSource$Pair.open(ContentSource.java:257)
	at org.eclipse.jgit.diff.DiffFormatter.open(DiffFormatter.java:1074)
	at org.eclipse.jgit.diff.DiffFormatter.createFormatResult(DiffFormatter.java:1001)
	at org.eclipse.jgit.diff.DiffFormatter.format(DiffFormatter.java:704)
	at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:36)
	... 6 more

Feb 13, 2024 11:05:02 AM com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository transform
WARNING: Failed to process a change to a file
java.io.UncheckedIOException: org.eclipse.jgit.errors.MissingObjectException: Missing blob 57306673cd3106a3fcbf204aa131dfced421bb1c
	at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:38)
	at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transform(CommitGraphCollector.java:410)
	at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transfer(CommitGraphCollector.java:343)
	at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:195)
	at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:129)
	at com.launchableinc.ingest.commits.CommitIngester.run(CommitIngester.java:149)
	at com.launchableinc.ingest.commits.CommitIngester.main(CommitIngester.java:77)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing blob 57306673cd3106a3fcbf204aa131dfced421bb1c
	at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:137)
	at org.eclipse.jgit.diff.ContentSource$ObjectReaderSource.open(ContentSource.java:112)
	at org.eclipse.jgit.diff.ContentSource$Pair.open(ContentSource.java:257)
	at org.eclipse.jgit.diff.DiffFormatter.open(DiffFormatter.java:1074)
	at org.eclipse.jgit.diff.DiffFormatter.createFormatResult(DiffFormatter.java:1001)
	at org.eclipse.jgit.diff.DiffFormatter.format(DiffFormatter.java:704)
	at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:36)
	... 6 more

Feb 13, 2024 11:05:02 AM com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository transform
WARNING: Failed to process a change to a file
java.io.UncheckedIOException: org.eclipse.jgit.errors.MissingObjectException: Missing blob 20630e58097[83](https://github.com/ruby/ruby/actions/runs/7885458586/job/21516643427?pr=9777#step:10:84)c5e43e42c436e80e245e6bdb518
	at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:38)
	at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transform(CommitGraphCollector.java:410)
	at com.launchableinc.ingest.commits.CommitGraphCollector$ByRepository.transfer(CommitGraphCollector.java:343)
	at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:195)
	at com.launchableinc.ingest.commits.CommitGraphCollector.transfer(CommitGraphCollector.java:129)
	at com.launchableinc.ingest.commits.CommitIngester.run(CommitIngester.java:149)
	at com.launchableinc.ingest.commits.CommitIngester.main(CommitIngester.java:77)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing blob 20630e5809783c5e43e42c436e80e245e6bdb518
	at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:137)
	at org.eclipse.jgit.diff.ContentSource$ObjectReaderSource.open(ContentSource.java:112)
	at org.eclipse.jgit.diff.ContentSource$Pair.open(ContentSource.java:257)
	at org.eclipse.jgit.diff.DiffFormatter.open(DiffFormatter.java:1074)
	at org.eclipse.jgit.diff.DiffFormatter.createFormatResult(DiffFormatter.java:1001)
	at org.eclipse.jgit.diff.DiffFormatter.format(DiffFormatter.java:704)
	at com.launchableinc.ingest.commits.CountingDiffFormatter.process(CountingDiffFormatter.java:36)
	... 6 more

Launchable recorded 1 commit from repository /Users/runner/work/ruby/ruby/build
Launchable recorded build 16f8f23f114b11[85](https://github.com/ruby/ruby/actions/runs/7885458586/job/21516643427?pr=9777#step:10:86)70347e6b82e1b980c8a2ab7c to workspace ruby/ruby with commits from 1 repository:

| Name   | Path   | HEAD Commit                              |
|--------|--------|------------------------------------------|
| .      | .      | 60cc647c1983035c7998d1a6fbfd1c832aea3e[96](https://github.com/ruby/ruby/actions/runs/7885458586/job/21516643427?pr=9777#step:10:97) |

Visit https://app.launchableinc.com/organizations/ruby/workspaces/ruby/data/builds/2505254 to view this build and its test sessions

@kohsuke
Any ideas? I'm not familiar with this part.

From the log, I see that git fetch was run with --filter=blob:none
According to https://stackoverflow.com/questions/49917616/what-is-the-git-clone-filter-options-syntax what this means is that BLOBs are not retrieved during this git-fetch. Instead, later git-clone appear to retrieve the needed objects.
That explains why BLOBs are reported as missing. The next question then is what can we do to retrieve those missing BLOBs as we run into them.

Let's also look for workaround.

https://github.com/actions/checkout/blob/b32f140b0c872d58512e0a66172253c302617b90/src/git-source-provider.ts#L163 sets --filter=blob:none which suppresses the retrieval of BLOBs
So I think if we set the filter to empty by adding filter: null, it should suppress that.
Combined with depth: 0 it's going to considerably increase the clone time. I think more reasonable depth setting should be OK. Something like depth: 10?


Severe performance degradation

FAQ

[X] I have checked the FAQ and it didn't resolve my problem.

Issues

[X] I have checked existing issues and there are no issues with the same problem.

Neovim Version
0.9.4
Dev Version?

[X] I am using a stable Neovim release version, or if I am using a dev version of Neovim I have confirmed that my issue is reproducible on a stable version.

Operating System
MacOs
Minimal Config
Default config, without any changes.

Steps to Reproduce
This is something that i have observed recently, i am mostly using null-ls to attach custom code actions through the generator api. And have just two formatters installed (active at most, stylua and clang format), besides a couple of formatters. However, recently on my trashy m1 at work i started noticing severe performance hits with two very hot paths - didChange and autocompletion. Whenever the internal neovim lsp client would trigger the text change notify event, to notify the server of the changes in the buffer, and whenever the autocompletion engine (nvim-cmp in my case) would request candidates i noticed a severe lag while having null-ls enabled. Went in and disabled it in my config, and after that the issue completely vanished.
The issue is very reproducible when we do changes in the buffer with a macro, say pasting over a word is what i mostly experimented with, over a few lines (dozen) in a buffer that is no bigger than 200 lines, the macro would be extremely laggy and slow, as far as completion goes, the moment you go into insert mode and start typing there is a significant lag / delay once i reach the minimal keyword count i have configured for nvim-cmp (which is 3 at the moment).
I tried disabling the primary language servers i tested with (lua and jdtls) and the issue persisted, only when i killed null-ls would i actually see sane performance, no stuttering, etc
Something to note, i increased the debounce which would help will avoiding the didChange spam , to something like 2seconds, and that fixed the issue while editing in normal mode, but nvim-cmp would still chug and stutter.
Reproducibility Check

[X] I confirm that my minimal config is based on the minimal_init.lua template and that my issue is reproducible by running nvim --clean -u minimal_init.lua and following the steps above.

Expected Behavior
No perceived lag, especially when there are no configured formatters or linters at all.
Actual Behavior
Severe drop in performance during the two most hot paths, which modifying the buffer and while auto completing.
Debug Log
None
Help
Yes, but I don't know how to start. I would need guidance
Implementation Help
No response
Requirements

[X] I have read and followed the instructions above and understand that my issue will be closed if I did not provide the required information.

After some investigation, i think the debug config option, is what nukes the performance so hard, but given the fact that i have pretty much no linters or formatters installed, i wonder why would it be so impactful, even with debug turned on, does it do something else besides logging the requests ?

This is in the docs:

As with LSP logging, debug mode will slow down Neovim. Make sure to disable
the option after you've collected the information you're looking for.


support ZUGFeRD-Extensions a.k.a. additional data

Andreas Starke had already published a way how ZUGFeRD "extensions"/additional data can accomodate special needs of industry branches (http://4s4u.de/additional_data/), e.g. if you need to specify the mileage in a logistics your invoice can say "one washing machne, one dry cleaner" and the additional data can say that the delivery truck transported the washing machine 500km but the dry cleaner only 25 because it was picked up elsewhere. There is a python library and a command line tool to embed/extract this files, the idea is that the mustangproject library and command line tool can do that too and ZUV can validate this additional data.

How you can help:

There is a generic schema(sic! not schematron) file for ZUGFeRD extensions and the extensions might get custom, extension specific schema files. We need a way to check schema files in the first place. For schematron we used https://github.com/phax/ph-schematron but I don't know if that supports schema files, if not we need an according library, which can load from our resources (we deploy in a single jar file) preferrably via maven central or other maven repos.
Add that, add the schema file from Andreas' website to the resources, use the most recent, unreleased version of mustangproject
git pull mustangproject, "mvn clean package" have maven cache the local file as 1.6.1
mvn install:install-file -Dfile=mustang-1.6.1-SNAPSHOT.jar -DgroupId=org.mustangproject.ZUGFeRD -DartifactId=mustang -Dversion=1.6.1 -Dpackaging=jar -DgeneratePom=true then in ZUV in

org.mustangproject.ZUGFeRD
switch version to
1.6.1
"mvn clean install" and "mvn eclipse:eclipse") to check if there are extensions and get the value with sth like mustangprojects'getAdditionalDataXML()
invent some section for the report additional to the existing  and  and probably BELOW , document that syntax, have the schema check run in case additional data is there, bundle it all together and send me a PR.

Andreas contributed a patch which was released with 0.8.0, if additional data is there it is checked and if it is not valid an error is thrown, unfortunately there is no notice that it's OK if there is no error


GPCCA Mode

This is not actually meant to be merged. It's just a very hacky preliminary way to use GPCCA.

This is an outdated reference and everything from here should be implemented. Closing.

agreed!


Individual URLs for custom tabs in Documentation landing page

Feature Suggestion
Currently Documentation landing page does not have URL routing to individual custom tabs, unlike the tabs in Explore or other main pages. Would be good to allow the custom config for the landing page also cause this to be set up.
Context
This is useful for bookmarking or passing links to individual tabs around.

Is this techdocs or backstage.io/docs? Not sure I understand this ticket?

@adrianke77 Can you include a screenshot so we can more fully understand what you mean with this enhancement.

Currently, if base-url/docs redirects to the Documentation page, selecting individual tabs does not add additional pathing to the URL. This makes it not possible for individual tabs to be bookmarked etc

For the Explore page, base-url/explore redirects to the page, but there is also a path suffix unique to the current tab, here being '/tools':

So the feature request is to have a similar indication in the URL of what tab the user is on for the Documentation page. This could be a suffix specified for each tab in the custom config for the Documentation landing page.

Aha I see. It looks to me like those tabs on the Docs page are specific to your Backstage instance. We don't have them internally and they are not on our demo installation. Not to say that it's not a good idea though of course - the tabs and the URL suffix. Feel free to have a go to implement this yourself.

Yes the tabs are custom to our Backstage instance. It was set up using the custom documentation page tab capability now available in Backstage.
The intent was to highlight that the way of setting up easily setting up custom tabs at the moment doesn't also map URLs to them and it would be nice if it did.


2组 房价预测与mnist

2组 房价预测与mnist

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Legends' Quest performance

My FPS dropped to around 20 during the segment in which you have to obtain crystal shards from the skeletons. Back to 50 after I formed the heart.
Found this spammed in my logs, I am not certain if this relates to the issue.
https://pastebin.com/Sj352B24

Should hopefully be fixed now, if anyone else encounters this issue I'm happy to re-open and investigate further.


Can't the defineexpose event be tested ?

I use vue-composition-api in my program
like
setup(){   const fn = ()=>{}   defineExpose({     fn    }); }
Can't I test this function in my test file?
I check wrapper.vm，its not on vm。
So is there a way to test this function?

Are you using Vue 2 or Vue 3?

Thank you for your reply. I have solved this problem. In fact, VM has exposed methods and properties, although I didn't see them in the console
I'm sorry for the delay

Thank you for your reply. I have solved this problem. In fact, VM has exposed methods and properties, although I didn't see them in the console
I'm sorry for the delay

@yangliguo7 Hello, May I know how did you solve the issue? cause I'm struggling with same issue

In general, I'd recommend testing from by interacting with the template - buttons, that kind of thing - as opposed to directly calling methods on vm.
If you can share an example of what you want to do @MahsaGoudarzi, I can give you some suggestions.

@lmiller1990 @yangliguo7 Hey, thank you so much, actually using defineExpose() solved my issue, but now I have another confusion, I have a ref variable like let showMe = ref(false) and I defined it into defineExpose({test}) as well,
when  I changed value of this ref variable in my test file the expected changed on html doesn't happen. for example this is my vue component:  <template>show me </template>  and in my test file when I changed it by wrapper.vm.showMe.value = true expect(wrapper.find("span").exists()).toBe(true) it returns falsy value .. Do you have any idea why it doesn't affect on my template?

maybe use flushPromises to solve this problem.  dom update is async
App.vue
<template>
  show me
</template>

test.spec.js
import Test from<EMAIL_ADDRESS>import { mount ,flushPromises} from '@vue/test-utils';

describe('test App.vue', () => {
  test('change showMe should show span', async () => {
    const wrapper = await mount(Test);

    expect(wrapper.find("span").exists()).toBe(false)

    wrapper.vm.showMe = true;

    await flushPromises()  // this way

    expect(wrapper.find("span").exists()).toBe(true)
  });
});

is this what you expected ?
and as lmiller1990  said
u shouldn't change the value by vm
data changes should be driven by events

Avoid using vm where you can - drive you test using trigger, etc. This is how a user will interact and how Vue expects things to be modified.
If you can share a minimal reproduction I can clone and run, I can make suggestions. I do not have time to try and reproduce your issue on my own. Thanks!

Hi, I have the same issue. I can't use trigger because there is no UI element inside the component itself that would trigger the exposed method (the whole reason it's exposed is because a potential parent component would call it), and I want to test the component on its own, which includes the exposed method.
my-component.vue

<template>

        exposed method has been triggered: {{ secretInternalValue }}

</template>

my-component.spec.js
import MyComponent from './my-component.vue';

describe('MyComponent', () => {
    it('changes secretInternalValue when changeSecret() is called', () => {
        wrapper = mount(MyComponent);
        wrapper.vm.changeSecret();
        expect(wrapper.text).toContain('exposed method has been triggered: true');
    });
});

Hi!

the whole reason it's exposed is because a potential parent component would call it

Not a direct solution to your issue, but can you explain more about why you are doing this? It seems like an antipattern in general for a parent to be able to "reach" inside a child and call a method.
Another way you might be able to test it is building the same scenario you are describing: a parent calling the method. Can you just do mount on the parent component, then interact with that however your user would to trigger the function?
Some more information about your scenario would be useful.

Not a direct solution to your issue, but can you explain more about why you are doing this? It seems like an antipattern in general for a parent to be able to "reach" inside a child and call a method.

It's certainly not the ideal way of doing things, but sometimes it's necessary. Otherwise there would be no reason for defineExpose to exist at all.
In my case it's an image component that creates a srcset from a single image, and it has lazy loading functionality that can get triggered by the parent. The component itself does not know the context of when it should start loading the images, and it has to be an irreversible state change, that's why it's a method and not a prop.

Another way you might be able to test it is building the same scenario you are describing: a parent calling the method. Can you just do mount on the parent component, then interact with that however your user would to trigger the function?

The parent component is not in the same repository, and it seems a bit silly to create one just for the test. Exposing methods to the parent so that they can be called from outside is a feature built into Vue, so I expected the Vue test utils to let me test it directly.

@itchwoot did you arrive to any valid solution to successfully test this? I'm facing exactly the same scenario and I would like to stick with vue-testing-library as my testing library, even though some times its approach makes difficult to test these corner cases.

@itchwoot did you arrive to any valid solution to successfully test this?

Sadly no, I ended up just not testing this functionality :(


Allow keyboard control for FancyZones win+` dialog window

You can press win-` to pick the layout that applies to the screen. That's great. Sadly, once the dialog shows, you need to use mouse to pick the layout. Can you please add support for cursor keys so you don't need to switch away from the keyboard?
Thanks!

would hot keys for just hot swapping work?

Probably, although this requires sacrificing more than one unique global shortcuts to powerzones. win-` brings up a dialog, so you can just support cursor keys there and avoid having to find yet another free keybinding :)

Yes, this!! I have a few different layouts that I switch between depending on my current task and having to move away from the keyboard after pulling up the dialog window is a real pain.

This is an accessibility issue as well.


Ask for advice about merging match_keypoints into hloc sfm pipeline

I've tried using superpoint and also dedode as keypoints extractor and then match with RoMa but can't get a proper result. How could I improve the result?
The basic idea is that I firstly extract all keypoints and then normalize the keypoints into range [-1, 1].
def to_reference_coord(kpt, W, H):
kpts = torch.stack((kpt[..., 0]/(W/2) -1, kpt[..., 1]/(H/2) -1), axis=-1)
return kpts
And then using roma model to calculate the warp and certainty. then using kpts, warp and certainty as input of match_keypoints. After getting the results I also used ransac to filter some of the matches.
I'm not able to reconstruct the model correctly by this way.

Thus I check the visual result of matches:

Anybody could give me some advice to improve the pipeline? Thanks!

By the way, this magic pickup truck is really hard to reconstruct. LOL

I would avoid matching when the roma confidence is very low. In the provided images I would guess that is the case.

I would avoid matching when the roma confidence is very low. In the provided images I would guess that is the case.

Actually RoMa reaches pretty accurate result by RegressionMatcher.match. Thank you for your reply, I will firstly have a review on my code.


Add a thundering herd test for various polling mechanisms.

This is a demonstration of Linux behavior with various configurations of multiple threads polling when a single file descriptor becomes readable.
Spoiler: in almost all configurations all threads wake up. The exceptions are:
epoll_wait on a shared epoll set, if the fd was added EPOLLET or EPOLLONESHOT
epoll_wait on separate epoll sets directly monitoring the fd, if added EPOLLEXCLUSIVE
We might expect epoll_wait of separate epoll sets monitoring (EPOLLEXCLUSIVE) the shared epoll set to work, but epoll_ctl rejects EPOLLEXCLUSIVE on an epoll set.

Do we still want to merge that one ? It'd require some updates first. And maybe someone assigned for review :)

I'm going to kill this PR for now -- at some point I'll revive it with the branch that includes signals/signalfd.


Update documentation with Cognito

Can be added to README.md or Gitbook (pending)

Documentation was done in GitBook, issue can be closed


Installation issue  with univ of washington

Running setup.py develop for align
ERROR: Command errored out with exit status 1:
command: ALIGN_installation_trial2/ALIGN-public/general/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'ALIGN_installation_trial2/ALIGN-public/setup.py'"'"'; file='"'"ALIGN_installation_trial2/ALIGN-public/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps
cwd: ALIGN_installation_trial2/ALIGN-public/
Complete output (15 lines):
/tmp/pip-build-env-0meokkn7/overlay/lib/python3.7/site-packages/cmake/data/bin/cmake: /lib64/libc.so.6: version GLIBC_2.14' not found (required by /usr/um/gcc-7.1.0/lib64/libstdc++.so.6)     /tmp/pip-build-env-0meokkn7/overlay/lib/python3.7/site-packages/cmake/data/bin/cmake: /lib64/libc.so.6: version GLIBC_2.16' not found (required by /usr/um/gcc-7.1.0/lib64/libstdc++.so.6)
/tmp/pip-build-env-0meokkn7/overlay/lib/python3.7/site-packages/cmake/data/bin/cmake: /lib64/libc.so.6: version GLIBC_2.17' not found (required by /usr/um/gcc-7.1.0/lib64/libstdc++.so.6)     /tmp/pip-build-env-0meokkn7/overlay/lib/python3.7/site-packages/cmake/data/bin/cmake: /lib64/libc.so.6: version GLIBC_2.14' not found (required by /usr/um/gcc-7.1.0/lib64/libgcc_s.so.1)
/usr/um/python-3.7/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'cmake_process_manifest_hook'
warnings.warn(msg)
File "/tmp/pip-build-env-0meokkn7/overlay/lib/python3.7/site-packages/skbuild/setuptools_wrap.py", line 560, in setup
cmkr = cmaker.CMaker(cmake_executable)
File "/tmp/pip-build-env-0meokkn7/overlay/lib/python3.7/site-packages/skbuild/cmaker.py", line 95, in init
self.cmake_version = get_cmake_version(self.cmake_executable)
File "/tmp/pip-build-env-0meokkn7/overlay/lib/python3.7/site-packages/skbuild/cmaker.py", line 82, in get_cmake_version
"Problem with the CMake installation, aborting build. CMake executable is %s" % cmake_executable)
Problem with the CMake installation, aborting build. CMake executable is cmake
Traceback (most recent call last):
----------------------------------------

ERROR: Command errored out with exit status 1: ALIGN_installation_trial2/ALIGN-public/general/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'ALIGN_installation_trial2/ALIGN-public/setup.py'"'"'; file='"'"'ALIGN_installation_trial2/ALIGN-public/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

cmake/2.8.8(default)
python3.7

Error seems to indicate that the libstdc++.so.6 has been compiled with a newer glibc (that is not provided by the default /lib64/libc.so.6).
Few options at this stage:

Upgrade the system GLIBC using apt / yum etc (probably not practical if this is a shared server)
Modify LD_LIBRARY_PATH to point to a more recent libc if one exists already
Don't try to install from source at all but use pre-built binaries

For option 3, you will need to send them the tar archive generated by publish-wheel on master. Latest generated wheel directory appears to be here. Instructions for wheel based installation can be found in the last comment in #572

My availability is extremely limited until 5/24. Please tag @stevenmburns @soneryaldiz if you need additional help.

Thanks Parijat.
option 2 worked by setting LD_LIBRARY_PATH: setenv LD_LIBRARY_PATH /usr/lib/libc.so.6:$LD_LIBRARY_PATH
option 3 works but if I update python files locally it does not get reflected. Reinstallation also does not work in this case as python files are also picked from wheel. To make it work we had to overwrite files in the installed directory.

Yes installation from wheel does not allow development mode. Glad it got resolved!


Remove numba dependency/extra for quaternion.

The docs to numpy-quaterion
(https://quaternion.readthedocs.io/en/latest) state:

However, certain advanced functions in this package (including squad,
mean_rotor_in_intrinsic_metric, integrate_angular_velocity, and related
functions) require scipy and can automatically use numba.

Thus I think it is save to remove this dependency extra that requires a
LLVM ≤ 11.

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@5822b9b). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main      #11   +/-   ##
=======================================
  Coverage        ?   91.68%
=======================================
  Files           ?       15
  Lines           ?      445
  Branches        ?        0
=======================================
  Hits            ?      408
  Misses          ?       37
  Partials        ?        0

Flag
Coverage Δ

pytest
91.68% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Thanks for the heads up, I will try and look at this today.

Fixed by #12


Generate features directly from TBD Cilium configuration schema

How to categorize this issue?
/area networking
/kind enhancement
/priority 4
/assign @briantopping
What would you like to be added:
Generate configuration option passthrough from some type of configuration schema such as https://github.com/cilium/cilium/blob/master/api/v1/openapi.yaml. Some build pattern needs to be defined so older versions of the configuration may be properly matched.
Why is this needed:
Cilium and gardener-extension-networking-cilium have a reasonable challenge with configuration matching between the two. Cilium configuration is in regular flux and the Gardener extension often cannot pass through new features that might be defined without changes to the container.

Some implementation ideas, in order of increasing disruption to the Cilium team. I'm kind of partial to the flags validator idea, second in the list:

We could just send all flags unvalidated to Cilium and let it crash if it doesn’t like them. This means taking out the existing flags parse logic.
Some form of flags validator on the Cilium entry point could basically send all the flags to Cilium and it could validate them. If the parse was bad, the existing cilium parser would reject it with output to stderr and and that would be passed through the provider to the user.
Various structured exports from Cilium are possible:

Use something like https://github.com/octago/sflags. The struct arguments would be reusable in the provider by pulling across the annotated structure from Cilium into the provider and recompiling it
If the flags structure in Cilium was generated by a well-known schema that could also generate the structure in the provider. Then the shared content was more abstract than just an annotated Go structure.

High level comment: I'm unsure whether that's ("Generate features directly from TBD Cilium configuration schema") really a goal we should strive for. Typically, we consciously decide which of the configuration options are exposed via the APIs. This is since not all of the configuration switches (1) make sense for end-users to configure, or worse, (2) could potentially harm the system's robustness. I don't know how critical this is for Cilium and whether this also applies here, but for the Shoot API, for example, we - by far - do not expose all the configuration options for the Kubernetes components.
Obviously, this has the down-side that exposing each configuration flag means extra effort, OTOH, it also means that the risk for end-users is minimized.

Thanks Rafael, your additional issues are very helpful for consideration.
The primary use case I'm seeing is users can't configure something they need without code changes to the provider to pass them through. A savvy user would have to know how to push a PR and then wait for a release if they had an organizational requirement in that regard. In less ideal situations, they would be running from a local fork or simply choose the Calico provider for lack of options. It becomes something of a self-fulfilling outcome that the Cilium provider never gets a following with advanced users -- exactly who it's best for.
I had started a PR for accepting ConfigMap as providerConfig, but your response helped me realize that this should coexist with the current config code rather than replace it. This gives users a backdoor if they need it and could be removed at a later date once config mapping from the provider to Cilium is better featured. Does that sound agreeable?

I can't judge since I'm not very familiar with Cilium (actually, not at all), hence I'm not sure how "risky" this is w.r.t. user experience and damage that can be done to the users themselves. OTOH, what you propose sounds familiar to what is possible with CoreDNS today, though this depends on a very stable "API" as end-users cannot control the CoreDNS/Cilium versions themselves.
Anyways, I don't want to either block nor push this topic, I just added a few observations when reading the issue description. It's probably better if the @gardener/gardener-extension-networking-cilium-maintainers folks add their opinion and give some guidance here.

Just to add my two cents:
@briantopping could you please provide a couple of examples where you would have needed such a functionality?
In general, I agree with what Rafael already wrote. It is rather a non-goal for us to expose each and every little option/switch as this usually makes it more complicated for most users/operators to understand what they actually need to set. However, if there is a useful scenario we can expose almost everything on a case-by-case basis.
Rafael's CoreDNS example is good that it also shows the limits of this approach. You cannot configure everything you might want to using this approach, but it may serve well for certain scenarios.

Hello @ScheererJ, thanks for your feedback. The first time I needed this functionality, I went through the PR process in https://github.com/gardener/gardener-extension-networking-cilium/pull/46. This time, it is for a feature flag that might not even make it to production in supporting https://github.com/cilium/cilium/issues/9207. As it stands, I am unaware how to support this flag without code changes and started asking myself where improvements could be made. Maybe I am missing something.
I completely agree it's a non-goal to manually support every switch, and they change over time in a project moving as fast as Cilium (or may never reach production as above). The original proposal was to make support automatic. The discussion has been valuable to realize validation may be harmful. Your questions expose requirements for security of a global delegated deployment for which RBAC on the config may be the only practical solution.
The current implementation of the provider does no validation of any switches, it just blindly passes them through to Cilium. Is there intent to add validation to avoid harmful combinations of configuration? Such an endeavor seems computationally infeasible unless the number of allowed switches was extremely small. And then we're back to the original problem of supporting smaller environments with a tiny attack surface and a dependency on bleeding-edge switches.
My k8s CNI experience is with the spec and Cilium. I assume Calico must have extensive permissions like Cilium for route maintenance. It seems plausible that any CNI instance could advertise <IP_ADDRESS>/0 and MITM for any node that accepts the advertisement, leading to potential flow compromise of the entire garden (or more through BGP peering). Can the Calico provider currently protect against a rogue seed like that? Cilium's dependence on BPF maps should also be evaluated to ensure a rogue Cilium container could not be specified for a seed in a manner that the seed can modify root cluster maps.
There are just so many ways that this could all go wrong that I believe configurations should be trusted by RBAC and Gitops methodology and Gardener should treat this as a documentation problem rather than one of implementation. Promotion of any CNI configuration should require sign-off by global org security rather than depend on trust of the delegated local org. Smaller environments that can't handle this burden may be left vulnerable if they open up their services prematurely, I just don't think it can be handled by any validator that the Gardener team could keep current and flexible over time.

The cilium issue you referenced is indeed intriguing. As you have mentioned, we are at least affected with our coredns deployment, even though we should mostly have a somewhat working solution due to having at least two replicas. However, one replica will likely handle all the load as udp is much more common for dns than tcp. Therefore, in general I all for resolving the issue, helping you verify that the proposed solution from cilium works and getting the fix into a cilium release. (Unfortunately, I did not understand, which feature flag to set from the issue. Maybe, I have skipped too much through it.)
Anyway, in the past the provider config was more or less free form, which also meant what you are asking for would have been easier to achieve. With https://github.com/gardener/gardener-extension-networking-cilium/pull/40 strict validation of the provider config was enabled. Indeed, validation may not be complete, but is a good first step to ensure the provider config has valid content.
What is not yet clear to me is whether you want to pass the cilium options only to certain shoot clusters or to all of them. If you simply want a passthrough for all of them I guess the ControllerDeployment values could be used as parameters. The other open question for me is the following: Do you simply want to set these feature flags in the cilium config map or somewhere else?

I found this problem through RFC3126 support for external DNS. You point out DNS requests are typically UDP, but dynamic updates are TCP. So it's difficult to support rfc2136 without this fix. I have some code around to pick up once the other issues worked out.
Wasn't familiar with #40, but believe it will be undisturbed by this proposal. ConfigMap is valid if the data entities can be parsed as YAML. That sidesteps the intent of validated Cilium configuration, but it will work. If consensus is we should not step away from validating Cilium config at the individual option level, it seems like we revert back to the issues originally raised above. I'm sure there are many reasons I'm not aware of why #40 was created in the first place.
Regarding the config option source scopes: If I understand your question correctly and given the security considerations, it seems like the local config should be overwritten by KVs from the global. That would allow a garden admin to overwrite values from an RBAC scope that the less-trusted local user does not have access to. It seems like both local and global scopes should be supported and shouldn't be a problem if providerConfig from each scope both allow the type to be set as desired. Using a map[string]string as the backing store would be sufficient to normalize KVs from both source types.


[WIP] [Woptim] RADIUSS Spack Configs update

Summary

This PR updates the submodule RADIUSS Spack Configs

Purpose
In RADIUSS Spack Configs:

The compilers (wrappers) on Lassen have been updated to better support the use of gcc 8.3.1 toolchain.
The local spack packages have been updated to be as close as possible to what we are pushing in https://github.com/spack/spack/pull/41375.

Notes
Packages in RADIUSS Spack Configs still have some specific changes compared to the PR mentioned above.
Supersedes https://github.com/LLNL/Umpire/pull/876

Closing in favor of #875


nasa-veda: Pin to using JupyterLab 3

This breaks jupyterlab-git, and maybe also dask-labextension. Use older tag here until this is fixed.
Ref https://2i2c.freshdesk.com/a/tickets/798

Looks good @yuvipanda, thanks!

I've done one better, and worked with our upstream image to pin to JupyterLab < 4 so jupyterlab-git and dask-labextension don't break! Brings in https://github.com/pangeo-data/pangeo-docker-images/pull/469 now.


DOT Login

Project Abstract
DOT Login uses OAuth2 and zk-SNARKs to bridge the gap between Web2 and Web3. This integration is facilitated by a series of custom Substrate pallets, the existing FRAME balances pallet, an off-chain worker that connects to the OAuth2 provider's JWK registry as well as a web-based wallet. DOT Login users will be able to create a wallet using their web2-native OAuth2 account of choice (e.g. Gmail), avoiding common ux painpoints such as remembering keyphrases, learning how to use hardware wallets, etc. without compromising on wallet functionality.
Grant level

[ ] Level 1:  Up to $10,000, 2 approvals
[x] Level 2:  Up to $30,000, 3 approvals
[ ] Level 3:  Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

[x] The application template has been copied and aptly renamed (project_name.md).
[x] I have read the application guidelines.
[x] Payment details have been provided (bank details via email or Polkadot (USDC & USDT) address in the application).
[x] I am aware that, in order to receive a grant, I (and the entity I represent) have to successfully complete a KYC/KYB check.
[x] The software delivered for this grant will be released under an open-source license specified in the application.
[x] The initial PR contains only one commit (squash and force-push if needed).
[x] The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
[ ] I prefer the discussion of this application to take place in a private Element/Matrix channel. My username is: @_______:matrix.org (change the homeserver if you use a different one)

I have read and hereby sign the Contributor License Agreement.

@semuelle thanks for your quick reply.

target audience

We've extended the overview and ecosystem fit chapters to provide more information about our target audience.

Are you aware of https://futures.web3.foundation/?

Yes, we're aware of it. We've added the reasons for our preference for the grants program to the additional information chapter.
I hope this helps to clarify your questions, @semuelle.

@keeganquigley

Couldn't there be a way to allow users to connect their existing wallet extension instead?

We've originally intended to create our own wallet due to the following reasons:

the generation of the ephemeral keys, the ability to interact with the OAuth2 protocol and the generation zk-SNARKs are essential to our solution and are not supported by any existing wallet
to be more flexible in the future, also with respect to our longer term plans. For example, at a later stage we'd envision to use Polkadot SDK's asset-conversion pallet to enable the wallet to pay tx fees in stables. That way the web2-native user doesn't have to hold any DOT, which would greatly improve UX and avoid confusion.

However, after re-consideration we're willing to finance the wallet for this stage by ourself, if you think it doesn't add value to the ecosystem.
We've adjusted the application accordingly, while leaving the option open to implement a wallet extension, rather than a complete wallet.

How would you compare your project with something like Dauth which uses TEEs for authentication?

To be honest, we were not aware of this project before your comment; thanks for bringing this up. We've added a section dedicated to the comparisons between the two projects. In essence, it seems like they're not building on EVM-tech rather than Polkadot SDK-related tooling while DOT Login is a Substrate-first solution. There are also design differences, e.g. they seem to require the TEE to guarantee the safety of the mail <-> address mapping they're storing there, while our solution doesn't foresee the storage of any PID.

on-chain JWT validation

We don't anticipate the JWT validation itself to be expensive since signature validation is a task that collators already heavily are involved with. For example, parachain collators would usually validate transaction signatures or parachain block signatures.
As for the verification of the ZK proof, the expensive part (generation of the proof) is carried out on the wallet (extension), while the significantly cheaper part (proof verification) is carried out by the collators.

@semuelle thanks for your reply. I've removed M2 and M3 from the scope of this grant. Does this setup work better for you? Financing the remaining scope through the DF program sounds like a good option indeed.

@semuelle I noticed that the changes requested label is still active, so I wanted to check if there's anything missing from our side.

@takahser regarding allocating enough time to this project, please allow me to quote from the grant application:

Our current situation is that we've reserved this month for setting up our businesses to be less reliant on us, so we can start focusing on the implementation of DOT Login from February.

We're currently shifting our focus to this project. I've already entrusted people close to me with governing and running my current ventures for the month of February, so this has been dealt with. Likewise, my colleagues have temporary reserved most of February for this project, so an allocation of 3 FTE is to be expected. After that, the plan is to secure follow-up funding for the other milestones through the Decentralized Futures program (as suggested by @semuelle), so our focus is going to remain on this project. As you can see in the Milestone 4+ section, we do already have plans beyond that scope, and we're confident to be able to secure long-term funding once the initial 3 milestones have been implemented. I hope this clarifies your doubts, but if not, please don't hesitate to reach out again.

@Noc2 thanks for your reply, let me break down your questions.

zkLogin: Their solution varies vastly in various factors:
a. data processing: they allow dApps to use web2 data, while DOT Login doesn't process any user data
b. their focus: they add an additional barrier to safe-guard dApps, we're aiming to deliver a seamless bridge to onboard web2 users
c. UX: they require a native mobile app, we're offering a web-based wallet with no extra components
d. trust assumptions: their solution includes trusting the attestation service in addition to trusting the OAuth2 providers
A detailed technical comparison of the on-chain components is difficult: Although their sequence diagram is easy to digest, it doesn't go in deep detail when it comes to the runtime implementation. More details can be found on the newly added zkLogin section.

Our GitHub accounts: While they've been created as early as 2014, 2015 and 2020, we acknowledge the fact that we've been focusing on the development of proprietary software rather than open-source - something we'd like to change (see also here).

Preference for Grants over Dec. Futures: We're currently streamlining our capacities in an effort to fully focus on DOT Login which is why a rapid decision on the way forward is a priority to us. According to the X Space from 25 Jan over 100 projects have been submitted to the Dec. Futures Program, of which 2 have been approved so far which is why we think the Grants Program is the better option for us (see also here). Initially we've applied for a larger scope, but @semuelle suggested to go for a smaller grant and to apply to the Dec. Futures program for the remaining scope - a lean approach that allows the Web3 Foundation to verify our code quality in this first iteration, before potentially committing to a bigger amount for the extended scope.

@singkeo, if you haven't yet, could you submit your company information through the KYB link above?

Thanks for onboarding us ! @semuelle thanks for the link, @keeganquigley done it today, waiting for the review.


Fix: Repo tabs are hidden in Actions on Firefox

This rule was conflicting with GitHub’s own overflow-nav logic, which ended up hiding every tab.

Fixes https://github.com/refined-github/refined-github/issues/5921
Reverts the ancient https://github.com/refined-github/refined-github/issues/3548

Test URLs

https://github.com/jerone/eslint-formatter-checklist/runs/7950750517?check_suite_focus=true

Before

After

cc @jerone

✅ If I disable this style via Dev Tools, the issue indeed disappears.


Update to a more recent Plutus repo version

I think I fixed it. Needs a constraint due to a weird dependency issue. Will fix upstream.

I tried to push the fix to this branch but I don't have permissions. We'll need to copy over a constraint from the Plutus cabal.project since I can't actually see a way to fix this in the .cabal file constraints, weirdly.


ci: Run non-fork, non-dependabot jobs again

Going to try backporting this to the web branch to try and understand why we're getting token errors: https://github.com/PRQL/prql/actions/runs/7351852156/job/20017282449


Webview_flutter Scroll listeners for checking scroll to bottom and top

Need scroll listeners for checking the users whether he/she completely red the web pages

@siraj-logicvalley
Does this issue sound similar to yours ?

I have checked this in ------->  _controller.evaluateJavascript('window.onscroll = function(ev) { if ((window.innerHeight + window.scrollY) >= document.body.offsetHeight) { messageHandler.postMessage(document.title);  } };'); in iOS Simulator. working correctly. but encounters the platform exception Error Domain=WKErrorDomain Code=5 "JavaScript execution returned a result of an unsupported type" UserInfo={NSLocalizedDescription=JavaScript execution returned a result of an unsupported type}, null)

@siraj-logicvalley
Where do you get that platformexception ? on iOS or Android ?

on iOS

But will the above referred issue once fixed, will address your issue as well, what I am asking.

Without additional information, we are unfortunately not sure how to resolve this issue.
We are therefore reluctantly going to close this bug for now.
Please don't hesitate to comment on the bug if you have any more information for us; we will reopen it right away!
Thanks for your contribution.
Could everyone who still has this problem please file a new issue with the exact description of what happens, logs, and the output of flutter doctor -v.
All system setups can be slightly different, so it's always better to open new issues and reference related issues.


unix: Clear -Waddress-of-packed-member warnings

Getting the new GCC 9.x -Waddress-of-packed-member warnings for some lines in the current master branch, e.g. in unix/linux-core.c:
loop->watchers[loop->n_watchers] = (void*) events;

The epoll_event struct on my system is packed and 1-byte aligned, so IIRC the behavior should be undefined when assigning (void*) events to a uv__io_t*, at least in C11:

A pointer to an object type may be converted to a pointer to a different object type. If the
resulting pointer is not correctly aligned for the referenced type, the behavior is
undeﬁned.

This PR clears the warnings by introducing two new private members in struct uv_loop_s to hold the values that are otherwise being appended to the watchers.

Alas changing the loop structure would break the ABI, wouldn’t it?

Sure would ._.
Do changes in the master branch make it to the v1.x ABI though?
If so, what's the way to propose a change for v2?

This could land for v2 no problem... but there is a proposal to never have a v2 so there is that.
@bnoordhuis thoughts?

I opened #2584 with a proposal that doesn't break ABI. This PR is the proper fix but since it can't land in a v1.x release, it's not immediately useful. There's no target date for a v2.x release, if indeed it ever happens.

Nice hack :) Closing in favor of #2584


Drop original capacity and implement truncate_front

Drop original_capacity
Currently bytes crate have a logic of preserving "original capacity" during split.
So
drop(bytes.take());
// `bytes` here will have original capacity preserved if original capacity was <= 16K

I'm not sure if this logic is actually useful, and it may be confusing (e. g. you got a 100 bytes buffer, you called reserve(1) and got 64K allocated).
I'd simply remove it (even without implementing truncate_front).
Implement truncate_front
On the other hand, without "original capacity" field it would be possible to implement cheap truncate_front operation.
truncate_front is the same as truncate, but truncates from front, not from back.
truncate_front is implemented trivally for KIND_ARC, KIND_STATIC and KIND_INLINE.
And for KIND_VEC truncate_front can be implementing by changing arc field of Inner struct to store a pointer to original Vec.
So for KIND_VEC
struct Inner {
    arc: AtomicPtr<Shared>, // vec.as_mut()
    ptr: *mut u8, // position to current view
    len: usize, // len of current view
    cap: usize, // capacity to current view
}

So to reconstruct Vec from Inner (e. g. to convert to Arc) you could write something like:
fn reconstruct_vec(Inner inner) {
    debug_assert!(inner.kind() == KIND_VEC);
    let offset = inner.ptr - inner.arc;
    Vec::from_raw_parts(inner.arc, inner.len + offset, inner.cap + offset)
}

Why truncate front
I believe there are lot of sitations when you need to take a Bytes and simply truncate front and back. For example, if you parse HTTP/2 frame, you can take Bytes by value, and truncate headers and padding to retrieve payload as Bytes. If kind == KIND_VEC this operation will be cheaper than currently with split_off, because Bytes won't need to be converted to KIND_ARC.

The intention of tracking the original allocation size was to handle a regression when transitioning to tokio-io. Part of the problem is that Decoder don't know the size of the originally allocated buffer. So Decoder is expected to reserve only as much as is needed for the current frame, but then, you can get into a situation where tiny reallocations are required.
Being able to truncate_front would be nice... but I am unsure what the best trade off is...  At this point, removing the original capacity tracking would require either figuring out how to do so w/o it being a breaking change OR figure out a way to migrate off for tokio-io 0.2.

Shouldn't FramedRead store initial capacity explicitly, and use it in reserve, and do not rely on Bytes property of preserving capacity in split_* and take operations?
It should be a mostly backward-compatible change of tokio-io (it will be source-compatible, but behavior will change slighly), no need to wait for tokio-io 0.2.

FramedRead does not call reserve. The Decoder impl calls reserve as needed.

Even if Decoder impl calls reserve, FramedRead could also call reserve before read_buf with initial capacity.

How does FramedRead know when to call reserve? If it calls reserve w/ the initial capacity each "tick" then there is no point in using Bytes because every iteration will allocate a brand new chunk.
Decoder is the thing that knows when to call reserve correctly, but it doesn't know the "right" amount... FramedRead knows this.

OK, I think I got it.
FramedRead could call reserve(initial_capacity) right before read_buf. However, in that case it will be one unnecessary reallocation per read_buf if last reserve call in Decoder reserved less than initial_capacity.
So it is not possible to make this change without breaking/changing tokio-io.

Somewhat related, what's the cheapest way of "consuming" the first N bytes of Bytes right now?
I'm using split_off(..)

So, tokio-io Decoder::decode could be changed to something like:
// Ok(Ok(item)) means it was decoded successfully
// Ok(Err(44)) means that decoder needs 44 bytes to decode something
fn decode(&mut self, src: &mut BytesMut) -> Result<Result<Self::Item, usize>, Self::Error>;

And FramedRead::poll should do reserve with error value of inner result.
Right?

Technically, yes. However, the goal of Encoder and Decoder is to be as simple as possible. I think we already have strayed from that goal too much.

Closing this issue in favor of tokio-io issue I've just created.


kurator install supports specifying component configuration files

Please provide an in-depth description of the question you have:
The components component configuration file of kurator is embedded in the binary file. If the user needs to modify the components component configuration file, it will be very inconvenient.
What do you think about this question?:
Can kurator install add a flag to use the local component configuration file.I don't know if this is possible.
Environment:

Kurator version:
Others:

IIRC, this should be supported, maybe I missed, let me figure out.


Changed logger adquisition from Log4j LogManager to our own LoggerFac…

…tory
This is used to be able to inject specific loggers on bundles that need
to be used multiple times, like Mongo Core and Mongo Repl

Codecov Report

Merging #159 into sharding will increase coverage by 1.2%.
The diff coverage is 36.33%.

@@              Coverage Diff              @@
##             sharding     #159     +/-   ##
=============================================
+ Coverage       17.12%   18.33%   +1.2%
- Complexity       1412     1511     +99
=============================================
  Files             769      780     +11
  Lines           26847    26803     -44
  Branches         2337     2337
=============================================
+ Hits             4597     4913    +316
+ Misses          21926    21556    -370
- Partials          324      334     +10

Impacted Files
Coverage Δ
Complexity Δ

...ava/com/torodb/mongodb/guice/MongodCoreModule.java
100% <ø> (ø)
2 <0> (ø)
:x:

...ds/signatures/diagnostic/ListDatabasesCommand.java
4.76% <ø> (+4.76%)
2 <0> (+2)
:white_check_mark:

.../repl/impl/AlwaysConsistentConsistencyHandler.java
20% <ø> (ø)
1 <0> (ø)
:x:

...odb/mongodb/repl/oplogreplier/ReplSyncFetcher.java
0% <0%> (ø)
0 <0> (ø)
:x:

...a/com/torodb/backend/AbstractDbBackendService.java
0% <0%> (ø)
0 <0> (ø)
:x:

...a/com/torodb/backend/meta/SnapshotUpdaterImpl.java
0% <0%> (ø)
0 <0> (ø)
:x:

...com/torodb/backend/meta/AbstractSchemaUpdater.java
0% <0%> (ø)
0 <0> (ø)
:x:

...ain/java/com/torodb/core/d2r/D2RLoggerFactory.java
0% <0%> (ø)
0 <0> (?)

...in/java/com/torodb/backend/BackendServiceImpl.java
0% <0%> (ø)
0 <0> (ø)
:x:

...odb/mongodb/repl/oplogreplier/ReplSyncApplier.java
0% <0%> (ø)
0 <0> (ø)
:x:

... and 153 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eb22305...7560233. Read the comment docs.


Quotely Laughs

Related Issue
Closes #248
Email id used to regsiter for VSoc'24
<EMAIL_ADDRESS>Description
Added quotely laughs app.
Type of PR

[ ] Bug fix
[ ] Feature enhancement
[ ] Documentation update
[ ] Security enhancement
[X] Other (specify): Project

Screenshots / Videos (if applicable)
11_7_2024, 10_22_12 am - Screen - Untitled video.webm
Checklist

[X] I have performed a self-review of my code.
[X] I have read and followed the Contribution Guidelines.
[X] I have tested the changes thoroughly before submitting this pull request.
[X] I have provided relevant issue numbers, screenshots, and videos after making the changes.
[X] I have commented my code, particularly in hard-to-understand areas.
[X] I have followed the code style guidelines of this project.
[X] I have checked for any existing open issues that my pull request may address.
[X] I have ensured that my changes do not break any existing functionality.
[X] Each contributor is allowed to create a maximum of 4 issues per day. This helps us manage and address issues efficiently.
[X] I have read the resources for guidance listed below.
[X] I have followed security best practices in my code changes.

merged @kom-senapati


reproducing the results of Figure 6

Hello. Great research work.
I have some questions about reproducing the results of Figure 6.

Do I only need to train the model once to obtain all the results in Figure 6 through inference, or do I need to train different models according to different situations?
In the prediction task in Figure 6, do you always keep the history window + prediction window = 48?
In code, how do I switch between Diffusion-TS-G and Diffusion-TS-R？
Thanks.

Hi, for your problems :

Yes, of course you can train it just once.
Yes, your right.
Regarding Diffusion-TS-R, just comment the code in gaussian_diffusion.py like following screeshot

Thanks for your reply, my problem is solved.


streaming watermarkStateStore type should be a class

Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[ ] [GOBBLIN-555] My Gobblin PR

https://issues.apache.org/jira/browse/GOBBLIN-555

@yukuai518 ,could you help review it? Thank you!


Disable finish drawing when existing vertex is clicked

When user is drawing a line and existing vertex is clicked, the drawing is finished. How can I disable this behaviour to finish drawing only when "Finish" button is clicked?

Currently this is not built in, but we will think about adding it.
You can use following code:
      const createMarkerProto = L.PM.Draw.Line.prototype._createMarker;
      const setTooltipTextProto = L.PM.Draw.Line.prototype._setTooltipText;
      L.PM.Draw.Line.include({
        _createMarker: function(latlng){
          const marker = createMarkerProto.call(this, latlng);
          marker.off('click', this._finishShape, this);
          return marker;
        },
        _setTooltipText: function() {
          setTooltipTextProto.call(this);
          const { length } = this._layer.getLatLngs().flat();
          if (length > 1) {
            this._hintMarker.setTooltipContent('Click finish in the Toolbar to create the Line');
          }
        },
      });
      map.pm.Draw.Line = new L.PM.Draw.Line(map);

@Falke-Design thank you, this is great improvement, but with this modification it's still possible to finish drawing when user clicks or double clicks the existing vertex.


Unity application on android not working

Required Info

Camera Model
D435i

Firmware Version
<IP_ADDRESS>

Operating System & Version
Win 10

Platform
Android 10

SDK Version
2.35

Language
unity

Issue Description
Hello everybody,
I'm trying to build a unity application for android that uses the depth camera. To do so I downloaded the unity package and I also built the native Android library (librealsense.aar) and placed it in the Plugins folder. As seen in another issue (#4155) I also add the request for the camera permission and the RsContext init but I'm still facing the same error:
ExternalException: rs2_pipeline_start_with_config(pipe:0xb988c8b8, config:0xb988c940)

Rethrow as Exception: No device connected
  at Intel.RealSense.ErrorMarshaler.MarshalNativeToManaged (System.IntPtr pNativeData) [0x000a7] in <8ba4a07bda0d406b93ef994ea9f0e3de>:0
    at (wrapper managed-to-native) Intel.RealSense.NativeMethods.rs2_pipeline_start_with_config(intptr,intptr,object&)
  at Intel.RealSense.Pipeline.Start (Intel.RealSense.Config cfg) [0x0000d] in <8ba4a07bda0d406b93ef994ea9f0e3de>:0
  at RsDevice.OnEnable () [0x000cf] in ...Assets\RealSenseSDK2.0\Scripts\RsDevice.cs:95

I successfully deployed from android studio the example "RS Camera".
Does the unitypackage work with android? If so, what I'm missing?
Thank you!

Hi @castanor   Android 10 devices have been known to have conflicts with the RealSense Android wrapper.   In non-Unity projects, this is usually resolved by setting targetSdkVersion to an earlier version.  '27' is a targetSdkVersion value that has been proven to solve the conflict.

Thanks, I was able to target version 27 of the sdk from unity and now it works.

Great news!   Thanks so much for the update.

@castanor I am also struggling to make unity realsense android, can I ask more details how you did that? Thanks

@qnach10 Hi, can you give me more info? Where is it exactly that you are having problems?

@castanor I made interactive installation, which using unity and Realsense (playing .bag file in Unity). and I am trying to make Oculus Quest version for it.
I succeeded to put bag files to android but I had an error when I debug with adb catlog "Unable to find realsense2 at(wrapper managed to native) Intel.Realsense.NativeMethods.rs2_get_api_version(object&).
I changed to target version 27.
I read another post from you and I tried to do "Build RealSense SDK for Android OS" but I am having trouble to do that (I tried both build with gradle and build with android studio)
If you have any suggestions, it would be really helpful for me!
I appreciate for your time.
Thanks

@qnach10 I remember having the same problem when I started so I suppose the problem is indeed the missing .aar file. I'm not an expert of Android Studio but I simply did "Build->Make Module 'librealsense'" and then I found the .aar inside ..build/outputs/aar. Try and see if you manage to build this way.


test: Add test to verify that null cells are displayed properly in spatial embeddings

Related bug: #858
This will require modifying the test dataset to have null values in spatial embeddings.

Thanks for creating all the tickets, Seve!!


load_csv not showing up in litstudy=1.0.4

I came across issue #29 when searching for how to resolve AttributeError: module 'litstudy' has no attribute 'load_csv'  for litstudy.load_csv(...)
I see that load_csv was added in 1.0.4. This is the version installed in my env.
PS: I installed litstudy today with pip install litstudy
I went searching in my ...\envs\litstudy\lib\site-packages\litstudy folder.
I noticed csv.py is missing \sources.
I observed in __init__.py in \sources that there's no entry for load_csv as well.
How do I go resolving this.  Thanks.
NB: I'm trying to use a csv file from my query in Scholarly

#walkaround as at 25 Mar '23

download csv.py from repository
edit __init__.py to include entry for load_csv

In my working file (JupyterLab)
import litstudy.sources  ##load subfolder with __init__
#from litstudy import *  #load_csv

## reload library
from importlib import reload  # Python 3.8.16
reload(litstudy.sources)

Thanks for using litstudy!
This was fixed by c669fab and load_csv is available in litstudy 1.0.5