DRILL-7855: Provide a secure mechanism for specifying storage plugin credentials

DRILL-7855: Provide a secure mechanism for specifying storage plugin credentials
Description
Please see https://github.com/apache/drill/pull/2162/files?short_path=f46bd00#diff-f46bd001ba51b40d94a6988bbcb2e16357e4c423da3b8e681073f9b100aadc55 for a detailed description.
This PR preserves the backward compatibility with the previous versions of Drill unless plugin configuration will use credentialsProvider (it is a new field, so older versions cannot handle it).
Documentation
Docs on the Drill Web site should be added for this feature.
Testing
Added unit tests, tested locally storage plugins with all credential providers. Will also test providing a custom pluggable credential provider.

@johnomernik
Could you take a look at this PR?

So, I didn't review all the code (and that wouldn't be effective review
anyhow).  I did hone in on the markdown doc at  164
docs/dev/PluginCredentialsProvider.md
https://github.com/apache/drill/pull/2162/files#diff-f46bd001ba51b40d94a6988bbcb2e16357e4c423da3b8e681073f9b100aadc55
Some comments:

I like the plugin credentials provider. This is handy.
One challenge in drill may be that the user accessing the cluster may
have very little control over many of the ways the credentials are
provided. For example:

HadoopCredentialsProvider: How are credentials provided by a "user"
here. Not the "admin" who started the cluster (the core-site.xml
would need
to be edited by the user. How do we handle different users connecting?
EnvCredentialsProvider: How does a user provide environmental
variables if they are connecting via ODBC, JDBC, Rest API?  Even SQLLine
running remotely may not work here, correct?
VaultCredentialProvider: I like this, however, as above, how does a
USER provider a vault location and token? If it's in the
drill-override.conf, how will a user specify this information
separate from
a different user?  Is there a way for Drill to provide an interface to
store vault tokens that can be unlocked via user
authentication?.   I know
this is messy, but essentially you have to have the User Tell Drill I am
who I say I am, and then you need Drill, once it verifies this,
to be able
to access the vault on behalf of the authenticated user in order to get
passwords for various data stores.  This all must be done security, and
ideally in a way that is simple for the user.

That's my first thought as of now, let me know if my rambling doesn't make
as much sense typed out as it does in my head.
John
On Sat, Feb 6, 2021 at 5:56 PM Charles S. Givre<EMAIL_ADDRESS>wrote:

@JohnOmernik https://github.com/JohnOmernik
Could you take a look at this PR?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/drill/pull/2162#issuecomment-774562189, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOXGUOEV5RTHSLV6YZEPGLS5XJJRANCNFSM4XGXCLSQ
.

Hello @JohnOmernik
Thanks a lot for reviewing the document and your comments, they all are major.
Doing these changes I had some similar objections, but finally decided to do it as it is.
The problem we trying to solve here is to add functionality for hiding credentials from the user that creates the storage plugin. Users can only specify how to access credentials, but it doesn't mean that the user knows actual values stored there, and he shouldn't. Actual credentials may change, but references may leave the same without affecting target users.
Providing some functionality for storing credentials automatically means providing functionality for displaying their values, but as mentioned above, I don't think we want to do that.
Having HadoopCredentialsProvider is consistent with the Hadoop way of providing configuration, but not more. It may be useful for the case when we have some pre-defined credentials that may be used in storage plugins, but it is not aimed for the case of frequently added/updated credentials.
EnvCredentialsProvider has a similar use case, but for the k8s world. Some pre-defined credentials may be exposed into the container as env variables, so Drill can use them, it may also be useful, but for limited cases as the earlier one.
I would expect VaultCredentialProvider to be used more compared to the two previous ones. I assumed the flow would be the following: administrator configures Drill with specified vault address and token. After that, users either ask the admin or use some tool for adding/viewing/updating credentials.
Currently, Drill differentiates only two user kinds: admins and regular users. Admins can create storage plugins, and all plugins created by one admin are visible to all other admins. This PR doesn't aim to improve Drill user management, it just provides minimal functionality for storage plugins credentials, so currently we use the same Vault token for all users.
There is also a discussion for improving Drill roles and permissions: https://lists.apache.org/thread.html/rbb4a83a67ee92b081d0ce3a17533dce5bfaf31b4dcc10ee9fd3cff33%40<dev.drill.apache.org>. One of the ideas @vdiravka proposed is integrating with Vault. So in this case, the user will already be associated with some Vault token and this value may be used instead of providing the token in configs, hope we will follow that way.


WIP: pkg/fuzzer: factor out the fuzzing engine

Almost all of the changes are in the last commit of the PR, but since it's a big rewrite of the very complicated logic, I don't see reasons to split it into smaller chunks. It won't help reviewers much and will only make CR note fixing unnecessary hard.

This is the first step for https://github.com/google/syzkaller/issues/1541.
Move the fuzzing engine that used to be interleaved with other syz-fuzzer
code into a separate package.
For now, the algorithm is more or less the same as it was, the only
difference is that a pkg/fuzzer instance scales to the available
computing power.
Add an executor-based test that performs real fuzzing.
TODO:

[ ] Better document the code.
[ ] Improve pkg/fuzzer tests. Add comparison argument collection for the TestOS.
[ ] Run syz-testbed experiments to see the difference in the fuzzing performance.

Bugs TODO:

[x] Debug sporadic panic: timed out waiting for an execution result errors.
[x] Debug zero coverage & non-zero signal inputs.

The race detector has found something:
2024-02-22T16:22:45.9417729Z WARNING: DATA RACE
2024-02-22T16:22:45.9417994Z Write at 0x00c0009be578 by goroutine 8:
2024-02-22T16:22:45.9418412Z   github.com/google/syzkaller/pkg/fuzzer.(*Request).sent()
2024-02-22T16:22:45.9418801Z
2024-02-22T16:22:45.9419022Z pkg/fuzzer/fuzzer.go:90 +0x11d
2024-02-22T16:22:45.9419420Z   github.com/google/syzkaller/pkg/fuzzer.(*Fuzzer).NextInput()
2024-02-22T16:22:45.9419825Z
2024-02-22T16:22:45.9420034Z pkg/fuzzer/fuzzer.go:193 +0x2da
2024-02-22T16:22:45.9420393Z   github.com/google/syzkaller/pkg/fuzzer.TestFuzz()
2024-02-22T16:22:45.9420748Z
2024-02-22T16:22:45.9421069Z pkg/fuzzer/fuzzer_test.go:73 +0x84a
2024-02-22T16:22:45.9421448Z   testing.tRunner()
2024-02-22T16:22:45.9421763Z       /usr/local/go/src/testing/testing.go:1595 +0x238
2024-02-22T16:22:45.9422137Z   testing.(*T).Run.func1()
2024-02-22T16:22:45.9422460Z       /usr/local/go/src/testing/testing.go:1648 +0x44
2024-02-22T16:22:45.9422731Z
2024-02-22T16:22:45.9422884Z Previous read at 0x00c0009be578 by goroutine 27:
2024-02-22T16:22:45.9423356Z   github.com/google/syzkaller/pkg/fuzzer.(*Fuzzer).execWait()
2024-02-22T16:22:45.9423751Z
2024-02-22T16:22:45.9423966Z pkg/fuzzer/fuzzer.go:294 +0x244
2024-02-22T16:22:45.9424343Z   github.com/google/syzkaller/pkg/fuzzer.(*genJob).run()
2024-02-22T16:22:45.9424719Z
2024-02-22T16:22:45.9424921Z pkg/fuzzer/job.go:53 +0x168
2024-02-22T16:22:45.9425342Z   github.com/google/syzkaller/pkg/fuzzer.(*Fuzzer).NextInput.func1()
2024-02-22T16:22:45.9425781Z
2024-02-22T16:22:45.9426002Z pkg/fuzzer/fuzzer.go:184 +0x50
2024-02-22T16:22:45.9426195Z
2024-02-22T16:22:45.9426315Z Goroutine 8 (running) created at:
2024-02-22T16:22:45.9426620Z   testing.(*T).Run()
2024-02-22T16:22:45.9426928Z       /usr/local/go/src/testing/testing.go:1648 +0x82a
2024-02-22T16:22:45.9427305Z   testing.runTests.func1()
2024-02-22T16:22:46.5726213Z       /usr/local/go/src/testing/testing.go:2054 +0x84
2024-02-22T16:22:46.5726987Z   testing.tRunner()
2024-02-22T16:22:46.5727561Z       /usr/local/go/src/testing/testing.go:1595 +0x238
2024-02-22T16:22:46.5728221Z   testing.runTests()
2024-02-22T16:22:46.5728798Z       /usr/local/go/src/testing/testing.go:2052 +0x896
2024-02-22T16:22:46.5729350Z   testing.(*M).Run()
2024-02-22T16:22:46.5729736Z       /usr/local/go/src/testing/testing.go:1925 +0xb57
2024-02-22T16:22:46.5730196Z   main.main()
2024-02-22T16:22:46.5730486Z       _testmain.go:57 +0x2bd
2024-02-22T16:22:46.5730724Z
2024-02-22T16:22:46.5730859Z Goroutine 27 (running) created at:
2024-02-22T16:22:46.5731380Z   github.com/google/syzkaller/pkg/fuzzer.(*Fuzzer).NextInput()
2024-02-22T16:22:46.5731904Z
2024-02-22T16:22:46.5732211Z pkg/fuzzer/fuzzer.go:184 +0x26b
2024-02-22T16:22:46.5732648Z   github.com/google/syzkaller/pkg/fuzzer.TestFuzz()
2024-02-22T16:22:46.5733084Z
2024-02-22T16:22:46.5733352Z pkg/fuzzer/fuzzer_test.go:73 +0x84a
2024-02-22T16:22:46.5733731Z   testing.tRunner()
2024-02-22T16:22:46.5734104Z       /usr/local/go/src/testing/testing.go:1595 +0x238
2024-02-22T16:22:46.5734557Z   testing.(*T).Run.func1()

@dvyukov  I've addressed the biggest CR notes in a separate commit. PTAL.
So far it seems that 2GB qemu VMs have already become too small (they tend to fail with OOMs). I've set up a syz-testbed experiment to see how it works on bigger VMs.

The extra memory consumption was caused by having multiple choice table versions (each referenced by some subset of goroutines). I've fixed it by using a proxy version of the choice table object.


CSUB-423: Update Polkadot.js

Description of proposed changes:
Practical tips for PR review & merge:

[ ] All GitHub Actions report PASS
[ ] Newly added code/functions have unit tests

[ ] Coverage tools report all newly added lines as covered
[ ] The positive scenario is exercised
[ ] Negative scenarios are exercised, e.g. assert on all possible errors
[ ] Assert on events triggered if applicable
[ ] Assert on changes made to storage if applicable

[ ] Modified behavior/functions - try to make sure above test items are covered
[ ] Integration tests are added if applicable/needed

Codecov Report

Merging #961 (caf5a88) into dev (1485a71) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##              dev     #961   +/-   ##
=======================================
  Coverage   77.14%   77.14%
=======================================
  Files          72       72
  Lines       11332    11332
=======================================
  Hits         8742     8742
  Misses       2590     2590

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


[DNM] SElinux denial fixes

Generated with
adb logcat -b all -d | audit2allow -p policy
includes part 2 of rfkill permission fix.

This change is

the second should be rebased on first one

rebased and squashed now. I'm still looking for more denials.

File_contexts priv_app two whitespace please clean these too ... Thanks

done

@Bitti09 I have seen others PR ... if this is related to them then please let only rfkill stuff then we can fix others in new PR

splitted the commit . now rfkill has extra PR


Auctions#420: live links login

https://github.com/artsy/auctions/issues/420 tracks this.
This pr standardizes our live auction linking to use a method on the sale model. It also adds view helpers on the auction page and my_active_bids components.
I searched the repo for use of PREDICTION_URL to narrow down places where this was an issue.
The liveAuctionUrl method takes one argument, user which if present appends /login to the url. The my active bids module just passes true because a bid implies a user.
I didn't do anything with our react stuff, which all looked fine.

LGTM! I think Sarah makes some good points about the Backbone vs. non-Backbone worlds—something we've pondered before and often just move the logic to Metaphysics in those cases. FWIW I don't think it's necessarily wrong to wrap partial data in Backbone for it's model functionality and therefore wouldn't consider this a blocker—but more and more it becomes a fork the road we should probably make a decision on and stick to.

Ah! Thanks for linking to that slack convo. 😄 I know @damassi had started creating utils folders (nested as far as possible) where necessary to accomplish this on the force side. Seems like we could continue that pattern here and keep all of the MP-model-helper logic in one place. /shrug

@sweir27 I'll come back to this with your recs in mind and then we can merge. Thanks for your comments.

I'd still very much like to have a domain/auctions bucket somewhere to drop miscellaneous stuff :)

Just noticed that CURRENT_USER is not available from sd when utils get loaded. I could go back to my original method of taking a second param forceLogin that could use true, false, or something like CurrentUser.orNull(), or we could load that part of the data sooner?

bid_form.coffee

my active bids- user settings screen

MAB - auctions list [hard to capture in screenshot]

banner - logged in:

banner - not logged in:

This looks good to me! Just had one question (and I think you need to rebase).

I think this is good to go but I cannot make its tests pass. I'm out of ideas.

Ah, @craigspaeth, I was going to ask if you wanted to pair on the failing tests but I was able to resolve them today- the issue with PREDICTION_URL seems to be fixed, or at least I can't reproduce it any more. Thanks for pointing out where things were going wrong in tests. This seems to be working with the correct redirect URLs now. Ready for merge.

👍


LinkedHashSet field to Set field, elements sort not correct.

List<String> -> Set<String> does not maintain order, even through the final underlying implementation/concrete type mapping of Set is a LinkedHashSet , this seems quite odd to me.
The same thing happens with Set<String> -> Set<String> when both are LinkedHashSet's (due to the default concrete type mapping of Set being LinkedHashSet).

I have same problem. Sorting not correct because HashSet type hardcoded in MapperFacadeImpl:
   public final <S, D> Set<D> mapAsSet(final Iterable<S> source, final Type<S> sourceType, final Type<D> destinationType,             final MappingContext context) {         return (Set<D>) mapAsCollection(source, sourceType, destinationType, new HashSet<D>(), context);     }


API question

Hello,
I'm using PEX' API for managing players groups.
I've got few problems:

When changing groups results don't take effect until player relogs/permissions reloaded
Reloading with manager.reset(); doesn't work?

At the moment I have such code:
PermissionUser u = PermissionsEx.getUser(player.getPlayer());
u.getParents(null).forEach(u::removeGroup);
String newgroup = (String) winner1.value;
u.addGroup(newgroup);
PermissionManager manager = PermissionsEx.getPermissionManager();
try {
    manager.reset();
} catch (PermissionBackendException e) {/*smth here*/}

I get an unexpected result: when changing player's group first time (let's think that I have 20 groups and I choose random one each time I execute this code) after executing this code player will have no groups until this code will be run again, then he'll get pre-last (sic!) group. This can be fixed running /pex reload.
So, it looks like:
Action: Set the group to group 'A'
player has no groups now
Action: /pex reload
player is in group 'A' now
Action: Set the group to group 'B'
player is in group 'A' now
Action: Set the group to group 'C'
player is in group 'B' now
Action: /pex reload
player is in group 'C' now
So, questions:

What is the proper way to save the results?
What is the difference between manager.reset(); and /pex reload?
How to refresh the player's cache or whatever to make him get his group without any additional movements?

Actually, that appeared with last version, removing the reload code works fine with latest version. Though, please make it clear do I need to clear the cache and reload anything or not.


Fixed unresolved import from removed ast::{Method_}.

This should fix #170. Thanks to @zeekay for the help!

Awesome! thanks v much @marvelm


Implement list directive

The list directive is currently a stub, which does nothing. However, it should look up the list (by name) in the domain data, find out if there are any known values for the list (either in* any* context, or in contexts with no matches) and render it like a table, much in the same way that the command-table command does.

Fixed by #144


Add some space at the end of Git Machete graph

For now (esp. for large graphs or when listing commits is enabled) the last branch name is very near the IDE tool window buttons:

That makes invoking actions on the last branch less comfortable and just looks less clear.
Probably adding one empty line at the end of the graph (or even better some Swing's spacing widget/component or just padding) solve this problem.

But how exactly do we imagine the graph table with "some space at the end"???
What do you mean by adding an extra line?

If it is to be another row the "issue" will remain. One will be able to resize & scroll te graph table to the situation above.
If it is to be some "sticky footer" we will get a strange scrollbar that does not fill the whole height of the window.

The same "issue" appears everywhere...
Debug:

Git Log:

Just simple visualization:

It should be something that is scrollable.
It should be something like in the main editor but smaller:

You can scroll down below the last line.

We decided to drop this idea.

It is not that problematic. The table looks good overall.
In most cases one can adjust the toolwindow height.


chic-hdr:0.1.0

I am submitting

[X] a new package
[ ] an update for a package

I have read and followed the submission guidelines and, in particular, I

[X] selected a name in conformance with the guidelines
[X] added a typst.toml file with all required keys
[X] added a README.md with documentation for my package
[X] have chosen a license and added a LICENSE file or linked one in my README.md
[X] ensured that my submission does not infringe upon the rights of a third party
[X] tested my package locally on my system and it worked
[X] named this PR as name:version of the submitted package
[X] agree that my package will not be removed without good reason

Description: Typst package for creating elegant headers and footers, as fancyhdr for LaTeX.

Thank you!


[AKS] Monitoring addon MSI auth - use existing tags on container insights extension dcr for PUT

Description

For MSI auth of Monitoring addon, there will be DCR created  with format MSCI--. If the customer tags and this resource exists, in case of customer try to re-onboard the monitoring addon with MSI auth, existing tags (if any) will be used for the PUT to retain the customer tags.
Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change
[Component Name 2] az command b: Add some customer-facing feature

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] The PR title and description has followed the guideline in Submitting Pull Requests.

[x] I adhere to the Command Guidelines.

[ ] I adhere to the Error Handling Guidelines.

AKS

Please make sure the changes won't break aks-preview, which refers the code from azure-cli.
@ganga1980 Could you please reply this question?

@ganga1980   Could you address these comments and  refer to submitting-pull-requests to modify the PR title by tomorrow?
If not, this PR will not be able to catch up with the release of the sprint, and it can only be postponed to the next sprint (05-24)

/azp run

Rebased the branch from dev and queued a pipeline to check some live test cases involving the monitoring addon.

@ganga1980 Is this change perceptible to users? If not, please use {} in the PR title so that it will not be written in the history note

Please make sure the changes won't break aks-preview, which refers the code from azure-cli.

@ganga1980 Could you please answer this question?

yes, these changes have been validated with aks-preview.


[WIP] Add script for flattening

This is still a work in progress, and I'm not sure how much time I'll have to work on this now that I'll have another team after my time.

@ikennaokpala have you tried it locally with a couple of countries?
@chao-xian Why did you change the title to [WIP]? The PR is approved, but still the title does not match the status.

I don't this is a script that is meant to be perfect.. given the anomalies that may come up..I see it as a script that jet start the flattening process.. Hence why it serve as a starting point. The developer that uses this script should then review the changes and make sure that it is in order before checking it in.

@ikennaokpala I don't think so. The script is supposed to flatten countries following a set of steps. It should generally work, and the developer has to review the output in any case.
I don't think we should approve this PR without running it locally for a couple of countries.

That is my humble opinion.. There is not point trying to perfect it knowing that Marriage abroad templates may present a few anomalies

@pmanrubia I added the [WIP] to the title because that's what @cbaines said the status is in the PR, which I did before the PR was approved.

@chao-xian could you please run the script locally and see if it is fit for purpose? if not, could you please find out if there are any steps we could take?

@ikennaokpala there is no point in approving a PR without even running it locally. This is not how we work.

FYI.. I have run it locally and found it sufficient

@chao-xian thank you for having a look at it.
Could you please add a comment to this PR with the countries you have run the script against? and the anomalies that you have found, if any?
If the script is running smoothly, we would only need to update the documentation before merging.

@pmanrubia this seems fine. I can't see any obvious problems with it. All it does is create the outcome per path files that would normally be quite a laborious manual process - assuming you are doing an 18 outcome flattening.
And it's definitely not complete because you are left with files that need cleaning up, so for a _opposite_sex.erb file it has alongside it a _opposite_sex.erb-e that needs removing.
I ran this against:

Malaysia
Singapore
Finland

And it's definitely not complete because you are left with files that need cleaning up, so for a _opposite_sex.erb file it has alongside it a _opposite_sex.erb-e that needs removing.

Interesting, I don't think I encountered this. I just ran the flatten script locally for the 3 countries you mentioned, and I didn't get any odd files generated. Perhaps the perl or sed you are using works differently to the one I'm using?

@cbaines I had been running it on OS X.
Another issue is that it doesn't work on the VM at the moment
vagrant@development:/var/govuk/smart-answers$ script/flatten.sh malaysia
script/flatten.sh: 12: read: Illegal option -d
script/flatten.sh: 19: script/flatten.sh: Bad substitution

But weirdly if I run that read command with the same options on the command line it doesn't complain.

But weirdly if I run that read command with the same options on the command line it doesn't complain.

I'm running it with bash, but I guess that in the govuk-puppet development VM, you are using dash. Try running it explicitly with bash perhaps? (e.g. bash ./scripts/flatten.sh ...).

@cbaines yeah that worked and without the -e erb templates.

@cbaines yeah that worked and without the -e erb templates.

Great :+1:

@mgrassotti have you been using this script at all?

@chao-xian yes I am

@mgrassotti both on your Mac or just in the VM? Any changes that you've made? If you have been using it in anger and it's fine, then I think we need to update this PR and look to merge it in (unless that's already been done somewhere else).
What do you reckon?

Hi @mgrassotti we really should merge this into master if we are actively using it. Can you please add your review to this PR. I'd be happy to update our docs if you can let me know that the script is ok.

@mgrassotti hi I'm going to close this PR as we are slowing down on the number of requests we get for flattening Marriage Abroad. The only other option is to merge into master. I'll close it by the end of tomorrow. Cc @pmanrubia

Matteo is no longer active, and I'm currently trying to cut down on the number of Pull Requests I have open. I'm going to close this for now, but I'll leave the branch around. I think this script is still a real time saver when it comes to flattening.


How can I access to currentUser in getServerSideProps of Next.js

ref. https://firebase.google.com/docs/hosting/nextjs#:~:text=This can be accessed in getServerSideProps

The Express res.locals object will optionally contain an authenticated Firebase App instance (firebaseApp) and the currently signed-in user (currentUser). This can be accessed in getServerSideProps.

Could you tell me how to access to currentUser in getServerSideProps, Next.js? At first, I thought that I can used it like this.
export const getServerSideProps: GetServerSideProps = async (context) => {
  console.log(context.res.locals currentUser);
  return { props: { data: 'aaa' } };
};

However I couldn't do it because the res.locals is Express res.locals object. What is it? And How can I access to it?

Did you find any solution?

Not yet.

Is there something fundamentally wrong with my question?

context.res.locals?.currentUser

Locals is just an object, be mindful it might not be present so nullish coalesce context.res.locals?.currentUser

Nice explanation, if this was working @SugiKent  wouldn't have created this ticket in the firstplace, i spent two days looking for this, i even  changes the ctx type to get  context.res.locals but id did not give any results

Sorry, I miss spelled in first comment. I forgot to use ?..
Now I implemented below code but firebaseApp and currentUser were undefined.
export const getServerSideProps: GetServerSideProps<Props, Params> = async (
  context
) => {
  console.log({ firebaseApp: context.res.locals?.firebaseApp });
  console.log({ currentUser: context.res.locals?.currentUser });

  return {
    props: { aaa: null },
  };
};

I thought that these property were not implemented and the official doc was advanced, right?

@jamesdaniels

locals property doesn't exist on context.res;


Measurement metadata registration does not work for specific ontology terms

What happened?
Measurement registration is not possible for some ontology terms.

ngs_measurement_registration_sheet.xlsx
What browsers are you seeing the problem on?
No response
Relevant log output
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

On the first glance it looks like the terminology service is not queried but the data manager is looking in the database instead.


Add workspace delete confirmation

[x] joeferner
[x] mwizeman joeybrk372 jharwig sfeng88

Disables the "Are you sure?" button for 1 second after Delete click

Points of Regression: Case delete
CHANGELOG
Added: Confirmation before case deletion

LGTM


hostnames still aren't getting released

Even after closing a deployment & waiting for it to disappear from the kubernetes cluster you can't reuse a hostname for some reason
You get this error
$ ~/ovrclk/akash/akash provider send-manifest --provider akash1vqthd5lk0rdc67v9v89f7zal6e3e2fw4ejhmfn --keyring-backend=test --from=user0 --home=$PWD/home --node=tcp://<IP_ADDRESS>:26657 ./minesweeper_hosts.yaml --dseq 79798
provider error messsage:
hostname not allowed: "minesweeper.hydrogen18.com" already in use

The object that tracks the in use hostnames either isn't clearing out state properly, or the deployment manager isn't sending the message to do so

Sorry for the confusion - I made #1123


added binary search in python

This Pull Request resolves Issue #25
Description:
In this Pull Request we have done the following additions/edits for the code:
added binary search in python
Note: Delete all the text including this!

Make sure you replace #<IssueNumber> with the particular Issue that you are referencing with!
Make sure you describe the Issue properly and ask as many people as possible to review your Pull Request so that it can be merged as soon as possible!

Thanks for contributing!


Add installation guide on how to install on MacOS

Hey, it's a super cool idea, to provide an offline-editor. Would you mind, to provide a documentation on how to install this on MacOS?

oh, sorry, I've finally figured it out.
Maybe, some additional lines might help:
Even on MacOS the installation of NPM/NodeJS is required.

Create an down Folder for oxocard-editor and run an terminal in that folder.

clone the git-repository in that folder (z.B /oxocard)
```sh
git clone https://github.com/oxocard/offline-editor
```

change to the Folder offline-editor and prepare the program
```sh
cd offline-editor
npm install
npm run build
```

After a while (depending on the PCs performance) you can start the offline-editor.
```sh
npm start
```

finally you can start the offline-editor

http://localhost:1777/

I close the issue


Trying to animate datapoints. Almost have it, but not quite.

I was trying to use the below to animate the datapoints. I feel like it is close, but it's just not happening. If I click on the graph the points all jump into place, but it's not actually performing the tween. I am guessing it is missing a set state, but I am not quite sure how I would implement that. Any suggestions?
An example can be seen here. https://i.imgur.com/jn7rEib.gifv
Thanks,
-MH
  double numberAnimate(double order, double current) {
    var adjustedValue = new Tween(begin: 1.0, end: current).animate(
      new CurvedAnimation(
        parent: controller,
        curve: new Interval(
          0.350, // + (order * 0.1),
          0.650,
          curve: Curves.easeIn,
        ),
      ),
    );
    return adjustedValue.value;
  }

          data:  [
                  DataPoint<double>(
                      value: widget._controller
                          .numberAnimate(date7.day.truncateToDouble(), 10.0),
                      xAxis: date7.day.truncateToDouble()),
                  DataPoint<double>(
                      value: widget._controller
                          .numberAnimate(date6.day.truncateToDouble(), 30.0),
                      xAxis: date6.day.truncateToDouble()),
                  DataPoint<double>(
                      value: widget._controller
                          .numberAnimate(date5.day.truncateToDouble(), 50),
                      xAxis: date5.day.truncateToDouble()),
                  DataPoint<double>(
                      value: widget._controller
                          .numberAnimate(date4.day.truncateToDouble(), 20),
                      xAxis: date4.day.truncateToDouble()),
                  DataPoint<double>(
                      value: widget._controller
                          .numberAnimate(date3.day.truncateToDouble(), 80),
                      xAxis: date3.day.truncateToDouble()),
                  DataPoint<double>(
                      value: widget._controller
                          .numberAnimate(date2.day.truncateToDouble(), 14),
                      xAxis: date2.day.truncateToDouble()),
                  DataPoint<double>(
                      value: widget._controller
                          .numberAnimate(date1.day.truncateToDouble(), 30),
                      xAxis: date1.day.truncateToDouble()),
                ],

is this an issue about the current library?

I am not certain, to be honest. I use the same animation code in many other places in my app and it works just fine. The animations in the gif that are on the same screen as the graph are using the same controller and are all coded just like the graph is, but the graph doesn't update. So I am not sure if maybe something special needs to be added to it to make it animatable? As of the current code base though, I can't seem to animate it by typical means.

Hmmm I see. Let's do this, create a new file named 'sample11.dart' , like this, https://github.com/aeyrium/bezier-chart/blob/master/example/lib/sample10.dart ,  then add this file here (simple file that I could run) and I will check.

This is the one I am currently using along with the file that has the animation in it.
chart.zip

Try creating a code that I could download and run please.

Ok, I will make a more complete one and post it. One second while I put it together.

Actually, I might have figured something out while trying to get a demo project together. Having simplified it down, I went to test it, and it moved differently now instead of just popping at once, it stopped midway through as if it was trying to animate but... got stuck? I think I might be able to mess with this particular thing and figure something out with it.

I believe I might have figured out what is going on.  All other demos I tried, when the pointer enters the widgets area onPointerEnter fires once, then onPointerExit fires once when you leave, then onPointerHover changes as you move the pointer around if you are not holding the button (if you do hold it onPointerMove gets called instead).
So for some reason, something is causing the pointer callbacks to go nuts and just keep repeating constantly.
Thanks,
-MH

Maybe there is a setState call
On Wed, Jul 31, 2019 at 11:01 PM MostHated<EMAIL_ADDRESS>wrote:

I believe I might have figured out what is going on.
All other demos I tried, when the pointer enters the widgets area
onPointerEnter fires once, then onPointerExit fires once when you leave,
then onPointerHover changes as you move the pointer around if you are not
holding the button (if you do hold it onPointerMove gets called instead).
So for some reason, something is causing the pointer callbacks to go nuts
and just keep repeating constantly.
Thanks,
-MH
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/aeyrium/bezier-chart/issues/26?email_source=notifications&email_token=ABFL3UBFJZSLUVWVKXAMC4TQCJNX3A5CNFSM4IHLVQL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3JHN6A#issuecomment-517109496,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABFL3UDKZRLPHOT3C6WTQCDQCJNX3ANCNFSM4IHLVQLQ
.

It looks like there is. This was your position update code in bezier_chart_widget
void _updatePosition(Offset globalPosition) {
    RenderBox renderBox = context.findRenderObject();
    final position = renderBox.globalToLocal(globalPosition);
    if (position == null) return;
    return setState(
      () {
        final fixedPosition = Offset(
            position.dx + _scrollController.offset - horizontalPadding,
            position.dy);
        _verticalIndicatorPosition = fixedPosition;
      },
    );
  }


using equirectangular images

Hi, thanks for the implementation.
Could this be used with 360º images as source, like the OmniNerf implementation
Thanks in advance and kind regards

@monschine Sorry, I'm not familiar with that. I guess it can't.

equirectangular images are rarely geometrically correct. it's physically impossible to build a real 360 camera with a single optical center that takes a snapshot photo. The only way to build it is by physically rotating some optical device.  If the data you want to use comes from these cameras, you should use the unstitched raw images instead.


Movement swapping oscillation

the two repairers on the top left are going back and forth forever.
almost 100% to due with the fastFiller creeps fighting over the same fastFiller stationary hauling position.

Noticed this with haulers and other creeps (scouts and maintainers). this doesn't happen only due to fastfiller but maybe in this occurrence. For me this occurs near my hub

Will check traffic priorities if I can find it. But if you made a commit that modifies this behavior recently can you link it?

I've got unpushed commits that appear to fix the fastFillerPosition issue that seemed to be the underlying cause of the oscillation

Noticed this with haulers and other creeps (scouts and maintainers). this doesn't happen only due to fastfiller but maybe in this occurrence. For me this occurs near my hub

If see this happening without fastFiller creeps involved, please link history :)


test

Codecov Report
Patch coverage has no change and project coverage change: -3.69 :warning:

Comparison is base (dad9050) 74.19% compared to head (25ec3cb) 70.50%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main       #8      +/-   ##
==========================================
- Coverage   74.19%   70.50%   -3.69%
==========================================
  Files         176      176
  Lines       17209    17209
  Branches      286      201      -85
==========================================
- Hits        12768    12134     -634
- Misses       4441     5075     +634

Flag
Coverage Δ

bot
40.64% <ø> (ø)

bot-unit
40.64% <ø> (ø)

discordeno
100.00% <ø> (ø)

discordeno-unit
100.00% <ø> (ø)

gateway
29.98% <ø> (-44.22%)
:arrow_down:

gateway-integration
?

gateway-unit
29.98% <ø> (ø)

rest
71.97% <ø> (ø)

rest-unit
71.97% <ø> (ø)

types
100.00% <ø> (ø)

types-unit
100.00% <ø> (ø)

utils
95.24% <ø> (ø)

utils-unit
95.24% <ø> (ø)

see 2 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


I get the "Uh oh... Refresh your browser to get back in business. "

I get this dialogue box immediately after logging in and my data is loaded up.
Name of Defective Feature
System Settings (Required)
ToolKit Version: 1.2.0
Operating System (Mac/Windows/Etc):  Mac OS 10.12.6
Browser (Chrome/Firefox/Safari): Both Chrome & Firefox
Browser Version: Chrome 61.0.3163.100, Firefox 56.0
Other Extensions:
Export of Toolkit Settings (Bottom Left of Settings Page)
[{"key":"AccountsDisplayDensity","value":"0"},{"key":"AccountsEmphasizedOutflows","value":false},{"key":"AccountsStripedRows","value":false},{"key":"AdditionalColumns","value":true},{"key":"AdjustableColumnWidths","value":true},{"key":"AutoCloseReconcile","value":false},{"key":"BetterScrollbars","value":"0"},{"key":"CategoryActivityCopy","value":false},{"key":"ChangeEnterBehavior","value":false},{"key":"CheckNumbers","value":true},{"key":"ColourBlindMode","value":false},{"key":"CustomFlagNames","value":false},{"key":"DisplayTargetGoalAmount","value":"0"},{"key":"GoalWarningColor","value":false},{"key":"HideAgeOfMoney","value":false},{"key":"HideReferralBanner","value":false},{"key":"ImportNotification","value":"0"},{"key":"PrintingImprovements","value":false},{"key":"PrivacyMode","value":"0"},{"key":"RemovePositiveHighlight","value":false},{"key":"RowHeight","value":"0"},{"key":"RowsHeight","value":"0"},{"key":"RunningBalance","value":"1"},{"key":"ShowCategoryBalance","value":false},{"key":"SquareNegativeMode","value":false},{"key":"StealingFromFuture","value":false},{"key":"TargetBalanceWarning","value":false},{"key":"accountsClearSelection","value":false},{"key":"activityTransactionLink","value":true},{"key":"betterScrollbars","value":"0"},{"key":"budgetBalanceToZero","value":true},{"key":"budgetProgressBars","value":"0"},{"key":"budgetQuickSwitch","value":false},{"key":"budgetRowsHeight","value":"0"},{"key":"calendarFirstDay","value":"0"},{"key":"categoryActivityPopupWidth","value":"0"},{"key":"checkCreditBalances","value":false},{"key":"collapseExpandBudgetGroups","value":false},{"key":"collapseSideMenu","value":true},{"key":"compactIncomeVsExpense","value":false},{"key":"currentMonthIndicator","value":false},{"key":"daysOfBuffering","value":false},{"key":"daysOfBufferingHistoryLookup","value":"0"},{"key":"editButtonPosition","value":"0"},{"key":"enableRetroCalculator","value":false},{"key":"enlargeCategoriesDropdown","value":true},{"key":"goalIndicator","value":false},{"key":"googleFontsSelector","value":"0"},{"key":"hideAccountBalancesType","value":"0"},{"key":"hideHelp","value":false},{"key":"highlightNegativesNegative","value":false},{"key":"importNotification","value":"0"},{"key":"incomeFromLastMonth","value":"0"},{"key":"l10n","value":"0"},{"key":"largerClickableIcons","value":false},{"key":"monthlyNotesPopupWidth","value":"0"},{"key":"navDisplayDensity","value":"0"},{"key":"pacing","value":"0"},{"key":"popupCalculator","value":true},{"key":"privacyMode","value":"0"},{"key":"reconciledTextColor","value":"0"},{"key":"removeZeroCategories","value":false},{"key":"reports","value":true},{"key":"resizeInspector","value":false},{"key":"rightClickToEdit","value":true},{"key":"seamlessBudgetHeader","value":false},{"key":"showIntercom","value":true},{"key":"spareChange","value":false},{"key":"splitKeyboardShortcut","value":true},{"key":"splitTransactionAutoAdjust","value":false},{"key":"swapClearedFlagged","value":false},{"key":"toBeBudgetedWarning","value":false},{"key":"toggleSplits","value":true},{"key":"toggleTransactionFilters","value":"0"},{"key":"warnOnQuickBudget","value":false}]
[paste settings here]
</details>

# Steps to Reproduce the Behavior
<!-- Please write down the steps necessary to reproduce the behavior you're seeing. Please be as thorough as possible. -->

1.  Log in.
2. Select any account.
3.

# Expected Behavior
<!-- What did you expect would happen after completing the steps? -->

# Actual Behavior
<!-- What actually happened after you completed the steps. -->

Gee, it worked for me about 2 hours ago, after an update earlier in the day. But I just applied today's latest update, and it's back where it was before; enabling the split transaction button causes Accounts not to load, with the same error message as before.

Same here. It was fine until yesterday but now I've had to turn off the extension to even be able to use YNAB.

Thanks for the bug report. Closing as a duplicate of #969. Turn off the Toggle Splits feature to get around the problem. A fix has been developed and will be in the next release of the Toolkit.


Fix accordion items wrongly registering as childs on …

…the (wrong) parent, e.g. on tab component.
Fixes the weird tab/accordion issue demonstrated in https://github.com/mqje/ember-bootstrap-accordion-example: accordion items were being registered as child to the tab containing them, that's why the tab nav got rerendered whenever a new accordion item was added... 🙄
@mqje this should fix it! Thanks for the awesome reproduction!

Who thanks, thqt was quick, Ill check it out monday on the project haha


Circuit range constraints

Pull request checklist
Please check if your PR fulfills the following requirements:

[x] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been reviewed and added / updated if needed (for bug fixes / features)
[x] Build (yarn build) was run locally and any changes were pushed
[x] Lint (yarn lint --check) has passed locally and any fixes were made for failures

Pull request type

Please check the type of change your PR introduces:

[x] Bugfix
[ ] Feature
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] Documentation content changes
[ ] Other (please describe):

What is the current behavior?

Closes #419
Closes #418
Closes #417
What is the new behavior?

Does this introduce a breaking change?

[ ] Yes
[x] No

Other information

Yes, this PR doesn't close #420


remove Option from KeyEvent

I just wrote a small event-loop and noticed the Some I needed to place everywhere.
After a while it gets a little annoying.
So I'm suggesting the following change:

KeyEvent will be KeyEvent(Key)
Key enum will contain something like an Unhandled or Other element wrapping an u16
Key::from_code(u16) will not return an Option but a Key and in the _ branch returns Key::Unhandled(code)

Sorry I did not understand what the function is for

The functionality stays the same but solved more "rust-like" as we can use more beautiful match-branches.
Edit: If you are okay with the proposed changes I would like to send this as a PR to make your life a bit easier.

I made the changes to my fork, if you want to have a look.
I if you want to merge it, just say so, I'll send a PR (the version bump would then be up to you).

@benaryorg sorry for the super long delay on getting back to you here.
A PR for this would be welcomed! Thank you for taking the time on it.


Lock Timeout Usage / Understanding

I'm struggling to understand why the following test is failing.
using NUnit.Framework;
using System;
using System.Threading.Tasks;
using AsyncKeyedLock;

namespace UnitTests.ThreadingRelated;

public class AsyncKeyedLockerTest
{
    [TestFixture]
    public class When_locking : AsyncKeyedLockerTest
    {
        [Test]
        public async Task and_a_timeout_value_is_exceeded()
        {
            var tcs = new TaskCompletionSource();
            var sut = new AsyncKeyedLocker<string>(x =>
            {
                x.PoolSize = 1;
            });

            var task1 = Task.Run(async () =>
            {
                using (sut.Lock("key", TimeSpan.FromSeconds(20), out _))
                {
                    tcs.SetResult();
                    await Task.Delay(TimeSpan.FromSeconds(5));
                }
            });

            // wait here until the tcs has started. This will ensure that task2 starts after a lock has already been started from task1.
            await tcs.Task;

            var task2 = Task.Run(() =>
            {
                try
                {
                    // Using a timeout of 1 second. I'm assuming this value should be small enough that the timeout value should be met.
                    using (sut.Lock("key", TimeSpan.FromSeconds(1), out _))
                    {
                        throw new Exception("this should not get called because the other task should have a lock on the string");
                    }
                }
                catch (Exception e)
                {
                    // I'm expecting this to be a TimeoutException, but I haven't been able to replicate an actual timeout yet, so I'm not sure.
                    // I'm definitely not expecting the following.
                    Assert.AreNotEqual("this should not get called because the other task should have a lock on the string", e.Message);
                }
            });

            await Task.WhenAll(task1, task2);
        }
    }
}

The Assert.AreNotEqual is throwing because the strings ARE equal.
I feel like I'm missing something very obvious here so please forgive me if that's the case.  I thought the sut.Lock method within Task2 would wait until one of the following is true

task1 releases the lock
the timeout of 1 second is met

It seems like neither of those constraints are met but even so, task2 continues without waiting.
Regarding first constraint above, what does it mean to release a lock? I thought it meant that the lock would not be released until the lock is disposed of which is not done until after the using clock finishes.
Again, I think I'm missing something fundamentally.

Hi,
The issue is that you're discarding the out parameter on the synchronous Lock method. That's a boolean that denotes whether or not the semaphore was entered. In case of a timeout, it would be false.
Also I'd recommend using LockAsync in an async method.

@MarkCiliaVincenti Thanks for your reply.
Unfortunately, I'm still missing something fundamental here. I've gone ahead and updated the test to use LockAsync but am still experiencing unexpected behavior (a failed test).
using NUnit.Framework;
using System;
using System.Threading.Tasks;
using AsyncKeyedLock;

namespace UnitTests.Core.ThreadingRelated;

public class AsyncKeyedLockerTest
{
    [TestFixture]
    public class When_locking : AsyncKeyedLockerTest
    {
        [Test]
        public async Task and_a_timeout_value_is_exceeded()
        {
            var tcs = new TaskCompletionSource();
            var sut = new AsyncKeyedLocker<string>(x =>
            {
                x.PoolSize = 1;
            });

            var task1 = Task.Run(async () =>
            {
                using (await sut.LockAsync("key", TimeSpan.FromSeconds(20)))
                {
                    tcs.SetResult();
                    await Task.Delay(TimeSpan.FromSeconds(5));
                }
            });

            // wait here until the tcs has started. This will ensure that task2 starts after a lock has already been started from task1.
            await tcs.Task;

            var task2 = Task.Run(async () =>
            {
                try
                {
                    // Using a timeout of 1 second. I'm assuming this value should be small enough that the timeout value should be met.
                    using (await sut.LockAsync("key", TimeSpan.FromSeconds(1)))
                    {
                        throw new Exception("this should not get called because the other task should have a lock on the string");
                    }
                }
                catch (Exception e)
                {
                    // I'm expecting this to be a TimeoutException, but I haven't been able to replicate an actual timeout yet, so I'm not sure.
                    // I'm definitely not expecting the following.
                    Assert.AreNotEqual("this should not get called because the other task should have a lock on the string", e.Message);
                }
            });

            await Task.WhenAll(task1, task2);
        }
    }
}

This test still fails for me.
I originally wrote the test using Lock because my usage will eventually be within a sync / void method.

Interesting... I removed the timeout parameters altogether from both LockAysnc method calls in my test and in debug, I'm seeing the wait/lock within the task2 as expected.
So maybe I'm miss-understanding what the timeout parameter is supposed to do. I assumed it would make the LockAsync method throw if the timeout is exceeded. Maybe instead, it is ignoring the lock after the timeout exceeds.
Can you clarify this behavior?

I've updated my original test to the following code and it passes as expected. It turns out I was not understanding the Timeout parameter behavior. Thanks for your help!
using NUnit.Framework;
using System;
using System.Threading.Tasks;
using AsyncKeyedLock;

namespace UnitTests.ThreadingRelated;

public class AsyncKeyedLockerTest
{
    [TestFixture]
    public class When_locking : AsyncKeyedLockerTest
    {
        [Test]
        public async Task and_a_timeout_value_is_exceeded()
        {
            var tcs = new TaskCompletionSource();
            var sut = new AsyncKeyedLocker<string>(x =>
            {
                x.PoolSize = 1;
            });

            var task1 = Task.Run(async () =>
            {
                using (sut.Lock("key", TimeSpan.FromSeconds(20), out _))
                {
                    tcs.SetResult();
                    await Task.Delay(TimeSpan.FromSeconds(5));
                }
            });

            // wait here until the tcs has started. This will ensure that task2 starts after a lock has already been started from task1.
            await tcs.Task;

            var task2 = Task.Run(() =>
            {
                try
                {
                    // Using a timeout of 1 second. I'm assuming this value should be small enough that the timeout value should be met.
                    using (sut.Lock("key", TimeSpan.FromSeconds(1), out var entered))
                    {
                        if (!entered) throw new TimeoutException();

                        throw new Exception("this should not get called because the other task should have a lock on the string");
                    }
                }
                catch (Exception e)
                {
                    // I'm expecting this to be a TimeoutException, but I haven't been able to replicate an actual timeout yet, so I'm not sure.
                    // I'm definitely not expecting the following.
                    Assert.AreNotEqual("this should not get called because the other task should have a lock on the string", e.Message);
                }
            });

            await Task.WhenAll(task1, task2);
        }
    }
}

@MarkCiliaVincenti Are there any other reasons why the out parameter would be false except for when/if a timeout has been reached?

LockAsync was failing because that also returns a type with a boolean.
As for your last question, no.


Port log4j-to-slf4j changes from 2.x

This looks good to me but we might require an SLF4J/Logback upgrade:

bump SLF4J to 2.0.9 (like 2.x),
bump Logback to 1.4.11 (2.x uses the 1.3.x branch, since the 1.4.x branch requires Java 11).

I will address this while sync'ing log4j-parent. I would like to proceed one module at a time.

This looks good to me but we might require an SLF4J/Logback upgrade:

bump SLF4J to 2.0.9 (like 2.x),
bump Logback to 1.4.11 (2.x uses the 1.3.x branch, since the 1.4.x branch requires Java 11).

I will address this while sync'ing log4j-parent. I would like to proceed one module at a time.

I cherry picked the "Bump SLF4J to version 2.0.9" commit from 2.x. It should be OK now.
Logback was bumped to 1.3.11 only, because 1.4.11 has a broken OSGi descriptor.


Length less than zero causes get grid dataset to fail

Length cannot be less than zero.
Parameter name: length
    + CategoryInfo          : NotSpecified: (:) [Get-GRIDDataset], ArgumentOutOfRangeException
    + FullyQualifiedErrorId : System.ArgumentOutOfRangeException,PSAtlasDatasetCommands.GetGRIDDataset
    + PSComputerName        : localhost

The main stuff comes down ok... (like the catalog)... but something else is going wrong. Need to run this carefully.

Problem is caused by a small change in how the text is being printed out - it used to give the filename, with a bracket. That isn't the case any longer. No close bracket. So a index fails.


[Bug]:

/mv version -p output
https://paste.gg/8c74af1128764c3e8e940a210d6acb7e
Server logs
[01:33:55 INFO]: FAILED.  [01:33:55 INFO]: 'spawn' does not appear to be a world. It is lacking a .dat file
Server Version
This server is running Paper version git-Paper-36 (MC: 1.20.1) (Implementing API version 1.20.1-R0.1-SNAPSHOT) (Git: faf9a65)

You are 122 version(s) behind

Bug Description
[01:33:55 INFO]: FAILED.  [01:33:55 INFO]: 'spawn' does not appear to be a world. It is lacking a .dat file
Hello Iam trying to import a world but it tolds me lacking a  .dat file i tried pn different worlds and its the same error
Steps to reproduce
[01:33:55 INFO]: FAILED.  [01:33:55 INFO]: 'spawn' does not appear to be a world. It is lacking a .dat file
Agreements

[X] I have searched for and ensured there isn't already an open or resolved issue(s) regarding this.
[X] I was able to reproduce my issue on a freshly setup and up-to-date server with the latest version of Multiverse plugins with no other plugins and with no kinds of other server or client mods.

Proper logs would help. Assume it's missing the level.dat file.
Where did you get the world from? Is it an overworld, nether or end?


Install convos

Hello,
Im trying to set up your application on a VPS ubuntu however i have a problem when i use the first cmd indicated
curl https://convos.by/install.sh | sh -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1186  100  1186    0     0   4553      0 --:--:-- --:--:-- --:--:--  4561
> /usr/bin/curl -s -L https://github.com/Nordaaker/convos/archive/stable.tar.gz | /bin/tar xz -C convos --str
ip-components 1
> /usr/bin/perl convos/script/convos install
>>> /usr/bin/perl /root/convos/script/cpanm -n -l /root/convos/script/../local IO::Socket::SSL
--> Working on IO::Socket::SSL
Fetching http://www.cpan.org/authors/id/S/SU/SULLR/IO-Socket-SSL-2.043.tar.gz ... OK
Configuring IO-Socket-SSL-2.043 ... OK
==> Found dependencies: Net::SSLeay, Mozilla::CA
--> Working on Net::SSLeay
Fetching http://www.cpan.org/authors/id/M/MI/MIKEM/Net-SSLeay-1.80.tar.gz ... OK
Configuring Net-SSLeay-1.80 ... N/A
! Configure failed for Net-SSLeay-1.80. See /root/.cpanm/work/1484153254.1833/build.log for details.
--> Working on Mozilla::CA
Fetching http://www.cpan.org/authors/id/A/AB/ABH/Mozilla-CA-20160104.tar.gz ... OK
Configuring Mozilla-CA-20160104 ... OK
! Can't configure the distribution. You probably need to have 'make'. See /root/.cpanm/work/1484153254.1833/b
uild.log for details.
! Installing the dependencies failed: Module 'Mozilla::CA' is not installed, Module 'Net::SSLeay' is not inst
alled
! Bailing out the installation for IO-Socket-SSL-2.043.
cpanm failed!

Do you have any idea of what could be the problem ?

I'm guessing you need some dev tools, since the output contains:
You probably need to have 'make'

Also, it seems like you're installing Convos as root. I highly suggest running it as a different user.

root@vps361672:~# curl https://convos.by/install.sh | sh -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1186  100  1186    0     0  19891      0 --:--:-- --:--:-- --:--:-- 20101
mkdir: cannot create directory ‘convos’: File exists
> /usr/bin/curl -s -L https://github.com/Nordaaker/convos/archive/stable.tar.gz | /bin/tar xz -C convos --str
ip-components 1
> /usr/bin/perl convos/script/convos install
>>> /usr/bin/perl /root/convos/script/cpanm -n -l /root/convos/script/../local IO::Socket::SSL
--> Working on IO::Socket::SSL
Fetching http://www.cpan.org/authors/id/S/SU/SULLR/IO-Socket-SSL-2.043.tar.gz ... OK
Configuring IO-Socket-SSL-2.043 ... OK
==> Found dependencies: Net::SSLeay
--> Working on Net::SSLeay

root@vps361672:~# curl https://convos.by/install.sh | sh -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1186  100  1186    0     0   7333      0 --:--:-- --:--:-- --:--:--  7366
mkdir: cannot create directory ‘convos’: File exists
> /usr/bin/curl -s -L https://github.com/Nordaaker/convos/archive/stable.tar.gz | /bin/tar xz -C convos --str
ip-components 1
> /usr/bin/perl convos/script/convos install
>>> /usr/bin/perl /root/convos/script/cpanm -n -l /root/convos/script/../local IO::Socket::SSL
--> Working on IO::Socket::SSL
Fetching http://www.cpan.org/authors/id/S/SU/SULLR/IO-Socket-SSL-2.043.tar.gz ... OK
Configuring IO-Socket-SSL-2.043 ... OK
==> Found dependencies: Net::SSLeay
--> Working on Net::SSLeay
Fetching http://www.cpan.org/authors/id/M/MI/MIKEM/Net-SSLeay-1.80.tar.gz ... OK
Configuring Net-SSLeay-1.80 ... OK
Building Net-SSLeay-1.80 ... FAIL
! Installing Net::SSLeay failed. See /root/.cpanm/work/1484165413.8261/build.log for details. Retry with --fo
rce to force install it.
! Installing the dependencies failed: Module 'Net::SSLeay' is not installed
! Bailing out the installation for IO-Socket-SSL-2.043.
cpanm failed!

Thanks for your answer. I have a very limited knowledge about linux. As a consequence, i searched to install make and it is installed.
Now, the prblem is about Module 'Net::SSLeay' is not installed Do you know how i could install it ? Thanks for these fast answer

Try this:
sudo apt-get install libssl-dev

Or maybe just this?
sudo apt-get install libnet-ssleay-perl libcrypt-ssleay-perl libio-socket-ssl-perl

CONVOS is launched. You're cool thanks!

Could you please tell me which packages you installed so I can document it on https://convos.by/doc/faq.html ?

Honestly I used first one and the second one directly also and launched and it worked

Cool 👍


Method Init not exist

I tried to start the init method call in Ios, but it does not exist in the project.
CarouselViewRenderer.Init();

Hi,
You should put the
using CarouselView.FormsPlugin.iOS namespace at the header of your .cs file or ...
call CarouselView.FormsPlugin.iOS.CarouselViewRenderer.Init(); directly.
Regards...

Tanks


pre-trained models

Hello,
Can you upload somewhere the trained models?
THank you!

Sorry, due to the limited hardware, it takes a long time to run the model. So I didn't train the model completely. I just trained 100 epochs in full. So there is no saved model.

Hello,
Can you upload somewhere the trained models?
THank you!
Sorry, due to the limited hardware, it takes a long time to run the model. So I didn't train the model completely. I just trained 100 epochs in full. So there is no saved model.

Hello,
Trying to download the data from the link provided and I am getting the following message:
2019-05-31 15:12 失效时间：永久有效
Can you re-upload it
Thank you!


-2 doesn't always work..

Lori, So very sorry about this have got you to add the '-2 words' (for which many thanks!).
It turns out that you can set -1 into the times for the file \System Volume Information\tracking.log.tmp but if you try to set it to -2 then you'll get back
Error code: (NTSTATUS) 0xc000000d<PHONE_NUMBER>) - An invalid parameter was passed to a service or function.

Do you care? Not sure I do, SVI is really pretty much "hands off".    Feel free to close this as "mistaken", or we can add some extra words,,,

Document details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 30d6a301-2028-bba5-0e43-092824810891
Version Independent ID: 5c8c373a-b34a-5e73-8002-57076540e6ce
Content: _FILE_BASIC_INFORMATION (wdm.h) - Windows drivers
Content Source: wdk-ddi-src/content/wdm/ns-wdm-_file_basic_information.md
Product: windows-hardware
Technology: windows-devices
GitHub Login: @tedhudek
Microsoft Alias: tedhudek

@rodwiddowson, finally got to circle back on this issue. I've updated the page to say that only NTFS and ReFS recognize and handle the -2 value. Craig indicated that SVI doesn't really mean anything to NTFS. Might a filter have gotten involved and failed it? Or are you running on an older version of Windows? (-2 was added in Win 10 Threshold) I can't do much more from a doc standpoint so gonna close this out.

Pretty sure it was NTFS (the SVI is a giveaway), but Its a long time ago.  I'm also reasonably sure that it was clean of lower filters (apart from the standard Windows one), but I'll believe Craig..


[Enhancement] Pass the NetPeer that sent the packet to the subscribed handler method.

Currently when subscribing to a packet there is seemingly no direct way to get the connected peer that the packet came from. It would be really useful for situations where you want the client might need a response from the server.
If the ReadPacket/ReadAllPackets method took the optional NetPeer that is provided to the NetworkReceiveEvent handler method and then passed it along, it would allow you to process packets with the NetPeer, this is also useful if you need access to the tag parameter of the NetPeer.

@Orsidus this is already done:
https://github.com/RevenantX/LiteNetLib/blob/master/LibSample/PacketProcessorExample.cs


ENG-1187 - add standard ESLint config

Description
Use @basis-theory/eslint-config

Testing required outside of automated testing?

[x] Not Applicable

Screenshots (if appropriate):

[x] Not Applicable

Rollback / Rollforward Procedure

[x] Roll Forward
[ ] Roll Back

Reviewer Checklist

[ ] Description of Change
[ ] Description of outside testing if applicable.
[ ] Description of Roll Forward / Backward Procedure
[ ] Documentation updated for Change

excellent! just a few questions and small suggestions

@brigonzalez everything addressed, lmk if there is anything else!

looks like @sabriskay or @lcschy will have to approve
I have no power here


SLAS shortcode and tenant options switched

I was trying to use the new slas commands, but I believe on v2.8.0 the options are accidentally switched. It looks like sfcc-ci is picking up the tenant from the shortcode option and the shortcode from the tenant option
sfcc-ci slas:client:list -help
Usage: slas:client:list [options]

Lists all SLAS clients that belong to a given tenant

Options:
  --tenant <tenant>        the tenant id used for slas
  --shortcode <shortcode>  the organizations short code
  -j, --json               Formats the output in json
  -h, --help               output usage information

Steps to reproduce:

Auth sfcc-ci
Attempt to list clients: sfcc-ci slas:client:list --shortcode kv7kzm78 --tenant zzrf_001

Expected results:
List of SLAS clients is displayed for tenant zzrf_001 for shortcode kv7kzm78
Actual results:
Error is displayed:
Could not get tenants: request to https://zzrf_001.api.commercecloud.salesforce.com/shopper/auth-admin/v1/tenants/kv7kzm78/clients failed, reason: getaddrinfo ENOTFOUND zzrf_001.api.commercecloud.salesforce.com
However, if you flip the tenant and shortcode it works:
sfcc-ci slas:client:list --tenant kv7kzm78 --shortcode zzrf_001

@hnestmann can you reproduce this?

Related... i'm not sure if this is working as expected for listing tenants:
 ~/code/ sfcc-ci slas:tenant:list -help
Usage: slas:tenant:list [options]

Lists all tenants that belong to a given organization

Options:
  --shortcode <shortcode>  the organizations short code
  -j, --json               Formats the output in json
  -h, --help               output usage information

 ~/code/ sfcc-ci slas:tenant:list --shortcode kv7kzm78
Could not get tenants: tenantId is not defined

Hi Drew, I heard from a partner the same thing- Thanks for raising - we'l have a look

WIP here: https://github.com/johnboxall/sfcc-ci/tree/fix-slas


WIP: Tool for flushing the CDN from wagtail, replaces Picard

Replaces picard with a new Wagtail admin view
Additions

a wagtail admin view that allows for flushing either the whole site

Removals

references to the 'picard' optional app

Changes

Testing

Screenshots

Notes

Todos

We should probably put this behind a permission check?
Needs tests

Checklist

[ ] PR has an informative and human-readable title
[ ] Changes are limited to a single goal (no scope creep)
[ ] Code can be automatically merged (no conflicts)
[ ] Code follows the standards laid out in the development playbook
[ ] Passes all existing automated tests
[ ] New functions include new tests
[ ] New functions are documented (with a description, list of inputs, and expected output)
[ ] Placeholder code is flagged
[ ] Visually tested in supported browsers and devices
[ ] Project documentation has been updated
[ ] Reviewers requested with the Assignee tool :arrow_right:

This looks great and I am very excited for it!  I know this is still a WIP, but it'd be neat to have some integration or unit tests on this (e.g. person submits url, gets success page, or person submits url but no credentials have been set up, gets helpful error page, etc)

@richaagarwal -- definitely, it think testing is the bulk of the remaining work.


[Feature] Documentation rework

Proposal is to join all files that contain test-related documentation into one file with fixed formatting that will contain all info for developers and users.
Currently, test-related documentation is located in these files:  LIST_OF_TESTS.md, RATIONALE.md, USAGE.md, TEST-CATEGORIES.md.
Issues with current documentation:

Hard to navigate (to get complete information about the test you need to browse multiple files)
Redundant information (links to other files, repetition of test details)
Hard to maintain (changes for one test result in need for change in multiple documentation files)
Hard to verify (some or all info about test might be missing, it's time-consuming to validate that all documentation is present)

Proposals:

Join documentation into TESTSUITE-TESTS.md file with fixed structure and formatting.
Create GitHub pipeline that will validate that file and ensure that structure isn't broken and all needed sections are present

In addition to fixing current issues, new format for documentation will have these possibilities in future (not in the scope of the issue):

Usage of documentation in code, directly parsed from the documentation. For example "remediation" in case if a test fails.
Moving content of points.yml, including tags and points, to the TESTSUITE-TESTS.md, and using this in the code
Possible automated generation of specialized documents, that will contain only subset of information for tests.

About having separate file with reasoning for users: is it really needed? If documentation is nicely structured, then it would be easy for user to find all desired information including reasoning for the tests.
Proposed structure of the documentation:
Category 1
Some info about the category
Test 1
Overview:
(task_name) Description of the test, what does it do
Reasoning:
Reasoning for the tests, why is it needed, how does it help for cloud-nativeness
Remediation:
What needs to be done if the test is failing.
Usage:
Command for running the test, extra dependencies if present.
Test 2
...

WIP: example of the file https://github.com/kosstennbl/cnf-testsuite/blob/test-doc-unification/docs/TEST_DOCUMENTATION.md


Pls help, stuck with AutoGGUF

I tried to make ggufs of different models (one that was already available and one which I made using the lazymergekit).
I always get the same error how ever. It's this one (I edited the model name out but it happens with both I tested. They are Mistral 7b based ones):
GML_CUDA_FORCE_MMQ:   no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
  Device 0: Tesla T4, compute capability 7.5, VMM: yes
main: build = 2151 (704359e2)
main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
main: quantizing 'ModelName/modelname.fp16.bin' to 'ModelName/modelname.Q4_K_S.gguf' as Q4_K_S
llama_model_quantize: failed to quantize: failed to open ModelName/modelname.fp16.bin: No such file or directory
main: failed to quantize model from 'ModelName/modelname.fp16.bin'

Also, before that error, I get another error:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
lida 0.0.10 requires fastapi, which is not installed.
lida 0.0.10 requires kaleido, which is not installed.
lida 0.0.10 requires python-multipart, which is not installed.
lida 0.0.10 requires uvicorn, which is not installed.
tensorflow-metadata 1.14.0 requires protobuf<4.21,>=3.20.3, but you have protobuf 4.25.2 which is incompatible.
torchaudio 2.1.0+cu121 requires torch==2.1.0, but you have torch 2.1.2 which is incompatible.
torchdata 0.7.0 requires torch==2.1.0, but you have torch 2.1.2 which is incompatible.
torchtext 0.16.0 requires torch==2.1.0, but you have torch 2.1.2 which is incompatible.
torchvision 0.16.0+cu121 requires torch==2.1.0, but you have torch 2.1.2 which is incompatible.
Successfully installed gguf-0.6.0 numpy-1.24.4 nvidia-cublas-cu12-<IP_ADDRESS> nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-<IP_ADDRESS> nvidia-cufft-cu12-<IP_ADDRESS> nvidia-curand-cu12-<IP_ADDRESS> nvidia-cusolver-cu12-<IP_ADDRESS> nvidia-cusparse-cu12-<IP_ADDRESS> nvidia-nccl-cu12-2.18.1 nvidia-nvjitlink-cu12-12.3.101 nvidia-nvtx-cu12-12.1.105 protobuf-4.25.2 torch-2.1.2

WARNING: The following packages were previously imported in this runtime:
  [numpy]
You must restart the runtime in order to use newly installed versions.

Is there any solution? I would like to try the model I merged locally, I was even able to evaluate it in the leaderboard but I can't turn it into GGUF.
Also is there a dedicated GitHub page for that notebook?

If you notice, you're not replacing the variables in your model name and directory, it seems to me:
'ModelName/modelname.Q4_K_S.gguf'  <------- The real format is Username/ModelName
Example: Kukedlc/Neuronal-7b

If you notice, you're not replacing the variables in your model name and directory, it seems to me:
'ModelName/modelname.Q4_K_S.gguf' <------- The real format is Username/ModelName Example: Kukedlc/Neuronal-7b

No as I wrote above I just edited the model name out because my models are just for test purposes and I don't want much attention for them.
I think the problem is that the code expects an older version of phyton but the colab has a new version but I'm not sure about that, just a guess.

If you notice, you're not replacing the variables in your model name and directory, it seems to me:
'ModelName/modelname.Q4_K_S.gguf' <------- The real format is Username/ModelName Example: Kukedlc/Neuronal-7b

But you are right that the username is not in the error for what ever reason. Instead the model name is twice with uppercase letters and without.

If you notice, you're not replacing the variables in your model name and directory, it seems to me:
'ModelName/modelname.Q4_K_S.gguf' <------- The real format is Username/ModelName Example: Kukedlc/Neuronal-7b

This is straight from the notebook:
qtype = f"{MODEL_NAME}/{MODEL_NAME.lower()}.{method.upper()}.gguf"

I still insist that I believe the issue lies in the path, the directory you are passing. llama_model_quantize: failed to quantize: failed to open ModelName/modelname.fp16.bin: No such file or directory main: failed to quantize model from 'ModelName/modelname.fp16.bin'. If you look there, you should have a folder created where I saved the downloaded model; I also use it in Colab, and the code works fine for me.

Share your code here if you want, and I'll review it.

Hey @GameOverFlowChart could you share a screenshot with all your input values? I did a lot of quants yesterday, I didn't run into this issue.

Hey @GameOverFlowChart could you share a screenshot with all your input values? I did a lot of quants yesterday, I didn't run into this issue.

Here is an example which doesn't work. Also thanks for the tools you provide by the way.

And here is the outcome

Try this: MLABONNED QUANTIZATION CODE

I will try, it seems like @mlabonne also tried to quantize it, but I'm not sure if he was successful (doesn't seem so).

Now that I look closely, your model comes from some Llama model, right? I had the same problem. Lately, I've been using models derived from Mistral, but with Llama models, I encountered the same issue. I think the problem lies in being a Llama model; something happens when it creates the directory (in the same folder as the Llama models /models), and it can't load the model when quantizing it. Only @mlabonne  can help us with this, haha.

The code I provided, run it and see if it performs quantization. It shouldn't fail because it's a model derived from Mistral.

Now that I look closely, your model comes from some Llama model, right? I had the same problem. Lately, I've been using models derived from Mistral, but with Llama models, I encountered the same issue. I think the problem lies in being a Llama model; something happens when it creates the directory (in the same folder as the Llama models /models), and it can't load the model when quantizing it. Only @mlabonne can help us with this, haha.

No it's a merge between two Mistral based models.

And did you try the new code?

I'm getting the same error. It could be something in the config.json as well. I'll try to troubleshoot it in a bit.

I'm getting the same error. It could be something in the config.json as well. I'll try to troubleshoot it in a bit.

Yes I also failed. It might help you maybe but I had the same problem as I tried to do the quantization with another model it's merged with:
argilla/distilabeled-OpenHermes-2.5-Mistral-7B
I thought I'm doing something wrong but maybe the colab can't quantize that model and models which are based on that.

I think the issue lies in how you're doing the merge. In your repository, I found some oddities in config.json, and the tokenizer.model and generation_config.json files are missing. Compare it with one of my models or those from Labonne. I'd review the merge code.

I think the issue lies in how you're doing the merge. In your repository, I found some oddities in config.json, and the tokenizer.model and generation_config.json files are missing. Compare it with one of my models or those from Labonne. I'd review the merge code.

Look at my comment above the same problem did happen with an unmerged model (maybe I should try that again to be sure).
The merge was made with the lazymergekit the code is here: https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing

I'm pretty sure it's what I mentioned last. I uploaded two of my files and it went past the previous error, but it's giving a discrepancy error now, obviously because it's from another model. Please check the config.json, it should have your model's name, not the merged one.

I'm pretty sure it's what I mentioned last. I uploaded two of my files and it went past the previous error, but it's giving a discrepancy error now, obviously because it's from another model. Please check the config.json, it should have your model's name, not the merged one.

Did I get it right, at the model on hugging face, the config should contain it's own name in the "_name_or_path" attribute? I hope hugging face has an editor which let's me change that simply. But as I said the mergekit did that, it probably took the name of the model which I have chosen as the base model for the merge. I will try, it might take some time again. But I'm really surprised that the open llm leaderboard did not fail during the evaluation, I expected that to be much more complex than the gguf step.

Uhm wait but I have no idea where to get tokenizer.model and generation_config.json. Would I need to merge again? I see it now the other model which I merged does have a tokenizer.model file.

I think the issue lies in how you're doing the merge. In your repository, I found some oddities in config.json, and the tokenizer.model and generation_config.json files are missing. Compare it with one of my models or those from Labonne. I'd review the merge code.

Uhm wait but I have no idea where to get tokenizer.model and generation_config.json. Would I need to merge again? I see it now the other model which I merged does have a tokenizer.model file. Uhm yeah it must be because both models use different models as the base. Look, the base model in question also doesn't have a tokenizer.model file which is probably the reason why the merge didn't generate one.

I try to gguf my second model first, if that works I think I know how I need to recreate the first one.

The other one worked


Add joinMulti type inference to TROUBLESHOOTING

Thanks @pmellati for reporting this. I think we can treat it as a known issue.

👍
If it is useful to declare the binder separately, we could generate joinInner, joinLeft, joinInnerInner, joinInnerLeft, ... methods, but if we do that we should probably get rid of the joinMulti method to avoid confusion.


Insert completion

Is it possible to use the insert functionality?

You can use the suffix parameter
const resp = await openai.createCompletion({
	model: "text-davinci-002",
	prompt: "Hi there.\nWe're writing to",
	suffix: 'Congrats from OpenAI!'
});
console.log(resp.data);


Style is broken, when using positionY : 'bottom'

If I use position : 'bottom', the alert tries to fill the entire column.
The alert text is not shown at all.

Nevermind, It was just broken in the Plunker demo when i tried to use it because of the hardcoded top: -100 in the css.


Cherry-pick for release branch: Fix race condition

Signed-off-by: Olga Lavtar<EMAIL_ADDRESS>

Tested against lab via deploying my own images just to sanity-check normal create process is good. LGTM!


[suggestion] Simplify genesis config

Description
I propose to simplify genesis configuration a little bit by moving it to config file & env and dropping the related CLI flag.
How it works now
There are the following configurations:

--submit-genesis CLI flag to enable submitting for a particular field
IROHA_GENESIS env var with path to genesis file (defaults to genesis.json)
genesis.public_key (GENESIS_PUBLIC_KEY) parameter that must be set for ALL peers
genesis.private_key (GENESIS_PRIVATE_KEY) parameter that must be set when --submit-genesis is set.

What is wrong:

IROHA_GENESIS can only be set via ENV
Intention to submit genesis is duplicated by setting --submit-genesis and genesis.private_key

Proposed change

genesis.public_key, required for all
genesis.private_key and genesis.file for the peer that submits the genesis
genesis.file might have a default value (genesis.json) which is applied if only genesis.private_key is set. Although it might make behaviour somewhat unclear. I think enforcing genesis.file to be set explicitly will be more reliable.

Benefits:

Flexible: genesis might be configured via only config file, or via only ENV, or mixed
Simple: no extra CLI flag

Thanks, that looks reasonable. If this issue is copied to the docs with minimal changes, it will help the users a lot in the setup stage.

Related discussion: https://github.com/hyperledger/iroha-2-docs/pull/397#discussion_r1342220725

What is the error handling procedure in case multiple peers have genesis private key specified?

What is the error handling procedure in case multiple peers have genesis private key specified?

@Mingela, I don't know how the situation of multiple peers submitting genesis blocks is handled now. Anyway, this proposal doesn't change anything here. It only changes the way of configuration.

Fine to me then


The wiki redirects to unrelated page

The wiki link on the README redirects to this page: http://sandbox.jpn.org/rcrswiki/index.php?title=Main_Page

Thank you for your report. I fixed it.

http://sandbox.jpn.org/rcrswiki/index.php?title=Main_Page
Since anyone can change the content a bot is accessing your wiki page to put advertising content. I think you should add authentication just before editing, to make sure that only the registered users will change the page. Best,

I'm sorry for the delay in reporting.
I temporarily closed the wiki because there was over 100  thousand page view per a day.
We will provide a new web page for the agent developer.


sd.Stream sounds not smooth

When I use this code, the audio sounds not smooth.  I am trying recording and playing at the same time, but I found it worded bad on mac, but normal on windows. I got a noisy sound on mac, when the recording is open. But it sounds normal when I stopped the recording process.
def stream_generator(blocksize, *, channels=1, dtype='float32',pre_fill_blocks=10):

    data, fs = sf.read("tmp.wav", always_2d=True)

    def callback(indata, outdata, frame_count, time_info, status):
        global current_frame
        chunksize = min(len(data) - current_frame, frame_count)
        outdata[:chunksize] = data[current_frame:current_frame + chunksize]
        if chunksize < frame_count:
            outdata[chunksize:] = 0
            raise sd.CallbackStop()
        current_frame += chunksize

    stream = sd.Stream(samplerate=22050,blocksize=4096, device=[0,1], callback=callback, dtype=dtype,
                       channels=1)
    with stream:
        event.wait()

stream_generator(1024)

The event variable is not defined, or am I missing something?
Do the example programs work on mac/windows?


Question: about this SDK and eBay 2017 Restful API

Not sure where to ask this, and I am a little confused about this SDK and the new REST API eBay launched last year.   Does this package use any of the new API, or is it strictly for the older SOAP based API.
Thanks!

git offers a search, nice thing if u ask me 😏

Thanks.  So the answer is both API's are mixed in this package.
Does anyone have any opinions if creating a new APP we should use the new API(does anyone know if eBay will deprecate the non REST version)?
The reason I ask is because I have working code working from an old app with non-REST version of the API, but I should update it won't work in a couple years.

That question does not really has something to do with this SDK, but anyway, it's up to you if you use rest or not. I'm not aware of any statement that says, old APIs will be shut down in the near future.
You can ask ebay developer support for a reliable answer.

@timramseyjr , this SDK is for REST API and for SOAP API, you can use both.

Yes @timramseyjr, if ebay does not declare bankruptcy crushed by amazon, the SOAP api will be deprecated in about 3/4 years (eBay not say when).


set_terminal_states does not calculate terminal_states_probabilities

Thanks so much for developing this tool.
I am analysing my own dataset for which RNA velocity does not work well, so I have followed the "CellRank beyond RNA velocity" tutorial and it worked very well for fate probability inference.
Now I would like to infer the lineage drivers but when running the tl.lineage.drivers, the following error occurred.
KeyError: "Unable to find transition matrix in `adata.obsp['T_fwd']`."
Since this indicates that the transition matrix is not updated to the adata object, I updated it manually by:
adata.obsp['T_fwd'] = g_fwd.transition_matrix
Then, tl.lineage.drivers gave a different error:
RuntimeError: Compute absorption probabilities first as `cellrank.tl.lineages(..., backward=False)`.
In fact, adata does not have .obs['terminal_states_probs'] column, and what is more, I found that g_fwd.terminal_states_probabilities is empty after manually setting the terminal states by set_terminal_states
This issue was reproduced with the pancreas dataset, and I attach the notebook including my codes and the reproduced errors.
I would be grateful if you could enlighten me on how to update an adata with required information (eg, obsp['T_fwd'], obs['terminal_states_probs'], etc.) for the downstream lineage driver inference.
cellrank_pancreas_PseudotimeKernel.ipynb.txt

Hi @tt2190 , sorry for the late reply.

Now I would like to infer the lineage drivers but when running the tl.lineage.drivers, the following error occurred.
KeyError: "Unable to find transition matrix in adata.obsp['T_fwd']."
Since this indicates that the transition matrix is not updated to the adata object, I updated it manually by:
adata.obsp['T_fwd'] = g_fwd.transition_matrix

Since I assume you have g: GPCCA object, why not just run g.compute_lineage_drivers? That way, you don't have to set the transition matrix in AnnData.

In fact, adata does not have .obs['terminal_states_probs'] column, and what is more, I found that g_fwd.terminal_states_probabilities is empty after manually setting the terminal states by set_terminal_states

For lineage drivers computation, you need to have data in, e.g., adata.obsm['to_terminal_states'], which contain the fate (absorption) probabilities. .obs['terminal_states_probs'] is juts a vector that gives you the probability of each cell being a terminal state (doesn't matter which) and is not used in driver computation.

I found that g_fwd.terminal_states_probabilities is empty after manually setting the terminal states by set_terminal_states

Yes, this is by design. Once you set your terminal states, you still need to compute the absorption probabilities towards them (this also needs the transition matrix) using g.compute_absorption_probabilities.
Then you should be able to run g.compute_lineage_drivers. Am attaching a small code snippet below in case anything is not clear:
import cellrank as cr

adata = cr.datasets.pancreas_preprocessed()

k = cr.tl.transition_matrix(adata)
g = cr.tl.estimators.GPCCA(k)
g.set_terminal_states({"Beta": adata[adata.obs['clusters'] == "Beta"].obs_names[:30],
                       "Alpha": adata[adata.obs['clusters'] == "Alpha"].obs_names[:30]})
g.compute_absorption_probabilities()
df = g.compute_lineage_drivers()

Many thanks @michalk8,
I was not aware of the g.compute_lineage_drivers method and it now works with my data!


Factories should be linted before specs run

Probably like this: https://github.com/thoughtbot/factory_girl/blob/master/GETTING_STARTED.md#linting-factories


continue in repeat loop gives invalid code

Lua language allows checking local variables from repeat loop in until statement:
When continue is used in repeat loop in yuescript, invalid code is generated:
x = 0
repeat
  x += 1
  y = x
  continue if x < 5
  print y
until y == 10

generated lua:
local x = 0
repeat
  local _continue_0 = false
  repeat
    x = x + 1
    local y = x
    if x < 5 then
      _continue_0 = true
      break
    end
    print(y)
    _continue_0 = true
  until true
  if not _continue_0 then
    break
  end
until y == 10

y is defined in nested repeat loop in generated code and is invisible for until y == 10 check, so the loop will never return.

Made a new commit for continue used in repeat.
x = 0
repeat
  x += 1
  y = x
  continue if x < 5
  print y
until y == 10

now compiles to:
local x = 0
repeat
  local _cond_0 = false
  local _continue_0 = false
  repeat
    x = x + 1
    local y = x
    if x < 5 then
      _continue_0 = true
      break
    end
    print(y)
    _cond_0 = y == 10
    _continue_0 = true
  until true
  if not _continue_0 then
    break
  end
until _cond_0

x = 0
repeat
  x += 1
  y = x
  continue if x < 5
  print y
until y > 0

continue currently skips the until check completely. Shouldn't the above loop exit on the first iteration without printing anything?

while + continue generates 2 times _continue_0 = true around break.
while true do continue:
while true do
    local _continue_0 = false
    repeat
        _continue_0 = true
        break
        _continue_0 = true
    until true
    if not _continue_0 then
        break
    end
end

x = 0
repeat
  x += 1
  y = x
  continue if x < 5
  print y
until y > 0

continue currently skips the until check completely. Shouldn't the above loop exit on the first iteration without printing anything?

I just forgot about the control flow check.
Tried to make it work this way.
local x = 0
repeat
  local _cond_0 = false
  local _continue_0 = false
  repeat
    x = x + 1
    local y = x
    if x < 5 then
      _cond_0 = y > 0
      _continue_0 = true
      break
    end
    print(y)
    _cond_0 = y > 0
    _continue_0 = true
  until true
  if not _continue_0 then
    break
  end
until _cond_0

while + continue generates 2 times _continue_0 = true around break.
while true do continue:
while true do
    local _continue_0 = false
    repeat
        _continue_0 = true
        break
        _continue_0 = true
    until true
    if not _continue_0 then
        break
    end
end

The extra _continue_0 = true is used for implementing the break keyword when both continue and break being used in a loop.
while cond
  continue if condA
  break if condB
  print "work"

compiles to:
while cond do
  local _continue_0 = false
  repeat
    if condA then
      _continue_0 = true
      break
    end
    if condB then
      break
    end
    print("work")
    _continue_0 = true
  until true
  if not _continue_0 then
    break
  end
end

x = 0
repeat
    x += 1
    continue if x < 5
    y = x
    print y
until y == 10

what about this one? _conf_0 checks y before it is defined in the loop.

Seems to generate the expected result.
local x = 0
repeat
  local _cond_0 = false
  local _continue_0 = false
  repeat
    x = x + 1
    if x < 5 then
      _cond_0 = y == 10 -- continue and get the until check before y declared
      _continue_0 = true
      break
    end
    local y = x
    print(y)
    _cond_0 = y == 10
    _continue_0 = true
  until true
  if not _continue_0 then
    break
  end
until _cond_0


HWY-283: Initialize round success meter with current timestamp.

Initializing it with the era start time makes it likely to unnecessarily slow down after downtime, because it will count all rounds since the era's beginning as "failed".
https://casperlabs.atlassian.net/browse/HWY-283

bors merge

bors r-

Actually...

bors merge


Sending test coverage to Code Climate

testing PR with coverage report

https://codeclimate.com/github/codeclimate/codeclimate/badges

So my understanding is if we merge this, we'll have a coverage metric on our code climate dashboard, which is now missing.

Perhaps CC needs a but more configuring. And while it's not a show stopper, I also see a lot of weird analysis from CC on other PRs, which I am not sure are useful ATM. Is there a way to disable those, until we can commit ourselves to fixing them?

Yes, analysis can be configured using the config file I just added - which currently turns on some extra analysis plugins. Here's a description of what can be fine tuned and how:
https://docs.codeclimate.com/v1.0/docs/configuring-your-analysis


Fix the generated code of FactoryWithParam with asType

I faced an error of  generated code when I used FactoryWithParam with asType.
The original generated code will produce like this
exampleLocator.registerFactoryParam<FactoryService, String?, double?><IFactoryService>((param1, param2) => FactoryService(key: param1, value: param2));

I tried to fix and then the revised generated code will produce like this
exampleLocator.registerFactoryParam<IFactoryService, String?, double?>((param1, param2) => FactoryService(key: param1, value: param2));

This functionality has been added to the newest PR that adds testing around all generators. It will be in production soon.


Ctf improvments

Hi,
we have hosted a couple of CTF events in the last month and have made a few improvements to the challenge notification we would like to merge back to you ;)
The improvements include:

Ability to resend a challenge solved notification, in case the user closed the notification before copying the key. The re sending is triggered by clicking on the challenge solved notification on the scoreboard.
Ability to copy the key via a copy to clipboard button (using the already imported ngclipboard package)
Ability to hide the ctf key in the challenge solved notification via a config options as they can be distracting to user not in a ctf environment. I added a ctf config file to demo this

I have also removed the fallback to the config application name to better be able to hide the application name during ctf events to try to prevent cheating a bit better.
Thank you for reading this and for making this awesome application!

Wow, that is awesome! Thanks a lot! I'll merge it into develop and do some testing, maybe a bit of re-wording the descriptions and then it'll be part of v2.26! :+1:

I think it makes sense to only offer the re-send feature if notifications and CTF-mode are enabled, right? Otherwise it might confuse people. Following @wurstbrot's explicit and self-explaining naming scheme in the config, we could also call the property e.g. showCtfFlagsInNotifications instead of ctfEnabled?
showChallengeSolvedNotifications: true
showCtfFlagsInNotifications: true
# = Show notifications with flags.
# Also show copy/paste-button for flags as well as repeat-notification feature via scoreboard.

showChallengeSolvedNotifications: true
showCtfFlagsInNotifications: false
# = Show notifications but hide flags.
# Hide repeat-notification feature from scoreboard.

showChallengeSolvedNotifications: false
showCtfFlagsInNotifications: true|false
# = Hide notifications and re-send feature.

Do you maybe want to do those changes before I merge?

Yeah that makes sense.
I'll look into implementing this this afternoon.

I added the functionality.
But having the ctf mode disabled by default means it is also disabled in the e2e test which leads to failing the test case i wrote for this feature.
Do you have an idea how to test features which are disabled by default in a nice way?

Hm, I didn't realize CTF would be disabled by default... The corresponding test could enable it before it runs, I guess. Just enabling it by default would also be okay as a workaround.

Fixed the issue with that e2e test (it checks now wether CTF is on or off and uses a correspondig it() definition)  and reduced the ctf.yml to only overriding the necessary setting.
Thanks again to @J12934 and @kkottke!!! 🥇


Suggested improvements for the proof of Lindenbaum's lemma

I have some thoughts on improving the proof for Lindenbaum's lemma. I am not sure if they can be approved if I make a pull request, so I am opening a discussion here first.

I found this paragraph quite hard to follow:

I think it is clearer if it is written this way:
From this, it follows that every finite subset $\Gamma'\subseteq\Gamma^\ast$ is a subset of $\Gamma_n$ for some $n\in\mathbb N$, since for each $B\in\Gamma'$ not already in $\Gamma_0$ is added at some stage $i$. If $n$ is the last one of these stages, then all $B\in\Gamma'$ are in $\Gamma_n$, i.e. $\Gamma'\subseteq\Gamma_n$.

The overall proof is a bit chunky to digest. I think the third and fourth paragraphs contribute to unnecessary bloat. Why can't we reuse the fact from the proof of lemma 12.6 that if $\Gamma^\ast$ is inconsistent, then $\Gamma_n$ is inconsistent for some $n$ to quickly finish the proof for $\Gamma^\ast$ is consistent?

The "fact" isn't proved in lemma 12.6, and what's done here is in fact the proof.
How about this for a rewrite:
From this, it follows that $\Gamma^\ast$ is consistent. Let $\Gamma' \subseteq \Gamma^\ast$ be finite. Each $B \in \Gamma'$ is also in $\Gamma_i$ for some $i$. Let $n$ be the largest among these. Since $\Gamma_i \subseteq \Gamma_n$ if $i < n$, every $B \in \Gamma'$ is also $\in \Gamma_n$, i.e., $\Gamma' \subseteq \Gamma_n$. Thus, every finite $\Gamma' \subseteq \Gamma^\ast$ is consistent. By Propositions 10.17 and 11.17, $\Gamma^\ast$ is consistent.

Ah, I see. So, this lemma's proof contains the solution to the exercise from lemma 12.6's proof?
Yes, this rewrite is more clearer. I'd appreciate it if you update the proof with this new writeup.

Thanks!