(feature) Design tokens

Pull Request
📖 Description
This PR enables working with DesignTokens from the c# side
🎫 Issues
#114
👩💻 Reviewer Notes
A DesignTokens class which exposes all the predefined tokens as properties has been added. All the token names have been defined as public c# const in the DesignTokenNames class.
In Progam.cs you add builder.Services.AddFluentUIComponents(). This is an extension method that adds the new DesignTokens 'service' to the DI container. This is also being leveraged for adding the IconService for the FluentIcon implementation.
When needed you can inject the DesignTokens instance and call the DesignToken methods (SetValueFor, GetValueFor, DeleteValueFor and WithDefaults) on an ElementReference , i.e
public partial class Index
{
    [Inject]
    private DesignTokens DesignTokens { get; set; } = default!;

    private FluentAnchor ref1 = default!;
    private FluentAnchor ref2 = default!;
    private FluentAnchor ref3 = default!;
    private FluentButton ref4 = default!;

    private int? theValueBeforeDelete;
    private int? theValueAfterDelete;

    protected override async Task OnAfterRenderAsync(bool firstRender)
    {
        if (firstRender)
        {
            // Use Dark mode for this button
            await DesignTokens.BaseLayerLuminance.SetValueFor(ref1.Element, 0);

            //Enabling this line below will generate an error because no default is set
            //await DesignTokens.BaseHeightMultiplier.SetValueFor(ref2.Element);

            //Make this button bigger
            await DesignTokens.BaseHeightMultiplier.WithDefault(25).SetValueFor(ref3.Element);

            theValueBeforeDelete = await DesignTokens.BaseHeightMultiplier.GetValueFor(ref4.Element);

            //Make this button huge
            await DesignTokens.BaseHeightMultiplier.SetValueFor(ref4.Element, 52);

            //ToDo: Create your own DesingToken
            //DesignToken<string> specialColor = new(jsRuntime!, "specialColor");
            //await specialColor.SetValueFor(ref3.Element, "#FF0000");
            StateHasChanged();
        }
    }

    public async Task OnClick()
    {
        await DesignTokens.BaseHeightMultiplier.DeleteValueFor(ref4.Element);

        theValueAfterDelete = await DesignTokens.BaseHeightMultiplier.GetValueFor(ref4.Element);
    }
}

for this index.razor file
@page "/"

<PageTitle>FluentUI WC home page</PageTitle>

Blazor Fluent UI Web Components home page!

<FluentAnchor @ref="ref1" Appearance="Appearance.Filled" Href="handlers" >Web components handlers</FluentAnchor>
<FluentAnchor @ref="ref2" Appearance="Appearance.Filled" Href="bindings" >Web components bindings</FluentAnchor>
<FluentAnchor @ref="ref3" Appearance="Appearance.Filled" Href="componentbindings" >Blazor component bindings</FluentAnchor>
<FluentButton @ref="ref4" Appearance="Appearance.Filled" @onclick=OnClick>FluentButton</FluentButton>

The value before calling Delete: @theValueBeforeDelete
The value after calling Delete: @theValueAfterDelete

✅ Checklist
General

[X] I have added tests for my changes.
[X] I have tested my changes.
[ ] I have updated the project documentation to reflect my changes.
[X] I have read the CONTRIBUTING documentation and followed the standards for this project.

⏭ Next Steps
TBD

Unfortunately, we aren't able to prioritize the JavaScript side of this work right now.
Not sure if @javiercn has time to review the questions/concerns above.

@EisenbergEffect I'm a bit overloaded with other work, but I'll try to get some time next week to look at it in more detail.

@vnbaaij Let's use this PR as an opportunity to start the collaboration with @williamw2. After you both have reviewed it, I think it's fine to merge and make a new release.

@EisenbergEffect Totally agree!

This is much nicer with the new script integration! For the future, let's try to keep PRs a bit more focused though. This PR has a bunch of refactoring that doesn't appear to be related to the DesignToken work. Next time, let's split that into its own PR.

Totally agree. I got a bit caried away with reorganizing things that should not have been done here.

Fix #114


docs: fix simple typo, domention -> dimension

There is a small typo in codes/python/advanced/custom_training.py, codes/python/advanced/dataset_generator.py, codes/python/neural_networks/cnns.py.
Should read dimension rather than domention.
Semi-automated pull request generated by
https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


[RFC]: improve README examples of stats/iter namespace

Description
This RFC proposes to improve the README examples of the stats/iter namespace package. Currently, namespace package READMEs only include minimal examples which are not particularly informative and do not showcase namespace functionality.
Historically, namespace examples were not given much effort due to the rapidly changing nature of the project; however, now that development and organization has settled, it would be good to revisit these packages and add proper examples to better communicate to users how to use namespace contents.
For those wanting to contribute to this issue, you can use the following steps:

Browse through the contents of the stats/iter namespace and identify interesting functions and packages to showcase.
Update the example code block in the "Examples" section of the README for the stats/iter namespace and open a PR.
When updating the README, examples should follow existing project conventions (e.g., ES5).
Examples may require functionality from other packages to help make examples more interesting.

Related Issues
None.
Questions
No.
Other
No.
Checklist

[X] I have read and understood the Code of Conduct.
[X] Searched for existing issues and pull requests.
[X] The issue name begins with RFC:.

i am working on this issue.

Hey @Planeshifter  there are already average of about 60-70 examples do you want to increase the examples or you want some interesting examples to showcase .

@pankaj-2503
Specifically, this PR is about changing the namespace example code block, which currently looks like this:
var getKeys = require( '@stdlib/utils/keys' );
var ns = require( '@stdlib/stats/iter' );

console.log( getKeys( ns ) );

Instead, it would be good to show a few functions and how they are used. It's better here to be selective, for example we could show one example with for a regular iterator and then also one using an iterator that computes a moving statistic, i.e. one supporting a window (W) parameter. Would be good to not just reuse examples from the individual packages, but for example come up with one conceptual example that involves multiple packages.
Hope this makes sense. Please let me know if you would like to give this a shot!

Yeah i would like to work on this updation of readme file .

I would like to contribute to this read me files . Also can you show an example for the same, i just landed on this repo


Constraints with an existential variable produced by kmxwasm cannot be internalised by Booster

Executing a proof test_change_quorum-int2val-spec.json) using the current version of kore-rpc-booster produces s number of warnings that boil down to Booster's inability to internalize constraints with existential variables.
The pretty-printed constraint looks like this:
#And #Not ( #Exists CODE:ModuleDecl . { AbstractCode_1:Code #Equals CODE:ModuleDecl } )

and seems to be introduced by abstracting the contents of the <code> cell (I'll need @virgil-serbanuta input on this to confirm).
I cannot say if the proof is hindered in any way by the Booster's inability to internalize these. I think Kore can still deal with them if necessary.
I'm attaching a minimized bug report that contains:

the Haskell and LLVM definition.kore
an execute request that leads to a three-way branch, the third branch containing an existential;
a simplify request with the predicate from the third branch that contains the existential;
responses to the requests for kore-rpc-booster, booster-dev and kore-rpc-dev

kmxwasm_bug_report_existential.tar.gz
booster-dev fails with:
[NotSupported (KJExists {sort = SortApp {name = Id {getId = "SortGeneratedTopCell"}, args = []}, var = Id {getId = "VarCODE"}, ...

while kore-rpc-dev and kore-rpc-booster return the predicate as-is (that is fine).
I do not think there's an immediate action needed. @virgil-serbanuta please let us know your views on this.

Hmmm.... I'm not sure how you go that constraint, but the backend should not interact at all with abstracted cells. Execution should stop just before touching them (Kasmer passes some cut point rules to the backend), Kasmer should replace the abstracted cells with normal content, then it should take one step to move past the instruction that touches the cells (this step can be repeated if needed), and then it should resume normal execution.


fix: improve compute new state root when producing block

Motivation
Improve "computeNewStateRoot" function
Description

computeNewStateRoot was so slow because we didn't compute hash tree root to cache in "prepareNextSlot". The main fix for this is to do it in "prepareNextSlot"
Also track state.hashTreeRoot() call in 3 places: verifyBlocksStateTransitionOnly(), prepareNextSlot() and computeNewStateRoot()
Also correct the bucket of that metrics as current values are not suitable for different networks

in holesky, it could be up to 1.8s, see #6194
in mainnet, if the value is <50ms we're good so no need to track values less than that. Need to track more values between 0.5s and 1.5s

Closes #6194

On mainnet, state.hashTreeRoot() for computeNewStateRoot() runs mostly < 50ms

On holesky, state.hashTreeRoot() for computeNewStateRoot() runs mostly 200ms to 500ms

:tada: This PR is included in v1.13.0 :tada:


Potential fix for #190

See comments in #190. Someone please test first! 😮

This, my friend, will not work, because the word "Unpublished" has the word "published" in it as well. So if we look for anything containing "published", we'll mark both "Unpublished" and "Published" things as "Published".
I'll take another swing at it and see if I can get it landed. Looks like fixing the back corpus won't be too bad either.

Ha! Of course. Good catch there.


Kingfisher Support Base64 ?

If I use the Kingfisher library and want to use it with Base64 images, can this library work? How can I do it?

Yes, the Kingfisher library supports loading images from Base64 encoded strings. You can achieve this by providing the Base64 encoded string directly to Kingfisher's KFImage or UIImageView instance.
Here's how you can load a Base64 encoded image using Kingfisher in Swift:
import Kingfisher
import UIKit
// Assuming you have a Base64 encoded string representing your image
let base64String = "YOUR_BASE64_ENCODED_IMAGE_STRING_HERE"
// Convert the Base64 string to Data
guard let imageData = Data(base64Encoded: base64String) else {
// Handle invalid Base64 string
return
}
// Create a Kingfisher image provider from the Base64 encoded Data
let provider = ImageDataProvider(data: imageData)
// Use Kingfisher to load the image
KFImage(provider)
.resizable()
.placeholder(UIImage(named: "placeholder")) // Optional: Provide a placeholder image
.onFailure { error in
// Handle image loading failure
print("Error loading image: (error)")
}
.onSuccess { _ in
// Image loaded successfully
print("Image loaded successfully")
}
.load()
By following these steps, you can easily load Base64 encoded images using the Kingfisher library in your iOS app.
Replace "YOUR_BASE64_ENCODED_IMAGE_STRING_HERE" with your actual Base64 encoded image string.
Convert the Base64 string to Data using Data(base64Encoded:).
Create a ImageDataProvider instance with the Base64 encoded image data.
Use KFImage to load the image, passing the ImageDataProvider as the image provider.
Optionally, you can specify a placeholder image using the placeholder() modifier.
Handle loading success or failure using the onSuccess and onFailure modifiers.

@Theeranail
There is a Base64ImageDataProvider you can use once you have the base64-encoded string.
https://github.com/onevcat/Kingfisher/wiki/Cheat-Sheet#image-from-base64-string


Changed validation message by type

I think it's better change the validation message by type and ensure it sending a "undefined" message, because an empty string message is already a message. Besides, we avoid multiple conditionals into an if declaration.

The Travis build failed because this error: Uncaught CredentialsError: Missing credentials in config

A log is must contain something. An empty string is not a log.


CommitMsg?

When using apply from HERMIT.Plugin, what does the CommitMsg parameter mean?

Sorry, forgot to reply to this.
The CommitMsg argument specifies whether to record the rewrite in the history tree. If you don't care about being able to use step/goto/etc, you can pass Never. Always causes an entry to be added even if nothing changes, Changed only adds an entry if something changes.
There is probably a better abstraction there... it was mostly hacked in during the run up to our last paper.

Ah. Sounds like it's relevant some HERMIT territory I haven't discovered yet. Thanks for the explanation.


Building a Stopwatch with Javascript and Tailwind CSS

Proposal Submission
Proposed title of article
Building a Stopwatch with Javascript and Tailwind CSS
Proposed article introduction
Are working on a project that needs a stopwatch or you are performing a task that requires timing? in this article, we will be building a simple stopwatch with javascript and styled with a CSS framework called Tailwind CSS.
If this is your first time using Tailwind, we will be learning briefly how to set up Tailwind CSS using the docs  and get it ready for use.
Key takeaways

The reader will learn how to set up Tailwind
The reader will how to use Tailwind in their application
The reader will learn how to build a simple stopwatch

References
N/A

Please include more key takeaways. I believe introductory aspects of tailwind have already been covered by other articles. Thanks

@WanjaMIKE
Proposal Submission
Proposed title of article
Building a Stopwatch with JavaScript and Tailwind CSS
Proposed article introduction
Are you working on a project that needs a stopwatch or performing a task that requires timing? in this article, we will be building a simple stopwatch with JavaScript and styled with a CSS framework called Tailwind CSS.
If this is your first time using Tailwind, we will be learning briefly how to set up Tailwind CSS using the docs and get it ready for use.
Key takeaways
The reader will learn how to set up Tailwind
The reader will learn how to use Tailwind in their application
The reader will learn how to build a simple stopwatch
The reader will learn what JavaScript EventListener is and how to use it
The readers will learn how to automate activities in an application with the use of the stopwatch they have created
References
N/A

Hello @WanjaMIKE I added new key takeaways are they enough or should I add more


Accepting a JSON Request Body

Thank you for Mappersmith! It's an amazing client ;)
The documentation says that a urlencoded version of the object will be used when sending values in the request body.
I wonder if that's possible to check the content type before creating a urlencoded version. In my case, the API only works with json objects.
Is there any way to send a json object in the request body?
thanks!

Hi @albertoleal, thanks. It's possible to send whatever you want to :)
Mappersmith will convert only objects to urlencoded, for a JSON body you can do:
Client.Photo.save({
  category: 'family',
  body: JSON.stringify({year: 2015, tags: ['party', 'family']})
})

Great, @tulios! Thanks!


Use blazor version in project json

place blazors version inside the generated package json

Ah I think there are a couple more changes here than should. It's very important that the v5 is Blazor v5 (and therefore .NET 5.0) compatible.


Should track_algo.track_with_expanded_transitions() return a Track instead of a list?

Hi!
While working on #363 I discovered that track_algo.track_with_expanded_transitions() returns a new "track" based on an empty list. Should this be an actual schema.Track?
def track_with_expanded_transitions(in_track):
    """Expands transitions such that neighboring clips are trimmed into
    regions of overlap.

    For example, if your track is:
        Clip1, T, Clip2

    will return:
        Clip1', Clip1_t, T, Clip2_t, Clip2'

    Where Clip1' is the part of Clip1 not in the transition, Clip1_t is the
    part inside the transition and so on.
    """

    #result_track = []
    result_track = schema.Track(kind=in_track.kind)
    ...

The reason I'm asking is that when trying to figure out if a Clip's parent is an audio track I don't get a valid result when calling item.parent(). This I believe is because it's not from a proper Track.

I got this working by changing another line in the same function replacing
result_track.append(_expand_transition(thing, in_track)) with result_track.extend(_expand_transition(thing, in_track))
Also had to do some tweaks to the indexing in test_track_algo.test_expand_surrounded_by_clips(self) to make all test pass.
Should I create a PR for this so you can check it out?

Its a good question, and if we don't make it return a track we should at least explain why we aren't returning a track back.
In a track, transitions are next to items and "eat into" the items  they're next to.  Given a track like this:
[A, B, t, C, D]
Where A, B, C, and D are clips and t is a transition, track_with_expanded_transitions() will return a list like this:
[A, B', (Bt, t, Ct), C', D]
Where B' is everything in B outside the transition, Bt is the chunk of B that is inside the transition.
This doesn't really follow the behavior of a track because the (Bt, t, Ct) tuple plays at the same time, with whatever the transition is applied to it.  It also makes it easy to find the parts of the track that are 'inside' the transition.
I think we should probably rename this function so that it doesn't sound like it returns a track.  In your case, can you cache whether it was audio or video before you pass it in?  What are you doing with these objects?

Hi!
Thanks you your insights.
I'm working on support for audio clips in the RV adapter. In the adapter I want to use the Clips.parent() method to determine if it lives in an AudioTrack or not.
The clips in the list returned from track_with_expanded_transitions() yield None when parent() is called.

In your case, can you cache whether it was audio or video before you pass it in? What are you doing with these objects?

Yes, I most definitely can. I can't understand why I didn't think of that. (palm face)
Thanks again!

I'll close this for now.


Robert Columbia: Watch jobsatags.com

Robert Columbia requests the watch of the watch_keyword jobsatags\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

jobsatags\.com has been seen in 3 true positives, 0 false positives, and 0 NAAs.

Approved by Mast in Charcoal HQ


Add to mainnet blockchain registries

TFTC -> FTC
TBTC -> BTC

Also move hyperswarm protocol to config so that mainnet and testnet nodes can operate on different networks.


Execute One-Off Task

We are currently building out our infrastructure of long running services on top of marathon and were wondering if there were a way to execute a task once (or at least once) and then not relaunched if something like the exit code was zero?
The use case, for us, is to run database migrations from our Docker containers.

Have you looked into chronos for this scenario? It seems to be a fit.

https://github.com/mesosphere/chronos

@jamescun @elingg @mlunoe
For such a simple use case, I don't think it's worth having a dedicate framework that you need to care for like Chronos.
Also there is (and was when this was created) an issue for this feature in marathon already https://github.com/mesosphere/marathon/issues/344


Match on generated controller specs

It would be great if the gem would match (and not regenerate) controller specs that are generated by Rails scaffolding.
Rails prepends the describe description with the HTTP method used for that controller action, e.g.:
describe "GET #index" do
describe "GET #show" do
describe "GET #new" do
describe "GET #edit" do
describe "POST #create" do
describe "PUT #update" do
describe "DELETE #destroy" do

Good idea. Welcome a PR!


copr: fix decimal

@coocood @siddontang @disksing PTAL

Coverage increased (+0.1%) to 81.868% when pulling 572ea30a9c743c1df3acf8539a40edd3c9066d4a on busyjay/fix-decimal into 5322d2bdb17fb353ceda41002020ea4ebe68d96d on master.

LGTM

should we add some invalid string test?

Added already, see test_parse_decimal_fail.

LGTM

This bug was found by evaluating a comparison expression, we should add some test cases in util/xeval.

LGTM

Coverage decreased (-0.09%) to 81.71% when pulling da06c2bd0b3a0d7e2dbb0ec151ac5b540a3d9fb6 on busyjay/fix-decimal into f59c2a38e68c3b68d5056febf81fcd49b423afd1 on master.


feat(controller): Allow to disable leader election (#5638)

Leader election uses the coordination.k8s.io kubernetes api with is only available in kubernetes >= 1.14.
By settings the environment variable LEADER_ELECTION_IDENTITY=off leader election can be disabled for the workflow controller.

1.14 is a quite old distribution. Any reason to stay below 1.14?

Alas, Openshift 3.11 is still used in some enterprise environments and it is based on kubernetes 1.11.

Can you please fix the make lint issue? I've assigned to @sarabala1979 as he worked on this originally.

@terrytangyuan Thb, I don't know if there are other parts which are not compatible with k8s < 1.14, but so far I have been running it on OpenShift 3.11 successfully.

Will RBAC ok or do we need to add/remove it based of flag?
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - create
  - get
  - update```

Will RBAC ok or do we need to add/remove it based of flag?
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - create
  - get
  - update```

RBAC wil have no problem with this.

Syncing with master should fix tests.


urlencode encodes space with +, youtrack expects space encoded with %20

We have subsystems (and other values) with spaces. e.g.
$youTrack->getSubsystem('KS', 'Rechnung, Vertrag');
results in a
404 Error for 'https://server/rest/admin/project/ST/subsystem/Rechnung%2C+Vertrag': 404: Subsystem [ Rechnung,+Vertrag ] not found.
Youtrack expects spaces encoded with %20, but urlencode uses + instead.

Thanks!


Remove http client events on normal requests, write if an error occurs

Remove http client events on normal requests, write if an error occurs

Do we need the empty functions? Can we remove them?

Do we need the empty functions? Can we remove them?

We can, but I didn't want to change the api, It's just like #126 that we didn't want to change the original structure of the code.


rm_file doesn't remove from toggle_quick_menu

If you M.add_file() and then M.rm_file(), the file can appear as "(empty)" in the quick_menu.
If you add_file() 3 files:
1.txt
2.txt
3.txt
and then rm_file() 2.txt
and then pull up M.toggle_quick_menu(), the 2nd entry is "empty"

I believe this is the intended functionality. You would not want the 3rd mark to disappear, so you get a "holed" config. If you add after, the empty spot will be taken, or at the tail.
The (empty) is just a label to show that that spot is not filled: https://github.com/ThePrimeagen/harpoon/blob/21f4c47c6803d64ddb934a5b314dcb1b8e7365dc/lua/harpoon/ui.lua#L102-L104
The lua to json conversion does not like holes in the config: https://github.com/ThePrimeagen/harpoon/pull/14#issuecomment-786976954
I might be wrong - it's been a while since I did changes in harpoon, and I know the codebase has evolved.


Incompatible with Lolex

First of all, I'm not sure if this is an issue with Bottleneck or Lolex (used by Sinon to stub setTimeout and friends, usually to speed up tests).
If I install Lolex and then use Bottleneck to schedule two asynchronous operations, only the first of them will execute; the other one will stay pending. Internally, Bottleneck uses setTimeout to schedule, so it looks to me like there's a callback that's not getting executed.
I've created a minimal working example. The same thing happens if I use submit with a callback instead of schedule with a Promise, a well as with Bluebird instead of the native Promise object. Also, the version of Node doesn't seem to matter. Finally, I'm on Windows, but I sure hope that's not the cause.
I've tried to dig into the Bottleneck code but wasn't able to come up with anything useful. I did notice that there's only one clearTimeout and it doesn't get called when I run my example.
Thanks!

Actually, if I replace Lolex with my own implementation, it works. I'm going to file an issue with Lolex instead. Closing this.

I've only glanced at this but won't this never run because it requires a tick first?
https://github.com/timdp/bottleneck-lolex-issue-demo/blob/lolex-mock/index.js#L27

On 12 Jun 2016, at 19:01, Tim De Pauw<EMAIL_ADDRESS>wrote:
Actually, if I replace Lolex with my own implementation, it works. I'm going to file an issue with Lolex instead. Closing this.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

@tjenkinson Not with my mock implementation, which only touches setTimeout, not setInterval.

Cool ok :+1:

On 12 Jun 2016, at 19:39, Tim De Pauw<EMAIL_ADDRESS>wrote:
@tjenkinson Not with my mock implementation, which only touches setTimeout, not setInterval.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.


Linkage monitor test

test

Closing due to inactivity. @suztomo feel free to reopen if you would still like to test something.


Inserts "end" when already inside valid block

Example:

The method already has an end. The extension should only insert an end when the AST is unbalanced.

Solved via 9b7f5a4df08cedf39bed1b25162894310cbd55cf

👍


Consider using JiraPS Module

https://atlassianps.org/docs/JiraPS
Replace home grown actions with this Powershell module.

Might be difficult to gracefully handle Jira connection issues without direct access to the requests.

JiraPS does not have the ability at this time to use the update operations.

JiraPS is not as maintained as once thought.  Better to stick with your API integration.


SHACL - Support for remote configuration

Users who talk to the ShaclSail through a HttpRepository and HttpRepositoryConnection should be able to:

enable or disable validation in the ShaclSail
call revalidate() in the ShaclSailConnection

@jeenbroekstra this is a use case that Ontotext has come with. Could you shed some light on how remote configuration is currently handled in RDF4J.
My natural instinct is to create a new class for HttpShaclSail and HttpShaclSailConnection that could be retrieved from the HttpRepository and HttpRepositoryConnection respectively. These would then expose the relevant methods to the user.

How do people who work locally currently do this?

What I'm getting at with that is that we should think about a generic mechanism to control such settings at the Repository API level, so that (either locally or remotely), people can control this without having to directly interact with the SAIL.
Normally, remote configuration is handled by sending an RDF-serialized RepositoryConfig (with a SailImplConfig etc) through the RemoteRepositoryManager, but that mechanism is not really suitable for these kinds of "per-connection" or even "per-operation" config settings.

I personally get the sail from the repo and cast it to a ShaclSail. Same for the connection. When working locally that is.
The RemoteRepositoryManager would work for disabling and enabling validation, since this is a global setting.
For interacting with the connection I really like the casting approach:
HttpRepositoryConnection connection ... ;
HttpShaclEnabledRepositoryConnection shaclConnection = connection.as(HttpShaclEnabledRepositoryConnection.class);

// revalidation
ValidationReport = shaclConnection.revalidate();

// custom transaction support
shaclConnection.begin(Validation.disabled);
....
shaclConnection.commit();

One thing I'm wondering about. Does the HttpRepositoryConnection support transactions fully. Essentially, do we have a POST /startTransaction endpoint?

On my phone so brief, but I'm not a fan of solving this through inheritance.
I'm thinking instead we add a new ConnectionSettings object that you can get/set on a RepositoryConnection.

One thing I'm wondering about. Does the HttpRepositoryConnection support transactions fully. Essentially, do we have a POST /startTransaction endpoint?

It does. Or rather: the RDF4JProtocolSession (which is used by the HTTPRepositoryConnection) does.

I personally get the sail from the repo and cast it to a ShaclSail. Same for the connection. When working locally that is.

We need a better mechanism for that. Users shouldn't need to interact with the Sail objects directly.
Let's do some rough sketches, how about making something like this possible:
RepositoryConnection conn  = ...;
conn.getSettings().set(ShaclSettings.validate, false);  // disable validation
conn.begin();
...

The result of RepositoryConnection.getSettings() would be a ConnectionConfig object, similar to how we have a ParserConfig in Rio, which takes a ConnectionSetting object and a value (similar to how ParserConfig takes a RioSetting object and a value). Once we have this in place it shouldn't be hard to add a generic serialization mechanism so that we can transmit these settings over the REST api.
If I understand the purpose of the revalidate method correctly, it is to kick off validation of the entire store without having to add new data (for example after you've added some data with validation disabled). If that is the case, I'm thinking that we could support this by having a generic RepositoryConnection.refresh() (and accompanying SailConnection.refresh()) that in case of the ShaclSail can interally call revalidate. The reason I want this generically solved is that it opens up other use cases (outside of SHACL validation) where internal components need to recompute things, and moreover it allows interaction with SHACL-enabled repositories without having to couple client code to specifics of the SHACL sail.

I think we should treat validation as a separate feature, like how we treat reasoning as a separate feature. Even if you don’t use a reasoner sail you still get the option to retrieve inferred triples.
Refresh would be a good option for a lazy reasoner.
Revalidate would be a good option for a lazy validator.
But this still doesn’t handle my wish to add a transaction begin option without validation. Though I’m not having second thoughts about that :P
I like the settings approach for the repository. Would make it immutable so the user would have to call repo.setSettings(my settings); or something like that. Makes it easier to handle for the http repo. I think i will look into coding that.

I think we should treat validation as a separate feature, like how we treat reasoning as a separate feature. Even if you don’t use a reasoner sail you still get the option to retrieve inferred triples.

To be honest, in retrospect I would have preferred that we had solved that differently - it's regularly caused confusion in our users who think that by supplying true to that parameter they "turn on inferencing".

Refresh would be a good option for a lazy reasoner.
Revalidate would be a good option for a lazy validator.

I don't see why there should be a naming difference for this at the core API level - both operations signal a message to refresh the state of the store. But if you really think revalidate is better, we can facilitate this at the Repository API level by providing a Wrapper or a util class of some sort, e.g. a ValidatingRepository(Connection) which wraps a normal Repository(Connection) and has an additional validate() method that calls refresh on the base RepositoryConnection.

But this still doesn’t handle my wish to add a transaction begin option without validation. Though I’m having second thoughts about that :P

I suggest we treat that as a separate issue for now. One possibility is that we add an optional argument to the begin method that takes a ConnectionConfig object. Or, alternatively, and more flexibly, we don't actually change the begin method at the core API level, but let that same wrapper or util at the Repository API level take care of this for us.

I like the settings approach for the repository. Would make it immutable so the user would have to call repo.setSettings(my settings); or something like that. Makes it easier to handle for the http repo. I think i will look into coding that.

That'd be great. Keep in mind that making the config immutable will make it harder to manipulate on the fly. I do agree that RepositoryConnection.getConfig() should retrieve a copy of the current config, but it should still be possible to tweak the individual settings in that config, so that you can do one-line config tweaks like this:
 connection.setConfig(connection.getConfig().set(ShaclSettings.VALIDATE, false));

unless that is what you meant with making it immutable, in which case, great :)

The problem with refresh() is that it doesn't tell the underlying store anything about what it should be refreshing.

For the ElasticsearchStore it could signal an index level refresh. Maybe rebuild the statistics.
For the NativeStore it could signal a flush and maybe a recalculation of statistics (dunno if we have statistics for the NativeStore). It would also signal a compaction of the hashmap. Realignment of the values files. Or maybe re-indexing data into new trees if the user changes which indexes they want. Or maybe dropping all caches.
For any inferencer it would likely mean to remove all inferred triples and re-infer them.
For the memory store it could mean to flush to disk if a disk is specified. Or sort all the statement lists. Or maybe compact all lists to take up less memory. Or intern all strings.

Another issue is the refresh() would have to either not return anything. Or return an object that the user needs to cast.
Essentially I feel that ´refresh()` is too fuzzy and it tries to solve a problem we don't have. The problem we have is the need for a user to interact with some specific feature of the underlying sail.
And for the immutable config. I meant an immutable config object. Calling  connection.getConfig().set(ShaclSettings.VALIDATE, false) would return a clone.

The problem with refresh() is that it doesn't tell the underlying store anything about what it should be refreshing.
* For the ElasticsearchStore it could signal an index level refresh. Maybe rebuild the statistics.

* For the NativeStore it could signal a flush and maybe a recalculation of statistics (dunno if we have statistics for the NativeStore). It would also signal a compaction of the hashmap. Realignment of the values files. Or maybe re-indexing data into new trees if the user changes which indexes they want. Or maybe dropping all caches.

* For any inferencer it would likely mean to remove all inferred triples and re-infer them.

* For the memory store it could mean to flush to disk if a disk is specified. Or sort all the statement lists. Or maybe compact all lists to take up less memory. Or intern all strings.

That was actually precisely the point.

Another issue is the refresh() would have to either not return anything. Or return an object that the user needs to cast.

That's not a problem. At the sail level, we can return a generic Report interface (or even a List if we want to suport multiple sails in the stack each giving their own report). At the Repository API level, we can have code that inspects the lists of reports and fishes out the one that is fit for purpose (so e.g. a ValidatingRepository could fish out a ValidationReport).
But I appreciate now that all of this may be a bit YAGNI. I'm just trying to make sure we can keep the core SAIL and Repository interfaces as minimal and generic as possible.

Essentially I feel that ´refresh()` is too fuzzy and it tries to solve a problem we don't have. The problem we have is the need for a user to interact with some specific feature of the underlying sail.

Users don't interact with specific features of an underlying Sail - at least not directly. A Sail is hidden. What a user wants is to make sure is that they get a report that tells them if anything in their repository violates the shape constraints. It's the Repository API's job to translate that high-level request into specific operations on the SAIL stack.
All that being said, I think we can't avoid the fact that for validation, we will need some sort of first-class citizenship in both the Repository and SAIL interfaces. The generic refresh idea might be nice but it's a bit overengineered for our immediate needs, I've come to agree.
I will want it a bit generalized/abstracted from the specifics of the ShaclSail implementation. Can we call it validate() instead - the semantics being "validate the current state of the connection and produce a report" or something along those lines. It will give us flexibility to in future maybe consider more sophisticated behavior (e.g. when used in a transaction it only validates the added data in the transaction, but when used outside it, it re-runs validation on the entire store - not saying that that is a good idea right now, just saying that maybe we might want something like that).
I suggest that we move the ValidationReport class out of the SHACL Sail package, and move it to the repository-api module so it is reusable both locally and remote. We can then add a method RepositoryConnection.validate() which returns an (optional) ValidationReport. At the SAIL level, we can add a SailConneciton.validate() which can return an iteration of statements. Default implementation will be a no-op, and we'll need to ensure the wrapper implementations all delegate it.
Actually, perhaps slightly better would be to introduce new interfaces for the new method in both APIs: ValidatingRepositoryConnection and ValidatingSailConnection. We can then make sure that the relevant implementations implement those interfaces. Similar to how we have, for example, NotifyingRepositoryConnection and NotifyingSailConnection.

And for the immutable config. I meant an immutable config object. Calling connection.getConfig().set(ShaclSettings.VALIDATE, false) would return a clone.

I think we agree :) I guess I was just confused by using the term "immutable" in this context (as far as I can tell no object is actually made immutable, we are simply not giving direct access to modify the internal copy).

I've split this out into two cards: this current card can focus on the config stuff, while a new issue (#2325) can focus on the new validate() API method.

Relevant part of the REST API documentation: https://rdf4j.org/documentation/reference/rest-api/#repository-configuration

closing since this problem is solved with transaction settings


upgrade-threshold-crypto-libs

Update the threshold_crypto related libs.
See also
https://github.com/fedimint/hbbft/pull/7
https://github.com/fedimint/threshold_crypto/pull/3

Codecov Report

Merging #499 (5b3c33c) into master (dabd9e4) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #499      +/-   ##
==========================================
- Coverage   72.80%   72.78%   -0.02%
==========================================
  Files          83       83
  Lines       10821    10832      +11
==========================================
+ Hits         7878     7884       +6
- Misses       2943     2948       +5

Impacted Files
Coverage Δ

fedimint-api/src/rand.rs
59.66% <100.00%> (+2.52%)
:arrow_up:

fedimint-core/src/epoch.rs
99.59% <100.00%> (ø)

fedimint/src/config.rs
66.85% <100.00%> (+0.18%)
:arrow_up:

fedimint/src/lib.rs
82.93% <100.00%> (ø)

modules/fedimint-ln/src/config.rs
87.75% <100.00%> (ø)

client/client-lib/src/api.rs
81.38% <0.00%> (-1.10%)
:arrow_down:

crypto/tbs/src/poly.rs
21.23% <0.00%> (-0.66%)
:arrow_down:

fedimint/src/net/peers.rs
92.68% <0.00%> (-0.27%)
:arrow_down:

fedimint/src/rng.rs
14.47% <0.00%> (+9.21%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Closing because we already did this, but using my threshold_crypto branch.


Custom theme not working?

Trying to make a custom theme, but it's not working, but without a clear error message, so I'm not sure what I'm doing...
import { theme } from 'mdx-deck/themes'

export default {
  ...theme,
  font: 'Roboto, sans-serif',
  colors: {
    text: 'black',
    background: 'white',
    link: 'black',
  },
  css: {
    fontSize: 'calc(16px + 6 * ((100vw - 320px) / 680))',
    textAlign: 'left'
  }
}

# Yo!

This is my first mdx-deck slide.

---

# Hey!

This is my second.

---

and I get a blank page with some vague (at least to me) error messages in the console:

sooooooo...don't know what I'm doing wrong here, so any help would be appreciated :-)

nvrmnd, I did not realize that it was supposed to be a separate file.

MDX treats default exports as layouts. To set a theme with mdx-deck, you'll want to export a theme – it should work if you have it inline, but ymmv


cannot configure a cluster with multiples "latest" containers

I tried to follow the documentation [1] to create a multiple containers clusters but did not succeed.
The setup loooks like it works but i don't see other nodes when polling _membership.
I could not understand how you did it in the 2.0-dev image. It looks like it's done in the /usr/src/couchdb/dev/run pythn script but i could not figure out where the nodes and the cluster is declared.
Thanks if you can give me a bit of light :)
[1] : http://docs.couchdb.org/en/2.0.0/cluster/setup.html#the-cluster-setup-wizard

Hey,
you are right, the 2.0-dev image uses the python script that the CouchDB developers created for testing purposes. Unfortunately, I'm completely unfamiliar with 2.0's cluster config. Maybe @robertkowalski or anyone from CouchDB could help out here?

I'm closing this. The 2.0-dev will automatically configure a 3 node cluster. You can override that by doing something like this:
docker run -p 5984:5984 klaemo/couchdb:2.0-dev --no-join
But you will need to expose each nodes ports and then configure them. It would be better to use the
2.0 image as its a single node.


docs ($compile): add error documentation for noslot error in $compile

there was no error page for the $compile:noslot error
this resolves #15790
What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
What is the current behavior? (You can also link to an open issue here)
What is the new behavior (if this is a feature change)?
Does this PR introduce a breaking change?
Please check if the PR fulfills these requirements

[ ] The commit message follows our guidelines: https://github.com/angular/angular.js/blob/master/CONTRIBUTING.md#commit-message-format
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

Other information:

It says that I need CLA but I've already filled one out.

I checked and it was because my git signature was for my work email. I've added my work email as an alternate email to my google account, how do I get the CLA bot to rerun the check? Thanks.

Is there anything else I need to do/update etc to get this merged? :)

I will review the PR this week

Thanks @JoshuaJWilborn - for future PRs, additional commits with adjustments don't have to have a "perfect commit message" because we will squash them into one in the end. It's also a good idea to create a new branch for each PR.


Added ReactCasts

ReactCasts is a free series on Youtube, with weekly short videos explaining React's APIs and concepts.

Thanks!


haplotypecallerspark OOM issues on rna-seq variant pipeline

Hello, I running a re-analysis of about 400 rna-seq samples and added variant calling to the config.  I'm currently using the stable version of bcbio. I'm running an ubuntu instance on google, 96 cores, 360 GB memory, all the transcript tools run fine the variant calling is throwing alot of errors, im not familiar enough with the new spark gatk integration to know where to start on this one, any help would be great.
Config is:
Template for human RNA-seq using Illumina prepared samples

details:

analysis: RNA-seq
metadata:
batch:
phenotype:
genome_build: hg38
algorithm:
aligner: hisat2
adapters: [truseq, polya]
tools_on: [qualimap_full, gemini_allvariants]
tools_off: [gemini]
vcfanno: [gemini]
variantcaller: [gatk-haplotype]
effects: snpeff
expression_caller: [stringtie, sailfish, salmon]
transcript_assembler: [stringtie]
upload:
dir: ../final

system config:
resources:
default options, used if other items below are not present
avoids needing to configure/adjust for every program
default:
memory: 2G
cores: 16
jvm_opts: ["-Xms750m", "-Xmx2500m"]
gatk:
jvm_opts: ["-Xms500m", "-Xmx2500m"]
snpeff:
jvm_opts: ["-Xms750m", "-Xmx2500m"]
qualimap:
memory: 4g
express:
memory: 4g
dexseq:
memory: 4g
macs2:
memory: 4g
seqcluster:
memory: 4g
i lowered all the memory limits to what is shown but am still getting the same issues.  my output is attached.
nohup.out.txt

Hi Patrick,
Sorry about that, it looks like not enough memory is getting allocated to GATK, so it is failing.  If you bump up the memory allocated to GATK it should fix the issue.

Thanks, in the system config, are the gatk memory settings per core? or total?
gatk:
jvm_opts: ["-Xms500m", "-Xmx2500m"]

Patrick;
Sorry about the continued problems. If you further increase memory allocated to GATK, by trying, say 8g, bcbio should automatically scale down the cores used. It would be useful to see if that helps avoid the issue. From the logs it's a bit hard to tell if Spark is hitting memory or disk access issues since it's complaining about both and I'm not sure what's a cause or a symptom.
Apologies, this is so problematic. Both the RNA-seq and HaplotypeCallerSpark bits are new functionality we're working on and will be attempting to scale better in the future. Thanks for the help debugging this.

No worries, just happy for the help.  I upped the mem to 8g and am getting a similar crash.
nohup4.out.txt

Patrick;
Thanks for continuing to help debug. From the log it appears as if you're making more progress, so moving in the right direction. Your machines appear to be pretty high memory since giving 8gb/core doesn't scale down core usage at all. As before, it's hard to tell if it is memory or disk from 16 simultaneous cores reading/writing.
What type of resources do your machines have? Can you scale up memory more to see if you can reduce total core usage? Maybe try bumping to 16Gb/core as a test. Additionally looking at top as
before would help differentiate if you're hitting memory problems or just stressing out the disk too much.
My way to debug this would be to adjust until we can make it work then determine how less stringent requirements can be made to work on your system. Thanks again for the patience and work debugging.

Hi Brad,
The instance has 96 cores, 360 GB, ill scale up to 16g/core and see how it goes, thanks so much for helping with this.

Hi Brad, i've updated from the stable to development, and tried both 16g and 32g for gatk on both stable and development, still having the same issues.  Most recent run was with system config:
default:
memory: 4G
cores: 16
jvm_opts: ["-Xms500m", "-Xmx4
bcbio-nextgen-debug.log.txt
g"]
gatk:
jvm_opts: ["-Xms500m", "-Xmx16g"]
snpeff:
jvm_opts: ["-Xms750m", "-Xmx4g"]
qualimap:
memory: 4g
express:
memory: 4g
dexseq:
memory: 4g
macs2:
memory: 4g
seqcluster:
memory: 4g
I have attached my debug log.  During this last run, total memory usage never went over 200g out of 360g available.

Patrick;
Sorry this is continuing to fail. I wish I had a good answer for you but my only suggestion is to turn down cores and turn up memory until you can eventually make it work and then we can try to walk back from there to figure something that is reasonable. This is way more memory than it should need but what I don't know is if it's causing some kind of filesystem issue by running with 16 cores. Did the 32g run scale back from 16 cores and still fail in the same way?
If we can't make it work with less cores my next suggestion would be to try and produce a small example that fails in this way and we could report to the GATK team to see if they have any insight.
Sorry I know this is unsatisfying help but I'm not positive what to suggest with this failure. Hope this helps some.

As a quick trip report - I was having similar issues, that were a bit sporadic. My problematic sample got through with a config like this:
resources:
  default:
    memory: 3G
    cores: 32
    jvm_opts: ["-Xms800m", "-Xmx3600m"]
  gatk:
    memory: 3G
    cores: 32
    jvm_opts: ["-Xms800m", "-Xmx3600m"]

Which led to memory requests like this:
[2018-05-10T16:10Z] ccgihpc.hpc.bcgsc.ca: Resource requests: gatk, vardict; memory: 3.00, 3.00; cores: 32, 32
[2018-05-10T16:10Z] ccgihpc.hpc.bcgsc.ca: Configuring 3 jobs to run, using 32 cores each with 96.1g of memory reserved for each job

And a GATK call like this:
[2018-05-10T16:12Z] eccfx01n3: export SPARK_USER=rdocking && unset JAVA_HOME &&
export PATH=/projects/rdocking_prj/software/bcbio-nextgen/data/anaconda/bin:$PATH &&
gatk-launch --java-options '-Xms800m -Xmx94349m -Djava.io.tmpdir=/projects/karsanscratch/rdocking/KARSANBIO-1390_rna_seq_runs/molm13_replicates/work/bcbiotx/tmpZHeHfM'
HaplotypeCallerSpark

... etc ...

Watching the job with top, it looked like it was using close to the 90G of RAM throughout. This one eventually finished, but then I ran into #2375
I'm going to try a few more things on my end overnight to see if I can get a cleaner run.

Rod -- thanks for the report and testing different configuration parameters. I'm sorry I can't help much from afar without a reproducible case but am open to tweaking what bcbio does to provide a more consistent, working, experience here. Thanks to everyone for working through these pain points with RNA-seq calling.

Thanks for the help, so far I am making progress, I had a suspicion that something was buggy on the vm i was using so i moved my drive to a fresh vm, a clean bcbio development install, with 64 cores and 256g memory. I altered to system config as Rod suggested, now using:
resources:
default:
memory: 3G
cores: 32
jvm_opts: ["-Xms800m", "-Xmx3600m"]
gatk:
memory: 3G
cores: 32
jvm_opts: ["-Xms800m", "-Xmx3600m"]
When I restarted bcbio with the command:
bcbio_nextgen.py -n 32 ../config/Analysis.yaml &
I immediately began seeing the same error messages, Not enough space to cache rdd_... in memory!
So I re-started bcbio with the command:
bcbio_nextgen.py ../config/Analysis.yaml &
without core specification everything is currently running smoothly, and i have for the first time an actual, viewable, vcf in bcbiotx
patrickreed@bcbio-metapsych:/mnt/disks/metapsych/Analysis/work$ sudo ls bcbiotx/*
bcbiotx/tmpHBSW42:
bcbiotx/tmpmPfTvb:
14501_1_P5_S8_R1_001-gatk-haplotype.vcf.gz  root
bcbiotx/tmpRcBlGI:
The problem now is resource utilization, its really low.

One thing i noticed in the log file is that the inital java gatk Haplotypecaller call shows up like this:
[2018-05-11T15:34Z] GATK: HaplotypeCaller
[2018-05-11T15:34Z] Using GATK jar /usr/local/share/bcbio/anaconda/share/gatk4-<IP_ADDRESS>-0/gatk-package-<IP_ADDRESS>-local.jar
[2018-05-11T15:34Z] Running:
[2018-05-11T15:34Z]     java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xms727m -Xmx3272m -XX:+UseSerialGC -Djava.io.tmpdir=/mnt/disks/metapsych/Analysis/work/bcbiotx/tmpmPfTvb -jar /usr/local/share/bcbio/anaconda/share/gatk4-<IP_ADDRESS>-0/gatk-package-<IP_ADDRESS>-local.jar HaplotypeCaller -R /usr/local/share/bcbio/genomes/Hsapiens/hg38/seq/hg38.fa --annotation MappingQualityRankSumTest --annotation MappingQualityZero --annotation QualByDepth --annotation ReadPosRankSumTest --annotation RMSMappingQuality --annotation BaseQualityRankSumTest --annotation FisherStrand --annotation MappingQuality --annotation DepthPerAlleleBySample --annotation Coverage -I /mnt/disks/metapsych/Analysis/work/align/14501_1_P5_S8_R1_001/14501_1_P5_S8_R1_001-sort-dedup.splitN.bam -L /mnt/disks/metapsych/Analysis/work/bedprep/ref-transcripts-rnaseq_clean.bed --interval-set-rule INTERSECTION --annotation ClippingRankSumTest --annotation DepthPerSampleHC --emit-ref-confidence GVCF -GQB 10 -GQB 20 -GQB 30 -GQB 40 -GQB 60 -GQB 80 --output /mnt/disks/metapsych/Analysis/work/bcbiotx/tmpmPfTvb/14501_1_P5_S8_R1_001-gatk-haplotype.vcf.gz
specifically the -Xmx is -Xmx3272m
previously, when i called bcbio with a -n of 32 or 64 the initial call would be something like:
[2018-05-10T21:09Z] GATK: HaplotypeCallerSpark
[2018-05-10T21:09Z] Using GATK jar /usr/local/share/bcbio/anaconda/share/gatk4-<IP_ADDRESS>-0/gatk-package-<IP_ADDRESS>-local.jar
[2018-05-10T21:09Z] Running:
[2018-05-10T21:09Z]     java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xms500m -Xmx45864m -Djava.io.tmpdir=/mnt/disks/metapsych/Analysis/work/bcbiotx/tmpAGP6GY -jar /usr/local/share/bcbio/anaconda/share/gatk4-<IP_ADDRESS>-0/gatk-package-<IP_ADDRESS>-local.jar HaplotypeCallerSpark --reference /usr/local/share/bcbio/genomes/Hsapiens/hg38/ucsc/hg38.2bit --annotation MappingQualityRankSumTest --annotation MappingQualityZero --annotation QualByDepth --annotation ReadPosRankSumTest --annotation RMSMappingQuality --annotation BaseQualityRankSumTest --annotation FisherStrand --annotation MappingQuality --annotation DepthPerAlleleBySample --annotation Coverage -I /mnt/disks/metapsych/Analysis/work/align/20026_1_P5_S9_R1_001/20026_1_P5_S9_R1_001-sort-dedup.splitN.bam -L /mnt/disks/metapsych/Analysis/work/bedprep/ref-transcripts-rnaseq_clean.bed --interval-set-rule INTERSECTION --spark-master local[16] --conf spark.local.dir=/mnt/disks/metapsych/Analysis/work/bcbiotx/tmpAGP6GY --annotation ClippingRankSumTest --annotation DepthPerSampleHC --emit-ref-confidence GVCF -GQB 10 -GQB 20 -GQB 30 -GQB 40 -GQB 60 -GQB 80 --output /mnt/disks/metapsych/Analysis/work/bcbiotx/tmpAGP6GY/20026_1_P5_S9_R1_001-gatk-haplotype.vcf
with -Xmx of -Xmx45864m
should this be calling java / gatk with an Xmx of 46g when my system config was setting the -Xmx for both java and gatk to 3g?

Rod and Patrick;
Thanks again for all of this. I've been working on this and pushed a version of GATK HaplotypeCalling that splits the jobs by chromosomes. It won't be any faster right now since it does them serially, but reduces the amount of work for each step. I was hoping this would allow it to power through each component before running into the issues you're hitting. If you could update to the latest development:
bcbio_nextgen.py upgrade -u development

and re-try it would be helpful to know if this improves things.
Patrick, running without -n uses a single core and won't parallelize variant calling. So it'll work but take unreasonably long. Regarding heap size, -Xmx is setting that in the bcbio runs, so the default without setting is 32g but it's not a hard limit. I know there are some weirdnesses that can happen around the 32Gb size so it could be useful experiment to try setting below (say, 8 cores with 3Gb each) to see if that has any impact.
Hope the by-chromosome approach helps power through this issue. Thanks again for all the help debugging this.

Update:
Splitting is working great, and i'm no longer getting any OOM errors even with full core usage.  A new error has occurred from the FixVcfHeader though, please advise if you would like me to open a new issue.  I attached the debug log.
bcbio-nextgen-debug.txt

@PatrickJReed  - that's the same one we're seeing over at #2375

@rdocking Glad this isn't a new issue, thanks.  Original OOM has been solved so I'm going to close this issue.


UDP typo

@vitawrap : Thanks for your contribution! The author(s) have been notified to review your proposed change.


Automated cherry pick of #3606: influxdb: 记录结果中的tags

Cherry pick of #3606 on release/2.10.0.
#3606: influxdb: 记录结果中的tags

Codecov Report

:exclamation: No coverage uploaded for pull request base (release/2.10.0@4451c60). Click here to learn what that means.
The diff coverage is 0%.

@@               Coverage Diff                @@
##             release/2.10.0   #3610   +/-   ##
================================================
  Coverage                  ?   7.47%
================================================
  Files                     ?     416
  Lines                     ?   66436
  Branches                  ?       0
================================================
  Hits                      ?    4966
  Misses                    ?   61042
  Partials                  ?     428

Flag
Coverage Δ

#aFlag
7.47% <0%> (?)

Impacted Files
Coverage Δ

pkg/compute/models/loadbalancerbackends.go
2.46% <0%> (ø)

pkg/compute/models/loadbalancers.go
1.53% <0%> (ø)

pkg/compute/models/loadbalancer_backendstatus.go
0% <0%> (ø)

pkg/compute/models/loadbalanceragents.go
2.33% <0%> (ø)

pkg/util/influxdb/influxdb.go
0% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4451c60...1f069fd. Read the comment docs.

/lgtm

/approve

/approve


Add GridRooms as a generalization of FourRooms

Add GridRooms environment, which is a generalization of FourRooms, in that, we can arrange rectangular rooms in an arbitrary rectangular grid-like fashion. Also addresses #70
Deprecate FourRooms.

Codecov Report

Merging #92 (1fc65e1) into master (df420a1) will increase coverage by 0.74%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #92      +/-   ##
==========================================
+ Coverage   77.32%   78.07%   +0.74%
==========================================
  Files          15       16       +1
  Lines         657      684      +27
==========================================
+ Hits          508      534      +26
- Misses        149      150       +1

Impacted Files
Coverage Δ

src/envs/gridrooms.jl
100.00% <100.00%> (ø)

src/envs/sequentialrooms.jl
99.09% <100.00%> (-0.91%)
:arrow_down:

src/envs/utils.jl
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update df420a1...1fc65e1. Read the comment docs.


Comparing BCEWithLogitsLoss with Focal Loss

Thanks for your wonderful work! Have you tried to use Focal Loss (i.e. sigmoid focal loss in Deformable DETR) as the classification loss instead of nn.BCEWithLogitsLoss? I am looking forward to see which one is better :)

@ifzhang We have tried Focal Loss. There isn't any different.

Thank you very much!

@ifzhang We have tried Focal Loss. There isn't any different.

cls loss and obj loss in code would divide num_fg, if we replace bce loss with focal loss, maybe we should divide batch_size instead of num_fg? however, this would lead to a large loss compared with other loss(such as iou loss 4.x, obj focal loss 22.x)

@ifzhang我们尝试了 Focal Loss。没有什么不同。

请问你都是过哪几个损失函数

@ifzhang我们试点了Focal Loss。没有什么不同。

请问你都试过哪几个损失函数


Fix output_folder for co2 evaluation

lol surprised we had this bug; it literally errors out for me 😁

Previously, I tried to remove the output_folder parameter and pass it through kwargs, since it’s only needed for retrieval. However, it seems that during refactoring, I accidentally removed it.

No worries! Thanks for merging 🚀


feat(#226): Print 'PHI' Expressions in 'spring' Integration Test

In this PR I fixed all the problems related to printing 'PHI' expressions for 'spring' integration test.
To do so, I added a few crutches and updated eo-maven-plugin up to 0.38.1 version.
Risk: opeo-maven-plugin changes XMIRs before printing PHI expressions. It might cause some incosistencies between XMIR transformations. It's better to remove all the crutches from opeo-maven-plugin and solve the original problems. To do so, I added a few new puzzles.
Related to #226.
History:

feat(#226): successfully print PHI representation of FactorialApplication
feat(#226): add javadoc for FormattingDecompiler
feat(#226): handle the case when 'modified' folder is absent
feat(#226): adjust jacoco limits

PR-Codex overview
The focus of this PR is to introduce a new FormattingDecompiler class that fixes issues with XMIR generated by two plugins.
Detailed summary

Adjusted limits in pom.xml for test coverage
Added a new plugin in spring/pom.xml
Updated XmirEntry.java with new method and return types
Modified DecompileMojo.java to use FormattingDecompiler
Added FormattingDecompiler class to fix XMIR formatting issues

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

@rultor merge

@rultor merge

@volodya-lombrozo OK, I'll try to merge now. You can check the progress of the merge here

@rultor merge

@volodya-lombrozo Done! FYI, the full log is here (took me 7min)


Can't run 8ball in #bot-commands

[5:45 PM] rob-3: !8ball
[5:45 PM] Lion: Please use this command in a Public channel.
Here are 3 of the 47 supported channel(s):
general,
memes,
programming_help.

I think 8ball should work in #bot-commands. Once I finish rewriting it, I will fix this bug myself if it still exists.

This is intended
The channel permission is public
Channel permission is checked here at runtime
Channel types are defined by name in Constants.ts
However, I would be down for changing the way this is defined as it could be useful to have a channel with multiple types
(ie #bot_commands could be public and bot)

for some historic context- we defined bot commands as not a public chan because we didn't want to start conversation in that channel/wanted to keep the messages sent in that chan strictly for class registration/garage status. we believed commands that lean more social should stick to the other public chans

for some historic context- we defined bot commands as not a public chan because we didn't want to start conversation in that channel/wanted to keep the messages sent in that chan strictly for class registration/garage status. we believed commands that lean more social should stick to the other public chans

Agree with this

However, I would be down for changing the way this is defined as it could be useful to have a channel with multiple types (ie #bot_commands could be public and bot)

Or maybe less intrusively, the ability to union types for a certain plugin
Just something like an array
public permission: ChannelType = [ChannelType.Public, ChannelType.Bot;]


Error 500 when trying to webview a specific repository

Gitea version (or commit ref):
Git version: 1.1.4
Operating system: Debian 8
Database:

[x] MySQL

Can you reproduce the bug at https://try.gitea.io:

[x] No

Log gist: in log/gitea.log:

[...routers/repo/view.go:52 renderDirectory()] [E] GetCommitsInfo: GetCommitByPath (/Iconr): Length must be 40:

Description
I manage several repositories via Gitea, and they a working like a charm. But one specific throws an 500 server error when I try to open it in the browser. In the logs I can stat this message:
[...routers/repo/view.go:52 renderDirectory()] [E] GetCommitsInfo: GetCommitByPath (/Iconr): Length must be 40:

But I can clone and push without any problem.
Where to start debugging?
Cheers,
Marianne
Screenshots

It seems there was already an issue to describe that.

I think this was already fixed in 1.2

@sysadmama could you confirm that?

@lunny + @lafriks Nope; started 1.2.0+rc2 yesterday and still error 500 on this specific repository. gitea.log.
[...routers/repo/view.go:53 renderDirectory()] [E] GetCommitsInfo: GetCommitByPath (/Iconr): Length must be 40:

Cheers,
Marianne

@sysadmama is that repository public so we could reproduce that issue?

@lafriks No it's reachable via VPN only :( Are there any further logfiles I could consult? Even strace doesn't enlighten me.

@lafriks @sysadmama I remember I have tried to resolve this issue but I failed.

Here is an example for an 500 with Version 1.2.0+rc3-6-g03ff7687:
https://try.gitea.io/AndyTest/TestDev.git
But with try.gitea.io it hangs.
The strings (of the filenames) are not escaped. I could imagine, that this would be a security issue, too.

Tried it with my repository: I accidentally had an empty File called Icon?. I removed it completely, and after a push I can access my repo via web interface. So I also would assume it's the (wrong or non existent) escaping.

@sysadmama The filename contains a special character gitea didn't parse correctly.

@Andlix It seems that we fail with filenames containing newlines (i.e. \n). I believe that none of the other filenames in your test repo cause problems.

same issue: link

same issue with file named ":w"

This is both a duplicate and fixed likely by one of @mrsdizzie escaping patches.


I need api-ul api- For face. FaceTime

I need an api -ul api for FaceTime live broadcasting ipa-ca wak-code .,/code/htlm.’__c broadcasting  ! ,ecc’em/.v vi
Feature description

Problem and motivation

Is this something you're interested in working on

I think we can close this, or are we now developing a FaceTime API? 😂
(cc @daniel-eder)

@Ein-Tim thanks .. not sure where that came from!


Class extends error has wrong result with instanceof (in ES5 target)

Bug Report

🔎 Search Terms

🕗 Version & Regression Information

This is a crash
This changed between versions ______ and _______
This is the behavior in every version I tried, and I reviewed the FAQ for entries about error inheritance.
I was unable to test this on prior versions because _______

⏯ Playground Link
https://www.typescriptlang.org/play?target=1#code/MYGwhgzhAEAiCmAnAlgN3gE2vAHgF3gDsMYBRRRAe0WgG8BfAKGEsIj2gyTU2gF5oheAHc43dBgAUASkbNWESiHgA6EJQDmkgOSkAygFZtslmyWr1WrignRkbPGELB4lAGbRyVRCYXm1mpLWPFj27E4u7mI2mNJAA

Playground link with relevant code
💻 Code
class Derived extends Error {}
const derived = new Derived()

console.log('ES5')
console.log(derived instanceof Error)
console.log(derived instanceof Derived)

🙁 Actual behavior
ES5
true
false

🙂 Expected behavior
ES5
true
true

In es6 runtime it works fine.

and I reviewed the FAQ for entries about error inheritance.

It's right in the FAQ, though.
https://github.com/Microsoft/TypeScript/wiki/FAQ#why-doesnt-extending-built-ins-like-error-array-and-map-work

Thanks for answering 😁


graphcool setup

I was hoping you could share the data schema in SDL syntax, or offer another way to utilize the already integrated Graphcool schema

hi, @Dialectician , this is schema that I made and that works, the problem that I found with graphcool is that schemas is not shareable, this schema wont work if you copy-paste it to your graphcool cli
# projectId: cj6alpbu60z8s0196ryeqngxs
# version: 8

type File implements Node {
  contentType: String!
  createdAt: DateTime!
  id: ID! @isUnique
  name: String!
  secret: String! @isUnique
  size: Int!
  updatedAt: DateTime!
  url: String! @isUnique
}

type Post implements Node {
  createdAt: DateTime!
  id: ID! @isUnique
  title: String!
  updatedAt: DateTime!
  url: String!
  votes: Int @defaultValue(value: 0)
}

type User implements Node {
  createdAt: DateTime!
  email: String @isUnique
  firstName: String!
  id: ID! @isUnique
  lastName: String!
  password: String
  updatedAt: DateTime!
}

The resolve may be found under https://www.graph.cool/docs/faq/graphql-how-to-download-graphql-sdl-schema-maiv5eekan/
Download the schema from the server
graphql get-schema
This downloads a GraphQL schema in SDL syntax

https://github.com/este/este/blob/master/data/project.graphcool

Hi @Dialectician , I updated the schema. You can find the updated one on libraries/graphql_schemas/schema.graphql. Is it okay for you?

And also @Dialectician @BjornMelgaard
I also added Graphcool Schema -> /libraries/graphql_schemas/Graphcool-simplified.schema


Create a POC for integrating a Colab classification model with weights and biases

Link to our W&B team -> https://wandb.ai/petal
I'm going to put our datasets for the text classification up in Github. Standby.

@elkong Can you sync up with @Simarkohli24 and see if you can publish your MATCH experiments to our W&B team?


nixos/adguardhome: Add settings option

Motivation for this change
It was not possible to configure AdGuard declaratively before.
This PR adds services.adguard.settings to generate the AdGuardHome.yaml. It is merged with yaml-merge on start.
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[X] Fits CONTRIBUTING.md.

These changes are really reasonable.
Thanks for the input, I tried checking a few options for links and the checked one didn't contain any links.

Have you tested this? If so, I'm happy to merge once @lunik1 reviews. Please ping back in a few days if we don't hear anything.

I already migrated my config to it, using:
  disabledModules = [ "services/networking/adguardhome.nix" ];
  imports = [ ../../nixpkgs/nixos/modules/services/networking/adguardhome.nix ];

Redeploying didn't result in any new closures, so the result should be identical.

Perfect! Thanks.

Sorry for the delay, I really like this PR too. Good work! My one suggestion is that I think it now makes sense to make the config location configurable à la #131356, could you add this option?

@CRTified nice addition, a lot more refined than my initial PR.

My one suggestion is that I think it now makes sense to make the config location configurable

I think this PR strikes a nice balance because adguardhome still manages its own configuration. This PR grafts configuration on top of what adguardhome manages, it doesn't attempt to provide any user management of the configuration in a sense that conflicts with the app. Personally, I think this PR is perfect as it is...

My one suggestion is that I think it now makes sense to make the config location configurable à la #131356, could you add this option?

@lunik1 @rhoriguchi Do you have a specific use case in mind for making the configuration path configurable? Maybe this can be integrated, but the settings-merging step makes symlinks and hardlinks impossible right now (due to this line).
It would be possible to accommodate for a dynamic path, but if there is no need for this, I'd prefer to keep it simple.
@aanderse Would it make sense to add a test for the service? I don't see a way to test the merging-step, but at least the initial start would be covered.

My one suggestion is that I think it now makes sense to make the config location configurable à la #131356, could you add this option?

@lunik1 @rhoriguchi Do you have a specific use case in mind for making the configuration path configurable? Maybe this can be integrated, but the settings-merging step makes symlinks and hardlinks impossible right now (due to this line). It would be possible to accommodate for a dynamic path, but if there is no need for this, I'd prefer to keep it simple.
@aanderse Would it make sense to add a test for the service? I don't see a way to test the merging-step, but at least the initial start would be covered.

The proposal your are making works fine for me, I'm not a big fan of the merging of the config, would be nice if that could be controlled by a option. Since i personally want to have an idempotent configuration.
I currently solve it like this which is more or less how you do it.

I'm not a big fan of the merging of the config, would be nice if that could be controlled by a option.

That's easy enough, by conditionally always cp-ing the config over. That would still keep the promise of checking the configuration while building due to the check phase for the config file. I'll edit the PR in a few minutes, just a bit busy right now.

My one suggestion is that I think it now makes sense to make the config location configurable à la #131356, could you add this option?

@lunik1 @rhoriguchi Do you have a specific use case in mind for making the configuration path configurable? Maybe this can be integrated, but the settings-merging step makes symlinks and hardlinks impossible right now (due to this line). It would be possible to accommodate for a dynamic path, but if there is no need for this, I'd prefer to keep it simple.

Maybe I am missing something, but could the target of that mv not just be $SOME_USER_SPECIFIED_CONFIG_LOCATION?. The originial PR points to some use cases, but if this will overcomplicate the module I am happy to merge as-is.

If the target was a symlink/hardlink, that mv would replace the symlink with an actual file.

I've now added services.adguardhome.mutableSettings as an option. If it is true (default), we allow changes from the web interface to persist between restarts by choosing the yaml-merge path in preStart. Otherwise, we always override the configuration on service start.
@rhoriguchi Let me know if this is sufficient for you.
@lunik1 I think adding a configurable path would complicate things a lot, as we need to consider whether merging/overriding should happen at all if that configuration is set. I'm also unsure whether the minimal isolation provided by systemd would require some attention for arbitrary config paths.
But it appears that when having multiple --config arguments, only the last one is considered (at least with 1-2 quick'n'dirty tests in a nix-shell). I don't know whether this should be documented somewhere, but it might be sufficient for the "mutable config, not controlled by nix, and backups somewhere else"-usecase from the other PR, as it allows overriding the config path without problems (not even a warning).
Please let me know what you think.

I've now added services.adguardhome.mutableSettings as an option. If it is true (default), we allow changes from the web interface to persist between restarts by choosing the yaml-merge path in preStart. Otherwise, we always override the configuration on service start.
@rhoriguchi Let me know if this is sufficient for you.
@lunik1 I think adding a configurable path would complicate things a lot, as we need to consider whether merging/overriding should happen at all if that configuration is set. I'm also unsure whether the minimal isolation provided by systemd would require some attention for arbitrary config paths.
But it appears that when having multiple --config arguments, only the last one is considered (at least with 1-2 quick'n'dirty tests in a nix-shell). I don't know whether this should be documented somewhere, but it might be sufficient for the "mutable config, not controlled by nix, and backups somewhere else"-usecase from the other PR, as it allows overriding the config path without problems (not even a warning). The module already provides extraArgs, so the user should be able to do this easily.
Please let me know what you think.

Thank you for adding the option.
I don't really see a reason to add a path since the settings options just creats a file and if someone wants to use a config file, they can just read the yaml and input it to the settings option. Also I agree, being able to add a path might cause some strange behavior, if as example adguardhome is not able to read or write to the file.

@aanderse I think this can be merged now :)

Thanks for the ping back!


Fix too many points or lines crash

Simplified the point cloud builder further in the process - we almost always have all the fields ready so we can remove a lot of complexity the "nicer"/fine-grained interface brings with it.
Note that the second commit here (which removes chaining the iterators with fallback values) got frame time for max-points in nyud down to about 131ms from 145ms before that commit (in release).
Edit: Tried on main with a quick & dirty fix, gives me for the same scenario about 143ms. Meaning this PR is perf win :)
Fix #1779
Checklist

[x] I have read and agree to Contributor Guide and the Code of Conduct

    Finished dev [optimized + debuginfo] target(s) in 0.18s
     Running `target/debug/examples/depth_cloud`

thread 'main' panicked at 'attempt to subtract with overflow', /home/cmc/dev/rerun-io/rerun/crates/re_renderer/src/point_cloud_builder.rs:220:75
stack backtrace:
   0: rust_begin_unwind
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/panicking.rs:64:14
   2: core::panicking::panic
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/panicking.rs:111:5
   3: draw_backprojected_point_cloud<re_renderer::renderer::lines::LineDrawData, re_renderer::renderer::rectangles::RectangleDrawData>
   4: draw
             at ./crates/re_renderer/examples/depth_cloud.rs:345:13
   5: {closure#0}<depth_cloud::RenderDepthClouds>
             at ./crates/re_renderer/examples/framework.rs:262:40
   6: sticky_exit_callback<(), depth_cloud::framework::{impl#1}::run::{closure_env#0}<depth_cloud::RenderDepthClouds>>
   7: run_return<(), depth_cloud::framework::{impl#1}::run::{closure_env#0}<depth_cloud::RenderDepthClouds>>
             at /home/cmc/.cargo/registry/src/github.com-1ecc6299db9ec823/winit-0.28.1/src/platform_impl/linux/wayland/event_loop/mod.rs:548:21
   8: winit::platform_impl::platform::wayland::event_loop::EventLoop<T>::run
             at /home/cmc/.cargo/registry/src/github.com-1ecc6299db9ec823/winit-0.28.1/src/platform_impl/linux/wayland/event_loop/mod.rs:222:25
   9: winit::platform_impl::platform::EventLoop<T>::run
             at /home/cmc/.cargo/registry/src/github.com-1ecc6299db9ec823/winit-0.28.1/src/platform_impl/linux/mod.rs:792:56
  10: winit::event_loop::EventLoop<T>::run
             at /home/cmc/.cargo/registry/src/github.com-1ecc6299db9ec823/winit-0.28.1/src/event_loop.rs:305:9
  11: depth_cloud::framework::Application<E>::run
             at ./crates/re_renderer/examples/framework.rs:189:9
  12: {async_fn#0}<depth_cloud::RenderDepthClouds>
             at ./crates/re_renderer/examples/framework.rs:363:5
  13: block_on<depth_cloud::framework::run::{async_fn_env#0}<depth_cloud::RenderDepthClouds>>
             at /home/cmc/.cargo/registry/src/github.com-1ecc6299db9ec823/pollster-0.3.0/src/lib.rs:128:15
  14: start<depth_cloud::RenderDepthClouds>
             at ./crates/re_renderer/examples/framework.rs:383:9
  15: call_once<fn(), ()>
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/ops/function.rs:507:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

only tested in the viewer in the end and didn't notice that the default adding there was broken (the viewer doesn't seem to make use of this ever)


Role_grants

Removed role grants, updated MySQL AMI

Two internal tests succeeded, so I'll commit.


Move .xcworkspace / bridging header task warning to pod init error

Move .xcworkspace message from task warning to pod init error
Fixes #554

lgtm + comment

Build failed due to Gradle download. Merging anyway.


Adds installation of Xray tools

This PR adds a cmake command to install the built tools. This is needed to have prjxray set as a conda package.

@litghost Python tests are now green. With the utils addition to the package, some modules couldn't be found

Waiting for green CI, but mostly LGTM

@litghost CI green, is this ok to merge? Once merged I will get conda-packages to build also prjxray


refactor: let target be a trait

This way target can be updated, and works better in combination with
Reacton and Solara.

Current dependencies on/for this PR:

master

PR #5

PR #9

PR #11   👈

PR #6

PR #7

This comment was auto-generated by Graphite.


Throwing errors in preprocess

Hi there,
I'm trying to implement string coercion using z.preprocess, but there seems to be no way to indicate coercion errors:
export const Str = z.preprocess((value) => {
  switch (typeof value) {
    case 'bigint':
    case 'boolean':
    case 'number':
      return value.toString();

    case 'object':
      if (value == null) {
        return value;
      }
      if (value instanceof Date) {
        return value.toISOString();
      }
      throw new Error('Could not interpret value as string');

    case 'string':
      return value.trim();

    case 'undefined':
      return value;

    default:
      throw new Error('Could not interpret value as string');
  }
}, z.string());

Str.parse([]); // Error: Could not interpret value as string (rather than a ZodError)

I tried to create verbose workarounds but none ended up working. Thanks!
Kind regards,
Hampus Kraft.

The few times I've played around with preprocess, I actually use zod schemas with the "looser" version of what I expect. For instance, if you're using preprocess to get from a serialized query-parameter style string-heavy object to a parsed object, something like:
// Query parameter schema
const loose = z.object({
  aBoolean: z.string().transform(Boolean),
  aNumber: z.string().transform(Number),
});

// Domain schema
const query = z.object({
  aBoolean: z.boolean(),
  aNumber: z.number(),
});

const fromQueryParams = z.preprocess(
  loose.parse,
  query
);

I have to admit that I haven't used preprocess in a serious manner yet, so there might be others who have come up with something more generic like what you we originally attempting.

Perhaps you should be able to add issues inside preprocess similar to superRefine. I'll look into this. It would save you from having to call coerceString twice.

+1 for adding issues in preprocess. @colinhacks Would you be willing to accept a PR on this?
IMO this would make it much more straightforward to build a reusable set of "lenient preprocessors" on top of zod (#804 #1019), while keeping good error messages.
I currently work around it with something like this:
function maybeParseNumber(v: unknown): unknown | number {
  if (typeof v !== "string") {
    return v;
  }
  if (!v) {
    return v; // Otherwise "" is parsed as 0
  }
  const ret = Number(v);
  return isNaN(ret) ? v : ret;
}

// Preprocessors for stringly-typed data.
const zs = {
  number: (augment?: (s: ZodNumber) => ZodNumber) => {
    let schema = z.number({ invalid_type_error: "Not a number", required_error: "Required" });
    schema = augment ? augment(schema) : schema;

    return z.preprocess((v) => maybeParseNumber(v), schema);
  },
}

Notice that:

The invalid_type_error needs to be overriden on the inner z.number() schema, otherwise you get the generic Expected number, received string error, which is not ideal for stringly-typed data like query parameters and environment variables.
If required_error is not defined, then you get invalid_type_error instead of the default required_error.

Consequently, the only way to allow adding constraints on the inner z.number() schema (like .min()) is the augment callback, e.g.:
zs.number((s) => s.min(5).max(10))

Unfortunately, this means that I can't simply stick a pre-existing number schema into a lenient schema. For instance, I'd like to be able to do:
const age = z
  .number({
    required_error: "Age is required",
    invalid_type_error: "Age must be a number",
  })
  .lte(5, { message: "this👏is👏too👏big" });

const queryParams = z.object({
  age: zs.number(age)
})

Or even just take a full "strict schema" and automatically loosen it recursively (similar to what znv does), e.g.:
// age is defined as above
const CreatePerson = z.object({
  age: age
});

const CreatePersonLenient = lenient(CreatePerson);
CreatePersonLenient.parse(...);

Perhaps you should be able to add issues inside preprocess similar to superRefine. I'll look into this. It would save you from having to call coerceString twice.

Couldn't the same be said for a transform you want to validate?

Took a stab at the PR for transform(). Wouldn't be hard to do for preprocess?

Well @selimb this would not break from the idea that a transform would ONLY operate on valid data that has been "refined" before it.
The feature here is that as you transform it into new data you can catch new errors that PREVENT transform.
Put another way - the data might be INVALID in that it CANNOT be transformed correctly.
Let's consider this note from the README on .transform():

⚠ Transform functions must not throw. Make sure to use refinements before the transform to make sure the input can be parsed by the transform.

What if you are doing a transform that might not work? We shouldn't have to look far for a transform that might NOT work.
For example, from the README on async-transforms:
const IdToUser = z
  .string()
  .uuid()
  .transform(async (id) => {
    return await getUserById(id);
  })

If you wrote this - hopefully your PR reviewer says, "what if getUserById doesn't work?" Your answer cannot be, "well .transform must assume it operates on valid date".  So I guess the README should update their code snippet:
const IdToUser = z
  .string()
  .uuid()
  .transform(async (id) => {
    return await getUserById(id)
       .catch(e => {
          logger.log('Who would have guessed this person could not be found')
          // zod say you can't throw an error in a transform
          return null
       })
  })
  .refine(v => v, {
     message: 'See that transform above, must be that getUserById did not find our user',
  })

You know what else smells? You don't have access to id in your message. Isn't that a shame?
Here's a playground: https://runkit.com/morgs32/zod-transform
Thoughts?? @colinhacks whaddya think?

@scotttrinh same stuff then inside #330, all of this is actually only OK with parse but not with safeParse
Your — elegant, I admit — example :
// Query parameter schema
const loose = z.object({
  aBoolean: z.string().transform(Boolean),
  aNumber: z.string().transform(Number),
});

// Domain schema
const query = z.object({
  aBoolean: z.boolean(),
  aNumber: z.number(),
});

const fromQueryParams = z.preprocess(
  loose.parse,
  query
);

actually throws on
fromQueryParams.safeParse({ aBoolean: 'hu', aNumber: 'ho' })

Of course
const fromQueryParams = z.preprocess(
  loose.safeParse, // <= here
  query
)

is not an option because preprocess is not intendented to work like this,
and
const fromQueryParams = z.preprocess(
  val => {
    const parsed = loose.safeParse(val)
    // ^ HERE
  },
  query
)

leave me in a doubt of what to do on parsed.error

I think in that last case, ctx inside preprocessors would be a way to go. Any temporary workarounds ?

@scotttrinh I think I found a solution, less elegant but working apparently.
Based on your "loose then strict" suggestion; but scattered on each property (to actually make a plain big z.object; .mergeable in the process)
import { z, ZodType } from 'zod'

const stringToNumberSchema = (def: number) => (z.string().default(`${def}`).transform(Number))
const safePreprocessor = <O, Z extends ZodType<O>> (preprocessorSchema: Z) => (val: unknown): O | null => {
  const parsed = preprocessorSchema.safeParse(val)
  if (!parsed.success) {
    return null
  }
  return parsed.data
}

const _paginationSchema = z.object({
  skip: z.preprocess(
    safePreprocessor(stringToNumberSchema(0)),
    z.number().min(0)
  ),
  limit: z.preprocess(
    safePreprocessor(stringToNumberSchema(20)),
    z.number().min(0).max(100)
  )
}).strict()

export type PaginationQuery = z.infer<typeof _paginationSchema>

Can I have your thoughts ?

Perhaps you should be able to add issues inside preprocess similar to superRefine. I'll look into this. It would save you from having to call coerceString twice.

@colinhacks are you still considering this?


MIGRATIONS-867 Switched to HAProxy 2.7

Signed-off-by: Chris Helma<EMAIL_ADDRESS>Description
The mirroring SPOA suddenly stopped working, and switching from HAProxy 2.8 to 2.7 seemed to fix it.
Specifically, the HAProxy Primary w/ mirroring spun up fine and lived until you sent a request to it, at which point HAProxy dies with this message:
root@def18754cf21:/# haproxy -f /usr/local/etc/haproxy/haproxy.cfg
[NOTICE]   (125) : New program 'mirror' (127) forked
[NOTICE]   (125) : New worker (128) forked
[NOTICE]   (125) : Loading success.
[ALERT]    (128) : A bogus APPCTX [0x5651c586b880] is spinning at 4528372 calls per second and refuses to die, aborting now! Please report this error to developers [strm=0x5651c586b36
0,8 src=<SPOE> fe=mirror be=mirroragents dst=<SPOE> txn=(nil),0 txn.req=-,0 txn.rsp=-,0 rqf=80c020 rqa=0 rpf=8040a060 rpa=0 scf=0x5651c5738180,CLO,40 scb=0x5651c57368f0,EST,1 af=0x565
1c586af60,9 sab=0x5651c586b880,0 cof=(nil),0:NONE((nil))/NONE((nil))/NONE(-1) cob=(nil),0:NONE((nil))/NONE((nil))/NONE(-1) filters={} applet=0x5651c52eb180(spoe_applet) handler=0x5651
c511a770(main+0xf1930)]
A bogus APPCTX [0x5651c586b880] is spinning at 4528372 calls per second and refuses to die, aborting now! Please report this error to developers [strm=0x5651c586b360,8 src=<SPOE> fe=m
irror be=mirroragents dst=<SPOE> txn=(nil),0 txn.req=-,0 txn.rsp=-,0 rqf=80c020 rqa=0 rpf=8040a060 rpa=0 scf=0x5651c5738180,CLO,40 scb=0x5651c57368f0,EST,1 af=0x5651c586af60,9 sab=0x5
651c586b880,0 cof=(nil),0:NONE((nil))/NONE((nil))/NONE(-1) cob=(nil),0:NONE((nil))/NONE((nil))/NONE(-1) filters={} applet=0x5651c52eb180(spoe_applet) handler=0x5651c511a770(main+0xf19
30)]
  call trace(11):
  | 0x5651c5024e92 [c6 04 25 01 00 00 00 00]: main-0x3fae
  | 0x5651c51ff256 [e9 6d fe ff ff 0f 1f 44]: task_run_applet+0x2f6/0x7fb
  | 0x5651c51b5e5a [49 89 c7 eb 0f 90 4c 89]: run_tasks_from_lists+0x40a/0x89e
  | 0x5651c51b666c [29 44 24 1c 8b 4c 24 1c]: process_runnable_tasks+0x37c/0x6f1
  | 0x5651c5184aea [83 3d 6f fb 3e 00 01 0f]: run_poll_loop+0x13a/0x554
  | 0x5651c5185129 [48 8b 1d 70 14 17 00 4c]: main+0x15c2e9
  | 0x5651c502a87d [31 c0 e8 5c 00 1d 00 31]: main+0x1a3d/0x2eb7
[NOTICE]   (125) : haproxy version is 2.8-dev3-e74d77b
[NOTICE]   (125) : path to executable is /usr/local/sbin/haproxy
[ALERT]    (125) : Current worker (128) exited with code 139 (Segmentation fault)
[ALERT]    (125) : exit-on-failure: killing every processes with SIGTERM
[ALERT]    (125) : Current program 'mirror' (127) exited with code 0 (Exit)
[WARNING]  (125) : All workers exited. Exiting... (139)

Issues Resolved
N/A
Testing
Ran the demo, hit the HAProxy primary w/ traffic.
(.venv) chelma@3c22fba4e266 cluster_traffic_capture % ./demo_haproxy.py
Creating primary cluster...
Waiting up to 30 sec for cluster to be active...
Cluster primary-cluster is active
Creating shadow cluster...
Waiting up to 30 sec for cluster to be active...
Cluster shadow-cluster is active
Building HAProxy Docker images for Primary and Shadow HAProxy containers...
Executing command to build Docker images: ./build_docker_images.py --primary-image haproxy-primary --primary-nodes host.docker.internal:9200 host.docker.internal:9201 --shadow-haproxy host.docker.internal:81 --shadow-image haproxy-shadow --shadow-nodes host.docker.internal:9202 host.docker.internal:9203 --internal-port 9200
Subshell> Copying Docker-related files to: /tmp/cluster_traffic_capture
Subshell> Writing HAProxy Config to: /tmp/cluster_traffic_capture/haproxy_no_mirror.cfg
Subshell> Writing HAProxy Config to: /tmp/cluster_traffic_capture/haproxy_w_mirror.cfg
Subshell> Building HAProxy Docker image for Primary Cluster...
Subshell> Primary HAProxy image available locally w/ tag: haproxy-primary
Subshell> Building HAProxy Docker image for Shadow Cluster...
Subshell> Shadow HAProxy image available locally w/ tag: haproxy-shadow
Starting HAProxy container for Shadow Cluster...
Starting HAProxy container for Primary Cluster...

HAProxy is currently running in a Docker container, available at <IP_ADDRESS>:80, and configured to pass traffic to an
ES 7.10.2 cluster of two nodes (each running in their own Docker containers).  The requests/responses passed to the
cluster via the HAProxy container will be logged to the container at /var/log/haproxy-traffic.log.  The requests are
mirrored to an identical shadow cluster.

Stopping cluster primary-cluster...
Cleaning up underlying resources for cluster primary-cluster...
Stopping cluster shadow-cluster...
Cleaning up underlying resources for cluster shadow-cluster...
Cleaning up underlying resources for the Primary HAProxy container...
Cleaning up underlying resources for the Shadow HAProxy container...

chelma@3c22fba4e266 cluster_traffic_capture % curl -X PUT 'localhost:80/noldor/_doc/1' -H 'Content-Type: application/json' -d'{"name": "Finwe"}'
{"_index":"noldor","_type":"_doc","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}%
chelma@3c22fba4e266 cluster_traffic_capture % curl -X GET 'localhost:9203/noldor/_doc/1'
{"_index":"noldor","_type":"_doc","_id":"1","_version":1,"_seq_no":0,"_primary_term":1,"found":true,"_source":{"name": "Finwe"}}%

Check List

[ ] New functionality includes testing

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented
[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #105 (5ae3dd6) into main (e1841ab) will not change coverage.
The diff coverage is n/a.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@           Coverage Diff           @@
##             main     #105   +/-   ##
=======================================
  Coverage   91.74%   91.74%
=======================================
  Files          24       24
  Lines        1115     1115
  Branches      110      110
=======================================
  Hits         1023     1023
  Misses         80       80
  Partials       12       12

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


fix: use a gauge for mgas_processed metric

Previously we use a Counter which only accepts integer increments, however since we report in millions of gas processed, we lose a lot of precision on the metric.
For example:

Any block below 1 million gas processed is counted as 0
Any block where the amount of gas is 1m < x < 2m gets counted as 1m gas processed

(Taken from the cache branch)

Codecov Report

Merging #2353 (a269b98) into main (1026be0) will decrease coverage by 20.76%.
The diff coverage is 0.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff             @@
##             main    #2353       +/-   ##
===========================================
- Coverage   60.41%   39.65%   -20.76%
===========================================
  Files         480      480
  Lines       58130    58130
===========================================
- Hits        35119    23052    -12067
- Misses      23011    35078    +12067

Flag
Coverage Δ

integration-tests
?

unit-tests
39.65% <0.00%> (-14.74%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

crates/stages/src/stages/execution.rs
0.00% <0.00%> (-63.55%)
:arrow_down:

... and 243 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Several refactors and add type infer

close #18
Basic type inference on columns, constants, add/sub and comparison.

Codecov Report

Merging #94 (b0771d8) into main (b0a6484) will decrease coverage by 0.62%.
The diff coverage is 80.82%.

@@            Coverage Diff             @@
##             main      #94      +/-   ##
==========================================
- Coverage   87.17%   86.54%   -0.63%
==========================================
  Files          71       72       +1
  Lines       11996    12687     +691
==========================================
+ Hits        10457    10980     +523
- Misses       1539     1707     +168

Impacted Files
Coverage Δ

xngin-expr/src/error.rs
0.00% <ø> (ø)

xngin-expr/src/lib.rs
100.00% <ø> (ø)

xngin-expr/src/pred.rs
96.96% <ø> (ø)

xngin-plan/src/error.rs
50.00% <ø> (ø)

xngin-plan/src/rule/col_prune.rs
93.51% <ø> (-0.44%)
:arrow_down:

xngin-plan/src/rule/expr_simplify.rs
94.70% <ø> (-0.57%)
:arrow_down:

xngin-plan/src/rule/joingraph_initialize.rs
92.34% <ø> (-3.93%)
:arrow_down:

xngin-plan/src/rule/mod.rs
85.00% <ø> (ø)

xngin-plan/src/rule/op_eliminate.rs
74.84% <ø> (ø)

xngin-plan/src/rule/outerjoin_reduce.rs
80.00% <ø> (-2.36%)
:arrow_down:

... and 22 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b0a6484...b0771d8. Read the comment docs.