mautrix-{meta,signal,whatsapp}: build with goolm

Description of changes

After olm gained knownVulnerabilities in #334638, build these bridges using the pure-Go goolm library instead of libolm bindings. I also tried converting mautrix-discord, but it failed to build for what appears to be an upstream reason, but I saw no reason to block the other three. The other mautrix-* bridges in nixpkgs are Python bridges, and I didn't look into building those without olm.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Tagging @sumnerevans here as I expect he’ll have opinions about this – per https://github.com/NixOS/nixpkgs/issues/336052#issuecomment-2298809378 apparently goolm has known issues with these bridges.

I think this should be an option for now, i.e., add it as a package argument so it's easy to override. I'm impartial about which to make the default (libolm has the vulnerability, goolm is potentially not adhering to the spec yet and hasn't been audited).

I think this should be an option for now, i.e., add it as a package argument so it's easy to override.

Do you have any preferences about what shape the option should take? Is it withGoolm ? false?

We (Beeper) tried to use goolm in our next-generation Android client and ran into weird issues. It's not entirely clear whether the issues were due to goolm itself or the way that we were using the library. We ended up just using libolm to reduce the number of variables that could be causing our issues.
I definitely think compilation with goolm should be added as a disabled option for now. Selfishly, it would be great if some people started testing it out more in production and reported any issues with it.

Selfishly, it would be great if some people started testing it out more in production and reported any issues with it.

I doubt any of those issues would be fixed.
The upstream repo has only seen a total of 16 commits by a single author.
There's also nothing that indicates it's more secure than libolm. Rather the opposite.

Mautrix uses their own fork.

Ok, here we go. OfBorg and Hydra won't build these as-is, but I tested the builds with commands like
nix build --impure --expr 'let p = import ./. {}; in p.mautrix-whatsapp.override { withGoolm = true; }'

I intend to use these options for my self-hosted Beeper bridges, and I'll report any issues I run into.

How about:

This option enables the use of the goolm library instead of libolm for end-to-end encryption. Using goolm is not officially supported by the mautrix developers, but they are interested in people trying it out and reporting any issues they run into.

@sumnerevans , I'm particularly interested in your feedback here to make sure that I don't misrepresent what Beeper wants to promise/request.

We marked it as experimental as of https://github.com/mautrix/go/releases/tag/v0.17.0, so maybe:

This option enables the use of an experimental pure-Go implementation of the Olm protocol instead of libolm for end-to-end encryption. Using goolm is not recommended by the mautrix developers, but they are interested in people trying it out in non-production-critical environments and reporting any issues they run into.

Given the issues that people have already mentioned, I think we should warn people about using this in production-critical environments.


[RFE] Add ability to add environmental variables to LB's via the UI

Rancher Versions:
Server: v1.5.3
Steps to Reproduce:
Enhancement request to add environmental variables via the UI for load balancers. understandably we cant have all the things, but this is super helpful especially when working on a development setup that uses vagrant.

What exactly is this useful for?

This was created after I opened a support ticket.
I'm not able to start the rancher load balancers in a vagrant machine. They constantly restart with
time="2017-03-30T10:34:44Z" level=info msg="Starting Rancher LB service"
time="2017-03-30T10:34:44Z" level=info msg="LB controller: rancher"
time="2017-03-30T10:34:44Z" level=info msg="LB provider: haproxy"
time="2017-03-30T10:34:44Z" level=info msg="starting rancher controller"
time="2017-03-30T10:34:44Z" level=info msg="Healthcheck handler is
listening on :10241"
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x63 pc=0x7f841b940259]

runtime stack:
runtime.throw(0x278eca0, 0x2a)
/usr/local/go/src/runtime/panic.go:547 +0x90
runtime.sigpanic()
/usr/local/go/src/runtime/sigpanic_unix.go:12 +0x5a
[...]

To mitigate this issue I had to add the label GODEBUG=netdns=go to the load balancer (and the healthcheck as well).
So yes @vincent99 as you implied, fixing this isn't prio 1. And no I don't necessarily need a fix.
I'd prefer you to raise the prio on this: https://github.com/rancher/rancher/issues/6111 (open since october) which is the core of this issue.

Looks like this was addressed in 1.6.0 #6111


Support for json arrays on root level of body.

With using the groovy pact consumer there is no way, when using the withBody() dsl & matchers, to declare a json array on the root body. While I've heard of this being a bad idea for API design, it should still be supported.
I've tried looking at JsonBuilder as an example, it can build a root level array in a few different ways, seen here and here, but they don't feel like the right fit for the PactBodyBuilder, so I've come up with my own pretty basic solution.
It involves the keyword 'rootJsonArray' in the dsl.
withBody() {
  rootJsonArray maxLike(10) {
    id identifier
      lineItems minLike(1) {
        id identifier
          amount numeric
          productCodes eachLike { code string('A100') }
      }
  }
}

commit is here: :boom: stefan-lz/pact-jvm@7bd807fc689b35da0bf833d4a827e15d53938b4a
I was also toying around with the idea of something like another dsl method similar to maxLike/minLike.
withBody() {
  orders rootJsonArray maxLike(10) {
    id identifier
      lineItems minLike(1) {
        id identifier
          amount numeric
          productCodes eachLike { code string('A100') }
      }
  }
}

but failed to get anything working quickly.

Got arrays working by extending withBody to accept an array:
withBody([
    1, 2, 3
])

generates
[
    1,
    2,
    3
]

and
withBody([
        {
          id identifier(1)
          name 'item1'
        }, {
          id identifier(2)
          name 'item2'
        }
])

generates
[
    {
        "id": 1,
        "name": "item1"
    },
    {
        "id": 2,
        "name": "item2"
    }
]

It has to have the brackets,
withBody [
    1, 2, 3
]

generates a compiler error

This now works:
withBody eachLike {
        id identifier(1)
        name 'item1'
}

Cool, that seems like a nice fit to the DSL. I'll have a play tonight and check it out.

Was this ever done for the Java DSL? Or rather, is there a way to do it with the Java DSL? I want to specify that the payload of a post request can be an arbitrarily long list of strings like ["a", "b", ...] but I cant' find a way to do it.

I thought this was supported, but I can't seem to find the implementation.

Find it, but it was only implemented for arrays on objects: https://github.com/DiUS/pact-jvm/issues/256#issuecomment-222295887
I'll re-open that issue.


List does not work with Adaway or HostsMan

The following URL (https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts) is not able to be imported to HostsMan or Adaway apps. HostsMan returns the error "This source is not compatible. Server does not return all the necessary information." Adaway states "Online: Not Available".

I'm not a Windows, nor an Android user.   I've had several discussions with folks at Adaway so I'm pretty sure we're not doing anything majorly wrong here.
Does anyone have insight on this?
@nimkip this might be a hostsman bug...

@nimkip see https://github.com/AdAway/AdAway/wiki/HostsSources -- this repo is listed there.
However it looks like this is an open bug in Adaway.  See https://github.com/AdAway/AdAway/issues/700
Maybe you could comment there and urge them into action?
You could always just clone this repo, and pull as needed prior to updating adaway from a local file.

This seems to be a bug with AdAway. It doesn't seem to show any GitHub URLs as being available. I've added my own hosts file for blocking Spotify ads and it does the same thing - "Local: not available, Online: not available".
However, since I've added my own hosts file I haven't gotten a single Spotify ad. So I'm willing to guess that it shows "not available" but can still access the hosts file and merge it. I've also noticed it shows the right number of sources applied, even if a few (such as the hosts file for this repo) show up as "not available" in the app.

Good catch. Upon further inspection AdAway does indeed import the hosts correctly, even though the app says it does not. HostsMan on the other hand will not attempt to download the list.

Just installed HostsMan and was looking for the best distribution and found yours to try out but ran into the same issue.
HostsMan seems to be looking for 'hosts.zip' instead of RAW so if you offer a zipped link on GitHub (like is offered here) it might be all that is required for HostsMan to use your file.

@nimkip @mistermarmot @JasonD94 the repo now contains host.zip files and should now work with HostMan and Adaway.  😎

My initial assumption about the solution is wrong and HostsMan isn't using your HOSTS as a source yet.
HostsMan is looking for some handshaking information from the distribution server.
I'll create a post on their forums at http://forum.abelhadigital.com/index.php and see what they have to say about the requirements.  It would probably be resolved easily if you and their admin/coder/owner discuss it directly.

@StevenBlack it's still not letting it be used. It's something to do with GitHub I believe. Is there any way you could host it on another site similar to someonewhocares page?

This still isn't fixed. Why was this closed?

Because it's not my bug.

HostsMan recognize line
fe80::1%lo0 localhost

as ERROR/CORRUPTED line. But infact this line is VALID IPv6 localhost for OSX system.
http://osxdaily.com/2014/04/12/restore-original-hosts-file-mac/
@StevenBlack @aequis @Jezza-x3

@StevenBlack If you could host it on a site such as someonewhocares host file, then it would let us download. HostsMan doesn't work with github. That's the main problem.


Razor Components Generating g.cs with AddMarkupContent() instead of AddContent()

Some of the pages (I haven't found the pattern) are generated using AddMarkupContent() which will keep the HTML readable, but some with  AddContent() where all new lines, tabs etc. are replaced by respective
 characters.
Can be AddMarkupContent() widely used?

Thanks for contacting us, @mashbrno.
We track this fix in https://github.com/aspnet/AspNetCore/issues/6185


Add Hosterfy hosting provider instructions

Update informations about using Geyser on Hosterfy (hosting provider).

The provider list is on alphabetical order, please put your host in the correct spot.

Done.


fish vs carapace in 2024

Request
I'm a fish user, and I'm new to caparace
I followed the step describe in https://carapace-sh.github.io/carapace-bin/setup.html
I was surprised to see
carapace --list | awk '{print $1}' | xargs -I{} touch ~/.config/fish/completions/{}.fish # disable auto-loaded completions (#185)

see

#185

I can see the reference here
https://github.com/carapace-sh/carapace-bin/blob/1b1ecb18cac516d0e68ec373da718e14faf9315f/docs/src/setup.md?plain=1#L32
I found the code was moved via 6ad36ab6bdaabcea04b7e2c2dd244c888063b0b0 and was initially added via 79e988ed8c311b63b2fa62ba36849dd25ae2bb68 (via #880)
All these are pretty old, the #185 was mentioning

https://github.com/fish-shell/fish-shell/issues/6716

Is there any progress around this ?
I saw the status of fish-shell#6716 but I was curious if there is nothing else that evolved in the last few years. I find pretty painful to create empty file it's a strange logic
I found this https://github.com/cli/cli/issues/360#issuecomment-849593706
Proposed solution
I would have expected carapace to:

create the empty file via the go, not via an shell rule

the empty file could have a better content than simply being empty

they could contain comment about why the file is there and empty

use fish "complete -e"  can't help ?

maybe the fish file could contain the complete command instead of being empty

for example
~/.config/fish/completions/git.fish could contain this
complete -e 'git'
complete -c 'git' -f -a '(_carapace_lazy git)'

Anything else?
I might be wrong about the things I'm asking or the point I'm raising, but I would like to know if we couldn't do something better for the fish integration
Thanks

Polar

Well it is only a workaround and I expected fish to fix this fairly quickly (little did I know).
The problem is that this is a bug in fish and from the outside this was the only way to get around it.
Erasing existing completion doesn't work, because fish loads the system completion on first invocation (pressing tab).
It does so even if one is already defined which clutters it.
Shell startup time is critical which limits other potential workarounds.
Generating the files in go is possible, but it is quite an invasive step as it essentially kills the native completion of fish which might be unwarranted when one disables a completer from carapace.
To be honest, this just needs to be fixed in fish with the tombstone approach mentioned here.

Thanks for your reply. I understand.
Do you think I could quote your reply in the involved fish discussions, and ask them for a status?

Sure

A fix was provided. Let now wait for it to be released

Hello carapace is adding like 3 seconds to my fish prompt startup and I think is because of these completions. Is there any fix for this?

Hello carapace is adding like 3 seconds to my fish prompt startup and I think is because of these completions. Is there any fix for this?

Even with the workaround it shouldn't be this bad :thinking:.
The files in ~/.config/fish/completions just need to be created once for each new completer.
So the workaround doesn't need to be executed every time.
The complete command in fish also tends to be quite slow (which is a bit weird),
if you want the full startup speed you could also export the scripts instead of sourcing carapace.
carapace git fish > ~/.config/fish/completions/git.fish

But you'd need to update PATH and add the set-env/get-env functions manually.

The point raised by @azlekov is legitimate, and the workrounds you provided could provide a solution.
But @rsteube said a few months ago fish needed to change something in a way to use something called "tombstone"
But the issue is now fixed
https://github.com/fish-shell/fish-shell/commit/5918bca1eba861e3e8f69f055aac20e9cae1af3c
Shouldn't the solution be now updated as fish should support it.
Please note, I'm aware there is no new fish release since March 2024

Well, depends on how easy it is to install a nightly (and how stable the master is).
But I can add a notice to the instructions, yes.
Might slowly be time to have another look at this.
Invocation delay of carapace is in the range of 7-10ms, which I personally don't consider an issue.
But the slow performance of the complete command and the amount of completers registered due to the bridges makes the whole topic relevant again.
It's possible to have a near zero overhead for most shells, but it involves a slightly different approach (see above).


Can we achieve below scenario ?

: As soon as user start typing "@" it should highlight that word until user select any matches from suggestions.
: If next word is not starting with "@" then it should not be highlighted.
: Later in same sentence If user enters "@" again then words should again highlight until he select again from suggestions . Probably we need to have one index or check where it can be identified whether it's current word starts with @.

See in above image first and fourth word getting highlighted only after selecting it from mentioned suggestions.
But I want they should also gets highlighted while typing.

Hello @aman-sankey, thanks for your feedback.
You can add another pattern part type that will highlight your typing mention:
{
    pattern: /(@\w+\b)/i,
    textStyle: {
        color: 'blue',
    },
}


The MainViewModelTest.kt is not in the androidTest folder.

The Step 6 of codelab requires to review existing code and mentions to open MainViewModelTest.kt from the androidTest folder.
However this file is actually present in the test folder.

@ PratikBanodkar
Thank you for reaching out to us. The issue is fixed and will be published soon.

Duplicate of #66, 146


error: root manifest not found

I am getting this error while running the command:
swift run swift-doc coverage /Users/raj/my_ios_project/project/Sources --output "cov.json"

@RajDSS I don't have enough information to reproduce this. However, searching for this error message took me to this SO thread, which makes me think that you're running swift run swift-doc outside of the swift-doc project directory. If you're just using it, I'd recommend installing swift-doc using Homebrew instead of building from source.


Push job-scheduler Jar to Maven

As of now the ISM plugin team has to depend on infra team to push the notification and job-scheduler jars to maven before they start the build process for new release.
This job will push the notification jar to staging repository in maven from where the user can publish the jar into prod. The job will be triggered whenever a new tag is cut for the new release.

These tests have been flaky beforehand, unrelated to this PR.
They are failing because of     org.elasticsearch.discovery.MasterNotDiscoveredException
Will need to look into it/refactor the tests.


.Net Core 3.1 broke my JSON

Issue Title
HttpPost with FromBody was empty after upgrade
General
My javascript was incorrectly serializing an integer field using quotes, e.g. "MystringKey":"ValueX", "MyInt":"123"
This worked just fine in .Net Core 2.2
All of a sudden it stopped working.  The framework now requires "MystringKey":"ValueX", "MyInt":123
There is evidently no way to log deserialization issues in the framework.
In the past this code worked for me to get the raw request - but that has now been broken too.
Provide details on the problem you are experiencing including the .NET Core version, operating system version and anything else that is relevant.
It would be great if the framework had a way to optionally log raw POST request bodies, and to optionally log deserialization errors.
Please provide a code sample for your issue if it is relevant, either inline, link to a gist (or similar) or add it as a zipped attachment.
[HttpPost]
        public JsonResult CreateTable([FromBody] CreateTableRequest req)
        {
            CreateTableResponse r = new CreateTableResponse();

            if (req == null)
            {
                r.Error = true;
                r.Message = $"Internal Error. Invalid CreateTable input - unable to process request parameters";
                // THIS LOGGING OF RAW request now ONLY WORKS IF THE JSON CAN BE PARSED OK as of .Net Core 3.1!
                // as of .Net Core 3.1 this does not work "AutoDetectRowsToExamine":"200"
                // must be "AutoDetectRowsToExamine":200
                var m = new MemoryStream();
                Request.Body.Position = 0;  // Always fails as of .Net Core 3.1
                Request.Body.CopyTo(m);
                var rawRequest = System.Text.Encoding.UTF8.GetString(m.ToArray());
                _logger.LogError($"CreateTable deserialization failed for Json {rawRequest}");

For some issues, you will get a quicker and better response if you file it at a more specific .NET repo. For example, if the problem is with ASP.NET Core, you are encouraged to use the aspnet/home repo.

The Migration Guide contains details on how to still use the Newtsonsoft.Json based serialization in <3.0 or the guide How to migrate from Newtonsoft.Json to System.Text.Json which has a dedicated section Allow or write numbers in quotes for setting up a compatible behavior.

Looks like this was answered. Thanks for the question!

Looks like this was answered. Thanks for the question!


Restrict DEVELOPER_DIR regex to the start of the line

All these patterns should only only be replaced at the start of the line since they should always be replacing absolute paths to Xcode. This theoretically improves the regex performance but more importantly should make the regex slightly more clear.

yep, can verify adding this works, and adding ^foo for sure breaks it


TLS test timeout

Issue
TLS test is consistently failing on initial deploying
Solution
Increase the timeount for the initial deployment

Doesn't seem to be topical anymore.


Reducer is called multiple times

i have dispatched an action type from an action. when i try to call a function in my .js file, the reducer is getting called multiple times.
This is my function to display data :
showCategoryDetails(id){
      this.props.getLocalCategoryDetails(id, this.props.activeUser.token).then(()=>{
              this.setState({Details:this.props.selectedLocalCategory.CatDetails});
               this.props.getLocalCategoryObject(id, this.props.activeUser.token).then((id)=>{

          this.setState({categoryDetails:this.props.selectedLocalCategory.Category});

       }).catch((error)=>{
       });
          }).catch((error)=>{

          });

     }

This is my reducer:
import * as types from '../constants/actiontypes';

export default function(state = null, action) {
    switch (action.type) {
        case types.GET_LOCAL_CATEGORY:
        console.log("apiResult",action.apiResult);
             return Object.assign({}, state, {Category: action.apiResult});
            break;

          case types.GET_LOCAL_CATEGORY_DETAIL:
            console.log("detail apiResult",action.apiResult);
                 return Object.assign({}, state, {CatDetails: action.apiResult});
                break;

    }
    return state;
}

the GET_LOCAL_CATEGORY_DETAIL is called multiple times.
How do I fix this one?

Whatever's going on here, this is a usage question, not a bug in Redux.  You're better off asking this question on Stack Overflow instead.

thank u


Adopt a governance document

Description forthcoming. Opening now a placeholder now, for a link in the WebDX meeting notes.
At present, the feature-set repo is controlled by the repo committers, with no delegation or formal decision-making process beyond habit and preference. We ought to adopt a governance document and process that establishes guardrails against unwanted unilateral action and lowers a ladder to contributors to take increasing ownership of the project.
In the 22 June 2023 WebDX CG general meeting, I proposed adopting a governance document based on the mdn/browser-compat-data governance model. See web-platform-dx/feature-set governance plan proposal on Google Docs for more information on:

Why we should adopt something like BCD's governance doc
What we can learn from BCD's experience
How we can bootstrap the governance group in the short term

The linked document is open to comments and suggestions, or you can comment here.
If you'd like to nominate yourself to help edit and adopt the governance document, please mention it below.

For the record, I'm nominating myself to the interim governance group and volunteering to shepherd the document editing process.

I'm also nominating myself for the interim governance group.

I'd nominate myself.

I'm nominating myself to the interim governance group.

@dfabulich Could you email me (email in profile), from your preferred email address for this project? I have a draft governance doc to share. Thank you!

I'm nominating myself for the interim governance group.

Over the last few weeks, a draft governance document has been reviewed and edited by the interim governance group. We're now opening up this process to other members of the WebDX community. You can read, comment, or add suggestions to the draft governance doc in Google Docs, or you can leave a comment on this issue. Not a lot has changed from the BCD doc that this is based on, but some highlights include:

Adding project- and organization-specific details (e.g., referencing the W3C CEPC as a code of conduct, mentioning feature descriptions and statuses, etc.).
Putting the Baseline definition (and other feature status definitions) and changes to the statuses themselves under control of the Owners group.
Explicitly calling for the owners to exhaust attempts to reach a consensus before forcing a vote.
Explicitly making Peer and Owner nomination deliberations private (to avoid needlessly embarrassing nominees who can't achieve consensus or pass a vote).
Setting a quarterly-or-better meeting cadence, for resolving outstanding questions.

All that said, I imagine that once we adopt a document, it'll be revised over time as we gain practical experience with it.
I'm looking forward to your comments here or on the doc!

This was fixed by https://github.com/web-platform-dx/web-features/pull/359. Thanks everyone!


Add support for mobile

Currently, we use the official replicate library, which requires node (cfr https://github.com/replicate/replicate-javascript?tab=readme-ov-file#replicate-nodejs-client). This plugin is thus not compatible with mobile versions of Obsidian.
We could switch to this: https://github.com/zebreus/replicate-api
Unofficial, but supposed to work both in the browser and in node.
It's possible to pass a custom FetchFunction, so we could pipe the Obsidian fetchUrl function (?):
https://github.com/zebreus/replicate-api/blob/master/src/helpers/makeApiRequest.ts

Another alternative: https://github.com/nicholascelestin/replicate-js
That one also supports customizing how requests are being made:
// Example using axios instead of fetch
import axios from 'axios';

const httpClient = {
    // Method arguments use object destructuring
    // All arguments are optional, can be in any order, but cannot be renamed
    get: async ({url, token, event}) => {
        const response = await axios.get(url, {headers: {'Authorization': `Token ${token}`}})
        console.log(`Handling ${event} event`); // Possible values: getModel, getPrediction
        return response.data;
    },
    post: async ({url, body, token, event}) => {
        const response = await axios.post(url, body, {headers: {'Authorization': `Token ${token}`}})
        console.log(`Handling ${event} event`); // Possible values: startPrediction
        return response.data;
    }
}
const replicateAxios = new Replicate({pollingInterval:5000, httpClient: httpClient});
const model = await replicateAxios.models.get("replicate/hello-world") // getModel event
const prediction = await model.predict({ text: "test"});// startPrediction, getPrediction events


FirebaseAuth.DefaultInstance.CurrentUser is not updated

Hello,
More bugs for Firebase:
Please fill in the following fields:
Unity editor version: 2018.3.14f1
Firebase Unity SDK version: 6.2.2
Firebase plugins in use (Auth, Database, etc.): Auth, Analytics, Database, Crashlytics, Messaging
Platform you are using the Unity editor on (Mac, Windows, or Linux): MacOS Catalina 10.15 beta
Platform you are targeting (iOS, Android, and/or desktop): iOS, Android
Please describe the issue here:
In Unity Editor, when we call ReauthenticateAsync successfully, we are still unable to call UpdateEmailAsync or update any sensitive user information after:

This operation is sensitive and requires recent authentication. Log in again before retrying this
request.

We need to logout and re-signin "manually".
Also, after all update operations, including UpdateUserProfileAsync, we need to logout and signin again to see the changes in FirebaseAuth.DefaultInstance.CurrentUser, even after restarting the Editor.
Not tested on Android or iOS.
Thanks for your support!

Hi @QuentinGprd,
Unfortunately I wasn't able to recreate this issue.
I tested using the auth/test_app as a baseline.  In ReauthenticateAysnc I hard-coded an update to the user profile by updating the display name once the re-authentication completed.
Line 442, immediately after the LogTaskCompletion if {} block, add:
return UpdateUserProfileAsync(newDisplayName: "<something>");
This is successfully updates the profile (immediately after re-authenticating via email) without an error in my testing.
As an additional test, in UpdateUserProfileAsync I chain a task which updates the email address of the account. I then output both auth.CurrentUser and the DefaultInstance.CurrentUser when it completes.  In my tests they were both successfully updated:
Line 303, add:
auth.CurrentUser.UpdateEmailAsync("<some_test_email>")
   .ContinueWithOnMainThread(update_email_task => {
        EnableUI();
        if (LogTaskCompletion(update_email_task, "User Email")) {
            DisplayDetailedUserInfo(auth.CurrentUser, 1);
            DisplayDetailedUserInfo(Firebase.Auth.FirebaseAuth.DefaultInstance.CurrentUser, 2);
       }
});

Could you update this code to make it fail in the way that you're seeing errors in your app?  Thanks!

I just added the following button in the auth/test_app UIHandler:
if (GUILayout.Button("Update DisplayName")) {
            UpdateUserProfileAsync("Kiki");
}

Here is a video of the bug
As you can see, after re-entering Play Mode, the previous name is still displayed. I need to logout then sign in to get the new one.
Let me know if you need anything else.

Hi this was fixed in 6.4.0 https://firebase.google.com/support/release-notes/unity#version_640_-_august_26_2019


Implement a simplified version of from_json

This PR tries to implement a simple but efficient version of from_json SQL function. in particular, it simply extracts the key-value pairs from the input json strings. These pairs are extracted by copying the substrings of the input without any data conversion or validation.
Demo:
Input:
[
"{\"Zipcode\" : 704 , \"ZipCodeType\" : \"STANDARD\" , \"City\" : \"PARC PARQUE\" , \"State\" : \"PR\"}",
"{\"Integer\":12345,\"String\":\"ABCXYZ\",\"Double\":1.1245}",
"{\"category\": \"reference\", \"index\": [4,{},null,{\"a\":[{ }, {}] } ], \"author\": \"Nigel Rees\", \"title\": \"{}[], <=semantic-symbols-string\", \"price\": 8.95}"
]

Output:
List 0: [0, 4]
	"Zipcode" : "704"
	"ZipCodeType" : "STANDARD"
	"City" : "PARC PARQUE"
	"State" : "PR"
List 1: [4, 7]
	"Integer" : "12345"
	"String" : "ABCXYZ"
	"Double" : "1.1245"
List 2: [7, 12]
	"category" : "reference"
	"index" : "[4,{},null,{"a":[{ }, {}] } ]"
	"author" : "Nigel Rees"
	"title" : "{}[], <=semantic-symbols-string"
	"price" : "8.95"

Here is a simple performance comparison with fully parsing json from file (which has output is just a table, not yet the desired map type for from_json). The comparison shows around 4X-5X difference in performance:
## Fully parse json:

| string_size  | Samples |  CPU Time  | Noise |  GPU Time  | Noise | Elem/s | bytes_per_second | peak_memory_usage |
|--------------|---------|------------|-------|------------|-------|--------|------------------|-------------------|
| 2^15 = 32768 |     35x | 438.905 ms | 2.30% | 438.821 ms | 2.30% | 1.090G |<PHONE_NUMBER> |         3.695 GiB |

## Simplified `from_json`:

| string_size  | Samples | CPU Time  | Noise | GPU Time  | Noise | Elem/s | bytes_per_second | peak_memory_usage |
|--------------|---------|-----------|-------|-----------|-------|--------|------------------|-------------------|
| 2^15 = 32768 |    171x | 87.881 ms | 5.11% | 87.859 ms | 5.10% | 5.446G |<PHONE_NUMBER> |         4.082 GiB |

Depends on:

https://github.com/rapidsai/cudf/pull/12432
https://github.com/rapidsai/cudf/pull/12455

Closes:

https://github.com/rapidsai/cudf/issues/12051

Note that from_json doesn't check for key duplicate:
val jsonString="""{"Double": 2.23, "Integer":12345, "String": "ABCXYZ", "Double": a}"""
val data = Seq((1, jsonString))
import spark.implicits._
val df=data.toDF("id","value")
df.show(false)

import org.apache.spark.sql.functions.{from_json,col}
import org.apache.spark.sql.types.{MapType, StringType}
val df2=df.withColumn("map",from_json(col("value"),MapType(StringType,StringType)))
df2.printSchema()
df2.show(false)

Output:
df2: org.apache.spark.sql.DataFrame = [id: int, value: string ... 1 more field]
root
 |-- id: integer (nullable = false)
 |-- value: string (nullable = true)
 |-- map: map (nullable = true)
 |    |-- key: string
 |    |-- value: string (valueContainsNull = true)

+---+------------------------------------------------------------------+----+
|id |value                                                             |map |
+---+------------------------------------------------------------------+----+
|1  |{"Double": 2.23, "Integer":12345, "String": "ABCXYZ", "Double": a}|null|
+---+------------------------------------------------------------------+----+

Just hit a performance issue with substring operation. I've filed https://github.com/rapidsai/cudf/issues/12445 for it.

Assuming that the input doesn't have UTF8 character, the implementation can be simplified and will have a very good performance. With input json strings having around 250 fields:

Parsing json into cudf column (similar to the old way from https://github.com/NVIDIA/spark-rapids/pull/6211):

Run:  [1/7] nested_json_gpu_parser [Device=0 string_size=2^0]
Fail: Unexpected error: std::bad_alloc: out_of_memory: RMM failure at:/home/nghiat/Devel/cudf/1/cpp/build/_deps/rmm-src/include/rmm/mr/device/pool_memory_resource.hpp:193: Maximum pool size exceeded

It fails immediately without any success!
With this new implementation:

| string_size  | Samples |  CPU Time  | Noise  |  GPU Time  | Noise  |  Elem/s  | bytes_per_second | peak_memory_usage |
|--------------|---------|------------|--------|------------|--------|----------|------------------|-------------------|
|      2^0 = 1 |   1600x |   2.836 ms | 26.84% |   2.831 ms | 26.76% |   5.157M |          5157188 |       140.688 KiB |
|     2^4 = 16 |   1792x |   3.933 ms |  2.47% |   3.929 ms |  2.46% |  59.456M |         59456135 |         2.197 MiB |
|    2^8 = 256 |   1632x |   8.150 ms |  8.16% |   8.146 ms |  8.15% | 458.799M |        458798726 |        35.142 MiB |
|  2^12 = 4096 |    727x |  20.637 ms |  5.05% |  20.631 ms |  5.05% |   2.898G |<PHONE_NUMBER> |       562.277 MiB |
| 2^16 = 65536 |     30x | 513.180 ms |  0.85% | 513.170 ms |  0.85% |   1.864G |<PHONE_NUMBER> |         8.786 GiB |

I started to look at this, but I am a little confused about the algorithm being used. We only care about a very specific level to extract. The top level of each element in the list passed in. So we would have a ListBegin as the very first token and a ListEnd as the very last token. That means each row of input data would also be wrapped in a StructBegin and a StructEnd.
.....

Your logic is right. But we are process characters (i.e., tokens) in parallel. If we want to find a match token _End to each Begin_ token directly from the list a tokens, we have to do linear search. My (complex) implementation involves some sort of computation for node levels, node parents etc... so we can maximize parallelism. Even that, I still have to do linear search for nested values to find the value boundary: https://github.com/NVIDIA/spark-rapids-jni/pull/844/files#diff-60a759906a1fdb92b9b286d781d66a2f3a25e7f7232b6ad27f75309cdcc78e2aR470.

I get that you are trying to process the tokens in parallel. That is why I wanted to just look for the level. It looks like we could do a few very simple kernels with almost no walking, just a few set offsets from known locations. But I would have to do a lot more digging into how the tokenization code works to know exactly what the levels need to be and exactly what the tokens are that we would need to look for. I think all we want is something like the following psudo code.
key_start = copy offsets[index] if level[index] == 3 and token[index] == FieldNameBegin;
key_end = copy offsets[index] if level[index] == 3 and token[index] == FieldNameEnd;
value_start = copy_offsets[index] if level[index] == 3 and token[index - 1] == FieldNameEnd;
value_end = copy_offsets[index] if level[index] == 2 and token[index] == StructMemberEnd;

But like I said before, if this works I am good with how it is right now. This could be an optimization that we look into as a follow on issue.

Even though I approved this I am not enough of a CUDA expert to really find anything itnerresting.  @hyperbolic2346 and @nvdbaranec could one or both of you also take a look?

FYI: I'm implementing the plugin work to hook up with this so I can run a benchmark to tell you how much performance gain we can get with this.

Some performance tests with a dataset of 670k rows, each row has >100 fields:
CPU only:
Run time: 0:00:16.046214

Old GPU implementation (https://github.com/NVIDIA/spark-rapids/pull/6211):
Run time:  <to be updated>

This GPU implementation:
Run time:  <to be updated>

build

build

build


Add multi-GPU unit test environment

Add a new group of agents to Jenkins CI to run multi-GPU workload. The agents will use a p2.8xlarge instance.
Add a simple test for gpu_hist.

@RAMitchell @trivialfis @canonizer

@RAMitchell The test tests/python-mgpu/test_gpu_updaters.py has failed due to difference between CPU and multi-GPU results:
======================================================================
FAIL: test_gpu_hist (test_gpu_updaters.TestGPU)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/workspace/tests/python-mgpu/test_gpu_updaters.py", line 31, in test_gpu_hist
    assert_gpu_results(cpu_results, gpu_results)
  File "/workspace/tests/python-mgpu/test_gpu_updaters.py", line 15, in assert_gpu_results
    .format(gpu_res["eval"][-1], cpu_res["eval"][-1])
AssertionError: GPU result don' match with CPU result (GPU: 42.155769, CPU: 41.519295)
-------------------- >> begin captured stdout << ---------------------

Should we be concerned with this error?

@hcho3 I will look into gpu hist after #3643.  For now I don't have much confidence in multi-gpu setup.

@trivialfis Would you like to add a multi-GPU test to #3643? If so, I can disable the failing test for now and merge this PR to master.

@hcho3 I will add cpp test for that.

@hcho3 This is amazing thanks. My view is we should disable any failing tests, merge this, then work to re-enable the tests.

@RAMitchell Got it. Let me go ahead and disable the failing test.
@trivialfis For CPP test, do you think we should run the same suite of tests on multi-GPU machine or a different suite?

@trivialfis One idea I have is to prefix name of Google test cases with MGPU_ to indicate that they should run on the multi-GPU machine. Something like
TEST(FooBarTest, MGPU_RunBar) {
    // Use device ordinals 0, 1, 2, ...
}

Actually, I kind of like the idea of keeping all tests together. For Python tests, we can add mgpu attribute to every test that's to be run with multiple GPUs.

Currently I put the multi-gpu test code inside (at the end of) normal cuda test files, wrapped with:
#if defined(XGBOOST_USE_NCCL)
TEST(Foo, MultiGpuBar) {
  // do something.
}
#endif

But there are other options too, I would love to hear about other opinions.

@trivialfis I don't think that would work, because one of the test configurations compile with NCCL (hence XGBOOST_USE_NCCL is defined) but runs on a single GPU.
I'm open to suggestions, but for now I am using MGPU_ prefix (for cpp) and mgpu attribute (for python) to indicate that a given test should use more than one GPU.

@hcho3 I agree with keeping all tests together.  But I don't think running multi-GPU tests on single GPU would introduce problem, since any multi-GPU code should also run on single GPU.  If such difficulties do arise we can pass --gtest_filter="-*.MGPU*" flag to the test executable.
Another option for not spiting up tests would be making a definition like #define TEST_WITH_MULTI_GPU, but to me it feels like a little bit redundant.
I don't have much experience in this and it's still in early stage.

@trivialfis It is certainly true that one should be able to run multi-GPU code on a single-GPU machine. However, clearly indicating the target machine of each test is still a good idea. Personally, I prefer MGPU_ prefix than XGBOOST_USE_NCCL, since the former is more concise and communicates semantics more clearly.
My rationale for separating single- and multi-GPU tests is that provisioning a large instance like p2.8xlarge takes siginficantly longer than provisioning p2xlarge. So to reduce test backlog, we want to run on the multi-GPU instance only those tests that would necessitate the plurality of GPU devices.


writeArrowFeather not working with nested type ?

Hi, in my case, I want to create a arrow file in client side, then pass to server side. But when I just try run writeArrowFeather,  will show the IndexOutOfBoundsException issues.
Exception in thread "main" java.lang.IndexOutOfBoundsException: index: 31393, length: 2320 (expected: range(0, 32768))
	at org.apache.arrow.memory.ArrowBuf.checkIndex(ArrowBuf.java:701)
	at org.apache.arrow.memory.ArrowBuf.setBytes(ArrowBuf.java:765)
	at org.apache.arrow.vector.BaseVariableWidthVector.setBytes(BaseVariableWidthVector.java:1244)
	at org.apache.arrow.vector.BaseVariableWidthVector.set(BaseVariableWidthVector.java:1059)
	at org.apache.arrow.vector.VarCharVector.set(VarCharVector.java:255)
	at org.jetbrains.kotlinx.dataframe.io.ArrowWriterImpl$infillVector$1.invoke(ArrowWriterImpl.kt:111)
	at org.jetbrains.kotlinx.dataframe.io.ArrowWriterImpl$infillVector$1.invoke(ArrowWriterImpl.kt:111)
	at org.jetbrains.kotlinx.dataframe.api.ForEachKt.forEachIndexed(forEach.kt:34)
	at org.jetbrains.kotlinx.dataframe.io.ArrowWriterImpl.infillVector(ArrowWriterImpl.kt:111)
	at org.jetbrains.kotlinx.dataframe.io.ArrowWriterImpl.allocateVectorAndInfill(ArrowWriterImpl.kt:197)
	at org.jetbrains.kotlinx.dataframe.io.ArrowWriterImpl.allocateVectorSchemaRoot(ArrowWriterImpl.kt:223)
	at org.jetbrains.kotlinx.dataframe.io.ArrowWriter$DefaultImpls.writeArrowFeather(ArrowWriter.kt:114)
	at org.jetbrains.kotlinx.dataframe.io.ArrowWriterImpl.writeArrowFeather(ArrowWriterImpl.kt:61)
	at org.jetbrains.kotlinx.dataframe.io.ArrowWriter$DefaultImpls.writeArrowFeather(ArrowWriter.kt:125)
	at org.jetbrains.kotlinx.dataframe.io.ArrowWriterImpl.writeArrowFeather(ArrowWriterImpl.kt:61)
	at org.jetbrains.kotlinx.dataframe.io.ArrowWriter$DefaultImpls.writeArrowFeather(ArrowWriter.kt:133)
	at org.jetbrains.kotlinx.dataframe.io.ArrowWriterImpl.writeArrowFeather(ArrowWriterImpl.kt:61)
	at org.jetbrains.kotlinx.dataframe.io.ArrowWritingKt.writeArrowFeather(arrowWriting.kt:89)
	at com.phodal.chapi.arrow.MainKt.main(Main.kt:26)
	Suppressed: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (33024)
Allocator(ROOT) 0/33024/264192/9223372036854775807 (res/actual/peak/limit)

		at org.apache.arrow.memory.BaseAllocator.close(BaseAllocator.java:437)
		at org.apache.arrow.memory.RootAllocator.close(RootAllocator.java:29)
		at org.jetbrains.kotlinx.dataframe.io.ArrowWriterImpl.close(ArrowWriterImpl.kt:247)
		at kotlin.jdk7.AutoCloseableKt.closeFinally(AutoCloseable.kt:64)
		at org.jetbrains.kotlinx.dataframe.io.ArrowWritingKt.writeArrowFeather(arrowWriting.kt:88)
		... 1 more

FAILURE: Build failed with an exception.

Here is my demo code with writer and some debug information:
val dataFrame = DataFrame.read("https://raw.githubusercontent.com/phodal-archive/apache-arrow-chapi-demo/master/data/0_codes.json")
dataFrame.schema().print()

val toArrowSchema = dataFrame.columns().toArrowSchema()
println(toArrowSchema.toJson())

dataFrame.writeArrowFeather(File("codes.arrow"))

When i try to debug, in the dataFrame.schema().print(), it will return correct schema:
NodeName: String
Module: String
Type: String
Package: String?
FilePath: String
Fields: *
    TypeType: String
    TypeKey: String
    Modifiers: List<String>
    TypeValue: String?
    Annotations: *
        Name: String
        KeyValues: *
            Key: String
            Value: String

Implements: List<String>
Functions: *
    Name: String
    Package: String?
    ReturnType: String
    Parameters: *
        TypeValue: String
        TypeType: String
    FunctionCalls: *
        Package: String?
        NodeName: String?
        FunctionName: String
        Position:
            StartLine: Int
            StartLinePosition: Int
            StopLine: Int
            StopLinePosition: Int
        Parameters: *
            TypeValue: String
            TypeType: String
        Type: String?
    Position:
        StartLine: Int
        StartLinePosition: Int?
        StopLine: Int
        StopLinePosition: Int?
    LocalVariables: *
        TypeValue: String
        TypeType: String
    IsConstructor: Boolean?
    Annotations: *
        Name: String
        KeyValues: *
            Key: String
            Value: String

Imports: *
    Source: String
    AsName: String
Position:
    StartLine: Int?
    StopLine: Int?
    StartLinePosition: Int?
    StopLinePosition: Int?
Annotations: *
    Name: String

But, in dataFrame.columns().toArrowSchema() the type will be error:
{
  "fields" : [ {
    "name" : "NodeName",
    "nullable" : false,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  }, {
    "name" : "Module",
    "nullable" : false,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  }, {
    "name" : "Type",
    "nullable" : false,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  }, {
    "name" : "Package",
    "nullable" : true,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  }, {
    "name" : "FilePath",
    "nullable" : false,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  }, {
    "name" : "Fields",
    "nullable" : true,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  }, {
    "name" : "Implements",
    "nullable" : true,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  }, {
    "name" : "Functions",
    "nullable" : true,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  }, {
    "name" : "Imports",
    "nullable" : true,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  }, {
    "name" : "Position",
    "nullable" : true,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  }, {
    "name" : "Annotations",
    "nullable" : true,
    "type" : {
      "name" : "utf8"
    },
    "children" : [ ]
  } ]
}

I lost something?

No, you're right - nested typed are not yet supported. :( Interesting data you've got there

Thanks for share it. Any plan on it? or I just try to modifiy AnyCol.toArrowField to implementation it ?

Honestly, i overlooked that our Arrow support misses nested types, so this improvement isn't planned. Right now the team is occupied with improvements to the documentation and notebooks experience. I think nobody is going to work on Arrow in near weeks.
You can submit a PR if you want, but apart from toArrowField there will be modification in actual writing here: infillVector https://github.com/Kotlin/dataframe/blob/master/dataframe-arrow/src/main/kotlin/org/jetbrains/kotlinx/dataframe/io/ArrowWriterImpl.kt

Thank you, I will try to find a solution.

IndexOutOfBoundsException: index: 31393, length: 2320 (expected: range(0, 32768)) is unexpected error, I am working on this (just got same in my project). This is because VariableWidthVector (where String column is saved to) does not know it's actual size.
About nested types, @phodal, do you have any examples in other Java-based projects with Arrow support as an example? And what is your target Arrow schema (does it contain SructVector, ListVector or any other)?

Exception is fixed in #350
Nested types are still not supported natively, should be saved correctly as strings


corert for RHEL/Centos

Hello,
Is it possible to build native executables for RHEL/Centos? or only Ubuntu is supported?

We have not gotton to building CoreRT for RHEL or Centos yet; however it should be pretty straightforward to fix things up to make it work. @cocotr Would you like to give it a try?
The one obstacle where you would need our help is nuget package publishing. @ellismg Do we have "build from source" for .NET Core documented already? I am wondering whether we can have corert branch of it that would allow folks to build corert for platform that we do not have yet, without relying on nuget packages.

Definitely I would like to give it a try.
btw. Do you think will it be possible to compile to native on Windows to make it work on RHEL/Centos (the same as for standard .net core app) ? or compilation needs to be done on RHEL/ Centos?

@cocotr I just quickly tried to build CoreRT on CentOS 7. I've found that besides the need to fix the managed building, there is one problem in the native code build. The fix is to replace #include <inttypes.h> in src/Native/libunwind/src/DwarfParser.hpp by #include <cinttypes>.

or compilation needs to be done on RHEL/ Centos?

I would do it on RHEL/Centos natively first. Cross-compilation from Windows to Unix is harder orthogonal problem.

what are prerequisites for building CoreRT on Centos 7 ? Similar as for Ubuntu ? Do you have any documentation?

In my CentOS 7 box, I tried the following:
# add some deps

yum install libunwind libunwind-devel cmake
yum groupinstall "Development Tools"

# then fetch the clang3.9.1
# source: https://copr.fedorainfracloud.org/coprs/alonid/llvm-3.9.1/

yum install yum-utils
yum-config-manager --add-repo https://copr.fedorainfracloud.org/coprs/alonid/llvm-3.9.1/repo/epel-7/alonid-llvm-3.9.1-epel-7.repo

yum install llvm-3.9.1 clang-3.9.1 --enablerepo=alonid-llvm-3.9.1

# clone repo and build
git clone https://github.com/dotnet/corert && cd $_
PATH=$PATH:/opt/llvm-3.9.1/bin ./build.sh

After 100% of progress, I got these errors:
[ 96%] Building ASM object Runtime/Full/CMakeFiles/Runtime.dir/__/amd64/CallingConventionConverterHelpers.S.o
[ 97%] [ 98%] Building ASM object Runtime/Full/CMakeFiles/Runtime.dir/__/amd64/ExceptionHandling.S.o
Building ASM object Runtime/Full/CMakeFiles/Runtime.dir/__/amd64/Interlocked.S.o
[ 98%] Building ASM object Runtime/Full/CMakeFiles/Runtime.dir/__/amd64/PInvoke.S.o
[ 99%] [100%] Building ASM object Runtime/Full/CMakeFiles/Runtime.dir/__/amd64/StubDispatch.S.o
Building ASM object Runtime/Full/CMakeFiles/Runtime.dir/__/amd64/UniversalTransition.S.o
[100%] Building ASM object Runtime/Full/CMakeFiles/Runtime.dir/__/amd64/WriteBarriers.S.o
In file included from /root/corert/src/Native/libunwind/src/libunwind.cpp:27:
In file included from /root/corert/src/Native/libunwind/src/UnwindCursor.hpp:30:
In file included from /root/corert/src/Native/libunwind/src/AddressSpace.hpp:75:
In file included from /root/corert/src/Native/libunwind/src/EHHeaderParser.hpp:19:
/root/corert/src/Native/libunwind/src/DwarfParser.hpp:372:59: error: expected ')'
    fprintf(stderr, "parseInstructions(instructions=0x%0" PRIx64 ")\n",
                                                          ^
/root/corert/src/Native/libunwind/src/DwarfParser.hpp:372:12: note: to match this '('
    fprintf(stderr, "parseInstructions(instructions=0x%0" PRIx64 ")\n",
           ^
/root/corert/src/Native/libunwind/src/DwarfParser.hpp:402:61: error: expected ')'
        fprintf(stderr, "DW_CFA_advance_loc1: new offset=%" PRIu64 "\n",
                                                            ^
/root/corert/src/Native/libunwind/src/DwarfParser.hpp:402:16: note: to match this '('
        fprintf(stderr, "DW_CFA_advance_loc1: new offset=%" PRIu64 "\n",
               ^

[snip]

fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make[2]: *** [Runtime/Full/CMakeFiles/Runtime.dir/__/__/libunwind/src/libunwind.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [Runtime/Full/CMakeFiles/Runtime.dir/all] Error 2
make: *** [all] Error 2
Failed to build corert native components.

Then I applied this patch:
--- a/src/Native/libunwind/src/DwarfParser.hpp
+++ b/src/Native/libunwind/src/DwarfParser.hpp
@@ -12,7 +12,7 @@

 #ifndef __DWARF_PARSER_HPP__
 #define __DWARF_PARSER_HPP__
+#define __STDC_FORMAT_MACROS

 #include <inttypes.h>
 #include <stdint.h>
 #include <stdio.h>

to get it fixed.
Now onto the managed / CLI story..

Similar to https://github.com/dotnet/corert/issues/2856#issuecomment-315531778, current master (bad4b3e) builds on CentOS with the following Dockerfile (without tests):
FROM centos:7
# it should pick the latest in 7 (atm 7.3.1611)
RUN cat /etc/*-release

RUN yum -y -q  install epel-release
RUN yum-config-manager --add-repo https://copr.fedorainfracloud.org/coprs/alonid/llvm-3.9.1/repo/epel-7/alonid-llvm-3.9.1-epel-7.repo
RUN yum -y -q update
RUN yum -y -q install \
      git cmake icu libicu-devel  binutils make \
      lttng-ust liblttng-ust-devel libuuid-devel libunwind libunwind-devel \
      llvm-3.9.1 clang-3.9.1 lldb-3.9.1 --enablerepo=alonid-llvm-3.9.1

ENV PATH /opt/llvm-3.9.1/bin:$PATH

RUN git clone https://github.com/dotnet/corert -b master --single-branch corert \
    && cd $_ \
    && ./build.sh skiptests

With tests, ilc gives the same errors on CentOS as on openSUSE (see the linked issue).


Implements PatternOptimization

Codecov Report
Attention: 33 lines in your changes are missing coverage. Please review.

Comparison is base (02cabdf) 69.99% compared to head (51d531a) 69.78%.
Report is 1 commits behind head on main.

Files
Patch %
Lines

...mental_experiment/torch_exp/graph_builder_optim.py
82.57%
23 Missing :warning:

experimental_experiment/torch_exp/_onnx_helper.py
50.00%
5 Missing :warning:

...ntal_experiment/torch_exp/optimization_patterns.py
93.75%
3 Missing :warning:

experimental_experiment/torch_exp/graph_builder.py
96.77%
2 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #35      +/-   ##
==========================================
- Coverage   69.99%   69.78%   -0.21%
==========================================
  Files          32       36       +4
  Lines        4273     4677     +404
==========================================
+ Hits         2991     3264     +273
- Misses       1282     1413     +131

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


CLA sign is required only for Mallikarjuna54

URL to the linked Repo/Org:  https://github.com/rdkcentral/RDKSplashScreen/pull/7
Steps to reproduce the problem:

What is the expected result? "Mallikarjuna54"user id enough to CLA signed. Other id please remove.
What happens instead? here 2 user id's are registered instead of one.

Hi @mallikarjuna54,
Unfortunately, all the users involved in the pull request should sign the CLA.
Also please note. It looks like this commit "BD-909:Relicense RDKSplashScreen to Metrological Apache" from Mallikarjuna Masina is not recognised by GitHub because you need to add the email address which you used for this commit in your local git configuration.

Hi @mallikarjuna54,
Unfortunately, all the users involved in a pull request should sign the CLA.
Also please note. It looks like this commit "BD-909:Relicense RDKSplashScreen to Metrological Apache" from Mallikarjuna Masina is not recognised by GitHub because you need to add the email address which you used for this commit in your local git configuration.

HI ibakshay ,
Mallikarjuna54 as signed already in local git as well as Registered in git hub. So in Pull request please remove extra user (Mallikarjuna Masina) so that my commit will ready to merge.

So in Pull request please remove extra user (Mallikarjuna Masina) so that my commit will ready to merge.
You have to reach out to the project admin for this query.

Note: CLA assistant tool expects all the contributors in a Pull Request to sign the CLA.

Hi @mallikarjuna54,
seems like the problem was resolved. I'm going a head and close this issue. If you have further questions/issues please feel free to open a new issue. Many thanks.


Enums are ignored during the validation

Repro:
require 'bundler/inline'

gemfile do
  gem 'dry-types', '=1.0.0'
  gem 'dry-validation', '=1.0.0.rc1'
  gem 'minitest', '=5.11.3'
end

require 'dry-validation'
require 'minitest/autorun'

module T
  include Dry.Types()

  BlogType = String.enum('pending', 'active')
end

class BlogValidator < Dry::Validation::Contract
  params do
    required(:type).value(T::BlogType)
  end
end

class EnumValidations < Minitest::Test
  def test_invalid
    assert_equal([:type], errors('wrong').keys)
  end

  def test_valid
    assert_equal({}, errors('active'))
    assert_equal({}, errors('pending'))
  end

  private

  def errors(type)
    BlogValidator.new.call(type: type).errors.to_h
  end
end

Produced the output:
Run options: --seed 45904

# Running:

F.

Finished in 0.001827s, 1094.6909 runs/s, 1642.0364 assertions/s.

  1) Failure:
EnumValidations#test_invalid [-:27]:
Expected: [:type]
  Actual: []

2 runs, 3 assertions, 1 failures, 0 errors, 0 skips

My expectation was that no values other than specified in the enum were allowed.
Alternatively, if enum is not supported - there should be an error as the current behaviour is somewhat dangerous (that is provided I use it correctly)

Constrained types are not yet supported. I'll try to do it before 1.0.0, worst case it'll go into 1.1.0.

Any workarounds you can think of for now?

Other than to define the same rules as the enum has, no :/

Actually, this is a duplicate of #30


bazel: reorder PATH on macOS

Reorder PATH macOS so that package managers binaries take precedence
over system binaries.
Signed-off-by: Sergii Tkachenko<EMAIL_ADDRESS>
Commit Message: bazel: reorder PATH on macOS
Additional Description: Reorder PATH macOS so that package managers binaries take precedence over system binaries.

/opt/homebrew/bin: Homebrew
/opt/local/bin: commonly used by macports
/usr/local/bin:/usr/bin:/bin: user tools -> common tools -> fundamental system tools

Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #20750]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]
P.S. This bit me hard, I couldn't understand why bazel run was using different python version than bazel build genrule cmd.
ref #17467
cc @keith

Note: this also solved a problem with external/base_pip3_sphinx/BUILD.bazel complaining about missing @base_pip3_importlib_metadata rule.

cc @RyanTheOptimist

Until this is merged, the sphinx problem can be worked around by creating user.bazelrc with the following:
build:macos --action_env=PATH=/opt/homebrew/bin:/opt/local/bin:/usr/local/bin:/usr/bin:/bin
build:macos --host_action_env=PATH=/opt/homebrew/bin:/opt/local/bin:/usr/local/bin:/usr/bin:/bin

Make sure the first path with python3 binary points to python3.10. You can verify this with a target like so:
genrule(
    name = "which_python3",
    outs = ["which_python3.txt"],
    cmd = "which python3"
)

Happy to help, @phlax!

The general purpose here was actually to prefer the system one to the package manager one, since the system one is probably more likely not be broken (we've seen people do crazy things to their homebrew directories). But ideally in that case the order would be used everywhere so you wouldn't be swapping back and forth between them still.

ah apologies - i should have waited - do you want to revert @keith
im wondering if we can prefer just the python dirs - ideally this will be fixed with the hermetic toolchains - but tbh it would be great if some mac users could test that out for local builds before we land it - windows is already breaking

I think this should be fine in the common case, I imagine folks will be more likely to have differing versions in homebrew than they would installed via xcode, just since xcode updates less frequently, but we can roll with this until we hit issues, or ideally replace it with your toolchains change

@keith I understand your concern. However, we already have the case where python 3.10 is implicitly required (My builds were broken because typecase expected __required_fields__, which was added in 3.9), but the default one provided by the system is 3.8.  We already do recommend using brew to install dependencies though - so it's on user to have their setup in order. That said, we probably should add python 3.10 to that brew install list.

@sergiitk we are just chatting about this very issue on slack channel - please do jump on - another related issue is different HOMEBREW install paths

True, but we can't control crazy things people do. We already do recommend using brew to install dependencies though - so it's on user to have their setup in order.

Yea I wasn't suggesting we try to not recommend using brew ever, but it's less likely that they could do something unexpected with /usr/bin/python3 than with the homebrew version, which just translates to issues. But either way until we can move to actual toolchains I think this is fine


Use Homebrew curl for audit when specified in the formula

[ ] Have you followed the guidelines in our Contributing document?
[ ] Have you checked to ensure there aren't other open Pull Requests for the same change?
[ ] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.
[ ] Have you successfully run brew style with your changes locally?
[ ] Have you successfully run brew typecheck with your changes locally?
[ ] Have you successfully run brew tests with your changes locally?

Review period will end on 2021-07-27 at 10:40:24 UTC.

Please note that this fixes only the audit, not the fetching.
For example you still get:
==> Downloading https://downloads.xiph.org/releases/flac/flac-1.3.3.tar.xz
##O=#  #
curl: (35) error:1400442E:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert protocol version
Trying a mirror...
==> Downloading https://ftp.osuosl.org/pub/xiph/releases/flac/flac-1.3.3.tar.xz

I would propose to fix the fetching in a second pull request (because actually I am struggling to add the brewed curl usage in the curl download strategy, based on the formula's fetch dependencies. I think this can thus be done in a second pull request to not pollute the discussion here.

Review period ended.

I'm all done, except for the logic in expand_deps and expand_reqs. I won't have time until later today to check this, but that part needs more thoughts as the boolean logic in there is quite complex. This might warrant a few tests to be sure we do not break anything.

I think I am happy with the changes as-is. I need a new review.
The initial code in expand_reqs and expand_deps is slightly different, as expand_deps has an extra !dep.build?. it might be on purpose, or it's missing from expand_reqs. I did not investigate that difference further and added the req/dep.fetch? condition with the same logic as req/dep.test?.

If there are no objections, I'll merge this tomorrow.

I renamed to dependency type from :fetch to :fetch_and_audit to make clear that this is used for both. The fetch part is not implement in this PR and will be in a second PR. If we want we could also split the dependencies and have a separate :fetch and :audit type, but I don't think this is needed.

I introduced an --include-fetch-and-audit flag, to be able to write the logic in formula_installer.

I only implemented the logic for dependencies and ignored the requirements part of the code, as we do not have plans to use fetch/audit requirements for the time being.

Right now the curl dependency is only installed when brew install --include-fetch-and-audit <formula> is used.
We need to change:

brew audit to install the fetch and audit dependency
brew fetch to install the fetch and audit dependency (in a second PR maybe?)
brew install from source to install the fetch and audit dependency

Thoughts? This is getting slightly more complex than expected and I need more help.

Thanks for your work here Michka. I'll take a look at some of the remaining tasks this week, it might be better to create separate PRs for them.

Thoughts? This is getting slightly more complex than expected and I need more help.

One thing to add, if I'm understanding this right, is that we would also need to uninstall Homebrew curl (if it's a fetch/audit dependency) after it has been used – once for the fetch and once for the audit, because we're trying to prevent unnecessary linkage with Homebrew curl.
Is this thinking correct? If so, I wonder if we should handle that part in test-bot or here. Sorry for the slightly off-topic comment but I thought I'd get it cleared here (or in the issue) before going ahead.

One thing to add, if I'm understanding this right, is that we would also need to uninstall Homebrew curl (if it's a fetch/audit dependency) after it has been used – once for the fetch and once for the audit, because we're trying to prevent unnecessary linkage with Homebrew curl.

Yes, this is what this implementation will be doing: I need to modify both brew audit and brew fetch to install brew curl, and then uninstall it after it is done (unless it was already installed previously, so we need to keep track of the state of the system). So we will install/uninstall curl twice.

Another solution would be to build a portable curl executable, that we could ship to the macOS and Linux CI, and always use that. A little bit like portable ruby.
We did something similar (but not exactly as it is inside a docker image) in the Debian 7 Linux docker image, were we put our own pre-compiled git version in the PATH (see https://github.com/Homebrew/homebrew-linux-dev/blob/master/Dockerfile), because git is too on old on that docker image.

This has been suggested previously: https://github.com/Homebrew/install/issues/554#issuecomment-885519531

This has been suggested previously: Homebrew/install#554 (comment)

Ha yes. @MikeMcQuaid said our version would be hard to keep up to date: I think we would be doing better than the outdated version of Ubuntu 16.04 (and 18.04) soon. We could even auto build a portable curl on each new curl release, using github actions.

I implemented the automatic installation of the fetch_and_audit dependents in fetch.rb and audit.rb.

The fetch and audit dependencies are now automatically installed when building a formula from source.

Everything is now implemented except the last step: the CurlDownLoadStrategy class needs to know about the fetch and audit dependency, so I need to pass the flag to it and implement the logic.

This has been suggested previously: Homebrew/install#554 (comment)

I would very strongly like to avoid us maintaining a portable curl.

I made a new proposal for the brew fetchpart, please have a look at the change in curl.rb. If the result code is 35 and brewed curl is installed, retry the download with brewed curl.
brew fetch will still be responsible to install brewed curl, but we use a sort of "indirect" way to hijack the fix into curl.rb, without having to rewrite the whole url/resource/download strategy stack.

I think we also want to review the cc shim and superenv to double check that it's not adding compiler flags or modifying environment variables (e.g. PATH) for this new dependency type. I don't think it will, but I think it'd be good to make sure to avoid opportunistic linkage.
Feel free to poke me in the next few days about this if you need me to have a look.

Actually I could remove all the use_brewed_curl arguments from formula_auditor.rb now, as I could use the same logic for audit and fetch: the command will install/uninstall brewed curl, and curl_with_workaround will just handle the issue by itself, as implemented now. What do you think?

I made a new proposal for the brew fetchpart, please have a look at the change in curl.rb. If the result code is 35 and brewed curl is installed, retry the download with brewed curl.

We had a previous PR for this and: I don't like it, I'm afraid. This is something that should be flagged in individual formulae rather than have magic that reruns curl transparently.

I re-implemented the change with the :using strategy. Not everything is done but you'll see the main idea.
Right now I am struggling to automatically install brewed curl with these lines: https://github.com/Homebrew/brew/pull/11773/files#diff-95bfed356ec6790f13d81d690e41576edf8b9ad6475031f93e74237cbdadb86aR146-R149
To test this, I changed the hello formula:
url "https://ftp.gnu.org/gnu/hello/hello-2.10.tar.gz", using: :brewed_curl
And then brew fetch -s hello should automatically install brewed curl (make sure you brew uninstall curl before).
It does not install curl, which is not what I expected.

I'm quite happy with the change now.
The only remaining thing is the automatic installation of brewed curl: I removed the implementation I initially made, and thought https://github.com/Homebrew/brew/pull/11773/files#diff-95bfed356ec6790f13d81d690e41576edf8b9ad6475031f93e74237cbdadb86aR66 would do it. But it does not seem to install curl when doing brew fetch -s <formula>. I also tried brew fetch blastem --head, and fetching failed because mercurial was not installed: still, Dependency.new("mercurial", tags) is added in the download strategy selector, so I am wondering what this does? The install mechanism for these dependencies is maybe broken?

I removed the implementation I initially made, and thought https://github.com/Homebrew/brew/pull/11773/files#diff-95bfed356ec6790f13d81d690e41576edf8b9ad6475031f93e74237cbdadb86aR66 would do it.

Does this work for e.g. Subversion etc.? If not, I think it may be fine to leave this without auto-install. If it does, that may be a good place to debug this.

The install mechanism for these dependencies is maybe broken?

Yes, perhaps. On macOS or Linux?

@iMichka also: it may be worth comparing brew install to brew fetch here?

Does this work for e.g. Subversion etc.? If not, I think it may be fine to leave this without auto-install. If it does, that may be a good place to debug this.

Yes, perhaps. On macOS or Linux?

On both. We don't have that many svn cases, but you can try brew install abcl --head: it will fail with You must: brew install svn, though the svn dependency has been added. Looks like the return value of these new dependencies is never used.
Not a big deal though right now: my code will fail with Homebrew-installed curl is not installed for <url> locally and you'll have to install brewed curl manually. And we we will let test-bot install/uninstall brewed curl when necessary (I'll open a second PR for that later).

On both. We don't have that many svn cases, but you can try brew install abcl --head: it will fail with You must: brew install svn, though the svn dependency has been added. Looks like the return value of these new dependencies is never used.

Interesting. Might be worth an extension (in a new PR) to try and do this automatically. I can take a look at this tomorrow, if requested.

Thanks, merging. I will start working on the test-bot change and test this change in homebrew-core.


Conversation tests using yaml

Proposed changes:

always ensure UserUttered events contain the full parse_data
renamed occurrences of conversation_tests.md to test_conversations.yml (as well as remaining nlu.md and story.md ones)
added default tests to rasa init in yaml format
removed the need to pass an interpreter to story readers (wasn't actually needed, e2e evaluation is done slightly different instead)
some code style improvements here and there 😅

Status (please check what you already did):

[ ] added some tests for the functionality
[x] updated the documentation
[ ] updated the changelog (please check changelog for instructions)
[x] reformat files using black (please check Readme for instructions)

@erohmensing I've updated the documentation to reflect the changes to the story testing parts. Also updated the chitchat-faq's page along the way. (and some other smaller things I noticed in the docs). do you have time to review the docs changes?

@degiz this is ready for a code review, still need to add some tests but I think giving the code a review first helps speed this up.

Oops, I didn't see that you just wanted my review on docs changes, sorry 😅

No worries, always also happy about any code comments 🙂

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: tmbo:x: deepsource-autofix[bot]You have signed the CLA already but the status is still pending? Let us recheck it.

good point about the naming, I'll make sure it is test_stories everywhere instead of test conversations. I'll keep some occurrences in the docs page though for discoverability

@degiz addressed all the comments - still need an accept though 😉


Make compatible with fman 0.3.6

StatusBarExtended relies on fman setting Panes.json to a dictionary whose keys are DirectoryPane#id.
fman 0.3.6 persists the "show hidden files" setting across restarts. To implement this, Panes.json had to be changed from a dictionary to a list. Further, DirectoryPane#id has been removed from fman's API.
Both of the above changes break the current version of StatusBarExtended. This PR fixes it again.

Thanks!


Upcoming alarm tab is blank

Issue

upcoming is blank even after adding 5 courses

Possible explanation

It's taking too much time to render list item(In my case it is 40 items)

@hirenchalodiya1  @manav1403 made changes to the API call. Should work fine now.

addressed in #33

https://github.com/devlup-labs/meet-scheduler/blob/6bf9318766faaeac29f4ced9207c2b1158f7ce63/src/pages/Popup/scripts/fetchSheet.js
Solve something like this

You may face failed to execute 'json' on 'response' body stream already read issue

@DarshitJain04 @manav1403 look into this


Allow Flyway Edition Specification in Config File

We are in the process of upgrading from Community to Pro edition.  It would be nice if this could be specified within the configuration file instead of on every command.

There's a technical limitation at the moment which would make loading the edition from the config file challenging (basically, by the time it gets to that step, it has already loaded the relevant libraries).
In addition to using the command-line flag, you can set this information using the FLYWAY_EDITION environment variable. Would that work for you?

@alextercete either can work, but it would be much easier to do through the config file since it's so easily shared.  Hopefully this can be added some time in the future.

The technical limitations are round this are significant. We don't think it's valuable enough to pursue.


Ordinal-aware wallet derivation paths

Strongly suggest that ordinal-aware HD wallets use different derivation paths for all outputs, and to never mix ordinal-aware outputs and transactions with non-ordinal aware outputs and transactions.
Perhaps even come up with and suggest the use of a specific distinct derivation path.

Seems like you did not go ahead with a distinct derivation path? Any reasons for this? Using the default taproot derivation path is problematic for Bitcoin wallets that want to add support for ordinals due to mixing with regular outputs.

@yknl Can you describe your concern a little more fully? I think I understand what you might be getting at, but I don't want to guess. What is the way in which a different derivation path would help other bitcoin wallets?

An existing Bitcoin wallet might already be using the taproot derivation path for regular transactions. If they add support for ordinals, outputs would be mixed with regular payment outputs. Wallet balance will need to be calculated by excluding the ordinal outputs as well. You can have some kind of system to keep track of and separate the outputs, but that quickly becomes very complex. Using a separate derivation path just for ordinals makes it much easier for wallets. All your ordinal outputs are on a different path or "separate wallet" essentially, no special logic required to manage the wallet.
It would be great to standardize on a new derivation path so that more Bitcoin wallets like what we're building can add support.

Gotcha, that makes total sense. I think this would be a great idea. First step is to figure out what the alternative derivation path should be.

Since we're still dealing with Bitcoin. We probably want to use a different "purpose" here.
m / purpose' / coin_type' / account' / change / address_index
BIP44 for reference
It seems like all the "purposes" currently used are the BIP numbers in which it was proposed. If that's the case, it might be difficult get one. A BIP might be rejected depending on how Bitcoin core devs feel about ordinals.
Otherwise a unique coin type can be used, perhaps coin type 100000000 for the 100M sats in each bitcoin?
m/86'/100000000'/0'/0/0
Still available according to SLIP44:

Nice, purpose' seems like the right thing. I'm going to mark this as blocked until the BIP is merged or rejected.

I suggest to use purpose 1408' as it is the number of the PR to the bip repo.

This is waiting on a BIP number to come up with a derivation path.

i sent ordinals sparrow wallet to a xverse ordinal address. the transaction confirmed the ordinals where in the wallet. i then uninstalled xverse ,later i reinstalled xverse and my address changed.and now my ordinals are stuck on my old address.can it be recovered? this is Sparrows address bc1p5302u8kamv0jltap324unpw3pmle2lu95px6rlx34yu9rcfh629sn7cjhu
and this is Xverse address bc1p8vdtr2f9zakzyxu43pdlutd56leaekmvn46yf2uf5gv28k06f74sggtfp2
this is the Transaction ID b7c5f4e23a833158280479b7d5ebf71ba23b202532ada4f990f9d3c89b53f23c

I think actually this is probably a bad idea. Adding an additional derivation path is just a recipe for confusion, and makes it hard to use non-ordinals tools with ordinal wallets.


Incompatibility with Play 2.5

Play 2.5 has changed the way they make REST calls, and it has broken play-s3 compatibility. Here is the error I get:
java.lang.ClassNotFoundException: com.ning.http.client.Response
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at fly.play.s3.Bucket.fly$play$s3$Bucket$$extractHeaders(Bucket.scala:226)
at fly.play.s3.Bucket$$anonfun$get$1.apply(Bucket.scala:67)
at fly.play.s3.Bucket$$anonfun$get$1.apply(Bucket.scala:66)
at fly.play.aws.AwsResponse$.apply(AwsResponse.scala:8)
Please let me know if you intend to continue support for this library as Play evolves. They make a lot of changes even with minor versions, so I know it can be a lot to maintain.
Thanks.

Yes, we plan to support it. Not sure when we will have modified the code to reflect the new changes.
The main body of work is with the streams: https://www.playframework.com/documentation/2.5.x/StreamsMigration25
And the other stuff seems to be just renaming.

@Hayena has been working on this. He however had another project to focus his attention on. @Hayena , will you have time to work on this in the next week or so?

I will see if I have time tonight to finish this otherwise I'll do it this weekend

Apart from a single failing test it seems the 2.5 branch is functioning (see https://github.com/Kaliber/play-s3/pull/62 for details). As soon as I have some more time I will release it.

Release 8.0.0 supports Play 2.5

In order to initialize an S3 instance with S3(bucketName) in the README and tests, you import play.api.Play.current, which is a static reference to Application. This is deprecated.
As described in the documentation, the tests should reflect how to use dependency injection to provide the Application. Or possibly take a different approach entirely. Either way, the tests and documentation should reflect the preferred approach.

Very good point, I totally forgot about that part of the library. As soon as I have a chance, I will fix it. Here we only use the compile time dependency injection approach in combination with the following signature:
def fromConfiguration(client: WSClient, configuration: Configuration)

We should probably add some annotations that help with the Guice version. I however have no direct experience with that. I might add another ticket and accept pull requests on that one.
In any case, as soon as I have time I will update the documentation and tests. Any help with that is of course appreciated.

People using Guice can inject the Application and put it into implicit scope, and everything works great. Injecting the Application, however is considered to be a big no no, since it creates circular dependencies (Application depends on the controllers, which depend on everything else, so having something depend on Application completes the circle).
A quick fix could be to simply deprecate the def apply(...)(implicit app: Application) methods, and replace them with def apply(...)(implicit client: WSClient, configuration: Configuration) methods instead.

@GitsMcGee agreed. As you might have noticed, my time has been very limited the last few months. If you feel this is very important please create a pull request.

@EECOLOR Sounds good. I wouldn't say it's hugely important, but it would certainly be nice to have. I'll make a pull request as soon as I have the time.

Just got this with play-2.5 and play-s3-8.0.0:
Tried proxying play.api.Application to support a circular dependency, but circular proxies are disabled

Replacing implicit Application with implicit client: WSClient, configuration: Configuration didn't help.

@thehappycoder I don't see how this library can cause that problem. Could you maybe create a minimal example that reproduces the problem?

@thehappycoder oh and feel free to create a new issue

@EECOLOR All good. I had a mistake in code.

@EECOLOR  I tried to upload a file in S3 bucket using this library, but I see some strange error as below. I searched a bit about the originalUri in the error but could not find any clues.
I am using play 2.5, and using the following snippet to get the bucket.
lazy val s3 = S3.fromConfiguration(ws, configuration)
lazy val bucket = s3.getBucket(configuration.getString("s3.bucket").get)
play.api.http.HttpErrorHandlerExceptions$$anon$1: Execution exception[[NullPointerException: orginalUri]]
at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:280)
at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:206)
at play.core.server.netty.PlayRequestHandler$$anonfun$2$$anonfun$apply$1.applyOrElse(PlayRequestHandler.scala:100)
at play.core.server.netty.PlayRequestHandler$$anonfun$2$$anonfun$apply$1.applyOrElse(PlayRequestHandler.scala:99)
at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:346)
at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:345)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at play.api.libs.iteratee.Execution$trampoline$.execute(Execution.scala:70)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
Caused by: java.lang.NullPointerException: orginalUri
at org.asynchttpclient.util.Assertions.assertNotNull(Assertions.java:23)
at org.asynchttpclient.uri.UriParser.parse(UriParser.java:320)
at org.asynchttpclient.uri.Uri.create(Uri.java:36)
at org.asynchttpclient.netty.handler.intercept.Redirect30xInterceptor.exitAfterHandlingRedirect(Redirect30xInterceptor.java:123)
at org.asynchttpclient.netty.handler.intercept.Interceptors.exitAfterIntercept(Interceptors.java:86)
at org.asynchttpclient.netty.handler.HttpHandler.handleHttpResponse(HttpHandler.java:130)
at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:186)
at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:82)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278)
Please let me know if there is anyway to tackle this issue. Any help would be greatly appreciated.
Thanks

@ceecer1 Make sure you have region defined in s3.region configuration prop and make sure you define bucket name. It happened to me in both of these cases.

@thehappycoder Thanks it worked, my bucket didn't require a s3 region.

@thehappycoder I have the same issue. But I did set s3.region. Where do I define the bucket name except according to the readme: val bucket = S3("mybucket")?

@git4sun can you post the error (including the stacktrace)?

[debug] o.a.n.c.ChannelManager - Closing Channel [id: 0xa7f09993, L:/192.168.4.xx:55153 - R:s3-us-west-2.amazonaws.com/54.231.185.xx:443] [debug] i.n.h.l.LoggingHandler - [id: 0xa7f09993, L:/192.168.4.xx:55153 - R:s3-us-west-2.amazonaws.com/54.231.185.xx:443] CLOSE() [debug] o.a.n.r.NettyRequestSender - Aborting Future NettyResponseFuture{currentRetry=0, isDone=false, isCancelled=false, asyncHandler=play.api.libs.ws.ahc.AhcWSRequest$$anon$1@1d3f666c, nettyRequest=org.asynchttpclient.netty.request.NettyRequest@4a7e4d5d, content=null, uri=https://s3-us-west-2.amazonaws.com/mybucket/requestBody7117625290787684978asTemporaryFile, keepAlive=false, exEx=null, redirectCount=1, timeoutsHolder=org.asynchttpclient.netty.timeout.TimeoutsHolder@f298834, inAuth=false, statusReceived=false, touch=1473749356653} [debug] o.a.n.r.NettyRequestSender - orginalUri java.lang.NullPointerException: orginalUri at org.asynchttpclient.util.Assertions.assertNotNull(Assertions.java:23) at org.asynchttpclient.uri.UriParser.parse(UriParser.java:320) at org.asynchttpclient.uri.Uri.create(Uri.java:36) at org.asynchttpclient.netty.handler.intercept.Redirect30xInterceptor.exitAfterHandlingRedirect(Redirect30xInterceptor.java:123) at org.asynchttpclient.netty.handler.intercept.Interceptors.exitAfterIntercept(Interceptors.java:86) at org.asynchttpclient.netty.handler.HttpHandler.handleHttpResponse(HttpHandler.java:130) at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:186) at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:82) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278) [debug] o.a.n.c.ChannelManager - Closing Channel [id: 0xa7f09993, L:/192.168.4.xx:55153 ! R:s3-us-west-2.amazonaws.com/54.231.185.xx:443] [debug] o.a.n.h.HttpHandler - Unexpected I/O exception on channel [id: 0xa7f09993, L:/<IP_ADDRESS>:55153 ! R:s3-us-west-2.amazonaws.com/<IP_ADDRESS>:443] java.lang.NullPointerException: orginalUri at org.asynchttpclient.util.Assertions.assertNotNull(Assertions.java:23) at org.asynchttpclient.uri.UriParser.parse(UriParser.java:320) at org.asynchttpclient.uri.Uri.create(Uri.java:36) at org.asynchttpclient.netty.handler.intercept.Redirect30xInterceptor.exitAfterHandlingRedirect(Redirect30xInterceptor.java:123) at org.asynchttpclient.netty.handler.intercept.Interceptors.exitAfterIntercept(Interceptors.java:86) at org.asynchttpclient.netty.handler.HttpHandler.handleHttpResponse(HttpHandler.java:130) at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:186) at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:82) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278) [debug] o.a.n.c.ChannelManager - Closing Channel [id: 0xa7f09993, L:/192.168.4.xx:55153 ! R:s3-us-west-2.amazonaws.com/54.231.185.xx:443] [debug] i.n.h.l.LoggingHandler - [id: 0xa7f09993, L:/192.168.4.xx:55153 ! R:s3-us-west-2.amazonaws.com/54.231.185.xx:443] INACTIVE [debug] o.a.n.h.HttpHandler - Channel Closed: [id: 0xa7f09993, L:/192.168.4.xx:55153 ! R:s3-us-west-2.amazonaws.com/54.231.185.xx:443] with attribute INSTANCE [debug] i.n.h.l.LoggingHandler - [id: 0xa7f09993, L:/192.168.4.xx:55153 ! R:s3-us-west-2.amazonaws.com/54.231.185.xx:443] UNREGISTERED [debug] o.a.n.c.DefaultChannelPool - Closed 0 connections out of 0 in 0ms

Does calling bucket.url("test", 1) give you the same error?

no. The error is caused by this line: val result = bucket + BucketFile(file.getName, request.headers.get("Content-Type").get, byteArray, None, None). More specifically, the + causes the error. It seems that the error happened only when actually communicating with S3 server.

What is the output of bucket.url("test", 1)

https://s3-us-west-2.amazonaws.com/mybucket/test?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Signature={somestring}&X-Amz-Date=20160913T094502Z&X-Amz-SignedHeaders=host&X-Amz-Expires=1&X-Amz-Credential={mykey}%2F20160913%2Fus-west-2%2Fs3%2Faws4_request"

@git4sun That url already shows the problem. Amazon is responding with a 301, but without a location header. The netty code does not expect this:
String location = responseHeaders.get(HttpHeaders.Names.LOCATION);
Uri newUri = Uri.create(future.getUri(), location);

And the parse function throws a very cryptic error: assertNotNull(originalUrl, "orginalUri");
The body of the response of the url you provided contains some clues:

The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

From what I can tell this means that the bucket was created in a region that is not us-west-2.

@EECOLOR thanks.


Encoder Ring Not Working After Flashing

I recently purchased an M5Dial.
Everything was working fine until I downloaded the firmware from GitHub, built it, and flashed it onto the device. After the flashing process, the encoder ring stopped responding. The click functionality is still operational, allowing access to the LCD TEST menu, but the rotation of the encoder does not trigger any response.
Could anyone provide guidance on whether there is a specific configuration or setting adjustment needed during the build process to get the encoder ring working again? Any assistance would be greatly appreciated.
Thank you.

The encoder works if you comment these lines in the main file.
I still don't know why, but it works.
    // if (!hal.encoder.btn.read())
    // {
    //     uint16_t time_count = 0;
    //     while (!hal.encoder.btn.read())
    //     {
    //         delay(10);
    //         time_count++;
    //         if ((time_count * 10) > 1000)
    //         {
    //             factory_test_mode = true;
    //             break;
    //         }
    //     }
    // }

Regards

Hi, it's a bug comes with encoder library, will be fixed in the future

It should be fix now

Thanks for fixing the issue. It works great now!


Errors when opening or accessing a Borehole

I'm new to pywellcad and tried the README.md sample code:
import wellcad.com

app = wellcad.com.Application()
sample_file_path = r"C:\Program Files\Advanced Logic Technology\WellCAD\Samples\Classic Sample.wcl"
borehole = app.open_borehole(sample_file_path)
gr_log = borehole.get_log("GR")
gr_log.file_export(r"C:\Temp", "sample_gr", "csv")

After entering the the borehole =  line, the document opens correctly in WellCAD.  However, I get the following errors:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\xander\AppData\Local\Programs\Python\Python39\lib\site-packages\wellcad\com\_application.py", line 104, in open_borehole
    return Borehole(self._dispatch.OpenBorehole(path))
  File "C:\Users\xander\AppData\Local\Programs\Python\Python39\lib\site-packages\wellcad\com\_dispatch_wrapper.py", line 53, in __init__
    self._dispatch._FlagAsMethod(method_name)
  File "C:\Users\xander\AppData\Local\Programs\Python\Python39\lib\site-packages\win32com\client\dynamic.py", line 543, in _FlagAsMethod
    details = build.MapEntry(self.__AttrToID__(name), (name,))
  File "C:\Users\xander\AppData\Local\Programs\Python\Python39\lib\site-packages\win32com\client\dynamic.py", line 551, in __AttrToID__
    return self._oleobj_.GetIDsOfNames(0, attr)
pywintypes.com_error: (-2147352570, 'Unknown name.', None, None)

Because of the error, no Borehole object is returned, which makes it difficult to further explore the capabilities of the API.
I see the same behavior on app.get_borehole() and app.get_active_borehole() calls.
I see the errors in both Python 3.9 and 3.10.  I'm running the Python interpreter in PowerShell.  WellCAD 5.6 build 1028.
Is there a workaround for this?

It looks like it was broken by commit 81ba790. If I had to guess, it was adding the extra dispatch methods in preparation for v5.7 WellCAD release.
@ArnaudCrl, can you please look into this? Let's handle the exceptions when we register attributes/methods in DispatchWrapper.


FIO Testcase addition to LISA

Generic FIO Test case addition to LISA

What is the purpose of this case?
By default, it will run testing against the os disk, if it is a perf testing, do we have baseline for it?


 Almost every day, it crashes for a different reason, and it's extremely frustrating. I even recently helped fix a bug (https://github.com/TheLastBen/fast-stable-diffusion/issues/2343#issuecomment-1642716471). I know it's not entirely your fault because I see the minor changes you make to Colab, but still, I'd like to understand why we constantly encounter these breaking changes. Is it because Google keeps updating its packages all the time? Or is it because automatic1111's project is constantly evolving and breaking everything? I just want to understand

The reason is because this repository does not follow common software practices ;)
I have forked this repository and made static versions myself where everything is static.
Mostly it gets broken through A1111 changes. This repository pulls always the latest changes from the main branch (at every run). Of course this will break things. There is no versioning within this repository either.
The runpod images do help though. But even there things are getting downloaded from some github repos.

Google colab is updating their backend and A1111 is updating every 15 days, even then, any instability lasts only a few hours, most of the issues are caused by the users themselves not using high-RAM runtimes.

Thank you for your responses, at least that way it's clearer

thank you for your answer


Create plugin-venafi-codesigning.yml and plugin-venafi-vcert.yml

Description
Adds entries for the following plugins:

venafi-codesigning-plugin (HOSTING-990)
venafi-vcert-plugin (HOSTING-991)

Github users: @FooBarWidget, @Avwsolutions
Submitter checklist for changing permissions
Always

[x] Add link to plugin/component Git repository in description above

For a newly hosted plugin only

[x] Add link to resolved HOSTING issue in description above

For a new permissions file only

[x] Make sure the file is created in permissions/ directory
[x] artifactId (pom.xml) is used for name (permissions YAML file).
[x] groupId / artifactId (pom.xml) are correctly represented in path (permissions YAML file)
[x] Check that the file is named plugin-${artifactId}.yml for plugins

When adding new uploaders (this includes newly created permissions files)

[x] Use the Jenkins community (LDAP) account name in the YAML file, not the GitHub account name
[x] Make sure to @mention the users being added so their GitHub account names are known if they require GitHub merge access (see below).
[x] All newly added users have logged in to Artifactory at least once

Reviewer checklist (not for requesters!)

[ ] Check this if newly added person also needs to be given merge permission to the GitHub repo (please @ the people/person with their GitHub username in this issue as well). If needed, it can be done using an IRC Bot command
[ ] Check that the $pluginId Developers team has Admin permissions while granting the access.
[ ] In the case of plugin adoption, ensure that the Jenkins Jira default assignee is either removed or changed to the new maintainer.
[ ] If security contacts are changed (this includes add/remove), ping the security officer (currently @daniel-beck) in this pull request. If an email contact is changed, wait for approval from the security officer.

There are IRC Bot commands for it

When can we expect this to be reviewed / merged ? Just curious about the progress

Please fill out the PR template, it is there for a reason. Following PR templates will generally get faster response times.

@slide The PR template says "only applies to permission changes". I thought, "I'm not changing existing permissions, I'm adding a new plugin" so I didn't use the template.
But I've filled in the template now.