pt-weighted centroid for azimuth

I think the $p_T$-weighted centroid for $\phi$ is not being calculated correctly. Because we operate in the complex $x-y$ plane, when we take the sum of these complex numbers, they are already weighted by $p_T$, ie. $p_T$ is the magnitude of the complex number. We multiply it by $p_T$ again, which would make it a $p_T^2$-weighted centroid. $\phi$ is just a directional component, which is why $p_T$-weighting needs to be done if working with it directly. Please correct.

Problem turns out to be deeper than expected. $p_T$-weighted azimuth centre is typically taken as
$$
\bar \phi = \dfrac{\sum_j p_{T j} \phi_j} { \sum_k p_{T k}}
$$
Obviously assuming the periodic space of $\phi$. But my method of calculation
$$
\bar \phi = \text{PHASE}\left(\sum_j x_j + i y_j\right)
$$
Is actually not mathematically equivalent. To be honest, my method feels much closer to the desired result.
One interesting case to consider is when two equal magnitude and anti-parallel vectors on the complex plane are combined. The standard method would result in an angle directly between them, but which? There are two valid choices, with a separation of $\pi$, and whichever is used will depend on numerical artefacts, not conscious choice. My method, however, should give an undefined value, which feels more honest.
It also feels as though the whole point of this $p_T$-weighting business in the first place is to more accurately capture the collective coordinates of the set, as though it were one. This is explicitly what my method does, whereas the standard method seems like a poor approximation. Very poor, in fact, since there is no need to approximate at all (unless for minor speedups). With the $\phi$ and $p_T$ data inputs, my method could be reproduced with:
$$
\bar \phi = \text{PHASE}\left(\sum_j p_{T j} \exp i\phi_j \right)
$$
Still, the functions claim to do different things to their implementation (and ofc the previous version didn't even do my own formula correctly). I think I will deprecate the functions, and write a new one which gives (eta, phi) midpoint values out, but by first summing the momenta of all the particles, and then simply accessing the respective properties of the result.


Update docs to reflect single quoted strings now used in Ruby snippets

This is the documentation update to accompany cucumber/cucumber-ruby#1298.

Boom!


Bug - Flight Spell not working

When I try to cast Flight Spell

it just consumes my media (I used charged crystals in this example for simplicity), makes some particles at my position, seemingly changes my FOV for a split second and does nothing in the finale
First noticed it in a modpack and then tried it with only Hex Casting and dependencies, with the same result

Will be fixed in 0.9.


bug(mat-table): Wrong row height for flex

Is this a regression?

[ ] Yes, this behavior used to work in the previous version

The previous version in which this bug was not present was
No response
Description
I'm just checking the new MDC guidline / changelog and was wondering why in the demo the new row height (52px) was missing for the flex table.
This is because the height is reset here https://github.com/angular/components/blob/main/src/material/table/_table-flex-styles.scss#L3 to (the old) 48px but if I'm not mistaken the value needs to be 52px (or gets picked from the original style)?
Btw. the same is for padding. The flex table overrides the expected default values from MDC.
Reproduction
Flex-Table of the example does not have the new height:
https://material.angular.io/components/table/examples#table-flex-basic
Expected Behavior
Using flex / html table should not make a difference in row height?
Actual Behavior
Flex / Html table differ in row height ...?
Environment

Angular: 15.0.3
CDK/Material: 15.0.2

This is intentional as per the comments in table.scss:

// Flex rows should not set a definite height, but instead stretch to the height of their
// children. Otherwise, the cells grow larger than the row and the layout breaks.
.mat-mdc-table mat-header-row.mat-mdc-header-row,
.mat-mdc-table mat-row.mat-mdc-row,
.mat-mdc-table mat-footer-row.mat-mdc-footer-cell {
  height: unset;
}


Reading local IP from config property breaks CF login when resurrected

The cf-registrar gets the local IP used to register at the router from its config. The IP is determined when the configuration template is rendered by erb.
This is a problem when the UAA gets resurrected by the BOSH Healthmonitor. Rendered configuration templates are cached in the BOSH director's blobstore. These template are used when resurrecting the VM, although it might get a different IP address. The UAA will then register with a wrong IP at the router and you cannot log into CF anymore.
All other CF components should not be affected as they don't use the cf-registrar.
As a solution, the cf-registrar could make sure to get the actual IP when started and not from a config property.

@cppforlife This might also be of interest for you? The director's caching mechanism for rendered templates has some funny side effects when properties are used that don't come the deployment manifest.

Closed due to bug created in BOSH backlog: #5


Build improvements

Please have a look at the individual commit messages for the details.

This change is

Merged despite the unrelated Travis failure.


[CRASH][v2-beta1] App crashes on opening

Your Rocket.Chat.Android version: 2.0.0-beta1
Mobile device model and OS version: Samsung S8 Android 8.0
As soon as start the app, the app crashes. But before the crash i can see the pop up error related to this issue and PR.
https://github.com/RocketChat/Rocket.Chat.Android/pull/999
https://github.com/RocketChat/Rocket.Chat.Android/issues/978

@mmarif4u Thank you for reporting the crash! We already saw what's happening and it's related to versions of the servers below 0.63 (in RC right now). We'll work today to release another build containing the fixes. 👍

Thanks @rafaelks, I believe 0.63 is not released yet. Maybe disable the features which requires that version for now. And use the currently available APIs of 0.62.2.
Waiting for the fix release. :+1:


use git providers

Seprated Git providers.
Implemented gitea and github providers.

Amazing, thanks! will review later today but very excited about this.
cc @greghaynes


Timing issue with Joint Trajectory Controller in Github CI with very slow simulations

Describe the bug
We are using Joint Trajectory controllers in one of our demos with robotic arms and have a system test to check the functionalities.
Everything works fine on local (Ubuntu 20.04 with Foxy or Rolling) but when the test is run on Github CI (also Ubuntu with Foxy or Rolling), 15% of the time it will fail like here, as command are sent before the joint trajectory is ready (guess from the line with [spawner.py-62] [INFO] [1636363807.141690639] [abb.spawner_abb_joint_trajectory_controller]: Configured and started abb_joint_trajectory_controller shown after the error).
Here is the controller used for the abb arm, using a custom class that should wait for the action server to be ready with the function wait_for_server().
Here you can find the launcher of the simulation focused on the abb arm.
My guess is that maybe the wait_for_server() function will return when the joint trajectory controller is initialized but not yet ready to receive commands. Locally the code will run fast enough for the controller to be ready to receive commands but on CI as our simulation run very slowly sometimes it will not have sufficient time. Would such a thing be possible ?
I can solve my problems by adding time.sleep() but imo it is not a good way to solve this.
Maybe is there another function that will check that the trajectory controller  is ready to receive the commands, but all I can find is to use the function wait_for_server() or to use time.sleep().
I did open an issue on rclpy repo in order to know if maybe my issue comes from the function wait_for_server().

Anyone has an idea about my issue?

The issue has been solved on our side by waiting a JointTrajectoryControllerState message on the /state topic, confirming that the JointTrajectoryController is ready to receive goals.

Nice! We can probably fix our flaky rostests the same way!
On Wed, 17 Nov 2021, 08:14 Benjamin Hug, @.***> wrote:

Closed #265 https://github.com/ros-controls/ros2_controllers/issues/265.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ros-controls/ros2_controllers/issues/265#event-5630069642,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA24PYP76QIKXSPKVQHNRPDUMNP75ANCNFSM5HUT2KIQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.


Update README.md - fix grammatrical errors in README.

I have noticed some grammatical mistakes in README inside section "How it works"
"Room name and join up"  should be "** Room name, and join up**"
"(Play, Pause, Seek etc)"  should be "(Play, Pause, Seek, etc)"
Screenshot-

Please review my PR.


Revert "update file perm used when creating cf config"

Reverts common-fate/cli#95 - the directory here should be 0700. Fixes the same issue fixed in https://github.com/common-fate/granted/pull/755

only change was permission @awsconfig/config.go (permission update)


FreeMatch SAF loss is a negative value?

Bug
In your FreeMatch paper equation 11, the SAF loss is a negative value. Then the SAF loss is added to the total loss. Isnt this causing SumNorm (p˜t/h˜t) and SumNorm(p¯/h¯) to be more dis-simmilar?

Reproduce the Bug

Error Messages and Logs

Negative loss doesn't affect the gradient

thank you for your fast reply!
I am just curious about the intuition behind this loss. By minimizing $L_{f}$ in eq11, aren't you pushing SumNorm (p˜t/h˜t) and SumNorm(p¯/h¯) to be further apart? I thought you want to make them closer?

thank you for your fast reply! I am just curious about the intuition behind this loss. By minimizing Lf in eq11, aren't you pushing SumNorm (p˜t/h˜t) and SumNorm(p¯/h¯) to be further apart? I thought you want to make them closer?

The loss encourages fairness on average predictions. We expect the average predictions to be close to uniform. Reflected on the entropy loss, it corresponds to maximizing the entropy. We replace the target term with a momentum-smoothed average prediction for stability. That's the intuition behind this loss. I think the paper and relevant works mentioned in the paper discussed this in more detail.

thank you for explaining it!


Request to update builder-base Golang to 1.17.8

What would you like to be added:
Builder base currently uses Golang 1.16.12. The AWS Controllers for K8s (ACK) project would love to use this image as our base for building our controllers, except we need at least 1.17.
Why is this needed:
ACK needs 1.17 because we rely on that version for our latest go modules. We currently leverage the EKS-D nonroot image for our runtime images, but still need to use bitnami Golang 1.17 images for our builder until your image has been updated.

sh-4.3# find / -name go
/root/sdk/go1.17.5/pkg/linux_amd64/go
/root/sdk/go1.17.5/pkg/linux_amd64_race/go
/root/sdk/go1.17.5/misc/cgo/testcshared/testdata/go2c2go/go
/root/sdk/go1.17.5/bin/go
/root/sdk/go1.17.5/src/cmd/go
/root/sdk/go1.17.5/src/cmd/vendor/golang.org/x/tools/go
/root/sdk/go1.17.5/src/go
/root/sdk/go1.15.15/pkg/linux_amd64/go
/root/sdk/go1.15.15/pkg/linux_amd64_race/go
/root/sdk/go1.15.15/misc/cgo/testcshared/testdata/go2c2go/go
/root/sdk/go1.15.15/bin/go
/root/sdk/go1.15.15/src/cmd/go
/root/sdk/go1.15.15/src/cmd/vendor/golang.org/x/tools/go
/root/sdk/go1.15.15/src/go
/root/sdk/go1.14.15/pkg/linux_amd64/go
/root/sdk/go1.14.15/pkg/linux_amd64_race/go
/root/sdk/go1.14.15/misc/cgo/testcshared/testdata/go2c2go/go
/root/sdk/go1.14.15/bin/go
/root/sdk/go1.14.15/src/cmd/go
/root/sdk/go1.14.15/src/cmd/vendor/golang.org/x/tools/go
/root/sdk/go1.14.15/src/go
/root/sdk/go1.13.15/pkg/linux_amd64/go
/root/sdk/go1.13.15/pkg/linux_amd64_race/go
/root/sdk/go1.13.15/misc/cgo/testcshared/testdata/go2c2go/go
/root/sdk/go1.13.15/bin/go
/root/sdk/go1.13.15/src/cmd/go
/root/sdk/go1.13.15/src/cmd/vendor/golang.org/x/tools/go
/root/sdk/go1.13.15/src/go
/root/sdk/go1.16.12/pkg/linux_amd64/go
/root/sdk/go1.16.12/pkg/linux_amd64_race/go
/root/sdk/go1.16.12/misc/cgo/testcshared/testdata/go2c2go/go
/root/sdk/go1.16.12/bin/go
/root/sdk/go1.16.12/src/cmd/go
/root/sdk/go1.16.12/src/cmd/vendor/golang.org/x/tools/go
/root/sdk/go1.16.12/src/go
/usr/bin/go
/go
/go/go1.16/bin/go
/go/go1.17/bin/go
/go/go1.13/bin/go
/go/go1.14/bin/go
/go/go1.15/bin/go
sh-4.3#

@RedbackThomson You can use these utility methods to switch to any Go version you want from 1.13 to 1.17.

@TerryHowe
Hmmm that's good but only up to 1.17.5 sadly - 1.17.8 is the latest.
@abhay-krishna
Also good to know! I'll test this out in our image definition and get back to you!

@abhay-krishna Yeah so I've been playing around with the builder base this morning but I see it only installs 1.17.5. So even if we do want to switch with that utility method, it's still not up-to-date enough. Can I create a PR to change this to 1.17.8? Can we unpin it from a patch version and just keep it at the latest version, so it's rebuild with the latest 1.17 every night?

Yes, we would accept a PR to update to 1.17.8, if 1.16 is out of date, mind updating it too?
On our side, our builds are all reproducible with checksums checked in for validation.  When we bump golang versions we need to roll these updates out through our various builds to get all the checksums updated without causing crazy havoc across our presubmits and what not.  I wouldn't be ready to unpin our patch version for now since we need to consider the downstream effects.  That said, we will pretty much be fine updating the versions whenever there are new versions, we just need to keep it as a specific decision.

I wouldn't be ready to unpin our patch version for now since we need to consider the downstream effects. That said, we will pretty much be fine updating the versions whenever there are new versions, we just need to keep it as a specific decision.

Okay. I can understand this concern, and the vetting process you'll need to consider before merging anything. Given you've not updated from 1.17.5 to 1.17.8 means that this hasn't been a worry for you in the 3 months between those releases. ACK is choosing to update to 1.17.8 because of the crypto package security concerns in the former version - as advice from both internal and external security auditors. Do you expect to also share those concerns? That is, do you foresee the EKS-D team actively updating the builder-base to maintain security standards for your generated binaries?

Like I said though, we are fine to update whenever there are new versions, we just do not have a specific cadence we are currently following.  Your reasoning is as good as any to update.

Ill open a PR to bump this.

https://github.com/aws/eks-distro-build-tooling/pull/348


Cannot send mailgun email with only cc or bcc and without to

Laravel Version: 9.x
PHP Version: 8.x

Description:
It is imposible to send mailgun email without specifying to. The issue is here:
protected function createMailgunTransport(array $config)
    {
        $factory = new MailgunTransportFactory();

        if (! isset($config['secret'])) {
            $config = $this->app['config']->get('services.mailgun', []);
        }

        return $factory->create(new Dsn(
            'mailgun+api',
            $config['endpoint'] ?? 'default',
            $config['secret'],
            $config['domain']
        ));
    }

It should use mailgun+https. I am not sure if this would be a breaking change but otherwise I can make a PR.
Steps To Reproduce:
Try to send mailgun email without to

Why would you send an email without specifying "to"?

We also used the API of Mailgun in Laravel v8 so not sure how this would ever have worked?
https://github.com/laravel/framework/blob/8.x/src/Illuminate/Mail/Transport/MailgunTransport.php#L74

Example - when sending reports to clients who shouldn't see each other's emails. Shouldn't sending to bcc (or cc) only be available in any case?
I am not sure how it worked in L8 but it did work.
Anyhow, is there some specific reason for using mailgun+api instead of mailgun+https?

Hi, I think the reason that it worked than (and also works with mailgun+https transport) is that L8 used /messages.mime endpoint and mailgun+api uses /messages.

Does it work for you is you use mailgun+smtps?

Nope, just mailgun or mailgun+https
mailgun+smtps requires some different auth judging by the exception

I think you're correct. I'm a bit concerned with changing it right now in how people's credentials will be handled with the new protocol.
Can you send attachments with the https protocol? I've been comparing the https and api ones with each other but I don't see anything about attachments in the https one:
https://github.com/symfony/mailgun-mailer/blob/6.0/Transport/MailgunHttpTransport.php
https://github.com/symfony/mailgun-mailer/blob/6.0/Transport/MailgunApiTransport.php

Yeah, it works (tested with 1 png file)

Thanks, I've sent in a PR for this: https://github.com/laravel/framework/pull/41255


Update PHPUnit, add config for CS-Fixer

I updated PHPUnit to work with the versions 6.x. I also added a config for the CS-Fixer and fixed some things it was complaining.

Hey, there are many interesting things in this pull request, thanks!
But I prefer to improve the skeleton package (used as base to create new middlewares) than this specific package. Latter, I can run a script to update all existing middlewares with this improvements.
Just made some commits to skeleton with some of your changes. There're some parts not included (like support for php nightly making configuration so complicated in travis).
Feel free to improve the skeleton with more pull request, but I prefer small pull requests with specific features than a big one. It's easier to review and discuss.
Thank you.


Best way to define a pure Runnable from a command line?

Is there a recommended way of creating a target from something like a cmd_args or a string, that only has a RunInfo, using the standard prelude?
I've tried using genrule for this like so:
genrule(
    name = "run_os",
    executable = True,
    out = "dummy",
    cmd = "touch dummy && $(exe :qemu) -cdrom $(location :os_image) -monitor stdio",
)

But this has a couple of major downsides:

It requires a dummy output (Should this be relaxed when executable = True? It might not make sense for a gen rule to not generate any output though.)
It has a DefaultInfo, which makes it get picked up by any buck2 build commands with a pattern that matches this target

I've considered using a sh_binary instead, but I like the conciseness of defining everything in a single rule invocation, and I have other concerns mentioned in https://github.com/facebook/buck2/issues/664.
For now I'm going to write something like a runnable rule that will do what I want, but I was wondering if there was a better way that I missed.

For anyone interested in a workaround, the following works, although it doesn't guarantee hermeticity in any way:
runnable = rule(
    impl = lambda ctx: [DefaultInfo(), RunInfo(args = cmd_args(ctx.attrs.cmd))],
    attrs = {
        "cmd": attrs.list(attrs.arg()),
    },
)

Usage:
runnable(
    name = "run_os",
    cmd = ["$(exe :qemu)", "-cdrom", "$(location :os_image)", "-monitor", "stdio"],
)

Hello, all! I'm leaving this here in case anyone is interested: I recently used Buck2 for formatting and linting cxxrs. The general idea is:

each subcell (owned by a build file) defines targets that group its files by type—for example, a rust group: <cxxrs/packages/buck/cxxrs/tag.bzl>;

at the root, targets are defined that perform particular actions on such groups—for example, executing rustfmt on all rust groups: <cxxrs/packages/buck/cxxrs/root.bzl>, <cxxrs/BUCK>; and

the cxxrs_dev.buck_incremental_command rule then executes ::cxxrs_dev::action::buck::IncrementalCommand to perform an action incrementally as documented here.

    For context, here are the targets that end up being defined at the root.

buck2 targets --config=cxxrs.allow_slow=true : 2>/dev/null

root//:action.clang-format
root//:action.clang-tidy
root//:action.deadnix
root//:action.nixfmt
root//:action.prettier.json
root//:action.prettier.yaml
root//:action.ruff
root//:action.rustfmt
root//:action.shellcheck
root//:action.shfmt
root//:action.taplo
root//:cxxrs_tag.by_type.cxx
root//:cxxrs_tag.by_type.json
root//:cxxrs_tag.by_type.nix
root//:cxxrs_tag.by_type.rust
root//:cxxrs_tag.by_type.shell
root//:cxxrs_tag.by_type.starlark
root//:cxxrs_tag.by_type.toml
root//:cxxrs_tag.by_type.yaml

It's a very basic implementation, but it can be useful as a starting point for anyone interested. I know implementing the cxxrs_dev binary in Rust is an odd choice, but it can be easily implemented in Python like other scripts used in the prelude.
Feel free to let me know if you have any questions!

A note on hermeticity (#664): if you're using Nix, you can write derivations for runnables you want to run hermetically and have Buck2 invoke them by referring to their absolute paths in the Nix store. That is, isolating a runnable and "bundling" it with its dependencies (including resources and the like) is a strength of Nix that can be useful here. Interactions with caching may be a little tricky (i.e., having Buck2 determine when to rebuild these or update the absolute paths it uses to refer to them), but other than that I'd expect an implementation to be reasonably straightforward.

That's an interesting approach. So far I've settled on doing linting/formatting via BXL scripts. The ergonomics aren't ideal due to limitations in Buck2's core, which I've opened issues for, but it works pretty well as it leverage parallelism and caching from Buck2.
How do you scale your approach though? Rules (specifically filegroup in this case) don't have visibility over sources that are owned by more specific BUCK files, even with globbing. Also some files you might want to format/lint may not be sources (e.g. BUCK files). Finally, it looks like your actions affect the whole repo, which would quickly get out of hand for a large enough repo (of course this is fine for many use cases).

R.e. Buck2 + Nix indeed, it's very easy to get started:
# Could come from somewhere else.
NIXPKGS_FLAKEREF = "github:nixos/nixpkgs/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

def _nix_runnable_impl(ctx: AnalysisContext):
    nix_store_link = _nix_package_impl(ctx)
    binary_names = ctx.attrs.binary_names or [ctx.attrs.name]

    if len(binary_names) == 1:
        return [DefaultInfo(), RunInfo(args = cmd_args(nix_store_link.default_outputs[0], format = "{}/bin/" + binary_names[0]))]
    else:
        sub_targets = {}
        for binary_name in binary_names:
            sub_targets[binary_name] = [DefaultInfo(), RunInfo(args = cmd_args(nix_store_link.default_outputs[0], format = "{}/bin/" + binary_name))]
        return [DefaultInfo(sub_targets = sub_targets)]

nix_runnable = rule(
    impl = _nix_runnable_impl,
    attrs = {
        "package_name": attrs.option(attrs.string(), default = None),
        "binary_names": attrs.option(attrs.list(attrs.string()), default = None),
        "flakeref": attrs.string(default = NIXPKGS_FLAKEREF),
    },
)

But this doesn't play nicely with remote execution. My usages of RBE and Nix are mutually exclusive right now, so I haven't explored this yet.
Note also that writing some nix2buck tool that takes a Nix file as input and generates Buck targets for various languages (like Reindeer does for Rust and we do at my dayjob for Conda) should not be too much work (although I don't know if there is a compelling use case for that given the that pinning Nixpkgs already means that any set of nix_package/nix_runnable targets would be consistent out of the box.

I haven't explored using BXL so far; thanks for the pointer.

How do you scale your approach though? Rules […], even with globbing.

Yes, each build file tags its own sources (at least each build file whose
sources we want to act on). Notice how most BUCK files in the project call
cxxrs_tag.known (example). I'm doing this even for BUCK files that are
generated by Reindeer: <reindeer.toml> <reindeer.bzl>.

Also some files you might want to format/lint may not be sources (e.g. BUCK
files).

I'm not sure I follow; would you please clarify? For what it's worth, I am
formatting BUCK files, and it's working as intended.

Finally, it looks like your actions affect the whole repo, […].

Yes, I'm doing that for convenience. It would be simpler to have each BUCK file
define it's own actions; we'd do away with having to tag files to begin with
(the entire point of doing so is to have files exposed at the root). With that
approach, each BUCK file would define an action.<FOO> that performs <FOO> on
its sources only, as opposed to having a single action.<FOO> at the root.
Whether this split is better or worse is, I suppose, in the eye of the beholder.
I haven't made it clear in my previous comment, but also note that the approach
I'm using, albeit simplistic, performs actions incrementally. The initial run
performs actions on all (matching) files, but each run after that performs
actions on modified (i.e., having different hashes) files only. We'd get that
for free if we define a target for each file, but that would quickly get out
of hand, of course.

Demonstration
#!/usr/bin/env bash

set -o errexit -o nounset -o pipefail

_redact_cwd() {
    sed "s:$PWD:<ROOT>:"
}

_inspect_rustfmt_action() {
    buck2 build \
        --build-report=- \
        --config=build_report.unstable_include_other_outputs=true \
        --config=cxxrs.log_level=debug \
        :action.rustfmt |
        jq --raw-output0 '.results[].other_outputs.DEFAULT[]' |
        xargs -0 jq .fields.args
} 2>/dev/null

_invalidate_2_rust_files() {
    fd --extension=rs --max-results=2 --print0 | xargs -0 -I{} "$SHELL" -c 'printf // >>{}'
}

buck2 clean 2>/dev/null
{
    _inspect_rustfmt_action
    _invalidate_2_rust_files
    _inspect_rustfmt_action | _redact_cwd
} | jq --slurp

[
  [
    "packages/_/ad_hoc/lto_0/src/lib.rs",
    "packages/rust/cxxrs/lto_1_rs/src/lib.rs",
    "packages/rust/cxxrs/nix_rs/build.rs",
    "packages/rust/cxxrs/nix_rs/src/error.rs",
    "packages/rust/cxxrs/nix_rs/src/lib.rs",
    "packages/rust/cxxrs/trycmd/src/lib.rs",
    "packages/rust/cxxrs/trycmd/src/main.rs",
    "packages/rust/_0/reindeer/src/lib.rs",
    "packages/rust/_0/reindeer/extern/valuable/src/lib.rs",
    "packages/rust/cxxrs/dev/src/action.rs",
    "packages/rust/cxxrs/dev/src/action/buck.rs",
    "packages/rust/cxxrs/dev/src/error.rs",
    "packages/rust/cxxrs/dev/src/ext.rs",
    "packages/rust/cxxrs/dev/src/lib.rs",
    "packages/rust/cxxrs/dev/src/main.rs",
    "packages/rust/cxxrs/dev/src/wrapper.rs",
    "packages/rust/cxxrs/lto_2_rs/build.rs",
    "packages/rust/cxxrs/lto_2_rs/src/lib.rs"
  ],
  [
    "<ROOT>/packages/_/ad_hoc/lto_0/src/lib.rs",
    "<ROOT>/packages/rust/cxxrs/trycmd/src/main.rs"
  ]
]

So, I'd expect this approach to work decently well even for "medium"[^1]-sized
projects, especially with a better implementation—there are a bunch of easy
improvements to make, but I opted for simplicity. If possible, it would also be
nice to cache the state—even the state of a clean repository only, so we don't
have to perform that full run initially.

I agree with you on the Buck2 + Nix front. Unlike Reindeer, that nix2buck tool
wouldn't have access to structured (meta)data given that flakes can have
arbitrary outputs, but we can easily work around this by requiring certain
outputs. That is, "to use this tool, your flake must output a value of the
form […]"; we could even add a Flake Parts module or something to improve that
experience. Like you, I don't have an immediate use for any complex integration
between Buck2 and Nix, so I also haven't explored this any further; however, you
can imagine this being very useful: if we think bigger than formatters and
linters, having Buck2 use compiler toolchains (and the like) that are built with
Nix would be great for reproducibility. If you use Buck2 from inside of a Nix
devShell you get some benefits locally, but it's still far from ideal (as you
mentioned because of interaction with remote execution, for example).
[^1]: Loosely: what most would consider medium-to-large but not massive.


Please clarify networking requirements for zone redundancy

This article suggests use of a /24 for an ASE but provides no content for more intentional and responsible use of available address space.  It would be helpful if it included some kind of calculation for planning purposes.  It seems the /24 recommendation would mean that each App Plan consumes 1 IP, but what if you enable zone redundancy?  Does that mean you could have 600 plans consuming IP's in your subnet even though you only see 200 plans in the portal?
If there is no impact to networking with redundancy, it would still be helpful to understand what is happening with networking.  I've heard that the ASE itself takes up 13 IP's and the plans each take 1.  Is that true of ASE v2 and v3 or was that information just about ASE v2?  If true for ASE v3 does that mean we can calculate for n where n = number of anticipated app plans such that 13 + n = the number of IP's needed for an ASE subnet?
Appreciate any input.  Thanks!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 2dcd1a66-7ff7-c0da-b170-c302784221ae
Version Independent ID: 4766d1de-b033-029a-9860-12a306195c72
Content: App Service Environment overview - Azure App Service Environment
Content Source: articles/app-service/environment/overview.md
Service: app-service
GitHub Login: @ccompy
Microsoft Alias: ccompy

I read over this as well which also doesn't provide guidance on calculating subnet size.
https://docs.microsoft.com/en-us/azure/app-service/environment/networking

@mhaarbrink Thank you for your query . We will investigate and update the thread.

@mhaarbrink Thanks again for the feedback! I have assigned the issue to the content author to review further and update the document as appropriate.

Detailed information has been added to the networking section: https://docs.microsoft.com/en-us/azure/app-service/environment/networking
#please-close


Up to v5.5.0

Checklist

[x] Used a personal fork of the feedstock to propose changes
~Bumped the build number (if the version is unchanged)~
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

@conda-forge-admin, please rerender

The bot beat me creating and merging #127, so this PR can be closed.


Design boot page for new app

A landing page like nextjs one / rails when you boot a new app

create-portal-app
This would be very plain, still use our styling and give you instructions

Acceptance

[ ] mockup of boot page

Tasks

[x] mock some rough boot page structure
[x] take elements from the hero section design
[x] play around with different options
[x] pick the best ones
[ ] get feedback
[ ] implement changes
[ ] polish off

Outcome

@rufuspollock here as well.. please have a look and let me know what you think

@monikappv generally look good. these are pasted a bit too small to actually compare - other than the layout. Could you maybe share the XD with these side by side for comparison. You could also add a bit of info e.g. fonts etc. (and maybe add thatinfo to the title for each of them eg..
### 4. Tweak to X with Inconsolata font

@rufuspollock here's the design that we worked on during the call yesterday
https://xd.adobe.com/view/3040c464-6cc2-4fc0-b808-bd0a436e0014-6513/
any improvement suggestions?

@monikappv looks good for now ...

FIXED. We have a finalized design (see description link).


Make references appearing after non-trivia trigger an error

This implements the error as discussed in #1920.
I know I have not yet submitted a CLA. I have sent it to my employer asking them to sign it and I am waiting on it back. I know this will not be merged until a CLA is submitted, and I will submit it ASAP.

Hi @nathggns, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes. I promise there's no faxing. https://cla.microsoft.com.

TTYL, MSBOT;

@nathggns, Thanks for signing the contribution license agreement so quickly! Actual humans will now validate the agreement and then evaluate the PR.
Thanks, MSBOT;

This is a breaking change over the previous version of the compiler.

To be clear, the people who were broken wrote references to files that were silently being ignored (and then somehow didn't end up using anything from that reference or ignored all the resulting errors).

It's also a breaking change because if you had something like:   /// <reference
then we wouldn't report that as an error.  but now we will.

I'm also concerned about the perf impact here.  The batch compiler effectively ignores all comments, except for the ones before the very first token.
One approach that might work is to have scanner report any /// comments it sees along hte way.  As this would be very few, it would hopefully not be much of a perf impact.
Note that this would likely not affect incremental parsing.  Here's why.  Say we did see a triple slash comment (after the first token).  We'd report that to the parser, and hte parser would produce an error.  because of that error, the parser would mark that node as having errors, which would make it ineligible for incremental reuse.
--
An alternative approach, which i think i actually prefer, is to stop treating /// <reference comments as actual comments.  Instead treat them as a real language statement.  Once they're in the tree, then it will be easy to report hte error when we do our typecheck walk.
These really should not be treated as comments.  Comments are things that should only have meaning to the user and should be uninterpretted by the compiler.  As these have deep meaning to our compiler (effectively they're the same as an ##include/import in other languages), they should really get a real syntaxkind/node.
That's a more appropriate fix that would address the fundamental bug, would provide a low perf impact solution, would not have any incremental issues, and would fit into our current architecture properly.

Judging by the value of doing this and the cost, i would say we should close this issue as won't fix.


Runnable env

This PR adds on option to specify (in the settings.json) environment variables passed to the runnable.
The simplest way for all runnables in a bunch:
    "rust-analyzer.runnableEnv": {
        "RUN_SLOW_TESTS": "1"
    }

Or it is possible to specify vars more granularly:
    "rust-analyzer.runnableEnv": [
        {
            // "mask": null, // null mask means that this rule will be applied for all runnables
            env: {
                 "APP_ID": "1",
                 "APP_DATA": "asdf"
            }
        },
        {
            "mask": "test_name",
            "env": {
                 "APP_ID": "2", // overwrites only APP_ID
            }
        }
    ]

You can use any valid RegExp as a mask. Also note that a full runnable name is something like run bin_or_example_name, test some::mod::test_name or test-mod some::mod, so it is possible to distinguish binaries, single tests, and test modules with this masks: "^run", "^test " (the trailing space matters!), and "^test-mod" respectively.
Fixes #4450
I suppose this info should be somewhere in the docs, but unsure where is the best place.

I suppose this info should be somewhere in the docs, but unsure where is the best place.

I think we need top-level section in manual.adoc for this:
== Editor Features
==== VS Code

@matklad I added the doc section.

bors r+

Are those environment variables also applied when clicking the "Debug" code lens over a unit test in VSCode?

Yes, they are.


enable default sidecarscope

Enable get DestinationRule and Services from default sidecarScope

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: hzxuzhonghu
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

pilot/OWNERS [hzxuzhonghu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

We are in 1.2 period, and it's secure to enable it.

@hzxuzhonghu: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
350a782c77f839dd629c5ec2437bfe1e4f04886e
link
/test istio-integ-local-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
350a782c77f839dd629c5ec2437bfe1e4f04886e
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
350a782c77f839dd629c5ec2437bfe1e4f04886e
link
/test istio-integ-k8s-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
350a782c77f839dd629c5ec2437bfe1e4f04886e
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
350a782c77f839dd629c5ec2437bfe1e4f04886e
link
/test istio-integ-k8s-tests

prow/istio-pilot-multicluster-e2e.sh
350a782c77f839dd629c5ec2437bfe1e4f04886e
link
/test istio-pilot-multicluster-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test istio-integ-local-tests

@rshriram: changing LGTM is restricted to assignees, and assigning you to the PR failed.

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: hzxuzhonghu, rshriram
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

pilot/OWNERS [hzxuzhonghu,rshriram]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

What is the user impact for this?  Does this mean default sidecarScope is always configured?

What is the user impact for this?  Does this mean default sidecarScope is always configured?

Should have no impact on users.
Yes, for a namespace, if there is no sidecar defined, a default one will be generated, this is also 1.1 behavior. But in 1.1 in order to reduce risk, the Service() and DestinationRule() interface still use the original ones.

/approve
/lgtm

@rshriram: changing LGTM is restricted to assignees, and assigning you to the PR failed.

In response to this:

/approve
/lgtm

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: hzxuzhonghu, rshriram
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

pilot/OWNERS [hzxuzhonghu,rshriram]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Trainer auto_lr_find flag cannot be set to boolean through argparse

🐛 Bug
It seems that the Trainer auto_lr_find flag can only be set to str type via argparse. If it is not a boolean, Trainer will seek the finder with variable name defined via --auto_lr_find instead of using the default lr_finder. It's not a big issue as we can work around this by some processing after parsing the arguments. However, I hope the auto_lr_find flag can be set in a better way as it may cause trouble for people who don't know about this. Thanks!
To Reproduce
This works:
python train.py

trainer = pl.Trainer(gpus=args.gpus,
                            auto_lr_find=True)

This will have error because of the string type:
python train.py --auto_lr_find True

parser = pl.Trainer.add_argparse_args(parser)
args = parser.parse_args()
trainer = pl.Trainer.from_argparse_args(args)

Environment
0.8.4

on flags like this or for example fast_dev_run we should set the action="store_true" in the argparser.
Then we can just do
python train.py --auto_lr_find

but
python train.py --auto_lr_find True

is definitely not the desired way, but you can currently use "1" instead of "True"

Thanks for the help! It sounds okay if "1" would work for the case.

fixed!


Python Send API

Hey guys,
I really like the idea behind Postal (thanks for open sourcing!) and have set up my server. I can send via the UI and can receive messages no problem. I think my issue lies in not correctly using the API.
Here is my current python code:
import requests
import json

def send_msg(to, sender, subject, html):
    api_url="https://mail.domain.io/api/v1/send/message"
    api_key="ghghhg7373732jsdf" #not a real api key :)
    params_dict= {
      "to": to,
      "from": sender,
      "subject":subject,
      "html_body": html
    }
    headers = {'Content-Type': 'application/json', 'X-Server-API-Key': api_key}
    return(requests.post(api_url, data=json.dumps(params_dict), headers=headers))

<EMAIL_ADDRESS><EMAIL_ADDRESS>"Tester", "
Message
"))

After running the script, I get <Response [200]>.
But nothing comes up on the server, and I never receive the messages.
Has anyone had a similar issue, or can see why this isn't working?
Thanks everyone!

the API should return some data to you as well as HTTP 200, do you get a response body?

I don't, which is what's odd. If you have more pressing issues my boss decided to use mailgun for our project.

Unless someone who knows more python can step in, I guess you'd have to tinker with the API controller in the code to see how far your request got. Sorry to not be of more use!

No worries, thanks for the response!


[Hold #45548] [Landing page] Update UseDot to direct "individual" and "1-9" traffic to NewDot rather than Expensify Classic

As part of the August earnings, we're going to start directing traffic to expensify.com to land on New Expensify (GH).
The above covers most of the sign-ups and thus is CRITICAL for moving sign-ups to NewDot. But another signup source is UseDot. Thus, we need to update UseDot so that we direct traffic to newDot as well. Currently, if you sign up on useDot, we will redirect you if you sign up with a public email. This will update it so we do private domains as well.
In addition, we will be updating UseDot to have qualifying questions added like the home page. Thus, we will want to mirror the logic from that project when the time come. But if we end up doing this first, we would send all users to NewDot.
For background, please go here. I do not think this is going to be CRITICAL to move new signups to NewDot but @trjExpensify wanted to create an issue so we can add this as polish once the homepage is launched.

Proposal: Update Landing Page to Redirect Traffic to NewDot
Problem Statement
Currently, the landing page directs traffic to Expensify Classic. We need to update it to redirect all traffic to NewDot instead.
Root Cause
The current landing page configuration is set to point to Expensify Classic. This does not align with our goal to transition users to NewDot.
Changes to Solve the Problem
Update Redirection Logic
Change the landing page redirection logic to point to NewDot instead of Expensify Classic.
Verify URL Configuration
Ensure that all relevant URL configurations and endpoints are updated to reflect the new redirection target.
Code Changes
JavaScript
// Redirection logic in landing page script
const redirectToNewDot = () => {
    window.location.href = "https://new.expensify.com";
};

// Call the redirection function when the page loads
window.onload = redirectToNewDot;

Alternative Solutions Explored
Gradual Rollout
Implement a feature toggle to gradually redirect users from Expensify Classic to NewDot. This allows for a controlled transition and monitoring of any issues that may arise.
User Prompt
Instead of automatic redirection, prompt users on the landing page with an option to navigate to NewDot. This provides users with a choice and eases the transition.

Jumping in on this because I was told it was CRITICAL here

Niceee, thanks! If you could get this prepped and ready to rock, that would be awesome, Neil. We might time the merge with going live with the new home page on expensify.com and routing new sign-ups from there as well, but we can chat about that in channel.

Put the hold on for https://github.com/Expensify/App/issues/45360, as per here.

Just updated the title to reflect the current plan now that we're adding qualifiers to Usedot.

Head's up, I'll be working 50% for the rest of this week and next 🌲

Ah interesting, why? Aren't we still going to hold the "individual" and "1-9" redirect on UseDot until the HybridApp rollout like we are the new home page on expensify.com?

Oh, I wasn't aware that we wanted to hold the UseDot qualifier question redirect until the HybridApp rollout. I assume the project team is aware of and agrees with that? If so then I think it would make sense to put this under a beta, deploy it, and then add everyone to the beta when we want to launch it. Or some similar approach.

I assume the project team is aware of and agrees with that?

Yeah, we've been operating on that for weeks and why this particular issue has a hold for a HybridApp rollout to new users in the title and it's nested in the list of issues held in the updates: https://expensify.slack.com/archives/C036QM0SLJK/p1723483522773169?thread_ts=1723483502.541039&cid=C036QM0SLJK

If so then I think it would make sense to put this under a beta, deploy it, and then add everyone to the beta when we want to launch it. Or some similar approach.

Why don't we just deploy the code to redirect everyone that selects the individual or 1-9 options in a second PR like we are the new home page?

@neil-marcellini shall we close this? It's in place, and we have the issue to remove the beta to release it to the masses. So we're tracking that issue as being held on flicking the switch for new sign-ups.


[medTriggerParameter] empty file

I have a warning:
AutoGen warning:
  "/home/mathildemerle/Dev/MUSIC3/medInria-public/src/layers/medCore/parameter/medTriggerParameter.h"
The header file is empty

Looking at medTriggerParameter.h, it appears that it is completely empty.
It's not a huge problem, but it could be fine to remove it at some point.
If needed the original medTriggerParameter.h and medTriggerParameter.cpp

medAbstractDbController.cpp is empty also


CS reports VM running from appliance, when it didn't start properly

CS reports VM running from appliance but does not start properly - the reason is suspected to be missing configurations from the missing options as described in #3690
ISSUE TYPE

Bug Report

COMPONENT NAME

VMWARE

CLOUDSTACK VERSION

4.13

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

implemented in vmware advanced capabilities work.

implemented in vmware advanced capabilities work.


Error installing PrivacySandBoxKotlin client-app

Hi
I was following the steps mentioned for Android Studio Stable and was trying to install the client-app in an Android Emulator (Tiramisu privacy sandbox API level) when I encountered this error INSTALL_PARSE_FAILED_MANIFEST_MALFORMED when I ran ./gradlew client-app:installPreviewDebug
Partial console output
> Task :client-app:installPreviewDebug
Generating APKs for device 'pixel_4_tiramisu_apis(AVD) - 13' for :client-app:previewDebug

The APKs have been extracted in the directory: /var/folders/nr/yl59xl596bv1bn0m5_hf1vlrq3tcn7/T/apkSelect14758958386426302792
Installing APKs 'base-xxhdpi.apk, base-master.apk, base-en.apk' on 'pixel_4_tiramisu_apis(AVD) - 13' for :client-app:previewDebug

[SplitApkInstallerBase]: Failed to commit install session<PHONE_NUMBER> with command package install-commit<PHONE_NUMBER>.

Error: INSTALL_PARSE_FAILED_MANIFEST_MALFORMED: Failed parse during installPackageLI: /data/app/vmdl1105594988.tmp/base.apk (at Binary XML file line #67): Depending on multiple versions of SDK library com.example.privacysandbox.provider

Unable to install /var/folders/nr/yl59xl596bv1bn0m5_hf1vlrq3tcn7/T/apkSelect14758958386426302792/base-xxhdpi.apk,/var/folders/nr/yl59xl596bv1bn0m5_hf1vlrq3tcn7/T/apkSelect14758958386426302792/base-master.apk,/var/folders/nr/yl59xl596bv1bn0m5_hf1vlrq3tcn7/T/apkSelect14758958386426302792/base-en.apk
com.android.ddmlib.InstallException: Failed to commit install session<PHONE_NUMBER> with command package install-commit<PHONE_NUMBER>.

Error: INSTALL_PARSE_FAILED_MANIFEST_MALFORMED: Failed parse during installPackageLI: /data/app/vmdl1105594988.tmp/base.apk (at Binary XML file line #67): Depending on multiple versions of SDK library com.example.privacysandbox.provider

The AndroidManifest.xml file inside client-app
<?xml version="1.0" encoding="utf-8"?>
<!--
     Copyright (C) 2022 The Android Open Source Project
     Licensed under the Apache License, Version 2.0 (the "License");
     you may not use this file except in compliance with the License.
     You may obtain a copy of the License at
          http://www.apache.org/licenses/LICENSE-2.0
     Unless required by applicable law or agreed to in writing, software
     distributed under the License is distributed on an "AS IS" BASIS,
     WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
     See the License for the specific language governing permissions and
     limitations under the License.
-->
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.example.privacysandbox.client">

  <uses-permission android:name="android.permission.INTERNET"/>
  <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE"/>

  <application
      android:allowBackup="true"
      android:icon="@mipmap/ic_launcher"
      android:label="@string/app_name"
      android:roundIcon="@mipmap/ic_launcher_round"
      android:supportsRtl="true"
<EMAIL_ADDRESS>
    <activity
        android:name="com.example.client.MainActivity"
        android:exported="true">
      <intent-filter>
        <action android:name="android.intent.action.MAIN" />

        <category android:name="android.intent.category.LAUNCHER" />
      </intent-filter>
    </activity>
    <uses-sdk-library android:name="com.example.privacysandbox.provider"
        android:versionMajor="1"
        android:certDigest="<SHA256 fingerprint>" />
  </application>
</manifest>

The SDK provider app was installed in the emulator prior to this setup. I am running Mac OS Big Sur and Android Studio was recently updated (Android Studio Dolphin | 2021.3.1 Patch 1).
Any advice on how to troubleshoot this?

Hi @hellosagar
I did put the digest in the manifest file. I just omitted it from this Github post and replaced it with <SHA256 fingerprint>.

Is this the first time you try to install the app? If not, try to uninstall the app first before installing it on your device

Hi @hellosagar
I was not able to install it at all.

I was facing the same issue few weeks back, you can look at that convo if that might help you - https://github.com/android/privacy-sandbox-samples/issues/26
I was able to solve mine, but I was using the Android Studio Flamingo which is a canary one

Sorry, missed this issue.
Did you also comment out this line in the client-app build file?
https://github.com/android/privacy-sandbox-samples/blob/main/PrivacySandboxKotlin/client-app/build.gradle#L69
Trying to support stable and canary android studio instructions is tricky, so we are looking to simplify this setup

Hi @hellosagar and @cshmerling
Thanks for your help in this. After following the instructions from @cshmerling to comment out that line in the build.gradle file, I can build the application now.
Thanks again for the help.


Runtime error:zero-dimensional tensor(at position 0) cannot be contatenated

when I run train_p_net.py ,it shows these error:
Traceback (most recent call last):
File "dface/train_net/train_p_net.py", line 50, in
end_epoch=args.end_epoch, frequent=args.frequent, lr=args.lr, batch_size=args.batch_size, use_cuda=args.use_cuda)
File "dface/train_net/train_p_net.py", line 17, in train_net
train_pnet(model_store_path=model_store_path, end_epoch=end_epoch, imdb=gt_imdb, batch_size=batch_size, frequent=frequent, base_lr=lr, use_cuda=use_cuda)
File "/home/li/DFace/dface/train_net/train.py", line 97, in train_pnet
accuracy_avg = torch.mean(torch.cat(accuracy_list))
RuntimeError: zero-dimensional tensor (at position 0) cannot be concatenated

Hello, I have the same problem.Have you saved it?

Thank,I have solved it.

change the error line into accuracy_avg = torch.mean(torch.stack(accuracy_list, dim=0))


fix: local node fails to start with custom --network-tag

Description:

Related issue(s):
Fixes #759
Notes for reviewer:

Checklist

[ ] Documented (Code comments, README, etc.)
[ ] Tested (unit, integration, etc.)

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 57.59%. Comparing base (63003a3) to head (0e09d86).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #760   +/-   ##
=======================================
  Coverage   57.59%   57.59%
=======================================
  Files          30       30
  Lines        1403     1403
  Branches      173      173
=======================================
  Hits          808      808
  Misses        574      574
  Partials       21       21

Files with missing lines
Coverage Δ

src/state/InitState.ts
94.17% <100.00%> (ø)


HttpExchange Kotlin conversion

For discussion, the bug in https://github.com/square/okhttp/pull/4936 and the number of !! statements suggests it's a worthy update and ripe for discussion

Side-by-side diff? https://gist.github.com/jrodbx/dcc81cb213e4c300c0e1ddfde4ae553d

Side-by-side diff? https://gist.github.com/jrodbx/dcc81cb213e4c300c0e1ddfde4ae553d

https://github.com/square/okhttp/pull/4938/commits/dbadaa6b614d02b37b6c7c773a418fd0216d85d6

Thanks for the reviews, this was a good learning experience for the migration and the kotlin is so much nicer.

Failing on https://travis-ci.org/square/okhttp/jobs/519834499
> Task :okhttp:compileKotlin
e: /home/travis/build/square/okhttp/okhttp/src/main/java/okhttp3/internal/http1/Http1ExchangeCodec.kt: (31, 41): Unresolved reference: HTTP_CONTINUE
e: /home/travis/build/square/okhttp/okhttp/src/main/java/okhttp3/internal/http1/Http1ExchangeCodec.kt: (187, 46): Unresolved reference: HTTP_CONTINUE
e: /home/travis/build/square/okhttp/okhttp/src/main/java/okhttp3/internal/http1/Http1ExchangeCodec.kt: (188, 28): Unresolved reference: HTTP_CONTINUE
e: /home/travis/build/square/okhttp/okhttp/src/main/java/okhttp3/internal/http2/Http2ExchangeCodec.kt: (32, 41): Unresolved reference: HTTP_CONTINUE
e: /home/travis/build/square/okhttp/okhttp/src/main/java/okhttp3/internal/http2/Http2ExchangeCodec.kt: (103, 77): Unresolved reference: HTTP_CONTINUE


Discover Search Not Working

This was an environment variable issue with alpha-heighliner. Fixed by adding the SEARCH_CX key


reduce the buffer when using high dimensional data in distributed mode.

to fix #2484

These changes broke something in distributed training, now datasets that were fine before also throw an MPI error on an MPI_Recv call somewhere:
salloc -N 3 mpiexec --machinefile /shared/hostnames.txt ../../lightgbm config=train.conf data=/shared/data/avazu-app.val num_trees=3 tree_learner=data
salloc: Granted job allocation 21
[LightGBM] [Warning] data is set=/shared/data/avazu-app.val, data=/shared/kdda.t will be ignored. Current value: data=/shared/data/avazu-app.val
[LightGBM] [Warning] tree_learner is set=data, tree_learner=data will be ignored. Current value: tree_learner=data
[LightGBM] [Warning] num_iterations is set=3, num_trees=100 will be ignored. Current value: num_iterations=3
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Warning] data is set=/shared/data/avazu-app.val, data=/shared/kdda.t will be ignored. Current value: data=/shared/data/avazu-app.val
[LightGBM] [Warning] tree_learner is set=data, tree_learner=data will be ignored. Current value: tree_learner=data
[LightGBM] [Warning] num_iterations is set=3, num_trees=100 will be ignored. Current value: num_iterations=3
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Warning] data is set=/shared/data/avazu-app.val, data=/shared/kdda.t will be ignored. Current value: data=/shared/data/avazu-app.val
[LightGBM] [Warning] tree_learner is set=data, tree_learner=data will be ignored. Current value: tree_learner=data
[LightGBM] [Warning] num_iterations is set=3, num_trees=100 will be ignored. Current value: num_iterations=3
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Info] Local rank: 2, total number of machines: 3
[LightGBM] [Info] Finished initializing network
[LightGBM] [Info] Local rank: 1, total number of machines: 3
[LightGBM] [Info] Finished initializing network
[LightGBM] [Info] Local rank: 0, total number of machines: 3
[LightGBM] [Info] Finished initializing network
[LightGBM] [Info] Finished loading data in 2.873416 seconds
[LightGBM] [Warning] Starting from the 2.1.2 version, default value for the "boost_from_average" parameter in "binary" objective is true.
This may cause significantly different results comparing to the previous versions of LightGBM.
Try to set boost_from_average=false, if your old models produce bad results
[LightGBM] [Info] Finished loading data in 2.883152 seconds
[LightGBM] [Warning] Starting from the 2.1.2 version, default value for the "boost_from_average" parameter in "binary" objective is true.
This may cause significantly different results comparing to the previous versions of LightGBM.
Try to set boost_from_average=false, if your old models produce bad results
[LightGBM] [Info] Finished loading data in 2.892466 seconds
[LightGBM] [Warning] Starting from the 2.1.2 version, default value for the "boost_from_average" parameter in "binary" objective is true.
This may cause significantly different results comparing to the previous versions of LightGBM.
Try to set boost_from_average=false, if your old models produce bad results
[LightGBM] [Info] Number of positive: 252989, number of negative: 1700962
[LightGBM] [Info] Number of positive: 252989, number of negative: 1700962
[LightGBM] [Info] Number of positive: 252989, number of negative: 1700962
[LightGBM] [Info] Total Bins 9926
[LightGBM] [Info] Total Bins 9935
[LightGBM] [Info] Total Bins 9891
[LightGBM] [Info] Number of data: 650820, number of used features: 9901
[LightGBM] [Info] Number of data: 651651, number of used features: 9912
[LightGBM] [Info] Number of data: 651480, number of used features: 9869
[LightGBM] [Info] Finished initializing training
[LightGBM] [Info] Started training...
[LightGBM] [Info] Finished initializing training
[LightGBM] [Info] Started training...
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.129703 -> initscore=-1.903591
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.129677 -> initscore=-1.903820
[LightGBM] [Info] Finished initializing training
[LightGBM] [Info] Started training...
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.129048 -> initscore=-1.909405
[LightGBM] [Info] Start training from score -1.905605
[LightGBM] [Info] Start training from score -1.905605
[LightGBM] [Info] Start training from score -1.905605
[ip-172-31-59-36:05691] *** An error occurred in MPI_Recv
[ip-172-31-59-36:05691] *** reported by process [478871553,1]
[ip-172-31-59-36:05691] *** on communicator MPI_COMM_WORLD
[ip-172-31-59-36:05691] *** MPI_ERR_TRUNCATE: message truncated
[ip-172-31-59-36:05691] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[ip-172-31-59-36:05691] ***    and potentially your MPI job)
--------------------------------------------------------------------------
An MPI communication peer process has unexpectedly disconnected.  This
usually indicates a failure in the peer process (e.g., a crash or
otherwise exiting without calling MPI_FINALIZE first).

Although this local MPI process will likely now behave unpredictably
(it may even hang or crash), the root cause of this problem is the
failure of the peer -- that is what you need to investigate.  For
example, there may be a core file that you can examine.  More
generally: such peer hangups are frequently caused by application bugs
or other external events.

  Local host: ip-172-31-56-120
  Local PID:  5770
  Peer host:  ip-172-31-59-36
--------------------------------------------------------------------------
salloc: Relinquishing job allocation 21
salloc: Job allocation 21 has been revoked.

[LightGBM] [Info] Total Bins 9926
[LightGBM] [Info] Total Bins 9935
[LightGBM] [Info] Total Bins 9891

I find the total bin of different machines are different.
Did you use the same config for different nodes, and without pre-partition dataset?

[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.129703 -> initscore=-1.903591
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.129677 -> initscore=-1.903820
[LightGBM] [Info] Finished initializing training
[LightGBM] [Info] Started training...
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.129048 -> initscore=-1.909405

and different init_score, it seems the dataset used in these machine are different

That is indeed weird. My data and LightGBM distribution lie on shared NFS, so all nodes have access to the same data.
As I said, the exact same dataset (avazu-app.val) trains to completion on the current master, so I think this PR has some kind of side-effect on the data loading.

@thvasilo sorry, I realize there is one more place i need to fix it, I will update this soon.

Tried the current PR, avazu-app.val with 1M features will now error out with [LightGBM] [Fatal] Too many features for distributed model, buffer is not enough. It is better to pass categorical feature directly instead of sparse high dimensional feature vectors.
But I'm wondering if this is a regression as the same dataset trains fine on master, but as before with different numbers of bins and init values, which I guess is bug?:
salloc -N 3 mpiexec --machinefile hostnames.txt ../../lightgbm config=train.conf data=/shared/data/avazu-app.val num_trees=3 tree_learner=data
salloc: Granted job allocation 12
[LightGBM] [Warning] data is set=/shared/data/avazu-app.val, data=binary.train will be ignored. Current value: data=/shared/data/avazu-app.val
[LightGBM] [Warning] tree_learner is set=data, tree_learner=feature will be ignored. Current value: tree_learner=data
[LightGBM] [Warning] num_iterations is set=3, num_trees=100 will be ignored. Current value: num_iterations=3
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Warning] data is set=/shared/data/avazu-app.val, data=binary.train will be ignored. Current value: data=/shared/data/avazu-app.val
[LightGBM] [Warning] tree_learner is set=data, tree_learner=feature will be ignored. Current value: tree_learner=data
[LightGBM] [Warning] num_iterations is set=3, num_trees=100 will be ignored. Current value: num_iterations=3
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Warning] data is set=/shared/data/avazu-app.val, data=binary.train will be ignored. Current value: data=/shared/data/avazu-app.val
[LightGBM] [Warning] tree_learner is set=data, tree_learner=feature will be ignored. Current value: tree_learner=data
[LightGBM] [Warning] num_iterations is set=3, num_trees=100 will be ignored. Current value: num_iterations=3
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Info] Local rank: 0, total number of machines: 3
[LightGBM] [Info] Finished initializing network
[LightGBM] [Info] Local rank: 2, total number of machines: 3
[LightGBM] [Info] Finished initializing network
[LightGBM] [Info] Local rank: 1, total number of machines: 3
LightGBM] [Info] Finished initializing network
[LightGBM] [Info] Local rank: 2, total number of machines: 3
[LightGBM] [Info] Finished initializing network
[LightGBM] [Info] Local rank: 1, total number of machines: 3
[LightGBM] [Info] Finished initializing network
[LightGBM] [Info] Finished loading data in 2.554492 seconds
[LightGBM] [Info] Finished loading data in 2.557398 seconds
[LightGBM] [Warning] Starting from the 2.1.2 version, default value for the "boost_from_average" parameter in "binary" objective is true.
This may cause significantly different results comparing to the previous versions of LightGBM.
Try to set boost_from_average=false, if your old models produce bad results
[LightGBM] [Warning] Starting from the 2.1.2 version, default value for the "boost_from_average" parameter in "binary" objective is true.
This may cause significantly different results comparing to the previous versions of LightGBM.
Try to set boost_from_average=false, if your old models produce bad results
[LightGBM] [Info] Finished loading data in 2.561811 seconds
[LightGBM] [Warning] Starting from the 2.1.2 version, default value for the "boost_from_average" parameter in "binary" objective is true.
This may cause significantly different results comparing to the previous versions of LightGBM.
Try to set boost_from_average=false, if your old models produce bad results
[LightGBM] [Info] Number of positive: 252989, number of negative: 1700962
[LightGBM] [Info] Number of positive: 252989, number of negative: 1700962
[LightGBM] [Info] Number of positive: 252989, number of negative: 1700962
[LightGBM] [Info] Total Bins 3952
[LightGBM] [Info] Total Bins 3949
[LightGBM] [Info] Total Bins 3952
[LightGBM] [Info] Number of data: 650820, number of used features: 3925
[LightGBM] [Info] Number of data: 651651, number of used features: 3925
[LightGBM] [Info] Number of data: 651480, number of used features: 3925
[LightGBM] [Info] Finished initializing training
[LightGBM] [Info] Started training...
[LightGBM] [Info] Finished initializing training
[LightGBM] [Info] Started training...
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.129703 -> initscore=-1.903591
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.129677 -> initscore=-1.903820
[LightGBM] [Info] Finished initializing training
[LightGBM] [Info] Started training...

if the warning exists, i think the error is caused by overflow.
BTW, there is bug in the number of bin calculations in the master branch, for dist model. fixed in this branch.

Unfortunately a new error popped up, on the datasets I've tried (kdda & avazu-app.val)
salloc -N 3 mpiexec --machinefile hostnames.txt ./lightgbm config=examples/parallel_training/train.conf data=/shared/data/avazu-app.val num_trees=3 tree_learner=data
salloc: Granted job allocation 4
[LightGBM] [Warning] Config file examples/parallel_training/train.conf doesn't exist, will ignore
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Warning] Config file examples/parallel_training/train.conf doesn't exist, will ignore
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Warning] Config file examples/parallel_training/train.conf doesn't exist, will ignore
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Info] Finished loading data in 8.683312 seconds
[LightGBM] [Info] Total Bins 15846
[LightGBM] [Fatal] Check failed: offset == num_total_bin at /shared/LightGBM/src/treelearner/feature_histogram.hpp, line 747 .

[LightGBM] [Warning] Check failed: offset == num_total_bin at /shared/LightGBM/src/treelearner/feature_histogram.hpp, line 747 .

[LightGBM] [Fatal] Check failed: offset == num_total_bin at /shared/LightGBM/src/treelearner/feature_histogram.hpp, line 747 .

[LightGBM] [Warning] Check failed: offset == num_total_bin at /shared/LightGBM/src/treelearner/feature_histogram.hpp, line 747 .

[LightGBM] [Fatal] Check failed: offset == num_total_bin at /shared/LightGBM/src/treelearner/feature_histogram.hpp, line 747 .

<... same errors continue>

yeah, actually, the master branch is okay, expect the print information is not right.
My previous fix for it introduce a new bug. Now it should work.

Thanks for the prompt fix!
avazu-app.t will train fine now, still getting the same for kdda.t though.
 salloc -N 3 mpiexec --machinefile hostnames.txt ./lightgbm config=examples/mpi/train.conf data=/shared/data/kdda.t num_trees=3 tree_learner=data
salloc: Granted job allocation 12
[LightGBM] [Warning] data is set=/shared/data/kdda.t, data=binary.train will be ignored. Current value: data=/shared/data/kdda.t
[LightGBM] [Warning] tree_learner is set=data, tree_learner=feature will be ignored. Current value: tree_learner=data
[LightGBM] [Warning] num_iterations is set=3, num_trees=100 will be ignored. Current value: num_iterations=3
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Warning] data is set=/shared/data/kdda.t, data=binary.train will be ignored. Current value: data=/shared/data/kdda.t
[LightGBM] [Warning] tree_learner is set=data, tree_learner=feature will be ignored. Current value: tree_learner=data
[LightGBM] [Warning] num_iterations is set=3, num_trees=100 will be ignored. Current value: num_iterations=3
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Warning] data is set=/shared/data/kdda.t, data=binary.train will be ignored. Current value: data=/shared/data/kdda.t
[LightGBM] [Warning] tree_learner is set=data, tree_learner=feature will be ignored. Current value: tree_learner=data
[LightGBM] [Warning] num_iterations is set=3, num_trees=100 will be ignored. Current value: num_iterations=3
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Info] Local rank: 0, total number of machines: 3
[LightGBM] [Info] Finished initializing network
[LightGBM] [Info] Local rank: 1, total number of machines: 3
[LightGBM] [Info] Finished initializing network
[LightGBM] [Info] Local rank: 2, total number of machines: 3
[LightGBM] [Info] Finished initializing network
[LightGBM] [Fatal] Check failed: num_total_features_ == static_cast<int>(bin_mappers->size()) at /shared/LightGBM/src/io/dataset.cpp, line 225 .

[LightGBM] [Fatal] Check failed: num_total_features_ == static_cast<int>(bin_mappers->size()) at /shared/LightGBM/src/io/dataset.cpp, line 225 .

[LightGBM] [Info] Finished loading data in 4.564516 seconds
[LightGBM] [Warning] Starting from the 2.1.2 version, default value for the "boost_from_average" parameter in "binary" objective is true.
This may cause significantly different results comparing to the previous versions of LightGBM.
Try to set boost_from_average=false, if your old models produce bad results
^C
salloc: Relinquishing job allocation 12

Thanks! I will find out what cause that checking failed.

@thvasilo I think the latest commit should fix the check error. Could you try it again? Thanks very much!

Thanks @guolinke it does indeed work now!
One last question I have is about initscore is it supposed to be the same for all workers?
Here's some example outputs for kdda:
[LightGBM] [Info] Number of positive: 442845, number of negative: 67457
[LightGBM] [Info] Number of positive: 442845, number of negative: 67457
[LightGBM] [Info] Number of positive: 442845, number of negative: 67457
[LightGBM] [Info] Total Bins 44054
[LightGBM] [Info] Total Bins 44054
[LightGBM] [Info] Total Bins 44054
[LightGBM] [Info] Number of data: 170354, number of used features: 8663
[LightGBM] [Info] Number of data: 170291, number of used features: 8663
[LightGBM] [Info] Number of data: 169657, number of used features: 8663
[LightGBM] [Info] Finished initializing training
[LightGBM] [Info] Started training...
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.867359 -> initscore=1.877803
[LightGBM] [Info] Finished initializing training
[LightGBM] [Info] Started training...
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.867297 -> initscore=1.877268
[LightGBM] [Info] Finished initializing training
[LightGBM] [Info] Started training...
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.868772 -> initscore=1.890142
[LightGBM] [Info] Start training from score 1.881737
[LightGBM] [Info] Start training from score 1.881737
[LightGBM] [Info] Start training from score 1.881737

and avazu-app.val:
[LightGBM] [Info] Number of positive: 252989, number of negative: 1700962
[LightGBM] [Info] Number of positive: 252989, number of negative: 1700962
[LightGBM] [Info] Number of positive: 252989, number of negative: 1700962
[LightGBM] [Info] Total Bins 7850
[LightGBM] [Info] Total Bins 7850
[LightGBM] [Info] Total Bins 7850
[LightGBM] [Info] Number of data: 651480, number of used features: 3925
[LightGBM] [Info] Number of data: 651651, number of used features: 3925
[LightGBM] [Info] Number of data: 650820, number of used features: 3925
[LightGBM] [Info] Finished initializing training
[LightGBM] [Info] Started training...
[LightGBM] [Info] Finished initializing training
[LightGBM] [Info] Started training...
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.129703 -> initscore=-1.903591
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.129048 -> initscore=-1.909405
[LightGBM] [Info] Finished initializing training
[LightGBM] [Info] Started training...
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.129677 -> initscore=-1.903820
[LightGBM] [Info] Start training from score -1.905605
[LightGBM] [Info] Start training from score -1.905605
[LightGBM] [Info] Start training from score -1.905605

the init_score is sync. you can find:

[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.129703 -> initscore=-1.903591
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.129048 -> initscore=-1.909405
[LightGBM] [Info] Finished initializing training
[LightGBM] [Info] Started training...
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.129677 -> initscore=-1.903820
[LightGBM] [Info] Start training from score -1.905605
[LightGBM] [Info] Start training from score -1.905605
[LightGBM] [Info] Start training from score -1.905605

the last 3 lines are the sync score in 3 nodes.

Thanks @thvasilo very much! now the distributed mode can be efficiently ran over sparse features too.

One thing I noticed, when setting min_data_in_leaf = 1 I get the following error, but only for kdda.t, avazu-app.val is works fine:
[ip-172-31-52-7:06473] *** Process received signal ***
[ip-172-31-52-7:06473] Signal: Segmentation fault (11)
[ip-172-31-52-7:06473] Signal code: Address not mapped (1)
[ip-172-31-52-7:06473] Failing at address: 0x100011
[ip-172-31-52-7:06473] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f2e6c4e6390]
[ip-172-31-52-7:06473] [ 1] ./lightgbm(_ZN8LightGBM16GetConfilctCountERKSt6vectorIbSaIbEEPKiii+0x9)[0x4fd0b9]
[ip-172-31-52-7:06473] [ 2] ./lightgbm(_ZN8LightGBM10FindGroupsERKSt6vectorISt10unique_ptrINS_9BinMapperESt14default_deleteIS2_EESaIS5_EERKS0_IiSaIiEEPPiPKimiiib+0x802)[0x50a302]
[ip-172-31-52-7:06473] [ 3] ./lightgbm(_ZN8LightGBM19FastFeatureBundlingERKSt6vectorISt10unique_ptrINS_9BinMapperESt14default_deleteIS2_EESaIS5_EEPPiPKimRKS0_IiSaIiEEdiidbb+0x5ee)[0x50c2fe]
[ip-172-31-52-7:06473] [ 4] ./lightgbm(_ZN8LightGBM7Dataset9ConstructEPSt6vectorISt10unique_ptrINS_9BinMapperESt14default_deleteIS3_EESaIS6_EEiRKS1_IS1_IdSaIdEESaISB_EEPPiPKimRKNS_6ConfigE+0x23d)[0x50d21d]
[ip-172-31-52-7:06473] [ 5] ./lightgbm(_ZN8LightGBM13DatasetLoader31ConstructBinMappersFromTextDataEiiRKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EEPKNS_6ParserEPNS_7DatasetE+0x1d80)[0x527cf0]
[ip-172-31-52-7:06473] [ 6] ./lightgbm(_ZN8LightGBM13DatasetLoader12LoadFromFileEPKcS2_ii+0x1ed)[0x52aa8d]
[ip-172-31-52-7:06473] [ 7] ./lightgbm(_ZN8LightGBM11Application8LoadDataEv+0x24a)[0x443f4a]
[ip-172-31-52-7:06473] [ 8] ./lightgbm(_ZN8LightGBM11Application9InitTrainEv+0x181)[0x445171]
[ip-172-31-52-7:06473] [ 9] ./lightgbm(main+0x49)[0x440199]
[ip-172-31-52-7:06473] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f2e6c12b830]
[ip-172-31-52-7:06473] [11] ./lightgbm(_start+0x29)[0x442779]
[ip-172-31-52-7:06473] *** End of error message ***

If I set the parameter to 2 (or other values > 1)  this does not happen. Seems like a corner case :/
For me the current PR is good enough, just something to keep in mind.

@thvasilo it is not trival to locate what cause the fail, I add a possible fix, you can have a try 😄

Now it takes much longer, but still get an error. I'd just note this as a known issue if it is to be included in the release, as you said this would be hard to track down.
[ip-172-31-51-139:08446] *** Process received signal ***
[ip-172-31-51-139:08446] Signal: Segmentation fault (11)
[ip-172-31-51-139:08446] Signal code: Address not mapped (1)
[ip-172-31-51-139:08446] Failing at address: 0x5331
[ip-172-31-51-139:08446] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f9b541a9390]
[ip-172-31-51-139:08446] [ 1] ./lightgbm(_ZN8LightGBM16GetConfilctCountERKSt6vectorIbSaIbEEPKiii+0x9)[0x4fd0b9]
[ip-172-31-51-139:08446] [ 2] ./lightgbm(_ZN8LightGBM10FindGroupsERKSt6vectorISt10unique_ptrINS_9BinMapperESt14default_deleteIS2_EESaIS5_EERKS0_IiSaIiEEPPiPKimiiib+0x802)[0x50a302]
[ip-172-31-51-139:08446] [ 3] ./lightgbm(_ZN8LightGBM19FastFeatureBundlingERKSt6vectorISt10unique_ptrINS_9BinMapperESt14default_deleteIS2_EESaIS5_EEPPiPKimRKS0_IiSaIiEEdiidbb+0x5ee)[0x50c2fe]
[ip-172-31-51-139:08446] [ 4] ./lightgbm(_ZN8LightGBM7Dataset9ConstructEPSt6vectorISt10unique_ptrINS_9BinMapperESt14default_deleteIS3_EESaIS6_EEiRKS1_IS1_IdSaIdEESaISB_EEPPiPKimRKNS_6ConfigE+0x23d)[0x50d21d]
[ip-172-31-51-139:08446] [ 5] ./lightgbm(_ZN8LightGBM13DatasetLoader31ConstructBinMappersFromTextDataEiiRKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EEPKNS_6ParserEPNS_7DatasetE+0x1d80)[0x527cf0]
[ip-172-31-51-139:08446] [ 6] ./lightgbm(_ZN8LightGBM13DatasetLoader12LoadFromFileEPKcS2_ii+0x1ed)[0x52aa8d]
[ip-172-31-51-139:08446] [ 7] ./lightgbm(_ZN8LightGBM11Application8LoadDataEv+0x24a)[0x443f4a]
[ip-172-31-51-139:08446] [ 8] ./lightgbm(_ZN8LightGBM11Application9InitTrainEv+0x181)[0x445171]
[ip-172-31-51-139:08446] [ 9] ./lightgbm(main+0x49)[0x440199]
[ip-172-31-51-139:08446] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f9b53dee830]
[ip-172-31-51-139:08446] [11] ./lightgbm(_start+0x29)[0x442779]
[ip-172-31-51-139:08446] *** End of error message ***

Thanks, I think I need to debug for that? Could the fail reproduce when run in the single node?

Running on a single node (no MPI parallel training) seems to be working fine.
One question I had: when I set min_data_in_leaf = 1 the number of total bins and used features changes, is this expected behavior? May I ask why that is in that case?
min_data_in_leaf = 1
[LightGBM] [Info] Total Bins 2268113
[LightGBM] [Info] Number of data: 510302, number of used features: 1118008

min_data_in_leaf = 50
[LightGBM] [Info] Total Bins 83942
[LightGBM] [Info] Number of data: 510302, number of used features: 26463

@thvasilo yeah, it is expected behavior.
LightGBM will pre-prune the feature that cannot split.
when you set min_data_in_leaf to k, for a very sparse features with non-zero samples smaller than k, will be pruned.

@thvasilo could you try one more time? If this still failed, I think I cannot fix it temporarily.

@guolinke
Just noticed this warning:
[ 43%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/file_io.cpp.o
/__w/1/s/src/io/dataset_loader.cpp: In member function ‘void LightGBM::DatasetLoader::ConstructBinMappersFromTextData(int, int, const std::vector<std::basic_string<char> >&, const LightGBM::Parser*, LightGBM::Dataset*)’:
/__w/1/s/src/io/dataset_loader.cpp:984:49: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
       if (sample_values.size() <= start[rank] + i) {
                                                 ^

Hello @guolinke sorry I got caught up with other stuff. Can confirm that training on kdda.t (20M features) with min_data_in_leaf=1 works as expected.
Example output:
salloc -N 3 mpiexec --machinefile ~/hostnames.txt ../../lightgbm config=train.conf data=/shared/data/kdda.t num_trees=3 tree_learner=data min_data_in_leaf=1
salloc: Granted job allocation 21
[LightGBM] [Warning] data is set=/shared/data/kdda.t, data=binary.train will be ignored. Current value: data=/shared/data/kdda.t
[LightGBM] [Warning] tree_learner is set=data, tree_learner=feature will be ignored. Current value: tree_learner=data
[LightGBM] [Warning] min_data_in_leaf is set=1, min_data_in_leaf=50 will be ignored. Current value: min_data_in_leaf=1
[LightGBM] [Warning] num_iterations is set=3, num_trees=100 will be ignored. Current value: num_iterations=3
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Warning] data is set=/shared/data/kdda.t, data=binary.train will be ignored. Current value: data=/shared/data/kdda.t
[LightGBM] [Warning] tree_learner is set=data, tree_learner=feature will be ignored. Current value: tree_learner=data
[LightGBM] [Warning] min_data_in_leaf is set=1, min_data_in_leaf=50 will be ignored. Current value: min_data_in_leaf=1
[LightGBM] [Warning] num_iterations is set=3, num_trees=100 will be ignored. Current value: num_iterations=3
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Warning] data is set=/shared/data/kdda.t, data=binary.train will be ignored. Current value: data=/shared/data/kdda.t
[LightGBM] [Warning] tree_learner is set=data, tree_learner=feature will be ignored. Current value: tree_learner=data
[LightGBM] [Warning] min_data_in_leaf is set=1, min_data_in_leaf=50 will be ignored. Current value: min_data_in_leaf=1
[LightGBM] [Warning] num_iterations is set=3, num_trees=100 will be ignored. Current value: num_iterations=3
[LightGBM] [Info] Finished loading parameters
[LightGBM] [Info] Local rank: 2, total number of machines: 3
[LightGBM] [Info] Finished initializing network
[LightGBM] [Info] Local rank: 0, total number of machines: 3
[LightGBM] [Info] Finished initializing network
[LightGBM] [Info] Local rank: 1, total number of machines: 3
[LightGBM] [Info] Finished initializing network
[LightGBM] [Info] Finished loading data in 14.300335 seconds
[LightGBM] [Warning] Starting from the 2.1.2 version, default value for the "boost_from_average" parameter in "binary" objective is true.
This may cause significantly different results comparing to the previous versions of LightGBM.
Try to set boost_from_average=false, if your old models produce bad results
[LightGBM] [Info] Finished loading data in 14.709807 seconds
[LightGBM] [Warning] Starting from the 2.1.2 version, default value for the "boost_from_average" parameter in "binary" objective is true.
This may cause significantly different results comparing to the previous versions of LightGBM.
Try to set boost_from_average=false, if your old models produce bad results
[LightGBM] [Info] Finished loading data in 14.874291 seconds
[LightGBM] [Warning] Starting from the 2.1.2 version, default value for the "boost_from_average" parameter in "binary" objective is true.
This may cause significantly different results comparing to the previous versions of LightGBM.
Try to set boost_from_average=false, if your old models produce bad results
[LightGBM] [Info] Number of positive: 442845, number of negative: 67457
[LightGBM] [Info] Number of positive: 442845, number of negative: 67457
[LightGBM] [Info] Number of positive: 442845, number of negative: 67457
[LightGBM] [Info] Total Bins 2050839
[LightGBM] [Info] Total Bins 2050839
[LightGBM] [Info] Total Bins 2050839
[LightGBM] [Info] Number of data: 169657, number of used features: 1010155
[LightGBM] [Info] Number of data: 170291, number of used features: 1010155
[LightGBM] [Info] Number of data: 170354, number of used features: 1010155
[LightGBM] [Info] Finished initializing training
[LightGBM] [Info] Started training...
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.867297 -> initscore=1.877268
[LightGBM] [Info] Finished initializing training
[LightGBM] [Info] Started training...
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.868772 -> initscore=1.890142
[LightGBM] [Info] Finished initializing training
[LightGBM] [Info] Started training...


Hope to highlight the programming language

What type of request is this?
New feature idea
Clear and concise description of the feature you are proposing
Sometimes I need to share some code snippets and hope they can be highlighted
Additional context
No response
Validations

[X] Check the feature is not already implemented in the project.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.
[X] Check that the feature is technically feasible and aligns with the project's goals.

I think it might also support images and even videos.


CMG-71 After CP changes to create preprod env, uncomment circle-ci config to deploy to preprod

Welcome to Codecov :tada:
Once merged to your default branch, Codecov will compare your coverage reports and display the results in this comment.
Thanks for integrating Codecov - We've got you covered :open_umbrella:


Support records from C# 9

Hello,
if I use records, Ceras cannot serialize. It throws that the constructor is not parameterless. Support for this could be very helpful.
Thanks for your work.

Hi, that would be a great idea!


Instruction handling refactoring

Coverage decreased (-0.5%) to 98.305% when pulling 82ec90660b777e397577f960c52824e4188bd347 on instruction-handling-refactoring into d1408cc2416f025b0a7d842d662fc7df67f949f6 on master.


Temprorary disable sasl for zk.

Zookeeper SASL plugin does not seem to work properly, temporary
disable sasl in connection string.
Disable kernel watchdog.

Codecov Report

Merging #65 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #65   +/-   ##
=======================================
  Coverage   25.41%   25.41%
=======================================
  Files          41       41
  Lines        1794     1794
  Branches      192      192
=======================================
  Hits          456      456
  Misses       1316     1316
  Partials       22       22

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2c1062b...55c267c. Read the comment docs.


ESSD 4 - Leaf phys vignette

Leaf physiology vignette needs to be ship-shaped

@atkinsjeff Let me know if I'm missing anything from the To Do list below for this vignette.
**NOTE: It looks like the fd_photosynthesis() data set includes ONLY subcanopy measurements from 2018, whereas fd_leaf_spectrometry() includes ONLY canopy measurements from 2018. Not necessarily an issue but requires two distinct methodology sections, for sure.
TO DO:
--fleshed out/distinct methods sections for fd_photosynthesis() and fd_leaf_spectrometry()
--revised boxplot figures for each data set
--data availability figures for each data set
--references

@atkinsjeff Also noting that the fd_leaf_spectrometry() dataframe still returns as just the first 8 rows

reinstall the package from github
On Wed, Dec 9, 2020 at 11:36 AM Lisa T. Haber<EMAIL_ADDRESS>wrote:

@atkinsjeff https://github.com/atkinsjeff Also noting that the
fd_leaf_spectrometry() dataframe still returns as just the first 8 rows
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/FoRTExperiment/fortedata/issues/64#issuecomment-741892129,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB7XVVKPJ4L76IKVZ6LJ5HLST6RSJANCNFSM4USKNV7Q
.

--
Jeff Atkins, PhD
Post-Doctoral Fellow
Department of Biology
Virginia Commonwealth University
atkinsjeff.github.io
he/his/him

Duh. Geez.

😊
Jeff Atkins, Ph.D
Department of Biology
Virginia Commonwealth University
On Wed, Dec 9, 2020, 12:00 Lisa T. Haber<EMAIL_ADDRESS>wrote:

Duh. Geez.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/FoRTExperiment/fortedata/issues/64#issuecomment-741906347,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB7XVVLOZ2OZVW45M7MOMA3ST6UJJANCNFSM4USKNV7Q
.

@atkinsjeff Ok a couple of (annoying) issues to work out in fd_leaf_spectrometry():

Looks like the subplot_id column has "O" rather than "0" (letter rather than number zero) in the id name. Probably originally my fault.

The current "tree_id" column is still not right and I think we want to correct it with the later-appended (2019) tree ID's so that subsequent years' data can be easily compared to 2018's. Here's what to do:

--Use the attached look up table (ignore previously sent ones, use this one).
--final column ("actual_tree_id") is the value we want to replace with.
--To match these, you're going to need to concatenate the (corrected, with zeros) subplot_id + species + tree_id values in the current fd_leaf_spectrometry() data set. Example:
subplot_id: D01E
species: ACRU
tree_id: CAN01
string: D01E_ACRU_CAN01 -- > use look up table --> actual tree id = 1EA1
Again, I apologize for the clusterf@#$ and let me know if any of this needs clarification.  I had no idea the workflow or the objective that was being
alluded to with the tree is jam.
Jeff Atkins, Ph.D
Department of Biology
Virginia Commonwealth University
On Wed, Dec 9, 2020, 12:19 Lisa T. Haber<EMAIL_ADDRESS>wrote:

@atkinsjeff https://github.com/atkinsjeff Ok a couple of (annoying)
issues to work out in fd_leaf_spectrometry():

Looks like the subplot_id column has "O" rather than "0" (letter
rather than number zero) in the id name. Probably originally my fault.
2.
The current "tree_id" column is still not right and I think we want
to correct it with the later-appended (2019) tree ID's so that subsequent
years' data can be easily compared to 2018's. Here's what to do:
--Use the attached look up table (ignore previously sent ones, use this
one).
--final column ("actual_tree_id") is the value we want to replace with.
--To match these, you're going to need to concatenate the (corrected, with
zeros) subplot_id + species + tree_id values in the current
fd_leaf_spectrometry() data set. Example:
subplot_id: D01E
species: ACRU
tree_id: CAN01
string: D01E_ACRU_CAN01 -- > use look up table --> actual tree id = 1EA1
Again, I apologize for the clusterf@#$ and let me know if any of this
needs clarification.
Reply to this email directly, view it on GitHub
https://github.com/FoRTExperiment/fortedata/issues/64#issuecomment-741918215,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB7XVVJFWXYQAJCDRGB6NVDST6WSPANCNFSM4USKNV7Q
.

@atkinsjeff fd_leaf_spectrometry() missing disturbance severity and treatment. BUTTTT, because these measurements were taken from the Girdling 2.0 group, the treatment for ALL "subplots" here should be Top-down. So maybe we should leave that off, since it's a different sub-experiment?

Two more questions @atkinsjeff whenever you get to them, not urgent:
(1) How do we add images to vignettes using paths that won't break when we push changes? Looks like the pictures in other vignettes are sourced from github but I only have mine on my local
(2) Did you cut and paste your references from a manuscript, or format them another way?

@kdorheim  taught me this trick, but post the image into a comment on here and then copy the URL

I copied refs via Google Scholar using APA style as it is the one most similar to what a lot of journals in science do.

@lisahaber looks good. I am just updating w/ lower res photos


Support pydantic v1

@luiztauffer with this we now support both pydantic versions 1 and 2, and there are CI tests for both.

Codecov Report

Merging #47 (4c7773c) into main (d19195d) will increase coverage by 0.01%.
The diff coverage is 83.17%.

@@            Coverage Diff             @@
##             main      #47      +/-   ##
==========================================
+ Coverage   86.72%   86.73%   +0.01%
==========================================
  Files          54       55       +1
  Lines        3148     3167      +19
==========================================
+ Hits         2730     2747      +17
- Misses        418      420       +2

Flag
Coverage Δ

unittests
86.73% <83.17%> (+0.01%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files
Coverage Δ

python/dendro/__init__.py
100.00% <100.00%> (ø)

python/dendro/api_helpers/clients/db.py
100.00% <100.00%> (ø)

...api_helpers/routers/gui/compute_resource_routes.py
100.00% <100.00%> (ø)

...dendro/api_helpers/services/_create_output_file.py
65.00% <100.00%> (+0.89%)
:arrow_up:

...pi_helpers/services/processor/update_job_status.py
88.33% <100.00%> (+0.19%)
:arrow_up:

python/dendro/sdk/App.py
89.62% <100.00%> (ø)

python/dendro/sdk/OutputFile.py
82.60% <100.00%> (+3.44%)
:arrow_up:

python/dendro/api_helpers/core/_model_dump.py
75.00% <75.00%> (ø)

python/dendro/sdk/Job.py
97.29% <97.50%> (+12.29%)
:arrow_up:

python/dendro/sdk/AppProcessor.py
86.80% <54.54%> (-2.03%)
:arrow_down:

... and 1 more

:mega: Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in Chrome or Firefox today!


Bluetooth: Controller: Fix incorrect use of rx buffer in Tx ISR

Do not enqueue node rx buffers for generating invalid ISO
data from Tx ISR while supplying the node rx buffer to
Radio. This causes node rx pool corruption and also MIC
failures in the central ISO LLL.
Signed-off-by: Vinayak Kariappa Chettimada<EMAIL_ADDRESS>

Assuming this is the patch I've tested, it works :)

Yes, this is the change I built the hex for you to test, fix for failure symptoms being MIC failure.


Add metadata

This PR closes #429. It adds metadata in home/library views
@dialelo @raulraja Could you review please? Thank you

Note that I'm creating two new issues for:

Asking Maureen for a new tagline/description.
Asking Isra for a better poster image.

Apart from Raúl's comment LGTM :+1:

Addressed comments:

Now the URL of poster image is expressed as absolute path.
Now metadata url is sensible to the current path, the target link is consistently to the shared section.

@raulraja Is good to be merged?

LGTM