[RFC] Provide facilities for JS<->C++ conversion

Background
I have written some node addons, and I feel kind of pain in converting JS to C++ and back in the API bindings.  So I plan to write some utility APIs that can help me do the conversion stuff automatically (one time and save forever). I think Nan is the best place for these APIs rather than creating a new separate module (and more developers will benefit from it ;)).
Here is a proposal to provide new JS<->C++ facilities in NAN that make it easier to marshal types between JavaScript and C++.
Feedback and comments are welcome. If the community accepts the proposal, I'm happy to contribute it :-).
Problem
When writing a native node addon, developers need to write some defending code in their API bindings:

Check whether the passed arguments are valid (e.g. the numbers of arguments, the type of arguments are expected).
If they are valid, then convert them from V8 types to native C++ types (using raw V8 APIs or Nan APIs).

Although Nan has already provided two helpers Nan::To and Nan::New, which converts V8 types to C++ types and back, but they don't fully solve the problem:

Only basic native number types (bool, uint32, int32, int64, double) are supported.
Developers still need to check availability for each argument separately.

Proposed Solution
This proposal is to add two functions in Nan which would make it easier to marshal types between JavaScript and C++, so that developers won't need to re-implement their JS<->C++ bindings.
The APIs are defined below:
// Currently Supported template types:
//   * basic native types: int32, int64, bool, double, uint32
//   * std::string
//   * std::vector<T>
// We could provide some interfaces allowing developers to support their own C++ types.
//
// Convert V8's arguments to C++ native types. Returns false if fails.
//
// An example of a JS function "add(a, b)" which takes two Integer parameters.
//   NAN_METHOD(Add) {
//     int a, b;
//     if (FromV8Argument(info, &a, &b)) {
//       // Do anything with a, b.
//     }
//   }
template<typename... Args>
bool FromV8Argument(Nan::NAN_METHOD_ARGS_TYPE v8_args, Args*... args);

// Convert the supported C++ native types to V8 types.
// Similar to NAN::New(), but it supports more types.
template<typename T>
v8::Local<v8::Value> ToV8Value(T value);

With the help of these two functions, writing JS bindings could be easier, and most of the messy stuff of converting V8 types to C++ types is done automatically and carefully.
The original code without using FromV8Argument:
NAN_METHOD(add) {
  if (!info[0]->IsNumber() || !info[1]->IsNumber()) {
    Nan::ThrowTypeError("Wrong arguments");
    return;
  }
  double arg0 = info[0]->NumberValue();
  double arg1 = info[1]->NumberValue();
  info.GetReturnValue().Set(Nan::New(arg0 + arg1));
}

The code using FromV8Argument:
NAN_METHOD(add) {
  double arg0, arg1;
  if (!Nan::FromV8Arguments(info, &arg0, &arg1)) {
    Nan::ThrowTypeError("Wrong arguments");
    return;
  }
  info.GetReturnValue().Set(Nan::New(arg0 + arg1));
}

Again, suggestions are welcome.

In general, we try not to have pure utility functionality in NAN, save for what already is there. Nan::To and Nan::New are not just utility, they are actually necessary for things to work on all supported versions. Another problem is that C++11 features like variadic templates cannot be used for NAN's support of Node pre-4.
You might want to take a look at https://github.com/charto/nbind. I think it does what you proposed and much more in a very elegant way.
On July 30, 2017 4:42:37 PM GMT+02:00, Haojian Wu<EMAIL_ADDRESS>wrote:

Background
I have written some node addons, and I feel kind of pain in converting
JS to C++ and back in the API bindings.  So I plan to write some
utility APIs that can help me do the conversion stuff automatically
(one time and save forever). I think Nan is the best place for these
APIs rather than creating a new separate module (and more developers
will benefit from it ;)).
Here is a proposal to provide new JS<->C++ facilities in NAN that make
it easier to marshal types between JavaScript and C++.
Feedback and comments are welcome. If the community accepts the
proposal, I'm happy to contribute it :-).
Problem
When writing a native node addon, developers need to write some
defending code in their API bindings:

Check whether the passed arguments are valid (e.g. the numbers of
arguments, the type of arguments are expected).
If they are valid, then convert them from V8 types to native C++
types (using raw V8 APIs or Nan APIs).

Although Nan has already provided two helpers Nan::To and Nan::New,
which converts V8 types to C++ types and back, but they don't fully
solve the problem:

Only basic native number types (bool, uint32, int32, int64,
double) are supported.
Developers still need to check availability for each argument
separately.

Proposed Solution
This proposal is to add two functions in Nan which would make it
easier to marshal types between JavaScript and C++, so that developers
won't need to re-implement their JS<->C++ bindings.
The APIs are defined below:
// Currently Supported template types:
//   * basic native types: int32, int64, bool, double, uint32
//   * std::string
//   * std::vector<T>
// We could provide some interfaces allowing developers to support
their own C++ types.
//
// Convert V8's arguments to C++ native types. Returns false if fails.
//
// An example of a JS function "add(a, b)" which takes two Integer
parameters.
//   NAN_METHOD(Add) {
//     int a, b;
//     if (FromV8Argument(info, &a, &b)) {
//       // Do anything with a, b.
//     }
//   }
template<typename... Args>
bool FromV8Argument(Nan::NAN_METHOD_ARGS_TYPE v8_args, Args*... args);

// Convert the supported C++ native types to V8 types.
// Similar to NAN::New(), but it supports more types.
template<typename T>
v8::Local<v8::Value> ToV8Value(T value);

With the help of these two functions, writing JS bindings could be
easier, and most of the messy stuff of converting V8 types to C++ types
is done automatically and carefully.
The original code without using FromV8Argument:
NAN_METHOD(add) {
 if (!info[0]->IsNumber() || !info[1]->IsNumber()) {
   Nan::ThrowTypeError("Wrong arguments");
   return;
 }
 double arg0 = info[0]->NumberValue();
 double arg1 = info[1]->NumberValue();
 info.GetReturnValue().Set(Nan::New(arg0 + arg1));
}

The code using FromV8Argument:
NAN_METHOD(add) {
 double arg0, arg1;
 if (!Nan::FromV8Arguments(info, &arg0, &arg1)) {
   Nan::ThrowTypeError("Wrong arguments");
   return;
 }
 info.GetReturnValue().Set(Nan::New(arg0 + arg1));
}

Again, suggestions are welcome.


Update cookie-validity-update.md A.js Cookie methods

Proposed changes
Customers and Segmenters are often unaware of the native Analytics.js cookie methods that allow you to access client-side cookies and get their values returned. This is very useful when you need to access a certain cookie to pass from client-side to server-side.
Added the methods at the bottom of the Client-side persistence in Analytics.js doc. Priscila (PTL) proposed in the slack thread that it should go on this page.
Merge timing

ASAP once approved?

Related issues (optional)
Slack Thread : https://twilio.slack.com/archives/C9Z93GW76/p1675992398931969
Zendesk Ticket : https://segment.zendesk.com/agent/tickets/509663

This content has been approved by Team-1's PTL Priscila Giatti

@stayseesong the table doesn't look the best on the actual doc and doesn't allow you to scroll left-right to see the entire table. Is there anything that can be for this?


System Instructions isn't triggering

Description of the bug:
I am initializing the Model to have system instructions, although when I send in the requests it's not regerstring the system instructions and the model is replying as if its not receiving them.
here is how I am initializing in the code
 init() {       model = GenerativeModel(name: "gemini-1.5-pro-latest", apiKey: APIKey.default, systemInstruction: .init(parts: [.text("Your name is steve a nome")]))     chat = model.startChat()   }
Actual vs expected behavior:
No response
Any other information you'd like to share?
No response

Hi @MikesHorcrux, system instructions require Gemini 1.5 which is only available v1beta version of the API. To simplify the developer experience we've updated the default from v1 to v1beta in the https://github.com/google-gemini/generative-ai-swift/releases/tag/0.5.0 (just launched this morning). I'd highly recommend upgrading to 0.5.0 (though there is a workaround in https://github.com/google-gemini/generative-ai-swift/issues/128#issuecomment-2048461216 to set v1beta on older versions of the SDK).
I was able to run your code snippet after updating to 0.5.0 and the system instruction altered the response:
let model = GenerativeModel(
  name: "gemini-1.5-pro-latest",
  apiKey: apiKey,
  systemInstruction: .init(parts: [.text("Your name is steve a nome")])
)
let chat = model.startChat()
let response = try await chat.sendMessage("Why is the sky blue?")
guard let text = response.text else { fatalError() }
print(text)

As a gnome named Steve, I can tell you that the sky is blue because of something called Rayleigh scattering. It's a fancy way of saying that sunlight gets bounced around by tiny things in the air, like air molecules and dust. Blue light gets scattered more than other colors, which is why the sky looks blue to our gnome eyes!

Hope this helps! Closing the issue now but feel free to re-open if you're still seeing the issue.


Change how external contacts are defined, loaded, and stored from configuration files

I noticed that there is a bug in the RDE related to how I define external contacts and fts in the configuration file, both for the model loading and for the ukf config. Basically, if the contact and the ft are in the same frame and have the same name, we have a segmentation fault as the variable handler stores the variable only once due to the same name.
I have to change this behavior. The idea is to use the name variable as the definition of the contact and the frame as the definition of the frame where the contact happens. Same for the fts. In this way, the name can be chosen by the user, while the frame has to be contained in the urdf. At this point, to include a contact in the UKF state, we can use the name instead of the frame as we were doing before. The same can be done for the fts. This solution allows having different names in the state variable handler even in the case the ft and the contact share the same frame.
Once I implement this new logic, I'll open a PR.
cc @GiulioRomualdi

Associated PR https://github.com/ami-iit/bipedal-locomotion-framework/pull/759.

PR merged. Closing.


Enhancement - Classes in the same namespace are still automatically imported.

Hello
When using autocompletion for class names and confirming, a use statement is automatically appended at the top of the file. This is a neat feature, it appears however that this also happens if the class to import is in the same namespace as the file being edited, in which case the use statement is not necessary.
Thanks in advance

Hello,
Need to discuss, I'm not a fan of that. Why ? Because when you're refactoring your class, you need to add the missing uses if you don't add them everytime.
If more people want this feature, I will probably add it, but I'm cons.
:-1:

What you say does indeed make sense. It is not something I desperately need or want, perhaps this is something for the further future. (Perhaps a checkbox in the settings screen?)


Tag and release pypdfium2 fork of ctypesgen

We would like to build pypdfium2 with a custom build of libpdfium in an offline environment. It would make our lives easier if you could tag and release the current head of main on GitHub. We prefer to use released versions and like to avoid "random" git commits in our build pipeline.
You get extra kudos if you push your fork to PyPI, too.
Some ideas
pypdfium2-ctypesgen would be logical name for this fork of ctypesgen.
You could either stick to the version schema of upstream and release the package as 1.2.0. Or you could break with upstream versioning and align the version numbers with pypdfium2. The current HEAD of main is used for pypdfium2 4.30.0, so you could release the package as pypdfium2_ctypesgen-4.30.0.

Wow, I didn't expect a CPython dev to be interested in my amateur projects!
I'm glad the code is being used, but sorry to disappoint: I don't plan to make releases of pypdfium2-ctypesgen or publish it to PyPI, and I'm reluctant to rename the package.
This project isn't meant for a release-based development model. The intent is just to have our pypdfium2 and ctypesgen codebases coupled together to facilitate cooperative evolution (i.e. the HEAD of pypdfium2 should automatically point to the HEAD of ctypesgen).
Prefer to see this as an external development branch of ctypesgen, or as a setup-time utility module of pypdfium2, rather than as real own project.

It would make our lives easier if you could tag and release the current head of main on GitHub.

Note that our sdist releases auto-pin pypdfium2-ctypesgen to the current HEAD at compile time, so if you install from the sdist you should not have to do any tricky determining which ctypesgen commit a pypdfium2 release was built with.
That is, as long as you don't need to install with caller-provided data files. Otherwise, ctypesgen has to be managed on the caller side. I gather the main reason for downstreams to use that is currently offline setup, which is a bit unfortunate, as that could theoretically be done with an internal ctypesgen call.
We might be able to add new setup capabilities to better handle this case. Like taking a full pdfium version (to avoid web calls to resolve it from the build cipher), a path to custom pdfium headers (e.g. $PDFIUM_HEADERS), and adding new system-path: and bundle: strategies to $PDFIUM_PLATFORM.

The reason for my interest in your project is a bit ... complicated. :) I have filed the ticket as an engineer at Red Hat.
Long story short, pypdfium2 is a dependency of the IBM Deep Search package Docling, which itself is becoming a dependency of InstructLab, which is the core of RHEL AI. For various reasons like supply chain security and SBOM, we are not using any Python wheels and other binaries. Instead we build everything from upstream sources, either as RPMs or as Python wheels. The Python wheels span a range of trivial, pure Python packages to PyTorch builds for different hardware accelerators like AMD ROCm and NVIDIA CUDA.
We have created Fromager pipeline, which resolves dependencies, retrieves sdists, and then build each wheel in an isolated virtual env without network access. All build time dependencies must be resolved and pre-fetched before a build. The approach works out of the box for most Python packages, but there are some oddballs that download code at build time. (CMake FetchContent is popular in the AI space.) In general, we don't bother upstream projects with our downstream policies. We only engage with upstream, when we think that a change is going to help upstream and the broader Python community in the long run.
pypdfium2 is on the oddballs. We are building our own libpdfium.so, so we need to build a custom pypdfium2. Since we install all build-time dependencies offline, PDFIUM_PLATFORM="system:$PDFIUM_VER" does not work for us out of the box. The setting triggers a download of this fork at build time.
I see an opportunity to make everybody's life a bit easier by releasing and publishing this fork under a new name on PyPI.

release the fork as pypdfium2_ctypesgen on PyPI.
change https://github.com/pypdfium2-team/pypdfium2/blob/4.30.0/pyproject.toml#L10 to install pypdfium2_ctypesgen

Then you can remove the complicated ctypesgen git fetch code from https://github.com/pypdfium2-team/pypdfium2/tree/4.30.0/setupsrc/pypdfium2_setup and replace it with a simple import ctypesgen + importlib.metadata.metadata("pypydfium2_ctypesgen") to verify that the correct fork is installed. For local development and testing, you can still install ctypesgen locally in a virtual env.
If you are interested, then I can work with you and create PRs. If you don't think it's worth your while, then that's also okay. We can work around the issues internally.

Since we install all build-time dependencies offline, PDFIUM_PLATFORM="system:$PDFIUM_VER" does not work for us out of the box. The setting triggers a download of this fork at build time.
Then you can remove the complicated ctypesgen git fetch code from https://github.com/pypdfium2-team/pypdfium2/tree/4.30.0/setupsrc/pypdfium2_setup and replace it with a simple import ctypesgen + importlib.metadata.metadata("pypydfium2_ctypesgen") to verify that the correct fork is installed.

This is a bit imprecise: The download of this fork is triggered through standard pyproject.toml dependency means, which you can prevent by passing --no-build-isolation/--no-isolation to pip/build, and instead prepare ctypesgen on the caller side as you like due to security policies etc. This is not related to a PDFIUM_PLATFORM setting, and no code in pypdfium2_setup clones or installs ctypesgen. (Only for creating the sdist itself, there is a web call to get the ctypesgen HEAD commit hash for pinning, and some tmp editing of pyproject.toml, which I have to admit is indeed a bit dirty.)
However, another web call that remains is resolving the $PDFIUM_VER short cipher to a full version (e.g. 6611 -> 128.0.6611.0). This is an issue that I intend to address, as stated above (in the meantime, use caller-provided data files).
The only remaining problem I see then is figuring out the right ctypesgen commit hash for a pypdfium2 release on your side. Parsing it out of the sdist's pyproject.toml shouldn't be too difficult. If you feel like this should be made more accessible, I could add the hash e.g. to autorelease/record.json or to the auto-generated changelog headers.
While I'm interested in cooperating with downstreams to improve the offline installs situation, I do not want to move away from "smart" branch tracking. The release-free model is quite intentional. If people install from pypdfium2's git main, I want them to also get latest ctypesgen git by default, not just a snapshot from the previous pypdfium2 release date. One reason is to avoid breakage when we make API changes but have not released them yet. I do not want to be forced to make releases every time we change the API, as we tend to experiment (consider the kind of changes that are made one day and re-changed the other day across both codebases).
Even historically, when we didn't have the fork yet, pypdfium2 was tracking ctypesgen git due to lack of recent releases at that time. See also https://github.com/ctypesgen/ctypesgen/issues/129 and https://github.com/ctypesgen/ctypesgen/issues/201 for background.
My concern here is that you want us to put pypdfium2 information into ctypesgen, by tagging/releasing in sync with pypdfium2. However, ctypesgen is technically an independent project, and your suggestion would fall apart if this work were ever upstreamed and we still wanted to track a branch, as ctypesgen cannot make its release schedule or tagging dependent on a downstream project. So I think this is not logically the right solution.
PS: Before you spend time on PRs, I think it's best if we first agree on what to change, and some changes I might prefer to do personally due to being fussy about code style.


nanmean warnings

@eageo this is in reference to PR #29
There are numerous warnings in your Jupyter notebook:
<ipython-input-8-d0ab17fd4069>:60: RuntimeWarning: Mean of empty slice
  mean_ndvi = np.nanmean(ndvi_clean)

Perhaps related to using np to take the mean of an xr DataArray?

I noted this when I ran the code. I will try to sort out another way to get the mean ndvi. Thanks

@eculler I have kind of hit a wall with this and have just chosen to suppress the warnings in the np.nanmean block of code
From my research on this topic, I have found the mean of smpty slice warning to be a common issue with no actual consequence while using np.nanmean (see issue and stack overflow links below) in which much of the online advice (albeit 3-4 years old) is to simply ignore the warnings. This all seems way beyond my comfort level, but it appears it has something to do with the imbedded dask.array function that xarray uses.
https://github.com/dask/dask/issues/3245#issue-302182457
As suggested in this thread below, I used the code
with warnings.catch_warnings():     warnings.simplefilter("ignore", category=RuntimeWarning)
to suppress warnings for the block that runs the mean of empty slices
https://stackoverflow.com/questions/29688168/mean-nanmean-and-warning-mean-of-empty-slice
Hope that is acceptable.


Fix few possible shader generation issues

See few examples here: https://github.com/playcanvas/engine/pull/5090
related report: https://forum.playcanvas.com/t/what-is-this-error/29834

closing, fixed now


Support for Element Modifiers

It seems Element Modifiers for typography is not supported. Just wanted to mention this in case the rewrite is still going on.

This has been implemented in the v1 release: https://twind.style/preset-typography


Not all string are translatable

For example Czech is fully translated yet some strings in the UI are still in English. (the menu on the left)
 $ msgfmt --statistics po/cs.po
269 translated messages.

Those strings are just not marked in the code as translatable, should be easy fix.
I quickly went through the plugin and this was the only place I could find. It might be worth though if someone from the team would also give this a try as I am not so familiar with the plugin to know all the places where I can get to.

I reviewed the UI for missing translations. These are strings that are not tagged for translation:

In Create Image wizard

"Blueprint" (as shown in the example above)
All steps in the wizard navigation (as shown in the example above)
When RHEL for Edge is selected:
"rhel/8/$ARCH/edge is the default, where $ARCH is determined by the host machine"

In Edit Blueprint page > Available Components section > Pagination

"# per page" - It looks like the titles prop is where this string would be provided to the Pagination component.
Note that "#-# of #" is currently not translatable in the PF4 component.

@jgiardino I am updating the translations per this issue.

When RHEL for Edge is selected:
"rhel/8/$ARCH/edge is the default, where $ARCH is determined by the host machine"

I believe this string is marked for translation but has not yet been translated.

All possible strings are currently translated.


Added semver to ash and winit versions

Just thought I'd add this since I forked and am using my own version, but might as well do this and let you put it on crates.io 💃

Those version constraints are already interpreted that way by default; this commit is a no-op. See https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html.

d'oh


Repo manager accepts blank name

Although checks are in place to keep the inferred names from being empty, it appears one can manually set the name to be empty.
python repo_dev.py add-readgroupset l a ga4gh-example-data/HG00096.bam --name ""

Good catch. We should add a method to SqlDataRepository called checkNameFormat (or something similar) that is called during the insertX methods.

This could probably use some design up-front.  We're basically talking about adding a object validation layer before DB insertion.
In fact it might be good to refactor the data repo insertion code to make it more OO.  We could have some logic like:
errors = object.validate()
if not errors.isEmpty():
    sql = object.getSql()
    args = object.getArguments()
    try:
        cursor.execute(sql, args)
    except whateverException as exception:
        object.handleInsertionException(exception)
else:
    raise dbObjectValidationFailedException(object, errors)

Or some variation of that in the datarepo code.  Then we'd have an object hierarchy of DbObject where each subclass would implement validate, getSql, etc.
Thoughts?

This looks like a nice pattern @dcolligan, and I agree that we should try to add more structure to the admin process soon. However, I would like to build out the admin features more before we stand back and refactor/redesign things. The code is still reasonably clear, and there's a lot of features yet to go in. I'm wary of trying to do too much design before the full feature set has been fleshed out.
Plus, there's a big refactor in the pipelines changing the object ID structure, as well as using SQL to directly answer queries. I think it would be good to get these out of the way before we return to the insertion logic design.
How does this sound?

It's not a solution I'm thrilled about, but if you think a refactor would hamper existing/future efforts, then I suppose I can pepper the code with checkNameFormat calls...


NFT-333: Resevoir for mainnet collection stats

@adamgobes one minor thing, there's a lot of resevoir vs reservoir in this. Prob worth correcting now


Cannot reference in a browser script

I cannot access the typings of this file in the browser because of the module.
If I do an import, it requires me to use requirejs which I want to compile into a single file. I took off the module statement for a temporary fix but I cannot seem to find anything on the internet to just reference the file at compile time and not runtime.
Thanks,
Jason

realized this isn't for the browser...


argo workflow demo site not working 503

https://workflows.apps.argoproj.io/workflows/argo

Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

E1110 15:38:38.587896       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.5/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:argo:argo-server" cannot list resource "secrets" in API group "" at the cluster scope
72
E1110 15:38:45.482185       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.5/tools/cache/reflector.go:167: Failed to watch *v1.ServiceAccount: failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:argo:argo-server" cannot list resource "serviceaccounts" in API group "" at the cluster scope
71
time="2021-11-10T15:38:48.673Z" level=info msg="Alloc=4941 TotalAlloc=378830 Sys=20049 NumGC=3181 Goroutines=17"

Two fixes needed:
https://github.com/argoproj/argoproj-deployments/commit/2b0ded53951a5a850287b5c271f7d481309f0fd4
https://github.com/argoproj/argoproj-deployments/commit/f707eae22847380b3c143c3fa490bd7a2a778014


ASPL-532 (#111)

ASPL-532
Better message to be conveyed in Jenkins console when traffic file > 200Mb is uploaded

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Double images fix

There is a known issue with gatsby-plugin-mdx and gatsby-remark-issue which results in double images with the config described in the course.  This can be fixed by adding gatsby-remark-images as plugin of  gatsby-plugin-mxd.
    {
      resolve: 'gatsby-plugin-mdx',
      options: {
        defaultLayouts: {
          default: require.resolve('./src/components/layout.js'),
        },
        gatsbyRemarkPlugins: ['gatsby-remark-images'],
        plugins: ['gatsby-remark-images'],
      },
    },

This was added as an annotation, but is now also added as a code snippet to the course errata.


cbindgen: Disable tests rather than abusing RUSTC_BOOTSTRAP

Motivation for this change
Since https://github.com/NixOS/nixpkgs/pull/64733, the packaging of cbindgen uses the RUSTC_BOOTSTRAP environment variable.
https://github.com/NixOS/nixpkgs/blob/681d3c260d2947e21ccb4122e27b4404d3722b58/pkgs/development/tools/rust/cbindgen/default.nix#L19
Background on Rust
The Rust programming language has a concept of “unstable features”. They are APIs, language features, or tool parameters whose designed is not completed yet. They may still change in incompatible way, potentially breaking users relying on them.
Having them at all in the implementation allows experimenting and iterating, rather than freezing a design while it’s still only theoretical. It is important for the long-term health of the language that it can keep experimenting this way.
To avoid unstable features becoming de-facto stable (if enough users start depending on them that there is pressure not to change them and keep an unfinished design), these features are not available in normal releases of the compiler, only on Nightly versions. And they require explicit opt-in.
rustc is a bootstrapping compiler: it is written in Rust and relies on some unstable features. In order to make it possible to build each release with the previous one, rustc’s build system uses the RUSTC_BOOTSTRAP environment variable to unlock unstable features despite using a bootstrap version of rustc that normally does not allow them. There is non-trivial work involved to keep all this working when breaking changes are made to unstable features.
Using RUSTC_BOOTSTRAP in another project technically allows using unstable features with a compiler that normally doesn’t, but this environment variable was never intended for anything other than building rustc itself.
Abusing RUSTC_BOOTSTRAP is harmful
By relying on it anyway, NixOS contributes to normalize other projects relying on it and counting on unfinished features not changing. If enough projects adopt this strategy, this puts pressure on the Rust team to declare unfinished features to be de-facto stable, and robs them of their opportunity for experimentation.
Please don’t contribute to spoiling it for everybody.
Background on cbindgen
An optional feature of cbindgen (expanding macros) involve shelling out to rustc with some unstable options. Therefore this requires Rust Nightly. Building Firefox requires cbindgen, but not macro-expansion feature. It is a goal of Firefox that it can be built with stable rustc. I expect that Firefox is the motivation to package cbindgen in NixOS.
Some of cbindgen’s own test suite exercise the macro-expansion feature, and therefore fails when run on stable rustc: https://github.com/eqrion/cbindgen/issues/338. But it’s only some tests.
This PR
I have reproduced https://github.com/eqrion/cbindgen/issues/338 by running cargo test  in cbindgen, and verified that cargo build works correctly. I am told that doCheck = false is the way to make the nix packaging disable cargo test for a Rust package, but I have not tried it.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Would running cargo build without RUSTC_BOOTSTRAP=1 but running cargo test with it be acceptable ? This way the build artifacts are built with stable rust but we still run tests.

That would be no different from https://github.com/NixOS/nixpkgs/pull/64733.
It’s not about the final executable being “pure”. It’s about creating a situation where your process is gonna start failing when you update your toolchain to a version where a breaking change was made to an unstable feature, and normalizing other projects doing the same.
It’s an important goal of Rust that updating from one toolchain to the next on the Stable release channel be as painless as possible. There is a strong promise of stability, but it needs boundaries to be viable. Experimental stuff only being available when one knowingly opts into the Nightly channel (or compiles their own rustc) is part of that story.

Just briefly read through this.
@SimonSapin Thank you for taking the time and looking into this! Very appreciated. I think FireFox was not the driving force behind having it in nixpkgs but one of the main consumers these days :-)
Is it possible to introduce (test) flags in cbindgen to disable tests that need nightly features? Maybe even detect if you are running on a nightly rustc version.
That way we could still run most of the tests and catch errors but would not need to resort to these kinds of hacks.

Note that RUSTC_BOOTSTRAP is already abused elsewhere, and that should probably be fixed too.
https://github.com/NixOS/nixpkgs/blob/fd4a976dc3e86ccc7535ddf1cb227be7d203c03c/pkgs/tools/security/bitwarden_rs/default.nix#L17

@andir When passing a -- argument to cargo test, the rest of the arguments are passed through to the test executable. So yes it’s possible to add runtime flags to the test suite. I’m not planning to work on that though.

@andir When passing a -- argument to cargo test, the rest of the arguments are passed through to the test executable. So yes it’s possible to add runtime flags to the test suite. I’m not planning to work on that though.

I started implementing checkFlags for buildRustPackage so we can restrict the tests that will be run.
My current WIP can be seen at https://github.com/NixOS/nixpkgs/compare/master...andir:rust-checkflags?expand=1
Going this route is probably better then disabling tests and/or requiring RUSTC_BOOTSTRAP.

What's the status here? If I understood correctly, the usage in cbindgen is resolved following #70259, but 10 other packages still use RUSTC_BOOTSTRAP.

Someone has to put in the time to change all the other packages. I will most likely not invest that time.

It’s not about the final executable being “pure”. It’s about creating a situation where your process is gonna start failing when you update your toolchain to a version where a breaking change was made to an unstable feature, and normalizing other projects doing the same.

I think that this statement, while true, is missing the fact that nixpkgs is not a “normal” package repository: we statically know with which version of rustc the packages are built, and people overriding it already know what they are doing. As such, normal users cannot have such breakage happen regardless of whether new versions of rustc break formerly-valid tests without us knowing about it (via hydra failing CI for these packages).
This being said, I would love for RUSTC_BOOTSTRAP to not have to be used. However, disabling tests altogether is, for the nixpkgs project itself, a bigger issue than using RUSTC_BOOTSTRAP, due to its aforementioned particularity, as we have a choice between something we know builds and works, and something we know builds and can only hope works.
I think the case presented in this PR is a great one, and I would love for us to be able to get rid of RUSTC_BOOTSTRAP (eg. by patching all upstream software so it no longer requires nightly, or for cases where it's needed only for tests by introducing a feature flag for nightly-requiring tests), but it is going to be a lot more work.
As such, in the current state of things, I think the best we can do would be to add comments around uses of RUSTC_BOOTSTRAP that explicitly state that no other project should use it. And then, as upstream projects become more compatible with stable Rust, to slowly remove these points of use.
Another solution would be to package one instance of Rust nightly, but this has already been decided against due to the update churn it implies. I'm not sure whether this issue gives enough new information to relitigate this previous choice (by eg. keeping a less-regularly updated nightly), but if people think this should happen I think it'd deserve some discussion on the Discourse first and then either in an issue or in a PR that'd implement such an addition.
Anyway, seeing as this PR cannot be landed as is, I am going to close it so it reduces our unactionable PR backlog. Sorry about it, and hope the suggested alternative roads will inspire some contributions! And thank you everyone for the feedback :)

It’s not about the final executable being “pure”. It’s about creating a situation where your process is gonna start failing when you update your toolchain to a version where a breaking change was made to an unstable feature, and normalizing other projects doing the same.

I think that this statement, while true, is missing the fact that nixpkgs is not a “normal” package repository: we statically know with which version of rustc the packages are built, and people overriding it already know what they are doing. As such, normal users cannot have such breakage happen regardless of whether new versions of rustc break formerly-valid tests without us knowing about it (via hydra failing CI for these packages).
This being said, I would love for RUSTC_BOOTSTRAP to not have to be used. However, disabling tests altogether is, for the nixpkgs project itself, a bigger issue than using RUSTC_BOOTSTRAP, due to its aforementioned particularity, as we have a choice between something we know builds and works, and something we know builds and can only hope works.
I think the case presented in this PR is a great one, and I would love for us to be able to get rid of RUSTC_BOOTSTRAP (eg. by patching all upstream software so it no longer requires nightly, or for cases where it's needed only for tests by introducing a feature flag for nightly-requiring tests), but it is going to be a lot more work.
As such, in the current state of things, I think the best we can do would be to add comments around uses of RUSTC_BOOTSTRAP that explicitly state that no other project should use it. And then, as upstream projects become more compatible with stable Rust, to slowly remove these points of use.
Another solution would be to package one instance of Rust nightly, but this has already been decided against due to the update churn it implies. I'm not sure whether this issue gives enough new information to relitigate this previous choice (by eg. keeping a less-regularly updated nightly), but if people think this should happen I think it'd deserve some discussion on the Discourse first and then either in an issue or in a PR that'd implement such an addition.
Anyway, seeing as this PR cannot be landed as is, I am going to close it so it reduces our unactionable PR backlog. Sorry about it, and hope the suggested alternative roads will inspire some contributions! And thank you everyone for the feedback :)


Variables and macros feature request.

Hi!
Would it be possible to add system (username, date, time,) and user-defined variables and basic macros support (Javascript or other scripting language) to Beeftext?
Keep up your excellent work!

Please close this as an non-issue. I just figured out this was already there.
Keep up your excellent work!

Non-issue. Feature request on something already there.

As you already noticed, you can already retrieve environment variables. In addition to that, Beeftext v8.0 will introduce a new #{powershell:} variable that will let you run a PowerShell script and capture its output.


Local Storage Needs to be cleared

I was on 1.2.0 and updated to 1.2.1.
The extension wouldn't load because I had some cards still in local storage. I had clear my local storage to get the extension to load because it couldn't parse the old local storage.
I lost the error (something along the lines of C token couldn't parse JSON) but it happened on init() and was fixed after I cleared the old local storage.

This has been fixed and will be included in the next patch

This has been fixed and will be included in the next patch


ARROW-8060: [Python] Make dataset Expression objects serializable

The Scalar are not properly exposed yet and the C++ implementation will change soon so while the serialization will work, we need to refactor this code.

Is there a JIRA about defining a e.g. Flatbuffers-based serialization protocol (probably better than Protobuf so we avoid having to introduce a library dependency) for expressions? Seems like it might provide a more scalable solution for serialization and could eventually succeed the strategy here

Created JIRA https://issues.apache.org/jira/browse/ARROW-8214


Lutron Occupancy Sensor Not Detected

Hi Upsert,
I am not sure if this is a true issue yet, but I thought I would log it anyway.  I just added a  a ceiling mount occupancy sensor (Lutron LRF2-OCR2B-P-WH) to my radioRA 2 Select main repeater.  Currently that is ONLY thing connected to the system.  I have nothing else , no switches, nothing but the sensor, and perhaps that is the issue (i am still waiting for my switches to arrive).  The sensor shows up in the app and is assign to a room, but it is NOT set to control anything yet, obviously.
My debug logs show that the component sees the bridge and attempts to parse the devices on that bridge, but comes up with none (see log snippet).  The integration report shows the sensor as ID 2 and has a name and an area, but it does not have any "buttons" which i think is what is causing the issue. casetify.py seems to need to have the words "buttons" in the if else statement that identify the bridge, light/switchs, and remote.  Might simply need a catch for the occupancy sensors, which don't see to have buttons.  I've included my integration report as well, so you can see how the sensors are formatted.
I'll report back once I have a switch or two connected.  Maybe buttons will show up!
Thanks again for this awesome component!
report.txt
ha_log_snippet.txt

Interesting. Maybe if it is not under Zones and has no Buttons, assume an occupancy sensor. Worse case I will have to do a binary_sensor: [ 1, 2, 3] type of config override.
I am still waiting on mine to ship, but once I get one I plan to integrate it as a new platform for the component. Should be a Binary Sensor of type 'occupancy'.

Sadly, I don't think I will be able to support the occupancy sensors as they do not output anything to telnet interface. I initially thought it was working, but it turns out I was just seeing the light level of the fixture associated with the occupancy sensor, i.e. the light in the assigned area.
Unless they update the telnet interface to support DEVICE output for Occupied/Unoccupied (page 140 of the Integration Protocol), I'm not sure what else can be done. Maybe with the increasing use of Ra2 Select, the official integration partners will press for this to be added. The addition of occupancy sensors to Ra2 Select is a relatively new feature (March 2018).

Bummer. :(  Looks like these guys are trying to figure it out too.  link
One guy said he could see the device output using a serial interface

Yeah that makes sense. The last two posts are from people with the full RadioRA 2 Main Repeater and not RA2 Select. The RadioRA 2 Main Repeater has both a telnet and serial line interface and no doubt a full implementation of the Lutron Integration Protocol.
RA2 Select is based on their newer platform so I am still hoping one of their official integration partners like Control4 will want occupancy sensor triggers to work in their driver and Lutron will add it. So far they have announced Control4 support here and I found a driver here which definitely uses Telnet for integration. The driver documentation is dated March 30, 2018 so maybe once more integrators start using it, they will feedback to Lutron about the missing output for occupancy sensor events.
I will keep an eye out for firmware updates to the bridge. Their last update was April 16, 2018.

Yup, let's hope they push lutron to add it.  The occupancy sensors are really nice and super responsive.  I would love to use more of them.

Firmware 06.05.14f000 is out and sadly it does not appear to change the telnet interface to have output events for occupancy sensors on RA2 Select.
Maybe we can appeal to Lutron engineers like @Aditya90 on Github to sneak the feature request into their backlog? 😸

Firmware 06.07.07f000 is out. No changes found with respect to the telnet interface.

Firmware 06.08.04f000 is out. No changes for the telnet interface. Probably preparing for Caseta fan controller I spotted a couple of weeks ago.

Firmware 06.09.01f000 is out. No changes found with respect to the telnet interface.

Firmware 06.09.05f000 is out. No changes to occupancy sensors in RA2 Select. Likely the firmware required to bump Caseta up to 75 devices.

Probably adding this too: https://fccid.io/JPZ0122. A new Radio RA2 / RA2 Select dimmer. Model RRD-PRO, 120 V, 250 W LED/CFL load. Looks like the two-wire RRD-6CL but with 250 W LED and phase adaptive like the RRD-6NA.
Apparently they have an Advanced Programming Mode too where you can adjust the fade times, delayed fade to off, disable the LEDs and adjust trim. Wonder if they will add some of that to the mobile app. It already has trim.

Firmware 06.11.04f000 dropped. No change.

Firmware 06.12.01f000 is out. I expect it will be for the motion sensor and repeater they just added to Caseta. Unfortunately that does not mean motion sensors over telnet have changed.

Firmware 06.12.01f000 is out. I expect it will be for the motion sensor and repeater they just added to Caseta. Unfortunately that does not mean motion sensors over telnet have changed.

Well fingers crossed that they add it!  In any event it's good to see them adding motion sensors!

@upsert Where do you see these new firmware releases? And are they applied automatically to our hubs? (Sorry to go a little off-topic, I’ve been using RA2 Select for a couple years and have always wondered this and never found anything.)

@zwily The hub broadcasts a status including the firmware version using mDNS and DNS-SD and I periodically use the Bonjour Browser to check it.

@upsert have you got your hands on a motion sensor yet and tested it out?

@bdubsw Yes, since 2018 on the Ra2 Select line. They are using the same hardware for Caseta.

Was just curious if the PD-OSENS-WH is exposed over telnet like everything else and sends motion events

PD-OSENS-WH is NOT exposed over telnet

I've been researching this same issue for another application and stumbled on this bug.  Perhaps my info may help you.
I believe the answer here is in the JSON side of the protocol.  Please look at https://github.com/hassio-addons/addon-lutron-cert to see how to create the necessary 3 certs/keys.  You can validate that you can receive a stream with those files using "curl -k --cacert caseta-bridge.crt --key caseta.key --cert caseta.crt https://bridge-ip:8081"
From there I would suggest looking at https://github.com/gurumitts/pylutron-caseta/commit/c14487c7d8a71af36b5958d324e46b5ca0509215 for examples of JSON you can send to query about the area.
My theory, which I haven't been able to validate/test, is that you could regularly query through the JSON the occupancy state of a room/area.  Everything I can tell from my testing makes it seem like the occupancy sensor works like a pico remote as opposed to an actual switch.
I hope this helps.

It may be possible over HTTPS/JSON, but I will leave that to the Caseta integration to reverse engineer the protocol.
This integration is only for the Telnet-based Lutron Integration Protocol. The hope is that since Telnet is the method for professionals to integrate with Caseta Pro and RA2 Select (and virtually all the Lutron main product lines), that they will eventually add it to the Telnet protocol and bring it up to par with RadioRa2.
On a related note, firmware 08.01.12f000 is out and sadly, it does not change the Telnet output.

I guess it got added to the other protocol, wonder why Telnet is (still) absent.

Firmware 08.01.13f000 is out. No changes to the telnet interface.

I have a Caseta Occupancy Sensor, too, and was hoping it would be available :(
I notice that the default Caseta integration says that it supports occupancy sensors... but it doesn't give Pico events.
Since I have a non-pro hub as well, I suppose I can run both integrations with only the occupancy sensor as the only device connected to the one....

Oh, and for what it's worth... it does show up in the integration report.

Well, I think it’s a great idea. You can let your friend know that although I have many Lutron products (including as far back as RadioRa “Classic”) I can’t/won’t buy their occupancy sensors if they can’t be integrated. I have two useless sensors (the Caseta, and a RA2 one that I received as a promo…. Which is just enough different to require a whole new hub. ☹ )

Well, I will give it a shot. No promises of course, but it can’t hurt to ask.

Looks like there will be a new Ra2 Select and HomeWorks hub coming soon: https://fcc.report/FCC-ID/JPZ0129
Same hardware for both and each has a repeater variation. Bluetooth Low Energy (BLE), 802.15.4 wireless (likely for Ketra) and 431 MHz ClearConnect. Uses 48 Volt Power-over-Ethernet which would be nice to have on the current hubs. Model numbers HQP7-RF-2, HQREP-RF, RR-PROC3, and RR-REP3.

Looks like there will be a new Ra2 Select and HomeWorks hub coming soon: https://fcc.report/FCC-ID/JPZ0129
Same hardware for both and each has a repeater variation. Bluetooth Low Energy (BLE), 802.15.4 wireless (likely for Ketra) and 431 MHz ClearConnect. Uses 48 Volt Power-over-Ethernet which would be nice to have on the current hubs. Model numbers HQP7-RF-2, HQREP-RF, RR-PROC3, and RR-REP3.

Good find! Finally some love for RA2 Select! The label pics look like the existing "Clear Connect Gateway - Type X" that they have for Homeworks and Athena.
I bailed on Caseta for RA2 Select because it was missing Occupancy / Vacancy Sensors, had a low device limit (50 at the time), and didn't have the 15A appliance model. And soon it looks like Caseta will have all the things I left for, was starting to get a little disappointed that RA2 Select wasn't getting any new or cool features. Still holding out for them to unlock fade times, etc. (outside of scenes) and sensor change of state over telnet.

Good find! Finally some love for RA2 Select! The label pics look like the existing "Clear Connect Gateway - Type X" that they have for Homeworks and Athena.
I bailed on Caseta for RA2 Select because it was missing Occupancy / Vacancy Sensors, had a low device limit (50 at the time), and didn't have the 15A appliance model. And soon it looks like Caseta will have all the things I left for, was starting to get a little disappointed that RA2 Select wasn't getting any new or cool features. Still holding out for them to unlock fade times, etc. (outside of scenes) and sensor change of state over telnet.

The latest Lutron integration protocol guide has a lot of references to occupancy monitoring. Has it shown up on the Caseta Smart Bridge Pro yet? Rev. Dec 16
https://www.lutron.com/TechnicalDocumentLibrary/040249.pdf

The latest Lutron integration protocol guide has a lot of references to occupancy monitoring. Has it shown up on the Caseta Smart Bridge Pro yet? Rev. Dec 16
https://www.lutron.com/TechnicalDocumentLibrary/040249.pdf

Another note to follow up on @upsert's analysis, looks like the FCC specifically did Zigbee and BLE testing. Will have to wait until the full release, but maybe they'll be able to control Hue (like their Aurora dimmer)! I mean, I'm not holding my breath, but that'd be amazing. Give me Hue or Ketra or both!

Another curious device from Lutron:
https://fcc.report/FCC-ID/JPZ0136/
Bluetooth Low Energy Darter keypad. Described as a "... wireless remote keypad for installation into a standard WD wallbox and powered at 120-277V. Device does not have a load but provides wireless signals to a Lutron lighting control system." Operates at 2.4 GHz, but nothing at 431 MHz for ClearConnect. Model numbers look like Pico model numbers with 3 and 4-button variants but RRST and HRST prefix.

Looks like those devices are all for the newly launched RA3:
https://radiora3.lutron.com/


Reconsider how file formats are described here?

The way you intended the formats to be described is one file per format version. That'd work well, if the file formats were radically different, but they're not. Looking at w3i (Info), the file format evolved incrementally, where each version only added new fields in between, without major changes.
I really liked how Hodor described the formats here (in Russian). Open war3map.w3i as an example, additional fields of newer versions are conveniently described within IF VERSION > 28 tags, which makes multi-version support trivial when writing a parser.
Bringing everything to one file (as long as the formats stay roughly the same) has the advantage that all changes don't affect 2+ files. Further, while writing a parser it's practically zero effort to support multiple formats. IF-ELSE-PROFIT!
I'd rewrite the format descriptions later and make a PR if you agree. I would probably follow Hodor's approach too, its readability is miles ahead of this: https://github.com/stijnherfst/HiveWE/wiki/war3map.wtg-Triggers
Further resources:

https://xgm.guru/p/wc3/w3-file-format by Hodor from above
https://www.hiveworkshop.com/threads/parsing-metadata-from-w3m-w3x-w3n.322007/
@Drake53's .NET library: https://github.com/Drake53/War3Net/tree/master/src/War3Net.Build.Core (has EnemyPlayerPriorities described, last unknown thing in current Info)

Here's Hodor's article from XGM as a screenshot:

Thanks, I'll review the different formats and consider it. I think when I considered in the past I was concerned about the documents becoming messy with exceptions, and didn't consider the redundancy in the file versions to be a big deal. I'll think about it again.

Consider adding formal KaitaiStruct specifications https://kaitai.io/ for more structural exactness, parser generation and at some point also writer generation https://github.com/kaitai-io/kaitai_struct/issues/27 https://doc.kaitai.io/serialization.html
Web IDE: ide.kaitai.io
Here is a .ksy file for .w3i files I have begun to write:
meta:
  id: wc3_w3i
  file-extension: w3i
  endian: le
seq:
  - id: version
    type: u4
  - id: saves_amount
    type: u4
    if: version >= 16
  - id: editor_version
    type: u4
    if: version >= 16
  - id: game_version
    type: game_version
    if: version >= 27
  - id: map_name
    type: w3str
  - id: author
    type: w3str
  - id: recommended_players
    type: w3str
    if: version >= 8
  - id: unknown_v0_to_v3
    type: unknown_v0_to_v3
    if: version <= 3
  - id: unknown_v4_to_v8
    type: unknown_v4_to_v8
    if: version > 3 and version <= 8
  - id: unknown_string
    type: w3str
  - id: camera_bounds
    type: camera_bounds
  - id: margins
    type: margins
    if: version >= 14
  - id: map_width
    type: u4
    if: version >= 1
  - id: map_height
    type: u4
    if: version >= 1
  - id: unknown_v3_int1
    type: u4
    if: version >= 2 and version <= 8
  - id: flags
    type: flags
    if: version >= 2
  - id: tileset
    type: w3char
    if: version >= 8
  - id: loading_screen
    type: loading_screen
  - id: game_data_set
    type: u4
    if: version >= 17
  - id: unknown_path
    type: w3str
    if: version <= 17
  - id: prologue_screen
    type: prologue_screen
    if: version >= 13
  - id: fog
    type: fog
    if: version >= 19
  - id: global_weather_id
    type: u4
    if: version >= 21
  - id: sound_environment
    type: w3str
    if: version >= 22
  - id: light_environment
    type: w3char
    if: version >= 23
  - id: water_color
    type: water_color
    if: version >= 25
  - id: script_languageso
    type: u4
    if: version >= 28
  - id: supported_graphics_modes
    type: u4
    if: version >= 29
  - id: game_data_version
    type: u4
    if: version >= 30
  - id: players_chunk
    type: players_chunk
  - id: forces_chunk
    type: forces_chunk
  - id: upgrades_chunk
    type: upgrades_chunk
    if: version >= 6
  - id: techs_chunk
    type: techs_chunk
    if: version >= 7
  - id: random_unit_tables_chunk
    type: random_unit_tables_chunk
    if: version >= 12
  - id: random_item_tables_chunk
    type: random_item_tables_chunk
    if: version >= 24
  - id: script_language2
    type: u4
    if: version == 26 or version == 27

types:
  w3str:
    seq:
      - id: value
        type: str
        terminator: 0
        encoding: UTF-8
  w3char:
    seq:
      - id: value
        type: str
        encoding: UTF-8
        size: 1
  w3id:
    seq:
      - id: value
        type: str
        encoding: UTF-8
        size: 4
  point_2d:
    seq:
      - id: x
        type: f4
      - id: y
        type: f4
  game_version:
    seq:
      - id: major
        type: u4
      - id: minor
        type: u4
      - id: revision
        type: u4
      - id: build
        type: u4
  unknown_v0_to_v3:
    seq:
      - id: unknown_v0_int1
        type: u4
      - id: unknown_v0_int2
        type: u4
  unknown_v4_to_v8:
    seq:
      - id: unknown_v4_int1
        type: u4
      - id: unknown_v4_int2
        type: u4
      - id: unknown_v4_int3
        type: u4
      - id: unknown_v4_float1
        type: f4
      - id: unknown_v4_float2
        type: f4
      - id: unknown_v4_float3
        type: f4
  camera_bounds:
    seq:
      - id: camera_bound_left
        type: point_2d
      - id: camera_bound_bottom
        type: point_2d
      - id: camera_bound_right
        type: point_2d
      - id: camera_bound_top
        type: point_2d
  margins:
    seq:
    - id: left
      type: u4
    - id: right
      type: u4
    - id: bottom
      type: u4
    - id: top
      type: u4
  flags:
    seq:
      - id: hide_minimap_on_preview_screens
        type: b1
      - id: change_ally_priorities
        type: b1
      - id: melee
        type: b1
      - id: non_default_tileset_map_size_large_never_been_reduced_to_medium
        type: b1
      - id: unexplored_areas_partially_visible
        type: b1
      - id: fixed_player_parameters_for_custom_teams
        type: b1
      - id: use_custom_teams
        type: b1
      - id: use_custom_techs
        type: b1
      - id: use_custom_abilities
        type: b1
      - id: use_custom_upgrades
        type: b1
      - id: map_properties_menu_opened_at_least_once
        type: b1
      - id: show_water_waves_on_cliff_shores
        type: b1
      - id: show_water_waves_on_rolling_shores
        type: b1
      - id: use_terrain_fog
        type: b1
      - id: tft_required
        type: b1
      - id: item_classification
        type: b1
      - id: custom_water_tint_color
        type: b1
      - id: rest
        type: b1
        repeat: expr
        repeat-expr: 15
  loading_screen:
    seq:
      - id: loading_screen_index
        type: u4
        if: _root.version >= 17
      - id: custom_loading_screen_path
        type: w3str
        if: _root.version >= 10 and _root.version != 18 and _root.version != 19
      - id: loading_screen_text
        type: w3str
        if: _root.version >= 10
      - id: loading_screen_title
        type: w3str
        if: _root.version >= 11
      - id: loading_screen_subtitle
        type: w3str
        if: _root.version >= 11
  prologue_screen:
    seq:
      - id: prologue_screen_path
        type: w3str
        if: _root.version >= 18
      - id: prologue_screen_text
        type: w3str
      - id: prologue_screen_title
        type: w3str
      - id: prologue_screen_subtitle
        type: w3str
  fog:
    seq:
      - id: fog_type
        type: u4
      - id: fog_z_start
        type: f4
      - id: fog_z_end
        type: f4
      - id: fog_density
        type: f4
      - id: fog_color_red
        type: u1
      - id: fog_color_green
        type: u1
      - id: fog_color_blue
        type: u1
      - id: fog_color_alpha
        type: u1
  water_color:
    seq:
      - id: red
        type: u1
      - id: green
        type: u1
      - id: blue
        type: u1
      - id: alpha
        type: u1
  players_chunk:
    seq:
      - id: amount
        type: u4
      - id: player
        type: player
        repeat: expr
        repeat-expr: amount
  player:
    seq:
      - id: num
        type: u4
      - id: controller
        type: u4
      - id: race
        type: u4
      - id: fixed_position
        type: u4
      - id: nme
        type: w3str
      - id: position
        type: point_2d
      - id: ally_low_priority
        type: u4
        if: _root.version >= 5
      - id: ally_high_priority
        type: u4
        if: _root.version >= 5
      - id: enemy_low_priority
        type: u4
        if: _root.version >= 31
      - id: enemy_high_priority
        type: u4
        if: _root.version >= 31
  forces_chunk:
    seq:
      - id: amount
        type: u4
      - id: force
        type: force
        repeat: expr
        repeat-expr: amount
  force:
    seq:
      - id: flags
        type: u4
      - id: players
        type: u4
      - id: name
        type: w3str
  upgrades_chunk:
    seq:
      - id: amount
        type: u4
      - id: upgrade
        type: upgrade
        repeat: expr
        repeat-expr: amount
  upgrade:
    seq:
      - id: players
        type: u4
      - id: id
        type: w3id
      - id: level
        type: u4
      - id: availability
        type: u4
  techs_chunk:
    seq:
      - id: amount
        type: u4
      - id: tech
        type: tech
        repeat: expr
        repeat-expr: amount
  tech:
    seq:
      - id: players
        type: u4
      - id: id
        type: w3id
  random_unit_tables_chunk:
    seq:
      - id: amount
        type: u4
      - id: table
        type: random_unit_table
        repeat: expr
        repeat-expr: amount
  random_unit_table:
    seq:
      - id: num
        type: u4
      - id: name
        type: w3str
      - id: columns_amount
        type: u4
      - id: column_type
        type: u4
        repeat: expr
        repeat-expr: columns_amount
      - id: rows_amount
        type: u4
      - id: row
        type: random_unit_table_row
        repeat: expr
        repeat-expr: rows_amount
  random_unit_table_row:
    seq:
      - id: chance
        type: u4
        repeat: expr
        repeat-expr: _parent.columns_amount
  random_item_tables_chunk:
    seq:
      - id: amount
        type: u4
      - id: table
        type: random_item_table
        repeat: expr
        repeat-expr: amount
  random_item_table:
    seq:
      - id: num
        type: u4
      - id: name
        type: w3str
      - id: sets_amount
        type: u4
      - id: set
        type: item_set
        repeat: expr
        repeat-expr: sets_amount
  item_set:
    seq:
      - id: items_amount
        type: u4
      - id: item
        type: item
        repeat: expr
        repeat-expr: items_amount
  item:
    seq:
      - id: chance
        type: u4
      - id: id
        type: w3id

:cry: https://github.com/Luashine/wc3-file-formats
:cry: https://github.com/Luashine/wc3-file-formats/blob/b7f578771933a9f64c405ea9d7355513f80d6a51/w3i-finished-notpushed/WIP-stable.ksy how much time did you spend?

@Luashine ~4 hours, using https://github.com/ChiefOfGxBxL/WC3MapSpecification/blob/36c1abfe1d8a1d420e7be2b8d376f601924a3cb5/Info/0-31.md and wc3libs. It was an excuse to look into Kaitai Struct finally. But it's only a stub and was tested only on a single .w3i file. For example, all the string length limits are missing and enum mappings.

Very nice - I'll read up more about Kaitai on their website. Since Luashine's wc3-file-formats repo is mentioned here, I'll reach out to various people on The Hive (Discord) to see if we can consolidate efforts under a Hive GitHub organization. There's a bunch of duplication out there that we might be able to consolidate.

I often search by the file name or extension rather than "Units" or "Info". "Units" could also mean "war3map.w3u" or the unit slks. And I hope we will have more custom formats in the future. As folder organization, it is okay, but could the files be renamed or tagged otherwise, so it is easier to find them? Thus you can also more easily see what's still missing.


feat: use tl_expected of ros package

use tl_expected of ros package
depends on https://github.com/ros-perception/point_cloud_transport/pull/32

https://github.com/Mergifyio backport humble iron


Storing output in variable

Render function works great for most of the scenarios.
But sometimes there are requirements where we need to store the output in variable for further processing like Trimming text, getting plain text.
How do we store string in a variable instead of rendering?

Take a look at #14

@sendy34 Does #14 solve your issue? Can i close this?


Debugging the text search

Is there a way to debug why Zulip doesn't show any results when searching for anything?
https://github.com/galexrt/docker-zulip/issues/78

@galexrt this is almost certainly that the process_fts_updates job isn't running, since that's what maintains the search indices.

@timabbott Thanks for the answer!
I directly submitted a PR to make the user and dbname configurable too, especially for the Docker Zulip version, see #4205.


BorderSkin specific code

research way to move BorderSkin specific code out of WindowsHook as a library and put it in BorderSkin

research way to move BorderSkin specific code out of WindowsHook as a library and put it in BorderSkin

https://archive.org/stream/ebookSE18/4_E18_Math_djvu.txt


2.x/bugfix/integrate new handle destination

Rebase https://github.com/pypeclub/pype/pull/480 onto release/2.12.0-rc

Can we merge it?


Electron updates

Description of changes
Draft because: 26 branch will also get an update

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

It is a bit unrelated but since https://github.com/yu-re-ka/nixpkgs/commit/02266dc69eb5aa8f33df6b5ccbde1c128f64001c we rely on go for git-lfs through nix-prefetch-git. Do we want to avoid to not trigger a rebuild when go is being updated?

Built electron_28, electron_27 on x86_64-linux

Built electron_28 on x86_64-linux


M3-406 - Rebuild Linode Tests, Fixes for List linodes, env file documentation

Tagged rebuild.tsx with data-qa attributes
Updated max instances in wdio config to 2 (run two parallel chrome instances)
Added --dir flag to yarn e2e. Can now run yarn e2e --dir create and it will run all tests in e2e/specs/create/**/*.spec.js
Created page object for rebuild section of linode detail
Created rebuild spec
Added data-qa-title to List linodes landing (fixes breaking test)
Added conditional to check linode status prior to clicking the l linode action menu
documented: e2e test env variables
Added: example .env file and updated contributing.md to tell contributors about it!

Testing
yarn start
yarn selenium # in a new shell
yarn e2e # in a new shell

Beware the tests assume you have:

That at least one test linode exists on your test account
That you do not have 2FA enabled

Please rebase and squash duplicate commits.


Code not updated?

I am using version 1.3.0 from NPM, but the code on NPM seems to be much newer than the code in this repo. Is the code actually still maintained on Github?
Why I am asking, is that there seems to be a capability option missing from the Register Device node, namely power-consumption
I am trying to register a power-consumption device in a way that will show a graph in iCube. However, I can only find Power, Voltage and  Current, which show in iCube but only as current values, not as a graph.
I would like to try and fix this, but seeing the code in this repo is not up to date, I can not supply you with a Pull Request.

I checked the code in NPM and it seems the power-consumption capability seems to be supported, but somehow it is not showing in my node-red node? I see in the code there is some check for the version of iHost? I am running version 2.2.0 so that should not be the problem I think.


Check the description presence in the PR

Check the description presence in the PR
Discussion thread: https://groups.google.com/g/quarkus-dev/c/KHlsSiDhiEg/m/-rtxLgCDAgAJ
Examples:

https://github.com/rsvoboda/quarkus-github-app-playground/pull/5
https://github.com/rsvoboda/quarkus-github-app-playground/pull/6

Tests which had multi-line comment check were transformed to text blocks and line with description complain was added.
Other tests were not adjusted, relevant json files were updated not to touch test logic.
Please instruct me if you would prefer different approach to tests changes.

@gsmet will you look into it. You mentioned that you wanted to do some docs related checks. Or hint me in the comment about the classes to look into

I'm on PTO tomorrow so I'll push it on Monday.


feat(iterable_extension): ✨ allow overriding toNum in closestTo method

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

4 of 8 (50.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.4%) to 99.601%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

lib/utils/iterable_extension.dart
4
8
50.0%

Totals

Change from base Build<PHONE_NUMBER>:
-0.4%

Covered Lines:
998

Relevant Lines:
1002

💛  - Coveralls


[Floating Action Button] Touch effect goes beyond the edge of button

Problem description
You may change background color behind the button to see it better

Link to minimally-working code that reproduces the issue
http://www.material-ui.com/#/components/floating-action-button
Versions

Material-UI: 0.16.7
React: 15.4.2
Browser: Chrome 55

Sorry, I can't see anything special 💅 .


Infomax yields different results depending on Python distribution

In https://github.com/scot-dev/scot/issues/160, I noticed that I get different ICs depending on whether I run my script on Anaconda Python or Python installed via Homebrew. In addition, Homebrew Python gives different values when run from a script as compared to running interactively.
Since this problem occurred with real EEG data, it was a bit difficult to create a toy example that reproduces this issue. Here's what I came up with:
import numpy as np
from mne.preprocessing.infomax_ import infomax

np.random.seed(1)
data = np.random.randn(10000, 64)
data[1000:1010, 10] *= 10000
w = infomax(data, random_state=1)

Just like in #3047, the problem appears only with standard Infomax (extended=False), not with the extended version (extended=True) - although these two issues might or might not be related.
When I run the example in Anaconda Python 3.5.1 and then in Homebrew Python 3.5.1, I get a maximum difference in w of around 0.006 - which is way too large to dismiss as numerical inaccuracy. When I run the same script interactively from Homebrew IPython via %run, I also get a difference of 0.006 compared to Homebrew Python script mode. This difference script vs. interactive does not exist in Anaconda Python (there the difference is exactly 0.0).
In my real EEG data, components not only change in their order, but also in their values (as seen on topoplots for example).
When I set extended=True, the problem disappears and the maximum difference is 4.34e-18 (so essentially zero).

I noticed that I get different ICs depending on whether I run my script on Anaconda Python or Python installed via Homebrew. In addition, Homebrew Python gives different values when run from a script as compared to running interactively.

I was going to ask if you were setting the random seed, but it seems you are. Moving on...
My guess is that the differences are driven by small differences in numerical error between different implementations. Anaconda should be using MKL, other builds might be using OpenBLAS or even a naive implementation of the linear algebra functions, and these could all have slightly different numerical errors. If the Infomax algorithm does not deal with these errors properly, then it would give rise to different behaviors.
To test, what I would do is use debugging to dig into individual steps in two installations in parallel. It would be a pain to set up, but it would allow you to definitively say where the differences arise.

And the extended=True being more robust suggests to me that the extended algorithm is somehow more robust or better implemented than the non-extended version. But I know very little about ICA, so that's about all I can suggest :(

My guess is that the differences are driven by small differences in numerical error between different implementations.

This is certainly possible. However, why there's a difference within the same Python distribution (installed via Homebrew) depending on whether I run a script or I run the very same code interactively completely eludes me at this point.

To test, what I would do is use debugging to dig into individual steps in two installations in parallel. It would be a pain to set up, but it would allow you to definitively say where the differences arise.

Right, it was already a pain to come up with the toy example. I guess I'll wait and see how this discussion develops before I do something that time consuming.
Point is, the current standard Infomax implementation seems to be problematic, and I'd be more than happy to help solve this issue (but I can't do it on my own).

depending on whether I run a script or I run the very same code interactively

I missed this bit originally. You mean the only difference is python vs python -i, or by "run interactively" you mean e.g. launch from Spyder?

I missed this bit originally. You mean the only difference is python vs python -i, or by "run interactively" you mean e.g. launch from Spyder?

I run the script in four different ways:

Anaconda 3.5.1 via python script.py
Homebrew 3.5.1 via python script.py
Anaconda 3.5.1 interactively (%run script.py from IPython)
Homebrew 3.5.1 interactively (%run script.py from IPython)

1 and 3 yield identical results. 2 differs from 1 and interestingly also from 4.

One thing you could test is whether Anaconda with MKL (conda install numpy scipy mkl) versus without (conda install numpy scipy nomkl) gives equivalent results. You could do each in a virtual environment (conda create -n py27nomkl python=2.7; conda install numpy scipy nomkl) to keep them isolated. If it fails, it gives you two nice environments to work with.

numerical changes during the optimization can create big difference in the end.
@jmontoyam will comment soon as he reported this to me in the past.

@Eric89GXL yes, maybe MKL is an issue here. I created two Python 3.5 envs, one with mkl, the other with nomkl. The maximum absolute difference between the weights as returned by Infomax is as large as 0.006. Anaconda's nomkl env returns exactly the same weights as Homebrew Python (so it seems both distributions are using OpenBLAS or something).

if you use the same linear algebra libs (Accelerate on mac or mkl) it's
expected you find the same numbers. Otherwise there is no way you can
guarantee this.

Yes, but from a user perspective the effect is pretty huge: in real data, components look different and are in a different order. Using the exact same data and Python code, the only difference is the used numerical libs in the background. Don't you think this is an issue here? Since it only occurs for standard Infomax, I guess this algorithm could probably be made more stable. Alternatively, since everyone is using extended Infomax, maybe standard Infomax should be deprecated or at least a warning message should be generated.

@jmontoyam can you pitch in?

since everyone is using extended Infomax, maybe standard Infomax should be deprecated or at least a warning message should be generated.

This is my preference. "infomax" and "extended-infomax" should become the same thing (with a deprecation warning for "infomax"). If you want, you can still turn off extended via the fit params.

did you empirically observe gains with the extended version? any
performance drawback?

Hi @cbrnr, as @Eric89GXL and @agramfort have mentioned, if we use 2 different python envs, one with mkl (anaconda), and one with nomkl (e.g. homebrew), they will use different numerical lineal algebra libraries, therefore it is expected to get "slightly" different solutions. How "slightly" will the solutions differ?...that strongly depends on the algorithm and data used. For instance, when I run your example I noticed something interesting, the following warning message:
RuntimeWarning: overflow encountered in exp
y = 1.0 / (1.0 + np.exp(-u))
Therefore, I decided to turn on the numpy capability to print all the warning messages (np.seterr(all='print')), and then I run again your example. I noticed that using infomax with this example data, it raised a lot of "Warning: overflow encountered in exp" messages (I would also recommend to enable the verbose=True parameter when you call infomax, this way you will see the logging info across iterations, which facilitates the comparison of mkl vs nomkl performance). Under normal circumstances, 2 different numerical linear libraries will behave "slightly" different, but when they encounter with under/over-flow errors, they will respond different to these errors, and these differences will start to accumulating during the iterations, therefore the final result will be different.
To test this assertion, please comment the line "data[1000:1010, 10] *= 10000", then run again your example, and compare both matrices:
np.max(np.abs(w_nomkl - w_MKL)) = 1.9706458687096529e-15
The next question is: Using this example, why this behaviour happen when using infomax, and not when using extended-infomax?
When I run your example (without commenting the line mentioned previously) using extended-infomax, there were no "warning overflow" messages, therefore it is expected similar solutions from the mkl and nomkl envs:
np.max(np.abs(w - w_MKL)) = 2.2204460492503131e-16
Why extended-infomax did not raise any overflow error?
If we take a look at the infomax source code, we can see that the non-linearity of extended-infomax involves the hyperbolic tangent function, which is bounded, whereas the non-linearity of infomax involves the exponential function, which may be unbounded, therefore it may raise under/over-flow errors, which it turn may impact the comparison of two different numerical linear algebra libraries.
In summary, we should be very careful when comparing the same algorithm/data using different numerical linear algebra libraries.

@jmontoyam I know that the discrepancy only occurs when some samples are large in comparison to the rest. You're right, this is due to the exponential non-linearity in standard Infomax - thanks for the nice explanation!
However, the problem is not the example - these differences between MKL and noMKL appear in real EEG data. I've only constructed this toy example to show what's going on.

@jmontoyam https://github.com/jmontoyam I know that the discrepancy
only occurs when some samples are large in comparison to the rest. You're
right, this is due to the exponential non-linearity in standard Infomax -
thanks for the nice explanation!
However, the problem is not the example - these differences between MKL
and noMKL appear in real EEG data. I've only constructed this toy example
to show what's going on.

yes it will happen. There is no way around it. It's not a bug. What is it
you want to fix?

@agramfort making extended Infomax the default will fix the problem. In addition, if people manually choose to use standard Infomax, maybe a warning message could be generated saying that standard Infomax is sensitive to numerical errors or something. I don't know if deprecating standard Infomax is necessary - if people want to use it, they should be able to do so (but they should be aware of potential pitfalls).

@cbrnr, as @agramfort mentioned in the previous comment, this is not a bug. In general, if we compare the same algorithm using different numerical linear algebra libraries, we could get "slightly" different solutions, how different? this depends heavily on the algorithm and the data used in the comparison. Indeed, this behaviour could also happens in extended-infomax using real EEG data. To test this assertion, I did the following 2 experiments: using real EEG data from one subject, I run the extended-infomax version from eeglab in matlab, in one experiment I used the standard mkl version shipped with matlab (11.1.1), and in the other experiment I used the newer mkl version shipped with anaconda (11.3.1). In both experiments I used exactly the same parameters, and in both experiments
extended-infomax finished using exactly the same number of iterations. When I compared the unmixing matrices, I got the following absolute difference:
max(max(abs(w_mkl_new - w_mkl_old))) = 0.0033
It is exactly the same matlab code, using the same numerical linear algebra library but different version of it, why does this happen? Floating point "algebra" is really tricky and curious, and in general floating point operations are not even commutative. Different numerical linear algebra libraries (or different versions of the same library), could be generate "slightly" different Assembly/Machine code, for instance, let's say, they could slightly modify the order of the operations in order to optimize
the use of the processor cache or even the register usage, therefore, taking into account that floating point operations are not in general commutative, for some datasets this could introduce slightly differences in the results of each iteration, and these differences will start to accumulating during the whole execution, therefore the final result will be different.

Thanks for the explanation @jmontoyam. WDYT about @cbrnr's suggestion earlier:

since everyone is using extended Infomax, maybe standard Infomax should be deprecated or at least a warning message should be generated.

@jona-sassenhagen likes it, and I agree it might be a good way to go. I'm happy with either deprecation or a clear warning message like "infomax can be sensitive to differences in floating point arithmetic, we recommend using extended infomax for enhanced reproducibility and stability"?

Thanks for the explanation @jmontoyam. WDYT about @cbrnr's suggestion earlier:
since everyone is using extended Infomax, maybe standard Infomax should be deprecated or at least a warning message should be generated.

infomax is still the default in eeglab ... let's not deprecate it.

@jona-sassenhagen likes it, and I agree it might be a good way to go. I'm happy with either deprecation or a clear warning message like "infomax can be sensitive to differences in floating point arithmetic, we recommend using extended infomax for enhanced reproducibility and stability"?

what @jmontoyam is telling you is that saying that extended-infomax
being more stable is so far not clear. extended-infomax does suffer
from the same numerical issues.

If we take a look at the infomax source code, we can see that the non-linearity of extended-infomax involves the hyperbolic tangent function, which is bounded, whereas the non-linearity of infomax involves the exponential function, which may be unbounded, therefore it may raise under/over-flow errors, which it turn may impact the comparison of two different numerical linear algebra libraries.

@agramfort I took this to mean that extended-infomax would be more stable due to its use of a bounded nonlinearity, no?

infomax is still the default in eeglab ... let's not deprecate it.

Nope, extended is standard:

We should consider what it would mean to default to extended in MNE. In EEGLAB, the difference between extended and regular is a keyword. In MNE, it is, too, but we somehow also allow a string method; we have two different ways of accessing extended infomax, either as method='extended-infomax', or with method=infomax ... fit_params={'method':'extended'} (or something like that).

Actually, method= overwrites the 'extended' part of fit_params. This is less than optimal IMO.
In my opinion, infomax and extended-infomax should default to just the same thing, with fit_params defaulting to extended=True for both. You can manually set it to not extended by setting extended to false in fit_params. Same API, no breakage, arguably more what you'd expect it to do, more like EEGLAB.

@agramfort I'm currently looking a bit at extended vs. old infomax in the literature and my own data btw. Maybe normal infomax is not as bad as I've been told ...

In a quick experiment I did, extended infomax is indeed better at isolating line noise (the primary benefit claimed by Lee et al. 1998).

Informally speaking, it's also if at all faster than non-extended infomax.

And here is fastica ... It's actually the best!

see https://github.com/mne-tools/mne-python/pull/3108
basically ICA defaults to fastica and if you want infomax or
extended-infomax you need to change the method param explicitly
ok?

Ran the results again (I had a bug) and while not as striking, the overall picture is the same, with Fastica > extended > old.

ok so happy with my PR?

I'd rather go the other direction - deprecate the extended-infomax method keyword, and allow setting extended via fit params (with both 'infomax' and 'extended-infomax' defaulting to extended=True). That would be more like EEGLAB does it, and it would also make extended the default for infomax, as 1. it arguably should be, 2. EEGLAB does it. Do you understand what I'm trying to say?

I'd rather go the other direction - deprecate the extended-infomax method keyword, and allow setting extended via fit params (with both 'infomax' and 'extended-infomax' defaulting to extended=True). That would be more like EEGLAB does it, and it would also make extended the default for infomax, as 1. it arguably should be, 2. EEGLAB does it. Do you understand what I'm trying to say?

this requires an API change and since infomax is not the default then
if people want to change the method they need to do explicitly and if
they want the extended version they do it on purpose and give credit
to Lee et al. and not to Bell et al.

@jona-sassenhagen I don't think comparing the performance of ICA algorithms on line noise is necessarily representative of their performance on EEG/MEG data. That said, Infomax doesn't seem to do well on isolating line noise, but it excels in producing dipolar sources - much better than e.g. FastICA. Here's a nice comparison of some 20 ICA algorithms regarding their performance on EEG data: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0030135
Figure 4 is particularly interesting - it shows that Infomax-based algorithms (standard Infomax, extended Infomax, and AMICA) perform best in terms of the two criteria used in this comparison.
What I want to say regarding the use of Infomax in MNE, I think standard and extended Infomax are pretty much comparable in performance. However, almost all EEG people I know use extended Infomax, because it is an improvement upon the original algorithm. It is also the default choice in EEGLAB.
Given that plus the fact that standard Infomax is obviously more sensitive to numerical errors, I'm in favor of extended Infomax over standard Infomax.
Since I use the Infomax function directly, I cannot comment on defaults and API changes pertaining to MNE as a whole (but an API change is always something that shouldn't happen too lightly).

Clemens: the paper introducing the extended algorithm (lee et al 1999) presented the ability to isolate line noise (as a non Gaussian component) as the case where extended has benefits over old style.

Right, in theory extended Infomax can isolate sub-Gaussian components such as line noise, whereas standard Infomax should completely fail in this case (as it only separates super-Gaussian components). In my experience, extended Infomax greatly reduces line noise, but sometimes it fails to completely isolate it (maybe due to non-stationarity of the noise). In any case, this is another reason why I would choose the extended over the standard implementation.

Given that plus the fact that standard Infomax is obviously more sensitive to numerical errors, I'm in favor of extended Infomax over standard Infomax.

@agramfort this is what I referenced earlier -- can you add this to your PR? My original suggestion was "infomax can be sensitive to differences in floating point arithmetic, we recommend using extended infomax for enhanced reproducibility and stability" but perhaps that is too strong.

better?


Doc: Added some relevant documentation on zapier that will trigger a fix

remove authentication logic as well for this class

Noted!

Noted!

thank you!


Hermes behaviour different than JSC when using different date libraries.

Using some moment.js methods and date-fns functions returns nothing instead of the desired result when developing under hermes engine.
React Native version:
System:
OS: macOS 10.14.5
CPU: (8) x64 Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
Memory: 183.05 MB / 16.00 GB
Shell: 3.2.57 - /bin/bash
Binaries:
Node: 12.4.0 - /usr/local/bin/node
Yarn: 1.16.0 - /usr/local/bin/yarn
npm: 6.9.0 - /usr/local/bin/npm
Watchman: 4.9.0 - /usr/local/bin/watchman
SDKs:
iOS SDK:
Platforms: iOS 12.4, macOS 10.14, tvOS 12.4, watchOS 5.3
Android SDK:
API Levels: 23, 26, 27, 28
Build Tools: 26.0.0, 26.0.1, 26.0.2, 26.0.3, 27.0.0, 27.0.1, 27.0.2, 27.0.3, 28.0.0, 28.0.3, 29.0.0, 29.0.0
System Images: android-22 | Google APIs Intel x86 Atom, android-23 | Google APIs Intel x86 Atom, android-26 | Google APIs Intel x86 Atom, android-28 | Intel x86 Atom_64, android-28 | Google APIs Intel x86 Atom
IDEs:
Android Studio: 3.4 AI-183.6<IP_ADDRESS>22156
Xcode: 10.3/10G8 - /usr/bin/xcodebuild
npmPackages:
react: ^16.8.6 => 16.9.0
react-native: 0.60.5 => 0.60.5
npmGlobalPackages:
react-native-cli: 2.0.1
Steps To Reproduce

['2019-06-06', '2019-06-07].filter(d => moment(d).isBetween('2019-06-05'. '2019-06-08', null, '[]'));

Describe what you expected to happen:
Expected: returns both dates in the array
Result: returns an empty array

FYI You have a . instead of a , between the dates.

^ Yea my bad i misspelled that but theres still a problem.

We're unable to repro this with moment 2.24.0.  Debugging moment is out of our wheelhouse, but Hermes does support this date format with both versions which have been released.
$ ./node_modules/hermesvm/osx-bin/hermes-repl
>> new Date('2019-06-06')
[Date 2019-06-06T00:00:00.000Z]

If you can reproduce using Date and not moment, or post a standalone JS file which reproduces the problem with the hermes or hermes-repl tools, that would help us narrow down the problem.


Creating delete a single rating endpoint + postman testing

Note: Front end needs to go through ratings table, check each rating owner against logged in user, and if they match, provide delete and update abilities

postman tested URI: http://localhost/COMP333_HW3/index.php/deleterating

Backend now checks that user is allowed to delete a certain rating or not


Network policies for OHDSI

As architect we neet to define restrictive network policies for the OHDSI stack

Linking to lsc-sde/lsc-sde#38


Add channel restored event

Fixes #1143
Description
Adds the MS-Teams' event channelRestored to the life cycle.
Specific Changes

Added the channelRestored event in the channel_data verifications in teams_activity_handler.py.
Added the on_teams_channel_restored method in teams_activity_handler.py.
Added a test for the on_teams_channel_restored method in test_teams_activity_handler.py.
Added comments to all the methods in teams_activity_handler.py.

Testing
Below you can see a bot using the channelRestored event, adapted to send a message when the event is raised and in the last image with the test passing.

@denscollo You'll need to update from master before I can merge.

@denscollo Probably just want to merge your PR's with master one at a time since each time I merge one, it will require you to update again.

@tracyboehrer Hi Tracy, I just updated the branch. We don't have permissions to merge PRs. I'll check every PR and update with master each time something is merged. Thanks!


Game Crash

Describe the bug
Game crashes upon world generation.
To Reproduce
Start a world.
Expected behavior
Be able to play.
Version
Fabric 0.7.0
Error Log
Okay
Additional context
I run a bunch of mods, and it took ages to figure out that, somehow, Untitled Duck Mod was the culprit for crashing the game.
crash-2024-07-30_09.57.28-server.txt

c2me conflict, will be fixed in the next version


Peruse does not close properly

Migrating issue from joshuef/peruse: https://github.com/joshuef/peruse/issues/100
OS: Arch Linux
Process:
Close the Peruse window by last tab close or window close.
Expectation:
The window closes and the process ends.
Result:
The window closes, but ps -ef | grep peruse shows that the process is still running.

We don't think this will be an issue with 0.7.0 - TBC after release

I've tried several build variations on a Linux VM and on Windows to reproduce this reported bug but am unable to in latest peruse branch as of commit https://github.com/maidsafe/safe_browser/commit/7bb12daf308498252abbbecac3fce6c703a2672c
Passing this along to see if others can reproduce.

Tested on MacOS with Peruse 0.7.0 where behaviour is IMO expected to be slightly different - i.e. clicking x on MacOS doesn't always close an app, it still has the dot under the icon in the tray. This is the same behaviour as Chrome, FF and Safari.
Right clicking the icon and selecting "Quit" successfully kills the processes for Peruse, as expected, and as is consistent with other browsers.
Note that the processes on MacOS seem to be different to Linux & Windows as the ps -ef | grep peruse command from the original bug report didn't work for me. Investigation found that the processes for Peruse were Electron ones, as pictured below.

These processes were all successfully removed when quitting Peruse.
Discussed with @hunterlester who confirmed that all Peruse related electron processes were removed when he quit on Windows.

@S-Coyle Right. Here's an interesting piece of information to corroborate: https://github.com/maidsafe/safe_browser/blob/peruse/app/main.development.js#L207

Verified fixed in Peruse v0.7.0. Closing


Remove 5.0 as an option?

It's human nature to want the latest and greatest, however 5.0 is not ready yet. With this as an option I've seen many people running into weird build issues, perhaps its best to remove as an option?

How about we keep it but we name it in a less attractive way - as in e.g. 'mainline' (UNSTABLE)


Update Gridline Positioning When Scale Domain Changes

Gridlines are currently static. Would be great if they moved with the scale for geometric/semantic zooming and panning.

Agree with Mark, we'll have some documentation around this soon.


NO-JIRA: Updates for Azure ambient credentials

Update docs/cloud_credentials.md for adding Azure Workload Identity steps.

/cc @snarayan-redhat

/cc @xingxingxia @lunarwhite

@swghosh I had thought about non-workload-identy ambient support after testing Azure workload identity cluster and thus I created a "Test"-only Jira CM-262. But later I thought it might be not yet supported by our code, so we move it to To Do. I'm glad to see you also think of it and give support to it. Therefore, let's met retitle:
/title CM-262: Updates for Azure ambient credentials

@swghosh I had thought about non-workload-identy ambient support after testing Azure workload identity cluster and thus I created a "Test"-only Jira CM-262. But later I thought it might be not yet supported by our code, so we move it to To Do. I'm glad to see you also think of it and give support to it. Therefore, let's met retitle:
/retitle CM-262: Updates for Azure ambient credentials

/retitle CM-262: Updates for Azure ambient credentials

/hold
for now, I've a security concern regarding exposing the Client Secret into the environment variable directly in the deployment. This would mean for a non-Workload Identity Azure cluster is someone gets access to the Deployment yaml, they'll get access to cloud-credentials secret values and as they're long-lived credentials it might  not a good idea to do it this way.
I'll try looking for alternatives to instead use a secret mounted as a file, and still make it work, similar to what we do for AWS and GCP.