wgc commands are failing

Component(s)
cli
Component version
latest
wgc version
latest
controlplane version
latest
router version
latest
What happened?
Since a few days ago i'm not able to run any wgc commands, i get the following error:
/app # wgc federated-graph update qa --label-matcher env=qa --routing-url https://graphql-qa.com/cosmo/v1/graphql ConnectError: [internal] internal error     at errorFromJson (file:///usr/local/lib/node_modules/wgc/node_modules/@connectrpc/connect/dist/esm/protocol-connect/error-json.js:54:19)     at errorFromJsonBytes (file:///usr/local/lib/node_modules/wgc/node_modules/@connectrpc/connect/dist/esm/protocol-connect/error-json.js:93:12)     at next (file:///usr/local/lib/node_modules/wgc/node_modules/@connectrpc/connect/dist/esm/protocol-connect/transport.js:113:31)     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)     at async Object.unary (file:///usr/local/lib/node_modules/wgc/node_modules/@connectrpc/connect/dist/esm/protocol-connect/transport.js:63:20)     at async Object.updateFederatedGraph (file:///usr/local/lib/node_modules/wgc/node_modules/@connectrpc/connect/dist/esm/promise-client.js:67:26)     at async Command.<anonymous> (file:///usr/local/lib/node_modules/wgc/dist/commands/federated-graph/commands/update.js:27:22)     at async Command.parseAsync (/usr/local/lib/node_modules/wgc/node_modules/commander/lib/command.js:936:5)     at async file:///usr/local/lib/node_modules/wgc/dist/index.js:8:5 {   rawMessage: 'internal error',   code: 13,   metadata: HeadersList {     cookies: null,     [Symbol(headers map)]: Map(8) {       'access-control-allow-credentials' => [Object],       'access-control-expose-headers' => [Object],       'connection' => [Object],       'content-length' => [Object],       'content-type' => [Object],       'date' => [Object],       'keep-alive' => [Object],       'vary' => [Object]     },     [Symbol(headers map sorted)]: null   },   details: [],   cause: undefined }
This happens in the cosmo running in our infrastructure and also on cosmo running from the example locally.
Environment information
No response
Router configuration
No response
Router execution config
No response
Log output
No response
Additional context
No response

Hi @ricardogoncalvesmindera,
I think this might be the same issue as https://github.com/wundergraph/cosmo/issues/582.
Please makes sure you're using Node 18+.

Hi, if you are self hosting the controlplane as well, you should see relevant errors there. Please send as that as well, else it is difficult to debug. Thank you.

I did another testing locally, the problem seems to be on creating subgraphs, here's some info:

Logs on control plane:
`
2024-03-04 10:26:49 {"level":"info","time":"2024-03-04T10:26:49.616Z","pid":1,"hostname":"a325607512bb","reqId":"req-3","req":{"method":"POST","url":"/wg.cosmo.platform.v1.PlatformService/CreateFederatedSubgraph","hostname":"localhost:3001","remoteAddress":"<IP_ADDRESS>","remotePort":37810},"msg":"incoming request"}
2024-03-04 10:26:49 Query: select "apiKeys"."id", "apiKeys"."user_id", "apiKeys"."organization_id", "apiKeys"."name", "apiKeys"."key", "apiKeys"."last_used_at", "apiKeys"."created_at", "apiKeys"."expires_at", "apiKeys_user"."data" as "user" from "api_keys" "apiKeys" left join lateral (select json_build_array("apiKeys_user"."id", "apiKeys_user"."email", "apiKeys_user"."created_at") as "data" from (select * from "users" "apiKeys_user" where "apiKeys_user"."id" = "apiKeys"."user_id" limit $1) "apiKeys_user") "apiKeys_user" on true where "apiKeys"."key" = $2 limit $3 -- params: [1, "cosmo_669b576aaadc10ee1ae81d9193425705", 1]
2024-03-04 10:26:49 Query: select "id", "name", "slug", "user_id", "created_at", "is_free_trial" from "organizations" where "organizations"."id" = $1 limit $2 -- params: ["ddd791ff-4577-4636-9db9-0ff05b6d8224", 1]
2024-03-04 10:26:49 Query: update "api_keys" set "last_used_at" = $1 where "api_keys"."id" = $2 -- params: ["Mon, 04 Mar 2024 10:26:49 GMT", "36431c3f-794d-42a9-9622-fc73a3bb89fa"]
2024-03-04 10:26:49 Query: select "targets"."id", "targets"."name", "targets"."type", "targets"."created_at", "targets"."labels", "targets"."organization_id", "targets_subgraph"."data" as "subgraph" from "targets" left join lateral (select json_build_array("targets_subgraph"."id", "targets_subgraph"."routing_url", "targets_subgraph"."subscription_url", "targets_subgraph"."subscription_protocol", "targets_subgraph"."schema_version_id", "targets_subgraph"."target_id", "targets_subgraph_schemaVersion"."data") as "data" from (select * from "subgraphs" "targets_subgraph" where "targets_subgraph"."target_id" = "targets"."id" limit $1) "targets_subgraph" left join lateral (select json_build_array("targets_subgraph_schemaVersion"."id", "targets_subgraph_schemaVersion"."target_id", "targets_subgraph_schemaVersion"."schema_sdl", "targets_subgraph_schemaVersion"."is_composable", "targets_subgraph_schemaVersion"."composition_errors", "targets_subgraph_schemaVersion"."router_config", "targets_subgraph_schemaVersion"."created_at") as "data" from (select * from "schema_versions" "targets_subgraph_schemaVersion" where "targets_subgraph_schemaVersion"."id" = "targets_subgraph"."schema_version_id" limit $2) "targets_subgraph_schemaVersion") "targets_subgraph_schemaVersion" on true) "targets_subgraph" on true where ("targets"."name" = $3 and "targets"."organization_id" = $4 and "targets"."type" = $5) limit $6 -- params: [1, 1, "javagraph", "ddd791ff-4577-4636-9db9-0ff05b6d8224", "subgraph", 1]
2024-03-04 10:26:49 Query: insert into "targets" ("id", "name", "type", "created_at", "labels", "organization_id") values (default, $1, $2, default, $3, $4) returning "id", "name", "type", "created_at", "labels", "organization_id" -- params: ["javagraph", "subgraph", "{}", "ddd791ff-4577-4636-9db9-0ff05b6d8224"]
2024-03-04 10:26:49 Query: insert into "subgraphs" ("id", "routing_url", "subscription_url", "subscription_protocol", "schema_version_id", "target_id") values (default, $1, $2, $3, default, $4) returning "id", "routing_url", "subscription_url", "subscription_protocol", "schema_version_id", "target_id" -- params: ["http://localhost:5001/graph/v3/query", "ws://localhost:5001/graph/v3/subscriptions", "ws", "3309fa8b-83ab-4c5c-8a6f-6ba78ad5a706"]
2024-03-04 10:26:49 Query: select "targets"."name" from "targets" inner join "federated_graphs" on "federated_graphs"."target_id" = "targets"."id" left join "schema_versions" on "schema_versions"."id" = "federated_graphs"."composed_schema_version_id" where ("targets"."organization_id" = $1 and "targets"."type" = $2 and not exists ((select "target_id", "label_matcher" from "target_label_matchers" where ("target_label_matchers"."target_id" = "targets"."id" and not label_matcher && ARRAY[])))) order by "targets"."created_at" asc, "schema_versions"."created_at" asc -- params: ["ddd791ff-4577-4636-9db9-0ff05b6d8224", "federated"]
2024-03-04 10:26:49 {"level":"error","time":"2024-03-04T10:26:49.648Z","pid":1,"hostname":"a325607512bb","service":"wg.cosmo.platform.v1.PlatformService","method":"CreateFederatedSubgraph","err":{"type":"PostgresError","message":"cannot determine type of empty array","stack":"PostgresError: cannot determine type of empty array\n    at ErrorResponse (file:///app/node_modules/.pnpm/postgres@3.4.3/node_modules/postgres/src/connection.js:790:26)\n    at handle (file:///app/node_modules/.pnpm/postgres@3.4.3/node_modules/postgres/src/connection.js:476:6)\n    at Socket.data (file:///app/node_modules/.pnpm/postgres@3.4.3/node_modules/postgres/src/connection.js:315:9)\n    at Socket.emit (node:events:517:28)\n    at addChunk (node:internal/streams/readable:335:12)\n    at readableAddChunk (node:internal/streams/readable:308:9)\n    at Readable.push (node:internal/streams/readable:245:10)\n    at TCP.onStreamRead (node:internal/stream_base_commons:190:23)","name":"PostgresError","severity_local":"ERROR","severity":"ERROR","code":"42P18","hint":"Explicitly cast to the desired type, for example ARRAY[]::integer[].","position":"458","file":"parse_expr.c","line":"1995","routine":"transformArrayExpr"},"msg":"cannot determine type of empty array"}
2024-03-04 10:26:49 {"level":"info","time":"2024-03-04T10:26:49.650Z","pid":1,"hostname":"a325607512bb","reqId":"req-3","res":{"statusCode":500},"responseTime":33.873875000019325,"msg":"request completed"}
`

@ricardogoncalvesmindera what is the output of "which node"?

I found out the issue, the create subgraph command is missing the label option, if i add the --label env=qa the command work. I didn't know it was mandatory.

You can close it as non-issue. Thank you and sorry for the time you spent.

Hi @ricardogoncalvesmindera
It is no longer necessary to include labels in federated graphs or subgraphs. However, if your federated graph was already created without labels, its constituent subgraphs must match that label.
But please note that you will be limited to a single federated graph within a namespace if you choose not to use labels.
Sorry for the confusion.

Hi @ricardogoncalvesmindera. Seems you are not on the latest version of the control plane (although in the issue you have specified it as so) but your CLI is. Please upgrade so you don't face compatibility issues like these.


Unable to install surrealdb-migrations v0.9.12 on Mac

Describe the bug
I'm unable to install surrealdb-migrtions V0.9.12 on Mac. I'm getting this error:
error: failed to compile surrealdb-migrations v0.9.12, intermediate artifacts can be found at /var/folders/k5/5jb6fd397q533zjk9ql2d5pr0000gn/T/cargo-installMPq8Bd
Caused by:
failed to download replaced source registry crates-io
Caused by:
failed to verify the checksum of surrealdb v1.0.0-beta.9+20230402
To Reproduce
I've just installed cargo version cargo 1.71.1 (7f1d04c00 2023-07-29) using curl https://sh.rustup.rs -sSf | sh
Then I've simply run command cargo install surrealdb-migrations
And I get the error above
Expected behavior
I'd expect surrealdb-migrations to install
Information

Mac version [Ventura 13.4.1]
Mac chip [M1]
SurrealDB version: [1.0.0-beta.9+20230402.5eafebd for linux on aarch64]
surrealdb-migration version: [0.9.12]

This is related to a crates.io registry issue. See this issue here: https://github.com/surrealdb/surrealdb/issues/1850
Your only solution for the moment is to keep trying to install until it works. This will be fixed in beta-10.


Move away from deprecated method

Fixed a typo and moved away from deprecated method in README

@narun2001 thank you

You're welcome Mike!


[SPARK-39594][CORE] Improve logs to show addresses in addition to ports

What changes were proposed in this pull request?
This PR aims to improve logs to show address together in addition to ports.
Why are the changes needed?
To provide used address information explicitly .

For example, we can check if IPv4 and IPv6 is used correctly.

IPv4
StandaloneRestServer: Started REST server for submitting applications on <IP_ADDRESS> with port 54511

IPv6
StandaloneRestServer: Started REST server for submitting applications on [0:0:0:0:0:0:0:1] with port 54566

In NettyBlockTrasferService, hostName could be different from bindAddress like the following.

NettyBlockTransferService: Server created on otherHost localhost:54718

Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass the CIs.

All test passed except SparkR Linter failure which is irrelevant.
x Failed to parse Rd in histogram.Rd
ℹ there is no package called ‘ggplot2’
Caused by error in `loadNamespace()`:
! there is no package called ‘ggplot2’
Execution halted
                    ------------------------------------------------
      Jekyll 4.2.1   Please append `--trace` to the `build` command
                     for any additional information or backtrace.

I made a PR to fix master branch

https://github.com/apache/spark/pull/36987

Rebased to the master.

Rebased to the master.

Could you review this please, @viirya ?

Thank you so much, @viirya .
Merged to master.


[SFS] Add new field for "Funding code" as sub-item of "Basic information"

Description of the issue
Since the OEP will be a central hub for scenario data of publicly funded projects, identifying studies by funding codes will not only be useful information in the KG, but also be a demand of users and funding agencies.
Factsheet
Study
Field
Name of the field, see Documentation
Ideas of solution
Add new field for "Funding code" as sub-item of "Basic information"

I like the idea. Maybe also link to EnArgus database directly, if possible?

Transferring to oeplatform, too.


GH Action fix

Build asset gh action fix.  (GH CLI login step)

Follow-up from the previous PR https://github.com/joaonc/show_dialog/pull/37


chore: update and extend a bit for 2021

Found a few issues and missing packages.

Hmm, this seems to be trying to revert #13 - probably bad timing.

This also happened to address the things in #12 by moving to the public URLs for uproot/awkward, and noting the deprecation of root-numpy and root-pandas.

@henryiii Maybe I should have merged this and #12 in the opposite order.
I've resolved the conflicts. Let me know if I've done so as you want it (that I didn't undo any of your edits accidentally), and if you give me the thumbs-up, I'll (squash and) merge it as-is.


Purchase Order Approval Decision

Hi Alex,
We set up a flexible workflow for PO in S/4HANA on-premise. We activate /IWPGW/TASKPROCESSING gateway service. But while displaying Teams card, the approval/reject button (we guest so) does not display.
Can you share us the information of how workflow work in the Teams app.
Thanks,
Thien

Hi Alex,
We set up a flexible workflow for PO in S/4HANA on-premise. We activate /IWPGW/TASKPROCESSING gateway service. But while displaying Teams card, the approval/reject button (we guest so) does not display. Can you share us the information of how workflow work in the Teams app.
Thanks, Thien

Hi Thien good morning. This purchase order approval functionality is one of the use cases we built on top of the bridge framework.  In order to receive the notification card for the purchase order approval, we needs to add additional configurations on the bridge framework config server, and also schedule an ABAP job in the SAP S/4HANA On-Prem system.
Please kindly follow the instructions in this discovery center mission to have everything up and running.
Best Regards,
Weikun Liu

Hi Weikun,
Thanks for the good hints. We can implement the push notification from S/4HANA to Teams by Event mesh. Great to have the mission in the discovery center to follow.
But there comes a concern arises:

After approval, the adaptive card information is updated:

View the detail again; the button approve and reject still display

We expect the app should display the status of the purchase order already approve as sample https://github.com/SAP-samples/btp-bridge-framework/blob/main/documentation/config/Bridge Application JSON Configuration.md
Are we missing any step.
Thank you,
Thien

Hi Thien,
Apologies for the delay, we are looking into this unexpected behavior now, and will get back to you as soon as we can.
Best,
Alex

Hi Thien,
We found a bug in the code. Please update lines 117-136 of /frontend/src/components/ObjectPage.jsx with the following code:
            // Setting creation date
            if (data.hasOwnProperty(props.config.CreationDate)) {
              const date = helper.formatDate(
                JSON.parse(JSON.stringify(data[props.config.CreationDate]))
              );
              setCreationDate(date);
            }

            // Date on which order was last changed, including status changes
            if (data.hasOwnProperty(props.config.LastChangeDateTime)) {
              const date = helper.formatDate(
                JSON.stringify(data[props.config.LastChangeDateTime])
              );
              setLastChangeDateTime(date);
            }

            // Status whether order is approved, rejected, or pending
            if (data.hasOwnProperty(props.config.PurchasingProcessingStatus)) {
              setStatus(data[props.config.PurchasingProcessingStatus]);
            }

Once again, don't forget to build and then push your frontend! We will be including this in the next round of updates for Bridge.
Best,
Alex


add more hacktoberfest projects to the showcase.md

List more active hacktoberfest repositories to the showcase.md

Hey @Gismo1337 can i work on this

Hey @Gismo1337 can i work on this

@clairecharles I assigned this to you. Sorry for the delay. I am busy today. Feel free to start anytime.


Improved modularity of recipes and installation process of python sdk

Description of changes:

Split both the core and python sdk recipes into .inc and .bb files. This allows better flexibility when using an older version of the recipe.

Updated the python sdk for a setuptools3 installation, eliminating the need for the DEPENDS = "python3 python3-setuptools" which will cut down on the build time significantly.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Hello @ndjhartman I see you are posting up a boto3 recipe.  That's great!
It's a new directory off the base and given we are going to have a lot of other SDKs, I would like to change the directory.
Could you please change the directory path and recipe name to:
recipes-sdk/aws-sdk/aws-sdk-python-v3.bb

We will follow this pattern for all languages.  For example, for aws iot device sdks, it will be:
recipes-sdk/aws-sdk/aws-sdk-java-v1.bb
recipes-sdk/aws-sdk/aws-sdk-java-v2.bb
recipes-sdk/aws-iot-sdk/aws-iot-device-sdk-python-v1.bb
recipes-sdk/aws-iot-sdk/aws-iot-device-sdk-python-v2.bb

When you commit it with the directory fix, you can also commit it against Issue #26
Thank you!

No problem! I was working on the boto recipe, but actually came across a fully fledged recipe on meta-iot-cloud. It has several AWS related recipes.

https://layers.openembedded.org/layerindex/branch/master/layer/meta-iot-cloud/

Additionally meta-python contains python3-aws-iot-device-sdk-python:

https://layers.openembedded.org/layerindex/recipe/69048/

Not sure what you want to do about them. If I should keep creating the recipes or just use the existing ones in other layers.
Thanks,
Nathan.

Hello @ndjhartman we will be actively maintaining recipes here for AWS off-cloud software.  We cannot control (and shouldn't control) what other organizations do in other repositories with the open source software AWS provides, but this repository is the AWS-curated repository for Yocto recipes.   Your contribution is appreciated to seed the boto3 recipe from this repository.


igniter install ash_authentication_phoenix doesn't add pipe through browser in generated live session auth scope

When running mix igniter.install ash_authentication_phoenix
The generated ash_authentication_live_session scope block in router.ex is missing pipe through browser, which means that the current_user is not set.
  scope "/", HelpdeskWeb do
    # missing: pipe_through :browser

    ash_authentication_live_session :authenticated_routes do
      # in each liveview, add one of the following at the top of the module:
      #
      # If an authenticated user must be present:
      # on_mount {HelpdeskWeb.LiveUserAuth, :live_user_required}
      #
      # If an authenticated user *may* be present:
      # on_mount {HelpdeskWeb.LiveUserAuth, :live_user_optional}
      #
      # If an authenticated user must *not* be present:
      # on_mount {HelpdeskWeb.LiveUserAuth, :live_no_user}
    end
  end

Fix released in 2.1.9


cpp-hocon: 0.1.6 -> 0.1.7

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools.
This update was made based on information from https://repology.org/metapackage/cpp-hocon/versions.
These checks were done:

built on NixOS

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 0.1.7 with grep in /nix/store/xvra7ffhxjj2p4j80p253jwjxgwyp1rl-cpp-hocon-0.1.7

directory tree listing: https://gist.github.com/f58abd979d8b4e35bed032e929b39177

du listing: https://gist.github.com/361aabaa0ba139c553980b12090a1f95

meta.description for cpp-hocon is: '" A C++ port of the Typesafe Config library"'.
cc @womfoo for testing.
Experimental: this build is cached with Cachix. To use the cache follow these experimental instructions:
One time setup in nixpkgs Git checkout:
cachix use r-ryantm
git remote add r-ryantm https://github.com/r-ryantm/nixpkgs.git

Test this build:
git fetch r-ryantm && git checkout b9f4afa38cedaac1b426765e80f6b817e5ff357b && nix-shell --pure -I nixpkgs=. -p cpp-hocon

No attempt on x86_64-darwin (full log)
The following builds were skipped because they don't evaluate on x86_64-darwin: cpp-hocon
Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

Success on x86_64-linux (full log)
Attempted: cpp-hocon
Partial log (click to expand)
-- Installing: /nix/store/xvra7ffhxjj2p4j80p253jwjxgwyp1rl-cpp-hocon-0.1.7/include/hocon/config_includer.hpp
-- Installing: /nix/store/xvra7ffhxjj2p4j80p253jwjxgwyp1rl-cpp-hocon-0.1.7/include/hocon/config_parse_options.hpp
-- Installing: /nix/store/xvra7ffhxjj2p4j80p253jwjxgwyp1rl-cpp-hocon-0.1.7/include/hocon/types.hpp
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/xvra7ffhxjj2p4j80p253jwjxgwyp1rl-cpp-hocon-0.1.7
strip is /nix/store/4qvrxzxa535y8304mk195x50b6p9607d-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/xvra7ffhxjj2p4j80p253jwjxgwyp1rl-cpp-hocon-0.1.7/lib
patching script interpreter paths in /nix/store/xvra7ffhxjj2p4j80p253jwjxgwyp1rl-cpp-hocon-0.1.7
checking for references to /build in /nix/store/xvra7ffhxjj2p4j80p253jwjxgwyp1rl-cpp-hocon-0.1.7...
/nix/store/xvra7ffhxjj2p4j80p253jwjxgwyp1rl-cpp-hocon-0.1.7

Success on aarch64-linux (full log)
Attempted: cpp-hocon
Partial log (click to expand)
-- Installing: /nix/store/gfj8l39f0m4fpln9wg7pxkrkcbiyaaf0-cpp-hocon-0.1.7/include/hocon/config_include_context.hpp
-- Installing: /nix/store/gfj8l39f0m4fpln9wg7pxkrkcbiyaaf0-cpp-hocon-0.1.7/include/hocon/config_exception.hpp
-- Installing: /nix/store/gfj8l39f0m4fpln9wg7pxkrkcbiyaaf0-cpp-hocon-0.1.7/include/hocon/config.hpp
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/gfj8l39f0m4fpln9wg7pxkrkcbiyaaf0-cpp-hocon-0.1.7
strip is /nix/store/0pjsgkxz0rp5baycq5sp2s72lrr5q9sg-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/gfj8l39f0m4fpln9wg7pxkrkcbiyaaf0-cpp-hocon-0.1.7/lib
patching script interpreter paths in /nix/store/gfj8l39f0m4fpln9wg7pxkrkcbiyaaf0-cpp-hocon-0.1.7
checking for references to /build in /nix/store/gfj8l39f0m4fpln9wg7pxkrkcbiyaaf0-cpp-hocon-0.1.7...
/nix/store/gfj8l39f0m4fpln9wg7pxkrkcbiyaaf0-cpp-hocon-0.1.7


The usb-mic project hard fault

I'm trying to run the usb-mic project on a Nucleo-L4R5Zi board. I created an empty project for this board in STM32CubeIDE and drag/dropped the files into the project. It builds OK but I hit a hard fault in system init on the line:
RCC->CR |= (uint32_t)0x00000100;
I suspect the problem is with how I'm building it. Are there instructions on building this, or an updated project for the STM32CubeIDE?
Any guidance is greatly appreciated.

The usb_mic project will work only on L1 MCUs.
If you were seduced by the phrase "Will work on all STM32L family with hardware USB" from the README.MD, sorry for that...
That was written a long long ago, then there was no L4 series yet.


Improve FixedBy assertion

Add a custom assertion message pointing out which potential vulnerabilities bumped up the FixedBy, so we can check if a bump in the test is necessary or if it's a bug in FixedBy logic.
Tests
CI.

/retest

/test slim-e2e-tests


Allow publishing for selective package

This is to allow the publishing of just one package - necessary changes made in client.yml (and package.json, if required)
Workaround to support the release of eventhubs - client, since the release pipelines don't have filtering enabled.
@Azure/azure-sdk-eng

@KarishmaGhiya: Is this ready to review?  If not, just let me know when.

@mikeharder This is ready for review

This approach looks reasonable for now.
@bsiegel with your work on pmpm do you think we will be able to use it to orchestrate our builds and move away from npm-run-all? Or do you think that is orthogonal?


A11y adjustments for the Brexit checker views

The Brexit checker templates use the finder_layout layout file.
This layout file already adds a <main> wrapper around the content.
However, some of the brexit checker views also contain a <main> wrapper, which means that on those pages there would be two main landmarks, which goes against the rule that the <main> element must be unique to the document.
In addition to this, these views also contain breadcrumbs.
Since breadcrumbs are a navigation element, they should live outside of <main>. I updated these views to store the breadcrumbs using content_for. This way the breadcrumbs will be placed outside of <main> in the layout file.
Before: two main landmarks, breadcrumbs sit inside of main

After: only one main landmark, breadcrumbs sit outside of main

⚠ This repo is Continuously Deployed: make sure you follow the guidance ⚠

Thank you for approving @injms, if you're happy with this would you mind merging it in for me as well?


[TC-OTCCM-1.1, 1.2] Yaml script has to be updated as per Test plan

Description:

[ ] TC-OTCCM-1.1

Step 3  - In Test plan there is no features in feature map list, But Script is verifying the "OTCCM.S.F00(DEPONOFF)" feature.

Step  4c - In Test Plan there is no Feature dependent attribute(DEPONOFF) attribute in AttributeList, But Script is verifying OnMode(0x0003) attribute.

Test Plan References:

https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/cluster/mode_oven.adoc

Yaml References:

https://github.com/project-chip/connectedhomeip/blob/master/src/app/tests/suites/certification/Test_TC_OTCCM_1_1.yaml

[ ] TC-OTCCM-2.1

In Steps 5, 6, 9, 10 OTCCM.S.M.CAN_MANUALLY_CONTROLLED PICS has to be added.

In Step 8 CurrentMode attribute value response is Wrong, It should be same as value in the Step 3, But Script is verifying response value should be same as step 6. So Script has to be updated as per Test Plan

Test Plan References:

https://github.com/CHIP-Specifications/chip-test-plans/blob/master/src/cluster/mode_oven.adoc

Yaml References:

https://github.com/project-chip/connectedhomeip/blob/master/src/app/tests/suites/certification/Test_TC_OTCCM_2_1.yaml

Addressed in PR
https://github.com/project-chip/connectedhomeip/pull/32116


Ignore stdout longer than 10K chars

would need a better solution in long term. but this at least prevents a browser from freezing on larger stdout/stderr outputs.

cc @maasg


upstream: implementation of outlier detection extensions

Commit Message:
upstream: implementation of outlier detection extensions
Additional Description:
The idea and need for the extensions are described in RFC document: https://docs.google.com/document/d/1ZCZSoirVB39eOLdD0VPlsEUING8c23Sq5bzozrv6f4k/edit?usp=drive_link
In a nutshell, the design decouples types of result reported to outlier detector from an algorithm which marks a host as unhealthy. For example, an outlier detector may be configured to count 3 consecutive errors and type of those errors can be defined by a user. For example:

count http codes in range 500-503
count http codes in range 401-403
count locally originated errors (resets, timeouts)

So, the algorithm does not really care about the exact type of reported result. It is only interested whether reported result should be considered an error or not.
The idea of using user-defined errors can be expanded to database errors. See issue #24215 (I have working prototype for errors reported by Redis).
This design puts extensions on top of already existing outlier detector. It means that the solution is 100% backwards compatible. Previous configs are accepted. But a user may configure outlier detection extension to

use "old" outlier detection to react to 5xx errors and add another range of HTTP errors (say 4xx)
disable "old" outlier detection and configure everything using extensions

The implementation of extensions is built on top of already existing outlier detection structures. This minimizes code changes and re-uses event logger, timers, etc.
The implementation is built on top of already approved API for extensions: https://github.com/envoyproxy/envoy/pull/31205
Risk Level: Low (previous configuration still works. Extensions do not have to be configured)
Testing: Added unit tests for new code and tests checking co-existence of "old" outlier and "new" extensions
Docs Changes: Yes. Added.
Release Notes: Yes.
Platform Specific Features: No
Fixes #18789

/retest

@wbpcode I see that you have been assigned for API approval, but changes in proto files are not really API  changes, but rather changes to event log which is defined in terms of protobufs. Hope it helps!

will the old outlier detection extension be eventually deprecated? (given that the new one lands)

It depends on the community and adoption of extensions. In general, everything what the current outlier offers will be implemented in form of extensions, so deprecation should be easy. I believe that removing "old" outlier should improve performance a bit, because in the current implementation all monitors like frequency, success rate always run regardless whether users use them to eject nodes or not.

Looks like the extension docs (*.rst) also need to be added.

Do you have a specific doc in mind? Maybe I missed something. I checked the docs output and "outlier detection" extensions are automatically added to docs based on proto's annotations.

/retest

/retest

will the old outlier detection extension be eventually deprecated? (given that the new one lands)

It depends on the community and adoption of extensions. In general, everything what the current outlier offers will be implemented in form of extensions, so deprecation should be easy. I believe that removing "old" outlier should improve performance a bit, because in the current implementation all monitors like frequency, success rate always run regardless whether users use them to eject nodes or not.

Deprecating old mechanism would as well reduce maintenance load on cognitive load for outlier detection api (where same thing can be achieved via different config mechanisms). Think we should do a proper deprecation cycle going forward.

will the old outlier detection extension be eventually deprecated? (given that the new one lands)

It depends on the community and adoption of extensions. In general, everything what the current outlier offers will be implemented in form of extensions, so deprecation should be easy. I believe that removing "old" outlier should improve performance a bit, because in the current implementation all monitors like frequency, success rate always run regardless whether users use them to eject nodes or not.

Deprecating old mechanism would as well reduce maintenance load on cognitive load for outlier detection api (where same thing can be achieved via different config mechanisms). Think we should do a proper deprecation cycle going forward.

OK. Let me try to add warnings that "old" consecutive errors will be deprecated. Thanks!

/retest

/retest

@nezdolik @wbpcode Thanks for reviewing this PR. I think I addressed all the comments. In some cases, I provided explanation why I am not planning to change anything and leave it as is. I would appreciate if you could do another pass. Thanks!

@cpakulski thank you for all the work done! i added one more comment and believe this is now ready for further review. @wbpcode do you have capacity to review this one? (otherwise will tag senior maintainers)

/assign-from @envoyproxy/senior-maintainers

@wbpcode Thanks for your initial comments. I agree with you that this is large PR. The additional problem is that extensions should work along with "previous" implementation. I believe that code can be cleaner, but keeping backwards compatibility sometimes requires building not-very clean approach. Once we start deprecating "old" way we can start cleaning the code.
I addressed most of your comments. One or two comments require more thoughts and I will work on them tomorrow.

/retest

/retest

Is this ready for a new review? cc @cpakulski

@wbpcode Thanks for keeping an eye on this PR! I still have to refactor FACTORY to check config when it arrives not when monitor is created. I hope to finish it today and will ask you for another review.

@wbpcode I addressed your major comments. There are few small issues left, like CODEOWNERS. I think it is ready for another review. Thanks a lot for great suggestions!

/retest

@wbpcode This is ready for review. CI fail is not related to my changes. Other PRs fail as well. Whenever you have time to review it. Thanks a lot!

@wbpcode Friendly ping :-)

/retest

/wait

@wbpcode Thanks for great feedback. I addressed all your comments and refactored the code.

/retest

gentle ping @wbpcode

/wait:review

friendly ping @cpakulski

I will resume work on this PR early next week.

@cpakulski is it possible to mark a host as outlier based on some http header with this model? Similar to x-envoy-degraded in passive health check?

@ramaraochavali This PR (outlier extensions) have nothing to do with setting host's status via headers. And it also does not conflict with such mechanisms. So, if setting heath status via a header is supported in the future, it will also work with proposed extensions.

@cpakulski Sorry for being vague. Currently we can detect outlier via consecutive 5xx, consecutive gateway etc. With this extension model, if I want to write an extension that reads a response header and mark it an outlier - is this possible?

@ramaraochavali To pass a header or header value to an extension, some changes are required, but not drastic. You would have to define a new error type in API. But I do not think it would be the best way of solving the problem. Extensions are designed to run different statistical algorithms, not a single event, like presence of a specific header.

/retest

@wbpcode I refactored the code to address your comments. I know it took me long time, so I will patiently wait for your another review pass.

Thanks so much for you update. This is a huge work and changes our core code. I will take a detailed check this weekend, when I get a bulk of free time.
Thanks again for all your time for this work. I know this take much of your time, but it's no doubt that the PR is become better and better. 🙂

/wait

@wbpcode I changed API (protos) a bit. Logically they are the same, but:

I created a new message called MonitorBaseConfig and moved there params which must be defined by all types of monitors. Those parts are buckets and enforcing parameter. That allowed me to process MonitorBaseConfig in ExtMonitorConfig class.
renamed proto fields/classes not to use word Error, but Result. When I envision different types of extension monitors, buckets do not always have to represent errors. There may be types of monitors which want to "catch" only positive responses, so it is better to call them Results.

/retest

For sure not that each filter (core or contrib) requires its own new Result.

If not each filter require its own status, then means mapping to an abstraction. Then we back to the exist Result way, although now you may split it to multiple classes but doesn't change the intrinsic and maybe doesn't make things better (I am not sure about this point at this timepoint).

Maybe I am missing something, but if we move API which defines what is error and what is not-error to individual filters, then the Result will be binary (Error or Success) and the processing paths have to be changed. Currently, Result first traverses "old" implementations of outlier and later on the Result is passed to new extensions. If we change the meaning of Result, it cannot be handed to "old" implementation.

I mean we can keep current Result status and make current Result as our generic status of extensions. We needn't to change the meaning of Result.

I would not even call it protocol dependent, but rather category dependent. As I explained above, I believe that all databases will use a single category of Result: DBResult.

But there are lots of non database protocols (double, thrift, sit, sofa, brpc,...) that cannot share same result status. (I actually even doubt whether all databases could shared same result status, but i am not familiar to that area 🤣 , assume it ok)

Now, I see a value of separating logic of interpreting result as error/non-error from outlier logic. Having Result to have only binary state of Error/Success

I am not means that use only two statuses (error/non-error). I mean we need a generic result abstraction for different protocols. Because the exist Result class is used and seems that could meet most basic requirements. So, I think we could use the Result as our generic abstraction first.
Esp the Result is an exist class, so it doesn't add any new utility. So, even at the worest case, I am wrong, and we need new more specific result or status, we still could do it in new PR. And at that timepoint, we should get some specific materials/scenarios to support our decision.

I think I made a small mistake when I was talking about Result. Everywhere I used word Result, I really meant ExtResult.
Result is just enum. So, it should be:
For sure not that each filter (core or contrib) requires its own new ExtResult.
One more thing to mention. As it is right now, only terminal filters can feed Result/ExtResult to the cluster, as other filters do not have access to cluster_manager. So, current implementation of mysql, postgres, etc cannot use outlier methods.

Because the exist Result class is used and seems that could meet most basic requirements. So, I think we could use the Result as our generic abstraction first.

Do you think it is still a valid statement, given that Result is just enum?
Thanks for feedback.

Do you think it is still a valid statement, given that Result is just enum?

Yeah. I think a protocol independent result status is necessary to resolve multiple protocols outlier detection support. And current enum Result is fine, at least for now. If addtional message is necessary in the future, we can provide it as optional new paramter.
That also means we can make the PR much more simpler.

Yeah. I think a protocol independent result status is necessary to resolve multiple protocols outlier detection support. And current enum Result is fine, at least for now. If addtional message is necessary in the future, we can provide it as optional new paramter.

If everything is going to be expressed in enum Result, how a user can program http range which should be treated as error? Some users may want any 5xx, others may want only gateway 502-504 and other users want also treat 4xx as errors. One of the main goals I want to solve is https://github.com/envoyproxy/envoy/issues/18789.

If everything is going to be expressed in enum Result, how a user can program http range which should be treated as error? Some users may want any 5xx, others may want only gateway 502-504 and other users want also treat 4xx as errors. One of the main goals I want to solve is https://github.com/envoyproxy/envoy/issues/18789.

As I said, it's responsibility of protocol specific router. It should be done by adding addtional API to the router filter to provide custom code range.

That part makes sense.
One of the goals of https://github.com/envoyproxy/envoy/pull/4822 was to separate http errors (any 5xx) and locally originated events, like (resets, timeouts, etc). Some users may want to count them together , so 5xx, reset, 5xx causes a node to be marked as unhealthy, but other users do want to ignore 5xx and react only to locally originated events. Router does not control the part which sends locally originated events.

It should be done by adding addtional API to the router filter to provide custom code range.

That will work but will be super confusing for users, because "old" outlier monitors will not use that API.

That will work but will be super confusing for users, because "old" outlier monitors will not use that API.

We can make the default code range match the old outlier monitor's behavior. And if a custom code range is provided, we can use the code range's result for both old outlier detection and new extensions.
It's OKay to change old outlier monitors's behavior if there is an explict configuration.

Given this extension as specific example, only threshold and enforcing is necessary. ErrorBuckets/errors are unecessary now.

I think that is not a correct statement. The reason is that with that approach extensions will not be able to separate between events generated by a protocol, say http and events generated by underlying lower protocol like TCP. That was a goal of https://github.com/envoyproxy/envoy/pull/4822 and as you can see it was not trivial to deliver.
For a simple http transaction which returns 200 OK, the underlying tcp layer will generate LocalOriginSuccess and http protocol will deliver Success.
For 500 code, the underlying tcp layer will generate LocalOriginSuccess, but HTTP filter will generate Failure/Error to outlier detector. Outlier will receive two signals, one positive and one negative.
If there is a network problem, underlying tcp layer will generate LocalOriginPeerReset (or similar - I do not remember).
One of the very important use cases is counting LocalOriginxxxx and Http codes separately, and it cannot be achieved if we remove meaning of buckets from extensions.
I believe that adding few error/bucket types Http, LocalOriginEvents and DBResult strikes the right balance and allows for enough flexibility that users can configure what they want to filter and react to. Why do you think that we will need bunch of other error types/buckets?

I think that is not a correct statement. The reason is that with that approach extensions will not be able to separate between events generated by a protocol, say http and events generated by underlying lower protocol like TCP. That was a goal of https://github.com/envoyproxy/envoy/pull/4822 and as you can see it was not trivial to deliver.

Sorry, I didn't make it more clear. I mean we should keep only generic status error bucket here and at the initial implementation, we can even't don't provide any bucket to keep same behavior with our legacy connective error detection.
There core of this PR should be provide a solution mechanism, rather than to provide a solution directly.

I believe that adding few error/bucket types Http, LocalOriginEvents and DBResult strikes the right balance and allows for enough flexibility that users can configure what they want to filter and react to. Why do you think that we will need bunch of other error types/buckets?

Because I have experience of a bunch of various different protocols. Dubbo, thrift, sofa, various priviate protocols, they all run on the Envoy.
I agree error/bucket types Http, LocalOriginEvents and DBResult make sense. But again, I think it should be part of router filter. The outlier detection should only provide a putResult() to take a generic Result after mapping.
Now, actually we alreay have Http and LocalOriginEvents by the putResult(Result, std::optional<Http::Code>).
And following your current design, we need to refactor it to support DBResult in the future. But I don't think then DBResult could cover all protocols or even most protocols, it could only cover only very limited protocols.
When new requirements are raised, we will need to change our core code and API (putResult in code and error bucket list in proto) again and again.
And considering the scenario where the develop maintain their own protocol extension in their fork, it's unreasonable for them to change the core code and proto API for the outlier detection support. This make this new extension point acutally isn't extendable because it strong depends on the core code and core proto API.

Thanks for the work! I just came across this where I myself am thinking of some changes to outlier detection.
First, as ramaraochavali also mentioned previously here, I want to only count gateway error response which have a specific response header. This is for the case where the Envoy is acting as an L1 proxy, and the L2 proxy may proxy back gateway errors from the actual origin. Since The L2 may be connected to multiple origins, we only want the L1 count the response as an error if it actually originated from the L2 and not from the origin, which can be distinguished by the presence of a response header. It seem the discussion around this comment is whether this config should be given to the router filter to decide whether to report this error vs given the config (and reference to response headers) to outlier detection on how to classify the error?
Second, I also wanted to introduce some other custom errors. The one I need is a 302 response + some custom response header. The thinking is maybe the upstream server can somewhat detect that it itself is not feeling well, so want to send some signals to the downstream telling it of so, but is still lively enough that it can still handle the current request. Thus I want some config API that is customizable to be able to match any kind of HTTP response. How can this fit within the proposed framework?
And lastly, I wanted to introduce some kind of "healthiness score" or "degraded severity" to each host monitored (which seems to be a big change on it's own, as the goal is to have the host only receive a portion of traffic instead of outright ejected). So one requirement here is that different detection signals (origin generated vs gateway errors etc.) when detected have different severity when changing the health score of the host. Combined with the previous custom error (which potentially might have more than one), I wanted the core outlier detection logic to be more generalized to just count, check and act on errors, iterating in a container of all configured signals/errrors/monitors, which I think match what you are doing here?
Thanks!

@wbpcode I realized that there is some misunderstanding on both sides.
I actually think that making outlier protocol independent is very good idea. All my arguments were not against doing it, but rather that I would not be able to address all the scenarios I had in mind. That code is a bit fragile.
Anyways, maybe I just went too far into that rabbit hole and lost the big picture. Having other opinions always help, so thanks for all constructive comments.
And now, @iq250vip added yet another very interesting context (thanks!) to it, which has to be done at router level.
There is no other way, but just try to move the config/knobs into individual filters (router first) and see how other parts of the code fit. I have a feeling that it will come at some compromise. Not sure what yet. Maybe I will have to send new, protocol independent Result only to new extensions and leave "old" detectors as is. Not sure yet.

Expanding on making outlier protocol independent while keeping on your bucketing approach. Does it make sense to then change the PutResult() interface instead of receiving a fixed enum result (which specifies which kind of issue it has, and thus still replying on outlier detection to still be protocol aware), receive a generic success/failure boolean along with what type of (custom) signal it is reporting?
So something like:
void putResult(absl::string_view report_type ,bool is_success) {
  if (auto tracker = error_trackers_.find(report_type); tracker != error_trackers_.end()) {
    tracker->second->update_result(is_success);
  }
}

absl::flat_hash_map<std::string, SomeErrorTrackerSharedPtr> error_trackers_;

Where SomeErrorTracker is a class used to keep track of a particular type of errors (and SomeErrorTrackerSharedPtr is a shared pointer to it), and error_trackers_ is initiated on the host monitor creation with all the type of errors we want to track for it. Different keys can point to the same error tracker (in the case where say we don't distinguish between external gateway errors vs local timeout errors). And the error trackers itself is used to calculate when to eject a host (e.g. due to consecutive errors or below the configured success rate threshold, which can itself be implemented using pluggable algorithms (e.g. via providing callbacks). Obviously we are still keeping the current putResult() interface, which can be reimplemented by calling this new method with predefined report_type keys. Is this something that makes sense to you that is compatible with your changes? (Sorry didn't have time to go over all your changes)
As mentioned previously, I wanted something that is generic using a container for each kind of tracked signals, which I think makes sense if we want outlier detection to be generic, since the current reporting Result enum isn't very generic, and I don't think we want to add yet another enum value for every kind of error that we want to track for.
This also have the benefit that custom filters can be added which sole purpose is to detect one kind of specific custom error, and we don't need to put every bit of detection logic (more importantly, the config that tells the router filter what to detect for) in the router filter.

@iq250vip If I understood your use case properly you would like to define something like that:
if http_code == 302 AND http.header(name_of_your_header).contains(some_text)
treat the response as error.
Is this correct?

Yes, and also keep track of this kind of error separately with other other trackers (timeouts, (http_code >= 500 AND http_code < 600 AND http.header(name_of_your_header).contains(some_text)). so that the severity of each kind of error can be tracked differently.

Yep, which is why I mentioned changing the PutResult() interface, so it is the caller's responsibility to tell which bucket/monitor to account the current error to.
I can do this change myself later and create a pull request, so I'm not counting on you to implement all this yourself, so just asking does this makes sense for you and is compatible with what you are doing here.


Events as records, approach 3.

I took @sgny's #694 and rewrote it to get rid of type parameters and to be more similar to the pattern I used for native elements and their methods.
For the moment, I still kept the accessor functions for nullable fields, but I am thinking of removing them.
This compiles, but is otherwise untested.

I had already tried a similar approach except for defining
module ErrorEvent =
  Event.SyntheticEvent({
    type payload = {error: string};
  });

instead of
module ErrorEvent = {
  type payload = {error: string};

  include Event.SyntheticEvent({
    type _payload = payload;
  });
};

and kept getting errors trying to access fields, both with local opens and type annotations. However, as I kept getting errors even with fields such as nativeEvent, I didn't suspect inline definition of payload.
Anyway, I've tried your version and it works. Further, it can be broken with inline definition.  I switched to the simplified version and as that seems to work even with inline definition, perhaps my definition is an edge case that is slightly buggy.

Checked just now,
module ErrorPayload = {
  type payload = {error: string};
};
module ErrorEvent =
  Event.SyntheticEvent(ErrorPayload);

works, so the issue I had really is that the module was defined inline.

This doesn't seem to be a BuckleScript issue. I tested on OCaml and when module Layout is not defined, the signature of LayoutEvent has t as an opaque type.
However, that's not the case when the module passed as argument to SyntheticEvent is {type t = string;}.

I had already tried a similar approach except for defining
module ErrorEvent =
  Event.SyntheticEvent({
    type payload = {error: string};
  });

instead of
module ErrorEvent = {
  type payload = {error: string};

  include Event.SyntheticEvent({
    type _payload = payload;
  });
};

and kept getting errors trying to access fields, both with local opens and type annotations. However, as I kept getting errors even with fields such as nativeEvent, I didn't suspect inline definition of payload.
Anyway, I've tried your version and it works. Further, it can be broken with inline definition so there might be a bug with the compiler for this edge case, either it should work or it should lead to an error referring to the inline definition.

I just read it, did you try the following:

module ErrorEvent = {
  type payload = {error: string};

  include Event.SyntheticEvent({
    type payload;
  } with type payload := payload;;
};

This is the syntax to redefine types in an include.

I had already tried a similar approach except for defining
module ErrorEvent =
  Event.SyntheticEvent({
    type payload = {error: string};
  });

instead of
module ErrorEvent = {
  type payload = {error: string};

  include Event.SyntheticEvent({
    type _payload = payload;
  });
};

and kept getting errors trying to access fields, both with local opens and type annotations. However, as I kept getting errors even with fields such as nativeEvent, I didn't suspect inline definition of payload.
Anyway, I've tried your version and it works. Further, it can be broken with inline definition so there might be a bug with the compiler for this edge case, either it should work or it should lead to an error referring to the inline definition.

I just read it, did you try the following:

module ErrorEvent = {
  type payload = {error: string};

  include Event.SyntheticEvent({
    type payload;
  }) with type payload := payload;
};

This is the syntax to redefine types in an include.
Possibly we can even remove the need for it to be a functor (why is it a functor?)
This would possibly enable the following:

module ErrorEvent = {
  type payload = {error: string};

  include Event.SyntheticEvent with type payload := payload;
};

Much simpler 😄

That syntax doesn't work for me on 7.3.2 or 8.0.0. Something like
module SyntheticEvent = {
	type payload;

  type t = {
    bubbles: Js.Nullable.t(bool),
    cancelable: Js.Nullable.t(bool),
    currentTarget: float,
    defaultPrevented: Js.Nullable.t(bool),
    dispatchConfig: registrationName,
    eventPhase: Js.Nullable.t(float),
    isTrusted: Js.Nullable.t(bool),
    target: Js.Nullable.t(float),
    timeStamp: float,
<EMAIL_ADDRESS>"type"]
    _type: Js.Nullable.t(string),
  };

  include EventMethods({
    type event = t;
  });
};

module ErrorEvent = {
	type payload = {error: string};

	include (SyntheticEvent:  (module type of SyntheticEvent) with type payload := payload);
};

works only when t does not depend on payload as above. Once the nativeEvent key is added, it appears we have to override t as well.
Sorry for dropping off the conversation, I've been swamped with a work project in the past weeks, but now I'm free(r) now that I've submitted my part yesterday.

Sorry you probably need to do this in the module type definition instead of the module implementation.

module type ErrorEvent {
  type payload = {error: string};

  include Event.SyntheticEvent with type payload := payload;
};

module type ErrorEvent {
  type payload = {error: string};

  include Event.SyntheticEvent with type payload := payload;
};

I've tried similar ideas the other day, but I couldn't exactly figure out a solution. I believe we would need to either define a module of that type or a functor that outputs that type to actually make use of it. Further, it appears any of the additional definitions can't reference that signature so we would have to repeat definitions. Have you been able to make this work?

How about this:

module type SyntheticEvent_intf = {
  type payload;
  type otherType = int;
};

module SyntheticEvent: SyntheticEvent_intf = {
  type payload;
  type otherType = int;
};

module ErrorEvent = {
  type payload = {error: string};

  include (SyntheticEvent: {
    include SyntheticEvent_intf with type payload := payload
  });
  let payload = {error: "error"};
};

Js.log(ErrorEvent.payload.error);

See the following link for an example: https://reasonml.github.io/en/try?rrjsx=true&reason=LYewJgrgNgpgBAFwJ4Ad4GUkDsEAsYICWAxgKIBuMOA+oTgGZwC8cA3gFByKrwoCGSKCD5gA3J25o4IPDABOAFR7M4dBOIC+49qEiw4mHPiJlKOAFwHsskxSoJaDFRy7Ip-QcLES38GfkVlFjVNbV1oeFI5ORA5OxxnH2UPIRFneRi5SwBnBDk6AHMtdgk6YigIMHgACkMbEniES1Y4CS4yiqqrIwIGswc1RgB3QjxJXgFUsDhzFhSvVq4NAEpxLlgEOHm0llYM2MsAIn25Q61W4vYAKWyAOiEC6qjMxtvtsFuT1aA

I don't think that works when type t depends on payload. See this version where
  include (SyntheticEvent: {
    include SyntheticEvent_intf with type payload := payload_
  });

should bring the definition of payload into the module, but that doesn't seem to be the case. It's not an export issue either, even if you move the instance into the module, it still doesn't work as the constructor cannot be found.

There was a typo in your example, here a fixed version

I don't think that works when type t depends on payload, the interface has to specify t as well, as far as I have found.
Why don't you think it doesn't work? Yes type t needs to be in the interface (if you don't want to specify the module type you can do module type of Xxx, but I think it's good practice to specify module types.

There was a typo in your example, here a fixed version

That was my point actually. You get the type payload_ but not payload that should be brought into scope with the include statement.

That was my point actually. You get the type payload_ but not payload that should be brought into scope with the include statement.

Because the payload has been replaced with payload_.

That was my point actually. You get the type payload_ but not payload that should be brought into scope with the include statement.

Because the payload has been replaced with payload_.

I had understood that syntax as replacing the abstract type payload with the type payload  as defined in ErrorEvent, but that was incorrect I see that othertype is included in ErrorEvent.
Assuming otherType is a record where the key payload is of type payload, we can handle that as in this version, but that seems to require defining t for each type of event. As in the example, otherType can't be defined inline and now that I understand what the syntax actually does, that's hardly suprising - you cannot have an unnamed type.
However if we already have the definitions of payload and t in the module, what benefit do we get from the include statement? If the modules had further definitions, that did not depend on payload or t, sure we can simply replace those and proceed. However, all methods in EventMethods also depend on t, so we'd have to define them as well.

That was my point actually. You get the type payload_ but not payload that should be brought into scope with the include statement.

Because the payload has been replaced with payload_.

I had understood that syntax as replacing the abstract type payload with the type payload as defined in ErrorEvent, but that was incorrect I see that othertype is included in ErrorEvent.
Assuming otherType is a record where the key payload is of type payload, we can handle that as in this version, but that seems to require defining t for each type of event. As in the example, otherType can't be defined inline and now that I understand what the syntax actually does, that's hardly suprising - you cannot have an unnamed type.
However if we already have the definitions of payload and t in the module, what benefit do we get from the include statement? If the modules had further definitions, that did not depend on payload or t, sure we can simply replace those and proceed. However, all methods in EventMethods also depend on t, so we'd have to define them as well.

If you don't want to redefine t you can do that with a functor I think.

Ok how about this:
module type SyntheticEvent_intf = {
  type payload;
};

module MakeSyntheticEvent = (M: SyntheticEvent_intf) => {
  type t = {embeddedPayload: M.payload};
};

module ErrorPayload = {
  type payload = {error: string};
};

module ErrorEvent = {
  include MakeSyntheticEvent(ErrorPayload);
  let payload = {
    embeddedPayload: {
      error: "error",
    },
  };
};

Js.log(ErrorEvent.payload.embeddedPayload.error);

Working version here

I probably miss context here by why not just parameterize type t?

Checked just now,
module ErrorPayload = {
  type payload = {error: string};
};
module ErrorEvent =
  Event.SyntheticEvent(ErrorPayload);

works, so the issue I had really is that the module was defined inline.

My suggestion is very similar to this. What is the problem with this?

Checked just now,
module ErrorPayload = {
  type payload = {error: string};
};
module ErrorEvent =
  Event.SyntheticEvent(ErrorPayload);

works, so the issue I had really is that the module was defined inline.

My suggestion is very similar to this. What is the problem with this?

What I tried at first was
module ErrorEvent =
  Event.SyntheticEvent( type payload = {error: string};);

which seems reasonable enough. Doesn't throw any compilation errors anyway. However, the resulting module signature has an abstract t, so none of the fields are accessible. In contrast, if you try something like
module ErrorEvent =
  Event.SyntheticEvent( type payload = string;);

you'd have the type defined as expected in the module signature. Since I hadn't identifed the difference, I ended up with the other PR.

Maybe we should focus on UX instead of implementation to help us decide what is the best?
Is there any difference between the 3 pending implantation in term of developer experience?

Maybe we should focus on UX instead of implementation to help us decide what is the best?
Is there any difference between the 3 pending implantation in term of developer experience?

I've been sitting out on the UX question, but #2 and #3 are identical in use. Scope to access fields and use methods will be the actual event submodules, such as PressEvent, etc. This could allow a single local open, such as PressEvent.(....)
With #1, the methods are defined in Event and not included in each submodule (though it could be revised), so the scope for methods is Event but to access fields, one would need PressEvent.
I also think we have stalled a new release for too long, so we should quickly decide on this.

AFAIK 3 is just an improved 2 (better implementation). And 2-3 are better in UX than 1 because it allows a single local open. Or did I overlook something?

Ok thanks for the explanation.
We need to move on & work on 0.63 :)
Are you ok to merge this PR & close the 2 first implementations? @sgny @jfrolich
@sgny I know you worked hard on this & don't want you to feel that you don't get credit for this as this PR is by @cknitt

Ok thanks for the explanation.
We need to move on & work on 0.63 :)
Are you ok to merge this PR & close the 2 first implementations? @sgny @jfrolich
@sgny I know you worked hard on this & don't want you to feel that you don't get credit for this as this PR is by @cknitt

No worries, I'm fine with it.

Ok, merging & preparing release!!!


ARROW-3641: [Python] Remove unneeded public keyword from pyarrow public C APIs

According to
https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#c-api-declarations
it is not necessary to use public here. If we want to be able to refer to Cython extension types at the C API level (at some point, this may not be a bad idea), then we must use public with those.

Appveyor build https://ci.appveyor.com/project/wesm/arrow/builds/20902944


gcroot didn't work as advertised

I tried to follow this guide.
See screenshot below - I issued the two commands as documented, but the second one fails.

It seems it doesn't like the -all argument?

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 2913d994-4c72-cc6e-0bd4-ba122ae21753
Version Independent ID: 03f3761e-be31-bae9-6ac7-ad86db389212
Content: Debug a memory leak tutorial
Content Source: docs/core/diagnostics/debug-memory-leak.md
Product: dotnet-fundamentals
GitHub Login: @tommcdon
Microsoft Alias: tommcdon

Associated WorkItem - 156749

Same problem.

I'm guessing that's because you're using the .NET Framework version of SOS, which doesn't seem to have the -all option for gcroot. Instead, install the .NET (Core) version.

I'm guessing that's because you're using the .NET Framework version of SOS, which doesn't seem to have the -all option for gcroot. Instead, install the .NET (Core) version.

I am experiencing the same issue and did not directly install SOS, it came as part of installing dotnet-dump which I believe would be the .NET core version.


Sik-Ho Tang | Review -- DeepCluster: Deep Clustering for Unsupervised Learning of Visual Features.

Sik-Ho Tang. Review — DeepCluster: Deep Clustering for Unsupervised Learning of Visual Features.

Overview
DeepCluster, K-Mean Clustering to Generate Pseudo-Labels, a Pretext Task for Self-Supervised Learning.

Illustration of the Proposed DeepCluster.
In this story, Deep Clustering for Unsupervised Learning of Visual Features, DeepCluster, by Facebook AI Research, is reviewed. In this paper:

DeepCluster, a clustering method is proposed that jointly learns the parameters of a neural network and the cluster assignments of the resulting features.
DeepCluster iteratively

groups the features with a standard clustering algorithm, k-means,
and uses the subsequent assignments as supervision to update the weights of the network.

This is a paper in 2018 ECCV with over 900 citations.

K-Mean is applied to learned features, EM-learning???

Notations for Supervised Learning
Before talking about DeepCluster, let’s define some notations using supervised learning.
Given a training set $X=\{x_{1}, x_{2}, ..., x_{N}\}$ of $N$ images, we want to find a parameter $\theta^{*}$ such that the mapping $f$ produces good general-purpose features.
These parameters are traditionally learned with supervision, i.e. each image $x_{n}$ is associated with a label $y_{n}$ in $\{0, 1\}^{k}$. This label represents the image’s membership to one of $k$ possible predefined classes.
A parametrized classifier $g_{W}$ predicts the correct labels on top of the features $f(x_{n})$.
Therefore, the loss function is (Eq. (1)):

where $l$ is the multinomial logistic loss.
This cost function is minimized using mini-batch stochastic gradient descent and backpropagation to compute the gradient.

DeepCluster as Pretext Task in Self-Supervised Learning

Top: k-Means Clustering on Vectors Produced by CNN; Bottom: Using the clustering results as psuedo labels for backpropagation.
DeepCluster Procedures
The idea of this work is to exploit this weak signal to bootstrap the discriminative power of a convnet.
We cluster the output of the convnet and use the subsequent cluster assignments as “pseudo-labels” to optimize Eq. (1). This deep clustering (DeepCluster) approach iteratively learns the features and groups them.
A standard clustering algorithm, $k$-means, is used.
$k$-means takes a set of vectors as input, in our case the features $f(x_{n})$ produced by the convnet, and clusters them into $k$ distinct groups based on a geometric criterion.
More precisely, it jointly learns a $d\times k$ centroid matrix $C$ and the cluster assignments $y_{n}$ of each image $n$ by solving the following problem (Eq. (2)):

Overall, DeepCluster alternates between clustering the features to produce pseudo-labels using Eq. (2) and updating the parameters of the convnet by predicting these pseudo-labels using Eq. (1).

Avoiding Trivial Solutions
Empty Cluster
An optimal decision boundary is to assign all of the inputs to a single cluster. This issue is caused by the absence of mechanisms to prevent from empty clusters.
More precisely, when a cluster becomes empty, a non-empty cluster is randomly selected and its centroid is used with a small random perturbation as the new centroid for the empty cluster. The points are then reassigned belonging to the non-empty cluster to the two resulting clusters.
Trivial Parametrization
If the vast majority of images is assigned to a few clusters, the parameters $\theta$ will exclusively discriminate between them.
A strategy to circumvent this issue is to sample images based on a uniform distribution over the classes, or pseudo-labels.

DeepCluster Analysis
Normalized Mutual Information (NMI)

(a): Evolution of the clustering quality along training epochs; (b): evolution of cluster reassignments at each clustering step; (c): validation mAP classification performance for various choices of k.
Normalized Mutual Information (NMI), is used to measure the performance:

where $I$ denotes the mutual information and $H$ the entropy.
If the two assignments A and B are independent, the NMI is equal to 0. If one of them is deterministically predictable from the other, the NMI is equal to 1.

(a): The dependence between the clusters and the labels increases over time, showing that the learnt features progressively capture information related to object classes.
(b): The NMI is increasing, meaning that there are less and less reassignments and the clusters are stabilizing over time.
(c): The best performance is obtained with k= 10,000. Given that ImageNet has 1000 classes. Apparently some amount of over-segmentation is beneficial.

Visualizations

Filter visualization and top 9 activated images from a subset of 1 million images from YFCC100M.
As expected, deeper layers in the network seem to capture larger textural structures.

Top 9 activated images from a random subset of 10 millions images from YFCC100M for target filters in the last convolutional layer.
The filters on the top row contain information about structures that highly correlate with object classes. The filters on the bottom row seem to trigger on style, like drawings or abstract shapes.

DeepCluster Performance
Linear Classification on Activations on ImageNet & Places

Linear classification on ImageNet and Places using activations from the convolutional layers of an AlexNet as features.
ImageNet
Model:

A linear classifier is trained on top of different frozen convolutional layers.

Results:

On ImageNet, DeepCluster outperforms the state of the art from conv2 to conv5 layers by 1−6%. The largest improvement is observed in the conv3 layer.

Finally, the difference of performance between DeepCluster and a supervised AlexNet grows significantly on higher layers: at layers conv2-conv3 the difference is only around 4%, but this difference rises to 12.3% at conv5,

marking where the AlexNet probably stores most of the class level information.

If a MLP is trained on the last layer, DeepCluster outperforms the state of the art by 8%.
Places
DeepCluster yields conv3-4 features that are comparable to those trained with ImageNet labels.

This suggests that when the target task is sufficiently far from the domain covered by ImageNet, labels are less important.

If the target domain is exactly the same with the source domain, supervise learning is the best method. E.g. the same subject watching different pictures?
If the target domain is different from the source domain, there is little gap between supervise learning and un-supervise learning. E.g. different subjects watching the same pictures or different pictures?

Pascal VOC

Comparison of the proposed approach to state-of-the-art unsupervised feature learning on classification, detection and segmentation on Pascal VOC.
DeepCluster outperforms previous unsupervised methods, such as Context Prediction [13], Context Encoders [46], Colorization [71], Split-Brain Auto [72], Jigsaw Puzzles [42], on all three tasks, in every setting.

The improvement with fine-tuning over the state of the art is the largest on semantic segmentation (7.5%).
On detection, DeepCluster performs only slightly better than previously published methods. Interestingly, a fine-tuned random network performs comparatively to many unsupervised methods, but performs poorly if only fc6–8 are learned.

YFCC100M

Impact of the training set on the performance of DeepCluster measured on the Pascal VOC transfer tasks.

In YFCC100M, object classes are severly unbalanced, leading to a data distribution less favorable to DeepCluster.
This experiment validates that DeepCluster is robust to a change of image distribution, leading to state-of-the-art general-purpose visual features even if this distribution is not favorable to its design.

AlexNet vs VGGNet

Pascal VOC 2007 object detection with AlexNet and VGG16.
In the previous experiment, AlexNet is used. Here a deeper network VGGNet is tried.
Training the VGG-16 with DeepCluster gives a performance above the state of the art, bringing us to only 1.4% below the supervised topline.

Image Retrieval

mAP on instance-level image retrieval on Oxford and Paris dataset with a VGG16.
The above table suggests that image retrieval is a task where the pre-training is essential and studying it as a down-stream task could give further insights about the quality of the features produced by unsupervised approaches.

One of the major issues is that k-mean clustering takes quite plenty of time.

Reference

[2018 ECCV] [DeepCluster] Deep Clustering for Unsupervised Learning of Visual Features.


keychain: 2.8.3 -> 2.8.5

Semi-automatic update. These checks were performed:

built on NixOS
ran /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin/keychain -h got 0 exit code
ran /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin/keychain --help got 0 exit code
ran /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin/keychain help got 0 exit code
ran /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin/keychain -V and found version 2.8.5
ran /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin/keychain --version and found version 2.8.5
ran /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin/keychain version and found version 2.8.5
ran /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin/keychain -h and found version 2.8.5
ran /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin/keychain --help and found version 2.8.5
ran /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin/keychain help and found version 2.8.5
ran /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin/.keychain-wrapped -h got 0 exit code
ran /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin/.keychain-wrapped --help got 0 exit code
ran /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin/.keychain-wrapped help got 0 exit code
ran /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin/.keychain-wrapped -V and found version 2.8.5
ran /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin/.keychain-wrapped --version and found version 2.8.5
ran /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin/.keychain-wrapped version and found version 2.8.5
ran /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin/.keychain-wrapped -h and found version 2.8.5
ran /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin/.keychain-wrapped --help and found version 2.8.5
ran /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin/.keychain-wrapped help and found version 2.8.5
found 2.8.5 with grep in /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5
found 2.8.5 in filename of file in /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5

cc "@sigma"

Success on aarch64-linux (full log)
Partial log (click to expand)
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/hipz20mi2sw0ag4zy2qpwyb4i09mdlf4-keychain-2.8.5
gzipping man pages under /nix/store/hipz20mi2sw0ag4zy2qpwyb4i09mdlf4-keychain-2.8.5/share/man/
strip is /nix/store/lvx1acn1ig1j2km8jds5x3ggh3f2wa8v-binutils-2.28.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/hipz20mi2sw0ag4zy2qpwyb4i09mdlf4-keychain-2.8.5/bin
patching script interpreter paths in /nix/store/hipz20mi2sw0ag4zy2qpwyb4i09mdlf4-keychain-2.8.5
/nix/store/hipz20mi2sw0ag4zy2qpwyb4i09mdlf4-keychain-2.8.5/bin/.keychain-wrapped: interpreter directive changed from "/bin/sh" to "/nix/store/lnws52qanbi5vz8qg24v4ym57nznfndg-bash-4.4-p12/bin/sh"
checking for references to /build in /nix/store/hipz20mi2sw0ag4zy2qpwyb4i09mdlf4-keychain-2.8.5...
/nix/store/hipz20mi2sw0ag4zy2qpwyb4i09mdlf4-keychain-2.8.5

Success on x86_64-darwin (full log)
Partial log (click to expand)
build flags: SHELL=/nix/store/pkjmwq7sqrvjg7cjiph6hq0khsmfl6p8-bash-4.4-p12/bin/bash
make: Nothing to be done for 'all'.
installing
post-installation fixup
gzipping man pages under /nix/store/sz42rwswk77nf710hp1g3hv0sm1g8xgr-keychain-2.8.5/share/man/
strip is /nix/store/4sdh09gmvl15cy0zb6i7mbvxh5syz206-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/sz42rwswk77nf710hp1g3hv0sm1g8xgr-keychain-2.8.5/bin
patching script interpreter paths in /nix/store/sz42rwswk77nf710hp1g3hv0sm1g8xgr-keychain-2.8.5
/nix/store/sz42rwswk77nf710hp1g3hv0sm1g8xgr-keychain-2.8.5/bin/.keychain-wrapped: interpreter directive changed from "/bin/sh" to "/nix/store/pkjmwq7sqrvjg7cjiph6hq0khsmfl6p8-bash-4.4-p12/bin/sh"
/nix/store/sz42rwswk77nf710hp1g3hv0sm1g8xgr-keychain-2.8.5

Success on x86_64-linux (full log)
Partial log (click to expand)
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5
gzipping man pages under /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/share/man/
strip is /nix/store/b0zlxla7dmy1iwc3g459rjznx59797xy-binutils-2.28.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin
patching script interpreter paths in /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5
/nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5/bin/.keychain-wrapped: interpreter directive changed from "/bin/sh" to "/nix/store/zqh3l3lyw32q1ayb15bnvg9f24j5v2p0-bash-4.4-p12/bin/sh"
checking for references to /tmp/nix-build-keychain-2.8.5.drv-0 in /nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5...
/nix/store/455vx6ibapfgpy0n34j3y0qpd53v2zn2-keychain-2.8.5


Modeler fails to start with <#> in folder path

Steps to Reproduce

Download Camunda Modeler
Extract to path with # in the name
Run
Modeler shows white canvas

Expected Behavior

[ ] Modeler is able to handle # in path
[ ] Modeler plug-ins work with # in folder names

Alternative Solutions

Show a message that # in folder names are not supported

Environment

Camunda Modeler Version: v2.x, v3.x
Operating System: Any (confirmed on Windows and Linux)

Additional Context
The issue is that we do not properly escape # in the file we tell the app to open (cf. https://github.com/camunda/camunda-modeler/blob/master/app/lib/index.js#L328).
This results in the client opening for the URL file://parts-before-hash.
Unfortunately we cannot simply escape all hashes, as these may be used as actual path to search delimiters, too. Example: As anchors in SVGs (cf. https://github.com/camunda/camunda-modeler/blob/master/client/src/app/primitives/icon/Icon.less#L4).

Referenced via SUPPORT-5362

We are not going to address this issue any time soon. A simple workaround exists: Move the modeler into a path without a # and you are going to be :cool:.

FYI, we just had this issue in a training. A descriptive error message would have been helpful in that case. Maybe it is possible to do a sanity check, by checking if a certain file exists?

@yT0n1 Happy to take contributions that improve the situation.
We do regard # in folder names as edge cases and, thus, decided to close this one as won't fix a while back.

@nikku Just wanted to let you know, that this does happen in the wild and can cause confusion 😉. I agree it is an edge case. I'll leave it to you to decide if it remains won't fix


python3Packages.pontos: 21.10.2 -> 21.11.0

Motivation for this change
Update to latest upstream release 21.11.0
Change log: https://github.com/greenbone/pontos/releases/tag/v21.11.0
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 144214 run on x86_64-linux 1

2 packages built:

python38Packages.pontos
python39Packages.pontos


Potential Dataset Version Mismatch

It appears that there's a discrepancy between the dataset version mentioned in the paper and the dataset actually utilized by the repository. Currently, the repository is likely operating on CO3Dv1, while it should be using CO3Dv2. Please correct me if I'm mistaken.

Hi,
thank you very much for pointing this out.
You are correct, we have unfortunately mistaken CO3Dv1 with v2.
However, note that we still trained and evaluated all models and baselines on the same data.
We will correct this mistake in the paper and potentially run all experiments again on v2.
Best
Chris


[Bug] CSS Module + ReactRouter: ClassNames aren't added to the generated HTML

The inital load of any of my pages lacks the proper classes, thus the stylling is all wrong on initial load.
Environment
  System:
    OS: Windows 10
    CPU: (8) x64 AMD FX(tm)-8350 Eight-Core Processor
    Memory: 16.56 GB / 31.98 GB
  Binaries:
    Node: 10.15.3 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.15.2 - C:\Program Files (x86)\Yarn\bin\yarn.CMD
    npm: 5.3.0 - C:\Users\me\AppData\Roaming\npm\npm.CMD
  Browsers:
    Edge: 44.17763.1.0
    Internet Explorer: 11.0.17763.1
  npmPackages:
    react: ^16.8.2 => 16.8.6
    react-dom: ^16.8.2 => 16.8.6
    react-router-dom: ^5.0.0 => 5.0.0
    react-static: ^7.0.0 => 7.0.10
    react-static-plugin-css-modules: ^7.0.8 => 7.0.8
    react-static-plugin-react-router: ^7.0.8 => 7.0.8
    react-static-plugin-sitemap: ^7.0.0 => 7.0.0
    react-static-plugin-source-filesystem: ^7.0.0 => 7.0.0
    react-transition-group: ^4.0.0 => 4.0.0

Expected Behavior
yarn start and yarn build yield same result.
Initial load has classNames/styles as appropriate.
Note that navigation using <Link /> components causes rerendering of that particular element, thus causing the classNames to get applied.
Reproducible Demo
https://github.com/rickyrombo/react-static-css-modules-bug

This affects me too, would be great to see this fixed.

It looks fine, but it's a bit hard to figure out what you were trying to achieve because it's just random demo styles. Could you explain your demo?

@rickyrombo I'm assuming that when you run yarn start you get the correct styles, but not when you run yarn build?

@benedfit  correct. the outputted static HTML does not contain the class names on the elements.

@rickyrombo I'm assuming that when you run yarn start you get the correct styles, but not when you run yarn build?

Correct. Note that the index page has no styling on initial load. Then click the link to go to foo, and click the link to go back, and the styles get mounted.

This is the same behaviour I experience with my custom Stylus/CSS modules plugin. I’ve been able to get around it by also having everything export during the ‘node’ stage too, but this adds overhead to the build process

same as me!
I used emotion to styled my react-static app and i also have this problem.
I try to
yarn add react-static-plugin-emotion and also get the error like this

After updating to v7.1.0 this issue was resolved for me, and I was able to remove the additional code I mentioned in https://github.com/react-static/react-static/issues/1166#issuecomment-491948441 and classes rendered as expected https://github.com/newhighsco/destiny-clan-warfare/commit/fb803e7b7ff5cdcbdd421b1049be476d7ea5c8e5

I just ran into this issue with the latest<EMAIL_ADDRESS>version. The stylesheet gets generated correctly but the built DOM does not have the classes assigned - this only happens after the react app was mounted. @benedfit can you tell me how you solved the issue?

Seems to be solved / have a good workaround.


Inconsistent SDK usage.

Bug report
Describe the bug
subscription-cancel is inconsistent with the rest of the sdk usage.
To Reproduce
View this repo
    $subscription = $stripe->subscriptions->retrieve($body->subscriptionId);
    $subscription->delete();

Expected behavior
    $subscription = $stripe->subscriptions->cancel($body->subscriptionId);

Additional context
It looks like these are equivalent calls and the second is consistent with other examples in the repo.  If these calls are not equivalent please let me know.

You're right. I don't recall when that pattern changed in the PHP SDK, but the sample was likely written before that time. The API reference reflects your expected behaviour, which I think is right: https://stripe.com/docs/api/subscriptions/cancel?lang=php.
Going to experiment now.

Updated the fixed price example. Thanks again!


Changing branch alias to 2.0

Q
A

Bug fix?
yrs

New feature?
no

Tickets
none

License
MIT

Since we're going to skip 1.4 and go straight to 2.0, it is the next release.
Buuuut, a better solution is to create 1.x and 2.x branches. Then I believe we can remove this completely - e.g. Flex https://github.com/symfony/flex/commit/375e01daedd481501c29f3dea443cf885858382f#diff-d2ab9925cad7eac58e0ff4cc0d251a937ecf49e4b6bf57f8b95aab76648a9d34

Renaming branches to 1.x and 2.x instead


Add API key to header in SDKs

I think we should be merging this to the infra-update branch for now and not to main.

I'm also not sure if it should be Authorization: Bearer ... header (which I think is more for user access tokens) and not a simple X-API-KEY (https://swagger.io/docs/specification/authentication/api-keys/). If we are using Authorization header the Basic ... could probably be used (https://blog.stoplight.io/api-keys-best-practices-to-authenticate-apis)?


Does not work in BGSW using Plasmo framework

Environment
Framework: Plasmo 0.84.0
Client side Chrome Browser Extension
Reproduction
Can be reproduced by creating a BGSW in the Plasmo framework and importing unpdf. Error message is:
🔴 ERROR  | Build failed. To debug, run plasmo dev --verbose.
🔴 ERROR  | Failed to resolve 'unpdf/pdfjs' from<EMAIL_ADDRESS>Describe the bug
Bug is as aforementioned: unpdf seems to be looking for a pdfjs dependency that is inaccessible.
Additional context
No response
Logs
No response

Mmh, interesting. Which bundler does Plasmo use? unpdf provides a sub path export unpdf/pdfjs:
https://github.com/unjs/unpdf/blob/f640d4de5999ddbeedb731b8580a1e7e1cfddb80/package.json#L34
Seems to me more like a Plasmo issue than unpdf. I have tested unpdf with esbuild, Rollup and more, which all correctly bundle unpdf/pdfjs.
Could you please provide a reproducible example?

Bundler used is https://parceljs.org/ I believe... I know that I couldn't get plasmo to work with pdfjs either, so this very well could be a plasmo issue.
Will drop an example in here when I get a moment, sorry about not including that the first round.
My current workaround is to use unpdf on my proxy (vercel) server (normally used for other things) but this is really just to continue development, it would be really not-prime to handle everyone's PDF shenanigans when they could be doing it themselves on the client side.

Hey there, sorry for taking so long!
https://github.com/Fiattarone/unpdf-plasmo-example under core/test.ts is where I put it
Or, even easier to reproduce with a one-liner:
git clone git@github.com:Fiattarone/unpdf-plasmo-example.git  --recurse-submodules &&
cd unpdf-plasmo-example &&
pnpm install &&
pnpm dev

fails upon compiling
If this is more of an issue with plasmo, I'll take it up with them--just let me know!

Hi @Fiattarone, thanks for the reproduction. I tried to import the pdfjs subpath export, but somehow the Parcel bundler doesn't seem to resolve the dependency. A quick look at Parcel's dependency resolution docs didn't help.
Even if we point to the PDF.js source file, the bundler fails to locate it:
import { configureUnPDF, extractText } from "unpdf"

configureUnPDF({
  pdfjs: () => import("unpdf/dist/pdfjs.mjs"),
});

Sorry, but at the moment, I don't have a fix for the Plasmo framework. Noted that Vite (using Rollup) and esbuild correctly resolves the import…


Use Symfony\Component\Finder\Finder

Handy component to help find PHP files with less code.

This is done!


Update pufferpanel.css

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for the Review, i didn't tested loading a new page, i just used the browser inspector, so i forgot that happen, maybe using the other one (left instead of right) should fix that. And i can understand your second point of reusability, so i Will see it later how can i do that without writing the same code again(i think some ways to do this but i have to check them to be clear which Is better). About the 360 Rotation, i was planning do a 540 deg Rotation(just for the collapse, on other parts just use 180deg and not 540deg), but as the image Is changed it already does 180deg, so i added 360deg, a Solution to this problem is not change the image, and always use the same, so when you see the code it says 540deg instead of 360deg. Of course if you dont like 540deg, it can be 180deg.


error getting person credential (2033 - invalid params)

Description of problem
Expected behavior
Steps to reproduce
Screenshots and/or log output

Environment

[ ] Occurs on Android - works on Android
[x] Occurs on iOS

Build #: 1.0.18-1771
Android Device Model:
iOS Device Model:
Workaround
Severity

[x] High
[ ] Medium
[ ] Low

This was caused by a bug in our attestation controller v2 (recent refactoring). It only impacts BC Wallet v1.0.18. Fixed in the attestation controller via PR bcgov/mobile-attestation-vc-controller/62

tested on iOS (iPhone SE, 17.5.1) and Android (pixel 8, android 14).

Verified in 1.0.18.


An admin invite url fails if the email contains + sign

Subject of the issue
The admin invite URL interprets the email param incorrectly if the email contains a + sign
Steps to reproduce

Invites an admin to the system with a new email address
Click the verification link that's attached to the invite email
Sets the password and click "Create Account"
user:invite_not_found is displayed

Expected behavior
Password setting should be successful and the new user can login. The email address in the invite url should replace + with %2B.
Actual behavior
The + sign in the email address got converted to a space. So the email address is incorrect and the invite could not be found.

For reference: https://www.w3.org/TR/html401/interact/forms.html#h-17.13.4

Space characters are replaced by `+', and then reserved characters are escaped as described in [RFC1738], section 2.2


feat(auth): improve GNAP error responses

Changes proposed in this pull request

Expands error responses to include a code field that refers to a particular GNAP error and a description field that explains what went wrong.

Context

Fixes #2344.
Just a small PR to update all the GNAP error responses to be more detailed.
Checklist

[x] Related issues linked using fixes #number
[x] Tests added/updated
[ ] Documentation added
[x] Make sure that all checks pass
[ ] Postman collection updated

Should error responses in middleware such as this be update as well?
main/packages/auth/src/signature/middleware.ts#L44

Good catch, I forgot about those.

Just a small PR to update all the GNAP error responses to be more detailed.

Not so small anymore...


Derive Debug trait for all structs and enums

This allows all the structs and enums to easily be placed in a struct
which derives Debug itself.
The metrics::recorder::Recorder and metrics_tracing_context::LabelFilter
traits now also extend Debug.

This PR is unreasonably wide reaching. What is it exactly that you're trying to accomplish?

The core motivation was to within a library (Ratatui) to be able to centrally define, describe and store Counter/Gauge/Histogram away from the line of code where changes are recorded, to store them for usage from that code. However these don't implement Debug, which means that I'd have to replace all the derived Debug implementations with manual ones.
After implementing those, I decided that it probably made sense to implement it for everything (per https://rust-lang.github.io/api-guidelines/interoperability.html#c-common-traits) (I'm assuming this was a mistake now and I should have checked). Several of the items where deriving Debug were problematic, and I went with what seemed to be the right approach of manual implementations / adding a Debug constraint to the traits.
Are there any particular parts of this PR that fall into the unreasonable bucket more than others, or is your perspective more that you're unconvinced of the usefulness of generally implement Debug in this crate? My perspective is that implementing Debug is generally a useful thing to do on any item that would potentially ever need to be stored in another item. Is that something you might have differences of opinion on?

Overall, I think just blanket deriving Debug has the potential to rapidly clutter debug output, especially with some of the larger structs that are being updated here... but that's not a blocking opinion.
My two no-nos here:

deriving Debug on types in binary crates (metrics-benchmark, metrics-observer)
requiring Debug on traits

For #1, this mostly just strikes me as clutter. If there was a strong need to have debug output, they'd have Debug derived or implemented already.
For #2, it doesn't make sense to make traits a supertrait of common traits unless they actually require that trait's functionality for their own behavior (i.e. the supertrait's effective behavior is the sum of the subtraits) or they would otherwise solve a large ergonomics issue, like having Send and Sync as subtraits on a trait with async functions, where it makes it cleaner downstream by not forcing callers to have to add  + Send + Sync to all of their where clauses.
In this case, neither of those scenarios is at play, and more tactically, this would only matter if you're holding a Box<dyn Recorder> (or other smart pointer), which... I don't see why anybody's code would be doing that. :P
So, all of that said: if we remove the derives from the binary crates, and from traits, I'd be OK with merging the rest.

Makes sense. I've removed the derives from the metrics-observer and benches, and removed the trait changes.
Rebased on main to pick up the other merged changes.
Clippy lints are unrelated to this change (will submit another PR to clean these up).

Rebased on main.
Fixed the unnecessary import.
Added _len debug fields on Fanout, FanoutBuilder, Router, and RouterBuilder.

No probs.
For more on how this is useful to me, see https://github.com/ratatui/ratatui/pull/1351, in particular the code where I centralize the initialization of metrics for each struct that uses them (e.g. https://github.com/ratatui/ratatui/pull/1351/files#diff-2c068b77d24467c86dd276124445d84fa48752b9c38e5f444a54f7a27ea2e0e7). I think this pattern makes it easy to use consistent IDs and language in the descriptions.
static METRICS: Lazy<Metrics> = Lazy::new(Metrics::new);

#[derive(Debug)]
struct Metrics {
    pub clear_region_count: Counter,
    pub clear_region_duration: Histogram,
    pub draw_count: Counter,
    pub draw_duration: Histogram,
    pub append_lines_count: Counter,
    pub append_lines_duration: Histogram,
    pub hide_cursor_duration: Histogram,
    pub show_cursor_duration: Histogram,
    pub get_cursor_position_duration: Histogram,
    pub set_cursor_position_duration: Histogram,
    pub size_duration: Histogram,
    pub window_size_duration: Histogram,
    pub flush_count: Counter,
    pub flush_duration: Histogram,
}

Released as part of metrics@v0.24.0/metrics-util@v0.18.0/metrics-tracing-context@v0.17.0/metrics-exporter-prometheus<EMAIL_ADDRESS>Thanks again for your contribution!


Add bin (hard and soft) descriptions to node properties

Track bin description as a property of a node so that it can be used during program rendering if desired.

Hi @rlaj,
This looks better, though still not quite right.
In an AST the fields are recognized by position and not by a named key, the fact that you had to add the attributes to Node indicates that you have gone down a wrong path slightly. Note that the existing file, line_number, description attributes are references to the test program flow source file from where the code that created the node originated - it is meta data that is there to help provide debug feedback, it does not carry semantic information about the flow represented by the AST. The generated flow should be the same if all the meta data was stripped out.
I would have a read at this if you want some more background info, it does take a while to digest and think about: http://whitequark.github.io/ast/frames.html
So, we are really looking for this to produce bin nodes that look like this:
s(:bin, 5)
s(:bin, 5, "some description)
It is the responsiblity of the flow renderer to understand that a bin node can take these two forms and if present the description will always be the second argument.
Now it is very possible that an OrigenTester update is required in parallel with this. I did have a quick look before and I noticed that I had earlier gone down a wrong path of using a dedicated bin_description node.
Let me know if you want to have a call or something to walk through this, more than happy to do that.

Made the updates.  Not completely sure I like the 'desc' method, but everything behaves as expected/requested/suggested.

Removed the desc method.