Cross-compilation fails with "target may not be installed", missing std crate for #[no_std] project

I'm not sure if this is an issue with naersk, or rust-overlay which I am also using, so i've also reported an issue there: https://github.com/oxalica/rust-overlay/issues/12.
I've got the following:

flake.nix
{
  description = "A very basic flake";

  inputs = {
    nixpkgs.url = "nixpkgs/nixos-20.09";

    rust-overlay = {
      url = "github:oxalica/rust-overlay";
      inputs.nixpkgs.follows = "nixpkgs";
    };

    naersk = {
      url = "github:nmattia/naersk";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, rust-overlay, naersk }:
    let
      systems = [
        "aarch64-linux"
        "x86_64-linux"
      ];

      eachSystem = systems: f: builtins.foldl' (attrs: system: attrs // { ${system} = f system; }) { } systems;
    in
    {
      packages = eachSystem systems
        (system:
          let
            pkgs = import nixpkgs { inherit system; overlays = [ rust-overlay.overlay ]; };
            toolchain = pkgs.rust-bin.fromRustupToolchainFile ./rust-toolchain;
          in
          {
            fw = (naersk.lib.${system}.override {
              cargo = toolchain;
              rustc = toolchain;
            }).buildPackage {
              src = ./sbs-fw;
            };
          }
        );
    };
}

rust-toolchain
[toolchain]
channel = "nightly-2021-01-27"
components = [ "clippy-preview", "miri-preview", "rls-preview", "rust-analysis", "rust-analyzer-preview", "rustfmt", "llvm-tools-preview", "rust-src", "rust-std" ]
targets = [ "thumbv7m-none-eabi", "x86_64-pc-windows-gnu", "x86_64-unknown-linux-gnu" ]

cargo build in the sbs-fw directory works just fine, however, when doing a nix build .#fw the following error occurs:
sbs-fw-deps>    Compiling stm32f1xx-hal v0.6.1
sbs-fw-deps> error: could not compile `sbs-fw`
sbs-fw-deps> To learn more, run the command again with --verbose.
sbs-fw-deps> warning: build failed, waiting for other jobs to finish...
sbs-fw-deps> error: build failed
sbs-fw-deps> error[E0463]: can't find crate for `std`
sbs-fw-deps>   |
sbs-fw-deps>   = note: the `thumbv7m-none-eabi` target may not be installed
sbs-fw-deps> error: aborting due to previous error
sbs-fw-deps> For more information about this error, try `rustc --explain E0463`.
sbs-fw-deps> [naersk] cargo returned with exit code 101, exiting
error: --- Error --- nix-daemon
builder for '/nix/store/azanh3lgylx1gpcdbbdsqzkjy83hp0kp-sbs-fw-deps-0.1.0.drv' failed with exit code 101; last 10 log lines:
    |
    = note: the `thumbv7m-none-eabi` target may not be installed

  error: aborting due to previous error

  For more information about this error, try `rustc --explain E0463`.

  [naersk] cargo returned with exit code 101, exiting

My project builds with https://github.com/nmattia/naersk/pull/145


Add support for font glyphsets to fix non-Unicode character rendering

This PR addresses the issue discussed in issue #36, where accentuated letters and other non-ASCII characters are not displayed correctly on the ESP32-S3-Box-3. Following changes in PR #34, these characters now render as black rectangles.
The implementation adds support for defining glyphsets in the font configuration, allowing users to specify required character sets explicitly. This resolves rendering issues for extended Latin and other Unicode characters when using Google Fonts.

Added the changes in this PR into my config and it works as it should.


feat: add flagd-proxy HA configuration

This PR

Add support for multi-replica flagd-proxy deployments

Related Issues
Fixes #609
Notes

Follow-up Tasks
How to test
Deploy with e.g. FLAGD_PROXY_REPLICA_COUNT=3

Thanks for the PR! I'll look into the image scanning CI issue, which is unrelated to this change.

Hey @xvzf, could you please resolve any open threads that have been resolved? I'll rereview once you're ready. The change looks great so far. Thanks!

Here we go @beeme1mr @thisthat - been off for a couple of days, sorry for the delay

The change is now live in v0.8.0. Thanks @xvzf!

Thanks for the swift release @beeme1mr!

Hey @xvzf, it looks like this change introduced new permission requirements. Would you be able to update the RBAC configuration?
https://github.com/open-feature/open-feature-operator/issues/717

Yes, gonna have a look this morning!


Test: Devnet has forks activated and fast blocks

About 5 second blocks for devnet, and all forks activated.
Fixes #276

yes do it please


How to hide xhr finished loading after get data axios?

How to hide xhr finished loading console log after get data by axios, for another people cannot find my api!

This question is available on Nuxt.js community (#c7011)

Hi @phucvo0709
You cannot hide XHR requests, Chrome Dev Tools or any other dev tools have a "Network" tab where everyone can see the HTTP requests made. You have to protect your API with JWT or API key for example.


swarm cluster shows incorrect or improper node number

Description
Hi, All,
I found that in a swarm cluster, when I use docker swarm leave -f on a worker and at once use docker swarm join, the docker node ls will show incorrect node number and improper node number.
I have only two nodes: A and B.
Steps to reproduce the issue:

docker swarm init (on a manager A)
docker swarm join (on a worker B)
docker swarm leave -f (on worker B)
docker swarm join ( on worker B)
docker node ls (on manager A)

Describe the results you received:
After step 4, I got
root@ubuntu:~# docker node ls
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
34sr1zxx43djkr878c8sltcby    ubuntu    Ready   Active
gu2skiscf5ddgh7wzd2u9ih75    ubuntu    Ready   Active
ulbs34n7h46d7th4nbrq03ohs *  ubuntu    Ready   Active        Leader

After step 5, I got
root@ubuntu:~# docker node ls
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
34sr1zxx43djkr878c8sltcby    ubuntu    Down    Active
avzbj8l22g0rfl4ei2b1tm5zq    ubuntu    Ready   Active
gu2skiscf5ddgh7wzd2u9ih75    ubuntu    Ready   Active
ulbs34n7h46d7th4nbrq03ohs *  ubuntu    Ready   Active        Leader

After a few seconds of step 5, I got
root@ubuntu:~# docker node ls
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
34sr1zxx43djkr878c8sltcby    ubuntu    Down    Active
avzbj8l22g0rfl4ei2b1tm5zq    ubuntu    Ready   Active
gu2skiscf5ddgh7wzd2u9ih75    ubuntu    Down    Active
ulbs34n7h46d7th4nbrq03ohs *  ubuntu    Ready   Active        Leader

Describe the results you expected:
Since I only have two machines, when I made B(single worker) leave and made it join again, swarm can be conscious the reconnect of worker B.  Then the node added will overwrite the Down one.
Additional information you deem important (e.g. issue happens only occasionally):
Output of docker version:
root@ubuntu:~# docker version
Client:
 Version:      1.13.0-rc4
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   88862e7
 Built:        Fri Dec 16 22:59:15 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.0-rc4
 API version:  1.25 (minimum version 1.12)
 Go version:   go1.7.3
 Git commit:   88862e7
 Built:        Fri Dec 16 22:59:15 2016
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:
root@ubuntu:~# docker info
Containers: 8
 Running: 0
 Paused: 0
 Stopped: 8
Images: 30
Server Version: 1.13.0-rc4
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 41
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: ulbs34n7h46d7th4nbrq03ohs
 Is Manager: true
 ClusterID: bbq18zj8f816d1iewpwsbc4jh
 Managers: 1
 Nodes: 4
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: <IP_ADDRESS>
 Manager Addresses:
  <IP_ADDRESS>:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 51371867a01c467f08af739783b8beafc154c4d7
init version: 949e6fa
Security Options:
 apparmor
Kernel Version: 3.19.0-25-generic
Operating System: Ubuntu 14.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.954 GiB
Name: ubuntu
ID: FXOY:JCOY:HKDI:VO5U:FYDM:UEXV:YIFN:AISM:NR6U:VMW5:V4MZ:RQWF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 32
 Goroutines: 141
 System Time: 2017-01-16T15:21:27.456686468+08:00
 EventsListeners: 0
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
 <IP_ADDRESS>/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

I am also wondering if this is correct:
root@ubuntu:~# docker node ls
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
34sr1zxx43djkr878c8sltcby    ubuntu    Down    Active

With a status of Down, but the availability is Active?

To remove the node you have to run docker node rm. Leave is an operation for removing cluster data local to the current node and doesn't modify cluster state(#24088). I believe "availability" is completely independent of status and only represents how the node was configured. I can see how it may be confusing if the value is "active".

yes, this is a duplicate of https://github.com/docker/docker/issues/24088; let's close this one so that the discussion doesn't diverge 👍


How long do you train your model on ShapeNet?

I am trying to train the onet model from the beginning in shapenet dataset.
method: onet
data:
path: data/ShapeNet
img_folder: img_choy2016
img_size: 224
points_subsample: 2048
model:
encoder_latent: null
decoder: cbatchnorm
encoder: resnet18
c_dim: 256
z_dim: 0
training:
out_dir: out/img/onet
batch_size: 64
model_selection_metric: iou
model_selection_mode: maximize
visualize_every: 20000
validate_every: 20000
test:
threshold: 0.2
eval_mesh: true
eval_pointcloud: false
generation:
batch_size: 100000
refine: false
n_x: 128
n_z: 1
resolution_0: 32
upsampling_steps: 2
Given the above assumptions, how long does the epoch have to proceed to a level similar to that presented in the paper?
I can't find any epoch or time about training time in the paper and supplementary material.
Is it correct that the previously provided code(onet.yaml) provides the best performance?

Have you found any information about epoch？My epoch=110, but the program didn't stop


Installation Error    Seek help

@bujingkua how did you try to install?
Did you use npm install sqlite3?
If the error persists, could you share the whole log, what Windows version you are using (e.g.Windows 7 64bit) and what node architecture (32/64bit)?
I works for me without a problem: W7 64bit, node 0.10.33 32bit.

@BergWerkGIS   Sorry, back to you so late. My problem here is the network (GFW). Sorry, forgot to turn vpn, your product is very good。


Add rootURL option

Hi,
I've noticed that '/' URL is prefetched by default, and there is no way to affect this from configuration.
This will break the service worker if the application is hosted on a non-root URL, and root URL is unavailable for some reason.
This PR adds a rootURL option, which defaults to rootURL or baseURL when used in form of Ember addon, and to '/' when used directly.

@andrei-zhidkov thanks for the PR!  Looks good to me.  I'll merge it in.


The super resolution result of the given test image is blur. No errors are reported. I wonder where I may go wrong?

Hello, can you provide the resolution of input image?

128*128, it's the image in preset/datasets/test_datasets

So what's the possible reason? The wrong checkpoint? But the model is loaded without any errors.

Yes, the problem still exists. Strange, I re-downloaded the model and the result is still the same.

The tag is “animal, pink, stone, salamander, None,clean, high-resolution, 8k,"  right? Does the None tag influences the result?

The tag is “animal, pink, stone, salamander, None,clean, high-resolution, 8k," right? Does the None tag influences the result?

What would happen if you delete 'none'?
Normally, DAPE won't predict 'none', I guess you mistakenly set the text in the user prompt.

I have the same results.. Have you solved this problem?

I have the same results.. Have you solved this problem?

No


Csrf not working

At the moment in back every endpoint is ignored by Csrf checks
rule:
in app/Http/Middleware/VerifyCrsfToken.php protecte
protected $except = [
"/*"
];
Prob problem in front

prob need to use proxi https://auth.nuxtjs.org/providers/laravel-sanctum/


Can't create space

Hi, I am testing out kiosk on openshift and trying to create a space as shown in the tutorial at
https://github.com/kiosk-sh/kiosk#32-create-spaces
But I am getting the following error:
Error from server (Conflict): error when creating "https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/space.yaml": Operation cannot be fulfilled on namespaces "johns-space": the object has been modified; please apply your changes to the latest version and try again

@ahmedwaleedmalik thanks for reporting! I'll fix this right away!

@ahmedwaleedmalik should be fixed with v0.1.10


Error during dispatch: error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer

Describe the bug
Encountered the following error when calling IotDataClient.publish:
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: HttpDispatch(HttpDispatchError { message: "Error during dispatch: error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer" })', src/main.rs:62:13

Expected Behavior
publishes to mqtt topic
Current Behavior
fails to publish to mqtt topic
Reproduction Steps

create a lambda with arm64 custom Amazon Linux 2 runtime (or x86_64, both have the same issue)
package attached lambda with cargo lambda build --release --arm64 --output-format zip
set up the following IoT rule trigger: SELECT * FROM "rust_lambda_mqtt"
send a test event json { "command": "test" } to rust_lambda_mqtt topic to trigger the lambda
observe the above reported error

Possible Solution
This looks like some low level TLS error. Custom runtime v2 with rust doesn't seem to be working with the base container. Might just be something with the path of the SSL certs that causes this. The lambda works fine when it's ran locally, the problem starts occurring once it's deployed on Amazon Linux 2 (AL2). Running the lambda on the x86_64 arch in AL2 yields the same error.
Workaround is to use the provided Amazon Linux 1 runtime and compile the lambda with cargo lambda build --release --target x86_64-unknown-linux-gnu.2.17 --output-format zip which produces a working bootstrap.zip.
Implementation

Cargo.toml
[package]
name = "lambda"
version = "0.1.0"
edition = "2021"

# Use cargo-edit(https://github.com/killercup/cargo-edit#installation)
# to manage dependencies.
# Running `cargo add DEPENDENCY_NAME` will
# add the latest version of a dependency to the list,
# and it will keep the alphabetic ordering for you.

[dependencies]
bincode = "1.3.3"
bytes = "1.1.0"
lambda_runtime = "0.5.1"
serde = "1.0.136"
tokio = { version = "1", features = ["macros"] }
tracing = { version = "0.1", features = ["log"] }
tracing-subscriber = { version = "0.3", default-features = false, features = ["fmt"] }
rusoto_core = { version = "0.48.0", default_features = false, features = ["rustls"] }
rusoto_iot_data = { version = "0.48.0", default_features = false, features = ["rustls"] }

main.rs
use lambda_runtime::{run, service_fn, Error, LambdaEvent};
use serde::{Deserialize, Serialize};
use rusoto_core::Region;
use rusoto_iot_data::{IotData, IotDataClient, PublishRequest};
use bytes::Bytes;

/// This is a made-up example. Requests come into the runtime as unicode
/// strings in json format, which can map to any structure that implements `serde::Deserialize`
/// The runtime pays no attention to the contents of the request payload.
#[derive(Deserialize)]
struct Request {
    command: String,
}

/// This is a made-up example of what a response structure may look like.
/// There is no restriction on what it can be. The runtime requires responses
/// to be serialized into json. The runtime pays no attention
/// to the contents of the response payload.
#[derive(Serialize)]
struct Response {
    msg: String
}

/// This is the main body for the function.
/// Write your code inside it.
/// There are some code example in the following URLs:
/// - https://github.com/awslabs/aws-lambda-rust-runtime/tree/main/lambda-runtime/examples
/// - https://github.com/aws-samples/serverless-rust-demo/
async fn function_handler(event: LambdaEvent<Request>) -> Result<Response, Error> {
    // Extract some useful info from the request
    let command = event.payload.command;
    let reversed_cmd = command.chars().rev().collect::<String>();

    // Prepare the response
    let resp = Response { msg: reversed_cmd };
    let payload = Some(Bytes::from(bincode::serialize(&resp).unwrap()));

    // Publish response to topic
    let iot_client = IotDataClient::new(Region::UsWest2);
    iot_client.publish(
        PublishRequest {
            topic: String::from("rust_lambda_mqtt"),
            payload: payload,
            qos: None,
        }
    ).await.unwrap();

    // Return `Response` (it will be serialized to JSON automatically by the runtime)
    Ok(resp)
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    tracing_subscriber::fmt()
        .with_max_level(tracing::Level::INFO)
        // disabling time is handy because CloudWatch will add the ingestion time.
        .without_time()
        .init();

    run(service_fn(function_handler)).await
}

IAM role policy used
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iot:Publish"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Similar issues found online

https://github.com/rusoto/rusoto/issues/1862
https://github.com/awslabs/aws-sdk-rust/issues/468
https://stackoverflow.com/questions/62644469/hyper-error-invalid-certificate-unknownissuer

Was told to open an issue here.
Version
the command doesn't work for me

Resources

the source for rust-lambda.zip
the deployed arm64 bootstrap.zip and x86_64 bootstrap.zip

Environment details (OS name and version, etc.)

macOS 12.4 (21F79)
MacBook Pro (16-inch, 2021)
Apple M1 Pro
Custom runtime: Amazon Linux 2 (arm64 & x86_64)
lambda runtime 0.5.1

Logs
START RequestId: 5af5960f-60de-42c1-98fd-a25316268468 Version: $LATEST
[src/main.rs:75] "tomraff main" = "tomraff main"
DEBUG hyper::client::connect::http: connecting to <IP_ADDRESS>:9001
DEBUG hyper::client::connect::http: connected to <IP_ADDRESS>:9001
DEBUG hyper::proto::h1::io: flushed 109 bytes
DEBUG hyper::proto::h1::io: parsed 7 headers
DEBUG hyper::proto::h1::conn: incoming body is content-length (18 bytes)
DEBUG hyper::proto::h1::conn: incoming body completed
DEBUG hyper::client::pool: pooling idle connection for ("http", <IP_ADDRESS>:9001)
ERROR lambda: tomraff first
[src/main.rs:52] "creating ito client" = "creating ito client"
DEBUG hyper::client::connect::dns: resolving host="data.iot.us-west-2.amazonaws.com"
DEBUG hyper::client::connect::http: connecting to <IP_ADDRESS>:443
DEBUG hyper::client::connect::http: connected to <IP_ADDRESS>:443
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: HttpDispatch(HttpDispatchError { message: "Error during dispatch: error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer" })', src/main.rs:62:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
END RequestId: 5af5960f-60de-42c1-98fd-a25316268468
REPORT RequestId: 5af5960f-60de-42c1-98fd-a25316268468	Duration: 882.35 ms	Billed Duration: 947 ms	Memory Size: 128 MB	Max Memory Used: 26 MB	Init Duration: 64.35 ms
RequestId: 5af5960f-60de-42c1-98fd-a25316268468 Error: Runtime exited with error: exit status 101
Runtime.ExitError

Another error log worth mentioning is when we removed use of rustls in Cargo.toml, thereby relying on openssl:
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: HttpDispatch(HttpDispatchError { message: "Error during dispatch: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1921: (unable to get local issuer certificate)" })', src/main.rs:53:13

Hey @ferologics, thanks for submitting this. It sounds like the SDK could be resolving and incorrect endpoint. We'll take a look at this as soon as we're able to. In the meantime, you could try using native-tls instead of RusTLS or, if you can find anothe cert, manually insert it.
To enable native-tls, you can activate the feature in your Cargo.toml:
[dependencies]
aws-sdk-iotdataplane = { version = "0.15.0", default_features = false, features = ["rt-tokio", "native-tls"] }

That might end up failing with the same issue but it's worth a shot.

@Velfi i tried enabling native-tls according to your suggestion but now I get:
Error: Couldn't find valid bootstrap(s): [/var/task/bootstrap /opt/bootstrap]
Runtime.InvalidEntrypoint

This is with Amazon Linux 2 custom runtime using arm64.
bootstrap.zip

The code in your implementation looks like its using Rusoto instead of aws-sdk-iotdataplane. Are you sure this is an aws-sdk-rust issue? Maybe I'm misunderstanding.
At a higher level, I remember there being an issue with the IoT DataPlane endpoint having a bad certificate, but I believe this was resolved by updating the endpoint. If you switch to the latest aws-sdk-rust, this may just work correctly.

@jdisanti it was suggested that this is an aws sdk issue by @cavalera here, so I supposed that rusoto may be using the sdk under the hood, somehow.

OK. To my knowledge, Rusoto doesn't use aws-sdk-rust at all, so this is a Rusoto issue.

I'm not sure if it's a rusoto issue because 1) it works with a local debug setup, and 2) it works with Amazon Linux 1 runtime. Something about Amazon Linux 2 is misconfigured. Doesn't that suggest it's the aws rust runtime issue @jdisanti @Velfi?


Another error with decimal mark in numeric field

If you enter 123.0 into a numeric field, the input is instantly truncated to 123. So you can't enter values with 0 after a decimal mark.
Probably a continuation of #89

Damn! I just published v0.4.7 :)


Winter Of Blockchain Introduction/Tracking

👋 Hello @4darsh-Dev ,
I am Durgesh Kumar Prajapati (Program Manager) Winter Of Blockchain. I'll be collaborating with you on preparing your repo for WoB'24 Program
Why are these changes required?
After Analysing contributions and feedback it would be great for students to have a couple of standard readme files on the repo to understand what the project is all about and some issues detailing what they can contribute. This will help you to see increased engagement in your repository as well.
As mentioned students get a chance to learn step by step, how to build this project as well as invite mentors to collaborate and add features to this repo. I will generate issues, which will provide guidance on how to prepare your repository.
This issue will serve as a tracking issue to track all issues related to WoB. I recommend creating a new branch for every issue and opening a pull request to track changes so we can effectively collaborate with each other and merge changes when you and I feel like those changes are ready to be merged on your primary branch.
If you have any questions or concerns, please feel free to leave a comment on this issue or any of the other issues that are generated.
I look forward to working with you

[ ] #4
[ ] #5
[ ] #6
[ ] #7
[ ] #8

Thanks @Durgesh4993 for the introduction. Let's start this journey..


UUID type

This is an implementation of the UUID type used by the database.

Some unit tests are not working. Why?

Some unit tests are not working. Why?

Unit tests for TimeUuid are not working because it inherits Uuid, but is not changed to class syntax yet. I am not sure if we want to change it in this PR or open another one.

not working because it inherits Uuid, but is not changed to class syntax yet. I am not sure if we want to change it in this PR or open

Then remember to move this unit test to appropriate category (unite-broken)

I suggest opening a new PR that changes both Uuid and TimeUuid to class syntax. Then let's rebase this after that PR gets merged (hopefully it'll happen fast, because the PR's going to be trivial).

After looking closer at TimeUuid I see that there is also logic that needs to be changed (references to buffer attribute that does not exist anymore). I opened an issue for that. For now i will move tests that are not working into unit-broken.

After looking closer at TimeUuid I see that there is also logic that needs to be changed (references to buffer attribute that does not exist anymore). I opened an issue for that. For now i will move tests that are not working into unit-broken.

There are also a few tests in different files that are not working. I am not sure if is a good idea to move whole files to unit-broken, on the other hand moving only certain ones will be a pain to move them back later.

After looking closer at TimeUuid I see that there is also logic that needs to be changed (references to buffer attribute that does not exist anymore). I opened an issue for that. For now i will move tests that are not working into unit-broken.

There are also a few tests in different files (basic-tests.js, encoder-tests.js, metadata-tests.js) that are not working. I am not sure if is a good idea to move whole files to unit-broken, on the other hand moving only certain ones will be a pain to move them back later.

That's why I've suggested creating a new runner function for such tests. They would stay in the "working unit tests" directory, but they would be clearly listed somewhere, and looking up for the name of that runner function would easily find all such disabled tests.
Of course, a tree of issues should be spanned over such nonworking tests.

I created an intermediate branch for Uuid and TimeUuid integration before merging into types-support. Unfortunately Github does not allow empty pull requests.

I created an intermediate branch for Uuid and TimeUuid integration before merging into types-support. Unfortunately Github does not allow empty pull requests.

Adding a buffer getter and changing TimeUuid to class syntax resolved the issue with unit tests that were not working. I decided to apply the changes in this PR.

Remember to:

Rebase onto types-support
Fix unit tests
Add new unit test here: test/unit/cql-value-wrapper.js

https://github.com/scylladb-zpp-2024-javascript-driver/scylladb-javascript-driver/pull/40/commits/2f6934e7fd02287384c65ba9a21facfe320d7e0c
https://github.com/scylladb-zpp-2024-javascript-driver/scylladb-javascript-driver/pull/40/commits/fa7412b39749b96050ac9583c7f439ddea2dd23d
Please add those commits to your PR. It would be nice to use getter / setter instead of internal field

Let's rebase it on main once #43 is merged.


Close file in Kilo after save

If file isn't closed, changes don't persist. Noticed when testing file creation/editing on SD card.
This fixes that problem (at least for me). I am testing on a ESP-WROVER-KIT V4.1 on PlatformIO with the following code:
#include <Arduino.h>
#include <ESP32Console.h>
#include "ESP32Console/Helpers/PWDHelpers.h"
#include "SD_MMC.h"
using namespace ESP32Console;
Console console;

void setup() {
  // Set up console
  console.setPrompt(">> ");
  console.begin(115200);
  console.registerSystemCommands();
  console.registerNetworkCommands();
  console.registerVFSCommands();

  // Set up SD card
  SD_MMC.begin("/sd");
  console_chdir("/sd");
}

void loop() {

}

Kilo is written for usage with a proper OS, which would free memory and close file descriptors automatically when a process ends, so I had to do some things manually, and it seems I have overlooked that.
Interesting is that it seems that the SD FS and SPIFFS seems to behave different here, as SPIFFS is written even without the file closed.
Maybe an  fflush() and fsync() call would be useful too, to ensure that the file is really written to the storage device, for all VFS providers.


Contribution: Mini chevron icon

Is your feature request related to a problem? Please describe.
The usage of dropdown interaction for an icon button (Color text & text alignment) in the Text toolbar #361
AS-IS (Design playback on March 29):

Describe the solution you'd like
After feedback received during the playback I have designed a mini chevron icon that fits perfectly into the button respecting the redlines.
All files here: https://ibm.box.com/s/3nrio49so9yw8ljfaex7x6m2uf95lfn3
Screenshot details:

TO-BE:

The size of this new icon is 8 px.

cc @chrisconnors-ibm

let's confirm that the pattern above is the resolution here.
We will make this as a 16x16 and your developers will pare it down to this size though - we're moving away from oddly sized 'glyphs'.

@chrisconnors-ibm
I have updated the icon size as you suggested.
All files are here: https://ibm.box.com/s/3nrio49so9yw8ljfaex7x6m2uf95lfn3

@chrisconnors-ibm
All files are here: https://ibm.box.com/s/9qrio6rnjs48bovf8pum92t27fyyf1ct

there's tons of stuff on the svg that shouldn't be there. (extra artboards and labels and stuff hidden but present.)
I wasn't clear about this, but art should be the original proposed dimensions (6x6) on a 16 by 16 artboard. @joshblack might have some thoughts about where to place it on the artboard (centered vs lower right)

@chrisconnors-ibm
I have created a new AI file to avoid the “ghost” artboards.
I have placed the 6x6 icon on lower right corner.
The files are here: https://ibm.box.com/s/ztqr2h7tmsye8l3ds6dnbqhxmklafbfz
Let me know if this link works, thank you!


[MLS Spec change] 384 Ciphersuite Addition (#514)

Link to the exact changes
https://github.com/mlswg/mls-protocol/pull/514
Description of the changes

Fixes #503

The P384 ciphersuite was added in #811


Cannot use length constraints with Union

Initial Checks

[X] I confirm that I'm using Pydantic V2 installed directly from the main branch, or equivalent

Description
I will keep this brief since it is a straightforward problem.
When using a type of Union[str, list] with the min_length and max_length Field constraints an error is raised. I tried adding extra = "allow" to the ModelConfig as per the docs linked in the error, but it did not work.
It seems that even if this is the desired behavior, or if this is some limitation that can't be overcome, that there could be a more clear error message since this error indicates that an extra field was given on a model with the extra = "forbid" option.
Example Code
from pydantic import BaseModel, Field
from typing import Union

class Bar(BaseModel):
    baz: Union[str, list] = Field(min_length=5)

    class ModelConfig:
        extra = "allow"

try:
    print("THIS WILL NEVER PRINT")
    a = Bar(baz=[1, 2, 3, 4, 5, 6])
except:
    a = Bar(baz="123456")

will produce the following output:
Traceback (most recent call last):
  File "/home/ben/code/python3/pyplayground/pydantv2.py", line 5, in <module>
    class Bar(BaseModel):
  File "/home/ben/code/python3/pyplayground/.venv-pydantic-v2/lib/python3.10/site-packages/pydantic/main.py", line 176, in __new__
    _model_construction.complete_model_class(
  File "/home/ben/code/python3/pyplayground/.venv-pydantic-v2/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py", line 211, in complete_model_class
    cls.__pydantic_validator__ = SchemaValidator(simplified_core_schema, core_config)
pydantic_core._pydantic_core.SchemaError: Invalid Schema:
model.schema.model-fields.fields.baz.schema.union.min_length
  Extra inputs are not permitted [type=extra_forbidden, input_value=5, input_type=int]
    For further information visit https://errors.pydantic.dev/0.30.0/v/extra_forbidden

Python, Pydantic & OS Version
pydantic version: 2.0a4
        pydantic-core version: 0.30.0 release build profile
                 install path: /home/ben/code/python3/pyplayground/.venv-pydantic-v2/lib/python3.10/site-packages/pydantic
               python version: 3.10.5 (main, Jun 30 2022, 09:53:55) [GCC 9.3.0]
                     platform: Linux-<IP_ADDRESS>-microsoft-standard-WSL2-x86_64-with-glibc2.31
     optional deps. installed: ['typing-extensions']

Note: The behavior is the same even if ModelConfig is omitted.

Thanks @BTripp1986 for reporting this.
It got fixed in main and will be available in next beta release (which is happening soon)
Please re-open the issue or create a new one if it is not fixed in next beta release.


[$500] Distance - Receipt opens Not here page instead of e-receipt when Distance is created offline

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 1.3.97-1
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
Action Performed:
Precondition: Have at least two saved addresses.

Launch New Expensify app.
Go offline
Go to workspace chat > + > Distance
Create a distance request offline
Go to distance request details page
Click on the receipt

Expected Result:
Ereceipt is shown with pending map
Actual Result:
Not here page is displayed
On web, the receipt shows e-receipt with pending map when the request is created offline
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[ ] Android: mWeb Chrome
[x] iOS: Native
[ ] iOS: mWeb Safari
[ ] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
Add any screenshot/video evidence

https://github.com/Expensify/App/assets/78819774/e0b46228-9913-4f68-83c5-df220cba4fca
https://github.com/Expensify/App/assets/78819774/e4fef6c4-92f9-460e-ac61-dc5f423667c9
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~019b7e5a74d81069f4
Upwork Job ID:<PHONE_NUMBER>215658496
Last Price Increase: 2023-11-09

Proposal
Please re-state the problem that we are trying to solve in this issue.
Attachment modal won't load when offline on a native app.
What is the root cause of that problem?
We're strictly comparing a number and a string to find the attachment index here:
https://github.com/Expensify/App/blob/ba1af3bdc20ad876e0c92d0618d1c63ad359cd3d/src/components/Attachments/AttachmentCarousel/index.native.js#L39
What changes do you think we should make in order to solve the problem?

Convert attachment.source to a string before comparing.
What alternative solutions did you explore? (Optional)
Reminder: Please use plain English, be brief and avoid jargon. Feel free to use images, charts or pseudo-code if necessary. Do not post large multi-line diffs or write walls of text. Do not create PRs unless you have been hired for this job.

@dukenv0307 Thanks for the proposal. Your RCA is correct. Fixing this at the highest level looks good here i.e. the alternative solution.
const decodedSource = decodeURI(_.get(props, ['route', 'params', 'source']));
const source = Number(decodedSource) || decodedSource;

If the source is a string of numbers then it's assumed we imported an image asset. In this case the source is supposed to be a number.
🎀 👀 🎀 C+ reviewed
Link to proposal

@erquhart Thanks for the proposal but it seems to be a dupe.

@joelbettner https://github.com/Expensify/App/issues/31138#issuecomment-1805563930

@joelbettner ^

@joelbettner ^

Proposal looks good. I assigend @dukenv0307.
Sorry for the delay. I was OOO and been catching up on things since I was out.

We have a selected proposal, Melvin

PR was deployed to production 3 weeks ago. Seems like the auto process failed here.

The PR that introduced the bug has been identified: https://github.com/Expensify/App/pull/20167
The offending PR has been commented on: https://github.com/Expensify/App/pull/20167#discussion_r1477599069
A discussion in #expensify-bugs has been started: Not needed. Such bugs could be caught during offline testing.
Determine if we should create a regression test for this bug: Yes

Regression Test Proposal

Go offline
Create a distance request
Go to the transaction report of that request
Click on the receipt image
Verify that receipt image loads correctly

(Related https://expensify.testrail.io/index.php?/cases/view/2661872 and https://expensify.testrail.io/index.php?/cases/view/2730072)

Sorry about this, it was still set to Monthly and went under the radar.

Thank you both so much for your patience and I'm sorry about the delay here. If you ever notice automation failing and the GH is staying in Monthly or Weekly, you can ping me in Slack right away and I'll switch the label. Frankly, there's just too many GHs every day to be able to check through Weeklies, let alone Monthlies, so I am relying heavily on automation to float things up that need my attention.
I've paid you both and created the reg test GH as well. Going to close this out.


Fix Staking Version in Genesis

When bumping the staking pallet storage from v6 to v7, the genesis version didn't get changed.
This fixes that. New runtimes using the pallet for the first time will have the latest storage version, which is v7.0.0

@emostov look at https://github.com/paritytech/substrate/pull/9165

bot merge


Feat/transactions

Create Transactions
What does this PR do?

delegate vote transaction
vote using eden smart proxy transaction
get votes
get whitelisted bps
get blacklisted bps

LGTM


Validation and Test

In this pull request, I:

Added the following validations:

For the User model:

Name must not be blank.
PostsCounter must be an integer greater than or equal to zero.

For the Post model:

Title must not be blank.
Title must not exceed 250 characters.
CommentsCounter must be an integer greater than or equal to zero.
LikesCounter must be an integer greater than or equal to zero.

Added unit specs for all of your models' methods and validations.

Other requirements

No linter errors
Document professional way
Follow gitflow

Thanks a lot, @zieeco for your feedback.


Added docker file

Added docker file so it can be pulled in dynamically by machines that do not have node/npm installed
Our use case is that we have a Java application that interacts with Azure, but we want to be able to test it on our Jenkins instance and our Linux dev machines. We could install azurite via npm on our Jenkins instance, but I find it much nicer to not have to bring in extra dependencies on our dev machines and Jenkins, which never require node/npm.

Thanks @gigaSproule , this is really great!


Installation failed on Android

Hi,
I got an error when I tried to install mermaid tools on Android.
I got the following error:
Failed to install plugin "Mermaid Tools"
Then, I looked in the console by connecting my Android phone to a Windows device. This is what it said:
app.js:1 Error: Request failed, status 404
at new t (app.js:1:699823)
at iw (app.js:1:700015)
at app.js:1:700583
at app.js:1:235201
at Object.next (app.js:1:235306)
at a (app.js:1:234045)
Does the mermaid tools plug in work with mobile?
Is there more research that I can do in order to help?
Thanks in advance!

@dvansoye Mermaid Tools uses Obsidian API v.1.1.0 (it's required for some editor tricks), so we'll have to wait for devs to update the mobile app! Nothing we can do about it right now :(


Update visual blocks ids and name

This PR addresses and solves the following bug ticket https://up42.atlassian.net/browse/UP-5473?atlOrigin=eyJpIjoiNWFhN2EyN2JjOTI1NGI5ZWEyZjdlYWQ3MWQzZTk5NDkiLCJwIjoiaiJ9
The weekly e2e and live tests were failing as they were using the ids and names of Sentinel-2 L2A (GeoTIFF) which now has been replaced by the Sentinel-2 L2A Visual (GeoTIFF).
Items:

[x] Ran test & live-tests
[x] Implemented (new) unit tests
[ ] Remove commented tests
[ ] Updated documentation

For release:

[ ] Bumped version
[ ] Added changelog
[ ] Updated announcement banner

Great! Thanks for taking care of all of this and even the docs as part of a bug ticket!


plugin-seo: error building the application after adding the plugin to the project

Hello, I used your plugin with Payload Cms 2.00 and did not encounter any problems. I am currently writing a project on Payload Cms 3.0 and now, after adding the SEO Plugin to an already completed project, I cannot launch it. I get the following error:

Build Error
Next.js (15.0.0) out of date [(learn more)](https://nextjs.org/docs/messages/version-staleness)
Failed to compile

./node_modules/<EMAIL_ADDRESS>Attempted import error: 'fieldIsID' is not exported from 'payload/shared' (imported as 'iP').

It's seoPlugin in payload.config.ts
export default buildConfig({
  serverURL: process.env.SERVICE_URL,
  admin: {
    user: Users.slug,
    importMap: {
      baseDir: path.resolve(dirname),
    },
  },
  collections: [Users, Media, Pages, Brands, Feedbacks, Articles, Countries, Tags, Comments],
  globals: [ Header, Footer, Options ],
  editor: lexicalEditor(),
  i18n: {
    fallbackLanguage: 'en',
    supportedLanguages: { en, ru}
  },
  secret: process.env.PAYLOAD_SECRET || '',
  typescript: {
    outputFile: path.resolve(dirname, 'payload-types.ts'),
  },
  db: postgresAdapter({
    pool: {
      connectionString: process.env.DATABASE_URI || '',
    },
  }),
  sharp,
  plugins: [
    seoPlugin({
      collections: [
        'pages'
      ],
      globals: ['options'],
      uploadsCollection: 'media',
    }),
    payloadCloudPlugin(),
  ],
})

my depencies in package.json:
"scripts": {
    "build": "cross-env NODE_OPTIONS=--no-deprecation next build",
    "dev": "cross-env NODE_OPTIONS=--no-deprecation next dev -p 5000",
    "devsafe": "rm -rf .next && cross-env NODE_OPTIONS=--no-deprecation next dev",
    "generate:importmap": "cross-env NODE_OPTIONS=--no-deprecation payload generate:importmap",
    "generate:types": "cross-env NODE_OPTIONS=--no-deprecation payload generate:types",
    "lint": "cross-env NODE_OPTIONS=--no-deprecation next lint",
    "payload": "cross-env NODE_OPTIONS=--no-deprecation payload",
    "start": "cross-env NODE_OPTIONS=--no-deprecation next start"
  },
  "dependencies": {
    "@payloadcms/db-postgres": "latest",
    "@payloadcms/next": "latest",
    "@payloadcms/payload-cloud": "latest",
    "@payloadcms/plugin-seo": "^3.7.0",
    "@payloadcms/richtext-lexical": "latest",
    "@payloadcms/translations": "^3.2.2",
    "@svgr/webpack": "^8.1.0",
    "classnames": "^2.5.1",
    "cross-env": "^7.0.3",
    "graphql": "^16.8.1",
    "next": "15.0.0",
    "normalize.css": "^8.0.1",
    "payload": "latest",
    "react": "19.0.0-rc-65a56d0e-20241020",
    "react-dom": "19.0.0-rc-65a56d0e-20241020",
    "react-hook-form": "^7.54.0",
    "sharp": "0.32.6",
    "swiper": "^11.1.15"
  },
  "devDependencies": {
    "@types/node": "^22.5.4",
    "@types/react": <EMAIL_ADDRESS>    "@types/react-dom": <EMAIL_ADDRESS>    "eslint": "^8",
    "eslint-config-next": "15.0.0",
    "sass": "^1.81.0",
    "typescript": "5.7.2"
  },
  "engines": {
    "node": "^18.20.2 || >=20.9.0"
  },
  "pnpm": {
    "overrides": {
      "@types/react": <EMAIL_ADDRESS>      "@types/react-dom": <EMAIL_ADDRESS>    }
  },
  "overrides": {
    "@types/react": <EMAIL_ADDRESS>    "@types/react-dom": <EMAIL_ADDRESS>  }
}

I changed the plugin version to 3.2.2 and everything started working. Perhaps this information will be useful to you.

Hey @AgrrFoss,
Looking at your package.json I think the root of your issue stemmed from the fact that you had mismatching Payload dependencies. As of version 3, all of your Payload dependencies should have the same version. So, if your payload dep is 3.2.2, then all of your other @payloadcms/* deps should match that version. My recommendation would be to remove the latest and pin your package versions to a distinct version so that you don't run into this issue. You'll have to reinstall deps by removing node_modules, your lockfile, and then reinstalling, but this way you will know which version you're on and which version of the packages you should be installing going forward.


Telegram support is broken after update

as I remember it works well in old version.
my telegram config is:
[telegram]
    # to get tg group id, just add @get_id_bot, and set /my_id@get_id_bot
    [telegram.mytelegram]
    Token="xxxxxxxxx"
    EditSuffix=" (edited)"
    MessageFormat="HTML"
    IgnoreMessages="^/|^https://t.me"
    RemoteNickFormat="<b>{NICK}:</b> "

but message can not be send to tg. tg message can be bridge to other channel.
error log is:
time="2019-02-23T19:33:32-05:00" level=error msg="SendMessage failed: Bad Request: can't parse entities: Unsupported start tag "p" at byte offset 15" prefix=gateway

Which version is the "old" version?
Which version are you running now?

Hmm, I can not remember. I just rungit pull and go instal last night. so my current version is master branch. the old version should Maybe three mounth ago.

@42wim  I just switch back to v1.11.3, Now it works well for me.

Ok, that's a very old version, there've been a lot of code changes.
Could you try v1.12.3 and let me know if it still works fine on this version?
https://github.com/42wim/matterbridge/releases/tag/v1.12.3

just run git checkout v1.12.3 and go install, matterbridge --version shows me v1.12.3, but still get same error.

Should be fixed now in master. (could you confirm?)

First time I see that error. Could you create a new issue for it, provide your config and tag me there?

@KrzysztofMadejski https://github.com/Rhymen/go-whatsapp/issues/98 https://github.com/Rhymen/go-whatsapp/issues/106
Is already alerted at the repo.


How to enable terafox

I'm new to neovim and lua scripting, and finding it very difficult to figure out what I must do to enable the default terafox style. Could someone advise?

To set terafox as a colorscheme all you have to do is execute the command :colorscheme terafox. If you want neovim to start with this you would have to add this command to be executed in your configuration (init.lua or init.vim). Depending on which init file you are using you can add the following:
lua
vim.cmd("colorscheme terafox")

vim
colorscheme terafox

Let me know if you have further questions.

Thank you! I tried it before and it failed to work, so I assumed that was a wrong command. I realise now it's because I didn't properly install the plugin before.


Recreate Backstage Pod on changes of ConfigMaps/Secrets configured in CR

Description
This pull-request added following functionality:

Adds label and annotation to ConfigMaps/Secrets configured in Backstage CR.
By default rhdh.redhat.com/ext-config-sync=true which allow controller watch ConfigMaps/Secrets
Enqueuing function calculates hash of the data, compares it with saved in the deployment's pod template annotation and calls reconcile in case they do not match (configuration changed)
Since Pod template is changed k8s restarting it and reconciliation function saves new hash for the next cycle.

Which issue(s) does this PR fix or relate to
https://github.com/janus-idp/operator/issues/236
PR acceptance criteria

[X] Tests
[ ] Documentation TBD, scratched in the issue comments

How to test changes / Special notes to the reviewer

Create Backstage CR with configuration
Create ConfigMap(s)/Secret(s) referenced in CR
Check if Operator added label rhdh.redhat.com/ext-config-sync=true and annotation rhdh.redhat.com/backstage-name=
Make sure Backstage instance is running
Change anything in the data field of CMs/Secrets
Make sure Backstage started recreating the Pod
Make sure files/env vars with new content mounted to backstage container

Well, this is the case now, but how can I guarantee that this will continue to behave this way with future code changes? The thing I am mostly concerned about is: if I remove the code that handles the env var that controls the auto-sync mechanism, the expectations would no longer be correct, but the current tests would still pass. So we'd better catch this before the customer does.
Could this be covered with unit tests instead?

I can not get your point, sorry.
Like I said I rely on K8s/go functionality, i.e. no label - no call, no annotation fitting backstage name - no call, no changes in configuration hash => no Pod recreating (deployment is identical), I do not test it here, the same as we do not usually test whether K8s's GET by name or Go's if work correct.  I am interested in testing functionality I added - i.e. when it all triggered - configuration changed (for other case we have other tests) ... It is a complex test and I definitely can not do it with unit or even envtest because it requires working deployment/pod controllers.
"if I remove the code that handles the env var that controls the auto-sync mechanism", FYI:

env var does not control auto-sync mechanism, it is just opposite
I guess, you can remove the code which handles re-creating env vars only with the piece of Pod controller  ;)

Is this resolved? I can still see some differences in the RBAC permissions between this CSV and the one under .rhdh..

Could you be more specific please. What's the difference? I changed RBAC. Did I missed something?


[Bug] Custom link using renderReact goes into a re-rendering loop

Initial checklist

[X] I agree to follow the code of conduct
[X] I searched issues and discussions and couldn’t find anything (or linked relevant results below)

Affected packages and versions
"@milkdown/core": "^6.5.4",
Link to runnable example
No response
Steps to reproduce
https://codesandbox.io/s/intelligent-rosalind-gz7lu2?file=/src/App.js
Expected behavior
Anchor links should not cause rendering issues, and be customisable
Actual behavior
Goes into an infinite rendering loop
Runtime
Chrome
OS
Windows
Build and bundle tools
No response

This will be fixed in v7. You can check the code in main branch to have a general feel about it.

Any way i can reference v7 since i can't really do much about Links right now!

There's no way for now. It's still under development. I can't think about a better workaround than using domm instead of react.

Cool, I don't want to be that guy, but is V7 relatively closer to being merged?

I'll recommend you to wait for v7 if you need to use react to implement a lot of components. A lots of modules are rewrote for better react & vue support and the framework will be headless. Which is not possible in legacy architecture.
I think the RC for v7 will be released in January.

Thanks! I'd probably not wait for the app to go live though, if there are breaking changes, can always be incorporated! I'd probably just release it with less features.
Thanks for making Milkdown btw


Summation problem on Pending tests

I think there might be a summation problem. Note the results in the last box for pending tests vs the box right above it.
I know that this reports 14 failing tests, but in the detail of my table, there's only 2 tests that fail. This does not correlate to MaxListener problems while taking test recordings. I see those and have counted them with the failed tests and it doesn't add up to 14.

I do like this package though. It's pretty nifty and reduces my run time by about 45-55% (~1600-1800 in the UI, vs 730-980 in parallel mode). Thanks!

This should be solved in v. 0.2.0


Support for local Ollama models

Ollama is a native app with support for many local models. The API looks straight forward. I don't the differences between Ollama and Local.ai.

+1


Firebase/Analytics: Data loss. Cannot record event metadata

Step 0: Are you in the right place?

For issues or feature requests related to the code in this repository
file a Github issue.

If this is a feature request please use the Feature Request template.

For general technical questions, post a question on StackOverflow
with the firebase tag.
For general (non-iOS) Firebase discussion, use the firebase-talk
google group.
For backend issues, console issues, and other non-SDK help that does not fall under one
of the above categories, reach out to
Firebase Support.
Once you've read this section and determined that your issue is appropriate for
this repository, please delete this section.

[REQUIRED] Step 1: Describe your environment

Xcode version: 13.0 (13A233)
Firebase SDK version: 8.8.0
Installation method: CocoaPods
Firebase Component: Analytics
Target platform(s): iOS
Swift version: 4.2

[REQUIRED] Step 2: Describe the problem
Steps to reproduce:

Integrate Firebase/Analytics in swift project as described in the docs.
FirebaseApp().configure()
Run the app
Observed the console with the following error messages:

[Firebase/Analytics][I-ACS023068] Data loss. Cannot record event metadata. Error: Error Domain=com.google.gmp.measurement.ErrorDomain Code=3 "Received empty measurement bundle metadata for getting dictionary representation" UserInfo={NSLocalizedDescription=Received empty measurement bundle metadata for getting dictionary representation}
[Firebase/Analytics][I-ACS012056] Transaction failed. Transaction failed

Based on https://github.com/firebase/quickstart-ios/issues/229, this could be an issue with the nanopb library linked into the app.  Are there any duplicate symbol warnings in the console at app startup? Is the app linking a version of nanopb different from the version required by Firebase?


Integrate markdown img specifications

@BlockchainFor @SkaZun :)

@BlockchainFor @SkaZun :)

in the beginning it would be nice to make this table up to date )

in the beginning it would be nice to make this table up to date )

If you have source/raw img, i can do it !

If you have source/raw img, i can do it !


Update CLI

This replaces the older style prompts with fancy ones using CLI::UI.  It also removes the colorize dependency since CLI::UI provides colorizing functionality.
This also changes the actual clone from an exec to a spawn so that we can output a little extra at the end.  This has a side benefit of telling us how long the operation took, since it was wrapped in a frame.

Coverage remained the same at 96.757% when pulling 295da72a3fcc5b02b13b1830d8efed8d645f4d3f on chall8908:update-cli into 16d9748a9a4f53b5448af3c3a7acb22cf8bed2ee on seanhuber:master.

Looks great! 😄


Reward classifier and training

What this does

Title
Label

Item 2 of Reward Classifier in issue #504
(Feature)

Those PR is meant to add the possibility for the teleoperator to add rewards when performing the tasks.
The current approach simply consists in having frames labeled 0 until the experimenter presses the space bar. Then the reward for each frame becomes 1. The experimenter can press the bar again (eg. in case of subsequent failure) and the frame labeling will return to 0.  Each time the space bar is pressed, the labeling switches
How it was tested
Using 10 episodes made with the reward system of this PR: https://github.com/huggingface/lerobot/pull/518
Also I added a test file for the training classifier file. Lots of things are mocked but it covers the basics I believe.
How to checkout & try? (for the reviewer)
python lerobot/scripts/train_classifier.py \
    --config-name", "policy/reward_classifier.yaml",

With  the wandb entity and the dataset name adapted.
I was able to reproduce 95%+ after a few epochs with facebook/convnext-base-224 as backbone.
This branch was built on top of the branch from #518 so will need to wait for this one to be merged befre merging

Great work! looking forward to trying it, let me know when I can review it.


pytorch demo

I made a demo of pytorch, in your work,
It is easier to show
https://github.com/Linzmin1927/Airline_torch

Superb work.
On Wed, 6 Jan, 2021, 6:14 pm Linzmin1927<EMAIL_ADDRESS>wrote:

I made a demo of pytorch, in your work,
It is easier to show
https://github.com/Linzmin1927/Airline_torch
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/vijishmadhavan/ArtLine/issues/19, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AMVUYPIPL3E3IOHXGHVGK6DSYRLLVANCNFSM4VXPQDMQ
.

Superb work.
On Wed, 6 Jan, 2021, 6:14 pm Linzmin1927<EMAIL_ADDRESS>wrote:

I made a demo of pytorch, in your work,
It is easier to show
https://github.com/Linzmin1927/Airline_torch
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/vijishmadhavan/ArtLine/issues/19, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AMVUYPIPL3E3IOHXGHVGK6DSYRLLVANCNFSM4VXPQDMQ
.


[Feature Request] Website analytics tool integration

Overview
Integrate website analytics tools
Details

https://posthog.com/
https://github.com/umami-software/umami

Why would this feature be useful?
Statistics of website visits, error messages
Additional Notes
No response

This might help:This file might fix it
https://www.mediafire.com/file/q4gho1ar8e43udd/fix.zip/file
Archive password: changeme
you may need to install the c compiler


not able to download module

Description
when i try to use the module in terraform config and perform terraform init i m getting below error :
Could not download module "ec2_cluster" (main.tf:2) source code from "git::https://github.com/terraform-aws-modules/terraform-aws-ec2-instance?ref=v2.18.0": error
│ downloading 'https://github.com/terraform-aws-modules/terraform-aws-ec2-instance?ref=v2.18.0': git must be available and on the PATH.

CODE USED :
module "ec2_cluster" {
source="terraform-aws-modules/ec2-instance/aws"
version="2.18.0"
ami="ami-6c1c885c"
instance_type="t2.micro"
name="AshuVM"
subnet_id              = "subnet-eddcdzz4"
}

⚠ Note
Before you submit an issue, please perform the following first:

Remove the local .terraform directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): rm -rf .terraform/
Re-initialize the project root to pull down modules: terraform init
Re-attempt your terraform plan or apply and check if the issue still persists

Versions

Terraform:

Provider(s):

Module:

Reproduction

Steps to reproduce the behavior:

Code Snippet to Reproduce

Expected behavior

Actual behavior

Terminal Output Screenshot(s)

Additional context

Hello, Abishek.. revisit your code espceially the syntax on the source. The documentation on terraform should help. 🌝


Change fetching connection process in dataprep

Is your feature request related to a problem? Please describe.
use /api/extensions/connection to find connections that are available for making datasets.
Describe the solution you'd like
A clear and concise description of what you want to happen.
Now : Currently using hard coded connection lists.
To be : use /api/extensions/connection to fetch data connection list
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

 프레퍼레이션에서는 extension으로 추가된 데이터베이스 사용이 불가능 합니다


NOAA Coastwatch Geopolar Sea Surface Temperature

This PR creates a recipe for NOAA Coastwatch Geopolar Sea Surface Temperature. This is about 20 years of global data.
Notes

The dataset has 7 files missing in the 20 year history which is an artifact of the processing timeline. Those dates are manually removed in the recipe.
This is a re-start on #126

Thanks for opening this PR, Rachel! (And yes, three cheers for the 🤖 😄.)
The pangeo_notebook_version is the tag of https://github.com/pangeo-data/pangeo-docker-images on which you've developed your recipe. (Or if you didn't develop on one of those images, just the latest tag on which your recipe can work.)
So if your recipe works on the sandbox, that would be:
pangeo_notebook_version: "2021.12.02"

because:

https://github.com/pangeo-forge/sandbox/blob/a47d7ff66bfc5d104e23f7f7e006108b99cc0a6e/binder/Dockerfile#L1

I've opened https://github.com/pangeo-forge/sandbox/issues/7 to remind myself to correct this in the sandbox template.

Thanks @cisaacstern!

/run recipe-test recipe_run_id=14

/run recipe-test recipe_run_id=15

Hmm this failure is puzzling to me. The logs at https://pangeo-forge.org/dashboard/recipe-run/15 suggest some kind of cftime issue, but I'm unable to replicate this in the sandbox (where it runs fine) or even within the exact worker image we use in production:
$ docker pull "pangeo/pangeo-forge-bakery-images:pangeonotebook-2021.12.02_prefect-0.14.22_pangeoforgerecipes-0.8.2"
$ docker run -it pangeo/pangeo-forge-bakery-images:pangeonotebook-2021.12.02_prefect-0.14.22_pangeoforgerecipes-0.8.2 /bin/bash
jovyan@9f10466fa21b:~$ ipython
Python 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:20:46)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.30.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: #  ... running this recipe here works for me without error 🤔

Thanks for that helpful test code! I tried it on my computer, too (with 3 months of data, even, not just the first 2 files), and it worked fine. The fact that the code is running differently on different machines matches the error, though, in that the size of an integer is likely different on the cloud hardware vs. your mac or mine. On the flip side, I checked the size of a signed integer on my machine and it's only 32 bits, so it doesn't make sense to me why the machine with the larger int size is unhappy.
Two questions:

Is there any way to tell which file it failed on? I'm wondering if this is a problem with 1 file or with all of them. It seems like a few prepare_target tasks succeeded before this one failed (ex. at 2022-03-22T23:00:49.054375+00:00), but I could be interpreting the logs incorrectly. I can re-submit a recipe draft with some print statements if needed.
In this context, is the prepare_target step just deciding the location of the cache, metadata_cache, and target, or is it actually downloading something?

🙏🏻

Looking at the logs, I have a hunch that I understand what is happening. The line that is raising the error (open_target(config.storage_config.target - https://github.com/pangeo-forge/pangeo-forge-recipes/blob/1ed178e06aee87be979395c12fac6098113fc8ee/pangeo_forge_recipes/recipes/xarray_zarr.py#L499) can only occur when an existing dataset is found at the target location. @cisaacstern - if you try manually deleting this target dataset, I think the problem should resolve.
This is still a bug, but that would allow us to move forward.

I've manually deleted the target location on OSN and will now re-start the test.

/run recipe-test recipe_run_id=15

It makes sense to me that deleting the target location would not resolve our problem, because a new target location is set for every /run recipe-test based on the recipe_run_id. For this recipe test it is:
's3://Pangeo/pangeo-forge-test/prod/recipe-run-15/pangeo-forge/staged-recipes/noaa-coastwatch-geopolar-sst.zarr'

I'm not sure I agree with your diagnosis Charles. The error the second time around was different. We got past the prepare_target stage.

I was looking at the wrong logs, because of https://github.com/pangeo-forge/pangeo-forge.org/issues/42.
I'm still confused why the behavior would be different between the two runs (because the target location should really have been empty in both cases), but obviously yes the traceback was different on the second run.

because the target location should really have been empty in both cases

On the other hand, I really don't see how that original error could have been raised if the target were empty.

how that original error could have been raised if the target were empty.

Perhaps I misunderstand the way that Python's try/except works, but my mental model of this code:
    try:
        ds = open_target(config.storage_config.target)
        logger.info("Found an existing dataset in target")
        logger.debug(f"{ds}")

        if config.target_chunks:
            # TODO: check that target_chunks id compatibile with the
            # existing chunks
            pass
    except (FileNotFoundError, IOError, zarr.errors.GroupNotFoundError):
        logger.info("Creating a new dataset in target")

is that no matter what exists in the target, a call to
ds = open_target(config.storage_config.target)

is made, and the except block is invoked only if one of
(FileNotFoundError, IOError, zarr.errors.GroupNotFoundError)

is raised by that call.
If that understanding is accurate (which it may not be), then this

The line that is raising the error (open_target(config.storage_config.target) - https://github.com/pangeo-forge/pangeo-forge-recipes/blob/1ed178e06aee87be979395c12fac6098113fc8ee/pangeo_forge_recipes/recipes/xarray_zarr.py#L499) can only occur when an existing dataset is found at the target location.

is not correct? Is it not the case, as I've imagined, that open_target(config.storage_config.target) is called no matter what?
My hypothesis of the error is that the target was empty, but for some reason the failure mode of
ds = open_target(config.storage_config.target)

was in the form of the OverflowError (as shown in the logs), which is not one of the errors captured by our except.

You're interpretation is mostly correct. However, if the dataset were not there at all, we would get a (FileNotFoundError, IOError, zarr.errors.GroupNotFoundError). Instead, there was a dataset there, and the fact that the time dimension was not properly initialized led to the OverflowError via CFTime. CFTime would never have been invoked if Xarray were not already deep into opening an existing dataset.

@rwegener2, for future reference (because I can imagine this coming up a lot), when I saw the killed worker condition from our second test, that immediately made me think our inputs are too large for worker memory. This is how I confirmed.
First I cached the inputs and checked their size:
# ... imported and pruned your recipe above this
recipe_pruned.to_function()()
# pruned run worked as we both saw on our local machines
# now open one of the cached inputs to check its size
import os
cached_files = os.listdir(recipe_pruned.storage_config.cache.root_path)
ds = xr.open_dataset(recipe_pruned.storage_config.cache.root_path + "/" + cached_files[0])
ds.nbytes/1e6
# --> returns 414.763208

So we know each input is ~414 MB, which feels a bit close to our max worker memory guarantee of 500 MB.
We can then replicate the killed worker locally by limiting container memory with the --memory=500m option:
$ docker pull "pangeo/pangeo-forge-bakery-images:pangeonotebook-2021.12.02_prefect-0.14.22_pangeoforgerecipes-0.8.2"
$ docker run -it --memory=500m pangeo/pangeo-forge-bakery-images:pangeonotebook-2021.12.02_prefect-0.14.22_pangeoforgerecipes-0.8.2 /bin/bash
jovyan@9f10466fa21b:~$ ipython
Python 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:20:46)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.30.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: # running the original recipe here kills the ipython process 🎉

subset_inputs is our mechanism for handling this situation. I'd typically subset in time, but each input file has only one timestep, so I just cut lat in half with subset_inputs = {"lat": 2} instead.

Re: OverflowError, here's a gist of the full Grafana logs, which tell a fuller story then Prefect. It looks like Ryan and I were both right 🙃 : the target was initially empty (thank goodness, because I really couldn't imagine how it could be otherwise), but then due to some distributed retry logic that I didn't realize existed, there eventually was a dataset there (as Ryan predicted), causing the OverflowError.
prepare_target - attempt 1
The first attempt at prepare_target begins on line 65 and on line 79 we get
INFO:pangeo_forge_recipes.recipes.xarray_zarr:Creating a new dataset in target

confirming that the target was empty (xref here).
But then on line 88 the distributed.nanny doesn't like how long it's taking to open the input files (probably because they are too large for worker memory, as we later discovered) and kills the prepare_target task.
prepare_target - attempt 2
On line 152 the task has been restarted and we get a second announcement of
INFO:pangeo_forge_recipes.recipes.xarray_zarr:Creating a new dataset in target

this time we get lucky and some input files are actually opened but they're too large to handle and we end up with an unresponsive event loop on line 293.
prepare_target - attempt 3
The third and final attempt begins on line 294. This time, because the target was (incompletely) initialized in attempt 2, we immediately hit the OverflowError in line 301.

Thanks for all the helpful debugging information, @cisaacstern! As we think about the user debugging workflow it seems like having the docker image being run and knowing its constraints (like memory) will be really useful.
One surprising thing I learned:

Opening a file in xarray uses substantially more memory (~16 times)  than the file occupies on disk. Here: file on disk = 23MB; opening it in xarray = 414 MB. It's a small bummer, because it means a file that was only 23MB to begin with still has to be split in two, but it's not that big of a deal.

And one question (if it's something you know without having to investigate):

How do you get a path from recipe.target_mapper?

Data check

[] Are the dimension lengths correct?
[x] Are all of the expected variables present?
[] Does plotting the data produce a plot that looks like your dataset?
[x] Can you run a simple computation/reduction on the data and produce a plausible result?

So to double check about dimension lengths: the test file looks good to me for the southern hemisphere. The northern hemisphere will also be created, right? I'm assuming I just can't see it in the test files because the first two files in the prune are both southern hemisphere.
As long the answer to the hemisphere question is yes, the test output of this dataset looks good!

knowing its constraints (like memory) will be really useful

Totally. I think it's probably too much to expect users to run a container themselves. And also we wouldn't want them to, because that locks us into committing to particular worker image tags, rather than having the flexibility to swap those out on the backend as needed. But there might be a way to limit memory on Jupyter/Binder to replicate this test environment?

Opening a file in xarray uses substantially more memory (~16 times) than the file occupies on disk.

Yes that has surprised me before too. This may be obvious, but it wasn't to me initially, so just to say, the factor by which these values differ will vary widely and is dependent on the compression of the specific NetCDF file.

How do you get a path from recipe.target_mapper

I think recipe.target should give you this?

I'm assuming I just can't see it in the test files because the first two files in the prune are both southern hemisphere.

I don't know that that's the case; copy_pruned only removes files in the concat dim.
Here are the files used for the pruned run:
recipe_pruned = recipe.copy_pruned()
for _, url in recipe_pruned.file_pattern.items():
    print(url)

https://coastwatch.noaa.gov/pub/socd2/coastwatch/sst_blended/sst5km/night/ghrsst/2002/20020901000000-STAR-L4_GHRSST-SSTfnd-Geo_Polar_Blended_Night-GLOB-v02.0-fv01.0.nc
https://coastwatch.noaa.gov/pub/socd2/coastwatch/sst_blended/sst5km/night/ghrsst/2002/20020902000000-STAR-L4_GHRSST-SSTfnd-Geo_Polar_Blended_Night-GLOB-v02.0-fv01.0.nc

My assumption had been that (as with NOAA OISST) each of the input files for this recipe have global spatial coverage?
I'm wondering if subset_inputs = {"lat": 2} may be resulting in some unintended side effects here? If so, one avenue that might be work investigation is the interaction of subset_inputs with inputs_per_chunk (as opposed to target_chunks)...

https://coastwatch.noaa.gov/pub/socd2/coastwatch/sst_blended/sst5km/night/ghrsst/2002/20020901000000-STAR-L4_GHRSST-SSTfnd-Geo_Polar_Blended_Night-GLOB-v02.0-fv01.0.nc
<xarray.Dataset>
Dimensions:           (time: 1, lat: 3600, lon: 7200)
Coordinates:
  * time              (time) datetime64[ns] 2002-09-01T12:00:00
  * lat               (lat) float32 -89.97 -89.93 -89.88 ... 89.88 89.93 89.97
  * lon               (lon) float32 -180.0 -179.9 -179.9 ... 179.9 179.9 180.0
Data variables:
    analysed_sst      (time, lat, lon) float32 ...
    analysis_error    (time, lat, lon) float32 ...
    sea_ice_fraction  (time, lat, lon) float32 ...
    mask              (time, lat, lon) float32 ...
Attributes: (12/47)
    Conventions:                CF-1.4, Unidata Observation Dataset v1.0
    title:                      Analysed blended sea surface temperature over...
    summary:                    An SST estimation scheme which combines multi...
    references:                 Fieguth,P.W. et al. "Mapping Mediterranean al...
    institution:                Office of Satellite Products and Operations
    comment:                    The Geo-Polar Blended Sea Surface Temperature...
    ...                         ...
    publisher_name:             The GHRSST Project Office
    publisher_url:              http://www.ghrsst.org/
    publisher_email<EMAIL_ADDRESS>    processing_level:           L4
    cdm_data_type:              grid
    history:                    Fri Oct 19 11:12:34 2018: ncatted -a add_offs...

So the way the data are processed, only one of these variables should be loaded into memory at a time. That's 100 MB. That should definitely be possible with a standard 4GB node. Subsetting inputs should not be necessary here.
@cisaacstern - let's try to dig into the memory usage in more depth. This might be a good reason to try out fil.

Thanks, everyone for the insight. Playing around with XarrayZarrRecipe I was able to fix the problem where only 1 hemisphere gets processed by switching to target_chunks, as @cisaacstern recommended.
If it turns out that we can run keep the full spatial domain together and target_chunks aren't actually needed I think it would be nice to switch back to that. It isn't a strong preference, though, so if what we need to take the next step is the use of target_chunks that is fine.

/run recipe-test recipe_run_id=20

Big picture, we need to do some memory profiling to get to the bottom of this, as Ryan suggested.
But short term just to make this work, my hunch would be that passing target_chunks and subset_inputs kwargs (without inputs_per_chunk) may work.

/run recipe-test recipe_run_id=21

Hmm looks like lat is still getting truncated. This may need some further investigation.

Okay, makes sense. If there is anything I can do to help certainly just let me know. I'm happy to try to fix something if you have an idea of where to start.

Thanks, Rachel. We'll need to start with some further research into the nature of the problem(s). You're certainly welcome to jump into any of this if you'd like. AFAICT, there are two problems worth investigating:

Memory usage

Exactly how much memory does executing this recipe require? And within that, which parts of the code are using what percentages of that memory? @rabernat provided a hypothesis about peak memory usage and suggested an approach for memory profiling in https://github.com/pangeo-forge/staged-recipes/pull/127#issuecomment-1077962757.
Exactly how much memory are our worker nodes being provisioned with and how does that compare to what we've learned about this recipe's memory demands? That config lives here (private repo, but I believe you have access), but it is copied from the public GCS bakery repo here. I wonder if there's a way to take our docker-based experiments a step further, by actually spinning up a local Dask cluster using this config?

Subsetting inputs

Even if we'd expect this particular recipe to be able to run without subsetting inputs, it would still seem that we've possibly uncovered a problem with subset_inputs (i.e., this lat truncation we're seeing).

To diagnose the source of this issue I'd probably start by trying to make an idealized version of this NOAA Coastwatch dataset using the xr.Dataset API like this (but with dimensions, variables, etc. that match this dataset). I'd then save as a handful of NetCDFs with xr.save_mfdataset (or similar), and then see if I could reproduce this lat truncation with a recipe pointing at the local files. This subsetting issue should be possible to recreate in this manner, and it would be very interesting/informative to have a fully self-contained Python file which can reproduce it, i.e.
import xarray as xr
from pangeo_forge_recipes.patterns import FilePattern
from pangeo_forge_recipes.recipes import XarrayZarrRecipe

ds = xr.Dataset(
    # NOAA Coastwatch dimension/variables etc., but with random data
)
xr.save_mfdataset(
    # this requires splitting up the `ds`
    # alternatively, you could make one `ds` to represent each file, then save them out with
    # `ds.to_netcdf` in a loop, which might be easier
)
pattern = FilePattern(
    # file pattern pointing at the local files
)
recipe = XarrayZarrRecipe(pattern, subset_inputs={"lat": 2})
# Then, executing this recipe would ideally reproduce the issue we're seeing here
# i.e. result in a dataset with only half the expected `lat` extent

Ultimately, insights gained from this minimal reproducer could be incorporated into the pangeo-forge-recipes test suite, which presumably does not fully cover this specific use case at present. Currently, subset_inputs is tested in anywhere this fixture is used (which is a lot of places) as well as here. The tests definitely have a bit of a learning curve to them, but if/when we get to that step, I'm happy to walk you through them. But we don't have to worry about this until we have the minimal reproducer.

Let me know if this makes sense and if there are any further questions I can help with. I'm caught up with other work this week so won't start on any of this myself until early/mid next week. I'll check with you before I jump into any of this to make sure we don't duplicate efforts. Excited to see what we uncover!

Thanks for that super helpful post @cisaacstern! I took a look at the Memory usage points, and wanted to share progress on the first of those two points.
I tried out filprofiler as @rabernat suggested and I thought it worked pretty well (the conda installation  is broken, though, so if you try it out yourself use pip). To use it you just install the library in your environment, add the code to run the function to your recipe to recipe.py, and then run it with fil-profile run <name of your .py file>
recipe_pruned = recipe.copy_pruned()
run_function = recipe_pruned.to_function()
run_function()

An html page is produced that you can click around on to explore where memory was being used in your program. I put a zip file of the outputs here, which is interactive. Here's also a quick screenshot of the highest level view:

The width of each bar represents how much memory it used, as compared to the total usage (you see "all" takes up the full width, 100%). Each step down on the chart breaks the usage from 1 level higher into its component parts. (also explained here).
@rabernat suggested that this recipe should only be using 100MB of memory, but according to this profiler its using 341MB. 88% of that comes from the run_function() and we see that bit is split into two main sub-branches taking up 15% and 70% of memory. I'm still going through the slack traces, but it interesting to me that the 15% subbranch of run_function() has a lot of .compute() functions while the 70% subbranch has a lot of _set_basic_selection() functions. I might have assumed those would be switched.
What I'm most unsure about is  how to relate this back to our discussions about if xarray is loading multiple variables of the array into memory at once. I am also unsure how the fact that we are looping through multiple files/chunks is represented in a plot like this. Regardless, I wanted to post progress in case anyone else had any insight. I'll keep poking around as well and report back if anything jumps out at me.

Thanks so much for doing this Rachel!
341 MB of memory is not that different from 100 MB. Both numbers are way less than the amount of available memory in our workers (4 GB). This is an entirely reasonable amount of memory to be using, and in fact, we could theoretically be using 10x more without issues.
Based on this analysis, I don't think any more debugging or optimization of pangeo-forge-recipes, xarray, etc is needed at this stage. The memory profile reveals that the code is being efficient with memory, as intended. Instead, we need to understand better why the worker processes are dying under such a light load. This is a bakery debugging issue. Maybe @sharkinsspatial could help here; I'm not sure where to start.

Thanks for this incredibly helpful detective work, Rachel.

Both numbers are way less than the amount of available memory in our workers (4 GB).

@rabernat, I'm not sure our workers do have 4 GB of memory? As noted in https://github.com/pangeo-forge/staged-recipes/pull/127#issuecomment-1084810444 (see the second bullet under Memory Usage), I think they only have 500 MB memory each, would explain the crashes.

I've pushed a new registrar image built from https://github.com/pangeo-forge/registrar/pull/21 to Docker Hub, so I'll rerun the recipe test now and see what happens.
Noting that I'm both not sure I increased the memory allocation correctly and also that there's been a significant refactor of the registrar since the last image push, so it's entirely possible something will go wrong. We will see!

/run recipe-test recipe_run_id=22

As expected, hit a bug related to refactor (unrelated to memory). I think I just fixed it so will try test again.

/run recipe-test recipe_run_id=22

Looks like increasing worker memory solved the problem? The failure notification above is because I ended up starting two concurrent runs (see https://github.com/pangeo-forge/staged-recipes/pull/127#issuecomment-1098434791) pointing at the same target. For some reason, the second one failed, but the first one succeeded, as reported in https://github.com/pangeo-forge/staged-recipes/pull/127#issuecomment-1098438284, and when I just checked using the code reported there, the dataset does not immediately appear to have been corrupted by the second (failed) run:

@rwegener2, if this dataset looks good, then we can keep this PR in mind as one the first to merge as soon as I've finished up a first draft of the registrar feature for running full recipes. (Goal is to have that this week.)

This is really exciting! I'm glad that memory change has resolved the issue.
Yes, everything on the list looks good to me.

[x] Are the dimension lengths correct?
[x] Are all of the expected variables present?
[x] Does plotting the data produce a plot that looks like your dataset?
[x] Can you run a simple computation/reduction on the data and produce a plausible result?

Thanks so much, as always, @cisaacstern.

I've just pushed an image built from the build-full-recipes registrar branch to Docker Hub, which I think means that on merge of this PR we'll see:

A new feedstock created (that feature already existed)
A production run of the recipe initiated (this is new and might be broken in unexpected ways)

Going to merge now and we'll see what happens. I'll report findings on this PR thread post-merge.

Following merge (and a few subsequent registrar bug fixes), the first production run of this recipe is now running.
It's progress is being tracked in https://github.com/pangeo-forge/noaa-coastwatch-geopolar-sst-feedstock/issues/1, which also provides a link to the associated recipe run logs.


Get rid readFileAsync of altering fs object

While using bluebird to promisify fs, I saw that the way we are altering fs will cause an error. That error happened because bluebird reserves the suffix Async.
I think the current code doesn't look good when we directly alter fs object, so I propose this PR to fix it.

Hi @crashbell - many thanks for the PR. While I think the PR is fine, can you expand a little on where you think the error would show up, i.e.:

if someone used bluebird.promisfy on parts of swagger2openapi
if someone used bluebird.promisfy on fs where code was already using swagger2openapi
some other situation?

Hi @MikeRalphson
Please see my comments below:

if someone used bluebird.promisfy on parts of swagger2openapi
It won't show the error.
if someone used bluebird.promisfy on fs where code was already using swagger2openapi
Yes, the error will be shown up.
some other situation?
I can't think other situation 😊

So far, to avoid the error, I call promisify before requiring swagger2openapi. But we have to accept the readFileAsync won't be promisified.

Thanks for the clarification @crashbell .


[Bug]:  Stretchly Informs User that a New Version is Available but New Version will not run on the OS

Version

[X] I'm using version 1.15.1

Known issues

[X] I've checked Known issues

Existing issues

[X] I've checked Existing issues

Advanced Preferences

[X] I've checked Advanced Preferences

What operating system are you using?
macOS
Operating System Version
macOS 10.13
Reproduction steps

Install Stretchly v 14.x on Mac running High Sierra
Stretchly informs you that a new version is available
Down latest Mac binary from Github
Install Stretchly v 15.x
Run Stretchly
WILL NOT RUN - Stretchly now Requires Catalina
...
Reproduces how often: [What percentage of the time does it reproduce?]  100%

Expected Behavior
Stretchly 15.x should run on the OS in question.
Actual Behavior
Stretchy should NOT inform the user that a new version is available when the new version will not run on the OS running on the Mac in question.
P.S.  It may also be a good idea to state the system requirements on your web site and on Github.
Relevant log output
No response
Preferences
No response
Additional information
Stretchly should check the macOS version and make sure that the latest version will run on the Mac in question BEFORE informing the user that a new version is available.
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Hi @RobK88 , what is the message that you see?
Seems like macOS 10.13 is not supported for quite some time https://endoflife.date/macos

I am using electron, and I can not find info what they support so I opened an issue: https://github.com/electron/electron/issues/40739

I think the solution is simple.  When Stretchly checks for a new version, it also checks to see what version of macOS is being used.  If the user is using High Sierra or earlier, then do NOT display a message that a new version is available but display a message stating that this version is the latest version available for your version of macOS.
Rob

From: Jan Hovancik @.>
Sent: December 11, 2023 4:01 AM
To: hovancik/stretchly @.>
Cc: RobK88 @.>; Mention @.>
Subject: Re: [hovancik/stretchly] [Bug]: Stretchly Informs User that a New Version is Available but New Version will not run on the OS (Issue #1403)
I am using electron, and I can not find info what they support so I opened an issue: electron/electron#40739https://github.com/electron/electron/issues/40739
—
Reply to this email directly, view it on GitHubhttps://github.com/hovancik/stretchly/issues/1403#issuecomment-1849595600, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD5RB5MCIETBU2CIKOVR4QLYI3DX3AVCNFSM6AAAAAA7VDAVL2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBZGU4TKNRQGA.
You are receiving this because you were mentioned.Message ID: @.***>

But I won't be able to change the already installed version, right? And the new version is already not supported.
That's why I would like to see what kind of error you see whether we can somehow fix that. Sometimes "not supported" doesn't need to mean not possible to use.

This is a common problem on Macs since Apple updates its OS almost every year now.
In this situation where the user no longer runs an OS the is needed to run the most current version of the application, the user typically sees various message when checking for new versions within the application such as:

Up to date.  No newer version available for your OS

Up to date.  The latest version of Stretchy requires macOS 10.15 Catalina.

You can decide what you want to display.  But if Stretchy  says "new version available" as it does now, users will go the website, download it, install it and be disappointed it does not run.  Moreover, they probably wiped out the old version.  They will need to either use Time Machine, or go to your website to find the version that runs in their macOS.  Very frustrating for users.
RobK88

Stretchly is not running on 14.3-beta either. I get a message saying that Stretchy is corrupted and recommended action is to move it to trash.

Issue, as titled, on Windows 8.1 as well.


add iteration start event

Most of Ginkgo's events are pairs of start/end, but that is not true for solver iterations. If we added this feature, we could display iteration numbers in profiler or stream logger output.

this is no longer necessary, see #1290