SSH Service Endpoint and SSH Task for single host is too restrictive

This is around the combination of SSH Service Endpoint and SSH Task.
 This single connection is setup using SSH Service Endpoint. The endpoint setup requires to specify a single host.
This restriction to a single host does not scale well for us. We have loads of hosts, all of which use the same SSH key.  It is very restrictive to have to define SSH Service Endpoint for each host we have, and then have to setup individual tasks for each host (we may want to execute same thing across several hosts).
Ideally, if we continue with SSH Service Endpoint and SSH Task model, we would like to:

Have SSH Key Service Endpoint where we do not specify host. Effectively, this becomes just "SSH Key" kind of thing.

Have SSH Task v2 where we specify the "SSH Key Servide Endpoint" and then can have host(s) as parameters to the task itself.  This host list would ideally work as "multi-configuration" kind of setup (hopefully people know what I'm talking about!).

There could be alternative approaches that would be suitable for our scenarios, I will raise a separate issue for this.

I will put additional thoughts here first rather than opening new issue. It's all related to "run some tasks over ssh on the hosts we define in the build/release definition".
Install SSH Key task
It's kinda almost same as "SSH Key Service Endpoint" I was talking about. Except:

It's on the level of build/release definition -- we don't want to give people who create build/release definitions ssh key passphrases.

Has some strange paramters I'm not quite sure about -- "Known Hosts Entry"?

Probably does not work with SSH Task?

In any case, it does look very close to what we'd like, if only it was at the service endpoint level, with passphrase baked in there.
Our SSH scenarios
While SSH Task is fine, there is more to it.
First, we run on Linux agents (pretty much), and to be fair, we wouldn't need much need for SSH Task, as we can invoke ssh directly.
More than that, we'd invoke other things which run over ssh, and which SSH Task probably does not support, but all of which need to have the key added/installed to the SSH Agent (or similar).

Ansible
rsync and scp (hence we wouldn't need Copy Files over SSH Task)
sftp
ssh tunnels (e.g. SOCKS5 proxies)
whatever other things that work over ssh!
doing all this via a set of ssh proxies (bastions) with Forward Agent

Obviously all of these need installed SSH keys.
So if we could have "Install SSH Key" task pulled into "SSH Key Service Endpoint" , and then "Install SSH Key v2" which would simply use it -- that would be perfect.

Hope it makes sense to whomever reading this!

I ran also into this restrictions that the task can not override the Host. I want to create in a test release a VM in the cloud and run some commands in the machine with azure DevOps. After the test the VM should destroyed. Because it is not possible to configure the IP at the task time I have to keep the public IP all the time.

This issue is stale because it has been open for a year with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

Hello!
Is there any workaround for this issue? Can this be re-opened?

Hi @ppanyukov @StefanSchoof @mikedzikowski looks like this would require more effort than task changes only (some changes for service connection also). This repo is mostly for pipeline tasks - could you please open a ticket on https://developercommunity.visualstudio.com/search?space=21?


Feature/update dependencies

Codecov Report

Merging #5 into master will not change coverage.
The diff coverage is 100%.

@@            Coverage Diff            @@
##             master       #5   +/-   ##
=========================================
  Coverage     35.28%   35.28%
  Complexity       32       32
=========================================
  Files            31       31
  Lines           649      649
  Branches         16       16
=========================================
  Hits            229      229
  Misses          412      412
  Partials          8        8

Impacted Files
Coverage Δ
Complexity Δ

...in/java/org/interledger/plugin/lpi/LedgerInfo.java
100% <100%> (ø)
2 <1> (ø)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e328c7f...50cead1. Read the comment docs.


Include in the Readme an example of how to use it in CI/CD Github action

Hi!
I been googleing how to show the html generated but i couldnt success ¿someone has an example of it used in a github action?
I tried with this but doesnt work:
name: Pull Request Staging

on:
  pull_request:
    branches:
      - staging

jobs:
  Terraform:
    name: Terraform Plan
    runs-on: ubuntu-latest
    steps:

    - name: Checkout Repo
      uses: actions/checkout@v2

    - name: Terraform Setup
      uses: hashicorp/setup-terraform@v1

    - name: Terraform Init
      run: cd ./staging; terraform init
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        TF_ACTION_WORKING_DIR: './staging'
        AWS_ACCESS_KEY_ID:  ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY:  ${{ secrets.AWS_SECRET_ACCESS_KEY }}

    - name: Terraform Validate
      run: terraform validate

    - name: Terraform Plan
      run: cd ./staging; terraform plan -out=plan.out; terraform show -json plan.out > plan.json
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        TF_ACTION_WORKING_DIR: './staging'
        AWS_ACCESS_KEY_ID:  ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY:  ${{ secrets.AWS_SECRET_ACCESS_KEY }}

    - name: Terraform Visual Install
      run: cd staging; npm install -g @terraform-visual/cli

    - name: Generate the Report
      run: cd staging; terraform-visual --plan plan.json;

    - name: HTML preview
      id: html_preview
      uses: pavi2410/html-preview-action@v2
      with:
        html_file: terraform-visual-report/index.html

When you say this isn't working - can you expand on that?
I have noticed that your html_file is likely pointing to the wrong place, as you're not in the 'staging' directory.


Simplify subscriber factory threading

Codecov Report
Attention: Patch coverage is 25.00000% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 92.68%. Comparing base (58bf80a) to head (1a2345c).

Files
Patch %
Lines

google_nest_sdm/google_nest_subscriber.py
25.00%
6 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #995      +/-   ##
==========================================
+ Coverage   92.57%   92.68%   +0.10%
==========================================
  Files          19       19
  Lines        1818     1817       -1
==========================================
+ Hits         1683     1684       +1
+ Misses        135      133       -2

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Grunt it up

Use grunt to do a more normalized src/dist split, with uglify and minification and whatever else I can think of.

Grunted!


FilesIterable skips all the files after a hidden file

Describe the bug
When initializing FilesIterable with a list of file paths using FilesIterable.from_paths, it will discard all the files after a hidden file.
The problem is in this line where return should be replaced by continue.
Steps to reproduce the bug
https://colab.research.google.com/drive/1SQlxs4y_LSo1Q89KnFoYDSyyKEISun_J#scrollTo=93K4_blkW-8-
Expected behavior
The script should print all the files except the hidden one.
Environment info

datasets version: 2.14.1
Platform: Linux-5.15.109+-x86_64-with-glibc2.35
Python version: 3.10.6
Huggingface_hub version: 0.16.4
PyArrow version: 9.0.0
Pandas version: 1.5.3

Thanks for reporting. We've merged a PR with a fix.


[TT-Train][Bug] ttnn::experimental::all_reduce crashes

Describe the bug
TEST_F(N300UtilsTest, TestXTensorReplicateAllReduce) {
    auto* device = &ttml::autograd::ctx().get_device();
    auto mesh_shape = device->shape();

    // optimized branch
    xt::xarray<float> test_data = xt::arange(64 * 64 * 4) / 100.F;
    xt::xarray<float> xtensor = test_data.reshape({2, 2, 64, 64});
    // non optimized branch
    //xt::xarray<float> test_data = xt::arange(32 * 32 * 1) / 100.F;
    //xt::xarray<float> xtensor = test_data.reshape({1, 1, 32, 32});

    ttml::core::XTensorToMeshVariant<float> replicate_composer = ttml::core::ReplicateXTensorToMesh<float>(mesh_shape);
    auto tensor = ttml::core::from_xtensor(xtensor, device, replicate_composer);

    auto sum_tensor = ttnn::experimental::all_reduce(tensor, ttnn::operations::reduction::ReduceType::Sum);
    ttml::core::MeshToXTensorVariant<float> identity_composer = ttml::core::VectorMeshToXTensor<float>(mesh_shape);

    auto xtensors_back = ttml::core::to_xtensor(sum_tensor, identity_composer);
    auto reduced_tensor = xtensor + xtensor;

    EXPECT_TRUE(xt::allclose(reduced_tensor, xtensors_back[0]));
    EXPECT_TRUE(xt::allclose(reduced_tensor, xtensors_back[1]));
}

Crash message is: [ctest] C++ exception with description "vector" thrown in the test body.
To Reproduce
go to the PR: https://github.com/tenstorrent/tt-metal/pull/15637
And run the TEST_F(N300UtilsTest, TestXTensorReplicateAllReduce).
Expected behavior
If the issue in our side (wrong setup or something else) all_reduce should return actionable message. Not just exception from the stl.
If the issue is inside of the all_reduce it should be fixed.
Screenshots
If applicable, add screenshots to help explain your problem.
Please complete the following environment information:

OS: [e.g. Ubuntu 20.04]
Version of software (eg. commit)

Additional context
It is blocking us from the multi device support. I set P0.
cc @davorchap

fyi @omilyutin-tt , sync with @SeanNijjar , is the error in MeshDevice infra or in the CCL OP?

Update: I've got a fix for the crash. The test now runs to completion but does not pass. It fails all-close but I think this is inevitable given the tiny atol in allclose and the (relatively) massive numbers we are converting fp32 -> fp16, accumulating, converting fp16 -> fp32 (visually the numbers are similar).
However, this hasn't been confirmed by comparing against a single chip batch-sum or feeding the individual numbers through an fp16 addition and comparing to fp32. That being said I'll consider this issue solved on branch. Will open PR and get it merged asap.

fix has been merged. Reassigning to @dmakoviichuk-tt for fix confirmation.

Done!


⚠ Samsung has degraded performance

In 6f9a3a9, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2530 ms

Resolved: Samsung performance has improved in 5e4febe.


List triggers on views

In the properties window for a view, could you have a tab for triggers on that view like you have for tables?

What database?

Postgres. See https://www.postgresql.org/docs/current/static/sql-createtrigger.html. Triggers can be created for tables, views and foreign tables (foreign data wrappers).

Will be added in 3.8


Tags for releases

Currently there are no tags for any release.  Tags are helpful since they allow people to download the full, unaltered version of a release, rather than just what is chosen to be included in an sdist.

I had simply forgot to push the tags. Now we have them for the last four releases.


(feat): Automate link Validation in chart-hub -

Auto validation of link using selenium then needs to be hosted  on continuous integration service           (Travis.yml )

Validation will be headless browser
preferring using javascript selenium validation

Can the same approach used in docs be used here? i.e., https://github.com/bmuschko/link-verifier
Is there an equivalent for YAML (CSV) files?


Non-covalent QM/MM

Define APIs for QM/MM - need to be able to A) create minimal force field for ligand with only VDW interactions, B) insert point charges in QM calculations, C) configure the QM and MM region boundaries in a sane way.

Closing as not done for now - will reopen if appropriate (may be superseded by NWChem integration)


运行live网络区块产生慢

Backing off claiming new slot for block authorship: finality is lagging.

刚开始正常，到几十个区块后开始报Backing off claiming new slot for block authorship: finality is lagging.


feat: move pipeline to root of project

feat: move pipeline to root of project
Update pipeline ref in pipelinerun

/ok-to-test


cant deconstuct rods near the airlocks

Description
cant deconstuct rods near the external airlocks, nor with wirecutter, nor with RCD
Reproduction

place rods/tile near the external airlock
Try to deconstruct
Cargo cant dock their shuttle/evac docks to the cargo/evac spawns somewhere in the space

Screenshots

Can't replicate. Is it on a specific station?
https://user-images.githubusercontent.com/54602815/229636132-c557a9f5-d586-406e-9b3c-5fa09262419d.mp4

it has to be a shuttle airlock specifically

The issue is the airlocks extra fixture which extends beyond its own tile. The current method for checking if a tile is "blocked" by other entities for the sake of things like RCD or lattice cutting is very crude.


你好，请问目前taro-lottie支持微信小程序吗？

I'm submitting a ...
[ ] bug report
[ ] feature request
[ ] question about the decisions made in the repository
[ ] question about how to use this project

Summary

Other information (e.g. detailed explanation, stack traces, related issues, suggestions how to fix, links for us to have context, eg. StackOverflow, personal fork, etc.)

支持，见readme


[BUG] k3d private registries volume mount fail

What did you do?
I'm trying to configure a gcr private registry inside k3d. So I've just followed the instructions here :  https://k3d.io/usage/guides/registries/ but with a _json_key as username
like this :
configs:
  eu.gcr.io:
    auth:
      username: _json_key
      password: '{"type": "service_account","project_id": "foo","private_key_id": "xxxxx","private_key": "-----BEGIN PRIVATE KEY-----\nMIIEMXH3w...\n-----END PRIVATE KEY-----\n","client_email": "foo@foo.iam.gserviceaccount.com","client_id": "xxxxxx","auth_uri": "https://accounts.google.com/o/oauth2/auth","token_uri": "https://oauth2.googleapis.com/token","auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/foo.iam.gserviceaccount.com"}'

How was the cluster created?

k3d create cluster dev -w 3 -v $(pwd)/k3d/registries.yml:/etc/rancher/k3s/registries.yaml --wait

What did you do afterwards?

Then I try to connect to my cluster
export KUBECONFIG=$(k3d get kubeconfig dev)
kubectl config use-context k3d-dev
kubectl get nodes
Unable to connect to the server: EOF

But the nodes are down :
docker ps -a
CONTAINER ID        IMAGE                           COMMAND                  CREATED              STATUS                          PORTS                             NAMES
81444c7b2794        rancher/k3d-proxy:v3.0.0-rc.6   "/bin/sh -c nginx-pr…"   About a minute ago   Up About a minute               80/tcp, <IP_ADDRESS>:39187->6443/tcp   k3d-dev-masterlb
d3d310ffa90c        rancher/k3s:v1.18.4-k3s1        "/bin/k3s agent"         About a minute ago   Exited (1) About a minute ago                                     k3d-dev-worker-2
0819f8a8a572        rancher/k3s:v1.18.4-k3s1        "/bin/k3s agent"         About a minute ago   Up About a minute                                                 k3d-dev-worker-1
1d5143c71261        rancher/k3s:v1.18.4-k3s1        "/bin/k3s agent"         About a minute ago   Exited (1) About a minute ago                                     k3d-dev-worker-0
8f83a256367b        rancher/k3s:v1.18.4-k3s1        "/bin/k3s server --t…"   About a minute ago   Exited (1) About a minute ago                                     k3d-dev-master-0

What did you expect to happen?
The cluster should be started and I can deploy pods with gcr private registry enabled.
Screenshots or terminal output
Here is the output of the master node :
time="2020-06-18T15:45:29.023763496Z" level=info msg="Using registry config file at /etc/rancher/k3s/registries.yaml"
time="2020-06-18T15:45:29.025966879Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
time="2020-06-18T15:45:29.026144475Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
time="2020-06-18T15:45:29.026264801Z" level=info msg="Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory\""
I0618 15:45:29.030477       6 controller.go:606] quota admission added evaluator for: deployments.apps
time="2020-06-18T15:45:29.091586970Z" level=info msg="Starting /v1, Kind=Secret controller"
time="2020-06-18T15:45:29.097549107Z" level=info msg="Active TLS secret k3s-serving (ver=207) (count 8): map[listener.cattle.io/cn-<IP_ADDRESS>:<IP_ADDRESS> listener.cattle.io/cn-<IP_ADDRESS>:<IP_ADDRESS> listener.cattle.io/cn-<IP_ADDRESS>:<IP_ADDRESS> listener.cattle.io/cn-<IP_ADDRESS>:<IP_ADDRESS> listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/hash:302113f38dcb0fe8a25d9816621dba6b1bdfc00f49068a75f73f42a325363764]"
I0618 15:45:29.245805       6 controller.go:606] quota admission added evaluator for: helmcharts.helm.cattle.io
I0618 15:45:29.258464       6 controller.go:606] quota admission added evaluator for: jobs.batch
2020-06-18 15:45:29.261223 I | http: TLS handshake error from <IP_ADDRESS>:49128: remote error: tls: bad certificate
2020-06-18 15:45:29.266202 I | http: TLS handshake error from <IP_ADDRESS>:49144: remote error: tls: bad certificate
containerd: exit status 1

When I skip mounting registries.yml into the cluster nodes, the cluster works fine (but no private registry :cry:  )
Which OS & Architecture?

Linux  / amd64

Which version of k3d?
k3d version v3.0.0-rc.6
k3s version v1.18.4-k3s1 (default)

Which version of docker?
 Thanks for your help !

After inspecting the logs inside the master node of k3s k3d-dev-master-0:/var/lib/rancher/k3s/agent/containerd/containerd.log, I found a error :
containerd: Near line 27 (last key parsed 'plugins.cri.registry.configs.eu.gcr.io.auth'): expected a top-level item to end with a newline, comment, or EOF, but got 't' instead

:thinking: t ? that's the first letter of type the first word in my json password.
So I've updated my password with espacing for all quotes.
configs:
  eu.gcr.io:
    auth:
      username: _json_key
      password: '{\"type\": \"service_account\",\"project_id\": \"foo\",\"private_key_id\": \"xxx","private_key":......\"}

And it worked !

Hi thanks @seuf for opening this issue and closing it with an informative solution :pray: :+1:


Unity Fractal Server issue

Hi,
I am running signalling server locally. When running Unity Fractal Server there is nothing in drop down to select. Shall i run StreamingDirectxClient as client and Unity Fractal Server as server? If i run like that there is nothing in unity server drop down. Please help me to figure out this.

Hi,
It works now. Based the thread https://github.com/CatalystCode/3dtoolkit/issues/135 i have deleted Authentication failed in webrtcConfig. No its fine. But when connecting to a directx client i can see 3d cubes rendering in directx client. But in Unity server the UI is still there. When i connect unity server with unity client same thing happening in server. And there is no cubes showing in unity client.

Looks like there is something wrong with the Unity FractalStream server. Can you enable "Development build" so that we can see if there is any error? My guess is there is something wrong with the dll loading.

Is it possible to deploy Unity Fractal Server to hololens and unity client in desktop?

No, it's not supported since Unity FractalStream server uses Win32 APIs.

@parandham03 Hello, did you figure out why the Unity Fractal Server's initial connection UI was not disappearing after connecting to client? I have the same issue.
Also, on Hololens Unity client, I get short messages when signing-in to signalling server and after connecting to the streaming server, which each disappear with fade; but afterwards, I have only a black screen on Hololens Unity client.
If I connect the server to a win32 client, I get the spinning cubes on the client, but the UI is still there on the Unity Fractal Streaming server.

Hi phongcao, thanks a lot for the answer. Indeed, I am initialising the connection from the server.
I have tried the Dx Holo client, and everything looks the same. I have also tried some other combinations of servers and clients, but until now the only one that somehow works is Unity Fractal Server + DxWin32 client - I have the fractals on the client, but the server screen is still UI on black background.
To be more clear and specific, I have opened issue #187 where I have attached logs for server + client. The weird log on the signalling server seems to appear only when the WI-FI router used for comm and signalling is not connected to the internet - I presume it is not mandatory to connect to internet if I am running on local Wi-FI ?


Fix Sender to properly repeat a non-binary data file

Currently when reading a non-binary data file and repeating it
with Giles Sender, the file is only sent once and re-opened
again and again but does not send data. This fix disposes of
the file once there are no more lines so that we have access to
the file's data once re-opened.
Test:
Run the pony word_count example with the UI running with older version of Giles Sender with the repeat option on, verify that you don't see updated information.
Build and run again with this branch, verify that data is continuously being processed.

You have an error @JONBRWN
Connecting OutgoingBoundary to <IP_ADDRESS>:36635

Orders source attempting to listen on <IP_ADDRESS>:50001

Orders source is listening

Market Data source attempting to listen on <IP_ADDRESS>:50002

Market Data source is unable to listen

This should never happen: failure in /home/travis/build/WallarooLabs/wallaroo/lib/wallaroo/core/source/tcp_source/tcp_source_listener.pony at line 211

Looks transient so I am restarting. If you get again, that might be an issue. Seems like it couldnt grab the port though. So that is environmental.

@JONBRWN I think there is an open issue for this, no?

@JONBRWN Ive used repeat with non-binary data and not experienced any problems. What exactly is the issue?

@SeanTAllen I'll look for the open issue.
This doesn't have multiple files though and doesn't repeat on master
giles/sender/sender -h <IP_ADDRESS>:7000 -m 5000000 -s 300 -i 5_000_000 -f testing/data/word_count/bill_of_rights.txt --ponythreads=1 -r

@JONBRWN I do that for demos and it repeats constantly. I've never experienced the problem you are stating.

Specifically:
../../giles/sender/sender --host <IP_ADDRESS>:7010 --file bill_of_rights.txt \
--batch-size 5 --interval 100_000_000 --messages 10000000 \
--ponythreads=1 --repeat

@SeanTAllen CI passed, care to give this one last look over?

@JONBRWN looks reasonable


Offset is outside the bounds of the DataView

Hello, I am getting this error when trying to read my file:
RangeError: Offset is outside the bounds of the DataView
    at DataView.getInt32 (<anonymous>)
    at readRleBitPackedHybrid (webpack-internal:///(rsc)/./node_modules/hyparquet/src/encoding.js:258:27)
    at readDataPage (webpack-internal:///(rsc)/./node_modules/hyparquet/src/datapage.js:70:111)
    at readColumn (webpack-internal:///(rsc)/./node_modules/hyparquet/src/column.js:60:135)
    at eval (webpack-internal:///(rsc)/./node_modules/hyparquet/src/read.js:144:126)
    at async Promise.all (index 15)
    at async readRowGroup (webpack-internal:///(rsc)/./node_modules/hyparquet/src/read.js:203:5)
    at async parquetRead (webpack-internal:///(rsc)/./node_modules/hyparquet/src/read.js:56:31)

Code snippet:
    const numRows = Number(metadata.num_rows);

    let limitRows;
    if (limit > 0) {
      limitRows = Math.min(limit, numRows);
      appLogger.info({ limitRows, numRows }, "Limiting parsed data rows");
    }

    let parquetData = [];

    await parquetRead({
      file: arrayBuffer,
      onComplete: (rows) => (parquetData = rows),
      ...(limitRows && { rowEnd: limitRows }),
    });

I thought maybe it happens because I might be specifying limit that is out of range for the data length, but that's not the case since I see this log from the IF statement:
{"level":"INFO","time":"2024-04-10T12:53:42.207Z","limitRows":10,"numRows":1780,"msg":"Limiting parsed data rows"}

Hi @platypii , thank you for quick response. Unfortunately, I can't share any data file.
I also started getting heap out of memory errors.
How do you suggest to filter data efficiently? I'd like to filter data and limit the result.

For large files, you should filter by row and/or column in order to reduce memory usage. If you're displaying data, try to only request the data that is in view. If you are performing scans of the entire data, you may need to break it up into chunks and read each chunk. I just added an example in the README.
There is not much I can do to debug the "out of bounds" error without an example file to test. Do you know what parquet writer generated your file? The information should be available in the parquet metadata.
Hyparquet is open source, and this is a young project, so if you manage to debug it, contributions are certainly welcome and appreciated!

Hi, my parquet data is generated by Apache Spark. I resigned from using rowEnd param in favor of limiting the data at the end, after filtering, and the offset error is gone. I used the column filter to reduce parsing of unused data and it works much better now. I can do all the filtering/processing at once in memory.
This is my code in case anyone needs it. Suggestions for optimization are appreciated ;)
import fs from "fs";
import { parquetRead, parquetMetadata, toJson } from "hyparquet";
import { appLogger } from "./logger";

export async function readParquet(
  filePath,
  filterFunc,
  limit = -1,
  columnsToRead = [],
) {
  try {
    const buffer = fs.readFileSync(filePath);
    const arrayBuffer = new Uint8Array(buffer).buffer;

    const metadata = parquetMetadata(arrayBuffer);
    const columnsNames =
      columnsToRead.length > 0
        ? columnsToRead
        : metadata.schema.slice(1).map((meta) => meta.name);

    let parquetData = [];

    await parquetRead({
      file: arrayBuffer,
      ...(columnsToRead.length > 0 && { columns: columnsToRead }),
      onComplete: (rows) => (parquetData = rows),
    });

    const jsonParquetData = toJson(parquetData);

    // attach column names to each row data
    // @ts-ignore
    let parsedParquetData = jsonParquetData.map((row) => {
      const updatedRow = {};
      for (let i = 0; i < columnsNames.length; i++) {
        const colName = columnsNames[i];
        updatedRow[colName] = row[i];
      }
      return updatedRow;
    });

    if (filterFunc) {
      parsedParquetData = parsedParquetData.filter(filterFunc);
    }

    if (limit > 0) {
      parsedParquetData = parsedParquetData.slice(0, limit);
    }

    appLogger.info({ firsRow: parsedParquetData[0] }, "Parsed parquet file");

    return parsedParquetData;
  } catch (e) {
    appLogger.error(e, `Error while reading parquet file ${filePath}`);
    throw e;
  }
}

I found that I could reproduce the "Offset is outside the bounds of the DataView" error if byteLength is not provided as part of the file argument. I'm not sure that's related to your original issue. But I just added an additional check on the byteLength field to hopefully avoid this error in the future. Published v0.7.5 with additional input validation.

Hi @platypii , I was getting the same error again on v0.7.5 recently. I updated to the latest version v0.9.7 and its gone, but I am getting another error:
Error: parquet invalid rle width 3
    at readRle (webpack-internal:///(rsc)/./node_modules/hyparquet/src/encoding.js:73:11)
    at readRleBitPackedHybrid (webpack-internal:///(rsc)/./node_modules/hyparquet/src/encoding.js:46:7)
    at readDataPage (webpack-internal:///(rsc)/./node_modules/hyparquet/src/datapage.js:53:76)
    at readColumn (webpack-internal:///(rsc)/./node_modules/hyparquet/src/column.js:59:122)
    at eval (webpack-internal:///(rsc)/./node_modules/hyparquet/src/read.js:165:78)
    at async Promise.all (index 6)
    at async readRowGroup (webpack-internal:///(rsc)/./node_modules/hyparquet/src/read.js:197:3)
    at async parquetRead (webpack-internal:///(rsc)/./node_modules/hyparquet/src/read.js:63:25)
    at async readParquet (webpack-internal:///(rsc)/./src/lib/utils/parquet-reader.js:19:9)

Some  metadata of my file:
Created by: parquet-mr version 1.12.3-databricks-0002 (build 2484a95dbe16a0023e3eb29c201f99ff9ea771ee)
Properties:
                  org.apache.spark.timeZone: Etc/UTC
                   org.apache.spark.version: 3.4.1
               org.apache.spark.legacyINT96:
  org.apache.spark.sql.parquet.row.metadata: {.......}
            org.apache.spark.legacyDateTime:
Schema:
...

@maksp-lipscore good catch! I just published hyparquet v0.9.8 which contains a fix for RLE byte width 3. I don't have your file to test on, but this matches the behavior in duckdb. Please let me know if that works for you.
Re: bonus question - No the toJson function does not convert null to "null" string. I just added tests to confirm this behavior [1]. I'm not sure why you are seeing "null" strings unless you are calling JSON.stringify yourself, or if your dataset contains rows with the literal string "null".

It worked! Thank you.


Updated disk example with size in bytes without measurement unit

This Pull Request (PR) fixes the following issues

Fixes #214
Task list

[x] Added an entry under the Unreleased section of the change log in the CHANGELOG.md.
Entry should say what was changed, and how that affects users (if applicable).
[ ] Resource documentation added/updated in README.md in the resource folder.
[ ] Resource parameter descriptions added/updated in schema.mof
and comment-based help.
[ ] Comment-based help added/updated.
[ ] Localization strings added/updated in all localization files as appropriate.
[x] Examples appropriately added/updated.
[ ] Unit tests added/updated. See DSC Resource Testing Guidelines.
[ ] Integration tests added/updated (where possible). See DSC Resource Testing Guidelines.
[ ] New/changed code adheres to DSC Resource Style Guidelines and Best Practices.

This change is

Codecov Report

Merging #218 into dev will not change coverage.
The diff coverage is n/a.

@@        Coverage Diff         @@
##           dev   #218   +/-   ##
==================================
  Coverage   95%    95%
==================================
  Files        7      7
  Lines      980    980
==================================
  Hits       939    939
  Misses      41     41

Sorry about the delay @fullenw1 - we've been a bit focused on the migration process over to the DSC Community. I've updated the build so that we can get this through.

Can you move this up to be in the ## Unreleased section and remove the duplicate ## <IP_ADDRESS> header?

Done :)


Add Litematica

Justification: so people can have client side schematics overlay for things too large for Create's blueprint

Fine enough edition. It can always be installed on the client's end pretty trivially, but I can see the value in it being enabled by default.


Packaging fails if Blueprint Nativization Method = Inclusive

This was brought up by a user on the forum.
It seems like nativizing blueprints does work with Cesium for Unreal, and will cause packaging to fail. According to Lildreas on the forum, this is something we should support, as it can improve performance of packaged builds and is especially important for XR applications.
Lildreas also provided additional info on blueprint nativization - https://docs.unrealengine.com/4.26/en-US/ProgrammingAndScripting/Blueprints/TechnicalGuide/NativizingBlueprints/
The error log provided in the forum post is:
UATHelper: Packaging (Windows (64-bit)): Building 8 actions with 20 processes…
UATHelper: Packaging (Windows (64-bit)): [1/8] Default.rc2
UATHelper: Packaging (Windows (64-bit)): [2/8] SharedPCH.Core.cpp
UATHelper: Packaging (Windows (64-bit)): [3/8] CesiumForUnrealSamples.cpp
UATHelper: Packaging (Windows (64-bit)): [4/8] SharedPCH.Engine.cpp
UATHelper: Packaging (Windows (64-bit)): [5/8] Module.NativizedAssets.gen.cpp
UATHelper: Packaging (Windows (64-bit)): The contents of are available only with C++17 or later.
UATHelper: Packaging (Windows (64-bit)): C:\Program Files\Epic Games\UE_4.26\Engine\Plugins\Marketplace\CesiumForUnreal\Source\ThirdParty\include\CesiumGeospatial/Ellipsoid.h(102): error C2039: ‘optional’: is not a member of ‘std’
UATHelper: Packaging (Windows (64-bit)): C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30037\INCLUDE\cmath(714): note: see declaration of ‘std’
UATHelper: Packaging (Windows (64-bit)): C:\Program Files\Epic Games\UE_4.26\Engine\Plugins\Marketplace\CesiumForUnreal\Source\ThirdParty\include\CesiumGeospatial/Ellipsoid.h(102): error C2143: syntax error: missing ‘;’ before ‘<’
UATHelper: Packaging (Windows (64-bit)): C:\Program Files\Epic Games\UE_4.26\Engine\Plugins\Marketplace\CesiumForUnreal\Source\ThirdParty\include\CesiumGeospatial/Ellipsoid.h(102): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
UATHelper: Packaging (Windows (64-bit)): C:\Program Files\Epic Games\UE_4.26\Engine\Plugins\Marketplace\CesiumForUnreal\Source\ThirdParty\include\CesiumGeospatial/Ellipsoid.h(103): error C2238: unexpected token(s) preceding ‘;’

@kring previously suggested adding CppStandard = CppStandardVersion.Cpp17; to the project's build.cs, but that did not seem to solve the issue.

Hi,
any update on this?
Best
Andreas

Hi guys,
found a way around this issue.
First I tested if this is a problem with my Unreal and VS installation, I completely removed UE 4.27.1 and Visual Studio 2019 Pro from my machine and cleaned all files from VS from everywhere.
Then I downloaded 4.27.1 and VS 2019 Pro again and did a clean install. But the issue persisted.
The only solution that worked for me was to change the Unreal Build Tool and hardcode it to use C++17 standard, because adding this in the projects build.cs (using "CppStandard = CppStandardVersion.Cpp17;") alone did not work.
a) ChangeUnreal Build Tool in VS 2019
Open the project file "UE_4.27\Engine\Source\Programs\UnrealBuildTool\UnrealBuildTool.csproj".
Open the file "Platform\Windows\VCToolChain.cs" in the Solution.
Search for the method "AppendCLArguments_Global" and add this line at the end (for me it was line 574):
Arguments.Add("/std:c++17");
Now do a Build of the solution, a new UnrealBuildTool.exe will be created.
b) Open Unreal and your project
Open the projects build.cs and add the Cpp17, couldn't hurt (don't know if it's needed though):
public YourProject(ReadOnlyTargetRules Target) : base(Target)
{
CppStandard = CppStandardVersion.Cpp17;
...
}
And then build and package your game and use nativize Blueprints, and it works now, no more "...only with C++17 or later." errors.
Best
Andreas


[ENHANCEMENT] Provide support for new hardware platforms (e.g., heterogenous CPU)

Is your enhancement related to a problem? Please describe.
As of v1.1.2, Harvester is using kernel v5.14.21.  This kernel was release at the end of 2021, so it does not provide supports for newer platforms (e.g., heterogeneous CPU). Harvester could not be installed on my homelab Intel 13700k system (Intel 13th gen, Raptor Lake). I suppose the same applies to enterprise platform like Intel Sapphire Rapids or AMD Epyc Genoa.
Describe the solution you'd like
Upgrade to a newer kernel for the elemental images, and use that for future version of Harvester.
At least for Intel Raptor Lake, kernel v6.0 or above resolve multiple compatibilities issue. I'm running kernel v6.2.6 without any visible problems at the moment. The latest upstream kernel is v6.4.2 as of this issue.
devpc@pop-os:~$ uname -srm
Linux 6.2.6-76060206-generic x86_64

Describe alternatives you've considered
N/a
Additional context
The video below show Harvester behavior when being installed on my 13700k system.
Starting from 01:20, there is an error loop that lasts until around 02:46, then installation becomes extremely slow. I'm not certain what triggers this because the error loop is not always there. Installation runs for about 30 minutes then stop at  loop0: detected capacity changed from 0 to 1215312 indefinitely. I have a longer video, but cannot upload due to GitHub 100 MB limitation.
https://github.com/harvester/harvester/assets/62678592/8bd42f1f-fd0b-43d1-800e-dbc7832d9013
Note: I tested Harvester installation with older platforms from Intel 6th gen up to AMD Ryzen 5000, and they all work fine.

Can you enter installer UI? I have a 12th Gen Intel platform and I can enter installer without issue

Can you enter installer UI? I have a 12th Gen Intel platform and I can enter installer without issue

Unfortunately I cannot. I have VT, VT-d, and SR-IOV enabled in BIOS, but even at factory default, it does not work.
The following link has recordings of my full installation process on 13th Gen platform, if that helps.
https://drive.google.com/drive/u/0/folders/1qStvs1P-4FFZwU65_fXvi5YmTOGNVdhz

similar symptoms to https://github.com/harvester/harvester/issues/4086.
Resolved by editing boot options and removing "console=ttyS0".
For context I used v1.2.0


Fails to build development environment

Hello,
current-bench fails to start for me.
I'm on Windows 10, running Docker version 20.10.5.
The contents of ./environments/development.env are:
OCAML_BENCH_DOCKER_CPU=1
OCAML_BENCH_DB_PASSWORD=docker
OCAML_BENCH_GRAPHQL_KEY=secret
OCAML_BENCH_TARGET_ARCH=amd64

I run make start-development and get the following output (errors way down at bottom):
PS C:\Users\d4hin\repos\current-bench> make start-development
docker-compose \
        --project-name="current-bench" \
        --file=./environments/development.docker-compose.yaml \
        --env-file=./development.env \
        up \
        --remove-orphans \
        --build
Building db
failed to get console mode for stdout: The handle is invalid.
[+] Building 11.2s (7/7) FINISHED
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 151B                                       0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load metadata for docker.io/library/postgres:12             1.7s
 => [internal] load build context                                          0.0s
 => => transferring context: 87B                                           0.0s
 => [1/2] FROM docker.io/library/postgres:12@sha256:2f3f78532c9cc5435d1cf  9.2s
 => => resolve docker.io/library/postgres:12@sha256:2f3f78532c9cc5435d1cf  0.0s
 => => sha256:2f3f78532c9cc5435d1cf9c8f5fb1409f9abd43e5b7 1.86kB / 1.86kB  0.0s
 => => sha256:75646c2fb4101d306585c9b106be1dfa7d82720ba 27.14MB / 27.14MB  1.0s
 => => sha256:7e98825f6d67159fc035c0db985619c0635cc7edb89 1.80kB / 1.80kB  0.3s
 => => sha256:6ef141d922bc14123d24ddec34aa11d0f360702a2 10.63kB / 10.63kB  0.0s
 => => sha256:4217f50ff2ca28b7c3dddb25372379c97093df532c7 3.25kB / 3.25kB  0.0s
 => => sha256:2355d0ffeb5531909b8910928bcbaa874e9464657c7 4.18MB / 4.18MB  0.4s
 => => sha256:cfd3ce06be45a467d795e78a52a17330a56a7b85a3b 1.42MB / 1.42MB  0.5s
 => => sha256:c7b7bb83e8f78baa3e4e402b1704b6df53db00bec8b 7.97MB / 7.97MB  0.9s
 => => sha256:c6786930510827ff4067b1103131548886adcd7 391.17kB / 391.17kB  0.6s
 => => sha256:19614baa7ddd150dc7115b541a85cec0e8178fc9c6e62c8 149B / 149B  0.6s
 => => sha256:af508737d8137d59629c343294044cb7f854220431d 3.05kB / 3.05kB  0.7s
 => => sha256:562171e5e21c288ecc3ae718a64e16c2f17902deb 73.34MB / 73.34MB  6.1s
 => => sha256:9a1da7929f851b2f7db64c04a23cf5b2c7377b0db64 9.02kB / 9.02kB  1.0s
 => => extracting sha256:75646c2fb4101d306585c9b106be1dfa7d82720baabe1c75  1.3s
 => => sha256:83302896f1018b739b0cdcdd817cbc82c823cf8651cffd6 200B / 200B  1.2s
 => => sha256:1f2e5b2f6e536af3111e90491c622a3bd93179b70459f09 129B / 129B  1.2s
 => => sha256:8892c210e4afc8a8f8ece7cd0bcccfee3b234dd09b6 4.40kB / 4.40kB  1.4s
 => => sha256:46ddd2f5c65450c2987d3d2526b0a0b7d6f1655d312fb31 120B / 120B  1.4s
 => => extracting sha256:2355d0ffeb5531909b8910928bcbaa874e9464657c71213a  0.2s
 => => extracting sha256:7e98825f6d67159fc035c0db985619c0635cc7edb89aee89  0.0s
 => => extracting sha256:cfd3ce06be45a467d795e78a52a17330a56a7b85a3b3a25b  0.1s
 => => extracting sha256:c7b7bb83e8f78baa3e4e402b1704b6df53db00bec8b57266  0.3s
 => => extracting sha256:c6786930510827ff4067b1103131548886adcd7a25a012ba  0.1s
 => => extracting sha256:19614baa7ddd150dc7115b541a85cec0e8178fc9c6e62c82  0.0s
 => => extracting sha256:af508737d8137d59629c343294044cb7f854220431d23fc4  0.0s
 => => extracting sha256:562171e5e21c288ecc3ae718a64e16c2f17902deb1011cad  2.5s
 => => extracting sha256:9a1da7929f851b2f7db64c04a23cf5b2c7377b0db64b3b67  0.0s
 => => extracting sha256:1f2e5b2f6e536af3111e90491c622a3bd93179b70459f098  0.0s
 => => extracting sha256:83302896f1018b739b0cdcdd817cbc82c823cf8651cffd68  0.0s
 => => extracting sha256:8892c210e4afc8a8f8ece7cd0bcccfee3b234dd09b6b2637  0.0s
 => => extracting sha256:46ddd2f5c65450c2987d3d2526b0a0b7d6f1655d312fb310  0.0s
 => [2/2] COPY ./init.sql /docker-entrypoint-initdb.d/init.sql             0.1s
 => exporting to image                                                     0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:65222942325e8b01767e984270bbcbc7150b551d35577  0.0s
 => => naming to docker.io/library/current-bench_db                        0.0s
Successfully built<PHONE_NUMBER>5e8b01767e984270bbcbc7150b551d35577e19aa4f3177c8caa3e6
Pulling graphql-engine (hasura/graphql-engine:v1.3.3)...
v1.3.3: Pulling from hasura/graphql-engine
fcf970b8ac66: Pull complete
Digest: sha256:814e36b355177046195edd795ca561fe762a2ae117129f185a26f9d3f609efb9
Status: Downloaded newer image for hasura/graphql-engine:v1.3.3
Building frontend
failed to get console mode for stdout: The handle is invalid.
[+] Building 42.9s (17/17) FINISHED
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 554B                                       0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 52B                                           0.0s
 => [internal] load metadata for docker.io/library/nginx:stable-alpine     0.7s
 => [internal] load metadata for docker.io/library/node:lts                0.8s
 => [builder 1/7] FROM docker.io/library/node:lts@sha256:f6b9ff4caca9d4f  18.1s
 => => resolve docker.io/library/node:lts@sha256:f6b9ff4caca9d4f0a331a882  0.0s
 => => sha256:abea835c0b3b0ef244d92a1076602ed0891a35b1b77 7.83kB / 7.83kB  0.0s
 => => sha256:da61ad49fa9961d6dfe53dc067fd690133f205498 11.29MB / 11.29MB  0.7s
 => => sha256:af62e04c0f85ed10801df4314d8cb9ba5e6c391f6a3 4.34MB / 4.34MB  0.5s
 => => sha256:f6b9ff4caca9d4f0a331a882e560df242eb332b7bbbed2f 776B / 776B  0.0s
 => => sha256:61e6ae4231ac78eb737a10e70d8925e92e7a35ec0c3 2.21kB / 2.21kB  0.0s
 => => sha256:00168f89dbe8f3c9985e536784c27517f6cc35ea5 45.38MB / 45.38MB  2.9s
 => => sha256:aae835e2c68ef1d48a4ffef72547a79926019aec6 49.79MB / 49.79MB  3.6s
 => => sha256:5254dc317968c943c4321cdbf0364a6a47c8332 214.35MB / 214.35MB  8.3s
 => => sha256:b3f76fdfa8e8784238b141aaa54fa1862aef5348665 4.19kB / 4.19kB  2.9s
 => => extracting sha256:00168f89dbe8f3c9985e536784c27517f6cc35ea56263469  1.9s
 => => sha256:5681b30a17f656444f1bd37c07ee6f594c7e2e26c 34.58MB / 34.58MB  5.5s
 => => sha256:da9a613e2daeaae78b9cd247e52f3f731b701730d7a 2.38MB / 2.38MB  3.9s
 => => sha256:67891041a0a2d985480f4934fc94bc0b180f9f27741bd5c 292B / 292B  4.1s
 => => extracting sha256:da61ad49fa9961d6dfe53dc067fd690133f205498e8045e8  0.4s
 => => extracting sha256:af62e04c0f85ed10801df4314d8cb9ba5e6c391f6a3c1db1  0.2s
 => => extracting sha256:aae835e2c68ef1d48a4ffef72547a79926019aec6a2bbeb5  2.4s
 => => extracting sha256:5254dc317968c943c4321cdbf0364a6a47c833296b3f45e3  7.6s
 => => extracting sha256:b3f76fdfa8e8784238b141aaa54fa1862aef534866543116  0.0s
 => => extracting sha256:5681b30a17f656444f1bd37c07ee6f594c7e2e26c4ec796e  1.6s
 => => extracting sha256:da9a613e2daeaae78b9cd247e52f3f731b701730d7ab8bf6  0.1s
 => => extracting sha256:67891041a0a2d985480f4934fc94bc0b180f9f27741bd5c2  0.0s
 => [stage-1 1/4] FROM docker.io/library/nginx:stable-alpine@sha256:ec3f6  5.7s
 => => resolve docker.io/library/nginx:stable-alpine@sha256:ec3f6e4587a20  0.0s
 => => sha256:ec3f6e4587a2053f406b1b4e027c260e82e4e32c256 1.65kB / 1.65kB  0.0s
 => => sha256:7409b874fed478250a841bc67aa1451df7163543cf6 1.36kB / 1.36kB  0.0s
 => => sha256:d0d03989cd8e044493922e446203a9893eb662a3279 7.98kB / 7.98kB  0.0s
 => => sha256:9b794450f7b6db7c944ba1f4161edb68cb535052fe7 2.82MB / 2.82MB  4.4s
 => => extracting sha256:9b794450f7b6db7c944ba1f4161edb68cb535052fe7db8ac  0.1s
 => => sha256:71c851db865893ed287bbecc2934e756a25e4e78bab 6.67MB / 6.67MB  4.9s
 => => extracting sha256:71c851db865893ed287bbecc2934e756a25e4e78babcd419  0.3s
 => => sha256:fc9aed49b78e299c67c0ee890e26326969df6c564fda2b2 902B / 902B  5.0s
 => => sha256:bb846ca9b45ab911d714856abaca670ed6a798711e73325 665B / 665B  5.1s
 => => sha256:281095db834a17364db464c05304d89414f2ab118283136 600B / 600B  5.0s
 => => extracting sha256:281095db834a17364db464c05304d89414f2ab118283136b  0.0s
 => => extracting sha256:fc9aed49b78e299c67c0ee890e26326969df6c564fda2b28  0.0s
 => => extracting sha256:bb846ca9b45ab911d714856abaca670ed6a798711e73325d  0.0s
 => [internal] load build context                                          0.0s
 => => transferring context: 332.87kB                                      0.0s
 => [stage-1 2/4] COPY ./nginx.conf /etc/nginx/nginx.conf                  0.2s
 => [stage-1 3/4] RUN rm -rf /usr/share/nginx/html/*                       0.3s
 => [builder 2/7] WORKDIR /app                                             0.4s
 => [builder 3/7] COPY package.json yarn.lock /app/                        0.0s
 => [builder 4/7] RUN yarn install                                        14.7s
 => [builder 5/7] COPY . /app/                                             0.0s
 => [builder 6/7] RUN yarn build                                           2.1s
 => [builder 7/7] RUN yarn bundle                                          6.5s
 => [stage-1 4/4] COPY --from=builder /app/dist /usr/share/nginx/html      0.0s
 => exporting to image                                                     0.1s
 => => exporting layers                                                    0.0s
 => => writing image sha256:c536e74e0967b7344ff7988a302a7ae2068e7bdfcd00f  0.0s
 => => naming to docker.io/library/current-bench_frontend                  0.0s
Successfully built c536e74e0967b7344ff7988a302a7ae2068e7bdfcd00fa74233b8a508ada4d16
Building pipeline
failed to get console mode for stdout: The handle is invalid.
[+] Building 170.7s (22/22) FINISHED
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 1.78kB                                     0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load metadata for docker.io/library/debian:stable-slim      0.9s
 => [internal] load metadata for docker.io/ocaml/opam:debian-ocaml-4.12    0.9s
 => [internal] load build context                                          0.0s
 => => transferring context: 54.32kB                                       0.0s
 => [build 1/8] FROM docker.io/ocaml/opam:debian-ocaml-4.12@sha256:e39cf  49.1s
 => => resolve docker.io/ocaml/opam:debian-ocaml-4.12@sha256:e39cf5ddf42c  0.0s
 => => sha256:004f1eed87df3f75f5e2a1a649fa7edd7f713d130 50.43MB / 50.43MB  2.8s
 => => sha256:b704dfdc404aa80ee420ceb91d1974441bb41b 286.70MB / 286.70MB  20.9s
 => => sha256:dbb52d71bac56d7b6969ac8b0310d33588f700 189.72MB / 189.72MB  14.2s
 => => sha256:e39cf5ddf42ceff246eee0402466a81e47a1c29567a 1.66kB / 1.66kB  0.0s
 => => sha256:7671537e58b4c6a388caee4d445b9a16476fbb3eb6fd1d2 956B / 956B  0.0s
 => => sha256:f7c1669319a4c7bcd2269e33e7bf6950679512069 10.47kB / 10.47kB  0.0s
 => => extracting sha256:004f1eed87df3f75f5e2a1a649fa7edd7f713d1300532fd0  2.2s
 => => extracting sha256:b704dfdc404aa80ee420ceb91d1974441bb41b37fce81fc  20.2s
 => => extracting sha256:dbb52d71bac56d7b6969ac8b0310d33588f7009a3692b933  6.4s
 => [stage-1 1/8] FROM docker.io/library/debian:stable-slim@sha256:f31a1f  6.4s
 => => resolve docker.io/library/debian:stable-slim@sha256:f31a1f9e274ebf  0.0s
 => => sha256:f31a1f9e274ebf9a97b964c9aed3386e1720d56a3d2 1.85kB / 1.85kB  0.0s
 => => sha256:65c240929acadafb6cdf7447d7e3ba1abc214a8b008fe4e 529B / 529B  0.0s
 => => sha256:f052c451335940bea31ab0e58feff9fc657959c4915 1.46kB / 1.46kB  0.0s
 => => sha256:7fa62122c34635d215ecb61e3d515d9eb1676a0f2 27.14MB / 27.14MB  5.0s
 => => extracting sha256:7fa62122c34635d215ecb61e3d515d9eb1676a0f258d352e  1.3s
 => [stage-1 2/8] RUN apt-get update     && apt-get install --no-install  26.2s
 => [stage-1 3/8] WORKDIR /app                                             0.0s
 => [stage-1 4/8] COPY ./aslr_seccomp.json /app/aslr_seccomp.json          0.0s
 => [stage-1 5/8] COPY ./db/migrations /app/db/migrations                  0.0s
 => [build 2/8] RUN sudo apt-get update &&     sudo apt-get install -qq   16.0s
 => [build 3/8] WORKDIR /mnt/project                                       0.0s
 => [build 4/8] COPY --chown=opam:opam pipeline.opam pipeline.opam         0.0s
 => [build 5/8] RUN opam install -y --deps-only -t .                     100.1s
 => [build 6/8] COPY --chown=opam . .                                      0.0s
 => [build 7/8] RUN sudo chown opam .                                      0.5s
 => [build 8/8] RUN opam exec -- dune build --profile=release bin/main.ex  2.2s
 => [stage-1 6/8] COPY --from=build /home/opam/.opam/4.12/bin/omigrate /a  0.1s
 => [stage-1 7/8] COPY ./scripts/wait-for /app/bin/wait-for                0.0s
 => [stage-1 8/8] COPY --from=build /mnt/project/_build/default/bin/main.  0.1s
 => exporting to image                                                     1.5s
 => => exporting layers                                                    1.5s
 => => writing image sha256:943cfc7ba022701594600f0d26fd96ada9ff5fbbac566  0.0s
 => => naming to docker.io/library/current-bench_pipeline                  0.0s
Successfully built 943cfc7ba022701594600f0d26fd96ada9ff5fbbac566a9c7ad3c80558708e94
Creating current-bench_db_1 ... done
Creating current-bench_pipeline_1       ... done
Creating current-bench_db-migrate_1     ... done
Creating current-bench_graphql-engine_1 ... done
Creating current-bench_frontend_1       ... done
Attaching to current-bench_db_1, current-bench_graphql-engine_1, current-bench_db-migrate_1, current-bench_pipeline_1, current-bench_frontend_1
db-migrate_1      | standard_init_linux.go:219: exec user process caused: no such file or directory
db_1              | The files belonging to this database system will be owned by user "postgres".
db_1              | This user must also own the server process.
db_1              |
db_1              | The database cluster will be initialized with locale "en_US.utf8".
db_1              | The default database encoding has accordingly been set to "UTF8".
db_1              | The default text search configuration will be set to "english".
db_1              |
db_1              | Data page checksums are disabled.
db_1              |
db_1              | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1              | creating subdirectories ... ok
db_1              | selecting dynamic shared memory implementation ... posix
db_1              | selecting default max_connections ... 100
db_1              | selecting default shared_buffers ... 128MB
db_1              | selecting default time zone ... Etc/UTC
db_1              | creating configuration files ... ok
db_1              | running bootstrap script ... ok
db_1              | performing post-bootstrap initialization ... ok
db_1              | syncing data to disk ... ok
db_1              |
db_1              |
db_1              | Success. You can now start the database server using:
db_1              |
db_1              |     pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1              |
graphql-engine_1  | {"type":"startup","timestamp":"2021-04-02T20:12:48.629+0000","level":"info","detail":{"kind":"server_configuration","info":{"live_query_options":{"batch_size":100,"refetch_delay":1},"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","plan_cache_options":{"plan_cache_size":4000},"enabled_log_types":["http-log","websocket-log","startup","webhook-log","query-log"],"server_host":"HostAny","enable_allowlist":false,"log_level":"info","auth_hook_mode":null,"use_prepared_statements":true,"unauth_role":null,"stringify_numeric_types":false,"enabled_apis":["metadata","graphql","config","pgdump"],"enable_telemetry":true,"enable_console":true,"auth_hook":null,"jwt_secret":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"console_assets_dir":null,"admin_secret_set":true,"port":8080}}}
graphql-engine_1  | {"type":"startup","timestamp":"2021-04-02T20:12:48.629+0000","level":"info","detail":{"kind":"postgres_connection","info":{"retries":1,"database_url":"postgres://docker:...@db:5432/docker"}}}
graphql-engine_1  | {"type":"pg-client","timestamp":"2021-04-02T20:12:48.629+0000","level":"warn","detail":{"message":"postgres
connection failed, retrying(0)."}}
graphql-engine_1  | {"type":"pg-client","timestamp":"2021-04-02T20:12:48.629+0000","level":"warn","detail":{"message":"postgres
connection failed, retrying(1)."}}
graphql-engine_1  | {"type":"startup","timestamp":"2021-04-02T20:12:48.629+0000","level":"error","detail":{"kind":"catalog_migrate","info":{"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"db\" (<IP_ADDRESS>) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}}}
graphql-engine_1  | {"internal":"could not connect to server: Connection refused\n\tIs the server running on host \"db\" (<IP_ADDRESS>) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}
db_1              | initdb: warning: enabling "trust" authentication for local connections
db_1              | You can change this by editing pg_hba.conf or using the option -A, or
db_1              | --auth-local and --auth-host, the next time you run initdb.
pipeline_1        |    application [INFO] Logging ready.
db_1              | waiting for server to start....2021-04-02 20:12:48.618 UTC [47] LOG:  starting PostgreSQL 12.6 (Debian 12.6-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
pipeline_1        |    application [ERROR] Database connection error:
pipeline_1        | Connection failure: could not connect to server: Connection refused
pipeline_1        |     Is the server running on host "db" (<IP_ADDRESS>) and accepting
pipeline_1        |     TCP/IP connections on port 5432?
pipeline_1        |
db_1              | 2021-04-02 20:12:48.620 UTC [47] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1              | 2021-04-02 20:12:48.634 UTC [48] LOG:  database system was shut down at 2021-04-02 20:12:48 UTC
pipeline_1        | pipeline: internal error, uncaught exception:
pipeline_1        |           Postgresql.Error(_)
pipeline_1        |           Raised at Pipeline__Db_util.check_connection in file "lib/db_util.ml", line 10, characters 6-15
pipeline_1        |           Called from Pipeline.v in file "lib/pipeline.ml", line 245, characters 2-36
pipeline_1        |           Called from Cmdliner_term.app.(fun) in file "cmdliner_term.ml", line 25, characters 19-24
pipeline_1        |           Called from Cmdliner.Term.run in file "cmdliner.ml", line 117, characters 32-39
db_1              | 2021-04-02 20:12:48.637 UTC [47] LOG:  database system is ready to accept connections
db_1              |  done
db_1              | server started
db_1              | CREATE DATABASE
db_1              |
db_1              |
db_1              | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sql
db_1              | GRANT
db_1              |
db_1              |
current-bench_db-migrate_1 exited with code 1
db_1              | 2021-04-02 20:12:48.919 UTC [47] LOG:  received fast shutdown request
db_1              | waiting for server to shut down....2021-04-02 20:12:48.921 UTC [47] LOG:  aborting any active transactions
db_1              | 2021-04-02 20:12:48.922 UTC [47] LOG:  background worker "logical replication launcher" (PID 54) exited with exit code 1
db_1              | 2021-04-02 20:12:48.922 UTC [49] LOG:  shutting down
db_1              | 2021-04-02 20:12:48.935 UTC [47] LOG:  database system is shut down
db_1              |  done
db_1              | server stopped
db_1              |
db_1              | PostgreSQL init process complete; ready for start up.
db_1              |
db_1              | 2021-04-02 20:12:49.027 UTC [1] LOG:  starting PostgreSQL 12.6 (Debian 12.6-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bitgnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
db_1              | 2021-04-02 20:12:49.028 UTC [1] LOG:  listening on IPv4 address "<IP_ADDRESS>", port 5432
db_1              | 2021-04-02 20:12:49.028 UTC [1] LOG:  listening on IPv6 address "::", port 5432
db_1              | 2021-04-02 20:12:49.033 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1              | 2021-04-02 20:12:49.043 UTC [84] LOG:  database system was shut down at 2021-04-02 20:12:48 UTC
db_1              | 2021-04-02 20:12:49.047 UTC [1] LOG:  database system is ready to accept connections

Any help would be greatly appreciated 🙏

Gah! It was a CRLF issue. I re-checked out the repository with LF and made more progress, but not completely there:
Creating current-bench_db_1 ... done
Creating current-bench_pipeline_1       ... done
Creating current-bench_graphql-engine_1 ... done
Creating current-bench_db-migrate_1     ... done
Creating current-bench_frontend_1       ... done
Attaching to current-bench_db_1, current-bench_db-migrate_1, current-bench_pipeline_1, current-bench_graphql-engine_1, current-bench_frontend_1
db_1              | The files belonging to this database system will be owned by user "postgres".
db_1              | This user must also own the server process.
db_1              |
db_1              | The database cluster will be initialized with locale "en_US.utf8".
db_1              | The default database encoding has accordingly been set to "UTF8".
db_1              | The default text search configuration will be set to "english".
db_1              |
db_1              | Data page checksums are disabled.
db_1              |
db_1              | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1              | creating subdirectories ... ok
db_1              | selecting dynamic shared memory implementation ... posix
graphql-engine_1  | {"type":"startup","timestamp":"2021-04-02T20:42:28.484+0000","level":"info","detail":{"kind":"server_configuration","info":{"live_query_options":{"batch_size":100,"refetch_delay":1},"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","plan_cache_options":{"plan_cache_size":4000},"enabled_log_types":["http-log","websocket-log","startup","webhook-log","query-log"],"server_host":"HostAny","enable_allowlist":false,"log_level":"info","auth_hook_mode":null,"use_prepared_statements":true,"unauth_role":null,"stringify_numeric_types":false,"enabled_apis":["metadata","graphql","config","pgdump"],"enable_telemetry":true,"enable_console":true,"auth_hook":null,"jwt_secret":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"console_assets_dir":null,"admin_secret_set":true,"port":8080}}}
graphql-engine_1  | {"type":"startup","timestamp":"2021-04-02T20:42:28.484+0000","level":"info","detail":{"kind":"postgres_connection","info":{"retries":1,"database_url":"postgres://docker:...@db:5432/docker"}}}
db_1              | selecting default max_connections ... 100
db_1              | selecting default shared_buffers ... 128MB
pipeline_1        |    application [INFO] Logging ready.
pipeline_1        |    application [ERROR] Database connection error:
pipeline_1        | Connection failure: could not connect to server: Connection refused
pipeline_1        |     Is the server running on host "db" (<IP_ADDRESS>) and accepting
pipeline_1        |     TCP/IP connections on port 5432?
pipeline_1        |
db_1              | selecting default time zone ... Etc/UTC
pipeline_1        | pipeline: internal error, uncaught exception:
pipeline_1        |           Postgresql.Error(_)
pipeline_1        |           Raised at Pipeline__Db_util.check_connection in file "lib/db_util.ml", line 10, characters 6-15
pipeline_1        |           Called from Pipeline.v in file "lib/pipeline.ml", line 245, characters 2-36
pipeline_1        |           Called from Cmdliner_term.app.(fun) in file "cmdliner_term.ml", line 25, characters 19-24
pipeline_1        |           Called from Cmdliner.Term.run in file "cmdliner.ml", line 117, characters 32-39
db_1              | creating configuration files ... ok
db_1              | running bootstrap script ... ok
db_1              | performing post-bootstrap initialization ... ok
db_1              | syncing data to disk ... ok
db_1              |
db_1              |
db_1              | Success. You can now start the database server using:
db_1              |
db_1              |     pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1              |
db_1              | initdb: warning: enabling "trust" authentication for local connections
db_1              | You can change this by editing pg_hba.conf or using the option -A, or
db_1              | --auth-local and --auth-host, the next time you run initdb.
db_1              | waiting for server to start....2021-04-02 20:42:28.136 UTC [47] LOG:  starting PostgreSQL 12.6 (Debian 12.6-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
db_1              | 2021-04-02 20:42:28.138 UTC [47] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1              | 2021-04-02 20:42:28.162 UTC [48] LOG:  database system was shut down at 2021-04-02 20:42:27 UTC
db_1              | 2021-04-02 20:42:28.165 UTC [47] LOG:  database system is ready to accept connections
db_1              |  done
db_1              | server started
db_1              | CREATE DATABASE
db_1              |
db_1              |
db_1              | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sql
db_1              | GRANT
db_1              |
db_1              |
db_1              | 2021-04-02 20:42:28.440 UTC [47] LOG:  received fast shutdown request
db_1              | waiting for server to shut down....2021-04-02 20:42:28.441 UTC [47] LOG:  aborting any active transactions
db_1              | 2021-04-02 20:42:28.442 UTC [47] LOG:  background worker "logical replication launcher" (PID 54) exited with exit code 1
db_1              | 2021-04-02 20:42:28.442 UTC [49] LOG:  shutting down
db_1              | 2021-04-02 20:42:28.455 UTC [47] LOG:  database system is shut down
db_1              |  done
db_1              | server stopped
db_1              |
db_1              | PostgreSQL init process complete; ready for start up.
db_1              |
db_1              | 2021-04-02 20:42:28.549 UTC [1] LOG:  starting PostgreSQL 12.6 (Debian 12.6-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
db_1              | 2021-04-02 20:42:28.549 UTC [1] LOG:  listening on IPv4 address "<IP_ADDRESS>", port 5432
db_1              | 2021-04-02 20:42:28.549 UTC [1] LOG:  listening on IPv6 address "::", port 5432
db_1              | 2021-04-02 20:42:28.553 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1              | 2021-04-02 20:42:28.563 UTC [84] LOG:  database system was shut down at 2021-04-02 20:42:28 UTC
db_1              | 2021-04-02 20:42:28.567 UTC [1] LOG:  database system is ready to accept connections
db-migrate_1      | omigrate: [INFO] Database already exists
db-migrate_1      | omigrate: [INFO] Creating the migrations table
db-migrate_1      | omigrate: [INFO] Applying up migration<PHONE_NUMBER>0054
db-migrate_1      | omigrate: [INFO] Applying up migration<PHONE_NUMBER>3805
db-migrate_1      | omigrate: [INFO] Applying up migration<PHONE_NUMBER>5643
current-bench_db-migrate_1 exited with code 0
graphql-engine_1  | {"type":"pg-client","timestamp":"2021-04-02T20:42:28.484+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(0)."}}
graphql-engine_1  | {"type":"startup","timestamp":"2021-04-02T20:42:28.484+0000","level":"info","detail":{"kind":"catalog_migrate","info":"Successfully initialized the catalog (at version 40)."}}
graphql-engine_1  | {"type":"startup","timestamp":"2021-04-02T20:42:28.484+0000","level":"info","detail":{"kind":"schema-sync","info":{"thread_id":"ThreadId 40","instance_id":"f782e680-b498-41ed-9fb8-18278193f751","message":"listener thread started"}}}
graphql-engine_1  | {"type":"startup","timestamp":"2021-04-02T20:42:28.484+0000","level":"info","detail":{"kind":"schema-sync","info":{"thread_id":"ThreadId 41","instance_id":"f782e680-b498-41ed-9fb8-18278193f751","message":"processor thread started"}}}
graphql-engine_1  | {"type":"startup","timestamp":"2021-04-02T20:42:28.484+0000","level":"info","detail":{"kind":"event_triggers","info":"preparing data"}}
graphql-engine_1  | {"type":"startup","timestamp":"2021-04-02T20:42:28.484+0000","level":"info","detail":{"kind":"event_triggers","info":"starting workers"}}
graphql-engine_1  | {"type":"startup","timestamp":"2021-04-02T20:42:28.484+0000","level":"info","detail":{"kind":"scheduled_triggers","info":"preparing data"}}
graphql-engine_1  | {"type":"startup","timestamp":"2021-04-02T20:42:28.484+0000","level":"info","detail":{"kind":"telemetry","info":"Help us improve Hasura! The graphql-engine server collects anonymized usage stats which allows us to keep improving Hasura at warp speed. To read more or opt-out, visit https://hasura.io/docs/1.0/graphql/manual/guides/telemetry.html"}}
graphql-engine_1  | {"type":"startup","timestamp":"2021-04-02T20:42:28.484+0000","level":"info","detail":{"kind":"server","info":{"time_taken":0.7339076,"message":"starting API server"}}}

Apparently pipeline_1 fails because it can't connect to db.

Just in case it was still a Windows-specific error, I tried from a natively-installed Linux distro, and got the same results.

Hi @d4hines! Thanks for opening this issue, I'm looking into why the pipeline can't connect to the db and will update as soon as I have a solution or more insight. :)

The error message from the db isn't very descriptive unfortunately, maybe let's improve that first.
I find it strange that the migrations container did manage to connect to the db – they're using the same connection information.

@rizo  I think the problem is that the pipeline container tries to connect to the database before the postgres server is setup. If you look at the logs, the pipeline errors out first and then the logs say that the database is created.  Maybe we should introduce a time lag before the pipeline tries to connect to the db?
Also, the pipeline does log the error here.

Indeed, adding let () = Unix.sleep 5 at the top of ./pipeline/bin/main.ml seems to have temporarily fixed the issue.

Fixed by #91


Create script to generate Gatling simulation

Gatling recorder allows to generate simulation out of HAR file, but it doesn’t look like a good solution for us for few reasons:

The generated file has predefined load setup, that can’t be changed out of the box
We need to replace not only /bsearch but /login part as well to get Cookie header for authentication.
cli in headless mode is quite limited

Overall, it feels like it is easy to write our own simulation generator in nodejs and avoid HAR files but rather whatever format single user benchmark tool gives back.

Implemented


OSS the github/gh-ado2gh repo and github/gh-bbs2gh repo

gh-ado2gh
Security: https://github.com/github/security-iam/issues/5449
Release: https://github.com/github/open-source-releases/issues/264
gh-bbs2gh
Security: https://github.com/github/open-source-releases/issues/265
Release: https://github.com/github/security-iam/issues/5451


Add “CNCF” to glossary

This is a Feature Request
Child of #5993
What would you like to be added
Add a glossary entry for the Cloud Native Computing Foundation.
Why is this needed
Readers may not be familiar with the CNCF abbreviation or with the full term. Using a glossary entry lets us use the associated Hugo shortcodes to help readers out.
Comments

I've added a PR for this: https://github.com/kubernetes/website/pull/14537
However, the entry it adds has an unexpected CNCF logo showing as well as the text I added. I'd welcome advice on that.
/help

/kind feature
/area community

/remove-help

Done via #14537
/close


Implement app versioning, changelog & update popup

Abstract
For the longest time, MPW has had no proper versioning system #55 - and it's been detrimental for a few reasons;

Unclear release or update schedule for the public (negative for PR and marketing).
The inability to designate 'stable' releases, effectively keeping MPW always unstable.
Inability for users to easily see changes, updates, without looking at GitHub.
More that we could probably keep coming up with!

This PR implements such a versioning system, similar to Semantic Versioning, and based upon the package.json file, but with discretion from the Labs team in designating major releases, rather than being API-based, given MPW is a user-facing application, not a developer library.
A Labs maintainer such as myself will designate releases with exclusive 'Version Bump' PRs, these PRs will bump the version in package.json according to the magnitude of the changes since the last bump.

The maintainer producing the release will also edit the changelog.md file, which acts both as a historical store of changes (in the git tree) while also being processed by MPW in to a user-facing Changelog - this gives the devs minimum release friction, while giving MPW users super-easy access to update information.

Notes:

This Pull Request itself may be considered v1.0.0 of MyPIVXWallet, as designated by the package.json, thus a followup bump is not necessary, this is the only exception to the exclusive Version Bump guidance.
The version popup relies on localStorage rather than IndexedDB as I did not think it was efficient to store a single tiny string inside it, which has no dependency on network (Mainnet/Testnet).

Conflict resolved + Changelog updated, good for re-ACK cc; @Liquid369 @Duddino


ipld: consider moving into separate repo

Summary
Optimint and other projects might use the current code under ipld:https://github.com/lazyledger/lazyledger-core/tree/40acb17283ebfe1c49b804a786464b93c2213a97/p2p/ipld
It makes sense to eventually move this out into a separate repository once the APIs stabilized a bit.
Action Items

[ ] Move ipld package into separate repo but keep the commit history in tact (possible)
[ ] make the package more self-contained (documentation, CI etc)
[ ] potentially abstract away some of the concrete APIs

Context / Original Openening Comment
Decide if we want to support using the IPLD plugin easily with vanilla IPFS. Previously, this was extensively used for testing and there is even a unit test, that required this logic (currently skipped):
https://github.com/lazyledger/lazyledger-core/blob/874df76f139b4a3effcd2a74d6078c9d08b6d1ba/p2p/ipld/plugin/nodes/nodes_test.go#L129-L130
Merging #294 will make this less easy to actually deploy an ipfs node with the plugin because building it with the Makefile will use some of the dependencies from the lazyledger-core go.sum. These dependencies most likely will differ from the deps used in the IPFS version (even if compiled locally). e.g. @evan-forbes noticed that after compiling the version in #294
plugin.Open("/redacted/.ipfs/plugins/lazyledger-plugin"): plugin was built with a different version of package golang.org/x/sys/cpu.
There certainly might be a way to get this working without moving the plugin into a separate repository but I think it would be easier and it also would make sense.
If we decide to go down that route, we should definitely keep the git history intact (which is possible).

e.g. @evan-forbes noticed that after compiling the version in #294
plugin.Open("/redacted/.ipfs/plugins/lazyledger-plugin"): plugin was built with a different version of package golang.org/x/sys/cpu.

@liamsi, curious what info channel I missed where @evan-forbes reported that.

https://discord.com/channels/638338779505229824/804386443845042206/834807592060190760

ref: celestiaorg/celestia-node/issues/1

@liamsi, agree. To do this we need to cherry-pick some of the changes done here.

Since we have to keep the DataAvailabilityHeader in celestia-core, and will already have to import that in celestia-node, I actually think we should keep logic needed by both repos in an independent package(s) in celestia-core. Just to have one less repo to keep track of versions.

Having repo segregation is much cleaner in terms of code and the issue you create for that code. In long term, IMO, this only gives profits. The two problems with that are, as you've mentioned, versioning and CI. For the first on it is not a problem for me and I can handle that, for the second, we would need a unified CI, which we can postpone.

Per a sync discussion with @Wondertan I am now convinced that we actually should move the plugin, DAH, and common logic to a different repo, provided that we have a strict release policy. I think that's what was meant by

For the first on it is not a problem for me and I can handle that,

This policy needs to ensure that the same versions of each repo are being used across the board per each release. It should also be noted that we include this policy in our docs.

Another thing is that we should always track that releases of top-level repos rely on the number releases of repo deeper in the tree, not on the specific commits. However, that's ok if we depend on specific commits in branches.

can we close this?

I think it is OK to close this. I think opening a super short issue in node which points to the deleted code could help to find the relevant portions.

IPLD code is linked in celestia-node issue #106, but just in case we need a convenient lookup later :
https://github.com/celestiaorg/celestia-core/tree/c4f2a98f4928d3b04517621c5792744c106ba306/pkg/da/ipld


how to delete module

I've created a new module, but I don't want to use it. How do I delete it

Just delete the directory 🙌

Try this :bowtie:
// use Illuminate\Support\Facades\File;

File::deleteDirectory(module_path('blog'));


Benchmarking Rhai

I think some formal benchmarks will be great.
I don't really have large/complex enough scripts to really do a meaningful benchmark. And I'm not really an expert in this... nor do I have access to proper system for this.
Furthermore, what is more important to measure?  Parsing speed?  Evaluation speed?  Memory/CPU footprint?
Suggestions?

So if we'd be able to store ASTs; I'd consider parsing/optimization benchmarks separately from the actual engine benchmarks.
Some things I'd be interested in are:

Overhead of calling host functions (expensive computations can be offloaded to the host, if the overhead is low).
Some common benchmarks, such as Mandelbrot etc. These are easy enough to implement and give an indication of evaluation speed.
Memory and CPU are important as well; but unless we have a stdlib, or a set of prominent scripts to benchmark as part of the CI, we'd have little use for them? I'm guessing these benchmarks would be used to detect perf regressions.

With benchmarking, simply running them on your local laptop would provide little information, as the actual runtimes may wildly differ depending on the system it is run on.  That way we'd have some meaningful, reproducible feedback.

Overhead of calling registered native functions should be very low, as essentially it is basically calling through a function pointer.  All of the overheads should be in setting up and copying the arguments.

I did put in a prime numbers sample, and it doesn't seem like Rhai is very fast compared to a compiled program.  However, it seems fast enough... not sure when running thousands of times though.

We can probably start with the prime numbers sample to detect regression...

I can have a look at writing a PR this weekend, to add some benchmarks and possibly include them in CI.

I figured that, since Rhai is treating every single value as a dynamic trait object dyn Any, requiring an allocation for every single value that ever occurs in the script, it might be best to try to cut down on some of those allocations.  Otherwise large complex scripts are going to be heavily dragged down.
I have a new branch set up on my repo https://github.com/schungx/rhai/tree/enum-Dynamic that changes Dynamic to an internal enum that is essentially a union of most of the built-in types.  Other customer types are still represented by a dyn Any trait object.
During my preliminary testing on the example scripts, speed_test.rhai runs around 50% faster - it completes one million iterations in 1.7 seconds instead of 2.6 before - I have no idea what kind of 5 year-old laptop @jonathandturner  was using to run it in 0.75 second!  I'd love to have one myself!
primes.rhai, which is a more complex script with more calculations, runs 5x faster.  I think the difference being speed_test.rhai only ever uses one single variable (although it creates one million allocated instances of the values!) while primes.rhai creates a lot more values due to heavy calculations.

Update: changing toolchain (on Windows) to stable-i686-pc-windows-gnu from -msvc makes primes.rhai run even faster, 8x faster than before.  So, I think LLVM does a better optimization job than MSVC (or Rust is optimized to LLVM)...
64-bit stable-x86_64-pc-windows-gnu runs slightly faster than 32-bit on Windows.

@schungx would you interested in bench over: https://github.com/Knetic/govaluate

That would be good, although it does seem to only benchmark relatively small scripted expressions, but those are usually the bulk of a script's workload.
Should be quite simple to translate into Rhai.  However, I think maybe we need to throw in a larger, more complex test script somewhere...

@schungx - yeah, I must have hit some of those llvm improvements. If I remember right, I was on macOS at the time, so that would be llvm.

I was on macOS at the time

LOL probably one of those ultra-high-end, really expensive Macbooks...

@schungx would you be interested in doing a bench to compare with: https://github.com/Knetic/govaluate

@GopherJ  Before we do benchmarks, I suppose we should wait until I merge in the enum version of Dynamic values.  The current implementation based on dyn Any trait objects is never going to win any performance races...

Actually, we should get some benchmarks merged before that, and compare the differences between the enum and Any version. ;)

compare the differences between the enum and Any version. ;)

I'm quite sure that would be >10x to 100x on most complex benchmarks LOL!

https://github.com/Knetic/govaluate

@GopherJ  it took just a few minutes to actually port the govaluate benchmarks to Rhai.  Here are the results:

Test
Timing

test bench_iterations_1000
1,870,790 ns/iter (+/- 1,019,604)

test bench_parse_single_parse
763 ns/iter (+/- 341)

test bench_parse_simple_parse
4,873 ns/iter (+/- 1,054)

test bench_parse_full_parse
20,754 ns/iter (+/- 3,721)

test bench_parse_primes (primes.rhai)
56,155 ns/iter (+/- 11,196)

test bench_eval_expression_single
76 ns/iter (+/- 24)

test bench_eval_expression_number_literal
760 ns/iter (+/- 87)

test bench_eval_expression_number_operators
1,414 ns/iter (+/- 309)

test bench_eval_scope_single
768 ns/iter (+/- 168)

test bench_eval_scope_multiple
779 ns/iter (+/- 445)

test bench_eval_scope_longer
2,241 ns/iter (+/- 639)

test bench_eval_scope_complex
2,038 ns/iter (+/- 1,272)

test bench_type_field
832 ns/iter (+/- 436)

test bench_type_method
594 ns/iter (+/- 164)

test bench_type_method_with_params
840 ns/iter (+/- 283)

test bench_type_method_nested
1,557 ns/iter (+/- 778)

I have a Surface Pro 4 i5 which will not break any speed records, but still it seems like the Rust versions are quite a bit slower than govaluate in evaluations, but much faster in parsing.
Anyone interested in this can surf to my repo
https://github.com/schungx/rhai

I'm quite sure that would be >10x to 100x on most complex benchmarks LOL!

I was wrong!  Learnt yet the same lesson (over and over again) - always benchmark and profile before saying anything!

Test
enum Dynamic
dyn Any

test bench_iterations_1000
1,870,790 ns/iter (+/- 1,019,604)
2,267,275 ns/iter (+/- 1,770,665)

test bench_parse_single_parse
763 ns/iter (+/- 341)
776 ns/iter (+/- 305)

test bench_parse_simple_parse
4,873 ns/iter (+/- 1,054)
5,128 ns/iter (+/- 1,186)

test bench_parse_full_parse
20,754 ns/iter (+/- 3,721)
22,723 ns/iter (+/- 6,244)

test bench_parse_primes
56,155 ns/iter (+/- 11,196)
59,193 ns/iter (+/- 18,476)

test bench_eval_expression_single
76 ns/iter (+/- 24)
130 ns/iter (+/- 40)

test bench_eval_expression_number_literal
760 ns/iter (+/- 87)
906 ns/iter (+/- 367)

test bench_eval_expression_number_operators
1,414 ns/iter (+/- 309)
1,666 ns/iter (+/- 327)

test bench_eval_scope_single
768 ns/iter (+/- 168)
898 ns/iter (+/- 254)

test bench_eval_scope_multiple
779 ns/iter (+/- 445)
924 ns/iter (+/- 182)

test bench_eval_scope_longer
2,241 ns/iter (+/- 639)
2,592 ns/iter (+/- 990)

test bench_eval_scope_complex
2,038 ns/iter (+/- 1,272)
2,684 ns/iter (+/- 2,362)

test bench_type_field
832 ns/iter (+/- 436)
757 ns/iter (+/- 297)

test bench_type_method
594 ns/iter (+/- 164)
500 ns/iter (+/- 112)

test bench_type_method_with_params
840 ns/iter (+/- 283)
961 ns/iter (+/- 692)

test bench_type_method_nested
1,557 ns/iter (+/- 778)
1,624 ns/iter (+/- 1,093)

This shows really how useless micro-benchmarks are...  The actual primes.rhai script runs 6-8x faster with enum Dynamic...

@schungx Yes it seems slower on evaluations, benchmark is important to know the CS stuff:) I'd like that we can add this bench test suite in rhai.

Can we have bench also for creating instances of Engine and register functions because it seems heavy in the flamegraph that we did for casbin-rs: casbin/casbin-rs#109 (comment)

Try Engine::new_raw().  It should be much lighter, but does not register any standard library functions.  Should be perfect for your case.

I tried the benchmarks on my Windows machine with a desktop class i7-8700.
It shows the same ratios/results as seen by @schungx, the only difference being that this CPU appears to have twice the grunt.

Test
enum Dynamic
dyn Any

test bench_iterations_1000
833,840 ns/iter (+/- 26,362)
1,082,180 ns/iter (+/- 29,553)

etc
...
...

Some pretty good news.  An attempt to make the Dynamic type as small as possible (down to 8 bytes for 32-bit builds without floats and i64, and 16 bytes for 64-bit builds) yielded some pretty significant speed improvements.

PR https://github.com/jonathandturner/rhai/pull/144 is now significantly faster (20x for the primes.rhai script).
This is achieved via "deep-linking" - meaning to pass around mutable references as much as possible to minimize data copying.  I have been trying to make this change for a long time, did it four times before finally finding a solution to get the borrow checker off my back.
It also solves the ages-long problem of not able to modify a deeply-nested chain of arrays and dot properties.

PR https://github.com/jonathandturner/rhai/pull/152 has another speed boost.  Particularly, it eliminates an allocation during each function call, resulting in significant speed improvements for scripts with lots of operator usage.

Rhai appears to be fast enough right now. Some benches at https://schungx.github.io/rhai/dev/bench/
I believe most of the low-hanging fruits have already been plucked.  To gain more speed, we have to go into byte-codes.  We now need to balance this with new features we can add for the same amount of developer time.
To put things into context, the speed_test script now runs in 0.25 seconds on a 2.3GHz standard cloud Linux VM.  Running the fibonacci script for 28 levels is 0.7 seconds.  Comparing it with wren benchmarks (http://wren.io/performance.html), it is not too far off, considering that the CPU running the wren benchmark has roughly twice the processor throughput.  So I estimate the equivalent speed to be around 0.4-0.5 seconds on the wren machine, compared with 0.2 seconds for wren.  This puts Rhai up to the speed level of Python at least.
Again, your milage may vary based on your usage scenario.  If you only need to crunch basic data and not interop with Rust functions/types at all, you'll be penalized for this flexibility (because during every operator/function call, the engine asks... "is this function overloaded? ... Hhmmm... no, built-in version then.").


Compressed snapshot for backing up

As the question in https://github.com/elastic/elasticsearch-hadoop/issues/646:
If the snapshot files in the hdfs were compressed, such as in '.gz' format, we can't use the compressed snapshots to restore.
I have another question:
As the backup progress is incremental, if the existing snapshots were compressed, can we still continuing backing up new snapshots?
If not, how can we do back up work on conditions that the existing snapshots were compressed?
Thank you!

@weiqiangyuan Please use the forum for questions, not the issue tracker.


Add new Sponsors and partners logos

Sponsors

Typeform

Partners

Leo Boeck

Waiting for last sponsor logo to pull request


Move @types packages to dev-deps in package.json to keep typescript f…

…rom throwing duplicate identifier errors
This fixes https://github.com/zenoamaro/react-quill/issues/316
Thank you!

@types/react is already a dev dependency, and there haven't been any reported issues with @types/quill so far, so I will close this.


which one is for android

Hi @ZhangmingWei1994, could you please be more specific with your question? Please follow the issue template and feel free to re-open this issue.

i have to re-install it every time i restart my device , i don't know why, it just cannot open from desk, (my laptop and desktop have same proble, both win 10 system)


Not able to setflex-shrink=0

When I try to set flex-shrink=0, it's not showing up in the generated CSS. I believe this is because 0 would be the same as undefined/null/etc in the current code.
As a quick test, I was able to make a small change that worked (EDIT: Actually that does not work since the type is number, so the default I guess would be 0 -- I'm not too familiar with typescript):
  if (styleObj.getFlexShrink() || styleObj.getFlexShrink() === 0) {
    style += `flex-shrink: ${styleObj.getFlexShrink()};`;
  }

One reason for this issue is that that flex-shrink defaults to 1 (See https://developer.mozilla.org/en-US/docs/Web/CSS/flex-shrink).
So it by not including the style, I'm not able to get the flex-shrink = 0 behavior that I was looking for.
This may be an issue with other styles. We may want to explicitly check for null/undefined for all styles. So if it is set, then we should use the value that is specified.
Source location: https://github.com/google/mesop/blob/24f77303b6b5f014ae882e74d35464b06a76632d/mesop/web/src/utils/styles.ts#L5

Since proto3 doesn't allow us to set default values, maybe one option is to set a default of 1 in https://github.com/google/mesop/blob/main/mesop/component_helpers/style.py#L199
Then we can update the condition to check for if it equals 1 for flex shrink in https://github.com/google/mesop/blob/main/mesop/web/src/utils/styles.ts#L70

Alternatively we could also change the proto data type from int to string. That way an empty string could act as a not set and then default to whatever is the default for that style. Downside is we'd lose the type checking of int values. And would probably need some input validation at some point.
I see some cases where you have the possibility of int | str in style.py and the corresponding proto value is a str. So that could work here and keep things consistent with existing behavior.

Using a str proto value SGTM - thanks for investigating into this. I didn't know flex-shrink had a default value of 1 😂


CB-21966 Expose datahub endpoint to refresh the datalake configuration

See detailed description in the commit message.

@horadla23 @lnardai @doktoric can this PR be merged?


if tf.reduce_sum(y_hat, 1) == self.token2idx[""]: break

I don't understand the conditional sentence in the loop when infer, it is always false。
if tf.reduce_sum(y_hat, 1) == self.token2idx[""]: break

Yes, I have this question too.

+1

+1

you are right. the author is wrong. I also think the following "decoder_inputs = tf.concat((decoder_inputs, y_hat), 1)" is wrong. cuz what we need to concat is not the total y_hat but the last time step of y_hat. so y_hat[:,-1]


Factory restore for NSPanel

I am looking for a way to factory restore the NSPanel.
What i mean by factory restore is to flash the original HMI interface and the original BIN for the esp32.
Getting backup for the ESP is easy and i have done it. Now how to do the HMI/TFT part? or is there some place where i can get the original TFT. HMI would be much nicer but beggars can't be choosers.

There You have it:
https://cdn.discordapp.com/attachments/538814618106331139/925351768852951050/ns-panel.tft

Hello, the second link (US version) does not work anymore. Have someone a backup? Thanks!

There You have it:
https://cdn.discordapp.com/attachments/538814618106331139/925351768852951050/ns-panel.tft

DIsplay says: "Model does not match"

There You have it:
https://cdn.discordapp.com/attachments/538814618106331139/925351768852951050/ns-panel.tft

The link sadly doesn't work anymore. Is there an updated url?

There You have it:
https://cdn.discordapp.com/attachments/538814618106331139/925351768852951050/ns-panel.tft

Discords new attachment urls breaks this. anyone have a new link?

Same! Would like to have it too!

There You have it:
https://cdn.discordapp.com/attachments/538814618106331139/925351768852951050/ns-panel.tft

DIsplay says: "Model does not match"

Could somebody please re-upload the tft files. They seem to have been deleted by discord.
I need the US variant specifically but having both available could be nice.
thanks.

Have a look here:
https://github.com/Blackymas/NSPanel_HA_Blueprint/tree/main/hmi

Does anybody have a link to original EU firmware bin file?

@ilexpl01 just to be clear, do you mean that here (https://github.com/Blackymas/NSPanel_HA_Blueprint/tree/main/hmi) are the stock firmware files for NSPanel, in case we want to rollback from the custom firmware?
https://github.com/masto/NSPanel-Demo-Files/issues/4#issuecomment-1034608391

From what I can see these are the factory files for the screen, not the NSPanel itself. I don’t see any .bin files there for the esp32.

Hey folks. Couple of things:

There is no way to restore the original firmware (the one that it comes with, that works with their app) unless you backed it up before flashing. The firmware is locked to the device, so you can't use someone else's copy.

While you can restore the original TFT, Sonoff considers that file their property. It is not open source, they have not shared it, and they have not given permission for anyone else to share it. When it pops up somewhere, it gets taken down.

If you have already flashed your device and you're trying to restore because you can't get it working, my suggestion is to follow something like https://github.com/Blackymas/NSPanel_HA_Blueprint. The tutorial video I made 2 years ago was before other folks had built these turnkey solutions, and it's a very low-level DIY approach. It still works if you want a fully custom design, but most of the time it will be a lot easier to use the existing building blocks.
I am going to lock this issue because people turning up here from Google searches, but there's nothing in this repo that can help folks with a factory restore. These files and wiki were intended to facilitate the homebrew approach.


Miscellaneous security fixes have been made in the core expression and spark

Fixed security vulnerabilities-
Unreleased Resource: Streams
utils/OrderedPropertiesHelper.java, line 56
utils/OrderedPropertiesHelper.java, line 57
utilities/OrderedPropertiesHelper.java, line 56
PropertiesHelper.java, line 39
PropertiesHelper.java, line 55
PropertiesLoader.java, line 59
Unreleased Resource: Database
RunSQLComponent.java, line 58
RunSQLComponent.java, line 81
RunSQLComponent.java, line 104
RunSQLComponent.java, line 127
Portability Flaw: Locale Dependent Comparison
CommandLineOptionsProcessor.java, line 81
Added SafeResourceClose util class to safely close the open resources.

getProperties() method in PropertiesLoader class is not Properly handled in try catch.


APB-6706 wrong heading on page relating to number of selected members

I've already got this in my PR


The order of props should not matter

💬 Questions and Help
I noticed that
<x.div borderBottom={2} borderColor="blue-400">
  hello
</x.div>

and
<x.div borderColor="blue-400" borderBottom={2} >
  hello
</x.div>

will produce different results. It's because the last one will generate:
border-color: #60a5fa;
border-bottom: 2px solid;

and the default color (black) for border-bottom will override border-color. Is there a better way to write it so that I don't have to remember that ordering props matter?
Ideally, is it possible to make the following code a reality?
<x.div borderBottom="2px solid blue-400">
  hello
</x.div>

Thanks! :)

That's a fascinating bug, and well-explained, thanks!
Order matters in CSS, therefore order matters in props. Without preserving order, you wouldn't have a workaround. Additionally, the order of props is important because otherwise the caller has to know the specific props to override:
const Foo = props => <x.div borderColor="red" {...props} />

<Foo border="2px solid green" />

The final border should be green, not red, and the caller shoudn't need to know that the inner component used borderColor instead of the border shorthand. That wouldn't work if the order weren't preserved.
About your ideal case, we might be able to do something for simplistic cases. The problem is that it gets tricky with complex space-separated values, and there's limited value to increasing the complexity of the code to handle them. But I can imagine handling the simple cases at some point.
For now, though, have you considered using borderBottomStyle and borderBottomWidth to avoid the effect of the border shorthand?

Glad it got your attention :D.
I can use borderBottomStyle and borderBottomWidth indeed.

What's even more interesting about this behavior is that the docs generate something entirely different:
<x.input border={2} borderColor="red-600" />

produces:
border: 2px solid rgb(220, 38, 38);

Doing that same thing locally I get:
border: 2px solid;
border-color: rgb(220, 38, 38);


Duplicate names in model selection

Feature Request
Is your feature request related to a problem? Please describe.
Currently if you point openwebui at two different instances of the same backend; the models in the "users" section are all duplicated (because it does a request to /models and gets the same ones twice) making it near impossible to tell which entry corresponds to which backend
Describe the solution you'd like
Suffix all entries with the id of the backend e.g. llama-3.1 (#1) which would map to the backend order setup in connections tab
Describe alternatives you've considered
I suppose bruteforce until I see usage on the right backend? with 5+ connections thats quite a game though because the entries are not sorted in any way.
Thanks!

Already added to dev.


[Bug]: JS where filter has wrong types

What happened?
let results = await collection?.query({
  queryTexts: [query],
  where: {
    $and: [
      {
        c_date: {
          $lt: date
        }
      }
    ]
  }
})

This complains
Property '$and' is incompatible with index signature.
      Type '{ c_date: { $lt: number; }; }[]' is not assignable to type 'LiteralValue | OperatorExpression'.

The query works if you cast the types to any
where: {
  $and: [
    {
      c_date: {
        $lt: date
      }
    }
  ]
} as any

Versions
chroma js 1.5.2
Relevant log output
No response

thanks @tonisives - opened a PR https://github.com/chroma-core/chroma/pull/722


[Documenation] There is no styling information in docs

Their is no hints and tips on how we style the toast while it does appear its very plane looking

This library does not look after the styling of toasts. You can override the default styling provided by toast or Noty, whichever you are using by adding custom css rules.

U have provided no links so can complete this and close this can you at lest provide some samples plz


enhancement: option for anonymous configuration

Options to configure vsftpd for anonymous read-only access would be great

Hi, PRs are always welcome!