(Serverless: OS Support Initiative): Document how to use Prisma with Docker / Linux Distros

Setting up Prisma on Docker environments is non-trivial, as plenty of comments in our Github issues demonstrate.
We should prepare a tutorial page that describes step-by-step how to use Prisma in the most popular Docker images, both for deploying serverless services (e.g., via AWS Fargate + AWS ECS) and for local development.
We should also take system architecture differences between the users host machine and their runtime machine into account (e.g., macOS users with Silicon CPUs won't be able to use alpine-based images locally unless they use Docker Buildx with --platform="linux/amd64", as Prisma doesn't work on Alpine on arm64: https://github.com/prisma/prisma/issues/8478)
We should target the following system scenarios, inspired from our ecosystem-tests:

node:lts-alpine3.17 (and variants, like node:lts-alpine, node:alpine) on amd64 and arm64
node:16-slim (and variants, like node:lts-buster-slim) on amd64 and on arm64
gcr.io/distroless/nodejs18-debian11 on amd64

Opinable, these were mainly relevant when prisma didn't support openssl 3 or for users with weird linuxbrew/snap setups, which are arguably not generally used for serverless and local development on Docker:

ubuntu:20.04 on amd64
ubuntu:22.04 on amd64

We should show our users how to:

install Prisma on Docker (with the above scenarios, installing missing system dependencies when needed)
connect to a database (e.g., Postgres locally exposed via docker-compose.yaml, whose URL one can configure when building the Docker image)
running prisma db push
generate types
use the Prisma Client

For constrained, barebone environments (such as gcr.io/distroless images and actual serverless services) we should tell users to write multi-stage Dockerfiles and only deploy the Prisma Client, delegating commands like prisma db push and prisma generate to a previous docker stage.
On Linux Alpine we should tell users not to install libc.

This follows up on https://github.com/prisma/docs/issues/4303 .

Follow-up TODOs

[ ] Add a https://pris.ly/d/docker link to this new docs page

[ ] Add a https://pris.ly/d/docker-alpine link to the Alpine subsection (as required by https://github.com/prisma/prisma/pull/17389#issue-1537761994)

Working on this draft:

How to run Prisma on Docker
Prisma supports the most of the popular Docker images for Node.js out of the box.
However, due to the system dependencies the Prisma engines rely on and the complexities of providing compiled binaries for all operating systems and CPU architectures, some adjustments may be needed.
In this tutorial, we'll walk you through the process of setting up a Docker container for a Node.js application that uses Prisma. Given a simple Prisma schema, an empty Postgres database, and a basic Express.js server running on port 3000, we'll create a Docker image that initialises Prisma, runs the server and connects to the database.
Preliminaries
Create a new prisma-docker-tutorial directory, open it, and paste the following files in there:
// ./prisma/schema.prisma
generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "postgres"
  url      = env("DATABASE_URL")
}

model Author {
  id   Int      @id
  post Post[]
}

model Post {
  id     Int    @id
  user   Author @relation(fields: [userId], references: [id])
  userId Int
}

// ./server.js
const express = require('express')
const app = express()
const port = 3000

const { PrismaClient } = require('@prisma/client')

const client = new PrismaClient()

app.get('/', async (req, res) => {
  const data = await client.user.findMany()
  res.send(JSON.stringify(data))
})

app.listen(port, () => {
  console.log(`App listening at http://localhost:${port}`)
})

// ./package.json
{
  "name": "@prisma/docker-tutorial",
  "license": "MIT",
  "devDependencies": {
    "prisma": "4.10.0"
  },
  "scripts": {
    "start": "node server.js"
  },
  "dependencies": {
    "@prisma/client": "4.10.0",
    "express": "4.18.2"
  }
}

./docker-compose.yml
version: '3.7'

services:
  postgres:
    image: postgres:15
    hostname: postgres_db
    restart: always
    environment:
      - POSTGRES_DB=postgres
      - POSTGRES_USER=prisma
      - POSTGRES_PASSWORD=prisma
    ports:
      - '5432:5432'
    networks:
      - prisma-network

  prismacmd:
    image: prisma-docker-tutorial
    stdin_open: true
    environment:
      - DATABASE_URL=postgresql://prisma:prisma@postgres_db:5432/postgres
    depends_on:
      - postgres
    networks:
      - prisma-network

networks:
  prisma-network:
    name: prisma-network

This tutorial has been tested on Docker version 20.10.16.

We define the ./Dockerfile as follows:
Linux Alpine (node:alpine)
This is the default Docker image for Node.js. It is based on Alpine Linux, which is a lightweight Linux distribution that uses the musl C standard library.
Prisma supports Alpine on amd64 out of the box, and supports it on arm64 since<EMAIL_ADDRESS>Related Docker images:

node:lts-alpine
node:16-alpine
node:14-alpine

# syntax=docker/dockerfile:1
FROM node:lts-alpine3.17

WORKDIR /usr/src/app
COPY --from=app . ./

RUN npm i \
  && npx prisma generate

EXPOSE 3000
ENTRYPOINT ["node", "server.js"]

Note: when running on Linux Alpine, Prisma downloads engines that are compiled for the musl C standard library. Please don't install glibc on Alpine (e.g., via the libc6-compat package), as that would prevent Prisma from running successfully.

Linux Debian (node:slim)
This image is based on Linux Debian, and uses the glibc C standard library.
It is mostly supported out of the box on amd64 and arm64, but as some older versions of this image come without libssl installed, it is sometimes necessary to install it manually.
Related Docker images:

node:lts-slim
node:bullseye-slim
node:buster-slim
node:stretch-slim

# syntax=docker/dockerfile:1
FROM node:slim

# Note: uncomment the following block if you're using an image based on Debian Buster, which doesn't come with libssl pre-installed
RUN apt-get update -y \
  && apt-get install -y openssl

WORKDIR /usr/src/app
COPY --from=app . ./

 They do not contain package managers, shells or any other programs you would expect to find in a standard Linux distribution, which is great for the security of the system.
Although this image is originally based on Linux Debian Bullseye, some manual care is required before it can be used with Prisma. In particular, commands like npm install and prisma generate must be run in a separate build stage, as the distroless image doesn't come with npm or a shell pre-installed. Moreover, the zlib system dependency is missing, and must be copied from a compatible Debian image manually.
On Distroless, only the amd64 architecture is supported.
# syntax=docker/dockerfile:1

# Build stage on Debian 11
FROM node:18-bullseye-slim as deps

WORKDIR /usr/src/app
COPY --from=app . ./

# Uncomment the following block if you're using an image based on Debian Buster, which doesn't come with libssl pre-installed
RUN apt-get update -y \
  && apt-get install -y openssl

WORKDIR /usr/src/app
COPY . ./

RUN npm i \
  && npx prisma generate

# Runtime stage on Distroless
FROM gcr.io/distroless/nodejs18-debian11 as runtime

WORKDIR /usr/src/app
COPY --from=deps /usr/src/app/ ./

# Copy zlib from Debian, which is needed by Prisma
COPY --from=deps /lib/x86_64-linux-gnu/libz.so.1 /lib/x86_64-linux-gnu/libz.so.1

# the distroless' image entrypoint is /nodejs/bin/node
CMD ["server.js"]
EXPOSE 3000


Warning: Native component for "RNBarcodeScannerView" does not exist

Steps to reproduce

added option onBarCodeRead for IOS;
in emulator show "Warning: Native component for "RNBarcodeScannerView" does not exist"

Expected behaviour
work without warnings in Emulator
Actual behaviour
"Warning: Native component for "RNBarcodeScannerView" does not exist"
Environment

Node.js version:
**React Native version0.20:
React Native platform + platform version: iOS 9.3, etc

react-native-camera
Version: npm version or "master"

i do have this problem with an android phone (galaxy s5 neo) too

found a solution? I'm getting the same error

same issuue here

Hmm, maybe you want to post this issue to react-native-barcodescanner instead of react-native-camera?
(RNBarcodeScannerView is used by react-native-barcodescanner not react-native-camera, or at least I can't find anything related to RNBarcodeScannerView in this project)
https://github.com/ideacreation/react-native-barcodescanner


feat: support typst

Hello, I just added support for typst  in the plugin, and it seems to be working fine. However, I'm unsure if this PR can be merged, as it depends on too many files, and the files are not loaded using logseq.Experiments.loadScripts(). This is the only way I could do it; I can't find a better solution. If you have any other thoughts, please let me know.
https://github.com/user-attachments/assets/bce55714-3ad2-4572-b9a6-baa495d1b709

Thank you @xyhp915 ! The code you optimized is fantastic; it eliminated a lot of JS file imports and switched to using npm imports. 👍


can you tell me some exmaple for events

such as show ,shown,hide and so on

Just use on, like:
 $("#some-element").on('bl.show', (event) => { console.info("show", event) })  $("#some-element").on('bl.shown', (event) => { console.info("shown", event) })


Unable to authenticate after updating home assisant.

Upgraded to 2024.1.5 and the integration broke. I reverted back to 2023.11.3 and the problem followed.  I am unable to sign in from Home Assistant. I can sign in fine on the Dreo app.  I've got 3 entries in the logs. Did the API server change?
This error originated from a custom integration.
Logger: dreo
Source: custom_components/dreo/init.py:38
Integration: Dreo (documentation, issues)
First occurred: 5:31:50 PM (1 occurrences)
Last logged: 5:31:50 PM
Unable to login to the dreo server
This error originated from a custom integration.
Logger: pydreo
Source: custom_components/dreo/pydreo/helpers.py:166
Integration: Dreo (documentation, issues)
First occurred: 5:31:48 PM (1 occurrences)
Last logged: 5:31:48 PM
No response from API
This error originated from a custom integration.
Logger: pydreo
Source: custom_components/dreo/pydreo/init.py:237
Integration: Dreo (documentation, issues)
First occurred: 5:31:48 PM (1 occurrences)
Last logged: 5:31:48 PM
Error logging in with username and password

Have you upgraded to latest dreo integration version?
-Jeff
On Jan 20, 2024, at 3:38 PM, chiments @.***> wrote:

Upgraded to 2024.1.5 and the integration broke. I reverted back to 2023.11.3 and the problem followed. I am unable to sign in from Home Assistant. I can sign in fine on the Dreo app. I've got 3 entries in the logs. Did the API server change?
This error originated from a custom integration.
Logger: dreo
Source: custom_components/dreo/init.py:38
Integration: Dreo (documentation, issues)
First occurred: 5:31:50 PM (1 occurrences)
Last logged: 5:31:50 PM
Unable to login to the dreo server
This error originated from a custom integration.
Logger: pydreo
Source: custom_components/dreo/pydreo/helpers.py:166
Integration: Dreo (documentation, issues)
First occurred: 5:31:48 PM (1 occurrences)
Last logged: 5:31:48 PM
No response from API
This error originated from a custom integration.
Logger: pydreo
Source: custom_components/dreo/pydreo/init.py:237
Integration: Dreo (documentation, issues)
First occurred: 5:31:48 PM (1 occurrences)
Last logged: 5:31:48 PM
Error logging in with username and password
—
Reply to this email directly, view it on GitHubhttps://github.com/JeffSteinbok/hass-dreo/issues/69, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABX5XT27KXDUKIVMYV4BA6TYPRIN5AVCNFSM6AAAAABCDQZ74GVHI2DSMVQWIX3LMV43ASLTON2WKOZSGA4TEMRXGY2DIMY.
You are receiving this because you are subscribed to this thread.Message ID: @.***>

Yes, sorry. I was under the impression that I was on the latest release. Doing a manual re-install seemed to take care of it. Sorry to waste your time, you can close this.


Fastnlight: Watch +91-9063382769

Fastnlight requests the watch of the watch_number +91-9063382769. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

Approved by tripleee in Charcoal HQ


interactions was not working ,What was the wrong over here?

stdClass Object
(
    [completion_status] => incomplete
    [completion_threshold] =>
    [credit] => credit
    [entry] =>
    [exit] => suspend
    [launch_data] =>
    [student_id] => 123456789
    [student_name] => Ramesh Bharwad
    [location] => {"curPage":5,"completed":[0,1,2,3]}
    [max_time_allowed] =>
    [mode] => normal
    [progress_measure] =>
    [scaled_passing_score] =>
    [session_time] => PT16.91S
    [success_status] => unknown
    [suspend_data] => [["RokUsTwQ58","IQjIr4yheT","emusyAUc4f","v44THuNrq1","693395e7-a187-4ac6-877a-d2278b52889d","b4f42983-15bc-4f3f-b526-a7e8d982a477","0b02dcb3-1490-4426-b4e5-ae934f1339b2","5467e982-be88-46af-ba75-4d2ffc9539b1","7d73f654-9e04-4a6d-adf2-631a0787750c","a|0|1|2|3|4|5|6|7|8","completed","a|A","b|T","o|B|C","started","result","a|E|F|B","score","totalAnsweredCorrectly","totalAnsweredIncorrectly","totalAnswered","a|H|I|J|K","n|K","n|1","n|4","n|5","o|L|M|N|O|P","o|G|C|Q|C","answer","submitted","answeredCorrectly","a|S|T|U","brightest","blackest","Pontus","a|W|X|Y","dfd","dfgd","o|Z|a|b|b","b|F","o|V|c|C|d","dfgdf","o|V|f|C|d","270e9ee7-7156-4e26-ba05-953d2ba0aa26","o|V|h|C|d","response","a|B|j|H","droppable-destination-0513cb83-a810-4881-85b4-07d23dafb060","droppable-destination-d37ef74a-e471-425c-a447-ea47ae033537","a|l|m","draggable-0513cb83-a810-4881-85b4-07d23dafb060","draggable-d37ef74a-e471-425c-a447-ea47ae033537","o|n|o|p","o|k|C|q|N","o|V|r|C|C","hjjh","o|V|t|C|d","o|9|D|D|D|R|e|g|i|s|u"],"v"]
    [time_limit_action] => continue,no message
    [total_time] => 0
    [comments_from_learner] => stdClass Object
        (
            [childArray] => Array
                (
                )

        )

    [comments_from_lms] => stdClass Object
        (
            [childArray] => Array
                (
                )

        )

    [interactions] => stdClass Object
        (
            [childArray] => Array
                (
                )

        )

    [learner_preference] => stdClass Object
        (
            [audio_level] => 1
            [language] =>
            [delivery_speed] => 1
            [audio_captioning] => 0
        )

    [objectives] => stdClass Object
        (
            [childArray] => Array
                (
                )

        )

    [score] => stdClass Object
        (
            [scaled] => 0.2
            [raw] => 20
            [min] => 0
            [max] => 100
        )

)

Had a problem with Interaction just lost 2 hours thinking it was my code. Switch to native API call and it works.
Don't know if this repo is still working/active/maintained. Hope I saved somebody's else time cause I can't get mine back 🤣


Add constraint that files should be unique within the project

When data files get exported in the Terra staging area, they are located in the same directory under the project
e.g.
gsutil ls -l gs://broad-dsp-monster-hca-dev-ebi-staging/dev/1e9334fc-48e8-4b6b-a1fb-7ff44821bb0b/data

If data files are added in separate submissions, we should add a check that filenames should be unique within the project.


Nwang/count stmgr connection failures in all clients

We found out that when a bad stmgr is experiencing connection issue to
all other stmgrs, it takes the stmgr a lot longer to reach the per stmgr
client max attempt and quite becaus bad connections cause extra delays.
In our case, instead of quitting after 5 minutes (1s reconnect timeout,
300 max attempts), the bad stmgr quitted after more than 2 hours. At the
same time, all other good stmgrs were quitting every 5 minutes.
A total reconnection attempt checking is added in order to make the bad
stmgr quit sooner.

Dropping this PR in favor of the new solution: https://github.com/twitter/heron/pull/2711


Automate Wear Screenshots

First step towards keeping screenshots automatically inline with the app design.
$ fastlane wearScreenshots

Next steps would be mobile, then eventually uploading to the play store on each release.

Nice!


Add support for custom registry

This PR adds support for overriding the container registry used to pull required images. It assumes that if a user both specifies an override registry for all images and then explicitly specifies an image from a different registry on a CassandraDatacenter resource that the more specific statement should win (in this case, explicitly state the image to use on the resource wins over the override registry).

Integration tests are not working at present due to issues with k3d/kind. Once that is sorted, I'll run them.


Strict version=None for non-versioned archives

In this PR:

raise error when passing non-None versions to unversioned archive methods
improve test coverage

requires bumpversion patch

Coverage increased (+0.4%) to 93.495% when pulling e21746cf5b57e53e7c4c8dedf0ab96237dbe5520 on raise_versions into e18f9fec7c1289483ac96ca38a6ab7ca6ef66e86 on staging.

Coverage increased (+0.3%) to 93.453% when pulling 3aeb139d226caa1df43c9913557cd9de95174958 on raise_versions into 900ad2a52cfecd9816db7f2747655bfec445d2c7 on staging.

Coverage increased (+0.3%) to 93.458% when pulling 1fd8e0faf8343b9b213944acdc4aa70cba45a872 on raise_versions into 900ad2a52cfecd9816db7f2747655bfec445d2c7 on staging.


Allow configurable signing algorithms

Description
@tetsuo-cpp filed similar issues under Cosign and Rekor. We realise there's a lot of overlap in maintainers, but wanted to make sure that we discuss each project that we plan to touch. Apologies if this feels a bit spammy.
Hi there! At Trail of Bits, we're looking at potentially implementing part of the Configurable Crypto Algorithms proposal (specifically Phase 1). We wanted to float this idea to each of the relevant Sigstore sub-projects so we can hash out the details in a more concrete way.
Across the Sigstore stack, we default to using ECDSA for signatures and SHA256 for hashing. There's more detail in the linked proposal but there are a number of motivations for wanting to customise the signatures that are generated, including paving the way for post-quantum signatures. The proposed design includes having a "supported algorithm" registry (perhaps this can go in the Protobuf specs) that enumerates the approved signature/hash algorithm combinations. We specifically don't want to allow arbitrary mixing and matching of signature and hash algorithm to avoid some of the security pitfalls listed in the proposal.
For sigstore-go, we want to support this set of approved signing algorithms. This can be as simple as a --signing-algorithm flag. As a first pass, we'd like to support ECDSA with SHA256 and SHA384 as well as EdDSA, with ECDSA-SHA256 remaining as the default.

Hello! I think this should be relatively straightforward as sigstore-go today just supports verification, not signing. We can add additional signing algorithm support as needed to things like pkg/verify/signature.go and pkg/root/trusted_root.go.

One requirement of this work is that verifying clients will need to take information about what signing algorithm was used to generate the signature since we can no longer assume SHA-256.
I think that this information can be part of the Sigstore bundle's VerificationMaterials, but each client including sigstore-go will require a --signing-algorithm flag for the non-bundle verification path.
Does this seem reasonable to you: @steiza @haydentherapper?

{
  "mediaType": "application/vnd.dev.sigstore.bundle+json;version=0.2",
  "verificationMaterial": {
     ...
  },
  "messageSignature": {
    "messageDigest": {
      "algorithm": "SHA2_256",
      "digest": "Xd/81x5QNKXw3pdWCvhoA2H4aS7Yd9F8hxlfyUtUVwg="
    },
    "signature": "MEQCIEeH9Ktf5pkHgdksH7mJCm4Jl66LemacsZHQ0VLKpgPOAiBsSM+neKdZE3l3AhxII1RuQOSKEk+TZRCspMjhvPxB0g=="
  }
}

@tetsuo-cpp are you talking about the signature algorithm or the hash one? I think the hash algo is already included in messageDigest in the bundle file. For the signature algorithm, instead, you could probably already deduce it from the Public Key included in the certificate (which is in the bundle), however even adding a "SignatureAlgorithm" in the bundle might not be ideal?
I believe we want the signature algorithm to come from an external policy, because you can't trust the file itself. If I sign a file with ECDSA-SHA256 today and in ten years from now ECDSA-SHA256 is broken, you want to make sure that the signature are from e.g. ED25519 and not ECDSA-SHA256, even though the file says so.

@tetsuo-cpp are you talking about the signature algorithm or the hash one? I think the hash algo is already included in messageDigest in the bundle file.

I don't think we can use that because you can also get a DSSE envelope in the bundle. From the spec:
        oneof content {
                dev.sigstore.common.v1.MessageSignature message_signature = 3 [(google.api.field_behavior) = REQUIRED];
                // A DSSE envelope can contain arbitrary payloads.
                // Verifiers must verify that the payload type is a
                // supported and expected type. This is part of the DSSE
                // protocol which is defined here:
                // <https://github.com/secure-systems-lab/dsse/blob/master/protocol.md>
                io.intoto.Envelope dsse_envelope = 4 [(google.api.field_behavior) = REQUIRED];
        }

Since we need this information in both cases, I figured it made sense to add it into the bundle (and we should wire any code that uses the message digest algorithm, to refer to the signing algorithm instead). That has the side-effect of ensuring that the bundles don't refer to a signing/hash combination that isn't valid in the Sigstore registry.

I believe we want the signature algorithm to come from an external policy, because you can't trust the file itself. If I sign a file with ECDSA-SHA256 today and in ten years from now ECDSA-SHA256 is broken, you want to make sure that the signature are from e.g. ED25519 and not ECDSA-SHA256, even though the file says so.

To me, this is two separate things: knowing what the signature algorithm is and restricting what signature algorithms are allowed. I'm proposing that the bundle format will document what signing algorithm was used to produce the signature so that a client knows how to verify it. That leaves the door open for clients to toggle what signing algorithms are allowed so that you can restrict this similar to how we allow them to be restricted in Fulcio and Rekor via --client-signing-algorithms.

cc @woodruffw I think similarly to rekor, we want to limit the flexibility to the regular "hashedrekord" types, which I'm not sure include dsse_envelope. If that's the case, for dsse_envelope case we can just assume the default is whatever is used right now, while the flexibility is allowed only in the case of message_signature.

Can you use the envelope hash from the Rekor entry?


Could you support Web Cams?

I have one integrated on my laptops and an external one and managing them is as much of a pain as audio devices :)

In what respect?
How do you currently use multiple webcams, and what do you want to achieve?
It's highly unlikely that audioswitcher will support webcams in it's current state

Especially "startup device" - oh but wait -windows does not support marking
them as default does it?  More important it would be cool to switch both
mic and speakers to pairings I specify.
You know what would be super helpful!
When a communication app "rings" switch to my headphones, when there is
nothing on the mic for x minutes switch back to another source I specify (
speakers.  I constantly have to do this but don't want to leave the
headphones on to hear IM tones.
D.
On Nov 6, 2015 11:52 AM, "Sean Chapman"<EMAIL_ADDRESS>wrote:

In what respect?
How do you currently use multiple webcams, and what do you want to achieve?
It's highly unlikely that audioswitcher will support webcams in it's
current state
—
Reply to this email directly or view it on GitHub
https://github.com/xenolightning/AudioSwitcher_v1/issues/1457#issuecomment-154469709
.

Yeah, no idea if it's possible for Webcams; probably not. It's application specific.
The other idea would require an insane amount of planning and testing. Something that is unlikely to happen for free; or in Audio Switcher.
Closing as issue no longer related to Audio Switcher.


DeployDB should be able to reliably hit web hooks

Requirements for the queueing and backoff logic

2xx response codes are considered successes and discarded
3xx response codes are followed no more than 3 levels
400 response codes should be discarded
420 response codes are retried per back-off logic
5xx response codes are retried per back-off logic

Back-off

Exponential back-off from 500ms
No more than 5 attempts
Failures after 5 attempts are logged as errors and discarded

With com.github.lookout.whoas 0.1.2 I think I can safely close this out


Mistake in devices detection

is_ios , is_androidos and other dont't work without that patch

Thank you @defragmentator! :-)


Detectors for male and female words should be combined

Story: As a user, I want to compare words biased towards women with words biased towards men, so that I can see if my letter is strongly biased one way or the other.
If we combine the malewords and femalewords detectors, then the summary can include a ratio. Male words should have positive valence and female words should have negative valence.
@molliem told me she always meant these word lists to be handled in a single detector.

@molliem says this really agentic (+) vs communal (-). Positive and negative are in this context, of course.
Be clear that communal words do not need to go - instead we need to add more agentic words. This is similar to how we don't want to delete effort words, but they need to be balanced with accomplishment words.

The "recruiting" language seems like it was borrowed from Textio or something. It's not appropriate to the evaluation context.


[Bug] Restore from backup slows down massively

🐛 Bug
Testing restore of a 750M transaction chain on big testnet. Initially the restore progresses at ~25k TPS, but the speed then drops off significantly.
http://mon.big.aws.hlw3truzy4ls.com/d/storage/storage?orgId=1&from=1619026800000&to=1619031600000

System information
Please complete the following information:

3f7de435

Debugging thread: https://toroeng.slack.com/archives/C01CEC7QGJU/p1619037090018900
Issue does not seem to be caused by EBS, CPU or network throttling. Restarting the restore pod to resume the restore results in the same behaviour of full speed for a while, and then significant slowdown.

PR for fix: https://github.com/diem/diem/pull/8256


Fix not loading any CBRs if cache files missing

Task/Issue URL: https://app.asana.com/0/414235014887631/1203344021820506/f
Pre-release for testing purposes: https://github.com/duckduckgo/BrowserServicesKit/releases/tag/pre-34.0.5
iOS PR:  https://github.com/duckduckgo/iOS/pull/1379
macOS PR: https://github.com/more-duckduckgo-org/macos-browser/pull/835
What kind of version bump will this require?: Patch
Description:

Fixes waiting for Content Blocking Rules to be applied on startup when WebKit cache files are missing

Steps to test this PR:

Delete Content Blocking Rules cache files on the file system

~/Library/Developer/CoreSimulator/Devices/DEVICE-ID/data/Containers/Data/Application/APPLICATION-ID/Library/WebKit/com.duckduckgo.mobile.ios - for iOS Simulator
~/Library/WebKit/com.duckduckgo.macos.browser - for macOS

Run the app and load a website
Validate ContentBlockingRules are compiled before navigation

OS Testing:

[ ] iOS
[ ] macOS

Good find, I think we need tests for these scenarios - otherwise in a few months noone will remember about this case and regression may happen.


Search for search terms no further than n words apart

Is your feature request related to a problem? Please describe.
This request came as a potential future request from the People & Parliament team: they would like to search for two search terms and make sure that they occur within close proximity to each other, not just anywhere in a document
Describe the solution you'd like
It seems the Elasticsearch intervals query might fit the bill. So I think the technical implementation should not be too much of an issue, but how to reflect this different type of query of the UI requires some consideration.
Describe alternatives you've considered
We might also post-process documents which are matches to the simple query string query - but I'm not sure that would be a better solution, as the UI question remains.
Additional context
Add any other context or screenshots about the feature request here.

Does something like "firstterm secondterm"~5 OR "secondterm firstterm"~5 already do what is asked?

OR is redundant (and would be | in simple query string syntax), but yes, that would work :+1:
I was somewhat suprised because the query documentation on I-analyzer suggests that ~ for phrases has rather different semantics. It turns out that contradicts the elasticsearch manual.
That said, the query you formulate here isn't something I would expect a non-programmer to come up with. I would support making a more beginner-friendly option for this as part of #1436

I think the query I suggested will match secondterm apple banana cherry date elderberry firstterm, while a simplified version without the OR and the second branch would not. Other than that, I agree those two queries would be equivalent.

Ah, to clarify: I meant you could leave out the disjunction operator, so "firstterm secondterm"~5 "secondterm firstterm"~5.


⚠ Bofrost.lu has degraded performance

In c802819, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1280 ms

Resolved: Bofrost.lu performance has improved in 851e047 after 6 minutes.


TLS support for kubeflow

Name of the component: kubeflow-v1.0
What would you like to be added:
TLS support for kubeflow ingress gateway
Why is this needed:
User wants to access Kubeflow using convinient HTTPS at https://kubeflow.${dns.domain}

Seldon endpoint is stil under http. Here is the reasoning:

Not clear if we should public endpoint at all... In this case we might want to move seldon under cluster-local-gateway
Perhaps implement Seldon as a separate component. And manage possible TLS or MTLS there

cluster-local-gateway  doesn't use TLS. We are doing nothing for Seldon yet. In the future we should move Seldon outside of Kubeflow component


few capitalization

Why:
Closes [issue link]

What's being changed:

Check off the following:

[ ] I have reviewed my changes in staging (look for the latest deployment event in your pull request's timeline, then click View deployment).
[ ] For content changes, I have completed the self-review checklist.

Writer impact (This section is for GitHub staff members only):

[ ] This pull request impacts the contribution experience

[ ] I have added the 'writer impact' label
[ ] I have added a description and/or a video demo of the changes below (e.g. a "before and after video")

@BibhushanKarki
This seems to be opened accidentally, I am going to close it now :yellow_heart: Take a look at the contributing.md to learn how to contribute in this repo. You can open a discussion if you have any questions :sparkles:


Changes to be able to modify HBase master JVM heap size

Currently HBase master JVM runs with the default heap size of Java. This PR is to enable users to be able to modify the heap size of the master JVM process.

LGTM


conditional resources result in errors

I'm getting errors when using this simple config:
module "mig1" {
  source            = "GoogleCloudPlatform/managed-instance-group/google"
  region            = "europe-west2"
  zone              = "europe-west2-b"
  name              = "group1"
  size              = 2
  service_port      = 80
  service_port_name = "http"
}

Errors:
* module.mig1.output.region_depends_id: Resource 'null_resource.region_dummy_dependency' not found for variable 'null_resource.region_dummy_dependency.id'
* module.mig1.output.region_instance_group: Resource 'google_compute_region_instance_group_manager.default' not found for variable 'google_compute_region_instance_group_manager.default.instance_group'
* module.mig1.output.region_instances: Resource 'data.google_compute_instance_group.regional' not found for variable 'data.google_compute_instance_group.regional.instances'

I believe it's because of the 'zonal' variable resulting in a count of 0 for resources that are always in the output.
Terraform v0.11.0
+ provider.google v1.2.0
+ provider.null v1.0.0

@antimius I update the module to support the breaking changes in v0.11.0, in master and tagged as 1.1.6. Please re-test after updating the modules terraform get -update=true

@danisla This works, thank you. I see it's in the upgrade guide for 0.11 (https://www.terraform.io/upgrade-guides/0-11.html) and wow, that's a bad breaking change. I also had to do 'terraform init -upgrade' to get the latest google provider plugin for the 'network_ip' to 'address' change.


Operator '=' cannot be applied to operands of type 'Option' and 'Enum' (AL0175)

Is your feature request related to a problem? Please describe.
Not sure about what's supported / what's in the pipeline for enum conversion in the next release...
Our C/AL code is currently full of integer <> option conversions. To support the move to AL, we're gradually integrating enum properties on our table fields in C/AL. When converting the C/AL code to AL, AL0122 error shows up:

Cannot implicitly convert type 'Integer' to 'Enum NVTStatus'

procedure SetStatusType(piStatusType: integer) // Cast integer to ENUM = NOK
    begin
        bolFinancialStatus := (piStatusType = Type::Financial);

Since we're on C/AL, we're trying to fix the error in C/AL first. To avoid this error, we changed the integer parameters in to options. However, now we're running into AL0175

Operator '=' cannot be applied to operands of type 'Option' and 'Enum' (AL0175)

procedure SetStatusType(piStatusType: option) // Integer > Option
    begin
        bolFinancialStatus := (piStatusType = Type::Financial); // Compare option to ENUM = NOK

Seems like implicit conversion from Option to Enum is not supported.
Although when changing the code to first cast the option to an enum, no errors show up...
procedure SetStatusType(piStatusType: option)
    var
        MyEnum: enum NVTStatus;
    begin
        MyEnum := piStatusType; // Cast option to ENUM = OK
        bolFinancialStatus := (MyEnum = Type::Financial);

Describe the solution you'd like
Please provide support for enum <> option conversion.
Please provide better documentation on what conversion are / will be supported using enums. (Taking into account the fact that partners are still on C/AL and are preparing for AL conversion in C/AL)

We have lifted the restriction for converting/comparing enums for the spring release. We expect to long term add again to improve code-quality, but we need everything to be on AL first.
This should work if you work the bcinsider.azurecr.io/bcsandbox-master docker images.


Update appium-android-driver and adb creation

@imurchie Perhaps we could also address https://github.com/appium/appium/issues/10627 since we bumped the major version of android driver


Typo and Question Re:  MS Learning Pathways

Category

[ ] Question
[ X] Typo
[ ] Bug
[ ] Additional article idea
In the Getting Started, Your Digital Day Playlist, Collaborating with Your Colleagues, there is an extra bullet under Tools.

Thank you for you feedback. We'll correct the typo.


Heatmap error

Hi,
When I am using the heatmap plot, I am obtaining this error: The hclust order only supports 2D arrays right now!.
Could you help me? If I select the output shap values for first class for example for have 2D array I get the error:'numpy.ndarray' object has no attribute 'values'..
Could you help me please??
Thanks!
Pablo

Please provide a reproducible example. Will close the issue for now


LightGBM classifier training keeps failed, spark executor exit with code 134 due to unrecognized issue

SynapseML version
0.10.0
System information

Language version (e.g. python 3.8, scala 2.12): python3
Spark Version (e.g. 3.2.2): 3.0
Spark Platform (e.g. Synapse, Databricks): GCP data proc

Describe the problem
Hi team. My LightGBM classifier training job keeps failing due to an unrecognized issue. After checking the full log of yarn application, I can only find the following related message without clear expression on what exactly the error is. Could you please provide me more infos on how to solve this? My dataset is about 4.5 million records with 700 features. And I used 4 GCP n1-highmem-8, with spark executor=4 and executor cores=8 each.
Code to reproduce issue
 train_df = spark.read.parquet(data_path)
 train_df = train_df.select(columns_map['target']+columns_map['feature'])
 train_df = train_df.rdd.map(lambda x: (x[0], Vectors.dense(x[1:]))).toDF(["label", "features"])
 train_df = train_df.withColumn('label', train_df['label'].cast('int'))
 model = LGBClassifierOnSpark(**training_params)
 model.fit(train_df)

Other info / logs
2022-08-16T15:34:26Z][WARNING] Requesting driver to remove executor 2 for reason Container from a bad node: container_e01_1660602949467_0002_01_000004 on host: maglev-aml-cluster-63fdf8-w-1. Exit status: 134. Diagnostics: adoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000004/__app__.jar > /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000004/stdout 2> /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000004/stderr
[2022-08-16T15:34:26Z][WARNING] Lost task 20.0 in stage 3.0 (TID 23) (maglev-aml-cluster-63fdf8-w-2 executor 1): ExecutorLostFailure (executor 1 exited caused by one of the running tasks) Reason: Container from a bad node: container_e01_1660602949467_0002_01_000003 on host: maglev-aml-cluster-63fdf8-w-2. Exit status: 134. Diagnostics: adoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000003/__app__.jar > /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000003/stdout 2> /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000003/stderr
[2022-08-16T15:34:26Z][ERROR] Lost executor 2 on maglev-aml-cluster-63fdf8-w-1: Container from a bad node: container_e01_1660602949467_0002_01_000004 on host: maglev-aml-cluster-63fdf8-w-1. Exit status: 134. Diagnostics: adoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000004/__app__.jar > /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000004/stdout 2> /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000004/stderr
[2022-08-16T15:34:26Z][WARNING] Requesting driver to remove executor 1 for reason Container from a bad node: container_e01_1660602949467_0002_01_000003 on host: maglev-aml-cluster-63fdf8-w-2. Exit status: 134. Diagnostics: adoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000003/__app__.jar > /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000003/stdout 2> /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000003/stderr
[2022-08-16T15:34:26Z][ERROR] Lost executor 1 on maglev-aml-cluster-63fdf8-w-2: Container from a bad node: container_e01_1660602949467_0002_01_000003 on host: maglev-aml-cluster-63fdf8-w-2. Exit status: 134. Diagnostics: adoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000003/__app__.jar > /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000003/stdout 2> /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000003/stderr
[2022-08-16T15:34:26Z][WARNING] Container from a bad node: container_e01_1660602949467_0002_01_000004 on host: maglev-aml-cluster-63fdf8-w-1. Exit status: 134. Diagnostics: adoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000004/__app__.jar > /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000004/stdout 2> /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000004/stderr
[2022-08-16T15:34:26Z][WARNING] Lost task 10.0 in stage 3.0 (TID 13) (maglev-aml-cluster-63fdf8-w-1 executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Container from a bad node: container_e01_1660602949467_0002_01_000004 on host: maglev-aml-cluster-63fdf8-w-1. Exit status: 134. Diagnostics: adoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000004/__app__.jar > /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000004/stdout 2> /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000004/stderr
[2022-08-16T15:34:26Z][WARNING] Container from a bad node: container_e01_1660602949467_0002_01_000003 on host: maglev-aml-cluster-63fdf8-w-2. Exit status: 134. Diagnostics: adoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000003/__app__.jar > /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000003/stdout 2> /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000003/stderr
[2022-08-16T15:34:27Z][WARNING] Requesting driver to remove executor 3 for reason Container from a bad node: container_e01_1660602949467_0002_01_000005 on host: maglev-aml-cluster-63fdf8-w-0. Exit status: 134. Diagnostics: adoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000005/__app__.jar > /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000005/stdout 2> /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000005/stderr
[2022-08-16T15:34:27Z][WARNING] Exception in connection from /<IP_ADDRESS>:46852
[2022-08-16T15:34:27Z][ERROR] Lost executor 3 on maglev-aml-cluster-63fdf8-w-0: Container from a bad node: container_e01_1660602949467_0002_01_000005 on host: maglev-aml-cluster-63fdf8-w-0. Exit status: 134. Diagnostics: adoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000005/__app__.jar > /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000005/stdout 2> /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000005/stderr
[2022-08-16T15:34:27Z][WARNING] Lost task 10.1 in stage 3.0 (TID 27) (maglev-aml-cluster-63fdf8-w-0 executor 3): ExecutorLostFailure (executor 3 exited caused by one of the running tasks) Reason: Container from a bad node: container_e01_1660602949467_0002_01_000005 on host: maglev-aml-cluster-63fdf8-w-0. Exit status: 134. Diagnostics: adoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000005/__app__.jar > /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000005/stdout 2> /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000005/stderr
[2022-08-16T15:34:27Z][WARNING] Lost task 21.0 in stage 3.0 (TID 24) (maglev-aml-cluster-63fdf8-w-0 executor 3): ExecutorLostFailure (executor 3 exited caused by one of the running tasks) Reason: Container from a bad node: container_e01_1660602949467_0002_01_000005 on host: maglev-aml-cluster-63fdf8-w-0. Exit status: 134. Diagnostics: adoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000005/__app__.jar > /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000005/stdout 2> /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000005/stderr
[2022-08-16T15:34:27Z][WARNING] Container from a bad node: container_e01_1660602949467_0002_01_000005 on host: maglev-aml-cluster-63fdf8-w-0. Exit status: 134. Diagnostics: adoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000005/__app__.jar > /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000005/stdout 2> /var/log/hadoop-yarn/userlogs/application_1660602949467_0002/container_e01_1660602949467_0002_01_000005/stderr

[2022-08-16T22:34:26.393079Z][DEFAULT] #
[2022-08-16T22:34:26.393833Z][DEFAULT] # A fatal error has been detected by the Java Runtime Environment:
[2022-08-16T22:34:26.394238Z][DEFAULT] #
[2022-08-16T22:34:26.394513Z][DEFAULT] #  SIGSEGV (0xb) at pc=0x00007fd20ee83a24, pid=5797, tid=0x00007fd1af5c4700
[2022-08-16T22:34:26.394818Z][DEFAULT] #
[2022-08-16T22:34:26.39509Z][DEFAULT] # JRE version: OpenJDK Runtime Environment (8.0_332-b09) (build 1.8.0_332-b09)
[2022-08-16T22:34:26.395351Z][DEFAULT] # Java VM: OpenJDK 64-Bit Server VM (25.332-b09 mixed mode linux-amd64 compressed oops)
[2022-08-16T22:34:26.395576Z][DEFAULT] # Problematic frame:
[2022-08-16T22:34:26.395852Z][DEFAULT] # C  [libc.so.6+0x18ea24]
[2022-08-16T22:34:26.396078Z][DEFAULT] #
[2022-08-16T22:34:26.396328Z][DEFAULT] # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
[2022-08-16T22:34:26.396549Z][DEFAULT] #
[2022-08-16T22:34:26.396804Z][DEFAULT] # An error report file with more information is saved as:
[2022-08-16T22:34:26.397047Z][DEFAULT] # /mnt/1/hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000004/hs_err_pid5797.log
[2022-08-16T22:34:26.397258Z][DEFAULT] #
[2022-08-16T22:34:26.397499Z][DEFAULT] # If you would like to submit a bug report, please visit:
[2022-08-16T22:34:26.397731Z][DEFAULT] #   https://github.com/adoptium/adoptium-support/issues
[2022-08-16T22:34:26.397973Z][DEFAULT] # The crash happened outside the Java Virtual Machine in native code.
[2022-08-16T22:34:26.398219Z][DEFAULT] # See problematic frame for where to report the bug.
[2022-08-16T22:34:26.398437Z][DEFAULT] #
[2022-08-16T22:34:26.423668Z][DEFAULT] #
[2022-08-16T22:34:26.424387Z][DEFAULT] # A fatal error has been detected by the Java Runtime Environment:
[2022-08-16T22:34:26.424697Z][DEFAULT] #
[2022-08-16T22:34:26.424947Z][DEFAULT] #  SIGSEGV (0xb) at pc=0x00007f1c2ea55a24, pid=5883, tid=0x00007f1bcf98a700
[2022-08-16T22:34:26.425167Z][DEFAULT] #
[2022-08-16T22:34:26.425436Z][DEFAULT] # JRE version: OpenJDK Runtime Environment (8.0_332-b09) (build 1.8.0_332-b09)
[2022-08-16T22:34:26.42571Z][DEFAULT] # Java VM: OpenJDK 64-Bit Server VM (25.332-b09 mixed mode linux-amd64 compressed oops)
[2022-08-16T22:34:26.425954Z][DEFAULT] # Problematic frame:
[2022-08-16T22:34:26.426168Z][DEFAULT] # C  [libc.so.6+0x18ea24]
[2022-08-16T22:34:26.426393Z][DEFAULT] #
[2022-08-16T22:34:26.426633Z][DEFAULT] # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
[2022-08-16T22:34:26.426847Z][DEFAULT] #
[2022-08-16T22:34:26.427077Z][DEFAULT] # An error report file with more information is saved as:
[2022-08-16T22:34:26.427326Z][DEFAULT] # /mnt/1/hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000003/hs_err_pid5883.log
[2022-08-16T22:34:26.427542Z][DEFAULT] #
[2022-08-16T22:34:26.42778Z][DEFAULT] # If you would like to submit a bug report, please visit:
[2022-08-16T22:34:26.428Z][DEFAULT] #   https://github.com/adoptium/adoptium-support/issues
[2022-08-16T22:34:26.428229Z][DEFAULT] # The crash happened outside the Java Virtual Machine in native code.
[2022-08-16T22:34:26.428479Z][DEFAULT] # See problematic frame for where to report the bug.
[2022-08-16T22:34:26.428696Z][DEFAULT] #

[2022-08-16T22:34:49.471098Z][DEFAULT] Traceback (most recent call last):  File "task.py", line 188, in <module>    run_on_spark(task_func, task_func_kwargs, logger, log_level)  File "task.py", line 116, in run_on_spark    task_func(**task_func_kwargs)  File "<string>", line 29, in lightgbm_spark_training  File "/opt/conda/default/lib/python3.8/site-packages/aml/modelhub/lgbm_on_spark.py", line 63, in fit
[2022-08-16T22:34:49.471771Z][DEFAULT]     self.gbm = super(LGBClassifierOnSpark, self).fit(dataset, params=params)
[2022-08-16T22:34:49.472111Z][DEFAULT]   File "/mnt/1/hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000001/pyspark.zip/pyspark/ml/base.py", line 161, in fit
[2022-08-16T22:34:49.472388Z][DEFAULT]   File "/opt/conda/default/lib/python3.8/site-packages/synapse/ml/lightgbm/LightGBMClassifier.py", line 2017, in _fit
[2022-08-16T22:34:49.47265Z][DEFAULT]     java_model = self._fit_java(dataset)
[2022-08-16T22:34:49.472892Z][DEFAULT]   File "/mnt/1/hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000001/pyspark.zip/pyspark/ml/wrapper.py", line 332, in _fit_java
[2022-08-16T22:34:49.473129Z][DEFAULT]   File "/mnt/1/hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000001/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1304, in __call__
[2022-08-16T22:34:49.473359Z][DEFAULT]   File "/mnt/1/hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000001/pyspark.zip/pyspark/sql/utils.py", line 111, in deco
[2022-08-16T22:34:49.473608Z][DEFAULT]   File "/mnt/1/hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1660602949467_0002/container_e01_1660602949467_0002_01_000001/py4j-0.10.9-src.zip/py4j/protocol.py", line 326, in get_return_value
[2022-08-16T22:34:49.473919Z][DEFAULT] An error occurred while calling o770.fit.
[2022-08-16T22:34:49.474733Z][DEFAULT] : org.apache.spark.SparkException: Job aborted due to stage failure: Task 10 in stage 3.0 failed 4 times, most recent failure: Lost task 10.3 in stage 3.0 (TID 31) (maglev-aml-cluster-63fdf8-w-2 executor 6): java.net.ConnectException: Connection refused (Connection refused)	at java.net.PlainSocketImpl.socketConnect(Native Method)	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)	at java.net.Socket.connect(Socket.java:607)	at java.net.Socket.connect(Socket.java:556)	at java.net.Socket.<init>(Socket.java:452)	at java.net.Socket.<init>(Socket.java:229)	at com.microsoft.azure.synapse.ml.lightgbm.NetworkManager$.getNetworkTopologyInfoFromDriver(NetworkManager.scala:129)	at com.microsoft.azure.synapse.ml.lightgbm.NetworkManager$.$anonfun$getGlobalNetworkInfo$2(NetworkManager.scala:116)	at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:24)	at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)	at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)	at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)	at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)	at com.microsoft.azure.synapse.ml.lightgbm.NetworkManager$.$anonfun$getGlobalNetworkInfo$1(NetworkManager.scala:111)	at com.microsoft.azure.synapse.ml.core.env.StreamUtilities$.using(StreamUtilities.scala:28)	at com.microsoft.azure.synapse.ml.lightgbm.NetworkManager$.getGlobalNetworkInfo(NetworkManager.scala:107)	at com.microsoft.azure.synapse.ml.lightgbm.BasePartitionTask.initialize(BasePartitionTask.scala:179)	at com.microsoft.azure.synapse.ml.lightgbm.BasePartitionTask.mapPartitionTask(BasePartitionTask.scala:114)	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$executePartitionTasks$1(LightGBMBase.scala:589)	at org.apache.spark.sql.execution.MapPartitionsExec.$anonfun$doExecute$3(objects.scala:195)	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)	at org.apache.spark.scheduler.Task.run(Task.scala:131)	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:498)	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:501)	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)	at java.lang.Thread.run(Thread.java:750)
[2022-08-16T22:34:49.475027Z][DEFAULT] None
[2022-08-16T22:34:49.475263Z][DEFAULT] Driver stacktrace:
[2022-08-16T22:34:49.475493Z][DEFAULT] 	at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2304)
[2022-08-16T22:34:49.475729Z][DEFAULT] 	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2253)
[2022-08-16T22:34:49.475955Z][DEFAULT] 	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2252)
[2022-08-16T22:34:49.476171Z][DEFAULT] 	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
[2022-08-16T22:34:49.476416Z][DEFAULT] 	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
[2022-08-16T22:34:49.476672Z][DEFAULT] 	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
[2022-08-16T22:34:49.476894Z][DEFAULT] 	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2252)
[2022-08-16T22:34:49.477118Z][DEFAULT] 	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1124)
[2022-08-16T22:34:49.477364Z][DEFAULT] 	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1124)
[2022-08-16T22:34:49.477597Z][DEFAULT] 	at scala.Option.foreach(Option.scala:407)
[2022-08-16T22:34:49.47782Z][DEFAULT] 	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1124)
[2022-08-16T22:34:49.478043Z][DEFAULT] 	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2491)
[2022-08-16T22:34:49.478261Z][DEFAULT] 	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2433)
[2022-08-16T22:34:49.47848Z][DEFAULT] 	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2422)
[2022-08-16T22:34:49.478716Z][DEFAULT] 	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
[2022-08-16T22:34:49.478939Z][DEFAULT] 	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:902)
[2022-08-16T22:34:49.479153Z][DEFAULT] 	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2204)
[2022-08-16T22:34:49.479372Z][DEFAULT] 	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2225)
[2022-08-16T22:34:49.479603Z][DEFAULT] 	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2244)
[2022-08-16T22:34:49.479838Z][DEFAULT] 	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2269)
[2022-08-16T22:34:49.480063Z][DEFAULT] 	at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1030)
[2022-08-16T22:34:49.480284Z][DEFAULT] 	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
[2022-08-16T22:34:49.480545Z][DEFAULT] 	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
[2022-08-16T22:34:49.480776Z][DEFAULT] 	at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
[2022-08-16T22:34:49.480995Z][DEFAULT] 	at org.apache.spark.rdd.RDD.collect(RDD.scala:1029)
[2022-08-16T22:34:49.481216Z][DEFAULT] 	at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:390)
[2022-08-16T22:34:49.481436Z][DEFAULT] 	at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3709)
[2022-08-16T22:34:49.481672Z][DEFAULT] 	at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2978)
[2022-08-16T22:34:49.481944Z][DEFAULT] 	at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3700)
[2022-08-16T22:34:49.482195Z][DEFAULT] 	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
[2022-08-16T22:34:49.482432Z][DEFAULT] 	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
[2022-08-16T22:34:49.482675Z][DEFAULT] 	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
[2022-08-16T22:34:49.482895Z][DEFAULT] 	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
[2022-08-16T22:34:49.483132Z][DEFAULT] 	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
[2022-08-16T22:34:49.483351Z][DEFAULT] 	at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3698)
[2022-08-16T22:34:49.483576Z][DEFAULT] 	at org.apache.spark.sql.Dataset.collect(Dataset.scala:2978)
[2022-08-16T22:34:49.4838Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.executePartitionTasks(LightGBMBase.scala:597)
[2022-08-16T22:34:49.484127Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.executePartitionTasks$(LightGBMBase.scala:583)
[2022-08-16T22:34:49.484415Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.executePartitionTasks(LightGBMClassifier.scala:27)
[2022-08-16T22:34:49.48467Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.executeTraining(LightGBMBase.scala:573)
[2022-08-16T22:34:49.484898Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.executeTraining$(LightGBMBase.scala:545)
[2022-08-16T22:34:49.485133Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.executeTraining(LightGBMClassifier.scala:27)
[2022-08-16T22:34:49.485359Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.trainOneDataBatch(LightGBMBase.scala:435)
[2022-08-16T22:34:49.485591Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.trainOneDataBatch$(LightGBMBase.scala:392)
[2022-08-16T22:34:49.485819Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.trainOneDataBatch(LightGBMClassifier.scala:27)
[2022-08-16T22:34:49.486039Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$train$2(LightGBMBase.scala:61)
[2022-08-16T22:34:49.486258Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.logging.BasicLogging.logVerb(BasicLogging.scala:62)
[2022-08-16T22:34:49.48648Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.logging.BasicLogging.logVerb$(BasicLogging.scala:59)
[2022-08-16T22:34:49.486719Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.logVerb(LightGBMClassifier.scala:27)
[2022-08-16T22:34:49.486938Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.logging.BasicLogging.logTrain(BasicLogging.scala:48)
[2022-08-16T22:34:49.487158Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.logging.BasicLogging.logTrain$(BasicLogging.scala:47)
[2022-08-16T22:34:49.487396Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.logTrain(LightGBMClassifier.scala:27)
[2022-08-16T22:34:49.487631Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.train(LightGBMBase.scala:42)
[2022-08-16T22:34:49.487851Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.train$(LightGBMBase.scala:35)
[2022-08-16T22:34:49.48807Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.train(LightGBMClassifier.scala:27)
[2022-08-16T22:34:49.48829Z][DEFAULT] 	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.train(LightGBMClassifier.scala:27)
[2022-08-16T22:34:49.488547Z][DEFAULT] 	at org.apache.spark.ml.Predictor.fit(Predictor.scala:151)
[2022-08-16T22:34:49.488769Z][DEFAULT] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[2022-08-16T22:34:49.488988Z][DEFAULT] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[2022-08-16T22:34:49.489207Z][DEFAULT] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2022-08-16T22:34:49.48949Z][DEFAULT] 	at java.lang.reflect.Method.invoke(Method.java:498)
[2022-08-16T22:34:49.489738Z][DEFAULT] 	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
[2022-08-16T22:34:49.489974Z][DEFAULT] 	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
[2022-08-16T22:34:49.490194Z][DEFAULT] 	at py4j.Gateway.invoke(Gateway.java:282)
[2022-08-16T22:34:49.490411Z][DEFAULT] 	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
[2022-08-16T22:34:49.490643Z][DEFAULT] 	at py4j.commands.CallCommand.execute(CallCommand.java:79)
[2022-08-16T22:34:49.49086Z][DEFAULT] 	at py4j.GatewayConnection.run(GatewayConnection.java:238)
[2022-08-16T22:34:49.491076Z][DEFAULT] 	at java.lang.Thread.run(Thread.java:750)
[2022-08-16T22:34:49.491759Z][DEFAULT] Caused by: java.net.ConnectException: Connection refused (Connection refused)	at java.net.PlainSocketImpl.socketConnect(Native Method)	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)	at java.net.Socket.connect(Socket.java:607)	at java.net.Socket.connect(Socket.java:556)	at java.net.Socket.<init>(Socket.java:452)	at java.net.Socket.<init>(Socket.java:229)	at com.microsoft.azure.synapse.ml.lightgbm.NetworkManager$.getNetworkTopologyInfoFromDriver(NetworkManager.scala:129)	at com.microsoft.azure.synapse.ml.lightgbm.NetworkManager$.$anonfun$getGlobalNetworkInfo$2(NetworkManager.scala:116)	at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:24)	at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)	at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)	at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)	at com.microsoft.azure.synapse.ml.core.utils.FaultToleranceUtils$.retryWithTimeout(FaultToleranceUtils.scala:29)	at com.microsoft.azure.synapse.ml.lightgbm.NetworkManager$.$anonfun$getGlobalNetworkInfo$1(NetworkManager.scala:111)	at com.microsoft.azure.synapse.ml.core.env.StreamUtilities$.using(StreamUtilities.scala:28)	at com.microsoft.azure.synapse.ml.lightgbm.NetworkManager$.getGlobalNetworkInfo(NetworkManager.scala:107)	at com.microsoft.azure.synapse.ml.lightgbm.BasePartitionTask.initialize(BasePartitionTask.scala:179)	at com.microsoft.azure.synapse.ml.lightgbm.BasePartitionTask.mapPartitionTask(BasePartitionTask.scala:114)	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$executePartitionTasks$1(LightGBMBase.scala:589)	at org.apache.spark.sql.execution.MapPartitionsExec.$anonfun$doExecute$3(objects.scala:195)	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)	at org.apache.spark.scheduler.Task.run(Task.scala:131)	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:498)	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:501)	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)	... 1 more

What component(s) does this bug affect?

[ ] area/cognitive: Cognitive project
[ ] area/core: Core project
[ ] area/deep-learning: DeepLearning project
[X] area/lightgbm: Lightgbm project
[ ] area/opencv: Opencv project
[ ] area/vw: VW project
[ ] area/website: Website
[ ] area/build: Project build system
[ ] area/notebooks: Samples under notebooks folder
[ ] area/docker: Docker usage
[ ] area/models: models related issue

What language(s) does this bug affect?

[ ] language/scala: Scala source code
[X] language/python: Pyspark APIs
[ ] language/r: R APIs
[ ] language/csharp: .NET APIs
[ ] language/new: Proposals for new client languages

What integration(s) does this bug affect?

[X] integrations/synapse: Azure Synapse integrations
[ ] integrations/azureml: Azure ML integrations
[ ] integrations/databricks: Databricks integrations

AB#1936447

@svotaw can you please take a look at this LightGBM native-layer error?

apologies for the delay, I didn't get a tag notification. There's nothing that I can see from the logs that indicates what the problem is.  It's possible it's just a OOM error that crashes the JVM in the native layer.  Can you try with fewer number of rows?  We are working on a new smaller memory footprint for training, but it's not yet available.

Hi @svotaw  really appreciated for your reply. Actually those are all the relative log that I can get from my executors lol. I'm not sure about whether it's an OOM cuz our spark cluster and config is large enough for this dataset. We've also done some experiments to use this cluster to run a much large data size which works fine. In addition I checked that error code 137 indicates the OOM issue while 134 means 'bad container', which we are not sure about the exact meaning.

We also encountered this problem.
our dataset is 60G
executors: 128
memory: 16G
executor cores: 4
Is there any good solution now?

We are close to having a new execution engine, just waiting on a bug fix review from LightGBM.  I can update this thread when it is available.  With the error as is, there's not much I can diagnose about a crash in the native linux layer.

okay ,thanks for your reply

hello, @svotaw I solved the problem with this parameter useBarrierExecutionMode=True .  But it confuses me even more.

I looked at where the barrier works and can see why it can wait until all nodes succeed and get the correct topology, but I don't understand why it would be wrong without the barrier. can you help me with this question？
These questions are related,748,594,743， but I have never found the exact reason.

I can't see why it would be different either just looking at the code, although @imatiach-msft might have more context for the errors he was seeing without it and why he added it. I have mostly worked on new code soon to be released which does not use barrier execution mode.
Have you tried the latest release?  I know there were some fixes that went into 0.10.1 and after.

The version we are using now is the latest version. 0.10.1 . and thx for your reply.
hello, @imatiach-msft ，Can you help me why this problem occurs？

I think I found the root cause code of this error, @svotaw @imatiach-msft . if the number of tasks requested is inconsistent with the number of tasks allocated. There is currently an infinite loop of logic without the Barrier model.
Therefore, If not use Barrier Execution Mode, this error will occurs frequently when the cluster resources are tight, and disappears when the resources are sufficient.
Thanks again for your reply.

Maybe there can have a strategy here to skip the check logic   and set numTasks to the number of workers obtained.:
strategy:  when a certain ratio of workers are connected & Wait 5 minutes and still no new worker joins
Then let the developer can set this ratio.  This will allow users to run normally outside of Barrier mode.

Sorry for reply so late. In fact I am not sure why  numTasks not match the actual number of Tasks.While the driver is waiting for accept, will a task failure cause this problem? i don't know. and I tried several times, but it didn't reproduce(This may be because I reduced the number of taskNumbers to 128 when retrying). my taskNumber is not automatically assigned, it is set by param.
the config encounters the problem before  :
dataset ： 60G
executors: 128
memory: 16G
executor cores: 4
numTasks: 512

Even when my numTasks number is 512, it runs successfully most of the case.

I think I reproduced the problem. without Barrier Execution Mode.
dataset ： 40G
executors: 64
memory: 16G
executor cores: 2
numTasks: 128
I have checked all the 128 tasks log . they all end with : 22/11/21 13:14:55 INFO BulkPartitionTask: task 12488 sending status message to driver: ignore:xxxxxxx:12459:75:29  .

As can be seen from the driverlog, the driver has been waiting for a node A, driver log:
I counted all the information connected to the node on the driver, and found that node A is indeed missing.

node A log: But In fact, this node has already sent the connection information.

So a deadlock occurs.

The abnormal executor ID i 65.And It can be seen from the job graph that it be caused by a node 14  removed in the middle and restarting a new node 65. And the new node cannot connect to driver.

@svotaw  @imatiach-msft

thanks so much for this great investigation!  I will take a look in more detail soon at what you sent.

I started looking at it, and basically this is just a limitation of using LightGBM in a Spark context. We use LightGBM in "distributed" mode, which requires all nodes to be working at the same time to be processing the data (i.e., no time parallelism). Spark, however, is designed to be resilient, and just restart any executors that it wants to (due to error, unresponsiveness, scaling, etc.). So if Spark decides to restart an executor once we've already started, it will basically break LightGBM. Our SynapseML wrapper also works this way since it integrates with Spark.
However, I'm going to investigate this and see if there's anything we (SyanpseML) can do to mitigate when this happens, particularly at the preparation stage. Maybe we can recover if we haven't reached the training stage yet.
I will work on it as I can, but we just got our new execution engine code checked into the native LightGBM side today, and will be working to release it through SyanpseML. It saves an order magnitude of memory, so we're hoping that it will improve LightGBM excecution dramatically in SynapseML. As I'm free on the side, I will see if we can't also improve recovery for the scenario you posted above.

okay. Thanks for your detailed reply.

Also - where in the source code would you recommend I look at to also brainstorm a solution for this in the meantime? I would love to help contribute to this project

A solution for dynamic scaling in LightGBM?  This would be LightGBM code, so microsoft/LightGBM. We (SynapseML) just generally recommend to turn off dynamic executor scaling for LightGBM usage.  I will be looking into a shorter term solution in the next couple of weeks to at least see if we can fail fast when network initialization fails.


Hint for mudskipper point easy emote clue is missing

Wave on mudskipper point easy clue doesn't show up after teleporting to it
Get clue, see required stuff, go to place, don't see required stuff.
Happened twice today. Thought first time was a fluke.
Not the second time.
Dropping the clue, tping away and back, combinations of things like that, don't work.

Looks like the map icons get cropped even without menu widget being actually visible https://gyazo.com/ee99649ab8ddb060f5207ef21b1fcbaf. @Nightfirecat maybe you know why?

Hm... Will look into this tomorrow, that definitely shouldn't be happening.

I'm unable to reproduce this behavior... No matter what I click on, or how I resize the map (in fixed and resizable layouts), I am not able to get this same behavior.

To anyone who may also have this issue--does the issue persist after restarting your client?

I'll test that when I get the step again

You should be able to test this without a clue step by seeing if our custom world map icons get clipped.

Oh?
But it didn't show the "wave" emote outline either.
I should've taken screenshots.

Updated the issue title as this looks like it is related to different issue than we originally thought, e.g simply by missing clue step.

Please Close - This regression happened as a result of #5538 that went live with the 1.4.18 release on 9/20/18, and was then reverted by #5664 on 9/27/18 for 1.4.19


[FEATURE REQUEST] Implement Dark and Light Mode Toggle

Description
Is your feature request related to a problem? Please describe.
Many users prefer to have control over the appearance of their applications, including the ability to switch between dark and light modes for better readability and reduced eye strain. Currently, the application lacks this feature, limiting user customization options.
Describe the solution you'd like
Integrate a dark and light mode toggle within the application's interface, allowing users to switch between modes seamlessly. When enabled, the dark mode should adjust the color scheme to darker tones, enhancing readability in low-light environments.
Describe alternatives you've considered
An alternative solution could involve implementing a theme selector with predefined themes, including dark and light modes. However, a toggle switch provides a more intuitive and straightforward user experience.
Additional Information
Context
The addition of a dark and light mode toggle aligns with current design trends and user expectations for customizable application interfaces. Offering this feature enhances user satisfaction and usability.
Potential Implementation (optional)
The toggle switch can be placed in the application's settings menu or as a prominent button within the interface for easy access. Utilizing CSS variables and media queries can facilitate smooth transitions between dark and light modes.
Screenshots or Mockups (optional)
If applicable, include any screenshots, mockups, or visual representations of the requested feature.
Priority (optional)
Medium
Additional Notes (optional)
Considering the increasing demand for customizable user experiences, implementing a dark and light mode toggle can differentiate the application and attract more users.

not required in website currently.


fix: 🐛 upgrade requirements, immediate-fix for type issues

Pending to fix jest issue

@omkar-at-bridgewise
Merge conflicts

@ali-at-bridgewise pls approve


Enable rest framework's browsable API

This pull request switches on the BrowsableAPIRenderer class. This renderer displays the endpoint in a nice HTML page which is enabled when an Accept: text/html header is sent by the client (which would only be true when browsing to it directly).

Rebased

It looks like this requires adding 'rest_framework' to INSTALLED_APPS, in order to pick up the templates - this isn't currently documented. Would it be possible to gracefully handle the case where this is missing from INSTALLED_APPS, and fall back on the old behaviour (perhaps with a deprecation warning)?

@gasman Yep, I'll do that. Cheers!

Changes made!

Merged in c4738fc3d5099796f1820fd200bb3e427417f7da


fix: Gremlin batch size calc

Feature or Bugfix

Bugfix

Detail

Insert exact batch size

Relates

#2494

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: c28948764d4a7c5ca9500dfa5329e64af11e492d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository


Added support for parsing of doubles and floats.

Also package-lock updated due to out-of-date version currently checked into master.
Minor tweak to existing tests to better cover the parsing of double and float.
Background:
After reviewing the existing very old PR that fixes doubles and floats thought best to create a new one with a slightly different approach
See: https://github.com/vpulim/node-soap/pull/985
I've gone for Number(text) instead of parseFloat(text) as I think it's safer to throw an error on unexpected characters rather than allowing an assumption.
E.g. "1a" !== 1
There is a good explanation of the pro's con's of Number() vs parseFloat() here: https://stackoverflow.com/questions/12227594/which-is-better-numberx-or-parsefloatx

Coverage increased (+0.004%) to 93.33% when pulling 5aa056e56cbabcae83969c7ce96e87438ba853c8 on alexb-uk:master into b0891343da75ab1b31d539812d5f3fb3a89ea15c on vpulim:master.

Coverage increased (+0.004%) to 93.33% when pulling 5aa056e56cbabcae83969c7ce96e87438ba853c8 on alexb-uk:master into b0891343da75ab1b31d539812d5f3fb3a89ea15c on vpulim:master.

please revert package-lock.json.  we'll update it.

please revert package-lock.json. we'll update it.

Done, thanks a lot.

I'm a little confused as to why Travis failed for Node v9 and v7, is that a known issue?

If it helps the build passes on my branch so guess it was an intermittent test timeout?
See: https://travis-ci.org/alexb-uk/node-soap/builds/531663881

thanks @alexb-uk !


About Commands

Which commands should I use to start running the puzzle solving, Thank you in Advance

Select a puzzle = write 66 (or any puzzle) and ENTER
Select a scan type = write R or SB or SE and ENTER
Enter threads = write 8 (or 1-32) and ENTER

can you write me an example please? thank you in advance

can you write me an example please? thank you in advance

Run the app. It will ask you to enter a puzzle number. Type a puzzle number you want and press ENTER.
The application will ask you to select the scan type. Type "R" and press ENTER to scan randomly.
The application will ask how many threads it wants to use. Depending on the number of threads in your CPU, enter a value between 1 and 16 and press ENTER.
The app will now start scanning.

I am closing issue. If you have any further questions, please do not hesitate and open a new issue.

Thank you


import images appears to be importing many 3rd party images

What happened:
Our customer is having an issue when using eks-a to import images into their registry.  The import process appears to be pushing many 3rd party images to the registry.  This makes it so anyone who imports the images needs to have permission to many registry namespaces, or else the process fails with a permission error.  But the list of required namespaces unknown, and could be large.
Sample error message:
Error: writing images to destination with image mover: image processor worker failed, rest of jobs were aborted: unauthorized: The client does not have permission for manifest: org.artifactory.exception.CancelException: mark_rogers3 does not have permissions to create modify or delete objects in namespace dockerv2-local:retailappsatx:retailappsatx

What you expected to happen:
Eks-a should only push images that are produced by the eks-a team.
How to reproduce it (as minimally and precisely as possible):
eksctl anywhere download images --output "$TARGET_ARTIFACTS_FILE" -v 10 2>&1 | tee "$TMPDIR/download_images_output.txt"

grep 'Reading bundles manifest' "$TMPDIR/download_images_output.txt" | awk '{print $7}' | sed 's/}//g' | xargs -I {} curl -L -s {} > "$TARGET_BUNDLE_MANIFEST_FILE"

REGISTRY_USERNAME="$REGISTRY_USERNAME" REGISTRY_PASSWORD="$REGISTRY_PASSWORD" eksctl anywhere import images --input "$TARGET_ARTIFACTS_FILE" --registry $REGISTRY_URL --bundles "$TARGET_BUNDLE_MANIFEST_FILE" -v 10

Anything else we need to know?:
Environment:

EKS Anywhere Release: v0.12.2
EKS Distro Release: 0.115.0

As an experiment, I ran the same script using a local docker registry.Then I looked at the

As an experiment, I ran the same download and import on a Linux machine in my own environment.  I used docker run -d -p 5000:5000 --name registry registry:2 to create a local registry.
At the end of the import, I didn't see any unfamiliar repositories at http://localhost:5000/v2/_catalog.  Here is the list of repositories in the registry.


Hashmap and user input configuration in bookstore program

When looking at the program, I decided to try and add a hash map for the books inputted. I also decided to add in user input so that whoever is operating the program, this being the customer, can input their username, password, email, and title of the book they want. I then added two more books to the map that best correlate to with purpose of me adding on to this program. I also added a publication date to the Books class simply for extra information.

I think you forgot to git add all the changes, but you didn't forget about a IntelliJ configuration file! 😅
This is all you committed: https://github.com/CodeByAidan/online-bookstore-system/commit/1861a9d35fda0a07fdad080b2d25cdce219681d9


Remove exclusion, use optional for spring-boot-starter-web

Exclude spring-boot-starter-tomcat causes errors.
See samples : https://github.com/speralta/spring-boot-ext-logback-access-sample

This error happened to us as well

@akihyro please feel free to ask me any questions about this PR

Sorry for the late reply.
Thanks for contributing!
Looks good to me.

v1.6 has been released!
http://search.maven.org/#artifactdetails|net.rakugakibox.springbootext|spring-boot-ext-logback-access|1.6|

Thanks @akihyro !


Connection not closed (as stated in version 1.0.9 - 2014/10/20)

Hi,
Thanks for the great work on this websocket library. I am using version 1.0.22 (I know this thank to the first lines in CHANGELOG.md and README.md). After a longtime running, I see that the server still keeps a connection that is supposed to have been dead (because I explicitly powered off the client). Reading my logs, I see that connection never sends close event to my code although the client was turned off.
Reviewing in CHANGELOG.md, I learnt that this problem has been fixed since version 1.0.9, released on 2014/10/20. Somehow this zombie connection issue is not yet completely fixed?
Please let me know:

Am I right to determine my current websocket version by looking at the first lines of CHANGELOG.md and README.md? I must clarify this important point as I am afraid I am using a version lower than 1.0.9.
Has anyone reported this zombie connection since version 1.0.9?
How do I fix this problem?

Thanks,

Am I right to determine my current websocket version by looking at the first lines of CHANGELOG.md and README.md? I must clarify this important point as I am afraid I am using a version lower than 1.0.9.

No, you should query the version property:
console.log(require('websocket').version)

Alternately, look at the version specified in package.json.

Has anyone reported this zombie connection since version 1.0.9?

Not that I'm aware of.

How do I fix this problem?

I don't know nearly enough about your problem, your code, or your environment to have any hope of answering this question. :-(
If possible please provide a specific test case that can reliably reproduce the behavior that you're seeing and then we can investigating. However, I suspect you are using the library incorrectly.

Hi Brian,
The version is 1.0.22. I verified by printing out:

console.log(require('websocket').version)

Below is my code outline:

"use strict"
const PORT = SOME_PORT
var http = require("http")
var server = http.createServer(function(req, resp) {
// Do nothing
}).listen(PORT, function() {
console.log((new Date()) + " Server is listening on port " + PORT)
})
var connections = []
var webSocketServer = require("websocket").server
var wsServer = new webSocketServer( {
httpServer: server
})
wsServer.on("request", function(request) {
var connection = request.accept(null, request.origin)
connection.on("message", function(message) {
   // Do some logic...

// If "connection" is not found in "connections", then connections.push(connection) // A
// else connection.close() // B
})

connection.on("close", function(code, description) {
    // find "connection" index in "connections", then connections.splice(index, 1)
})

})

After a longtime running in about a week, when memory reaches over 90MB, one of my client connections keeps disconnecting due to B. I know for sure the code falls into B because before closing connection, I send to the client some message (I did not write it out in the code snippet above for your ease of reading). Obviously, the code falls into B since connection still exists in connections (close event has not been emitted).
Note that, close event works quite well in very first times, but things will happen when it runs in a long time (or maybe memory gets bigger?).
Regards,

Try pushing your connection in the request handler, not the message handler.

No, I cannot push my connection in the request handler, which is too early. Before pushing to connections, I need to verify it with // Do some logic.... If I fail to verify, then I will not push it into the connections.

Hi!
I'm facing the problem, when connection is not removed from websocket.server connections array after connection.close() or connection.drop() were called.
I'm using "websocket": "1.0.35"
Server is created with only  autoAcceptConnections: false
I have only 'request' listener on server

Can you provide a code example demonstrating the problem?
Brian
On Sun, Aug 18, 2024 at 3:38 AM Valeria Kononenko @.***>
wrote:

Hi!
I'm facing the problem, when connection is not removed from
websocket.server connections array after connection.close() or
connection.drop() were called.
I'm using "websocket": "1.0.35"
Server is created with only autoAcceptConnections: false
I have only 'request' listener on server
—
Reply to this email directly, view it on GitHub
https://github.com/theturtle32/WebSocket-Node/issues/225#issuecomment-2295213659,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAM7QDX6KRRCVQGIQPZ233ZSB2RLAVCNFSM6AAAAABMWH3IBWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJVGIYTGNRVHE
.
You are receiving this because you commented.Message ID:
@.***>


Client rate books

Refactor on the show modals to allow book details to be passed, however will be necessary to allow comments to also be viewable on these pages


Error: Init:CreateContainerError when GCS Fuse CSI driver enabled with GKE Cluster on apache airflow deployment

Source
While deploying apache airflow deployment to the GKE cluster which is enabled gcs-fuse-csi-driver for Mount filesystem, getting
Init:CreateContainerError. while describe the pod, we were get error like Error: failed to reserve container name "check-db_sample-airflow-db-migrations-5d99cc5b86-bjqd2_sample_c8e8766c-b2e0-486c-8e1a-f2db0fe20745_0": name "check-db_sample-airflow-db-migrations-5d99cc5b86-bjqd2_sample_c8e8766c-b2e0-486c-8e1a-f2db0fe20745_0" is reserved for "5738e4d666b4c6872c316b752cac8633b901218f71b3f63c3176f365e5e86e9e".
Here I will attach the source code for apache airflow helm chart,
https://github.com/airflow-helm/charts/tree/main/charts/airflow
I will share the values.yaml for  db-migrations
values.yaml
########################################
## COMPONENT | db-migrations Deployment
########################################
dbMigrations:
## if the db-migrations Deployment/Job is created
## - [WARNING] if `false`, you have to MANUALLY run `airflow db upgrade` when required
##
enabled: true

## if a post-install helm Job should be used (instead of a Deployment)
## - [WARNING] setting `true` will NOT work with the helm `--wait` flag,
##   this is because post-install helm Jobs run AFTER the main resources become Ready,
##   which will cause a deadlock, as other resources require db-migrations to become Ready
##
runAsJob: false

## resource requests/limits for the db-migrations Pods
## - spec for ResourceRequirements:
##   https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#resourcerequirements-v1-core
##
resources: {}

## the nodeSelector configs for the db-migrations Pods
## - docs for nodeSelector:
##   https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector
##
nodeSelector: {}

## the affinity configs for the db-migrations Pods
## - spec for Affinity:
##   https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#affinity-v1-core
##
affinity: {}

## the toleration configs for the db-migrations Pods
## - spec for Toleration:
##   https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#toleration-v1-core
##
tolerations: []

## the security context for the db-migrations Pods
## - spec for PodSecurityContext:
##   https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#podsecuritycontext-v1-core
##
securityContext:
  fsGroup: 3003
  runAsGroup: 2002
  runAsUser: 1001

## Labels for the db-migrations Deployment
##
labels: {}

## Pod labels for the db-migrations Deployment
##
podLabels: {}

## annotations for the db-migrations Deployment/Job
##
annotations: {}

## Pod annotations for the db-migrations Deployment/Job
##
podAnnotations:
  gke-gcsfuse/volumes: "true"

## if we add the annotation: "cluster-autoscaler.kubernetes.io/safe-to-evict" = "true"
##
safeToEvict: false

## the number of seconds between checks for unapplied db migrations
## - only applies if `airflow.dbMigrations.runAsJob` is `false`
##
checkInterval: 300

Persistence Volume and Persistence Volume Claim Configuration with terraform file
 pv_pvc.tf
 resource "kubernetes_persistent_volume" "sample-airflow-logs-pv" {
 metadata {
     name = "sample-airflow-logs-pv"
 }
 spec {
   capacity = {
   storage = "150Gi"
 }
 access_modes = ["ReadWriteMany"]
 storage_class_name = "sample-eks-efs-sc"
 mount_options = [
  "implicit-dirs",
  "uid=1001",
  "gid=3003" ]
 persistent_volume_source {
  csi {
    driver = "gcsfuse.csi.storage.gke.io"
    volume_handle = "sample-airflow-logs-fs"
       }
     }
   }
 }
resource "kubernetes_persistent_volume_claim" "sample-airflow-logs-pvc" {
metadata {
   name      = "sample-airflow-logs"
   namespace = "sample"
}

 spec {
   access_modes       = ["ReadWriteMany"]
   storage_class_name = "sample-eks-efs-sc"
   volume_name = "sample-airflow-logs-pv"
   resources {
      requests = {
        storage = "150Gi"
     }
   }
 }
 depends_on = [
   kubernetes_persistent_volume.sample-airflow-logs-pv
 ]
}

Error:
Command : kubectl get pods -n dataworkz
Error:
sample-airflow-db-migrations-5d99cc5b86-9zsdt   0/2     Init:CreateContainerError   0          71m

sample-airflow-redis-master-0                   2/2     Running                     0          71m

sample-airflow-scheduler-7b4c5bcdff-tt246       0/2     Init:CreateContainerError   0          71m

sample-airflow-sync-users-79d7d4f477-9gzph      0/2     Init:CreateContainerError   0          71m

sample-airflow-web-57755c9659-9hwh4             0/2     Init:CreateContainerError   0          71m

sample-airflow-worker-0                         0/2     Init:CreateContainerError   0          71m

Commnad : kubectl describe pod sample-airflow-db-migrations-5d99cc5b86-bjqd2 -n dataworkz
Error from Event:
Events:
Type     Reason     Age                 From               Message

----     ------     ----                ----               -------

Normal   Scheduled  2m14s               default-scheduler  Successfully assigned dataworkz/sample-airflow-db-migrations-5d99cc5b86-bjqd2 to gke-sample-all-in-on-sample-default-p-4e593424-qtww

Warning  Failed     13s                 kubelet            Error: context deadline exceeded

Normal   Pulled     0s (x3 over 2m13s)  kubelet            Container image "[xxxxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/dataworkz/airflow:2.5.3](http://xxxxxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/dataworkz/airflow:2.5.3)" already present on machine

Warning  Failed     0s (x2 over 13s)    kubelet            Error: failed to reserve container name "check-db_sample-airflow-db-migrations-5d99cc5b86-bjqd2_sample_c8e8766c-b2e0-486c-8e1a-f2db0fe20745_0": name "check-db_sample-airflow-db-migrations-5d99cc5b86-bjqd2_sample_c8e8766c-b2e0-486c-8e1a-f2db0fe20745_0" is reserved for "5738e4d666b4c6872c316b752cac8633b901218f71b3f63c3176f365e5e86e9e"

Observation:
As per the official document for gcs-fuse-csi-driver, we followed below steps
https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver

Step1: Added pod annotations to the deployment values.yaml
podAnnotations:
gke-gcsfuse/volumes: "true"

Step2: Added security context to values.yaml
securityContext:
fsGroup: 3003
runAsGroup: 2002
runAsUser: 1001

Step3: Tried more pod annotations to the values.yaml
podAnnotations:
gke-gcsfuse/volumes: "true"
gke-gcsfuse/cpu-limit: "2048m"
gke-gcsfuse/memory-limit: "2048Mi"
gke-gcsfuse/ephemeral-storage-limit: "50Gi"

we are getting error while creating init container on check db line, below are the init container helm deployment file
deployment.yaml
initContainers:
      {{- if $extraPipPackages }}
      {{- include "airflow.init_container.install_pip_packages" (dict "Release" .Release "Values" .Values "extraPipPackages" $extraPipPackages) | indent 8 }}
      {{- end }}
      {{- if .Values.dags.gitSync.enabled }}
      ## git-sync is included so "airflow plugins" & "python packages" can be stored in the dags repo
      {{- include "airflow.container.git_sync" (dict "Release" .Release "Values" .Values "sync_one_time" "true") | indent 8 }}
      {{- end }}
      {{- include "airflow.init_container.check_db" (dict "Release" .Release "Values" .Values "volumeMounts" $volumeMounts) | indent 8 }}

Please refer the below link to get full script of deployment.yaml
https://github.com/airflow-helm/charts/blob/main/charts/airflow/templates/db-migrations/db-migrations-deployment.yaml
Please guide to add any additional configuration to get work properly.

Thanks for reporting the issue.
Unfortunately, currently the driver does not support volumes for initContainers, see https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver/issues/38.

Hi @songjiaxun,
Thanks for addressing this issue.
We are using GKE version 1.26.5-gke, shall we use the same version to leverage as sidecar container that you recommended in #38 or we need to wait for kubernetes version  update v1.28 to use sidecar.

@Zudhakar I would recommend waiting for the sidecar feature is officially supported in k8s 1.28.
Also I am working on a workaround. It may or may not work for the init containers. Basically, manual injection of the sidecar may work. I will keep you posted with my experiment.

@songjiaxun
I just noticed that GKE  version 1.28 release date is 2023-08-15. Can you please confirm that this change will be part of it and available.
Thanks.

Hi @Zudhakar , the Kubernetes sidecar feature is still alpha, and GKE cannot use the feature until it becomes beta.
We are working on testing the Kubernetes sidecar feature, and will start to use the feature as soon as possible.

Thanks for the update @songjiaxun
Is there any approximate time frame?

@Zudhakar we are working to test the feature now, and we will update with the time frame after the tests.
Closing this item as this is duplicated with https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver/issues/38. Please keep following up on the issue https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver/issues/38.

Duplicate of #38


chore: release 0.10.0

:robot: I have created a release *beep* *boop*
0.10.0 (2021-06-11)
Features

add number separators, units, conditional required, hex strings (#137) (397680d)
add pinning reupload-all (#138) (31a69ea)

Bug Fixes

remove topic option from pss commands (#144) (76e7f51)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/ethersphere/swarm-cli/releases/tag/v0.10.0 :sunflower: