keycloak-backend alpha plugin doesn't seem to be working

Hi, I add this to my backstage deployment following the directions for adding it to the new backstage backend
I'm still unable to login since it can't find my user identify in the catalog, i'm assuming this plugin automatically syncs, which I don't see happening during start up
Loading config from MergedConfigSource{FileConfigSource{path="/app/app-config-from-configmap.yaml"}, EnvConfigSource{count=4}} 55 {"level":"info","message":"Found 1 new secrets in config that will be redacted","service":"backstage"} 54 {"level":"info","message":"Listening on :7007","service":"rootHttpRouter"} 53 {"level":"info","message":"Plugin initialization started: 'app', 'proxy', 'scaffolder', 'techdocs', 'auth', 'catalog', 'permission', 'search'","service":"backstage","type":"initialization"} 52 {"level":"info","message":"Serving static app content from /app/packages/app/dist","plugin":"app","service":"backstage"} 51 {"level":"info","message":"Starting scaffolder with the following actions enabled fetch:plain, fetch:plain:file, fetch:template, debug:log, debug:wait, catalog:register, catalog:fetch, catalog:write, fs:delete, fs:rename","plugin":"scaffolder","service":"backstage"} 50 {"level":"info","message":"Creating Local publisher for TechDocs","plugin":"techdocs","service":"backstage"} 49 {"level":"info","message":"Configuring \"database\" as KeyStore provider","plugin":"auth","service":"backstage"} 48 {"level":"info","message":"Performing database migration","plugin":"catalog","service":"backstage"} 47 {"level":"info","message":"Configuring auth provider: oauth2Proxy","plugin":"auth","service":"backstage"} 46 {"entry":"main","level":"info","message":"Injecting env config into module-backstage.686b5ec2.js","plugin":"app","service":"backstage"} 45 {"level":"info","message":"Added DefaultCatalogCollatorFactory collator factory for type software-catalog","plugin":"search","service":"backstage"} 44 {"level":"info","message":"Added DefaultTechDocsCollatorFactory collator factory for type techdocs","plugin":"search","service":"backstage"} 43 {"level":"warn","message":"Permission backend started with permissions disabled. Enable permissions by setting permission.enabled=true.","plugin":"permission","service":"backstage"} 42 {"level":"info","message":"Storing 291 updated assets and 0 new assets","plugin":"app","service":"backstage"} 41 {"level":"info","message":"Starting all scheduled search tasks.","plugin":"search","service":"backstage"} 40 {"level":"info","message":"Plugin initialization complete, newly initialized: 'proxy', 'scaffolder', 'techdocs', 'auth', 'catalog', 'search', 'permission', 'app'","service":"backstage","type":"initialization"} 39 {"level":"info","message":"Task worker starting: search_index_software_catalog, {\"version\":2,\"cadence\":\"PT10M\",\"initialDelayDuration\":\"PT3S\",\"timeoutAfterDuration\":\"PT15M\"}","plugin":"search","service":"backstage","task":"search_index_software_catalog"} 38 {"level":"info","message":"Task worker startin

Hi @airedwin ,
Thanks for reporting this issue. Can you please clarify:

version of the application?
version of the plugin?
the exact reproduction steps, especially what you did add exactly and which directions you did follow, when you said "I add this to my backstage deployment following the directions for adding it to the new backstage backend"?
the behavior you expected

That will help us better understand and scope this issue. Thanks.

backstage version 1.27.0 using the alpha backend, this was defaulted through their create-app
plugin 1.9.12
i followed the directions at https://janus-idp.io/plugins/keycloak/#new-backend-configuration
my app-config.yaml looks like
catalog:   providers:     keycloakOrg:       default:         baseUrl: https://host/auth         loginRealm: myrealm         realm: myrealm         clientId: client-confidential         clientSecret: ${KEYCLOAK_CLIENTSECRET}         schedule:           frequency: { minutes: 30 }           timeout: { minutes: 3 }           initialDelay: { seconds: 15 }   import:     entityFilename: catalog-info.yaml     pullRequestBranchName: backstage-integration   rules:     - allow: [Component, System, API, Resource, Location]   locations:     # Local example data, file locations are relative to the backend process, typically packages/backend`
- type: file
target: ../../examples/entities.yaml
# Local example template
- type: file
  target: ../../examples/template/template.yaml
  rules:
    - allow: [Template]

# Local example organizational data
- type: file
  target: ../../examples/org.yaml
  rules:
    - allow: [User, Group]

`
i expect to see the scheduler running in the logs and for my keycloak users to be imported into the catalog

any update on this? is there something that i am doing wrong?

@Zaperex Can you please help answer this? Thanks.

@airedwin I'm going to need a bit more information.

What version is your keycloak instance? If you're using keycloak v17+, you should omit the /auth endpoint in your baseUrl.
Also since you have set an initial delay on your scheduler, it won't actually trigger immediately so the logs you provided would not have captured the setup of this.
If you are using keycloak 16 or earlier, then your app-config.yaml looks correct.

I tested using your configuration of using:

npx<EMAIL_ADDRESS><EMAIL_ADDRESS>keycloak instance that is v22.0.3
Also needed to add the following resolution into the root package.json due to the 1.28.x backstage packages being pulled in preventing the backend from starting.

  "resolutions": {
    "@backstage/backend-app-api": "0.7.5",
    "@backstage/backend-dynamic-feature-service": "0.2.10"
  },

Do you see a log containing the following that indicates the keycloak backend task worker started?
2024-06-21T15:01:16.718Z catalog info Task worker starting: KeycloakOrgEntityProvider:default:refresh, {"version":2,"cadence":"PT30M","initialDelayDuration":"PT15S","timeoutAfterDuration":"PT3M"} task=KeycloakOrgEntityProvider:default:refresh

A successful sync would log out something similar to:
2024-06-21T15:01:31.722Z catalog info Reading Keycloak users and groups class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=94a7f8ba-4414-4173-bdfd-ae325c6217a6
2024-06-21T15:01:31.778Z catalog info Read 5 Keycloak users and 4 Keycloak groups in 0.1 seconds. Committing... class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=94a7f8ba-4414-4173-bdfd-ae325c6217a6
2024-06-21T15:01:31.783Z catalog info Committed 5 Keycloak users and 4 Keycloak groups in 0.0 seconds. class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=94a7f8ba-4414-4173-bdfd-ae325c6217a6

When I tested with /auth, I would get a 404 error:
2024-06-21T15:02:43.423Z catalog info Reading Keycloak users and groups class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=ba54d185-7ff5-4086-b033-6c8f6ae9552f
2024-06-21T15:02:43.443Z catalog error Error while syncing Keycloak users and groups Request failed with status code 404 class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=ba54d185-7ff5-4086-b033-6c8f6ae9552f name=Error stack=Error: Request failed with status code 404
    at createError (/home/frkong/coding/test-keycloak/node_modules/@keycloak/keycloak-admin-client/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/home/frkong/coding/test-keycloak/node_modules/@keycloak/keycloak-admin-client/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/home/frkong/coding/test-keycloak/node_modules/@keycloak/keycloak-admin-client/node_modules/axios/lib/adapters/http.js:322:11)
    at IncomingMessage.emit (node:events:530:35)
    at endReadableNT (node:internal/streams/readable:1696:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) status=404

Do you have any additional logs related to the keycloak backend appear? Having those would help debug your issue.
Also can you check if you have your keycloak instance setup correctly?
It should have the following

client access type set to confidential
service account roles are enabled

The following realm-management client roles are enabled:

query-groups
query-users
view-users

@airedwin I'm going to need a bit more information.

What version is your keycloak instance? If you're using keycloak v17+, you should omit the /auth endpoint in your baseUrl since that would usually result in a 404 Not Found error to be triggered when the keycloak-backend attempts to read from the keycloak instance.

Also since you have set an initial delay of 15 seconds on your scheduler, it won't actually trigger immediately so the logs you provided would not have captured the setup of this. You would need to wait up to 15s before it starts the reading process. Can you provide more logs?

If you are using keycloak 16 or earlier, then your app-config.yaml looks correct.

Are you using a postgres database when starting backstage? If so, the schedule for the keycloak-backend would be stored in the database, and might not startup immediately after backend starts due to that next_update_at value in the database.

You can verify by going into the postgresql database, connecting to the catalog database with \c backstage_plugin_catalog, then querying: SELECT * FROM backstage_backend_tasks__tasks;
If an entry for the keycloak backend does not exist in that table, then it probably never got started. In that case, you should check if you have properly installed the keycloak backend into the backend.

I tested using your configuration of using:

npx<EMAIL_ADDRESS><EMAIL_ADDRESS>keycloak instance that is v22.0.3
Also needed to add the following resolution into the root package.json when I tested due to the 1.28.x backstage packages being pulled in (due to transitive dependencies) preventing the backend from starting.

  "resolutions": {
    "@backstage/backend-app-api": "0.7.5",
    "@backstage/backend-dynamic-feature-service": "0.2.10"
  },

Do you see a log containing the following that indicates the keycloak backend task worker started?
2024-06-21T15:01:16.718Z catalog info Task worker starting: KeycloakOrgEntityProvider:default:refresh, {"version":2,"cadence":"PT30M","initialDelayDuration":"PT15S","timeoutAfterDuration":"PT3M"} task=KeycloakOrgEntityProvider:default:refresh

A successful sync would log out something similar to:
2024-06-21T15:01:31.722Z catalog info Reading Keycloak users and groups class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=94a7f8ba-4414-4173-bdfd-ae325c6217a6
2024-06-21T15:01:31.778Z catalog info Read 5 Keycloak users and 4 Keycloak groups in 0.1 seconds. Committing... class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=94a7f8ba-4414-4173-bdfd-ae325c6217a6
2024-06-21T15:01:31.783Z catalog info Committed 5 Keycloak users and 4 Keycloak groups in 0.0 seconds. class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=94a7f8ba-4414-4173-bdfd-ae325c6217a6

When I tested with /auth, I would get a 404 error:
2024-06-21T15:02:43.423Z catalog info Reading Keycloak users and groups class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=ba54d185-7ff5-4086-b033-6c8f6ae9552f
2024-06-21T15:02:43.443Z catalog error Error while syncing Keycloak users and groups Request failed with status code 404 class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=ba54d185-7ff5-4086-b033-6c8f6ae9552f name=Error stack=Error: Request failed with status code 404
    at createError (/home/frkong/coding/test-keycloak/node_modules/@keycloak/keycloak-admin-client/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/home/frkong/coding/test-keycloak/node_modules/@keycloak/keycloak-admin-client/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/home/frkong/coding/test-keycloak/node_modules/@keycloak/keycloak-admin-client/node_modules/axios/lib/adapters/http.js:322:11)
    at IncomingMessage.emit (node:events:530:35)
    at endReadableNT (node:internal/streams/readable:1696:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) status=404

Do you have any additional logs related to the keycloak backend appear? Having those would help debug your issue.
Also can you check if you have your keycloak instance setup correctly? It should have the following

client access type set to confidential

service account roles are enabled

The following realm-management client roles are enabled:

query-groups
query-users
view-users

I will give this a try, i think the only difference between yours and mine is the resolutions?
I dont see the Task starting in the logs, I will check the database shortly

My start up logs are
Loading config from MergedConfigSource{FileConfigSource{path="/app/app-config-from-configmap.yaml"}, EnvConfigSource{count=4}} 18 {"level":"info","message":"Found 1 new secrets in config that will be redacted","service":"backstage"} 17 {"level":"info","message":"Listening on :7007","service":"rootHttpRouter"} 16 {"level":"info","message":"Plugin initialization started: 'app', 'proxy', 'scaffolder', 'techdocs', 'auth', 'catalog', 'permission', 'search'","service":"backstage","type":"initialization"} 15 {"level":"info","message":"Serving static app content from /app/packages/app/dist","plugin":"app","service":"backstage"} 14 {"level":"info","message":"Starting scaffolder with the following actions enabled fetch:plain, fetch:plain:file, fetch:template, debug:log, debug:wait, catalog:register, catalog:fetch, catalog:write, fs:delete, fs:rename","plugin":"scaffolder","service":"backstage"} 13 {"level":"info","message":"Creating Local publisher for TechDocs","plugin":"techdocs","service":"backstage"} 12 {"level":"info","message":"Configuring \"database\" as KeyStore provider","plugin":"auth","service":"backstage"} 11 {"level":"warn","message":"Permission backend started with permissions disabled. Enable permissions by setting permission.enabled=true.","plugin":"permission","service":"backstage"} 10 {"level":"info","message":"Performing database migration","plugin":"catalog","service":"backstage"} 9 {"level":"info","message":"Configuring auth provider: oauth2Proxy","plugin":"auth","service":"backstage"} 8 {"level":"info","message":"Added DefaultCatalogCollatorFactory collator factory for type software-catalog","plugin":"search","service":"backstage"} 7 {"level":"info","message":"Added DefaultTechDocsCollatorFactory collator factory for type techdocs","plugin":"search","service":"backstage"} 6 {"entry":"main","level":"info","message":"Injecting env config into module-backstage.686b5ec2.js","plugin":"app","service":"backstage"} 5 {"level":"info","message":"Storing 291 updated assets and 0 new assets","plugin":"app","service":"backstage"} 4 {"level":"info","message":"Starting all scheduled search tasks.","plugin":"search","service":"backstage"} 3 {"level":"info","message":"Plugin initialization complete, newly initialized: 'proxy', 'techdocs', 'scaffolder', 'permission', 'auth', 'search', 'catalog', 'app'","service":"backstage","type":"initialization"} 2 {"level":"info","message":"Task worker starting: search_index_software_catalog, {\"version\":2,\"cadence\":\"PT10M\",\"initialDelayDuration\":\"PT3S\",\"timeoutAfterDuration\":\"PT15M\"}","plugin":"search","service":"backstage","task":"search_index_software_catalog"} 1 {"level":"info","message":"Task worker starting: search_index_techdocs, {\"version\":2,\"cadence\":\"PT10M\",\"initialDelayDuration\":\"PT3S\",\"timeoutAfterDuration\":\"PT15M\"}","plugin":"search","service":"backstage","task":"search_index_techdocs"}

my packages/backend/src/index.ts is
/*
 * Hi!
 *
 * Note that this is an EXAMPLE Backstage backend. Please check the README.
 *
 * Happy hacking!
 */

import { createBackend } from '@backstage/backend-defaults';

const backend = createBackend();

backend.add(import('@backstage/plugin-app-backend/alpha'));
backend.add(import('@backstage/plugin-proxy-backend/alpha'));
backend.add(import('@backstage/plugin-scaffolder-backend/alpha'));
backend.add(import('@backstage/plugin-techdocs-backend/alpha'));

// auth plugin
backend.add(import('@backstage/plugin-auth-backend'));
// See https://backstage.io/docs/backend-system/building-backends/migrating#the-auth-plugin
backend.add(import('@backstage/plugin-auth-backend-module-guest-provider'));
// See https://backstage.io/docs/auth/guest/provider
backend.add(
  import('@backstage/plugin-auth-backend-module-oauth2-proxy-provider'),
);

// catalog plugin
backend.add(import('@backstage/plugin-catalog-backend/alpha'));
backend.add(
  import('@backstage/plugin-catalog-backend-module-scaffolder-entity-model'),
);

// permission plugin
backend.add(import('@backstage/plugin-permission-backend/alpha'));
backend.add(
  import('@backstage/plugin-permission-backend-module-allow-all-policy'),
);

// search plugin
backend.add(import('@backstage/plugin-search-backend/alpha'));
backend.add(import('@backstage/plugin-search-backend-module-catalog/alpha'));
backend.add(import('@backstage/plugin-search-backend-module-techdocs/alpha'));

// custom keycloak plugin
// @ts-ignore
backend.add(import('@janus-idp/backstage-plugin-keycloak-backend/alpha'));

backend.start();

I assume that is how I "install" the plugin?

your packages/backend/src/index.ts looks correct.

It seems like I don't have those tables in my database, I'm using an AWS RDS instance, with a single database with multiple schemas

Hmm that most likely indicates that the keycloak backend did not start up at all.

I built in docker with this dockerfile
# Stage 1 - Create yarn install skeleton layer
FROM node:18-bookworm-slim AS packages

WORKDIR /app
COPY package.json yarn.lock ./

COPY packages packages

# Comment this out if you don't have any internal plugins
#COPY plugins plugins

RUN find packages \! -name "package.json" -mindepth 2 -maxdepth 2 -exec rm -rf {} \+

# Stage 2 - Install dependencies and build packages
FROM node:18-bookworm-slim AS build

# Install isolate-vm dependencies, these are needed by the @backstage/plugin-scaffolder-backend.
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && \
    apt-get install -y --no-install-recommends python3 g++ build-essential && \
    yarn config set python /usr/bin/python3

# Install sqlite3 dependencies. You can skip this if you don't use sqlite3 in the image,
# in which case you should also move better-sqlite3 to "devDependencies" in package.json.
#RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
#    --mount=type=cache,target=/var/lib/apt,sharing=locked \
#    apt-get update && \
#    apt-get install -y --no-install-recommends libsqlite3-dev

USER node
WORKDIR /app

COPY --from=packages --chown=node:node /app .

RUN yarn config set "strict-ssl" false -g
RUN --mount=type=cache,target=/home/node/.cache/yarn,sharing=locked,uid=1000,gid=1000 \
    yarn install --network-timeout 600000

COPY --chown=node:node . .

RUN yarn tsc
RUN yarn --cwd packages/backend build
# If you have not yet migrated to package roles, use the following command instead:
# RUN yarn --cwd packages/backend backstage-cli backend:bundle --build-dependencies

RUN mkdir packages/backend/dist/skeleton packages/backend/dist/bundle \
    && tar xzf packages/backend/dist/skeleton.tar.gz -C packages/backend/dist/skeleton \
    && tar xzf packages/backend/dist/bundle.tar.gz -C packages/backend/dist/bundle

# Stage 3 - Build the actual backend image and install production dependencies
FROM node:18-bookworm-slim

# Install isolate-vm dependencies, these are needed by the @backstage/plugin-scaffolder-backend.
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && \
    apt-get install -y --no-install-recommends python3 g++ build-essential && \
    yarn config set python /usr/bin/python3

# Install sqlite3 dependencies. You can skip this if you don't use sqlite3 in the image,
# in which case you should also move better-sqlite3 to "devDependencies" in package.json.
#RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
#    --mount=type=cache,target=/var/lib/apt,sharing=locked \
#    apt-get update && \
#    apt-get install -y --no-install-recommends libsqlite3-dev

# From here on we use the least-privileged `node` user to run the backend.
USER node

# This should create the app dir as `node`.
# If it is instead created as `root` then the `tar` command below will
# fail: `can't create directory 'packages/': Permission denied`.
# If this occurs, then ensure BuildKit is enabled (`DOCKER_BUILDKIT=1`)
# so the app dir is correctly created as `node`.
WORKDIR /app

# Copy the install dependencies from the build stage and context
COPY --from=build --chown=node:node /app/yarn.lock /app/package.json /app/packages/backend/dist/skeleton/ ./

RUN yarn config set "strict-ssl" false -g
RUN --mount=type=cache,target=/home/node/.cache/yarn,sharing=locked,uid=1000,gid=1000 \
    yarn install --production --network-timeout 600000

# Copy the built packages from the build stage
COPY --from=build --chown=node:node /app/packages/backend/dist/bundle/ ./

# Copy any other files that we need at runtime
COPY --chown=node:node app-config.yaml ./

# This switches many Node.js dependencies to production mode.
ENV NODE_ENV production

CMD ["node", "packages/backend", "--config", "app-config.yaml"]

and i'm using a helm chart from https://github.com/backstage/charts
i slightly customized it to not deploy the in cluster postgresql container, but instead to use environment values for the RDS

I see, can I ask if you had the same issue when running it locally?

sorry did not mean to close it

i have not tried running it locally

Can you try running it locally to verify that it doesn't work there as well? It should also make this debugging process faster so we can see where the issue stems from.

ok let me set up a local db

interesting, it worked locally with same docker image
{"level":"info","message":"Task worker starting: KeycloakOrgEntityProvider:default:refresh, {"version":2,"cadence":"PT30M","initialDelayDuration":"PT15S","timeoutAfterDuration":"PT3M"}","plugin":"catalog","service":"backstage","task":"KeycloakOrgEntityProvider:default:refresh"}

hmm that's very weird, if you try to run that image with the helm chart, it doesn't start the keycloak plugin?

hmm that's very weird, if you try to run that image with the helm chart, it doesn't start the keycloak plugin?

ya, that's what it seems like... makes no sense lol

i just repushed the image, it's the same

You changed the image field of the helm chart right?
https://github.com/backstage/charts/blob/864c683f5f459ad021254e5dee093201bf7bfa8b/charts/backstage/values.yaml#L93-L102

You changed the image field of the helm chart to refer to your image right? https://github.com/backstage/charts/blob/864c683f5f459ad021254e5dee093201bf7bfa8b/charts/backstage/values.yaml#L93-L102

yes
spec:
containers:
- args:
- '--config'
- /app/app-config-from-configmap.yaml
command:
- node
- packages/backend
env:
- name: APP_CONFIG_backend_listen_port
value: '7007'
- name: POSTGRES_HOST
value: myserver
- name: POSTGRES_PORT
value: '5432'
- name: POSTGRES_DATABASE
value: backstage_dev
- name: POSTGRES_USER
value: backstage_admin
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: backstage-dev-database-secret
- name: KEYCLOAK_CLIENTSECRET
valueFrom:
secretKeyRef:
key: client-secret
name: backstage-dev-keycloak-secret
- name: APP_CONFIG_app_baseUrl
value: 'https://backstage-dev'
- name: APP_CONFIG_backend_baseUrl
value: 'https://backstage-dev'
- name: APP_CONFIG_backend_cors_origin
value: 'https://backstage-dev'
- name: NODE_TLS_REJECT_UNAUTHORIZED
value: '0'
image: 'backstage:latest'
imagePullPolicy: Always

i ran locally with
docker run -e POSTGRES_HOST=host.docker.internal -e POSTGRES_PORT=5432 -e POSTGRES_DATABASE=backstage -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD="" -e KEYCLOAK_CLIENTSECRET=mysecret -e APP_CONFIG_app_baseUrl=https://backstage-dev -e APP_CONFIG_backend_baseUrl=https://backstage-dev backstage:latest

I don't think backstage:latest would work? The correct format should be:
image:
  registry: <your registry>
  repository: <your repository>/backstage
  tag: latest

Can you perhaps inspect your backstage deployment pods to verify what the image being used is? I suspect it's just using the defaults.

I don't think backstage:latest would work? The correct format should be:
image:
  registry: <your registry>
  repository: <your repository>/backstage
  tag: latest

Can you perhaps inspect your backstage deployment pods to verify what the image being used is? I suspect it's just using the defaults.

sorry, i redacted my private repo, but ya, i have the repo in there, it's like repo.com/library/backstage:latest

ya, it's definitely on the server

Oh wait, can I ask where you're adding your keycloak backend configurations?

You would either need to inline your configurations in the appConfig field of the helm values when you install it.
Alternatively, you can create a ConfigMap containing your custom-app-config.yaml configurations and add it to the extraAppConfig field. For example, in your values.yaml add:
extraAppConfig:
  - configMapRef: app-config-custom
    filename: custom-app-config.yaml

Then add the config map with your configurations, change the values to fit your needs
kind: ConfigMap
apiVersion: v1
metadata:
  name: app-config-custom

data:
  custom-app-config.yaml: |
    catalog:
      providers:
        keycloakOrg:
          default:
            baseUrl: http://localhost:8080
            realm: backstage
            loginRealm: backstage
            clientId: backstage
            clientSecret: <client-secret>
            schedule:
              frequency:
                minutes: 30
              timeout:
                minutes: 3
              initialDelay:
                seconds: 15

$ pwd
/app
$ ls
app-config-from-configmap.yaml  app-config.yaml  node_modules  package.json  packages  yarn.lock
$ cat app-config-from-configmap.yaml
app:
  baseUrl: https://backstage-dev.i2cv.io
  title: Clairvoyant Backstage
auth:
  environment: production
  providers:
    oauth2Proxy:
      signIn:
        resolvers:
        - resolver: forwardedUserMatchingUserEntityName
backend:
  baseUrl: https://backstage-dev.i2cv.io
  database:
    client: pg
    connection:
      database: ${POSTGRES_DATABASE}
      host: ${POSTGRES_HOST}
      password: ${POSTGRES_PASSWORD}
      port: ${POSTGRES_PORT}
      user: ${POSTGRES_USER}
    pluginDivisionMode: schema
  listen:
    port: :7007
catalog:
  import:
    entityFilename: catalog-info.yaml
    pullRequestBranchName: backstage-integration
  keycloakOrg:
    default:
      baseUrl: https://keycloak-dev.i2cv.io/auth
      clientId: backstage-confidential
      clientSecret: ${KEYCLOAK_CLIENTSECRET}
      loginRealm: cv
      realm: cv
      schedule:
        frequency:
          minutes: 30
        initialDelay:
          seconds: 15
        timeout:
          minutes: 3
  locations:
  - target: ../../examples/entities.yaml
    type: file
  - rules:
    - allow:
      - Template
    target: ../../examples/template/template.yaml
    type: file
  - rules:
    - allow:
      - User
      - Group
    target: ../../examples/org.yaml
    type: file
  providers: null
  rules:
  - allow:
    - Component
    - System
    - API
    - Resource
    - Location
enabled:
  keycloak: true
techdocs:
  builder: local
  generator:
    runIn: docker
  publisher:
    type: local

unless this isn't the file that is being used

ah you misconfigured it in your app-config-from-configmap.yaml.
you used
catalog:
  import:
    entityFilename: catalog-info.yaml
    pullRequestBranchName: backstage-integration
  keycloakOrg:
    default:
      baseUrl: https://keycloak-dev/auth
      clientId: backstage-confidential
      clientSecret: ${KEYCLOAK_CLIENTSECRET}
      loginRealm: realm
      realm: realm
      schedule:
        frequency:
          minutes: 30
        initialDelay:
          seconds: 15
        timeout:
          minutes: 3

instead of doing catalog.providers.keycloakOrg

i just noticed that too, i think it's in my helm somewhere that is missing, thanks!

let me know if it works, if it does I'll close the issue.

it's working, i think we should delete this, user error lol

I got this working but it doesn't seem like the provider can match up an identity with a user in the catalog. I assume it's a resolver issue? I'm using:
auth:
  providers:
    oauth2Proxy:
      signIn:
        resolvers:
          - resolver: forwardedUserMatchingUserEntityName

should I be using something different? the users are imported correctly
{"class":"KeycloakOrgEntityProvider","level":"info","message":"Reading Keycloak users and groups","plugin":"catalog","service":"backstage","taskId":"KeycloakOrgEntityProvider:default:refresh","taskInstanceId":"bed5788f-c893-447c-a659-43976c9bebd6"}
{"class":"KeycloakOrgEntityProvider","level":"info","message":"Read 9 Keycloak users and 4 Keycloak groups in 0.2 seconds. Committing...","plugin":"catalog","service":"backstage","taskId":"KeycloakOrgEntityProvider:default:refresh","taskInstanceId":"bed5788f-c893-447c-a659-43976c9bebd6"}
{"class":"KeycloakOrgEntityProvider","level":"info","message":"Committed 9 Keycloak users and 4 Keycloak groups in 0.0 seconds.","plugin":"catalog","service":"backstage","taskId":"KeycloakOrgEntityProvider:default:refresh","taskInstanceId":"bed5788f-c893-447c-a659-43976c9bebd6"}

this is the error i'm getting in the log
{"level":"info","message":"::1 - - [10/Sep/2024:21:01:51 +0000] \"GET /api/catalog/entities/by-name/User/default/1f2b03f8-bbe0-4ae2-8640-52e56873962e HTTP/1.1\" 404 281 \"-\" \"node-fetch/1.0 (+https://github.com/bitinn/node-fetch)\"","service":"rootHttpRouter","type":"incomingRequest"}
{"level":"error","message":"Request failed with status 500 Failed to sign-in, unable to resolve user identity","service":"rootHttpRouter","stack":"Error: Failed to sign-in, unable to resolve user identity\n    at /app/node_modules/@backstage/plugin-auth-node/dist/index.cjs.js:860:11\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Object.refresh (/app/node_modules/@backstage/plugin-auth-node/dist/index.cjs.js:952:24)","type":"errorHandler"}

it looks like it's querying on the keycloak id which is in metadata.annotations.keycloak.org/id or metadata.id, do i need to write a custom resolver for this?

this was the data put into the catalog by your plugin
06b9eb7e-9991-4fb0-9263-97f4a4c53e47	metadata.annotations.backstage.io/managed-by-location	url:https://keycloak/auth/admin/realms/myrealm/users/1f2b03f8-bbe0-4ae2-8640-52e56873962e	url:https://keycloak/auth/admin/realms/myrealm/users/1f2b03f8-bbe0-4ae2-8640-52e56873962e 06b9eb7e-9991-4fb0-9263-97f4a4c53e47	metadata.annotations.backstage.io/managed-by-origin-location	url:https://keycloak/auth/admin/realms/myrealm/users/1f2b03f8-bbe0-4ae2-8640-52e56873962e	url:https://keycloak/auth/admin/realms/myrealm/users/1f2b03f8-bbe0-4ae2-8640-52e56873962e 06b9eb7e-9991-4fb0-9263-97f4a4c53e47	metadata.annotations.backstage.io/view-url	https://keycloak/auth/admin/realms/myrealm/users/1f2b03f8-bbe0-4ae2-8640-52e56873962e	https://keycloak/auth/admin/realms/myrealm/users/1f2b03f8-bbe0-4ae2-8640-52e56873962e 06b9eb7e-9991-4fb0-9263-97f4a4c53e47	metadata.annotations.keycloak.org/id	1f2b03f8-bbe0-4ae2-8640-52e56873962e	1f2b03f8-bbe0-4ae2-8640-52e56873962e 06b9eb7e-9991-4fb0-9263-97f4a4c53e47	metadata.annotations.keycloak.org/realm	myrealm	myrealm 06b9eb7e-9991-4fb0-9263-97f4a4c53e47	apiversion	backstage.io/v1beta1	backstage.io/v1beta1 06b9eb7e-9991-4fb0-9263-97f4a4c53e47	kind	user	User 06b9eb7e-9991-4fb0-9263-97f4a4c53e47	spec.profile.email<EMAIL_ADDRESS><EMAIL_ADDRESS>06b9eb7e-9991-4fb0-9263-97f4a4c53e47	spec.profile.displayname	edwin lastname	Edwin Lastname 06b9eb7e-9991-4fb0-9263-97f4a4c53e47	spec.memberof	admin	Admin 06b9eb7e-9991-4fb0-9263-97f4a4c53e47	spec.memberof.admin	TRUE	TRUE 06b9eb7e-9991-4fb0-9263-97f4a4c53e47	spec.memberof	keycloak read only	Keycloak Read Only 06b9eb7e-9991-4fb0-9263-97f4a4c53e47	spec.memberof.keycloak read only	TRUE	TRUE 06b9eb7e-9991-4fb0-9263-97f4a4c53e47	metadata.name	edwin.lastname	edwin.lastname 06b9eb7e-9991-4fb0-9263-97f4a4c53e47	metadata.namespace	default	default 06b9eb7e-9991-4fb0-9263-97f4a4c53e47	metadata.uid	06b9eb7e-9991-4fb0-9263-97f4a4c53e47	06b9eb7e-9991-4fb0-9263-97f4a4c53e47 06b9eb7e-9991-4fb0-9263-97f4a4c53e47	relations.memberof	group:default/admin	group:default/admin 06b9eb7e-9991-4fb0-9263-97f4a4c53e47	relations.memberof	group:default/keycloak read only	group:default/keycloak read only

Hmm that's weird, it seems the issue on your end is that the x-forwarded-user header seems be resolving to be the id of the keycloak user instead of the username of the user since the name of the user entity that gets queried is determined by this header value: https://github.com/backstage/backstage/blob/f094dfd54a293bfe03cd00cb6dbd7f562cc26a8b/plugins/auth-backend-module-oauth2-proxy-provider/src/resolvers.ts#L31.
This may require configuring your keycloak instance or proxy to return the correct x-forwarded-user header, or a custom resolver might be needed to match to the id annotation instead?

Ya,.I think so too, thanks. I'm using the default oauth2proxy sidecar for backstage as part of their helm chart, I'll see if I can configure that forwarded header

Actually, I think that is correct, the user header is id and there is a preferred-username header that is the username... would it make sense to change your plugin to put the id in the catalog name field?

Or maybe it's a keycloak client config of what to put in the user claim

There is also the option of installing a transformer module to set the name of the entity to the id instead of the username.

There is also the option of installing a transformer module to set the name of the entity to the id instead of the username.

Thanks, is there any documentation on how to build the entity to return?

Nevermind, I foudn it https://github.com/janus-idp/backstage-plugins/blob/main/plugins/keycloak-backend/src/lib/read.ts

I'm not sure if i'm doing this right, but i created a custom module for the usertransformer and i'm getting an error
/app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1648
              provides: Array.from(moduleInit.consumes).map((c) => c.id)
                                                                     ^
TypeError: Cannot read properties of undefined (reading 'id')
    at /app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1648:70
    at Array.map (<anonymous>)
    at /app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1648:57
    at Array.map (<anonymous>)
    at /app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1642:33
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Promise.all (index 5)
    at async #doStart (/app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1633:5)
    at async BackendInitializer.start (/app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1562:5)
    at async BackstageBackend.start (/app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1762:5)

I'm assuming it's the user.id!
this is my module
import { createBackendModule } from '@backstage/backend-plugin-api';
import type UserRepresentation from '@keycloak/keycloak-admin-client/lib/defs/userRepresentation';
import { UserEntity } from '@backstage/catalog-model';

import {
  keycloakTransformerExtensionPoint,
  UserTransformer,
  GroupRepresentationWithParentAndEntity,
  KeycloakTransformerExtensionPoint,
} from '@janus-idp/backstage-plugin-keycloak-backend';

export const catalogModuleCustomUserTransformer = createBackendModule({
  pluginId: 'catalog',
  moduleId: 'custom-user-transformer',
  register(reg) {
    reg.registerInit({
      deps: { keycloak: keycloakTransformerExtensionPoint },
      async init({
        keycloak,
      }: {
        keycloak: KeycloakTransformerExtensionPoint;
      }) {
        keycloak.setUserTransformer(customUserTransformer);
      },
    });
  },
});

const customUserTransformer: UserTransformer = async (
  entity: UserEntity,
  user: UserRepresentation,
  realm: string,
  groups: GroupRepresentationWithParentAndEntity[],
) => {
  return {
    ...entity,
    metadata: {
      name: user.id!,
    },
  };
};

and i imported it like this
backend.add(import('@janus-idp/backstage-plugin-keycloak-backend/alpha'));
backend.add(import('@internal/backstage-plugin-catalog-backend-module-clairvoyant-user-transformer'));

@Zaperex i think the issue is the keycloakTransformerExtensionPoint, i don't know if it's a bug or if i'm doing something wrong, but i tried importing that from the alpha plugin (i changed my import above) and it says it's missing the id

maybe i need a newer version of the backstage backend, gonna try upgrading

I'm not sure if i'm doing this right, but i created a custom module for the usertransformer and i'm getting an error
/app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1648
              provides: Array.from(moduleInit.consumes).map((c) => c.id)
                                                                     ^
TypeError: Cannot read properties of undefined (reading 'id')
    at /app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1648:70
    at Array.map (<anonymous>)
    at /app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1648:57
    at Array.map (<anonymous>)
    at /app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1642:33
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Promise.all (index 5)
    at async #doStart (/app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1633:5)
    at async BackendInitializer.start (/app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1562:5)
    at async BackstageBackend.start (/app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1762:5)

I'm assuming the module isn't imported right
this is my module
import { createBackendModule } from '@backstage/backend-plugin-api';
import type UserRepresentation from '@keycloak/keycloak-admin-client/lib/defs/userRepresentation';
import { UserEntity } from '@backstage/catalog-model';

import {
  keycloakTransformerExtensionPoint,
  UserTransformer,
  GroupRepresentationWithParentAndEntity,
  KeycloakTransformerExtensionPoint,
} from '@janus-idp/backstage-plugin-keycloak-backend';

export const catalogModuleCustomUserTransformer = createBackendModule({
  pluginId: 'catalog',
  moduleId: 'custom-user-transformer',
  register(reg) {
    reg.registerInit({
      deps: { keycloak: keycloakTransformerExtensionPoint },
      async init({
        keycloak,
      }: {
        keycloak: KeycloakTransformerExtensionPoint;
      }) {
        keycloak.setUserTransformer(customUserTransformer);
      },
    });
  },
});

const customUserTransformer: UserTransformer = async (
  entity: UserEntity,
  user: UserRepresentation,
  realm: string,
  groups: GroupRepresentationWithParentAndEntity[],
) => {
  return {
    ...entity,
    metadata: {
      name: user.id!,
    },
  };
};

and i imported it like this
backend.add(import('@janus-idp/backstage-plugin-keycloak-backend/alpha'));
backend.add(import('@internal/backstage-plugin-catalog-backend-module-custom-user-transformer'));

Updating fixed it


Type added for MCE sub/csv status in MCH specs

Signed-off-by: Erin Murphy<EMAIL_ADDRESS>

Issue #20377


chore: update test suite to relevant binaries

This PR updates the binaries used in CI to a more relevant version. I've chosen ibc-go-simd v8.5.1.
Note: I am unsure if this will pass all tests. Feel free to do with this PR as you wish.

if any upgrade test do fail, it could be due to how halt heights work in v44/45 (at height) vs v47+ (height+1) in my experience. so any block off by 1 errors that you see are probably related to that
further, test are still flakey but I have a PR pending regiew for a month that fixes flakeyness. So if you want to merge that in here I can approve & merge once all CI passes

#1263

@boojamya Ill talk a look at this tomorrow if you can enable maintainer edits on this PR for me :)

Thanks dan!


Redundant code in filter method of RedactingFilter - unnecessary deepcopy

Please describe an inefficiency you see in the project code currently
There is redundant code currently present in the RedactingFilter class, specifically in the filter method.  In line 22, we are deepcopying the __dict__ of the records object of the logger. We then loop through this, setting the items of the deepcopy to have the redaction rules applied, and finally set the original record object equal to this. I claim that this is unnecessary and that we can avoid this extra deepcopy all together.
Describe the solution you'd like
We can simply remove the deepcopy, and the usage ofsetarr in line 29, to simply set d[k] = self.redact(content, k).

Fixed in release 0.0.2


Review

This PR is only for making a review possible on GitHub

Hi Krystof,
thanks for your review!
I hope I answered everything and did not miss anything.
I was looking for a way to send the comments and I guess I probably shouldn't have clicked to "resolve" :-D
if necessary, I will give unresolved again.
Michal

Hi Krystof,
thanks for your review!
I hope I answered everything and did not miss anything.
I was looking for a way to send the comments and I guess I probably shouldn't have clicked to "resolve" :-D
if necessary, I will give unresolved again.
Michal


[CELEBORN-1659][FOLLOWUP] Dockerfile should support copying CLI jars

What changes were proposed in this pull request?
Dockerfile should support copying CLI jars.
Why are the changes needed?
CLI jars are generated from make-distribution.sh. Therefore, Dockerfile could copy CLI jars to /opt/celeborn/ directory.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
No.

Ping @akpatnam25.


Uncaught TypeError: Cannot read property 'repo_name' of undefined

Open directory that has no repo
Git error!
Get error.

Atom: 1.24.0 x64
Electron: 1.6.16
OS: Mac OS X 10.13.4
Thrown From: ctags-status package 1.3.7
Stack Trace
Uncaught TypeError: Cannot read property 'repo_name' of undefined
At /Users/user/.atom/packages/ctags-status/node_modules/q/q.js:155

TypeError: Cannot read property 'repo_name' of undefined
    at StatusBarView.setProfile (/packages/my-github-profile/lib/main.coffee:57:62)
    at setProfile (/packages/my-github-profile/lib/main.coffee:116:25)
    at /packages/my-github-profile/lib/main.coffee:126:11
    at _fulfilled (/packages/ctags-status/node_modules/q/q.js:834:54)
    at self.promiseDispatch.done (/packages/ctags-status/node_modules/q/q.js:863:30)
    at Promise.promise.promiseDispatch (/packages/ctags-status/node_modules/q/q.js:796:13)
    at /packages/ctags-status/node_modules/q/q.js:604:44
    at runSingle (/packages/ctags-status/node_modules/q/q.js:137:13)
    at flush (/packages/ctags-status/node_modules/q/q.js:125:13)
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickCallback (internal/process/next_tick.js:98:9)

Commands
     -7:05.7.0 core:select-right (input.hidden-input)
     -7:03.1.0 core:save (input.hidden-input)

Non-Core Packages
activate-power-mode 2.7.0
atom-ctags 5.1.2
atom-material-syntax 1.0.8
atom-material-syntax-dark 1.0.0
atom-material-ui 2.1.3
atomic-chrome 0.3.3
auto-indent 0.5.0
autocomplete-ctags 0.4.1
base16-tomorrow-night-eighties-syntax 0.12.0
browser-refresh 0.9.0
busy-signal 1.4.3
case-conversion 0.2.0
color-picker 2.2.5
ctags-status 1.3.7
debug 0.0.1
dict 0.3.1
editorconfig 2.2.2
emmet 2.4.3
environment 1.3.1
file-type-icons 1.3.4
file-types 0.5.5
fold-functions 0.5.0
git-control 0.9.0
git-diff-details 1.4.0
go-config 1.2.4
hard-wrap 1.7.1
Hyrule-atom 1.2.1
intentions 1.1.5
last-cursor-position 0.9.2
linter 2.2.0
linter-ui-default 1.6.10
marked 0.1.10
material 2.2.0
material-ui 1.0.10
merge-conflicts 1.4.5
minimap 4.29.7
my-github-profile 0.2.1
nucleus-dark-ui 0.12.3
open-in-github-app 0.2.6
php-debug 0.2.6
pigments 0.40.2
pretty-json 1.6.4
remote-ftp 2.1.4
sass-autocompile 0.13.5
Sass-Syntax-Highlight 0.1.1
spacegray-tomorrowdark-ui 2.1.0
tag 0.5.0
term3 0.22.1
tomorrow-evening-syntax 1.1.0
tomorrow-night-eighties 0.5.0
tomorrow-night-eighties-syntax 1.0.2
tree-view-git-branch 0.1.2
tree-view-git-status 1.4.0
unity-ui 2.1.11
url-encode 0.5.0
vagrant 0.1.0

It seems that the error is actually raised by another package my-github-profile which also uses q.js according to the stacktrace
    at StatusBarView.setProfile (/packages/my-github-profile/lib/main.coffee:57:62)

Yeah, I opened one there, but "Latest commit 1ec7b4d on Jul 12, 2015", I probably won't get a reply. Closing this ticket, though. Thanks!

@marv3lls i think it's an issue in https://github.com/atom/atom/ that provided a trace at the wrong package ... Would you be able to reproduce it with Safe Mode: https://flight-manual.atom.io/hacking-atom/sections/debugging/#using-safe-mode ?


Issue with k8s.io/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/

This is a bug report
Unable to run Wordpress app on Kubernetes Minikube singlenode cluster using instructions available at https://kubernetes.io/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/
Page to Update:
https://kubernetes.io/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/

Any details you want to share?
Do you believe it is a documentation issue?


Added support for LocalDateTime and LocalDate

Modified CalendarConverter.java and DateConverter.java to add support for LocalDateTime and LocalDate formats. Modified AbstractDateConverterTest.java and CalendarConverterTest.java to include coverage for added code.

Hi @m-lefevre - Thanks for this PR, I'm really appreciate it. java 8 provides many convenient stuffs and I think to support it is really good. But introduce it in our core ModelMapper will cause ModelMapper only support for java 8 and above. I suggest maybe we can create a module for jdk8 stuff.

@chhsiao90, what exactly do you mean by "module"? You want to put it in a separate repo? If so, it should we someone with the right to create a repo inside modelmapper account to start with.

But to be honest it only makes sense if someone is actively developing new features that should be ported to several versions of the library and post java 8 projects. If there's no active development, and I don't see it, why can't we merge this PR and release a new major version?

@m-lefevre , am I correct that this PR will only support LocalDate(Time) as a source, not as a destination? If so, I would expand it and add a destination support.

Hi @yuranos - Because currently modelmapper still support Java 1.6, so we can't adds any Java 8 stuff into our core codebase. What I thought about the module, please refer how jackson do like https://github.com/FasterXML/jackson-modules-java8 or https://github.com/FasterXML/jackson-datatype-jsr310.

hi, I'm working on this and please take a look modelmapper-module-java8.

Close this pull request as we still need support java6 and java7.


feat: add sidebar nav to docs pages

Resolves #1156
This PR adds a (localized) sidebar to documentation pages. For tutorial docs, it displays the nested table of contents extracted from electron/electron/docs/README.md#guides-and-tutorials.
For API docs, the sidebar displays a list of all APIs.
To Do

[ ] Make it look better. Currently it's a vanilla unordered list.
[ ] Fix the left-alignment of the page heading and breadcrumbs.
[ ] Visually emphasize the currently active link (probably using client-side JavaScript)
[ ] Display the new nested tutorials nav on the /docs index page too
[ ] Make it look decent on mobile devices. Maybe the answer is to hide hide the sidebar entirely on mobile.

Here's what it looks like right now:

And an API doc page, in English:

Going to also add a right-hand navbar for a table of contents. See progress at https://github.com/electron/i18n/pull/1349


Not calculating usd total amount in confirmation modal

👋

here it works

@prury did you manage to reproduce the bug? 👀

@prury did you manage to reproduce the bug? 👀

i did manage do reproduce another one that its similar 😄
Polkadot Assethub:

@exezbcz could you tell me how to reproduce this bug? I tried several times but not catch this wierd dialog.

Did not manage to find any steps to reproduce, One thing that happened to me was the incorrect amount calculation on Polkadot asset hub


Select and drag dag

Support to multi drag task
What is the purpose of the pull request

Convenient for users to drag and drop, when user use the dag, he should drag one by one. Now, he can select one or more to drag by pressing the key of 'ctrl' or 'meta'.

Brief change log

Add multiDrag.js

Please don't change the code style of dag.vue page, otherwise I don't know what you changed

Good job,
We should commit an issue before we do this work. Can you commit a new feature issue about your work and link this PR to that issue?

Please don't change the code style of dag.vue page, otherwise I don't know what you changed

sorry about that, i will redo dag.vue page.

Good job,
We should commit an issue before we do this work. Can you commit a new feature issue about your work and link this PR to that issue?

thx, get it. it's my first time to create PR. I will recheck my code these days, then commit a new feature issue.

Good job,
We should commit an issue before we do this work. Can you commit a new feature issue about your work and link this PR to that issue?

thx, get it. it's my first time to create PR. I will recheck my code these days, then commit a new feature issue.

done

Good job,
We should commit an issue before we do this work. Can you commit a new feature issue about your work and link this PR to that issue?

thx, get it. it's my first time to create PR. I will recheck my code these days, then commit a new feature issue.

done

I don’t see your associated issue, please give an issue to describe this function, and then connect to this pr

Good job,
We should commit an issue before we do this work. Can you commit a new feature issue about your work and link this PR to that issue?

thx, get it. it's my first time to create PR. I will recheck my code these days, then commit a new feature issue.

done

I don’t see your associated issue, please give an issue to describe this function, and then connect to this pr

such like this? #3068

very good， front end +1

very good， front end +1

you can Approve +1 at “File Changed”

thanks for your first contribution, if you need help, please contact<EMAIL_ADDRESS>or wechat(510570367), when added, please tell your github id


fix: add missing space after running glyph

The other status icons/glyphs have an additional space defined, without this space the running task count number gets squeezed with the icon (at least with my terminal set-up).
Note: This is probably a grapheme cluster thing (https://mitchellh.com/writing/grapheme-clusters-in-terminals)
--
BTW: thanks for creating and sharing overseer, it is IMHO a beautiful piece of engineering which inspires me a lot 🙏🏻

Thanks! Looks like the space may have been dropped during an icon migration


[Improvement] Provide the sample source file in the CharacterIO sample

The sample can be better understand if the content of the source file is given and also the output written to the file can be printed.
Sample - https://ballerinalang.org/docs/by-example/character-i-o

Created https://github.com/ballerina-platform/ballerina-examples/issues/805 to track this.


Expose a general function for agent execution

Description
This PR adds an assistant client to both public(UI) and server which currently exposes the agent execute API
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[ ] All tests pass, including unit test, integration test.

[ ] New functionality has user manual doc added.
[ ] Commits are signed per the DCO using --signoff.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

How do we allow other plugins to overwrite assistantClient and provide their own customized functionalities?

Could you provide a use case or example?


WIP: test of the operator check for e2e

@sdake: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

gencheck_operator
66f7b10fdb628356a199493c7a0337d995eef9e0
link
/test gencheck_operator

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@sdake: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

gencheck_operator
66f7b10fdb628356a199493c7a0337d995eef9e0
link
/test gencheck_operator

e2e_operator
66f7b10fdb628356a199493c7a0337d995eef9e0
link
/test e2e_operator

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@sdake: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e_operator
66f7b10fdb628356a199493c7a0337d995eef9e0
link
/test e2e_operator

gencheck_operator
295f8491f817f4c2db770ca660b294a49ec75ff2
link
/test gencheck_operator

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@sdake: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

gencheck_operator
295f8491f817f4c2db770ca660b294a49ec75ff2
link
/test gencheck_operator

e2e_operator
295f8491f817f4c2db770ca660b294a49ec75ff2
link
/test e2e_operator

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@sdake: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e_operator
295f8491f817f4c2db770ca660b294a49ec75ff2
link
/test e2e_operator

gencheck_operator
4eed17d462cc46f21047c139a1648e03fded5625
link
/test gencheck_operator

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@sdake: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

gencheck_operator
4eed17d462cc46f21047c139a1648e03fded5625
link
/test gencheck_operator

e2e_operator
4eed17d462cc46f21047c139a1648e03fded5625
link
/test e2e_operator

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test e2e_operator

@sdake: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

gencheck_operator
4eed17d462cc46f21047c139a1648e03fded5625
link
/test gencheck_operator

e2e_operator
4eed17d462cc46f21047c139a1648e03fded5625
link
/test e2e_operator

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@sdake: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e_operator
4eed17d462cc46f21047c139a1648e03fded5625
link
/test e2e_operator

gencheck_operator
b22483e77bdd1d5cc2592d38717a4eff811bea29
link
/test gencheck_operator

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@sdake: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

gencheck_operator
b22483e77bdd1d5cc2592d38717a4eff811bea29
link
/test gencheck_operator

e2e_operator
b22483e77bdd1d5cc2592d38717a4eff811bea29
link
/test e2e_operator

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@sdake: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e_operator
b22483e77bdd1d5cc2592d38717a4eff811bea29
link
/test e2e_operator

gencheck_operator
1baa6bae8d7f2a48f7473648b6712b5e8566bbdd
link
/test gencheck_operator

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@sdake: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

gencheck_operator
1baa6bae8d7f2a48f7473648b6712b5e8566bbdd
link
/test gencheck_operator

e2e_operator
1baa6bae8d7f2a48f7473648b6712b5e8566bbdd
link
/test e2e_operator

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test e2e_operator

@sdake: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

gencheck_operator
1baa6bae8d7f2a48f7473648b6712b5e8566bbdd
link
/test gencheck_operator

e2e_operator
1baa6bae8d7f2a48f7473648b6712b5e8566bbdd
link
/test e2e_operator

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@sdake: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e_operator
1baa6bae8d7f2a48f7473648b6712b5e8566bbdd
link
/test e2e_operator

gencheck_operator
ad0d54fa66643a49615b94767b27b9c3349705c6
link
/test gencheck_operator

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@sdake: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

gencheck_operator
ad0d54fa66643a49615b94767b27b9c3349705c6
link
/test gencheck_operator

e2e_operator
ad0d54fa66643a49615b94767b27b9c3349705c6
link
/test e2e_operator

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Hi @sdake, will you be able to make progress on this? Otherwise we should find another owner since it's quite critical.

@ostromart looks like @irisdingbj will take this over.  Note my original work was passing in my local environment and also prow.  There is some minor issue - possibly related to the usage of metallb - but possibly something else.
I'll take a look today after my morning meetings finish up.
Cheers
-steve

subsumed by work from @irisdingbj .


add overlay in https://readthedocs.io/

Part of #14

DoD: Add to the README

Hey,
I am trying to tackle this one, but readthedocs package pages may be npm/pypi.
The current flow as I get it takes the type (npm/pypi) but it is not always shown in readthedocs URLs.
Any suggestions?

@Oferlis This issue is not described well.
There are two types of adding Overlay to ReadTheDocs:

Identifying the current page and adding the Overlay details as we do in npm.org and PyPI.
Find links to ReadTheDocs on other websites and add the indicator to them.

For those two options, we need first to collect examples of both, then do some research to understand what you said about different package ecosystems, different URLs, different page structures, and so on.


Update SDCard module to use Tock 2.0 system calls.

This has not been tested as no boards have an SDCard.

bors r+


Auto play plugin broken with the new update for Vue 3

Description
After the recent Vue3 update, it renders pretty well on my side with the latest version of Vue. However, the autoplay plugin for this instance only does one tick before it stops. i don't see any console errors so I'm assuming it is something with this plugin for Vue3.
Steps to check or reproduce
Attaching a codesandbox template that I replicated the issue on.
https://codesandbox.io/s/holy-morning-5otoz?file=/src/App.vue

Hello @megasanjay!
I've found out that AutoPlay is broken while I'm fixing the https://github.com/naver/egjs-flicking/pull/523
So basically it requires Flicking to be updated to fix this issue.
I'll do it asap :)

Fixed this issue in Flicking v4.2.5, please try that :)


Question: Drop shadow to the IndicatorView

Sorry for this question, but am i able to add a drop shadow to the indicator view? I believe it should be possible since the indicatorView is an open subclass of UIView. Ive tried using CALayer properties but to no avail. if it is possible, can anyone give me any guidance? thank you

hey @akeem554 , there was a bug that was fixed with https://github.com/gmarm/BetterSegmentedControl/pull/121.
It's possible to achieve that now. You can check the appleStyled class method on the control for an example.


Unable to create an AMV unless sort key is the first field

While creating AMV seeing different errors related to partition.
Steps to reproduce the behavior (Required)
Running:
create materialized view `order_products_amv`
PARTITION BY date_trunc('day', `created_ts`)
DISTRIBUTED BY HASH(`order_ref_num`) BUCKETS 18
refresh MANUAL
ORDER by (product_id)
AS
select
order_ref_num,
selling_site_id,
created_ts,
`line_item_id`,
`product_id`
from
order_state  osj,

Expected behavior (Required)
Success
Real behavior (Required)
SQL Error [1064] [42000]: Getting analyzing error from line 3, column 14 to line 3, column 42. Detail message: Materialized view partition function date_trunc check failed: date_trunc('day',  selling_site_id)

OR
ERROR 1064 (HY000): Getting analyzing error from line 2, column 14 to line 2, column 44. Detail message: Resolve partition column failed:null

StarRocks version (Required)
3.1.5

verified in main branch (0320), already fixed.


Add GATK-SV

Moving the initial gatk_sv workflow from the older branch https://github.com/populationgenomics/production-pipelines/tree/gatk-sv and updating according to the current cpg-utils/workflows package.
How to run it on the validation dataset:
gatk_sv/gatk_sv.py configs/seqr.toml configs/genome.toml configs/validation.toml configs/gatk-sv.toml configs/test.toml
(should be submitted as --dataset seqr --access-level test or --dataset validation --access-level test).
I tested only the first stage GatherSampleEvidence (I limit to this stage on line 447 get_workflow().run(stages=[GatherSampleEvidence])) and it worked.

hmm gatk_sv.py is getting really big (and will only get much bigger from here). Should we move the stages into separate files?

@vivbak let's start merging working parts of gatk-sv workflow. Do you mind if I create a new PR with GatherSampleEvidence and EvidenceQC stages only (since they already worked successfully), and we can work on other stages in separate PRs?

sounds good!

@vivbak please review https://github.com/populationgenomics/production-pipelines/pull/115 then


Move tests out of core

This PR does a couple different test-related things in an attempt to make it more approachable to write new tests (because we need better coverage :grin:):

Moves tests into __tests__/ subdirectories that are siblings of the modules that they test (like jest).
Makes page class paths passed to startServerBeforeAll relative to the spec file, rather than relative to a parent directory. (It requires using __filename in the spec, which I don't like, but I couldn't figure out how to get jasmine to tell me which spec was running, and I dislike this less than I dislike having paths be relative to non-obvious things.)
Adds some better error handling around the webpack build part of the tests.
Updates the integration test documentation

Going to merge to unblock more test-writing. Other feedback can be addressed in new PRs


Use localhost for notification server

Fixes https://github.com/bisq-network/btcd-cli4j/issues/5

Is this configurable? #5 discusses this. If you run in docker containers btcd and seednode run on different hosts. Please give some more context on how you interpret #5 and how this PR would work with docker ok installations

Is this configurable? #5 discusses this. If you run in docker containers btcd and seednode run on different hosts. Please give some more context on how you interpret #5 and how this PR would work with docker ok installations

Doesn't look easy to make configurable, best would be if you could forward the port to the docker container I think.

@mrosseel Feel free to provide a PR for supporting an option for the host.


Tab character ignored in signatureHelpTriggerCharacters

monaco-editor version: 0.10.1
Browser: chrome/electron
OS: W10
Steps or JS usage snippet reproducing the issue:
monaco.languages.register({id:'mine'})
monaco.languages.registerSignatureHelpProvider('mine', {
    provideSignatureHelp: function (model, position) {
         return {
            activeSignature: 0,
            activeParameter: 0,
            signatures: [{label: 'handler triggered', parameters: [{label: 'triggered', documentation: 'tab not working'}]}]
          }
     },
    signatureHelpTriggerCharacters:[' ', '\t', '\n']
})
monaco.editor.create(document.getElementById("container"), {
	value: "something",
	language: "mine"
});

space and enter work, tab does not

Tab is not programmed to do anything at this point when having parameter hints.

We closed this issue because we don't plan to address it in the foreseeable future. If you disagree and feel that this issue is crucial: we are happy to listen and to reconsider.
If you wonder what we are up to, please see our roadmap and issue reporting guidelines.
Thanks for your understanding, and happy coding!


Cron hpa sample

Ⅰ. Describe what this PR does
Ⅱ. Does this pull request fix one issue?

Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews

Codecov Report

Merging #701 (0c63f90) into master (af4e7d4) will increase coverage by 0.22%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #701      +/-   ##
==========================================
+ Coverage   16.53%   16.76%   +0.22%
==========================================
  Files          68       68
  Lines        2993     3006      +13
==========================================
+ Hits          495      504       +9
- Misses       2447     2450       +3
- Partials       51       52       +1

Impacted Files
Coverage Δ

pkg/ddc/alluxio/transform_optimization.go
78.46% <0.00%> (-0.36%)
:arrow_down:

pkg/ddc/alluxio/transform.go
0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update af4e7d4...0c63f90. Read the comment docs.


Readme should have a minimal "Getting Started" section

To get people quickly going with a library, it's useful to have a very minimal getting started section as the first or second thing visible on the readme. This way people don't have to hunt around trying to find the simplest example to get going.

That's true. I'll keep it open.

How about now? What do you think?

Looks good, this will certainly help people get slog up and running much easier

Would be nice to have something more elaborated, but that as much as I can realistically invest in it at this point. :)


⚠ 한경주요지수-코스피200 has degraded performance

In 169ca81, 한경주요지수-코스피200 (https://markets.hankyung.com/indices/kospi200) experienced degraded performance:

HTTP code: 200
Response time: 5570 ms

Resolved: 한경주요지수-코스닥 performance has improved in 396ba81.


Update AutoRest C# version

Update AutoRest C# version to 1.0.0-alpha.20201021.1

This update brings support for node 14


Extra "level" in TOC

The TOC current is structured as "Visual F#" / "Visual F# Development Portal" / "... all content ... "
Can we eliminate one (or both) of the two "top level" elements of the tree? Having to click through both adds no value.

Development portal has content in it, so does the Visual F# page. Don't think it's something that we can get rid off.
Would love to have @KevinRansom @dsyme to chime in on this.

There's a lot of duplication in there - it seems like "Development Portal"
could easily be folded into "Visual F#" to make one page,and eliminate an
entire "level" of the tree.
On Wed, May 25, 2016 at 12:44 PM, Den Delimarsky<EMAIL_ADDRESS>wrote:

Development portal has content in it, so does the Visual F# page. Don't
think it's something that we can get rid off.
Would love to have @KevinRansom https://github.com/KevinRansom @dsyme
https://github.com/dsyme to chime in on this.
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/Microsoft/visualfsharpdocs/issues/92#issuecomment-221685166

I believe both top-level nodes in the TOC provide value, but I do think that this child structure is a bit odd.  I think it would be good to flatten the TOC:
Visual F#/
Visual F# Development Portal/
Visual F# Guided Tour/
...

This is why I think that.
The "Visual F#" top-level TOC item feels like more of a "walkthrough of cool things about F#".  This is useful information, especially if people want some quick motivation for why they might pick F#.  I don't see that as being a parent of docs, though.
The Development Portal is a great central place that allows you to hop to various parts of the docs depending on the high level task.  I don't see this as being a parent to docs, either.  Although it's docs-related, it's really more of an organized set of groups that allow you to jump to various sections without having to hunt things down in the TOC.

I wouldn't mind @cartermp 's option here - my main "gripe" is that it's adding an extra parent in the hierarchy to everything, which makes browsing odd - things get indented in the tree one extra time, which seems unnecessary.
Having it as the top page, but at the same level in hierarchy as "Guided Tour" makes sense.

I would be somewhat against merging them into one, but I would be OK with "flattening" the structure, similarly to what @cartermp suggested.

Flattening would be good with me - I'm not sure how this would be done, however, given the files in the repo... I suspect it's something outside of the community content.

AFAIK it can be done just by editing the TOC file.  All doc files are in the same repo, so the relative links should still all work.

Structure-wise this is really a TOC change, as long as you don't touch the locations of the files.


Question:

I can't publish my osTv application, I have the following error:

This bundle is invalid. The key UIRequiredDeviceCapabilities in the Info.plist may not contain values that would prevent this application from running on devices that were supported by previous versions.

Can solve

Solution: change the UIRequiredDeviceCapabilities key in info.plist to arm64


Confdocs can't parse array examples correctly

Given this configuration:
Compression []string `yaml:"compression" conf:",example=['zstd', 'snappy', 'none']"`

The parser stops at the first comma and produces an incomplete example.
compression: ['zstd' # Compression sets the compression algorithm to use in order of priority. Valid values are "none", "gzip", "snappy","lz4", "zstd". Default is ["snappy", "none"].

I wonder if https://github.com/matt1484/spectagular might be useful for this.


version mismatch langhchan-openai

🐛 Describe the bug
With new version of langchain-openai (^0.1.6)`, embedchain is not working
Because no versions of embedchain match >0.1.103,<0.2.0
and embedchain (0.1.103) depends on langchain-openai (>=0.0.5,<0.0.6), embedchain (>=0.1.103,<0.2.0) requires langchain-openai (>=0.0.5,<0.0.6).
So, because giospatial-tl-report depends on both langchain-openai (^0.1.6) and embedchain (^0.1.103), version solving failed.

Thanks @amjadraza for creating the issue. Will fix it in few hours.

Still the same problem, any updates?
Otherwise I will be forced to remove from my project all the parts that includes embedchain, it looks like a stillborn library.

Sorry for the delay guys. We have been quite swamped with a lot of other stuff. Working on a fix right now. Will put up a PR soon.


Dockerfile with hrequest

I'm building a container using the hrequest library, however, when I try to use the function get, it doesn't work and stops. Thanks for your help!

Hello @mir3u ,
By any chance could you provide me with a dockerfile (to build an image) where hrequests works ?
I have been trying to use many images but always get errors.


Provide visual examples

In the "Options" chapter, it'd be great to actually be able to see the results
Add a distinct example (screenshot) to the README for a good 1st impression

@jhermann Those are great suggestions, thanks! I'll try to add some screenshots in the next couple of days.


Remove ContractSetStore

The ContractSetStore interface is redundant now that we add a many-to-many relationship with the contract store. Having the two interfaces does not really make sense since the ContractStore interface references contract set names, so I removed it.

waiting on #74

no need to review - need to make some minor changes now that #74 is merged


load mj-include templates from http url with mrml-wasm

What is the state of our support for mj-include? Have we ironed out the potential security issues, WASM support and other considerations?
If so, I'd like to see what work is left and I can take it on.
If not, I'd like to work together to flesh out what decision points we have left


fix docs authentication section links

Thanks for contributing. I was in the middle of a big typescript rewrite that took longer than expected...
This is now fixed!


Official website of the institution - https://uniao.ugv.edu.br/

Hello, I hope you are well.
I am an academic on the software engineering course at UGV - Universidade Grande Vale. I will also leave the institution's official website archived next to this comment.
https://uniao.ugv.edu.br/
Thanks

@MateusFKrinski I have to close this request without merging, because you requested to add a non-existent domain to this repository: Host ugv.edu not found: 3(NXDOMAIN) Please ensure that you are familiar with this repository rules located at the bottom of this page and submit a new request with the correct information. Thank you for your understanding!


[Question] Can I download wav file (not mp3 file) from youtube link?

Checklist

[ ] I'm asking a question
[ ] I've looked through the README and FAQ for similar questions
[ ] I've searched the bugtracker for similar questions including closed ones

Question

As mentioned title, Can I download wav file (not mp3 file) from youtube link??
I wanna download wav file (and 16k sample rate) from youtube link. Can I?
Thank You!!

Youtube does not provide wav.

Youtube does not provide wav.


Allow batch creation of jobs

Create SDB items representing the job graph with a batch insertion, to avoid large numbers of transactions when there are many child jobs.

Ready for review.

Thank you @adderan


Path support for image annotator

Is your feature request related to a problem? Please describe
I'm planning to build a generic annotator app for images. Currently, wave only supports bounding box annotation (rectangle). For the segmentation part, there is no way to annotate masks. Introducing that in the image annotator could be a great addition.
Describe the solution you'd like
path support (polygon) can be introduced to support annotating segmentation masks.
cc @mturoci

Resolved by https://github.com/h2oai/wave/pull/1643


How to change moment locale to "th" in chart options?

Please ask questions on StackOverflow: https://stackoverflow.com/questions/tagged/chart.js


Execute tests and pull screenshots

:tophat: What is the goal?
Continue shot development adding a new feature to be able to pull the screenshot from the Android device connected to our computer.
If your review this PR you'll see how the paradigm used is OOP and not FP. This is because I'd like to use this library named Freestyle but I couldn't get Scala macros working on Gradle.
How is it being implemented?
The steps followed to implement this feature are:

Add a Scala code formatter and integrate it into our TravisCI configuration. You can find more information about scala fmt here.
Create two main abstractions we will use for executing adb commands and interact with the view interface by printing a message in the standard output. The abstractions names are Adb and Console.
Connect our ShotPlugin and different tasks with a new domain model named Shot using Adb and Console as collaborators.
Implement all the logic needed to clean and pull screenshots from the connected Android device.
Create a new extension needed to get the application package easily. This could be improved in the future if needed.
Add some libraries needed to mock dependencies in our unitary tests

How can it be tested?
Clone this branch and execute these commands:
./gradlew install
cd shot-consumer
./gradlew executeScreenshotTests

You should see a new folder inside shot-consumer/app/screenshots/screenshots-default with a bunch of png files inside and some metadata files.
Next tasks?

Refactor part of the system initialization to use cake pattern.
Start working on the error handling to be able to break a task execution when needed by returning an error modeled using Either and sealed traits.
Start comparing images and getting the difference between every pair of screenshots. To be able to do this we will have to parse the metadata file we've pulled from the connected device.

I've just read the code "in diagonal" 'cause my Scala knowledge is pretty rusty but it's looking great. I've also run the tests, just out of curiosity and why do we need different Gradle versions?
Installing the Shot library I saw this:
Downloading https://services.gradle.org/distributions/gradle-3.1-bin.zip
but running the screenshot test I got this instead:
Downloading https://services.gradle.org/distributions/gradle-2.8-all.zip
Could those be the same?

@fjfdeztoro the issue related to the different Gradle version has to be related to the plugins we are using. They may require different Gradle plugins. This won't affect the final user because this is just needed for development purposes. Could you paste here the command you executed?

It could also be related to the Android project versioned inside the project and used as a plugin consumer. But that sounds weird because the project is not linked as a root Gradle module :S


[RFC]: add stats/incr/mkurtosis

Moving incremental kurtosis. See also @stdlib/stats/incr/kurtosis for the non-moving version.
For similar packages, see https://github.com/stdlib-js/stdlib/tree/develop/lib/node_modules/%40stdlib/stats/incr.

@kgryte I'd like to work on this issue. I was wondering if you could provide some guidance on how to get started with it or perhaps point me to a similar implementation for reference?

@headlessNode A couple of links:

Non-moving version: https://github.com/stdlib-js/stdlib/tree/develop/lib/node_modules/%40stdlib/stats/incr/kurtosis.
Example of moving implementation where we calculate the moving variance: https://github.com/stdlib-js/stdlib/tree/develop/lib/node_modules/%40stdlib/stats/incr/mvariance
Non-moving version for calculating the variance: https://github.com/stdlib-js/stdlib/blob/develop/lib/node_modules/%40stdlib/stats/incr/variance/lib/main.js

As can be inferred from the variance implementations, the key thing to figure out are the update equations. Once a window is filled, for each new value, the oldest value must be removed.
Similar to mvariance, we should support providing an optional known mean parameter. When providing a known mean, this also affects the update equations, as one no longer needs to compute the mean at the same time as the kurtosis.

@kgryte I've gone through the variance and mvariance code. I understand the implementations about both the accumulator1 and accumulator2, in case of both variance and mvariance.
From what I understood, the need for accumulator2 arises due to presence of optional mean parameter. So, in case of mkurtosis, we'd have both the accumulator1 and accumulator2.
I will now go through the kurtosis code, but I wanted to confirm what you mean by the update equations when you say that the key thing to figure out are the update equations? Do you mean the part of the code where the moment, mu, and delta are being calculated?

@kgryte Does this mean that we need to, first,  derive equations for implementation of moving kurtosis? If so, I'm willing to do it, but I would need a starting point and some guidance to ensure I'm on the right track.

You may; TBH, I am not sure. I'd need to do the R&D. I suggest consulting the reference literature linked to in the incr/kurtosis package and starting from there.

@kgryte First of all I want apologize for my recursive inquiries. And Thank you for your patience and guidance.
I was looking through other similar issues, mainly incr/wvariance and following the literature in #46 , I stumbled across some literature which were deriving numerically stable equations for weighted variance. I also found the equations for incr/variance and some other methods.
Although I haven't found something for mkurtosis yet, that is what you meant here right? Finding more numerically stable equations for our implementation of the methods?

Yes, it may be quite likely you'd need to derive the update equations yourself. IIRC, I remember sitting down with pen and paper when originally implementing kurtosis and skewness.


fix: replace library for FUSE support

This commit refactors the internals of proxy/fuse to use
github.com/hanwen/go-fuse/v2 instead of bazil.org/fuse, partly in
pursuit of better macOS support.
Fixes #937.

I tested this manually on macOS Big Sur (Apple M1 & x86) and confirmed it worked in both cases.
The FUSE mount will fail to create a Unix socket if the path exceeds the system limit (see https://github.com/golang/go/issues/6895) and the resulting error isn't very friendly (bind: invalid argument), but I'll fix that on a PR after this one.
Also, the Kokoro build for Go 1.16 on mac is stalled, because the installation of osxfuse requires a restart. We'll have to figure out how to work with that problem in a future change, too.

As a side note, installing macfuse on M1 machines is quite an undertaking. After using brew install macfuse, I had to restart into recovery mode, allow kernel extensions, restart again, and only then would the proxy start in fuse mode.


JIRA DRILL-5879: Like operator performance improvements

Recently, custom code has been added to handle common search patterns (Like operator)
Contains, Starts With, and Ends With
Custom code is way faster than the generic RegEx based implementation
This pull request is another attempt to improve the Contains pattern since it is more CPU intensive

Query: select  from <table> where colA like '%a%' or colA like '%xyz%';
Improvement Opportunities
Avoid isAscii computation (full access of the input string) since we're dealing with the same column twice
Optimize the "contains" for-loop
Implementation Details
1)
Added a new integer variable "asciiMode" to the VarCharHolder class
The default value is -1 which indicates this info is not known
Otherwise this value will be set to either 1 or 0 based on the string being in ASCII mode or Unicode
The execution plan already shares the same VarCharHolder instance for all evaluations of the same column value
The asciiMode will be correctly set during the first LIKE evaluation and will be reused across other LIKE evaluations
2)
The "Contains" LIKE operation is quite expensive as the code needs to access the input string to perform character based comparisons
Created 4 versions of the same for-loop to a) make the loop simpler to optimize (Vectorization) and b) minimize comparisons
Benchmarks
Lineitem table 100GB
Query: select l_returnflag, count from dfs.<source> where l_comment not like '%a%' or l_comment like '%the%' group by l_returnflag
Before changes: 33sec
After changes : 27sec

@sachouche Do you have a breakdown of how much gain we got with 1 vs 2.  Since the changes for 2 are not straightforward and easy to maintain, I am thinking  performance gain vs. maintainability of the code.

@sachouche, thanks for the first PR to Drill! Thanks for the detailed explanation!
Before reviewing the code, a comment on the design:

Added a new integer variable "asciiMode" ... this value will be set ... during the first LIKE evaluation and will be reused across other LIKE evaluations

The problem with this design is that there is no guarantee that the first value is representative of the other columns. Maybe my list looks like this:
Hello
你好

The first value is ASCII. The second is not. So, we must treat each value as independent of the others.
On the other hand, we can exploit the nature of UTF-8. The encoding is such that no valid UTF-8 character is a prefix of any other valid character. Thus, if a character is 0xXX 0xYY 0xZZ, then there can never be a valid character which is 0xXX 0xYY. As a result, starts-with, ends-width, equals and contains can be done without either converting to UTF-16 or even caring if the data is ASCII or not.
What does this mean? It means that, for the simple operations:

Convert the Java UTF-16 string to UTF-8.
Do the classic byte comparison methods for starts with, ends with or contains. No special processing is needed for multi-byte

Unlike other multi-byte encodings, UTF-8 was designed to make this possible.
If we go this route, we would not need the ASCII mode flag.
Note: all of this applies only to the "basic four" operations: if we do a real regex, then we must decode the Varchar into a Java UTF-16 string.

It was 50% for 1) and 50% for 2)
Notice this breakdown depends on
o The number of Contains pattern for the same value (impacts 1))
o The pattern length (impacts both 1) and 2))

Paul,

I think you misunderstood the proposal
Let me use an example
select .. c1 like '%pattern1%' OR c1 like '%pattern2'..
Assume c1 has 3 values [v1, v2, v3]
The generated code will create a new VarcharHolder instance for each value iteration
For v1: VarCharHolder vc1 is created, ascii-mode computed for pattern1, ascii-mode computation reused for pattern2, pattern3, etc since we're evaluating the same value
For v2: VarCharHolder vc1 is created, ascii-mode computed for pattern1, ascii-mode computation reused for pattern2, pattern3, etc since we're evaluating the same value
DITO for v3

Note that the test-suite has similar test cases that you are proposing; they all passed.

Paul, again thanks for the detailed review:

I was able to address most of the feedback except for one
I agree that expressions that can operate directly on the encoded UTF-8 string should ideally perform  checks on bytes and not characters
Having said that, such a change is more involved and should be done properly
o The SqlPatternContainsMatcher currently gets a CharSequence as input
o We should enhance the expression framework so that matchers can a) express their capabilities and b) receive the expected data type (Character or Byte sequences)
o Note also there is an impact on the test-suite since StringBuffer are being used to directly test the matcher functionality

@ppadma @paul-rogers I see that @sachouche addressed the comments in the JIRA - is this one ready to merge?

@priteshm @sachouche This PR needs to be updated on top of changes made for DRILL-5899.