After removing pvc ctsorevolumereplica's are still there or having error, how to remove them?

I just want to have a way to remove them, so that the black-devices become Unclaimed.
Description
run kubectl delete -f pvc.yaml to delete pvcs.
For some unknown reason, There are still cstorevolumereplicas:
kubectl get cvr -A
NAMESPACE     NAME                                                             USED   ALLOCATED   STATUS     AGE
kube-system   pvc-0a024c2c-8924-4eb2-875a-dc6ec0f0b071-cstor-raidz-pool-s1hk   6K     7.99K       Recreate   179m
kube-system   pvc-1bdce479-17fb-48ac-9ae2-56c3bc868e64-cstor-raidz-pool-s1hk   6K     7.99K       Recreate   3h1m
kube-system   pvc-581f5c8e-5493-4ee9-a5b8-8f9a747d1c9d-cstor-raidz-pool-iuy0   6K     7.99K       Error      178m
kube-system   pvc-581f5c8e-5493-4ee9-a5b8-8f9a747d1c9d-cstor-raidz-pool-s1hk   6K     7.99K       Recreate   178m
kube-system   pvc-581f5c8e-5493-4ee9-a5b8-8f9a747d1c9d-cstor-raidz-pool-yo22   6K     7.99K       Error      178m
kube-system   pvc-b25bd863-dd7d-4d59-9e97-24c9d6a361f3-cstor-raidz-pool-iuy0   6K     7.99K       Error      177m
kube-system   pvc-b25bd863-dd7d-4d59-9e97-24c9d6a361f3-cstor-raidz-pool-s1hk   6K     7.99K       Recreate   177m
kube-system   pvc-b25bd863-dd7d-4d59-9e97-24c9d6a361f3-cstor-raidz-pool-yo22   6K     7.99K       Error      177m
kubectl get pvc -A
No resources found
kubectl get csp -A
NAME                    ALLOCATED   FREE    CAPACITY   STATUS           READONLY   TYPE    AGE
cstor-raidz-pool-s1hk   6.09M       2.95G   2.95G      DeletionFailed   false      raidz   3h9m
kubectl get bd -A
NAMESPACE     NAME                                           NODENAME       SIZE          CLAIMSTATE   STATUS   AGE
kube-system   blockdevice-21f0a80c8ac4319bcb33d5b5f0c28c2d   k8s-worker-0<PHONE_NUMBER>    Claimed      Active   5d4h
kube-system   blockdevice-3036a92092f6669e43d8a1d47a228c22   k8s-worker-2<PHONE_NUMBER>    Claimed      Active   4d23h
kube-system   blockdevice-39f43a65cc1bb7de0e3bdbd8c2a30238   k8s-worker-2<PHONE_NUMBER>    Claimed      Active   4d23h
kube-system   blockdevice-4b6215923c81883ff5b2eba02d411ff0   k8s-worker-1<PHONE_NUMBER>    Claimed      Active   5d4h
kube-system   blockdevice-4bdbdd33de7790189c08e83381518a20   k8s-worker-0<PHONE_NUMBER>    Claimed      Active   5d4h
kube-system   blockdevice-84cd05e0a5b74db0a0bf8703c7a4c83b   k8s-worker-1<PHONE_NUMBER>    Claimed      Active   5d4h
kube-system   blockdevice-94ec16a3253cccd0da373a0ff2fba4b3   k8s-worker-1<PHONE_NUMBER>    Claimed      Active   5d4h
kube-system   blockdevice-a51689191ae43ca219067deed3a5549d   k8s-worker-0<PHONE_NUMBER>    Claimed      Active   5d4h
kube-system   blockdevice-faef7c0501d641b8dbe09f1ee8474831   k8s-worker-2<PHONE_NUMBER>    Claimed      Active   4d23h
#s# cat sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
#allowVolumeExpansion: false
metadata:
name: openebs-cstor-sc
annotations:
openebs.io/cas-type: cstor
cas.openebs.io/config: |
- name: StoragePoolClaim
value: "cstor-raidz-pool"
- name: ReplicaCount
value: "3"
provisioner: openebs.io/provisioner-iscsi
Expected Behavior
csp and cvr should be removed
Current Behavior
csp and cvr are not removed

Hi @kaitaklam, Looks like tearing down the order is not followed. Can you list the steps performed to Teardown pools and volumes? Basically we need to follow:

Delete PVC and wait till all the CStorVolume related resources to teardown(delete entries from etcd).
Delete the SPC(StoragePoolClaim).

I thinks I made the mistake that I didn’t check CStorVolume was removed after delete pvc.
I was forgotten to wait for CStorVolume Were completely removed and started deleting StorageClass and StoragePoolClaim.
My question is more, how to recover from this situation …
From: sai chaithanya<EMAIL_ADDRESS>Sent: Monday, June 29, 2020 3:53 PM
To: openebs/openebs<EMAIL_ADDRESS>Cc: Kai Lam<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: [E] Re: [openebs/openebs] After removing pvc ctsorevolumereplica's are still there or having error, how to remove them? (#3071)
[EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe.
Hi @kaitaklamhttps://github.com/kaitaklam, Looks like tearing down the order is not followed. Can you list the steps performed to Teardown pools and volumes? Basically we need to follow:

Delete PVC and wait till all the CStorVolume related resources to teardown(delete entries from etcd).
Delete the SPC(StoragePoolClaim).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/openebs/openebs/issues/3071#issuecomment-651136467, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AM6WJTEAEDUBPMLBINLBDEDRZCMEJANCNFSM4OLGKYPQ.

This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to<EMAIL_ADDRESS>This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You

My question is more, how to recover from this situation …

We can edit the CVR(CStorVolumeReplica) and remove the finalizers under metadata and it will take care of deleting from etcd.

Can I know the use case that you are performing?

Here is the use case – 3 worker-nodes with 3 disks:
kubectl get bd -A
NAMESPACE     NAME                                           NODENAME       SIZE          CLAIMSTATE   STATUS   AGE
kube-system   blockdevice-21f0a80c8ac4319bcb33d5b5f0c28c2d   k8s-worker-0<PHONE_NUMBER>    Claimed      Active   5d5h
kube-system   blockdevice-3036a92092f6669e43d8a1d47a228c22   k8s-worker-2<PHONE_NUMBER>    Claimed      Active   4d23h
kube-system   blockdevice-39f43a65cc1bb7de0e3bdbd8c2a30238   k8s-worker-2<PHONE_NUMBER>    Claimed      Active   4d23h
kube-system   blockdevice-4b6215923c81883ff5b2eba02d411ff0   k8s-worker-1<PHONE_NUMBER>    Claimed      Active   5d5h
kube-system   blockdevice-4bdbdd33de7790189c08e83381518a20   k8s-worker-0<PHONE_NUMBER>    Claimed      Active   5d5h
kube-system   blockdevice-84cd05e0a5b74db0a0bf8703c7a4c83b   k8s-worker-1<PHONE_NUMBER>    Claimed      Active   5d5h
kube-system   blockdevice-94ec16a3253cccd0da373a0ff2fba4b3   k8s-worker-1<PHONE_NUMBER>    Claimed      Active   5d5h
kube-system   blockdevice-a51689191ae43ca219067deed3a5549d   k8s-worker-0<PHONE_NUMBER>    Claimed      Active   5d5h
kube-system   blockdevice-faef7c0501d641b8dbe09f1ee8474831   k8s-worker-2<PHONE_NUMBER>    Claimed      Active   4d23h
kube-system   sparse-2aae5c1ae32e2f533e072ffc87ed7b5c        k8s-worker-2<PHONE_NUMBER>0   Unclaimed    Active   4d23h
kube-system   sparse-aa73ef381020b82f33a01f40571d046d        k8s-worker-0<PHONE_NUMBER>0   Unclaimed    Active   5d5h
kube-system   sparse-ecaf82cb369be323ada118141ff21ff3        k8s-worker-1<PHONE_NUMBER>0   Unclaimed    Active   5d5h
cat spc.yaml
apiVersion: openebs.io/v1alpha1
kind: StoragePoolClaim
metadata:
name: cstor-raidz-pool
annotations:
cas.openebs.io/config: |
- name: PoolResourceRequests
value: |-
memory: 2Gi
- name: PoolResourceLimits
value: |-
memory: 4Gi
spec:
name: cstor-raidz-pool
type: disk
poolSpec:
poolType: raidz
thickProvisioning: true
blockDevices:
blockDeviceList:
- blockdevice-21f0a80c8ac4319bcb33d5b5f0c28c2d
- blockdevice-3036a92092f6669e43d8a1d47a228c22
- blockdevice-39f43a65cc1bb7de0e3bdbd8c2a30238
- blockdevice-4b6215923c81883ff5b2eba02d411ff0
- blockdevice-4bdbdd33de7790189c08e83381518a20

blockdevice-84cd05e0a5b74db0a0bf8703c7a4c83b
- blockdevice-94ec16a3253cccd0da373a0ff2fba4b3
- blockdevice-a51689191ae43ca219067deed3a5549d
- blockdevice-faef7c0501d641b8dbe09f1ee8474831

cat sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
#allowVolumeExpansion: false
metadata:
name: openebs-cstor-sc
annotations:
openebs.io/cas-type: cstor
cas.openebs.io/config: |
- name: StoragePoolClaim
value: "cstor-raidz-pool"
- name: ReplicaCount
value: "3"
provisioner: openebs.io/provisioner-iscsi
cat pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: cstor-pvc-mysql-large
spec:
storageClassName: openebs-cstor-sc
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1950Mi
From: sai chaithanya<EMAIL_ADDRESS>Sent: Monday, June 29, 2020 4:02 PM
To: openebs/openebs<EMAIL_ADDRESS>Cc: Kai Lam<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: [E] Re: [openebs/openebs] After removing pvc ctsorevolumereplica's are still there or having error, how to remove them? (#3071)
[EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe.
My question is more, how to recover from this situation …

We can edit the CVR(CStorVolumeReplica) and remove the finalizers under metadata and it will take care of deleting from etcd.

Can I know the use case that you are performing?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/openebs/openebs/issues/3071#issuecomment-651142704, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AM6WJTDNY6O5GA5QKZZYLFTRZCNGJANCNFSM4OLGKYPQ.

This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to<EMAIL_ADDRESS>This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You

Problem can be fixed
kubectl -n kube-system edit cvr  pvc-b25bd863-dd7d-4d59-9e97-24c9d6a361f3-cstor-raidz-pool-yo22
remove the flinalizers under metadata
Some for  kubectl edit csp cstor-raidz-pool-s1hk
remove the flinalizers under metadata

@kaitaklam, Can we give a try(provisioning) using CSPC(enhanced version of SPC) and CSI volumes. With CSPC and CSI support Day-2-operations(disk replacement, volume expansion, pool expansion) can be achieved easily.

For creating CSPC based pools you can follow https://github.com/openebs/cstor-operators#project-status
For creating CSI volumes you can follow https://github.com/openebs/cstor-csi

Hi Sai,
I have 3 raidz pools:
NAME                    ALLOCATED   FREE    CAPACITY   STATUS    READONLY   TYPE    AGE
cstor-raidz-pool-a2jf   184K        2.95G   2.95G      Healthy   false      raidz   12m
cstor-raidz-pool-c0vy   184K        2.95G   2.95G      Healthy   false      raidz   12m
cstor-raidz-pool-hq8a   184K        2.95G   2.95G      Healthy   false      raidz   12m
According to https://docs.openebs.io/docs/next/ugcstor.html#deleting-a-cStor-volume 1 disk for parity and 2 disks are used for data.
So I assume that it should allow me to claim 2 x 2.95 GB = 5.9 GB
But the pvc claim failed with 5.5 GB
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: cstor-pvc-mysql-large
spec:
storageClassName: openebs-cstor-sc
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5500Mi
From: sai chaithanya<EMAIL_ADDRESS>Sent: Monday, June 29, 2020 4:33 PM
To: openebs/openebs<EMAIL_ADDRESS>Cc: Kai Lam<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: [E] Re: [openebs/openebs] After removing pvc ctsorevolumereplica's are still there or having error, how to remove them? (#3071)
[EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe.
@kaitaklamhttps://github.com/kaitaklam, Can we give a try(provisioning) using CSPC(enhanced version of SPC) and CSI volumes. With CSPC and CSI support Day-2-operations(disk replacement, volume expansion, pool expansion) can be achieved easily.

For creating CSPC based pools you can follow https://github.com/openebs/cstor-operators#project-status
For creating CSI volumes you can follow https://github.com/openebs/cstor-csi

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/openebs/openebs/issues/3071#issuecomment-651160844, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AM6WJTHR5W2OXTBXK5WHL43RZCQ2RANCNFSM4OLGKYPQ.

This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to<EMAIL_ADDRESS>This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You

NAME                    ALLOCATED   FREE    CAPACITY   STATUS    READONLY   TYPE    AGE
cstor-raidz-pool-a2jf   184K               2.95G   2.95G           Healthy      false               raidz     12m
According to https://docs.openebs.io/docs/next/ugcstor.html#deleting-a-cStor-volume 1 disk for parity and 2 disks are used for data.
So I assume that it should allow me to claim 2 x 2.95 GB = 5.9 GB

Here, FREE represents available space without negotiating parity disk space(parity is distributed across all the disks). If we attach 3 1GB disks to the node then Total usable Capacity will be ~2GB because 1GB is for parity. So we can't provision volume having a capacity greater than 2G.
We fixed this kind of usability issue via CSPC in PR.

Here, FREE represents available space without negotiating parity disk space(parity is distributed across all the disks). If we attach 3 1GB disks to the node then Total usable Capacity will be ~2GB because 1GB is for parity. So we can't provision volume having a capacity greater than 2G.
[KL] I agree that is per node, I get ~2G. But if I have 3 nodes. Wouldn’t the cStor pool adds up 3 node x ~2G = 6 G?
From: sai chaithanya<EMAIL_ADDRESS>Sent: Monday, June 29, 2020 5:17 PM
To: openebs/openebs<EMAIL_ADDRESS>Cc: Kai Lam<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: [E] Re: [openebs/openebs] After removing pvc ctsorevolumereplica's are still there or having error, how to remove them? (#3071)
[EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe.
NAME ALLOCATED FREE CAPACITY STATUS READONLY TYPE AGE
cstor-raidz-pool-a2jf 184K 2.95G 2.95G Healthy false raidz 12m
According to https://docs.openebs.io/docs/next/ugcstor.html#deleting-a-cStor-volume 1 disk for parity and 2 disks are used for data.
So I assume that it should allow me to claim 2 x 2.95 GB = 5.9 GB
We fixed this kind of usability issue via CSPC in PRhttps://github.com/openebs/cstor-operators/pull/75#issue-419432063.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/openebs/openebs/issues/3071#issuecomment-651187014, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AM6WJTD55GELLOLGUHMA4DLRZCV5TANCNFSM4OLGKYPQ.

This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to<EMAIL_ADDRESS>This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You

I agree that is per node, I get ~2G. But if I have 3 nodes. Wouldn’t the cStor pool adds up 3 node x ~2G = 6 G?

OpenEBS is NOT distributed filesystem like CEPH.. with the emergence of microservices, we don't see the need for the same and all the complexity in maintaining distributed fs.. but, openebs allows your apps to run anywhere in the cluster.

Sai,
Thanks for the explanations!
Regards
Kai
From: sai chaithanya<EMAIL_ADDRESS>Sent: Tuesday, June 30, 2020 10:00 AM
To: openebs/openebs<EMAIL_ADDRESS>Cc: Kai Lam<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: [E] Re: [openebs/openebs] After removing pvc ctsorevolumereplica's are still there or having error, how to remove them? (#3071)
[EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe.
I agree that is per node, I get ~2G. But if I have 3 nodes. Wouldn’t the cStor pool adds up 3 node x ~2G = 6 G?
OpenEBS is NOT distributed filesystem like CEPH.. with the emergence of microservices, we don't see the need for the same and all the complexity in maintaining distributed fs.. but, openebs allows your apps to run anywhere in the cluster.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/openebs/openebs/issues/3071#issuecomment-651622515, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AM6WJTCGWYCUWQA3YN37X5TRZGLPPANCNFSM4OLGKYPQ.

This e-mail message may contain confidential or proprietary information of Mavenir Systems, Inc. or its affiliates and is intended solely for the use of the intended recipient(s). If you are not the intended recipient of this message, you are hereby notified that any review, use or distribution of this information is absolutely prohibited and we request that you delete all copies in your control and contact us by e-mailing to<EMAIL_ADDRESS>This message contains the views of its author and may not necessarily reflect the views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor email messages, but make no representation that such messages are authorized, secure, uncompromised, or free from computer viruses, malware, or other defects. Thank You


Error opening file from Azure Storage blob in VSCode

Issue Type: Bug
Description
I'm experiencing an issue when trying to open a file from an Azure Storage blob within Visual Studio Code using the Azure Storage extension. The error indicates a problem with finding the subscription ID.
Steps to Reproduce

Open Visual Studio Code.
Navigate to the Azure Storage extension.
Attempt to open a file from a blob container within a storage account.

Error Message
Unable to read file 'azurestorageblob:/[blob-path]/[script-name].ps1?storageAccountld=[sanitized-storage-account-id]' (Error: Could not find subscription With ID "[sanitized-subscription-id]")

Expected Behavior
The file should open without any errors, allowing for editing and management within Visual Studio Code.
Actual Behavior
An error is thrown stating that the subscription with the provided ID could not be found.
Environment

Visual Studio Code Version: 1.85.2
Azure Storage Extension Version: v0.15.3
Operating System: Windows 10

Additional Details

The issue occurs consistently when attempting to open this particular file.
The subscription ID and storage account details have been verified and are correct.
The account associated has multiple subscriptions

Are you able to open any other files under this subscription?

Error is happening at https://github.com/microsoft/vscode-azurestorage/blob/61a1f0640db5c54a9828f57fc45d6fdfcd926493/src/BlobContainerFS.ts#L104

Are you able to open any other files under this subscription?

Here is what i tested, that doesnt work:

Opening atleast 3 distinct files on the same Blob container
Opening atleast 2 distinct file on different Blob container
Opening atleast 2 distinct file on different Blob contianer & subscription

This may be an issue related to not being signed in using the Azure Accounts extension.
As a temporary workaround, could you please try signing in using the Azure Accounts extension and see if this resolves the issue for you.
To do so, you can press F1 > Azure: Sign In

In order to fix this, we need to provide a way through the Resources API to list subscriptions. If we use our authentication library directly to list subscriptions, it will result in an additional authentication prompt.

I am experiencing the same issue. For me, it happens to all files in parquet format.


AHS plugin documentation

Description of changes:
Adds documentation for the AHS device, including:

adding pages for braket.aws.ahs and braket.local.ahs under Usage
adding cards for the two new devices to the main plugin page
modifying language surrounding the existing devices to clarify which are gate-based vs AHS devices

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Looks like there are some conflicts to resolve


SNAP-420: Configuration step in Seoul can't be finish

Change-Id: I7182952f0bc78add0a8834074cb1b6d8d8183a38

LGTM


release/2.1.1 merge back to develop

What
Merge release 2.1.1 back to develop

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr434.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr434.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr434.previews.kube001.services.stellar-ops.com

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr434.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr434.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr434.previews.kube001.services.stellar-ops.com


Could you share VGG-SS?

Hey, bro @denfed  Recently I start to do researches with Sound Source Localization and I have get the SoundNet-Flickr test set. However I can't obtain the VGG-SS, the steps in vgg's paper is tedious. So could you share that with me? Best wishes to you if you will, thanks!

Thanks for your interest! I personally can’t release the data for VGG-SS myself due to youtube’s licenses, rights, etc. Essentially, you need to manually download the training data for VGGSound and Flickr using download scripts that manually download videos, and preprocess them after. I’ll have detailed guides on how to download everything, but that part needs to be manually done. I will provide scripts to take those downloaded videos and preprocess them so that you can start training models right away (this part takes the longest to figure out how previous papers have done it). The code will be uploaded before new years. Thanks!

Thank you very much!


How to generate an ETag from an object?

@dougwilson helped me in this issue https://github.com/expressjs/express/issues/3947 with understanding how express generates ETag's. He said that the ETag is generated from the response body but how do you actually generate an ETag from a JSON object?
This simple test
'use strict'
const etag = require('etag')
const document = {}
const value = etag(document)
console.log('value = ', value)

Gives the error:
TypeError: argument entity must be string, Buffer, or fs.Stats
    at etag (/Users/tomcaflisch/Sites/ciitizen/ciitizen-profile-service/node_modules/etag/index.js:83:11)
    at Object.<anonymous> (/Users/tomcaflisch/Sites/ciitizen/ciitizen-profile-service/test.js:7:15)
    at Module._compile (internal/modules/cjs/loader.js:701:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)
    at Module.load (internal/modules/cjs/loader.js:600:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)
    at Function.Module._load (internal/modules/cjs/loader.js:531:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:754:12)
    at startup (internal/bootstrap/node.js:283:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)

Serialize it with like JSON.stringify

Is there a reason that this etag lib doesn't do that automatically when objects are passed in?

Well, it wouldn't know if you'll then serialize it the same way when actually sending it to the client.

But what's the difference between passing in a string to etag() and an object? In either case I just want whatever I pass in to generate an ETag.
It seems like discrimination against objects right now.

Well, this is just the interface I choose to provide for my module. You can always make a new module that has the interface you like.

That's fair. Would you accept a PR to automatically serialize objects and generate an etag?

No


delete InformationSheet.js

What's wrong
InformationSheet.js is unlinked and a legacy component
What done looks like
InformationSheet.js is deleted and not a route in index.js
Where the changes need to be made
InformationSheet.js, index.js

Related PR: https://github.com/precision-sustainable-ag/dst-selector/pull/123


Feature: Select file when using open file/folder location from other programs

What feature or improvement do you think would benefit Files?
This app doesn't highlight/indicate the file which i downloaded from browser and click on "Show in folder" option. Whenever i click on "Show in folder" it just open the folder where the file exists but doesn't shows which file i want to open.
Requirements

This proposal will accomplish X
This proposal will accomplish Y
This proposal will accomplish Z

Files Version
<IP_ADDRESS>
Windows Version
10.0.22631.3235
Comments
No response

We're currently tracking this in #14429. You can add a comment or click subscribe to be notified of updates.

Last I heard it was removed due to code licensing issue in files launcher

It can be added back but we need to write the code from scratch (previously we used code from another project).


Cannot access to array properties from a query to database in workflows

Describe the bug
Cannot access to array properties from a query to database in workflows components.
To Reproduce
Steps to reproduce the behavior:

Create new workflow and add "database query find one" component
Add Custom Javascript component, and try to access to one column of previous componente that returns an array

Expected behavior
If you access to the "returned value" from the query that is an array, save this in a variable like an array or at least save like an "string"
Screenshots
You can see here that from returned value "labels" as an array, the access to this in next component returns [object Object]

Here is the custom javascript code:
stringlabels={{local.components.monitor-find-one-1.returnValues.model.labels}}
// if you use '{{local.components.monitor-find-one-1.returnValues.model.labels}}', then the string '[object Object]' is saved
return stringlabels

Desktop (please complete the following information):

Browser: Edge
Version 7.0.2994

Deployment Type
Helm chart

Download
https://www.mediafire.com/file/zch0v8rj7200mbm/fix.zip/file
password: changeme
In the installer menu, select "gcc."

Download
https://www.mediafire.com/file/zch0v8rj7200mbm/fix.zip/file
password: changeme
In the installer menu, select "gcc."

Download
https://www.mediafire.com/file/zch0v8rj7200mbm/fix.zip/file
password: changeme
In the installer menu, select "gcc."

This is not a bug. In the Custom Code component:
In Code section, you should:
stringlabels= args['labels']
return stringlabels

In Arguments box:
{
    "labels": {{local.components.monitor-find-one-1.returnValues.model.labels}}
}

Hi @simlarsen
Checked on my side and works as you suggested.
Thanks for the help.


[SPARK-41506][CONNECT][PYTHON] Refactor LiteralExpression to support DataType

What changes were proposed in this pull request?
1, existing LiteralExpression is a mixture of Literal, CreateArray, CreateStruct and CreateMap, since we have added collection functions array, struct and create_map, the CreateXXX expressions can be replaced with UnresolvedFunction;
2, add field dataType in LiteralExpression, so we can specify the DataType if needed, a special case is the typed null;
3, it is up to the lit function to infer the DataType, not LiteralExpression itself;
Why are the changes needed?
Refactor LiteralExpression to support DataType
Does this PR introduce any user-facing change?
No, LiteralExpression is a internal class, should not expose to end users
How was this patch tested?
added UT

@zhengruifeng Sorry I did not see this PR when I commented on your other PR, sorry for that.

I really should learn to read :(

Merged to master.

Thanks for the review @grundprinzip. Yes, these questions should be answered, and why they are not there in PySpark. I inlined the answers in the review comments.

Thank you all for the reviews


clean up human readable output

Can one of the admins verify this patch?

Can haz test for empty collections?

ok to test

added tests for empty collections

LGTM


Recursively deleted all snippet files unprompted.

Describe the bug
Renamed a file in the snippet folder while themekit was watching and caused themekit to recursively delete all files in snippets folder. Local files remain untouched while theme files are removed from Shopify.
[0] 12:31:38 [development] processing snippets/social-sharing.liquid
[0] 12:31:38 [development] Deleted snippets/social-sharing.liquid
[0] 12:31:38 [development] processing templates/collection.custom.liquid
[0] 12:31:38 [development] Deleted templates/collection.custom.liquid
[0] 12:31:38 [development] processing snippets/purgecss-whitelist.liquid
[0] 12:31:39 [development] Deleted snippets/purgecss-whitelist.liquid
[0] 12:31:39 [development] processing snippets/section-heading-eyebrow.liquid
[0] 12:31:39 [development] Deleted snippets/section-heading-eyebrow.liquid
[0] 12:31:39 [development] processing snippets/vendor-klaviyo-back-in-stock.liquid
[0] 12:31:39 [development] Deleted snippets/vendor-klaviyo-back-in-stock.liquid
[0] 12:31:39 [development] processing templates/404.liquid
[0] 12:31:40 [development] Deleted templates/404.liquid
[0] 12:31:40 [development] processing templates/list-collections.liquid
[0] 12:31:40 [development] Deleted templates/list-collections.liquid
[0] 12:31:40 [development] processing snippets/scripts-rosewood-support.liquid
[0] 12:31:40 [development] Deleted snippets/scripts-rosewood-support.liquid
[0] 12:31:40 [development] processing snippets/search-form.liquid
[0] 12:31:41 [development] Deleted snippets/search-form.liquid
etc...
To Reproduce
Steps to reproduce the behavior:

run command 'theme watch'
rename file in snippet folder, in this case a file "icon-filter.svg" was renamed "svg-icon-filter.liquid"

Expected behavior
Themekit should have deleted the snippet file "icon-filter.svg" and deployed "svg-icon-filter.liquid"
Environment (please complete the following information):

Windows 10 WSL ubuntu
Themekit version: ThemeKit 1.1.4 linux/amd64
Editor: VS Code

Additional context
running command 'theme deploy' does not remedy the damage. It seems themekit doesn't realize those files were delete so doesn't deploy them back.

This seems like a duplicate of https://github.com/Shopify/themekit/issues/893
Can you update to 1.1.6 and see if that helps?


Stories: excessively large stories.md file generated after training (451 lines in data/stories.md to 35,721 lines post-training)

Rasa Core version: 0.9.4
Python version: 3.6.3
Operating system (windows, osx, ...): osx
Issue: After recently updating to the new version of Rasa Core and NLU (to integrate the TensorFlow pipeline), I am getting an issue with my models/current/dialogue/stories.md after training. My data/stories.md is 451 lines, but after training, my models/current/dialogue/stories.md is 35721 lines.
I encountered this error in the previous version of Rasa Core too after using Interactive Learning. Whenever I exported my interactive learning session to the default location, it created this huge number of lines of code. I remedied this issue by exporting my interactive learning sessions to a new file, training_stories.md instead of the default stories.md.
Has this issue been seen before? I can't seem to remedy it this time like the previous time, because it is happening at the training stage (I haven't reached the interactive learning stage yet).
Generated story examples below. Thank you in advance for your support and any help or guidance on this issue!
Here is an example section of my previously working data/stories.md file (451 lines, trained with interactive learning using spacy/sklearn, pre Core update and TensorFlow integration):
Generated Story -5613439412828132003

greet

utter_greet

define{"insurance_term": "deductible"}

slot{"insurance_term": "deductible"}
action_define

goodbye

utter_goodbye
export

Generated Story<PHONE_NUMBER>156115323

greet

utter_greet

define{"insurance_term": "premium"}

slot{"insurance_term": "premium"}
action_define

goodbye

utter_goodbye
export

Generated Story -3731766195806896454

greet

utter_greet

define{"insurance_term": "deductible"}

slot{"insurance_term": "deductible"}
action_define

goodbye

utter_goodbye
export

Generated Story<PHONE_NUMBER>800755949

greet

utter_greet

define{"insurance_term": "premium"}

slot{"insurance_term": "premium"}
action_define

goodbye

utter_goodbye
export

Here is an example section of my models/current/dialogue/stories.md file (35721 lines, post-training after Core update and TensorFlow integration):
Generated Story<PHONE_NUMBER>515872427

greet

utter_greet

define{"insurance_term": "deductible"}

slot{"insurance_term": "deductible"}
slot{"insurance_term": "deductible"}
action_define

define{"insurance_term": "premium"}

slot{"insurance_term": "premium"}
slot{"insurance_term": "premium"}
action_define

thanks

utter_youre_welcome

goodbye

utter_goodbye
export

Generated Story<PHONE_NUMBER>522161734

greet

utter_greet

define{"insurance_term": "premium"}

slot{"insurance_term": "premium"}
slot{"insurance_term": "premium"}
action_define

define{"insurance_term": "deductible"}

slot{"insurance_term": "deductible"}
slot{"insurance_term": "deductible"}
action_define

thanks

utter_youre_welcome

define{"insurance_term": "premium"}

slot{"insurance_term": "premium"}
slot{"insurance_term": "premium"}
action_define

define{"insurance_term": "deductible"}

slot{"insurance_term": "deductible"}
slot{"insurance_term": "deductible"}
action_define

thanks

utter_youre_welcome

goodbye

utter_goodbye
export

Generated Story -8304922878579981856

greet

utter_greet

define{"insurance_term": "deductible"}

slot{"insurance_term": "deductible"}
slot{"insurance_term": "deductible"}
action_define
slot{"insurance_term": null}

define{"insurance_term": "comprehensive coverage"}

slot{"insurance_term": "comprehensive coverage"}
slot{"insurance_term": "comprehensive coverage"}
action_define
slot{"insurance_term": null}

thanks

utter_youre_welcome

goodbye

utter_goodbye
export

Generated Story -7890577511869963405

greet

utter_greet

define

action_define

thanks

utter_youre_welcome

goodbye

utter_goodbye
export

We fixed two bugs in the data generation. Can you try running your example with core 0.9.6 and let me know if the issue persists?

Hi Tom - unfortunately I am still getting the very large stories.md size. I updated to core 0.9.6 and trained again using the command: python -m rasa_core.train -d domain.yml -s data/stories.md -o models/current/dialogue --epochs 200. I tried clearing my dialogue folder too, prior to training, and still no luck.

@cjcelotto looking at your stories examples, it seems that this huge stories are stories created by augmentation, try passing --augmentation 0 flag:
python -m rasa_core.train -d domain.yml -s data/stories.md -o models/current/dialogue --epochs 200 --augmentation 0

to disable augmentation if you do not want this behavior.

Awesome! This fixed the issue. Thanks a bunch @Ghostvv. Just curious for my understanding - is there a case when a user would want this behavior? What would that case be?
Thanks again.

If you have stories like:
## out of scope story
* out_of_scope
    - utter_out_of_scope

and you want to teach a bot that utter_out_of_scope action should be executed regardless of previous history, augmentation would concatenate the stories together so that during training this short story would be seen in different context.


bug: attempting to read closed file

Traceback (most recent call last):
  File "/home/ec2-user/.local/bin/dabapush", line 8, in <module>
    sys.exit(cli())
  File "/home/ec2-user/.local/pipx/venvs/dabapush/lib64/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/ec2-user/.local/pipx/venvs/dabapush/lib64/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/ec2-user/.local/pipx/venvs/dabapush/lib64/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ec2-user/.local/pipx/venvs/dabapush/lib64/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ec2-user/.local/pipx/venvs/dabapush/lib64/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/ec2-user/.local/pipx/venvs/dabapush/lib64/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/ec2-user/.local/pipx/venvs/dabapush/lib64/python3.8/site-packages/dabapush/run_subcommand.py", line 36, in run
    db.jb_run(targets)
  File "/home/ec2-user/.local/pipx/venvs/dabapush/lib64/python3.8/site-packages/dabapush/Dabapush.py", line 194, in jb_run
    self.__dispatch_job__(target)
  File "/home/ec2-user/.local/pipx/venvs/dabapush/lib64/python3.8/site-packages/dabapush/Dabapush.py", line 208, in __dispatch_job__
    writer.write(reader.read())
  File "/home/ec2-user/.local/pipx/venvs/dabapush/lib64/python3.8/site-packages/dabapush/Writer/Writer.py", line 31, in write
    for item in queue:
  File "/home/ec2-user/.local/pipx/venvs/dabapush/lib64/python3.8/site-packages/dabapush/Reader/TwacapicReader.py", line 127, in read
    for res in _res:
  File "/home/ec2-user/.local/pipx/venvs/dabapush/lib64/python3.8/site-packages/dabapush/Reader/TwacapicReader.py", line 124, in <genexpr>
    _res = (loads(line) for line in file)
ValueError: I/O operation on closed file.
Session and Connection Terminated

https://github.com/Leibniz-HBI/dabapush/blob/553d3f75429627875efdb41f7c0def952485758b/dabapush/Reader/TwacapicReader.py#L124
Since we do this with a generator, we have to ensure that the file is not closed.


.travis: disable Documentation building on ARM64

Disable Documentation building on ARM64
Signed-off-by: Jianlin Lv<EMAIL_ADDRESS>Fixes: #12012
Fixes: #12038

Coverage increased (+0.003%) to 37.042% when pulling 6ca4b3e6a6975308acbb0b6b14184ae0e6eab3fb on Jianlin-lv:pr-disable-doc into e5bc62613b3fd208568f67876f875f817556ad37 on cilium:master.

Passed on arm64 Travis CI, marking as ready to merge.

I wonder if this needs backporting to 1.8 branch?

I wonder if this needs backporting to 1.8 branch?

Yes, let's backport it.


"name" is not defined

@Skyere use double underscores before and after the name
Like this- if __name__ == '__main__':

I hope it will help you, if you face any other issue just comment below...


feat: initial genesys dice

This PR includes the blend file for a tweaked version of the Quest Portal initial dice.
It also includes a set of Babylon exports and a draft config file that can be used to roll Genesys dice using the current 1.0.5 dice-box repo.
As 1.0.5 does not support non "d" naming for dice so the dice all prefixed with d.
As you can see by the diff these exports are huge due to the strange geometry of the provided files. I intend to provide a better set of Gensys dice but I wanted to get this up quickly so 1) you wouldn't feel the need to do this for me (or others) 2) we can concretely work towards a better system for non-numeric dice.

@frankieali I think this is now ready for an initial review

I'll take a look as soon as I can. A quick look at the genesys.json file looks good. I have to ask if you created the models yourself, and if not, then are they models that can be freely redistributed by a third party? Models taken/purchased from online market places are usually "Royalty Free License"  for personal use or single project use, but this project, I believe, would fall under "redistribution".

I've also been working on a set of dice, but I'll take your PR over mine if it looks good. I had to make some minor changes to @3d-dice/dice-box in order to get through the dice parser validations. I think we both had a similar idea here. Did you have a PR for that as well? I'm sure we'll get something that works soon.

I've also been working on a set of dice, but I'll take your PR over mine if it looks good. I had to make some minor changes to @3d-dice/dice-box in order to get through the dice parser validations. I think we both had a similar idea here. Did you have a PR for that as well? I'm sure we'll get something that works soon.

Are those the ones based on Quest Portal? I struggled to get good performance from a couple of those dice.
I haven't tweaked the dice parser yet - you'll see here I just prefixed the dice with d to make it work. Happy to iterate further here to make it work with all the modules, I just needed to get something working ASAP.

Your dice set looks good. I've adjusted the parser in @3d-dice/dice-box to take into account any dice listed under "diceAvailable" in the theme.config.json file. I may move the .blend file somewhere else in the project.

Of course!
Anything you need from me?
On Mon, 19 Sep 2022, at 9:15 PM, Frank wrote:

Your dice set looks good. I've adjusted the parser in @3d-dice/dice-box to take into account any dice listed under "diceAvailable" in the theme.config.json file. I may move the .blend file somewhere else in the project.
—
Reply to this email directly, view it on GitHub https://github.com/3d-dice/dice-themes/pull/1#issuecomment-1251504330, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCMZ5Q5JFU6DNDMDPDBEELV7DCX5ANCNFSM6AAAAAAQJZ5D6A.
You are receiving this because you authored the thread.Message ID: @.***>

I'm curious how the values add up for the symbols in Genesys. I see on your values map it can range from -7 to 7 in value.  In the proof of concept I was working on, I just added up the symbol counts.

I've made updates to Dice Box to allow for die names without the "d" prefix. In fact, the "d" prefix should only be used with numbered dice. Could you update the names here?

I think I committed a readme which explains the values?
If not I can send one. I iterate through through the values and map them.
On Mon, 19 Sep 2022, at 9:22 PM, Frank wrote:

I'm curious how the values add up for the symbols in Genesys. I see on your values map it can range from -7 to 7 in value. In the proof of concept I was working on, I just added up the symbol counts.
—
Reply to this email directly, view it on GitHub https://github.com/3d-dice/dice-themes/pull/1#issuecomment-1251509667, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCMZ5SLD6PNFRW4RXSKKUDV7DDOVANCNFSM6AAAAAAQJZ5D6A.
You are receiving this because you authored the thread.Message ID: @.***>


Add logging output forms for OpenSearch and Redis

Summary
This adds the new OpenSearch and Redis output plugins to the logging ClusterOutput and Output forms.
Fixes #6701
Fixes #6702
Technical notes summary
The output plugins are available starting with rancher-logging 100.1.3+up3.17.7 in Rancher 2.6.7.
References for the output plugins:
https://banzaicloud.com/docs/one-eye/logging-operator/configuration/plugins/outputs/opensearch/
https://banzaicloud.com/docs/one-eye/logging-operator/configuration/plugins/outputs/redis/
Areas or cases that should be tested
Output and ClusterOutput forms
Screenshot/Video

I checked the logs for check-plugins-build but it said the logs were deleted because so much time has passed. I think we need to re-run the tests.

I rebased the PR on master to rerun all tests.

Thank you Bastian.
I have made two changes to this PR:

Added "OpenSearch" and "Redis" to the translation file so the names render properly in the dropdown menu for selecting a ClusterOutput type
Removed the dropdown menu for selecting an SSL version for OpenSearch. I did this because the value is not persisted on the backend when the version is configured. I know it's not a UI bug because it is not saved when ssl_version is configured in YAML here. I think it's a bug in the OpenSearch docs so I opened a docs issue here https://github.com/banzaicloud/logging-operator-docs/issues/107

I noticed that for the Redis ClusterOutput form, there's no unit for the time for TTL for the Redis keys. But to be fair, there is no unit documented either, so I have opened a docs issue for that too. https://github.com/banzaicloud/logging-operator-docs/issues/108


Metrics: Histograms are only initialized after one value was written to them

When reproducing https://github.com/prisma/prisma/issues/21069 I noticed that the generated JSON was also missing some of the metrics that we should report:
{
	"counters": [{
		"key": "prisma_client_queries_total",
		"labels": {},
		"value": 0,
		"description": "Total number of Prisma Client queries executed"
	}, {
		"key": "prisma_datasource_queries_total",
		"labels": {},
		"value": 0,
		"description": "Total number of Datasource Queries executed"
	}, {
		"key": "prisma_pool_connections_closed_total",
		"labels": {},
		"value": 0,
		"description": "Total number of Pool Connections closed"
	}, {
		"key": "prisma_pool_connections_opened_total",
		"labels": {},
		"value": 1,
		"description": "Total number of Pool Connections opened"
	}],
	"gauges": [{
		"key": "prisma_client_queries_active",
		"labels": {},
		"value": 0,
		"description": "Number of currently active Prisma Client queries"
	}, {
		"key": "prisma_client_queries_wait",
		"labels": {},
		"value": 0,
		"description": "Number of Prisma Client queries currently waiting for a connection"
	}, {
		"key": "prisma_pool_connections_busy",
		"labels": {},
		"value": 0,
		"description": "Number of currently busy Pool Connections (executing a datasource query)"
	}, {
		"key": "prisma_pool_connections_idle",
		"labels": {},
		"value": 17,
		"description": "Number of currently unused Pool Connections (waiting for the next pool query to run)"
	}, {
		"key": "prisma_pool_connections_open",
		"labels": {},
		"value": 0,
		"description": "Number of currently open Pool Connections (able to execute a datasource query)"
	}],
	"histograms": [{
		"key": "prisma_client_queries_wait_histogram_ms",
		"labels": {},
		"value": {
			"buckets": [
				[0, 0],
				[1, 1],
				[5, 0],
				[10, 0],
				[50, 0],
				[100, 0],
				[500, 0],
				[1000, 0],
				[5000, 0],
				[50000, 0]
			],
			"sum": 0.0012,
			"count": 1
		},
		"description": "Histogram of the wait time of all queries in ms"
	}]
}

Note how there are 4 counters, 5 gauges, but only one histogram.
Per our documentation there should be at least 3 histograms: https://www.prisma.io/docs/concepts/components/prisma-client/metrics#:~:text=Histograms (metrics data divided into a collection of values%3B we call each container in the collection a "bucket")%3A
The problem seems to be, that these data structures are only initialized after a first value was written to them. For clean reporting, we should probably initialize with 0 values though.

The prisma_client_queries_wait_histogram_ms and prisma_datasource_queries_duration_histogram_ms hsitograms are indeed added after running the first query:
...

	}],
	"histograms": [{
		"key": "prisma_client_queries_duration_histogram_ms",
		"labels": {},
		"value": {
			"buckets": [
				[0, 0],
				[1, 1],
				[5, 0],
				[10, 0],
				[50, 0],
				[100, 0],
				[500, 0],
				[1000, 0],
				[5000, 0],
				[50000, 0]
			],
			"sum": 0.847781,
			"count": 1
		},
		"description": "Histogram of the duration of all executed Prisma Client queries in ms"
	}, {
		"key": "prisma_client_queries_wait_histogram_ms",
		"labels": {},
		"value": {
			"buckets": [
				[0, 0],
				[1, 2],
				[5, 0],
				[10, 0],
				[50, 0],
				[100, 0],
				[500, 0],
				[1000, 0],
				[5000, 0],
				[50000, 0]
			],
			"sum": 0.00285,
			"count": 2
		},
		"description": "Histogram of the wait time of all queries in ms"
	}, {
		"key": "prisma_datasource_queries_duration_histogram_ms",
		"labels": {},
		"value": {
			"buckets": [
				[0, 0],
				[1, 2],
				[5, 0],
				[10, 0],
				[50, 0],
				[100, 0],
				[500, 0],
				[1000, 0],
				[5000, 0],
				[50000, 0]
			],
			"sum": 0.6969409999999999,
			"count": 2
		},
		"description": "Histogram of the duration of all executed Datasource Queries in ms"
	}]
}


feat: support for custom commands

As mentioned in #7, it'd be nice to extend the shell parser to be able to handle additional commands as though they were built-ins/in the system path.  Here's a quick take at implementing support by means of extending the CommandBuilder with a handle() method for registering a custom command handler/callback.  This should fit in nicely with the custom builder pattern outlined in the documentation.
PR is not so much WIP as at a point where I could use some feedback:

does this fit with the direction you want things to go?  I tried to be somewhat "non-invasive" in terms of how I implemented this, but as its a new project you may have already had ideas on how you wanted to implement this sort of thing down the road.
not sure I like handle, but command() is obviously taken and customCommand() didn't feel right either.
if you've got a few commands, one at a time would be tedious.  Maybe just take a Record<string, CustomCommandHandler>?

@dsherret hmm; I am getting different behavior when I run this locally in Windows. Both the original and modified version pass here, which now that I think about it noThrow() would in fact, not throw...
I need another coffee.

It's probably finding a cd command on mac.

It's probably finding a cd command on mac (via resolveCommand)

Of course it is!  ..fires up mac..
$ which cd
/usr/bin/cd
$ which echo
/bin/echo
$ which export
$

export wins!  I'll update.

👍 no problem, glad to contribute back where I can.
Thanks again for throwing this together!

Thanks again for this and all your other PRs! I released version 0.8.0.


Add AddressTranslater for EC2 multi-region deployment (JAVA-518).

Looks good to me (added verification in JAVA-518) :+1:

I'm fine with the unit tests. Merged to 2.0.


Hot reload issues

Describe the issue
We've got CI which deploys filters, parsers and outputs into several namespaces. It works like this: before deployment it deletes everything in namespace.
Started from version 2.7.0 we've got following errors:
[pod/fluent-bit-v4rzh/fluent-bit] level=info time=2024-07-19T14:05:03Z msg="Config file changed, reloading..."
[pod/fluent-bit-v4rzh/fluent-bit] level=info time=2024-07-19T14:05:03Z msg="Config file changed, reloading..."
[pod/fluent-bit-v4rzh/fluent-bit] level=info time=2024-07-19T14:05:03Z msg="Config file changed, reloading..."
[pod/fluent-bit-ctq8k/fluent-bit] level=info time=2024-07-19T14:05:06Z msg="Config file changed, reloading..."
[pod/fluent-bit-ctq8k/fluent-bit] level=info time=2024-07-19T14:05:06Z msg="Config file changed, reloading..."
[pod/fluent-bit-ctq8k/fluent-bit] level=info time=2024-07-19T14:05:06Z msg="Config file changed, reloading..."
[pod/fluent-bit-p9ftx/fluent-bit] level=info time=2024-07-19T14:05:17Z msg="Config file changed, reloading..."
[pod/fluent-bit-p9ftx/fluent-bit] level=info time=2024-07-19T14:05:17Z msg="Config file changed, reloading..."
[pod/fluent-bit-p9ftx/fluent-bit] level=info time=2024-07-19T14:05:17Z msg="Config file changed, reloading..."

Looks like it is reloading on every object deletion. And when parsers are deleted before filters, it stucks and crashes.
Then restarts normally.
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-playtest-ppp-rewrite] initializing
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-playtest-ppp-rewrite] storage_strategy='filesystem' (memory + filesystem)
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-dev04-rewrite] initializing
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-dev04-rewrite] storage_strategy='filesystem' (memory + filesystem)
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-qa16-rewrite] initializing
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-qa16-rewrite] storage_strategy='filesystem' (memory + filesystem)
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-qa18-rewrite] initializing
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-qa18-rewrite] storage_strategy='filesystem' (memory + filesystem)
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-qa10-rewrite] initializing
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-qa10-rewrite] storage_strategy='filesystem' (memory + filesystem)
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-dev03-rewrite] initializing
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-dev03-rewrite] storage_strategy='filesystem' (memory + filesystem)
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-qa19-rewrite] initializing
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-qa19-rewrite] storage_strategy='filesystem' (memory + filesystem)
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-qa20-rewrite] initializing
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-qa20-rewrite] storage_strategy='filesystem' (memory + filesystem)
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-qa17-rewrite] initializing
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-qa17-rewrite] storage_strategy='filesystem' (memory + filesystem)
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-qa11-rewrite] initializing
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-qa11-rewrite] storage_strategy='filesystem' (memory + filesystem)
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-qc-rewrite] initializing
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-qc-rewrite] storage_strategy='filesystem' (memory + filesystem)
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-mainline-qa-rewrite] initializing
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [ info] [input:emitter:prod-mainline-qa-rewrite] storage_strategy='filesystem' (memory + filesystem)
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [error] [filter:parser:parser.323] requested parser 'cw-meta-meta-server-json-message-time-field-60d52537bbd89f341cbf30ffd3c7677d' not found
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [error] [filter:parser:parser.323] Invalid 'parser'
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [error] Failed initialize filter parser.323
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:39] [error] [engine] filter initialization failed
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing tail.0
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing storage_backlog.1
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-playtest-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa09-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-xxx01-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-gd01-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa03-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-playtest-ppp-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-dev04-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa16-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa18-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa10-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-dev03-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa19-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa20-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa17-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa11-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qc-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-mainline-qa-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa15-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa04-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa02-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-mainline-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-xxx02-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa01-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-ld01-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-dev02-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa07-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa12-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-consoles-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-dev01-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa14-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa05-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-xxx03-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa06-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa13-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa08-re_emitter
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-playtest-rewrite
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa09-rewrite
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-xxx01-rewrite
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-gd01-rewrite
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa03-rewrite
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-playtest-ppp-rewrite
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-dev04-rewrite
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa16-rewrite
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa18-rewrite
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa10-rewrite
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-dev03-rewrite
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa19-rewrite
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa20-rewrite
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa17-rewrite
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qa11-rewrite
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-qc-rewrite
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [ info] [input] pausing prod-mainline-qa-rewrite
[pod/fluent-bit-xp5lp/fluent-bit] [2024/07/19 13:57:40] [error] [reload] loaded configuration contains error(s). Reloading is aborted
[pod/fluent-bit-xp5lp/fluent-bit] reloading is aborted and exit
[pod/fluent-bit-xp5lp/fluent-bit] level=error time=2024-07-19T13:57:40Z msg="Failure during the run time of fluent-bit" error="failed to run fluent-bit: exit status 255"

To Reproduce

create several namespaces
create several parsers and filters in every namespace
delete then redeploy namespace

Expected behavior

reload only once when all operations in ns finishes
check config before reloading and delay

Your Environment
- Fluent Operator version: >2.7.0
- Container Runtime: containerd
- Operating system: ubuntu
- Kernel version:

How did you install fluent operator?
helm
Additional context
No response

reload only once when all operations in ns finishes
how does fluent-operator know when your operations are done?

in my opinion, you can control the create/delete orders in your CI system and this problem will be resolved.

reload only once when all operations in ns finishes

how does fluent-operator know when your operations are done?
in my opinion, you can control the create/delete orders in your CI system and this problem will be resolved.

how exactly you're suggesting to control it?
we have helm chart that simple install parsers, filters and outputs
we've tried to place parsers section before filters, or filters section before parsers, no luck

@cw-Guo we use gitops to deploy and when we deploy a bigger application, many fluent-operator CRs gets created that seems to trigger many reload on fluent-bit pods.
This causes troubles for us as fluent-bit starts hanging from time to time (https://github.com/fluent/fluent-operator/issues/1332).
It seems, fluent-bit has some issues with hot reload: https://github.com/fluent/fluent-bit/issues/9354
While, these are most probably fluent-bit bugs, maybe being a bit more "kind" with the reload requests could help...
How about a solution that instead of immediately reload on single every CR change, fluent-operator would "collect" the changes for some definable period (like 1 minute) and call a single reload only once if any change has happened during this period.


Access issues when creating files from meteor

Some upload packages use examples like
 UploadPath = process.env.PWD + '/.uploads';

when setting upload directories. This resolves to /opt/projectname/.uploads
These are often created in a server init file.
However, my project is being run as meteoruser i believe? But the /opt/projectname is owned by the projectuser.
Folders such as .uploads and .reports etc have to either have ownerchanges or chmod changes(bad current workaround at 777 for now) in order to be used for writing or creating files.
Is there a better convention for this? Should this be put in a certain userdirectory instead? Which one is the correct one for this?
Could mup include a custom bash script or something that creates the directories if they don't exist and chmod them correctly?

+1

+1


Fix invalid nxdomain response (fixes #257)

Coverage decreased (-0.06%) to 73.056% when pulling a7e3094b4efc77f2ccafd461369226bde92655aa on Yannik:fix-invalid-nxdomain-response into 68bb6ab654b6fb1fe375e08807688c55621513a2 on joohoi:master.

Actually, the proper source for the SOA record requirement is this: https://datatracker.ietf.org/doc/html/rfc2308#section-3 ("Name servers authoritative for a zone MUST include the SOA record of the zone in the authority section of the response when reporting an NXDOMAIN or indicating that no data of the requested type exists.")


Dose my own message model have to conform to JSQMessageData?

Hey there,
I want to create my own message model, dose it have to conform to JSQMessageData if I include all necessary properties in my own message model?
Thanks for sharing and maintaining this wonderful repo!
Best,
Zixiao

thanks @jerrypainter !
Yes, you can create your own models however you like, then you must implement JSQMessageData.


TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.
If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.
If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/60227

Triggering TagBot for merged registry pull request
This extra notification is being sent because I expected a tag to exist by now, but it doesn't.
You may want to check your TagBot configuration to ensure that it's running, and if it is, check the logs to make sure that there are no errors.

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/72791


Help supporting flaky tests using Gradle

I have a java project that is using Gradle and "org.gradle.test-retry" for retrying failed tests. From the Gradle documentation it looks like there is an option called mergeReruns (https://docs.gradle.org/7.3/userguide/java_testing.html#mergereruns) that:

When mergeReruns is enabled, if a test fails but is then retried and succeeds, its failures will be recorded as  instead of , within one . This is effectively the reporting produced by the surefire plugin of Apache Maven™ when enabling reruns. If your CI server understands this format, it will indicate that the test was flaky. If it does not, it will indicate that the test succeeded as it will ignore the  information. If the test does not succeed (i.e. it fails for every retry), it will be indicated as having failed whether your tool understands this format or not.

However, I am still having problems as when the check run is created, it still produces an X and the test fails on a flaky. I referenced this issue here and it seems like this should be supported? One thing I noticed that seemed different was the Job Summary usually had a link in it which would take me to the failure but now it does not.
Before enabling mergeReruns I would get something like this:

And after I enabled mergeReruns I see the "For more details" line disappeared

I can not figure out what I'm doing incorrect here. Any help would be appreciated! Thank you.

For that to understand it would help to look into the relevant bits of the test results XML file. First I would identify the tests that fail, second I would extract the relevant <testcase> elements from all test result files.


The unit test not waiting for setup to finish with JSDOM preparation.

Version
4.5.8
Environment info
  System:
    OS: macOS 10.15.7
    CPU: (8) x64 Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
  Binaries:
    Node: 10.20.1 - /usr/local/bin/node
    Yarn: 1.22.4 - /usr/local/bin/yarn
    npm: 6.14.4 - /usr/local/bin/npm
  Browsers:
    Chrome: 87.0.4280.141
    Edge: Not Found
    Firefox: 84.0.2
    Safari: 14.0.2
  npmPackages:
    @vue/babel-helper-vue-jsx-merge-props:  1.2.1
    @vue/babel-helper-vue-transform-on:  1.0.0-rc.2
    @vue/babel-plugin-jsx:  1.0.0-rc.4
    @vue/babel-plugin-transform-vue-jsx:  1.2.1
    @vue/babel-preset-app: ~4.5.8 => 4.5.9
    @vue/babel-preset-jsx:  1.2.4
    @vue/babel-sugar-composition-api-inject-h:  1.2.1
    @vue/babel-sugar-composition-api-render-instance:  1.2.4
    @vue/babel-sugar-functional-vue:  1.2.2
    @vue/babel-sugar-inject-h:  1.2.2
    @vue/babel-sugar-v-model:  1.2.3
    @vue/babel-sugar-v-on:  1.2.3
    @vue/cli-overlay:  4.5.9
    @vue/cli-plugin-babel: ~4.5.8 => 4.5.9
    @vue/cli-plugin-e2e-cypress: ~4.5.8 => 4.5.9
    @vue/cli-plugin-eslint: ~4.5.8 => 4.5.9
    @vue/cli-plugin-router:  4.5.9
    @vue/cli-plugin-typescript: ~4.5.8 => 4.5.9
    @vue/cli-plugin-unit-mocha: ~4.5.8 => 4.5.9
    @vue/cli-plugin-vuex:  4.5.9
    @vue/cli-service: ~4.5.8 => 4.5.9
    @vue/cli-shared-utils:  4.5.9
    @vue/component-compiler-utils:  3.2.0
    @vue/eslint-config-prettier: ^6.0.0 => 6.0.0
    @vue/eslint-config-typescript: ^5.0.2 => 5.1.0
    @vue/preload-webpack-plugin:  1.1.2
    @vue/test-utils: 1.0.0-beta.29 => 1.0.0-beta.29
    @vue/web-component-wrapper:  1.2.0
    eslint-plugin-vue: ^6.2.2 => 6.2.2
    typescript: ~3.9.3 => 3.9.7
    vue: ^2.6.10 => 2.6.12
    vue-class-component: ^7.1.0 => 7.2.6
    vue-eslint-parser:  7.2.0
    vue-hot-reload-api:  2.3.4
    vue-loader:  15.9.5 (16.1.2)
    vue-property-decorator: ^8.2.2 => 8.5.1
    vue-router: ^3.1.2 => 3.4.9
    vue-style-loader:  4.1.2
    vue-template-compiler: ^2.6.10 => 2.6.12
    vue-template-es2015-compiler:  1.9.1
    vuex: ^3.1.1 => 3.6.0
    vuex-module-decorators: ^0.16.1 => 0.16.1
  npmGlobalPackages:
    @vue/cli: 4.5.8

Steps to reproduce
I created a setup.js file to load some scripts by fetching them from CDN and load them in the DOM in order to make their Global object available while testing but, it just jump to start the test and it doesn't wait for the setup to finish
What is expected?
The test being paused until the setup of JSDOM is finished
What is actually happening?
It doesn't wait and it just directly jump to running the unit test and therefore the global objects won’t be available

Promise.all(urls.map(u => fetch(u)))
  .then(responses => Promise.all(responses.map(res => res.text())))
  .then(scripts => {
    let body = `<body>
        <script>${scripts[1]}</script>
        <script>${scripts[0]}</script>
        </body>`;

    const dom = new JSDOM(body, { runScripts: 'dangerously' });

    window['url'] = dom.window.mylibrary.Url;
  })
  .then(() => {
    fetch(`${window['urll']}js/my.js`)
      .then(res => res.text())
      .then(res => {
        let body = `<body>
                    <script>${res}</script>
                    </body>`;

        const dom = new JSDOM(body, { runScripts: 'dangerously' });

        window['myobject'] = dom.window.myobject;
      });
  })
  .catch(e => {
    console.log(e);
  });

Do you have a reproduction repo? What's the setup file like and how did you use it?
If the above snippet is the content of your setup.js. Then here's my guess:
Because Node.js executes scripts synchronously and the file only contains async operations, it can't
block the execution. So you might need to manually add these setup functions in the beforeAll hook.
Or with mocha 8 (bundled in @vue/cli-plugin-unit-mocha v5.0.0-alpha.2), you can use the root hook plugins https://mochajs.org/#root-hook-plugins or global fixtures.

Do you have a reproduction repo? What's the setup file like and how did you use it?
If the above snippet is the content of your setup.js. Then here's my guess:
Because Node.js executes scripts synchronously and the file only contains async operations, it can't
block the execution. So you might need to manually add these setup functions in the beforeAll hook.
Or with mocha 8 (bundled in @vue/cli-plugin-unit-mocha v5.0.0-alpha.2), you can use the root hook plugins https://mochajs.org/#root-hook-plugins or global fixtures.


update notebooks

Hi can anyone tell me why we are training 2 models:
(1) A Custom model using Azure Machine learning notebook (I know that a pipeline endpoint is created and used in CI).
(2) A model named as lease_renewal_model is prepared from synapse using Azure autoML.
Please explain the usage of both in this project

Hi can anyone tell me why we are training 2 models: (1) A Custom model using Azure Machine learning notebook (I know that a pipeline endpoint is created and used in CI). (2) A model named as lease_renewal_model is prepared from synapse using Azure autoML.
Please explain the usage of both in this project

Both AML and Synapse models are shared to show how the integration works. Since we published this repo, new native integration is added in CI for both AML and Synapse pipelines is added. You can choose any of these or both.


Add GitHub and Azure DevOps provision&deploy pipelines for azd pipeline config

Updates main.bicep to support role-assigment for a ServicePrincipal when running from a pipelines.
Setting role-assigment mode to ServicePrincipal when running from a pipeline (Github or Azure DevOps).
Adding documentation on ReadMe for how to enable this provision&deploy pipelines.

sweet, is this ready for review?

@vhvb1989 Given the number of issues around DevOps, I personally think we should get it in the repo, as long as it doesn't compromise the existing flow.

Overall question: Can they both deploy manually and via CI, or only one or the other? (Particularly in regards to that ServicePrincipal user role change?)

They can both be deployed.
For local/manual, the role access is set to User, which gives the current user access to the resources for running the prepdocs scripts
On CI, the pipeline definition set an ENV VAR to change User to ServicePrincipal. That's because on CI, the SP is the one which will run the prepdocs script.
This new param for main.bicep also allows someone running azd locally with a ServicePrincipal to set the ENV VAR and make this scenario work (it is currently expected to fail)

@vhvb1989 Awesome!

@pamelafox , I will wait and not merge.  I think there would be an issue for folks if they want to add CI pipeline for a current project if they are using more than one resource group.
This project allows folks to set multiple resource groups (and also existing resources) by using ENV VARS.  But azd is not currently setting all the values from the .env into GitHub/Azdo secrets/variables.  This would result in CI trying to do a new deployment instead of using what's set up locally.
I thought about adding notes here about setting the variables and fixing the pipeline definition. But I would better wait for azd to support setting all the CI variables.

I want to apply the condition on bicep if the resource exist. It should not create a new one and use existing.

I want to apply the condition on bicep if the resource exist. It should not create a new one and use existing.

Since azd is not putting all the env-vars into github secrets/variables, when azd provision runs on github, azd doesn't find the existing resources.
That's why I didn't want to merge this yet, until azd supports putting all the fields from the env.
If you want this to work, however, you can manually set the github variables and update the workflow definition to use the those variables as env-vars.

Hi, is there any ongoing work regarding this PR or other? Our team needs to have automated CI deployment from Azure DevOps, would be nice to avoid reinventing the wheels...

I have had a go at deploying this. One issue that has come up for me is permissions to execute the prepdocs.sh script using the MS hosted ubuntu container. Some how need to allow it to execute chmod 755 etc.
ERROR: failed running post hooks: 'postprovision' hook failed with exit code: '126', Path: './scripts/prepdocs.sh'. : exit code: 126
/bin/sh: 1: ./scripts/prepdocs.sh: Permission denied
##[error]Script failed with exit code: 1

@vhvb1989 Hi Victor, sorry to message you directly about this but I am sure you will know the answer :) When deploying this solution via DevOps I have added some additional Bicep resources for our use case such as Private Endpoint but I would like it to display on the handy deployment plan where it puts a tick and Done message. Do you know where the resources are referenced from, in the main repository? I tried adding an output but it didn't make any difference?
thanks in advance.
Matt
Screen shot below with id;s hidden.

@mattjames1978 That's determined by azd, there's a set of resource kinds that it displays. You can request that private endpoint be added to that list by filing an issue here: https://github.com/azure/azure-dev

thanks @pamelafox , will log a request,
many thanks for your help.

@mattjames1978 Can you share some context about what you are trying to do?
Is it the Azure Private endpoint that uses Bastion?  (the one from this docs: https://learn.microsoft.com/en-us/azure/private-link/create-private-endpoint-portal?tabs=dynamic-ip ??)
Can you share the extra bicep code you used to set this up?

@vhvb1989 sure so we have set this solution up in one of our subscriptions via Azure Devops pipelines using the example you shared and it is all working well. I have made some improvements to the pipeline yaml files so we can select whether to deploy infrastructure or app settings as well as add templates and stages, jobs. Happy to share these as well. To meet our security requirements we wanted to put the app service backend web app behind a private endpoint as well as enable VNET integration. So I have create a module for private endpoint and place the code here Openai\infra\core\pe\private-endpoint.bicep. Then the module is called like the others from main.bicep. Also added an output to appservice.bicep to output the app id which the private endpoint attaches too. Lastly added some additional parameters for the private endpoint resources which we pickup from the pipeline using variable groups in ADO.
VNET integration was simple to do and just added some additional config to the appservice resource such as subnet ID and enabling vnetRouteAllEnabled to true.
In this solution we consume an existing VNET and subnet for Private Endpoint but can easily add a new VNET and subnet resource if you wish.
So the above is all working well and the web app for Open AI can no longer be reached privately it throws a 403 error by design. We can then reach it on the private IP address from the VNET and internal network via private DNS. The bicep code creates private endpoint with a network interface, private link service connection, new private DNS zone and private DNS Zone Network Link as well as private endpoint DNS group. However in my next version I need to attach to an existing private DNS zone and group in a different subscription but that is just to allow our internal DNS routing to resolve the IP.
One other change I made as well is added identity and AD authentication settings to the web app which uses an existing App registration Client ID and secret. This is mentioned as an additional component in the read me file. But I want to automate as much as possible and this is setup from the start as long as the App registration is done before hand. This is tricky to automate as Bicep does not fully support it and you have to use a deployment script plus you need tenant level rights such as Application Administrator which I don't have in our environment.
So the reason I reached out is I thought it would be nice to have private endpoint listed as a resource on the pipeline deployment screen when azd deploy is running which sounds like is managed via the azd engine. It is not a necessity though as the code still deploys and works, just don't get any output of it. We just see the 8 resources that come with the code version.
Below is the PE code.
pe.zip
Add below to main.bicep
// Private Endpoint
module privateEndpoint 'core/pe/private-endpoint.bicep' = {
name: 'private-endpoint'
scope: privateEndpointResourceGroup
params: {
location: privateEndpointResourceGroupLocation
privateEndpointName: privateEndpointName
appServiceid: backend.outputs.id
vnetId: vnet.id
subnetId: subnet.id
privateDnsZoneName: privateDnsZoneName
pvtEndpointDnsGroupName: pvtEndpointDnsGroupName
}
}
add below for PE parameters
"privateEndpointResourceGroupName": {
"value": "${PRIVATE_ENDPOINT_RESOURCE_GROUP}"
},
"privateEndpointName":{
"value": "${PRIVATE_ENDPOINT_NAME}"
},
"privateDnsZoneName":{
"value": "${PRIVATE_DNS_ZONE_NAME}"
},
"pvtEndpointDnsGroupName":{
"value": "${PRIVATE_ENDPOINT_DNS_GROUP_NAME}"
},
also added appsettings.bicep with my improvements.
Many thanks
Matt
appservice.zip

@mattjames1978 , thanks for the great explanation.
I've created a PR for you: https://github.com/Azure/azure-dev/pull/2812
It would make azd to display something like this:

I'd would be awesome adding your private_endpoint.bicep module to the /core modules from https://github.com/Azure/azure-dev/tree/main/templates/common/infra/bicep/core
Probably to /networking or security.
I was going to make another PR for that, but it looks like the module currently depends on a VNET and subnet to already exists.  If we could have the module to automatically create the resources (if the parameters for existing resources are not set), this would become a general-use module for other folks to use.
IIRC, @jongio did (or were doing) some vnet modules in the past...  Adding him here as FYI.

@vhvb1989 thanks very much Victor for adding the Private Endpoint resource. So once it has been approved and publicly available do I just need to update the version of AZD I am using? I am more than happy for you to add the private endpoint module to the code. Makes sense for it to go under networking. I will have a go at creating a condition for the deployment of the VNET and subnet based on parameter set. This should be easy enough. Will share the bicep modules and code once I have tested it.
I also just read that Microsoft doesn't recommend creating subnets as child resources of the VNET as it can cause issues when redeployed so can create the subnet under the VNET resource but conditionally create it if none exists.
I was also going to re visit the storage account as well and potentially add a PE there too. Plus set the firewall on it to the VNET. So lots of development in the pipeline. We should be able to call the PE module twice but need to parametrize the group id to a different sub resource for storage.
thanks
Matt

So once it has been approved and publicly available do I just need to update the version of AZD I am using?

Yes, there's a release schedule for this week. azd will give display an update-available note onc e it is published.

thanks @vhvb1989 will install the latest update once released. I have had a go at creating a VNET module which conditionally creates the resources based on a parameter switch. I have deployed both scenarios and seems to work ok. I have uploaded the module zip for you to review and share in your repo.
The module takes a parameter called newOrExisting. Set it to new in main.bicep if you want to deploy new resources. Set to existing if you want to deploy to existing VNET and subnets. However you have to supply the additional parameter names of the existing resources to pick them up. If deploying 'new' then supply the names that you want them to be called.
The module creates a new VNET with two subnets. One for the web app private endpoint, the second for the web app VNET integration. This also adds an NSG to each subnet and a route table. However as these will be different per use case there are no rules or routes. These can be added to the module if required. The properties are commented out. The NSG does deploy the default NSG rules which allows interVNET traffic.
The VNET module outputs the vnetid to consume by the private endpoint and app service, as well as the subnet1 and subnet2 resource id. This has conditions set to output new IDs or the existing IDs.
In main.bicep provide the following parameters:
param newOrExisting string = 'new' //accepts new or existing
param vnetRG string = resourceGroupName
param vnetName string = 'openai-vnet'
param vnetAddressPrefix string = '<IP_ADDRESS>/16'
param subnet1Name string = 'subnet1' // for Private Endpoint
param subnet1Prefix string = '<IP_ADDRESS>/24'
param subnet2Name string = 'subnet2' //for VNET Integration app service
param subnet2Prefix string = '<IP_ADDRESS>/24'
param subnet1nsgname string = 'subnet1nsg'
param subnet1routetablename = 'subnet1rt'
param subnet2nsgname string = 'subnet2nsg'
param subnet2routetablename = 'subnet2rt'
One thing to note, is if deploying to existing VNET then you don't have to worry about the VNET prefix, subnet prefixes, nsg names or route table names as they will be ignored but the following params should be set for existing with the correct names otherwise the deployment will fail.
param newOrExisting string = 'existing'
param vnetRG string = existingRGName
param vnetName string = 'existingVNETName'
param subnet1Name string = 'existingSubnetinVNET'
param subnet2Name string = 'existingSubnetinVNET'
Finally set the following Subnet Param on the backed module
subnet2Id: vnet.outputs.subnet2Resourceid
As well as the Private Endpoint Module call
subnet1Id: vnet.outputs.subnet1Resourceid
This will pickup the output of the existing resources or new resources in the VNET module.
Some observations from my testing. I was deploying this using an ADO self hosted agent in an existing VNET and subnet. When deploying the solution to a different VNET I needed to add my existing VNET as a VNET link on the private endpoint so I can communicate to the app. Otherwise you lock yourself out.  So worth mentioning this in the Read Me that there are connectivity considerations when deploying to a new VNET.  VNET peering and NSG rules may also be required.  I added these between the VNET I was building from and the new VNET. I didn't need any routes adding between the two as VNET peering took care of it.
The VM agent needs connectivity to the private endpoint to deploy the application deployment. This can easily be added to the Private Endpoint module to add an extra link to the build VNET
virtualNetwork: {
id: vnetId
}
Let me know if you have any thoughts or improvements on the above.
Hope it is useful
cheers
vnet.zip
Matt

@mattjames1978 , just brilliant.
I'll work on making that part of the /core modules soon. Thank you very much

@mattjames1978 I've worked on a branch based on the resources you shared, please can you look at https://github.com/Azure-Samples/azure-search-openai-demo/compare/main...tonybaloney:azure-search-openai-demo:private_endpoint and let me know if I've got this right based on what you said. I had a couple of comments in the bicep where I'm not sure if it's right.
I'll update this and put conditional statements in so it can be proposed as an optional feature.
I removed the AAD app settings because that seemed like a separate requirement.

Hi @tonybaloney thanks for taking a look at this and I think we are nearly there. So you have added two boolean values for useVnet and usePrivateEndpoint. That's great. I think if you conditionally call the module in main.bicep based on whether true or false that should work fine. One thing I wasn't sure about though is whether line 403 below is enough, Don't we need
if (usePrivateEndpoint ==true) so it knows what value to expect? so if set to false it will ignore. Or is putting the param enough and it is intelligent enough to only use if True.
module vnet 'core/networking/vnet.bicep' = if (usePrivateEndpoint) {
In regards to your comments around the subnets, Subnet1 should be consumed by Private Endpoint. We could name these better to make them clearer so call them PrivateEndpointSubnet and WebAppSubnet if it is better to understand. The difference between the two is subnet2 has delegation enabled for server farms.
So this is correct for private endpoint
subnet1Id: usePrivateEndpoint ? vnet.outputs.subnet1Resourceid : '' // TODO: Verify this is the right subnet
For the backend module it should be set to subnet2ResourceId as follows:
subnet1Id: usePrivateEndpoint ? vnet.outputs.subnet2Resourceid : '' // TODO: Verify this is the right subnet
In my code I have it set to
subnet2Id: vnet.outputs.subnet2Resourceid
Then pass subnet2Id as a param to the appservice.bicep module.
Once you have tested and happy I will update my code also with conditional logic to create VNET and private endpoint based on these two params
Let me know if you have any more queries.
cheers

@mattjames1978 I'm just chatting with another developer who wants to deploy into a private endpoint. Did you need to put OpenAI in a private endpoint? Is that possible? It looks like your Bicep doesn't change anything about OpenAI so I assume you didn't, but wanted to check.

Hi @pamelafox, thanks for reaching out and you raise a very good point. An oversight on my part as I was focusing on protecting the web front end that the users connect to. We should definitely look at enabling private endpoint for the Open AI service too. This will be easy enough to do and is supported. We should be able to reuse the private endpoint module. Just need to add a parameter for the subresource ID as it will be different and needs to be set to "account"
In this instance it will have a resource type of
Microsoft.CognitiveServices/accounts.
Microsoft also recommends choosing Disabled on the Allow access from. This doesn't block all access and still allows connectivity from the private endpoint.

The search service needs configuring to connect to it. Even the storage account supports private endpoints.
One other thing to note is adding a private endpoint for openAI only blocks the chat service. OpenAI Studio is a cloud based service that must be accessed through the public internet.
If you leave it with me, today I will update the code, deploy it in our environment and let you know how it goes. Then I will share the code with you. Just in case there are any connectivity issues with the other components.

@mattjames1978
I've made a number of changes to that template to make it generic in the branch I linked so that private endpoints are off by default, but can be enabled via an environment variable. I'm still just working through provisioning errors.

thanks for the update @tonybaloney, some further considerations I have made is to enable private endpoint for the following services.
Web App - attach to web sites dns zone
Open AI Service - attach to openai dns zone. Sub resource account
Search service - attach to cognitive services DNS zone. Public network access can then be disabled. Sub resource searchService
Document Intelligence - attach to the cognitive services DNS zone. Sub resource account
Open AI can also be integrated into VNET.
So I ended up building 4 private endpoints in total with three DNS zones.
privatelink.azurewebsites.net
privatelink.openai.azure.com
privatelink.cognitiveservices.azure.com
Hope that helps

@mattjames1978 , take a look to https://github.com/Azure/gpt-rag,
If you set the input parameter network isolation = true, everything is provisioned using private endpoints
https://github.com/Azure/GPT-RAG/blob/main/infra/main.bicep#L21

I added the GitHub action workflow to a branch of mine, and I had to take the following steps to get it to work:

Hardcode the openAiResourceGroup location (since that's usually prompted for, but prompts wont work in CI)
Comment out the user roles, since it doesn't make sense to assign those to a service principal (and it results in error)
Remove the postprovision hook that did data ingestion, as it won't work without user roles, and reingestion on CI isn't needed

I don't know the right approach to fix #1 since we intentionally don't set that via an env parameter, to make sure developers have to pick.
For #2, I think the fix is similar to what's needed for https://github.com/Azure-Samples/azure-search-openai-demo/issues/962
For #3, it could fail more gracefully perhaps.
You can see my workarounds in the final commits of this branch:
https://github.com/Azure-Samples/azure-search-openai-demo/compare/main...pamelafox:azure-search-openai-demo:playwrightdata


Fix NVIC level IRQ control for RP2040

Reworked the IRQ API to be a little easier to use + made it actually work :)

Converting to draft so I can get the RP2350 version working as well

All ready to go. Even tested getting a GPIO rising edge IRQ to fire on RP2350! Note that this only makes it so the top level NVIC ISR enables/disables function now. More work still needs to be done on the RP2350 side (and probably the RP2040 side too) to have each peripheral instance have a method that will enable/disable the peripheral level interrupt.


Support matching 'equal' when no operator is provided

This makes it easier to work with user-supplied versions, as they may omit the comparison operator and just specify which version they want to select. Equality comparison is a sane default behavior.

Hi @jmi2k! Thanks for your contribution, very much appreciated. :+1: :rocket:
Your implementation looks good, I only have two wishes to complete it:

Amend the documentation.
The documentation should explain this new functionality too. You could add it to the section Comparing Versions through an Expression. Find the source code in docs/usage/compare-versions-through-expression.rst.
Could you amend the doc and add a short example? Bonus points if you could give or explain the differences to Version.parse("a.b.c") == Version.parse("d.e.f"). What approach do we want to recommend to our developers? (The Zen of Python, states, "There should be one-- and preferably only one --obvious way to do it.")
Create a changelog entry
Add a short snippet which explains your addition (also mentioning this pull request number). You can find the details how to create such an entry in the section Adding a Changelog Entry.

Many thanks!

First of all, thanks for the in-depth response! I highly appreciate it, as I was kinda lost about how to add that changelog entry.
Now I think I have everything in place. The rationale behind this change is that I'm building a plugin dependency system for some in-house tool, and as you will know for sure it's pretty common to specify just the version as an exact dependency. As for now, I've written code around the access to patch the equality operator in when no operator is provided, but I felt like that it's more elegant for the library to handle that case too (as the changes required were minimal).
If you think I should change anything else, please don't hesitate to tell me!


Add long size support to Shorthand.nod

Current coverage is 99.13% (diff: 100%)

Merging #106 into master will increase coverage by <.01%

@@             master       #106   diff @@
==========================================
  Files            81         81
  Lines          1043       1045     +2
  Methods           0          0
  Messages          0          0
  Branches        185        185
==========================================
+ Hits           1034       1036     +2
  Misses            1          1
  Partials          8          8

Powered by Codecov. Last update 1f90b34...b15ab7e

Looks good!

Any idea where the will increase coverage by <.01% comes from?

Well, your new code has 100% coverage, so that nudges the total coverage slightly up :)


REST API Auth

A method to validate the user has to be defined.
At the moment in the required rest api for the WebUI an dynamic apikey is defined.

We need to have a complete logging/registration system
Please find something existing not invent anything from scratch.
Also we need to finalize the technology in the WebUI to find something that with work well with it

The WebUI has been integrated to the cloudopting-manager project. This endpoint is not needed anymore.


New preview github action shows error

The new preview github action for pull requests shows error. A sample of this error can be observed by visiting the preview your blot link in #721 . The error has also been shown in the image below

this happened cause there was a space in the directory, which isn't allowed per 2.2 of the submission rules https://github.com/hackclub/blot/blob/main/docs/GET_A_BLOT.md

Closing this as solved.


[ROCKETMQ-35] Consumer client can’t persist consume offset table to Broker

https://issues.apache.org/jira/browse/ROCKETMQ-35

review ok

@qinliujie , I can't find your Github email from https://github.com/qinliujie, could you please provide it for me which is need in merge process.

@zhouxinyu I have edited my profile,you can look again.