[SPARK-34869][K8S][TEST] Extend "EXTRA LOGS FOR THE FAILED TEST" section of k8s integration test log with the describe pods output

What changes were proposed in this pull request?
Extending "EXTRA LOGS FOR THE FAILED TEST" section of k8s integration test log with kubectl describe pods output for the failed test.
Why are the changes needed?
PR builds frequently fails as the k8s integration tests are very flaky now in Amplab Jenkins environment.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Locally by making temporary one of the test fail. The output is:
21/03/25 16:55:16.722 ScalaTest-main-running-KubernetesSuite INFO KubernetesSuite:

===== EXTRA LOGS FOR THE FAILED TEST

21/03/25 16:55:17.167 ScalaTest-main-running-KubernetesSuite INFO KubernetesSuite: BEGIN driver DESCRIBE POD
Name:         spark-test-app-a2b03971b7c049e8a2629f6a3198842b
Namespace:    35bdb17e308743afaec17538f89a7c3e
Priority:     0
Node:         minikube/<IP_ADDRESS>
Start Time:   Thu, 25 Mar 2021 16:52:10 +0100
Labels:       spark-app-locator=75f695685ae44314a99ec13bb39332bc
              spark-app-selector=spark-150230742d364a77927a08eed0222065
              spark-role=driver
Annotations:  <none>
Status:       Succeeded
IP:           <IP_ADDRESS>
Containers:
  spark-kubernetes-driver:
    Container ID:  docker://d6d27b0551060d9b094f12d1e232dfb5ae78ce38559680c7126c548996da4d95
    Image:         docker.io/kubespark/spark:3.2.0-SNAPSHOT_9575B805-9CB0-4A16-8A31-AA2F8DDA8EE5
    Image ID:      docker://sha256:3fc556c73a0d5187b5a14dbdc2f69ef292e60b544b4b4d3715f6749417c20918
    Ports:         7078/TCP, 7079/TCP, 4040/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      driver
      --properties-file
      /opt/spark/conf/spark.properties
      --class
      org.apache.spark.examples.SparkPi
      local:///opt/spark/examples/jars/spark-examples_2.12-3.2.0-SNAPSHOT.jar
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 25 Mar 2021 16:52:11 +0100
      Finished:     Thu, 25 Mar 2021 16:52:20 +0100
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  1408Mi
    Requests:
      cpu:     1
      memory:  1408Mi
    Environment:
      SPARK_USER:                 attilazsoltpiros
      SPARK_APPLICATION_ID:       spark-150230742d364a77927a08eed0222065
      SPARK_DRIVER_BIND_ADDRESS:   (v1:status.podIP)
      SPARK_LOCAL_DIRS:           /var/data/spark-dab6f1c9-e538-40c8-a7d9-3e88f9b82cfa
      SPARK_CONF_DIR:             /opt/spark/conf
    Mounts:
      /opt/spark/conf from spark-conf-volume-driver (rw)
      /var/data/spark-dab6f1c9-e538-40c8-a7d9-3e88f9b82cfa from spark-local-dir-1 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-nmfwl (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  spark-local-dir-1:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  spark-conf-volume-driver:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      spark-drv-c60832786a15ffbe-conf-map
    Optional:  false
  default-token-nmfwl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-nmfwl
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  3m7s  default-scheduler  Successfully assigned 35bdb17e308743afaec17538f89a7c3e/spark-test-app-a2b03971b7c049e8a2629f6a3198842b to minikube
  Normal  Pulled     3m7s  kubelet, minikube  Container image "docker.io/kubespark/spark:3.2.0-SNAPSHOT_9575B805-9CB0-4A16-8A31-AA2F8DDA8EE5" already present on machine
  Normal  Created    3m7s  kubelet, minikube  Created container spark-kubernetes-driver
  Normal  Started    3m6s  kubelet, minikube  Started container spark-kubernetes-driver
21/03/25 16:55:17.168 ScalaTest-main-running-KubernetesSuite INFO KubernetesSuite: END driver DESCRIBE POD

Test build #136529 has started for PR 31962 at commit a92cca8.

Test build #136529 has finished for PR 31962 at commit a92cca8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136529/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41113/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41113/

This is different error:
- Start pod creation from template
- PVs with local storage *** FAILED ***
  io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://<IP_ADDRESS>:8443/api/v1/persistentvolumes. Message: object is being deleted: persistentvolumes "test-local-pv" already exists. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=persistentvolumes, name=test-local-pv, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=object is being deleted: persistentvolumes "test-local-pv" already exists, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=AlreadyExists, status=Failure, additionalProperties={}).
  at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:570)
  at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:509)
  at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:474)
  at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:435)
  at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:250)
  at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:881)
  at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:341)
  at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:84)
  at org.apache.spark.deploy.k8s.integrationtest.PVTestsSuite.setupLocalStorage(PVTestsSuite.scala:87)
  at org.apache.spark.deploy.k8s.integrationtest.PVTestsSuite.$anonfun$$init$$1(PVTestsSuite.scala:137)
  ...
- Launcher client dependencies

@srowen @shaneknapp do you know how Minikube runs are isolated on Jenkins?

(No idea about any of that here, sorry)

No problem, let's hope the next error will be the one we look for.

jenkins retest this please

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41113/

Test build #136536 has started for PR 31962 at commit a92cca8.

Test build #136536 has finished for PR 31962 at commit a92cca8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136536/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41120/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41120/

===== EXTRA LOGS FOR THE FAILED TEST

21/03/25 13:08:25.283 ScalaTest-main-running-KubernetesSuite INFO KubernetesSuite: BEGIN driver DESCRIBE POD
Name:               spark-test-app-269a3ebb336a45c0b8485bcd9757bb01
Namespace:          048b7ef217b2443aa9ce20c72906872a
Priority:           0
PriorityClassName:  <none>
Node:               minikube/<IP_ADDRESS>
Start Time:         Thu, 25 Mar 2021 13:05:23 -0700
Labels:             spark-app-locator=5dad6c34644f4dd2864665239ad40d17
                    spark-app-selector=spark-a78c13817a9641b59801132dfde88a11
                    spark-role=driver
Annotations:        <none>
Status:             Pending
IP:                 <IP_ADDRESS>
Containers:
  spark-kubernetes-driver:
    Container ID:
    Image:         docker.io/kubespark/spark-py:3.2.0-SNAPSHOT_14049b88-ef56-4044-b1b7-fcc98803fe36
    Image ID:
    Ports:         7078/TCP, 7079/TCP, 4040/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      driver
      --properties-file
      /opt/spark/conf/spark.properties
      --class
      org.apache.spark.deploy.PythonRunner
      local:///opt/spark/examples/src/main/python/pi.py
      5
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  1433Mi
    Requests:
      cpu:     1
      memory:  1433Mi
    Environment:
      SPARK_USER:                 jenkins
      SPARK_APPLICATION_ID:       spark-a78c13817a9641b59801132dfde88a11
      SPARK_DRIVER_BIND_ADDRESS:   (v1:status.podIP)
      SPARK_LOCAL_DIRS:           /var/data/spark-ec6ef64c-722c-4568-b02e-4077498109c5
      SPARK_CONF_DIR:             /opt/spark/conf
    Mounts:
      /opt/spark/conf from spark-conf-volume-driver (rw)
      /var/data/spark-ec6ef64c-722c-4568-b02e-4077498109c5 from spark-local-dir-1 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-m6lgm (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  spark-local-dir-1:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  spark-conf-volume-driver:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      spark-drv-367160786afdd4ab-conf-map
    Optional:  false
  default-token-m6lgm:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-m6lgm
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    3m2s                 default-scheduler  Successfully assigned 048b7ef217b2443aa9ce20c72906872a/spark-test-app-269a3ebb336a45c0b8485bcd9757bb01 to minikube
  Warning  FailedMount  3m1s                 kubelet, minikube  MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-367160786afdd4ab-conf-map" not found
  Normal   Pulling      86s (x4 over 3m)     kubelet, minikube  pulling image "docker.io/kubespark/spark-py:3.2.0-SNAPSHOT_14049b88-ef56-4044-b1b7-fcc98803fe36"
  Warning  Failed       85s (x4 over 2m59s)  kubelet, minikube  Failed to pull image "docker.io/kubespark/spark-py:3.2.0-SNAPSHOT_14049b88-ef56-4044-b1b7-fcc98803fe36": rpc error: code = Unknown desc = Error response from daemon: pull access denied for kubespark/spark-py, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
  Warning  Failed       85s (x4 over 2m59s)  kubelet, minikube  Error: ErrImagePull
  Normal   BackOff      74s (x6 over 2m59s)  kubelet, minikube  Back-off pulling image "docker.io/kubespark/spark-py:3.2.0-SNAPSHOT_14049b88-ef56-4044-b1b7-fcc98803fe36"
  Warning  Failed       74s (x6 over 2m59s)  kubelet, minikube  Error: ImagePullBackOff
21/03/25 13:08:25.284 ScalaTest-main-running-KubernetesSuite INFO KubernetesSuite: END driver DESCRIBE POD

@srowen looking the events this must be a clue repository does not exist or may require 'docker login':
  Warning  Failed       85s (x4 over 2m59s)  kubelet, minikube  Failed to pull image "docker.io/kubespark/spark-py:3.2.0-SNAPSHOT_14049b88-ef56-4044-b1b7-fcc98803fe36": rpc error: code = Unknown desc = Error response from daemon: pull access denied for kubespark/spark-py, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41120/

Although I think the image should not be pulled from docker.io but be available in the minikube's own local repository.
It would be good to look around:
$ eval $(minikube docker-env)
$ docker images

@shaneknapp do you have any idea?
Moreover this warning is also interesting:
Warning  FailedMount  3m1s                 kubelet, minikube  MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-367160786afdd4ab-conf-map" not found.
I have to go soon as it is 22:15 here but I hope these events helps a bit.

cc @dongjoon-hyun, @holdenk

Thank you for pining me, @attilapiros . I agree with you. Apparently, Amplab Jenkins seems to have a setup issue still. I have a personal downstream Jenkins machine dedicated to run K8s integration test for all Apache branches (master/3.1/3.0/2.4). I usually keep them up-to-date. Currently, Minikube 1.18.1 and K8s 1.20.2. They never fails for last 7 days in all branches.

Test build #136552 has started for PR 31962 at commit 5ca3309.

Test build #136552 has finished for PR 31962 at commit 5ca3309.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Thanks @dongjoon-hyun . I also thought about checking the Minikube's bug/issue database for this v1.7.3 version but as you mentioned this is very likely a Minikube bug and the migration is already planned to the Minikube 1.18.1 via SPARK-34738 I think we have to wait until it is finished.

I quickly went through the issues for this version: https://github.com/kubernetes/minikube/issues?page=3&q=is%3Aissue+v1.7.3.
Although I haven't found a similar issue still I think this very likely a Minikube's bug.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136552/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41136/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41136/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41136/

Error processing tar file(exit status 1): write /opt/spark/jars/kubernetes-model-batch-4.13.2.jar: no space left on device

jenkins retest this please

Test build #136588 has started for PR 31962 at commit 5ca3309.

Test build #136588 has finished for PR 31962 at commit 5ca3309.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41172/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136588/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41172/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41172/

Merged to master.


Failed on Azure Resources Deployment - Microsoft.Web/Sites/control

Hi,
I got the following error on deploying the Azure components

Any news about this issue? we have the same problem. We need your help.
Thanks a lot!! Regards

Can you please try selecting higher App Service plan like S2 or S3 and try again.
Once deployment is completed, you can go back to smaller App Service plan.
Let us know if issue still persists.


@app_mention event is not being triggered sometimes

We are using our slack app by doing @app_mention from workflow message .
In most cases the app_mention wrks as expected but in some instances we see that the app_mention doesn't trigger. We thought it's a backend error from our side but it seems like the app_mention is not happening at all for some cases.
Does this happen when multiple events trigger simultaneously if so how to handle it .
The events  we are triggering are :

@app_mention
Some button actions
total 3 different button actions.

We are using app in 5 different channels simultaneously ,  and for each request an app_mention is triggered and other events as well . Is this happening due simultaneous events being triggered ?

Hi @TALUPURI-DIMPLE-HIMA-BINDU, thanks for asking the question.
It's totally fine to use your app within a few channels at a time. Thus, that shouldn't be the reason why your app server does not receive some of the events.
The only possible cause apart from network/infra issues and/or Slack outage is ratelimits for Events API. Your app can receive only 30K events per hour. This means that if your app subscribes other events that can be delievered a lot (e.g., message events in a busy channel), app_mention event delivery can be affected by that. Please refer to the document for more details: https://api.slack.com/apis/rate-limits#events
If that's not your case, please contact our customer support team to check what's happening with your app and workspace. Since this is a public place, we're unable to ask you to provide any sensitive information like your workspace / channel metadata. If you're observing some unexpected behavior / errors, customer support agents can check server-side logs upon your approval. You can reach out to them via either /feedback in your Slack workspace or https://my.slack.com/help/requests/new . I hope this helps.

Thank you  @seratch for the reply we will contact support team .


Sync

Sync with internal repository solving #143
2 important improvements:

allowing to not render cells outside of viewport (new allowCellsRecycling API in FixedDatatTableColumn)
significantly sped up calculateState in the FixedDataTable

With the "Replacing getDOMNode with findDOMNode" commit we will be breaking compatibility with react 0.12.x this seems fine to me but we should make this clear in the release notes and update the package.json.

This looks good to go, merge when ready.


Fix typos and improve readability.

Thanks for the feedback Ardon! The patch doesn't work anymore however due to the resolution of https://bits.nictiz.nl/browse/ZIB-1584, so I handpicked your edits into the profile (minus one or two that I didn't agree with ;))


When User changes Quota in Private Cloud and switches browser tab, the changes get reset

Description:
When User changes Quota in Private Cloud and switches browser tab, the changes get reset
Steps to reproduce:

open existing Private Cloud product
open additional browser tab if do not have one
in the Private Cloud product browser tab change any Quota value
switch to another browser tab
switch back to Private Cloud product browser tab

Expected result:
The changed value retain
Actual result:
The changed value gets reset

@funtigr When reporting an issue, it's important to identify the cause more clearly, especially for bugs reported by team members. During testing, the behavior occurs for all fields, not just for Quota changes.

Oh, I see. Didn't notice that on that time.
This is fixed now.


Add pose estimation training pipeline and RTMCC head

Description
Please include a summary in English, of the changes in this pull request. If it closes an issue, please mention it here.
Closes: #(issue)
You should link at least one existing issue for PR. Before your create a PR, please check to see if there is an issue for this change.
PRs from forked repository not accepted.
Change(s)

Add Dataset class for pose estimation task
Add pose estimation training pipeline
Add RTMCC head and its loss module
Add pose estimation pose processor (for SimCC style head)
Add pose estimation metric PCK

More metrics will be added later

Add keypoint parameter in augmentation
Add PoseTopDownAffine transform
Add yaml configuration

cfg module has not updated
RTMPose model should have CSPNeXt, and it will be updated later

Changelog
If you PR to dev branch, please add a brief summary of the change to the Upcoming Release section of the CHANGELOG.md file and include a link to the PR (formatted in markdown) and a link to your github profile.
For example,
- Added a new feature by `@myusername` in [PR 2023](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/2023)

Code Formatting
If you PR to either master or dev branch, you should follow the code linting process. Please check your code with lint_check.sh in ./scripts directory.
For more information, please read the contribution guide in CONTRIBUTING.md.

This PR contains #352.
Remain works are as following:

CSPNeXt backbone should be added
Heatmap based pose estimation head should be added
More metric functions should be added
More augmentation techniques should be added
Update configurations
Update docs


[ARM] az deployment sub/tenant/mg create: Fix the KeyError: 'resourceGroup' in outputting results in table format when deploying non-resource group level resources

Issue: #17744
Description

For deployments that are not under the resource group level, the return data does not contain 'resourceGroup'. Therefore, the KeyError: 'resourceGroup' will appear in the original code
Testing Guide

Local test results

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change.
[Component Name 2] az command b: Add some customer-facing feature.

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] The PR title and description has followed the guideline in Submitting Pull Requests.

[x] I adhere to the Command Guidelines.

[x] I adhere to the Error Handling Guidelines.

ARM


Add changelog for v0.1.0a0

References
Backfill the v0.1.0a0 changelog using github-activity: https://github.com/executablebooks/github-activity
This is in preparation for having automated releases (#121, probably using Jupyter Releaser), so we can have a consistent changelog file and release content.

Code changes
None

User-facing changes
None

Backwards-incompatible changes
None

Ah looks like something is off with the RTD build.
Just rebased https://github.com/jtpio/jupyterlite/pull/109 and it's also happening there: https://readthedocs.org/projects/jupyterlite/builds/14071447/

looks like flit ain't happy for the py-in-js packages... not sure what to make of that... will take a look locally


feat: add border for the avatar of donating campers

Checklist:

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] All the files I changed are in the same world language (for example: only English changes, or only Chinese changes, etc.)
[x] My changes do not use shortened URLs or affiliate links.

Additionally, we could display an icon with the word "supporter " similar to the way we handle top contributors.
for reference:

I would make the border golden, and I would also like to see a badge similar to the top contributor.

Looks much nicer, I would change the badge icon for the supporter.

changed with the a heart.

Looks great, just a quick check if this respects the privacy settings.


Calling wasm-opt on cvizzu.wasm, also printing gzipped size.

@veghdev do we set emscripten environment variables using emsdk.sh in the CI build env?

@simzer In the container, the vizzu-devenv.deb package comes preinstalled. During the post-installation process, it invokes emsdk/emsdk_env.sh. However, some functionalities may differ when running as the root user. For instance, emcmake may not be available on the PATH and needs to be explicitly called using the complete path /root/emsdk/upstream/emscripten/emcmake (if I remember well)

it seems that EMSDK env variable is not set.

is emsdk_env.sh invoked or sourced? it should be sourced to be able to set the PATH and other envs

is emsdk_env.sh invoked or sourced? it should be sourced to be able to set the PATH and other envs

https://github.com/vizzuhq/ppa/blob/master/src/vizzu-devenv/DEBIAN/postinst
echo "source "$userhome/emsdk/emsdk_env.sh"" >> $userhome/.bashrc

So the build runs as root? Can we call cmake like this? source "<path to emsdk>/emsdk_env.sh & emcmake cmake ....."

modify: https://github.com/vizzuhq/vizzu-lib/blob/main/ci/cloudbuild/vizzu-devenv-prebuild-docker.yaml


What is the difference between instruct_alpha and retrieval_alpha models ?

I found your models on huggingface and i was wondering what is the difference betwen chip2_12B_instruct_alpha and chip2_12B_retrieval_alpha ?
Are they different in terms of training datasets ?

They are extremely similar. The only difference is that for the retrieval alpha the context used to generate the answer is put inside of special brackets
User: Here is my question
--[Article name: Article text related to the question]--
Chip2: Here is my answer using the text I looked up in the article.
Chip2 without retrieval doesn't include the article used to generate the answer. It works the same as all the other models I uploaded on HF. We are working on getting a good retrieval setup that is easily compatible with these datasets and will pbulish some more soon. Also on LAION.

@Rallio67 thank you for reply. I have a further question based on your response.
Is the retrieval_alpha model pre-trained with the same manner - concat the external article with the user input ?
If this is the case, how did you prepare the training datasets for the retrieval_alpha model ?
Thank you

In building the datasets, one type of commonly used approach is to take a context from some source text and then generate some questions related to that source text. In general with instruction tuning, I just use the question / instruction for the training. For the retrieval alpha instead of a tuple of (instruction,response) I have a tuple that is (instruction, context, response) and then that example is included in the training in the same was as with rallio_test.txt
So if you now have some external information you want to the model to use in consideration of its answer, then you paste it in between the brackets --[paste it here]-- with that exact format shown below. So in practice to use this you need to stop generation at the bracket, retrieve whatever the relevant text is via some kind of database lookup, and then close the bracket and the agent will generate an answer incorporating (hopefully) the text in the bracket.
User: Here is my question

--[Article name: Article text related to the question]--

Chip2: Here is my answer using the text I looked up in the article.<|endoftext|>```


Difference in number of apps when checked from PowerShell and COE tool

We are seeing more number of apps in COE tool when compared to the report for all apps in our default environment, Avanade(PRODUCTION). Below are few examples of apps we see in COE but not in PowerShell report:
91c18836-e0c3-4097-8313-3cc7f90b6f9a -- [LAMA] Catalogue de Formations MVP 2
250a9242-a60b-4317-a882-9623ae40f8ad -- PMT on Growth Market Deal Desk & Financials Team Site forms
Upon checking, check deleted flow is running successfully for this environment. Can you please check on this and let us know what else might be the reason for this inconsistency,

Sorry we cannot diagnose with this information. Can you please check, for both of your examples:

What version of Core are you running?
The table has a column called App Deleted, can you please see if it's marked as Yes for these two apps?
Is the app actually there? You can check by giving yourself permissions in the Set App Permissions app and then browsing there.

Hi,
Thanks for your reply. Below are my responses for your questions:

Version is 1.98
It is marked as FALSE for the above mentioned apps.
I tried giving permissions but it says not found.

I am unbale to find app in Power platform admin center. Please check and let us know if there is any other dependency which needs to be checked for this.
.

It sounds like the app is, in fact, deleted. We cannot do the work to cross reference against the powershell as it has its own peculiarities and we are not product support.
When you run Check Deleted next, it should mark the app as deleted in the CoE.


Finalize DataCommons Imports List

Theme
Type
Table
Notes
Internal Table #

1
STEM Degrees Conferred by Level
Time Series 2011-2021
IPEDS Table 318.45
Must split inconsistent unit types (%)
1

2
Salaries
Time Series or 2021 standalone
NCSES Table 4-2

14

3
Research Doctorate Recipients
Time Series 2011-2021
NCSES Table 1-10
Not STEM-Specific
25

4
Employed Graduates by Field
Time Series 2003-2021
NCSES nsf23306-tab006-002
Missing some years
26c

5
Grad Student Demographics in STEM
Time Series
NCSES Table 2-1
Not a duplicate of #1 because students instead of grads; inconsistent unit types; one table per year
7

6
Bachelors Degrees Conferred by Field
Standalone or time series
NCES 322.50
Need to also upload analogous male table; very rich and could be standalone instead of time series
11

7
Tenure status for STEM doctorate holders
Time series 2004-2017
NCSES 9-26
Separate tables for every other year
32

8
HBCU enrollment by sex
Time Series 1990-2021
NCSES 313.20

29

Potential Geo Data

Theme
Type
Table
Notes
Internal Table #

1
Federal funding for STEM programs by state
Time series 2012-2020
[nsf22342-tab003](https://ncsesdata.nsf.gov/explorer/datatables?page=1&survey=FSS)

32

Spelman has confirmed interest in sets 1-8 above, and has requested additional geo data.


Issue/clarification on iOS crash traces

Describe the bug
I'm trying to integrate Rollbar into an iOS (Obj-C) app to help track crashes. However struggling to get it to report into Rollbar properly.
My setup is as follows (following the docs):
- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {
    // Rollbar
    RollbarConfig *config = [RollbarConfig new];
    config.destination.accessToken = @"abcdef";
    config.destination.environment = @"prod";
    id<RollbarCrashCollector> crashCollector = [[RollbarPLCrashCollector alloc] init];
    [Rollbar initWithConfiguration:config crashCollector:crashCollector];

Rollbar lib versions:
    pod "RollbarNotifier", "~> 2.3.3"
    pod "RollbarDeploys", "~> 2.3.3"
    pod "RollbarPLCrashReporter", "~> 2.3.3"

I am trying to replicate a crash using:
[NSException raise:@"Invalid foo value" format:@"foo is invalid"];

I have turned off "Debugging" from Xcode just incase it was capturing it before the Rollbar SDK.
All I'm getting in Rollbar on the web is (at an info level):
The app was exited at the end of last session.

I would have thought it would have sent the stack trace to Rollbar?
Have I missed something?
Following this:
https://docs.rollbar.com/docs/apple#optionally-define-a-crash-report-collector-instance
Tried to look for examples, but they were all over 2 years old so not sure how applicable they would be with Rollbar's new Apple lib.
https://github.com/rollbar/rollbar-apple/tree/master/Examples
Calling the SDK from (x-mark all suitable):
[X] Objective-C
[] Swift
[] Other:

Hi @jamiepmullan , it looks like you did all the expected SDK hookup correctly.
Please, make sure all the proper filters are on within the Rollbar Dashboard (website). Crash reports are coming with Critical RolbarLevel.
Also, please, keep in mind that the crash reports will be picked up and reported to Rollbar.com on the next subsequent application startup after the crash (assuming the crash does not happen before the Rollbar was properly initialized with a valid config and a crash reporter module).
Feel free to let me know here if those suggestions did not resolve the problem.
Yes, the Examples are from the SDK's previous incarnation (as Rollbar-iOS) and eventually will be adopted for this SDK. You should be able to find up-to-date examples within Demos. All the Demos are parts of the Xcode workspace for the SDK. If you clone the repo and open the workspace in Xcode the Demos will be available within the Xcode and can be built and debugged individually right there.

@akornich , when you select that Error alert with (unknown) title on the RollbarDashboard and open up the latest occurrence of it, do you see a Raw Crash Report on the occurrence details page? Somewhat similar to this one...

@jamiepmullan , that is good news. Did you have dSYM for the build uploaded to the project?
If you have this message above the raw crash report:
This backtrace could not be symbolicated because a matching dSYM upload was not present
View the dSYM uploads received for this project here. Check out the docs for help in setting up iOS symbolication.
please, follow the instructions within the links of that message.
Once dSYM is uploaded, new reports should be symbolicated and titles of new occurrences should be updated when the symbolication succeeds.

@akornich ah! that might be it... interestingly 1 worked without having the dSYM uploaded...
Thanks for helping me clear that up.

@jamiepmullan , please, keep me posted on the outcome. By the way, the item title will not be updated even when the occurrence titles get fixed since the item title is based on the very first occurrence grouped into an item...

@jamiepmullan , can we close this item? or, are you still having some problems?


Explicit dependencies

Did you try to run the app locally? It doesn't work.

@shestakoven

Did you try to run the app locally? It doesn't work.

fixed

approved

rebased, squashed


Logo design proposal

Hi, my name is Valeria. I'm a graphic designer and I'd like to collaborate on your open source project and propose free logo/icon design for your app. If it's something you're interested in, please, let me know!
Best,
Valeria

Hi, @teneresa! Thanks for your kind offer.
Currently, there may be some higher-value UI/layout design work to do, but it may fall outside graphic design of logos, icons, etc.; https://github.com/zulip/zulip-mobile/issues/3632 comes to mind. If you're interested in that kind of work, please let us know! 🙂  But I'm closing this, as it's been almost two years, and there hasn't been a lot of activity.


Problem with installing chatterbot with pip

The error that I got:
Installing build dependencies ... error
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [91 lines of output]
Collecting setuptools
Downloading setuptools-67.4.0-py3-none-any.whl (1.1 MB)
---------------------------------------- 1.1/1.1 MB 2.5 MB/s eta 0:00:00
Collecting wheel<0.33.0,>0.32.0
Downloading wheel-0.32.3-py2.py3-none-any.whl (21 kB)
Collecting Cython
Downloading Cython-0.29.33-py2.py3-none-any.whl (987 kB)
-------------------------------------- 987.3/987.3 kB 6.9 MB/s eta 0:00:00
Collecting cymem<2.1.0,>=2.0.2
Downloading cymem-2.0.7-cp311-cp311-win_amd64.whl (28 kB)
Collecting preshed<2.1.0,>=2.0.1
Downloading preshed-2.0.1.tar.gz (113 kB)
---------------------------------------- 113.7/113.7 kB ? eta 0:00:00
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Collecting murmurhash<1.1.0,>=0.28.0
Downloading murmurhash-1.0.9-cp311-cp311-win_amd64.whl (18 kB)
Collecting thinc<7.1.0,>=7.0.8
Downloading thinc-7.0.8.tar.gz (1.9 MB)
---------------------------------------- 1.9/1.9 MB 7.2 MB/s eta 0:00:00
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Collecting blis<0.3.0,>=0.2.1
Downloading blis-0.2.4.tar.gz (1.5 MB)
---------------------------------------- 1.5/1.5 MB 7.1 MB/s eta 0:00:00
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Collecting wasabi<1.1.0,>=0.0.9
Downloading wasabi-0.10.1-py3-none-any.whl (26 kB)
Collecting srsly<1.1.0,>=0.0.6
Downloading srsly-1.0.6-cp311-cp311-win_amd64.whl (194 kB)
------------------------------------- 195.0/195.0 kB 11.5 MB/s eta 0:00:00
Collecting numpy>=1.7.0
Using cached numpy-1.24.2-cp311-cp311-win_amd64.whl (14.8 MB)
Collecting plac<1.0.0,>=0.9.6
Downloading plac-0.9.6-py2.py3-none-any.whl (20 kB)
Collecting tqdm<5.0.0,>=4.10.0
Downloading tqdm-4.64.1-py2.py3-none-any.whl (78 kB)
---------------------------------------- 78.5/78.5 kB 4.3 MB/s eta 0:00:00
Collecting colorama
Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Installing collected packages: wasabi, plac, cymem, wheel, srsly, setuptools, preshed, numpy, murmurhash, Cython, colorama, tqdm, blis, thinc
DEPRECATION: preshed is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
Running setup.py install for preshed: started
Running setup.py install for preshed: finished with status 'error'
error: subprocess-exited-with-error
    Running setup.py install for preshed did not run successfully.
    exit code: 1

    [26 lines of output]
    C:\Users\subho\AppData\Local\Programs\Python\Python311\Lib\site-packages\setuptools\installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
      warnings.warn(
    WARNING: The wheel package is not available.
    running install
    C:\Users\subho\AppData\Local\Programs\Python\Python311\Lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    running build
    running build_py
    creating build
    creating build\lib.win-amd64-cpython-311
    creating build\lib.win-amd64-cpython-311\preshed
    copying preshed\about.py -> build\lib.win-amd64-cpython-311\preshed
    copying preshed\__init__.py -> build\lib.win-amd64-cpython-311\preshed
    creating build\lib.win-amd64-cpython-311\preshed\tests
    copying preshed\tests\test_counter.py -> build\lib.win-amd64-cpython-311\preshed\tests
    copying preshed\tests\test_hashing.py -> build\lib.win-amd64-cpython-311\preshed\tests
    copying preshed\tests\test_pop.py -> build\lib.win-amd64-cpython-311\preshed\tests

╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.

You need an older version of python - 3.7.8 is the latest functioning version

How can we contribute to add Python 3.10+?

As noted in the link in the deprecation warning  you can add --use-pep517 to the install or prevent pip from reaching that version. the issue seems to be preshed needs to migrate to the new packaging standard https://packaging.python.org/en/latest/tutorials/packaging-projects/

How can we contribute to add Python 3.10+?

I don't think we can, as the problem is with one of the dependencies in the program, which hasn't been updated for a while-i can't remember what it is called off the top of my head.

I use python 3.6 installed ok


Raise the addon memory limits to 1G

The limits were chosen based on basic tests; they aren’t sufficient in various production scenarios. Pending actual scale testing which will help determine accurate limits, this raises the limit to 1G which should be enough for common clusterset sizes.

/approve

/cc @vthapar

We'll also need backport to 2.9.

We'll also need backport to 2.9.

/cherrypick release-2.9
/cherrypick release-2.8

/cherrypick release-2.8


Feature: Secrets backend support for sensitive config values

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3048
Originally created by @pnadolny13 on 2021-11-08 14:54:51

Problem to solve
This is based on a comment from https://gitlab.com/meltano/meltano/-/issues/2918#note_726141457.
As we start to think about supporting secrets backends we should implement it in a way that makes the multi tenant deployment use case easier too. Its common for users to ask how they should implement a multi tenant application that uses meltano as the backend where they need to have multiple credentials and settings per plugin (i.e. client 1 tap-postgres, client 2 tap-postgres,...). We usually recommend injecting environment variables to a single generic plugin config vs appending a new inherited plugin config to the meltano.yml. That approach leaves it up to the app developer to manage credentials and settings.
Target audience
Users with multi tenant deployments.
Further details
Proposal
If we build the secrets backend with properly namespaced keys then we could start recommending that app developer run meltano CLI commands to store and retrieve environment variables. Something like (idk the ordering so this is just for illustration):
meltano --environment=prod config <plugin> set --secret <key> <value> <optional_job_id>
In AWS Parameter store this could become a key like:
/meltano/prod/<plugin>/key>/<optional_job_id> = <value>
/meltano/prod/tap-postgres/password/job_123 = admin_password_123
My comment for retrieving them at runtime from here https://gitlab.com/meltano/meltano/-/issues/2918#note_726141457:

If the job_id isnt specified in an elt execution or theres no job_id specific secret then it just defaults to the generic plugin_name. Or we could use some inheritance there, take all generic secrets but job specific keys override generic.

What does success look like, and how can we measure that?
This would allow multi tenant users to not have to build a custom config/settings manager and not have to retrieve and inject the environment variables before execution.
Success would be if multi tenant app developers decided to migrate from their custom config managers to meltano managed.
Regression test
(Ensure the feature doesn't cause any regressions)

[ ] Write adequate test cases and submit test results
[ ] Test results should be reviewed by a person from the team

Links / references

We've created a multi-part set of specs that aims to resolve this:

https://github.com/meltano/meltano/discussions/6999 (this item)
https://github.com/meltano/meltano/discussions/7055
https://github.com/meltano/meltano/discussions/7056

Just a comment from office hour today, I see some of this is pretty close to what I am thinking of. My initial idea was a syntax like
config:
password: @azurekv('my-kv-name', 'my-kv-secret')

So there would be different "functions" for each type of secrets store, and Meltano would at runtime fetch the secret. This, however, would necessitate some secrets_config block that could contain some info about how to authenticate to the Secret Store. In Azure that would typically be related to Managed Identity (Identity IDs etc), with AWS that might be something like a role name.
Some other suggestions have a type of URI: akv://my-keyvault/my-secret or similar instead of the at-sign and pseudo-function - which might be a better idea for all I know.
Thinking this through though, I realized it is easier than I had thought to jinja this out completely:
from jinja2 import Template

def get_the_secret(value: str):
    """Cleverly get the secret from a Secret Store"""
    lv = list(value)
    lv.reverse()
    return ''.join(lv).upper()

template = Template(
    """
    properties:
      password: {{ akv('this-super-secret') }}
    """
)

rendered_template = template.render(my_secret="some_value", akv=get_the_secret)
print(rendered_template)

This is something a lot of users could implement themselves, using jinja inside their orchestrator. I know jinja-support has been a separate discussion, I don't know where that stands. Effectively these secrets would have to be evaluated at runtime, I don't know the innards of Meltano well enough to have an opinion about how easy/hard that would be. Half-baked thoughts are welcome.
fyi @pnadolny13 @edgarrmondragon @cjohnhanson @visch

#8276 might serve as inspiration for this


Preserve information about Excel's named ranges

In Excel, one can define a "named range". It would be nice to be able to preserve those ranges in the dataset. Perhaps something like this:
let workbook = excelReader.AsDataSet(namedRangesInTheirOwnTables: true)

This could create additional tables in the DataSet, one table for each named range.

I'd be happy to take this on, at least for the XLSX format. I would like to get some feedback my suggested approach, though. I can't think of a simple/elegant way without ending up reading the referenced worksheets twice. You can't do that now because you don't have random access to worksheets (it's just a forward-only cursor of forward-only cursors). I think having random access would be valuable in its own right performance-wise for large workbooks where you want only a fraction of the worksheets (my use case for this project).

I am ok with supporting random access. I think my only requirement would be that it doesn't negatively affect performance for the usual forward-only use.

Great, I'll have at it.

Thanks.  Can you do it in a branch off of develop?
On 6 Feb 2015 00:54, "Steve1961"<EMAIL_ADDRESS>wrote:

Great, I'll have at it.
--
Reply to this email directly or view it on GitHub
https://github.com/ExcelDataReader/ExcelDataReader/issues/25#issuecomment-73162093
.

Is this still something being looked into? I have a current requirement to find named ranges.
I like the product very much and don't want to have to move to OleDb
Thanks

Hi,
Probably not in the short term.
As far as I know, the ranges are expressed as formulas, so this feature depends on issue #332 initially. Next we need to parse the formulas in order to extract the ranges. I suspect both are non-trivial, alas.


Feature/improved python templates

Checklist

[ ] I tested the changes and all related features on the test server https://artemistest.ase.in.tum.de.
[X] I documented my source code using the JavaDoc / JSDoc style.
[ ] I added integration test cases for the server (Spring) related to the features
~[ ] I added integration test cases for the client (Jest) related to the features~
~[ ] I added screenshots/screencast of my UI changes~

Motivation and Context

According to ARTEMIS-359 there was no proper Python template included for Python programming exercises. At least, was not as extensive as the Java template.
Description

I added a new Python template, including structural and behavioral tests. I used the same instructions as in the Java template in order to be consistent. In the browser, while creating a new exercise, the markdown and preview also get updated when choosing a different programming language.
Steps for Testing

Log in to ArTEMiS
Navigate to Course Administration
Create a new programming exercise for any course
Switch between languages, the template (incl. the preview) should change accordingy

Please don't request a review before:

All tasks from the description are done and checked off
The build is successful

@thilo-behnke Sorry about the early PR. Since we are close to a release, I wanted to get some early feedback, so I can implement the changes and merge it. Bu you are right.


Fit Galacticus magnitudes/luminosities to CatSim SED library

In order to generate PhoSim-ready InstanceCatalogs, we have to be able to tell PhoSim where to find SEDs for the sources we are simulating.  Since Galacticus only outputs magnitudes and luminosities, it has been decided that we will just do a best-fit association with the SEDs in CatSim's library and feed those SEDs into PhoSim.  Performing this association has proven to be more tricky than I thought.
The attached plot shows the apparent magnitude residual as a function of R_v = A_v/E(B-V) for the six LSST filters.  I am plotting against R_v because the way that Galacticus assigns dust parameters has proven to be one of the difficulties (I have already backchanneled @evevkovacs about this, and she says that the Galacticus team is working on it).  In each plot, the horizontal axis is R_v.  The vertical axis is the apparent magnitude of the best-fit SED with dust applied minus the same quantity as reported by Galacticus.  The colorbar corresponds to the number of sources in that pixel of the plot.  This plot is only for the disk components of galaxies.
As I said, there appears to be a problem with how Galacticus assigns dust parameters, because we get some very large values of R_v.  Even if we were to cut out those anomalously high-dust galaxies, we would still have magnitude residuals of the order a few 0.1 magnitudes, especially in the bluer bands.  Is this acceptable?
Another question: The Galacticus magnitudes in this plot are calculated as follows:

take f = LSST_filters/diskLuminositieStellar:LSST_*:observed:dustAtlas
set m = -2.5*log10(f)
set m = m - 5.0*log10(1+z)

That last step confuses me.  I was told to expect to be off by a single 2.5*log10(1+z), since Galacticus does not apply the stretching of photon frequencies when calculating its 'observed' luminosities.  However, to get good agreement with the CatSim fits, I had to subtract 2*2.5*log10(1+z).  I am both surprised by the extra factor of 2 and by the fact that I had to subtract rather than add.  Could the Galacticus experts speak to what I should have to do to f in order to get the actual magnitude observed by LSST? @evevkovacs @dkorytov

We will look into this. BTW, for the record, in Galacticus, the relationship between the observer-frame luminosities and the apparent magnitude  (which measures flux) should be:
m_apparent= -2.5log_10(Luminosity in observer frame)+ mu -2.5log_10(1+z)
where mu is the distance modulus. (The reason is that Galacticus does not bother to carry the (1+z) common factor throughout the luminosity calculation)
@egawiser This catalog currently contains both luminosities and magnitudes. Will the filter luminosities be of use to the working groups, or are they solely interested in magnitudes?

What distance is used in the distance modulus?  My understanding is that, in cosmological contexts, the -2.5*log10(1+z) factor should already be included in the distance modulus, because of the (1+z) factor in the luminosity distance.  Are you using the angular diameter distance to calculate the distance modulus?

My understanding is that mu is the luminosity distance and the factor of (1+z) comes from the change of  integration variable  when integrating the SED over the  blue-shifted filter band pass. ie the endpoints of the integration change, and there is a scale factor change of 1+z.

@evevkovacs I agree with that explanation.  The solution to some of these questions would be to use flux densities in microJansky instead of magnitudes.  The familiar formula is
L = f* 4 pi d_luminosity^2 .  But for flux density in f_nu units (which is the case for microJansky or AB magnitudes), you have to divide d_luminosity^2 by (1+z).  You would think that the distance modulus would include this, since it converts between absolute and apparent magnitude but it doesn't - this was initially defined for low-redshift applications, so that term is in the infamous "k correction" that is part of the official M-m formula but seldom written out explicitly in textbooks.
Filter luminosity densities (in L_nu units) would seem to be potentially of interest, but if they can be derived from the filter magnitudes and true redshift using a stated formula, it shouldn't be necessary to list them in the catalog.

@danielsf I'm really baffled why you're varying R_V, and over such a wide range while trying to find the best-fit CatSim SED template to the reported Galacticus photometry.  I don't know what dust model Galacticus is using, but I'd bet R_V is in the range of 3 to 4 and possibly a fixed value if a single dust law is being assumed for all galaxies.  Without more information on how Galacticus modelled the dust, I don't think this approach to fitting will be very meaningful.

I am not varying R_v.  I am taking it from Galacticus.  Galacticus also returns V and B band magnitudes with and without dust.  I am using those magnitudes to find the A_V and E(B-V) that Galacticus reports and using that to set R_V.  The anomalously large R_V values are due to a known bug in how Galacticus interpolates fluxes from its star formation model (at least, this my understanding from private communications with Eve).

Thanks for the explanations on the magnitude-versus-flux problems.  I will compare them to what I am finding in my fits and make sure that the minus signs all line up.

Thanks for explaining the R_V situation.  As long as everything you're using is in rest-frame B and V bands, that should be self-consistent - though it's still hard to understand how the apparent bug produces such crazy R_V.

@egawiser  @danielsf  It's not the star-formation model but the dust correction that is the problem. The dust correction is computed by interpolating or extrapolating a set of look-up tables which depend on galaxy properties. Sometimes that interpolation/extrapolation goes haywire. Andrew Benson worked a lot on fixing this, but it seems to have cropped up again. Rv is crazy because the dust correction produces a totally extincted galaxy.

And just to be completely explicit, here is the formula for converting Galacticus observed luminosity to apparent magnitude:
m = -2.5log_10(L) - 2.5log_10(1+z) + 5*log_10(DL(z)) + 25
where L is the luminosity, DL(z) is the luminosity distance in Mpc, and the 25 comes from converting Mpc to pc (because absolute magnitude is defined at a distance of 10 pc). For observed luminosities, the SED of the galaxy is integrated over the blue-shifted wavelength range of the observer-frame filter.

Okay.  I now have the SED fitter working such that the magnitudes I get from the fit SEDs are consistent with the conversion between Galacticus flux and magnitude given by Eve above.  That being said, there is still a bit of slop.  These figures compare the Galacticus magnitudes to the SED magnitudes with and without dust for a random set of 56,000 galaxy disks. The numbers next to the plot titles are the first, second, and third quartiles in |m_galacticus - m_SED|.  I am posting these in case anyone feels that a disagreement of 0.1 magnitudes is totally unacceptable.

Cool, thanks Scott! This looks like it needs some science team inspection: @fjaviersanchez which DC2 analysis paper authors should be looking at these plots?

Hey Scott (@danielsf), which redshift are you are using to calculate the magnitudes? The default "redshift" has the effect from peculiar velocities included. "redshift_true" is cosmological redshift and it's the value that I use to compute magnitudes in protoDC2.  Maybe this is the cause? I've did a quick test:

@dkorytov Right... I am using the total redshift.  I will change that.  I probably won't have time to re-run my validation test today.  I'll try to rerun the validation test and post new results in the next few days.
Nice catch.

Pardon my ignorance, but which is the best one to use? The luminosity distance is usually calculated in terms of the cosmological redshift, and as you go out in z, the peculiar velocity matters less.  But I would think that the (1+z) factor for the photon frequencies would be affected by the peculiar velocity, as would the limits of integration over the filter.  The last point is moot, because we are not even k-correcting the filter luminosities for the change from snapshot redshift to lightcone redshift.  What exactly do the observers measure? I have asked Eric Gawaiser the above question, but he hasn't answered yet.

I agree with @evevkovacs that luminosity distance only depends on the cosmological redshift but that a doppler shift caused by peculiar velocity further affects the frequency of each photon and hence flux density (=magnitude).  Peculiar velocity should cause scatter but not bias, as seen in Dan's plot.   Scott could test this by plotting magnitude discrepancy versus redshift (and even more directly, magnitude discrepancy versus peculiar velocity).  But sure, if the discrepancy disappears after switching to redshift_true, no need to worry about this further.

Distinguishing between true redshift and full redshift does not improve the scatter plots above.  If people want to check my work, the data for the plots is generated by this script
https://github.com/danielsf/gcr-catalogs/blob/check_redshift/GCRCatSimInterface/validation/test_sed_fitting.py
@evevkovacs Regarding the 2.5*log10(1+z) factor that must be added to the catalog magnitudes to get the actual magnitudes: is that z true or full?

Correcting that error on my part did not seem to help the scatter

I took a quick look at your code but haven't digested it yet. Can you see if there is any z dependence to the scatter? That might give us a clue.

@danielsf @dkorytov  OK, sorry I have been asleep at the switch. The Galacticus luminosities for both the LSST and the SED filters are all evaluated at the snapshot redshifts. If you are doing a consistency check between your fits to the SED filters and the Galacticus values for the LSST filters, you need to use the snapshot redshifts for each galaxy, rather than the lightcone redshifts. I believe that Dan has saved the snapshot  number for each galaxy in the galaxy properties. We will need to give you a table that converts that to redshifts. I have it somewhere and will find it and will send it to you. In the next catalog iteration we will add that table to the catalog metadata.

@evevkovacs what's the difference between redshift_true and snapshot redshift?

redshift_true is the lightcone redshift (ie redshift obtained when galaxies, which have been evaluated  at the simulation snapshots are interpolated onto the observer's lightcone). (Recall that the SAM is run on the simulation snapshots)

@evevkovacs's insight that the observer frame luminosities are at snapshot redshifts improves the fit, but only marginally.  I also plotted the magnitude fits in the rest frame (which should be immune to the effects of diverse redshift definitions).  There is still a delta_magnitude between Galacticus' calculations and the CatSim SEDs on the order of a few 0.01 magnitudes.  Recall that the numbers in the title of each plot denote the first, second, and third quartiles in delta_magnitude.
(the plots are all combinations of rest/observer frame and dust/no dust)

Focussing on "rest-frame no dust", you say the differences are ~few x .01 mags. Can this reasonably be accounted for by the differences between the SED template  fit by CatSim and the "true SED", which is only coarsely approximated by the SED filter luminosities supplied by Galacticus? The width of the wavelength bins for the SED filters in Galacticus varied from 250- 600 angstroms over the wavelength range 3000-10000 angstroms, so there's some slop there. For example, u-band effectively covers the range 3200 -4050, and is spanned by 3 Galacticus SED filters. And most of the filter  throughput  goes into just one SED filter.  I have no feel for how much SED templates vary in this wavelength range. How good are the SED fits?

I am attaching a text file that lists the magnitudes (unnormalized by any distance modulus) of the CatSim SEDs in each of the Galacticus tophat bands.  I'm not sure which plots would be most helpful, so I'm just going to put the data here.
I do suspect that what we are seeing is the natural slop one would expect from approximating one SED grid (Galacticus) with another (CatSim).  I don't have a strong opinion whether or not this much slop is acceptable or not.
CatSimMagGrid.txt

For now we agreed that due to the approximations the slop is expected. So I will close this issue. If people feel like we need more tests we should reopen.


Test and implement cascade deletes for many to many

We want to cascade destroy dependent many to many join relationships
when the main record is destroyed.
Fixes #76

@tmfrnz I think this does what you wanted but please let me know if you find any that are missing!

looks good, though measure_actors and actor_measures it appears that this may still be missing for measures
https://github.com/dumparkltd/marine-defrag-server/blob/master/app/models/measure.rb

looks good, though measure_actors and actor_measures it appears that this may still be missing for measures
master/app/models/measure.rb

give that a go now @tmfrnz 😄


Type 9 SOG is inconsistently coded

Hey,
Thanks for go-ais!
I ran into a little surprise this week when we noticed that type 9 StandardSearchAndRescueAircraftReport does not encode its Sog field like the others:
https://github.com/BertoldVdb/go-ais/blob/fd4fd5b2687ae6ab50065f6702d2645c54d32c98/messages.go#L126
It has a plain uint16 rather than a Field10 which decodes to scaled float.
Fixing this is a breaking change, but it also seems like a better experience to be able to count on all 10-bit SOG fields being handled the same (Type 27 6-bit SOG is obviously an exception).
Thoughts?

I just noticed that this is probably intentional and correct, sorry about the false alarm.
https://gpsd.gitlab.io/gpsd/AIVDM.html#_type_9_standard_sar_aircraft_position_report says:

Speed over ground is in knots, not deciknots as in the common navigation block; planes go faster.
The special value 1023 indicates speed not available, 1022 indicates 1022 knots or higher.

So the type 9 definition should be different from the other 10-bit deciknots fields.
Closing.


The webonary-create-site plugin is not sending welcome email

I created a new site on webonary.work with my<EMAIL_ADDRESS>email, after deleting this user with all the attached sites.
This is what it said:

This is the email I received on<EMAIL_ADDRESS>Dear User,
Your new Webonary site has been successfully set up at:
https://www.webonary.work/test-stutzman13/
Your new site comes with a standard set of pages for Front Matter and Back Matter dictionary materials. Please do not remove these pages. The Webonary team can help you provide the necessary information for these pages. It is also important that you do not remove or reorganize the menu items. They have been chosen with great thought.
You can log in to the administrator account with the following information:
Username: verna
Password: 5R8yRNn4WtM7
Log in here: https://www.webonary.work/test-stutzman13/wp-admin
You may access helpful information here: http://www.webonary.work/help/. When you are ready to publish or if you have any questions, please contact us at<EMAIL_ADDRESS>In order to stay informed about Webonary developments, please sign up for the Dictionary & Lexicography Services newsletter at https://www.webonary.work/join-the-dictionary-lexicography-services-mailing-list/. You may also join the Google User group, https://groups.google.com/g/webonary-list, for interaction with other users.
You are invited to take the Dictionary-Making and Lexicography Course at https://sites.google.com/sil.org/dls-course/.
We hope you enjoy your new site. Thanks!
--The Webonary Team

I was able to log in successfully using the username and password from the email.


Addressing Comment #3 - Analysis by institute

The end goal is to have a table that looks like this:

Avg. APT (95% CI) - "0.9 (0.85-0.95)"
CPOF/availability - "###.#"
Here's how we could get there. Data is saved from each award here, "administration" is the institute that gave the award:
https://github.com/sopeeweje/NLP-AI-Medicine/blob/378991ac290e414e6ea01068d9b322b8fba51c3c/feature_extraction.py#L59-L71
Get papers by institute by doing the following:

Load data.pkl from the main directory (list of dictionaries representing each award). Use these lines to get the number of awards and total value of the awards by institute:
https://github.com/sopeeweje/NLP-AI-Medicine/blob/378991ac290e414e6ea01068d9b322b8fba51c3c/feature_extraction.py#L167-L176

Group the awards by "administration", where project_number is the value of the "project_number" key in each awards dictionary (called "by_institute":
{"Institute1": [project_number1, project_number2, ...] , "Institute2": [project_number1, project_number2, ...], ...}

Steal these lines directly to get the info on the papers:
https://github.com/sopeeweje/NLP-AI-Medicine/blob/378991ac290e414e6ea01068d9b322b8fba51c3c/analyze_clusters.py#L384-L397

"output" will be structured as:
{PMID1: {citations: ###, apt: ###, project: ###, year: ###}, PMID2: {citations: ###, apt: ###, project: ###, year: ###}...,
"project" is the project number for that paper, which corresponds to project_number above.

Iterating over "by_institute": For each institute -> for each project_number "idd" ->
papers_institute_citations = [output[key]["citations"] for key in output if output[key]["project"]==idd]
papers_institute_apt = [output[key]["apt"] for key in output if output[key]["project"]==idd]
papers_institute_availability = [output[key]["year"] for key in output if output[key]["project"]==idd] ***list of years that will need to be converted to availability

Sum over each of these lists for each institute. 95% CI for APT can be calculated with:
https://github.com/sopeeweje/NLP-AI-Medicine/blob/378991ac290e414e6ea01068d9b322b8fba51c3c/analyze_clusters.py#L427-L430

Return a CSV that matches the above picture

This can all be done in an independent function in analyze_clusters.py

Looks good 👍🏾


Enable CI for DDS Record Replay v3

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Please upload report for BASE (integration/bump-3.0.x@e048a87). Learn more about missing BASE report.

Additional details and impacted files
@@                    Coverage Diff                    @@
##             integration/bump-3.0.x     #155   +/-   ##
=========================================================
  Coverage                          ?   26.25%
=========================================================
  Files                             ?       56
  Lines                             ?     3668
  Branches                          ?     1813
=========================================================
  Hits                              ?      963
  Misses                            ?     2023
  Partials                          ?      682

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Ability to scrape javlibrary actresses including other names in parenthesis?

I'm wondering if there is a way to have javlibrary scrape the actress names without stripping out the alternate names.  The ability to look at the csv and pick the oldest name is actually really useful but my library has already been scraped using the complete javlibrary name so I need to stick with the same convention.
Is there a way to, for example, have it keep Hasumi Kurea (Adachi Ami) as the name instead of stripping out the (Adachi Ami ) part?

I think the main issue is that if javlibrary ever adds additional actress aliases to the site, the name will change. I will add a way to include all aliases as separate actresses in the metadata as per #290.

Added sort.metadata.nfo.addaliases in 2.5.7


Could not load local trace TypeError: Failed to fetch

Hi, I install the Chrome Trace Viewer extension on VS Code. I generate the json file on the remote workstation and want to view the json through chrome trace viewer extension. I got this error from ui.perfetto.dev "Could not load local trace TypeError: Failed to fetch". Any reason why ? Is it because I need password every time I need to download files from remote workstation to my local machine ?

Good find, this sounds like a use case that I didn’t initially imagine when I wrote this, and sounds like a bug in the extension. This extension works by starting up a local web server that Perfetto requests for the trace file — this is necessary because web pages can’t load arbitrary files from your computer without user interaction for security.
My guess is that the server runs on the remote workstation, and so the browser on your dev machine fails to connect to the server. The solution is probably to make sure the extension copies the file over to the dev machine and launches the server from there. Since I don’t currently have a workflow with a remote workstation, is this an issue you’re interested in debugging?

Thanks for the reply. I am interested in debugging this issue. But I do not have a clear strategy on how to debug this ? May I ask whether there is some guidance on debugging this issue ?

I actually have the same usecase, and found this issue


Apply code suggestions and formatting from VS2022

Apply code suggestions and formatting from VS2022 with default settings
Add an editorconfig file based on these settings

The changes are very good in my opinion and as an F# user I find the code analysis that C# users get is pretty nice.
The diff is huge because of the auto-formatting so this would mess with any other active branches if any. Formatting rules could be changed in .editorconfig to reduce this diff.
Fixes https://github.com/peteroupc/CBOR/issues/62

You should update this pull request with indent_size = 2 and csharp_indent_braces = true, and in general any other formatting rule that minimizes the size of this pull request.

I assume the current formats are based on customization on your system? Can you export an .editorconfig file for this? For example from VS via https://stackoverflow.com/a/56954297/4903561 . That would allow me and others to contribute to the project. Currently if you for example delete a closing curly bracket and re-add it, a large and incompatible formatting change results since VS assumes standard formatting.

I have created an .editorconfig with the changes indent_size = 2, tab_width = 2, and dotnet_diagnostic.SA1210.severity = none (to disable enforcing alphabetical order on using directives).  Rebase this pull request with these changes, please.

Redone PR in https://github.com/peteroupc/CBOR/pull/67


As a Developer, I need Mina hash implementation for both p2p and ledger data types, so I can use it in all layers of the Node

Definition of Done: interfaces for (most) p2p and ledger types that are:

consistent with OCaml
covered with tests
easy to modify as type structure changes

Assuming we use our fuzzing data generator and the driving code is a Rust program, on the OCaml side we would need a mapping from a string identifier to a pair of a binprot_decoded_function and a hash function. Then as inputs (either through FFI or stdio IPC) the Rust driving code will pass a string identifying the type, and a binprot encoded value. The OCaml code will get the matching decoder and hashing functions, decode the data, hash it, and send back the result to the Rust code which will then compare both hashes.


Produce Containers for Branches

A commit or merge into any branch causes a rebuild of the main branch's containers.
Need to figure out how to restrict that to just main and possibly do builds for other branches.

GitHub already does this.


Unknown parameter in input: "AdminId", must be one of: DetectorId, MasterId, InvitationId

Errors in cloud watch when lambda is deployed
line 458: AdminId=str(admin_account)
full log from cloudwatch:

Sorry duplicate of:
https://github.com/aws-samples/aws-control-tower-guardduty-enabler/pull/21


PLA-221 Rework PullRequestTest to work with non npm package managers

Add a run-script-job that sets up a generic workflow capable of working with npm, pnpm and yarn; update the PullRequestTest workflow to use the new jobs; update tests and templates accordingly.

@quesabe can we release it to *.*.*-canary version? So that @igorosip0v can play with it in real repo?

@quesabe can we release it to *.*.*-canary version? So that @igorosip0v can play with it in real repo?
@gvidon, I guess we can. The workflow we have does not support such a variety, but I can do it manually.

BTW do we expect canaries to be rarely used or not? If we want a long lasting solution for canary releases maybe I should create a branch canary for such releases and configure the workflow... And for now is it ok to release from the PR branch? Otherwise I could set up the canary workflow first and then use it here, but it would slow down the advance with this PR.
Please let me know which path you consider more suitable.


Nodemailer is not sending all the mails, missing in Openshift hapijs

I m using nodemailer in openshift and hapijs , Nodemailer is not sending all the mails, example if i try to fill contact us form requests more then one at a same time from different computers, nodemailer do not send confirmation email to all users who submitted form, may be due to busy in sending first email. how to solve this problem.

Server throw [Error: Connection timeout] code: 'ETIMEDOUT' error few times...
Can you tell me why server is throwing this error few times.


[RFC] Authentication & authorization for workload/pod Identity in CoCo

Background
For a long time, identities are ignored in CoCo, which could cause some problems.

Evidence factory attack: Currently, once a pod passes the Remote Attestation, it can access KBS via AA. If a container inside the TEE escapes, and calls corresponding API to get evidence of the TEE, the container can then access arbitrary KBS resources because the evidence will pass the remote attestation.
If a malicious CSP deploys many TEE with the same TCB without informing the user, these TEEs can pass Remote Attestation of AS provided by the Relying Party, and then request different resources from KBS.
Identity of KBS: For now, we do not have a mechinism to efficiently deliver KBS public keys from the Relying Party to the TEE, especially in those scenarios where a private CA is involved. This will make the TEE connect to a fake and malicious KBS and do unexpected behaviors.

Introduction
This proposal will aim to deal with the problems all above, mainly via a new mechinism named identity provisioning. The core idea is to bring authentication and authorization mechanism in CoCo. Before describe the high level design, let's briefly introduce some different identities.
Let's map three classic identity concepts to CoCo:

TEE environment TCB: s.t. device identity traditionally. It is determined by [TEE hardware TCB] + [underlying softwares artifacts like guest firmware/kernel/kernel parameter/rootfs]. Currently, we use TEE environment TCB to ensure the TEE is safe. This identity can confirm the confidentiality of contents inside a Pod. If an entity does not have a TEE environment TCB identity, it cannot access any KBS. However, this identity can be faked via evidence factory attack.
Tenant/User/Relying Party: s.t. user identity traditionally. This identity can determine who owns the pod/workload. Indirectly, it can also affect whether it currently has access to a certain KBS resource or a certain KBS. For example, Alice has set up a KBS whose uri is http://alice-host:8080, then Alice has permission to access kbs://alice-host:8080/* but Bob does not.
Workload: s.t. application identity traditionally. The identity of the workload describes the "usage" of a set of containers inside the guest. In this way, workload identity shares the same as guest identity. This identity will also affect whether it has access to a certain KBS resource. For example, an workload identity can realize the following scenario. Suppose Alice has a KBS whose uri is http://alice-host:8080. Alice has a machine learning workload which may access to kbs://alice-host:8080/my/machine-learning-data-set/*. At the same time, however, Alice's MySQL workload can not access kbs://alice-host:8080/my/machine-learning-data-set/*, but it can access kbs://alice-host:8080/my/mysql/* which the machine learning workload cannot access.

Overall, when a KBS resource is requested, a token or credential which is derived from the combination identity of the three identities should be carried. With the identity of user and workload, only a limited range of KBS resources can be accessed. Even if a container escapes, it can only access the pre-defined range of resources that the entire workload can access. In this way, the evidence factory attack does not matter.
Among the different identities, TEE environment TCB can be verified by the remote attestation process. The other two identities, tenant identity and workload identity require more design, s.t. authentication and authorization. We want to propose a protocol independent of the underlying hardware.
Overall Design
Let's quickly clarify what we want to authenticate and authorize.

Authentication: To confirm the identity of a workload. After a VM is started, an authentication process should occur. The VM needs to access Relying Party side with workload-specific ticket (same as workload id in the following). The ticket should be provisioned to CSP by tenant before the TEE starts. The tenant, knowing the corresponded ticket of every workload, then validates the ticket. There are two things the tenant should do in this stage.

Validate the legitimacy of ticket to check whether it has been defined and registered before
Mark the ticket as registered. When a duplicated ticket comes, reject it

Authorization: When the authentication process succeeds, the tenant can ensure the VM is legitimate. Then it can give enough but limited permission to the VM to access different KBS resources. The permission could be a token or certification. Then, when any confidential resource is needed, the VM/workload can use the token to request.

Pre-provisioned Identity Protocol
Let's see detailed design.

Let's see the different componants:

PS: Provisioning Service. This service is owned by relying party. It is responsible for:

Provision workload id/ticket to CSP.
Authentication: Based on some workload specific information to identify the VM.
Authorization: According to the KBS resources that the workload needs to use, generates a token with a specific identity and permissions and provisions

AS: Attestation Service. Verify different TEE evidence, to ensure the execution environment is a legal TEE environment TCB.
KBS: Key Broker Service. The service to store confidential resources. Current KBS implementation will firstly verify TEE environment TCB, then distribute confidential resources. The difference of this proposal is that here KBS verifies a token generated by PS and then distributes confidential resources. It will not do verification of TEE environment TCB or remote attestation.
AA: Attestation Agent. Currently AA is responsible for collecting TEE evidence which is needed in remote attestation and accessing KBS. In this proposal, AA should be the requester of authentication. Also, it should work as a stub inside guest to handle all requests inside the guest to access confidential resources from KBS.

Authentication
The first question is what we can use to authenticate a specific VM/workload.
The answer is "Embed workload id information into the remote attestation report" or "leverage nonce-like field in the remote attestation report as workload id".
In this way, a remote attestation report can be represented abstractly as [TEE environment TCB] + [workload id]. Attestation-Service can help us to verify the [TEE environment TCB] is expected. Then it can extract the [workload id] out of the report, and deliver to PS to check whether the [workload id] passes authentication. It is significant that every [workload id] must be used ONLY ONCE. In this way, we can prevent a malicious CSP to use a same [workload id] to do replay attacks.
Suppose a tenant wants to launch one MySQL workload. The basic flow is as following:

She needs to prepare one [workload id] -- for this MySQL. She knows this [workload id] is associated with this workload. She register the [workload id] in the PS hosted by herself, and at the same time the [workload id] then is sent to CSP
CSP launch the guest and use a proper way to inject the [workload id] into the guest. For example, TDX could use MRCONFIG, SEV-SNP could use HOSTDATA.
Once the guest is started, AA will make an authenticating request to PS. Let's assume it is a rpc named "registration". The request will carry the remote attestation report together with the [workload id]. The integrity of the [workload id] is protected by the report signature.
PS delivers the remote attestation report to AS. AS verifies the attestation report for [TEE environment TCB]. Then extract the [workload id].
PS validates the [workload id].
PS signs and provisions a token using the user's private key. The token contains its permissions, which is decided by PS due to the usage of the workload. Together, PS can also send back the KBS public keys together.
Whenever a confidential resource is requested, AA just carries the signed token.

Authorization
After the authentication, the tenant is able to authorize the workload enough permission to access KBS. It can send back a token, or a certificate which contains enough information about the identity of the tenant and the workload. The token itself can be signed by the tenant to show that this token is provisioned after a successful confirming of [TEE environment TCB], s.t. remote attestation.
In the same send-back process, the tenant can also send the public key of the KBS to the VM.
Then, when the VM needs to retrieve a confidential resource, like an image security policy JSON file, it can use this token to make requests.
Implementation
Certificates and Keys

There are some keys and certificates in the proposal:

Relying party's private key. Every tenant should keep an asymmetric key pair. The private key is used to sign token by PS to provision to AA.
Token: After a successful authentication and authorization (we once used "registration" to refer in previous parts), AA will get a token. This token is signed by the user's private key. It includes accurate permission information about where it can access.
KBS's HTTPS public key certificate. The "registration" process can also provision KBS's public key certificate to AA. It is used by AA to ensure the identity of KBS when requesting confidential resources.
Relying party's public key certificate. This certificate is used by KBS to verify the token.

Here we distinguish the Relying party's public key certificate and KBS's public key certificate. As potentially this can be easily to extend to support a third party attestation service if needed, where KBS is kept by user but PS+AS by the third party.
Software Stack
We can extend the functionality of HW-TEE attesters in Attestation-Agent cc-kbc, by which we can inject workload into remote attestation report.
And for the AS side, we can add related logic into the hardware verifier to extract the workload id. Then it can be sent to PS.
Confidential Resource Distribution Overview
Resource Registration

Tenants deliver their public key certificates to a KBS, which could be a process for tenants to register themselves to a KBS.
Tenants push different resources into a KBS. The resource can be uniquely identified by a KBS Resource URI

Resource Retrievement

Tenants use different workload id to distinguish different workloads. Different workload corresponds to a subset of KBS resources.
AA does "registration" using remote attestation report and the workload id.
Tenant verifies the attestation report, and validate the workload id. Then provisions a token with access permission of the subset of KBS resources that the workload needs. The granularity of authorized resources can be marked with KBS Resource URI.
The workload will then access different KBS resources using the token.

How can this protocol resolve the problems mentioned

For evidence factory attack, it can be prevented by a mapping maintained by PS. This mapping contains information of different workload id. Each id can register only once. When a request of a duplicated workload id comes, it will reject provisioning new tokens, which prevents from "cloning" new TEEs. Also, a token inside a VM has limited permission to KBS. Even if a container escapes, it will not get "free" permission to all KBS resources, but the limited permission that the token has.
For KBS identity, the public key can be distributed by the PS. There might be a new problem of the identity of PS. This doesn't matter as once a fake PS is connected, the tenant can find that the workloads whose id are kept in the PS are not registered in a timeout.

@Xynnn007 thanks for bring this concept, which looks much similar as the SVID described in spiffe.io, specifically the approach described in section A Day in the Life of an SVID in https://spiffe.io/docs/latest/spire-about/spire-concepts/, possible we offering the service which meet the spiffe standard?

@Xynnn007 thanks for bring this concept, which looks much similar as the SVID described in spiffe.io, specifically the approach described in section A Day in the Life of an SVID in https://spiffe.io/docs/latest/spire-about/spire-concepts/, possible we offer the service which meet the spiffe standard? I know some other services also use spiffe APIs, like istio.

To my understanding, SPIRE gives a good solution of layered-authorization for cloud native. In CoCo, we will face the same issue but now we actually pay little attention to. Let me try to explain:

Now we are ignoring the privileges of the entities inside a Pod. That means anyone who can get the evidence could pass the KBS protocol and achieve the whole access to the KBS. But actually, different components (attestation-agent, app container) MAY have different privileges.
We SHOULD limit who can call the API of get_evidence, s.t. only something like attestation-agent who can represent a pod and have root privileges. If not, the risk is any entity like an app container can get the evidence and be involved in the protocol from the AA and AS, thus letting the app container get the "role" and privilege of the whole pod. Tobin once made a great speech about a similar scenario, where an escaped and malicious container could get the evidence of the CVM.
After the authorization, a SVID in cert should be given from PS to AA. Then, all app containers' identity should be certified by AA's SVID. This is important when different container with different privilege is required.

Apart from SPIRE & SPIFFE's SVID, we still have some more ways to deal with different application scenarios. They are based on how to access the confidential resources. About this, we need to combine RA with different auth schemes like OIDC, etc. The core idea is ONLY the one who passes RA-based authentication protocol can get the authorized credential.


read event fires twice.

Somewhy the 'read' function is fired multiple times. It will fire for eevery photo that has been taken, so every new photo increases the number of times 'read will be called'

@dsteinborn's update should have fixed this. Closing now, let me know if not.


Add osx-arm64

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge-admin, please rerender

This is ready to go @conda-forge/geographiclib-cpp

I'll take a look at this today.

Thanks @cffk - you are the only maintainer of this repo, so it would be great if you could merge :)

Happy to add myself as maintainer if you want.


Cleanup: deprecate and remove qps_driver

Now when we have qps_json_driver and run_performance_tests.py scripts, I think we can get rid of the old qps_driver and some of the bash scripts that run performance test scenarios.

Yes, we need to do this. This means some migration of the contents of my recent PR.


add solution for these problems in C++

Add solution for these problems and add their solution links under solution section in readme file
1. Char
2. Calculator
3. First digit !
4. Coordinates of a Point

Sir, could you please assign this task to me?

Sir, could you please assign this task to me?

Hi @avinashdhakne , Yes just fork the repository and give a start to it.
and start working on it.


[basisu] update to v1.50.0

Patches upstreamd here: https://github.com/BinomialLLC/basis_universal/pull/383

[x] Changes comply with the maintainer guide.
[x] SHA512s are updated for each updated download.
[x] The "supports" clause reflects platforms that may be fixed by this new version.
[x] Any fixed CI baseline entries are removed from that file.
[x] Any patches that are no longer applied are deleted from the port's directory.
[x] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[x] Only one version is added to each modified port's versions file.

For optional targets, it is sometimes less intrusive to only control build and install instead of adding conditions everywhere. i.e.
# not conditional; not modified
add_library(examples ...)
target_link_libraries(examples PRIVATE mainlib)

# only condition
if(EXAMPLES)
    install(TARGETS examples ...)
else()
    set_target_properties(examples PROPERTIES EXCLUDE_FROM_ALL 1)
endif()

Thanks for the tip. Here I went with the proper way as I did the same in the upstream PR.

Usage test passed on x64-windows triplet.


pull request for e2e test at 01/31/2017 10:45:26

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Next build failed: Can't resolve 'react'

Hi, whenever I npm run build my Next.js app, I get a webpack error. This may be something to do with #129 but I'm not building with any local modules/dependancies.
wiktor@MANTIS:~/Development-Linux/workspacemodels$ npm run build

><EMAIL_ADDRESS>build
> next build

info  - Loaded env from /home/wiktor/Development-Linux/workspacemodels/.env.local
warn  - SWC minify release candidate enabled. https://nextjs.org/docs/messages/swc-minify-enabled
Generated 9 documents in .contentlayer
info  - Checking validity of types
info  - Creating an optimized production build
Failed to compile.

./node_modules/next-contentlayer/dist/hooks/useLiveReload.js
Module not found: Can't resolve 'react' in '/home/wiktor/Development-Linux/workspacemodels/node_modules/next-contentlayer/dist/hooks'

Import trace for requested module:
./node_modules/next-contentlayer/dist/hooks/index.js
./pages/about.jsx

./node_modules/next-contentlayer/dist/hooks/useMDXComponent.js
Module not found: Can't resolve 'react' in '/home/wiktor/Development-Linux/workspacemodels/node_modules/next-contentlayer/dist/hooks'

Import trace for requested module:
./node_modules/next-contentlayer/dist/hooks/index.js
./pages/about.jsx

./node_modules/next-contentlayer/dist/hooks/useMDXComponent.js
Module not found: Can't resolve 'react/jsx-runtime.js' in '/home/wiktor/Development-Linux/workspacemodels/node_modules/next-contentlayer/dist/hooks'

Import trace for requested module:
./node_modules/next-contentlayer/dist/hooks/index.js
./pages/about.jsx

./node_modules/next-contentlayer/dist/hooks/useMDXComponent.js
Module not found: Can't resolve 'react-dom' in '/home/wiktor/Development-Linux/workspacemodels/node_modules/next-contentlayer/dist/hooks'

Import trace for requested module:
./node_modules/next-contentlayer/dist/hooks/index.js
./pages/about.jsx

./node_modules/next-themes/dist/index.modern.js
Module not found: Can't resolve 'react' in '/home/wiktor/Development-Linux/workspacemodels/node_modules/next-themes/dist'

Import trace for requested module:
./pages/_app.js

> Build failed because of webpack errors

Here are my packages:
    "@mdx-js/loader": "^2.0.0",
    "@next/mdx": "^12.1.0",
    "classnames": "^2.3.1",
    "contentlayer": "^0.1.2",
    "date-fns": "^2.28.0",
    "next": "12.1.0",
    "next-contentlayer": "^0.1.2",
    "next-themes": "^0.1.1",
    "react": "17.0.2",
    "react-dom": "17.0.2",
    "swr": "^1.2.2"

And here's a repo link.
Any help very much appreciated!

Hi @wiktor-jurek - thanks for opening this issue and providing a link to your repo. I've taken a brief look and it seems the problem is related to you patching the config.resolve.alias Webpack config - I assume for the sake of using Preact? However, I don't see any Preact deps installed.
Simply commenting out these lines in your next.config.js seemed to make the next build work for me.

In case you actually intend to use Preact, please refer to this issue https://github.com/contentlayerdev/contentlayer/issues/62.
Closing this issue for now but happy to re-open in case I misunderstood the situation.

Thanks so much. That's it. That's what I get for blindly coping .next configs... 🙄


fix: Can navigate back after data selection and browsing state is kept

Search, order and includeDrafts are passed from browsing page to dataset result and are passed into the back link : now the user goes back to the exact state she left.
fix #238

Good idea, it was an artifact of the past, I refactored to have only the router query as the source of truth. a8f3e90 (#243)


Bug: move some packages into peerDependencies

Lexical version: 0.3.5
Steps To Reproduce

Using yarn berry as package manager under PNP mode.

Link to code example:
Just run the demo of "Powerful Features" on official home page of lexical.dev.
https://codesandbox.io/s/serene-wiles-1w0s5h?file=/src/Editor.js

The current behavior

The expected behavior
yarn berry uses different virtual packages while trying to import from different packages, it makes the two HeadingNodes are not same.

Correct me if I'm wrong, but this seems like more of an issue with that particular CodeSandbox and running it with Yarn Berry, instead of NPM which we use at the moment. I'm unable to reproduce the error with our current Package Manager.


Voicing needs listeners to fire on disabled Nodes

From a voicing meeting today with @jessegreenberg and others, we  discussed the need for listeners to fire on disabled components. Like if you click on a disabled checkbox, it may voice "show force values, disabled." Currently inputEnabled (which enabled runs on) won't let this happen. It would be good to discuss this with @jonathanolson to see what could be done.  Do you have any ideas?

The down is just an example, the main point is that we have a case where we need some input listeners on the Node to fire when the Node is disabled. Even for disabled Nodes, we would like to add a highlight around the Node when a mouse is over it and to describe the Node with speech when it gets clicked.
The inputEnabled property doesn't impact picking, so we are are still detecting hits just preventing listeners from being fired. What if we had a separate list of input listeners that could still get called when inputEnabled is false?

What if we had a separate list of input listeners that could still get called when inputEnabled is false?

This is specifically something that was considered when @zepumph and I were discussing approaches. We considered a flag on listeners where they would still have events fired even when inputDisabled, AND the listeners then would need to be very aware of when they were called when inputDisabled.
That sounds like a reasonable approach to me, however it might be good to brainstorm and meet about this, since this effectively changes our general needs from what I was aware of.

We will need this for the "lock ratio" checkbox in Ratio and Proportion. Marking for meeting to initially discuss with @jessegreenberg. We can pull in @jonathanolson with what our findings are, and more brainstorming.

We also need it for the Range combo box in Ratio and Proportion.

@jessegreenberg, is this similar to the issue about voicing invisible Nodes? Should it be part of that design while we are working in scenery?

@terracoda, can you note here whether or not you feel like this is blocking the publication of Ratio and Proportion with voicing. If so, we can get some of @jonathanolson's time on the books (likely for early Q2) to get this supported.

@terracoda had a fun idea about this today. Let's refocus this issue into our actual case, which is that disabled nodes that support voicing would look like a mouse-only reading block (not available with keyboard), and it would say the name response and "unavailble" (mimicing aria-disabled in NVDA). Basically we could somehow wrap the node in a reading block, only when disabled though. We wouldn't want any content in the PDOM (text content or button).
I'll see how it could go!

@jessegreenberg and I discussed this a bit further today, and the conclusion we came to really was that this needs to be a scenery-level change. We have 3 paths forward:

The "real" solution. @jessegreenberg and I found that over in https://github.com/phetsims/scenery/issues/1116#issuecomment-790216912 we almost supported disabledInputOptIn on the inputListener interface, and doing that here would be the best path forward to letting these readingBlock listeners fire when clicking on a disabled Node. After a bit of poking, it seems like a pretty huge change:

We would need to change Input.ts to not just ignore Nodes after an inputDisabled: false
We would need to investigate why/how some PDOM events like focus/blur are listening to pickable still instead of inputEnabled, and try to align that, perhaps with this feature, or using the dispatchEvent.fireOnInputDisabled option.
We would need to really change Voicing/ReadingBlock/InteractiveHighlighting to support listening to inputEnabled, and doing the right thing on disabled Nodes. We most likely want this feature for all voicing Nodes, and perhaps for all InteractiveHighlighting ones, so that would require messing with the responsibilities of each of those traits.

We can workaround this with DisabledReadingBlockParent.ts (a name I just made up) that can surround the Nodes we need to immediately support this with. It may be worth some investigation.
We can continue to do nothing, and not support this.

Last Friday, @terracoda and I brainstormed what (2) may look like above, and we came to the conclusion that it would be more trouble than it was worth. The primary reason is because we would need to make sure to support the parent with keyboard interaction with voicing while also keeping the actual element focusable with keyboard (but disabled) for description.
From this, as well as further discussion during the quarterly team planning meeting today, we aren't going to devote time to this at least until Q3 2022. Makring on hold.