Add container.image.digest and make container.image.tag array

This PR adds container.image.digest field and make container.image.tag an array of strings.
This is to cover https://github.com/open-telemetry/semantic-conventions/issues/48.
Also related to https://github.com/open-telemetry/semantic-conventions/issues/72.
More analysis can be found at https://github.com/open-telemetry/semantic-conventions/issues/48#issuecomment-1614425878.
cc: @AlexanderWert @kaiyan-sheng @mlunadia

Hey @lmolkova thanks for the feedback here!
From my perspective it would give a better user experience to provide a more generic naming. In a use case where a collector ships data to a data-store when the end user will go to search for them it would be easier to have all the related container fields under the container.* namespace. I suspect that the average user would struggle to understand what oci.manifest.digest stands for even if they know what they are looking for.
In addition I don't see any tags specific attribute in the OCI spec: https://github.com/opencontainers/image-spec/blob/main/manifest.md. Do you see sth there that totally reflects the container.image.tag? If only .digest is reflected in OCI directly then that's an extra reason for not splitting the fields in container.* and oci.* at the same time.
To my mind we should be following what runtimes/orchestrators provide which at the same time follow the OCI under the hood. Following the OCI looks more like an implementation detail and should not be exposed to the end user to my mind since it slightly changes the scope of interest. You can also have a look at my comment at https://github.com/open-telemetry/semantic-conventions/issues/48#issuecomment-1614425878 where I analyse how the various runtimes/orchestrators define the specific fields. There I also mention that k8s is based on Container Runtime Interface (CRI) which follow the OCI spec but nowhere this detail is exposed to the end user.  Indeed the CRI reports as RepoDigests while in OCI it's just digest from what we can see.
Let me know what you think :).

@ChrsMark I agree on the tag part - there is no formal tag definition in the OCI.
I don't however agree that OCI manifest digest should be recorded with container.image.digest.
OCI manifest is wider and more general-purpose thing than container image digest.
It describes not just container images, but non-runnable artifacts, VM images, helm charts, or anything else.
As an owner of Azure Container Registry SDK, when I report client calls, I would not even know what users are packaging, how they intend to use it. Distribution platforms such as image/artifact registries don't know either - they just implement OCI/docker v2 APIs.
container.image.digest focuses on the containers and the execution side only.  Introducing it would cause the following problems:

container/artifact registries and their client libraries will have to use container.image attributes for non-container things (and not images).
OR we'll have to introduce vm.image, helm.chart, oci.manifest namespaces, etc

OCI image manifest is a standard thing defined by the spec, while container image is a vague thing. k8s docs refer to the OCI spec.
TL;DR: OCI manifest digest covers a wider set of use-cases, allows to have consistent attribute in client libraries, artifact registries, and container environments. It provides common unambiguous terminology.
The only downside, that there is a tiny learning curve to discover OCI.  For my point of view, benefits of oci.manifest.digest  outweigh it.

Thanks for the detailed explanation @lmolkova !
To your use-case I see the point, however I would avoid using a generic naming like oci.image.digest to report a Container's Image Digest specifically. Based on your example the oci.image.digest might be populated for other entities not just containers so I think it would better to use a specific field for the container entity. I would prefer using container.image.digest and then registry.image.digest to refer to Registry Images in general. I would see value in a query like container.image.digest==registry.image.digest since the second field creates a super set that includes the first one.
Also, I made some extra research to try and get things together and I think that for the container.image.digest we need to clarify some things along with the container.image.id. Let me try to summarize my findings:
So the image ID is the equivalent of the .config.digest from
https://github.com/opencontainers/image-spec/blob/main/manifest.md
We already depict this at container.image.id since https://github.com/open-telemetry/semantic-conventions/pull/39/files#r1202644093.
The current PR intends to add the Digest information that is capable to be used for downloading an image.
In Docker and the CRI it is called RepoDigest and it is not part of the Image Manifest at https://github.com/opencontainers/image-spec/blob/main/manifest.md.
Indeed it seems to be a different field which is part of the Manifest List or the "fat manifest" as it is called (instead of the Image Manifest). But still the ID and the Digest are depicted in different parts of the spec.
Example:
➜  ~ docker pull prom/prometheus:v2.16.0@sha256:efd99a6be65885c07c559679a0df4ec709604bcdd8cd83f0d00a1a683b28fb6a
docker.io/prom/prometheus@sha256:efd99a6be65885c07c559679a0df4ec709604bcdd8cd83f0d00a1a683b28fb6a: Pulling from prom/prometheus
Digest: sha256:efd99a6be65885c07c559679a0df4ec709604bcdd8cd83f0d00a1a683b28fb6a
Status: Image is up to date for prom/prometheus@sha256:efd99a6be65885c07c559679a0df4ec709604bcdd8cd83f0d00a1a683b28fb6a
docker.io/prom/prometheus:v2.16.0@sha256:efd99a6be65885c07c559679a0df4ec709604bcdd8cd83f0d00a1a683b28fb6a
➜  ~ docker manifest inspect --verbose prom/prometheus:v2.16.0 | jq '.[0].Descriptor'
{
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "digest": "sha256:efd99a6be65885c07c559679a0df4ec709604bcdd8cd83f0d00a1a683b28fb6a",
  "size": 2824,
  "platform": {
    "architecture": "amd64",
    "os": "linux"
  }
}

➜  ~ docker inspect prom/prometheus:v2.16.0 --format 'Id: {{.Id}}
Repo Digest: {{index .RepoDigests }}'
Id: sha256:e935122ab143a64d92ed1fbb27d030cf6e2f0258207be1baf1b509c466aeeb42
Repo Digests: [prom/prometheus@sha256:efd99a6be65885c07c559679a0df4ec709604bcdd8cd83f0d00a1a683b28fb6a prom/prometheus@sha256:e4ca62c0d62f3e886e684806dfe9d4e0cda60d54986898173c1083856cfda0f4]

➜  ~ docker manifest inspect --verbose prom/prometheus:v2.16.0 | jq '.[0].SchemaV2Manifest.config'
{
  "mediaType": "application/vnd.docker.container.image.v1+json",
  "size": 6669,
  "digest": "sha256:e935122ab143a64d92ed1fbb27d030cf6e2f0258207be1baf1b509c466aeeb42"
}

So to summarize it, we already have the container.image.id which is the equivalent of Image ID in Docker, Kubernetes and CRI as it is also mentioned at https://github.com/open-telemetry/semantic-conventions/blob/main/model/resource/container.yaml#L35-L47. In terms of OCI it is the digest of the .config section of the manifest.
So here we talk about adding the Repo Digest identifier. Thus, if my claims above are correct we already skipped using the oci. specific namespace and to my mind it would be better if we continue doing this. It's easier to just use a mapping more friendly to the Container runtimes' (like ImageID, RepoDigest) users rather than going deep into the OCI spec which took me significant time to analyse and correlate with the fields provided by the runtimes. Plus the fact that the intend to use oci would extend the scope.
I would propose sth like container.image.digest and registry.image.digest (for registry use cases) as I already mentioned.
@lmolkova out of curiosity which part of the OCI spec would you be willing to use in your use-case specifically? The one that is depicted as Repo Digest and is used for downloading an image or the Image ID one?
Also I think having input from more people here to get more opinions would help a lot :).

To your use-case I see the point, however I would avoid using a generic naming like oci.image.digest to report a Container's Image Digest specifically. Based on your example the oci.image.digest might be populated for other entities not just containers so I think it would better to use a specific field for the container entity.

@ChrsMark , this is where we disagree. Attribute-based correlation is one of the features OTel provides. If we give the same thing multiple names we would not be able to correlate using attributes.
E.g. when building an artifact, I want to:

record data how the artifact is obtained regardless of container environment
record how it's pushed to the registry on the client side
record how it's pushed on the registry side
record how it's pulled on both sides
record how it's used (e.g. as a resource attribute on an application)

Assuming a bright future where I can get access to all this telemetry, using oci.manifest.digest  I can find everything that has ever happened to this specific artifact digest.
Querying gets more complicated with container.image.digest, and registry.artifact.digest, now you need to know they are both available and are the same thing.
Once you add config digest and layers digest into the picture, the bigger the need for unambiguous and externally defined image id becomes (which is defined in OCI and manifest digest).
I'm still struggling to understand the problem with oci.manifest.digest except a small learning curve. Can you explain if there are other reasons to make attribute-based correlation more complicated and use ambiguous and vague definitions while we have a spec that defines id?

I'm still struggling to understand the problem with oci.manifest.digest except a small learning curve. Can you explain if there are other reasons to make attribute-based correlation more complicated and use ambiguous and vague definitions while we have a spec that defines id?

Thank's @lmolkova! Let me try to collect my concerns bellow:

The learning curve is not quite small, see my explanation at https://github.com/open-telemetry/semantic-conventions/pull/159#issuecomment-1641674353 of how the OCI information is depicted in runtimes. Also there was a struggle to clearly define those at https://github.com/open-telemetry/semantic-conventions/pull/39/files#r1202644093. And still the schema needs to be validated again, see point 4 bellow. All these make me feel that the learning curve would not be negligible.
Coming from an Infrastructure Observability background I would assume that developers crafting Observability tools/collectors would find it more straight forward to use a terminology more relative to container runtimes (ImageDigest, ImageID). Otherwise they also need to go through the learning curve.
If we follow the oci.image.digest we might hit an issue with this field containing a super sets of objects/entities: A container image is an OCI image but an OCI image is not necessarily a container image. So if I plot in a pie chart the top 10 OCI images this might be misleading if I don't know that this dataset does not only contain container image information. This is an example illustrating the points 1 and 2. It's a clear issue of being generic VS being specific.
Last but not least, what is the plan about the container.image.id field that we have already introduced with https://github.com/open-telemetry/semantic-conventions/pull/39/files#r1202644093? As I explained at https://github.com/open-telemetry/semantic-conventions/pull/159#issuecomment-1641674353 we would need to validate it again since this is also part of the OCI spec. @joaopgrassi @marcsanmi what are your thoughts on this?

Still open question:
A) which part of the OCI spec would you be willing to use in your use-case specifically? The one that is depicted as Repo Digest and is used for downloading an image or the Image ID one?
I'm not totally against the oci.* proposal but I would like to address the above first.

@ChrsMark
Thank you for the update!

I'm not sure I understand the concern, please take a look at p4 below
My take (coming from an observability perspective) is that using standard, unambiguous terminology and being able to correlate telemetry using attributes is quite important.
It probably means that the pie chart needs to take other information into account - such as manifest media type. BTW, for the registry scenario, it does not matter what type manifest has since the registry pulls and pushes arbitrary data and does not care.
The  container.image.id, as we discussed before is a container-runtime-specific thing that represents the image id - in case of docker it's (as you figured out, the config digest), in the case of k8s it's something else. The same image has different ids on docker and k8s.
If there are other environments container.image.id would be different in each of them,  oci.manifest.digest would be a cross-environment, unified digest, common across different environments.
I.e. in your Prometheus example above,

container.image.id=sha256:e935122ab143a64d92ed1fbb27d030cf6e2f0258207be1baf1b509c466aeeb42
oci.manifest.digest=sha256:efd99a6be65885c07c559679a0df4ec709604bcdd8cd83f0d00a1a683b28fb6a.
I don't see how calling it container.image.digest would be any easier to understand or use (on the contrary, we'll have yet another attribute to correlate it to registry telemetry). Plus if I google for container.image.digest I will get nothing definitive, oci.manifest.digest will bring me right to the spec.

A) which part of the OCI spec would you be willing to use in your use-case specifically? The one that is depicted as Repo Digest and is used for downloading an image or the Image ID one?

If I defined oci namespace for my SDK, I'd start with:

oci.manifest.digest
oci.manifest.media_type
oci.schema_version  (2 by default)

thinking more about it, I'd be useful for me to record config and layers digests on telemetry, so I'd also consider

oci.blob.digest (blob is coming from docker v2 API, but )
oci.blob.media_type - this can be used to distinguish configs from layers
oci.schema_version

Since SDK (or registry) has no knowledge of container environment the image/artifact will be used in (or if  it will be used in the container environment at all), it would not know anything about containers or their ids.

Hey @lmolkova and thank you for the feedback! I see how container.image.id is different and now things are clear to me.
I think we can decide on using the oci.manifest.digest, I will update this PR accordingly :).

Thanks for reviewing this folks! In adacd83 , I changed the fields to plural form since I think it's more accurate based on https://opentelemetry.io/docs/specs/otel/common/attribute-naming/#name-pluralization-guidelines. Runtimes also report those in plural form.
I also tuned the descriptions accordingly.
@lmolkova let me know what you think about the oci.manifest.digests description specifically.

Sorry @ChrsMark for not responding earlier. I read all the conversation above and I'm glad we reached an agreement. I agree with the points @lmolkova brought up, I think adopting OCI is more "future proof" and gives us the power of correlating things. I left a review with a few points.

@open-telemetry/specs-semconv-maintainers I think this can be merged?

@joaopgrassi Chris is out this week. I just resolved the conflict in the Changelog file.

@lmolkova as I explained at https://github.com/open-telemetry/semantic-conventions/pull/159#discussion_r1283357024, container runtimes like Docker and CRI can provide multiple Repo Digests for a container image, see for example https://docs.docker.com/engine/api/v1.43/#tag/Image/operation/ImageInspect.
Based on this I think we should give the option to populate this field with multiple values as well. Otherwise if we make it singular/string, it will not be able to cover many important use cases. If this does not make sense from "oci" perspective then maybe we should reconsider the decision of using the "oci" generic namespace and maybe think of introducing 2 different fields oci.manifest.digest and container.image.digests(maybe that's more accurate based on this last observation/evaluation?).
Both options would be ok from my side, so let me know what you think @joaopgrassi @lmolkova :).

@lmolkova as I explained at #159 (comment), container runtimes like Docker and CRI can provide multiple Repo Digests for a container image, see for example https://docs.docker.com/engine/api/v1.43/#tag/Image/operation/ImageInspect.
Based on this I think we should give the option to populate this field with multiple values as well. Otherwise if we make it singular/string, it will not be able to cover many important use cases. If this does not make sense from "oci" perspective then maybe we should reconsider the decision of using the "oci" generic namespace and maybe think of introducing 2 different fields oci.manifest.digest and container.image.digests(maybe that's more accurate based on this last observation/evaluation?).
Both options would be ok from my side, so let me know what you think @joaopgrassi @lmolkova :).

The oci.manifest.digest does not capture the RepoDigests and there could be just one for the running container.
RepoDigest is docker-specific thing. E.g. k8s uses the repo-scoped-digest in the imageID (e.g.  mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9)
And the repo-digest (whatever they are) would be already populated in the container.image.id attribute.
If we need to capture docker-only RepoDigests, let's do it in the container.docker.image.digests or just docker.image.digests since other runtimes don't share the same property.

Hey @lmolkova , let me try to clarify what I have in mind :) :

The oci.manifest.digest does not capture the RepoDigests and there could be just one manifest for the container instance. The digest has specific format (hash_algo:hash), so it cannot contain the internal.registry.example.com:5000/example@ part.

From the docs' page example we are not going to add the internal.registry.example.com:5000/example@ part but the rest of it.
From a previously used example:
➜  ~ docker pull prom/prometheus:v2.16.0@sha256:efd99a6be65885c07c559679a0df4ec709604bcdd8cd83f0d00a1a683b28fb6a
docker.io/prom/prometheus@sha256:efd99a6be65885c07c559679a0df4ec709604bcdd8cd83f0d00a1a683b28fb6a: Pulling from prom/prometheus
Digest: sha256:efd99a6be65885c07c559679a0df4ec709604bcdd8cd83f0d00a1a683b28fb6a
Status: Image is up to date for prom/prometheus@sha256:efd99a6be65885c07c559679a0df4ec709604bcdd8cd83f0d00a1a683b28fb6a
docker.io/prom/prometheus:v2.16.0@sha256:efd99a6be65885c07c559679a0df4ec709604bcdd8cd83f0d00a1a683b28fb6a

➜  ~ docker inspect prom/prometheus:v2.16.0 --format 'Id: {{.Id}}
Repo Digest: {{index .RepoDigests }}'
Id: sha256:e935122ab143a64d92ed1fbb27d030cf6e2f0258207be1baf1b509c466aeeb42
Repo Digests: [prom/prometheus@sha256:efd99a6be65885c07c559679a0df4ec709604bcdd8cd83f0d00a1a683b28fb6a prom/prometheus@sha256:e4ca62c0d62f3e886e684806dfe9d4e0cda60d54986898173c1083856cfda0f4]

Both docker pull prom/prometheus@sha256:efd99a6be65885c07c559679a0df4ec709604bcdd8cd83f0d00a1a683b28fb6a and docker pull prom/prometheus@sha256:e4ca62c0d62f3e886e684806dfe9d4e0cda60d54986898173c1083856cfda0f4 will work, so those are valid digests.
So in the bare minimum example of docker developers that want to capture the digests would need to iterate over the RepoDigests field array and extract the digests' part.

RepoDigest is docker-specific thing. E.g. k8s uses the repo-scoped-digest as the imageID (e.g. mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9)
And the repo-digest (whatever it is) would be already populated in the container.image.id attribute.

For k8s, I don't think that is the case. Kubernetes Container objects will not hold that information anyways. The runtime, or better the CRI, will provide the RepoDigests.
For example if you exec/ssh inside a k8s node you can crictl to get the container images (for example with containerd as runtime).
An example:
➜  ~ crictl image -o json
 {
      "id": "sha256:e935122ab143a64d92ed1fbb27d030cf6e2f0258207be1baf1b509c466aeeb42",
      "repoTags": [
      ],
      "repoDigests": [
        "docker.io/prom/prometheus@sha256:efd99a6be65885c07c559679a0df4ec709604bcdd8cd83f0d00a1a683b28fb6a"
      ],
      "size": "54439175",
      "uid": null,
      "username": "nobody",
      "spec": null,
      "pinned": false
    },

From this it's clear to me that CRI provides that info in plural form (not just for docker but in general) and we should be aligned with that because that's what the APIs provide. See how the API's proto is defined: https://github.com/kubernetes/cri-api/blob/c75ef5b473bbe2d0a4fc92f82235efd665ea8e9f/pkg/apis/runtime/v1/api.proto#L1238.
In the above example the sha256:efd99a6be65885c07c559679a0df4ec709604bcdd8cd83f0d00a1a683b28fb6a digest can be used to pull the specific image from the registry, so it's an actual digest.
So again an easy way to extract the digests in k8s containers is to access the CRI API and use the repoDigests field array in order to extract the digests.

If we need to capture docker-only RepoDigests, let's do it in the container.docker.image.digests or just docker.image.digests since other runtimes don't share the same property.

Based on the above examples it should be container.image.digests since it's also aligned with the CRI not just docker.

I'm far from being an expert in this subject, but I read all again and I see points in both sides.
Reading again the OCI image spec it has a property called config which holds the descriptor. In the descriptor docs it's clear that it talks about a single "targeted content", and that has a single digest - the spec even says:

A Content Descriptor includes the type of the content, a content identifier (digest), and the byte-size of the raw content. Optionally, it includes the type of artifact it is describing.

It seems if we take OCI for it, makes sense to use what they also say there - a single digest. But with the cases that @ChrsMark brought, I'm not sure. Maybe we need to think about how instrumentations would populate this field, so we can have more info to base the decision here?
If we don't know and want to be flexible, maybe going with the array approach might be a good trade off? Not sure..

Thanks for the feedback @joaopgrassi!
I agree that based on OCI image spec the manifest's digest should be only one and in singular form.
My intention to make it plural/array was based on the the need to support the Runtimes' use cases (see https://github.com/open-telemetry/semantic-conventions/pull/159#issuecomment-1685786852). I see that it is a compromise and less accurate.

Maybe we need to think about how instrumentations would populate this field, so we can have more info to base the decision here?

To illustrate my point I'm mostly interested to cover use cases of infra/platform observability/security: As an analyst I want to  look for a specific digest if exists in the container images that are present in my cluster. This should be equivalent to fetching everything from the CRI API manually and analyzing manually the information.
Reflecting all the above conversations I have the feeling that in order to be accurate we need to be specific and avoid any generalization at this point.
From my point of view the most accurate is to introduce 2 different fields:

oci.manifest.digest to cover all the generic usecases as described at https://github.com/open-telemetry/semantic-conventions/pull/159#issuecomment-1651077018
container.image.digests to cover information retrieved from the CRI APIs as described at https://github.com/open-telemetry/semantic-conventions/pull/159#issuecomment-1685786852 to specifically reflect what the CRI API provides.

Let me know what you folks think.

Sounds reasonable to me, but we would have duplication of data right? As container.image.digests would be a superset of oci.manifest.digest. In cases where @lmolkova described where it deals with "non container" scenarios we would only populate oci.manifest.digest if I got it right?

I think it depends on the perspective/method we collect these data:

but we would have duplication of data right? As container.image.digests would be a superset of oci.manifest.digest.

In cases we only have access to the already stored container images of our observed system we can retrieve the information from the CRI's API and hence we would populate container.image.digests accordingly. We don't have a straight forward way to retrieve the single manifest's digest so far. In some cases we might be able to populate both fields but that's a fair compromise for now. Maybe in the long run the CRI definition will change (to singular) and we can deprecate the digests in order to only use the oci one.

In cases where @lmolkova described where it deals with "non container" scenarios we would only populate oci.manifest.digest

In cases we have direct access to know the specific manifest's digest we will be using the oci.manifest.digest (for example in download time/execution).

There is 1:1 relationship between container and an image. One container can run only one image.
There is also 1:1 relationship between an image and it's manifest.
So having multiple digests for one container does not make sense.
The smart image can be pushed to multiple repositories, but if it's the same image, it will have the same oci digest anywhere (as it's a sha256 of manifest). Check out answers on this thread: https://stackoverflow.com/questions/45533005/why-digests-are-different-depend-on-registry
If we need to support docker v1 or something else where the same image can have multiple manifest digests, let's create contaiee.docker.repp_digests or something similar.

All right, based on the above discussions I have changed the oci.manifest. field accordingly to singular and introduced a new one to reflect what CRI's api provides:

oci.manifest.digest
container.image.repo_digests to reflect what CRI provides (https://github.com/kubernetes/cri-api/blob/c75ef5b473bbe2d0a4fc92f82235efd665ea8e9f/pkg/apis/runtime/v1/api.proto#L1238) for infra observability and security use-cases.

I hope that covers all that we have discussed so far.

@lmolkova I think the requested changes are now covered :) . Are we good to go with this one?

@open-telemetry/specs-semconv-maintainers this one should be ready to for merge?


Option to not redownload existing files if there are no index files

It would be nice if the program did not redownload the already downloaded files but just skipped them in the case of absence index files for them. This would be actual if, for example, the index files were lost or deleted. I just have lots of downloaded files from blogs, that were downloaded year ago, I would like to update them to download new posts, but the index files have been deleted because I deleted blogs from the program due bug crash occurs after adding new blogs if existing blogs were so many in the program so now I have no index files for older blogs. And if I add all the old blogs to the program, then it will redownload all the files in the folders and this process will take a long time because I have over 500 thousands files. I know that you are currently working on program development or other things, so I do not expect that this option will appear in the program anytime soon or at all, but if there is an opportunity to introduce this option I will be happy because it will save a lot of time and internet traffic.

It's actually three lines of code to check in the directory instead I guess.
I've decided against in the beginning because for me at least on my crappy hard drives it takes some time to only open an explorer view in a folder with several ten thousand files. I can add that though as an option somehow ...

Okay, so is it possible to write this in the code and this option will appear in the program? If so, I will be very glad, for me it is really necessary. Thanks!

Its implemented. I'll probably upload a release quite soon, but it might have other problems I've not stepped over yet caused from my code refactoring.
On the other hand it should be more stable now, easier for other people to join and maybe someone has time a write some unit tests to that it can be automatically tested.

Should be working, but I can not guarantee for anything right now. I've shuffled around quite a bit of code.
It can not load the old binary databases, only the new text based ones. So you might have to run TumblThree v<IP_ADDRESS> or v<IP_ADDRESS> to convert the databases to text files and then run this one. If you used the recent version anyways everything is already set.
Uploading TumblThree-v<IP_ADDRESS>.zip…

Maybe I leave this open if there are any people how are willing to do the testing.
It's basically at the state of the latest commit johanneszab@ef4e270 with only the version number changed.
If you notice anything wrong, please write it here.

The version <IP_ADDRESS> program does not download images at all, after scanning the blog disappears from the queue and there is not one downloaded file in the folders.

Whoops, sorry. It's probably too late for me already to think about logic. This should be working now. I've tested it with files and a database.
TumblThree-v<IP_ADDRESS>.zip

Thank you! It seems that it works. But there is one thing: a year ago, when I downloaded blogs, there was no function to download files with specified size. And now I'm downloading only with the function of downloading specified size. In practice, this means that I have many files in the older blogs with digits in the filenames like "_400", "_250", "_540" and now the program creates new same files next to them with the new digits "_1280". This is a small inconvenience, could this be fixed?


Set no precontioner in GMRESSolver

Hi,  I would like to test GMRESSolver without calling any preconditioner.  Is this easy to do in the MFEM,
by comment out
solver.SetPreconditioner(*L2ElliptOptConPr);
Will this be an issue without calling this setpreconditioner  ?
Thanks.

@YHDAUT, you can omit setting a preconditioner with no issues.
Best,
Socratis

Thank you @psocratis ,  it works now..


Using headless service instead of endpoint in multicluster, single control-plane, multi-network configuration on EKS

(This is used to request new product features, please visit https://discuss.istio.io for questions on using Istio)
Describe the feature request
In a Shared control plane (multi-network) Istio installlation on EKS.
I would like to be able to reference a DNS name for my LOCAL_GW_ADDR instead of an IP address since ELB resolved IP addresses can vary
Describe alternatives you've considered

Reference one of the resolved IP addresses of my principal cluster's ELB hostname => it works but the IP address could change and break the setup

Using an NLB => IP addresses are fixed but the gRPC trafic between components of the secondary cluster and the pilot (as an example) is'nt working.

Affected product area (please put an X in all that apply)
[X] Configuration Infrastructure
[ ] Docs
[X] Installation
[X] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Additional context
I'm working on a multi-cluster, shared control-plane, multi-network Istio deployment on EKS
The only way I had it working at the moment is with the istio-ingressgateway in both cluster populating TCP ELB's
cat override.yaml
gateways:
istio-ingressgateway:
serviceAnnotations:
service.beta.kubernetes.io/aws-load-balancer-type: "elb"
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
Problem could come from the lack of support stated here:
https://istio.io/docs/setup/kubernetes/install/multicluster/shared-gateways/#setup-cluster-2
The command fails if the load balancer configuration doesn’t include an IP address. The implementation of DNS name support is pending.
Could this issue be resolved if, instead of populating endpoints:
---
# Source: istio/templates/endpoints.yaml

apiVersion: v1
kind: Endpoints
metadata:
  name: istio-pilot
  namespace: istio-system
subsets:
- addresses:
  - ip: <LOCAL_GW_ADDR>
  ports:
  - port: 15003
    name: http-old-discovery # mTLS or non-mTLS depending on auth setting
  - port: 15005
    name: https-discovery # always mTLS
  - port: 15007
    name: http-discovery # always plain-text
  - port: 15010
    name: grpc-xds # direct
  - port: 15011
    name: https-xds # mTLS or non-mTLS depending on auth setting
  - port: 8080
    name: http-legacy-discovery # direct
  - port: 15014
    name: http-monitoring
...
---
apiVersion: v1
kind: Endpoints
metadata:
  name: istio-policy
  namespace: istio-system
subsets:
- addresses:
  - ip: <LOCAL_GW_ADDR>
  ports:
  - name: grpc-mixer
    port: 9091
  - name: grpc-mixer-mtls
    port: 15004
  - name: http-monitoring
    port: 15014
...
---
apiVersion: v1
kind: Endpoints
metadata:
  name: istio-telemetry
  namespace: istio-system
subsets:
- addresses:
  - ip: <LOCAL_GW_ADDR>
  ports:
  - name: grpc-mixer
    port: 9091
  - name: grpc-mixer-mtls
    port: 15004
  - name: http-monitoring
    port: 15014
  - name: prometheus
    port: 42422
...

Where we are obliged to use IP addresses
We could use Headless Service where we reference the DNS name of the LoadBalancer exposing the istio-ingressgateway service from the primary cluster?
---
apiVersion: v1
kind: Service
metadata:
  name: istio-pilot
  namespace: istio-system
spec:
  type: ExternalName
  externalName: <ELB-DNS-NAME>
  clusterIP: None
  ports:
    - port: 15003
      name: http-old-discovery # mTLS or non-mTLS depending on auth setting
    - port: 15005
      name: https-discovery # always mTLS
    - port: 15007
      name: http-discovery # always plain-text
    - port: 15010
      name: grpc-xds # direct
    - port: 15011
      name: https-xds # mTLS or non-mTLS depending on auth setting
    - port: 8080
      name: http-legacy-discovery # direct
    - port: 15014
      name: http-monitoring
...
---
apiVersion: v1
kind: Service
metadata:
  name: istio-pilot
  namespace: istio-system
spec:
  type: ExternalName
  externalName: <ELB-DNS-NAME>
  clusterIP: None
  ports:
    - name: grpc-mixer
      port: 9091
    - name: grpc-mixer-mtls
      port: 15004
    - name: http-monitoring
      port: 15014
...
---
apiVersion: v1
kind: Service
metadata:
  name: istio-pilot
  namespace: istio-system
spec:
  type: ExternalName
  externalName: <ELB-DNS-NAME>
  clusterIP: None
  ports:
    - name: grpc-mixer
      port: 9091
    - name: grpc-mixer-mtls
      port: 15004
    - name: http-monitoring
      port: 15014
    - name: prometheus
      port: 42422
...

thanks in advance!
Niels

It looks to be working,
We have now 3 EKS clusters,
For the principal cluster, We followed the doc as is, using the DNS Hostname of the loadBalancer exposing the istio-ingressgateway for the "network1" address in the istio configMap
For the secondary clusters:
We changed the helm templates to:

endpoints: Not use them

services:
Use :
type: ExternalName
externalName: < DNS Hostname >

configMap:
Always use the {{ .Values.global.remotePilotAddress }}, {{ .Values.global.remotePolicyAddress }} and {{ .Values.global.remoteTelemetryAddress }}

you can find here a diff between the official configMap template and the one I used for the secondary clusters
And used the DNS Hostname of the loadBalancer exposing the istio-ingressgateway of the principal cluster as LOCAL_GW_ADDR
export LOCAL_GW_ADDR=< DNS Hostname >

So you then have used LOCAL_GW_ADDR in remotePilotAddress, remotePolicyAddress and remoteTelemetryAddress right?

@mazzy89 Yes, once I edited the helm templates I use the same command as stated in the documentation :)
helm template --name istio-remote --namespace=istio-system \
  --values install/kubernetes/helm/istio/values-istio-remote.yaml \
  --set global.mtls.enabled=true \
  --set gateways.enabled=true \
  --set security.selfSigned=false \
  --set global.controlPlaneSecurityEnabled=true \
  --set global.remoteUseDnsHostname=true \
  --set global.remotePilotAddress=${LOCAL_GW_ADDR} \
  --set global.remotePolicyAddress=${LOCAL_GW_ADDR} \
  --set global.remoteTelemetryAddress=${LOCAL_GW_ADDR} \
  --set gateways.istio-ingressgateway.env.ISTIO_META_NETWORK="network2" \
  --set global.network="network2" \
  -f override.yaml \
  install/kubernetes/helm/istio > istio-cluster-2.yaml

The override.yaml file being
gateways:
  istio-ingressgateway:
    serviceAnnotations:
      service.beta.kubernetes.io/aws-load-balancer-type: "elb"
      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"

/assign @Nielsoux

Which version of istio are you using @Nielsoux ? I am trying to setup pretty much the same thing (istio mesh with 2 AWS clusters, but using external load balancers), and the crush map templates do not correspond to the one you show in the diff file.

Hi,
I applied the modifications to the templates and the helm configuration and I was able to get my istio-ingressgateway up and running in my cluster2 (AWS). However, it seems like the clusters 1 and 2 are able to communicate only in one way (1->2), and the ingress-gateway of cluster2 does not actually forwards requests. Have you experienced something similar?
I am unsure if it is an issue with the AWS load balancer or with the istio configuration.
Thank you,
Camille

Hello @camille-rodriguez ,
To be honest, we dropped this solution and went to a kind of distributed control-plane mode,
Why:
1/ Cross-cluster resolution is not supported by default in Kubernetes,
So we had to create the ServiceEntry CRD in the clusters where we did'nt deploy the istio control-plane so that we could create them, and following this doc, stub the .global domain.
2/ Pilot is not watching those Istio's CRD in the secondary Istio cluster so we were not able to have a working RBAC configuration
for example:
The creation of a ServiceRole + ServiceRoleBinding in the secondary cluster was'nt seen by the control-plane, even if the multi-cluster ServiceAccount from the secondary cluster would have the rights to watch them
But, to reply to your question, did you use the "override.yaml" that I mentioned when you installed Istio?

Hi @Nielsoux,
Thank you for the additional information! I can see that I am still going to face issues with this type of deployment even if I get past the issue I currently have. I haven't tried yet a deployment of replicated control planes, but that was in my to-do list if having a common control plane is not working. Did you follow the official doc for it and did you face issues regarding AWS load balancers hostname resolution in that case as well?
Regarding the setup, I did omit the override file because I wanted to use an external load balancer instead of internal. I will try again with this configuration and see how it goes.
Thank you!

@camille-rodriguez,
Yes we followed the doc you mentioned and therefore had no more issue with the name resolution since every components are referring to the pilot from their own clusters and can then use the endpoints :)
Regarding the load balancer:
In the override.yaml,
You can just remove the following annotation to use an external LB as this is the defat behaviour to have a puic-facing LB populated
service.beta.kubernetes.io/aws-load-balancer-internal: "false"
https://docs.aws.amazon.com/eks/latest/userguide/load-balancing.html

Thanks for bringing up this feature request!
It would be great to have "Shared control plane (multi-network)" available on EKS
It's been a while since 2019, wondering whether Istio road-map has the plan of supporting "Shared control plane (multi-network) + EKS" or not? Thanks!
p.s.
Services “istio-pilot” not found, when trying “Shared control plane (multi-network)” on EKS 1.14
https://stackoverflow.com/questions/60673173/services-istio-pilot-not-found-when-trying-shared-control-plane-multi-netwo

We are facing the same issue. We want to have a shared controlplane and form a mesh with 3 other EKS clusters on the same network but it's not properly working out 😕


Python modules structure

Implementation of every Python module following this structure:
Import of libraries
Initialization

def publish(client):
      """Do stuff and communicate some data to the given Paho client"""

Doing this way, the modules can be imported at the main program and scheduled accordingly to the desired planning. Every module implements the same function publish(client) which receives a properly configured Paho client.

Find a dummy example at https://github.com/ESAI-CEU-UCH/raspi-monitoring-system/blob/master/raspi_mon_sys/ExampleMonitor.py


Change a little the API to retrive the error.

Attempt to fix #18

Coverage increased (+2.2%) to 84.0% when pulling 4f9233ec45d5c8c5cc69eb75a9c0ab52716825f9 on yageek:hotfix/download_error into 3de339a55934c49fa256181e63b9ecf0234c15b7 on auth0-community:master.


Prepare for 4.0.0~pre1

🎈 Release
Preparation for 4.0.0 release.
No previous release!
Checklist

[x] Asked team if this is a good time for a release
[x] There are no changes to be ported from the previous major version
[x] No PRs targeted at this major version are close to getting in
[x] Updated changelog
[x] Updated migration guide (as needed)

Note to maintainers: Remember to use Squash-Merge

Focal failure is a flaky performance test.  Probably unrelated to the changelog :)


[Feature, Request], Table, filter, future

Existing Component
是
Component Name
table
Description
now if the table has many filter must be reset for each， future can I reset them once

We have no plan to support this.


Reverse port forwarding feature

Hi there, you mentioned there is no reverse port forwarding because it is equivalent to spawning another server on the client side
I have a client behind the router that connects to the remote server, how can I expose a local SSH port on a remote server?
server does not have a way to connect to the client so how would launching another server on the client side would help this?
Thanks

You are absolutely right. Now that you think about it, the "equivalence" part isn't true and reverse port forwarding is really required to expose a local port behind a NAT.
Honestly, I was just lazy to think of a way to fit this feature in the current design. I'll keep that in mind and maybe implement it one day. In the meantime I'll update README and if you or anyone else have good solutions, feel free to put it here.


can you use a third party app registration to access api.powerbi.com on behalf of another tenant?

We have an app registration that can access management.azure.com APIs on behalf of our clients (other tenants). But it doesn't work with api.powerbi.com... I followed this tutorial,
https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-api-authentication#check-if-user-can-assign-access-to-subscription
How can I reproduce the "getAppOnlyToken" that would allow me to access our client's api.powerbi.com  apis? Through azure management I can create, update, delete, list, suspend, resume Capacities. But I can't assign them to a workgroup. Why did you split those functionalities into two seperate apis?
Thanks

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0fdac5f7-cd43-34c5-5905-dceeb9813952
Version Independent ID: 6fd53f84-dbb2-2a9c-2745-ea1999779113
Content: Embedded analytics to embed Power BI content in your application for your customers - Power BI
Content Source: powerbi-docs/developer/embed-sample-for-customers.md
Service: powerbi
Sub-service: powerbi-developer
GitHub Login: @rkarlin
Microsoft Alias: rkarlin

Thanks for your comment, @mlegris . This issue has been assigned to the doc author for review.

@KesemSharabi

Thanks for your comment, @mlegris.
This looks like a support request or product question rather than documentation feedback.
You might find a solution in the Power BI Community. If not, try our support page.


text alignment

Tried using .textalignment but didn't work and I couldn't see any attributes with NSTextContainer that dealt with text alignment.
Is this possible?

Looks like a bug, good catch! In the mean time, you can create an NSMutableParagraphStyle and set the alignment property, and then set that object in the attributes dictionary of your NSAttributedString using the NSParagraphStyleAttributeName key.

@indragiek Cool, thanks for the suggestion!

/code var txtstyle = NSMutableParagraphStyle()
txtstyle.alignment = NSTextAlignment.Center
    let RTFPath = NSBundle.mainBundle().pathForResource("text", ofType: "rtf")!
    let RTFData = NSData(contentsOfFile: RTFPath)!
    let options = [NSDocumentTypeDocumentAttribute: NSRTFTextDocumentType, NSParagraphStyleAttributeName : txtstyle]

    policyLabel.numberOfLines = 0
    policyLabel.attributedText = NSAttributedString(data: RTFData, options: options, documentAttributes: nil, error: nil)
    policyLabel.delegate = self

Just tried this and it didn't work

'''
var txtstyle = NSMutableParagraphStyle()
txtstyle.alignment = NSTextAlignment.Center
    let RTFPath = NSBundle.mainBundle().pathForResource("text", ofType: "rtf")!
    let RTFData = NSData(contentsOfFile: RTFPath)!
    let options = [NSDocumentTypeDocumentAttribute: NSRTFTextDocumentType, NSParagraphStyleAttributeName : txtstyle]

    policyLabel.numberOfLines = 0
    policyLabel.attributedText = NSAttributedString(data: RTFData, options: options, documentAttributes: nil, error: nil)
    policyLabel.delegate = self

'''
Just tried this and it didn't work!

 var txtstyle = NSMutableParagraphStyle()
        txtstyle.alignment = NSTextAlignment.Center

        let RTFPath = NSBundle.mainBundle().pathForResource("text", ofType: "rtf")!
        let RTFData = NSData(contentsOfFile: RTFPath)!
        let options = [NSDocumentTypeDocumentAttribute: NSRTFTextDocumentType, NSParagraphStyleAttributeName : txtstyle]

        policyLabel.numberOfLines = 0
        policyLabel.attributedText = NSAttributedString(data: RTFData, options: options, documentAttributes: nil, error: nil)
        policyLabel.delegate = self

Just tried this and it didn't work!

NSParagraphStyleAttributeName is a key that can't be passed in the options parameter as you have there. In this case, you will need to create an NSMutableAttributedString and apply the attribute like this:
let attrString = NSMutableAttributedString(data: RTFData, options: options, documentAttributes: nil, error: nil)
attrString.addAttribute(NSParagraphStyleAttributeName, value: style, range: NSMakeRange(0, attrString.length))
policyLabel.attributedText = attrString;

@indragiek Thanks! Got it to work

This ticket shouldn't be closed. Is still an issue :)

@raulriera Good point!


Server Crashes with TPM

Tmux 3.3, top of tree tpm.
When mixed with TPM it crashes the tmux server. Tmux installed via homebrew on osx. Crash deletes the plugins folder.

Note crash does not happen with a vanilla config. It only happens with oh my tmux

Looks like it might be an upstream issue. https://github.com/tmux/tmux/issues/3203#issuecomment-1146777548

Hello @cujomalainey 👋
I cannot reproduce the issue with tmux 3.3 on macOS.
Which plugins did you enable?

confirmed its an upstream issue (https://github.com/tmux/tmux/issues/3206), thanks for the prompt response


Transposition of data attributes to support target analytics in web components

The problem
In order for Adobe Target Experiments to transmit information to Google Analytics, a series of html attributes have been created for the DBDM analytics code to identify elements modified by Adobe Target.
Example:
data-experimentid="12345"
data-dynamic-content="true"
data-driverintent="interactive"
data-driverformat="CF603"
data-driversource="target"
data-driverlocation="dds-leadspace__cta-0"
data-drivertype="TY702"
The solution
When placing these attributes on elements with href, need for these attributes to be transposed to the create html elements with the anchor
<dds-some-item href="https://www.google.com"
data-experimentid="12345"
data-dynamic-content="true"
data-driverintent="interactive"
data-driverformat="CF603"
data-driversource="target"
data-driverlocation="dds-leadspace__cta-0"
data-drivertype="TY702"

Should be transposed to:
<a href="https://www.google.com"
data-experimentid="12345"
data-dynamic-content="true"
data-driverintent="interactive"
data-driverformat="CF603"
data-driversource="target"
data-driverlocation="dds-leadspace__cta-0"
data-drivertype="TY702"

Additional Information

DDS will implement as a generic solution that will transpose all data attributes except autoid.
Tracking the data attributes is difficult for this team without this feature in place.

@leetsao Recommend attending the Engineering Office Hours.  The meeting takes place every Wednesday morning at 9 ET.  Here is a link to the meeting information:  https://ibm.ent.box.com/notes/799374147228

This was previously discussed in an office hours and I was asked to create this feature request.
I have time on the next office hours and we can continue to discuss there.

Lee TsaoSenior Solutions Architect, IBM IXe: @.:<PHONE_NUMBER>

----- Original message -----From: "RobertaJHahn" @.>To: "carbon-design-system/carbon-for-ibm-dotcom" @.>Cc: "Lee Tsao" @.>, "Mention" @.***>Subject: [EXTERNAL] Re: [carbon-design-system/carbon-for-ibm-dotcom] Transposition of data attributes to support target analytics in web components (#6813)Date: Tue, Aug 10, 2021 9:41 AM      @leetsao Recommend attending the Engineering Office Hours. The meeting takes place every Wednesday morning at 9 ET. Here is a link to the meeting information: https://ibm.ent.box.com/notes/799374147228 — You are receiving this because    ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization. ZjQcmQRYFpfptBannerEnd

@leetsao Recommend attending the Engineering Office Hours. The meeting takes place every Wednesday morning at 9 ET. Here is a link to the meeting information: https://ibm.ent.box.com/notes/799374147228
—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.

QA URL that is modified by Target: https://www.ibm.com/topics/edi-electronic-data-interchange?at_preview_token=lIs_-apTKomYU1rYN9Q8BQ&at_preview_index=1_1&at_preview_listed_activities_only=true

Will need to transpose for any anchors and cards. Example above would need to include this for dds-link-list-item-card, for example.


Use isarray in order to support IE8. Fixes #2

thanks! I'll get this merged in and publish a new version sometime today

@jonschlinkert ping?

sorry for the delay! thanks for pinging me and for the pr!


[Feature Request] [LevelUp] Prestige XP conversion

Right now, if you prestige while being over the allowed prestige level. It will set you to prestige 1 level 1 zero experience. This feature request suggests that if you prestige while being over the minimum required level, all the extra experience past X level will be put towards your post-prestige level. This could be a toggle as well to preserve behavior others might prefer in this scenario.
Example: Level 40 is prestige requirement while member is level 43. All experience past level 40 would mean the user would most likely be Prestige 1 Level 12 after prestiging.

Thats a good idea, i'll implement it in the next patch

This feature has been implemented


Add ability to suppress multiple Slack alerts for same failing tests

Is your feature request related to a problem? Please describe.
Currently, Elementary Slack alerting is great but getting notified every 30 minutes (how frequently we run dbt)  about the same test failing is clogging our alerting channel resulting in people muting it, defeating the purpose of alerting to test failures.
Describe the solution you'd like
We would like to be able to set a parameter to allow us to receive all alerts or only the first failure alert depending on the importance of the model.
Describe alternatives you've considered
Without this feature available right now, we are looking at a custom Slack notification based on querying the Elementary schema to check to see if a test failure already exists for that unique test. But this feature seems like it would be useful for everyone and Elementary already has a great Slack alerting setup.
Additional context
Add any other context or screenshots about the feature request here.
Would you be willing to contribute this feature?
Yes.

Linking a different problem that could relate in terms of the solution to this issue.

we often have transient DB errors like connection dropped or something so we have a secondary run to catch these before sending alerts for failures only if they fail a second time using something like: dbt build --select result:error+ result:fail+ --defer --state ./target - In the case that a model has failed once, the post hook runs to catch the alert and then the model passes on the 2nd go - Elementary currently sends a failure alert for the first failure (or 2 alerts if it fails both times) - how would you recommend getting around this? I was thinking of maybe disabling the auto upload of alerts for the 1st run (though a var to the command) cos in theory everything that fails is run again anyway and should be caught the 2nd time.

It seems like both could be solved by allowing users to specify an optional where to the alerts views to filter the alerts.

Released in version 0.6.5


Make import all work with eslint

At Brigade we have recently switched from jshint/jsxhint to eslint, and it would be nice if this plugin worked with the eslint tool in the same way that it works with jshint/jsxhint.

As a matter of fact, you can use eslint right now. If you add this configuration, it should work:
    "jshint_cmd": "eslint --stdin --format compact"

The name of that configuration option isn't great. Perhaps we should make a change to "find_undefined_variables_cmd"? And then make it compatible with anything that outputs a standard format (the "compact" format of eslint coincidentally (?) matches the output from jsx?hint. But there are other formatting options as well ("checkstyle" seems to be one of them).

(The json formatting of the configuration is a commit in the making)

Neat!

Here's a commit to switch to eslint in the brigade repo: https://github.com/brigade/brigade/commit/d82748b026e43b38fadb74060736f5a4658cf66a
Unfortunately I have some issues pushing to our internal code review tool, so it'll live in that branch for a while.


Add release-please

Converting to draft until we figure out how to hook up to cargo release

Closing in favour of #244


Make notify return a promise derived from onProgress return values/throws.

See https://github.com/promises-aplus/progress-spec/issues/3#issuecomment-11609297. This would replace our current strategy of crashing if onProgress throws.
I believe it would be something along the lines of Q.all(progressHandlers.map(Q.fcall)).

I'm trying to abort the hashing of a number of files and I'm using Q's promises to handle the async part.
So since this is never going to work in 1.0, I guess its not possible to stop the execution of a promise from the outside or is it?

@tomdaub cancellation is an unrelated concern to progress/notify. Q promises have multiple consumer semantics, so cancellation is tricky. I am contemplating building a library with a "task" primitive that is similar to a promise but with explicit fork() and cancel(), and @domenic is actively working out whether promises can become cancelable promises with reference counting on observers.
See: https://github.com/kriskowal/gtor


copy over unreferenced contrib to new contrib repo

When this goes in, it should be the opposite of kubernetes/kubernetes#12693

LGTM


Add segment-anything-fast perf/acc benchmarks to torchao

This PR adds in segment-anything-fast evaluation to torchao, and also adds benchmarks for int8 quantization + 2:4 sparsity.
With this we can run combined perf/accuracy benchmarks for segment-anything. This should give us a starting point for the relative perf vs relative acc graph for PTC.

Model Type
Technique
img/s
memory (MiB)
mIoU
relative speedup
relative accuracy

ViT-h
baseline (bfloat16, max-autotune)
22.75
15172
0.5811

int8 dynamic quant (attn + mlp)
24.91
15154
0.5822
1.09x
100.19%

2:4 sparsity (mlp only)
24.81
15632
0.5672
1.10x
97.61%

2:4 sparsity (attn + mlp)
24.30
13429
0.5306
1.07x
91.31%

int8 dynamic quant (attn)int8 dynamic quant + 2:4 sparsity (mlp lin1)2:4 sparsity (mlp lin2)
26.46
14865
0.5668
1.16x
97.54%

This just copies over the evaluation scripts. Eventually I think we should move over the modeling code too, but plan to do that in a subsequent PR.

@pytorchbot merge

@pytorchbot merge

@huydhn @clee2000 another example of mergebot not merging stuff


Missing some comments

use ieee.std_logic_1164.all;

entity test_entity_name is
generic (
    a : integer;
    b : std_logic := '1';
    c, d : std_logic_vector(1 downto 0)
  );
port(
  --! over comment 0
  x: in std_logic;
  y: out std_logic; --! inline comment 0
  z: inout std_logic;
  --! over comment 1
  m: in std_logic_vector(31 downto 0); --! inline comment
  b: in std_logic_vector(31 downto 0) := "0010";
  --! over comment 2
  c,d : in std_logic := '1' --! inline comment 2
);
end test_entity_name;

architecture e_arch of test_entity_name is
  --! test signal
  signal a : integer;
  signal b,c : std_logic_vector(1 downto 0);

  constant d : integer;
  constant e,f : integer := 0;

  function counter(minutes : integer := 0; seconds : integer := 0)
    return integer is variable total_seconds : integer;
  begin
  end function;

  type state_0 is (INIT, END);

  begin

--! test process
label_0: process begin
end process;

label_1: process (clk0, reset) begin
end process;

process begin
end process;
  uut_0 : half_adder port map (a => a, b => b);
  half_adder port map (a => a, b => b);
end e_arch;```

The comment "--! test signal" is missing in the syntax tree

I just tested this and I don't see the issue.  Attached is the output of tree-sitter parse.  Note the comment on line 22 (0 based indexing for line numbers).  If I add (comment) @comment to the highlights, it shows up highlighted as a comment in tree-sitter highlight.
syntax-tree.txt

You are right.


Can we simplify this part of the grammar?

Apparently, this specific section of the grammar is difficult to parse in zed's CI. See this issue.
I'm wondering if it could be simplified somehow?

closed by #85


add landing message #1

This is an example comment. "April is not convinced about the  code."

This is an example comment. April is not convinced about the code.


Square on large complex(64) inputs return incorrect results

Description
As reported in MPMath vs Jax square, CPU and MPMath vs JAX square, CUDA, there exists regions in complex plane where jax.numpy.square returns incorrect results. For example:
>>> with jax.default_device(jax.devices('cpu')[0]):
...   jnp.square(jnp.array(4.468666049354532e+32-5.8683295066770075e+26j, dtype=jnp.complex64))
...
Array(-inf-infj, dtype=complex64)
>>> with jax.default_device(jax.devices('cuda')[0]):
...   jnp.square(jnp.array(4.468666049354532e+32-5.8683295066770075e+26j, dtype=jnp.complex64))
...
Array(nan-infj, dtype=complex64)

The expected result is inf - infj as suggested by:
>>> (4.468666049354532e+32-5.8683295066770075e+26j)**2
(1.9968976260619403e+65-5.244720966582595e+59j)

or
>>> with jax.default_device(jax.devices('cpu')[0]):
...   jnp.square(jnp.array(4.893543125372159e+30-4.893543125372159e+30j, dtype=jnp.complex64))
...
Array(-inf-infj, dtype=complex64)

where the correct result is -infj.
Other samples problematic to jax square include:
                                                      x ->      jax x**2   mpmath x**2
        (4.468666049354532e+32-5.8683295066770075e+26j) ->   (-inf-infj)   (inf-infj)
        (4.468666049354532e+32+5.8683295066770075e+26j) ->   (-inf+infj)   (inf+infj)
       (-4.468666049354532e+32+5.8683295066770075e+26j) ->   (-inf-infj)   (inf-infj)
                           (inf-5.358817286521974e+28j) ->    (nan-infj)   (inf-infj)
                           (inf+4.893543125372159e+30j) ->    (nan+infj)   (inf+infj)
                          (-inf-5.358817286521974e+28j) ->    (nan+infj)   (inf+infj)
                          (-inf+4.893543125372159e+30j) ->    (nan-infj)   (inf-infj)
         (4.893543125372159e+30-4.893543125372159e+30j) ->   (nan-infj)         -infj
         (4.893543125372159e+30+4.893543125372159e+30j) ->   (nan+infj)          infj
        (-4.893543125372159e+30-4.893543125372159e+30j) ->   (nan+infj)          infj
        (-4.893543125372159e+30+4.893543125372159e+30j) ->   (nan-infj)         -infj

What jax/jaxlib version are you using?
0.4.23
Which accelerator(s) are you using?
CPU/GPU

This issue is fixed via https://github.com/jax-ml/jax/pull/24874 . The current state of JAX square accuracy is summarized in https://github.com/pearu/functional_algorithms/issues/52.


Let methods in shellcraft be documented

Currently all files under shellcraft/templates cannot be documented.
Need redesign the shellcraft module to do so.
Features we want for shellcraft are listed as follows(so we can make sure our new design fulfills them):

Support multi-architecture (currently i386/amd64).
When one invokes shellcraft.sh, it would find the correct assembly according to context.arch.
All public methods are documented.
Since there are many common codes between these two architecture, we need DRY (currently we use x86 directory to achieve this)
Don't confuse between shellcraft.connect and shellcraft.syscall.connect, which was a bug in pwntools-python.
[Discuss] Do we really need to support something like shellcraft.i386.linux.sh under amd64 context? One can use context.local to achieve calling another architecture's assemblies.

Need suggestions @peter50216 @ShikChen @hanhanW

I would say no for the last point.

I think the main problem is the 'module' in python is separated by '.', i.e. shellcraft.amd64.linux.sh()
While in ruby style it should be Shellcraft::Amd64::Linux.sh
But the ruby one is hard for typing.
I suggested that we define methods in ruby way:
# amd64/common/common.rb
module ::Pwnlib::Shellcraft::Amd64
  # document goes here
  module Common
     # defines method `cat`
     include Pwnlib::Shellcraft::Helper
  end
end

# amd64/common/mov.rb
module ::Pwnlib::Shellcraft::Amd64::Common
  # document goes here
  def mov(src, dst)
     # ...
     cat "mov #{src}, #{dst}"
  end
end

# amd64/linux/sh.rb
module ::Pwnlib::Shellcraft::Amd64::Linux
  # document goes here
  def sh(*args)
     context.local(arch: 'amd64') do
        cat Shellcraft::X86::Linux.sh(*args)
     end
  end
end

And add an instance shellcraft that defines all methods like shellcraft.amd64.common.mov, shellcraft.amd64.linux.sh
And it adopts calls like shellcraft.${arch}.(common|${os}).xx to be abbreviated to shellcraft.xx


[AutoDiff] Add tests.

Add the following:

Crasher tests.
Activity analysis tests.
Control flow differentiation known incorrect derivative tests.

Crasher tests are in test/AutoDiff/compiler_crashers.
They each reference a JIRA issue and start with RUN: not --crash.
They should be moved to test/AutoDiff/compiler_crashers_fixed when they are fixed.

The following issues are newly tracked:
TF-429, TF-756, TF-781, TF-881, TF-923, TF-928, TF-945.
Some of these issues are in-progress!

@swift-ci Please test tensorflow

Added activity analysis tests in test/AutoDiff/activity_analysis.swift.

@swift-ci Please test tensorflow

@swift-ci Please test tensorflow


Github.dev crashing both on safari and chrome while trying to use iPad without connected keyboard

I want to edit my scripts with ipad screen keyboard but i can not use github.dev it keeps saying “setting up your editor” for several times and crashing after that. This is same for chrome and safari.

@RuzgarEken I will have to close this since we can not reproduce this.
If you can reproduce still can you please attach a video showing the crash. Then we can reopen this and forward to the owners. Thank you


As the proper implementation of ADR-003 won't be ready soon, the interface may be defined locally in this repository (KISS).
Ref: #64

CoreAPI in ADR-003 refers to IPFS CoreAPI interface.

This issue is replaced by https://github.com/celestiaorg/optimint/issues/127.


typo: Double word "cluster"

Link was also didn't work because of period between label and url parts

@nschonni Thank you for your submission! #sign-off


Document heapy diff limitations

Heapy diff can yield both false positives
and false negatives due to keying on heap
pointers. This is difficult to fix, so
we're just documenting this for now.
Refs #31

Thanks!


Post release PR did not get created for TOML module

Description:

$Subject during 0.2.0 release.
Below is the log:
Run gh release create v$VERSION --title "module-ballerina-toml-v$VERSION"
  gh release create v$VERSION --title "module-ballerina-toml-v$VERSION"
  gh pr create --base main --title "[Automated] Sync main after $VERSION release" --body "Sync main after $VERSION release"
  shell: /usr/bin/bash -e {0}
  env:
    JAVA_HOME: /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/11.0.19-7/x64
    VERSION: 0.2.0
    GITHUB_TOKEN: ***
https://github.com/ballerina-platform/module-ballerina-toml/releases/tag/v0.2.0
aborted: you must first push the current branch to a remote, or use the --head flag
Error: Process completed with exit code 1.

https://github.com/ballerina-platform/module-ballerina-toml/actions/runs/4955420167/jobs/8864813296
Need to check on this.

Closing since the PR is now sent: https://github.com/ballerina-platform/module-ballerina-toml/pull/14


Enable Orion to start using default options

Raised on gitter:
It's possible to start Orion without specifying any options but Orion is not usable because keys are not generated and there's no value specified for nodenetworkinterface.
For testing and to enable users to get up and running faster, Orion should behave in the same way as Besu.  If there is no key specified or existing database, then generate a key and define default values enabling Orion to start and the endpoints to be accessible.

Another option is display help instead of starting if not options are specified.


No license?

Hey! I'm looking into potentially using or adapting parts of this code in a project of mine (with credit given of course), but I'm not sure if that's permitted when there's no license.
If you're unsure of what license to use or just want more information about what they are, what they do, and why you need them, check out choosealicense.com. It's a site created by GitHub themselves for the exact purpose of informing people of what licenses are, what they do, and why the're needed.

Thanks for letting me know, I've added a license.


Runner listener service fails to start

I have a deployment where the runners report that they are running, but are not appearing in the github actions menu.  Looking at the logs of the runner pod, it appears that the runner listener is not starting properly:
Github endpoint URL https://github.com/

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication

√ Connected to GitHub

# Runner Registration

A runner exists with the same name
√ Successfully replaced the runner
The signature is not valid.
16c16
< ./externals/node12/bin/node ./bin/RunnerService.js &
---
> ./externals/node12/bin/node ./bin/RunnerService.js $* &
20c20
< wait $PID
---
> wait $PID
\ No newline at end of file
27c27
<                 listener = childProcess.spawn(listenerExePath, ['run'], { env: process.env });
---
>                 listener = childProcess.spawn(listenerExePath, ['run'].concat(process.argv.slice(3)), { env: process.env });
30c30
<                 listener = childProcess.spawn(listenerExePath, ['run', '--startuptype', 'service'], { env: process.env });
---
>                 listener = childProcess.spawn(listenerExePath, ['run', '--startuptype', 'service'].concat(process.argv.slice(2)), { env: process.env });
34c34
<
---
>
59c59
<
---
>
91c91
< });
---
> });
\ No newline at end of file
.path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Starting Runner listener with startup type: service
Started listener process
Started running service
An error occurred: Not configured
Runner listener exited with error code 2

the log then loops forever repeating the last five lines.
We are using the two container setup with a runner container based on summerwind/actions-runner:v2.274.2 and the v0.16.0 controller

It turns out that an incorrectly configured environment variable was causing the runner config.sh script to (silently) fail, which caused startup loop to enter this spin loop.

It turns out that an incorrectly configured environment variable was causing the runner config.sh script to (silently) fail, which caused startup loop to enter this spin loop.

@BrendanGalloway - what was the incorrect env var that caused this?

@BrendanGalloway - what was the incorrect env var that caused this?

At some point during the setup I had gotten the error that RUNNER_NAME was not set in the runner startup, so I added one to get further into debug and then forgot to remove it later.  The non-unique runner names were messing with the registration in the config script

At some point during the setup I had gotten the error that RUNNER_NAME was not set in the runner startup, so I added one to get further into debug and then forgot to remove it later.  The non-unique runner names were messing with the registration in the config script


Error code 139, Segmentation fault

When I set up the .env file, and run docker compose up in the project root directory, the backend is being built, but the fronted doesn't seem to work. I suspect this is because of the version of node in /frontend/Dockerfile. Because right after I changed the node version from FROM node:20.9-bullseye-slim to FROM node:22-bullseye-slim, the frontend website started to show.

Awesome, thanks for debugging and opening the PR. Will merge it shortly.


[msal-extensions] Change sample location. Add documentation and diagrams.

Coverage remained the same at 83.339% when pulling 31874bdc1bc6c466a674c659a6be74ceaa7a1a1e on extensions/update-sample into 1c1a65f9ccdf7f31b53e10f0a1fe5397afd4d788 on dev.

@jo-arroyo Thanks for the detailed review! I need to do a better job with spell checking documentation and error messages. 😅


Update go_register_toolchains version or allow the caller to specify it

What version of gRPC and what language are you using?
com_github_grpc_grpc: 1.46.0
language: Kotlin and Go (using the GRPC Gateway package https://github.com/grpc-ecosystem/grpc-gateway)
What operating system (Linux, Windows,...) and version?
Linux x86-64
What runtime / compiler are you using (e.g. python version or version of gcc)
Didn't get this far (failing at bazel build)
What did you do?
Please provide either 1) A unit test for reproducing the bug or 2) Specific steps for us to follow to reproduce the bug. If there’s not enough information to debug the problem, gRPC team may close the issue at their discretion. You’re welcome to re-open the issue once you have a reproduction.
In my Bazel Workspace file, I'm trying to load the dependencies load("@com_github_grpc_grpc//bazel:grpc_extra_deps.bzl", "grpc_extra_deps") in my project. However, when setting up GRPC Gateway, I need to add go_register_toolchains(version = "1.19.4"). However, in your code, it's hardcoded to be version 1.18. https://github.com/grpc/grpc/blob/master/bazel/grpc_extra_deps.bzl#L56
When I tried to use both in my Bazel Workspace, I would get an error that go_register_toolchains was already called with a different version. GRPC Gateway is not compatible with version 1.18, I get an error that files cannot be found.
What did you expect to see?
I would like to see the go_register_toolchains version not to be hardcoded so I can pass in a different compatible version. A note to recommend a default and changing at your own risk is OK.
What did you see instead?
go_register_toolchains version is hardcoded, so I need to copy the grpc_extra_deps file into my project and manually update the version.
Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).
See TROUBLESHOOTING.md for how to diagnose problems better.
Anything else we should know about your project / environment?

Something similar like this could be done: https://github.com/envoyproxy/envoy/blob/main/bazel/dependency_imports.bzl#L29

Hi, I'd love to help and can get this pretty easily. I was thinking basically the same as yijiem. I'm getting a permission denied error though and can't make a branch/PR.


Docs at http://django-vanilla-views.org/ not up to date

I just looked to adding this to a project, I was surprised to read that only Django up to 1.8 is supported on: http://django-vanilla-views.org/#requirements . This isn't the case on the repo, has the documentation build been failing?

tox.ini or .travis.yml?

Ah, yes. My pet hate. 😄
Alas, the answer is probably both. (But we might argue we don't really need tox locally for development...)

Just .travis.yml would be fine. I'd be okay with us dropping tox.

I reckon it's worth us bringing this back up to date and getting it under the encode umbrella. https://github.com/encode/

Can I do anything to help with move to encode/updating docs?

Ok so I figured out changing the docs in #75 , #76, and #77. Currently it relies on running the mkdocs.py script in the repo (not mkdocs the tool, perhaps an early prototype @tomchristie ?) and copying the results over to the gh-pages branch as I did in #77. They're up to date now at least.
@tomchristie can you give me access to the repo settings, then I'll move them to HTTPS using the new free Lets Encrypt on github pages :)

I've just updated the docs for the 1.1.0 release.
If you could check "Enforce HTTPS" @tomchristie that'd be neat.

@tomchristie I appreciate you're busy - perhaps you could add me as collaborator on the repo so I can do this and not ask you in the future?


Afwezigheidsoverzicht is unreadable

its all white

Thanks for the report! My (previous) school unfortunately hasn't enabled the new absences page yet. I'll be sure to fix this as soon as they do. For now, consider disabling dark mode.


Allow colon : to be used as key,val separator

The Ruby gem, which is quoted in the README, permits this, and as such, I expected this crate to allow it as well. Turns out, it doesn't :cry:

Thoughts on this? @sgrif @alicemaz

My kneejerk reaction is "what no that breaks sourcing the file in the shell." But I suppose .env files aren't typically shared so it's unlikely to impact anyone that didn't do it that way themself. No strong opinion either way.

The ruby gem allows this and I've never seen a good reason for it. I don't think it's worth supporting unless there's a legitimate use case

I'm closing this one for now. Thanks for your contribution, @elifoster, but I'm not sure we need this at the moment. If you'd like to discuss this further, please open an issue about it.


Google PaLM API

I have early Access to this API Now.
Added hook with google-palm.js.
Quick and dirty added. Currently raw output to screen.
https://developers.generativeai.google/guide

Need to add:

local storage
message array (memory)
formatting
tbd

local storage, message array memory and formatting added.
Now I need to handle bad responses ie:

{
"messages": [
{
"author": "0",
"content": "Hi Steven! It's nice to meet you. How can I help you today?"
},
{
"author": "0",
"content": "Strawberry ice cream is a delicious and refreshing treat that is perfect for a hot summer day. It is made with fresh strawberries, cream, and sugar, and it has a light and fluffy texture. Strawberry ice cream is a popular flavor all over the world, and it is often served with whipped cream and a cherry on top.\r\n\r\nStrawberry ice cream is a good source of calcium and vitamin C. It is also a good source of antioxidants, which can help protect your cells from damage. Strawberry ice cream is a low-fat food, and it is a good source of protein.\r\n\r\nIf you are looking for a delicious and refreshing treat, strawberry ice cream is a great option. It is a healthy and nutritious food that is perfect for any time of day.\r\n\r\nHere are some fun facts about strawberry ice cream:\r\n\r\n* The first strawberry ice cream was made in France in the 17th century.\r\n* Strawberry ice cream is the most popular flavor of ice cream in the United States.\r\n* The average American eats about 20 pounds of ice cream per year.\r\n* The world's largest ice cream cone is 10 feet tall and weighs 1,000 pounds.\r\n* The world's largest ice cream sundae weighs 10,000 pounds and contains 100 gallons of ice cream.\r\n* The world's most expensive ice cream is made with edible gold and costs $250 per scoop.\r\n\r\nI hope you enjoy your strawberry ice cream!"
},
{
"author": "0",
"content": "Yes, I remember your name. You said your name is Steven. It's nice to meet you, Steven."
},
{
"author": "0",
"content": "What is your name?"
}
],
"filters": [
{
"reason": "OTHER"
}
]
}

Memory is currently stored in the variables, so it wont clear unless we refresh. Need to have it save to local storage and recalled from local storage to allow the clear memory button to work.

Memory part is now good. Actually most of it should be good.


Server error: dispatch(HTTP.ParserError.invalidStartLine)

Upon installing example project, We are seeing this  error when trying to login the first time:
Server error: dispatch(HTTP.ParserError.invalidStartLine)

Hi @douglance are you running the SteamPressExample page with the engine as well? What OS are you experiencing the issue on and are you running straight Vapor or are you running it behind a proxy (like Nginx)? I haven't seen this issue at all I must admit (sounds like it could be a Vapor issue, though more detail will help diagnose)

Also what version of Vapor are you pulling in?

https://github.com/miswift/miswiftdotcom/blob/mailchimp/Package.swift
Is this your Package.swift that you are using? If so you need to update to Vapor 1.5

Good call! Thank you!

Actually, with the new Vapor version, I'm still receiving the same error and pages take an awful long time to load. It hangs on establishing a secure connection.

Have you tried a different browser? And are you running the VaporSecuityHeaders browser? I had an issue where it refused to load a CSS file in Safari but restarted the browser and the app and it all seemed fine

PS If you have used the Example site, that has a Content Security Policy to block non HTTPS requests, so if running locally your CSS download will just hang. The latest code on the example site on master has a fix for this

@douglance are you still having issues? Will close this issue if not

I cloned a copy of the Example Site, built it and ran it as a local host.  It now appears to be working fine.  I get the login and password on the terminal page.  I am able to log in.  It requires a new login password.  I was able to make a new blog post.  I believe this supports closing this issue


Standardize message packing

Currently wire messages are JSON and release messages are MessagePack. I think it would be nice to standardize our messaging. Either switching to MessagePack for everything, or ProtoBuf would be my personal suggestion.
Biggest pros I see to MessagePack is it's simple, and available in many languages if WH gets a community of folks who want to build out clients in languages other than protobuf supported languages.
With ProtoBuf, the static typing and ability to have back-compatible messages seem to be the biggest draw.
From what I gather perf differences should have relatively little impact since we see a relatively little amount of these messages in comparison to through-traffic.
Personally, I don't see much use in having clients in other languages(maybe c for better perf, but it supports proto bufs anyways). And I think having static typing/back compat could be really nice for expanding features in the future without breaking outdated clients. So my vote here is for full protobuf.
Thoughts @mrkurt @mbyczkowski

Another item of interest in my exploration is that protobuf doesn't support very many types. your standard variety of int/uint/float etc. Strings, and that's about it. Given you could have application specific twiddling to hack a solution if we need it. But MessagePack has facilities out of the box for everything protobuf does, plus arrays and maps.
I don't know if we need those extra features or not. Right now the only non-natural representation in WH is the VCSTimestamp which we represent as a time.Time. But again, we could just as easily rep it as a uint64 as seconds post epoch

Derp... Actually scratch that last one. They're about even on that front. Missed some protobuf docs. Protobuf may have the lead there actually on the power to add custom types

All in all, the 2 arguments I see for protobufs are:

Use a template file, so always ensured to generate correctly(don't change the template, code doesn't break) - however, extensions for messagepack exist for templates
Designed to allow back-compat for messages for clients running old versions. Careful use of messagepack can achieve the same result though

Personally I'm leaning to messagepack

And with the custom extensions as a first-class feature in this implemention I really think running with this repo would be awesome. https://github.com/tinylib/msgp. Fixes our Template issue above - and will also cleanup the use of the reflection package we're currently using in message/wire.go


[DOCS] Update welcome page

Problem/Concern
Update to include simple description & either embed video or link to it.
Proposed Solution

Issue completed by Dan


Add DataFrame.write.json and DataFrame.write.parquet functions

Please answer these questions before submitting your pull requests. Thanks!

What GitHub issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes SCT-10733, SCT-10735

Fill out the following pre-review checklist:

[x] I am adding a new automated test(s) to verify correctness of my new code
[ ] I am adding new logging messages
[ ] I am adding a new telemetry message
[ ] I am adding new credentials
[ ] I am adding a new dependency

Please describe how your code solves the related issue.
Add the functions DataFrame.write.json and DataFrame.write.parquet to internally call the copy_into_location function and unload data from a DataFrame into one or more JSON/parquet files in a stage.

I have read the CLA Document and I hereby sign the CLA


Should pos-resource support blank nodes?

There might be an architectural decision to make here. I'll describe my use case and current blockers.
I use pos-resource to make a resource available to descendants. This includes creating pos-resource elements and setting their uri programmatically (usually with lazy=true)
When the resource is a blank node, pos-resource errors with e.g. NamedNode IRI "_g_L9C317" must be absolute
I would need to confirm, but it appears the error is thrown by os.store.get when checking editable, which in turn assumes that the Uri is a named node: https://github.com/linkeddata/rdflib.js/blob/cd9a5960ef01bf9350c55a349db4410bc029a796/src/update-manager.ts#L154
There are other places that the PodOs code currently assumes a Uri is a named node/sym, e.g. https://github.com/pod-os/PodOS/blob/e4904287890fa3c03a37fb485d0b78ddd893cc34/core/src/thing/Thing.ts#L99
Current workaround is to rewrite data to avoid using blank nodes.
Issues to address:

Should it be possible to specify a blank node as URI for pos-resource (with lazy=true)?
If yes, how should editable status be determined for blank nodes? E.g. how is this usually handled by rdflib.js?
If yes, what other changes/tests are required?
If no, what is the preferred mechanism to access the relations of a blank node within PodOS?

You are right, PodOS mostly assumes named nodes currently and I did not spent much thoughts on blank nodes(yet). I am trying to avoid them as described in your workaround.
Some things come to my mind, but the whole blank node situation must be carefully thought through:

a blank node is also a resource just without a name (URI), so pos-resource could apply
using uri attribute to refer to a blank node is definitely not right, we either need to rename it or add another attribute to handle blank node
how would one refer to blank nodes?
pos-resource does 2 things: fetch and set the context for siblings. Fetching does not apply to blank nodes
we also need to consider collections, it is another but maybe similar topic

My current use case actually relates to collections - my list component creates pos-resource elements and sets their uri. Calling lazy=true suppresses fetching but allows descendants to receive the resource.
If it's a blank node, the uri starts with an underscore, e.g. _g_L9C317
Options for renaming would be resource or about. consistent with https://www.w3.org/TR/rdfa-lite/#resource or https://www.w3.org/TR/rdfa-primer/#alternative-for-setting-the-context-about
PodOS has the potential for its DOM to become valid RDFa.
Just checking now, it turns out RDFa supports explicitly defining blank nodes, which would work well for our case here.
https://www.w3.org/TR/rdfa-core/#s_blankNodes


Remove external-attacher from the repo.

Describe the bug
The presence of external-attacher adds on overhead and issues wrt attachment in various scenarios. One of them would be the lack of performance on syncing volumeattachment from api server..etc. More or less we don't have controller publish and unpublish capabilities , so we should not make use of this sidecar and cause unnecessary addon here thus other issues.
The solution:
Make use of skip attach field in CSI Driver object.
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: mycsidriver.example.com
spec:
  attachRequired: false

/me on it.

This needs extensive testing to make sure there won't be an issue for RWO PVC

this is already tried in https://github.com/rook/rook/pull/4172 and reverted back https://github.com/rook/rook/pull/4332 due to the issue we have seen in RWO PVC

This needs extensive testing to make sure there won't be an issue for RWO PVC

Yeah, rather this need extensive testing for all types of PVCs :).

this is already tried in rook/rook#4172 and reverted back rook/rook#4332 due to the issue we have seen in RWO PVC

The difference here is, this is GA in 1.18 compared to previous versions. So ideally it should work without issues. Lets try out with 1.18 version and see.

Tagging @gnufied
On a prior discussion regarding removal of the attacher, as it improved performance of overall pod startup times, it was pointed out by @gnufied that this essentially means double attach is detected and prevented by the storage plugin, and the basic AD controller will not ensure that the older node running the pod has actually detached etc. (this is even when the csi plugin ControllerPublish/Unpublish are NOPs) (unable to find a link to the discussion as yet)
Also, as @Madhu-1 points out, the prior attempt led to the related problems.
A subsequent speed up of the attach-detach operation was attempted here and was not fully successful.
HTH, in taking this forward as appropriate.

@humblec are you working on the fix? would you like to track this one for v3.0.0 if not please move this out of v3.0.0 release milestone

Moving it to release-v3.1.0

@humblec can this be done in 3.1.0? or do we need to move it out to the next release?

Yeah, we have to move this from 3.1.0. Done.. thanks :+1:

Considering skipAttach has been GAd with 1.18 and take care of the burden on unncessary operation (round-trip) in the preparation of a volume for a container, and requires CSI Drivers to deploy an unnecessary sidecar container (external-attacher), this is a good time to take it out.


Support for self-hosted instances?

Hi there
Is it possible to configure for a self-hosted instance of goatcounter?
Thanks in advance.
Edit: Wonder is pasting part of my url ending 'count' will work....seems not :-/
<script src="//gc.zgo.at/count.js" data-goatcounter="https://<selfhostedfqdn>/count" data-nscript="afterInteractive"></script>
instead of
<script data-goatcounter="https://<selfhostedfqdn>/count" async src="//<selfhostedfqdn>/count.js"></script>

As mentioned in the README, you can use the scriptSrc parameter to specify self hosted count.js location

As mentioned in the README, you can use the scriptSrc parameter to specify self hosted count.js location

Okay.


Feature: Search for a transaction by hash

The Request
The ability to search for a transaction by it's hash
Problem
Right now, there is no way to search for a transaction by it's hash.
Solution
As a user, when I navigate to the "Transactions" tab, I would like to search for a transaction by it's hash.

I'd like to get this one in as my first open source contribution to open source.

https://github.com/paradigmxyz/rivet/pull/51


Improve client side docs for authentication

Improve documentation
Link
https://supabase.io/docs/guides/auth/auth-apple
and other external provider pages
Describe the problem
Currently when using external ouath2 providers, you can't use the following code:
const { user, session, error } = await supabase.auth.signIn({
  provider: 'discord'
})

The reason being that when you call this function the user is redirected to the provider to sign in, which exits the function call and when you are returned to the page, unless supabase runs on page load the authtoken in the url isn't caught.
My solution to this was to have a listener on supabase.auth.onAuthStateChange which runs on every page, and allows me to have a switch statement like the following to reflect the state change:
    supabase.auth.onAuthStateChange((event, session) => {
        const data = session?.user;
        switch (event) {
            case 'SIGNED_IN':
                $user = {
                    id: data.user_metadata.provider_id,
                    supabaseId: data.id,
                    username: data.user_metadata.full_name,
                    avatarUrl: data.user_metadata.avatar_url,
                    email: data.email,
                };
                break;
            case 'SIGNED_OUT':
                $user = null;
                break;
        }
    });

Describe the improvement
Either the js client should improve to open a popup for external providers allowing for you to actually await the return of the session, or the need for what I said above should be added in the documentation. I noticed quite a few people in the discord faced a similar problem and this took me quite a while to figure out a good solution. And even this solution isn't perfect, I had to update it yesterday as it just stopped working in particular cases. I had to add a callback page witch ran supabase.auth.getSessionFromUrl(); to update the authStateChange function.

A duplicate of #460.


fix: add emptyDir to PSP

PSP is missing emptyDir referenced in deployment
Fixes #

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Revert OpenJ9 RHEL7 images to state of last build

Sorry for not addressing this in #120 . I think this should finally resolve the matter.
For OpenJ9/RHEL7 images this time.
This reverts commit 9f844fb4e1008cdf5973d7f0f13739012526be11 and also
rolls back the cekit reference. The sources now reflect exactly the
state they were in when the last builds were produced: confirmed by
cekit osbs build and comparing the dist-git commits.

@jmtd this is still not reverting the offending commits in release, namely:
1a896a325687848701a9eba239ade50b405f8a64
358bdf0be082b85449613ab45439cc16e76df30f
bfdf8bf4872c17185344c969a94b2ae88d085c43
d6c96d9c7fb0c77c5bce44a9a17b5de6a8cffc5d
e3b3d88c25b975fbfd855d8fd89dfd10058c84f8
0e6463fc7cae01e2e0b83bf6f8a2c07f0553e88c

Hi @yselkowitz, my intent here and in #120  was to revert the openj9 image sources to the "released" versions. Are there still problems with the image-stream metadata? What would you like to see the present state as?

openshift/library is using the release branch to reflect the currently available imagestreams.  Since OpenJ9 has yet to be released, either release needs to have all those commits reverted, or we need another (temporary?) branch which does reflect the current available state.

Filed #123 for the imagestreams reverts.


Enable cloud-provider in GCE based deployments.

This PR is in continuation to migrate deployment mechanism used by federation to k8s-anywhere as listed out in https://github.com/kubernetes/test-infra/issues/3858. This is the 5th task in that series.
Federation CI jobs needs k8s clusters deployed with cloud-provider enabled for load-balancer type services and persistent volumes.
/cc @luxas @pipejakob @madhusudancs @kubernetes/sig-federation-pr-reviews

/assign @pipejakob

Big apologies -- I've been out with a bad case of the flu for the past few days, but am catching up on things today.

No problem @pipejakob, take your time. Hope all is well now.

We'll just merge this after upgrade support is merged, and @shashidharatd can just put the new args in the config file as well
However, I really hope we don't have to do this, and instead can use the out-of-tree GCE cloud provider as an addon. cc @cheftako as your PR is needed for this to work

@luxas, @cheftako, i am curious to know when is the timeline we could use out-of-tree GCE cloud provider. federation need to start using the kbeadm based cluster deployment soon (like within a week or two). Should we wait until out-of-tree GCE cloud-provider is available or continue to use the current in-tree GCE cloud-provider?
Also @luxas, --cloud-provider flag is already removed from kubeadm init, so i cannot use the args passing method to kubeadm and instead should rely on config file to pass args to kubeadm. So i am in a  fix about what to do. Really appreciate any suggestions on way forward. Thanks !

@luxas, @cheftako, i am assuming external cloud-provider feature for GCE is not yet available and raised this PR to use the in-tree GCE cloud-provider and we shall update the code once it is available.
@jessicaochen i have modified the argument passing to kubeadm init from flags to config file. i have also updated the code path for upgrade scenario and require your review.
@pipejakob, i hope to have addressed your comments. request you to PTAL again.
Thanks !

cc @jagosan

--config has been around a very long time (~1.5), but the format of the file has changed slightly between versions. From what I saw, the args used there are all good (has been the same for many versions)

In that case /lgtm from an upgrade perspective.

@pipejakob, @luxas, request to have a look into this PR.
I have tested the installation scenario and it works well. But i have not tested the upgrade scenario. @jessicaochen request any suggestion or pointers to test the upgrade scenario.
Thanks !

/cc @marun, @irfanurrehman

I did the upgrade test by first deploying v1.7.8 cluster. Then i upgraded to v1.8.0.
Upgrade scenario is working along with the changes in this PR.
@pipejakob, can you please take it forward. Thanks !

Taking a look now.

/lgtm