[SPARK-8160][SQL]Support using external sorting to run aggregate

add config 'spark.sql.planner.sortMergeAggregate' to turn on sortMerge Aggregate, now default is false. and add two class to support sortMergeAggregate:
SortMergeAggregate is for Aggregate that cannot be codegened.
SortMergeGeneratedAggregate is for GeneratedAggregate that can be codegened.

Build triggered.

Build started.

Test build #35110 has started for   PR 6875 at commit daab5ba.

Test build #35110 has finished for   PR 6875 at commit daab5ba.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
trait GeneratedAggregate
case class HashGeneratedAggregate(
case class SortMergeAggregate(
  case class ComputedAggregate(
case class SortMergeGeneratedAggregate(

Build finished. Test FAILed.

Build triggered.

Build started.

Test build #35122 has started for   PR 6875 at commit 44a3e62.

Test build #35122 has finished for   PR 6875 at commit 44a3e62.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
trait GeneratedAggregate
case class HashGeneratedAggregate(
case class SortMergeAggregate(
  case class ComputedAggregate(
case class SortMergeGeneratedAggregate(

Build finished. Test PASSed.

@JoshRosen this may be of interest to you

@andrewor14, yes, definitely :smile:
I have a patch (#6444) to implement an optimized binary processing sort for use in Spark SQL and the change here will amplify the benefits of the new sort, so I'm super excited about this.

@JoshRosen yes, i have looked at  patch (#6444). this pr has no conflict with (#6444) because it just implement based-sort aggregate after sort by groupKey. so it can run on both external sort and binary processing sort.

@lianhuiwang Can we do hash based aggregation first, then switch to sort based if We can not hold all of  them in memory? (we still can have a flag to disable it)

@davies if we can not hold all of them in memory and then switch to sort based, it should re-shuffle data to do sort. so its computation cost is very expensive. i think it is determined by statistics before physical plan execution. this problem is similar as hash join or sort-merge join. now sort merge join is determined by spark.sql.planner.sortMergeJoin(default is false). like sort merge join, sort based aggregation of this PR is also determined by spark.sql.planner.sortMergeAggregate(default is false).

@lianhuiwang Aggregation is different than join, because aggregation could aggregation could reduce the data size, but join cannot. Optimizer could figure out whether use broadcast join or sort merge join based on the size of table, but it's very hard to guess what's the memory assumption will for aggregation (which is determined by the number of unique groups and aggregation algorithm).
All the aggregations happens within a partition, so no shuffling is needed. Usually, there are two aggregations  happen before and after shuffling.

@davies yes,i get it. at first, i think we can have a flag to do sort based aggregation on reduce stage that replace with hash based aggregation. and then based on memory of a task we can determine hash based aggregation or sort based aggregation and automatically set this flag.

@davies, i have implemented it in #7423, please take a look, thanks.

I think we should close this PR for now while we review the other one; let's re-open if necessary.

ok,@JoshRosen, thanks, i close this PR.


Working Dockerfile, nothing fancy

as the title suggests, I don't have any strong opinions about my work here. I tested it insofar as running in docker and pulling up localhost:7777 As I mentioned before this could be a nice way to get up and running quickly as local dev environments are so finicky

Excellent, thanks!


Structure member cross references can become stale

Bug Description:
Structure member cross references are not cleared by undoing the creation of a structure.
https://user-images.githubusercontent.com/10556585/223724605-26796d7f-4095-486a-af6f-bfd73cc1ae8a.mov

This issue can be triggered in a simpler way: just undefine the function that has the xref to the type field, and observe (in the type view or using Python) the type field xref still exists.


several cloud_to_cloud node_server interop tests failing

https://sponge.corp.google.com/target?id=586f9e41-c20c-4b5e-871e-2de4a19cc059&target=github/grpc&searchFor=&show=ALL&sortBy=STATUS
The following tests are failing:
cloud_to_cloud:ruby:node_server:unimplemented_method
cloud_to_cloud:ruby:node_server:custom_metadata
cloud_to_cloud:javaokhttp:node_server:unimplemented_service
cloud_to_cloud:ruby:node_server:status_code_and_message
cloud_to_cloud:ruby:node_server:unimplemented_service
cloud_to_cloud:ruby:node_server:timeout_on_sleeping_server
In all cases, the failure is related to the server returning UNAVAILABLE status.

It looks like those tests may have run before #13433 and grpc/grpc-node#95 were merged. They were supposed to fix that problem. I'm going to close this for now, and we'll reopen it if this problem shows up again.


gnome: don't include gnome-themes-extra by default

Description of changes
For #277944
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Could you please include in the commit message the reference to the removal from buildstream?

Some programs don't use dark theme if this package is not installed, e.g. Libreoffice and the file picker of JabRef.

Then that sounds like an insufficient heuristic in those programs. (I guess they detect if GTK theme ends with -dark.) The proper method would be checking the x-d-p settings portal. Or if the portal is not available, they could try looking at gtk-application-prefer-dark-theme setting (make sure you have both of those set properly).

Do the two dark theme settings listed in https://wiki.archlinux.org/title/GTK#Dark_theme_variant not work? It is hard to tell how to make random programs do anything without checking their source code.

No, it doesn't work for me.

If non-gnome applications have a surprise dependency on adawaita-dark for having a dark theme, we should probably add a release note as well mentioning this is no longer installed by default - as much as it's on the software in question to get fixed.

I encountered that with Sublime Text, which appears to use theme name https://github.com/sublimehq/sublime_text/issues/5194#issuecomment-1020709999
I tried to reproduce the described algorithm but I am always just getting black color:
let
  pkgs = import ../nixpkgs {};
in

let
  testSrc = pkgs.writeText "test.c" ''
    #include <gtk/gtk.h>
    #include <gio/gio.h>

    char* get_gtk_theme_name() {
        GSettingsSchemaSource* gnome_source = g_settings_schema_source_get_default();
        g_autoptr(GSettingsSchema) gnome_schema = g_settings_schema_source_lookup(gnome_source, "org.gnome.desktop.interface", FALSE);

        if (gnome_schema == NULL) {
            return NULL;
        }

        g_autoptr(GSettings) gnome_settings = g_settings_new_full(gnome_schema, NULL, NULL);

        return g_settings_get_string(gnome_settings, "gtk-theme");
    }

    char* get_gtk_background_color(const char* theme_name, GdkRGBA* color) {
        g_autoptr(GtkStyleContext) style_context = gtk_style_context_new();
        g_autoptr(GtkCssProvider) css_provider = gtk_css_provider_get_named(theme_name, NULL); // also tried "light" and "dark" instead of NULL

        gtk_style_context_add_provider(style_context, GTK_STYLE_PROVIDER(css_provider), GTK_STYLE_PROVIDER_PRIORITY_USER);

        gtk_style_context_get_background_color(style_context, GTK_STATE_FLAG_SELECTED, color);
    }

    int main() {
        gtk_init(0, NULL);
        g_autofree char* theme_name = get_gtk_theme_name();
        printf("theme = %s\n", theme_name);
        GdkRGBA background_color = { 1.0, 1.0, 1.0, 1.0 };
        get_gtk_background_color(theme_name, &background_color);
        printf("rgb(%f, %f, %f)\n", background_color.red, background_color.green, background_color.blue);
        printf("dark = %d\n", background_color.red < 0.5 && background_color.green < 0.5 && background_color.blue < 0.5);

        return 0;
    }
  '';
in
pkgs.runCommandCC
  "gtk-theme-color"
  {
    nativeBuildInputs = with pkgs; [
      pkg-config
    ];

    buildInputs = with pkgs; [
      gtk3
      glib
      makeWrapper
    ];
  }
  ''
    mkdir -p "$out"

    cc -ggdb \
      $(pkg-config --cflags gtk+-3.0 gio-2.0) \
      $(pkg-config --libs gtk+-3.0 gio-2.0) \
      "${testSrc}" -o "$out/test"
  ''

Regarding LibreOffice, it supports the proper FDO colors-cheme standard: https://github.com/LibreOffice/core/blob/06aea80b0482c03e8f0a5f735d3a31ec816a07fe/vcl/unx/gtk3/gtkframe.cxx#L1349 I can confirm that it works here, at least with libreoffice-fresh.

I find the problem in my config. I set /org/gnome/desktop/interface/gtk-theme = "Adwaita-dark"; which breaks libreoffice. I set it to Adwaita and libreoffice use the dark theme correctly. JabRef still has the problem.

Tauri based programs are also affected.


Bug: status endpoint 'KeyError'

Bug Description

Started a new version creation (dataset had one previous version) and it seemed to have finished successfully (looking at the STEP function logs), but get the error below when using the status endpoint function.
The end po
How to Reproduce

Create a new version on a dataset with an exising version
Wait until the STEP function finishes
Run the status endpoint, example payload that shows the error.

{
  "http_method": "GET",
  "body": {
    "execution_arn": "arn:aws:states:ap-southeast-2:632223577832:execution:processingnonproddatasetversioncreation969559C1-bA6hvJSBF4WR:2022-06-12T23-43-57-941Z_3VP1G23B47GM5DYM"
  }
}

See error

{
  "errorMessage": "'new_version_id'",
  "errorType": "KeyError",
  "stackTrace": [
    "  File \"/var/task/geostore/import_status/entrypoint.py\", line 16, in lambda_handler\n    return handle_request(event, REQUEST_HANDLERS)\n",
    "  File \"/var/task/geostore/api_responses.py\", line 42, in handle_request\n    return request_handlers[method](event[BODY_KEY])\n",
    "  File \"/var/task/geostore/import_status/get.py\", line 33, in get_import_status\n    response_body = get_import_status_given_arn(body[EXECUTION_ARN_KEY])\n",
    "  File \"/var/task/geostore/step_function.py\", line 106, in get_import_status_given_arn\n    version_id = step_function_input[NEW_VERSION_ID_KEY]\n"
  ]
}

What did you expect to happen?
What actually happened?
Software Context
Operating system:
Environment:
Relevant software versions:

AWS CLI:
Poetry:

Additional context

Definition of Done

[ ] This bug is done:

[ ] Bug resolved to user's satisfaction
[ ] Automated tests are passing
[ ] Code is peer reviewed and pushed to master
[ ] Deployed successfully to test environment
[ ] Checked against
CODING guidelines
[ ] Relevant new tasks are added to backlog and communicated to the team
[ ] Important decisions recorded in the issue ticket
[ ] Readme/Changelog/Diagrams are updated
[ ] Product Owner has approved as complete
[ ] No regression to functional or
non-functional
requirements

As discussed, this is because the test overlapped a deployment:

Your pipeline ran 11:43
The commit with the new version ID finished deploying at 11:52
You ran the status check at 11:54, resulting in the error


fix: time unit not match in ThreadMetricUtil.cpp

Signed-off-by: Lloyd-Pottiger<EMAIL_ADDRESS>What problem does this PR solve?
Issue Number: close #5595
Problem Summary:
What is changed and how it works?
Check List
Tests

[ ] Unit test
[ ] Integration test
[ ] Manual test (add detailed scripts or steps below)
[ ] No code

Side effects

[ ] Performance regression: Consumes more CPU
[ ] Performance regression: Consumes more Memory
[ ] Breaking backward compatibility

Documentation

[ ] Affects user behaviors
[ ] Contains syntax changes
[ ] Contains variable changes
[ ] Contains experimental features
[ ] Changes MySQL compatibility

Release note

None

/cc @fuzhe1989 @ywqzzy @bestwoody
PTAL again 😭

/merge

/hold

/merge

/unhold

/merge


Add registry proxying section

Define repository namespace query parameter for proxying.
Closes #12
Giving time for registry operators to weigh in
Maintainer approval

[x] @dmcgowan
[ ] @jdolitsky
[x] @jzelinskie
[ ] @mikebrow
[x] @stevvooe
[ ] @vbatts

Why do clients need to know anything about pull-through caching if its implemented server-side?

Why do clients need to know anything about pull-through caching if its implemented server-side?

The clients should know how the registry host was resolved from a given image reference. The clients don't care how the server is implemented, but they SHOULD provide information to the server which indicates what the reference being asked for is. Just as when an HTTP client connects with a PROXY server it must communicate what the upstream server is, the same is true here. Today the protocol doesn't define anyway to communicate what the upstream is and proxies end up be hardcoded to a single upstream. In a few cases you can see proxies use custom domains per upstream and require users to change the name of their images in order to use them.

Today the protocol doesn't define anyway to communicate what the upstream is and proxies end up be hardcoded to a single upstream.

Right... isn't that the point? If I encode that "myregistry/mynamespace/myrepo" goes to "upstream/foo/bar", that's a detail for the maintainer of myrepo and one that the client, ideally, doesn't need to know; the whole point is the puller thinks they are getting "myregistry/mynamespace/myrepo".
If the goal is to allow the client to specify "upstream/foo/bar", then I'd that the target is not really a repository anymore, but simply a working proxy, and thus, a different protocol parameter might be useful, but registries should therefore have the option to not support said parameter.

Right... isn't that the point?

That is one use case that will still work. In the example you mentioned, when a repository is proxied in that fashion, the puller often does know of this detail as they must explicitly provide myregistry with the intent of getting some upstream content. The use case where myregistry is some sort of blessed version of upstream is reasonable, but not the intent of the namespace parameter here.

If the goal is to allow the client to specify "upstream/foo/bar"

This is the use case here and proxy may be better terminology here, but that is really a detail of the registry. The registry may act as a proxy, proxy-cache, or active mirror, that is out of scope for definition here. This parameter just enables all of those features to work across multiple namespaces. For example if you want public images from both docker.io/* and quay.io/* to be cached in the same registry proxy today, you would need the server to have two hostnames (something like docker.io.myproxy and quay.io.myproxy) then have clients configured to do that mapping for each namespace. This simple query parameter provides a much simpler option to clients and servers. If a server does not support it, it ignores the parameter. If a client is configured to send all requests to a server which does not support it, that is not different than any other misconfiguration by clients today.

For example if you want public images from both docker.io/* and quay.io/* to be cached in the same registry proxy today, you would need the server to have two hostnames (something like docker.io.myproxy and quay.io.myproxy) then have clients configured to do that mapping for each namespace

Or configure two repositories, one for each? (especially since combining them could lead to merge conflicts).
I'm concerned we're adding quite a bit of complexity to address a use case that has simpler solutions when configured on the registry side.

I'm concerned we're adding quite a bit of complexity to address a use case that has simpler solutions when configured on the registry side.

Can you elaborate here? Configuring a repository for each mirror is non-trivial. Configuring a domain for each upstream and routing to the upstream based on the domain is not easier, that would still requires the same routing on the server side that an implementation of this would require. The client side implementation to support per-registry configuration is not simple and inherently requires catch -all conditions when trying to enforce proxying through a gateway.
I did do a client side implementation of this to demonstrate the feature and allow server side implementations a client to test against. On the client side, it is not complex at all since clients should already know how to handle 404s when multiple registry endpoints are configured. On the server side, the complexity to support this isn't much more than existing proxy-cache support.

Configuring a repository for each mirror is non-trivial.

Its non-trivial, but its not that difficult either :)
My concern remains around complexity: the document as outlined, for example, says that the ns should not be sent to non-mirroring registries... but how does the client know that? Is it the registry's job to report back an error if that argument is found but unsupported? How will clients know to be able to check for this capability?
If we feel that pass-through proxying of other registries is, in and of itself, a feature of the protocol (rather than something configured on the registry side), then I suspect we need to give significantly more thought to the end-to-end user experience. For example, I could imagine some paths supporting proxying and others not.

how does the client know that?

The clients have the most context and really does not need to be defined here, only that a client SHOULD make that distinction to avoid sending unnecessary redundant information. The clients themselves have both the configuration and endpoint resolution logic, so it has multiple options for determining this. In the implementation I sent I just simply did this by checking whether the endpoint was configured without push support, as this could indicate the registry being communicated with may not be the upstream source. However, I will probably add a check there for ns != host since there are never push configurations (such as with a Kubernetes runtime). Either way, this is trivial and not required.

Is it the registry's job to report back an error if that argument is found but unsupported?

No, the registry can simply ignore it. This is like asking a registry today which was configured to mirror docker.io to return an error if the client actually meant quay.io, the registry just isn't expected to have the same amount of context as a client in regards to the intent of the entire pull process. If the registry chooses to be handle the ns parameter and not support it, it is as easy as returning a 404 for unconfigured upstreams.

How will clients know to be able to check for this capability?

They aren't expected to check for it, but rather be explicitly configured for it. A client will know if it is configured to always use a specific mirror or a mirror for multiple namespaces. I think what you are suggesting here though is the idea of registry discovery. That is a much larger topic that I would still love to see happen, in that feature a client could start with zero knowledge (except of course the domain quay.io, docker.io, etc) and discover registry capabilities and endpoints.

discussion ensues on the call today.
This sounds like a decent addition, but with a clear use-case for the behavior, and whether a registry implementation MUST support it.

pinging @thomasmckay and @kurtismullins who are implementing mirroring on Quay -- they probably have feedback and want to track this thread

Pulp container plugin team will want to keep an eye on this thread as well. Any feedback @ipanova @dkliban @asmacdo ?

is ns already used, so best to continue with that mnemonic? or could
it be something to not collide with the outdated concept that images
would on be named "transport/namespace/name:tag"?

@vbatts I use ns or namespace here because namespace is a very generic concept. Certainly as a generic concept it has been used to mean many things. However generally namespace would refer to additional context (such as a prefix) on another name. In the distribution spec case, the name given to the registry would be the part on the URL path, the namespace just gives that additional context to the name. Existing distribution clients today parse the name as you described <sometransport/host/whatever>/<name given to the registry>, in which case when given just <name given to registry>, the <sometransport/host/whatever> would be the namespace of that. You could continue to divide those parts in smaller names in other namespaces, such as the Docker hub does with usernames/reponames, but that is out of scope here. Capturing this in elegant words is kind of tough, recommendations on which parts are unclear or how to make it better are appreciated.

@dmcgowan one thing i'm unclear on here is: can i have a single registry mirror that will be usable for more than one remote registry (i.e. remote of docker.io/..., quay.io/..., etc)

Notes from today's call:

(jz) quay does something different for mirroring. This should be called "proxying"
(dmcg) this is really for client side caching proxy, and is needed.

Please lets find a way to classify this language (whether client or server side). So we can close  out or merge this

Reiterating our convo from the meeting:
I actually see a lot of value in adding this query parameter, but removing any connotation that it is the blessed solution for repository mirroring. I think that by including this value, a proxy could implement lots of different behavior for the client that need not be directly related to repository mirroring.

I note that harbor uses the term "replication" rather than proxy-cache or mirroring, which I quite like https://goharbor.io/docs/1.10/administration/configuring-replication/

Updated section to remove use of the term "mirror". A mirror is just a special case of a proxy or could be a completely different configuration. For example, the most common "mirroring" case which is used by Docker is actually a pull-through cache. Using the more generic terminology here instead of continuing the confusion :)

LGTM

big time rebase needed @dmcgowan

Hi team, just checking to see when this is likely to be merged in? Thanks

@dcarrion87 it's going to be a while. Effort is still continuing on the 1.1 release. After that, we'll revisit this.

I don't mind rebasing this after 1.1 to target 1.2. It really is a pretty light change, it has been included by containerd since 1.4. The issue on the client is whenever a proxy is used, the client is throwing away information that could be important to the registry. We only include ns whenever the proxy host name does not match the requested image host name. I would love to see a registry proxy make use of it even before it hits an official 1.2.0 release.

I just wanted to check in if this proposed change has just stalled or is abandoned? I know that these things take a long time, so it can at times be hard to tell.
I have been working in depth with mirroring OCI artifacts over the last year now. The addition of an ns query parameter solves a lot of problems seen in different projects. The major issue right now seems to be that all workarounds are non standard, causing incompatibility. Containerd adds the query parameters whil CRIO does not. Some registries will look for the query parameter while others either only support a single registry or suggest including the root url in the namespace.
I just wanted to add my input that I think that adding this to the spec would be very useful and could potentially help pain points that exist today when mirroring.
If there is any additional work required before merging this, I would be happy to help.

I've been watching this as well, and would like to voice my support for getting this officially included in the spec.

+1

Operating https://registry.k8s.io  we see a TON of requests that are obviously from pull-through caches/mirrors like artifactory for images that we don't host. If this API was standardized we could eventually start requesting that these tools switch to ns-aware pulls and stop spamming us with requests for images we don't provide.
While not the top of our concerns, this feels wasteful and generates a lot of log noise, and isn't free.
The current state where pull-through mirroring pretty much has to request all upstreams feels bad, what concerns remain with this proposal?

Operating https://registry.k8s.io  we see a TON of requests that are obviously from pull-through caches/mirrors like artifactory for images that we don't host. If this API was standardized we could eventually start requesting that these tools switch to ns-aware pulls and stop spamming us with requests for images we don't provide.

I'd guess this is from users thinking they have setup a proxy, and docker automatically falling back when the proxy pull fails. This is made worse by docker only supporting a proxy to Docker Hub. The result is the proxy does nothing, never used to pull k8s images (since the client would pull directly) and never finding a match (since the proxy isn't pointing to Hub) for the Hub pulls.

The current state where pull-through mirroring pretty much has to request all upstreams feels bad, what concerns remain with this proposal?

Are there proxies doing this today? The few I've used are configured for a single upstream registry, at least for a given namespace or port.
I worry we'll all agree this is a good change, but the key players in the space won't implement it. It would be good to see this being actively used by both clients and proxies.
We also need to consider existing clients and proxies in various combinations. E.g. if a proxy for a single registry is working with older clients and gets upgraded to use the headers, those older clients still need to go to the original registry. Similarly, newer clients configured with an older proxy should not pull content from the wrong upstream registry, since that would expose them to attacks by dependency confusion and namespace squatters.

I'd guess this is from users thinking they have setup a proxy, and docker automatically falling back when the proxy pull fails. This is made worse by docker only supporting a proxy to Docker Hub. The result is the proxy does nothing, never used to pull k8s images (since the client would pull directly) and never finding a match (since the proxy isn't pointing to Hub) for the Hub pulls.

The issue where docker only supports configuring a mirror for dockerhub is also pretty unfortunate but not what I'm referring to, the headers are for agents like artifactory (and other similar tools).
My understanding is that users setup a central catch-all pull-through cache and have no way to disambiguate which upstream the pulls were meant for, so we get pulls for ... a lot of images clearly not associated with us.
This seems to be common with self hosted options, versus something like ECR/AR remote repositories. It's understandable that adding a distinct IP / hostname for every proxied host is heavy handed, and I don't think most users even think to use a port other than 5000.

Are there proxies doing this today? The few I've used are configured for a single upstream registry, at least for a given namespace or port.

AFAICT, yes, for example:
https://jfrog.com/help/r/jfrog-artifactory-documentation/virtual-docker-repositories
"This allows you to access images that are hosted locally on local Docker repositories, as well as remote images that are proxied by remote Docker repositories, and access all of them from a single URL defined for the virtual repository. Using virtual repositories can be very useful since users will continue to work with the virtual repository while the admin can manage the included repositories, replace the default deployment target and those changes will be transparent to the users. [...]"

I worry we'll all agree this is a good change, but the key players in the space won't implement it. It would be good to see this being actively used by both clients and proxies.

This seems a bit circular? If we don't document the spec for this optional feature then most projects are not going to enable support (e.g. https://github.com/distribution/distribution/pull/3864#issuecomment-1974748544)

We also need to consider existing clients and proxies in various combinations. E.g. if a proxy for a single registry is working with older clients and gets upgraded to use the headers, those older clients still need to go to the original registry. Similarly, newer clients configured with an older proxy should not pull content from the wrong upstream registry, since that would expose them to attacks by dependency confusion and namespace squatters.

That's true, but a proxy can behave more intelligently when the namespace parameter is present and over time more clients will have it, there are various references to waiting for this to be standard. The proxies can fall back to existing behavior such as querying all upstreams when the header is missing.
Containerd alone represents a lot of clients that already have this header in place. Were it to become standard and proxies begin supporting it there's actually a pretty large client base sending it already, but there's resistance to supporting it more widely referring back to waiting for this PR.

The issue where docker only supports configuring a mirror for dockerhub is also pretty unfortunate but not what I'm referring to, the headers are for agents like artifactory (and other similar tools).

I'm not sure how you'd differentiate the scenarios from the k8s registry. It would just look like a bunch of failed image pulls to that registry with either scenario.

Are there proxies doing this today? The few I've used are configured for a single upstream registry, at least for a given namespace or port.

AFAICT, yes, for example:
https://jfrog.com/help/r/jfrog-artifactory-documentation/virtual-docker-repositories

Ugh, someone should open a dependency confusion CVE against them if that's really what's happening.

I worry we'll all agree this is a good change, but the key players in the space won't implement it. It would be good to see this being actively used by both clients and proxies.

This seems a bit circular? If we don't document the spec for this optional feature then most projects are not going to enable support (e.g. https://github.com/distribution/distribution/pull/3864#issuecomment-1974748544)

We've always standardized things that were already implemented somewhere to show the feasibility first. You can't untag a standard, and implementations are effectively the CI for a standard. If the distribution project doesn't want to add it, or maintain a feature branch, then we'd need another project.

We also need to consider existing clients and proxies in various combinations. E.g. if a proxy for a single registry is working with older clients and gets upgraded to use the headers, those older clients still need to go to the original registry. Similarly, newer clients configured with an older proxy should not pull content from the wrong upstream registry, since that would expose them to attacks by dependency confusion and namespace squatters.

That's true, but a proxy can behave more intelligently when the namespace parameter is present and over time more clients will have it, there are various references to waiting for this to be standard. The proxies can fall back to existing behavior such as querying all upstreams when the header is missing.

Registries falling back when the header isn't seen is probably easy enough. That needs to be to a single registry to avoid a dependency confusion attack.
The fallback for new clients talking to older registries is less clear. The API needs to ensure new clients either know to fallback when talking to older proxies, or that the API doesn't exist on old proxies to avoid the security risk of sending requests to the wrong upstream registry.

Containerd alone represents a lot of clients that already have this header in place.

With Docker updating that integration, the single proxy setting to Hub will hopefully stop being an issue in the future.

If you need an example of the ns query parameter being used in the wild here is one. I am currently using it in Spegel do enable tag resolving in a simple way.  This is less of an issue for me as I only support Containerd which has been adding this parameter to all of its image pulling request for a few years now. This has been working really well as from a user perspective it becomes  a seamless experience.
https://github.com/spegel-org/spegel/blob/ecfdd57cb2a22513c5785cce89e5c8eb6771dbf3/pkg/registry/registry.go#L180-L185
The other solution that I have seen used out in the wild is to add a header to the mirror configuration so that the receiving registry is aware of the original registry the image originates from. This is how Drangonfly solves the problem for example. Which means that they would not be able to support wild card mirror configuration.
https://d7y.io/docs/next/operations/integrations/container-runtime/containerd/#multiple-registries
Right now every registry out there seems to come up with their own standard for how to deal with mirroring multiple namespaces. A common practices is to to join the original image name reference with the mirror registry. For example docker.io/library/ubuntu:20.04 would become example.com/docker.io/library/ubuntu:20.04. While it may work it creates another set of challenges forcing users to modify image reference and creating incompatibility with other tools that do not understand the logic implemented in the name.

Can this be used more generically? Not just for image pulls from a proxy, but also for tag listings, referrers, possibly even a push-through proxy? Is there a way users can directly reference content on a proxy with a namespace parameter, e.g. an added field in the image reference like proxy.example.org/some/image:some-tag[registry.example.com/some/image]? I'm considering how this could be extended to accessing content in an OCI Layout were an export has multiple images with the same tag but different source repositories.


Bug in Rating API error message 118400

Hi,
it seems to me that in rating API error message 118400 as well as in the same message in Shipping API 128400 there is a new bug inttroduced a few months ago. It kind of made it's way even to the documentation

The variables %xyz% are not replaced by meaningful information but appear explicit as such.
For us it would be important to restore the correct behaviour since, e.g., we use the information in %dest.AdjCountry% to correct an earlier request for a destination region we couldn't identify in the first place.
Thx for your suggestions.

Thank you for bringing this concern to our attention.  Our technical team is looking into it and will provide a more substantive response shortly.

This is confirmed as a production defect and will follow the process to address it.  Thanks again for bring this to our attention.

I'd appreciate if you updated us here once issue is resolved.
BTW, the same problem applies to other messages, e.g. 128400, but I assume you figured that out yourself.
cheers.

Definitely!  And appreciate your patience.

Would also really love an update on this @chuntoulee

Hi, thank you for your comments. The Rate API has been updated to replace the variables.

since we moved to the REST API meanwhile I do unfortunately have to confirm and restate that the error still persists.
Please dear UPS devs, give it a chance and repair it, it might not be relevant in your main market US but it very much is in Europe.

Hi, thank you for your comment. To confirm, have you experienced this behavior recently while using the RESTful Rate API?

HI UPSRahul,
yes. Recently (sept 4) with the RESTful API
Request fragment(anonymized)
"ShipTo":{"Name":"XXXX","AttentionName":".","Phone":{"Number":"XXXX"},"Address":{"AddressLine":[". ","XXX"],"City":"ADEJE","PostalCode":"38670","CountryCode":"ES"}}
Response:
{"response":{"errors":[{"code":"128400","message":"The destination postal code dest.postal is not a valid dest.country postal code. Verify your postal code or select dest.AdjCountry as your destination country or territory."}]}}

for clarification, the fragments above are from shipping API


Problems syncing globalState on web ENOENT argv.json

Right after turning on sync this happens (selfhost web, local server).
[2020-02-06 06:26:14.856] [userDataSync.log] [error] GlobalState: Unable to read file 'vscode-userdata:/User/argv.json' (EntryNotFound (FileSystemError): Error: ENOENT: no such file or directory, open '/Users/daimms/.vscode-web-dev/data/User/argv.json')

argvResource is not available in resource.
@bpasero I remember we discussed about this that argvResource has to be made optional in web?

@sandy081 but it is defined for web:
https://github.com/microsoft/vscode/blob/bb3edc40e6430179721ce2c7945557958fd94b7e/src/vs/workbench/services/environment/browser/environmentService.ts#L137
I am not sure how useful it is for web though, I do not think we currently reference it for web anywhere.


Fix mekabow autofire not working properly when attack speed too high

The Meka Bow's autofire unit does not work when player's attack speed is vastly increased. To replicate: Equip God's Crown from Reliquary mod (+120% all stats), Meka Bow's autofire stops working properly. (MC version 1.19.2, but probably happens on other versions too). Bug can probably be replicated with any other way to increase attack speed.
Changed firing check from == to <= as a hotfix.

hello, is this still a bug in the latest 1.21 version? could you check it please?

hi, unfortunately i'm busy with other things at the moment and will be unable to test the issue 😞 I have not checked if this bug happens in other versions other than 1.19.2


IOS : Some pages of the app gets blinks rapaidly ,when user goes to online from offine

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:
1.login to app
2.turn off the  wifi/mobile data data
3.open app
4.again  turn on the wifi/mobile data
5.Observe that, chat composer, about page, main chat listing page gets blinks
rapidly
Expected Result:
chat composer, about page, main chat listing page shouldn't blink ,when moves back to online from offline
Actual Result:
chat composer, about page, main chat listing page gets blinks
rapidly user, when moves back to online from offline
Workaround:
Can the user still use Expensify without this being fixed? Have you informed them of the workaround?
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[x] iOS / native
[ ] iOS / Safari
[ ] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number:
Reproducible in staging?: Needs reproduction (Applause not able to reproduce)
Reproducible in production?: Needs reproduction (Applause not able to reproduce)
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://github.com/Expensify/App/assets/43996225/e4f276c9-2d03-4ef8-b3d6-1c907ebab65e
Expensify/Expensify Issue URL:
Issue reported by: @lincolnb17
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1689825356877209
View all open jobs on GitHub

What iPhone is this on? I can't reproduce it on iPhone 14 Pro.

@JmillsExpensify  i tested iPhone 12 Pro Max

Going to open this issue up for the community, though of questionable value given that it's visual and doesn't appear on all iOS devices.

No proposals yet

Let's give this one another test and then close if not reproducible.

@JmillsExpensify @mvtglobally  i am still able to reproduce in my iphone 12 pro max
https://github.com/Expensify/App/assets/47911127/059266b3-e251-4f6f-ad4d-7bfacf4243dc

@lincolnb17 Can you share what version of iOS you are using?

@JKobrynski 16.5.1 (c)

@JmillsExpensify I am also unable to repro the issue. And just to note we don't have any proposals yet.

@JmillsExpensify Bump

Not sure if we have a good process for this one. Several aren't able to reproduce but one person can. Still no proposals in any case.

If we don't want to close this, should we increase bounty since there is no proposal yet?

@JmillsExpensify Bump

Pinged @JmillsExpensify in slack. He mentioned to look into this tomorrow

Given that QA hasn't been able to reproduce, I'm inclined to close rather than increase the bounty.

@JmillsExpensify  Fine by me as well. We can close this. Not able to reproduce

Bump @JmillsExpensify We can close this out IMO

Agreed, closing.


Services fail to start when deployed on Kubernetes

Hello,
I am trying to install all Edgex Foundry micro services on Kubernetes. I created a Helm Chart for all services following the template from https://github.com/edgexfoundry/developer-scripts/blob/master/releases/hanoi/compose-files/docker-compose-hanoi-no-secty.yml
After Helm install command being applied all services except Consul, Redis, Kuiper and UI are failing to start.
I guess the main difference from running via docker-compose.yaml is services are able to find Core Consul service when they try to connect for the first time. If I start the services in a Docker environment they can not find the Core Consul service when they try to connect at first attempt.
The logs from Edgex Core Command service when running on Kubernetes:
level=INFO ts=2020-12-07T10:44:12.816281446Z app=edgex-core-command source=config.go:193 msg="Loaded configuration from /res/configuration.toml" level=INFO ts=2020-12-07T10:44:12.817027768Z app=edgex-core-command source=variables.go:394 msg="Variables override of 'Registry.Host' by environment variable: REGISTRY_HOST=edgex-core-consul" level=INFO ts=2020-12-07T10:44:12.817103407Z app=edgex-core-command source=variables.go:394 msg="Variables override of 'Clients.Metadata.Host' by environment variable: CLIENTS_METADATA_HOST=edgex-core-metadata" level=INFO ts=2020-12-07T10:44:12.817142795Z app=edgex-core-command source=variables.go:394 msg="Variables override of 'Service.Host' by environment variable: SERVICE_HOST=edgex-core-command" level=INFO ts=2020-12-07T10:44:12.81733108Z app=edgex-core-command source=variables.go:394 msg="Variables override of 'Databases.Primary.Host' by environment variable: DATABASES_PRIMARY_HOST=edgex-redis" level=INFO ts=2020-12-07T10:44:12.817573469Z app=edgex-core-command source=config.go:168 msg="Using Configuration provider (consul) from: http://edgex-core-consul:8500 with base path of edgex/core/1.0/edgex-core-command" level=INFO ts=2020-12-07T10:44:12.833754274Z app=edgex-core-command source=variables.go:394 msg="Variables override of 'Databases.Primary.Host' by environment variable: DATABASES_PRIMARY_HOST=edgex-redis" level=INFO ts=2020-12-07T10:44:12.834489558Z app=edgex-core-command source=variables.go:394 msg="Variables override of 'Registry.Host' by environment variable: REGISTRY_HOST=edgex-core-consul" level=INFO ts=2020-12-07T10:44:12.834565738Z app=edgex-core-command source=variables.go:394 msg="Variables override of 'Clients.Metadata.Host' by environment variable: CLIENTS_METADATA_HOST=edgex-core-metadata" level=INFO ts=2020-12-07T10:44:12.834679033Z app=edgex-core-command source=variables.go:394 msg="Variables override of 'Service.Host' by environment variable: SERVICE_HOST=edgex-core-command" level=INFO ts=2020-12-07T10:44:12.835172578Z app=edgex-core-command source=config.go:304 msg="Configuration has been pulled from Configuration provider (4 envVars overrides applied)" level=ERROR ts=2020-12-07T10:44:12.835209369Z app=edgex-core-command source=logger.go:74 msg="logTarget cannot be blank, using stdout only" level=INFO ts=2020-12-07T10:44:12.835314116Z app=edgex-core-command source=registry.go:80 msg="Using Registry (consul) from http://edgex-core-consul:8500" level=INFO ts=2020-12-07T10:44:12.877779584Z app=edgex-core-command source=database.go:147 msg="Database connected" level=INFO ts=2020-12-07T10:44:12.878271505Z app=edgex-core-command source=telemetry.go:86 msg="Telemetry starting" level=INFO ts=2020-12-07T10:44:12.878330912Z app=edgex-core-command source=httpserver.go:99 msg="Web server starting (edgex-core-command:48082)" level=INFO ts=2020-12-07T10:44:12.878352662Z app=edgex-core-command source=message.go:50 msg="Service dependencies resolved..." level=INFO ts=2020-12-07T10:44:12.87836673Z app=edgex-core-command source=message.go:51 msg="Starting edgex-core-command 1.3.0 " level=INFO ts=2020-12-07T10:44:12.878389886Z app=edgex-core-command source=message.go:55 msg="This is the Core Command Microservice" level=INFO ts=2020-12-07T10:44:12.878416073Z app=edgex-core-command source=message.go:58 msg="Service started in: 63.345277ms" level=INFO ts=2020-12-07T10:44:12.879632782Z app=edgex-core-command source=httpserver.go:107 msg="Web server stopped" level=INFO ts=2020-12-07T10:45:41.874835546Z app=edgex-core-command source=httpserver.go:116 msg="Web server shutting down" level=INFO ts=2020-12-07T10:45:41.874927848Z app=edgex-core-command source=httpserver.go:118 msg="Web server shut down" level=INFO ts=2020-12-07T10:45:41.874959337Z app=edgex-core-command source=database.go:161 msg="Database disconnected" level=INFO ts=2020-12-07T10:45:41.939392666Z app=edgex-core-command source=telemetry.go:98 msg="Telemetry stopped" level=INFO ts=2020-12-07T10:45:41.939508883Z app=edgex-core-command source=bootstrap.go:127 msg="Un-Registering service from the Registry"
Thanks,
Sukru

It almost looks like Kubernetes is shutting the service down after 30 seconds.  Is a healthcheck configured that is causing the POD to be deemed failed?

Can you run the services without Consul?
Please try to override the CMD to only "--confdir=/res"
Consul is using by default when the container starts up
https://github.com/edgexfoundry/edgex-go/blob/7f9dd25f9d981fc062fb35c9729d56f5b41711bb/cmd/core-command/Dockerfile#L42

This is not an issue with Consul (as I indicated in the Slack discussion). As @bnevis-i commented and I pointed out in Slack, something is shutting down the services. Closing this issue. Discussion can continue in Slack.


No captions on 18.02sc video

Steps to Reproduce

See https://ocw.mit.edu/courses/18-02sc-multivariable-calculus-fall-2010/resources/clip-total-differentials-and-chain-rule/
The legacy site has captions (displayed in a pop-up window)

Expected Behavior

should be able to run captions on, and use the interactive transcript

Actual Behavior

no caption control
no interactive transcript
no button to download captions

I haven't looked to see how widespread this issue is in this course.
This scholar course was authored in a very different way from other scholar courses, and all other courses.

@pdpinch I did some digging into this today. At least part of what's going on here seems to be that the videos in question don't have 3Play captions. In ocw-to-hugo, we find captions using:
  const captionsFile = media["embedded_media"].find(
    embeddedMedia =>
      embeddedMedia["id"].endsWith(".vtt") &&
      embeddedMedia["title"] === "3play caption file"
  )

Looking up the example video from above, it does not have 3Play captions and instead has this:
{
  "uid": "0ed7fe55a1b025da274a7bf76a0a45e8",
  "parent_uid": "3c74015a1e39d58c6c35744224686cd7",
  "id": "Caption-OCW-MP4",
  "title": "Caption-OCW-MP4-English - US",
  "type": "Caption"
}

Some of the other videos in the course do have 3Play captions, which look like this:
{
  "uid": "78c777bec50c5543869c26bf10fe0a44",
  "parent_uid": "faa0b05cc4a5b2e55ff56453d12c69c6",
  "id": "CCoTAyZ14XM.vtt",
  "title": "3play caption file",
  "type": null,
  "technical_location": "https://ocw.mit.edu/courses/mathematics/18-02sc-multivariable-calculus-fall-2010/4.-triple-integrals-and-surface-integrals-in-3-space/part-b-flux-and-the-divergence-theorem/session-84-divergence-theorem/flux-and-the-divergence-theorem/CCoTAyZ14XM.vtt"
}

Looking up the UUID for that caption across the rest of the JSON export for that course comes up with nothing. There are no exported static files in the course with that UUID in their filename either. Looking at the type of Caption-OCW-MP4 makes me think that maybe the captions for these videos are embedded in the MP4 files themselves? Other videos that have a 3play caption also have a transcript property that has an entire HTML based transcript of the video, but the "mp4" captioned videos do not have a transcript in the export either.
If you look here you can see the video with its interactive transcript, so clearly it does exist somewhere. Going back to my local copy of the raw Plone exports, I don't even see 18.02SC anywhere. Tomorrow I will dive into S3 and see if I can find anything there.
Another potentially notable thing is that template_type is set to thumbnail_popup on these, yet on the old site they seem to be displayed inline. Do you have any further background information on these types of videos?

@abeglova or @pt2302 can you check in 3Play to see if the captions exist on their site? (based on Carey's note about the legacy site having an interactive transcript, I believe the captions should be on 3Play)

Since Carey's out until 12/12, I removed his assignment


picom: 12.3 -> 12.4

Updated to 12.4, fixing a bug introduced in 12.3 which caused saved images of windows to remain on screen when switching workspaces
Release Notes
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

this caused a hash missmatch now. See #355643


unknown flag: --deploy-agent when doing addon

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
follow this guide
https://open-cluster-management.io/getting-started/integration/app-lifecycle/
and when executing
kubectl -n open-cluster-management get deploy multicluster-operators-subscription --context ${CTX_HUB_CLUSTER}
found the following error and use source install succeed
not sure why the --server-agent is appended and whether anything wrong ...
# clusteradm version
client          version :v0.1.1
server release  version :v1.20.2

multicluster-operators-subscription-75f858fb79-rl9qq     0/1     CrashLoopBackOff   1          30s
root@jitest19:~# kubectl -n open-cluster-management logs multicluster-operators-subscription-75f858fb79-rl9qq
f
Usage of /usr/local/bin/multicluster-operators-subscription:
      --add_dir_header                   If true, adds the file directory to the header of the log messages
      --agent-image string               Image of the agent to be deployed on managed cluster. (default "quay.io/open-cluster-management/multicloud-operators-subscription:latest")
      --agent-install-all                Configure the install strategy of agent on managed clusters. Enabling this will automatically install agent on all managed cluster.
      --alsologtostderr                  log to standard error as well as files
      --cluster-name string              Name of this endpoint.
      --debug                            if debug is true, hub github webhook listener will be disabled
      --disable-tls                      Disable TLS on WebHook event listener.
      --hub-cluster-configfile string    Configuration file pathname to hub kubernetes cluster
unknown flag: --deploy-agent
      --kubeconfig string                The kube config that points to a external api server.

Expected behavior
A clear and concise description of what you expected to happen.
Environment ie: Kubernetes version:
Additional context
Add any other context about the problem here.

https://github.com/open-cluster-management-io/clusteradm/pull/146
We have it fixed already but we are waiting for a clusteradm release.
In the mean time, you can get around the error by modifying the deployment and remove the --deploy-agent flag.

thanks, then I will close this , appreciate the info


as anyone I can list all contributions in OPEN status

closes #37

I immediately thought of cairo-streams when I read the Issue. Very nice usecase, good job! lgtm 💪

Too bad we cannot have a parameter in the filter() function, the next one would be easy as well (list issues assigned to a given contributor)


fix tools/feature_supports_matrix

extracted from https://github.com/ManageIQ/manageiq/pull/23140
Overview
I don't know the true purpose of this code.
But it was referencing QUERYABLE_FEATURES
which went away 5dae6457fc (in 3/15/2021)
I went in here because I was adding supports feature mixin to all models and grep showed me this class
Before
threw exception
After
Lists every model
Alternate
Delete this tool.
I know it was used by Martin H, not sure who else would need it.

I believe the original purpose of this was so that we could generate the capabilities matrix in the docs.  That being said, it might be worth just deleting or, if it's actually used for the docs, move it over to the docs repo?  @agrare Thoughts?

I don't know if it was intended to build the docs I think it was just informational.  The capabilities_matrix in the docs came years after this tool was introduced.

I'd be ok with dropping this tool, though I have to admit it's pretty cool we can create an at-a-glance table.

Merging anyway, since the bug is fixed.  If we want to delete we can in a follow up PR, and @kbrock it's really up to you.  Since you're in this space, maybe this tool it helpful for you.


When content contains \n, the editor can't render anything

This might be fixed in https://github.com/10play/10tap-editor/pull/118
I'll try get it in this week

Should be fixed in 0.5.1
If you want a new line you should use <br />

Thank you


fix users list appears again if there are multiple users with same name and one of them uses the first name as username

This is an extended fix to @bijanebrahimi's fork.
This commit fixes issue #34.

If @jakiestfu could merge the code would be great.

@marcosrocha85
This pull request is already 2yrs old, but the Author @jakiestfu seems not responding to anything despite the fact that he is still actively using github.

@WisdomSky I having issues with At.js because of dot in usernames, like marcos.rocha, so I decided to move to another plugin. If Mention can handle that I will use your fork, but no worry, I will test and give a feedback. Thanks btw.


lazyload arcpy

It's nice if you can run something like slap -h without waiting for arcpy to load; we can do this by only importing arcpy after we've parsed command line arguments and determined we need to.

Fixed in #40


Getting console messages via MonitoringApi

Hi
Is there a possibility to get messages logged by Hangfire.Console using JobStorage.Current.GetMonitoringApi().JobDetails(jobId) method?

Not at the moment. But could be done by providing a few extension methods for IMonitoringApi.

Thanks for answer!
So there is no "out of box" way to get ConsoleLine objects by jobId?
Only idea I have for the moment is to get ConsoleStorage instance by reflection and try to get some using this class.

Please take a look at v1.2.1, which adds an initial version of IConsoleApi interface (similar to IMonitoringApi) to access console data. It only contains a single method GetLines() for now, but may be extended in the future.
Here's an example:
using (var consoleApi = JobStorage.Current.GetConsoleApi())
{
    var job = JobStorage.Current.GetMonitoringApi().JobDetails(jobId);

    foreach (var state in job.History.Where(s => s.StateName == ProcessingState.StateName))
    {
        var lines = consoleApi.GetLines(jobId, JobHelper.DeserializeDateTime(state.Data["StartedAt"]));
        // do something with lines
    }
}

There's also an optional argument to get only lines of a certain type (LineType.Text or LineType.ProgressBar).

Great! Thanks a lot for new version ;)
I'll try it in a few next days, and post some feedback.

Solution works perfectly ;)
Thanks for your help ;)

@pieceofsummer what is the purpose of JobHelper.DeserializeDateTime(state.Data["StartedAt"])? Why do you need this timestamp to get the lines?

The data is stored with a key based on the start time so that is why is needed to look it up. As a side note it is an Epoc so is millisecond accurate.
The XML comments say it is the time processing was started and then point to create date property.
https://github.com/pieceofsummer/Hangfire.Console/blob/a1fea1356c46fe70b0a9df1785cf5c138aa18ae0/src/Hangfire.Console/Monitoring/IConsoleApi.cs#L16
However, these are not the same; the started processing date is needed. This is only available in the data dictionary on the processing task and is a string.
Finally to make use of the lines you will need to cast them to either TextLineDto or ProgressBarDto  otherwise you only have access to the colour and timestamp.


wasi-parallel: implement CPU parallelism

This change implements wasi-parallel in Wasmtime. It addresses only the CPU parallelism (not GPU), since this would introduce additional complexity to an already difficult review (if you're interested about the GPU progress, talk to @egalli). Each commit implements a separate piece to the puzzle. Though most of the changed lines are not the core implementation (see tests, LICENSE, etc.), due to the size of the PR, it may be convenient to look at this commit by commit.
I see several issues highlighted by this change:

lacking toolchain support: in the absence of any other feasible path, the WebAssembly bytes of the kernel code to be executed in parallel are embedded in the host WebAssembly module itself. This is problematic for many reasons, not least of which is the ability for the host WebAssembly module to tamper with the kernel (on the flip side: an intra-module JIT compiler feature!). But that is not the end of it: toolchain support is also lacking simply to produce modules that use atomics and shared memory. For C programs: https://github.com/WebAssembly/wasi-libc/issues/326. For Rust programs: https://github.com/rust-lang/rust/issues/102157. Due to all this, most tests and examples in this crate are meticulously hand-crafted WAT files — once toolchain support improves these difficult-to-maintain files should be replaced.
uncomfortable Wasmtime/Wiggle APIs: this implementation of wasi-parallel works by creating new instances of the kernel in each thread of a thread pool — the host module exports a shared memory and the kernel imports it. Getting access to the shared memory from within a wasi-parallel invocation is difficult with Wasmtime/Wiggle: the best way I could think to resolve this is to teach Wiggle to skip a WITX function and then manually implement add_to_linker for that function. This is tedious and error-prone, so I would be excited to discuss better solutions. Note that an almost identical version of this problem will arise when I try to implement wasi-threads.

Why now? The rationale behind merging this code behind the wasi-parallel feature flag is to enable users to try this out locally and to iterate on this in-tree (e.g., toolchain, GPU parts).

lacking toolchain support: in the absence of any other feasible path, the WebAssembly bytes of the kernel code to be executed in parallel are embedded in the host WebAssembly module itself. This is problematic for many reasons, not least of which is the ability for the host WebAssembly module to tamper with the kernel (on the flip side: an intra-module JIT compiler feature!). But that is not the end of it: toolchain support is also lacking simply to produce modules that use atomics and shared memory. For C programs: How to distribute THREAD_MODEL=posix builds? WebAssembly/wasi-libc#326. For Rust programs: Fail to compile with WebAssembly atomics and shared memory rust-lang/rust#102157. Due to all this, most tests and examples in this crate are meticulously hand-crafted WAT files — once toolchain support improves these difficult-to-maintain files should be replaced.

I think it is important to note that this PR does not depend on toolchain - implementing an API in consumer should not require a particular producer to be on board; multiple memories are a a different example of this. Also keep in mind that libc support is already implemented, WebAssembly/wasi-libc#326 is strictly about packaging and detecting it.

That's true, yes, and I feel pretty similarly to the wasi-threads proposal as what's here as well. I also don't see a future for the wasi-threads proposal as-is in the component model right now. If these proposals want to integrate with the component model I think work needs to happen in the component model itself rather than assuming the functionality can be implemented with special host functions.

Just out of curiosity, as I haven't been following wasi meetings very closely, what is the background of the requirement that wasi interfaces only use component model?

My understanding of how wasi-parallel is expected to fit into the component model is that it could be packaged as an import of a core-module, with the core-module importing the linear memory and exporting functions that operate on it directly. That isn't supported in wit or wit-bindgen today, so we'll need interim approaches, but eventually we should be able to pull these into the wit framework.

@alexcrichton thank you for the detailed answers.

I'm not sure what you mean though by if this only applies to wasmtime.

I meant to ask whether this is the approach adopted specifically by wasmtime or by component model, but you have answered that.

My understanding of how wasi-parallel is expected to fit into the component model is that it could be packaged as an import of a core-module, with the core-module importing the linear memory and exporting functions that operate on it directly. That isn't supported in wit or wit-bindgen today, so we'll need interim approaches, but eventually we should be able to pull these into the wit framework.

Does component model allow this yet? What can the interim approaches be? Maybe we should take it the component-model repo though.

The component model does support importing a core wasm module, but that's not reflected in the *.wit textual description of APIs at this time. For interim things I suspect that'd be good to bring up on the component model repo yeah.

@alexcrichton, I think there are several points you should reconsider:

In the component model the hack you've implemented here with getting the shared memory to work will not work and has no way of working, there's no escape hatch to implement as you've done here.

Then there's a bug in the component model specification. If there is no way to support multiple WASI proposals (wasi-parallel, wasi-threads), no way to satisfy the recurring requests for access to the host (latest of which is https://github.com/bytecodealliance/wit-bindgen/issues/313), and no way to then make use of the core WebAssembly threads proposal, then the WIT specification is not complete enough. If there isn't an escape hatch, then one should be built.

Additionally the existing wasi-common, wasi-nn, and wasi-crypto will likely all need to get removed or refactored or something with a transition away from wiggle

Sure, but that should not block progress. The issue you point out is not with those WASI contributions but rather with the unstable state of wit-bindgen. I tried to refactor wasi-nn to use WIT a month or two ago and the tools were simply not ready. If the spec is not ready and the tools are not ready, it seems unreasonable to block all other work on that. I think the more productive approach would be to allow WASI proposals to continue and ensure that both the WIT spec and tooling can do everything wiggle could.

I am not sure how useful it's been to have wasi-nn and wasi-crypto in this repository, I am not aware of how much usage they're getting and feedback they're providing to the upstream proposals.

You should talk to @geekbeast, also a Fastly employee. He was attempting to use wasi-nn in Viceroy and was able to do so based on the reference implementation here.

I also don't see a future for the wasi-threads proposal as-is in the component model right now. If these proposals want to integrate with the component model I think work needs to happen in the component model itself rather than assuming the functionality can be implemented with special host functions.

I am not involved in component model design... but you are 😁. This PR — along with https://github.com/WebAssembly/wasi-parallel/issues/4, https://github.com/bytecodealliance/wit-bindgen/issues/130, https://github.com/bytecodealliance/wit-bindgen/issues/313 — should be great feedback for a component model issue that you would know how to craft but I would not. I don't understand all of the component model requirements, history, etc. so my ability to contribute there is limited. Why not just take this PR as feedback to the component model and propose a solution so that these WASI proposals can have a future?

Agreed with @sunfishcode above that the right way to support various kinds of shared-memory multi-threading in WASI in a component-model world is by expressing the imported threading functionality as core module imports, which are then instantiated by the component to explicitly share core memory with the other core modules inside the component.  When core wasm eventually gets a fork instruction for creating threads purely from within core wasm, then these core module imports would even be virtualizable from core wasm, satisfying WASI's general goal to specify virtualizable interfaces.
This does raise the question of how to write the interface of a core module in Wit, since there currently isn't any syntactic support for this.  But it's easy to imagine a Wit-like syntax for core:moduletype with a new top-level module keyword that can then be mentioned via import in a world.  IIUC, this same Wit syntax would be independently useful for reflecting the types of components that import shared-everything dynamically-linked libraries, like a libc or libspidermonkey, using CanonicalABI.md saying how things get linked up by default, which is something @sunfishcode have also been talking about recently.  So it's good to have 2 rather-symmetric motivating use cases for this new Wit functionality.
I don't have enough understanding (particularly around Wiggle) to say what we should do in the right-now term before the above Wit functionality gets implemented.  But switching out Wiggle for Wit does seem like it should be our goal state.

Thanks @sunfishcode and @lukewagner for pointing out that the shared memory import/export will be expressible with WIT core modules. I still have questions on this (When and how can the core module syntax be implemented? Is help needed there and, if so, what? Should that block this PR?) but it seems like long term there should not be a problem with wasi-parallel moving to WIT related to shared memory import/export.
But, to take up what I think is @alexcrichton's point, the issue is actually about host access. Proposals like this one and wasi-threads and other use cases (like the ones linked above) want to touch the host itself, e.g., Wasmtime APIs like Caller, Module, Store, Engine. In wasi-threads, e.g., wasi_thread_spawn will need access to the Module of the parent thread in order to create new instances of it and access to the SharedMemory in the Store to import that in each new child thread. I think what @alexcrichton is saying is that the WIT syntax and tooling do not support this and never will (?), which I was asking him to reconsider.
I actually see several possibilities along a continuum:
a. dead in the water: the WIT syntax never has knowledge of "host access," the tools don't support it, and this class of proposal is dead in the water (from @alexcrichton's argument that engines like Wasmtime will not want to support non-WIT-able proposals)
b. hack: the WIT syntax still has no knowledge of "host access," the tools still don't support it, but this class of proposal can be implemented as a core module with some hacking on the linker (as done in this PR) to get access to the host (I don't understand how this would look in WIT!)
c. escape hatch: the WIT syntax still has no knowledge of "host access," the tools do support it as an escape hatch (e.g., an additional parameter to the WIT macros like host_access_functions: ["..."]), and this class of proposal can be more cleanly implemented
d. full support: the WIT syntax does gain knowledge of "host access" (perhaps with a new host keyword in the signature of functions, e.g., wasi_foo(host, x: i32, y: i32)), then the tools would support it, etc.
I've implemented "b" and I believe @alexcrichton was suggesting "a." I would prefer "c." @lukewagner, @sunfishcode: I can't tell from your comments but I think you are not saying "a"?

(Also, please prefix every sentence above with "IIUC" since I am not too familiar with the details of the component model and I'm reading into some of @alexcrichton's comments).

To some degree I'm trying really hard here to be an impartial messenger, along the lines of "please don't shoot the messenger". I'm pretty intimately aware of all the constraints here ranging from the threads proposal to the desire for WASI threads to the component model itself, but I am trying to not take on the design myself and leave that to others. To that end I don't have great responses to comments such as:

If there isn't an escape hatch, then one should be built.

I'm pointing out the systems that are in play and their various designs, and all of these systems have intentional design behind them. I am not personally going to go to the component model repository and open a "please add a small escape hatch" issue because I'm aware of why one hasn't been added yet. Furthermore I'm not going to go to the wasi-parallel proposal and open an issue saying "you will never work with the component model" because I'm aware of the strong motivations for such a proposal. I'm trying to ideally state impartial facts here without trying to solve everything myself.
That being said I naturally have a lot of opinions about how this should all work. If a parallelism story for wasm requires an "escape hatch" that seems like a fundamental problem because parallelism is such a foundational part of any runtime environment that if it's not designed for from day one it seems to me like it has little chance of gaining any amount of larger traction. For example just because Wasmtime is capable of implementing escape hatches like this in no way means other runtimes can do so. The Caller context is not available on the web, for example, meaning that if the wasi-parallel proposal crucially relies on such an escape hatch then that would be unimplementable or otherwise problematic for the web.
Personally I even have hesitation about the whole kernel model in wasi-parallel. I don't understand how anyone could productively write a kernel that literally only imports shared memory and nothing else. Virtually no language can target a world where nothing is imported and if nothing is imported then a language can't even do something like print a line to standard out. While it's theoretically possible and technically possible to get something working here I feel that this is a far cry from a productive developer experience.
Again though I'm trying hard to not insert myself into the design here. These are some thoughts of mine but honestly I don't consider them relevant since I'm not the one championing or being involved in the proposal. At the same time though I feel like someone has to point out the actual limitations of the systems that are being built at some point as otherwise it feels like things are carried on as everyone's just blindly merging everyone else's work without thought to the consequences.

Why not just take this PR as feedback to the component model and propose a solution so that these WASI proposals can have a future?

My hope is that design issues like running wasi-parallel in the component model would have come up much earlier in the design process long before a PR came to Wasmtime here. This isn't really the right place to discuss deep design details of the proposal or modifications to the component model. Otherwise though I don't really think it's a productive developer process to create a proof of concept and then turn around to the component model and say "you figure out how to get this working it can't regress". The design here should be a collaborative process of all parties involved which doesn't just shove all the work onto one person to figure out everything.

Sorry for the long post, I have a lot of thoughts on this but now's probably not the time or the place. I'll reiterate that I personally would like to see threads-on-wasm via a standard like WASI. I don't think that the component model and WASI are in a state where things can be plucked off the shelf, mashed together, and a workable standard created. I feel there are deeper designs that need to be implemented and accounted for before moving forward with this iteration of wasi-parallel. For example the import-a-core-wasm-module idea seems plausible, but AFAIK the wasi-parallel proposal has not been designed with this in mind, runtime support in Wasmtime has not been implemented with this in mind, and support in developer tooling (a la *.wit) has not been designed with this in mind. That's a lot of hypothetical work to leave on the table while asking to merge this as an interim solution.

I'd be curious to see how others feel about this though. I am not sure how useful it's been to have wasi-nn and wasi-crypto in this repository, I am not aware of how much usage they're getting and feedback they're providing to the upstream proposals. In that sense I'm not sure how beneficial it will be to land this here vs have it externally.

Is there a good mechanism to track usage of various features in Wasmtime? Usually I have no knowledge of certain features being used by a customer until they contact us with some issues, or they present/publish their usage somewhere. And these are probably a subset of total user base. For example, I know a big device manufacturer is in the process of embedding wasi-nn into their products. For wasi-crypto, it is an essential function for a Wasm runtime to be selected for a SASE project we are involved.
Here are a few links of wasi-nn usages from googling (not all on Wasmtime):
https://deislabs.io/posts/wasi-nn-onnx/
https://www.secondstate.io/articles/openvino-inferencing-using-wasmedge-wasi-nn/
https://wasmedge.org/book/en/write_wasm/rust/wasinn.html
https://cfp.cloud-native.rejekts.io/cloud-native-rejekts-eu-valencia-2022/talk/NXLTRQ/
https://www.youtube.com/watch?v=48ORmla7mak&list=PLj6h78yzYM2Ni0u-ONljTkv4uOutyjwq9&index=11

Just as an update: I realized my comment above was forgetting that we're talking about the core-instance-per-thread approach to multi-threading, for which the core module import approach I suggested above would need the imported core module to itself have a module import (the module to be instantiated for each new thread), which is not yet possible in core wasm.  So that might not be the right approach here.
It does really seem like the best approach to exposing multi-threading to core wasm in a standard way is as proposed by Watt & Rossberg, as a core wasm feature.  Although what we do to allow short-term experimentation and prototyping in Wasmtime is a different question.

For example just because Wasmtime is capable of implementing escape hatches like this in no way means other runtimes can do so. The Caller context is not available on the web, for example, meaning that if the wasi-parallel proposal crucially relies on such an escape hatch then that would be unimplementable or otherwise problematic for the web.

Web engines support this via a JavaScript callback which creates another copy of the instance. Web polyfill was part of the motivation for wasi-threads proposal, the other part being the fact that at least one more standalone engine supports it as well. Core spec supports multiple instance executing in parallel, this isn't an unknown either.

@mingqiusun for sure yeah, I don't mean to say definitively whether or not those proposal should be removed from the repository, and it's great to hear that wasi-nn is getting such usage!
@penzn indeed you are correct. There are a huge number of caveats with the support on the web for threading, however, including:

The threads proposal is not offically merged into the wasm spec yet
I'm not aware of any general purpose "spawn a thread library" due to the instance-per-thread model which requires somehow getting a module's instance imports onto web workers. All solutions I know of are very runtime-specific or specific to the shape of module and what it imports. Everything I'm aware of at least is very far from "just call pthread_create and it all works"
The web's abstraction layer is not components, which I believe wasi-{parallel,threads} want to target at this time. The component abstraction layer, at this time, does not allow for precisely the same capabilities of what core wasm allows.

One point that's probably worth clarifying is that it's not like I think Wasmtime shouldn't be able to implement a web-style threading scenario. In fact you can, already today, implement "threading" with Wasmtime since you as the host can spawn threads and figure out how to instantiate modules on new threads and share host state. There's nothing inherent to Wasmtime preventing this with Wasmtime's core wasm embedding API.

I talked with @abrown at great length yesterday about this and we're going to try to allocate time next week to discuss what it might look like for wasi-parallel to get integrated into the component model more completely. In the meantime my personal feelings on this PR is that I would like to make sure that this iteration of wasi-parallel and wasi-threads can be implemented with Wasmtime's embedding API, but otherwise this implementation I think might be best to reside in a different repository than the main Wasmtime repository.

@abrown, @lukewagner, @sunfishcode, and I talked last week at great length about this PR and the various issues here, zooming out mostly to threads in the component model. We concluded a number of things that I hope to summarize here, and please correct me y'all if I miss anything.
Our general topic was what to do with threads, the component model, and having the two work in harmony with each other. To that end we broke down a possible plan of action in to three "stages" ordered from shorest-time-to-implement to longest-time-to-impelment.
Stage 1
The first thing for Wasmtime and threads is to get anything working all. The goal here would effectively, through some wasm-defined API using Wasmtime's embedding API, achieving the functionality proposed by wasi-{threads,parallel} today (I'm going to say only wasi-threads from now on but I mean both). There's a lot of things even here that need implementing/deciding such as:

How are existing WASI hostcalls synchronized?
How is a Store<T> created on the new thread?
How does the new thread acquire a Module to instantiate?
How does the new thread acquire a Linker<T> to create a new instance on this thread?

This is all just the bare bones of getting anything working, but should hopefully provide valuable implementation feedback to Wasmtime, the component model, and the wasi-threads proposal. All of this is likely design work that needs to be done anyway for any eventual formal wasi-threads proposal is the hope.
This local maximum, however, is not immediately compatible with the component model. Some of the fundamental abstractions here, such as importing a shared linear memory, do not work the same way in the component model (components can't import a linear memory). This means that this stage does not have access to host APIs defined by the component model, such as a hypothetical wasi-snapshot-preview2.
Stage 2
This stage is intended to be sequenced after the prior stage with the goal of hooking up component-model-level host definitions into a core wasm module that wants to use threads. As a base assumption the idea here is that the user-supplied "thing" is still a core wasm module. The high-level idea is that host-defined component-model definitions are "automatically lowered" into the corresponding core wasm functions to get imported into a core wasm module.
This sort of means that a wasmtime::Linker could be blended with a wasmtime::component::Linker to instantiate a wasmtime::Module. The thinking is that the work from stage 1, probably mostly present in a wasmtime::Linker, could pull in definitions from a wasmtime::component::Linker to get access to all host APIs while still retaining access to the abstraction layer of threads.
The main body of work in this stage would be to figure out how to robustly take a component-model level API and translate it into a core wasm API. This is roughly already done with the canonical ABI but I personally believe there's a lot of fiddly bits to work out such as how to reuse the lifting/lowering in Wasmtime without duplicating it. For example somehow a VMContext and a VMComponentContext need to be bridged between one another or ... something like that.
In discussing this though we wondered but weren't sure of how important such feature would be for migrating from wasi preview1 to preview2. There will, for a long time, be binaries that import wasi_snapshot_preview1 and there's a number of strategies to load that into a preview2-supporting runtime, but something it might be quite useful to allow merging of newly-defined and implemented APIs into older modules.
The downside of this stage is that there still isn't full integration with the component model. Everything is still working at the level of abstraction of a core wasm module and inherently continues to be incompatible with components. The main distinction from stage 1 is that component-model-level functionality, which is the presumed path that WASI is going, can still be imported into the module.
Stage 3
The next stage of evolution for support here would be to integrate threads into the component model proposal itself. This is a far-in-the-future vision though where there's a difference between a func and a shared func for example. there's a ton of design questions here that (at least to me) don't have obvious answers. There's quite a lot of design work here not only for core wasm but also the component model. This is the hypothetical end goal of components that support threads, but I'm mostly listing this here for completeness as the timescale for implementing this in in the many-numbers-of-years.

Given the possible staging here I concluded that my original stance of "we probably don't want to merge this" is no longer true. The staging here would explicitly continue to work at the core wasm layer and not the component layer. Furthermore for the time  being there would be no path forward to implement this in the component layer but there would be a path to taking some of the major benefits of the component layer and bringing them to core wasm. I'd revise my previous stance, then, to it's ok to merge this into the main Wasmtime repository along the lines of all other WASI proposals.
That's not to discount the technical review here, however. This is an example of a fundamental issue in wiggle which will need to get ironed out before landing. Furthermore I personally at least feel that wasi-parallel may not be a best fit for the first proposal implemented here (the kernel feels a bit odd, it's JIT compiled so doesn't work with AOT compilation, the kernel itself can't import anything useful, unclear how toolchains will produce this kernel from "real" source code, etc). @abrown what would you think of implementing wasi-threads first? That feels like a more low-level proposal which would, for now at least, subsume the functionality implemented here for wasi-parallel I think?

Thanks @alexcrichton for the detailed write-up; I think it's a pretty accurate reflection of what we discussed and a decent high-level vision for how to move forward. I agree that wasi-threads is a more foundational proposal and, actually, I think it highlights some of the issues to overcome to get to stage 1 better than wasi-parallel does. wasi-parallel side steps a few things with its "bag of bytes" paradigm. I'll see if there's a way to get wasi-threads done first.
The current issues to get to stage 1 that @alexcrichton describes above are many and span multiple repositories (this one, wasi-libc, LLVM, Rust, etc.). For anyone else interested in progress on getting wasi-threads to the stage 1, this Zulip channel has the latest discussion. @haraldh has taken some of the pieces I and others have been working and has a working PoC! I will post my take on the remaining items to upstream, probably in an issue on the wasi-threads spec repository.

I'm going to close this as this is a pretty old PR at this point and efforts have shifted significantly in the intervening two years. Nowadays most things are happening around "shared-everything-threads" which is the most likely path forward I think. If others disagree though I'm happy to reopen.


Fix panic on DatabaseInstance failing to create

Description of your changes
In some cases when a DatabaseInstance fails to create, such as when it cannot create a subnetwork if there's no IP space available, a panic would occur as there's no certificate info in the certSlice. Fixed this by just checking if the certSlice is empty.
Fixes #343
I have:

[X] Run make reviewable test to ensure this PR is ready for review. (just ran make test as the rest fails)

How has this code been tested
Tested the following scenarios:

Successful creation of DatabaseInstance: confirmed certificate details are still added to status.atProvider.serverCaCert
Failed creation does not cause panic anymore and does not add certificate details to status.atProvider.serverCaCert. Status condition is still added with the error from GCP.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

/test-examples="examples/sql/instance.yaml"


Add ErrorSuppressionFixer

#3035
Initial commit - fixer and tests are based on SilencedDeprecationErrorFixer thus @HeahDude is added as author.
Option allows only boolean value meaning to do what is should or not.
Having also null to ignore option would make false doing the opposite to true:

for silence_deprecation_error it would sense - to remove @ from deprecation notices
but for noise_remaining_usages what would that mean? Add @ everywhere?

Feel free to propose better naming for options, descriptions and interesting test cases.
My doubt is line 106 in ErrorSuppressionFixer.php - I thought it should be something like:
$indexes[$prev] = (true === $this->configuration['noise_remaining_usages']) ? false : null;

but that failed some tests. Anyway it's just a hint where to look for flaws.

Is there a specific reason not to implement some exclude option as proposed in #3035? IMO it's a must have, as some @ usages are legit, e.g.:
// failure is handled so one doesn't care about the E_WARNING
if (false === $file = @fopen(...)) {
    // throw some exception
}

ping @kubawerlos
what is the state of this PR ?

Thank you @kubawerlos.


examples/jit :: Error: TorchError { c_error: "can't fopen" }

Hi! Awesome package you've got here. I'm running into some trouble trying to run
cargo run --example jit model.pt image.jpg
cargo run --example pretrained-models -- resnet18.ot tiger.jpg
Any thoughts on how I can resolve? Thanks!

I'm on mobile so cannot check but the fopen in the error message may mean that there is no model.pt or resnet18.ot file in the current directory ?

aha! I was indeed missing the .jpg file. I should go to sleep now, thanks a bunch!!


Bump version to 0.11.0

Version bump is pretty trivial, should I push them directly into master?

Codecov Report
Patch coverage has no change and project coverage change: -0.78% :warning:

Comparison is base (1cbaf3c) 94.81% compared to head (27dc342) 94.04%.
Report is 1 commits behind head on master.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##           master      #55      +/-   ##
==========================================
- Coverage   94.81%   94.04%   -0.78%
==========================================
  Files          10       10
  Lines         772      772
==========================================
- Hits          732      726       -6
- Misses         40       46       +6

see 4 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


[fargate] Support FargateService in MonitoringFacade.monitorFargateService

Feature scope
MonitoringFacade.monitorFargateService
Describe your suggested feature
Currently monitoring using MonitoringFacade.monitorFargateService API only supported on the following Fargate constructs: NetworkLoadBalancedFargateService, ApplicationLoadBalancedFargateService.
Can you please add support to FargateService type as well.
https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ecs.FargateService.html
Thank you.

+1 on request. One work-around available today seems to be:
import * as monitoring from "cdk-monitoring-constructs";

// monitor = new monitoring.MonitoringFacade(...);
// fargateService = cdk.FargateService instance

monitor.addSegment(
  new monitoring.FargateServiceMonitoring(monitor, { fargateService })
);

In other words, FargateServiceMonitoring supports a raw FargateService instance while the method on MonitoringFacade does not. Thus, I think the fix here is to swap FargateServiceMonitoringProps for CustomFargateServiceMonitoringProps.

Doesn't facade.monitorSimpleFargateService({ fargateService }) work?
https://github.com/cdklabs/cdk-monitoring-constructs/blob/edfc4e4bcb68f7f71df18fc2acab84f130ce3988/lib/facade/MonitoringFacade.ts#L456-L463

facade.monitorSimpleFargateService({ fargateService }) should indeed fulfill this need, but let me know if I'm missing something.


"Link Sticker" of Story

Hi,
Firstly thanks for who contributed this amazing project.
I am trying to do a instagram bot which automaticall yshares stories with LINK sticker.
https://adw0rd.github.io/instagrapi/usage-guide/story.html in here, StoryLink works well, it adds link to story but only problem is there is no "link sticker".
this is what i mean:

https://github.com/adw0rd/instagrapi/issues/644 here it explains how it stickers works and says check it with charles.
then i tried to do what this article says: https://adw0rd.com/2020/3/26/sniffing-instagram-charles-proxy/en/ and finally got something in charles.

in the end of article, author says if headers-accept contains "zstd" you can decompress then but i couldn't do it :/
for adding link with stickler to my story what should i do or where do i look.

@sametcopur knew how to add Link Sticker ?

@idrizdi https://adw0rd.github.io/instagrapi/usage-guide/story.html

@adw0rd it's add they link but just the link, there is no stickler. that is the problem.

Thats an Instagram App StoryEditor Feature and not the api feature.

Good day, everyone!
Has anyone got more information on this issue?
I'm trying to make stories with link stickers as well and am facing the same issues (lack of documentation/examples of URL sticker use).
Any help will be immensely appreciated!

https://adw0rd.github.io/instagrapi/usage-guide/story.html

https://adw0rd.github.io/instagrapi/usage-guide/story.html

Hi adw0rd, I really appreciate your work, but this issue can't be clarified with a reference to the documentation. Alone the hint that link stickers need x and y coordinates and size information like hashtags is missing at all. Moreover, this results in the sticker remaining invisible (my workaround is to make it huge (e.g. width and height = 5)) so that it doesn't matter where the user taps. But answering the request with a link to the documentation is really not helpful.

@juppcamper Thank you very much for the comment, send your improvements to the documentation please, where you share your experience with StorySticker

hi @adw0rd , can you help me understand how adding link feature works, currently both the links i.e charles and usage-guide are not opening.

hi @adw0rd , can you help me understand how adding link feature works, currently both the links i.e charles and usage-guide are not opening.

Hello did you find a way? Anybody?


Wires disappear after duplicating or connecting to the server

At first I thought it was a bug in Advanced Duplicator 2, as wires it pasted were invisible.
But it also happened with Advanced Duplicator 1.
Then I tried reconnecting to the server, and all wires on all existing contraptions on the map were gone too.
This happens on both my cleanly installed dedicated server, and cleanly installed game, with these workshop addons: https://steamcommunity.com/sharedfiles/filedetails/?id=136821175
I have tried in singleplayer with no addons except Wiremod. The issue persists.
When dupping with adv. dup 2, only the last wire is restored. And if you do a wire with say 20 waypoints, then redo the wire with 3 waypoints; in the dup those 20 waypoints will still be there, and the 3 first were just changed. (Happens whether you delete the existing wire first or not).
Behavior is identical with adv. dup 1, so I'm pretty sure it's a bug with the wire itself.
https://youtu.be/8AyUEMzLx6c

Looks like the problem is in the networking. I can check it out.

https://github.com/wiremod/wire/pull/1688

Thanks. I'm using the workshop, so I just took your changes and dumped the entire thing into my server's lua folder. This fixes dupes, but not connects. Don't know if that's a different bug, or just due to how I'm just dumping the script in my lua. That's good enough until this gets approved, I guess.

https://github.com/wiremod/wire/pull/1691


Missing RazotLight

Hi,
I'm using:

#addin nuget:?package=Cake.Graph&version=0.8.0&loaddependencies=true

and getting an error

Error: Unable to find a version of 'RazorLight' that is compatible with 'Cake.Graph 0.8.0 constraint: RazorLight (>= 2.0.0-beta1)'.

Do you have some tip for me?
Thank you in advice

Because RazorLight is only available in a beta release, you will also need to add &prerelease (could be &prerelease=true, can't remember) for the dependency to correctly download.

The following works for me, if you're still having issues:
#addin nuget:?package=RazorLight&version=2.0.0-beta1&prerelease&loaddependencies=true

Thanks,
I'm trying with

#addin nuget:?package=RazorLight&version=2.0.0-beta1&prerelease&loaddependencies=true

The error message changes...

Could not load C:\cake\Addins\System.Reflection.Metadata.1.4.2\lib\portable-net45+win8\System.Reflection.Metadata.dll (missing System.Collections.Immutable,
Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
Could not load C:\cake\Addins\Microsoft.CodeAnalysis.Common.2.3.1\lib\netstandard1.3\Microsoft.CodeAnalysis.dll (missing System.Collections.Immutable, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)

We're going to exit release candidate status soon.
In mean time, would like to update the README to include this note:
and update to RazorLight 2.0.0-rc.3
Thanks for using the project!  Was thinking of moving to Cake for my build scripts.