bug loading vizgen data

Description
Trouble using vizgen data with sq.read.vizgen(). Function is expected 8 columns in metadata file mine has 9
Minimal reproducible example
adata = sq.read.vizgen(
    path=data_path,
    counts_file=os.path.join(data_path,section,file_path,cbg_file),
    meta_file=os.path.join(data_path,section,file_path,meta_file),
    transformation_file=os.path.join(data_path,section,'region_0/images/micron_to_mosaic_pixel_transform.csv'),
)

Traceback
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [54], in <cell line: 1>()
----> 1 adata = sq.read.vizgen(
      2     path=data_path,
      3     counts_file=os.path.join(data_path,section,file_path,cbg_file),
      4     meta_file=os.path.join(data_path,section,file_path,meta_file),
      5     transformation_file=os.path.join(data_path,section,'region_0/images/micron_to_mosaic_pixel_transform.csv'),
      6 )

File /allen/programs/celltypes/workgroups/rnaseqanalysis/mFISH/michaelkunst/miniconda3/envs/scvi-tool/lib/python3.9/site-packages/squidpy/read/_read.py:146, in vizgen(path, counts_file, meta_file, transformation_file, library_id, **kwargs)
    144 # fmt: off
    145 coords = pd.read_csv(path / meta_file, header=0, index_col=0)
--> 146 coords.columns = ["fov", "volume", "center_x", "center_y", "min_x", "max_x", "min_y", "max_y"]
    147 # fmt: on
    149 adata.obs = pd.merge(adata.obs, coords, how="left", left_index=True, right_index=True)

File /allen/programs/celltypes/workgroups/rnaseqanalysis/mFISH/michaelkunst/miniconda3/envs/scvi-tool/lib/python3.9/site-packages/pandas/core/generic.py:5588, in NDFrame.__setattr__(self, name, value)
   5586 try:
   5587     object.__getattribute__(self, name)
-> 5588     return object.__setattr__(self, name, value)
   5589 except AttributeError:
   5590     pass

File /allen/programs/celltypes/workgroups/rnaseqanalysis/mFISH/michaelkunst/miniconda3/envs/scvi-tool/lib/python3.9/site-packages/pandas/_libs/properties.pyx:70, in pandas._libs.properties.AxisProperty.__set__()

File /allen/programs/celltypes/workgroups/rnaseqanalysis/mFISH/michaelkunst/miniconda3/envs/scvi-tool/lib/python3.9/site-packages/pandas/core/generic.py:769, in NDFrame._set_axis(self, axis, labels)
    767 def _set_axis(self, axis: int, labels: Index) -> None:
    768     labels = ensure_index(labels)
--> 769     self._mgr.set_axis(axis, labels)
    770     self._clear_item_cache()

File /allen/programs/celltypes/workgroups/rnaseqanalysis/mFISH/michaelkunst/miniconda3/envs/scvi-tool/lib/python3.9/site-packages/pandas/core/internals/managers.py:214, in BaseBlockManager.set_axis(self, axis, new_labels)
    212 def set_axis(self, axis: int, new_labels: Index) -> None:
    213     # Caller is responsible for ensuring we have an Index object.
--> 214     self._validate_set_axis(axis, new_labels)
    215     self.axes[axis] = new_labels

File /allen/programs/celltypes/workgroups/rnaseqanalysis/mFISH/michaelkunst/miniconda3/envs/scvi-tool/lib/python3.9/site-packages/pandas/core/internals/base.py:69, in DataManager._validate_set_axis(self, axis, new_labels)
     66     pass
     68 elif new_len != old_len:
---> 69     raise ValueError(
     70         f"Length mismatch: Expected axis has {old_len} elements, new "
     71         f"values have {new_len} elements"
     72     )

ValueError: Length mismatch: Expected axis has 9 elements, new values have 8 elements

Version
'1.2.2'
...

I'm experiencing something similar except it's 16 elements rather than 9?
ValueError: Length mismatch: Expected axis has 16 elements, new values have 8 elements

Hi @mkunst23 and @andrewjkwok , this should've been fixed in #648, installing squidpy from main should fix this.

@michalk8 Thanks for pointing us in the right direction - it works correctly now!

Sorry it seems it isn't entirely working yet. When I read the data in, my obs dataframe gets all NaNs, but if I check my cell_metadata.csv file, it's looks populated with the various cell coordinates etc. to me. The result is that I have no spatial coordinates to plot.
A second quick thing is that previous merscope outputs gave the cell coordinates in a set of hdf5 files, but that has since the merscope software update to v232 and onwards becomes a single parquet file instead - does squidpy need this info at all? I can't seem to find anywhere in the squidpy documentation that uses this file and am wondering if that would help the issue of the lack of spatial coordinates.

hey @andrewjkwok ,
thanks for reporting this, I'm afraid it's a bit tricky to help out without the data available. Could you share the data download so we can test it out? thanks!
Best,
Giovanni

@giovp Yes very happy to. Is there an email I could share a google drive link to? Many thanks in advance.

Sorry just a quick follow up @giovp @michalk8 was wondering if there was somewhere to share my data with your team to take a look?

@andrewjkwok any chance you could point us to some public data? for example, some data shared by vizgen?

@giovp hmm the problem is that the cell metadata file from my MERSCOPE output (running their latest v232 software)  doesn't look the same as the ones that are on vizgen's website.
So if I go to the squidpy website and follow the tutorial (https://squidpy.readthedocs.io/en/stable/external_tutorials/tutorial_vizgen.html) for the data download (https://info.vizgen.com/mouse-brain-map?submissionGuid=a66ccb7f-87cf-4c55-83b9-5a2b6c0c12b9), the cell_metadata.csv file doesn't look the same.
I've attached a truncated version of my cell metadata file - as you can see, the columns aren't the same as the one on the vizgen website?
Vizgen website data:
datasets_mouse_brain_map_BrainReceptorShowcase_Slice1_Replicate1_cell_metadata_S1R1.csv
Output from my merscope:
cell_metadata_truncated.csv

hi @andrewjkwok I am unfortunately unable to look into this in the next two weeks, thanks for sharing the data, I'll get back to you soon

Hi - just wanted to quickly check whether there was any progress with this?

There was an issue with indexing but installing squidpy from main should fix the metadata not populating.
The spatial coordinates are being populated by the center_x and center_y from the metadata. The sq.read.vizgen function doesn't use the cell segmentation output, either the older hdf5 or the newer parquet formats.

hi all, I would suggest to take a look at the https://github.com/scverse/spatialdata-io package to read in spatial omics data. We won't be maintaining the IO reading functions here but update specifications from the commercial platforms only in spatialdata-io


Fix of missing semicolons.

Hello, I found 2 missing semicolons and they were breaking the build.

@ahmadalfy can you merge this?


Update README.md screenshots

Update README.md screenshots to include all the new showcases.
I'll take care of this soon.

Hi Patrick! I just submitted a PR for this issue. Please let me know if I did anything incorrectly, as this is my first PR!

Hi Patrick! I just submitted a PR for this issue. Please let me know if I did anything incorrectly, as this is my first PR!


fix: remove swagger2 legacy rules

PR summary
This commit removes the remaining
swagger2-specific legacy rules from the project.
This includes:

invalid_type_format_pair
no_consumes_for_put_or_post
get_op_has_consumes
no_produces

PR Checklist
General checklist
Please make sure that your PR fulfills the following requirements:

[x] The commit message follows the Angular Commit Message Guidelines.
[ ] Tests for the changes have been added (for bug fixes / features)
[x] Docs have been added / updated (for bug fixes / features)
[ ] Dependencies have been updated as needed
[ ] .secrets.baseline updated as needed?

Checklist for adding a new validation rule:

[ ] Added new validation rule definition (packages/ruleset/src/rules/*.js, index.js)
[ ] If necessary, added new validation rule implementation (packages/ruleset/src/functions/*.js, updated index.js)
[ ] Added new rule to default configuration (packages/ruleset/src/ibm-oas.js)
[ ] Added tests for new rule (packages/ruleset/test/*.test.js)
[ ] Added docs for new rule (docs/ibm-cloud-rules.md)

Checklist for removing an old validation rule:

[x] Removed old rule implementation (packages/validator/src/plugins/validation/*.js)
[x] Removed or adjusted testcases (packages/validator/test)
[x] Updated default configuration to deprecate old rule (packages/validator/src/.defaultsForValidator.js)
[x] Removed docs of old rule (docs/ibm-legacy-rules.md)

:tada: This PR is included in version 0.72.2 :tada:
The release is available on npm package (@latest dist-tag)
Your semantic-release bot :package::rocket:


Implement series recipes for interpolations

With this PR, you should be able to do:
@series
    seriestype := :loess # any *_interp or *_spline also
    x := xs
    y := ys
end

and have it "just work", generating a smoothed series.

Test failures on MacOS seem unrelated.

Still WIP - need to fix the quadratic interpolation, and individually test.

Looks like it's just floating point issues. Would you mind submitting a PR to relax the tolerances a bit before we tag?


Tour steps via data attributes

Is there a way to add steps via data attributes so that I can set up the steps in my templates and not have to programmatically create a new tour on every page?  I'd like to set up one global tour that just loads the steps based on if there are any data-steps in the current page.

+1

+1


Add broadcast functionality to FFN component

The Fully Connected NN will operate only on the last dimension of the input tensor:
[dim 0, dim 1, ..., dim n-2, input dim] -> [dim 0, dim 1, ..., dim n-2, output dim]
The total number of dimensions has to be explicitly set in the config of the FFN, where 2 is the default value for the case [Batch size, input dim].

this is great functionality! thanks!! :)


[ownership] Once we strip ownership ignoring transparent functions fr…

…om our module, strip ownership as we serialize.

@swift-ci smoke test and merge

@swift-ci smoke test

@swift-ci smoke test and merge


Error when initialize database InitializeDatabase(IApplicationBuilder app)  Cannot insert the value NULL into column 'NonEditable'

Hi.
I'm trying to Using EntityFramework Core for configuration and operational data. According to your article https://github.com/IdentityServer/IdentityServer4/blob/master/docs/quickstarts/4_entityframework.rst i successfully performed
dotnet ef migrations add InitialIdentityServerPersistedGrantDbMigration -c PersistedGrantDbContext -o Data/Migrations/IdentityServer/PersistedGrantDb
dotnet ef migrations add InitialIdentityServerConfigurationDbMigration -c ConfigurationDbContext -o Data/Migrations/IdentityServer/ConfigurationDb

But when i call private void InitializeDatabase(IApplicationBuilder app) it fails with following exception
SqlException: Cannot insert the value NULL into column 'NonEditable', table 'aspnet-AspNetCoreIdentityServer2.dbo.Clients'; column does not allow nulls. INSERT fails.
The statement has been terminated.
Code is following:
public static IEnumerable<Client> Clients =>
    new List<Client>
    {
        // machine to machine client (from quickstart 1)
        new Client
        {
            ClientId = "client",
            ClientSecrets = { new Secret("secret".Sha256()) },

            AllowedGrantTypes = GrantTypes.ClientCredentials,
            // scopes that client has access to
            AllowedScopes = { "api1" }
        },
        // interactive ASP.NET Core MVC client
        new Client
        {
            ClientId = "mvc",
            ClientSecrets = { new Secret("secret".Sha256()) },

            AllowedGrantTypes = GrantTypes.Code,
            RequireConsent = false,
            RequirePkce = true,

            // where to redirect to after login
            RedirectUris = { "http://localhost:5002/signin-oidc" },

            // where to redirect to after logout
            PostLogoutRedirectUris = { "http://localhost:5002/signout-callback-oidc" },

            AllowedScopes = new List<string>
            {
                IdentityServerConstants.StandardScopes.OpenId,
                IdentityServerConstants.StandardScopes.Profile
            }
        }
    };

Project file is following:
<PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
    <UserSecretsId>aspnet-AspNetCoreIdentityServer-AFD4C82E-A3A7-4B45-BDA5-4EF3A9BE1DF0</UserSecretsId>
  </PropertyGroup>
<ItemGroup>
    <PackageReference Include="IdentityServer4.EntityFramework" Version="2.1.0" />
    <PackageReference Include="IdentityServer4.AspNetIdentity" Version="2.1.0" />
    <PackageReference Include="Microsoft.AspNetCore.All" Version="2.1.4" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="2.1.4" PrivateAssets="all">
      <IncludeAssets>runtime; build; native; contentfiles; analyzers</IncludeAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Design" Version="2.1.4" PrivateAssets="All" />
  </ItemGroup>

If i set this field to nullable explicitely, the same bug occurs with IdentityResources.NonEditable , Created and etc ...
How to fix this?
Thanks in advance!

These properties should have default values set if you are using the ToEntity mappers found in the IdentityServer4 EF library.
That being said, you are using quite an old version of IdentityServer4 and I can't remember exactly when those were added.

As i assume you call my nugets too old and you suggest to move to identity server 3.0.x.
Please tell

if this server has support for Windows authorization i if there are any samples on github.

What benefits Using EntityFramework Core for configuration and operational data gives me if i choose to move to IS 3.0.x

You could try the patches for 2.1 or even upgrade to 2.5

Thank you, i upgraded to  <PackageReference Include="IdentityServer4.EntityFramework" Version="2.4.0" />    , InitializeDatabase works


[Feature] Bitbucket Addon [OSF-8192]

Updates #6954

Purpose
Add support for a read-only bitbucket OSF add-on.  Still targets develop-backup and needs to be upgraded to work with DjangOSF.
Changes
Adds a whole new provider in website/addons/bitbucket.  All changes outside that directory go directly towards supporting the new addon.
Side effects
Unavoidable grief.
Ticket
[#SVCS-268] - JIRA
[#OSF-8192] - JIRA
Entangled with WB PR: https://github.com/CenterForOpenScience/waterbutler/pull/198

MIGRATION WARNING
Not only does this share a migration number (0042) with #7418 , that migration has a conflicting action -- altering possible types for BaseFileNode.
Running makemigrations --merge will NOT be sufficient.
Instead, whichever PR goes in last will need the migration recreated before being merged.

The corresponding WB PR has been merged and pushed to WB's develop branch.

@sloria , finagling complete.


Print debug messages using human-readable time format

What problem are you facing?
The time formatting of output messages is not in a human readable format.
How could Upbound help solve your problem?
Add the ISO8601 or RFC33339 time encoder to the log output formatter

Actually this is handled in the latest release of crossplane-runtime so updating to 0.19 will fix it

@bobh66 we running atm v0.20.0rc candidate for crossplane-runtime so we can close this issue https://github.com/upbound/upjet-provider-template/blob/main/go.mod#L6 - thanks


docs: add API documentation

This is our initial draft of an API documentation.
The document is aspirational in nature, meaning it does not reflect the current state of the API, but rather where we would like to be. Code implementing this API is already available in separate branches, so once we have accepted a version of this API doc we can immediately bring the implementation up to speed.

@danyill I also just added your "plugin bundle archive format" suggestion to the "Missing" section at the bottom.
Like all features which I'd rather implement in a distribution or within a plugin, I added it for completeness of our discussion only, and would suggest removing it from API.md before merging, as soon as we're all agreed that it doesn't belong in core.

I just removed the "Missing" section from the bottom. If you think any of the points it listed need to be handled by OpenSCD core in particular, please discuss here:
Missing
Plugin manifest, searchable plugin store
This is still needed if we want to enable plugins to be installed and updated from within OpenSCD. This could be done by a "plugin management" or "plugin store" plugin. Otherwise, this could be done by the OpenSCD distribution itself.
A good candidate for a plugin manifest format is the Plugin type defined above, with the addition of a kind flag indicating whether it is a "menu" or "editor" plugin (or maybe "both"), and possibly a version property.
Plugin bundle archive format
For offline installation of plugins (e.g. by dragging and dropping a .zip file into OpenSCD), a format for bundling the plugin's JavaScript and resource files along with a manifest file is needed. This could either be implemented by the OpenSCD distribution or by a "plugin management" plugin as described in the previous section.
Plugin update notifications
Given a plugin manifest with a version number, OpenSCD core could check for updates to the plugins and notify the user when updates are available.
Within the current architecture, the plugin's own service worker is expected to handle this.
Deep linking to editor plugins
Currently, the only way to open an editor plugin is by clicking on its tab in the tab bar. It would be nice to be able to deep link to a particular editor plugin, for example by clicking on a link in another plugin.
This could be handled by the OpenSCD distribution, which could e.g. listen for hashchange events and open the appropriate editor plugin by setting the OpenSCD core's editorIndex property.

I think a lot of the concept are thought through, but I can imagine if you are not familiar with the history some things could harder to understand. For example I don't know why there are no more validator plugins :)
Do you think this is the document to add these reasoning or some other place?

Do you think this is the document to add these reasoning or some other place?

I think an extra "History" section in the README.md might be a great idea, too. Do you think it would be better to have this section in the API.md?

You could also document the individual decisions as ADR and link them here ;)


Define UriData == operator

  Uri uri1 = new Uri(host: "host", path: "some path");
  Uri uri2 = new Uri(host: "host", path: "some path");
  print(uri1 == uri2); // true

  UriData uriData1 = new UriData.fromString("Some data");
  UriData uriData2 = new UriData.fromString("Some data");
  print(uriData1 == uriData2); // false

Like an Uri == operator,  UriData == operator should return true if data Uri's are not the same object but equals

We haven't defined the equality to use for UriData.
Should two UriData objects be equal if they have the same data, the same data and encoding, or the exact same representation? Either is possible, so there isn't a "natural" equality to use. The "exact same representation" is the simplest one to implement (compare the text values), but is probably too specific.
The other cases are more complex, and make having a compatible hashCode much harder.
So, the choice was to not add an equality to the UriData class. You can extract the data, the text or the URI and compare those as you wish, but we won't make one of them the default equality.


Investigate utility of Portainer CE to help us manage our Docker Swarms

Describe the solution you'd like
See if Portainer CE might be useful for cluster management.
Additional context
Mentioned in a highlighted piece from Docker Community Forums 'SAAS based on Docker, 1 container per client'

Just a note we tried portainer but it was pretty shaky with the scale we have of containers.. Very often it would fail.. Hence we all went back to cli..
But worth trying again for sure

No need to go over old ground. It's rare for these things to suddenly get better.


[SPARK-37784][SQL] Correctly handle UDTs in CodeGenerator.addBufferedState()

What changes were proposed in this pull request?
This PR fixes a correctness issue in the CodeGenerator.addBufferedState() helper method (which is used by the SortMergeJoinExec operator).
The addBufferedState() method generates code for buffering values that come from a row in an operator's input iterator, performing any necessary copying so that the buffered values remain correct after the input iterator advances to the next row.
The current logic does not correctly handle UDTs: these fall through to the match statement's default branch, causing UDT values to be buffered without copying. This is problematic if the UDT's underlying SQL type is an array, map, struct, or string type (since those types require copying). Failing to copy values can lead to correctness issues or crashes.
This patch's fix is simple: when the dataType is a UDT, use its underlying sqlType for determining whether values need to be copied. I used an existing helper function to perform this type unwrapping.
Why are the changes needed?
Fix a correctness issue.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
I manually tested this change by re-running a workload which failed with a segfault prior to this patch. See JIRA for more details: https://issues.apache.org/jira/browse/SPARK-37784
So far I have been unable to come up with a CI-runnable regression test which would have failed prior to this change (my only working reproduction runs in a pre-production environment and does not fail in my development environment).

Please let me know if you have suggestions for good ways to write a regression test for this bug.  So far I've been unable to adapt my existing reproduction into something which fails in CI.
Given enough time, I might be able to contrive a failing regression test by manually instantiating a SortMergeJoinExec operator and controlling its input iterators such that the non-copied values are mutated when the iterator advances (I'd use the SparkPlanTest helpers for this).
OTOH this particular helper function changes very infrequently, so I think the risk of future regression might be small enough that it might be okay to forgo writing the more complicated test. If anyone has strong opinions here then please let me know.

I'm now curious about whether there could be other similar UDT-related bugs in our code generation. I plan to search through the code for all other places where we generate copy() / clone() logic to check whether they properly handle UDTs.

I'm going to merge this to master, branch-3.2, branch-3.1, and branch-3.0.

Thank you!


How to declare dependency in Chart.yaml when installing directly from chart archive URL or local archive

Issue
I'm able to install the following Helm chart by directly using the URL to the chart archive:
helm upgrade --install aws-ebs-csi-driver \
    --namespace kube-system \
    --set enableVolumeScheduling=true \
    --set enableVolumeResizing=true \
    --set enableVolumeSnapshot=true \
    https://github.com/kubernetes-sigs/aws-ebs-csi-driver/releases/download/v0.7.0/helm-chart.tgz

However, I want to include the above chart as a dependency in a different chart. How can I declare that as a dependency in the Chart.yaml dependencies section? I've tried various ways but it seems that the dependencies.repository field always requires it to be a proper Helm repository, it appears that I can not just link directly to the archive. I've also tried download the archive directly and using the "file://path/to/helm-chart.tgz" form, but that ends up in a Save error occurred:  validation: chart.metadata is required because it tries to re-archive the already archived chart.
I'm not sure if this is as intended as I couldn't find anything about this in the documentation. The documentation does mention that I should be able to manage dependencies manually within the charts directory as a chart archive, but doesn't describe how to do with regards to declaring the dependency from the Chart.yaml:

Versioning Info
Output of helm version:
version.BuildInfo{Version:"v3.3.4", GitCommit:"", GitTreeState:"", GoVersion:"go1.15.2"}

Output of kubectl version:
kubectl version
Client Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-eks-4c6976", GitCommit:"4c6976793196d70bc5cd29d56ce5440c9473648e", GitTreeState:"clean", BuildDate:"2020-07-17T19:00:19Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.9", GitCommit:"4fb7ed12476d57b8437ada90b4f93b17ffaeed99", GitTreeState:"clean", BuildDate:"2020-07-15T16:10:45Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.):
Minikube / EKS

@Chili-Man I couldn't find this documented anywhere but you can use https://kubernetes-sigs.github.io/aws-ebs-csi-driver/ and your repository when specifying the dependency. They have a Helm chart repo.

@mattfarina indeed they do have helm chart repo but it is out date, unfortunately. They haven't updated the repo's index.yaml to point to the newer version; I created a ticket on their yesterday to get that fixed: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/587 ;  That does give me an idea though, I suppose I can fork, update, and publish it myself instead for the time being.
However, I'm still trying to understand whether or not /how a chart archive / direct URL to the chart can actually be declared in the Chart.yaml's dependency section as how we can helm install directly from that.

Helm does a certain amount of dependency management for you. It deals with versions, updating versions based on ranges, and so forth. To do that, it needs to know about the chart and it's versions rather than an individual file with a single chart version.
When you specify a repository and version in the dependencies you declare what you would like. My versions are often set with something like ^1.2.3 (shorthand for >=1.2.3 < 2.0.0). When you run helm dep up it will figure out which is the latest version in the range and set that in the Chart.lock file. If a new version of a dependency comes out... running helm dep up will find it, update your chart to use it, and lock to it.
This is why Helm is looking for more information.
I Helm were to use a direct archive of a chart the version field would be irrelevant.
Does that help explain what Helm is doing? Many package managers operate this way.


Support for inline macro as value in configurations

Configuration value can be a macro using syntax
warp.configuration = <% 'this is a macro' %>

configuration shadow.macro.FUNC can parse an inline macro

fix a bug related to configuration shadow.macro.FUNC

I'm not sure it's a good idea to allow the possibility to write inline macros that can be indefinitely long.
Maybe we should restrict to a certain number of statements ?
Also, instead of macros, it could be explicit warscript code (not enclosed in <% %> but detectable with another convention).

The change cannot be made global like this, it would have many implications in different places throughout the code base and in terms of operations could lead to errors because WarpScript execution cannot happen before some of the configuration keys are actually set.
Closing the PR for this reason.


Add PyTuple::to_list

Companion to #3043. I've included benchmarks which suggests that this method is indeed about 25% faster on my machine.

Based on #3043 having already gone in, I'm just going to proceed to merge this so I can include it in a rebased 0.18.2.
bors r+


stdlib: Update for stricter enforcement of @usableFromInline

The enforcement patch isn't ready yet (https://github.com/apple/swift/pull/16586) but this patch keeps getting merge conflicts so I'm going to merge it now since it's harmless.

@swift-ci Please smoke test


Dynamically select upstream cluster in TCP Proxy and Proxy Protocol

I'm forwarding traffic to Envoy via Proxy Protocol and would like to configure Envoy to tunnel the traffic upstream using HTTP/2 Connect, similar to the encapsulate example provided here.
The goal is to use the original destination address (from the Proxy Protocol header) to determine the upstream Envoy proxy, while the original destination port is used to route the traffic to the appropriate destination service at the other end of the tunnel. My configuration looks something like this:
static_resources:
  listeners:
    - name: egress_listener
      address:
        socket_address:
          address: <IP_ADDRESS>
          port_value: 8080
      listener_filters:
        - name: envoy.filters.listener.proxy_protocol
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol
      filter_chains:
        - filters:
            - name: envoy.filters.network.tcp_proxy
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
                stat_prefix: egress_tcp_non_mtls
                cluster: other_envoy_proxy_cluster
                tunneling_config:
                  hostname: "<IP_ADDRESS>:%DOWNSTREAM_LOCAL_PORT%"
                  headers_to_add:
                  - header:
                      key: original_dst_port
                      value: "%DOWNSTREAM_LOCAL_PORT%"

  clusters:
    - name: other_envoy_proxy_cluster
      connect_timeout: 5s
      # This ensures HTTP/2 CONNECT is used for establishing the tunnel.
      typed_extension_protocol_options:
        envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
          "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
          explicit_http_config:
            http2_protocol_options: {}
      type: STATIC
      load_assignment:
        cluster_name: other_envoy_proxy_cluster
        endpoints:
        - lb_endpoints:
          - endpoint:
              address:
                socket_address:
                  address: "%DOWNSTREAM_LOCAL_ADDRESS_WITHOUT_PORT%"  # This is not supported
                  port_value: 10000

However, it seems that variable substitution (like %DOWNSTREAM_LOCAL_ADDRESS_WITHOUT_PORT%) is not supported for the SocketAddress, which makes this configuration invalid.
I am aware of the option to use the ORIGINAL_DST cluster type to automatically use the original destination IP and port, but in my case, I need to override the port (because the tunnel port and the final service destination port are different).
Is there any workaround to either:

Use variable substitution for the SocketAddress, or
Use the ORIGINAL_DST cluster type but still allow overriding the port for tunneling purposes?

Maybe you can use envoy_v3_api_field_config.core.v3.SocketAddress.resolver_name?
add @zuercher who is probably the expert here and code owner of tcp_proxy

Where can I find more information on custom resolvers? From what I understand, setting this up would require implementing a separate service that exposes either a gRPC-based resolution API or DNS-over-HTTPS (DoH). However, passing the original destination information to this service feels like a lot of complexity, whereas I was hoping to handle this in-process with something like a Lua script or another extension mechanism.

Hi, have a look:
Network Filter (network_filters/set_filter_state)
in combination with cluster type ORIGINAL_DST and upstream port override (originaldstlbconfig) can work for you..
I haven't tested this but theoretically maybe something like this might work here:
set_filter_state network filter config:
on_new_connection:
- object_key: envoy.network.transport_socket.original_dst_address
  format_string:
    text_format_source:
      inline_string: "%DOWNSTREAM_LOCAL_ADDRESS_WITHOUT_PORT%"

and in the ORIGINAL_DST cluster:
"original_dst_lb_config { upstream_port_override": 10000 }


Tests fail with ghc 9.6 on stackage nightly

Add to expected test failures for now.
       Test suite failure for package cardano-coin-selection-1.0.1
           unit:  exited with: ExitFailure 1
       Full log available at /var/stackage/work/unpack-dir/.stack-work/logs/cardano-coin-selection-1.0.1-test.log

           Cardano.CoinSelection
             CoinMap properties
               CoinMap coverage is adequate [✔]
                 +++ OK, passed 200 tests:
                 39.0% coin map has multiple entries
                 39.0% coin map has several unique values
                 34.0% coin map has one entry
                 34.0% coin map has one unique value
                 27.0% coin map is empty
               CoinMapEntry coverage is adequate [✔]
                 +++ OK, passed 3200 tests:
                 86.72% coin map entry list has multiple entries
                 86.34% coin map entry list has multiple unique keys
                 40.16% coin map entry list has multiple duplicate keys
                 27.56% coin map entry list has no duplicate keys
                 15.69% coin map entry list has one duplicate key
                 11.34% coin map entry list has two duplicate keys
                  5.25% coin map entry list is empty
                  4.53% coin map entry list has one unique key
                  4.38% coin map entry list has one entry
                  3.88% coin map entry list has two unique keys
                  3.66% coin map entry list has two entries
               coinMapFromList preserves total value for each unique key [✔]
                 +++ OK, passed 100 tests.
               coinMapFromList preserves total value [✔]
                 +++ OK, passed 100 tests.
               coinMapToList preserves total value [✔]
                 +++ OK, passed 100 tests.
               coinMapFromList . coinMapToList = id [✔]
                 +++ OK, passed 100 tests.
               coinMapToList . coinMapFromList = id (when no keys duplicated) [✔]
                 +++ OK, passed 100 tests.
               coinMapToList order deterministic [✔]
                 +++ OK, passed 100 tests (43% shuffled).
             CoinSelection properties
               monoidal append preserves keys [✔]
                 +++ OK, passed 100 tests.
               monoidal append preserves value [✔]
                 +++ OK, passed 100 tests.
             CoinSelectionData properties
               CoinSelectionData coverage is adequate [✔]
                 +++ OK, passed 200 tests (98.5% amountAvailable ≥ amountRequested).
           Cardano.CoinSelection.Algorithm.LargestFirst
             Coin selection: largest-first algorithm: unit tests
               Expect success: case #1:
           	max=100, UTxO=[10,10,17], Output=[17] --> Right [17] [✔]
               Expect success: case #2:
           	max=100, UTxO=[12,10,17], Output=[1] --> Right [17] [✔]
               Expect success: case #3:
           	max=100, UTxO=[12,10,17], Output=[18] --> Right [12,17] [✔]
               Expect success: case #4:
           	max=100, UTxO=[12,10,17], Output=[30] --> Right [10,12,17] [✔]
               Expect success: case #5:
           	max=3, UTxO=[1,2,10,6,5], Output=[11,1] --> Right [6,10] [✔]
               Expect success: case #6:
           	max=100, UTxO=[12,20,17], Output=[40,1,1,1] --> Right [12,17,20] [✔]
               Expect success: case #7:
           	max=100, UTxO=[12,20,17], Output=[40,1] --> Right [12,17,20] [✔]
               Expect success: case #8:
           	max=100, UTxO=[20,20,10,5], Output=[41,6] --> Right [10,20,20] [✔]
               Expect success: case #9:
           	max=2, UTxO=[1,2,10,6,5], Output=[11,1] --> Right [6,10] [✔]
               Expect success: fewer inputs than outputs: case #1:
           	max=1000, UTxO=[100,100], Output=[1,2,3,4] --> Right [100] [✔]
               Expect success: fewer inputs than outputs: case #2:
           	max=1000, UTxO=[100,10], Output=[1,2,3,4] --> Right [100] [✔]
               Expect success: fewer inputs than outputs: case #3:
           	max=1000, UTxO=[10,10], Output=[1,2,3,4] --> Right [10] [✔]
               Expect success: fewer inputs than outputs: case #4:
           	max=1, UTxO=[100], Output=[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1] --> Right [100] [✔]
               UTxO balance not sufficient: case #1:
           	max=100, UTxO=[12,10,17], Output=[40] --> Left (InputValueInsufficient (InputValueInsufficientError {inputValueAvailable = Coin 39, inputValueRequired = Coin 40})) [✔]
               UTxO balance not sufficient: case #2:
           	max=100, UTxO=[12,10,17], Output=[40,1,1,1] --> Left (InputValueInsufficient (InputValueInsufficientError {inputValueAvailable = Coin 39, inputValueRequired = Coin 43})) [✔]
               UTxO balance sufficient, but maximum input count exceeded:
           	max=9, UTxO=[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1], Output=[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1] --> Left (InputLimitExceeded (InputLimitExceededError {calculatedInputLimit = 9})) [✔]
             Coin selection: largest-first algorithm: properties
               forall (UTxO, NonEmpty TxOut), for all selected input, there's no bigger input in the UTxO that is not already in the selected inputs [✔]
                 +++ OK, passed 100 tests; 1 discarded.
               The algorithm selects just enough inputs and no more. [✔]
                 +++ OK, passed 10000 tests; 81 discarded.
               The algorithm produces the correct set of change. [✘]
           Cardano.CoinSelection.Algorithm.Migration
             idealBatchSize
               Eventually converge for decreasing functions [✔]
                 +++ OK, passed 100 tests:
                 47% BatchSize 42
                 27% BatchSize 32768
                 26% BatchSize 21845
             accuracy of selectCoins
               dust=1% [✔]
                 +++ OK, passed 1000 tests:
                 89.9% MEDIOCRE (<=  99%)
                  9.8% PERFECT  (== 100%)
                  0.3% OKAY     (>   99%)
               dust=5% [✔]
                 +++ OK, passed 1000 tests:
                 72.5% PERFECT  (== 100%)
                 23.1% MEDIOCRE (<=  99%)
                  4.4% OKAY     (>   99%)
               dust=10% [✔]
                 +++ OK, passed 1000 tests:
                 89.5% PERFECT  (== 100%)
                  6.6% MEDIOCRE (<=  99%)
                  3.9% OKAY     (>   99%)
               dust=25% [✔]
                 +++ OK, passed 1000 tests:
                 96.6% PERFECT  (== 100%)
                  2.2% OKAY     (>   99%)
                  1.2% MEDIOCRE (<=  99%)
               dust=50% [✔]
                 +++ OK, passed 1000 tests:
                 98.6% PERFECT  (== 100%)
                  1.1% OKAY     (>   99%)
                  0.3% MEDIOCRE (<=  99%)
             selectCoins properties
               No coin selection has outputs [✔]
                 +++ OK, passed 10000 tests.
               Every coin in the selection change > dust threshold [✔]
                 +++ OK, passed 10000 tests.
               Total input UTxO value >= sum of selection change coins [✔]
                 +++ OK, passed 10000 tests.
               Every selection input is unique [✔]
                 +++ OK, passed 10000 tests.
               Every selection input is a member of the UTxO [✔]
                 +++ OK, passed 10000 tests.
               Every coin selection is well-balanced [✔]
                 +++ OK, passed 10000 tests.
             selectCoins regressions
               regression #1 [✔]
                 +++ OK, passed 1 test.
           Cardano.CoinSelection.Algorithm.RandomImprove
             Coin selection : random algorithm unit tests
               Cardano.CoinSelectionSpec[355:5] [✔]
               Cardano.CoinSelectionSpec[355:5] [✔]
               Cardano.CoinSelectionSpec[355:5] [✔]
               Cardano.CoinSelectionSpec[355:5] [✔]
               Cardano.CoinSelectionSpec[355:5] [✔]
               Cardano.CoinSelectionSpec[355:5] [✔]
               cannot cover aim, but only min:
           	max=4, UTxO=[1,1,1,1,1,1], Output=[3] --> Right [1,1,1,1] [✔]
               REG CO-450: no fallback:
           	max=4, UTxO=[1000000,1000000,1000000,1000000], Output=[2000000,500000] --> Right [1000000,1000000,1000000,1000000] [✔]
               enough funds, proper fragmentation, inputs depleted:
           	max=100, UTxO=[10,10,10,10], Output=[38,1] --> Left (InputsExhausted InputsExhaustedError) [✔]
               Cardano.CoinSelectionSpec[355:5] [✔]
               each output needs <maxNumOfInputs:
           	max=9, UTxO=[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1], Output=[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1] --> Left (InputLimitExceeded (InputLimitExceededError {calculatedInputLimit = 9})) [✔]
               each output needs >maxNumInputs:
           	max=9, UTxO=[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1], Output=[10,10,10,10,10,10,10,10,10,10] --> Left (InputLimitExceeded (InputLimitExceededError {calculatedInputLimit = 9})) [✔]
               Cardano.CoinSelectionSpec[355:5] [✔]
               Cardano.CoinSelectionSpec[355:5] [✔]
               Cardano.CoinSelectionSpec[355:5] [✔]
             Coin selection properties : random algorithm
               forall (UTxO, NonEmpty TxOut), running algorithm gives not less UTxO fragmentation than LargestFirst algorithm [✔]
                 +++ OK, passed 100 tests; 2 discarded.
               forall (UTxO, NonEmpty TxOut), running algorithm gives the same errors as LargestFirst algorithm [✔]
                 +++ OK, passed 100 tests.
           Cardano.CoinSelection.Fee
             Fee calculation : unit tests
               CoinSelection (inps=[20]outs=[17]chngs=[3]), UTxO=[]), fee=3 --> Right (FeeOutput {csInps = [20], csOuts = [17], csChngs = []}) [✔]
               CoinSelection (inps=[20,20]outs=[16,18]chngs=[4,2]), UTxO=[]), fee=6 --> Right (FeeOutput {csInps = [20,20], csOuts = [16,18], csChngs = []}) [✔]
               CoinSelection (inps=[20,20]outs=[18,18]chngs=[2,2]), UTxO=[]), fee=2 --> Right (FeeOutput {csInps = [20,20], csOuts = [18,18], csChngs = [1,1]}) [✔]
               CoinSelection (inps=[20,20]outs=[17,18]chngs=[3,2]), UTxO=[]), fee=2 --> Right (FeeOutput {csInps = [20,20], csOuts = [17,18], csChngs = [2,1]}) [✔]
               CoinSelection (inps=[20,20,20]outs=[14,18,19]chngs=[6,2,1]), UTxO=[]), fee=3 --> Right (FeeOutput {csInps = [20,20,20], csOuts = [14,18,19], csChngs = [4,1,1]}) [✔]
               CoinSelection (inps=[20,20,20]outs=[14,18,19]chngs=[6,2,1]), UTxO=[]), fee=3 --> Right (FeeOutput {csInps = [20,20,20], csOuts = [14,18,19], csChngs = [6]}) [✔]
               CoinSelection (inps=[20]outs=[17]chngs=[3]), UTxO=[]), fee=4 --> Left (CannotCoverFee (Fee (Coin 1))) [✔]
               CoinSelection (inps=[10]outs=[7]chngs=[3]), UTxO=[1]), fee=5 --> Left (CannotCoverFee (Fee (Coin 1))) [✔]
               CoinSelection (inps=[10]outs=[7]chngs=[3]), UTxO=[1,1]), fee=5 --> Right (FeeOutput {csInps = [10,1,1], csOuts = [7], csChngs = []}) [✔]
               CoinSelection (inps=[10]outs=[7]chngs=[3]), UTxO=[3]), fee=5 --> Right (FeeOutput {csInps = [10,3], csOuts = [7], csChngs = [1]}) [✔]
               CoinSelection (inps=[10,10]outs=[7,7]chngs=[3,3]), UTxO=[2,2]), fee=10 --> Right (FeeOutput {csInps = [10,10,2,2], csOuts = [7,7], csChngs = []}) [✔]
               CoinSelection (inps=[10,10]outs=[7,7]chngs=[3,3]), UTxO=[3,3]), fee=10 --> Right (FeeOutput {csInps = [10,10,3,3], csOuts = [7,7], csChngs = [1,1]}) [✔]
               CoinSelection (inps=[20,20]outs=[16,18]chngs=[4,2]), UTxO=[]), fee=6 --> Right (FeeOutput {csInps = [20,20], csOuts = [16,18], csChngs = []}) [✔]
               CoinSelection (inps=[1]outs=[1]chngs=[]), UTxO=[2]), fee=1 --> Right (FeeOutput {csInps = [1,2], csOuts = [1], csChngs = [1]}) [✔]
               CoinSelection (inps=[]outs=[]chngs=[]), UTxO=[3]), fee=3 --> Right (FeeOutput {csInps = [3], csOuts = [], csChngs = []}) [✔]
               CoinSelection (inps=[]outs=[]chngs=[]), UTxO=[2,2]), fee=3 --> Right (FeeOutput {csInps = [2,2], csOuts = [], csChngs = [1]}) [✔]
               CoinSelection (inps=[]outs=[]chngs=[]), UTxO=[2,2]), fee=3 --> Right (FeeOutput {csInps = [2,2], csOuts = [], csChngs = [1]}) [✔]
             Fee Calculation: Generators
               Arbitrary CoinSelection [✔]
                 +++ OK, passed 100 tests.
             Fee Adjustment properties
               Fee adjustment is deterministic when there's no extra inputs [✔]
                 +++ OK, passed 100 tests.
               Adjusting for fee (/= 0) reduces the change outputs or increase inputs [✔]
                 +++ OK, passed 100 tests; 10 discarded.
             distributeFee
               fee portions are all within unity of ideal unrounded portions [✔]
                 +++ OK, passed 100 tests.
               fee portions are allocated optimally [✔]
                 +++ OK, passed 100 tests.
               Σ fst (distributeFee fee outs) == fee [✔]
                 +++ OK, passed 3200 tests.

                 properties (6400 in total):
                 50.00% fee > 0
                 40.55% nOuts=2+
                  4.88% nOuts=1
                  4.58% nOuts=2
               snd (distributeFee fee outs) == outs [✔]
                 +++ OK, passed 3200 tests.

                 properties (6400 in total):
                 50.00% fee > 0
                 40.55% nOuts=2+
                  4.88% nOuts=1
                  4.58% nOuts=2
               expectFailure: not (any null (fst <$> distributeFee fee outs)) [✔]
                 +++ OK, failed as expected. Falsified (after 1264 tests and 49 shrinks):
                 (Fee (Coin 1),Coin 1 :| [Coin 1])
             coalesceDust
               sum coins = sum (coalesceDust threshold coins) [✔]
                 +++ OK, passed 12800 tests:
                 91.016% sum coins ≠ 0
                  8.984% sum coins = 0
               all (/= Coin 0) (coalesceDust threshold coins) [✔]
                 +++ OK, passed 12800 tests:
                 91.141% ∃ coin ∈ coins . coin = 0
                  8.984% ∀ coin ∈ coins . coin = 0
                  8.859% ∀ coin ∈ coins . coin > 0
               leaves at most one dust coin [✔]
                 +++ OK, passed 12800 tests:
                 52.805% length result ≥ 2
                 38.211% length result = 1
                 26.687% ∀ coin ∈ coins . coin ≠ threshold
                 13.891% have mixture of coin values in relation to threshold
                  8.984% length result = 0
                  8.906% ∀ coin ∈ coins . coin = threshold
                  7.953% ∀ coin ∈ coins . coin < threshold
                  3.180% ∀ coin ∈ coins . coin > threshold
               length coins >= (coalesceDust threshold coins) [✔]
                 +++ OK, passed 100 tests.
             reduceChangeOutputs
               data coverage is adequate [✔]
                 +++ OK, passed 400 tests:
                 100.0% fee >= 0
                 51.5% several non-empty coins
                 30.8% fee = 0
                 23.5% 0 < fee < sum coins
                 21.8% fee > sum coins
                 21.2% fee = sum coins
                 21.2% two non-empty coins
                 15.5% one non-empty coin
               the fee balancing algorithm converges for any coin selection [✔]
                 +++ OK, passed 100000 tests.
             splitCoin
               data coverage is adequate [✔]
                 +++ OK, passed 200 tests:
                 73.0% coin to split is non-zero
                 54.0% list of coins has at least one non-zero coin
                 42.5% list of coins has at least one zero coin
                 37.0% list of coins has multiple entries
                 33.5% coin to split is smaller than number of coins to increase
                 33.5% list of coins is empty
                 29.5% list of coins is singleton
                 27.0% coin to split is zero
               preserves the total sum [✔]
                 +++ OK, passed 100 tests.
               results are all within unity of ideal unrounded results [✔]
                 +++ OK, passed 100 tests.

           Internal.Coin
             Coin properties
               Only construction of non-negative values is possible. [✔]
                 +++ OK, passed 3200 tests:
                 50.03% input is negative
                 46.53% input is positive
                  3.44% input is zero
               Coverage of generated values is acceptable. [✔]
                 +++ OK, passed 100 tests:
                 45% value is more than one
                 28% value is zero
                 27% value is one
               Addition [✔]
                 +++ OK, passed 100 tests.
               Multiplication [✔]
                 +++ OK, passed 6400 tests:
                 48.64% scaling factor is positive
                 48.20% scaling factor is negative
                  3.16% scaling factor is zero
               Subtraction [✔]
                 +++ OK, passed 100 tests:
                 50% x > y
                 29% x < y
                 21% values are equal
               Division [✔]
                 +++ OK, passed 6400 tests:
                 48.64% denominator is positive
                 48.20% denominator is negative
                  3.16% denominator is zero
               Modulus [✔]
                 +++ OK, passed 6400 tests:
                 48.64% denominator is positive
                 48.20% denominator is negative
                  3.16% denominator is zero
               Distance [✔]
                 +++ OK, passed 100 tests:
                 50% x > y
                 29% x < y
                 21% values are equal
           Test.Vector.Shuffle
             shuffle
               every list can be shuffled, ultimately [✔]
                 +++ OK, passed 100 tests (91% shuffled).
               shuffle is non-deterministic [✔]
                 +++ OK, passed 100 tests (95% not deterministic).
               sort (shuffle xs) == sort xs [✔]
                 +++ OK, passed 100 tests (98% non-empty).
             shuffleNonEmpty
               every non-empty list can be shuffled, ultimately [✔]
                 +++ OK, passed 100 tests (91% shuffled).
               shuffleNonEmpty is non-deterministic [✔]
                 +++ OK, passed 100 tests (92% not deterministic).
               sort (shuffleNonEmpty xs) == sort xs [✔]
                 +++ OK, passed 100 tests (100% non-empty).
             shuffleWith / mkSeed
               shuffling with the same seed is deterministic [✔]
                 +++ OK, passed 100 tests (98% non singleton).
               different seed means different shuffles [✔]
                 +++ OK, passed 200 tests; 23 discarded:
                 89.5% different

                 66.0% big list
                 29.0% small list
                  5.0% singleton

           Failures:

             src/test/Cardano/CoinSelection/Algorithm/LargestFirstSpec.hs:274:9:
             1) Cardano.CoinSelection.Algorithm.LargestFirst, Coin selection: largest-first algorithm: properties, The algorithm produces the correct set of change.
                  *** Gave up! Passed only 51199 tests; 416 discarded tests:
                  98.742% amountSelected > amountRequired
                   1.258% amountSelected = amountRequired

             To rerun use: --match "/Cardano.CoinSelection.Algorithm.LargestFirst/Coin selection: largest-first algorithm: properties/The algorithm produces the correct set of change./"

           Randomized with seed 624635623

           Finished in 43.6456 seconds
           121 examples, 1 failure

Need to revisit and determine if this is still an issue.


docs: move API changes to the correct API version

These changes were made as part of https://github.com/docker/docker/pull/17478 but inadvertently added to the v1.21 API version.
Also added a short description to the API changelog.

ping @SvenDowideit @moxiegirl @jamtur01 :smile:


method access modifiers

would we be marked down if we do not declare methods to be private? (methods are public by default)

It should be fine to leave your methods public, but if the methods are only used within your class, I think it'll be better to leave them as private.

What @weikiatt said ^^^. In general, it is best practice to limit the scope of your methods using private/protected etc. if they are only used internally in the class. Although, it is technically not wrong to leave them all as public, it is still highly recommended to set them to private.

same! i normally leave them all as public first then assign methods as private when checking if i realise they havent been used outside of the class (e.g. overloaded constructors)


Party Overview doesn't pick up Custom Currencies

I see that the module is supposed to have compatibility with party overview, but it's not working for me. Party overview shows the default 5e curriences. I'm using Custom Currencies 1.3.5, Party Overview 2.5.13, and Foundry VTT 0.8.9.

This error occurs for me as well, and is caused by a data shape failure within Party Overview.
I found the location of the error after some investigation. I suspect that Custom Currency is not following the expected data shape contract.
Party Overview appears to be expecting CONFIG.DND5E.currencies to have a different shape than it actually has when Custom Currencies is installed. I suspect the proper values for the properties are not strings, but rather objects with a toString method which outputs the currency's name.
See https://github.com/mclemente/party-overview/blob/2.5.19/module/SystemProvider.js#L126 for the failing method in Party Overview
Party Overview expects CONFIG.DND5E.currencies to have a shape like:
class Currency {
  constructor({name, each}){
    this.name = name;
    this.conversion = { each };
  }

  toString = ()=> this.name;
}
CONFIG.DND5E.currencies = {
  pp: new Currency({ name: 'Colonial Dollar', each: 0.1}),
  gp: new Currency({ name: 'Gold Piece', each: 1}),
  ep: new Currency({ name: 'Pound', each: 10}),
  sp: new Currency({ name: 'Shilling', each: 20 }),
  cp: new Currency({ name: 'Pence', each: 12 }),
}


"Not showing x further errors and warnings" increases diagnostic severity in explorer

Issue Type: Bug

Enable "problems.decorations.enabled": true in settings
Open a folder with a file that has many (e.g 100+) "warning" severity diagnostics
Observe explorer decoration for file

Expected:
Green "warning/info/hint" colouring and "+9" text
Actual
Red "error" colouring and "+9" text
Note, the problems list behaves correctly and doesn't consider the "fake" error when sorting files like it would a real error.


[Feature Request] Automatic Outlook addin sideload/installation on Mac

Can I install/side-load Outlook custom add-in for Mac programatically without manually installing?
Why it is possible on Windows and not possible on Mac?

Outlook for Windows, uses the registry for config storing.
The script behavior for windows is storing manifest path under the key:
HKEY_CURRENT_USER\SOFTWARE\Microsoft\Office\16.0\Wef\Developer
And outlook automatically sideloads that addin.
For Mac, the situation is quite different.
It has windows registry emulation through sqlite database, but that emulation does not support that key.
I have try to store the manifest path under that key and Mac Outlook didn't sideload my addin.
Also, Mac Outlook has native config storage - plist file.
But I have no idea under what key I should store the manifest path, to be addin automatically sideloaded.
I found in the documentation, that there is OfficeWebAddinDeveloperExtras flag in that plist.
What exactly things, this flag turns on or off?

The problem is the code that will look up this setting and use it to load the add-in is part of the product and not the tooling.  It was added to Win Outlook, but the Mac Outlook is a different code base and needs to be updated separately and it hasn't been.  This is something we'll need to add to our backlog.


IPv4/6 error during setup

Bug Report
System Information

Operating System: Debian 12
AMP version: just now executed setup script

I confirm:

[x] that I have searched for an existing bug report for this issue.
[x] that I am using the latest available version of AMP.
[x] that my operating system is up-to-date.
[x] that my checkboxes above look exactly like the one on the left with no extra spaces. Remove this line to confirm you can follow instructions - if this line shows up in the bug report then the whole report will be ignored

Symptoms
I try to install AMP using the provided setup script from getamp.sh as user root on a fresh installed Debian 12 VPS at Hetzner.
I added A and AAAA records for the new VPS to my DNS zone, resolving to the VPS IP adresses.
Regarding HTTPS encryption for the panel, the script checks if the given domain resolves to the systems IP address and reports:
The specified domain my.server.com resolves to 'X.X.X.X' but your external IP is 'Y:Y:Y:Y::1'.
So it retrieves the external resolved IPv4 Address and compares with the systems IPv6 address. Both, as listed in the error message, absolutely correct and resolvable with nslookup from my workstation.
Reproduction
Have a VPS with IPv4 and IPv6 Adresses, add them as A and AAAA records to your dns zone and execute the setup script.

i had to disable ipv6 then reenable it again

This has been fixed in a subsequent update.


Fix Pokedex tab and Update Login page

Fixes #53
New login screen preview

Great job! I had been going a little crazy trying to fix that pokedex tab


impl Display for Statement

What type of enhancement is this?
API improvement
What does the enhancement do?
sqlparser's Statement implements Display, we need to implement Display for our Statement, named sql::statement::Statement also.
See
match self.query_statement(stmt, query_ctx.clone()).await {
    Ok(output) => {
        let output_result =
            query_interceptor.post_execute(output, query_ctx.clone());
        results.push(output_result);
    }
    Err(e) => {
        let redacted = sql::util::redact_sql_secrets(query.as_ref());
        error!(e; "Failed to execute query: {redacted}");

        results.push(Err(e));
        break;
    }
}

We currently print all the query if any of the statement failed to execute. We should write instead:
match self.query_statement(stmt, query_ctx.clone()).await {
    Ok(output) => {
        let output_result =
            query_interceptor.post_execute(output, query_ctx.clone());
        results.push(output_result);
    }
    Err(e) => {
        error!(e; "Failed to execute query: {stmt}");

        results.push(Err(e));
        break;
    }
}

with stmt probably redacted. And formatting structural stmt would be more reliable to redact fields, instead of depending on brittle regexps.
Implementation challenges
No response

I'd like to give it a try.

@irenjj a reminder to confirm that you still work on this issue - do you find any trouble here?

@irenjj Thanks for your update! That's great.


installation of release v3.0 fails

Hi,
the 3.0 release lacks the 'metaphlan.py' file in main directory and in the one found in /metaphlan
the function 'check_and_install_database' in undefined:
metaphlan/metaphlan.py", line 915, in main     pars['index'] = check_and_install_database(pars['index'], pars['bowtie2db'], pars['bowtie2_build'], pars['nproc'], pars['force_download']) NameError: name 'check_and_install_database' is not defined
best,

Hi,
have you installed the software with pip install . or just executed the py
script?

Francesco Beghini
PhD Student
Lab. of Computational Metagenomics
Department of Cellular, Computational and Integrative Biology - CIBIO
University of Trento
Via Sommarive 9, 38123 Trento, Italy
On Fri, Apr 3, 2020 at 7:29 PM genio0815<EMAIL_ADDRESS>wrote:

Hi,
the 3.0 release lacks the 'metaphlan.py' file in main directory and in the
one found in /metaphlan
the function 'check_and_install_database' in undefined:
metaphlan/metaphlan.py", line 915, in main pars['index'] =
check_and_install_database(pars['index'], pars['bowtie2db'],
pars['bowtie2_build'], pars['nproc'], pars['force_download']) NameError:
name 'check_and_install_database' is not defined
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/biobakery/MetaPhlAn/issues/77, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGKRKK46V5HXAUJDZVBTXTRKYMJDANCNFSM4L4JZSKQ
.

HI,
i tested the source code tarball tagged as v3.0
https://github.com/biobakery/MetaPhlAn/archive/3.0.tar.gz
cheers,

From the root folder, the one that contains the setup.py file, you should
run "pip install ." in order to install MetaPhlAn and make it runnable
Francesco Beghini
PhD Student
Lab. of Computational Metagenomics
Department of Cellular, Computational and Integrative Biology - CIBIO
University of Trento
Via Sommarive 9, 38123 Trento, Italy
On Tue, Apr 7, 2020 at 6:14 PM genio0815<EMAIL_ADDRESS>wrote:

HI,
i tested the source code tarball tagged as v3.0
https://github.com/biobakery/MetaPhlAn/archive/3.0.tar.gz
cheers,
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/biobakery/MetaPhlAn/issues/77#issuecomment-610480791,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGKRKP6D6VXE5NUBJI3S63RLNGN3ANCNFSM4L4JZSKQ
.

Sry, obviously I did not look closely enough at how this is packed (was previously slightly different).
Thanks


Super expression must either be null or a function

I'm not sure what this error means, but my babel just updated, and now stimulus throws this error.
import { Application } from "stimulus"
import EmailSignupController from "./controllers/email_signup_controller"

const application = Application.start()
// the offending line. Commenting this out stops the error.
application.register("email_signup", EmailSignupController)

I'm using "stimulus": "^1.1.1" and "@babel/core": "7.6.2". When I was on babel 7.5.5 things worked fine.

Can you share a working example that reproduces the issue? I tried using Babel 7.6.2 and the same registration code without error:

https://glitch.com/edit/#!/tundra-sneeze?path=src/index.js:3:0
https://tundra-sneeze.glitch.me/

When I try to update on that site, I get an error: 😒

My EmailSignupController is really simple:
import { Controller } from 'stimulus'

export default class extends Controller {
  static get targets() {
    return [ 'email', 'agree', 'form', 'complete' ]
  }

  submit() {
    let errors = [];
    if(this.emailTarget.value === '') {
      errors.push("An email address is required")
    }
    if(!this.agreeTarget.checked) {
      errors.push("Please check the agree box.")
    }
    if(errors.length > 0) {
      errors.forEach(err => document.querySelector(".signup-form .errors").innerHTML += `<p>${err}</p>`)
    } else {
      this.completeTarget.style.display = "block"
      this.formTarget.style.display = "none"
    }
  }
}

As per normal JS stuff, my yarn.lock is pretty massive, so it's possible it's not a babel core issue, but maybe one of the other side deps. Here's the deps from my package.json
"devDependencies": {
    "@babel/core": "7.6.2",
    "@babel/plugin-proposal-class-properties": "7.5.5",
    "@babel/preset-env": "7.6.2",
    "@fortawesome/fontawesome-free": "5.10.1",
    "babel-loader": "8.0.6",
    "bootstrap": "4.3.1",
    "browser-sync": "2.26.7",
    "del": "5.0.0",
    "gulp": "4.0.2",
    "gulp-concat": "2.6.1",
    "gulp-imagemin": "^6.1.0",
    "gulp-newer": "^1.4.0",
    "gulp-noop": "^1.0.0",
    "gulp-rev": "^9.0.0",
    "gulp-sass": "^4.0.2",
    "gulp-size": "^3.0.0",
    "gulp-sourcemaps": "^2.6.5",
    "gulp-uglify": "^3.0.2",
    "jquery": "3.3.1",
    "normalize-scss": "^7.0.1",
    "photoswipe": "^4.1.3",
    "plyr": "^3.5.6",
    "popper.js": "1.15.0",
    "rails-ujs": "^5.2.3"
    "rev-del": "^2.0.0",
    "stimulus": "^1.1.1",
    "webpack-stream": "^5.2.1"
  }

Hmm, well, I can't reproduce using your controller code and dependency list. Does it work if you rm yarn.lock && yarn install? Does the stack track from the error point to anything?

I tried blowing everything out, and I still get it. Unfortunately, the error isn't helpful. It says the error is on line 1 of bundle.js 😝  It doesn't show anything else. I'm using Firefox 69 on macOS 10.14.6 if that matters. I'll have to see what error I get in other browsers, or if it's just my machine being wonky.

Happy to try if you can share an example repo (or .zip of one).

I appreciate the help. I wish I could share more of the code, but it's all work stuff 😞  I'll try it again and see if I can create a mini example somehow

Unfortunately, the error isn't helpful. It says the error is on line 1 of bundle.js

Try turning off minification / production mode / etc.

Closing for now. Will reopen if you can trace this back to Stimulus. ✌

Good news, looks like it's not Stimulus. It seems to be uglify. Removing that solves the issue. The weird thing though was that I was using uglify before stimulus, and things worked fine. So some how a mix of having the two? But specifically related on the uglify side it seems.
Anyway. Thanks for the help!


feat(pgbouncer): add support for service account  & sidecars in the pod

This contrib allows to spin out a sidecar in the pod (e.g. gcloudsqlproxy to provide an extra layer of security/ease between our apps and PG instances) this feature require a service-account to be mounted in the workload.
Thanks !

Thanks for contributing @OlivierCharuel
Looks good to me but please update configuration docs in README.md accordingly.

Hello @pmint93. I updated the configuration docs in README.md.
Thank you


Added fatal error for "Returning error because of i/o error...

I added a test using a simulated fastq-multx case.
I'll probably submit another change after I update IO::Pipe::Producer to set the error code of the system call in $?.
To run the tests yourself:
cd tests
./run_tests.sh

For now, I think we can ignore the "All checks have failed".  I'll create an issue to address this:
Error: Unable to process command '::add-path::/home/runner/work/htseq2multx/htseq2multx/fastq-multx' successfully.
[9](https://github.com/hepcat72/htseq2multx/runs/7620069135?check_suite_focus=true#step:3:10)
Error: The `add-path` command is disabled. Please upgrade to using Environment Files or opt into unsecure command execution by setting the `ACTIONS_ALLOW_UNSECURE_COMMANDS` environment variable to `true`. For more information see: https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/

Supplanted by #9


Open the folder

Description
When I have launched the files app and I have a folder open, I minimize the interface and then I open a folder on the desktop, but the interface doesn't show up at the top, but I check and find that it does open the folder.
Steps To Reproduce

Open folder a
Minimize the interface to the taskbar
Double-click to open another folder b
The software still appears in the taskbar, not at the top.
Check the software interface and find that folder b has been opened.

Requirements

e.g. Fix crash that occurs when...

Files Version
<IP_ADDRESS>
Windows Version
10.0.22635.3500
Log File
debug.log

Thank you for the feedback!
So, the issue you reported is where opening a folder doesn't bring the app window to the top foreground, where you can visually see?

Yes, the folder is opened, but the software interface doesn't pop up and remains minimized in the taskbar.


Addon.save() does not work

When trying to save addon (e.g. https://www.moddb.com/mods/stalker-anomaly/addons/anomaly-hd-models-addon),
class File(BaseMetaClass):
    def save(self, path = None):
        mirror = join(download.find("a", string=f"download {self.name}")["href"])

throws TypeError: 'NoneType' object is not subscriptable because it tries to search for an addon title instead for a filename

Simply needs to be changed to:
mirror = join(download.find("a", string=f"download {self.filename}")["href"])

Fixed in 22c92e3


Hide tasks in run task menu,

I would like the ability to hide tasks in the run tasks menu. For example we have PreLaunchTasks that we don't want to ever show in the menu.
This feature has been requested multiple times: #125065, #116892, #110928
@Alejandro-FA, @herrberk, @CarlPer are you still interested in this feature?

@mjrist Yes, absolutely. Those tasks certainly pollute the Run Task drop-down...

https://github.com/microsoft/vscode/issues/197390#issue-1976644953 implies adding "hide": true may achieve this.

#197390 (comment) implies adding "hide": true may achieve this.

You're right, seems like this has been implemented since the original feature request!

Excellent. Confirmed this works. I will close out the feature request.


add linkcheck

Adds linkcheck to mdbook build/mdbook serve so we can avoid breaking links in the future.

NOTE: requires everyone
cargo install mdbook-linkcheck


*: replace grpc-gateway endpoint with /v3

This PR replaces the current /v3beta with the stable /v3 API. Currently, support for /v3beta is maintained for one minor iteration and will remove completely as of 3.5. This can be removed now but as we offered the convenience with the last bump I kept it in while completely removing /v3alpha.
I will attempt to improve e2e test coverage in a separate PR.
ref: https://github.com/coreos/etcd/issues/9189

Codecov Report

Merging #9298 into master will increase coverage by 0.16%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #9298      +/-   ##
==========================================
+ Coverage   75.55%   75.72%   +0.16%
==========================================
  Files         365      365
  Lines       30702    30702
==========================================
+ Hits        23198    23248      +50
+ Misses       5879     5839      -40
+ Partials     1625     1615      -10

Impacted Files
Coverage Δ

embed/serve.go
69.62% <100%> (ø)
:arrow_up:

proxy/grpcproxy/register.go
69.44% <0%> (-11.12%)
:arrow_down:

rafthttp/msgappv2_codec.go
71.3% <0%> (-1.74%)
:arrow_down:

etcdserver/api/v3rpc/watch.go
80.85% <0%> (-1.71%)
:arrow_down:

pkg/schedule/schedule.go
81.69% <0%> (-1.41%)
:arrow_down:

clientv3/watch.go
95.83% <0%> (+0.24%)
:arrow_up:

etcdserver/server.go
79.83% <0%> (+0.46%)
:arrow_up:

internal/mvcc/watchable_store.go
86.71% <0%> (+0.73%)
:arrow_up:

etcdserver/v3_server.go
82.77% <0%> (+0.77%)
:arrow_up:

pkg/netutil/netutil.go
69.79% <0%> (+1.04%)
:arrow_up:

... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a0e17f6...ddc5a23. Read the comment docs.

lgtm. Defer to @xiang90
Thanks

Well since I am here I am going to refactor the curl testing and probably break it out over a few different files.

sgtm

@xiang90 PTAL, if this seems reasonable I would like to get this merged and add the breakout test files in a new commit as this is quite large as is. Will rebase once we are OK.

@xiang90 kind ping :)

@hexfusion i am fine with this large commit.

thank you


Device Monitoring: errors calling asset-library don't show as lambda errors

Aws Connected Device Framework Affected Module(s):

device-monitoring
I'm submitting a ...

[x] bug report
[ ] feature request

Description:

When there is an error calling into asset library (specifically an Internal Server Error, but i'm sure other errors), that does not translate into a lambda error.
Current behavior:

When an error happens, the lambda still returns a successful return code
Expected behavior:

When an error happens, the lambda should return an unsuccessful return code
Steps to reproduce:

Additional Information:

@aaronatbissell Do you happen to know the block of code that causes the error or the steps to replicate?
This seems like a fairly simple thing to fix, unless it is an issue caused by incorrect data in the asset library.


ENH PHP 8.1 compatibility

Issue https://github.com/silverstripe/silverstripe-framework/issues/10250

Builds are failing - can you please take a look?

Required a ^4 of webauthn, which is currently not php 8.1 compatible.  I've changed to 4.x-dev, it'll still fail until https://github.com/silverstripe/silverstripe-webauthn-authenticator/pulls is merged

Ah, might have been this older commit that was required - https://github.com/silverstripe/silverstripe-webauthn-authenticator/commit/371c5bbb08dc793ff46f053e5bf09a5d76ecbf10

Sweet. I'll add this to my merge list for this afternoon.


Provide verbose logging mode for Argo CLI

Is this a BUG REPORT or FEATURE REQUEST?:
FEATURE REQUEST

TLDR: Like kubectl's -v option, argo should provide verbose logging optionally so users can inspect the API requests being made.
We're running Argo workflows on EKS and we're currently experiencing significant performance issues. argo list operations often take > 1min to return or just timeout altogether. This is in a namespace with ~ 100 workflows total.
In release 2.4.0 the --chunk-size option from kubectl was put in argo, and that certainly has helped reduce timeouts but argo list operations are still extremely slow.
It'd be great to be able to see detail of what's happening in the time a list operation is being executed.

Environment:

Argo version: 2.4.1

imho this is almost a must in order to be able to interact with workflows from other apps - we need to be able to see the api calls made in order to be able to emulate them.

Inspecting requests can be done using Wireshark or similar. That said - in v2.4 - all API requests are basically calls to Kubernetes, e.g. argo list is same as kubectl get workflows. Might be enough to get your data.

Thanks for the reply.

e.g. argo list is same as kubectl get workflows.

Is this true for all argo CLI commands, and will it be true going forward? If the Argo CLI is making different requests to kubectl in any way, it will be useful to use --verbose to see exactly how they differ in the event of performance differences.
--verbose likely has other uses too.
(Can say though our performance problems with Argo have passed 🙂)

This should be fixed now, try:
argo --loglevel DEBUG submit ...

This produces debugging if you're running Argo Server:
GODEBUG=http2debug=2 argo submit examples/hello-world.yaml

I'm closing this as the solution meets the description. However, printing the requests does not happen (it is GRPC anyway - not HTTP+REST). Feel free to open a new issue.


std.experimental.allocator.make!T must either call GC.addRange or be marked @system

sludwig (@s-ludwig) reported this on 2016-12-18T18:19:08Z
Transfered from https://issues.dlang.org/show_bug.cgi?id=16982
CC List

ag0aep6g
greeenify
johannespfau (@jpf91)

Description
When creating objects that contain references, make() doesn't currently call GC.addRange(), which may lead to dangling references in case the GC frees such an unseen reference.

To fix this, make() must either be marked as @system as soon as an object with references is created, or preferably, it should call GC.addRange(). A vanilla version ("makeNoGC"?) could be added to be able to opt-out of this, but it should be very clear from its name that it is unsafe in a GC scenario, because this is a pitfall that is very easy to hit.

The same reasoning applies to makeArray().

greeenify commented on 2016-12-27T11:37:46Z
(I just closed https://issues.dlang.org/show_bug.cgi?id=15790 as a duplicate, but it contains a good example that illustrates this issue)

greeenify commented on 2016-12-27T11:36:10Z
*** Issue 15790 has been marked as a duplicate of this issue. ***

b2.temp commented on 2016-12-18T18:58:40Z
We could instead imagine a system based on introspection (discover data that should trigger addRange()) and annotations (trust data for for which addRange() should be called but wont). Depending on the result of the introspection, GC.addRange() would be called automatically.

See also:

-<EMAIL_ADDRESS>-<EMAIL_ADDRESS>-<EMAIL_ADDRESS>

sludwig (@s-ludwig) commented on 2016-12-18T19:55:16Z
Yeah, I used to simply `static if (std.traits.hasIndirections!T) GC.addRange(...);` in vibe.d's allocator module. Using annotations instead of controlling this at the call site is an interesting idea, although I can see applications for both approaches.

Using @nogc sounds good, too, although if that approach is taken, I'd propose to define @nogc as a UDA in object.d instead of adding a new special trait for detecting its presence.


Maintain an array of KML nodes not yet supported by Cesium

So the individual developer has access them and decide what to do. In my case, I want to support KML ScreenOverlay(s) but there is no support for them yet in Cesium.

Thanks @dwhipps!  This is an interesting idea.
If you haven't done so, can you please submit a Contributor License Agreement (CLA) so we can review this?  Let me know if you have any questions.

I submitted the CLA just now.

Thanks for the contribution, @dwhipps, we received your CLA.

@tfili do you want to review this?

Any decision on this? Really hoping it'll make it into the next release. (From what I can tell, the only downside is some extra memory usage.)

@dwhipps The issue with this solution is that the whole KML document ends up staying in memory because nodes have references to their parents nodes.
The better solution would be to have an event called unsupportedFeature or something like that. This event would get fired whenever an unsupported feature is encountered and would allow the app to handle it if it wants to without forcing everything to be held in memory. Also, we would need unit tests for the changes.

I agree with @tfili here, an event should work really well.  We'll be happy to take this for the next release if you can get those changes in.  Thanks @dwhipps!

Makes sense. I’ll implement this as an event.

On Jun 21, 2016, at 2:02 PM, Matthew Amato<EMAIL_ADDRESS>wrote:
I agree with @tfili https://github.com/tfili here, an event should work really well. We'll be happy to take this for the next release if you can get those changes in. Thanks @dwhipps https://github.com/dwhipps!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/AnalyticalGraphicsInc/cesium/pull/4021#issuecomment-227521774, or mute the thread https://github.com/notifications/unsubscribe/ACN_wLzoYQMb1QV9_DtrXXWbj9W-V-tRks5qOCcYgaJpZM4I1keh.

Ok, I've pushed a commit that raises an event when attempting to parse an unsupported node type. I added a spec test in there too.

Thanks Tom. The test was doing exactly what it was supposed to... failing! I had inadvertently added GroundOverlay, which in fact is already implemented in Cesium. Tests passing now.

@dwhipps The tests were broken again from your latest change.

Thanks @dwhipps. Looks good!

Thanks again for the contribution, @dwhipps!
@tfili can you please update CHANGES.md in master?

@tfili can you please update CHANGES.md in master?

Ah, I see #4050 now.


🖌 style: auth dark theme

Summary
I have added a dark theme for all the auth files

Change Type

[x] Style update

Checklist

[x] My code adheres to this project's style guidelines
[x] I have performed a self-review of my own code
[x] I have commented in any complex areas of my code
[x] I have made pertinent documentation changes
[x] My changes do not introduce new warnings
[x] I have written tests demonstrating that my changes are effective or that my feature works
[x] Local unit tests pass with my changes
[x] Any changes dependent on mine have been merged and published in downstream modules.

When #1754 is merged, I will fix any new style bugs that I cannot currently see, in a new PR

This is so cool. I just tried it and it looks amazing! I'm looking forward to using the login page at night without hurting my eyes! 😎


Remove experimental tag from includes

These parameters have been around for a while and are used by the CLI.
Slack discussion: https://cloudfoundry.slack.com/archives/CCRLX88DD/p1589818952216600?thread_ts=1589356454.154100&cid=CCRLX88DD

@Gerg this is now out of date due to a doc styling refactor


Allow generic custom objects for Cyber Observables

The recent changes made to the library allows parsing generic custom objects, but parse_observable() does not have the feature implemented.
https://github.com/oasis-open/cti-python-stix2/blob/master/stix2/v20/observables.py#L903
It also looks like we are not using allow_custom when we call it.
https://github.com/oasis-open/cti-python-stix2/blob/master/stix2/v20/observables.py#L36

I started work on this: https://github.com/oasis-open/cti-python-stix2/commit/b633fd37856ccd15257d7d6bae3223411a90e9ef
However, parsing fails even with allow_custom=True when parsing an Observed-Data object that contains a custom observable object, or parsing an observable object with a custom extension because _check_property has no knowledge of the allow_custom value that was provided when the parent object was created. I'm hesitant to add allow_custom to every object type; is there another way to get around this?

Random thought (might be crazy):
Maybe we're thinking about this backwards (specifically in ObservedData and _Observable, but also throughout the library). What if we always allow custom properties and types, but just set a flag (like _has_custom) on STIXBase objects that contain custom properties. Then we can check if that flag is True once the constructor returns, and raise an error or not based on whether allow_custom is True). Does that make sense? Does it sound crazy?

When you allow a custom SDO type, it gets returned as a python dictionary, not a _STIXBase object. So having a self.__has_custom in _STIXBase wouldn't work in those cases. We could create a GenericCustomObject class that subclasses STIXBase and overwrite methods like clean and _check_property, but that gets a little more complicated (and won't make it into a 1.0 release).


Add maxHealth ability (#237)

Added to towers baby-steps and tick-tick-boom

Codecov Report

Merging #238 into master will increase coverage by 0.02%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #238      +/-   ##
==========================================
+ Coverage   92.65%   92.67%   +0.02%
==========================================
  Files          92       93       +1
  Lines        1089     1092       +3
  Branches      165      165
==========================================
+ Hits         1009     1012       +3
  Misses         65       65
  Partials       15       15

Impacted Files
Coverage Δ

packages/warriorjs-abilities/src/maxHealth.js
100% <100%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 028975b...4ce9f8b. Read the comment docs.

Added maxHealth ability and included it in existing towers, updating the baby-steps tip to introduce it.
Let me know if I missed anything!

Thanks for the feedback!  The code base was a pleasure to work with.
Good catch, I'll push a fix shortly.

Thanks, and welcome to the Contributors!


Add support for storage->pubsub notifications

See the concepts doc and the registering object changes doc.
Pull requests target the 3956-storage-notifications branch
Steps:

[x] Add a BucketNotification class, mapping the notification resource.  Note that a BucketNotification may point to a topic which belongs to a different project than the one which owns its bucket. (#3957)
[x] Add Bucket.notification factory. returning new BucketNotification. (#3958)
[x] Add BucketNotification.create method, mapping notifications/insert.  Update the bucket's properties from the response (e.g., to get its server-generated ID).  Should we barf if the ID is already set? (#3966)
[x] Add BucketNotification.delete method, mapping notifications/delete.  Presumably barf if the ID is not set. (#3983)
[x] Add BucketNotification.exists method, mapping notifications/get onto boolean (False for NotFound).  Presumably barf if the ID is not set. (#3984)
[x] Add BucketNotification.reload method, mapping notifications/get,  Presumably barf if the ID is not set. (#3986)
[ ] Add Bucket.list_notifications method, mapping notifications/list.  Return an iterator of BucketNotification instances. (#3990)
[ ] Add a system test which exercises the new API wrapper methods.

Questions:

Do we need to support de novo creation of BucketNotfication instances w/ a user-supplied ID?  The spec says that the ID is generated during notifications/insert, which suggests not.
I assume that access to the new features does not depend on being able to import google.cloud.pubsub.  The counter-example would be that BucketNotification.create must verify the existence of its topic, and maybe create a new one.  Or does the back-end do that? The docs say that a topic is created if it does not exist.
There is no documented support for update / patch of existing notifications.  Is that intentional?
What are the intended semantics of passing {'payload_format`: 'NONE'}?  I don't think subscribers will find message with an empty payload useful, so I must be misunderstanding something.  Do the non-payload attributes provide enough information for the subscriber to do something useful?

/cc @lukesneeringer, perhaps to loop in the appropriate GCS back-end contact.

Another question:  do we need a "round trip" system test?  E.g., one which verifies that notifications arrive at the subscriber?  Or can we get away with just doing BucketNotification CRUD in the system tests?

Hi @tseaver,
Answers to questions:

Do we need to support de novo creation of BucketNotfication instances w/ a user-supplied ID? The spec says that the ID is generated during notifications/insert, which suggests not.

Indeed, GCS will generate IDs for you.

I assume that access to the new features does not depend on being able to import google.cloud.pubsub. The counter-example would be that BucketNotification.create must verify the existence of its topic, and maybe create a new one. Or does the back-end do that? The docs say that a topic is created if it does not exist.

GCS will not create a Cloud Pub/Sub topic for you if one does not exist. It will, though, check to see if the topic you specify exists and permissions have been appropriately configured for GCS to be able to publish topics to it, and the call to create a subscription will fail if either of these two requirements are not met. The "gsutil" command-line utility will, on the client side, create a topic if it does not exist.
I suggest a similar, single "subscribe()" command exist, that fetches the correct robot name, attaches appropriate permissions for it to a Pub/Sub topic (and maybe creates that topic), and then creates the subscription. Otherwise pretty much every user will have to write out that same chain of commands.

There is no documented support for update / patch of existing notifications. Is that intentional?

Yes. Subscriptions cannot be modified. They can only be deleted and created.

What are the intended semantics of passing {'payload_format`: 'NONE'}? I don't think subscribers will find message with an empty payload useful, so I must be misunderstanding something. Do the non-payload attributes provide enough information for the subscriber to do something useful?

The attributes provide enough information to accomplish most tasks. They tell you the event that happened (an object was created, an object was deleted), and they also tell you which object it happened to and when. The payload may be useful if you want to see, for instance, the object's type or its size, but it's not necessary in many cases.

Another question: do we need a "round trip" system test? E.g., one which verifies that notifications arrive at the subscriber? Or can we get away with just doing BucketNotification CRUD in the system tests?

A round trip test would be ideal, but getting back a successful "subscription created" message from GCS is a good sign that you've done things correctly.

@lukesneeringer Working on system tests for the feature, and running into permissions errors:
Traceback (most recent call last):
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/storage/tests/system.py", line 609, in test_notification_minimal
    self.assertTrue(notification.exists())
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/storage/.nox/sys-3-6/lib/python3.6/site-packages/test_utils/retry.py", line 95, in wrapped_function
    return to_wrap(*args, **kwargs)
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/storage/google/cloud/storage/notification.py", line 243, in create
    data=properties,
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/storage/.nox/sys-3-6/lib/python3.6/site-packages/google/cloud/_http.py", line 293, in api_request
    raise exceptions.from_http_response(response)
google.api.core.exceptions.Forbidden: 403 POST https://www.googleapis.com/storage/v1/b/notification-minimal-1505764404665/notificationConfigs: The service account<EMAIL_ADDRESS>does not have permission to publish messages to to the Cloud Pub/Sub topic '//pubsub.googleapis.com/projects/precise-truck-742/topics/notification-1505764383884', or that topic does not exist.

Does the feature require whitelisting?  The docs say that the notifications/insert API will create the topic if needed:  I changed the test to create it, without changing the error.

You cannot create a notification to publish to a given Cloud Pub/Sub topic until the following conditions are met:
1.) the topic exists.
2.) the GCS service account associated with the project that owns this bucket has permission to publish to that topic.
GCS enforces this constraint, which is what you're seeing. The error message is correct.

@BrandonY thanks for the followup.  Two points:

The docs for enabling notification say:

If you use a [TOPIC_NAME] that doesn't exist in your project, one is created for you.

I'm guessing that the described behavior is a feature of the gsutil notification create subcommand, rather than the notifications/insert API.

I see that error message even when I precreate the topic using the same service account as the one used to create the bucket, and which I'm using to attempt to invoke notifications/insert.  I can successfully verify that the account has the 'pubsub.topics.publish` permission on that topic, e.g.:

(pdb)  pp self.publisher_client.test_iam_permissions(self.topic_path, ['pubsub.topics.publish'])
permissions: "pubsub.topics.publish"

@tseaver If you're using gsutil to understand the behavior, you can actually log the requests made (as well as the payloads sent and received).

@dhermes I'm not running gsutil:  I'm reading docs.

@tseaver OK, no worries.

@BrandonY, @dhermes  See this gist.

Looking at the registering object changes doc vs. the notifications/insert doc, I have tried using the shorter form for the topic reference (projects/<project-id>/topics/<topid-id>), but still get the same 403 error (it even re-expands the path reference to the longer form).

Hi @tseaver,
GCS will not create a topic for you if one does not exist.
I believe you're looking at the instructions under the "gsutil" tab. It is true that the gsutil program will create a topic if you do not specify one. If you click the other tab header, JSON API, you should not see that text.
That is a bit confusing, though. I'll see about changing that bit of doc.

@BrandonY thanks for the follow-up.
I had decided already that auto-creation of the topic was not an API feature.  The system tests I am working on create the topic before creating the bucket:  they then attempts to create the notification, but fail with the 403.
The Ruby and PHP guys implementing the feature followed up on my query.  Maybe their phrasing makes the issue clearer?

@BrandonY 7d9ee94 adds the binding to the topic's IAM policy, required to enable creating the notifications.  Those system tests are now passing.

@BrandonY re: round-trip tests.  The concept docs say:

When you add a notification configuration, Cloud Storage may take up to 30 seconds to begin sending notifications associated with it.

That makes writing a non-flaky system test for the round-trip a good deal harder:  how can we know when the notification is active?

Yes, the service account for the project must have permission to publish to the topic before the call to create the subscription will succeed.
Writing a system test that obeys this constraint will require waiting 30 seconds after you create a notification before verifying that changes are being received by Cloud Pub/Sub. In practice, you can probably get away with 10-15 seconds.


Node 14

We ought to verify that the toolbox is running on Node 14, resolve any issues, and eventually move our pipeline to Node 14. Thanks @burdettadam for pointing this out.

Made the switch in #213


Usage of this project to another project as a template

Hello, I am planning to create a project as learn-julia containing every basic of Julia programming language. I was searching for a project which I can use as a template and follow the sequences. This project seems like an ideal one to me. Can I use this project structure and everything as a template and for following the sequence for my own project?
Please do let me know.

@FahimFBA sure, you may re-use the structure. Good luck with your new learn-julia project 👍🏻


Option to not check sha256 when reading .yaml file for pipeline

Currently when supplying a .yaml file for a pipeline release, the kabanero operator is still trying to check for sha256. It should at least have an option to stop checking for the sha256 when the content is a .yaml file.

@mchenggit - Is there a particular release this is needed by?

@marikaj123 I can work around it, but fixing it definitely helps improve pipeline development experience.

This is fixed by #627 and will be available in the first 0.9.0 alpha or rc release.


1.20.6 support

Currently crashing with 1.20.5 version (or maybe because of Vulkan Mod) would still love to give this mod a try

Closed by https://github.com/jaredlll08/no-shade/commit/5d75966d64dcd4e10786a7dc8d05e82dccdb0767