Updating the guidance for securing your Helm installation

@jbeda @anguslees @gabrtv @technosophos for eyeballs! anyone else, please wade in!

I gave it a quick read through, and mostly lgtm (with joe's suggestions).
Sorry for not being able to spend more time on it - I'm mid-transit to
Austin atm :)

Gus

On Mon, 4 Dec 2017 at 06:41 Joe Beda<EMAIL_ADDRESS>wrote:

@jbeda commented on this pull request.
Looks great @squillace https://github.com/squillace! I put some notes
in there. Feel free to ignore if you like.
In docs/securing_installation.md
https://github.com/kubernetes/helm/pull/3228#discussion_r154533923:

@@ -0,0 +1,108 @@
+# Securing your Helm Installation

+Helm is a powerful and flexible package-management and operations tool for Kubernetes. Installing it using the default installation command -- helm init -- quickly and easily installs Tiller, the server-side component with which Helm corresponds.
+
+This default installation applies no security configurations, however. It's completely appropriate to use this type of installation when you are working against a cluster with no or very few security concerns, such as local development with Minikube or with a cluster that is well-secured in a private network with no data-sharing or no other users or teams. If this is the case, then the default installation is fine, but remember: With great power comes great responsibility. Always use due diligence when deciding to use the default installation.
+
+## Who Needs Security Configurations?
+
+For the following types of clusters we strongly recommend that you apply the proper security configurations to Helm and Tiller to ensure the safety of the cluster, the data in it, and the network to which it is connected.
+
+- Clusters that are exposed to uncontrolled network environments
Perhaps add "or run untrusted workloads that have access to the network"?
In docs/securing_installation.md
https://github.com/kubernetes/helm/pull/3228#discussion_r154534023:

+Helm and Tiller are designed to install, remove, and modify logical applications that can contain many service interacting together. As a result, often its usefulness involves cluster-wide operations, which in a multitenant cluster means that great care must be taken with access to a cluster-wide Tiller installation to prevent improper activity.
+
+Specific users and teams -- developers, operators, system and network administrators -- will need their own portion of the cluster in which they can use Helm and Tiller without risking other portions of the cluster. This means using a Kubernetes cluster with RBAC enabled and Tiller configured to enforce them. For more information about using RBAC in Kubernetes, see Using RBAC Authorization.
+
+#### Tiller and User Permissions
+Tiller in its current form (2.7.x) does not provide a way to map user credentials to specific permissions within Kubernetes. When Tiller is running inside of the cluster, it operates with the permissions of its service account. If no service account name is supplied to Tiller, it runs with the default service account. This means that all Tiller operations on that server are executed using the Tiller pod's credentials and permissions.
+
+The properly limit what Tiller itself can do, the standard Kubernetes RBAC mechanisms must be attached to Tiller, including Roles and RoleBindings that place explicit limits on what things a Tiller instance can install, and where.
+
+This situation may change in the future. While the community has several methods that might address this, at the moment performing actions using the rights of the client, instead of the rights of Tiller, is contingent upon the outcome of the Pod Identity Working Group, which has taken on the task of solving the problem in a general way.
+
+
+### The Tiller gRPC Endpoint and TLS
+
+In the default installation the gRPC endpoint that Tiller offers is available inside the cluster (not external to the cluster) without authentication configuration applied. Without applying authentication, any process in the cluster can use the gRPC endpoint to perform operations inside the cluster. In a local or secured private cluster, this enables rapid usage and is normal. (From outside the cluster, Helm uses the normal authentication that Kubernetes supports to reach a gRPC endpoint.)
The gRPC endpoint may be available outside of the cluster but on the same
network depending on how networking is set up. When users use a network
config that bridges natively the the host network (for example, calico w/
BGP integration) the endpionts for gRPC will be accessible while the
service endpoint (ClusterIP) will not. This means it is still reachable.
In docs/securing_installation.md
https://github.com/kubernetes/helm/pull/3228#discussion_r154534053:

+Helm and Tiller are designed to install, remove, and modify logical applications that can contain many service interacting together. As a result, often its usefulness involves cluster-wide operations, which in a multitenant cluster means that great care must be taken with access to a cluster-wide Tiller installation to prevent improper activity.
+
+Specific users and teams -- developers, operators, system and network administrators -- will need their own portion of the cluster in which they can use Helm and Tiller without risking other portions of the cluster. This means using a Kubernetes cluster with RBAC enabled and Tiller configured to enforce them. For more information about using RBAC in Kubernetes, see Using RBAC Authorization.
+
+#### Tiller and User Permissions
+Tiller in its current form (2.7.x) does not provide a way to map user credentials to specific permissions within Kubernetes. When Tiller is running inside of the cluster, it operates with the permissions of its service account. If no service account name is supplied to Tiller, it runs with the default service account. This means that all Tiller operations on that server are executed using the Tiller pod's credentials and permissions.
+
+The properly limit what Tiller itself can do, the standard Kubernetes RBAC mechanisms must be attached to Tiller, including Roles and RoleBindings that place explicit limits on what things a Tiller instance can install, and where.
+
+This situation may change in the future. While the community has several methods that might address this, at the moment performing actions using the rights of the client, instead of the rights of Tiller, is contingent upon the outcome of the Pod Identity Working Group, which has taken on the task of solving the problem in a general way.
+
+
+### The Tiller gRPC Endpoint and TLS
+
+In the default installation the gRPC endpoint that Tiller offers is available inside the cluster (not external to the cluster) without authentication configuration applied. Without applying authentication, any process in the cluster can use the gRPC endpoint to perform operations inside the cluster. In a local or secured private cluster, this enables rapid usage and is normal. (From outside the cluster, Helm uses the normal authentication that Kubernetes supports to reach a gRPC endpoint.)
The line "from outside the cluster.." is a little misleading. I'd say
something like "When running outside the cluster the helm tool
authenticates through the Kubernetes API server to reach helm and as such
leverages existing Kuberenetes auth".
In docs/securing_installation.md
https://github.com/kubernetes/helm/pull/3228#discussion_r154534099:

+Specific users and teams -- developers, operators, system and network administrators -- will need their own portion of the cluster in which they can use Helm and Tiller without risking other portions of the cluster. This means using a Kubernetes cluster with RBAC enabled and Tiller configured to enforce them. For more information about using RBAC in Kubernetes, see Using RBAC Authorization.
+
+#### Tiller and User Permissions
+Tiller in its current form (2.7.x) does not provide a way to map user credentials to specific permissions within Kubernetes. When Tiller is running inside of the cluster, it operates with the permissions of its service account. If no service account name is supplied to Tiller, it runs with the default service account. This means that all Tiller operations on that server are executed using the Tiller pod's credentials and permissions.
+
+The properly limit what Tiller itself can do, the standard Kubernetes RBAC mechanisms must be attached to Tiller, including Roles and RoleBindings that place explicit limits on what things a Tiller instance can install, and where.
+
+This situation may change in the future. While the community has several methods that might address this, at the moment performing actions using the rights of the client, instead of the rights of Tiller, is contingent upon the outcome of the Pod Identity Working Group, which has taken on the task of solving the problem in a general way.
+
+
+### The Tiller gRPC Endpoint and TLS
+
+In the default installation the gRPC endpoint that Tiller offers is available inside the cluster (not external to the cluster) without authentication configuration applied. Without applying authentication, any process in the cluster can use the gRPC endpoint to perform operations inside the cluster. In a local or secured private cluster, this enables rapid usage and is normal. (From outside the cluster, Helm uses the normal authentication that Kubernetes supports to reach a gRPC endpoint.)
+
+Shared and production clusters -- for the most part -- should use Helm 2.3.0 at a minimum and configure TLS for the Tiller gRPC endpoints to ensure that within the cluster usage of the gRPC endpoint is only for the properly authenticated identity. Doing so enables any number of Tiller instances to be deployed in any number of namespaces and yet no unauthenticated usage of any gRPC endpoint is possible. Finally, usage of Helm with the --tiller-tls-verify option to enforce TLS in any communication with gRPC endpoints.
The minimum should be Helm 2.7.2 as TLS was pretty much busted before
that. See the 2.7.2 release notes:
https://github.com/kubernetes/helm/releases/tag/v2.7.2
In docs/securing_installation.md
https://github.com/kubernetes/helm/pull/3228#discussion_r154534129:

+Specific users and teams -- developers, operators, system and network administrators -- will need their own portion of the cluster in which they can use Helm and Tiller without risking other portions of the cluster. This means using a Kubernetes cluster with RBAC enabled and Tiller configured to enforce them. For more information about using RBAC in Kubernetes, see Using RBAC Authorization.
+
+#### Tiller and User Permissions
+Tiller in its current form (2.7.x) does not provide a way to map user credentials to specific permissions within Kubernetes. When Tiller is running inside of the cluster, it operates with the permissions of its service account. If no service account name is supplied to Tiller, it runs with the default service account. This means that all Tiller operations on that server are executed using the Tiller pod's credentials and permissions.
+
+The properly limit what Tiller itself can do, the standard Kubernetes RBAC mechanisms must be attached to Tiller, including Roles and RoleBindings that place explicit limits on what things a Tiller instance can install, and where.
+
+This situation may change in the future. While the community has several methods that might address this, at the moment performing actions using the rights of the client, instead of the rights of Tiller, is contingent upon the outcome of the Pod Identity Working Group, which has taken on the task of solving the problem in a general way.
+
+
+### The Tiller gRPC Endpoint and TLS
+
+In the default installation the gRPC endpoint that Tiller offers is available inside the cluster (not external to the cluster) without authentication configuration applied. Without applying authentication, any process in the cluster can use the gRPC endpoint to perform operations inside the cluster. In a local or secured private cluster, this enables rapid usage and is normal. (From outside the cluster, Helm uses the normal authentication that Kubernetes supports to reach a gRPC endpoint.)
+
+Shared and production clusters -- for the most part -- should use Helm 2.3.0 at a minimum and configure TLS for the Tiller gRPC endpoints to ensure that within the cluster usage of the gRPC endpoint is only for the properly authenticated identity. Doing so enables any number of Tiller instances to be deployed in any number of namespaces and yet no unauthenticated usage of any gRPC endpoint is possible. Finally, usage of Helm with the --tiller-tls-verify option to enforce TLS in any communication with gRPC endpoints.
It should be clear that each instance of Tiller should have its own TLS
configuration (including a separate CA).
Tiller currently doesn't look inside the certificate presented but just
verifies it is signed by the CA. If the same CA is used across tiller
instances it will allow users to talk to any of them.
In docs/securing_installation.md
https://github.com/kubernetes/helm/pull/3228#discussion_r154534211:

+For the proper steps to configure Tiller and use Helm properly with TLS configured, see Using SSL between Helm and Tiller.
+
+When Helm clients are connecting from outside of the cluster, the security between the Helm client and the API server is managed by Kubernetes itself. You may want to ensure that this link is secure. Note that if you are using the TLS configuration recommended above, not even the Kubernetes API server has access to the unencrypted messages between the client and Tiller.
+
+### Tiller's Release Information
+
+For historical reasons, Tiller stores its release information in ConfigMaps. We suggest changing the default to Secrets.
+
+Secrets are the Kubernetes accepted mechanism for saving configuration data that is considered sensitive. While secrets don't themselves offer many protections, Kubernetes cluster management software often treats them differently than other objects. Thus, we suggest using secrets to store releases.
+
+Enabling this feature currently requires setting the --storage=secret flag in the tiller-deploy deployment. This entails directly modifying the deployment or using helm init --override=..., as no helm init flag is currently available to do this for you. For more information, see Using --override.
+
+### Thinking about Charts
+
+Because of the relative longevity of Helm, the Helm chart ecosystem evolved without the immediate concern for cluster-wide control, and especially in the developer space this makes complete sense. However, charts are a kind of package that not only installs containers you may or may not have validated yourself, but it may also install into more than one namespace.
I don't have it handy right now (I want to say --dry-run --debug), but
there is a way to have the helm client dump the yaml that will be
installed. Paranoid users should use this to get a concrete idea of what is
going to happen before they install a chart. Might be good to note that
here.
In docs/securing_installation.md
https://github.com/kubernetes/helm/pull/3228#discussion_r154534232:

+### Thinking about Charts

+Because of the relative longevity of Helm, the Helm chart ecosystem evolved without the immediate concern for cluster-wide control, and especially in the developer space this makes complete sense. However, charts are a kind of package that not only installs containers you may or may not have validated yourself, but it may also install into more than one namespace.
+
+As with all shared software, in a controlled or shared environment you must validate all software you install yourself before you install it. If you have secured Tiller with TLS and have installed it with permissions to only one or a subset of namespaces, some charts may fail to install -- but in these environments, that is exactly what you want. If you need to use the chart, you may have to work with the creator or modify it yourself in order to use it securely in a mulitenant cluster with proper RBAC rules applied.
+
+### gRPC Tools and Secured Tiller Configurations
+
+Many very useful tools use the gRPC interface directly, and having been built against the default installation -- which provides cluster-wide access -- may fail once security configurations have been applied. RBAC policies are controlled by you or by the cluster operator, and either can be adjusted for the tool, or the tool can be configured to work properly within the contraints of specific RBAC policies applied to Tiller. The same may need to be done if the gRPC endpoint is secured: the tools need their own secure TLS configuration in order to use a specific Tiller instance. The combination of RBAC policies and a secured gRPC endpoint configured in conjunction with gRPC tools enables you to control your cluster environment as you should.
+
+## Best Practices for Securing Helm and Tiller
+
+The following guidelines reiterate the Best Practices for securing Helm and Tiller and using them correctly.
+
+1. Create a cluster with RBAC enabled
+2. Configure each Tiller gRPC to use TLS
I'd say "Configure each Tiller gRPC endpoint to use a separate TLS CA"
In docs/securing_installation.md
https://github.com/kubernetes/helm/pull/3228#discussion_r154534263:

+As with all shared software, in a controlled or shared environment you must validate all software you install yourself before you install it. If you have secured Tiller with TLS and have installed it with permissions to only one or a subset of namespaces, some charts may fail to install -- but in these environments, that is exactly what you want. If you need to use the chart, you may have to work with the creator or modify it yourself in order to use it securely in a mulitenant cluster with proper RBAC rules applied.
+
+### gRPC Tools and Secured Tiller Configurations
+
+Many very useful tools use the gRPC interface directly, and having been built against the default installation -- which provides cluster-wide access -- may fail once security configurations have been applied. RBAC policies are controlled by you or by the cluster operator, and either can be adjusted for the tool, or the tool can be configured to work properly within the contraints of specific RBAC policies applied to Tiller. The same may need to be done if the gRPC endpoint is secured: the tools need their own secure TLS configuration in order to use a specific Tiller instance. The combination of RBAC policies and a secured gRPC endpoint configured in conjunction with gRPC tools enables you to control your cluster environment as you should.
+
+## Best Practices for Securing Helm and Tiller
+
+The following guidelines reiterate the Best Practices for securing Helm and Tiller and using them correctly.
+
+1. Create a cluster with RBAC enabled
+2. Configure each Tiller gRPC to use TLS
+3. Release information should be a Kubernetes Secret
+4. Install one Tiller per user, team, or other organizational entity with the --service-account flag, Roles, and RoleBindings
+5. Use Helm with the --tiller-tls-verify flag to enforce verification
There are other flags necessary for using TLS. And a different set needs
to be used per tiller. I don't think there is a good outline on how to
juggle multiple tillers. That can come later but would help make this guide
realistic.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/helm/pull/3228#pullrequestreview-80710637,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AANr4G2_5X8Iioja9UlTqZ8GJx-Yixv5ks5s8vlzgaJpZM4Qz2A-
.
Message protected by MailGuard: e-mail anti-virus, anti-spam and content
filtering.
http://www.mailguard.com.au/mg
Report this message as spam
https://console.mailguard.com.au/ras/1ScQpGZIPf/3gF5luXi4it05u9x3VPKPB/0.2

This is all really great commentary; thanks for having at it. I just got off a flight, so I'll be going through this tonight to make modifications that reflect our current efforts here, and sketch out where we might improve the documentation going forward. As I said, I'm very much looking forward to iterating.

@bacongobbler alsoL: thanks for the provenance link. I'll catch up there.

Having left more comments above, I again thank the reviewers. I'll submit a revision tomorrow for merging, assuming that we are all committed to continuing some interactive iteration here. It's a fruitful area for work.

@bacongobbler @jbeda @anguslees I want to thank you for the comments. @bacongobbler, I've addressed the technical issues you highlighted, and have addressed the larger concerns in what seems like a good way for this week, but I want to assure you that I will submit a new PR to continue updating this documentation throughout this week and month.
I think that there are many ways to deploy and use Helm and Tiller, and some less common patterns will become more common as the tool is adopted for use by different people for different things. As a result, I think we'll be back here on these docs and others quite a bit. And that's a good thing for everyone, I think.
I'm requesting a merge on these bases, if you see your way clear. If there are additional concerns, please raise them and I'll address them immediately.


Meganav and Vorto

Hi,
Im trying to create a multilanguage site in umbraco (7.13.2) using Vorto. I also installed the meganav plugin, I have created a vorto datatype of meganav and added the following code in master template:
@{     var homeNode = Model.Content.Site().OfType<HomePage>();     @Html.Partial("ExampleNavigation", homeNode.MainNavigation) }
and i get the following error
System.InvalidOperationException: The model item passed into the dictionary is of type 'Our.Umbraco.Vorto.Models.VortoValue1[System.Collections.Generic.IEnumerable1[Cogworks.Meganav.Models.MeganavItem]]', but this dictionary requires a model item of type 'System.Collections.Generic.IEnumerable`1[Cogworks.Meganav.Models.MeganavItem]'.
That means that meganav is not working with Vorto?
George

Hey @geo782!
Thanks for using Meganav!
I haven't personally used Vorto and Meganav together, but I don't believe there is any reason why it wouldn't work.
From a quick look at the Vorto documentation / Github I think I have found your issue.
Normally when using Vorto the value of homeNode.MainNavigation will be VortoValue, rather than IEnumerable<MeganavItem> as Meganav expects. You should be able to get the correct value for the culture you need with the .GetValue() extension method. This Github thread may help explain: https://github.com/umco/umbraco-vorto/pull/107
If I'm correct, homeNode.MainNavigation.GetValue("en") (with "en" being whatever culture code you need) would return the correct value that Meganav expects.
Hope this makes sense!
Callum

Hello,
I am closing this issue due to inactivity.
If you still need help please don't hesitate to re-open it again or start a new thread!
Callum


Additional useful info

For the Asp.Net core section it would be useful to mention the following as it is not always included by default in Asp.Net core templates as witnessed by a few StackOverflow questions. Adding the using statement for Microsoft.Extensions.Configuration does not fix compile time errors on it's own:
Do not forget to add via DI:
public IConfiguration Configuration { get; }
public Startup(IConfiguration configuration) {     Configuration = configuration; }

Document details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: de9db02d-7310-1141-6aeb-539c02326a36
Version Independent ID: 90062036-aa01-3f73-dac9-b4a092245721
Content: Connection Strings - EF Core
Content Source: entity-framework/core/miscellaneous/connection-strings.md
Product: entity-framework
Technology: entity-framework-core
GitHub Login: @rowanmiller
Microsoft Alias: avickers

@chriswhitts Can you point to the templates involved?

EF Team Triage: Closing this issue as the requested additional details have not been provided and we have been unable to reproduce it.
BTW this is a canned response and may have info or details that do not directly apply to this particular issue. While we'd like to spend the time to uniquely address every incoming issue, we get a lot traffic on the EF projects and that is not practical. To ensure we maximize the time we have to work on fixing bugs, implementing new features, etc. we use canned responses for common triage decisions.


CopperheadOS compatibility status

Hello,
I'm on my way to try microg, but I got a few questions first. I start by saying I'm gonna try it on CopperheadOS. I've read it's quite easy to install it with the provided APK, has anyone had any experience with microg+copperheados already? Will I be able to use Google Maps application or is there any as-similar-as-possible alternative (in terms of reliability and efficiency)? Are social apps (e.g. Facebook, Whatsapp, Signal, ...) gonna work? and finally, I've got an LG G Watch R W110. Judging by what I could see in the wiki, there's no support whatsoever for Android Wear yet. Is it correct?
If there's something major you also feel like I should know before diving into this please let me know.
Thank you very much!

AFAIK, CopperheadOS does not include support for signature spoofing out of the box for security reasons.
For app support, there should not be any differences between Copperhead OS and other custom ROMs, so I expect all the apps you mentioned to be running. There are some apps not working, if you encounter problems, please report a bug here.
Android Wear support is not yet supported. There is experimental support to communicate with an Android Wear emulator, but no bluetooth support yet.


Add getOrCreate

Thanks for the PR! If you don't mind, could you provide the motivation/use case for this function?
Also, please update docs (readme) and add jsdocs.

Just checking in on this. @evantrimboli would you mind addressing the original comment? Thanks!

Ping @evantrimboli

Closing this due to lack of responses.
Please re-open if you wish :)


Question about working with MTurk developer Sandbox

Hi,
I have two questions regarding Reseval and I would really appreciate any help with them:

While try working with the MTurk Sandbox, I am succeeding to run the reseval.create but there is a weird behavior:
In the opened environment on AWS, I can see the application which I had opened, and it looks like that:

But, on the AWS mturk worker page, the whole HIT is looks like:

After accepting the request, I tried to click the link but I get this page:

I was wondering maybe the sandbox platform is not support with opening that kind of html files?

While using MUSHRA test, on the prescreen_questions, There is a wav file which I need to upload? If not, from where Reseval brings that file? for now, it looks like that on the application page in AWS:

Thank you very much,
Roi

For 1) the MTurk display looks correct. There will be a survey link at the end of the eval. Participants are instructed to copy-paste that link into MTurk. However, you've caught a bug that I have not formatted that URL correctly. I'll take a look at that today/tomorrow. The Heroku backend should work if you need to run something in the meantime (although it gave me some issues a few months ago which prompted the change to AWS).
For 2) you pass the directory to use to reseval.create. The directory should be formatted as described here.

Nevermind, this issue is still ongoing. Currently working on it.

Figured it out. The URL also needs to be prefixed with http://. Fixed in version 0.1.2.


Implements exercise solution

@tanyapowell Yes, this is related to that ticket, although we have yet to provide a review for the new session material Efua created

Looks good to me


Pipeline Job Support

Default support for pipeline jobs in JJB is a bit lacking.
Offhand the configs don't quite match:

pipelines render as “flow-definition”s rather than “project”s
Actions are used to track properties declared in the pipeline, only known after the fact. (Could hack a actions passthrough in JJM)
many standard properties use workflow-specific tags.

If JJM could resolve this similar to the RawJob extension, would make managing pipeline jobs with JJM much easier.

Duplicate of #12


run Gratipay 139

138

28 (22) account reviewed.

Droplet creation is taking a loooooooong time.

Backup taken and verified.

Droplet spun up and updated.

Script started.

Script ran for 14 minutes (0:14:44.253428).

Pinned https://twitter.com/Gratipay/status/560869396953190400.

Log downloaded and droplet destroyed.

MassPay done and posted back for 82 users.

Escrow shuffled.

Gratipay support base:

Logs uploaded in https://github.com/gratipay/logs/commit/ecfdfb730ac1f1f11c49e0ea33a8340f09609bb8.


[vtk] fix find lz4 hdf5 pegl

Describe the pull request

What does your PR fix? Fixes #15502

Basically it fixes LZ4, HDF5, PEGL import problems, allowing to use VTK / ITK with FindPackage.

Please ping me if this PR is ready for review.

Please ping me if this PR is ready for review.

Please ping me if this PR is ready for review.

unfortunately not yet. I didn't manage to fix correctly the initial problem (hdf5 double inclusion when using VTK + ITK)

Please ping me if this PR is ready for review.

unfortunately not yet. I didn't manage to fix correctly the initial problem (hdf5 double inclusion when using VTK + ITK)

Please ping me if this PR is ready for review.

unfortunately not yet. I didn't manage to fix correctly the initial problem (hdf5 double inclusion when using VTK + ITK)

For users who only use vtk, do you have any way to deal with this problem? I tried your changes, for the modification of lz4, the compilation did not report an error, but for hdf5 pegl did not seem to solve the problem.

Please ping me if this PR is ready for review.

unfortunately not yet. I didn't manage to fix correctly the initial problem (hdf5 double inclusion when using VTK + ITK)

For users who only use vtk, do you have any way to deal with this problem? I tried your changes, for the modification of lz4, the compilation did not report an error, but for hdf5 pegl did not seem to solve the problem.

Please ping me if this PR is ready for review.

unfortunately not yet. I didn't manage to fix correctly the initial problem (hdf5 double inclusion when using VTK + ITK)

For users who only use vtk, do you have any way to deal with this problem? I tried your changes, for the modification of lz4, the compilation did not report an error, but for hdf5 pegl did not seem to solve the problem.

Well, I removed the hdf5 patch as it breaks VTK build (as you noticed). I've tried many things with no acceptable solutions. I'm trying to remove the FindHDF5.cmake after build as a workaround for now...
If any of you have an idea, I'm taking it.

Please ping me if this PR is ready for review.

unfortunately not yet. I didn't manage to fix correctly the initial problem (hdf5 double inclusion when using VTK + ITK)

For users who only use vtk, do you have any way to deal with this problem? I tried your changes, for the modification of lz4, the compilation did not report an error, but for hdf5 pegl did not seem to solve the problem.

Well, I removed the hdf5 patch as it breaks VTK build (as you noticed). I've tried many things with no acceptable solutions. I'm trying to remove the FindHDF5.cmake after build as a workaround for now...
If any of you have an idea, I'm taking it.

I didn't manage to fix the issue with hdf5. So I propose this workaround (simply removing FindHDF5.cmake after build from VTK and use the regular config from HDF5 directly), which seems to be effective for us.
If anyone have a better solution, I'll take it.

I didn't manage to fix the issue with hdf5. So I propose this workaround (simply removing FindHDF5.cmake after build from VTK and use the regular config from HDF5 directly), which seems to be effective for us.
If anyone have a better solution, I'll take it.

Since you've closed the modify permission of other mantainers, please do the following steps:

update this branch from vcpkg master
rebuild vcpkg
use command ./vcpkg x-add-version --all
update changes

Since you've closed the modify permission of other mantainers, please do the following steps:

update this branch from vcpkg master
rebuild vcpkg
use command ./vcpkg x-add-version --all
update changes

@dweckmann Thank you very much for your contribution！
I tried your changes, I can successfully configure the VTK library.
During the test, a problem was found.
Normally, the library is only searched once, but in my development, the library is searched again in the subdirectory (due to design issues).
For other libraries, there is no problem with repeated search:
find_package(Qt5 REQUIRED Widgets)
find_package(Qt5 REQUIRED Widgets)
For the VTK library, an error message will be reported for repeated searches:
find_package(VTK REQUIRED)
find_package(VTK REQUIRED)
Searching for PEGTL Searching for PEGTL - found target taocpp::pegtl CMake Error at libwvl/3rdParty/scripts/buildsystems/vcpkg.cmake:489 (_add_library):   _add_library cannot create imported target "PEGTL::PEGTL" because another   target with the same name already exists. Call Stack (most recent call first):   libwvl/3rdParty/installed/x64-windows/share/vtk/FindPEGTL.cmake:26 (add_library)   libwvl/3rdParty/scripts/buildsystems/vcpkg.cmake:634 (_find_package)   libwvl/3rdParty/installed/x64-windows/share/vtk/VTK-vtk-module-find-packages.cmake:585 (find_package)   libwvl/3rdParty/installed/x64-windows/share/vtk/vtk-config.cmake:129 (include)   libwvl/3rdParty/scripts/buildsystems/vcpkg.cmake:634 (_find_package)   CMakeLists.txt:14 (find_package)
Do you have any suggestions for this situation? If finding the library twice is indeed abnormal behavior, I will try to adjust the project logic to solve it.

@dweckmann Thank you very much for your contribution！
I tried your changes, I can successfully configure the VTK library.
During the test, a problem was found.
Normally, the library is only searched once, but in my development, the library is searched again in the subdirectory (due to design issues).
For other libraries, there is no problem with repeated search:
find_package(Qt5 REQUIRED Widgets)
find_package(Qt5 REQUIRED Widgets)
For the VTK library, an error message will be reported for repeated searches:
find_package(VTK REQUIRED)
find_package(VTK REQUIRED)
Searching for PEGTL Searching for PEGTL - found target taocpp::pegtl CMake Error at libwvl/3rdParty/scripts/buildsystems/vcpkg.cmake:489 (_add_library):   _add_library cannot create imported target "PEGTL::PEGTL" because another   target with the same name already exists. Call Stack (most recent call first):   libwvl/3rdParty/installed/x64-windows/share/vtk/FindPEGTL.cmake:26 (add_library)   libwvl/3rdParty/scripts/buildsystems/vcpkg.cmake:634 (_find_package)   libwvl/3rdParty/installed/x64-windows/share/vtk/VTK-vtk-module-find-packages.cmake:585 (find_package)   libwvl/3rdParty/installed/x64-windows/share/vtk/vtk-config.cmake:129 (include)   libwvl/3rdParty/scripts/buildsystems/vcpkg.cmake:634 (_find_package)   CMakeLists.txt:14 (find_package)
Do you have any suggestions for this situation? If finding the library twice is indeed abnormal behavior, I will try to adjust the project logic to solve it.

Do you have any suggestions for this situation? If finding the library twice is indeed abnormal behavior, I will try to adjust the project logic to solve it.

No sorry, VTK cmake files are too complex for my basic knowledge about it. I've tried to tweak a bit this part, but either my project won't build, either VTK won't.

Do you have any suggestions for this situation? If finding the library twice is indeed abnormal behavior, I will try to adjust the project logic to solve it.

No sorry, VTK cmake files are too complex for my basic knowledge about it. I've tried to tweak a bit this part, but either my project won't build, either VTK won't.

Do you have any suggestions for this situation? If finding the library twice is indeed abnormal behavior, I will try to adjust the project logic to solve it.

All find modules should have a check if(NOT TARGET <targetname>) and only create the target if it does not exists. Maybe that if is missing here

Do you have any suggestions for this situation? If finding the library twice is indeed abnormal behavior, I will try to adjust the project logic to solve it.

All find modules should have a check if(NOT TARGET <targetname>) and only create the target if it does not exists. Maybe that if is missing here

Same for this one, you'll have to rebase on upstream/master.

Same for this one, you'll have to rebase on upstream/master.

LGTM, @Neumann-A  do you have any other questions?

LGTM, @Neumann-A  do you have any other questions?

no

no

Can you please merge this branch from master and re-generate the version record?
Thanks,

Can you please merge this branch from master and re-generate the version record?
Thanks,

Can you please merge this branch from master and re-generate the version record?
Thanks,

Done !
Thank you for your time.

Can you please merge this branch from master and re-generate the version record?
Thanks,

Done !
Thank you for your time.

@dweckmann Seems you need to update the version record again.

@dweckmann Seems you need to update the version record again.

Will rerun the pipeline test after all test finished.

Will rerun the pipeline test after all test finished.

/azp run

(Rerunning due to replacing the macos vms)

Please add add paraview:x64-windows-static-md=fail to scripts\ci.baseline.txt.

Please add add paraview:x64-windows-static-md=fail to scripts\ci.baseline.txt.
The qt executable run failed, not related to this PR:

@JackBoosY: Why? This looks like the same issue marble had in another PR. Maybe there is some issue with the newer CMake version?

@Neumann-A So can you make a new PR to fix that?

Can you please merge to master?

Ping @dweckmann for response.

Ping @dweckmann for response.

Sorry, I don't have time to spend on this anymore as we are focused on other subjects right now. I really wish I could help further, but I cannot make any promise, sorry again. Feel free to close the PR or re-author it.

If you want to continue this PR, please reopen.

@dweckmann Is still work for this PR? Please reply me

@dweckmann Is still work for this PR? Please reply me

@FrankXie05 The last time I checked, only the patch for lz4 in VTK is still relevant. I guess you have to strip all other patch and only keep that.
Feel free to take this pr as basis and reopen.

@dweckmann Thanks for your reply, I will submit a PR  to verify it. :)


Fix variable reassignment type inference

Make sure the following simple test pass, with full type inference
let x = 1;
let y = x;
let z = y;
z

Done


Support get_backend_for_uri_with_options in Read Path

Description
Goal: I am trying to read a Delta Table from a non-AWS S3 object store (like minio). I seem to be able to read the table metadata but then when reading data I get an error that seems to indicate that my endpoint_override is being ignored and requests are sent to aws instead.
My code looks like this:
from deltalake import DeltaTable
storage_options = {"AWS_ACCESS_KEY_ID": ACCESS_KEY, "AWS_SECRET_ACCESS_KEY":SECRET_KEY, "AWS_ENDPOINT_URL": ENDPOINT_URL, "AWS_REGION": 'us-east-1'}
dt = DeltaTable("s3://warehouse/tpcds_sf1000_dlt/item/", storage_options=storage_options)
print(dt.dt.pyarrow_schema()) # SUCCESS
print(dt.to_pandas().head()) # FAILS

In looking through the code, I wonder if the line here needs to use the "with_options" variant:
https://github.com/delta-io/delta-rs/blob/d10b428f15906dd74f5d33cf592288cdf7dd1e66/python/src/lib.rs#L485
in order to pass through my configs?
Use Case
Delta on S3 object stores like Minio, Swift, FlashBlade, Vast, ECS, etc.
Related Issue(s)

Thanks for reporting this! Our current API is problematic in that we can't retrieve the storage backend once it's been created in the DeltaTable. We'll need to refactor a bit, probably changing the storage backend to use an Arc, instead of an owned Box pointer.
For now, the workaround is to explicitly pass a configured PyArrow filesystem:
from deltalake import DeltaTable
from pyarrow.fs import S3FileSystem

storage_options = {
    "AWS_ACCESS_KEY_ID": ACCESS_KEY,
    "AWS_SECRET_ACCESS_KEY":SECRET_KEY,
    "AWS_ENDPOINT_URL": ENDPOINT_URL,
    "AWS_REGION": 'us-east-1'
}
dt = DeltaTable("s3://warehouse/tpcds_sf1000_dlt/item/", storage_options=storage_options)

pa_fs = S3FileSystem(
    access_key = storage_options["AWS_ACCESS_KEY_ID"],
    secret_key = storage_options["AWS_SECRET_ACCESS_KEY"],
    endpoint_override = storage_options["AWS_ENDPOINT_URL"],
    region = storage_options["AWS_REGION"],
)
print(dt.dt.pyarrow_schema()) # SUCCESS
print(dt.to_pandas(filesystem=pa_fs).head()) # This *should* work

thanks @wjones127 for the quick response.  This helps me get a little further but I think I then run afoul of differing path expectations. After making the change, I get the following error:
pyarrow.lib.ArrowInvalid: Expected an S3 object path of the form 'bucket/key...', got a URI: 's3://warehouse/tpcds_sf1000_dlt/item/part-00000-1805aaca-4959-40a2-87b1-c42385e763e8-c000.snappy.parquet'

I think the problem is coming from here: https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/s3fs.cc#L364
Where here it's expecting a "bucket/key" path without a URI (i.e., no "s3://"), but I could be wrong.
Am I still missing something or is this maybe just not quite functional yet?
If there's any other issues tracking the work needed, I'd be happy to take a look and see if I can't contribute.

Am I still missing something or is this maybe just not quite functional yet?

Darn. I guess that workaround doesn't work. Thanks for reporting that failure too.
We don't yet have integration testing with object stores, but that's high priority in the next few months.

If there's any other issues tracking the work needed, I'd be happy to take a look and see if I can't contribute.

Fixing #696 should make the workaround function. Fixing #690 should make this work without the workaround. You are welcome to submit a PR on either!

Whilst working on #632 I was moving the storage backend behind an Arc as well, so I opened a quick PR with just that included (#697). This will not yet fix the issue, but prepares the rust side for the python side to use a shareble reference to the store, and avoid re-creating it.

@joshuarobinson with the release of 0.6.1 we now - whenever possible - reuse the same backend store also in the read path. Could you check if that works for you?

Hi @roeap
I've just tried again with 0.6.2 and the issue is now fixed!
fyi, manually specifying the pyarrow.filesystem doesn't work, but letting Delta take care of it behind the scenes does work (and is the better option, IMO)


Release CI Improvements

This PR improves our CI in a few ways:

When pushing to main, semantic-release is run in dry-run mode, and artifacts are not pushed
When manually running in main, the above steps do occur

this is just safer
and allows for editorial changes to the various CHANGELOG.md files

Releases now include committing docs to the rdkcentral/firebolt repository, which is wired up to rdkcentral.github.io/firebolt

:tada: This PR is included in version 0.12.0-next.6 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 0.12.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Unable to compile with Maven 2?

Hi,
I've tried this with the latest release, and the master branch. I'm getting this error when compiling with maven 2:
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Error resolving version for 'net.alchim31.maven:scala-maven-plugin':      Plugin requires Maven version 3.0
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8 seconds
[INFO] Finished at: Tue Sep 03 09:48:03 EDT 2013
[INFO] Final Memory: 7M/75M
[INFO] ------------------------------------------------------------------------`

Here's my maven version info:
Apache Maven 2.2.1 (rdebian-8)
Java version: 1.7.0_25
Java home: /usr/lib/jvm/java-7-openjdk-i386/jre
Default locale: en_US, platform encoding: ANSI_X3.4-1968
OS name: "linux" version: "3.8.0-29-generic" arch: "i386" Family: "unix"

I tried with the most recent version of Maven (3.1.0) and was able to compile successfully. So, maybe the requirements should be updated in the documentation?

I think depending on maven 3 is intentional. I'll change the documentation.

i encountered network issue while compiling which  get stucked in the Downloading, can you give me some suggestions to accomplish the compile?
Downloading from info-bliki-repository: http://gwtwiki.googlecode.com/svn/maven-repository/org/json4s/json4s-ast_2.12/3.5.3/json4s-ast_2.12-3.5.3.pom


Calendar event is not recognised by iPhone

I generate a QRCode Calendar event like this:
QRCode::calendar($appointment_start, $appointment_end, $summary, $description, $location)->setOutfile(storage_path('app/public/images/qrcode/test.png'))->png();
It read well by my Samsung android phone but on an iPhone it just said cannot read data. Anyone has same problem and any advice to make it work? Thank you!

Find out why, it is due to QR_CalendarEvent.php file had VCALENDAR outter wrapped VEVENT which is no longer valid.


Testing the app

Hi again 🙂
I did not find any documentation on how to test the app. Although it has some tests, actually the Modal spec was the only one I found, I have no clue on how to run and debug specific parts of the app, especially the ones I'm implementing & customizing in the src.
Do you have any recommended way of testing specific features? Perhaps it would be great to at least describe it in the docs or put some references.

Since it is a vue application most of the tests can be done with vue devtools. The sync parts need to be checked in magento directly


short-term solvers to add

GCSpy (ongoing)
Exact (ongoing)
RC2 (see pull request)

also:

MaxHS from Helsinki (their side is ready, maybe not on pip?)
pySMT

It would be nice to have Haddock as an accessible solver (c++, Hybrid CP + MDDs) on cpmpy, since we have no MDD based solver yet


docs: explain default of --min-py-version

After our earlier discussion on #230 I now understood why the default is that way and why it makes sense to update require_python.
To avoid this problem in the future, I suggest to add a explanation to the README about the default behavior.
I hope this matches the conses reached on #207, where the mention of the specific default version was removed to reduce maintenance burdon

no thanks, you can check --help

sorry for the maybe stupid question, but how does checking --help do that?
running setup-cfg-fmt --version  on a v2.5.0 install results in:
usage: setup-cfg-fmt [-h] [--include-version-classifiers] [--min-py-version MIN_PY_VERSION] [--max-py-version MAX_PY_VERSION] [filenames ...]

positional arguments:
  filenames

options:
  -h, --help            show this help message and exit
  --include-version-classifiers
  --min-py-version MIN_PY_VERSION
  --max-py-version MAX_PY_VERSION


Add an event filter with the resource.DesiredStateChanged predicate to filter status updates out

Description of your changes

Depends on: https://github.com/upbound/upjet/pull/241
This PR adds an event filter using the resource.DesiredStateChanged to the generated resource controllers so that updates to only labels, annotations (Crossplane's create pending/failed annotations will be ignored) and to the spec will result in reconcile requests. The relevant upjet PR is https://github.com/upbound/upjet/pull/231.
It also adds support for granular management policies and the spec.initProvider API to allow the specification of the configuration parameters that will only be used during a create call. Please refer to the relevant upjet PRs: https://github.com/upbound/upjet/pull/224, https://github.com/upbound/upjet/pull/237
It also introduces a change in the external client that reconciles the MRs of this provider so that the external client now explicitly re-queues a reconcile request if a shared provider has expired. Please refer to the relevant upjet PR for more details: https://github.com/upbound/upjet/pull/241
I have:

[x] Run make reviewable to ensure this PR is ready for review.

How has this code been tested

/test-examples="examples/applications/application.yaml"

/test-examples="examples/applications/application.yaml"


make sure downloader image is present and called non short name

Addresses: https://github.com/containers/ai-lab-recipes/issues/592
Please review @n1hility @rhatdan
cc @cooktheryan @prarit
Change: model downloader is not present as well as short name resolution (not sure if this is a problem yet until it becomes available)

I am working on a similar fix for this.  Want to add --pull never to the podman run commands.

I was putting the common code into ../common/

Ive both tagged this as local instead of latest as well as add the pull never

I now like mine better, lets continue on that one.


CORS errors when loading assets

Is there any way to disable CORS in the webview?
To avoid an X-Y problem: I am loading local JS/HTML files with a custom protocol wry://. These JS files then talk to a backend service that I control.
The problem is, neither setting Access-Control-Allow-Origin: * nor Access-Control-Allow-Origin: wry:// work on the server, because it seems like the webview hardcodes https and http URL as the only kind of CORS origin. Can CORS just be turned off completely?
Or is there a way of loading local JS/HTML files with a host that can be matched against by Access-Control-Allow-Origin?

If we just talking about CORS, I think it won't be possible.
Because platform like macOS doesn't support disabling it.
If you just want to load files from a host server, I suppose just load the URL with http://localhost should work.

I see. The files are in the filesystem, so I'd have to hack together a localhost listener to serve them...

Custom protocol is usually used for loading assets within a certain data directory.
It's also possible to do something like file://.
If you can all load them with it, I don't think you need a backend server.
But this is the first time, I encounter CORS issue in wry. Perhaps @lucasfernog and @lemarier could also give some advice?

This whole issue is because CORS doesn't support custom protocols.

In macOS there is a private API, but not sure it's something we want to include.
https://trac.webkit.org/browser/trunk/Source/WebKit/mac/WebView/WebPreferences.mm?rev=111350#L1011

Looks like IONIC had to boot a local HTTP server :/
https://docs.google.com/document/d/19VQ-n7hGr9IDPPstQqU8_8WgqUh7R6sgQfL2neoT-Xw/edit

@nullchinchilla We would like to look into this issue. Could you share your server setup and minimum example with wry?
Anyone with similar problem is welcome to share the mcve.

Here is a minimal test case: https://github.com/raguay/TestIt
This example shows the problem I'm having with CORS.

BTW: I'm on a macpro M1 system with the latest Big Sur OS.

Well, my opinion here is that CORS exists in browsers to prevent unauthorised access. If you are using something like window.fetch, I would expect it not to work if your service requires it (e.g. using helmet within some kind of express server). My recommendation is to use tauri's http API - which is not bound by the same limitations of CORS in a browser-like environment.

just use the http api from tauri here:
function getMessage() {
    		fetch("http://localhost:9878/api/", {
    			method: "get",
    			headers: { "Content-type": "application/json" }
    		}).then(data => {
    			return data.text();
    		}).then(data => {
    			$$invalidate(0, message = data);
    			console.log(data);
    		});
    	}
    	```

Okay, that does work.  I didn't know about that API. I guess I need to study the API more. Thanks for the pointer.

Closing this with the solution above. Note that CORS can still be solved by backend servers handle it properly.
We should add documents about this on the HTTP endpoint API.


A couple of minor issues when running from WSL

Issue Type: Bug
I'm excited to find when I run code . in WSL on Windows 11 it runs code in WSL and not WIndows with remote debugging. However a few of points

default integrated shell is Powershell not WSL Ubuntu (say) and I can't see how to change for WSL on windows (not Linux)
Updates to the view like type script errors seem to take a long time

Thanks for all the excellent work.
VS Code version: Code 1.63.2 (899d46d82c4c95423fb7e10e68eba52050e30ba3, 2021-12-15T09:40:02.816Z)
OS version: Windows_NT x64 10.0.22000
Restricted Mode: No

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz (12 x 2592)

GPU Status
2d_canvas: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: enabledopengl: enabled_onrasterization: enabledskia_renderer: enabled_onvideo_decode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
15.71GB (4.97GB free)

Process Argv
. --crash-reporter-id d4f7b7d4-f906-432a-a781-81f25dff1b73

Screen Reader
no

VM
0%

Extensions (4)

Extension
Author (truncated)
Version

astro-vscode
ast
0.8.5

vscode-markdownlint
Dav
0.45.0

vscode-versionlens
pfl
1.0.9

svelte-vscode
sve
105.8.0

A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vstes627:30244334
pythontb:30283811
pythonptprofiler:30281270
vsdfh931cf:30280410
vshan820:30294714
vstes263:30335439
vscorecescf:30384386
pythondataviewer:30285071
vscod805cf:30301675
pythonvspyt200:30340761
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593:30376534
pythonvs932:30410667
vscop804:30404766
vs360:30404995
vsrem710cf:30416617
py55gd98:30411514
vscexrecpromp3t1:30407762

default integrated shell is Powershell not WSL Ubuntu

This depends on the type of window. In a WSL window (as seen in the lower left corner) you get a Linux terminal.
In a local window, it's Powershell by default.
The idea is that when working in WSL, you open a WSL window, with terminal and the all extensions running in Linux.

Updates to the view like type script errors seem to take a long time

Please add more details and steps to reproduce, e.g. a screencast. Are you working on a /mnt/... drive?

@aeschli Thanks, I think my expectations may be wrong and I;m not sure how to check which version of code is running. Basically I very confused about what context code is running in.

Open WSL in terminal
go to to a WSL folder - eg ~/project
run code .

The VS code i get was running a clean new config (eg no extensions as set in the Pre Windows 11 launched WSL version using code .) As this is WSLg on windows 11 I assumed the code was now running using WSL context and X11 windows (or whatever Linux uses these days and is now supported for WSL GUI apps) and not the WIn64 (?) version.
However it use powershell as the default I hit a number of problems including those mentioned and any attempt to delete files tries to save to recycle which fails and it offers to permanently delete.
To add to the confusion I tried and the remote WSL extension as I had to used before windows 11 upgrade  and all the problems went away.
All very confusing to be honest.
Do we run the windows version of code or WSLg now?

Ah, I see. You don't have the Remote-WSL extension installed. In your setup all tools run on the Windows side, just using the WSL file system.
Remote WSL is very likley what you want: https://code.visualstudio.com/docs/remote/wsl
You should have gotten a recommendation for the extension. Maybe you clicked it away?


Gatewayd unique identifier

Today we currently use the node_pub_key as the identifier of a Gatewayd instance. But it is possible for multiple gatewayd instances to share a single lightning node (i.e in the LSP case like Zeebedee). switch_active_gateway should use this identifier instead of the public key of the lightning node.
We can probably use the public key of the redeem_key as this identifier.

We can probably use the public key of the redeem_key as this identifier.

Yes, that not being the case seems like a bug to me.

I was mistaken, we actually do already use the redeem_key as an identifier:
pub fn to_gateway_registration_info(         &self,         route_hints: Vec<RouteHint>,         time_to_live: Duration,         api: Url,     ) -> LightningGateway {         LightningGateway {             mint_channel_id: self.mint_channel_id,             mint_pub_key: self.redeem_key.x_only_public_key().0,             node_pub_key: self.node_pub_key,             api,             route_hints,             valid_until: fedimint_core::time::now() + time_to_live,             fees: self.fees,         }     }
I thought mint_pub_key was identifying the federation, not the gateway. This should probably just be renamed.


"Import could not be resolved" after installing the package

Type: Bug
The problem is that pylance continues to report error even after package installation.
The sequence of action was as on the picture.  Steps to reproduce:

install anaconda
open attached notebook, run it under base environment. You will get an error in the last cell because the package is missing
pip install gap-stat
run the last cell. it works.  but pylance is still reporting the problem (which does not exist any more)

The notebook is attached (rename to ipynb):
kmeans.ipynb.txt

Extension version: 2024.6.1
VS Code version: Code 1.90.2 (5437499feb04f7a586f677b155b039bc2b3669eb, 2024-06-18T22:33:48.698Z)
OS version: Linux x64 6.4.6-060406-generic
Modes:

System Info

Item
Value

CPUs
Intel(R) Xeon(R) CPU E5-2696 v4 @ 2.20GHz (88 x 1197)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okskia_graphite: disabled_offvideo_decode: enabledvideo_encode: disabled_softwarevulkan: disabled_offwebgl: enabledwebgl2: enabledwebgpu: disabled_off

Load (avg)
1, 2, 1

Memory (System)
251.76GB (241.06GB free)

Process Argv
--crash-reporter-id 27d42247-63fb-4d9e-9cb0-87d9974843dc

Screen Reader
no

VM
50%

DESKTOP_SESSION
ubuntu-xorg

XDG_CURRENT_DESKTOP
Unity

XDG_SESSION_DESKTOP
ubuntu-xorg

XDG_SESSION_TYPE
x11

A/B Experiments
vsliv368:30146709
vspor879:30202332
vspor708:30202333
vspor363:30204092
vscoreces:30445986
vscod805:30301674
binariesv615:30325510
vsaa593:30376534
py29gd2263:31024239
c4g48928:30535728
azure-dev_surveyone:30548225
2i9eh265:30646982
962ge761:30959799
pythongtdpath:30769146
welcomedialog:30910333
pythonnoceb:30805159
asynctok:30898717
pythonregdiag2:30936856
pythonmypyd1:30879173
h48ei257:31000450
pythontbext0:30879054
accentitlementst:30995554
dsvsc016:30899300
dsvsc017:30899301
dsvsc018:30899302
cppperfnew:31000557
dsvsc020:30976470
pythonait:31006305
jchc7451:31067544
chatpanelt:31048053
dsvsc021:30996838
9c06g630:31013171
pythoncenvpt:31062603
a69g1124:31058053
dvdeprecation:31068756
dwnewjupyter:31046869
2f103344:31071589
legacy_priority:31077528

will it go away if you close and re-open the notebook? or if you reload vsocde?

yes, it goes away if I close and open the notebook.
But this is not an option and here is why:   I have Spark context running in this notebook, and many initialized pandas dataframes, which are quite hard to calculate. And if I close the notebook and re-open it again, I will have to re-initialize the Spark app, recalculate of my dataframes, etc., which is a hard routine and takes a lot of time. Therefore, keeping a notebook running (even with false errors from pylance) is better than closing and re-opening of it.
But anyway, it would be good to have a command  in Ctrl + Shift + P   named  "Pylance:  restart scanning of files/notebooks"   which would just do the job

I don't know how the guys from PyCharm team did it, but they re-scan of all installed packages of the current environment just as soon as the new package arrives.  It would be good to have this here too.

I see that this bug in "needs repro"  status. I don't know if you noticed this: I provided a notebook file in my  original message, together with steps to reproduce.

This could be this issue here:
https://github.com/microsoft/vscode/issues/199800
The root cause of the problem is you change your kernel and we (Pylance) never get told about it. So we think the current interpreter is a different one.
That's why Heejae asked if it went away if you closed and opened the notebook, we get the initial interpreter correctly then.

The log from the Pylance Language Server should tell us which interpreter we think is being used. (If it's the VS code issue I mentioned, it won't match the kernel).

Just following up on the issue. Does it still repro for you? If so, could you please provide more logs as outlined in our troubleshooting guide here? Thanks.

 It would show if we actually are using the correct interpreter for the kernel selected. My guess is that this is the VS code issue which is caused by the kernel being selected having the same python version.

logs.txt
And here is how I reproduced this situation and recorded those logs:   https://www.youtube.com/watch?v=AjeRVlLrNcA
As you may have noticed on my video,  there are no actions performed (based on the logs) after I installed that  missing package.  If it was, I would see some messages in my log  just AFTER I installed it.

From watching the video and looking at the log it seems like it's not the VS code issue. It's more like we're not getting a file system events when you install the gap_statistic module.
According to the log we should be watching these folders here:
2024-07-18 19:35:58.602 [info] [Info  - 7:35:58 PM] (151781) Adding fs watcher for library directories:
 file:///opt/anaconda3/lib/python3.10
file:///opt/anaconda3/lib/python3.10/lib-dynload
file:///opt/anaconda3/lib/python3.10/site-packages
file:///opt/anaconda3/lib/python3.10/site-packages/PyQt5_sip-12.11.0-py3.10-linux-x86_64.egg
file:///opt/anaconda3/lib/python3.10/site-packages/mpmath-1.2.1-py3.10.egg

If you print out gap_statistics.__file__, where does it say it comes from?

It will be in
/opt/anaconda3/lib/python3.10/site-packages/gap_statistic/init.py

If you touch that file (or maybe just add a file in that directory), do you get any fs watcher output in the Python Language Server?
For example, I get this is I add a file in my site-packages folder:
2024-07-18 10:45:43.131 [info] [Info  - 10:45:43 AM] (26484) LibraryFile: Received fs event 'add' for path 'c:\Users\rchiodo\AppData\Local\anaconda3\envs\Pesnet\Lib\foo.py'
2024-07-18 10:45:43.131 [info] [Info  - 10:45:43 AM] (26484) LibraryFile: Received fs event 'add' for path 'c:\Users\rchiodo\AppData\Local\anaconda3\envs\Pesnet\Lib\foo.py'
2024-07-18 10:45:43.131 [info] [Info  - 10:45:43 AM] (26484) LibraryFile: Received fs event 'add' for path 'c:\Users\rchiodo\AppData\Local\anaconda3\envs\Pesnet\Lib\foo.py'
2024-07-18 10:45:43.131 [info] [Info  - 10:45:43 AM] (26484) LibraryFile: Received fs event 'add' for path 'c:\Users\rchiodo\AppData\Local\anaconda3\envs\Pesnet\Lib\foo.py'

I tried to edit the file, add lines, save it, creating a new file in that directory - nothing helped:  https://youtu.be/P7cBw8YPSTE
I know for sure that I launched multiple times  the command
chmod a+rw -R /opt/anaconda3

So there are permissions to read/write from that directory. But the question is:  are there permissions for tracking file changes? How to find out in Linux?  Could you give some script/code which tracks for changes, so that I will run it and tell it to you?

Sorry I'm entirely sure how we track changes. It's part of the VS code API, we just tell it which files to track. I believe it uses something called https://www.npmjs.com/package/chokidar. You could potentially write a javascript file that uses chokidar as in the npm examples and then try editing a file that's being watched?

I reproduced a small app in js (with usage of chokidar) and it works just fine. But there is one catch.
The code of my js file:
// Import the chokidar package
const chokidar = require('chokidar');

// Define the directory you want to watch
const directoryToWatch = '/opt/anaconda3/lib/python3.10/site-packages/gap_statistic';

// Initialize the watcher
const watcher = chokidar.watch(directoryToWatch, {
  ignored: /(^|[\/\\])\../, // ignore dotfiles
  persistent: true,
});

// Define event handlers
watcher
  .on('add', path => console.log(`File ${path} has been added`))
  .on('change', path => console.log(`File ${path} has been changed`))
  .on('unlink', path => console.log(`File ${path} has been removed`))
  .on('addDir', path => console.log(`Directory ${path} has been added`))
  .on('unlinkDir', path => console.log(`Directory ${path} has been removed`))
  .on('error', error => console.log(`Watcher error: ${error}`))
  .on('ready', () => console.log('Initial scan complete. Ready for changes'))
  .on('raw', (event, path, details) => {
    // This event should be used only if you need raw event details
    console.log('Raw event info:', event, path, details);
  });

console.log(`Watching for changes in ${directoryToWatch}`);

then I am running it as
node ./1.js

and here is what I see when I add a new file to this directory:

So, it looks like my simple app is working fine.  But please, look how I initialized it:
const directoryToWatch = '/opt/anaconda3/lib/python3.10/site-packages/gap_statistic';

Let us make one change:
const directoryToWatch = '/opt/anaconda3/lib/python3.10/site-packages';

If this is done and node ./1.js  is launched, then tracking for changes also has place, but it slowly enumerates all files (there are a lot of them in site-packages because this is anaconda's base env). However, after adding a file I see this:

So, the question is this:  are you sure you are using chokidar correctly and not facing a similar issue?

So, the question is this: are you sure you are using chokidar correctly and not facing a similar issue?

That same error you got should show up in the VS code output somewhere if there was the same problem. Not the pylance output (as we don't create the file watchers) but likely this one:

See this issue we had a while back. You can control this problem using globs in VS code:
https://github.com/microsoft/pylance-release/issues/5639

This might help too:
https://code.visualstudio.com/docs/setup/linux#_visual-studio-code-is-unable-to-watch-for-file-changes-in-this-large-workspace-error-enospc

In the extension host I see this:
logs_extension_host.txt
It doesn't look I am having any issues with number of files.  I mentioned this issue related to my own js script, but that doesn't mean that VS Code does monitoring the same way I did when I wrote this little script with the help of CharGPT (because I am not a js developer at all).
So, that was just my guess, but no confirmation in logs.   And the question is:  how to find the reason why it doesn't work?
There is a second question:  is it really difficult to make me a command "rescan libraries" in Ctrl+Shift+P?   So that after library installation, I will just run this command, and pylance will re-index everything.  Please....

Technically you could just use "Python : Restart Language Server". It would have the same effect.

And the question is: how to find the reason why it doesn't work?

We'd have to repro it in house to see if we could suss out why the file watchers are not actually watching. It's likely the limit thing, but I'm not entirely sure where that error crops up. You could also try upping the limit as specified in that link I sent to see if that fixed it.
or just restart the language server.

Rich, thank you very much for the command name - now I will know that the name is "Restart Language Server" :)
If you know how to dump all logs using one command (if there is any) from all vs code subsystems and just zip them all an give to you - I will do this. This may help to find the reason.
Currently I know only how to see them separately in the bottom window, and that is why I did not find anything about limits, so there is still no proof.

Yeah it might not even be VS code that's outputting the error. Some other system might. Not sure how they even detect it.
But you can get to where all the logs are here:
Developer : Open Extension Logs Folder
You might have to also set VS code's log level too. That's with the Developer: Set Log Level command.


Implementing featured parameter in API posts.browse method and post model

Closes #5152

Adds featured filter option to posts.browse method modifying the model to take it too

@ErisDS  Any update on this one? waiting this one to be merged to move on to the next change

Sorry for the delay, left a comment on one line but this is otherwise looking good to go :+1:

Sorry, just spotted the commit message - can you please update it to match our guidelines? It needs to please have a short description as the first line - that's what goes into the changelog.


yubikey-manager tests fail with "dictionary size changed during iteration"

This appears to have started recently: https://hydra.nixos.org/job/nixpkgs/trunk/yubikey-manager.x86_64-linux
I'm disabling the tests for now but I'm guessing something changed in the python infrastructure that would have caused that.
  File "nix_run_setup.py", line 8, in <module>
    exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
  File "setup.py", line 63, in <module>
    'Topic :: Utilities'
  File "/private/var/folders/6j/566zczg137j_zbvhjh2zcpr80000gn/T/nix-build-python2.7-yubikey-manager-0.4.0.drv-2/yubikey-manager-0.4.0/ykman/yubicommon/setup/__init__.py", line 124, in setup
    return _setup(**kwargs)
  File "/nix/store/w8zld7z4gq4b36z0szgrh6yv5zi30915-python-2.7.13/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/nix/store/w8zld7z4gq4b36z0szgrh6yv5zi30915-python-2.7.13/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/nix/store/w8zld7z4gq4b36z0szgrh6yv5zi30915-python-2.7.13/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/nix/store/j6mdcbimr51ac4an2isad9f3y1xl0hi5-python2.7-bootstrapped-pip-9.0.1/lib/python2.7/site-packages/setuptools/command/test.py", line 215, in run
    self.run_tests()
  File "/nix/store/j6mdcbimr51ac4an2isad9f3y1xl0hi5-python2.7-bootstrapped-pip-9.0.1/lib/python2.7/site-packages/setuptools/command/test.py", line 238, in run_tests
    **exit_kwarg
  File "/nix/store/w8zld7z4gq4b36z0szgrh6yv5zi30915-python-2.7.13/lib/python2.7/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/nix/store/w8zld7z4gq4b36z0szgrh6yv5zi30915-python-2.7.13/lib/python2.7/unittest/main.py", line 149, in parseArgs
    self.createTests()
  File "/nix/store/w8zld7z4gq4b36z0szgrh6yv5zi30915-python-2.7.13/lib/python2.7/unittest/main.py", line 158, in createTests
    self.module)
  File "/nix/store/w8zld7z4gq4b36z0szgrh6yv5zi30915-python-2.7.13/lib/python2.7/unittest/loader.py", line 130, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/nix/store/w8zld7z4gq4b36z0szgrh6yv5zi30915-python-2.7.13/lib/python2.7/unittest/loader.py", line 103, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/nix/store/j6mdcbimr51ac4an2isad9f3y1xl0hi5-python2.7-bootstrapped-pip-9.0.1/lib/python2.7/site-packages/setuptools/command/test.py", line 43, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/nix/store/w8zld7z4gq4b36z0szgrh6yv5zi30915-python-2.7.13/lib/python2.7/unittest/loader.py", line 91, in loadTestsFromName
    module = __import__('.'.join(parts_copy))
  File "/nix/store/w8zld7z4gq4b36z0szgrh6yv5zi30915-python-2.7.13/lib/python2.7/test/regrtest.py", line 184, in <module>
    for module in sys.modules.itervalues():
RuntimeError: dictionary changed size during iteration

cc @benley @FRidh
Edit: the doCheck = false is in https://github.com/NixOS/nixpkgs/commit/15502421c72de368f23ee13f1df56ebb1f424935

Duplicate of #29103.
The yubikey-manager archive contains tests so the test runner invocation should be fixed instead.

@FRidh I don't quite understand how the distutils.cfg thing causes this error. I checked the other ticket but that didn't clarify for me; I'm happy to fix something in yubikey-manager if needed but am not sure what to do yet.

Our default checkPhase runs python setup.py test, which asks setuptools to find and execute tests. setuptools then uses easy_install to download and install test dependencies. This can be configured with distutils.cfg. Therefore, this is likely a bug somewhere in setuptools or distutils. An easy solution would likely be to manually specify the checkPhase.


Add support for flipping arcs.

I was running into issues flipping footprints that were acquired through Mouser's part library, due to the footprints often having arc features (which flip was not supported for). After fixing those I ran into an issue of the flip_layer functionality choking on features that were in layers that were not tied to the front or the back of a board (ex. Dwgs.User).
So, I additionally made the flip functionality more robust to handle layers that aren't either on the top or bottom of the board (such as user drawings). When the flip function is called and it encounters one of these layers a warning is printed, and the feature stays on the original layer instead of being moved to the None layer as was done previously.
This allowed me to to use this library to layout footprints that were acquired from Mouser's part library.

Thanks!


Project naming on commandline

Project naming on commandline

If there's an issue, please reopen and elaborate.


Generic select function for specific column of type

original thread
import { SelectQueryBuilder, SqlBool } from "kysely"

type DB = {
	likes: { isDeleted: SqlBool }
	reviews: { isDeleted: SqlBool }
	users: { foo: SqlBool }
}

type Column<C extends string, Type> = { [K in C]: Type }

type TablesWithColumn<
	DB,
	C extends string,
	Type,
	Col extends Column<C, Type> = Column<C, Type>,
> = keyof {
	[K in keyof DB as DB[K] extends Col ? K : never]: never
}

type Withdeleted = TablesWithColumn<DB, "isDeleted", SqlBool>
//    ^? type withdeleted = "likes" | "reviews"

// works
const withDeleted1 = <Type, O>(
	qb: SelectQueryBuilder<DB, Withdeleted, O>,
) => qb.where("isDeleted", "=", false)

// doesn't work
const withDeleted2 = <
	TB extends TablesWithColumn<DB, "isDeleted", SqlBool>,
	O,
>(qb: SelectQueryBuilder<DB, TB, O>) => qb.where("isDeleted", "=", false)

Playground Link
Maybe would this be due to limitation of how typescript works?

Hey 👋
Both variants don't work since they require to always have exactly 'likes' | 'reviews' in query context.
So things like: 'likes' | 'users' or 'likes' | 'reviews' | 'users' cause a type error, which is not what you want.
Regarding issues with type errors within helper functions, it's a limitation of TypeScript, it won't narrow the generic properly when used inside the helper. You should only really focus on making sure the input and output types of a helper are correct.
type TablesWithColumn<C extends string> = {
  [TB in keyof DB]: C extends keyof DB[TB] ? TB : never;
}[keyof DB];

type TablesWithIsDeletedColumn = TablesWithColumn<"isDeleted">;

type IfIntersects<G0, G1, R> = [G0 & G1] extends [never] ? never : R;

// easiest escape is - don't bother with anything but input and output correctness and strictness.
// anything inside the helper cannot be type-safe anyway due to typescript limitations.
const withDeleted = <TB extends keyof DB, O>(
  qb: IfIntersects<TB, TablesWithIsDeletedColumn, SelectQueryBuilder<DB, TB, O>>
): SelectQueryBuilder<DB, TB, O> => qb.where("isDeleted" as any, "=", false);

// if you want better ergonomics within the helper, you could cast it as a narrower builder.
// this is still not quite as type-safe as you'd like given that the tables in `TablesWithIsDeletedColumn`
// could have very different structures, and that nothing guarantees all of them are in query context.
const withDeletedAlt = <TB extends keyof DB, O>(
  qb: IfIntersects<TB, TablesWithIsDeletedColumn, SelectQueryBuilder<DB, TB, O>>
): SelectQueryBuilder<DB, TB, O> =>
  (qb as unknown as SelectQueryBuilder<DB, TablesWithIsDeletedColumn, O>).where(
    'isDeleted',
    "=",
    false
  ) as any;

https://tsplay.dev/NDAG4m


Enable dask-expr in benchmarks

I can reproduce those import errors locally if I do something like:
import dask

dask.config.set({"dataframe.query-planning": True})

import dask_expr as dd

import dask.dataframe as dd

but not I I remove either one of those imports

AB tests here: https://github.com/coiled/benchmarks/actions/runs/8081452589
Good to merge now


days should support numeric format too

I've seen NASA-style countdown timers that include days as a top-level unit, with display like this:T-3:23:58:59 which means almost 4 days before launch time. The days use the same separator as between hours and minutes.
Seems like we may want to support the days: 'numeric' style to support this case.
Here's one example I found online.

The issue here would be the lack of CLDR data.

The current proposal doesn't include support for numeric days, but it can be easily added later, once CLDR data is available. @justingrant can this be closed?

Actions :

[ ] Open CLDR issue to support this feature @sffc


Build fails on Linux

Hi,
When I built nana on Linux, there was an error on source/detail/linux_X11/platform_spec.cpp.

Clang 3.6.1

[ 12%] Building CXX object CMakeFiles/nana.dir/source/detail/platform_spec_selector.cpp.o
/usr/bin/clang++   -DNANA_ENABLE_PNG -DNANA_LIBPNG -DNANA_LINUX -DNANA_UNICODE -DNANA_X11 -DPLATFORM_SPEC_HPP="<nana/detail/linux_X11/platform_spec.hpp>" -DSTD_CODECVT_NOT_SUPPORTED -std=c++11 -Wall -I/usr/include/freetype2 -I/home/cocoa/.ghq/github.com/cnjinhao/nana/include    -o CMakeFiles/nana.dir/source/detail/platform_spec_selector.cpp.o -c /home/cocoa/.ghq/github.com/cnjinhao/nana/source/detail/platform_spec_selector.cpp
In file included from /home/cocoa/.ghq/github.com/cnjinhao/nana/source/detail/platform_spec_selector.cpp:21:
/home/cocoa/.ghq/github.com/cnjinhao/nana/source/detail/linux_X11/platform_spec.cpp:47:34: error: invalid operands to binary expression ('std::ifstream' (aka 'basic_ifstream<char>') and 'int')
                        return static_cast<bool>(ifs_ != 0);
                                                 ~~~~ ^  ~
1 error generated.

GCC 5.1.0

[ 12%] Building CXX object CMakeFiles/nana.dir/source/detail/platform_spec_selector.cpp.o
/usr/bin/c++   -DNANA_ENABLE_PNG -DNANA_LIBPNG -DNANA_LINUX -DNANA_UNICODE -DNANA_X11 -DPLATFORM_SPEC_HPP="<nana/detail/linux_X11/platform_spec.hpp>" -DSTD_CODECVT_NOT_SUPPORTED -std=c++11 -Wall -I/usr/include/freetype2 -I/home/cocoa/.ghq/github.com/cnjinhao/nana/include    -o CMakeFiles/nana.dir/source/detail/platform_spec_selector.cpp.o -c /home/cocoa/.ghq/github.com/cnjinhao/nana/source/detail/platform_spec_selector.cpp
In file included from /home/cocoa/.ghq/github.com/cnjinhao/nana/source/detail/platform_spec_selector.cpp:21:0:
/home/cocoa/.ghq/github.com/cnjinhao/nana/source/detail/linux_X11/platform_spec.cpp: In member function ‘bool nana::detail::conf::open(const char*)’:
/home/cocoa/.ghq/github.com/cnjinhao/nana/source/detail/linux_X11/platform_spec.cpp:47:34: error: no match for ‘operator!=’ (operand types are ‘std::ifstream {aka std::basic_ifstream<char>}’ and ‘int’)
    return static_cast<bool>(ifs_ != 0);
                                  ^

Could you check this, please?

Please, simple erase !=0 and try againg. Now ifstream have a bool operator not more a void*

Thanks for reporting this bug. A fix has been checked into the hotfix branch.
fix it manually by yourself
return static_cast(ifs_);

Thank you for the fix! :star2:


[DO NOT MERGE] Switch rendering app to government-frontend

Part of: https://trello.com/c/LnV838V3/193-penultimate-stage-re-publish-specialist-content-to-switch-rendering-app-1
It can be merged and deployed when government-frontend can correctly render specialist content.

There are a few more references to specialist-frontend that should be updated or removed:
https://github.com/alphagov/specialist-publisher/search?utf8=✓&q=specialist-frontend&type=

Should SearchPresenter also be updated?

@fofr possibly yes


Don't require settings.local

Resolves #1392
Make all settings have a default value, so we don't require settings.local for this project.
This would allow us to run tests without requiring us to copy example-local.py to local.py. It also makes it possible to run the project without customizing local.py. This also means that the test script could be the same at the CI and developer level, since we don't need to copy example-local.py to local.py for testing anymore.
What this means is that for the project to work correctly, ALL SETTINGS are required in settings.base, and the settings you want overridden should be placed in settings.local (or as an environment variable).
Why?

When 30d11f07106db584acd6b42313c01830e3996508 was introduced, every single developer in the project had to open settings.example-local, copy four lines regarding some new module into settings.local for the project to even run. If we make sure that all settings have a default setting in settings.base, we don't need to configure settings.local except for when we want to do so.
When running tox locally one would overwrite local.py with a fresh example-local.py, which removes custom local.py settings like logging, databases and Stripe / ReCaptcha keys.

We've talked about unclogging the base settings file and this doesn't exactly help. We should probably see this in relation to #976

It's not supposed to help that issue. Anyways, see my input on #976 regarding your comment.
If you don't like this solution, what do you think of the case I described under "Why - 1", and how should we solve it? Or is it not an issue?

I'll close this in favour of the ongoing discussion in #976.


Update access-tokens.md to fix some verbiage and organization

Fix "with the following modifications" incomplete list. Move items up from other parts of the document where they did not really belong.

@davidmu1

Can you review this PR?
IMPORTANT: When this content is ready to merge, you must add #sign-off in a comment or the approval may get overlooked.

#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#please-close.
this PR was moved to azure-docs-pr:

https://github.com/MicrosoftDocs/azure-docs-pr/pull/247024
https://github.com/MicrosoftDocs/azure-docs-pr/pull/247038


add compiled sql to manifest

The manifest.json file should contain compiled SQL for every node. This will be used for rendering documentation for a dbt project.

add any sql that we have to the manifest

Ignoring the metadata for the moment (that looks easy since we have the project config), here's the current code layout. Things can get a bit confusing as we use the name 'compile' a lot:

dbt.runner.RunManager.run_from_graph() ends up getting called
which calls dbt.runner.RunManager.compile()
which calls dbt.compilation.Compiler.compile().
which is the only place that has access to the manifest, and it throws it away after writing the manifest to disk (which currently happens even pre-linking).

The node compilation that generates the actual compiled_sql value has a different chain:

we start at dbt.runner.RunManager.run_from_graph() again
which calls dbt.runner.RunManager.execute_nodes()
which calls, via a thread pool, dbt.runner.RunManager.call_runner()
which calls runner.safe_run() (which should always be dbt.node_runners.BaseRunner.safe_run())
which calls self.compile() (which should always be dbt.node_runners.CompileRunner.compile() in this case)
which calls dbt.node_runners.CompileRunner._compile_node(), which does the actual jinja rendering (this nuance is important as _compile_node() has multiple entry points)

So the place that we'd have to do this is in run_from_graph(), but a lot has to change - we have to pass the manifest back up the stack from Compiler.compile(), and then pass it down all the way through the node runners.
My intuition is this is the way to go:

Change Compiler.compile() and its callers to return the manifest as well.
Store the manifest as an attribute on the RunManager. This is not elegant or anything, but at least it doesn't have to be a new parameter on all the RunManager method (call_runner is potentially extra-annoying due to threading and having to stuff multiple references into the args_list)
pass the manifest into the runner as an argument (ugh). Runners that don't compile will need some thought here.
(ab)use the fact that Python dicts are thread-safe for storage to just live-update the nodes in the manifest with the compiled SQL. Alternatively we could invent a new manifest type here + new node types (CompiledManifest/CompiledNode?) but I am not sure how much that buys us, it's basically the same issues.
Move writing out the manifest to RunManager.run_from_graph. The various threads have been updating the manifest along the way, so we can just move dbt.compilation.Compiler.write_manifest_file into the RunManager or something.

One big downside: this will fall apart pretty horribly if we ever were to switch from threading to multiprocessing, which seems very limiting. Also this will probably break many tests due to new method signatures, but that's to be expected.

@beckjake thanks for the detailed writeup! I agree - no part of this sounds pleasant. What I was picturing was a much simpler implementation in which we effectively just call the code in the CompileTask, then inspect the compiled results to pluck out the compiled_sql.
I think that, for example, the results var here should contain a list of all of the nodes in the graph, and they should all have compiled_sql attributes. I totally recognize that at this point, we already need to have created the manifest based on the current flow of things.
I wonder if it makes sense to invent a new CompiledManifest, created by merging a ParsedManifest and the output of runner.run(query, CompileRunner). I don't want to do something totally hacky here - we can ship docs without the compiled code if we need to - but I did want to suggest an alternative way of approaching this problem.
More broadly, we're eventually going to want to record the output from dbt invocations so that we can surface information like test failures, model build times, etc, in the docs. It could be that trying to force this compiled_sql attribute into the manifest is the wrong idea, and we're actually better suited to tackle the question of serializing "run results" instead.
cc @cmcarthur

serializing "run results" instead

This is a really interesting way of reframing the problem. The manifest is supposed to be a representation of the project with no understanding of the warehouse, but incorporating compiled SQL kind of breaks that abstraction. And, in all of these cases, we have the compiled SQL right there in the target directory. We could always just serve it up from the filesystem

Lots of great questions here!
I think passing the ParsedManifest along as we go, and updating it with CompiledNodes (and more, seeds for instance end up with their own situation that is neither a ParsedNode nor a CompiledNode), is the cleanest way we'll get things. If we do it right, at any point in the run we can serialize the current manifest and write it to disk. One example that comes to mind is on a failure, you will be able to get compiled_sql for the failing node in your manifest, which seems pretty awesome to me.
Regarding multiprocessing, sometimes for cross platform stuff like dbt you end up having to use processes because of the subtle differences in how threads behave across platforms - processes are a bit more of a level playing field. If you need to be able to kill background tasks cleanly no matter what, threads aren't viable either. Hopefully we can avoid those as threads are much more pleasant in the happy path at least. In the past I've rolled my own on top of python's standard library - if your data is pickle-able, it's not that hard of a problem, it just introduces some design constraints.
And we are actually going to be effectively serializing the run results in this new model, because this is where we probably have to get our compiled node to put into the manifest (result.node is the result of CompileRunner.compile())


Clustering

Is it possible make clustering in harp.gl like google maps or other?

Thanks @BoryaMogila for reaching out and contributing to harp.gl
We certainly do have clustering on our roadmap for this year. Please stay tuned for further updates.


deps: Bump rules_python -> 0.7.0

Fix #20317
Signed-off-by: Ryan Northey<EMAIL_ADDRESS>Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

/lgtm deps
I'm running this locally without issue

cool, it brings python toolchains amongst other things - ill experiment with that after it lands

cc @mattklein123 apologies - this needs to be re-approved as i entered the date incorrectly

/lgtm deps


Fix PHONT

Issue #, if available:
Description of changes:
Found one more misspelling of PHONY. Fixes it.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/approve


flake.nix: init

I like flakes. 🤷🏻♂

Added flake.nix

Wired up the package to the default build output for nix build
Wired up the shell to the default shell for nix shell
Wired up the two checks (for formatting and nixpkgs) to nix flake check.
Added the hydra-built systems as the default systems

Added nix flake update to the update scripts
Reshaped default.nix to accept pkgs as an input
Added treefmtWrapper (per the docs on treefmt-nix) that fits into nix fmt.

Turns out that nix flake update is failing in the CI scripts. I wonder if it's a "nix from inside nix" problem. There's no output, which is sad.

@infinisil, it is a tiny layer above default.nix! I did it differently than the nixfmt one because the Nix code is shaped differently, but there's no getting around the flake's use of flake.lock. I thought the simplest way to go was duplication rather than trying to make npins flake compatible.

there's no getting around the flake's use of flake.lock

There is though! If you just don't specify any flake inputs and instead import the default.nix (which then uses the niv-pinned ones) with its default arguments, it should work just fine

if you don't specify flake inputs

But that's what I want the flake for -- specifying inputs. Without any inputs, as a flake consumer, I have to use whatever's pinned, and there's no ability to override.
I think the right way to go is to update npins to 0.2.4 (per the linked review), then use the import-flake to base the npins off flake.lock.

Would .envrc be use flake or use nix?

Would .envrc be use flake or use nix?

It would be able to contain use flake should this PR be merged.

Without any inputs, as a flake consumer, I have to use whatever's pinned, and there's no ability to override.

You can override the pinned sources using the default.nix. E.g.
{
  inputs.ncbn.url = "github:NixOS/nixpkgs-check-by-name";
  inputs.ncbn.flake = false;

  outputs = { ncbn, ... }:
    let
      result = import ncbn {
        nixpkgs = <something else>;
      };
    in
    {
      packages.ncbn = result.build;
    };
}

This notably has the advantage that you can even override it with a version of Nixpkgs that Flake inputs can't represent, e.g. one patched via a derivation, or fetched with a non-builtin fetcher, etc.

Feel like I don't know enough about the pros/cons of flakes despite using them for like 4 years.
I think at a minimum we shouldn't have two lock files for the same inputs.
Is there any tradeoff in evaluation caching between using flakes vs npins lock file?

I'm returning to this PR after a couple weeks, but I'm going to be experimenting with how GitHub actions runs the code a bit so please bear it no mind until it's out of draft.

Would .envrc be use flake or use nix?

It would be able to contain use flake should this PR be merged.

I noticed this recently use flake || use nix.


Fixed three imports from gnomad package

When working with the v3 data I found a few imports that failed. I've updated those three imports to the proper paths in the gnomad package.

Sorry I didnt realize #49  was not yet merged. Closing this.


ServerRepository renaming

ServerRepository should be renamed.
Saying Server here is kind of a lie... What if I plug a ServerRepository with a TestingBackend?
This protocol is meant to add elemental CRUD functionality to the Repository protocol. A better, and more appropriate, name for this one could be CRUDRepository, or ElementalRepository.

⚠ Watch out for documentation updates when renaming!

Renaming process checklist:

[ ] Every appearance in code is renamed
[ ] Every according file is renamed
[ ] Project builds successfully
[ ] Tests pass successfully
[ ] There's a deprecation/migration mechanism implemented in code
[ ] README is updated
[ ] CHANGELOG is updated


A better API for specifying extra arguments

Sadly this is not compatible with v1.3.0, but I don't think anyone had time to use that.


[WIP][SPARK-41775][PYTHON][ML] Adding support for PyForch functions

NOTE: If you want to only view the diff from the other WIP PR regarding the baseline changes, look at the LAST COMMIT in this PR's commit history (titled "WIP adding notebook functionality"). Since I am sending out parallel PRs that are related, you should view this commit to see the diff pertaining to this ticket.
What changes were proposed in this pull request?

This is an addition to https://github.com/apache/spark/pull/39299 to add support for using functions as the input for distributed training. The users would follow the first workflow in the design document to run training.
Why are the changes needed?

We want to make it easier for users to run distributed training from a notebook setting.
Does this PR introduce any user-facing change?

Users can now input a training function into the PyTorchDistributor().run(...) api. This will require a lot of additional documentation though since since we are internally using cloudpickle to run training so we need to be able to have the user's train() function be picklable.
How was this patch tested?

This is a WIP PR so it hasn't been tested yet.

Can one of the admins verify this patch?

overall good.

Merged to master.


Put hyperresistivity term in transport term

We should put HyperresistiveDiffusionTerm inside ConstructTransportTerm, and have its diffusion coefficient (the current input Lambda) be set either by dB/B (according to section 5.4 in doc/notes/theory) or directly as an input.
Perhaps we could also replace the arguments bool kinetic, bool heat to ConstructTransportTerm by an enum transportType or similar, that could be for example TRANSPORT_TYPE_DENSITY/HEAT/KINETIC/POL_FLUX?

The hyperresistive diffusion term has now been added to the code, albeit in a slightly different way (i.e. not part of the general transport interface).


Integration with Kakoune Lint throw errors

Hello there,
I got a problem using rubocop in the Kakoune Lint module
I guess I just have to set buffer lintcmd 'rubocop -f emacs' then :lint but it logs ton of errors here debug buffer
Doest anyone got the catch explaining why it does not work in this context ?

Expected behavior
It should lint the current buffer
Actual behavior
Rubocop crash then the logs are pushed in the debug buffer
Steps to reproduce the problem

run a vanilla Kakoune session with the lint module (provided by default)
open a Rails file (in my case, a controller)
config the linter set buffer lintcmd 'rubocop -f emacs'
run :lint

It works sometime for no reason
:set buffer lintcmd 'rubocop -f emacs'
:lint # do not works, crash
:set buffer lintcmd 'rubocop -f emacs -l'
:lint # no warning nor errors
:set buffer lintcmd 'rubocop -f emacs'
:lint # works well

RuboCop version
$ rubocop -v
0.88.0

I installed Kakoune and was able to reproduce the problem. I didn't manage to find the root cause, though. What I noticed was, that the cops that crash are in extension gems rather than in RuboCop itself. In your linked debug buffer it was Rails cops. By adding --debug to the rubocop command I could see
Scanning /tmp/kak-lint.ola2iidt/sel-0/text/Gemfile
For /tmp/kak-lint.ola2iidt/sel-0/text: Default configuration from /home/jonas/.rvm/gems/ruby-3.0.0/gems/rubocop-1.18.2/config/default.yml
For /home/jonas/dev/rubocop: configuration from /home/jonas/dev/rubocop/.rubocop.yml
configuration from /home/jonas/.rvm/gems/ruby-3.0.0/gems/rubocop-performance-1.11.3/config/default.yml
configuration from /home/jonas/.rvm/gems/ruby-3.0.0/gems/rubocop-performance-1.11.3/config/default.yml
configuration from /home/jonas/.rvm/gems/ruby-3.0.0/gems/rubocop-rspec-2.4.0/config/default.yml
configuration from /home/jonas/.rvm/gems/ruby-3.0.0/gems/rubocop-rspec-2.4.0/config/default.yml
configuration from /home/jonas/.rvm/gems/ruby-3.0.0/gems/rubocop-rake-0.6.0/config/default.yml
configuration from /home/jonas/.rvm/gems/ruby-3.0.0/gems/rubocop-rake-0.6.0/config/default.yml
Inheriting configuration from /home/jonas/dev/rubocop/.rubocop_todo.yml
undefined method `[]' for nil:NilClass
/home/jonas/.rvm/gems/ruby-3.0.0/gems/rubocop-rspec-2.4.0/lib/rubocop/cop/rspec/base.rb:22:in `on_new_investigation'
/home/jonas/.rvm/gems/ruby-3.0.0/gems/rubocop-1.18.2/lib/rubocop/cop/commissioner.rb:153:in `block (2 levels) in invoke'

which indicates that configuration loading for the extensions isn't working, although it seems like all the right default.yml files have been read.
In the end, what worked for me was to follow the instruction at https://github.com/mawww/kakoune/wiki/Lint#rubocop which uses set-option window lintcmd rather than set buffer lintcmd.


added confirm message to delete phase

Deleting a phase now shows a confirmation message #1089

yes. will send through in a separate PR


Fixing model validation errors for Azure Backup

Fixes #6956

Automation for azure-sdk-for-python

Automation for azure-sdk-for-go

Can one of the admins verify this patch?


tests(tests): added tests for config, git, project, and utils components

:vertical_traffic_light: Quality checklist

[x] Did you add the changes in the CHANGELOG.md?
[x] Did you run all the code checks? (go test)
[x] Are the tests passing?

I need this PR to enter first, so we can correct Horusec and Semgrep here.

Amazing work my friend!!!! I liked very much 💯 🥇