Workflow for fleet managers that don't support specifying path

As mentioned in https://github.com/osrf/rmf_core/issues/104#issuecomment-643994871, interfacing with a vendor fleet manager involves creating a fleet adapter and fleet driver.
For creating the fleet adapter, one needs to implement rmf_fleet_adapter::agv::RobotCommandHandle which includes the function follow_new_path.  I assume this function is for rmf_traffic to instruct the fleet manager that a specific AGV is to follow a specific path.
How would this work for fleet managers that plan the paths internally and do not support specifying the path?
I suppose this means the fleet adapter, on receiving a call to follow_new_path, only sends DestinationRequest messages to the fleet driver using only the destination.  Will such discarding of path information affect the internal operation of rmf_core?

interfacing with a vendor fleet manager involves creating a fleet adapter and fleet driver.

This used to be the case back in December, but it is no longer the recommended way to interface with RMF (support does still exist for this in the first release, but we will likely drop it by the time RMF is considered ready for production).
Instead the recommended way to interface with a vendor fleet manager is to use the rmf_fleet_adapter library (i.e. the RobotCommandHandle interface and other related APIs in this directory) to create your own fleet adapter executable that talks between RMF and the fleet manager. The simple way to think about the difference between this new approach and the old one is that the old approach had two executables:

the fleet driver (written by the vendor or system integrator)
the fleet adapter (provided out of the box by RMF)

The new approach uses only one executable:

the fleet adapter (written by a vendor or system integrator using the APIs provided by RMF)

How would this work for fleet managers that plan the paths internally and do not support specifying the path?

This depends on what level of path planning the fleet manager can/must do. If we're talking about local planning around environmental obstacles, then I would recommend feeding the fleet manager the waypoints given by follow_new_path one at a time as the robot reaches the waypoints. The fleet adapter does not attempt to deal with environmental obstacles, so leaving that up to robot (or fleet manager, depending on the design I suppose) would make for really great synergy between the fleet adapter and the fleet manager. We've had great success with this approach on the MiR.
If, for some reason, the fleet manager must have complete control over the entire path from start to end, with no ability to specify intermediary waypoints or to feed in waypoints during task execution, then it would not qualify for the top-tier of RMF fleet compliance: the Full Control category. Instead it would fall under the Read Only category (or Traffic Light if it can at least start or stop along the way, but we don't have a user-friendly API for that yet, so you'd have to implement the logic for that if you need it soon).

Will such discarding of path information affect the internal operation of rmf_core?

Discarding the path information and trying to go directly to the destination is not an acceptable way to use the rmf_fleet_adapter::agv API. The fleet adapter reports the path that it planned to the schedule, so if your robot follows a different path, then it's not complying with the schedule at all. You would be better off writing your own fleet adapter from scratch using rmf_traffic_ros2 and rmf_traffic_msgs. I don't recommend doing this because, frankly, it's very difficult.
I should also note, when interfacing with the rmf_fleet_adapter::agv API, you should not just follow the waypoints that are given, but you should also be conscientious of the timing on those waypoints. Specifically, you should not pass a given waypoint until the clock has passed that waypoint's time value. This timing information is a crucial aspect of the fleet adapter's strategy for avoiding conflicts, because it may have your vehicle wait at a spot for some period of time in order to allow another robot to pass. So not only would it be wrong to discard the path, but it would also be wrong to discard the timing information. Note that it's perfectly okay for your robot to be late in arriving at a waypoint, but it must not leave a waypoint until the designated time, or else it is likely to create conflicts that should have been avoided.
In summary
If your fleet manager's API allows it, I would strongly advise feeding the path waypoints to it one at a time as a series of DestinationRequests. Do not send the next DestinationRequest until both conditions are met:

The robot has arrived at the current waypoint
The clock has reached or gone past the time specified for the current waypoint

I think my situation falls under the Read Only category (technically Traffic Light but since it's not available at the moment, I'll treat it as Read Only for now).
In that case, what is the procedure for implementing a read-only fleet adapter?
The rmf_fleet_adapter/src/read_only/FleetAdapterNode.hpp doesn't seem like it supports inheriting or extending.

Before proceeding down that path, are you certain it is impossible to incrementally issue a sequence of DestinationRequests to your fleet manager? If you're working with Sesto, I believe that was successfully done with the old fleet adapter+driver API, so it should be entirely possible with the new API.
This basically means you issue a DestinationRequest that targets the first waypoint given by follow_new_path and then wait until the robot arrives before issuing the next DestinationRequest to the next waypoint... etc until the robot reaches the final waypoint.
I want to make sure that is truly not an option before proceeding, because it would be a shame to needlessly demote a robot.

The traffic light API has now been merged as of #147 .
Since the initial issue here was the absence of the traffic light API, I'll close this ticket.


Provide support for Websocket source

As described in  Stackoverflow there is an interest in providing support for Websocket source especially since there is already a Websocket sink

@krishnaprasadas: Looking at the SO discussion, it appears you have close to a working sample in your sandbox. Would you be interested in contributing the websocket-source?

The latest progress is here: https://stackoverflow.com/questions/47283166/kafkaproducer-stops-on-application-startup-saying-closing-the-kafka-producer-w.
The related @krishnaprasadas ' code on the matter: https://github.com/krishnaprasadas/websocket/tree/websocket-source/spring-cloud-starter-stream-source-websocket

Sure, what is the procedure, Can I raise a PR? I haven't used all properties in the source like server port (which can be configured through server.port anyway).

The existing Spring Boot server.port is definitely enough. But we should mention it in the README anyway.
There are many other ServerWebSocketContainer options to be exposed as configuration properties.
The payloadType would be great option as well.
However all of them we can discuss already on the PR or in the separate issues.
Thank you for considering contribution!

I have created a PR. Please have a look.


Updated poetry.lockfor devo image

Status
Ready
Related Content Pull Request
Related PR: https://github.com/demisto/content/pull/24842
Description
Updated the packages because of the following issue: AttributeError: module 'lib' has no attribute 'OpenSSL_add_all_algorithms'

I am confused, why did we added cryptography to this image as a fix?

We didn't, It already existed in the docker image.
But the latest update that happened upgraded the version, which broke the integration
I just reverted back to the previous version

@Shellyber if we need the version locked we also need to specify it in the dependabot config, otherwise dependabot will upgrade it anyway

@Shellyber we cant make the fix on the integration side instead?

@JudahSchwartz No, this is a partner integration so I can't really fix it locally
Also when trying to update all (so we won't need to lock it) it caused a different problems with their SDK
So the best thing is to lock the version


Did we know?: IBM NASA Geospatial

https://huggingface.co/ibm-nasa-geospatial

Yes, we have already covered this in #226.


GSD Request

--- GSD JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from  to before v5.10.102",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=b9a229fd48bfa45edb954c75a57e3931a3da6c5f"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "b9a229fd48bfa45edb954c75a57e3931a3da6c5f",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "ax25: improve the incomplete fix to avoid UAF and NPD bugs\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v5.10.102 by commit b9a229fd48bfa45edb954c75a57e3931a3da6c5f. For more details please see the references link."
}
--- GSD JSON ---

This issue has been assigned GSD-2022-1000389


Stop reading templates when there are no errors

Purpose
Templates are read from disk only when needed to report errors.
Line by line read changed to readAllBytes.
Fixes #
Approach

Describe how you are implementing the solutions along with the design details.

Samples

Provide high-level details about the samples related to this feature.

Remarks

List any other known issues, related PRs, TODO items, or any other notes related to the PR.

Check List

[x] Read the Contributing Guide
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[ ] Added necessary tests

[ ] Unit Tests
[ ] Spec Conformance Tests
[ ] Integration Tests
[ ] Ballerina By Example Tests

[ ] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[ ] Ballerina By Examples

Not related to https://github.com/ballerina-platform/ballerina-lang/issues/32849 . This just improves performance.


in a nested data structure, model_dump_json fails to dump attributes of a subclass

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description
I am using pydantic 2.3.0 to create a JSON dump of a nested data structure where inheritance is used.
The nesting structure is this:

Materials is the root structure, which contains a list of Material instances (or of subclasses - like the one called DataMaterial).
DataMaterials contain a list of DataView instances.
DataViews have a name property.

My example DataView instance has an attribute name set to "foo". While model_dump_json on the DataMaterial (nesting level 2) dumps the name property, it is no longer dumped when the root structure, Materials, is dumped.
When I change the type annotation from list[Material] to list[DataMaterial], this issue is gone - so my understanding of the issue is that model_dump fails to check if the list elements are of a subclass of the specified class - and thus does not use the subclass' type definition.
The reproducer can also be found at https://gist.github.com/mephinet/abfd1ceafe29085c02e429111df1e348
Example Code
import pydantic

# nesting level 3:
class DataView(pydantic.BaseModel):
    name: str

# nesting level 2:
class Material(pydantic.BaseModel):
    id: int

class DataMaterial(Material):
    views: list[DataView] = []

# nesting level 1
class Materials(pydantic.BaseModel):
    materials: list[Material]

dm = DataMaterial(id=42, views=[DataView(name="foo")])
materials = Materials(materials=[dm])

# the name attribute is available at nesting level 2
d1 = dm.model_dump_json()
print(d1)
assert "foo" in d1

# but not in nesting level 3
d2 = materials.model_dump_json()
print(d2)
assert "foo" in d2

Python, Pydantic & OS Version
pydantic version: 2.3.0
        pydantic-core version: 2.6.3
          pydantic-core build: profile=release pgo=true
                 install path: /home/.../env/lib/python3.11/site-packages/pydantic
               python version: 3.11.5 (main, Aug 28 2023, 20:02:58) [GCC 13.2.1 20230801]
                     platform: Linux-6.4.12-arch1-1-x86_64-with-glibc2.38
     optional deps. installed: ['typing-extensions']

Selected Assignee: @adriangb

What about doing this
class Materials(pydantic.BaseModel):
    materials: list[Material|DataMaterial]

Pydantic dumps according to the schema, not the objects. Since you declared materials: list[Material] it will only look for the id: int field in the items of the list, not the views: list[DataView] = [] field. For more info see https://github.com/pydantic/pydantic/issues/6403#issuecomment-1620252270.
I would make it a union for now.

Thanks for this pointer! Having seen SerializeAsAny, maybe a SerializeAsSubclass would be an option...

SerialzieAsAny already exists, it just can't be applied "globally" or at the "model level"

@mephinet I faced the same issue
this https://docs.pydantic.dev/latest/usage/serialization/#serializing-with-duck-typing solved my problem
Refer to https://docs.pydantic.dev/latest/usage/serialization/#modeldumpjson

Subclasses of standard types are automatically dumped like their super-classes
More specifically, only the fields from the annotated type will be included in the dumped object

Migration warning :
This behavior is different from how things worked in Pydantic V1, where we would always include all (subclass) fields when recursively dumping models to dicts. The motivation behind this change in behavior is that it helps ensure that you know precisely which fields could be included when serializing, even if subclasses get passed when instantiating the object. In particular, this can help prevent surprises when adding sensitive information like secrets as fields of subclasses.

Closing, as this is expected behavior and the desired behavior can be configured with the help of the docs links above.


pull request for e2e test at 11/17/2016 17:56:05

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

optest/docs/updated_at.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


No game window even opens - it crashes during the time it takes to create a window

Modpack Version Used
1.5.3
What happened?
Minecraft won't even open. Allocated 6gb of ram to it, it doesn't seem to be the problem.  I'm on Java 17 as well. I tried with the original modpack to remove oculus since I saw "Problematic frame" and in known issues they said removing oculus could help. It had not.
I expected the window to at least open...
Screenshots (if applicable)
No response
Relevant latest.log
debug.log (latest.log had less information) https://gist.github.com/Vnersack/5e4f0cd1c70e9dbeebea753cc8556529

crash report, not in crash-reports but outside of it: https://gist.github.com/Vnersack/82086b42722b71e81546240acf8f4588

Hi, this is an issue with some AMD video cards, and can usually be fixed by
updating your graphics drivers!
On Sun, Jun 30, 2024, 6:59 PM Vnersack @.***> wrote:

Modpack Version Used
1.5.3
What happened?
Minecraft won't even open. Allocated 6gb of ram to it, it doesn't seem to
be the problem. I'm on Java 17 as well. I tried with the original modpack
to remove oculus since I saw "Problematic frame" and in known issues they
said removing oculus could help. It had not.
I expected the window to at least open...
Screenshots (if applicable)
No response
Relevant latest.log
debug.log (latest.log had less information) https://gist.github.com/Vnersack/5e4f0cd1c70e9dbeebea753cc8556529
crash report, not in crash-reports but outside of it: https://gist.github.com/Vnersack/82086b42722b71e81546240acf8f4588
—
Reply to this email directly, view it on GitHub
https://github.com/katubug/CottageWitchRecast/issues/14, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AE7XE5TGXJYBE62IYIQJMGDZKCLXBAVCNFSM6AAAAABKEQPDTGVHI2DSMVQWIX3LMV43ASLTON2WKOZSGM4DENJQHA4TIMY
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>


Readme.md spelling issue...

You have this under:  What is Pythonic code and why does it matter?

Most importantly perhaps, you are writing code that is easily read and understood by your follow and senior Python developers.

Pretty sure that is supposed to be "...understood by your FELLOW and senior Python developers."

Yikes, it should be! Thanks.

Fixed, thanks.


Autocompute AmbiguousAssignOrExtract as extract

Fixes #149
This adds new classes such as MatrixIndexExpr, which are able to be auto-computed.  PR #157 made this relatively easy.  I still need to add a few tests.

@jim22k the Louvain notebook is failing sometimes.  I don't recall it ever being flaky before.  Would you be able to take a look at it when you have the chance?  Thanks.
In other news, it's nice to see the tests passing again when installing psg from source or wheels!

Alright, this is ready to go in!  Yet another fairly large PR that has backwards-incompatible changes.


Update for 2021

we have to change some of the readme tests according to the new title of the workshop

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: akudev:x: dirkelkoYou have signed the CLA already but the status is still pending? Let us recheck it.


handle dynamic port on Sonic Pi

These changes update sonic-pi-cli to work with Sonic Pi 3.2dev which uses a dynamically allocated port for server communication. The changes are compatible with other earlier versions.

Bump this. New release due out in just over a week. Can you incorporate this now please. Tested and works on latest beta  for Mac, Windows and RPi. Also still works with existing release,

Incorporated and released, thanks for this.
I don't have access to the 3.2beta so I can't test this, but I was wondering if there might be an issue where Sonic Pi picks different ports each time it runs, while the old port is still earlier in the log file. If that was the case, this approach might still fall short.
I guess we'll have to wait and see!


External endpoint isn't creating

When I deploy this micros service demo application using this repo https://github.com/microservices-demo/microservices-demo , end-pint isn't creating for accessing the sock shop front end ui.
steps:
git clone https://github.com/microservices-demo/microservices-demo.git
cd deploy/kubernetes
kubectl apply -f manifests/sock-shop-ns.yaml -f complete-demo.yaml
I couldn't access the sock shop ui using nodeip:30001 and not creating end points to access it.

This Kops setup currently creates all cluster nodes in private subnet and single bastion host in public subnet. The nodes in private subnet are not accessible from internet. So you will not be able to do nodeip:30001 from anywhere. You will be able to do it only from the bastion host or from some other node in the same cluster.
To make it accessible from anywhere, you would need to use a load balancer (either directly with a service with type Loadbalancer or through Ingress). But I don't know what exactly does this do microservice demo do any why does it use node port instead of load balancer - so I'm not sure what impact would it have on the application when you change it.
What I can do if it would help you is to add some options to the Ansible tooling which would allow you to setup a public cluster (i.e. without bastion host and with the cluster nodes in public subnets directly accessible from internet). Let me know if you would like to have this feature.

Best Regards,
Palanisamy
On Fri, Dec 22, 2017 at 1:22 AM, Jakub Scholz<EMAIL_ADDRESS>wrote:

This Kops setup currently creates all cluster nodes in private subnet and
single bastion host in public subnet. The nodes in private subnet are not
accessible from internet. So you will not be able to do nodeip:30001 from
anywhere. You will be able to do it only from the bastion host or from some
other node in the same cluster.
To make it accessible from anywhere, you would need to use a load balancer
(either directly with a service with type Loadbalancer or through
Ingress). But I don't know what exactly does this do microservice demo do
any why does it use node port instead of load balancer - so I'm not sure
what impact would it have on the application when you change it.
What I can do if it would help you is to add some options to the Ansible
tooling which would allow you to setup a public cluster (i.e. without
bastion host and with the cluster nodes in public subnets directly
accessible from internet). Let me know if you would like to have this
feature.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/scholzj/aws-k8s-kops-ansible/issues/17#issuecomment-353442635,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AP0_I1riDAeURajBGor3ijzEoUZGVkSYks5tCrcSgaJpZM4RKGd8
.

I appreciate bastion host kind of approach, but when we deploy application in K8s, we should want access that application as public e.g webserver port 80
Either you make some links for accessing this application endpoints via bastion host dns (since we already created loadbalancer in AWS, we can try to leverage this same for accessing apps with public dns custom port)or create one more approach like public cluster.

Well, the load balancer which is already created is for the access to the Kubernetes API. I would not mix that. When you want to expose something to the internet, you do it normally two ways:

Create a service with Type: Loadbalancer. That will create a new AWS load balancer for each such service and make it accessible form the internet. This is the easiest way how to do it. But if you have many services it can get a bit more expensive (The classic AWS ELB loadbalancer has some fixed costs even when there is no traffic).
You can use Ingress (the Ingress controller is among addons). Ingress will create a single AWS load balancer and use it to route traffic for different addresses (which you can setup for example through Route53). To use Ingress, you should first create a Type: ClusterIP service and in addition to it an Ingress resource which will tell Ingress how to do the routing. Ingress is good for HTTP(S) based traffic. It is not that suitable for TCP/UDP traffic.

Additionally, you can do some manual tasks. For example, you can create manually your own AWS Load balancer and configure it to route traffic to different NodePorts.


Best way to open a trace file in the UI programmatically?

I'd like to use the Perfetto UI to visualize traces automatically through an IDE extension. I'm able to generate a JSON file with the trace events I'm interested in, and I'm able to automatically open up ui.perfetto.dev in my browser.
However, I haven't found a straightforward way to open the trace file programmatically — for example, is it possible to pass the file path as a query parameter? The behavior I'm looking for would be as if the user had clicked "open trace file" from the sidebar and selected a file, but without those manual steps.
If this isn't currently possible but would be accepted as a contribution, please suggest next steps.

Have a look at this script which allows opening a trace from the command line: https://github.com/google/perfetto/blob/master/tools/open_trace_in_ui
You can use it directly or adapt it to your needs.

Also see https://perfetto.dev/docs/visualization/deep-linking-to-perfetto-ui

Yes that's correct. Due to security, web pages are not able to access the filesystem directly without explicit user selection (this is not restricted to the Perfetto UI but really to any website).

Yep browsers forbid that.
Imagine for a moment  if you could just follow a linke like http://evil.com?file=~/.gitcookies   and the browser allowed that action to pass through the file without an  user UI interaction (file picker). Would be quite bad :)
For the same reason you can't just pass a trace file path in the URL in the Perfetto UI

Fair enough. I thought perhaps it would be possible to include the file contents in the request, but that would require sending the file to your server. Then I thought perhaps it would be possible to send the file to the web page over a socket or WebRTC, but that's not any simpler than spinning up an HTTP server. I wondered if perhaps you had some other clever solution.
At the end of the day, the only problem I have with the local HTTP server is that now I'm on the hook to configure it securely — which isn't that big of a deal. If perfetto.dev were to start acting like this ominous sounding evil.com, nothing should be compromised. 😈


Remove device queue in reviewer tools (bug 1098333)

https://bugzilla.mozilla.org/show_bug.cgi?id=1098333
Feature detection does not work correctly and that queue does not match how the reviewers are doing their work anyway.

r+wc


Changing country dropdown crashes the app

Go to app - https://mentors.netlify.com/
Change the country - India to Russia
Crashes the app. Error in console -
backend.js:6 TypeError: Cannot read property 'filter' of null
at t.a.filter (Mentor.js:87)
at hi (react-dom.production.min.js:2997)
at pi (react-dom.production.min.js:2983)
at react-dom.production.min.js:6205
at t.unstable_runWithPriority (scheduler.production.min.js:266)
at Wo (react-dom.production.min.js:2739)
at wu (react-dom.production.min.js:5984)
at ou (react-dom.production.min.js:5266)
at react-dom.production.min.js:2774
at t.unstable_runWithPriority (scheduler.production.min.js:266)

I would like to fix the bug :) PR coming soon

Thanks for your help guys :)


Getting wrong Cloud Platform Project ID after changing from default appscript project

Working with the Hangouts Chat Bot template to add access to chat.bot REST API.  Prior to adding this library, I had changed the Project from within the "Resources > CloudPlatform Project" menu.
Now after adding the Library and Template, running the default "sendPushMessage()" function returns the following error:
{"error":{"code":403,"message":"Hangouts Chat API has not been used in project<PHONE_NUMBER>22 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/chat.googleapis.com/overview?project=879964091322 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.","status":"PERMISSION_DENIED","details":[{"@type":"type.googleapis.com/google.rpc.Help","links":[{"description":"Google developers console API activation","url":"https://console.developers.google.com/apis/api/chat.googleapis.com/overview?project=879964091322"}]}]}}
Looking at the CloudProject that is now associated with the project, it is completely different and attempting to reassociate with the referenced project id returns an error.

Making a copy of the project worked, but the bug/issue is related to when the Cloud Platform Project is changed.

This is working an intended. As per the documentation, Google APIs need to manually enabled for standard GCP projects. Once you've switched off of the default GCP project it's no longer possible to get it back.

Yes, however the oAuth script sees the original project I'd rather than the
associated script.
On Mon, May 6, 2019, 08:30 Eric Koleda<EMAIL_ADDRESS>wrote:

This is working an intended. As per the documentation
https://developers.google.com/apps-script/guides/cloud-platform-projects#enabling_an_api_in_a_standard_gcp_project,
Google APIs need to manually enabled for standard GCP projects. Once you've
switched off of the default GCP project it's no longer possible to get it
back.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/gsuitedevs/apps-script-oauth2/issues/184#issuecomment-489641972,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADNMZ5NHCTQR5CPS77JSTSLPUA6IHANCNFSM4HKXZVEA
.

The library just uses the client ID and client secret you pass to it. If you change projects, you'll need to create a new client ID and secret in that project and then update your code.

Hi Eric,
Seems to be sorted now.  It may have been a propagation issue.  I had
changed the project earlier, that evening and was getting a returned error
message pointing to the incorrect cloud console URL.  After retrying this
morning, the API returns the expected results.
Thanks for your help for maintaining this project.
On Mon, May 6, 2019, 09:40 Eric Koleda<EMAIL_ADDRESS>wrote:

The library just uses the client ID and client secret you pass to it. If
you change projects, you'll need to create a new client ID and secret in
that project and then update your code.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/gsuitedevs/apps-script-oauth2/issues/184#issuecomment-489667368,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADNMZ5MVY4WGR37KGM7DORDPUBGNTANCNFSM4HKXZVEA
.


create some sort of proxy for problems on non-standard ports

Many schools seem to block these ports. If we had a proxy that went to them (and not anywhere else on the internet!), that might help.

Does the availability of the webshell address the problem of access?
Students in a restricted environment might not be able to configure proxies. Another solution could be along lines of a simple nginx proxy_pass that redirects https://proxy.picoctf.com/<port>/<requestURL> to the shell server(s).

The webshell doesn't really address this. If we are hosting a web challenge on port 50000, they can't use a web browser from the shell. Something like the proxy_pass you suggest would be a nice enhancement.

The presence of multi shell-servers makes me think this proxy URL would need to be in every problem description. Given something like:
There is a website running at http://{{server}}:{{port}}. Try to see if you can login!,
turns into:
There is a website running at http://{{server}}:{{port}} or http://{{proxyserver}}/{{port}}. Try to see if you can login!,
We might as well make the 'proxy' the default one displayed.

I agree.

In attempting to apply the aforementioned proxy scheme on one of the test servers, two problems arise that make this unlikely to sort out in short term:

If the port, say 1234 is specified in the request path (e.g. https://shell.picoctf.com/1234/) then the pages served will not be sitting at root path, and absolute links such as /login` will wind up breaking. To avoid this would require problem writers to make all resources (css, js, images) as well as links to use relative paths.

Trying to be clever with a subdomain, e.g. https://1234.shell.picoctf.com/ would address the path issue, but this doesn't work with our wildcard SSL cert. The one our registrar currently provides works only for 1 level subdomain wildcard, *.picoctf.com. It's not as simple as adding a CNAME to get a new second level wildcard domain cert for *.proxy.picoctf.com. If this were possible, we could have possibly something like https://2019shell2-1234.proxy.picoctf.com.

For brainstorming...
Regarding idea 1, would the following pair of mod_rewrite rules work? (I didn't think through this too much and I assume nginx has the same capability in rewriting as apache.)
(a) https://shell.picoctf.com/1234/ internal proxies (P) to https://shell.picoctf.com:1234/. (Obviously, we can't use an external redirect and so we need to use P.)
(b) https://shell.picoctf.com/xyz with a referrer of https://shell.picoctf.com/1234/ proxies to https://shell.picoctf.com:1234/xyz. (Since this depends on the referrer field, this still assumes the problem text does not tell the user to access a URL directly, i.e., not through a hyperlink.)

Almost...I added step b into the mix. Problem is any subsequent action loses the referral.
Very specific example from problem Logon:
https://shell.picoctf.com/1234/ has a link to /logout.
This is rewritten+proxied properly by part b to https://shell.picoctf.com/1234/logout.
Now to the user's browser, the url has become https://shell.picoctf.com/logout which means subsequent referrer field has changed and loses all info on its original port origin.

Hmm, how about in (b) we do an external redirect (R in mod_rewrite) so that the URL at the browser becomes https://shell.picoctf.com/1234/xyz?

I think I got it, I had quickly given up on redirect due to what I thought was the referrer getting lost in my nginx directives. Turns out it was because the problem's own flask redirect function was setting Location to HTTP, which made chrome's default Referrer-Policy: no-referrer-when-downgrade lose the referrer.
Setting proxy_redirect http:// $scheme://; that intercepts those Location type of responses and retains the referrer throughout.
Only remaining thing that I can see, is to make sure problem writers reference external css/js with HTTPS urls.

Only remaining thing that I can see, is to make sure problem writers reference external css/js with HTTPS urls.

To that end, I've adjusted handful of 2019 committed problems as well as the public problem example/templates


Support multiple lists for an IN clause

Fixes #737
Fixes #953

Coverage increased (+0.06%) to 83.522% when pulling df4faaba20f8b3ffd4d8e504241a735d75d08c9e on AnEmortalKid:multi-ins into d34c885ba5a8c933a29ba3663d2273f7a75f24b1 on JSQLParser:master.

@wumpz thank you 👏 , I can go back through some issues and see which other ones could be closed by this!


Update cancel.yaml to use ASF GHA

[x] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #. NO NEED
[x] Update the CHANGES log. NO NEED

Thanks @potiuk for quick response, btw, do you need an ASF release for that GHA to be used?

No need for a release @kezhenxu94. This is not software officially "released" by ASF the idea is that its origin must be controlled by the ASF committers.

do you need an ASF release for that GHA to be used?

From my understanding, if we use internally(inside ASF), this is not a strict required. As this is not included in a source release :) But if they are going to post on the airflow website as a thing called official release(or similar), then it is .

One comment here. I thought about the new policy and consequences, and I think it makes much more sense for each projects to keep their own copies of the actions. I am still waiting for the infra/Greg to comment on it but I raised my concerns here about using such actions cross-project:
https://lists.apache.org/thread.html/rfc3269829edc672f928d8304d7b0aa38d1d2a5f5cb177af95e98a1e5%40<users.infra.apache.org>
Happy if you also join the discussion, I'd love to hear what others think about it.

One comment here. I thought about the new policy and consequences, and I think it makes much more sense for each projects to keep their own copies of the actions. I am still waiting for the infra/Greg to comment on it but I raised my concerns here about using such actions cross-project:
https://lists.apache.org/thread.html/rfc3269829edc672f928d8304d7b0aa38d1d2a5f5cb177af95e98a1e5%40<users.infra.apache.org>
Happy if you also join the discussion, I'd love to hear what others think about it.

Thanks. I'm also watching that thread, will reply there

@potiuk I think your(airflow) fork is fine, then SkyWalking doesn't need a new one, from my understanding of ASF policy.
2 key points,

INFRA is making sure the action is security, and doesn't include write operation.
Airflow and SkyWalking are in the same foundation, we are just 2 groups of people interesting in different tech directions, but basically, we are sharing branding(ASF) and source code IPs.

INFRA is making sure the action is security, and doesn't include write operation.

This is not happening now. INFRA is not involved in review of the actions. And some of those actions actually perform write operations.

Airflow and SkyWalking are in the same foundation, we are just 2 groups of people interesting in different tech directions, but basically, we are sharing branding(ASF) and source code IPs.

The problem is that I (more any on the commiters/PMCs of Airflow) we are not reviewing all the commits in those repos. I simply cloned someone else's action repo (in this case it is mine, so I am pretty sure I know what I am doing but there are other actions that aren't mine). So I just cloned the repo, In my case, I am reviewing only the particular commit I am referring to in 'apache-airflow' but someone could add malicious code in any of the previous commits - I have no physical possibility to review all that code, but I want to keep the history so that I can bring new changes, review and switch to new versions.

This is not happening now. INFRA is not involved in review of the actions. And some of those actions actually perform write operations.

Yes, I know they haven't. I mean, all these things are related to this purpose :)

The problem is that I (more any on the commiters/PMCs of Airflow) we are not reviewing all the commits in those repos. I simply cloned someone else's action repo (in this case it is mine, so I am pretty sure I know what I am doing but there are other actions that aren't mine). So I just cloned the repo, In my case, I am reviewing only the particular commit I am referring to in 'apache-airflow' but someone could add malicious code in any of the previous commits - I have no physical possibility to review all that code, but I want to keep the history so that I can bring new changes, review and switch to new versions.

We(SkyWalking) are only using the cancel task for now, so, I could say this is safe, right :). That is my point of now doing a duplicated clone, specifically for this case.

We(SkyWalking) are only using the cancel task for now, so, I could say this is safe, right :). That is my point of now doing a duplicated clone, specifically for this case.

For now yes I assume you reviewed my code :). I am more into creating a good policy that anyone could follow.


--ds-color-border-focus-inverse is a missing token

Please verify the version of Auro design tokens you have installed
4.3.0
Please describe the bug

Go to https://auro.alaskaair.com/getting-started/developers/design-tokens
Scroll to "Border" section
See missing token
Go to deprecated tab
Find var(--auro-color-border-focus-on-dark)
See how new token is n/a

Reproducing the error
This issue cannot be reproduced on the Auro docsite
Expected behavior
This token is present in the design system to be consumed by Auro components.
What browsers are you seeing the problem on?
Chrome
Additional context
Unless I have missed something, this token needs to be added back into the design system, as it's legacy token value is still needed in Auro components
Exit criteria
The design token is added back into the repo with the new naming convention

This token is not needed, as we are no longer using it in any of our components


mec172xevb_assy6906: tests/arch/arm/arm_irq_vector_table failed to run

Describe the bug
Failed to run the test case of tests/arch/arm/arm_irq_vector_table
Please also mention any information which could help others to understand
the problem you're facing:

What target platform are you using? -> mec172xevb_assy6906
What have you tried to diagnose or workaround this issue? Do the same thing on MEC15, it can pass

To Reproduce
Steps to reproduce the behavior:

. zephyr-env.sh
. ~/set_env_mec172x.sh
west build -p always -b mec172xevb_assy6906 tests/arch/arm/arm_irq_vector_table/
west flash

Expected behavior
The test should pass. The correct log on MEC15 is pasted below
*** Booting Zephyr OS build zephyr-v3.0.0-823-gc3c6bf9f2aea  ***
Running test suite vector_table
===================================================================
START - test_arm_irq_vector_table
Test Cortex-M IRQs installed directly in the vector table
isr0 ran!
isr1 ran!
isr2 ran!
 PASS - test_arm_irq_vector_table in 0.8 seconds
===================================================================
Test suite vector_table succeeded
===================================================================
PROJECT EXECUTION SUCCESSFUL

Impact
Can not pass daily test
Logs and console output
E: ***** MPU FAULT *****
E:   Instruction Access Violation
E: r0/a1:  0x000c3839  r1/a2:  0x00000000  r2/a3:  0x000c3839
E: r3/a4:  0x00118924 r12/ip:  0x00000000 r14/lr:  0xfffffffd
E:  xpsr:  0x4000001f
E: Faulting instruction address (r15/pc): 0xbf004770
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Fault during interrupt handling

E: Current thread: 0x118328 (main)
E: Halting system
E: ***** MPU FAULT *****
E:   Instruction Access Violation
E: r0/a1:  0x000c3839  r1/a2:  0x00000000  r2/a3:  0x000c3839
E: r3/a4:  0x00118924 r12/ip:  0x00000000 r14/lr:  0xfffffffd
E:  xpsr:  0x4000001f
E: Faulting instruction address (r15/pc): 0xbf004770
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Fault during interrupt handling

E: Current thread: 0x118328 (main)
E: Halting system

Environment (please complete the following information):

OS: Ubuntu 20.04
Toolchain: zephyr-sdk-0.13.2
Commit SHA or Version used: fef6e46f011aa8829e39b79c1861dfad3a19020d

Additional context
None

It seems the following patch fixed this problem
commit 8ef2403d3775ce1bbcc1d73a719ddc2f17d53c39 (HEAD -> main)
Author: Jay Vasanth<EMAIL_ADDRESS>Date:   Thu Apr 14 15:22:30 2022 -0400
boards: arm: mec172xevb_assy6906: Include limited drivers

Include only required drivers for hello world application

Signed-off-by: Jay Vasanth<EMAIL_ADDRESS>

Close fixed


A  question about this project.

Hi Juan.
I was just looking at the source code for this project and wanted to ask you about it.
The first question is why by default it breaks up  the SQL into multiple statements by default. Is there some advantages to that?  I can't think of any reason why that would be preferable to just sending over a large SQL with all the statements in one shot. It would be faster and more transactionally safe. I know you used the go project so maybe that's why but maybe I am missing something. If you got rid of that it would simplify the parsing a great deal too.
Also for those times when you want to create a migration file manually it would be easier to have the markers be simpler like --+up and --+down or even --^ and --v
Thanks for writing it! I am learning crystal and it was instructional to read the code.

Hello Tim! I'm glad you found this useful.
The project started as a (quick but functional) port from goose restricted to postgres. So I admit I didn't give that part much thought then.
Right now the only advantage I can think of executing the statements one at a time is the possibility of having a more detailed trace of the execution of the migration (even though we are not doing it right now). Note that we could still wrap the migrations in a transaction without needing to execute the script as a whole. On the other hand, I agree that we should drop the parsing code regarding if we can do without it.
I believe it is a priority to have micrate use the crystal db API so we can target more databases (there is a working branch, but I still want to check the state of the available drivers and think about what will happen to the prebuilt executable). Maybe we can pick this (and the transaction issue) up later. I wouldn't want to make any postgres-driver specific changes for the moment.
Thanks for the comment! I hadn't thought about this before.

+1 for sqlite support here :)

@rdp Check out the latest release which includes support for SQLite and MySQL :)

Closing this. For the moment I can't see a strong reason change the tool to run the scripts as a whole. Other databases are already supported and transaction support will be tracked in #11.


Remove destination self from NAT 1:1

It is not needed and caused problems if used and was not remembered on
edit anyway.

This is the replacement for #3255


java.lang.ArithmeticException: uint64 underflow

On Spadina launch, I see this about twice per second:
tekulog3.zip
13:01:35.148 ERROR - PLEASE FIX OR REPORT | Unexpected exception thrown for event 'interface tech.pegasys.teku.util.time.channels.TimeTickChannel.onTick' in handler 'tech.pegasys.teku.services.beaconchain.BeaconChainController' java.lang.ArithmeticException: uint64 underflow (See log file for full stack trace)

I believe this issue must have been fixed with the PRs: #2863 and #2865.


fix(54458): The inlay hint does not handle TypeScript's spread operator well

Fixes #54458

@typescript-bot pack this


feat(textwriters): added text writer system to dialogue nodes

What Has Changed

Added a central resource ScriptableObject called UVNFGameResources
Created an ITextWriter system that writes text to the UI
Added an TextWriterSettings struct that can be used to define custom fonts, font sizes, etc.

Still missing documentation, so I'll merge this at a later point.


feat: initial work on v3

Been investigating some weirdness in the Quoter:

These functions are not marked view because they rely on calling non-view functions and reverting to compute the result. They are also not gas efficient and should not be called on-chain.
https://docs.uniswap.org/reference/periphery/interfaces/IQuoter

Found @shanefontaine's post https://medium.com/authereum/getting-ethereum-transaction-revert-reasons-the-easy-way-24203a4d1844, but would need the equivalent for Python lol

All v3 tests are now passing for me locally. Not sure why CI seems to fail so hard...
However, two tests for v2 are failing, as well as one for v1.
Probably going to merge this for now, we can continue the work in a follow-up PR.

I just remembered that CI is likely failing due not having access to secrets in PRs (maybe it's just because the branch is on my own fork?).
After some debugging, I got the tests to print this:
<Result MissingSchema("Invalid URL '': No schema supplied. Perhaps you meant http://?")>

Tried setting the secret in my own fork, but no success.
Gonna merge this and hope for the best...


Task definition is not created in crew.py

Watching you web_scraper project video on Loom at:
https://www.loom.com/share/68d796b13cd94647bd1d7fae12b2358e?sid=7fdf595b-de84-4d51-9a81-ef1e9c8ac71c
after using command
agentstack g t scrape_site
no tool definition is created in crew.py
Tested in Windsurf v1.0.5
Python 3.12.7.
AgentStack version: 0.1.13

hi @vanetreg! thank you for the issue! there is a fix ready in the upcoming release. we have a ton of updates coming in that release as well so it's been a little slow to get out the door.
In the meantime for the next day or so, if you add # Task definitions under the agent definitions in crew.py it will fix this issue. Alternatively, you can install via github pip install https://github.com/AgentOps-AI/AgentStack.git

I've added definition manually


can't find any state types in type selection

I am trying to complete the "hello world" tutorial from the ros flexbe wiki.i followed the tutorials closely but cannot seem to find any state types even though i installed all the state packages mentionned in the tutorial and they are detected in the flexbe app.
here is a screenshot:

Do you see the entry flexbe_states when you go to the Configuration panel
in the app?
If not, can you run "roscd flexbe_states" in the same terminal where you
launched the app before? Does it work?
If yes, can you check whether the package.xml of flexbe_states contains the
export statement for flexbe_states?
Not sure right now what the problem is because some states are apparently
found, but not the standard ones which come with FlexBE.

@pschillinger thank you for your quick reply.this is what my configuration panel looks like:

i solved it. i don't know what the issue was but deleting my workspace and reinstalling everything made it work.i already did try this before posting here,but i forgot to remove the old shortcut and create a new one.what gave me the idea was when i read your reply and launched the app with roslaunch i found the states(before uninstalling everything i didn't even with roslaunch) .Thank you again for the quick reply.


Document TAKAHE_CSRF_HOSTS

I was playing around with Takahē (great work, looks amazing). In my situation I was attempting to host it behind CloudFront. CloudFront was configured to forward host headers, cookies, and parameters, however every time I tried to login I kept hitting a CSRF verification failed and with debug mode turned on does not match any trusted origins.
I didn't see any documentation on how to set the CSRF hosts, but I did see TAKAHE_CSRF_TRUSTED_ORIGINS in the docker compose file. It seems at some point this was replaced with TAKAHE_CSRF_HOSTS.
It would be nice if TAKAHE_CSRF_HOSTS was documented somewhere to avoid having to dive into source code to work out how to setup the CSRF hosts for Django and save time troubleshooting. Likewise it might be worth revisiting docker-compose.yml as I suspect some options are no longer relevant or correct (even if the docker-compose.yml file isn't designed for production use).

Thanks for spotting that! I've added some docs for now and fixed up the compose file, but you're right, it probably needs a review; unfortunately, it's the one bit I don't use locally since I'm all on podman.


xhunter-server to deploy on heroku

alternate method, instead of ssh server, localtunnel server

create new branch for heroku deploy button


Allow config relative binary paths and configs placed outside of the …

This PR enables the specification of a repo external config folder. Also it enables specification of a relative binary path in config.c (relative to the config folder, e.g., ./image.bin => in the specified config folder).

I've implemented this in a different way which precludes the need for replicating the linker script in each external config folder (d890ba8).


Flake while running coverage

unhandled error during test:
/b/build/slave/Linux/build/flutter/packages/flutter/test/widgets/sliver_list_test.dart
NoSuchMethodError: The getter 'pathSegments' was called on null.
Receiver: null
Tried calling: pathSegments
#0      Object.noSuchMethod (dart:core-patch/dart:core/object_patch.dart:46)
#1      collect (package:coverage/src/collect.dart:16)
<asynchronous suspension>
#2      CoverageCollector.collectCoverage (package:flutter_tools/src/test/coverage_collector.dart:55)
<asynchronous suspension>
#3      CoverageCollector.onFinishedTests (package:flutter_tools/src/test/coverage_collector.dart:23)
<asynchronous suspension>
#4      _FlutterPlatform._startTest (package:flutter_tools/src/test/flutter_platform.dart:501)
<asynchronous suspension>
#5      _FlutterPlatform.loadChannel (package:flutter_tools/src/test/flutter_platform.dart:266)
#6      PlatformPlugin.load (package:test/src/runner/plugin/platform.dart:63)
<asynchronous suspension>
#7      Loader.loadFile.<anonymous closure> (package:test/src/runner/loader.dart:252)
<asynchronous suspension>
#8      new LoadSuite.<anonymous closure>.<anonymous closure> (package:test/src/runner/load_suite.dart:89)
<asynchronous suspension>
#9      invoke (package:test/src/utils.dart:297)
#10     new LoadSuite.<anonymous closure> (package:test/src/runner/load_suite.dart:88)
#11     Invoker._onRun.<anonymous closure>.<anonymous closure>.<anonymous closure>.<anonymous closure> (package:test/src/backend/invoker.dart:403)
<asynchronous suspension>

Still happening

/cc @aam

More recent stack trace:
NoSuchMethodError: The getter 'pathSegments' was called on null.
Receiver: null
Tried calling: pathSegments
#0      Object.noSuchMethod (dart:core-patch/dart:core/object_patch.dart:46)
#1      collect (package:coverage/src/collect.dart:16)
<asynchronous suspension>
#2      CoverageCollector.collectCoverage (package:flutter_tools/src/test/coverage_collector.dart:55)
<asynchronous suspension>
#3      CoverageCollector.onFinishedTest (package:flutter_tools/src/test/coverage_collector.dart:23)
<asynchronous suspension>
#4      _FlutterPlatform._startTest (package:flutter_tools/src/test/flutter_platform.dart:510)
<asynchronous suspension>
#5      _FlutterPlatform.loadChannel (package:flutter_tools/src/test/flutter_platform.dart:266)
#6      PlatformPlugin.load (package:test/src/runner/plugin/platform.dart:63)
<asynchronous suspension>
#7      Loader.loadFile.<anonymous closure> (package:test/src/runner/loader.dart:252)
<asynchronous suspension>
#8      new LoadSuite.<anonymous closure>.<anonymous closure> (package:test/src/runner/load_suite.dart:89)
<asynchronous suspension>
#9      invoke (package:test/src/utils.dart:297)
#10     new LoadSuite.<anonymous closure> (package:test/src/runner/load_suite.dart:88)
#11     Invoker._onRun.<anonymous closure>.<anonymous closure>.<anonymous closure>.<anonymous closure> (package:test/src/backend/invoker.dart:403)
<asynchronous suspension>
#12     new Future.<anonymous closure> (dart:async/future.dart:174)
#13     StackZoneSpecification._run (package:stack_trace/src/stack_zone_specification.dart:209)
#14     StackZoneSpecification._registerCallback.<anonymous closure> (package:stack_trace/src/stack_zone_specification.dart:119)
#15     _rootRun (dart:async/zone.dart:1122)
#16     _CustomZone.run (dart:async/zone.dart:1023)
#17     _CustomZone.runGuarded (dart:async/zone.dart:925)
#18     _CustomZone.bindCallbackGuarded.<anonymous closure> (dart:async/zone.dart:965)
#19     StackZoneSpecification._run (package:stack_trace/src/stack_zone_specification.dart:209)
#20     StackZoneSpecification._registerCallback.<anonymous closure> (package:stack_trace/src/stack_zone_specification.dart:119)
#21     _rootRun (dart:async/zone.dart:1126)
#22     _CustomZone.run (dart:async/zone.dart:1023)
#23     _CustomZone.bindCallback.<anonymous closure> (dart:async/zone.dart:949)
#24     Timer._createTimer.<anonymous closure> (dart:async-patch/dart:async/timer_patch.dart:21)
#25     _Timer._runTimers (dart:isolate-patch/dart:isolate/timer_impl.dart:382)
#26     _Timer._handleMessage (dart:isolate-patch/dart:isolate/timer_impl.dart:416)
#27     _RawReceivePortImpl._handleMessage (dart:isolate-patch/dart:isolate/isolate_patch.dart:165)

Just saw this twice in a row.
This seems to occur because the observatoryUri passed to collect() is null. That comes off the ProcessEvent passed here.

Here's a full log from travis. If you need any reference: https://travis-ci.org/CrochetLand/app/jobs/394125608
From what I could investigate, it's happening only in OSX, I have the same build in travis with linux and it doesn't happen.
As for now I'm disabling coverage inside the OSX box.
The error is random, sometimes does not happen as you can see in this build: https://travis-ci.org/CrochetLand/app/builds/393343349

Not sure it's useful, but here's an additional log:
https://build.chromium.org/p/client.flutter/builders/Linux/builds/6229/steps/run test.dart for coverage shard/logs/stdio

Is this still an issue?

Yep, seen just yesterday :(

Just see it today...

I see this reliably on my home machine.
We should probably all start putting export FLUTTER_TOOL_ARGS=--checked in our .bash_profiles.

Attempt #1 at fixing this: https://github.com/flutter/flutter/pull/20506

Is this still happening or did #20506 fix it?

Haven't seen this in a while - closing.


Diff abund: exclude collections based on number of members

Eventually, compute plugin can declare the min number of members a collection must have in order to run the compute. The frontend can read this number, look at the collection, and then gray out in the dropdown those collections that do not meet the requirement.

Closing. Was a nice idea but i dont think we'll do this


Using postgis & postgresql inside a nix-shell

There's pkgs.postgis but it's designed to be used by NixOS services. https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/databases/postgresql.nix
Is there a way to get postgresql and postgis working within a nix-shell? I'm guessing this requires manual setup of the postgresql database files (which may involve manual installation of postgis then).

Maybe take a look at nixos-shell

I've had some success re-wrapping postgresql in a new build environment. You need to call CREATE EXTENSION etc to actually install your extensions though, so you could probably add that as a shell hook?
  mypg = let pg = postgresql; in pkgs.buildEnv {
    name = "my-postgresql";
    paths = [ pg pg.lib ] ++ [
      # your extensions here
    ];
    buildInputs = [ pkgs.makeWrapper ];
    postBuild =
        ''
          rm $out/bin
          mkdir -p $out/bin
          cp --target-directory=$out/bin ${pg}/bin/{postgres,pg_config,pg_ctl,initdb,createdb}
          wrapProgram $out/bin/postgres --set NIX_PGLIBDIR $out/lib
        '';
  };

I just found the problem.
In my shell.nix I'm also bringing in psycopg2 which propagates the postgresql-9.6.6 dependency. Because for some reason psycopg2 was ahead of my custom package environment, then postgresql-9.6.6 binaries was placed ahead of it.
By removing psycopg2, it ended up working.
I found another issue discussing this problem of collisions between psycopg2 and postgresql. https://github.com/NixOS/nixpkgs/issues/22311
I think 2 things should happen. psycopg2 shouldn't need the binaries, only the libraries that it binds to. (It currently propagates the entire thing).
Secondly, is there a way to make psycopg2 to use my custom postgresql environment?
@FRidh

@johnchildren I think I have an idea as to what the purpose of cp --target-directory=$out/bin ${pg}/bin/{postgres,pg_config,pg_ctl} is for. The custom build environment creates its own store path. It looks almost exactly the same the normal postgresql store path. Except that it now has its own share/extensions directory. That contains the postgis symlinks.
The postBuild command removes the symlinks to pg_config, postgres, pg_ctl, and instead copies the postgres, pg_config, pg_ctl into the custom build environment. However all the other bin commands that postgresql exports is still in the custom environment, it's just they are symlinks back to the normal postgresql store path.
I understand why this is needed for postgres since it later is applied to the wrapProgram command. But what is the point of removing the symlinks and copying over the commands of pg_config and pg_ctl? Wouldn't they still work as symlinks?
The only thing I can imagine is that by copying it to the custom build environment, the programs are aware of the postgis extension that is in the share/extensions. However I'm not sure what commands would need to be aware of the postgis extensions to work properly. Surely pg_ctl doesn't need to be aware of it.
Also the standard postgresql service expression also does the exact same thing, and it is meant to work across any kind of extensions, so it only copies over postgres, pg_config and pg_ctl, who made this choice and why?

https://github.com/NixOS/nixpkgs/blob/b2f53c76f83cbe91bb9c765b7a3aaaa50336d653/nixos/modules/services/databases/postgresql.nix#L122-L136

It's now possible. See https://github.com/NixOS/nixpkgs/issues/65452 for an example

as for psycopg2, pg has to overriden there. Unfortunately, there isn't an easy way.
self: super: {
   python = super.python.override {
    self = self.python;
    packageOverrides = pself: psuper: {
      psycopg2 = psuper.psycopg2.override { postgresql = self.postgresql_11; };
    };
   };
}

Something like that. But I agree, libpg should be extracted to separate package.


Feature Request: MP3 Normalizer

What is the purpose of your issue?

[ ] Bug
[x] Feature Request
[ ] Question
[ ] Other

Description
Add an option which acts like http://mp3gain.sourceforge.net/ which allows you to normalize the volume of downloaded MP3s, to a custom dB value (e.g. 89 dB by default)
Since it downloads from YT, the volume is all over the place so an option to normalize it across tracks would be great.
It'd entail either coding customly a system to analyze the volume of the MP3, find an average volume then apply a gain based on that, or just find a library in py already to do it.

+1. I find this very important!

As far as I can see spotify-downloader uses ffmpeg for conversion, so normalizing the audio should be straightforward with its loudnorm filter. We would need to add the -af loudnorm=I=-16:TP=-1.5:LRA=11 parameter to the ffmpeg command (probably in the convert.py somewhere).
Meanwhile you can also try to use the ffmpeg-normalize tool to do it manually after you have your mp3 files.

Implemented by default, spotDL v3 uses EUB R128 algorithm.
I know it's late, just clearing out old issues.


tasks: POC for split pool into there type

Have been tried in feat-cp-r, this commit will lead to 5G memory footprint.


[ENH] Add electrode coregistration GUI for aligned CT/MRIs for ieeg processing

@larsoner said to open an issue to discuss this, since a new GUI is pretty major maintenance but @adam2392 @jasmainak and others (me) have expressed interest in incorporating this where it would be high quality and have a lot of users as a part of mne-python. I have a manual version written using PyQt5 but it's too slow and Eric suggested a VTK embedding would be much better. This is based on the img_pipe code (cc @libertyh) but has been rewritten and Liberty and Eddy Chang expressed that they would be interested in migrating it into a mature community.
Basically, the GUI would either have users input file paths or navigate using buttons to an MRI, CT (coregistered) and ieeg data file (and optionally a pial surface from a freesurfer recon which helps with ECoG) and then navigate in 3D space to each contact on an electrode/grid and select the channel that corresponds to that location. I'm currently working on a local_max algorithm that almost works that could help make it less work.
I think the main question is whether this would be a good fit in mne-python? cc @agramfort

Adding some more features that would make it even more highly attractive for iEEG:

automatic sEEG interpolation backend if at least 2 contacts are manually localized per sEEG electrode (I have some existing code with an talented ugrad here, and I can link an overleaf document with more info). We simply would have to refactor/test.
we can add a magnet feature; knn brightest pixel type algorithm that fieldtrip localization GUI also has. Easy to implement I presume.
automatic saving to BIDS :p

@alexrockhill just spoke w/ @agramfort  and @hoechenberger and I think there potentially is interest assuming it doesn't rely on mayvi :p
Agram suggested to us: "let’s get some screenshots/videos for iEEG localization using the GUI to make things more explicit"
cc: @jasmainak
FYI: iEEG Analysis Workflow Generically Looks Like

pipeline: handle different imaging, and clinical metadata to get your T1w and CT images lined up in the space you want them.
contact-localization (what is proposed here): can be semi-automated depending on EcoG vs sEEG, but always requires a manual GUI step.
merging anatomy and iEEG representation: representing the relevant anatomy for each iEEG channel (very important for sEEG case), which can be like temporal-pole, or white-matter-hippocampus, or csf, or ventricle, or skull, which are all mapped from a specific atlas (e.g. the Desikan image from FreeSurfer aparc+aseg.mgz)

hey @alexrockhill and @adam2392 ! Great idea to start a issue. I can't see the video though :( File seems corrupt ...
Do you have an idea how many lines of code a simple GUI-based thing would involve? How big/mature (=how many contributors/users) is the community around img_pipe/seek? I'm asking because I'm trying to figure out which of the following options makes most sense: a) help improve img_pip or seek b) start a new repo (mne-ieeg ?) for this kind of things, or c) port everything to MNE.
Each has its advantages and disadvantages. MNE will give you visibility but each pull request will require careful vetting to ensure nothing else breaks. img_pipe already has an established community (does it?) that can be leveraged but you'll have to probably spend some effort in cleaning the existing code up and building a contributor community (easier for MNE). Starting a new repo has the advantage that everything goes faster, you can control the code quality from the start and things can be tailormade for ieeg/seeg without being bogged down by MNE API. But disadvantage is that community needs to be built and depending on what 3D viz you want to do, you might have to use a bunch of private functions from MNE (not sure though ...)

In terms of workflow for research and clinical usage related to your question "should X go in MNE-Python":

pipeline stuff: imo at least, this doesn't need to be in MNE-Python, or MNE even, but if it was an MNE product, it would definitely gain visibility :p. I think this can either be in the form of nipye in img_pipe, or snakemake in seek-pipeline (oh god plz no custom shell scripts :p). Most pipelines will require FreeSurfer, FSL, SPM12, acpcdetect, and MRTrix3  commands and thus should be ideally containerized in a pipeliney software framework (nipy, snakemake, airflow, prefect, etc.).
channel anatomical representation: I think this should be considered for MNE-Python for the reasons we talked about for sEEG. I.e. need to be aware of sEEG analysis, which signals might be wm, or ventricle, etc.
visualizations: I think certain viz can be in MNE-Python, like the plot_seeg example that Eric helped build out, which can be built on. I feel that JS has significant advantages for more general sEEG 3D viz for developing visualizations for clinical usage and publication figures/videos.

With that said, I think the localization GUI could be a stand-alone MNE.tools app (not in mne-python), with something like mne-ieeg-localization or something? It has mne-python as a dependency to interface with DigMontage for the channel names, ch positions, fiducials and coord_frames. There is strong interest from seek (n = 2 labs :p), since I find going thru fieldtrip to just do this one step cumbersome rather then just having a MNE-interfacing tool.
@libertyh and @alexrockhill can prolly speak more from img_pipe side, but looking at GH, they have labs that consistently use it. Not sure for what portion of it tho (pipeline, localization, or viz), but I'm sure there's more ppl facing this consistent issue at the localization step.

Huh the video works just fine on my browser...
Seems like around 30 people have posted issues to/starred/forked img_pipe so I would guess there is maybe a few times that that have been at least interested.
I would be in favor of having a separate package that has everything you need hosted somewhere besides mne that allows users to have more functions to do everything you might need (a lot of that includes wrapping freesurfer commands). But, I think having the core GUI would be really helpful to the broader base of users and it would only be like ~1000 lines, hopefully less.

@alexrockhill do you mind linking to the repo that houses the GUI you showed us during sprint? I will try to find some time to PR ;)
Might try to get some undergrads too to help out if possible.

Sure it's here https://github.com/alexrockhill/img_pipe, thanks so much for being interested in this. It would be great to have the picker use PyQt for the widgets (which it already does) with a VTK embedded window for the graphics (backend currently matplotlib) as @larsoner suggested if you're looking to do a PR or two.
Eventually maybe we could have the GUI in GitHub and then the rest of the pipeline for convenience either where it is on the ChangLabUCSF github or an mne-ieeg-type package that could be hosted in mne-tools.

It looks like the main branch hasn't had any updates for awhile, and you have two branches named update and updates :p. Which branch should I reference off of?
@agramfort is it okay for us to stick it into mne-tools now for dev, or wait till later when it's done?

Haha poor naming and I haven't had a chance to work on this for a couple months... update is stable for the manual picker and updates is where I'm trying to add some automation to the picking.

@agramfort is it okay for us to stick it into mne-tools now for dev, or wait till later when it's done?

can you be more specific? you want to move a repo to mne-tools org? if so sure

Hmmm we should think of a name that maybe pays homage to img_pipe but is also consistent with mne style...
mne-ieeg is okay but maybe not specific, mne-ieeg-pipeline might be good

Perhaps mne-iEEG-localize for the GUI?

@agramfort is it okay for us to stick it into mne-tools now for dev, or wait till later when it's done?
can you be more specific? you want to move a repo to mne-tools org? if so sure

Yep to have an easier time centralizing the eyes of MNE-devs on this push! What is the best way of doing so?
We can tidy up the code from @alexrockhill 's branch and rename it mne-ilocalize? or something that screams iEEG electrode localization :P . In README, we obviously pay props to the original creators in img_pipe, and summarize our new direction (w/ vtk, etc.)
Then we can upload as a new repository in alpha version?

Sounds good to me, just making sure @libertyh is in the loop and this is okay with her also

Fixed by #9586 I think, feel free to reopen if I'm missing something


Re-add beautysh to pre-commit once it works again.

Pre-commit has stopped working. Maybe fixable by just updating everything?
beautysh.................................................................Failed
- hook id: beautysh
- exit code: 1

Traceback (most recent call last):
  File "/home/runner/.cache/pre-commit/repo3we8pamt/py_env-python3.12/bin/beautysh", line 5, in <module>
    from beautysh import main
  File "/home/runner/.cache/pre-commit/repo3we8pamt/py_env-python3.12/lib/python3.12/site-packages/beautysh/__init__.py", line 9, in <module>
    import pkg_resources  # part of setuptools
    ^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'pkg_resources'

Pretty format YAML.......................................................Failed
- hook id: pretty-format-yaml
- exit code: 1

Traceback (most recent call last):
  File "/home/runner/.cache/pre-commit/repo9838j_so/py_env-python3.12/bin/pretty-format-yaml", line 5, in <module>
    from language_formatters_pre_commit_hooks.pretty_format_yaml import pretty_format_yaml
  File "/home/runner/.cache/pre-commit/repo9838j_so/py_env-python3.12/lib/python3.12/site-packages/language_formatters_pre_commit_hooks/__init__.py", line 2, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'

Now it's just beautysh that doesn't work so I assume that it's the tool's fault. I'll try to re-introduce it again later.


HamburgerMenu.Buttons Hover Bug

Minimal Template :

background flickers while mouse moved slowly over button
button hover don't work after busy state

To be honest I suspect this is a localized problem that is more reflective of your graphics driver than anything in Template 10. In my tests (as a result of your q) I don't see this. Then to issue number 2. You are right. After busy state the menu appears to be disabled. Need to investigate I think. Maybe someone else can see what's happening here.
I wonder if anyone in @Windows-XAML/community has seen this?

i will figure out why i have those flickering backgrounds. Maybe someone else with windows 10 build 10537 / 10547  and latest nvidia driver can confirm this issue ? (Evga gtx660ti) I can also imagine a complete different reason but i will check 1st.
The other issue i will investigate but im mainly a java programmer and debugging in vs 2015 is to be honest unexpected uncomfortable compared to popular IDEs in Java or it`s just that those are unfamiliar to me :).

Confirmed I see it also when I move the mouse over the area in red above. Very annoying. As you suggest - could be a driver issue. I have another machine to test on and will update.

i just took a quick look but can`t test it right now,
solution could be a simple re-size of the button area like :
Width="{Binding ActualWidth, ElementName=itemsControl}"
Height="{Binding ActualHeight, ElementName=symbolIcon}"

?

Um, @darenm, that might be the coolest comment of all times.

Fixed it. Strange one, that.


Include external files

What this PR does / why we need it:
This PR adds the flag --include-path which allows you to include external files in the install, upgrade and template commands.
Example
Given the configmap:
---
apiVersion: v1
kind: ConfigMap
metadata:
    name: {{ .Release.Name }}-test
data: {{ (.Files.Glob "/tmp/externals/*").AsConfig | nindent 2 }}

And the directory structure:
├── /tmp/externals
│   ├── external1.txt
│   └── external2.txt

Running helm template test_chart --include-path /tmp/externals will generate the manifest:
---
# Source: test_chart/templates/test.configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
    name: RELEASE-NAME-test
data:
  external1.txt: |
    External file 1
  external2.txt: |
    External file 2

If the --include-path is not passed, the output will be:
Error: template: test_chart/templates/test.configmap.yaml:6:41: executing "test_chart/templates/test.configmap.yaml" at <(.Files.Glob "/tmp/externals/*").AsConfig>: error calling AsConfig: must pass files

Use --debug flag to render out invalid YAML

Special notes for your reviewer:
This PR continues the work that @hunsche started in his PR: https://github.com/helm/helm/pull/8841.
I tried contacting him to discuss his work, but unfortunately, I couldn't reach him so I implemented this myself after discussing this at the Helm dev call which took place on August 19th.
This is my first contribution to Helm, and my first time working with Go. I'd be glad to get feedback for this regarding code style, best practices, and design.
close #3276
close #8227
Related documentation PR: https://github.com/helm/helm-www/pull/1180
Thanks in advance! :)

There's feedback on #8841 that needs to be taken into account here.

@mattfarina Can you please address what feedback?

@mattfarina Btw, I thought about it more and I think I should revert this commit: https://github.com/helm/helm/pull/10077/commits/d1f9bcb0fbf00bfbb76c0c1e844f87689d284d41
My initial thought was that I want to be able to access the file in the chart by its path and not by an alias.
On the other hand, this doesn't allow us to treat relative paths, because the user can pass --include-path ../external.txt, but then if he references /external.txt in the chart, it won't recognize this file.
If I revert this commit I can do: --include-path ../external.txt:externals/external.txt and then reference externals/external.txt in the chart.
What do you think about this?

@itaispiegel Using a colon as the separator won't work on Windows. Helm is used on Windows, too.
As for the idea of having a key/value pair, that requires the Helm user (chart user) to know about the internal workings of the chart. For many that is not the case. The consumer of a package (just like someone using apt to install mariadb) doesn't usually know the app that's being installed or how it works. I'm not sure the UX is a great experience for that setup.
Am I missing something?

@itaispiegel Using a colon as the separator won't work on Windows. Helm is used on Windows, too.

I'm not sure this is an issue. For example, you can see that in Docker they use a colon as a delimiter for the -v flag:
https://docs.docker.com/engine/reference/commandline/run/#mount-volume--v---read-only

As for the idea of having a key/value pair, that requires the Helm user (chart user) to know about the internal workings of the chart. For many that is not the case. The consumer of a package (just like someone using apt to install mariadb) doesn't usually know the app that's being installed or how it works. I'm not sure the UX is a great experience for that setup.
Am I missing something?

How is this different than passing values to a chart during the installation? For example, if I want to install the MariaDB chart, I'd need to go over the README file, understand the internal values, and set them accordingly.
I don't think there's really a difference with external files. If a chart requires them, then its documentation should explain how to pass them correctly.
That's also why I think that changing the behavior to fail if the file isn't included (instead of considering it empty) is a necessary behavior. Because then if you include the wrong file, the chart will fail to install, instead of it being installed in a weird state. I don't think though that it should block this PR.

Hey guys, I'm going on vacation so I won't be able to continue working on this until I come back in October.
I'm very motivated to get the work done, so I'll continue this when I come back.
If anyone wants to contribute to this PR by then, feel free to contact me by email<EMAIL_ADDRESS>:)

@mattfarina Can you please address what feedback?

I believe @mattfarina is referring to these comments:
https://github.com/helm/helm/pull/8841#pullrequestreview-661474703
https://github.com/helm/helm/pull/8841#issuecomment-748243016
We discussed them in a developer call, but I don't believe there are any concrete test cases covering this scenario with the --include-file flag.

Why was --include-file removed?

You need not forget about helm get all.

What about named aliases for each individual directory?  With this approach, the chart does not have to rely on absolute paths.  And when some existing release is updated, it will be possible to individually replace the named directory.  It's kind of like a docker  mount named volumes only in the opposite direction vol_name:/path/to/dir.
Also, my proposal may be related to https://github.com/helm/helm/issues/1892#issuecomment-939436851

Hi everyone, I would like to contribute to this feature.
Can I pick up the work and continue the implementation :)
My plan would be to:

rebase the current work to the current master
extend the loadExternalPaths implementation to support named and default aliases like

--include-path vol_name1:/path/to/dir1 --include-path vol_name2:/path/to/dir2

 ├── /path/to/dir1
│   ├── external1.txt
│   └── external2.txt

├── /path/to/dir2
│   ├── external1.txt
│   └── external2.txt

---
apiVersion: v1
kind: ConfigMap
metadata:
    name: {{ .Release.Name }}-test
data: {{ (.Files.Get "vol_name1/external1.txt") | nindent 2 }}

---
apiVersion: v1
kind: ConfigMap
metadata:
    name: {{ .Release.Name }}-test
data: {{ (.Files.Glob "vol_name1/*").AsConfig | nindent 2 }}

example with multiple --include-path without named alias. Helm would define a default alias like 0 and 1.
--include-path /path/to/dir1 --include-path /path/to/dir2

---
apiVersion: v1
kind: ConfigMap
metadata:
    name: {{ .Release.Name }}-test
data: {{ (.Files.Get "0/external1.txt") | nindent 2 }}

Also I thought about not only including files from file directories but also maybe github repostiories. Does it make sense to incorporate such a functionality as part of this PR, to be able to include files from github repositories?

Be my guest :)
Regarding the option to include files from Github repositories - why would you want that?
Remember you can always clone the repository locally and use a symlink. I'm not sure about this addition, so we should wait and see the maintainers' opinion on this.

To give more context :)
From my experience working on different enterprise projects, it is common to have different gh repositories for application and ops.
A common use case in a DevOps automated pipeline could be that the helm chart is part of an ops repository that wants to get the application configuration present in a different application gh repository.
It would make sense to have a capability for helm to automatically get the application configuration and make it available to the chart. Additionally, it would play well with tools like ArgoCD and in an environment where there is a huge amount of services that have different application configurations.
But its just a proposition :)

Hello,
Somebody has any update about this?
Thanks a lot!

Hello, Somebody has any update about this? Like @anamarija said, we're using Argo CD too, and it's a common approach getting the configuration from another repository. Thanks a lot!

You're right. I also really want this feature released!
I'm sorry for delaying this for so long. I went on a trip after working on this, and then when I came back I couldn't find the time to continue this. I'll try to put some effort on this in the next days so we can get this done!
You're welcome to join me :)

@bacongobbler @mattfarina Hey, I finally fixed the comments on the PR!
I'd be glad if you could review this. To be honest, it's been a while since I've been in contact with you about this.
If you think that there are major gaps in this PR, please make contact with me, and I'd be glad to further discuss this :)

Hello, Somebody has any update about this? Like @anamarija said, we're using Argo CD too, and it's a common approach getting the configuration from another repository. Thanks a lot!

You're right. I also really want this feature released! I'm sorry for delaying this for so long. I went on a trip after working on this, and then when I came back I couldn't find the time to continue this. I'll try to put some effort on this in the next days so we can get this done! You're welcome to join me :)

Hi, It's alright dude, thanks a lot for your time.
I can help you with whatever you need, I have revised all the changes and I think it's correct. It is just what we need :)

Hello, Somebody has any update about this? Like @anamarija said, we're using Argo CD too, and it's a common approach getting the configuration from another repository. Thanks a lot!

You're right. I also really want this feature released! I'm sorry for delaying this for so long. I went on a trip after working on this, and then when I came back I couldn't find the time to continue this. I'll try to put some effort on this in the next days so we can get this done! You're welcome to join me :)

Hi, It's alright dude, thanks a lot for your time. I can help you with whatever you need, I have revised all the changes and I think it's correct. It is just what we need :)

Great. So let's wait until one of the maintainers updates us how to proceed.

any news on this here?

Hey, unfortunately I couldn't get enough attention on this so I didn't get a CR...

that is very sad :) - we also need this very much. I will keep my fingers crossed :)

that is very sad :) - we also need this very much. I will keep my fingers crossed :)

I agree. Great opportunity for one of the maintainers to try help us here :)

@joejulian
Please, it is not possible that for 3 consecutive times, the same code will be passed from pull request to pull request.
The basics have already been coded, making the feature work.
This issue has been opened here:
https://github.com/helm/helm/issues/3276 - 19 Dec 2017
Blocked and played to closed, here:
https://github.com/helm/helm/pull/8227 - 29 May 2020
Is here:
https://github.com/helm/helm/pull/8841 - 2 Oct 2020
https://github.com/helm/helm/pull/8840 - 2 Oct 2020
I live in Brazil, not even here bureaucracy takes 2 years for a stamp.
Or is it project policy to take what you don't want to do and keep closing pull requests?

Bureaucrats get paid to do their job. I'm doing this as a kindness to the community. I'd be happy to have help. Care to volunteer?

I'm already helping before you even ask.
Now, do you want help or do you want an employee?
Example is this thread. It's ready, it's tested, it's functional.
What else do you need?
If you tell me, what's missing in this thread for it to be coded, I'll do what's missing.
Or do you want an employee who doesn't have permission to run the project and just cleans up Issues and Pulls?
If it's something coding, programming, I'll gladly help.

What do I need? I need a couple core maintainers to review it. I only have permissions to do triage.

@joejulian we are the same.

Unfortunately I don't have that power either.

It frustrates me, too, Paulo. That's why I closed the dead one and added a milestone to this one. I'm hoping to get some eyes on it.

Hey, any news from the maintainers?

@itaispiegel I've added it to the January milestone. It's either going to get reviewed or bumped to the next milestone. Either way, you'll get some sort of feedback from them by then. :crossed_fingers:

In some cases, its great to create helm charts to have a whole software stack setup the right way and have the end user bring his own configuration (BYOC maybe? 😅), a bit like the BYOL model. Would allow to bring a "flat" file, potentially templated out from another software. Kubernetes is already complex enough for most people, if you add Helm, the step to get into the ecosystem is quite high IMO.

To give more context :)
From my experience working on different enterprise projects, it is common to have different gh repositories for application and ops. A common use case in a DevOps automated pipeline could be that the helm chart is part of an ops repository that wants to get the application configuration present in a different application gh repository. It would make sense to have a capability for helm to automatically get the application configuration and make it available to the chart. Additionally, it would play well with tools like ArgoCD and in an environment where there is a huge amount of services that have different application configurations.
But its just a proposition :)

Sounds like a neat idea, especially for GitOps workflows with config stored in other directories. To play devil's advocate however, it seems like this could introduce an expectation by Helm users to maintain other integrations, like GitLab repositories for example. Not the worst thing, but it seems beyond the scope of this particular PR. Maybe a --from-repo flag, separate from this, would serve better?

LFTI 👀 this thing is gonna help a lot with DevOps & GitOps especially within monorepos, as well as with multi-culti aka regional deployments 🚢
Coz, atm it require plenty of shenanigans 🕹to achieve those goals 🎯

@bacongobbler @mattfarina
is this something that can be progressed? it looks like everything is ready here? this would be super useful and it seems to just be waiting for something to happen...... thanks!

LFTI 👀 this thing is gonna help a lot with DevOps & GitOps. Especially within monorepos, as well as with multi-culti aka regional deployments 🚢 Coz, atm it require plenty of shenanigans 🕹to achieve those goals 🎯

For what it's worth, my stopgap solution has been to dynamically symlink the external files directory into the helm chart inside a small shell wrapper around the helm binary.

@mattfarina Thanks for your reply!

First, Helm is both a client and an SDK. There are many applications, such as controllers and operators, that use the SDK. To support both we need to keep the separation of concerns in mind. For example, the Helm client loads a chart and passes the loaded chart to the action. See the cmd/helm/install.go file for an example. This allows the application to load the chart as it needs to and pass it to the install/upgrade/etc logic to operate on. In the same way an operator could load a chart through a different means and pass it to the logic. This means we can't assume the SDK should be accessing the local file system.

The local files need to be loaded by the application (i.e. cmd/helm) and then passed into the business logic. This way something like a controller could load files through a different means and pass them to the business logic.
The idea is that we need to keep applications in mind that are not the Helm CLI.
If I understand correctly - the problem here is that applications using the Helm SDK might not be able to access the local filesystem (for example they might be running from within a container/pod). I think this can be solved well by handling the file loading only in the cmd package. The action package shouldn't be aware of this logic, because the person using it can simply pass all external files directly to the Chart's Files attribute. Did I understand this correctly?

Why do a directly instead of the specific files? I'm asking for a use case for this as it's not been shared before. Please think of the application installer first.
The use of directories is needed, as it simplifies the use of this feature. If the user wants to include many files, he can pass only a single directory instead of passing all of the files separately.

Helm needs to internally store all of these files to deal with things like rollback. These are stored, by default, on a secret. This means there are size implications. Being lazy with paths and loading things one does not need will cause size issues that need support.

Correct me if I'm mistaken - but I think this should be true about everything that the user does. If one decides to include many files, he should be aware that it might have size implications on the cluster.
Maybe a good idea though should be to have a limit on the file size, and if the user wants to include large files, he can pass a flag like --no-file-size-limit. What do you think about this?

There is a security implication here. One could create a copy and paste prompt and point to the external files of /etc. Many people may blindly copy/paste and then leak everything in that directory. Even without the directory support someone could leak something in a protected directory due to copy/paste. How do you see handling something like this?

This has been discussed and I think that the user should be aware that he might be exposing secret files with this feature.
I think first of all this should be documented on Helm's website (I already opened a PR, but it was closed because it was inactive). In addition, we can try to think of some heuristics we can run to decide if this feature is used unsafely. For example, we can fail the installation if Helm detects a symlink in the included directory, or disallow certain file types.
I'll be glad to hear your opinion on this :)

Hope we all had a great break - its amazing to come back and see this moving forward, does this needs a final push to get there?

@mattfarina @bacongobbler Hey guys, can we please merge this? 🙏

The 3.12 release is scheduled for May 10th. I don't really know why they do it this way, but they usually don't merge things until preparing the release.
Reviews, however, are still needed. I haven't seen @bacongobbler around in a while so hopefully another @helm/helm-core-maintainers can review his change request and dismiss his review if he's unable to do so himself.
I know @mattfarina has been really busy lately but he always comes back in time for some release reviews.

The 3.12 release is scheduled for May 10th. I don't really know why they do it this way, but they usually don't merge things until preparing the release.
Reviews, however, are still needed. I haven't seen @bacongobbler around in a while so hopefully another @helm/helm-core-maintainers can review his change request and dismiss his review if he's unable to do so himself.
I know @mattfarina has been really busy lately but he always comes back in time for some release reviews.

I hope we can at least merge this soon so I can consider my job here done...

@mattfarina @bacongobbler Any update? I'm still waiting for my PR to be reviewed.

Hey @jan-lucansky, I want this to be merged too, and I know there are many other users who want this feature and need it.
Unfortunately the project maintainers keep ignoring the existence of this PR, although I tried contacting them.

@itaispiegel I see that 2 changes are requested from reviewers, isn't the PR stuck on you making those changes?

@itaispiegel I see that 2 changes are requested from reviewers, isn't the PR stuck on you making those changes?

I already updated the PR according to the requested changes about 5 months ago, but never got a reply from the reviewers...

@mattfarina @joejulian @bacongobbler any updates on reviewing this?

IMHO one the best things about helm is that it's almost all yaml-in-yaml-out. Exceptions being for crypto and lookup functions. This adds another exception to that. It seems like to me it would be better to add option to turn the files into values that get processed like anything else.  Could the use case here be solved by just allowing -set-file to take a directory like kubectl create secret generic does?

Given there is some concern with this approach, it might be prudent to look at the root problem to see if there is another solution.
The problem (IMO):
users frequently want to override configuration files in a helm template. The only way to override large configuration files wholesale is through a yaml key with a multiline file as the value. Editing yaml files as sub-keys requires a certain level of indentation for the file and makes it harder to copy paste files around. The conversion from raw config file to yaml mostly is done manually adding an element of human error to making sure those details are exactly correct. It is also harder to remember how to do, requiring looking at examples and hoping you've done it right.
The other alternative takes a lot more effort from a chart development. Requiring identifying the entire configuration file and adding template-able sections based on individual keys in the values.yaml. The problem with this approach is that adding or changing configuration template sections now requires a chart change instead of just a values.yaml change. This is a big enough QoL issue I would imagine a robust solution would go a long way making helm better.
Z4ce mentioned you can automate this with a tool or a flag to remove the human element. I think that's somewhat of a bandaid. Would it be possible to have core support in helm for this feature? i know it's very difficult, but how hard would it be in a values.yaml to specify a file syntax so a key can be populated from a local file that way?
Regardless of solution, I hope we can all agree this is a core issue of helm and should be prioritized.

@ostlerc I think that last line of agreement might be the hangup. If you think of helm as a package installer, like apt or yum, they don't offer ways to replace part of the package during the installation or upgrade process.
side story: back a decade or so ago, if you went into most distro's IRC channels and told them you were replacing packaged files, they'd call you names and make you vow to never try to get community help again.
Now clearly we have no desire to ever see that behavior in our community but their point, that the packaging that the chart provider is offering is done with intent is what makes helm charts supportable by their community. As a packaging tool, I don't agree that it's the responsibility of helm to facilitate overriding pieces of the package. We can leave that to (and to provide the hooks to do so) plugins or post-render tools.

@joejulian  that's insightful thank you!
But I think fundamentally helm would be useless without the ability to template. Also most package installs do allow several flags to alter the way it is used. And lastly, linux package managers are fundamentally different than helm install. Once those are installed, users are intended to change user configuration files, but with kubernetes and helm that should be done at the time of install, not at a later step.
I could be wrong, but that's how I see it anyways. My team uses helm extensively for the ability to deploy to different environments the same packages. If we took out the ability to change the configuration per environment, it's just a bunch of yaml files. So I think at the core, this is what helm was meant to solve.

I think it's most important to think about the core interface with Helm. Right now it's values.yaml and chart in, and on the other side you get a set of Kubernetes resources. To change the contract to be values.yaml and another set of files, I think would be a mistake.  Especially when you start to think about things like Argo and Flux.
I would like to understand the use cases more and figure out how we could still keep the contract of helm as it is, while still accommodating whatever use cases are applicable here.
Why would tooling to help people put files and directories into the values structure not be sufficient?

Honestly the comparison here to apt or yum etc falls short to me. It's 2023 - we are delivering software in portable containers configured to use the 12 Factor philosophy. The entire point is they run via injected configuration. We aren't (often) SSH'ing into VMs and running apt-get install anymore. With Kubernetes and Helm - in my mind the point is to pass in configuration, and get a functional software stack out of it using the supplied configuration.
Not being able to pass in the most common method of configuration (files..... pg_config, myTemplate.pug, nginx.conf) in a coherent way from local / repository filesystems is crazy to me

@zfalen-deloitte if your chart supports passing configuration files via values, I guess the --set-file option is what you need :wink:

So... it's 2024, is there any chance that this can be merged? I am still doing the workaround of copying the configs into the chart folder.

Well, it's mid-2024! :(
Eagerly waiting for this feature.