Could not resolve org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0.

Error occured building
A problem occurred configuring root project 'android'.

Could not resolve all artifacts for configuration ':classpath'.
Could not resolve org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0.
Required by:
project :
Cannot choose between the following variants of org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- gradle70JavadocElements
- gradle70RuntimeElements
- gradle70SourcesElements
- gradle71JavadocElements
- gradle71RuntimeElements
- gradle71SourcesElements
- gradle74JavadocElements
- gradle74RuntimeElements
- gradle74SourcesElements
- gradle75JavadocElements
- gradle75RuntimeElements
- gradle75SourcesElements
- gradle76JavadocElements
- gradle76RuntimeElements
- gradle76SourcesElements
- gradle80JavadocElements
- gradle80RuntimeElements
- gradle80SourcesElements
- gradle81JavadocElements
- gradle81RuntimeElements
- gradle81SourcesElements
- javadocElements
- runtimeElementsWithFixedAttribute
- sourcesElements
All of them match the consumer attributes:
- Variant 'gradle70JavadocElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'documentation' but wasn't required.
- Found org.gradle.docstype 'javadoc' but wasn't required.
- Required org.gradle.jvm.version '11' but no value provided.
- Required org.gradle.libraryelements 'jar' but no value provided.
- Found org.gradle.plugin.api-version '7.0' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle70RuntimeElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'library' but wasn't required.
- Found org.gradle.jvm.environment 'standard-jvm' but wasn't required.
- Found org.gradle.plugin.api-version '7.0' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.jvm.version '11' and found compatible value '8'.
- Required org.gradle.libraryelements 'jar' and found compatible value 'jar'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle70SourcesElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'documentation' but wasn't required.
- Found org.gradle.docstype 'sources' but wasn't required.
- Required org.gradle.jvm.version '11' but no value provided.
- Required org.gradle.libraryelements 'jar' but no value provided.
- Found org.gradle.plugin.api-version '7.0' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle71JavadocElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'documentation' but wasn't required.
- Found org.gradle.docstype 'javadoc' but wasn't required.
- Required org.gradle.jvm.version '11' but no value provided.
- Required org.gradle.libraryelements 'jar' but no value provided.
- Found org.gradle.plugin.api-version '7.1' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle71RuntimeElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'library' but wasn't required.
- Found org.gradle.jvm.environment 'standard-jvm' but wasn't required.
- Found org.gradle.plugin.api-version '7.1' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.jvm.version '11' and found compatible value '8'.
- Required org.gradle.libraryelements 'jar' and found compatible value 'jar'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle71SourcesElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'documentation' but wasn't required.
- Found org.gradle.docstype 'sources' but wasn't required.
- Required org.gradle.jvm.version '11' but no value provided.
- Required org.gradle.libraryelements 'jar' but no value provided.
- Found org.gradle.plugin.api-version '7.1' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle74JavadocElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'documentation' but wasn't required.
- Found org.gradle.docstype 'javadoc' but wasn't required.
- Required org.gradle.jvm.version '11' but no value provided.
- Required org.gradle.libraryelements 'jar' but no value provided.
- Found org.gradle.plugin.api-version '7.4' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle74RuntimeElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'library' but wasn't required.
- Found org.gradle.jvm.environment 'standard-jvm' but wasn't required.
- Found org.gradle.plugin.api-version '7.4' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.jvm.version '11' and found compatible value '8'.
- Required org.gradle.libraryelements 'jar' and found compatible value 'jar'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle74SourcesElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'documentation' but wasn't required.
- Found org.gradle.docstype 'sources' but wasn't required.
- Required org.gradle.jvm.version '11' but no value provided.
- Required org.gradle.libraryelements 'jar' but no value provided.
- Found org.gradle.plugin.api-version '7.4' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle75JavadocElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'documentation' but wasn't required.
- Found org.gradle.docstype 'javadoc' but wasn't required.
- Required org.gradle.jvm.version '11' but no value provided.
- Required org.gradle.libraryelements 'jar' but no value provided.
- Found org.gradle.plugin.api-version '7.5' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle75RuntimeElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'library' but wasn't required.
- Found org.gradle.jvm.environment 'standard-jvm' but wasn't required.
- Found org.gradle.plugin.api-version '7.5' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.jvm.version '11' and found compatible value '8'.
- Required org.gradle.libraryelements 'jar' and found compatible value 'jar'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle75SourcesElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'documentation' but wasn't required.
- Found org.gradle.docstype 'sources' but wasn't required.
- Required org.gradle.jvm.version '11' but no value provided.
- Required org.gradle.libraryelements 'jar' but no value provided.
- Found org.gradle.plugin.api-version '7.5' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle76JavadocElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'documentation' but wasn't required.
- Found org.gradle.docstype 'javadoc' but wasn't required.
- Required org.gradle.jvm.version '11' but no value provided.
- Required org.gradle.libraryelements 'jar' but no value provided.
- Found org.gradle.plugin.api-version '7.6' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle76RuntimeElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'library' but wasn't required.
- Found org.gradle.jvm.environment 'standard-jvm' but wasn't required.
- Found org.gradle.plugin.api-version '7.6' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.jvm.version '11' and found compatible value '8'.
- Required org.gradle.libraryelements 'jar' and found compatible value 'jar'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle76SourcesElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'documentation' but wasn't required.
- Found org.gradle.docstype 'sources' but wasn't required.
- Required org.gradle.jvm.version '11' but no value provided.
- Required org.gradle.libraryelements 'jar' but no value provided.
- Found org.gradle.plugin.api-version '7.6' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle80JavadocElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'documentation' but wasn't required.
- Found org.gradle.docstype 'javadoc' but wasn't required.
- Required org.gradle.jvm.version '11' but no value provided.
- Required org.gradle.libraryelements 'jar' but no value provided.
- Found org.gradle.plugin.api-version '8.0' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle80RuntimeElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'library' but wasn't required.
- Found org.gradle.jvm.environment 'standard-jvm' but wasn't required.
- Found org.gradle.plugin.api-version '8.0' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.jvm.version '11' and found compatible value '8'.
- Required org.gradle.libraryelements 'jar' and found compatible value 'jar'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle80SourcesElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'documentation' but wasn't required.
- Found org.gradle.docstype 'sources' but wasn't required.
- Required org.gradle.jvm.version '11' but no value provided.
- Required org.gradle.libraryelements 'jar' but no value provided.
- Found org.gradle.plugin.api-version '8.0' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle81JavadocElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'documentation' but wasn't required.
- Found org.gradle.docstype 'javadoc' but wasn't required.
- Required org.gradle.jvm.version '11' but no value provided.
- Required org.gradle.libraryelements 'jar' but no value provided.
- Found org.gradle.plugin.api-version '8.1' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle81RuntimeElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'library' but wasn't required.
- Found org.gradle.jvm.environment 'standard-jvm' but wasn't required.
- Found org.gradle.plugin.api-version '8.1' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.jvm.version '11' and found compatible value '8'.
- Required org.gradle.libraryelements 'jar' and found compatible value 'jar'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'gradle81SourcesElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'documentation' but wasn't required.
- Found org.gradle.docstype 'sources' but wasn't required.
- Required org.gradle.jvm.version '11' but no value provided.
- Required org.gradle.libraryelements 'jar' but no value provided.
- Found org.gradle.plugin.api-version '8.1' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'javadocElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'documentation' but wasn't required.
- Found org.gradle.docstype 'javadoc' but wasn't required.
- Required org.gradle.jvm.version '11' but no value provided.
- Required org.gradle.libraryelements 'jar' but no value provided.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'runtimeElementsWithFixedAttribute' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'library' but wasn't required.
- Found org.gradle.jvm.environment 'standard-jvm' but wasn't required.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.jvm.version '11' and found compatible value '8'.
- Required org.gradle.libraryelements 'jar' and found compatible value 'jar'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.
- Variant 'sourcesElements' capability org.jetbrains.kotlin:kotlin-gradle-plugin:1.9.0:
- Unmatched attributes:
- Found org.gradle.category 'documentation' but wasn't required.
- Found org.gradle.docstype 'sources' but wasn't required.
- Required org.gradle.jvm.version '11' but no value provided.
- Required org.gradle.libraryelements 'jar' but no value provided.
- Found org.gradle.status 'release' but wasn't required.
- Compatible attributes:
- Required org.gradle.dependency.bundling 'external' and found compatible value 'external'.
- Required org.gradle.usage 'java-runtime' and found compatible value 'java-runtime'.

Probably a duplicate of #729 since both refer to Kotlin 1.9.x


Use epoch seconds to allow applying timezone offset

I live in the US so the word clock example wasn't showing the right time. Now it does.
This is obviously not great, because it doesn't handle DST. But at least now there's a place to change it twice a year.

I just noticed that vector_clock uses time.localtime() and that seems to do the right thing (although I'm not sure how). Maybe I'll look into that instead; it'd work around DST issues too.

I also see that https://github.com/pimoroni/presto/pull/28 includes time zone fixes as well as a bunch of other changes, so if you want to prefer that one, closing this is fine.


Fix rest of RenderPipeline types

Description
There are plenty of RenderPipeline types not covered. However this is very low prio.
🔜 TODO

[ ] (RenderPipeline) UnityEngine.Rendering.AttachmentDescriptor
[ ] (RenderPipeline) UnityEngine.Rendering.BlendState
[ ] (RenderPipeline) UnityEngine.Rendering.CameraProperties
[ ] (RenderPipeline) UnityEngine.Rendering.CullingResults
[ ] (RenderPipeline) UnityEngine.Rendering.DepthState
[ ] (RenderPipeline) UnityEngine.Rendering.DrawingSettings
[ ] (RenderPipeline) UnityEngine.Rendering.LODParameters
[ ] (RenderPipeline) UnityEngine.Rendering.RasterState
[ ] (RenderPipeline) UnityEngine.Rendering.RenderStateBlock
[ ] (RenderPipeline) UnityEngine.Rendering.RenderTargetBlendState
[ ] (RenderPipeline) UnityEngine.Rendering.ScopedRenderPass
[ ] (RenderPipeline) UnityEngine.Rendering.ScopedSubPass
[ ] (RenderPipeline) UnityEngine.Rendering.ScriptableCullingParameters
[ ] (RenderPipeline) UnityEngine.Rendering.ScriptableRenderContext
[ ] (RenderPipeline) UnityEngine.Rendering.ShaderTagId
[ ] (RenderPipeline) UnityEngine.Rendering.ShadowDrawingSettings
[ ] (RenderPipeline) UnityEngine.Rendering.ShadowSplitData
[ ] (RenderPipeline) UnityEngine.Rendering.SortingSettings
[ ] (RenderPipeline) UnityEngine.Rendering.StencilState
[ ] (RenderPipeline) UnityEngine.Rendering.SupportedRenderingFeatures
[ ] (RenderPipeline) UnityEngine.Rendering.VisibleReflectionProbe

👌 TESTED BUT NO CONVERTER (works as-is):

[x] (RenderPipeline) UnityEngine.Rendering.FilteringSettings
[x] (RenderPipeline) UnityEngine.Rendering.RenderQueueRange
[x] (RenderPipeline) UnityEngine.Rendering.SortingLayerRange

❌ KNOWN NOT TO WORK (contains ID to Object):

[x] (RenderPipeline) UnityEngine.Rendering.VisibleLight

Motivation
Want to cover all types. Even though these types are not commonly used, it should still be proven to work via the suite of tests.

This is overkill. I'm closing, and if someone needs any of this then create an issue on it and I'll look into it.


Tasks not showing up in Command Palette

Tested using windows version

Not sufficient information, please provide steps and expected results.
Also notice tasks show up in the Command palette when entering 'task '.

There seems to be nothing related to tasks showing up in the Command Palette. At the > prompt, if I type task, absolutely nothing shows up. For instance, the instructions at https://code.visualstudio.com/docs/languages/typescript#_transpiling-typescript-into-javascript about how to transpile TypeScript code do not work. There is no "Configure Task Runner" command.
This is on Win10,
Version 0.10.3
Commit 783ecf841a2b21edca6d86853670cae89b4c244f
Date 2015-11-26T14:06:31.538Z
Shell 0.34.1
Renderer 45.0.2454.85
Node 4.1.1

Same here @msiniscalchi on Ubuntu 14.04. I can't follow instructions to configure. There is no "Configure Task Runner" and no "task", just nothing related to tasks.

Can you attach a screenshot of what you see. Here is what I see:

Is there any message in the console of the developer tools Help>Toggle Developer Tools>Console.

I toggled the console too.

It now works for me! I did a clean reinstall again and made a symlink too and now it works. Don't know what was going wrong. I have the application folder on my desktop. Here's a screenshot too.

I can't reproduce this neither on Linux nor on Windows using 0.10.6. @msiniscalchi can you still reproduce this using 0.10.6 ?

Here. Just tried this on OS X, right after installing.

@msiniscalchi very wired. Can you check if there is something in the log. From the menu bar select Help > Toggle Developer Tools and repeat the action. Thanks !

Absolutely nothing! There is not a single line in the console log (I assume that's what you wanted me to check). I tried several times to pul up tasks.... BUT I think I got it!!!
Task-related items DO show up in the palette if you open a folder---any folder. And when you close the folder, task items are no longer shown.  Not sure whether this is a bug or not, but it should be documented.

@msiniscalchi tasks are only available if you open a folder. The underlying reason is that task rely on a .vscode/tasks.json file and this can only exist in a folder case not when editing single files. I fully agree that this should be documented. I will do so for the next release

As suggested I address this by updating the corresponding documentation.

To verify check the documentation for tasks.json. There is now a sentence at the beginning addressing this.

On the Command Palette i type the literal 'code'. I need to get the below -

But i get the below -

Please help. What should i do to get the former? Palette options?

The first one is contributed by the C# extension. The next two once are to my knowledge not necessary anymore. @joaomoreno can you clarify.


Verify that live requests are saved and reused in tests (Closes #18)

Create a helper that can be included in tests to use REAL login info. It should/would be included during debugging mode ONLY when running your specs (perhaps via ENV variable later)


Replace Reese's Sodium Options with Xander's Sodium Options

Is your request related to a problem? Please describe if so.
The Sodium options menu isn't very vanilla-like
Describe the mod you'd like to add, remove, or change
Replace https://modrinth.com/mod/reeses-sodium-options with https://modrinth.com/mod/xanders-sodium-options
List alternative mods or solutions you've considered
none
Additional context
Xander's Sodium Options is a more vanilla like alternative to the Sodium menu. It can integrate with compatible mods like Iris and Sodium Extra as well, alongside being able to be disabled incase the user doesnt like it

I've considered this since I saw v2 come out :)
I might ping opinions on the discord just to see which one people would prefer, I personally prefer xander's


Data fixer performance - flatten composition and restructure optimisation rule

Follow-up to #71 with the "not easy bits" - the more structural / logic changes.
To copy words from #71:

In basic: the optimisation process works by applying various rules that restructure the produced datafixing functions based on the algebraic rules given by the point-free style of functions. Many of these rules are applied one-at-a-time (many(once(...)) rule), where a single node in the point-free tree is rewritten before packing up the new modified tree. This particularly means that the same logic runs many, many times on the same data.

Although the previous PR made changes that improved the performance of rewriting with the once() rule, this PR replaces those entirely with a new optimisation rule that applies every rule within a single traversal of the tree. To do this, this PR flattens point-free function compositions into a linear array instead trees of nested compositions. This removes the need for the association rewriting rules, which took up a large portion of time themselves. Additionally, due to how they restructured the tree, it was infeasible to apply rewrites "all at once" to the same degree as is done here.
Flattening composition is enabled/made easier by a change to encode Type information in each PointFree instance directly, instead of only encoding intermediate types. This means PointFreeRule no longer needs to accept a Type. There is some weird behaviour with creating View instances: some types generate Views with different types to the underlying function. This is used in cases such as tagged fields, which need to tag the final type such that the overall View can construct a correct Codec.
Results
With the latest changes to #71, I measure an average of 8s to build fixers on a single thread. With this PR, the average time is ~1.2s.
The majority of this time is now spent building the initial fixer function, as opposed to optimisation:

Force-pushed a change to the optimisation rule that is equivalent, but optimises everything in 3 total passes without the need to retry. We can theoretically merge all 3 passes into a single one with some additional trickery, but it doesn't seem to have any performance benefit.

I am still kind of swamped with s23 work so it's better if someone else reviews this instead of me after all. I think I can contribute better if I can do random small reviews instead. But I see we have one more already so maybe it's not even needed?


Feature request: Multiline expressions on call keyword

It would be great if we could have the ability to pass the final parameter of the call keyword (https://karatelabs.github.io/karate/#call) as a multi-line expression (https://karatelabs.github.io/karate/#multi-line-expressions), so that we don't have to define an additional variable in a previous statement. e.g.
Currently we do:
Feature: Some scenarios

    Background:
        * def CreateThing = read('classpath:util/api.feature@name=CreateThing')

    Scenario: First scenario
        * def thingRequest =
          """
          {
            "prop1": "value1"
          }
          """
        * def thing = call CreateThing thingRequest

But it would be nice to do:
Feature: Some scenarios

    Background:
        * def CreateThing = read('classpath:util/api.feature@name=CreateThing')

    Scenario: First scenario
        * def thing = call CreateThing
          """
          {
            "prop1": "value1"
          }
          """

The purpose is to reduce the noise when calling other functions - which when setting up a scenario which requires complex prerequisites, every little helps!

I would like to contribute. This is my first time contributing to open source project.

@pradeepprabhakar sure. this will require some independent digging from your side into how the code works

@ptrthomas I understand.


 Tag: v0.28.0  flags: (if any)
Bug or Regression?
Regression
Description
I'm on  Arch linux, version is:
Linux me 6.4.12-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 24 Aug 2023 00:38:14 +0000 x86_64 GNU/Linux

I replaced kitty with alacritty. when inside alacritty, I can run hyprctl command fine. But if started a tmux session inside alacritty, then running hyprctl <anysubcommand>, it will error out like this:
Couldn't connect to /tmp/hypr/9654749244117f7f150c6f2a2ce4dede6e8cbb25_1693567633/.socket.sock. (3)

I checked my /tmp/ dir, and made sure all user have read/execute permission along the path, but the error still happens.
How to reproduce
just to inside a tmux session, then run hyprctl with any subcommands.
Crash reports, logs, images, videos
No response

This is not a hyprland/hyprctl bug, rather you need to update the value of the HYPRLAND_INSTANCE_SIGNATURE environment variable in tmux, as tmux may use the older value of this variable, since it uses the first instance started as a server for other instances.

You can import variables into tmux automatically when hyprland is started by running a script, for example,
create a script as ~/.local/lib/import_env
Then use exec-once to execute the script when hyprland starts.
exec-once = $HOME/.local/lib/import_env tmux

Another option is to let tmux auto-update those e.g.,
$ tmux set -ag update-environment " HYPRLAND_INSTANCE_SIGNATURE HYPRLAND_CMD"

or (more complete)
$ cat ~/.tmux.conf
# non-systemd
set -ag update-environment " DBUS_SESSION_BUS_ADDRESS XDG_RUNTIME_DIR"
# wlroots, QtGui; rarely used outside of DEs
set -ag update-environment " XDG_SESSION_TYPE"
# Cursors (see also gsettings for Gtk, xrdb for X11)
set -ag update-environment " XCURSOR_SIZE XCURSOR_THEME"
# Generic Wayland
set -ag update-environment " WAYLAND_DISPLAY"
# Arcan
set -ag update-environment " ARCAN_CONNPATH"
# Xorg or Xwayland
set -ag update-environment " DISPLAY XAUTHORITY"
# Hyprland + defaults in main.cpp
set -ag update-environment " HYPRLAND_INSTANCE_SIGNATURE HYPRLAND_CMD _JAVA_AWT_WM_NONREPARENTING MOZ_ENABLE_WAYLAND XDG_CURRENT_DESKTOP"
# Sway or i3
set -ag update-environment " SWAYSOCK I3SOCK"
# Wayfire IPC plugin
set -ag update-environment " WAYFIRE_SOCKET"
# labwc --exit/--reconfigure
set -ag update-environment " LABWC_PID"
# etc

hyprctl --instance 0 clients
or the above

Both of these two works, thanks!

Thank you for explaining solution, but could you please also explain what causes tmux to fail to pick up this environment variable? Even a brief explanation would be useful.
I'm concerned that there may be other things missing when starting Hyprland that will lead to silent errors later down the line like this one.
I'm happy to read through the source code, but any hint as to where to start reading would be really appreciated.

Is this problem specific to any Wayland compositor or only Hyprland? I spent hours trying to understand why playerctl and hyprctl don't work.
Could this please be added to FAQ or README?


fix customSpecificationsMapping for kind 'class' and 'struct'

This PR fixes a bug in exhale/utils.py:specificationsForKind(kind). When using a custom directive function with customSpecificationsMapping, the specifications for the kinds struct and class were overwritten.

Good catch, thanks for providing the fix!  Released in v0.3.1 just now :slightly_smiling_face:


osrs_quest_viewer does not complete child-dependency quests when parent is completed

I noticed if you mark Desert Treasure 1 as complete the quests required for its completion stay in the graph. I think it would make more sense to the users if child quests were marked completed automatically in a circumstance like this.

That's a good point. The "Complete" functionality has two purposes 1) to mark quests as complete, and 2) to hide quests. For example, a player who is restricted to 10HP may not be able to complete a quest like DT1, so for them, they would want the quest hidden - not completed. It is equivalent for the quest to not exist, so it should be hidden but it shouldn't impact the prereqs.
So, I think the best way forward would be to add a toggle between whether all prerequisites should be completed, or if the user just want's to complete 1 specific quest.
An alternative could be three states per node: Uncompleted, Completed, and Hidden, but that complicates things. You can also check this short video for an example of me walking through the quests manually in a chain: https://youtube.com/shorts/V_icyISouVk?si=wVrM3ijrgbNLuwHH

I added a second button "Finish", which will complete all the prerequisites. "Complete" will still only hide the selected quest.
Here is an example of finishing While Guthix Sleeps.
Before:

After Finishing While Guthix Sleeps (1 click):

Thanks for the suggestion. If there are issues we can reopen.


ci: add Python version to cache key

PR Checklist

[x] A description of the changes is added to the description of this PR.
[ ] If there is a related issue, make sure it is linked to this PR.
[ ] If you've fixed a bug or added code that should be tested, add tests!
[ ] Documentation in docs is updated

Description of changes
https://github.com/fpgmaas/deptry/actions/runs/3320886881/jobs/5487809759#step:4:20 highlights that we are currently running tox under Python 3.10 regardless of the Python version sent as an input, because we share the cache key regardless of the version.

Codecov Report

Merging #173 (06a634b) into main (6f54023) will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff          @@
##            main    #173   +/-   ##
=====================================
  Coverage   95.1%   95.1%
=====================================
  Files         28      28
  Lines        895     895
  Branches     146     136   -10
=====================================
  Hits         852     852
  Misses        28      28
  Partials      15      15

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


fix(sec): upgrade gopkg.in/yaml.v3 to 3.0.0

What happened？
There are 1 security vulnerabilities found in gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b

CVE-2022-28948

What did I do？
Upgrade gopkg.in/yaml.v3 from v3.0.0-20210107192922-496545a6307b to 3.0.0 for vulnerability fix
What did you expect to happen？
Ideally, no insecure libs should be used.
The specification of the pull request
PR Specification from OSCS

Since this is indirect, we should push upstream to update their dependencies:
$ go mod why gopkg.in/yaml.v3
# gopkg.in/yaml.v3
github.com/opencontainers/image-spec/schema
github.com/xeipuuv/gojsonschema
github.com/xeipuuv/gojsonschema.test
github.com/stretchr/testify/assert
gopkg.in/yaml.v3

It doesn't look like gojsonschema has been touched in a while, so that may be a sign that we should investigate other options.

I think it can be ok to update the version to be ahead of what go modules resolved to, if we have a good reason. Go modules define the minimum compatible version, but SemVer allows for more current (minor/patch) releases to be used. Updating the indirect would  allow us to "recommend" / "force" our consumers to follow.

We fixed this be removing the yaml dependency entirely.


Experiments during review

All changes are present inside a different folder; so we can simply merge the branches.


How to bootstrap setuptools from source

Hi, I'm a developer for the Spack package manager. We've had a setuptools package that has worked fine for a while now, and the vast majority of our Python packages depend on it. However, I went to update to the latest version of setuptools and noticed that it now depends on six, packaging, and appdirs. Unfortunately all 3 of these packages also depend on setuptools. six and packaging can fall back on distutils.core if necessary, but the setup.py for appdirs has a hard setuptools import.
It seems to me like it is no longer possible to build setuptools from source. Is this correct??

An easy solution to this problem would be to replace:
import setuptools

with:
try:
    from setuptools import setup
except ImportError:
    from distutils.core import setup

in the appdirs setup.py. That's what six and packaging do. If you agree, I can reach out to the appdirs devs and see what they think.

Ok, I hacked our package manager so that it can still install setuptools. https://github.com/LLNL/spack/pull/3198 contains all of the changes necessary to build setuptools from source now that the dependencies are no longer vendored. I understand the frustration that comes with having to maintain vendors of all of your dependencies; I just hope that you don't add any dependencies that cannot be built with distutils.core instead of setuptools.

Can you use pip to install the dependencies from wheels (which don't require setuptools to install)? That's the recommended procedure. Or can your package manager install the wheels?

Unfortunately, a package manager that requires another package manager kind of defeats the purpose, don't you think? It looks like other developers have already contributed patches to fix appdirs and pyparsing so that they can be built without setuptools. Assuming the developers approve those patches, we should be good for now.

Yes, I sort of see that. I worry this approach may conflict with the long-term strategy for setuptools, which is that it might require arbitrary dependencies, some of which require setuptools. In fact, an early goal is to move pkg_resources into its own packages, and that package would probably require setuptools to install. Is there a reason your package manager can't have pre-built versions of these packages available or vendor them itself (into the setuptools build recipe)?

Is there a reason your package manager can't have pre-built versions of these packages available or vendor them itself (into the setuptools build recipe)?

In general, yes, there is a reason. Our package manager is designed to install software in the world of HPC, where you may need to use exotic compilers like xlc to install something on a bluegene-q or cray supercomputer. Since we need to support hundreds of operating systems and dozens of compilers, we haven't spent much time on supporting binary package installation like Python's wheels or Homebrew's taps. We have plans to do so in the future, but it has been low priority. Obviously, the compiler/OS doesn't matter much for non-compiled packages like setuptools, but the mechanism would have to be the same.
We have dealt with circular dependencies before. For example, pkg-config depends on glib and vice versa. Luckily the developers realized this, and pkg-config comes with it's own internal copy of glib, much like setuptools used to come with its dependencies. This can be annoying since we end up having to add the same patches for glib to both packages, but it prevents a circular dependency which is nice.
We could theoretically vendor the dependencies ourselves. That seems like the easiest solution to me if we ever run into a setuptools dependency where distutils.core is insufficient to handle the installation. I see a lot of packages that use distutils.core as a fallback. Out of curiousity, how common is it to find a package that uses features of setuptools that are not available in distutils.core?

I'd say it's fairly common. And some packagers might be depending on those features and not realizing it, because pip will pre-import setuptools and setuptools monkey-patches distutils, so even a package that only imports distutils.core will be installed with all the setuptools hooks when installed under pip. Of course, this all should change when PEP 518 lands and makes it possible to declare build dependencies and thus enable dropping the implicit invocation of setuptools.

@tgamblin @svenevs may also want to get in on this discussion.

Muahahahaha!  @adamjstewart I hope this discussion pushes spack over the edge and you ditch the efforts of replicating PyPi.  Things like numpy and scipy -- totally, keep them, if people pip install them then they are at fault (spack has never been easy, you can make minimal assumptions).
For things like setuptools or sphinx, there's literally no reason for spack to be providing those.
The below may not be appropriate for this thread, and some of it gets convoluted quickly.  Please disregard that which you feel is not appropriate here, apologies in advance!
@jaraco is it correct that the general expectation will be

A user installs python (aka their OS, which came with python in most cases)
setuptools will never be packaged directly into this (the whole distutils split -- different conversation), but with the changes in #581 are now expected to install a pre-built version.
This pre-built version should be independent of anything managed by pip in the ideal case.

AKA frown upon ye who pip install setuptools?

Is this correct, or should pip be used to install setuptools as the preferred approach on systems that have it available in both their native package manager and they received pip out of the box?  The conversation got a little confusing on the other thread, setuptools is fundamental to so many things on pip, it seems reasonable to just assume it exists after a certain point.
Do you happen to know if there is a "safe" way of configuring pip to not install / touch anything underneath the site-packages directory?  E.g. only supporting virtualenv or ~/.pip?  Adam and I had both looked into doing this, but it sounds like you are saying even if we found a way to prevent pip from behaving this way, that might cause problems (the "monkeypatching" business).

@jaracao: how do the plans for setuptools affect efforts by OS distro folks to create reproducible builds (podcast)of packages? It seems like these changes will make it very hard to validate a Python package entirely from source, and there are good reasons to make that possible.
Even if you have build dependencies and can install the build deps from wheels, that is still a binary in the chain that someone has to trust. So I worry a bit about this direction.  I suppose that we could implement some bootstrapping logic that goes back to older versions of setuptools, if we need to, or we could rely on a baseline set of "trusted" wheels for setuptools (adamjstewart is right that we don't have binary installs yet, but they're not that hard to add).  But I'd rather reproduce from source, and I suspect other distros would too.
@svenevs: other than reproducibility, the main issue with relying on pip in spack is that we need to be able to mirror builds on airgapped networks. Spack traverses a dependency DAG, downloads all the sources, and lets you build that on networks that aren't connected to the internet. If we rely on pip, we can't do that.

In general, requiring yourself as a dependency to bootstrap yourself leads to a lot of headaches. For example, we are having a lot of nightmares trying to package openjdk/icedtea right now. If you need a java compiler to build a java compiler, how do you install the first java compiler? C/C++ compilers generally come with the operating system, but I don't think java compilers do.

suppose that we could implement some bootstrapping logic that goes back to older versions of setuptools, if we need to, or we could rely on a baseline set of "trusted" wheels for setuptools

I wouldn't recommend using older versions of setuptools. That's an unsustainable approach in the long run.
You could rely on trusted wheels, or you could even vendor your own bootstrapping logic. That is, you could write your own routine that resolves the setuptools build dependencies (or hard-codes them), builds them into a viable form for import, and injects them onto sys.path before building setuptools.
Hmm. This makes me wonder if setuptools should bundle its build dependencies. Rather than vendor its dependencies in general, it could bundle its build dependencies. I'll give that a go.

I've drafted a new release in the feature/bundle-build-deps branch and released b1. Install it with pip install -i https://devpi.net/jaraco/dev/+simple/ setuptools==34.3.0 or grab the sdist and test it in your build environment.
I notice that this change won't affect the most common use-case, that of pip installing setuptools from source when dependencies aren't met. It fails because the setuptools shim is invoked before setuptools has a chance to inject its build dependencies.

This makes me wonder if setuptools should bundle its build dependencies. Rather than vendor its dependencies in general, it could bundle its build dependencies.

Can you elaborate on the distinction between the two options?

Prior to Setuptools 34, setuptools would vendor its dependencies, including them as hacked-in copies of the dependencies. Thus import pkg_resources.extern.six would give the vendored version of six if present as pkg_resources._vendor.six or would fall back to the top-level six otherwise. It would do this even when installed. In this model, the dependencies were undeclared (nothing in install_requires).
With this new proposed technique, the dependencies are bundled into the sdist only to facilitate building and installing from source... but the dependencies are still declared as install_requires, thus relying pip or easy_install or whatever install those requirements, but providing a zip of those dependencies to use to build the package prior to installation. In the installed environment, there are no import hacks and the packages must have been installed properly and naturally.

To summarize, this would allow us to build setuptools from an sdist on PyPI without other dependencies, correct?

With this new proposed technique, the dependencies are bundled into the sdist only to facilitate building and installing from source... but the dependencies are still declared as install_requires, thus relying pip or easy_install or whatever install those requirements, but providing a zip of those dependencies to use to build the package prior to installation. In the installed environment, there are no import hacks and the packages must have been installed properly and naturally.

Ok, so if understand this correctly, we could then declare setuptools to have no (circular) build dependencies, and it would install fine from source.  That is good.  Then we could use the installed setuptools to install other packages as needed.
One question: are any of the build dependencies also run dependencies?  If they are, it seems like we'll still need to make sure we install the bundled run dependencies along with setuptools when bootstrapping, or the installed setuptools won't run.

are any of the build dependencies also run dependencies?

Yes. At the moment, the build dependencies and install dependencies are identical. So you would need those installed to invoke setuptools... which I can see is problemmatic if you then want to install the setuptools dependencies from source and they require setuptools.
I guess if you're using setup.py install, that will invoke easy_install, which will grab the installed dependencies and install them, but I don't recommend that.
I guess what it's coming down to is that whatever installs from source needs a way to install setuptools and its dependencies in one operation (as pip does with wheels or the setuptools did when the packages were vendored).

I guess if you're using setup.py install, that will invoke easy_install, which will grab the installed dependencies and install them

Won't work on air-gapped clusters that have no connection to the outside world. We've been very careful about making sure that all of our packages can be installed without an internet connection.

I guess what it's coming down to is that whatever installs from source needs a way to install setuptools and its dependencies in one operation (as pip does with wheels or the setuptools did when the packages were vendored).

Ok, so I guess I have two questions:

Could we have an option to install the bundled dependencies along with setuptools?  That would get us started.

Is that not basically the same as vendoring the dependencies?  If so, what was so bad about vendoring, especially since setuptools is effectively a leaf in every DAG?  I'm trying to figure out what problem is being solved by vendoring (feel free to point me at some other discussion...)

It's different than just vendoring because it only takes place at build time, thus it allows people to say, upgrade the packaging that setuptools is using without having to also upgrade setuptools itself.

@dstufft: ok, so if we can get the bundled dependencies installed from source initially, so that we can use setuptools to install the other packages from source, does that fit the model, assuming the installed setuptools will use the newer versions when it finds them?

I would assume you could still run the same step yourself that is used to generate the sdist if that is what you are meaning.

This all seems overly complicated. If setuptools's dependencies do not require setuptools to build, then Spack doesn't have a problem. At that point, it sounds like pip and its import setuptools shim becomes the real problem.
I understand the desire to behave like any other Python package, where you declare your dependencies and the latest and greatest version always gets installed first. But setuptools isn't like any other Python package. If every Python package in existence requires setuptools to build with pip, then setuptools cannot have any dependencies. If this can't be achieved internally, then I don't see the problem with vendoring your dependencies. Yes, it becomes more difficult to build setuptools with, say, an arbitrary version of packaging. But there are also downsides to allowing arbitrary versions of packaging. If an update to packaging breaks setuptools, it breaks every Python package in existence. If instead the version of packaging was vendored, it would have to go through various tests before the vendored version is updated.
I'm sure there was a lot of debate that went into the decision to remove vendored packages. If you can point me to that thread, I would love to hear more arguments as to why it is necessary.

There was very little debate. The discussion was hosted in #581 and I'll write up some more detail on the tradeoffs.

Could we have an option to install the bundled dependencies along with setuptools?

An option to which. setup.py install? I think that would require having the dependencies not only in an importable form but also in an installable form. That form could be .eggs, except that eggs are deprecated in favor of wheels.

What was so bad about vendoring, especially since setuptools is effectively a leaf in every DAG?

See the motivations in #581, one of which is to help make setuptools not a leaf in every DAG. But you're right, everything essentially depends on setuptools, which means that every system needs a way to make setuptools available to build packages. Pip is one way to make that happen. rwt is another (relies on pip). virtualenv also seems to be able to create environments with setuptools without a network connection, perhaps you could rely on that to build the packages and then pip to install them. I'll experiment with that.

@adamjstewart Here's an example of building setuptools from source without hitting the network. It of course relies on having virtualenv installed (which itself ensures that setuptools is installed and available):
grab the sources
$ pip download --no-binary :all: setuptools
Collecting setuptools
  Downloading setuptools-34.2.0.zip (621kB)
    100% |████████████████████████████████| 624kB 1.5MB/s
  Saved ./setuptools-34.2.0.zip
Collecting packaging>=16.8 (from setuptools)
  Using cached packaging-16.8.tar.gz
  Saved ./packaging-16.8.tar.gz
Collecting six>=1.6.0 (from setuptools)
  Using cached six-1.10.0.tar.gz
  Saved ./six-1.10.0.tar.gz
Collecting appdirs>=1.4.0 (from setuptools)
  Downloading appdirs-1.4.1.tar.gz
  Saved ./appdirs-1.4.1.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools)
  Using cached pyparsing-2.1.10.zip
  Saved ./pyparsing-2.1.10.zip
Successfully downloaded setuptools packaging six appdirs pyparsing

disable network
create a build environment
$ virtualenv builder
Using base prefix '/Library/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/jaraco/builds/builder/bin/python3
Also creating executable in /Users/jaraco/builds/builder/bin/python
Installing setuptools, pip, wheel...done.

build the sources
$ builder/bin/pip wheel --no-cache --find-links . --no-index ./setuptools-34.2.0.zip
Processing ./setuptools-34.2.0.zip
Collecting packaging>=16.8 (from setuptools==34.2.0)
Collecting six>=1.6.0 (from setuptools==34.2.0)
Collecting appdirs>=1.4.0 (from setuptools==34.2.0)
Collecting pyparsing (from packaging>=16.8->setuptools==34.2.0)
Building wheels for collected packages: setuptools, packaging, six, appdirs, pyparsing
  Running setup.py bdist_wheel for setuptools ... done
  Stored in directory: /Users/jaraco/builds
  Running setup.py bdist_wheel for packaging ... done
  Stored in directory: /Users/jaraco/builds
  Running setup.py bdist_wheel for six ... done
  Stored in directory: /Users/jaraco/builds
  Running setup.py bdist_wheel for appdirs ... done
  Stored in directory: /Users/jaraco/builds
  Running setup.py bdist_wheel for pyparsing ... done
  Stored in directory: /Users/jaraco/builds
Successfully built setuptools packaging six appdirs pyparsing

All of this is predicated on setuptools being available before building from sources. Perhaps that's an assumption Spack should guarantee when building from sources (and either vendor that functionality or provide a mechanism to bootstrap it). Eventually, PEP 518 will become a reality and be able to declare their build dependencies, but until then, setuptools should be considered an implicit dependency for building any python package.
Is there a reason an approach like this wouldn't work for Spack?

So there exist systems where given a set of dependencies (python and otherwise) build everything from source is a requirement. This gets complicated when you have circular dependencies. Imagine a situation where you want to install setuptools:

You have a source tarball of setuptools, so you build a setuptools binary (it can be a wheel or some other system it doesn't matter). This works with the PR in #982 just fine.
You discover that setuptools requires packaging, six, appdirs.
You attempt to build appdirs from source, which requires setuptools to build, so you attempt to go and install setuptools, which fails because while you have a setuptools binary package you don't have it's runtime packages.

AFAICT having setuptools depend on anything which itself depends on setuptools (build or runtime) is always going to make an assumption that you have somehow managed to have a copy of setuptools + all dependencies already built already available. Any system which cannot have that assumption will not work.
This isn't unique to setuptools either. Even with PEP 518 a build tool is not going to be able to depend on itself directly or transitively without breaking from scratch source rebuilds.

Given that this is not unique to setuptools and any build tool is going to possibly have this problem in addition to the fact that pip needs all of this too, perhaps rather than de-vendoring the right answer is to just get the relevant stakeholders together to come up with a vendoring solution that works well for all involved, and roll that up as a tool that can be used by everyone.
We have sort of the start of this in pip with an invoke task: https://github.com/pypa/pip/blob/master/tasks/vendoring/init.py which has made handling our vendoring a lot easier. This coupled with pip/_vendor/init.py to allow downstream distributions like Debian to sort of unvendor if they really want to (it's actually still vendored, but the vendoring just done slightly differently).
Having used that task recently, it really is a lot easier to deal with, the primary problem really just becomes needing to make sure you update if one of those have a big bug or something.

How do you create an sdist in 34.x? Maybe by answering this question we can get a better idea of how one is to bootstrap setuptools.

It's done in Travis-CI, such as https://travis-ci.org/pypa/setuptools/jobs/204763612. It invokes bootstrap.py to generate the requirements.txt then installs those with pip.

@dstufft @jaraco: I'm sold on @dstufft's comments and suggestion:

rather than de-vendoring the right answer is to just get the relevant stakeholders together to come up with a vendoring solution that works well for all involved, and roll that up as a tool that can be used by everyone.

I am not convinced any of the other approaches are going to work for folks building from source, and I am pretty convinced that being able to build from source is important.
What do we need to do to get this started?  I'm sure I don't know all the relevant stakeholders, but I can think of Spack (me, @adamjstewart, @lee218llnl), easybuild (@boegel), and maybe Gentoo (@junghans?)?  What about Debian and others distro folks?

Debian would be @warsaw, though Barry has participated pretty heavily in hammering out pip's current approach. There's also @torsava from Fedora that might be interested.

FWIW in Debian we "rewheel" pip's dependencies from installed platform packages.  I don't know where that term came from, but it's basically a technique to turn the OS package back into a .whl.  We use dirtbike to do it in Debian, but the intent is for that tool to be cross platform (contributions welcome!).  In the Debian source package for pip, we indicate which Debian binary packages are used to create the wheels using a Built-Using field in debian/control.  That's primarily for dependency and other distro tracking.
So yes, there's a bit of a chicken and egg problem because we need a setuptools to interact with the platform build tools to build the Debian binary packages (.debs), but once we have those, we've so far been able to rewheel those dependencies.  Then we modify pip source to devendorize and import from those special wheels.  It does become a bit of a problem when pip adopts a new dependency, and theoretically dirtbike might have trouble rewheeling some future package, but so far this approach has worked for us.  Since Debian is in freeze right now (and so is Ubuntu), I haven't looked at any of the new stuff, but no doubt will have to adjust our packaging for the upstream-devendorized versions.

So coincidentally I'm at Hackillinois 2017 and noticed setuptools is one of the projects (along with Spack). So if some setuptools maintainer is here and wants to discuss in person, I'm around all weekend. 😁

Sorry for being late to the party, @radhermit is Gentoo's setuptools packager, but in short Gentoo patches setuptools's dependencies (six, appdirs and packaging) to use distutils.

So after some thorough, thoughtful consideration and conversation with Todd, I've come to the conclusion that it's not possible for a build tool to have dependencies that rely on that build tool if we want to retain the expectation that a target environment can be built entirely from sources without any pre-built artifacts. In #581, we violated that expectation by requiring that installations of setuptools have access to pre-built artifacts of its dependencies. Further complicating the situation is the implicit build dependency of setuptools' dependencies on setuptools itself.
It's not enough that pip can resolve this situation in the general case by relying on pre-built binaries to execute builds because other build systems, even if they use pip to build/install, still want to build from source.
I pushed back hard on re-vendoring the packages because of the detriments that re-introduces, but also because I wish to create an ecosystem in which there's only one package that requires special handling w.r.t. vendoring (pip) and other build tools, which is mainly setuptools now but could eventually include other python package build tools, can be implemented as normal packages with normal dependencies.
One question that come up for which I didn't have a good answer was 'could pip and setuptools merge?' In that case, all the specialness of vendoring with pkg_resources and setuptools would once again be consolidated in one package. I hadn't considered this proposition much in the past, but thought maybe some had, though I didn't find anything in a search. What's your reaction @dstufft to this proposition? I'm slightly resistant to this because it does run contrary to enabling other build tools to exist.
Ultimately, I think we settled that setuptools should be responsible for monitoring and working with its dependencies to ensure that they can all be installed from source without setuptools being installed in advance (essentially must rely on distutils only). I've started down this road by writing tests (above) to perform such a check and assert this guarantee (and detect regression). The test still needs work as it hard-codes the dependencies (mainly to include nested ones), but also because pyparsing currently fails the assertion.
Although the tests do assert that the "unknown option" warnings are ignorable (through a whitelist), I still have a little uncertainty as to how reproducible these builds are (as they might appear different when built with and without setuptools). I'd like to expand the tests to assert some guarantees about this case as well. There's also a concern that I'd like to explore that if pkg_resources became its own package, it too could not depend on setuptools for installation. The current draft of the independent package does rely on setuptools to build.
@junghans, @radhermit, @warsaw, and others, how does this proposal suit your use cases? Does this provide a sustainable model in which build tools such as setuptools can have dependencies.

For Gentoo, everything should work fine as long as unvendored setuptools deps don't require setuptools to build/install. Otherwise we'll have to patch them internally as we did before :smile:.

I'd really not like for pip and setuptools to merge.  In Debian, our setuptools package is relatively straightforward, though we're at 33.1.1 due to the various release freezes in effect (as it happens, for both Debian and Ubuntu).
pip on the other hand is anything but a straightforward package.  We have to devendor all the bundled packages, and we have a rather elaborate mechanism for "rewheeling" system packages for those dependencies.  Basically it means taking an apt-installed package and turning it back into a wheel.  Then we arrange for those wheels to show up early in the import path of pip.  @dstufft has been amazing at helping us simplify this process and making it fairly robust, but it's not something we'd like to not have to replicate for other packages (pip as you say is rather special), especially for something so fundamental as setuptools.  Unlike pip, which is mostly a leaf package, setuptools is pretty much the bottleneck for a huge swath of the Python ecosystem.
Philosophically, I don't think merging setuptools and pip makes sense.  I thought the consensus was toward separating build and install, and that seems like it would be going backward.
For a while I've thought that we need some kind of meta-package for vendoring which would make it easier for downstreams to devendorize.  We have complications for requests, urllib3, etc. which all have to be devendorized.  I think support like that belongs in Python, but I don't have a clear vision of how it would work, so I've not spent any time on it.
If I understand your proposal correctly, you're going to 1) not vendor dependencies; 2) ensure that dependencies can always be installed with distutils - without downstream hacking.  If so, +1.  I'd like to get Matthias to weigh in since he's technically the sole maintainer of setuptools on Debian, but I don't know if he has a GH account so I can't nosey him in.  (I have forward the url for this issue to him though.)

@doko42

I would also be against merging pip and setuptools, for many of the reasons already listed.
One possible glitch in the plan to only depend on distutils using packages is that distutils using packages are not fully functional. Since distutils does not support install_requires you technically end up with a slightly broken package (in that it doesn't declare any dependencies). To my knowledge, this is unlikely to be a major problem given that I think all of the systems where a complete, from source, rebuild is a requirement tend to have their own dependency handling anyways (even if it can use install_requires as an input to that). Distutils also lacks the ability to create wheels, so for the Python toolchain specifically we still can't do a from source/"zero" rebuild (at least, not automatically. It'd have to be done more manually) but I'm not sure that is as big of a deal as it is for these other systems since there is less of an impetus to do source rebuilds from zero there.

So I just wanted to chime in and say I agree with this solution and that if setuptools' dependencies are always installable from source using distutils, we're good!  Thanks for talking this over at HackIllinois, @jaraco!
RE: @dstufft:

I think all of the systems where a complete, from source, rebuild is a requirement tend to have their own dependency handling anyways (even if it can use install_requires as an input to that).

This is true for Spack.  If we do one day delegate to pip for some installs (e.g., the ones that don't need compilers) then leveraging install_requires could still be useful as you suggest.
The only thing I still have questions about is this, RE: @jaraco:

I still have a little uncertainty as to how reproducible these builds are (as they might appear different when built with and without setuptools)

What are we missing if we install setuptools' dependencies using distutils vs. setuptools?  Surely the installed Python code isn't different, right?  My understanding is that setuptools can install some extra metadata that distutils doesn't, but I don't know who consumes that metadata.  We would just need to be able to ensure that nothing breaks if the extra stuff is missing, as we'll have to install some things with distutils.
@jaraco and I talked about having setuptools' dependencies only use distutils, so there's only one authoritative version of these packages.  Is that an option?

six is no longer installable without setuptools, see https://github.com/benjaminp/six/issues/183

@mjpieters: we patch this in Spack and the build still works, but ideally @jaraco would work with the six maintainers to ensure they're always buildable without setuptools.

I was recently looking at some code in setuptools that could take advantage of requests, and I started to think about what it would take to use requests in setuptools in the model I've proposed. Setuptools would need to work with requests to see that it's installable without setuptools, which may not be possible with some of the declarations it uses like extras_require or include_package_data and maybe others. I'm pretty sure requests can't be built on distutils only.
Plus, with the additional bootstrapping constraints that this approach creates, I'm inclined to reconsider vendoring. The general constraint I'm trying to avoid is "any Python build library must depend only on stdlib", but this constraint runs in conflict with the goal to "allow a packaging system to bootstrap an environment entirely from source distributions."
In this archived  branch, I explored only vendoring the dependencies for building only, but as we've observed, that approach is only sufficient to get setuptools installed (but not its dependencies).
As I'm tempted to re-introduce vendoring of its dependencies, I'd like to find a technique that doesn't lose the advantage of declared dependencies.
If re-introducing vendoring, I'd like to achieve the following:

The setuptools code to simply import the dependency modules or at least have a clean (one-line) equivalent for importing (no try/except).
Default installation mode would declare and install dependencies.
Setuptools would prefer natural dependencies to vendored ones (so someone installing the package naturally would be using the resolved dependencies).
Enable system packagers to remove vendored packages and rely entirely on declared/managed dependencies.
Mechanism for incorporating dependencies should be derived from simple dependency declaration, not relying on commits and edits of other code bases copied into the setuptools code base.

The pip vendoring technique and the Setuptools 33 vendoring technique achieves 1 and 4 but not 2, 3, and 5.
And worse, (2) is what runs in conflict with the goal to avoid circular build-time dependencies in the build tools. But, if setuptools were to re-vendor its dependencies, it could be granted special consideration when resolving build dependencies, allowing it to satisfy the build dependencies even when its dependencies aren't met. What I struggle with still is how to declare this special consideration. I don't want setuptools to be special because it's called setuptools. I'd like for any build tool to be able to do the same (such that any given package could opt-opt of setuptools and opt into another build system with a compatible interface). Even in Setuptools 33 or the revendor branch, there's no obvious reason why setuptools doesn't need its dependencies to run.
I still don't feel like I have a clean solution here that doesn't end up leaving Setuptools (and thus any build tool) as a special-needs player in the ecosystem.
I'll continue to mull about this issue, which I don't feel is resolved cleanly yet.

FWIW it would be fairly trivial to adopt the latest version of the pip based technique so that it got you 3, although I'm not sure it's a great idea.
The latest version of pip's toolchain more or less gives you 5 if I understand what you're saying correctly, while the code does get committed to the repository, managing that is done by a invoke task which completely blows away any local changes and reimports the code from upstream. It does rewrite any imports as needed so that they are relative imports and it has the capability to carry manual patches against the dependencies (applied automatically when importing the upstream code base).
Ultimately I think 2 is what is causing the problem here though, and I can't think of a solid way of handling that which doesn't end up as an even "grosser" solution than vendoring.

The Nix package manager is quite similar to Spack and so in Nixpkgs we've encountered the exact same issue. We prefer to build from source/sdist but we have also a function to build packages from wheels so that gives us a bit more flexibility.
Some time ago I looked into how we could bootstrap setuptools, now that it doesn't vendor its dependencies anymore. The easiest solution for us would be to use the wheel for all its dependencies, but we prefer not to because we ideally refer to a "true" source and not a build product.
I agree that pip and setuptools should not be merged. They really have different functionality and now we're also getting more methods for building Python packages, like e.g. flit, it becomes even more important to have them separated.

Taking another stab at this issue, I have some thoughts, which ramble a bit, so I've broken it down into three segments.
The Two-Package Solution
I have another idea, almost a proposal. What if setuptools provided two packages, the current, natural one as is currently in place, and another, perhaps named setuptools_built or setuptools_bundle or setuptools_all, which is essentially a pre-built, dependency-free package (perhaps just setuptools with its dependencies vendored). This special package would be "build dependency" of setuptools itself and any of its dependencies. As a build-only dependency, installers would be advised only to install this package in a transient manner for the duration of the build step, enabling packages like appdirs and requests and pyparsing to rely on setuptools for their builds (as they did before). This package, much like Setuptools 33 and earlier, would be buildable and installable with distutils only from source.
This approach is potentially messy as it relies on installers not to install this package permanently, and might create weird behaviors that might prove difficult to troubleshoot (i.e. which setuptools is present when an error occurred). It does, however, nicely solve the bootstrapping problem (providing a bootstrapping package) and also elegantly solves the "which setuptools is required to install a given setuptools" problem, as setuptools would declare its own build dependency, allowing setuptools to eschew much of the logic in setuptools.sandbox which was created to support the self-install model.
To some extent, this idea is dependent on the implementation of PEP 518 and would required build system like Spack and system packages to honor that spec when building packages. That feels like a fairly small investment if it ultimately means that setuptools (and any other build tool) can declare its dependencies naturally, allow itself and its dependencies to depend on it (or other build tools), but can also own the responsibility for providing bootstrapping.
Build Systems Should Special-Case Build Dependency Resolution
Alternatively, these build systems could be adapted to have a special handling for build-time dependencies, such that if a package has a build-time dependency on setuptools (or other build tools), the resolution isn't to build/install those packages, but instead has a distinct mechanism for providing those packages. This approach would allow each build tools to have control over how these bootstrapping issues are addressed. Some might just vendor the known build dependencies (namely setuptools today, but sure to include others). Others might do some clever trick to resolve the dependency tree and put the sources on sys.path while building the build dependencies.
PyPA Provides Build Dependency Resolver
As I think about it more, perhaps this clever trick is exactly what we should develop - a tool for building and installing (transiently or permanently) a set of packages that are mutually dependent on each other. This tool would have special handling for working with source packages and making them available without installing them.
For the following example, I'll call this tool "source-resolver".
So for example, a system package builder wants to build 'appdirs'. 'appdirs' declares it has a build-time dependency on "setuptools" (any version), so it wraps the build step in call to source_resolver('setuptools'). That routine will retrieve the setuptools source, extract it to a temporary directory, add it to sys.path, and then recurse on its dependencies. It will do this in a context and undo its changes at the close of the context. In this context, the builder invokes the build steps on appdirs, producing the package (and possibly installing it).
source_resolver, like pip, could take an argument of a local directory where the sources are already present.
This approach would require that all build dependencies are importable from source without an intermediate build or install step. Or alternatively, the spec could be expanded to allow packages to declare which of their build dependencies require this special handling. Or maybe this routine is used as a fallback when a build fails using the normal process.

There's a lot going on here, so I'm going to break for now and let these thoughts continue to simmer.

Reposted from distutils-sig:
Today, I ran into trouble working with an old project that had six pinned to version 1.1.0.  The install failed because buildout tried to install it as 1.10.0 and failed because 1.10.0 was already installed.
The problem arose because six's setup.py imports setuptools and then imports six to get version.  When Buildout runs a setup script, it puts it's own path ahead of the distribution, so the setup script would get whatever version buildout was running.  IMO, this is a six bug, but wait, there's more.
I tried installing a pinned version with pip, using pip install -U six==1.9.0. This worked. I then tried with version 1.1.0, and this failed, because setuptools wouldn't work with 1.1.0. Pip puts the distribution ahead of it's own path when running a setup script. setuptools requires six >= 1.6, so pip can't be used to install pinned versions (in requirements.txt) earlier than 1.6. Six is a wildly popular package and has been around for a long time. Earlier pins are likely.
I raise this here in the broader context of managing clashes between setuptools requirements and requirements of libraries (and applications using them) it's installing.  I think Buildout's approach of putting it's path first is better, although it was more painful in this instance.
I look forward to a time when we don't run scripts at install time (or are at least wildly less likely to).
Buildout is growing wheel support. It should have provided a work around, but:

I happened to be trying to install a 1.1 pin and the earliest six wheel is for 1..
I tried installing six 1.8. Buildout's wheel extension depended on pip, which depends on setuptools and six.  When buildout tries to load the extension, it tries to get the extension's dependencies, which includes six while honoring the version pin, which means it has to install six before it has wheel support. Obviously, this is Buildout's problem, but it illustrates the complexity that arises when packaging dependencies overlap dependencies of packages being managed.

IDK what the answer is. I'm just (re-)raising the issue and providing a data point.  I suspect that packaging tools should manage their own dependencies independently. That's what was happening until recently IIUC for the pypa tools through vendoring. I didn't like vendoring, but I'm starting to see the wisdom of it. :)

This (depending on six, appdirs, pyparsing, etc) really complicates packaging on Heroku, FWIW.

@kennethreitz: Is that because Heroku builds these packages from source? Why not install from wheels using pip? What is the complication?

@jaraco Heroku installs setuptools (and pip) by default into those environments, but people tend to also depend on libraries like six, appdirs, etc in their own projects hosted on Heroku. The unvendoring means that setuptools and the project code are now competing over who gets to define what an acceptable version is for these libraries to be installed with.

This is an issue every other distribution is going to face eventually as well. Therefore, I recommend that each build tool vendors its dependencies. At least setuptools should because to install a setuptools-based package one needs to import from setuptools.

@jaraco people do pip freeze > requirements.txt from older setuptools installations, which then conflicts with our installation.
Here's what I'm doing about it. https://github.com/heroku/heroku-buildpack-python/pull/397

Sorry if this has already been said, this discussion is quite long and have unfortunately lost the time to read it fully. That said, am still very much struggling with this issue on many fronts.
What if setuptools provided a mechanism to vendor in-place and then used that functionality for managing these dependencies? Given the discussion here it seems that vendoring needs to happen for setuptools, but also this needs to be done in a way where dependencies can be tracked and easily update. This could also be a feature for those that also need vendoring for other reasons. Admittedly vendoring is not a great thing to use, but some cases (like setuptools) seem to need it. Might as well make it less painful.

With Setuptools 36, it once again vendors its dependencies.

\o/

If I'm not too overenthusiastic... "when will it be released?" (I don't see it on pypi yet).

Well, it should have been released automatically when I tagged the commit. I'll have to investigate why.

Aah, there was an error in the deploy step due to the removal of requirements.txt.

It should now be on PyPI.

Seems to not have been deployed as that change was not on a tag. Would it make sense to add a 36.0.1 tag?

IMHO it would make sense to just upload the release manually using twine.

That's weird. I definitely ran the commands to cut a release. Oh, I ran setup.py release which built the dists but didn't upload them. :/

Using twine, as recommended, the dists are now (verifiably) in PyPI.

Great thanks for doing this.

@jaraco Hi! Another developer from Spack doing some necromancy to maybe resurrect an old issue :skull: We were hit by this problem today:

https://github.com/spack/spack/issues/31109

which was triggered by the new 64.0.0 release. The gist of it seems to be that, trying to install from source setuptools 62.3.2 with the following command:
/tmp/py38/bin/python3 '-m' 'pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/home/culpo/.spack/bootstrap/store/linux-ubuntu20.04-x86_64/gcc-9.4.0/py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne' '.'/tmp/py38/bin/python3 '-m' 'pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/home/culpo/.spack/bootstrap/store/linux-ubuntu20.04-x86_64/gcc-9.4.0/py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne' '.'

from a virtual environment that has setuptools 64.0.0 installed, fails when trying to create the metadata from the wheel. The error looks like the following:

Excerpt of the stacktrace
[ ... ]
Non-user install due to --prefix or --target option
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /tmp/pip-ephem-wheel-cache-z64i02nz
Created temporary directory: /tmp/pip-build-tracker-x6romquo
Initialized build tracking at /tmp/pip-build-tracker-x6romquo
Created build tracker: /tmp/pip-build-tracker-x6romquo
Entered build tracker: /tmp/pip-build-tracker-x6romquo
Created temporary directory: /tmp/pip-install-6b6sr5bk
Processing /tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src
  Added file:///tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src to build tracker '/tmp/pip-build-tracker-x6romquo'
  Created temporary directory: /tmp/pip-modern-metadata-jcsudk0s
  Running command Preparing metadata (pyproject.toml)
  running dist_info
  creating /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info
  writing /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/dependency_links.txt
  writing entry points to /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/entry_points.txt
  writing requirements to /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/requires.txt
  writing top-level names to /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/top_level.txt
  writing /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/dependency_links.txt
  writing entry points to /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/entry_points.txt
  writing requirements to /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/requires.txt
  writing top-level names to /tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-jcsudk0s/setuptools.egg-info/SOURCES.txt'
  Traceback (most recent call last):
    File "/tmp/py38/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/tmp/py38/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/tmp/py38/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
      return hook(metadata_directory, config_settings)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/build_meta.py", line 188, in prepare_metadata_for_build_wheel
      self.run_setup()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/build_meta.py", line 174, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 87, in <module>
      dist = setuptools.setup(**setup_params)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/__init__.py", line 87, in setup
      return distutils.core.setup(**attrs)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/core.py", line 148, in setup
      return run_commands(dist)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/core.py", line 163, in run_commands
      dist.run_commands()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/dist.py", line 967, in run_commands
      self.run_command(cmd)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/dist.py", line 1229, in run_command
      super().run_command(command)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/dist.py", line 986, in run_command
      cmd_obj.run()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/command/dist_info.py", line 35, in run
      egg_info.run()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/command/egg_info.py", line 308, in run
      self.find_sources()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/command/egg_info.py", line 315, in find_sources
      mm.run()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/command/egg_info.py", line 550, in run
      self.add_defaults()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/command/egg_info.py", line 587, in add_defaults
      sdist.add_defaults(self)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/command/sdist.py", line 226, in add_defaults
      self._add_defaults_python()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/command/sdist.py", line 111, in _add_defaults_python
      build_py = self.get_finalized_command('build_py')
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/cmd.py", line 299, in get_finalized_command
      cmd_obj.ensure_finalized()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/cmd.py", line 107, in ensure_finalized
      self.finalize_options()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/command/build_py.py", line 33, in finalize_options
      orig.build_py.finalize_options(self)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/command/build_py.py", line 43, in finalize_options
      self.set_undefined_options('build',
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/cmd.py", line 286, in set_undefined_options
      src_cmd_obj = self.distribution.get_command_obj(src_cmd)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_distutils/dist.py", line 858, in get_command_obj
      klass = self.get_command_class(command)
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/dist.py", line 966, in get_command_class
      self.cmdclass[command] = cmdclass = ep.load()
    File "/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src/setuptools/_vendor/importlib_metadata/__init__.py", line 194, in load
      module = import_module(match.group('module'))
    File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
    File "<frozen importlib._bootstrap>", line 991, in _find_and_load
    File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
  ModuleNotFoundError: No module named 'setuptools.command.build'
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /tmp/py38/bin/python3 /tmp/py38/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmppj2rk5qg
  cwd: /tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src
  Preparing metadata (pyproject.toml) ... error
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
[ ... ]

and from:
ModuleNotFoundError: No module named 'setuptools.command.build'
  error: subprocess-exited-with-error

I got the impression that the "system" setuptools v64.0.0 was injected by pip during the install process. Part of it is maybe our fault, since we explicitly pass:
--no-build-isolation

for good reasons explained above, but since setuptools specifies no requirements:
[build-system]
requires = []
build-backend = "setuptools.build_meta"
backend-path = ["."]

I'm uncertain if this is a bug on setuptools, or on pip or or just our misunderstanding of how setuptools should be bootstrapped with PEP 518 now in place. Do you see better ways to bootstrap setuptools from sources?

Hi @alalazo, there might be some variation of commands between versions of setuptools (that is expected as setuptools development progresses).
This is the case of setuptools.command.build, which does not exist for older versions.
It seems that your setup is using an older version of setuptools to build a new one, which would explain the error...
Ideally we want setuptools to build itself without the help of older versions.
I am not sure how pip handles the --no-build-isolation, but for normal builds  some paths might leak into the build environment depending on the operating system (e.g. Debian-based).

I run some experiments, and I think the problem the problem is the following:

It is uncommon, but newer versions of setuptools may have a different set of entry-points. This is part of the natural evolution of the project (this will be the case for example when we implement PEP 660).

If you run the build with pip/--no-build-isolation on pip or build/--no-isolation, existing entries of PYTHONPATH are left behind. This means that existing setuptools-*.dist-info directories will be picked up by importlib.metadata when using entry-points for auto-discovery.

Entry-points from different versions can be incompatible and cause this type of problem.

I don't know if we can support this kind of usage right now (maybe we can do it in the future). For the time being I recommend bootstrapping setuptools in an environment without setuptools, or use build isolation.

Just as a curiosity (that might help us to understand what is going on), would you face the same error if you use build to create a wheel? (It also has the --no-isolation flag).

I tried some manual experiments and build works fine both with and without isolation:
$ ls
changelog.d  conftest.py  _distutils_hack  exercises.py  LICENSE      PKG-INFO       pyproject.toml  README.rst  setup.py    setuptools.egg-info  tox.ini
CHANGES.rst  dist         docs             launcher.c    MANIFEST.in  pkg_resources  pytest.ini      setup.cfg   setuptools  tools

$ python -m build --no-isolation
[ ... ]
Successfully built setuptools-62.3.2.tar.gz and setuptools-62.3.2-py3-none-any.whl

This is not the case with pip[^1], which makes me think pip might be at fault here?
[^1]: I tried pip 20.0.2 and 22.1.2
Another data point is that currently we rely heavily on setting PYTHONPATH ourselves. Since we install each package into its own hashed directory (see https://github.com/pypa/setuptools/issues/980#issuecomment-281232729 for some background of the Spack project) and we want control of how things are built, PYTHONPATH is the mechanism by which we ensure a build environment with --no-isolation finds its requirements.

@abravalheri You're correct, build works fine only with isolation - or without if setuptools is not installed on the system.

I hacked with my virtual environment and I confirm that the issue is indeed https://github.com/pypa/setuptools/issues/980#issuecomment-1154535704 Just before the failure these variables are set:
PEP517_BACKEND_PATH=/tmp/culpo/spack-stage/spack-stage-py-setuptools-62.3.2-fde2yob53gvhzxmq3qwc2ym3fkcyrdne/spack-src
PEP517_BUILD_BACKEND=setuptools.build_meta

and they seem to respect what pyproject.toml prescribes. Inspecting the entry points though, one could see those stemming from the setuptools v62.4.0 installed on the system.


refactor(autoware_autonomous_emergency_braking): add getObjectOnPathData and getLongitudinalOffset functions

Description
I moved some functions to utility functions in a util file. No logic changes were made.
Related links
Parent Issue:

Link

How was this PR tested?

[x] run psim

Notes for reviewers
None.
Interface changes
None.

Effects on system behavior
None.

@danielsanchezaran

Could you please check for degradation
you mean in CI/CD?

There seems to be an issue with the end of the predicted path? @kyoichi-sugahara
https://github.com/user-attachments/assets/c87756fd-9732-45d9-98fe-7c987bfbd5f3

normal OSS autoware does not seem to have that issue
https://github.com/user-attachments/assets/3368a26c-7cf0-4207-96f1-442fd5a99f18

@danielsanchezaran

There seems to be an issue with the end of the predicted path?

I couldn't understand issue directly sorry :pray:
Can you specify what's the issue?

@danielsanchezaran

There seems to be an issue with the end of the predicted path?

I couldn't understand issue directly sorry 🙏 Can you specify what's the issue?

Did you watch the video? There is no AEB activation when we set a bus over the goal pose

@danielsanchezaran

yes, scenario evaluation

I started to run the simulation :+1:
I did some debugging and it seems all points are being filtered out by this processing:
    obj_arc_length > path_length + longitudinal_offset) {
    return {};
  }

You can see the attached video.
https://github.com/user-attachments/assets/4f03864f-990a-4fa4-8efd-1bb8d2cbc49a
Regarding the difference between before and after the changes - while it may not be the intended behavior, from the perspective of excluding points that exceed the path length, the behavior after the change seems more correct. What do you think?
By the way, when I commented that "I couldn't understand issue directly sorry" I meant that after watching the video, I couldn't understand why the behavior was problematic. Sorry if that wasn't clear.
I can provide route files that can reproduce the issue in recrete_p_sim_scenario along with debug print differences if needed - please let me know if you'd like those.

@danielsanchezaran

yes, scenario evaluation

I started to run the simulation 👍
I did some debugging and it seems all points are being filtered out by this processing:
    obj_arc_length > path_length + longitudinal_offset) {
    return {};
  }

You can see the attached video.
2024-11-26_19.30.10.mp4
Regarding the difference between before and after the changes - while it may not be the intended behavior, from the perspective of excluding points that exceed the path length, the behavior after the change seems more correct. What do you think?
By the way, when I commented that "I couldn't understand issue directly sorry" I meant that after watching the video, I couldn't understand why the behavior was problematic. Sorry if that wasn't clear. I can provide route files that can reproduce the issue in recrete_p_sim_scenario along with debug print differences if needed - please let me know if you'd like those.

the object arc_length should not be larger than the path length + longitudinal offset, you can visually see the predicted path going inside the parked bus. One of the variables (path length , longitudinal offset, object arc length) is wrong I believe

@kyoichi-sugahara I will approve this PR, the changes look good, but please fix the front_overhang and use the max_longitudinal_offset_m before merging!


get real MAC address in Android 6+

Offical document : https://developer.android.com/about/versions/marshmallow/android-6.0-changes.html?hl=zh-cn#behavior-hardware-id
Android 6+ cann't get real MAC address by call wm.getConnectionInfo().getMacAddress();

@Tneciv please send your pull request conforming to rules specified in Readme. i.e to only develop branch.


filezilla: migrate to wxGTK32

Description of changes
From https://filezilla-project.org/versions.php:

All official FileZilla binares now link against wxWidgets 3.2.1

Things done

Built on platform(s)

[ ] x86_64-linux
[x] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 221148 run on aarch64-linux 1

1 package built:

filezilla