Configure hosts separately from targets when --target is specified.

This prevents target configs from accidentally being picked up when cross compiling from hosts that have the same architecture as their targets.
closes #3349

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @ehuss (or someone else) soon.
If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.
Please see the contribution instructions for more information.

One quick bit of feedback: I don't think this should just be host. This should be host.(name of host's target triple), so that you can specify different flags/etc for different host architectures.

This should be host.(name of host's target triple), so that you can specify different flags/etc for different host architectures or operating systems.

I don't see a realistic use case for having host target triples since each build host should only need one host configuration for all targets as the build scripts must run on the build host and shouldn't change depending on targets, using target triples here seems to complicate things unnecessarily.

@jameshilliard I have several use cases for that. Projects often check in .cargo/config.toml files, and users may do something similar to share their personal .cargo/config.toml files. Things like flags or linkers may vary by platform; for instance, you may want one setting for Windows hosts and another for Linux, or one setting for x86-64 and another for ARM64.

Thanks for the PR @jameshilliard! We talked about this in the Cargo meeting today for quite awhile and I'm going to try to summarize our thoughts all down in writing. As you've discovered this is a particularly large thorn in Cargo's side, and it's been there for quite some time!
Overall we generally favored this design, but one thing we're worried about is backwards compatibility. This will break any usage of [target] configuration that would apply to both the host and the target (when they're the same). Another concern that I have is that this is close but not quite to fixing some surprising behavior with cargo build. From our discussion, however, I think we've got a way forward on all this, and I'm curious to get your thoughts on it!
The changes we were thinking for this PR are:

Instead of host.linker we'd have host.$target.linker in the same manner as the target table today (to future-proof config used simultaneously for different hosts, e.g. on Windows).
Add a new flag to the target table, something like target.$target.applies-to-host which is default true (Cargo's historical default). If this flag is set to false, though, then the [host] table is consulted for host builds.
This lends itself well to a fix for the cargo build situation as well. With these configuration knobs in place cargo build would configure the target/host build separately if host configuration is present or target configuration which doesn't apply to the host (e.g. applies-to-host is false).
Once all this support has been stabilized and has sat on the stable channel for a release or so, then we'd switch the default of applies-to-host from true to false.

How does that sound? This is definitely a more long-term plan to get this fixed but the hope is that we can do this with minimal breakage and if something breaks there's recourse for users to fix it. Additionally I think this should fix all if not almost all of the issue about cargo build and cross compilation and such.
The points about "fixing" cargo build and updating its behavior with respect to the host table is a little more involved than just this PR itself. It's fine for that to be a separate PR or implemented by someone else later. Until that's all implemented though I would personally prefer to land this support unstable in Cargo because it's so closely related to those changes as well (and it seems good to evaluate/stabilize all at once).

Instead of host.linker we'd have host.$target.linker in the same manner as the target table today (to future-proof config used simultaneously for different hosts, e.g. on Windows).

I think this is unnecessarily complex, these host config flag settings aren't exactly portable in most cases so having multiple sections for different host build script targets doesn't appear to be useful.

This will break any usage of [target] configuration that would apply to both the host and the target (when they're the same).

This should behave as before as long as one doesn't manually set the target(which one generally only does when cross compiling AFAIK). The only thing I could see this breaking are some obscure edge-cases(obscure enough they aren't even tested for) where one is manually setting a target that also needs custom flags for the build script(most of the time build scripts don't need custom flags so this generally shouldn't cause any issues).

Add a new flag to the target table, something like target.$target.applies-to-host which is default true (Cargo's historical default). If this flag is set to false, though, then the [host] table is consulted for host builds.

Well it doesn't exactly unconditionally default to true here but rather it inconsistently chooses what to do based on the host and target architectures being the same, in most cases when cross compiling the default will be false as long as the target arch is different from the host. Relying on setting target.$target.applies-to-host to false would result in the existing highly inconsistent/unexpected behavior as applying target configs to host scripts is wrong/unnecessary in almost all cases, the only case where applying target flags to host build scripts is generally safe is when the host is the target(which I have retained the existing behavior for). There's no good reason IMO that the config for cross compiling for a x86_64 target from a x86_64 host should require special casing vs a config for compiling for a aarch64 target from the same x86_64 host.

Once all this support has been stabilized and has sat on the stable channel for a release or so, then we'd switch the default of applies-to-host from true to false.

I really don't think this is a good idea, currently the behavior of cargo arbitrarily applying target configurations to host scripts is objectively wrong but only breaks in a fatal way for specific configurations, and by providing a host table we ensure that in the rare case build scripts need custom flags that there is a way for them to be provided still. Since this change should not break any common use cases I think we should just default to doing the sane thing and not apply target flags to build scripts except in cases where the host is the target(target not set).

This is definitely a more long-term plan to get this fixed but the hope is that we can do this with minimal breakage and if something breaks there's recourse for users to fix it.

While this does change a default it doesn't appear likely to cause significant breakage, and with a [host] section it should be trivial to fix for those where this breaks.

The points about "fixing" cargo build and updating its behavior with respect to the host table is a little more involved than just this PR itself.

I'm not really seeing a scenario that's not covered by my approach, is there some use case I've missed here?

Projects often check in .cargo/config.toml files, and users may do something similar to share their personal .cargo/config.toml files. Things like flags or linkers may vary by platform; for instance, you may want one setting for Windows hosts and another for Linux, or one setting for x86-64 and another for ARM64.

Do you have an example of what one of those checked in cargo configs would look like? I'd like to see one that this change would break ideally.
Custom build script flags tied to the host architecture just seems like a bad idea for multiple reasons, it means the build output gets affected by both the build host as well as the target which I don't see as desirable in general.
Normally the build output should only be affected by the target settings, the main use case for the [host] table IMO is to give the user a way to fix an unexpected build script compile failures, I wouldn't expect that section to be used at all when cross compiling except to workaround unanticipated issues.

I have done a lot of "complicated" criss-crossed compilation--mostly in the world of C/C++--and am one of the people who has been long affected by this Cargo issue, whilst slowly drowning in failing workarounds ;P, and I am also not sure I see the use case for host-triple specific build rules? The only thing I ever feel I need to pass that is special on the host is super high-level language confirmation flags (the only one I can think of being -std=, but I could see a use case for -funsigned-char). I generally just need my host compiler to "work at all", not be configured in some extra special way (and I thereby try to get away if at all possible with just running "cc" and passing -I/c/o, though in practice I often "care" a bit and run "gcc" or even "clang"). It could be that people are doing something very different in Rust-land with their host build steps, or there is an entire level of complexity far past where I have ever gone?

@jameshilliard
A few examples of things I'd actively like to do that would require different options depending on the target triple the host uses:

Use lld for performance when building on hosts where it works reliably, but not on hosts where it doesn't.
Use target-feature flags for host code, for build scripts that do CPU-intensive work (such as building complex data structures) and would benefit from optimization. (These are the same kinds of build scripts that motivate turning on optimizations even in debug mode.)
Use a specific compiler, but only on targets where that path makes sense (e.g. using a specific compiler on Windows and a different specific compiler on Linux).

Also, to clarify, I don't object to having [host] for the case of options identical on every host; I just want to make sure, when we're adding this, that it's possible to specify options specific to the architecture or OS of the host.

@joshtriplett The first and third bullets don't actually seem like they should be checked in as files to your project as part of your cargo configuration, as that is probably going to make them attempt to get applied to a build of your project even if you are just some embedded dependency; at a "high-level", I'd argue that the person doing the build should get to make these determinations if at all possible, as these are both like, toolchain issues and not target issues. Like, to the extent to which there are faster ways of doing these builds, shouldn't that just be defaults built into cargo, or decisions made by the system toolchain distributor? (I'd even argue that is mostly true of the second one as well, though I can appreciate some stored guidance on math flags or something, but even there I find it a bit sketchy and I'm highly concerned that projects using it will become hard for me to compile.)

.cargo/config.toml files checked into a crate do not apply when that crate is used as a dependency, only when it's the top-level crate you're building, and they don't apply when using cargo install of a remote crate (e.g. from crates.io). They primarily make sense for top-level binary crates. Also, those options have value in ~/.cargo/config.toml as well.

Use target-feature flags for host code, for build scripts that do CPU-intensive work (such as building complex data structures) and would benefit from optimization. (These are the same kinds of build scripts that motivate turning on optimizations even in debug mode.)

Ah, I hadn't realized that build scripts were used for CPU-intensive work like that.

Also, to clarify, I don't object to having [host] for the case of options identical on every host; I just want to make sure, when we're adding this, that it's possible to specify options specific to the architecture or OS of the host.

I've reworked this a bit so that cargo will first look for a arch specific host table along the lines of [host.$hosttripple] if one that matches does not exist it will then fallback to a non arch specific [host] table.

Hm so fwiw it'd be nice to have a little more engagement with the concerns of the Cargo team. It seems like after writing up our thoughts you response was basically "no", which isn't really engaging and can come across as dismissive. As the Cargo team we try to guide Cargo with our shared experience and history with Cargo, and dismissing that because this PR works for you sort of feels like you're ignoring all the past experience we've learned maintaining Cargo over the years.
The problem of different host/target configurations has been with Cargo since the beginning, and it's pretty deeply ingrained in the design of Cargo itself. (no one's attempted to fix this in the ~8 years I've been maintaining Cargo) That alone basically means that trying to fix this is going to be a somewhat nuanced and difficult problem to tackle. Not impossible of course!
From what I can recall there's two primary issues with Cargo's treatment of this configuration. One is that cargo build does not "implicitly cross-compile" which means that you can't separately configure things like build scripts and the final artifact. This gives rise to problems such as when you enable sanitizers you have to specify --target because you don't want to sanitize build scripts and proc macros. The second problem is that even if you do specify --target it'll pick up the cargo configuration if the host target matches the --target value.
Your PR is attempting to address the second of these problems, but the two problems, in my opinion, are quite intertwined. Your PR is a breaking change to Cargo beacuse it is changing Cargo's behavior. If someone's project builds on stable/beta/nightly this means that if this PR were to land then that project would have no recourse to build across stable/beta/nightly. To be clear, the breakage here is that if --target is specified and it matches the host value then the [target] configuration values which previously applied are no longer applied. Also to be clear I'm not saying which behavior is write or wrong, I'm stating that behavior is changing and therefore breaking. This breakage namely means that this change needs to be handled with care.
If you don't want to change your PR from how it is then it will not land in Cargo due to it being a breaking change. If you're willing to change your PR, however, then we can all engage together and try to work towards a design that solves everyone's goal which is to satisfy Cargo's stability guarantees while also solving the underlying problems.

It seems like after writing up our thoughts you response was basically "no", which isn't really engaging and can come across as dismissive. As the Cargo team we try to guide Cargo with our shared experience and history with Cargo, and dismissing that because this PR works for you sort of feels like you're ignoring all the past experience we've learned maintaining Cargo over the years.

I'm looking for the best solution to the problem, which is why I was trying to understand the reason one would want to pass arch specific flags to the build scripts, some of your suggestions seemed strange to me so I was trying to understand the use cases behind them, clearly I had missed at least one use case as performance optimizing build scripts like this is somewhat unusual in general. I have a lot of experience with cross compilation in as I maintain a number of buildroot packages so I'm trying to find a clean maintainable solution here that behaves/integrates well with downstream projects, so don't take my response as a "no" but rather a request for additional context/info.

The problem of different host/target configurations has been with Cargo since the beginning, and it's pretty deeply ingrained in the design of Cargo itself. (no one's attempted to fix this in the ~8 years I've been maintaining Cargo) That alone basically means that trying to fix this is going to be a somewhat nuanced and difficult problem to tackle. Not impossible of course!

I'm quite familiar with systems that have different host/target configurations, we do this sort of host/target split in buildroot as well in various ways for other languages, and this is probably not the most complex situation I've seen there. I don't think this is a fundamental or particularly difficult problem to solve so much as a thorny edge case that got missed.

Your PR is a breaking change to Cargo beacuse it is changing Cargo's behavior.

Based on @joshtriplett's description of how non-default host configs would be used the main use case for passing configurations to host scripts is for performance optimization or passing certain feature flags to the host build scripts. This change may result in a performance regression in some existing build scripts but would not typically cause fatal errors based on my understanding, so even though it's a behavior change it's probably not a breaking one in practice for most users and is easily fixable.

If someone's project builds on stable/beta/nightly this means that if this PR were to land then that project would have no recourse to build across stable/beta/nightly.

I think my approach here has a good migration path as a cargo config that has host tables should still build fine on older cargo versions that don't recognize them(older cargo versions would just look at the target tables), so retaining the existing behavior while remaining backwards compatible does not appear to be an issue here(I reworked my PR this way after I was explained the use case for arch specific host configs) and one should be able to provide a single cargo config still that will work across the stable/beta/nightly versions still.

To be clear, the breakage here is that if --target is specified and it matches the host value then the [target] configuration values which previously applied are no longer applied. Also to be clear I'm not saying which behavior is write or wrong, I'm stating that behavior is changing and therefore breaking. This breakage namely means that this change needs to be handled with care.

Projects can effectively retain the existing behavior in a backwards compatible way if they want to by simply adding host copies of the target tables, but this is unlikely to be needed/desirable for most projects from my understanding as this behavior has buggy edge cases a lot of the time in practice.

Hm so fwiw it'd be nice to have a little more engagement with the concerns of the Cargo team.

Is there some other place I should be trying to discuss this with the cargo team, it has been difficult getting any serious engagement relating to this issue(as you can see it's a longstanding issue that does not appear to have had a seriously attempted root cause fix until this PR) from a cargo maintainer that is familiar with this area of the code-base. While I'm an expert in cross compilation I am not an expert in rust or cargo, I barely even know any rust at all and was working on this because an important python library in one of my dependency trees decided to make rust mandatory recently.

If you're willing to change your PR, however, then we can all engage together and try to work towards a design that solves everyone's goal which is to satisfy Cargo's stability guarantees while also solving the underlying problems.

Yes, that's what I've been trying to do, is there a better channel for discussing potential solutions to this problem? Is there documentation regarding what cargo's stability guarantees are exactly? It's unclear what exactly the stability constraints are for a potential fix.

I'm sorry that it seems like it is difficult to get engagement.  I realize and sympathize that it is frustrating when something downstream breaks, and it can be particularly difficult if you are unfamiliar with the tool or language. I've been there before, too.
The Cargo Team is a very small group of volunteers with very limited time to contribute to the project. Unfortunately, we are unable to respond to every request for help, as there are just too many.
There is a process documented here for working through new changes. This change lands somewhere closer to the "large features" section, where we encourage people to work with the community to come up with ideas and to consider the design space.  But even then, it is impossible for us to help with everything people want.

Is there documentation regarding what cargo's stability guarantees are exactly?

The stability guidelines are documented here.

The Cargo Team is a very small group of volunteers with very limited time to contribute to the project. Unfortunately, we are unable to respond to every request for help, as there are just too many.

I get that, but IMO this issue should have been prioritized due to it being relatively simple to fix and having a high impact on certain projects.

There is a process documented here for working through new changes. This change lands somewhere closer to the "large features" section, where we encourage people to work with the community to come up with ideas and to consider the design space. But even then, it is impossible for us to help with everything people want.

So IMO this is essentially two different changes combined, the first change is to fix the bug where cargo picks up target configs for host scripts if the target and host archs are the same. The second is to add additional config tables so that anyone who actually happened to depend on setting host config flags would still have a way to do so with the first bug fixed.
Should I split the first bug fix out separately from the host table feature or keep them combined? I guess I need to open a RFC for the host table feature right?

The stability guidelines are documented here.

Ok, so it looks like the first bug fix would be allowable per the policy of allowing "Changes to resolve possibly unsafe or unreliable behavior." for "situations where backwards-incompatible changes are made".

Since it seems the host config table feature will take some time to work through the design process I went ahead and split out the bug fix part of this PR in #9346 which should be enough to get host==target cross compilation working in most cases.

Based on @joshtriplett's description of how non-default host configs would be used

My comments were regarding why we need a [host.$target_triple] table. That's separate from the concerns that @alexcrichton posted about backwards compatibility.
We absolutely agree that eventually Cargo should stop looking at [target.$target_triple] for host configuration. However, today, that's the only mechanism to configure either host or target, and Cargo's de-facto behavior is that it looks at [target.$target_triple] for both host and target. Breaking that (by causing host compilations to stop looking at it) will potentially break code that may be counting on the application of those options. That's not just an optimization issue, it's a potential correctness issue, given the wide range of things that may be done via rustflags or linkers or similar.
The transition plan we're proposing is this:

Introduce the [host] and [host.$target_triple] tables, as you're doing.
Introduce a new applies-to-host field in the existing [target.$target_triple] tables, defaulting to true.
When looking up options for the host, if any host tables are present or if the applicable [target.$target_triple] has applies-to-host = false, don't apply options from the target table. But otherwise, if there are no host tables and the applicable target table has applies-to-host = true (either by default or specified explicitly) continue applying options from an applicable [target.$target_triple], for backwards compatibility, and perhaps emit a transition warning if applies-to-host is defaulting to true.
Wait a while (6+ months) for people to notice.
Then change the default of applies-to-host to false.

Does that transition plan make sense?

So, here's what I'm seeing: there are three changes that need to be made. The first is to add a new feature using like 30 lines of code--the host configuration mechanism--for which it seems like there isn't disagreement any longer. The second is to add maybe 15 lines of code and move another 5 lines of code to fix the current issue blocking cross-compilation on Cargo from being viable (at least without tons of workarounds); this is also now "done" (modulo the third change).
The third change is that the Cargo team has decided that they have a backwards-compatibility strategy and breaking change transition plan that they would like to see for these two changes: there was an internal discussion, and Alex and Josh seem pretty certain about what they want to see done here. It sounds like this will require one new config file variable and a couple if statements to hack at the behavior of the previous two changes... maybe another 10 lines of code?
Is there something about Cargo's process that makes it seem more efficient to argue with James (btw: I don't speak for James; my only knowledge of James' existence is because he was kind enough to fix this bug that has deeply affected me, as someone who does cross-compilation) rather than just type these 10 lines of code, close the issue, and allow everyone to move on with their lives? If James decides he just doesn't want to type out this code himself, will Cargo never fix this?

Breaking that (by causing host compilations to stop looking at it) will potentially break code that may be counting on the application of those options. That's not just an optimization issue, it's a potential correctness issue, given the wide range of things that may be done via rustflags or linkers or similar.

So the correctness aspect is a big reason I'm not a fan of using a applies-to-host option for migration, because the existing behavior is fundamentally incorrect/broken when cross compiling but tends to only cause breakage in limited circumstances.
I suspect the amount of projects that are effectively already broken when cross compiling by the existing incorrect behavior is far greater than the amount that would be broken by fixing this.

perhaps emit a transition warning if applies-to-host is defaulting to true

Forcing all projects to apply this flag to silence a warning just doesn't seem like a good idea to me when most projects don't actually depend on the current broken behavior but would typically work either way most of the time(with the new behavior being most correct).
Since there's no realistic way for cargo to determine if a project relies on the broken target flags being passed to the host scripts behavior the best approach IMO is to correct the behavior and let projects that break update their configs when they break.
Due to this change allowing for the previous behavior to be replicated exactly with a cargo config that's also fully backwards compatible with cargo versions prior to the change I would argue that immediately defaulting to the correct split behavior will cause far less maintenance burden than requiring all projects to set a flag to silence a warning which should only actually be an issue for a small amount of projects.
For the projects this would break their configs are almost certain to already have thorny edge cases around cross compilation anyways so I see little reason to not try and force them to fix the issue when upgrading cargo as a migration path with a warning most projects shouldn't have to care about would IMO just cause confusion.
The projects that rely on the ability to pass target flags to host scripts are also more likely to have experienced developers that can quickly fix their configs when they break as requiring target configs being passed to the host scripts is a fairly advanced/uncommon use case already.

Does that transition plan make sense?

How about this instead for a transition plan?:

Introduce the [host] and [host.$target_triple].
Default to the new correct split behavior.
Add a message explaining this behavior change that shows up only if the following three conditions are met: a host target fails to compile, a --target is specified, there are no host tables present in the config.
Add an environmental variable(which we would mention in the above error message) that can be set to revert to the previous behavior temporarily.
Remove the behavior overriding environmental variable(which we should always warn on when used before removal) after projects have had a chance to update their configs to split the target and host flags properly.

Is there something about Cargo's process that makes it seem more efficient to argue with James (btw: I don't speak for James; my only knowledge of James' existence is because he was kind enough to fix this bug that has deeply affected me, as someone who does cross-compilation) rather than just type these 10 lines of code, close the issue, and allow everyone to move on with their lives? If James decides he just doesn't want to type out this code himself, will Cargo never fix this?

I'm not saying I'm not willing to make such a change, but I want to make sure that the cargo maintainers properly understand my reasoning for why I don't think the applies-to-host migration strategy is a good idea before I go try and implement it as there are a number of negative implications to that strategy that I am seeing which may not be all that obvious to the cargo maintainers.

The third change is that the Cargo team has decided that they have a backwards-compatibility strategy and breaking change transition plan that they would like to see for these two changes: there was an internal discussion, and Alex and Josh seem pretty certain about what they want to see done here.

I'm trying to make sure I understand all the reasoning behind the applies-to-host migration strategy suggestion so that I can make suggestions for improvements but this is a bit tricky when I'm unable to review the details of this discussion(if there are public logs for this discussion I haven't been able to find them at least). I'm also trying to make sure I'm not missing anything non-obvious here myself as I have very little experience with rust/cargo in general. When the bulk of these sort of discussions are internal it's difficult for outside contributors like myself to get a proper understanding of all the issues/implications.

On Sat, Apr 10, 2021 at 09:28:05PM -0700, James Hilliard wrote:

Breaking that (by causing host compilations to stop looking at it) will potentially break code that may be counting on the application of those options. That's not just an optimization issue, it's a potential correctness issue, given the wide range of things that may be done via rustflags or linkers or similar.

So the correctness aspect is a big reason I'm not a fan of using a applies-to-host option for migration, because the existing behavior is fundamentally incorrect/broken when cross compiling but tends to only cause breakage in limited circumstances.

The difference is that the existing behavior has existed for many years,
and there may be people currently counting on it.
Nobody is arguing that the current behavior is correct, just that we
need a transition to avoid abruptly breaking people with no warning.
If someone has a configuration that previously didn't work, making it
work is great. However, if someone has a configuration that previously
did work, making it stop working is a regression. On balance, in
almost all cases, we very strongly favor avoiding regressions. We want
to make sure that people who currently have a working configuration
don't get broken.
Once [host]/[host.$target_triple] support exists, it'll be possible
to immediately fix configurations that didn't previously work, without
waiting for Cargo's defaults to change, just by adding a host table.

perhaps emit a transition warning if applies-to-host is defaulting to true

Forcing all projects to apply this flag to silence a warning just doesn't seem like a good idea to me when most projects don't actually depend on the current broken behavior but would typically work either way most of the time(with the new behavior being most correct).
Since there's no realistic way for cargo to determine if a project relies on the broken target flags being passed to the host scripts behavior the best approach IMO is to correct the behavior and let projects that break update their configs when they break.

We don't want to break people's existing configuration.
That said, regarding a transition warning, I said "perhaps" for
precisely this reason: such a warning may produce a fair bit of
unnecessary noise.
But on the other hand, it would be trivial to silence the warning,
once, by either adding an empty [host] table or by adding
applies-to-host = false.

Does that transition plan make sense?

How about this instead for a transition plan?:

Introduce the [host] and [host.$target_triple].
Default to the new correct split behavior.
Add a message explaining this behavior change that shows up only if the following three conditions are met: a host target fails to compile, a --target is specified, there are no host tables present in the config.

"fails to compile" is the best case scenario. "Subtly doesn't work as
expected" would be much worse.
And even if that weren't the case, it's much more disruptive to break
people and tell them they need to change their configuration, rather
than warning people that they need to change their configuration soon
and then giving them time to do so.

When the bulk of these sort of discussions are internal

The summary Alex posted was intended to be a complete record of the rationale; there's no hidden discussion you're missing that wasn't summarized into the issue. It came up in the standing weekly Cargo meeting, we were quite happy that someone was working on this, and we just needed to address compatibility with existing users.

why I don't think the applies-to-host migration strategy is a good idea

You've talked a little about why you don't feel it's necessary, and why you feel like users will be able to notice and fix brokenness without a transition.
However, I haven't seen your explanation for why you think a transition plan would be harmful. As soon as the initial change goes in, it'll immediately be possible to write Cargo configuration that works correctly for cross-compilation, and that handles both cargo build and cargo build --target .... The only thing the transition plan delays is changing the interpretation of existing configurations. Anyone directly impacted by this problem doesn't need to wait for the transition plan.

The difference is that the existing behavior has existed for many years, and there may be people currently counting on it. Nobody is arguing that the current behavior is correct, just that we need a transition to avoid abruptly breaking people with no warning.

I'm certainly aware of this, but I'm just not seeing a great way to avoid breaking things for some people.

If someone has a configuration that previously didn't work, making it
work is great. However, if someone has a configuration that previously
did work, making it stop working is a regression. On balance, in
almost all cases, we very strongly favor avoiding regressions. We want
to make sure that people who currently have a working configuration
don't get broken.

I would normally agree that maintaining bug for bug compatibility is generally a good idea, however in this case IMO we kinda need a way to force projects to be made aware of this issue so that they can update their configs.

That said, regarding a transition warning, I said "perhaps" for
precisely this reason: such a warning may produce a fair bit of
unnecessary noise.

Yeah, this is a bit tricky and needs to be fully thought through.

But on the other hand, it would be trivial to silence the warning,
once, by either adding an empty [host] table or by adding
applies-to-host = false.

It's not really an issue for my use case as that would be a one line change, I'm just trying to cover all my bases here.

"fails to compile" is the best case scenario. "Subtly doesn't work as
expected" would be much worse.

Sure, but that's at least probably fairly rare, and the current behavior is already a different type of "Subtly doesn't work as
expected" situation so it's a tradeoff there as well.

And even if that weren't the case, it's much more disruptive to break
people and tell them they need to change their configuration, rather
than warning people that they need to change their configuration soon
and then giving them time to do so.

That would be ideal, I'm just not seeing an effective way to do this that:

Doesn't create a lot of likely unnecessary noise.
Ensures those that need to be aware of this change are actually made aware of it.

If it's decided that creating a bunch of noise with a warning and a flag is the best approach I can implement that, but I'm looking to see if we can come up with something better first.

The summary Alex posted was intended to be a complete record of the rationale; there's no hidden discussion you're missing that wasn't summarized into the issue.

I'm not saying that it's being intentionally hidden or anything, but as someone that has very little background with rust/cargo I had already misunderstood things like why a host.$target.linker option is needed based on Alex's original summary and I want to make sure I'm not missing any additional context.

However, I haven't seen your explanation for why you think a transition plan would be harmful. As soon as the initial change goes in, it'll immediately be possible to write Cargo configuration that works correctly for cross-compilation, and that handles both cargo build and cargo build --target .... The only thing the transition plan delays is changing the interpretation of existing configurations. Anyone directly impacted by this problem doesn't need to wait for the transition plan.

I'm not saying a transition plan is harmful but in order to be beneficial we need to figure out how to handle the discoverability issue otherwise we'll just be delaying the breakage in most cases. I don't see much benefit to a transition period without dealing with that somehow and it's not clear what the best approach is.

@jameshilliard That's true.
I think there are two reasonable ways we could handle discoverability of this transition, with tradeoffs between them:

We could emit a transition warning, unless you have a host table or set applies-to-host one way or the other. This will definitely make sure people are aware, but it'll produce a lot of noise, and the changes needed to silence the warning wouldn't actually be needed after the transition.
We could just make a post on the Inside Rust blog and add a prominent notice to the release notes. This may not notify everyone, but it'll notify many people, so that when we change the default it'll catch fewer people off guard.

I'm fine with either of those alternatives, along with the transition plan from https://github.com/rust-lang/cargo/pull/9322#issuecomment-817240912 . What do you think, @alexcrichton?

We could emit a transition warning, unless you have a host table or set applies-to-host one way or the other. This will definitely make sure people are aware, but it'll produce a lot of noise, and the changes needed to silence the warning wouldn't actually be needed after the transition.

Yeah, having to keep the variable around to make pre-default change versions of cargo behave appropriately would be a bit ugly IMO.

We could just make a post on the Inside Rust blog and add a prominent notice to the release notes. This may not notify everyone, but it'll notify many people, so that when we change the default it'll catch fewer people off guard.

Yeah, due to the likely very low percentage of projects this change would impact I think that's probably the best option.

Sorry I was not working the first half of this week and I was inundated with other work for the rest of the week. I'll set aside time to digest the responses here and respond here again early next week.

Ok I've read over this thread and I don't have anything really to add over what @joshtriplett has already said (thanks @joshtriplett for being active here).
I don't think that it's worthwhile to implement a warning here since it seems like it'll just be overly complicated figuring out emission and suppression, I think it's fine to make a post somewhere about this and also perhaps post on a subset of the ... 11 issues linked here. That will probably get word to all the relevant folks that a change is coming and testing would be appreciated.

The load_host_triple and load_target_triple functions are largely duplicated between each other and should have more logic shared between them. Otherwise what's here seems fine by me so far.

The load_host_triple and load_target_triple functions are largely duplicated between each other and should have more logic shared between them.

This look good?

The approach looks good now, thank you for working on it.
One last thing: whenever we add new syntax like this, we normally introduce a feature-gate, in the form of a named -Z flag. (It's also possible to enable such feature-gates in ~/.cargo/config.toml itself; see https://doc.rust-lang.org/cargo/reference/unstable.html for details.) We can then let people test the new feature, report their experiences, and then we can stabilize it once we have some feedback.
Could you please add a feature-gate using Cargo's existing mechanism for that?

Could you please add a feature-gate using Cargo's existing mechanism for that, as well as adding documentation of the new fields to the unstable-features page linked above?

Should I just feature gate the host table feature part? Is there a good example for how I should do feature-gate checks like this in the codebase?

@jameshilliard Yes, the feature gate should be applied to the host table.
See the documentation at the top of src/cargo/core/features.rs . I would suggest the feature name host-config.

Could you please add a feature-gate using Cargo's existing mechanism for that, as well as adding documentation of the new fields to the unstable-features page linked above?

Added and rebased, does that look right? FYI tests should be passing but github actions seems to be partially down right now.

Looks good to me!
@rfcbot merge

Looks good to me!
@rfcbot merge

Tests can probably be rerun now since it seems the github actions issue is resolved. Tests are passing in my fork here.

I was under the impression that applies-to-host was going to be implemented? As-is I do not think this should merge because it's the same breaking change that was outlined above?

I was under the impression that applies-to-host was going to be implemented?

I thought the we had decided against that approach since there were issues there.

@alexcrichton I think I may have been confused myself. I was under the impression, based on the conversation so far, that we would just make this have no effect unless you specify a host table, which would not be a breaking change. Then later we could plan a transition to always separate host and target, and to default to "every build looks like a cross-compile", which would be the change requiring a careful transition and announcement.
So, two points to be very explicit about to make sure we're all on the same page:

@alexcrichton Do you feel like we need a separate applies-to-host flag, separate from "does the host table exist" and "did the user pass -Z host-config or do the equivalent in the config file"? I thought it would suffice to key this off of the presence/absence of a host table, until we do a future transition to entirely stop looking at the target table for hosts.

@jameshilliard Is this PR intended to have any effect if you don't have an applicable host table? I thought, based on the logic as I read it, that it would continue to fall back to a target table if you don't have a host table. If that's not the case, that's an issue.

FWIW, the thing @joshtriplett is suggesting feels like it would only solve the use cases I have--to consistently cross compile existing Rust libraries (I am not the author of any of the Rust libraries I am compiling: I am just consuming them in my C/C++ program)--if I can somehow have an empty host "table" (which I hope I can do with an environment variable and not a configuration file I need to put somewhere, particularly if I have to write that configuration file into the folder of the library I am compiling... if I have to edit the library I am compiling that would almost be a non-starter for me, and I would continue to try to use ridiculous workarounds such as "set the target linker to /bin/true"), as I have no interest in applying any host-specific configuration.

@saurik It sounds like the problem you have would be best solved once we complete the transition we're talking about. In the meantime, though, yes, you could have an empty host table.

2\. Is this PR intended to have any effect if you _don't_ have an applicable host table? I thought, based on the logic as I read it, that it would continue to fall back to a target table if you don't have a host table. If that's _not_ the case, that's an issue.

Right now it uses target tables when not cross compiling and host tables when cross compiling for host scripts. This at least allows for single config backwards compatible exact replication of the previous behavior if one needs that.

if I can somehow have an empty host "table"

This is currently the default when cross compiling.

@jameshilliard Then yes, that's a problem. The expectation was that that (changing the defaults) was the transition that would happen at some future point (with widespread announcement/communication), not in this PR.
In this PR, we'd like to see the host table introduced, and used if present, so that people can start working with it. We can then make an announcement about it, and once it's stabilized we can mention it in the release notes as well. Then, once that announcement has been made and people can write a configuration that'll work before and after the transition, we can start the transition of the defaults.

In the meantime, though, yes, you could have an empty host table.

Currently the host table is only used when cross compiling so that backwards compatible configs are possible while getting correct behavior when cross compiling in most cases(since usually one doesn't really intend to pass target flags to host scripts when cross compiling). So when cross compiling it means no host table is equivalent to an empty host table.

There needs to be some explicit indication that the user wants the new behavior of not using a matching target table; without an explicit indication of that, we'll break users who expected the target configuration to apply (because at the moment, in the absence of a change in this area, the target configuration is the only way users have to do platform-specific configuration, whether for the target or host).
That explicit indication could be:

The presence of a host table
The presence of an applies-to-host flag
Or, possibly, the presence of -Z host-config, but I'm not sure if that's what @alexcrichton intended and I want to confirm that before suggesting that's sufficient.

What if you're cross-compiling and there is no host table?

It's treated as an empty host table which is usually what one wants when cross compiling.

That needs to look at the target table, just as it does now, for backwards-compatibility.

So it does this when not cross compiling still by default.

In this PR, we'd like to see the host table introduced, and used if present, so that people can start working with it.

Well I think we generally don't actually want to set a host table most of the time when cross compiling since the host defaults should generally work for build scripts.

Then, once that announcement has been made and people can write a configuration that'll work before and after the transition, we can start the transition of the defaults.

Configurations that work now can be updated in a backwards compatible way still.

There needs to be some explicit indication that the user wants the new behavior of not using a matching target table; without an explicit indication of that, we'll break users who expected the target configuration to apply (because at the moment, in the absence of a change in this area

I think the issue with this was that it looked tricky to do in a way that was discoverable for those that needed to know about it while also not creating a bunch of log noise with warnings that most people don't really need to care about.

That explicit indication could be

Well right now the use of target tables for host scripts is tied to the presence of a --target flag or equivalent. But you can copy a target config to it's arch equivalent host config to get previous behavior while being backwards compatible.

@jameshilliard and I discussed this extensively on Zulip. Here's a summary of that discussion:

We agreed that there needs to be an opt-in for the new behavior where host compilations don't look at the target table.
I proposed that the opt-in be a top-level target-applies-to-host = false option. That has the advantage of not being target-specific, so you only have to specify it once, not once per target section. You'll also be able to specify it with CARGO_TARGET_APPLIES_TO_HOST=false as an environment variable.
target-applies-to-host will default to true for now to avoid breaking backwards-compatibility, and after an announcement  and transition period, we can change the default to false.
The logic for both cross-compilations and native compilations will be: if there's a host table use that configuration, otherwise if target-applies-to-host is not false then use the matching target configuration, otherwise use no configuration.
The host table and target-applies-to-host will both be gated behind -Z host-config.
-Z host-config will just be a feature-gate; it won't affect the defaults.
As soon as this goes in, we should start asking people to test -Z host-config, so that we can stabilize it in a timely fashion.
As soon as -Z host-config is stable, people can start using target-applies-to-host = false to opt into the better defaults.
We can make an announcement and start the transition to the new default for target-applies-to-host as soon as we stabilize -Z host-config and that change releases in a stable Cargo.

I'm cancelling the FCP for now, and I'll restart it as soon as the above logic is implemented.
@rfcbot cancel

So I made an update that refactored things that should hopefully handle all cases, but I changed some of the behavior from what we discussed for a few reasons after looking over the current implementation in more detail, let me know how this looks.

We agreed that there needs to be an opt-in for the new behavior where host compilations don't look at the target table. (The current PR doesn't have an opt-in.)

Well I don't actually think it's actually all that necessary myself in practice but doing this looked workable so I've implemented it in my latest commit. This last set of changes gates the new default behavior behind -Z host-config so that we can still test the new behavior ahead of time.

I proposed that the opt-in be a top-level target-applies-to-host = false option. That has the advantage of not being target-specific, so you only have to specify it once, not once per target section. You'll also be able to specify it with CARGO_TARGET_APPLIES_TO_HOST=false as an environment variable.

This is implemented now.

target-applies-to-host will default to true for now to avoid breaking backwards-compatibility, and after an announcement and transition period, we can change the default to false.

This is implemented, although I currently have -Z host-config changing to the new defaults to make it easier to test during the transition.

The logic for both cross-compilations and native compilations will be: if there's a host table use that configuration, otherwise if target-applies-to-host is not false then use the matching target configuration, otherwise use no configuration.

This is a bit tricky to implement cleanly and is kinda confusing I think, it looks easier/cleaner to just couple the new defaults to -Z host-config and change the defaults at the same time as we stabilize -Z host-config. Then by having -Z host-config also gate the new defaults we don't need any host table detection logic that changes target-applies-to-host behavior.

The host table and target-applies-to-host will both be gated behind -Z host-config.

So currently I have this inverted, target-applies-to-host always takes precedence while -Z host-config changes to the new default behavior for target-applies-to-host when target-applies-to-host isn't explicitly set.

-Z host-config will just be a feature-gate; it won't affect the defaults

I do have this changing the defaults currently, I think for the transition we need a way for users(and the cargo test suit) to test the new defaults.

As soon as this goes in, we should start asking people to test -Z host-config, so that we can stabilize it in a timely fashion.

Yeah, this is one reason I think it makes sense to have -Z host-config change the defaults ahead of time.

As soon as -Z host-config is stable, people can start using target-applies-to-host = false to opt into the better defaults.

Currently I have the availability of target-applies-to-host ungated since it's separate from the host table feature and since the host table feature fully depends on target-applies-to-host = false behavior while the target-applies-to-host flag does not actually depend on the host table feature at all.

We can make an announcement and start the transition to the new default for target-applies-to-host as soon as we stabilize -Z host-config and that change releases in a stable Cargo.

So I think the new defaults should be enabled at the same time -Z host-config is stabilized, otherwise it makes the transition rather confusing as you would end up with partial intermediary migrations.
Hopefully these changes should make the transition easier overall by simplifying things a bit and reducing the amount of potential config combinations during the stabilization period.
FYI github decided to break normal test behavior for outside contributors in pull requests by default but tests are passing here at least.

@jameshilliard FWIW, I think the argument for the defaults not being tied to -Z host-config is that Rust has a weird release-cycle concept (which I'm guessing is the non-obvious quirk here that will make the plan make sense) where -Z is fundamentally only available on nightly builds of the toolchain. So, I'm guessing the idea is to spend a week testing that the behavior that will go live a week later--which wouldn't have "the new default"--works, by letting people on nightly opt into the testing using -Z host-config (which, importantly, isn't something people actually using any of these features can include in their builds as this would be for the nightly initial test phase only to verify the feature works). This let's them extra-cautiously test the target-applies-to-host feature doesn't cause some additional bug before it gets launched to everyone, which is then unrelated to the second phase of this--which I guess would be made many months later--to changes the default behavior, remove target-applies-to-host, and actually fix this bug "for real".

This let's everyone extra-cautiously test that the target-applies-to-host feature doesn't cause some additional bug before it gets launched to everyone, which is then unrelated to the second phase of this--which I guess would be made many months later--to changes the default behavior, remove target-applies-to-host, and actually fix this bug "for real".

Actually with the current design I think we would keep target-applies-to-host indefinitely(probably useful for enabling for native builds in some cases), however the default for it would change so that it would no longer be needed to actually be set most of the time. I wanted to ensure the new defaults were actually tested ahead of time which is why I'm using -Z host-config to enable them.

This let's everyone extra-cautiously test that the target-applies-to-host feature doesn't cause some additional bug before it gets launched to everyone, which is then unrelated to the second phase of this--which I guess would be made many months later--to changes the default behavior, remove target-applies-to-host, and actually fix this bug "for real".

Yeah, I think that might make things more complex/tricky though and my current implementation seemed to fit best with the current code at least, at least it seems the least likely approach to cause bugs from what I can tell, testability is a big concern here for me, I'm trying to make sure I get good coverage of a number of potential scenarios up front for the upcoming defaults. Could add another unstable feature flag I suppose to flip the defaults but I think that's probably not needed.

If the defaults and the -Z host-config are tied together, then people wouldn't be able to use the nightly testing phase to verify the behavior of target-applies-to-host (as they might not even be using it).

Well I think it's best to at least have -Z host-config flip the defaults since that should actually replicate what things would look like after stabilization, this I think should reduce the amount of different combinations we need to test as well, since you then only have "previous defaults+no host tables" and "new defaults+host tables" scenarios to test for effectively.

Well I think it's best to at least have -Z host-config flip the defaults since that should actually replicate what things would look like after stabilization, this I think should reduce the amount of different combinations we need to test as well, since you then only have "previous defaults+no host tables" and "new defaults+host tables" scenarios to test for effectively.

That's not what things will look like after stabilization. We need to provide time for people to update their configurations on stable before we change the defaults, and people can't update their configurations on stable until the options to do so are stable. So this needs to be a three-step process: add the configuration behind a feature gate, stabilize the feature gate, after a transition period change the default.
Please make -Z host-config just a feature gate, without having it change the default.

We need to provide time for people to update their configurations on stable before we change the defaults, and people can't update their configurations on stable until the options to do so are stable.

I think I didn't explain this properly but dealing with this issue was a reason I didn't feature gate the target-applies-to-host flag behind -Z host-config but rather made it immediately available without changing the defaults when it is not set, because it essentially allows people to update their configs ahead of the combined default change+host-config feature as setting target-applies-to-host effectively fully overrides the default entirely so that behavior would not change for anyone who ads this.

So this needs to be a three-step process: add the configuration behind a feature gate, stabilize the feature gate, after a transition period change the default.

Yeah, so technically what I was going for is a slightly modified variant of this three step process:

add the target-applies-to-host configuration option so projects can explicitly set their desired default behavior, as soon as we have this projects can start updating their configs by adding this explicit flag ahead of the default switch
add the host configuration behind a feature gate
stabilize the feature gated host table config option and change the defaults simultaniously, as long as projects have set their chosen target-applies-to-host setting they will

The main reason I preferred this approach is that it simplifies testing combinations, if you think the -Z host-config option needs to be stabilized ahead of the default change I think we would want to add a -Z target-applies-to-host-new-default flag or something to allow testing of the new defaults ahead of the default transition, but I think that is probably more error prone than linking it with the host table feature. I'm mostly wary of stabilizing the host table feature ahead of the default change due to increased testing complexity.

Adding target-applies-to-host as an immediately stable option would be unusual for Cargo; normally such things go through a nightly-only testing period just like everything else. The Cargo team will have to discuss the possibility of doing otherwise; that would be an exception to our usual process, and there would have to be a good reason for granting it. In any case, even if we did add that option immediately, that would then be the mechanism to test this with. Making -Z host-config change the default does not make this easier to test, it makes it harder to test, because it mixes the stabilization process for these new options with the planned transition.
I'd also like to express that it feels frustrating, in this PR, when you've repeatedly decided to implement something other than what was previously requested, and rather than asking if that would work, you just do what you'd rather do and then try to convince us to accept that instead.
If you want to suggest a different approach, please talk about it first, and get agreement for that change, rather than just ignoring what we've asked for and doing something else. We're asking for those things for a reason, and it feels like you keep trying to push us to move faster and ignore our normal stabilization processes.

Yesterday was officially the one-year anniversary of my running into this issue cluster, for which I've slowly had to add more and more silly workarounds as my set of platforms, build environments, and notably unfortunately Rust library dependencies (which I'm no longer going to allow in my project: I have burned way way way too much time into Cargo issues that literally never get fixed... this was looking to be the first) have expanded.
I've thereby been watching this issue with baited breadth now for the past month, watching James's discussions on other issues from a variety of other affected projects even before this pull request was filed, ever since he commented on #3349, an issue that has been open for almost five years (and one of the issues I recognized early on was likely heavily related to #8147, an issue I filed on Cargo, which is apparently blocking cc from a fix).

I'd also like to express that it feels frustrating, in this PR, when you've repeatedly decided to implement something other than what was previously requested, and rather than asking if that would work, you just do something different and then try to convince us to accept that instead.

You do realize that you are now seriously arguing over what must amount to a single if statement, right? If you don't want it to work the way James wants it to work, it would take less time to just type the if statement than to type even the single sentence required to express your opinion over said if statement. I get that you're frustrated, but I assure you that you are nowhere near as frustrated as the rest of us are in the audience watching this go down.

If you want to suggest a different approach, please talk about it first, and get agreement for that change, rather than just ignoring what we've asked for and doing something else. We're asking for those things for a reason, and it feels like you keep trying to push us to move faster and ignore our normal stabilization processes.

But... you have all the power in this situation :(. It isn't like anyone is holding you hostage over getting this bug fixed, and certainly not James: if you wanted to, right now, you could take a few minutes out of your day and fix this issue for the numerous developers--and James has identified a reasonably large number of affected projects near the top of this pull request--who have either had to build towers of ridiculous workarounds or are outright blocked on this issue... James couldn't even stop you!
And look: there are a lot of reasons I am generally super sympathetic towards the plight of a project maintainer. I have been there, for a super popular project used by not only tens of thousands of developers but tens of millions of end users (who are often way way more demanding than developers ever could be, as they aren't used to working on software), and I appreciate that it really sucks sometimes, and I wouldn't blame anyone for burning out on it.
I've been in the situation where I've had bugs that I simply don't know how to fix, despite a number of people insisting it was easy to do... and I'd have been willing to believe (not that it was true :/) this was one of them based on prior conversations with Alex. I've been in the situation where people decide to ship a fork with that fix anyway, "compatibility be damned", and if James were intending to do that (he clearly isn't) I'd understand being super super angry.
(I also will admit to having a kind of related "pet peeve": people who show up with massive pull requests they want to get into a project without having first talked to anyone about the approach or even the feature itself... this tiny tiny patch by someone who isn't even a Rust developer clearly isn't that: the vast majority of the time James has spent on this patch has been in this discussion, not on the code; it wouldn't have been some tragic waste of time to re-do the work.)
I've even had bugs where there simply isn't a fix, whether due to technological, legal, or moral reasons: that really sucks... you get to spend every single day for the rest of your life apologizing to people until, once in a while, you snap as your long explanations--which everyone calls "excuses"--are tiresome to write (I am so sick of people demanding that I solve some problem or another by making my federated system centralized... not going to happen, no matter if they are "correct" or not).
Hell: if you simply didn't have either the time or emotional stamina to type the single if statement required to merge this patch, or were conflicted with doubt over your chosen position and felt a need for the community to first "come to consensus" on the solution before you inflicted a change of any form on them, I'd get that also... but neither of those things are happening here: you clearly have a lot of time to devote to typing text towards this issue and you know precisely what you want to have done.
Instead, what I'm seeing, as an outsider to this project, is that James has heroically waded into a codebase in a language that isn't one he normally develops in--in fact, James admitted "I barely even know any rust at all"--to try to get traction on fixing an issue that has been known for at least so long that the key issue filed for it is much closer to the birth of Cargo than it is to today, and despite numerous roadblocks, he pulled it off: not only did he fix the problem, not only was he able to test the fix, but he managed to cause the Cargo team to agree to fix the issue and even come to internal consensus on how the fix should work!
James is my hero today. I feel like, when I'm at my best (and I admit I'm often not), I'm channeling the kind of energy James brought to this patch. James inspires me to work even harder than I already do to fix things upstream, and I hope other people end up feeling the same way about James.
Now, is James part of the "internal consensus" on exactly how to fix this issue? Clearly not, but need he be? If James doesn't want to sit around and be forced--with someone breathing down his neck--to type the one if statement he doesn't agree to, is anyone better off by apparently-how-Cargo-works process of spending weeks (and it literally has been two full weeks since I first noted this issue had left the phase of "reasonable roadblocks" and entered "argumentative deadlock") trying to guilt him into typing it?
Of course / obviously, the answer is "no": no one is better off :(.
You aren't better off, as this is a clear-cut case where it would take you less time to type the code you want than to even explain your interest... if this had been an issue on one of my projects, I'd have just fixed it myself and then rejected the pull request (potentially crediting James if I felt he had been useful).
Developers like me aren't better off: as someone affected by this bug, I'm pretty sure that even the most annoying half-solutions cited on this thread would be "good enough" to remove all of my unstable workarounds (I'd still need workarounds, but I'm pretty sure I could get them to work "consistently").
I can't imagine James is better off: spending weeks arguing with him and making him feel bad for having had the unmitigated gall to try to fix this issue is absolutely worse than if you had just pocket rejected (aka, ignored) the pull request; and knowing that you seem to think he's somehow acting in bad faith here must suck.
Sadly, I think the thing that is probably the most disappointing is that, as someone who cares a lot about cross-compilation--and who has at times been the maintainer of a gcc+llvm/binutils fork for a popular platform, in addition to a rather complete distribution of Unix that I cobbled from scratch and cross-compiled with said toolchain--I think James has been correct to fight you, at least a bit, on many of these points: he's clearly an expert here on cross-compilation scenarios and I don't feel like that background is being in any way respected.
There are projects, today, broken and even blocked with no reasonable workaround, and those projects are being given very little weight versus unsubstantiated theories that either there might exist projects which will be broken so badly by this change that they won't know how to fix it (a scenario I find patently ridiculous, as I can't imagine that any project relying on the current behavior works reliably anyway)... literally the strongest argument here that has so far been made against James' original patch was that there might theoretically exist a project that wouldn't actually break but might simply build a bit slower until they adopt the host table.
But like, this is your project: commit what you want, or don't commit anything at all; decide that James is worth debating with (which isn't what this has been), or decide that you are better off just ignoring him; you have your beliefs about how what stability guarantees mean to you, and you should follow them! ...but, as it stands, I beg you to ask yourself "what message am I sending with how I'm reacting to this pull request", as so far what I'm hearing--loud and clear--from this stand-off is "Cargo is a frozen project that doesn't fix bugs", "the developers behind Cargo prefer to argue than type code", and "non-Rust external project maintainers don't matter to the Cargo team, only Rust developers" :/.

Adding target-applies-to-host as an immediately stable option would be unusual for Cargo; normally such things go through a nightly-only testing period just like everything else. The Cargo team will have to discuss the possibility of doing otherwise; that would be an exception to our usual process, and there would have to be a good reason for granting it. In any case, even if we did add that option immediately, that would then be the mechanism to test this with.

Yeah, I could add a separate gate for that as well as I mentioned and I think it would be workable.

Making -Z host-config change the default does not make this easier to test, it makes it harder to test, because it mixes the stabilization process for these new options with the planned transition.

I think the main thing is just getting the config option target-applies-to-host available to early on as this seems to make everything else easier down the line. I'm trying to avoid testing the host table feature in combination with the old defaults as that's what appears to complicate testing to me. Whether the defaults are changed before or at the same time as the host table matters less I think.

I'd also like to express that it feels frustrating, in this PR, when you've repeatedly decided to implement something other than what was previously requested, and rather than asking if that would work, you just do something different and then try to convince us to accept that instead.

The problem is that I'm not familiar with cargo enough to anticipate ahead of time implementation roadblocks with a particular approach before I make an attempt at implementing it, so I'm basically trying an approach running into an issue then trying to rework it in a way that still handles all required use cases.

If you want to suggest a different approach, please talk about it first, and get agreement for that change, rather than just ignoring what we've asked for and doing something else. We're asking for those things for a reason, and it feels like you keep trying to push us to move faster and ignore our normal stabilization processes.

I'm just trying to iterate on the implementation until I get something that is workable for everyone, but I don't really know ahead of time all the issues I'm going to run into with a particular approach before I actually try and attempt it, I figure it's easier to put up the implementation for review than to try and explain it without also showing an implementation(since I'm likely to not explain everything completely). I'm leaning quite heavily on the automated tests to make sure I'm getting the implementation right as the current implementation is somewhat difficult to fully reason about in regards to implications of changes.

You do realize that you are now seriously arguing over what must amount to a single if statement, right? If you don't want it to work the way James wants it to work, it would take less time to just type the if statement than to type even the single sentence required to express your opinion over said if statement.

This existing implementation is just difficult to work with IMO, I had assumed the same originally but it seems to be a bit more complex than that due to unanticipated issues.

I've been in the situation where people decide to ship a fork with that fix anyway, "compatibility be damned", and if James were intending to do that (he clearly isn't) I'd understand being super super angry.

I ran into this issue because a different infrastructure project with cargo in its dependency tree decided to break compatibility with everything that doesn't have a usable rust toolchain...so I'm kind of coming from a point of view that things are already so severely broken that a fix should probably take higher priority over compatibility(minimizing disruption is fine but I think fixing such a severe bug should take precedence over uncommon edge cases in general).

You aren't better off, as this is a clear-cut case where it would take you less time to type the code you want than to even explain your interest... if this had been an issue on one of my projects, I'd have just fixed it myself and then rejected the pull request (potentially crediting James if I felt he had been useful).

Yeah, so normally when I report a toolchain bug like this I don't immediately try and fix it myself since it takes me a while to get up to speed on these sort of high complexity foreign codebases so I just let a maintainer fix it(since they will usually know the codebase so much better that any fix they come up with would be much better than anything I come up with), but since this bug was so severe(it effectively breaks rust cross compilation for every single cross compilation meta build system like buildroot, openembedded, openwrt, and others with no maintainable workarounds) that I decided I needed to write a fix to unblock things for them as all those previous bug reports never got any serious engagement from a maintainer.

I can't imagine James is better off: spending weeks arguing with him and making him feel bad for having had the unmitigated gall to try to fix this issue is absolutely worse than if you had just pocket rejected (aka, ignored) the pull request; and knowing that you seem to think he's somehow acting in bad faith here must suck.

It's not that big of an issue to go back and forth on this and I really want to get this right up front to avoid problems in the future so we don't have to deal with a complex migration like this again. In particular I'm trying to make sure the test suite has good enough coverage that we don't miss any important edge cases. Faster review cycles would be appreciated though.

But like, this is your project: commit what you want, or don't commit anything at all; decide that James is worth debating with (which isn't what this one-sided argument has been), or decide that you are better off just ignoring him; you have your beliefs about what stability guarantees mean to you, and you absolutely should follow them! ...but, as it stands, I beg you to ask yourself "what message am I sending with how I'm reacting to this pull request", as so far what I'm hearing--loud and clear--from this stand-off is "Cargo is a frozen project that doesn't fix bugs", "the developers behind Cargo prefer to argue than type code", and "non-Rust external project maintainers don't matter to the Cargo team, only Rust developers" :/.

I think this bug still existing should probably be looked at as a process failure that should be learned from and corrected, this is a bug that has been reported many many times over the years and should have been triaged/categorized early on as a P1/blocker type of issue due to the extremely high severity(fatal error), wide impact(it breaks essentially every cross compilation meta build system) and lack of viable/maintainable workarounds.

And look: there are a lot of reasons I am generally super sympathetic towards the plight of a project maintainer. I have been there, for a super popular project used by not only tens of thousands of developers but tens of millions of end users (who are often way way more demanding than developers ever could be, as they aren't used to working on software), and I appreciate that it really sucks sometimes, and I wouldn't blame anyone for burning out on it.

I've been part of an open source project where we(maintainers and those working with them closely) have coordinated(successfully) largely optional and backwards compatible features/changes that have not only taken years but have had enormous amounts of users that were straight up actively hostile(to the point where a number of misguided and relatively non-technical users literally spent millions of dollars funding a coordinated and wide-scale propaganda campaign to try and effectively destroy said open source project), it's not fun. I think you just have to have a thick skin here to some degree when it comes to successfully dealing with users like this.

I've been in the situation where I've had bugs that I simply don't know how to fix, despite a number of people insisting it was easy to do... and I'd have been willing to believe (not that it was true :/) this was one of them based on prior conversations with Alex.

Yeah, even if there wasn't an obvious fix right away this should have at least been properly root caused early on.

I've even had bugs where there simply isn't a fix, whether due to technological, legal, or moral reasons: that really sucks... you get to spend every single day for the rest of your life apologizing to people until, once in a while, you snap as your long explanations--which everyone calls "excuses"--are tiresome to write (I am so sick of people demanding that I solve some problem or another by making my federated system centralized... not going to happen, no matter if they are "correct" or not).

In this particular case I think it was quite clear from the start that this wasn't an unfixable bug, that it was simply a minor logic error seemed clear to me from the start, even though I had trouble finding where in the codebase it was actually located for a while.

Sadly, I think the thing that is probably the most disappointing is that, as someone who cares a lot about cross-compilation--and who has at times been the maintainer of a gcc+llvm/binutils fork for a popular platform, in addition to a rather complete distribution of Unix that I cobbled from scratch and cross-compiled with said toolchain--I think James has been correct to fight you, at least a bit, on many of these points: he's clearly an expert here on cross-compilation scenarios and I don't feel like that background is being in any way respected.

I've generally had quite a good experience with reporting bugs to toolchain maintainers, that this particular bug in a major language toolchain went unfixed without even being root caused for many years is very atypical IMO. I've reported obscure(unusual configuration) low impact(very few users impacted) low severity(has trivial and maintainable workarounds) gcc bugs that have been root caused almost immediately by the maintainers and triaged with a fix being made in a reasonable amount of time. If bugs that are clearly fixable, high impact and high severity go unaddressed for a long period of time like this people aren't going to consider rust a reliable infrastructure platform to build upon which is detrimental to the goals of rust and the ecosystem as a whole.

There are projects, today, broken and even blocked with no reasonable workaround, and those projects are being given very little weight versus unsubstantiated theories that either there might exist projects which will be broken so badly by this change that they won't know how to fix it (a scenario I find patently ridiculous, as I can't imagine that any project relying on the current behavior works reliably anyway)... literally the strongest argument here that has so far been made against James' original patch was that there might theoretically exist a project that wouldn't actually break but might simply build a bit slower until they adopt the host table.

Yeah, this bug has been clearly wasting an enormous amounts of time for many highly experienced developers(those that do cross compilation like this are not usually amateurs) without any serious engagement from the maintainers to try and fix it. These are the sort of experienced developers who proper engagement with should be prioritized as they tend to provide high quality feedback in general and are most helpful at ultimately advancing the goals of the project.
In any case please don't take this as me as trying to bash or be hostile to the maintainers, I'm trying to provide constructive criticism and recommendations so that the project can improve. As someone who deeply cares about security I think rust has really innovated a lot here, but appropriately triaging severe but less interesting issues like this bug which may not impact the maintainers own use cases is also important in order to get wider adoption of rust. With rust now on its way to being integrated into the Linux kernel this is only going to become more important going forwards.
IMO we have a shared social responsibility as infrastructure project contributors/maintainers in general to our users and those developers who depend on us, especially when it comes to use cases(such as cross compilation here) that we explicitly claim to fully support. We all make mistakes, the important part is that we learn from them so that they don't keep happening.

@saurik @jameshilliard
I understand, genuinely, that this issue is causing problems for people. It has caused problems for me in the past. I'm deeply glad to see someone working on it.
Yes, in theory, we could work on this ourselves, as we could with any issue. We tend to have much less bandwidth than we do issues to work on, and we spend quite a lot of that bandwidth reviewing and fixing issues. And it's not always obvious which issues are going to take one line to fix and which issues will turn out to be surprisingly complex; Cargo's internals do sometimes tend to mirror the problem space in their complexity.
As with many Rust projects, we tend to place a higher priority on mentoring others rather than on changing things ourselves, precisely because we want to foster more people with the knowledge to work on things. This is especially true for Cargo, because we have a serious dearth of people with internal knowledge of Cargo, and doubly so people with that knowledge and spare bandwidth. As a result, we tend to bend over backwards trying to provide guidance for others working on changes, in preference to working on those changes ourselves.
(This is in addition to the reason above, that attempting to dive into a problem like this can lead to surprising amounts of complexity, and we only have so much bandwidth for those experiments and adventures.)
Backwards compatibility is, in particular, one of the areas we often find ourselves having to explain in detail, because it isn't always obvious. And we could absolutely use some better contributor documentation in this area. The procedures for starting a new feature in nightly, testing it, and later stabilizing it are not self-evident, and require some knowledge.
We do tend to strongly prioritize the possibility of regressions over the speed of fixes. This is partly because of Rust's very strong compatibility guarantees that we strive to uphold, and partly because when regressions do happen we find ourselves in "emergency drop everything fix it now" mode.
And for the record, we don't mind getting pushback on our proposed approaches to doing things. We do tend to be unusually sensitive to the case where those alternatives come in the form of code being written; we've had the same experience you have, @saurik, of people showing up with large PRs with zero discussion, and not everyone is willing to accept feedback or understand the constraints Cargo needs to meet.
I'm sorry for expressing those frustrations here. @jameshilliard, in particular, I appreciate you working on this, and for going through revisions.  As you said, these kinds of issues can take a disproportionate amount of time. In case it isn't abundantly clear, we recognize and appreciate not just the final lines of code, but all the experimentation that goes into that work. Thank you, @jameshilliard, for doing those experiments and taking the time to make this work. And thanks as well for the detailed testsuite coverage; that gives us much more confidence in a change like this.

@jameshilliard

I'm trying to avoid testing the host table feature in combination with the old defaults as that's what appears to complicate testing to me.

If it helps, any time any host table is present, the presence or absence or value of target-applies-to-host should not matter; a host table always takes precedence for the host if it's present, because it indicates a configuration that's been updated. A target table can only apply to the host if there are no host tables and target-applies-to-host is true. You'd need one test case to confirm that, but otherwise, you don't need to worry about the case where the host table is present but the default value of target-applies-to-host hasn't changed yet. Does that make your testing and test-case writing easier?
@saurik This kind of test that @jameshilliard is taking the time to write is an excellent example of why this is substantially more than a one-line change.

As with many Rust projects, we tend to place a higher priority on mentoring others rather than on changing things ourselves, precisely because we want to foster more people with the knowledge to work on things. This is especially true for Cargo, because we have a serious dearth of people with internal knowledge of Cargo, and doubly so people with that knowledge and spare bandwidth. As a result, we tend to bend over backwards trying to provide guidance for others working on changes, in preference to working on those changes ourselves.

I think the biggest problem is that this never even really got root caused before I isolated the bug, and because of that a number of developers had to implement some really insane workarounds that were far more complex than the actual fix, someone probably would have been able to fix this sooner if just the root cause/bug isolation was done as there were clearly many highly experienced developers hitting this bug.

Backwards compatibility is, in particular, one of the areas we often find ourselves having to explain in detail, because it isn't always obvious. And we could absolutely use some better contributor documentation in this area. The procedures for starting a new feature in nightly, testing it, and later stabilizing it are not self-evident, and require some knowledge.

Yeah, I generally try really hard to make stuff backwards compatible as much as feasible, this bug is just particularly thorny since it's a logic flaw that results in highly inconsistent behavior. My typical approach to dealing with inconsistent behavior is to try and come up with a solution that fixes things for the most amount of use cases while breaking things for the least amount of use cases.

We do tend to strongly prioritize the possibility of regressions over the speed of fixes. This is partly because of Rust's very strong compatibility guarantees that we strive to uphold, and partly because when regressions do happen we find ourselves in "emergency drop everything fix it now" mode.

IMO this is a case where that should have happened as well, in terms of severity this is arguably far worse than a typical regression as it's almost completely fatal with no workarounds. I agree this is a case where one needs to take extra care to get things right.

And for the record, we don't mind getting pushback on our proposed approaches to doing things. We do tend to be unusually sensitive to the case where those alternatives come in the form of code being written; we've had the same experience you have, @saurik, of people showing up with large PRs with zero discussion, and not everyone is willing to accept feedback or understand the constraints Cargo needs to meet.

Yeah, this stuff often takes a number of review cycles, doing my best to come up with something that meets Cargo's constraints, but I'm frequently hitting unexpected issues when I actually try and implement things. Effectively the only rust code I wrote before this PR is my PR fixing what is effectively this exact same logic bug in pyo3 so I'm not able to anticipate a number of issues before I actually hit them while working on the implementation.

I'm sorry for expressing those frustrations here. @jameshilliard, in particular, I appreciate you working on this, and for going through revisions. As you said, these kinds of issues can take a disproportionate amount of time. In case it isn't abundantly clear, we recognize and appreciate not just the final lines of code, but all the experimentation that goes into that work. Thank you, @jameshilliard, for doing those experiments and taking the time to make this work. And thanks as well for the detailed testsuite coverage; that gives us much more confidence in a change like this.

Yeah, I'm used to these sort of extended review cycles, just please try not to assume that me pushing back on something is a refusal to address the issue so much as me hitting a roadblock and needing additional feedback to work the problem. I really don't want to implement something that I don't feel is logically clear, consistent and properly tested if possible, especially since I'm having to lean heavily on the test suite to validate the implementation behaves as expected.

If it helps, any time any host table is present, the presence or absence or value of target-applies-to-host should not matter; a host table always takes precedence for the host if it's present, because it indicates a configuration that's been updated.

I'd really prefer to avoid this if possible, which is one reason why I didn't want to have the host table feature ever made available without the default change. This is rather tricky I to implement cleanly from what I can tell and can lead to potentially confusing behavior.
The main issue is that we have both generic and target specific host tables, so it's more complex than looking for just one table, we have to check target specific and non-target host tables specific. How would we handle a case where only an arch specific host table is defined(without a generic table) but where that host config which doesn't match the running host arch? Do we default to using the target table(old defaults) or do we default to no table(new defaults)? I think we really should avoid this by not introducing the host table feature before the default is changed.

A target table can only apply to the host if there are no host tables and target-applies-to-host is true. You'd need one test case to confirm that, but otherwise, you don't need to worry about the case where the host table is present but the default value of target-applies-to-host hasn't changed yet. Does that make your testing and test-case writing easier?

Not really, I think this is difficult to implement and to properly test for, I don't see an obvious way to check if there are any non-arch-matching host tables defined. I also am not seeing any reason to make the host table feature available without the default change as the host table is effectively non-functional without the ability to set target-applies-to-host behavior/defaults to false anyways, especially since the default change being done before or at the same time as the host table feature should fully sidestep this particular issue. Any implementation of this seems to me like it would be a significant complexity increase for something that would just be removed after the migration, and that complexity increase is likely to result in bugs IMO.

@saurik This kind of test that @jameshilliard is taking the time to write is an excellent example of why this is substantially more than a one-line change.

Yeah, there's a bunch of non-trivial edge cases we need to deal with and I really want to get these right up front.

As a simple possibility, could you detect host tables at parse time, and flag the presence of any host table, rather than checking for one later?
If that isn't feasible, then I'd suggest just splitting the feature-gate as you suggested, and we'll aim to stabilize target-applies-to-host before host-config.

As a simple possibility, could you detect host tables at parse time, and flag the presence of any host table, rather than checking for one later?

I'm not seeing an obvious way to do that, or to test all combinations properly.

If that isn't feasible, then I'd suggest just splitting the feature-gate as you suggested, and we'll aim to stabilize target-applies-to-host before host-config.

Yeah, this seems to be the best option that I'm seeing, what should I name the feature gate flag for target-applies-to-host?

I'd suggest -Z target-applies-to-host, to keep it simple.
Note that -Z target-applies-to-host should not change the value, just gate access to that configuration option.

:umbrella: The latest upstream changes (presumably #9030) made this pull request unmergeable. Please resolve the merge conflicts.

I'd suggest -Z target-applies-to-host, to keep it simple.

Added, so this gives us the following migration path:

add -Z target-applies-to-host and -Z host-config options with the new defaults being enabled for testing by -Z host-config
stabilize the config option target-applies-to-host
stabilize -Z host-config with new defaults

Note that -Z target-applies-to-host should not change the value, just gate access to that configuration option.

Implemented, only -Z host-config changes the defaults(we don't want the host table feature enabled with the old defaults as that would make testing difficult/complicated), in order to help ensure testing accuracy the -Z host-config option can not be enabled unless -Z target-applies-to-host is also enabled. This is to ensure that the tests appropriately replicate the stabilization path ordering.

Alright, this design seems reasonable to me, thank you!
I saw a few code cleanups that I'd like to get in before applying this. Rather than force a feedback cycle for those, I fixed them locally and pushed to the branch.
I think this is ready for @rust-lang/cargo review now.
@rfcbot merge

I've lost steam discussing this. It's unstable for now so more discussion can happen later during stabilization. I continue to think there should only be one feature gate but I no longer have the energy to pursue this.

It's unstable for now so more discussion can happen later during stabilization.

Does that mean it's in an unstable branch now somewhere? Not sure where to find that.

I continue to think there should only be one feature gate but I no longer have the energy to pursue this.

I mean, there's ways to do that and some of my earlier approaches only had one flag...but it ended up like this to workaround other issues/requirements.

Based on the above FCP and the discussion in the @rust-lang/cargo meeting today:
@bors r+
We may still revise this further in the future before it's stabilized, but we can go ahead and merge this as-is.

:pushpin: Commit 5338bb2af1365a8f50f67768754034fecfbbfc4f has been approved by joshtriplett

:hourglass: Testing commit 5338bb2af1365a8f50f67768754034fecfbbfc4f with merge 0cecbd67323ca14a7eb6505900d0d7307b00355b...

:sunny: Test successful - checks-actions
Approved by: joshtriplett
Pushing 0cecbd67323ca14a7eb6505900d0d7307b00355b to master...

Hello, I am currently facing this issue with target==host==x86_64.  Cargo version is 1.75.  Is this version has the fix?  Does it require any config settings?  Appreciate the help.

@vithalsm, it is behind some Z flags. See https://doc.rust-lang.org/cargo/reference/unstable.html#host-config

Cargo version is 1.75. Is this version has the fix?

Yes but it's a bit annoying to enable since for unclear reasons #9753 has yet to be merged, for buildroot we use these env variables.

Thanks for the pointers.  However -Z expects cargo from 'nightly' channel or use the env variables as being used by buildroot.  Using nightly is not feasible for us, so will try with env variables.
But, would it be possible to merge and make the options available regular stable release please?  That will be very helpful for many