New software configuration and provisioning mechanism

Introduction
This issue deals with a topic that has been the center of multiple previous discussions and existing issues and PRs. This is an attempt at briefly summarizing the problem in order to solve it in a way that is agreeable to all potential users and developers alike.
IMPORTANT: Please keep the discussion around this particular feature in this GitHub issue, so that we do not have to collect feedback from multiple sources.
Problem description
Zephyr currently lacks a mechanism to configure multi-instance software components at compile-time well above the hardware. Instead, as described multiple times, we have:

Devicetree: Multi-instance, but currently restricted to represent (mostly) actual hardware
Kconfig: Single-instance, mostly restricted to software enablement and configuration

A third mechanism is therefore required. This issue describes the use cases and requirements in order to end up concluding with a final proposal.
Zephyr also lacks a provisioning mechanism, which would be able to generate a pre-compiled "image" of the data that is required in order to provide the required data for certain subsystems to execute.
Use cases
The following use cases have so far been identified:

Network stack interface description: https://github.com/zephyrproject-rtos/zephyr/pull/68127
Configuration of init priorities: https://github.com/zephyrproject-rtos/zephyr/pull/73836
Multi-instance software (non-driver) configuration (e.g. buffer sizes in CHAT/CMUX)

We foresee additional users of this mechanism in the future, but the two use cases above exist today and require a solution.
Requirements (WIP)
Scope

Single serialization format?
Multiple outputs?
Single file for all use cases?
Generation of C structures directly instead?

An additional use case is buffer sizes for instances of CHAT (at command handling), CMUX (uart multiplexing), UARTs etc. A CMUX instance can have multiple channels, where only one needs a huge buffer (for network traffic for example) and others require small buffers for typically AT commands.

Sources:

Provisioning a PSA secure storage target (https://github.com/zephyrproject-rtos/zephyr/issues/75275). This needs to be sourced from a secure container at prod time. If we don't want to complicate things at dev and test time then we should integrate this with a simple dev-time config approach for testing.
Several people have asked for support of alternative source formats (not now but via a migration path) as input which they generate from internal systems. The requirement here is not that we need multi source at once but it should be clear how we want to migrate there.
UX for first-time users, this ideally includes a single source for single- and multi-instance configuration.
A way to cross-reference DT based on a common data model as the entities are mostly the same (required by both, init and net conf)
A well-defined distinction of Kconfig/DT/sw config that can be reviewed and enforced based on easy-to-observe criteria.
We need a method to map properties in Kconfig, DT and the new config approach for backwards compatibility as many properties will have to be migrated from both DT and Kconfig over time (as in net config).
Overlays (required by init right away and probably net conf, too, clearly required in the future)

Targets:

type bindings (obvious - just mentioning it for completeness), ideally kept close to usage sites and (as many have suggested) split up based on frequency of updates (maintainability)
Several people have asked for integration with the settings subsystem (and other NVS). Not required right away, but for net config this is on the migration path.
Specific configuration properties should be updatable at runtime (net config is an example).
zero footprint by default (core Zephyr standard for all kinds of config systems)

Documentation:

Several people have asked for self-documenting bindings, ideally integrated with the doc build system.

I also suggest looking at the use cases I collected from community contributions. Ignoring those would mean ignoring those contributions IMO (none of those comes from me, they've all been collected from previous discussions or asking stakeholders on Discord). Few need to be supported initially but any proposal should be aware of them and include a credible migration path.

my comment copy pasted from the Arch WG chat as requested:

"can we be clear that provisioning requirement is obvious when talking about networking IP address but most multi instance configuration needed to develop software is probably not about provisioning, ie buffer sizes and stack sizes of drivers or properties of how parts of the system will act that doesnt change between serial numbers"

and

"i agree with brix about separating the provisioning problem space from the static multi instance configuration problem space"

For network configuration, without having run-time configuration, the whole thing can be done in Kconfig, e.g.
# subsys/logging/Kconfig.network_interfaces file:
# total_interfaces would need to come from some sort of DTS processing or macro for counting number of "okay" devices"
if "$(interface)" <= "$(total_interfaces)"

config NETWORK_$(interface)_IP_ADDRESS
    string "IP address of network interface $(interface)"

endif

# Kconfig file in test:
interface = 0
source "subsys/logging/Kconfig.network_interfaces"
interface = 1
source "subsys/logging/Kconfig.network_interfaces"
interface = 2
source "subsys/logging/Kconfig.network_interfaces"
interface = 3
source "subsys/logging/Kconfig.network_interfaces"
interface = 4
source "subsys/logging/Kconfig.network_interfaces"

Where total_interfaces would need to be set in dts directly or using macros to count the number of devices of a certain type

Arch WG:

Devicetree vs this new mechanism:

@henrikbrixandersen mentions that if there's a tie to a struct device when the property is in Devicetree
@fgrandel explains how Devicetree is currently being used in many instances to configure "software"
@nashif states that we should be able to serialize this in a format that can be used at runtime, independent of the image (i.e. re-provisioning without recompiling)
@nordicjm states that multi-instance configuration could be done with Kconfig, no need for an additional mechanism to configure this
@jfischer-no suggests using CBOR as a serialization language
@henrikbrixandersen states that we may be mixing up different concepts with this: provisioning (serialized data that can be used across multiple instances of a single image) and build-time configuration that is different in every build. @henrikbrixandersen uses the setting subsystem downstream in order to handle this
@fgrandel reminds us of the provisioning use case for Bluetooth Mesh, which is a very common one. Differentiating between provisioning and multi-instance configuration in terms of language and infrastructure may complicate things, so we may want to have the ability to generate either serialized data or just macros that can be used by the code directly.

Note on images build for MCUboot:
when planning provisioning mechanism, note that it may be hard to generate some hex and add something directly to image (as part of binary), as outcome of building image for MCUboot may have it already signed and encrypted. So in case when some blob is inserted as binary section, where you could directly address constants, that blob would require signing with the image.
Note on using external, out of image, blob:
such blob will require careful check, the same level as any user input as it may be used as attack vector on firmware. And what to do when these are tampered with? What to revert to? Note that something that may look like tampering may be just a problem that should not brick device, but rather, with annoying effect for user, default to factory setting - the factory settings that provisioning provides.
I have also a question: how to avoid using RAM for things that always be constant for a device? For example you may not change embedded MAC or serial number of device, yet when accessing these, from what has been written during provisioning, the device will have to load them to RAM, at least temporarily. This also brings cost of enabling Flash API and other subsystems required, for device to get the factory provisioned info, even if every unique instance of a device only reads the info and never attempts to store updates, or chooses other subsystem of storage (like FS).

Thanks @carlescufi for subsuming our discussion. A few corrections to avoid misunderstandings:

@fgrandel explains how Devicetree is currently being used in many instances to configure "software"

Not exactly. I don't use the software/hardware distinction myself. I was rather saying that almost all properties that we introduce change at the same frequency as driver software and not at the same frequency as vendor hardware. From a well-established encapsulation perspective this indicates a requirement to keep driver-specific properties apart from vendor-specific properties and de-facto standardized stable Linux properties (as different files, not as different formats).
I was citing others who had laid out alternative heuristics to define the sw/hw line:

"Everything that ties to a struct device is 'hardware'" as mentioned by @henrikbrixandersen and others. This heuristic breaks down in both directions: 1. By normalization rules, this definition would include almost all of the network interface configuration which is 1-to-1 to drivers. 2. We have a considerable number of DT properties that do not tie to struct device instances.
"DT should contain everything that changes at the frequency of hardware" as @decsny seems to have been informed by several community members (but does not find himself being practicable): This heuristic also breaks down in practice. As pointed out above by that definition almost all Zephyr-specific properties in DT would be "software".

I pointed out that as long as we do not have an intersubjectively observable heuristic that defines the line between sw and hw, each maintainer will define their own personal criteria as we observe in practice. I disagreed with @nashif that we have only few properties that are contended. I'd rather say that almost all newly introduced properties are contended as @decsny has confirmed based on practical experience over the last few months.

@jfischer-no suggests using CBOR as a serialization language

I mentioned that this is part of the solution space. Others have voted for protobuf or Thrift.

@henrikbrixandersen states that we may be mixing up different concepts with this: provisioning (serialized data that can be used across multiple instances of a single image) and build-time configuration that is different in every build.

I responded, that we cannot cleanly define per property whether they are "provisioning only" or "build only". On a per-app basis this line will change considerably in practice.
Maybe I misunderstood what was proposed, but if it means that we'd introduce another "a priory" (ontological) distinction then I'd strongly oppose such a strategy. Here a few examples others mentioned in earlier discussions:

build-time "hw" property (e.g. gain of ADC channels)
provisioning-time "hw" property (e.g. CDC ECM MAC address)
build-time "sw" property (e.g. buffer sizes)
provisioning-time "sw" property (e.g. IPv6 address)

I could immediately think of products (apps) or use cases (tests, debugging) where those properties would not fit those categories.
I therefore argued that any attempt to bind properties to a specific configuration source or target is conceptually flawed and will not work in practice. The fix is: Define a common abstract data model for everything closely tied by encapsulation rules to software programming models (which includes drivers) based on well-established normalization rules (i.e. shared entities and properties) independently from any specific representation or usage context. This can be done w/o any ontological assumptions with mathematical precision and has therefore been well-established best practice in application configuration for decades. This model then needs to be serializable from/to multiple files on a per use-case and encapsulation basis and/or overlayed  at any level.
The settings subsystem is part of the solution space but was introduced to the discussion. That's why I voted against having to depend on the settings subsystem for all configuration as it severly limits extensibility of the config subsystem and introduces unnecessary complexity and resource usage to certain use cases like tests or simple custom applications that otherwise won't need it. I vote for technology that allows us to flexibly merge/split serialization from/to any sources/targets depending on specific application requirements. Several technologies with excellent UX exist for all proposed serialization formats to implement such a merge/split from/to a common intermediate representation.
I mentioned that as a bonus, this will make it easier to switch between alternative source and target representations at dev-time (e.g. for easy reproduction of bugs and debugging with minimal dependencies).

@fgrandel reminds us of the provisioning use case for Bluetooth Mesh

Not only B. Mesh, my comment was including BLE proper as both seem to support the settings subsystem as config target according to @jhedberg. This becomes even more interesting when adding secure storage or large-scale provisioning systems as possible config sources. I also mentioned several other immedately relevant use cases introduced to the discussion by others and that are not yet part of the use case list.

@nashif talks about the developer experience: language, syntax, tooling can all be shared.

I added to this that IMO tooling will be 90% of the maintenance effort and will be mostly shared between all use cases discussed so far (including init and net config). Assuming a common intermediate representation, sourcing from different formats and serializing into different targets can be added flexibly at very low cost. A common intermediate representation will help to debug the configuration space (similarly to .config and merged DTS) no matter whether done in YAML, JSON or DT.
I therefore vote for extracting and re-using existing DT tooling (not format) and re-use it for whatever configuration system and hopefully common intermediate format we decide upon. Any addition to the DT and/or config world will then equally profit both sides, ie. we can expect that DT tooling will benefit from improvements in the config area, too (e.g. UX improvements, validations, bindings, documentation, etc.). This also has obvious advantages from the user perspective as a lot of target-side and tooling knowledge could be re-used, independently of context.
As long as those use cases and basic requirements (including those mentioned in my first comment) are not explicitly made part of the RFC I'll find it very hard to agree to any configuration system. They are all immediately relevant to the initial use cases mentioned by the OP and are therefore part of a "minimal set" as required by this issue. And please note that none of those requirements favors DT over YAML (or whatever other solution space technology) in any way.

I disagreed with @nashif that we have only few properties that are contended. I'd rather say that almost all newly introduced properties are contended as @decsny has confirmed based on practical experience over the last few months.

you need to provide some context and details here, I have no idea what @decsny said and where he said it and in what context.

I therefore vote for extracting and re-using existing DT tooling (not format) and re-use it for whatever configuration system and hopefully common intermediate format we decide upon.

why would you use a tooling for a format if you are not using the format itself? Also, we established that runtime is important, most of the DT tooling is build time and based on macros.

@nordicjm @de-nordic I think your comments are really relevant but they both rather belong to rather specific solution spaces unless we extract abstract requirements from those examples. I find it hard myself to cleanly distinguish between problem and solution w/o loosing sight of feasibility. But it would be nice if we tried to keep very specific implementation problems always linked to more abstract requirements as I understood @carlescufi 's approach.
@nordicjm In more abstract terms you're making the case - if I understand correctly - that almost any source format will be able to represent an abstract data model (including Kconfig) and that we should not introduce additional formats unless required for a good reason (i.e. simplicity and UX as a requirement).
@de-nordic If I understand you correctly you're requiring at a more abstract level that configuration needs not only to integrate with the settings subsystem alone but also with other subsystems (e.g. MCUBoot signing in your case). From a requirements perspective this confirms that we cannot tie the configuration subsystem target to a single subsystem alone. It must be sufficiently open to integrate with whatever target and process we already have or invent in the future.

@nashif

you need to provide some context and details here, I have no idea what @decsny said and where he said it and in what context.

Sure: https://github.com/zephyrproject-rtos/zephyr/issues/76902#issuecomment-2307710931

@nordicjm In more abstract terms you're making the case - if I understand correctly - that almost any source format will be able to represent an abstract data model (including Kconfig) and that we should not introduce additional formats unless required for a good reason (i.e. simplicity and UX as a requirement).

No, I'm stating that without runtime configurability (i.e. provisioning and able to change it) the entire network configuration PR is moot because no tooling is required, static configuration can be wholly done in Kconfig, thus the prime feature that this must support is provisioning

@jfischer-no suggests using CBOR as a serialization language

I mentioned that this is part of the solution space. Others have voted for protobuf or Thrift.

Not introducing new dependencies or minimizing dependencies should be a requirement. IIRC it was mentioned somewhere, but I cannot find it here. (Side note, we already use CBOR implementation and IMO it fits better to use it as a DTB alternative).

@nashif

why would you use a tooling for a format if you are not using the format itself? Also, we established that runtime is important, most of the DT tooling is build time and based on macros.

Considerable parts of DT tooling are not specific to DT as a source format (which as such I'm not questioning). Re-using those parts not only means less maintenance cost but also considerable re-use of existing community knowledge.
Just a few examples off of the top of my head w/o trying to be complete:

Bindings are in YAML, not in DT. Binding format. All validation and generator infrastructure around that is re-usable. This tooling could benefit from a common approach as the existing binding approach has several modeling and usability flaws that could at the same time be fixed for everyone. I even suspect that the fact that many seem to find DT "awkward" is at least partly due to bindings rather than DT itself. This alone is a very important investment if we have to re-invent the wheel and maintain it on both sides.
Many low-level generic DT macros are not tied to a specific source format. They only tie to an abstract hierarchical node (aka entity) vs property (aka attribute) distinction that we'll most probably need in the config area, too.
In the config area we'll need a zero-footprint (probably macro) default target, too. Code that generates macros from typed entities and nodes (bindings plus intermediate representation) can be re-used among DT and config.
Nordic and others have built visualization and maintenance tooling around DT that from a structural viewpoint could be (partly) re-used for for any entity-relationship graph including config which would make it easier to learn and maintain tooling.
We need to represent and validate links between DT and config. This is much easier in a common intermediate infrastructure.

I'm sure I could find more examples if looking at the code. But I think this is enough to make the point.

@nashif

you need to provide some context and details here, I have no idea what @decsny said and where he said it and in what context.

Sure: #76902 (comment)

@decsny : "From being on the (recently removed, by me) DT binding maintainer area, I saw many things being added to DT that aroused this type of argument over HW/SW distinction."

This is very vague and a general obvervation which might be valid and to the point but this needs to be supported with examples that show whether this is going from bad to worse or something manageable that can be tolerated depending on the use case and context. If someone thinks this is a serious problem that needs attendion, someone needs to open an issue with examples and bring this to the attention of everyone.

I disagreed with @nashif that we have only few properties that are contended. I'd rather say that almost all newly introduced properties are contended as @decsny has confirmed based on practical experience over the last few months.

you need to provide some context and details here, I have no idea what @decsny said and where he said it and in what context.

I think florian is referring to my engagement of him on #76902, he mentioned that I was taking a definition for a software and hardware distinction as for granted, I just responded to him that since I have been on the DT reviewer list for a while, I actually did see that it was quite a gray area and everybody seems to have different opinions.
As far as the point that florian is trying to make, I have not created a compiled list of detailed examples to support his argument, which IMO sounds a bit zealous but I think his spirit is going in the right place regardless, it's an important issue (as everybody interested in this thread would agree I think). And to be clear, I would not say that almost all newly introduced properties are contended, that is a bit of an exaggeration, but there are multiple cases:

First of all, yes, I don't think it's uncommon to see disagreements and change requests on PRs from somebody who thinks something being added to devicetree doesn't belong there or vice versa. I think everybody who has been involved in the zephyr upstream has observed this plenty of times, and it's even discussed almost regularly in the working groups and discord forums as well as github. I do not think this claim needs to be proved to anybody who is a regular here.
On the other hand, I couldn't tell you an exact proportion, but I think a good amount of DT changes and additions are not controversial and rather routine.

As far as how much we are adhering to Linux I'm not sure and don't know a good way to measure it, but I think a lot of MCU stuff being added to Zephyr is not found in Linux and a lot of bindings being added for platforms added to zephyr are new territory as far as relevant inter-project DT most of the time.
As far as how much we are respecting the various flavors of hardware vs software distinction, like I said above there can be disagreements, but for the most part, I think most changes and additions to DT are due to drivers being expanded or added to account for more features or aspects of the hardware and enable awareness of them through a more thorough DT description. I think there is a high frequency of changes, as florian pointed out, but this is not necessarily because DT is being mapped 1:1 to software constructs and is being forced to change with the same frequency, but I think most of the cases are just because there is a rapid amount of new hardware enablement development work going on in Zephyr due to it being a growing project with a lot of interest from silicon vendors who are shifting resources to support it. There might be some cases in DT where there is a problem with strong software mapping, and we should avoid it, but I don't think it's this dire as I am being quoted saying.

Finally, there is the case where there is something being changed in DT that is in my opinion questionable, but nobody questions it or contends it. And to be clear, this case is not about my personal interpretation of the hardware config vs software config debate, but I think it's a different issue, where a lot of people (including myself) just don't want to put in the energy to fight against the tidal wave of interest from whoever is adding the thing when there is no alternative to suggest to them anyways since we don't have a good multi instance static software configuration scheme (which is the point of this issue, I suppose).

just a nother note here, can we please STOP talking about implementation and formats and DTS please? thought we had a good discussion yesterday while remaining at a high level and now we are going down the path again talking tooling and formats, this is very disruptive.

just a nother note here, can we please STOP talking about implementation and formats and DTS please? thought we had a good discussion yesterday while remaining at a high level and now we are going down the path again talking tooling and formats, this is very disruptive.

you should quote reply whoever you're talking to with what specifically you think is not relevant.

you should quote reply whoever you're talking to with what specifically you think is not relevant.

too much to quote reply. hint: I am not referring to you, just seen your message :)

you should quote reply whoever you're talking to with what specifically you think is not relevant.

too much to quote reply. hint: I am not referring to you, just seen your message :)

I do think it's not productive to make vague statements about people's discussion being irrelevant, it doesn't help not to be specific, you want to avoid discouraging the participation you actually do want by being too broad with your meta-criticism of the discussion

I do think it's not productive to make vague statements about people's discussion being irrelevant, it doesn't help not to be specific, you want to avoid discouraging the participation you actually do want by being too broad with your meta-criticism of the discussion

where am I being vague?

can we please STOP talking about implementation and formats and DTS please?

I am saying we need to first collect requirements before we go into implementation details and the discussion is shifting toward what was already discussed in the issues that were abrubtly closed. Why at this stage people are "voting" for implementations and formats and tooling without first agreeing on the requirements? Why is this discussion shifting toward how DTS is being used or mis-used?

I'm not disagreeing with you that we need to talk about the requirements before proposing myriads of solutions, but a lot of these requirements are going to come from observations of weaknesses about the current implementation we have, so saying not to talk about it at all is too vague. Clearly there is some direction of the discussion above that you didn't like and I think it would be more useful to be more specific about what direction it went that you didn't like and by who. On the other hand, if by your quoted request you really flatly think that any mention of DT or implementation (especially the current) is irrelevant, then I tend to disagree for the reason I just gave.

I'm not disagreeing with you that we need to talk about the requirements before proposing myriads of solutions, but a lot of these requirements are going to come from observations of weaknesses about the current implementation we have, so saying not to talk about it at all is too vague.

and how does a "vote"" for an implementation in a discussion about requirements and as you say, weaknesses in such implementations help here?  We will get there, sooner or later, but when this forum goes into the same rabbit hole about the same topic and the same arguments, most of us will  just give up and lose interest.
Some DTS topics being touched on here deservee their own issues and tracks.

I would not say that almost all newly introduced properties are contended, that is a bit of an exaggeration, but there are multiple cases.

Yes, I admit this was an exaggeration. Sorry for that. :-( It's enough that there are many instances to make my argument.

but I think most of the cases are just because there is a rapid amount of new hardware enablement development work going on in Zephyr due to it being a growing project

From my observations I don't 100% agree. IMO a sufficiently large part of changes are made due to driver-specific features for existing hardware to support the argument that those properties should be packaged with drivers rather than hardware. This is very well established encapsulation practice, why deviate? That does not invalidate your argument, though.
I acknowlege that we try heroically hard to make our configuration models independent of specific vendors and Zephyr driver programming models. But I've seen too many times that for custom out-of-tree drivers or custom hardware this model won't sufficiently match project requirements. We just don't hear much of this upstream as people will usually not comment their custom drivers/hardware in public. Most of the time one can re-use existing drivers, though, and then this is not an issue.

just another note here, can we please STOP talking about implementation and formats and DTS please?

I said the same thing above - so why re-iterate? There was nothing deviating being brought to the discussion in the meantime except for your own questions re DT and tooling, right?
If you go back you will observe that you were the very first who asked back about DT and tooling specifically leaving the fundamental aspects of my contribution unmentioned and picking on those two side aspects out of context just because you seem to have found those being worth commenting? No problem from my side, that's why I answered your questions.
I therefore have to say that it's at least partly your fault if the discussion has been digressing again from the problem space. You just got what you asked for, IMO.
My contribution, for example, was exclusively arguing about the software/hardware distinction - not any specific format at all as I explicitly made clear in my comment. I had to correct the same false assumption in yesterday's arch WG meeting, remember?
Focussing on conceptual distinctions is in the problem space, focussing on DT as a format is not. Also feasibility cannot be completely excluded from our discussion either, as has been remarked by several yesterday, too. I think it's totally fine to make a proposal in the problem space more credible by showing that at least one realistic implementation option exists as long as one doesn't immediately jump to that option as the only "good" one or arguing in favor of it at the expense of other options.

thought we had a good discussion yesterday

I don't agree. Yesterday's discussion was almost entirely redundant to what had been discussed online before (except for the truly original proposal to forget about config altogether and do everything in Kconfig). This was only necessary because a large part of the audience was not aware of arguments made offline (as we can again see in this discussion). None of what was said here had not been said previously elsewhere. It's a pity that so many comments are ignored. Plus we had to intervene twice to re-focus yesterday's discussion on the problem space.
I also have to say: This whole discussion thread would be almost 100% unnecessary if requirements mentioned in other contexts would have been collected in the RFC in the first place before opening it for discussion. I'm forced to repeating everything here that I had already stated elsewhere w/o exception as others, too.
I'm still waiting for the substantial contributions of those who've not commented on any of those prior issues before.

too much to quote reply

IMO this amounts to weaseling. Everybody might implicitly feel indirectly accused unless you make this specific (as I did) so that people can defend themselves (as they rightfully did in my case because I made wrong assumptions).

This is very vague and a general obvervation which might be valid and to the point but this needs to be supported with examples that show whether this is going from bad to worse

I personally find this comment a bit unfair - at least targeting @decsny specifically. Many others (including myself) have made this point over and over including specific examples. @decsny's comment was embedded in an RFC that pointed to specific examples and only confirmed the observations made there. And his new comment is equally balanced and well argued.
I believe that someone really acquainted with the area making such an observation should be enough to cast doubt on any prior statement that there is no issue here. @nashif  You haven't given any specific examples either to support your own argument, right? I think there have been plenty claims made in yesterday's WG call, too, that have been accepted at face value which I personally found 100% acceptable because I fully trust that those making them knew what they were talking about in their area, too.
But here some specific examples then as you asked for those:

I pointed to the USB maintenance area yesterday to make the point.
I can also point to my own maintenance area. IEEE 802.15.4 configuration as I found it established when I arrived has always made a very liberal interpretation of the hw/sw split, too. We even got a distinction between ieee802154 and ieee802154g accepted upstream some years ago without anyone mentioning its ambiguity.
You'll hopefully not deny that in the past there have been many PRs where @gmarull repeatedly mentioned questionable use of the hw/sw distinction - and I totally agree with him on all accounts if one takes DTSpec's OS independency requirements seriously. I'm sure @gmarull will easily be able to name a few of those examples right away if challenged.

We also should not ignore that it is documented practice in Zephyr to mix multi instance software configuration into the hardware container:

Because Kconfig is unable to flexibly control some instance-specific driver configuration parameters, such as the size of an internal buffer, these options may be defined in devicetree. However, to make clear that they are specific to Zephyr drivers and not hardware description or configuration these properties should be prefixed with zephyr,, e.g. zephyr,random-mac-address in the common Ethernet devicetree properties.

If this doesn't explicitly blur the hw/sw line then what else do we need to exemplify??

If someone thinks this is a serious problem that needs attendion, someone needs to open an issue with examples and bring this to the attention of everyone.

There are at least two PRs and one RFC where this argument has been brought forward in the context of this topic alone not counting all the unrelated PRs where the hardware/software line has been questioned by commenters in other contexts.
What more should be expected in terms of specific issues, problems and suggestions being brought forward in detailed form?
@nashif It would also be nice to get your personal requirements list or at least some confirmation of existing requirements that others have mentioned. This would add tangible content to the discussions rather than just focussing on the meta-discussion alone (which I otherwise find worthwile, too btw, but not exclusively so).

I'm not disagreeing with you that we need to talk about the requirements before proposing myriads of solutions, but a lot of these requirements are going to come from observations of weaknesses about the current implementation we have, so saying not to talk about it at all is too vague.

and how does a "vote"" for an implementation in a discussion about requirements and as you say, weaknesses in such implementations help here? We will get there, sooner or later, but when this forum goes into the same rabbit hole about the same topic and the same arguments, most of us will just give up and lose interest.
Some DTS topics being touched on here deservee their own issues and tracks.

I think we are having a misunderstanding here, I am not saying that I think we should be voting on implementations or even proposing them, what I'm saying is the observation current implementation is going to be an important source of deriving requirements. As for what anybody else in this thread is this saying about implementations, I'm not defending anybody, I honestly only skimmed what's going on because I got pinged like 5 times. That's why I asked you to clarify what you were talking about, since your meta-criticism sounded to broad to me since I personally think discussion of the current implementation is relevant to requirement gathering since most of the motivation for talking about this stems from the limitations people are discovering with it. But your brief request to filter discussion if taken completely as worded would exclude discussion of that, which is why I asked you to be more specific and clarify.

I personally find this comment a bit unfair - at least targeting @decsny specifically.

@decsny  do you really feel targetted here? I emphasized that all of this might be true and valid, but we need specific examples and we need to discuss this on a different track if it is really that bad.

@nashif You haven't given any specific examples either to support your own argument, right?

right, and I will not provide examples because I am not making any claims or arguments, you are the one making them.

most of us will just give up

Before most of us give up they'll have to make their first substantial contribution first or at least refer explicitly to what others have brought to the table. ;-) We have a lack of people listening to each others' argument and a lack of original content.

@decsny do you really feel targetted here? I emphasized that all of this might be true and valid, but we need specific examples and we need to discuss this on a different track if it is really that bad.

no, i don't feel targeted, from the comment he's referring to that was before I even came to this thread, and you're not wrong that specific examples are needed. Which is why in my original response to all the pings I specifically clarified in response to the claim that I knew about all this that I do not have a compiled list of examples, that would take time and effort which I have not had the luxury to spend. And I want to be clear also based on some comments in this thread that I personally don't desire to be the vehicle for an appeal to authority about what's being discussed here and don't completely see what value it adds anyways, since another point of what I was trying to say above is that those experiences I've seen a lot I would expect should also to have probably been commonly seen also by everybody else interested in this issue.

Now that we all seem to converge on a common problem statement (pending a few more examples I trust will be easy to find) we can hopefully see what the essence of this back and forth would be wrt the concern of this issue.
Requirements:

We have to make any conceptual distinction between different "a priori" (=application/use case independent) config containers (Kconfig vs. DT vs. X) so precise that they could ideally be enforced mostly automatically to avoid further conflict and confusion.
The more such "a priori" containers we propose the more it is important to make those distinctions objectively observable, otherwise users will find it very hard to implement and understand those concepts in practice. Imprecise distinctions and overlap in the config area have cost us as a community and even more our users too much energy already - as was again exemplified by all those comments here re sw vs hw and Kconfig vs DT.

What I mean by precise has been explained in this comment, see the definition of "axiomatic" vs. "ontological": Our configuration space must be accompanied by easy-to-observe (axiomatic) heuristics that refer to intrinsic properties of our software rather than relying to extrinsic "real world" concepts that cannot be mapped to code artifacts without contextual interpretations that we will by definition (and backed by a lot of research) never be able to agree upon objectively.
I've shown that the distinctions and concepts I'm proposing (feature selection vs. configuration, normalization, encapsulation) can be mostly automated or at the very least be defined with mathematical precision. The same can not be said about distinctions like hw vs sw or provisioning vs build time properties. At least not until someone has come up with a heuristic that really represents what we actually do. Note that I've shown that heuristics that have been proposed so far are quite obviously not good enough.
I'm of course open to being proved wrong or convinced otherwise. But certainly not by "hand wavy" concepts that no one has been able to define precisely so far.

I personally don't desire to be the vehicle for an appeal to authority

It's funny that you're saying this because you've made it credible that you are just one of those with authority. Citing you once more:

Actually, I would say as the only active person on the DT collaborator list for the last couple months, it's probably more apparent to me than most people that the gray area is an issue.

But I agree with you that no one should hide behind anyone else. I hope this is quite obvious from the way I'm defending my argument against strong opposition although I would certainly have many, many reasons by now to give up. Much more than others who didn't even join the party yet.

I personally don't desire to be the vehicle for an appeal to authority

It's funny that you're saying this because you've made it credible that you are just one of those with authority. Citing you once more:

Actually, I would say as the only active person on the DT collaborator list for the last couple months, it's probably more apparent to me than most people that the gray area is an issue.

But I agree with you that no one should hide behind anyone else. I hope this is quite obvious from the way I'm defending my argument against strong opposition although I would certainly have many, many reasons by now to give up. Much more than others who didn't even join the party yet.

The context of what you're quoting clearly was just to say that I in particular have been privy to a lot of instances where people had different opinions and it's clear to me that the definitions are gray, which you were in that thread claiming that I was not cognizant of when I asked you some questions. I don't see how my clarification that I am on the DT collaborator list and do in fact recognize it's current usage is not completely ideal (which I think at this point is not a revolutionary position for any community member), is grounds for people to use my name as a fire blanket to throw over any chaos in every other discussion regarding DT that they get themselves into as somehow a major point in some bombastic blather of bafflegab.

thought we had a good discussion yesterday

I don't agree. Yesterday's discussion was almost entirely redundant to what had been discussed online before (except for the truly original proposal to forget about config altogether and do everything in Kconfig). This was only necessary because a large part of the audience was not aware of arguments made offline (as we can again see in this discussion). None of what was said here had not been said previously elsewhere. It's a pity that so many comments are ignored. Plus we had to intervene twice to re-focus yesterday's discussion on the problem space.

I overlooked the fact that this is only about requirements, sorry. But it is hard for me to separate them here. Mostly here and in other discussions it is about DT vs. X. Count how often DT (devicetree) and Kconfig appear in your comments. After reading most of the comments here, I think we should go back to a more precise description of where devicetree can be used, including examples of acceptable gray areas. (Also, some guidelines on why the Kconfig option should not be used, e.g. in parts instantiated by C macros or devicetree.) Sampling requirements for the new configuration system without knowing exactly where others may be used is a step too far.

I also have to say: This whole discussion thread would be almost 100% unnecessary if requirements mentioned in other contexts would have been collected in the RFC in the first place before opening it for discussion. I'm forced to repeating everything here that I had already stated elsewhere w/o exception as others, too.

There are many comments from you that lack conciseness and are bloated. This takes more time to read and analyze. Please keep your requirements comments concise, and try not to take other people's statements out of context. I am confident that they will get more attention.

yes I also agree @fgrandel to be clear I think you can help the project as a valuable force for changing things in the right direction with your energy and systematic thinking but please take into consideration like johann said that the communication is the oil that lets the gears turn in the project

I hope this is quite obvious from the way I'm defending my argument against strong opposition although I would certainly have many, many reasons by now to give up. Much more than others who didn't even join the party yet.

I don't think there is as much opposition to you as you are perceiving, it's just you are a lot of steps ahead of everybody all the time and there's a lot of people that are involved in this discussion, we all have to be patient. This has been an issue for years and I have never seen so much community momentum about solving it as there is now, so this is the right time for you to involve yourself in the discussion about it, you just can't expect it to be solved immediately

There are many comments from you that lack conciseness and are bloated.

Thanks @jfischer-no and @decsny for contributing to the meta-discussion. I'm happy that we take the time to lead it, because I was the first requesting to focus on the problem space and also questioning our debating culture. And of course I'm open to feedback here, too.
We agree on one point: Our debating culture is often overly emotional, therefore often unnecessarily tiring and even if not so emotional then often very, very inefficient. This discussion is a rather representative example I'd say.
My problem is: I cannot disagree more about your definition of "conciseness", no matter how many thumbs appear under that comment. I'm aware that my communication style differs considerably from that of most in this community but it differs because I want it to differ. I fiercly resist the increasingly acceptable tendency to ignore any text that is longer than three lines because we've unlearned to concentrate on a line of thought for longer than five minutes. I also fiercly resist the tendency to speak before listening or thinking.
If anyone's concentration span has already been exhausted by these two paragraphs then, yes, we'll not be able to communicate because I am writing for an audience that I expect to routinely read specifications, white papers or datasheets with a few hundred or even thousands of pages patiently and not consider them "bloated". So I'm going to write provocatively long again (from your perspective) because it's my personal style and I expect respect for my style. Adaptation is required from both sides, so you can adapt to my style - as I usually do to yours, too.
Most comments here are short but redundant, contribute nothing original but shallow assumptions and emotions and clearly show that the person commenting has not taken the time to read respectfully and refer to what others had contributed before. I can open whatever Zephyr discussion thread and there'll be a high probability that I'll observe the same there, too. This is what is really detrimental to our efficiency of communication.
My comments usually take several hours to write so, yes, I do expect you to take a few minutes to follow along and digest, too.
If you all had taken the time to read and truly understand these two dense and conscise comments maybe asking back a few times to clarify your understanding I swear you'd then not have heard me speak up a single more time because everything I was forced to say since was 100% redundant (including this statement itself).
For more background and a more thorough explanation you can still read my RFC if you really care, just leave out the whole (now obsolete) DT/CT part and concentrate on the requirements sections. I know of at least three community members who understood it immediately so it is doable. But it certainly takes more than five minutes and rightfully so, because it took me over a week to interview everybody, think hard about what they have said and then come up with and think through a compromise that takes into account all requirements I had heard so far. It is all but perfect but it is certainly much better quality, more dense and less bloated than this discussion.
I also suspect that some seem to confuse "bloated" with "he seems to be talking about things I don't understand so, TLDR". I'm aware that most in this community do probably not have a background in epistemology, have probably not much practice in applying Conway's law or have probably not done much event storming to transform ontologies into shared language. Trust me I know exactly what I'm talking about. If you think this is bloated then you probably didn't understand.
To be honest: If I find a comment with words in it I don't understand I'd Google them, ask ChatGPT or ask the person commenting. I would certainly be patiently explaining in detail if someone cared to ask. If you think I'm arrogant I can ensure you that I had to read up on concepts I learnt from this community for many months non-stop - so certainly this means giving and taking. The reason why I might talk in a different language is because I have a strong background in humanities and application development. This is called diversity. And I love diversity because it is really healthy to a community. Unfortunately I don't find this community to be very diverse at times. And the way I'm getting feedback here confirms that a bit more practice in accepting diversity could do no harm to this community either.
So as long as I can see that most here have not really understood what I'm talking about I'd say: Be patient with yourself rather than asking for patience from me. Take your time to read up on concepts and others' arguments, think thoroughly about what you have read before you write again - even if it takes a few hours or days. And then feel free to ask if something is still not clear. In the best case I'll see immediately that I misunderstood something or was not precise and learn something.
Here are my desiderata for a more efficient debating culture (also cited from a prior contribution):

All comments must either strive to establish community requirements or refer to requirements brought forward by others before. (About half of the comments in this discussion would be filtered by this rule - but I can truly see improvement here.)
All suggested design improvements shall refer to well-established state-of-the-art design criteria including keywords that empower everyone to read up on them if they are not aware of them. (At least three fourths of the comments do not comply and so far I cannot see much questions being asked either, so I suppose this is due to a lack of common awareness of sota design methods from the application development space?),
Comments must show that the author is fully aware of the RFC to be decided upon and all arguments that were brought forward previously in the same context and explicitly refer to those if they contradict own proposals. (This kills almost every contribution made in this discussion.)

If we'd just followed these three simple rules then this discussion would not have to exist at all - not even in a more concise form. I'd probably win the bet that my initial RFC used less words and contains double the content of this whole discussion.
I ask everyone to do what I've done all the time: Make your own list of what others have said, think about it until you can make a proposal that represents a good compromise to your very best knowledge of all stakes that have been brought forward and only then comment (paying attention to explain your specific choice of compromise to those whose stake you'll not be able to fully represent). Then this goes through a few rounds of clarifications and cleaning up of misunderstandings and then we have a result. You'll see that then you'll be exactly as many steps ahead as I'm now because this is all I did.
If you all had done the same then not only would it have been much easier for you to follow along but also you would not have been such a big waste of my time. Sorry to say it so clearly but this is what it looks like from my side.

try not to take other people's statements out of context

As everyone I sometimes misunderstand people, and I'm sorry for it - and I've always said so when it happened. I strongly disagree with the assumption being made here that I consciously cited people out of context. AFAICS I was the only one to admit and apologize for misunderstandings so far because I truly care.
When I cite a lot of people then this is only because I've listened and I want to make sure that everyone's stake is being heard - even when they are not present. Those who were present in the last arch WG will have noticed that I didn't say a single word about my own requirements but I only ensured that requirements by those community members who were not present in person would be heard. And the same is true for this whole discussion - don't forget, I'm still just helping out Jukka to get a solution to his problem as he's the only other one who's really invested a lot into this topic so far. I have no personal stake at all but my promis to Jukka that I'd find a more acceptable solution once I put a red flag on his PR (which is still there for a reason). I call this responsability.
Once you think as much about my concerns as I've thought about yours, you can come back and cite me, too. I'll make sure to correct you patiently if I'm misunderstood as I've done with Carles summary above who certainly did not cite me wrong consciously either, I'm 100% sure.
(Overall measured reading time of this bloated contribution: 10 minutes of your lifetime - took me 3 hrs of mine to write it.)

IMHO the separation between configuring struct devices and other would be a good separation between devicetree configuration and other (yaml based) configuration.
I would say there are 2 reasons to configure struct devices:

For multi-instance support,
To allow reuse of driver code where only a limited set of parameters are changed,

These 2 reasons have not always been followed, but it is possible to correct this. We should however try to avoid anyone from using devicetree for configuration by making whatever needs configuration into a struct device. Some cases will remain border case (e.g. reserving some hardware region to store a network MAC), but these should be accepted.
The proposed yaml "configuration" can also be considered as provisioning instead of configuration, and we already have a system for provisioning: the settings subsystem. Now is this yaml configuration incompatible with the settings subsystem ? Not at all. The yaml configuration can be used to generate an in image provisioning solution, and using it as a settings based configuration can be done as a later step in the network config.
There might be some misconceptions about the settings system:

It requires persistent storage: this is not true. Whatever data that is available can be injected into a subsystem, this can come from external "storage" (as cbor, nanopb, thrift, ...), internal persistent storage, but also from in image data. The routine settings_runtime_set(key, data) can be used to inject data into a subsystem, and this doesn't need any persistent storage.
The settings system is a key-value storage: this is not true. The key that is used is only to help settings handlers to set/get the correct data of a subsystem.

The settings subsystem can be configured to use multiple sources of settings that can be added and removed as needed. There can be only one destination to store the settings, but also this destination can be changed when required.
The settings subsystem can also take some (sometimes hard to handle) work from you. E.g. disable a subsystem while configuration changes are being applied and reenabling when the all changes are done (disable when set is called, enable when commit is called).
The proposed yaml configuration method can also be used as a base to generate a set of factory defaults that are stored in a separate flash region and even configure where this flash region is to be found/written.
The yaml configuration method as provided in #68127 is a good first step to allow easier configuration and it is a versatile solution that does not break any other configuration/provisioning methods that zephyr already has. It is an enabler for more items that are missing configuration or have been (wrongly) added to devicetree.

I like the approaches and thoughts which @Laczen put into how the settings subsystem can be used for combining the configuration and provision approach. But, I think we still have a big issue to solve:

We should however try to avoid anyone from using devicetree for configuration by making whatever needs configuration into a struct device. Some cases will remain border case (e.g. reserving some hardware region to store a network MAC), but these should be accepted.

We still lack a clear definition of what is hardware and what is software. Exactly these "some cases" which you mention are very problematic, and not even that rare. Therefore, if we would like to keep hardware and software settings separated we will have to come up with a precise definition. If we are not able to find one we should avoid this separation, as it will be only the cause for countless other discussions and debates in the future. Which means time and effort, which we can invest in something more productive and useful.

@Laczen It would be nice if you could position yourself wrt to the argument given by others above:

"Everything that ties to a struct device is 'hardware'" as mentioned by @henrikbrixandersen and others. This heuristic breaks down in both directions: 1. By normalization rules, this definition would include almost all of the network interface configuration which is 1-to-1 to drivers. 2. We have a considerable number of DT properties that do not tie to struct device instances.

@benediktibk, @fgrandel,
In my comment there was no clear definition of what hardware is and what is not, because this definition is not needed. What I'm saying is that devicetree configuration is limited to the construction of struct devices. The struct device creates a kernel object and has a specified layout, these come with limitations (userspace, api, ...) and whatever is created like a struct device needs to adhere to these limitations. For the creation of struct devices dts can be used as a configuration tool.
When using devicetree for the configuration of struct devices also software items are allowed to be created and configured by devicetree. The discussion on whether or not something is added as a struct device is moved into a discussion on whether this representation as a struct device is efficient or not. And this is the correct discussion.
In the present situation there are already exceptions in devicetree. We should identify them and decide if they will remain as exceptions or they should be changed into a struct device definition. Both methods are acceptable, but a clear path for the near future is set. Are future exceptions allowed? My cristal ball is failing on this.
When we look further in the future the yaml configuration system could be used to move any configuration items from dts and limit dts to a pure hardware definition. Whether this is wanted/needed/... is not something I'm taking a position on.
@fgrandel, as should be clear from the above there is no need for me to take any position. Both configuration from dts and yaml can be accepted, but for dts they are limited to the construction of struct devices. If someone wants to configure something that is not a struct device yaml is the way to go.

The discussion on whether or not something is added as a struct device is moved into a discussion on whether this representation as a struct device is efficient or not. And this is the correct discussion.

Could you please clarify what you consider as efficient in this context? Are we talking flash footprint, runtime, maintainability, ... ?

@Laczen I will try to summarize what you are aiming for, just for the purpose of checking if I understood everything correctly. Please correct me, if I am wrong.
You intent to have two separate configuration systems/sources. One is the devicetree, from which the struct devices are constructed. The other one is the yaml, which will be used for software items.
From this I conclude, that there will be a separation between hardware and software. Which brings me back to my original question: Where is the line? I'm sorry to be so nitpicky on this, but if we intent to use such a system we will also have to come up with a definition for this separation. And yes, this definition is urgently needed in such a system, as there will be otherwise lots of confusion and discussion about this topic. Therefore, as long as we cannot come up with a precise and practical definition for this separation we won't be able adapt this architecture which you have proposed.

I agree with @benediktibk about having a properly defined limit in what belongs to hardware space versus what belongs to software space.
In a more general way: I think we need to step back and clearly define all the scopes of what is being addressed here. If not, we will end up arguing forever on technical details which - at this stage - do not belong here, imo.
Couple of questions:

What are the domains this will be targeting? I see 3 at least: device drivers, subsystems and user applications. We definitely want to solve the 2 firsts in a generic way, do we want to solve the 3rd as well?
Built-time vs Run-time? Either because of hw constraints and/or because of the targeted usage, we will need to evaluate both being supported and in a flexible way (switching from one type to another). #68127 has been done that way - at built time - to solve complex network configuration for test cases, obviously there are field cases which will need runtime network configuration.
What do we want to do with configuration options in Kconfig when these are about specific settings? (a subsystem thread priority, a subsystem thread stack size, ...). Do we want to leave them there or throw them away into the whatever-new-solution-we-will-have?

I do not think we can properly answer these without getting more use cases, listing their requirements and cross-checking them in order to have a better overview.
We could go stakeholder by stakeholder:

device drivers? answering the very first question above would already solve most of this issue there
networking? this stack is complex enough to have various aspects such as the stack itself, the network interfaces, the sub-domains such as WiFi, canbus, 15.4, etc...
Bluetooth?
storage?
...
any subsystem that would take advantage of such configuration mechanism.
...

@Laczen

In my comment there was no clear definition of what hardware is and what is not, because this definition is not needed.

I agree that what we're looking for is a good heuristic that precisely defines what goes where based intrinsically on our software, not on something external. I agree, that the heuristic you're proposing is such an intrinsic heuristic. Unfortunately this specific heuristic does not work as has been pointed out several times above.
Meta: So yes, you do contradict what was said before w/o explaining how you want to cover the weaknesses that your argument re-introduces. As you seem to have difficulties to derive this from what has already been said yourself, I'll try to re-explain, although this is 100% redundant and produces the kind of bloat, that we can and should avoid. Plus you're forcing me to talk about DT because your argument leaves the problem space.

What I'm saying is that devicetree configuration is limited to the construction of struct devices.

No, it is not - and that's exactly the problem. DT contains both, more and less, than the configuration of struct devices and there is no way we're going to change this in practice:

More: Interrupts, CPUs, clocks, memory, intermediate helper nodes, etc. (to name just a few). As Linux (and DTSpec) explicitly require those nodes inside DT we can not move them elsewhere.
Less: Almost all properties from the network configuration belong 1:1 to struct device (just as driver configuration does - api and iface have the same intrinsic status inside struct device). Based on the rough heuristic you propose all those iface properties would go inside DT, too. I assume that this is not what you intend?

In the present situation there are already exceptions in devicetree. We should identify them and decide if they will remain as exceptions or they should be changed into a struct device definition.

This is a recursive definition: You introduce a definition that contains undefined exceptions which you want to define "later" based on what? As can be seen above, those "exceptions" are too important to ignore them now: They are the OP's very problem statement. So as long as you cannot say how precisely we are going to deal with them, your proposal simply reproduces the current situation and I therefore cannot see its merit.
I proposed alternative heuristics in my RFC, which do not have these weaknesses but you seem not to be aware of - please read my proposal then and position yourself with regard to it.
Meta: So yes, surely you should have read and understood (and if not understood: asked back) first to be able to position yourself accordingly w/o forcing others to waste their time and produce bloated redundancy that makes it hard for everyone to follow the discussion.

I do not think we can properly answer these without getting more use cases, listing their requirements and cross-checking them in order to have a better overview.

Agreed. Are you aware of the use cases in my RFC? I think they cover at least all use cases that have been brought forward by community members so far. Many more than the OPs initial list.

We could go stakeholder by stakeholder:

I agree that this would be a possible solution. But I still find it rather arbitrary and way too complicated. I'd very much prefer a single heuristic that:

can be applied w/o having to discuss and define separate scopes for every single use case and subsystem separately,
is way simpler than a complicated definition that requires loads of documentation to be understood (that no one will read anyway),
can be applied to future subsystems w/o having to load the arch WG with a decision each time.

The heuristics based on feature selection (Kconfg vs. DT) and normalization/encapsulation (DT vs. X) are exactly such rules that can be documented in two simple paragraphs (see my RFC, I used just two paragraphs to define both) and extend to all current and future subsystems (or software modules more generally).

@benediktibk, @tbursztyka, I am carefully avoiding any use of software/hardware because this can lead to difference in understanding. There are drivers that are a pure software representation of some hardware (e.g. all simulator devices), is this software (space), is this hardware (space) ? Is an fpga with uploadable image software/hardware? Is a driver that emulates flash over an IPC channel software/hardware?
Stating "devicetree is used to configure struct devices" does not imply anything (software/hardware) on what will be struct devices and what not. Trying to define what can be a struct device (driver vs subsystem) is something that I'm not trying to do, it is not needed (although it would help to avoid dissussions). There will always be a grey area where something could fit into both and there is no wrong selection in that case.
Stating "devicetree is used to configure struct devices" does not imply that anything that has a structure that resembles a struct device should be configured trough devicetree.

@Laczen If I understood it correctly you are saying it is not necessary to separate between hardware and software, it is only necessary to define what should be in devicetree or X (possible yaml)?
If that is the case you won't have to define the difference between hardware and software. Which is hard, as you mentioned yourself. But this only shifts the problem: How do you then define what should be in devicetree or in X?
We should definitely avoid grey areas and try to make it as clear as possible to avoid useless lengthy discussion in the future.

@benediktibk, @tbursztyka, I am carefully avoiding any use of software/hardware because this can lead to difference in understanding. There are drivers that are a pure software representation of some hardware

Ok I see your point, perhaps we need to stick to "devices" instead of hardware, in order to be semantically right. Emulated/simulated/virtualized devices are seen like as actual devices from an OS point of view. Even if that does not mean any physical evidence. Let's take ivshmem for instance, found in Qemu and ACRN: it will never be a physical hardware, but it is a device on a bus with registers, memory and features.

Stating "devicetree is used to configure struct devices"

I do not like this extreme short-cut so let's not use it.
Devicetree if used to describe the devices present in the target. Which devices can be related to an actual physical device or not as seen above.
When it comes to defining what should go into DT and what should not, perhaps we could relate to hard vs soft settings instead.
For instance, anything that relates to the internals of a device (registers, memory, internal feature enablement) would be interpreted as hard setting. On the other side, anything that relates to the device driver internals (thread, stack, mapping, priorities, API features etc....)  could be seen as soft-settings. So far, I believe this has been managed in the right way. Where soft settings have landed generally in Kconfig (at a cost of loosing the ability to manage these settings per-instance). I have very few example where a setting landed in DT where it should not have like "zephyr,deferred-init" for instance. But all in all, this differentiation has been done pretty well.

Devicetree if used to describe the devices present in the target. Which devices can be related to an actual physical device or not as seen above.
When it comes to defining what should go into DT and what should not, perhaps we could relate to hard vs soft settings instead.
For instance, anything that relates to the internals of a device (registers, memory, internal feature enablement) would be interpreted as hard setting. On the other side, anything that relates to the device driver internals (thread, stack, mapping, priorities, API features etc....) could be seen as soft-settings. So far, I believe this has been managed in the right way. Where soft settings have landed generally in Kconfig (at a cost of loosing the ability to manage these settings per-instance). I have very few example where a setting landed in DT where it should not have like "zephyr,deferred-init" for instance. But all in all, this differentiation has been done pretty well.

I like this method of differentiating between the two.

I'd like to add the following sidenote about hard vs soft settings;
Currently an IRQ priority is a hard setting defined in device tree (per instance), but thread priorities are a soft setting.
This looks evident at first, but if a device driver has a thread for each instance, there is no way of prioritizing them currently with kconfig.
If we keep this hard/soft setting distinction, and would add whatever soft setting language/framework, this is confusing IMO.

If we keep this hard/soft setting distinction, and would add whatever soft setting language/framework, this is confusing IMO.

Looks to me that you already pointed out the core of the issue to remove any confusion in your previous statement. Unless I misunderstood this last sentence.
As long as a setting is not instance centric (i.e.: is a general setting), it should be in Kconfig.
Or to rephrase: as soon as a setting is valuable to be set per-instance, it should not be into Kconfig.
I do not see this too confusing, as long we properly define the scope of each configuration component: Kconfig, DTS, else.

Grey areas between hardware and software configuration are there in some cases because we are missing a software service on top of the hardware, so we tend to do that directly in DT instead of going one level up in to the software layer. So for example, there are sensors that are initialized and setup using DT, but we have a sensing subsystem setting on top that manages such sensors, same thing can be said more or less about networking and many other subsystems that deal with devices.
So, as said above already multiple times. DT shall be used for hardware initialization, Kconfig for software selection and modularizing the resulting image and optimizing it for the intended use-case and the configuration system shall be used for configuring zephyr services (subsystems) and provisioning both at build time, with so-called factory settings and later at run-time through some management system and for example OTA protocols for connected systems.
Basically, anything that can be changed at runtime using some subsystem interface, should be fair game for this configuration mechanism.
The runtime aspect excludes Kconfig and DT already. Kconfig is a build time configuration tool, DT and its tooling in Zephyr is used at build time as well.
@Laczen  in https://github.com/zephyrproject-rtos/zephyr/issues/77638#issuecomment-2324390603 is the closest so far to what I think we want.
We need to turn things around and think a bit differently about things, here a few items to start with:

Each subsystem or software component, where applicable, should be configurable both using Kconfig for static and initial use and at run-time using a subsystem specific configuration.
Subsystem configuration shall use a common infrastructure and syntax, managed in a single globale namespace
Each subsystem defines the schema needed for its configuration, managed under a global configuration namespace.
A configuration subystem (using existing settings subsystem) should be used to manage system settings. System here is the collection of subsystems that are configured in. The configuration subsystem is how build-time and runtime configuration are managed and how they get translated into setting data that can be saved into the setting db.
A subsystem is started with defaults coming for example from Kconfig. Such default settings can be overriden at build time using a configuration file following the schema of the subsystem. Certain configurations, i.e. multi-instance might not come from Kconfig and in this case will need to depend on the subsystem configuration...

When I think about all of this and where want to go, I think of redhat/fedora kickstart and SuSE's Autoyast (Disclaimer: I wrote and started Autoyast long long time ago). Just take a look at https://doc.opensuse.org/projects/autoyast/#CreateProfile-Network, does this remind of something?  See https://github.com/zephyrproject-rtos/zephyr/pull/68127/files#diff-37db3efef3521dc752b198d36cdb9de1e8f6f7a0433d43c6d8ffa959791d0b0b
Device Tree vs Kconfig was a hot topic long time ago, we have a section in the docs describing the difference and guidelines already: https://docs.zephyrproject.org/latest/build/dts/dt-vs-kconfig.html

I do not see this too confusing, as long we properly define the scope of each configuration component: Kconfig, DTS, else.

Agreed. Please note that we already have documentation on Kconfig vs. Devicetree usage: https://docs.zephyrproject.org/latest/build/dts/dt-vs-kconfig.html Perhaps this should be moved up in the documentation hierarchy and expanded (and eventually also cover whatever configuration infrastructure, we come up with here).

@tbursztyka

When it comes to defining what should go into DT and what should not, perhaps we could relate to hard vs soft settings instead.

This comes very close to what I have in mind. There seem to be edge cases, though, that require an even more precise definition, eg. we have several driver subsystems where, depending on hw capability, the same driver feature could be a driver internal or offloaded to hardware, e.g. synchronization of PTP timers and to a rather extreme extent in IEEE 802.15.4 radio drivers, where large parts of the MAC may optionally be offloaded to and configured in hardware. Maybe I didn't fully understand your criteria, yet, but in those cases I would find it hard to categorize a priori.
If we take usage sites of settings as the central easy-to-observe criterion, then what you have in mind seems to fall into place quite naturally.
A hard setting would be anything that has all usage sites inside the soc, boards, driver and arch folders, a soft setting the rest.
Seems that this would place settings like thread priorities and stack sizes quite naturally in the soft settings category while the kind of registers etc. you list would probably only be used in the drivers folder. In the edge case of soft MAC vs hard MAC features, settings would land in the soft settings category as they are accessed from the subsystem folder, too. This would again correspond to my intuition where such settings belong.
If we add or remove usage sites and for backwards compatibility, moving settings between the two categories is not a problem as long as we join both models in a common intermediate config tree.
What's nice about this criterion is that it can be automated for CI/CD or as an Initial analysis of what needs to be deprecated and moved from Kconfig and ot DT to X. Even the migration could probably be partially automated.

@nashif
Nice reqs list. :-) Thanks.

Each subsystem or software component, where applicable, should be configurable both using Kconfig for static and initial use and at run-time using a subsystem specific configuration.
A subsystem is started with defaults coming for example from Kconfig. Such default settings can be overriden at build time using a configuration file following the schema of the subsystem. Certain configurations, i.e. multi-instance might not come from Kconfig and in this case will need to depend on the subsystem configuration...

I find the a priori build/provisioning distinction non-intuitive and unnecessarily complicated as it duplicates the same settings in different places. It becomes more confusing when separating multi-instance vs. single-instance config. And what if settings need to be migrated from single to multi instance?
We needed such workarounds in the past because we had no other choice. But now we have the chance to improve UX defining one canonical semantically defined source/target per property that is easy to document for beginners and easy to review (even automatically).
Of course some will like to do mass-deployment of independently produced provisioning images, but this should be opt-in IMO and needs to be customizable on the source and target side (individual build/provisioning split, thrift/protobuf/cbor/custom dbs/key vaults, ...).
I think concentrating Kconfig on feature selection (things that change code) while concentrating DT/X on hard/soft settings (things that change data) with optional source/target splits is the simplest zero-footprint-by-default and at the same time maximally scalable approach.

https://docs.zephyrproject.org/latest/build/dts/dt-vs-kconfig.html

Agreed, cited this myself above. We should simply deprecate the no longer required workarounds in that doc which will make it much easier to understand - especially if accompanied by tooling that helps contributors to validate the canonical in-tree split automatically.

Subsystem configuration shall use a common infrastructure and syntax, managed in a single global namespace

This alone doesn't define the internal structure of the global namespace w/o ambiguity. That's where encapsulation and normalization come in as precise and easy-to-automate rules. Otherwise 100% agreed (= "common normalized intermediate representation").
We have to also map DT and the deprecated parts of Kconfig into this global namespace, though. This is required for backwards compat/migration, to ensure that cross-references can be validated at build time with useful error messages and to enforce data integrity across all sources and targets, no matter how many we'll add in the future.

Each subsystem defines the schema needed for its configuration, managed under a global configuration namespace.

Yep (= "splitting the global data model based on encapsulation aka folders/files").
You did not mention specific binding requirements, though. The currently proposed binding system does not work:

It does not support all C types
It requires us to re-invent the wheel (and later maintain separately) every aspect of typing, documentation and validation although we know from experience that inventing a good binding system is hard.
It is not zero-footprint
It doesn't provide clear overlay semantics
etc.

I also hope our users will not have to learn two binding languages unnecessarily. Binding is our own invention, no one can say it is "hard settings only". Binding is a UX hurdle currently so it should be improved across hard and soft settings. I list several deficiencies mentioned by community members in my RFC.
We should therefore require converging towards a common binding system in the solution space w/o concentrating too much on a specific implementation path for now.

The configuration subsystem is how build-time and runtime configuration are managed and how they get translated into setting data that can be saved into the setting db.

I'd say, that this doesn't comply with the zero-footprint and minimal dependency by default requirements above. I prefer an initial macro-based approach for many reasons:

simpler, less initial development effort, more iterative learning curve (especially in the OPs specific initial use cases - which do not require any additional config/settings subsystem to get us started).
zero-footprint, no subsystem dependency for low resource applications, improved decoupling of config space from targets
re-use of existing macros (raw macro nomenclature, access props, parents, nodes, cross-references, etc.) and community know-how (different prefixing, e.g. CONFIG_NODE/PARENT/PROP(...) but internal code-reuse/aliases).
ensures a soft migration path and backwards compat
... more args given in my prior comments

This does not exclude any additional target in the future. To the contrary: the additional layer of decoupling makes this approach more open to extensions, including the mentioned config subsystem/settings db that we will need at some time in the future, but not for initial use cases.

When I think about all of this and where we want to go, I think of redhat/fedora kickstart and SuSE's Autoyast

Good inspiration. Acquainted myself with the central notions that mostly seem to fit the "mass deployment" use case. Fits my own proposals. My proposal just goes a bit beyond by further elaborating the concept based on our Zephyr-specific low-resource/application dev environment requirements (which differ considerably from those of Suse).

One more thing just came to my mind: While I do not question the settings subsystem as an (optional) config target at all, we might run into a chicken and egg situation, when early subsystem configuration is required to bring up the settings backend itself or other subsystems require config before the settings subsystem is able to provide them.
In the case of the network subsystem "early" means before POST_KERNEL as configurable interface flags must be updated before net_core.c tries to auto-start devices in it's init hook.
This is covered by the "macros + optional settings" approach, but not by the "settings only" approach.

One more thing just came to my mind: While I do not question the settings subsystem as an (optional) config target at all, we might run into a chicken and egg situation, when early subsystem configuration is required to bring up the settings backend itself or other subsystems require config before the settings subsystem is able to provide them.

The settings subsystem does not require a backend to operate, so it also does not require bringing up the settings backend. The settings subsystem consists mainly of two parts: the first part is an agreement how configuration changes will be applied (through the settings handlers), the second part is the storage backend that uses the first to apply configuration values stored in the backend. To use settings without storage backend the routine settings_runtime_set is available.

TLDR;
Another requirement derived from a recent example:

It SHALL be possible to easily move properties between hard and soft settings (DT and YAML files) and between compats/types (binding files) w/o losing backwards compatibility of existing compats.

Corollaries:

It SHALL be possible to use the same typing (binding) syntax for both, hard and soft settings.
It SHALL be possible to compose types (in addition to type inheritance) in a single DT/config node.

Full Rationale:
https://github.com/zephyrproject-rtos/zephyr/pull/76087#issuecomment-2346896173 is another recent instance of the futile attempt to define the appropriate driver-supporting hardware model forever rather than accepting that the model will always partially be defined by the driver model. And worse: We'll not be able to precisely say which is which from the outset. Not even if we use reviewer-independent attribution rules based on usage sites.
This example also shows once again how we currently force every contributor to claim that her new DT model is more appropriate for DT (aka "much closer to hardware", cf #76087) implying that the previous was "partially wrong" until the new model will turn out to be partially "wrong" itself due to unexpected requirements.
Please don't tell me that if we had done it "right" in the first place then we wouldn't have to deprecate a "wrong" model now. I find this unfair towards both, previous contributors and reviewers. Only hindsight bias allows one to claim something like this.
Of course some initial hard/soft settings split (or the split between compats) might prove largely appropriate over time in some cases. But no human being should be forced to precisely predict where it will break down in the future due to unforeseen new requirements, information or viewpoints. Therefore we need to be able to deal routinely with such cases - even more so with the additional DT/YAML distinction - w/o having to claim that what was done previously must have been "wrong".
IMO absolute categories like "wrong" and "right" are to be banned from reviews unless they mean "good enough" to cover for known requirements - and not a bit more. In that sense the previous schema now replaced in #76087 was clearly appropriate IMO. What we're trying to do with DT is outdated practice from waterfall development based on some "correct model" in absolute terms which we clearly disapprove of in the more agile Zephyr community elsewhere. Why not here?
Side note: If we follow my proposal, the routine solution for a situation as the one in #76087 will be relying on flexible type composition and the intermediate common data model. We'll be able to re-split and re-distribute properties from release to release between compats and/or hard/soft settings along criteria of factual requirements and re-use w/o assuming anything about the future. Unless s/o comes up with a better proposal for type composition and normalization in the solution space, these are therefore requirements, too.

@Laczen

The settings subsystem does not require a backend to operate

TLDR;
Your statement is not true for persistent configuration / provisioning. I therefore uphold my requirement unchanged:

Requirement: Availability of config as early as possible, at the very least before POST_KERNEL.

Note that I wrote "config" - not "provisioning".
And additionally:

Availability of provisioned settings (not configuration) SHALL be restricted to the worst case settings backend setup-time by default so that backends can be switched w/o breaking it.

Rationale
I took my time to answer because I wanted to become more acquainted with the settings subsys first. What you're saying is true in general but untrue for boot configuration and persistent provisioning (including persistent runtime re-configuration) which is what we're discussing here. Most persistency backends will routinely require a backing driver to by "ready" (e.g. a flash driver) or even a file system to be mounted before they can hand out settings.
So setting backend setup time will matter at least in some cases like the one I mentioned.
It is true, though, that clients can force subsys initialization early (if and only if the specific backend supports it) as the subsystem itself can be initialized from everywhere and does not rely on system initialization order per se.
As I've shown based on the OPs actual use case, we cannot restrict availability of boot-time hard/soft settings (not provisioning) to the worst case settings backend driver setup time, though. We also do not want to rely on implementation-specific backend setup times for provisioning because that would mean that it may break just by switching the settings backend.

TLDR;
Another requirement derived from a recent example:
It SHALL be possible to easily move properties between hard and soft settings (DT and YAML files) and between compats/types (binding files) w/o losing backwards compatibility of existing compats.

Corollaries:
It SHALL be possible to use the same typing (binding) syntax for both, hard and soft settings.
It SHALL be possible to compose types (in addition to type inheritance) in a single DT/config node.

Where was this decided? What group of people/working group decided this is a must have feature?

let's not get carried away with what "requirements" mean in this context.  The way I see it, those are just wishes or inputs, that can be accepted or rejected. Already many of the requirements here are conflicting or out of scope, depending on how you understand what the scope is. The title of this RFC is actually is very limiting and already was rejected, i.e. it calls for compile-time and multi-instance, but we want runtime and we want to be able to deal with all the types configurations, not only multi-instance.

@Laczen

The settings subsystem does not require a backend to operate

TLDR;
Your statement is not true for persistent configuration / provisioning. I therefore uphold my requirement unchanged:

Requirement: Availability of config as early as possible, at the very least before POST_KERNEL.

Note that I wrote "config" - not "provisioning".

@fgrandel, I'm going to repeat myself. The settings subsystem consists of 2 parts:

A set of methods to apply changes to a subsystem (the settings_handlers),
A settings backend that allows persistent storage,

Item 1. can be used at any time, and does not require a settings backend to be available. This can be used to configure or provision any subsystem. E.g. if a "net" subsystem provides handling of "/net/cfg" this can be used to inject whatever config into the subsystem (also to assign a constant struct pointer, that is the result of data defined in a yaml, to whatever pointer in the subsystem struct). This also allows e.g. configuring buffers, ... by defining them in a include file and assigning them through the set handler.
Using this method would just need the creating of a init routine that uses runtime_set("/net/cfg/xxx", ...).
Item 1. cannot only be used for configuration, but also to apply changes when coming from e.g. a terminal input or any other remote configuring.
Item 2. would allow making extra config changes that are stored persistently. But this of course only when this is available. Whatever is stored persistently might never contain the above "/net/cfg" (but it can).
Defining one (proper) settings_handler allows the use of configuration coming from: in image structs, remote provisioning, persistent storage...

Requirement: Availability of config as early as possible, at the very least before POST_KERNEL.

what comes before POST_KERNEL is kernel....
A subsystem that supports the loading  of configuration at runtime will depend on the 'configuration' service which on its part will depend on any storage backends and devices, all of that can be done in POST_KERNEL with proper dependency setup that @tbursztyka  is working on.

A subsystem that supports the loading of configuration at runtime will depend on the 'configuration' service which on its part will depend on any storage backends and devices, all of that can be done in POST_KERNEL with proper dependency setup that @tbursztyka is working on.

Why does the configuration service need to depend on the storage backend and devices? The storage backends are sources of configuration data that need to be processed by the configuration service. The storage backends are only one source of configuration data.

A subsystem that supports the loading of configuration at runtime will depend on the 'configuration' service which on its part will depend on any storage backends and devices, all of that can be done in POST_KERNEL with proper dependency setup that @tbursztyka is working on.

Why does the configuration service need to depend on the storage backend and devices? The storage backends are sources of configuration data that need to be processed by the configuration service. The storage backends are only one source of configuration data.

@Laczen What provides what configuration data and where that data is stored and using what mechanism is TBD, my point here is that whatever design we come with, it will need to work based on dependencies, most likely all running at the same runlevel, i.e. after the kernel is up.
But to touch briefly on what you are saying here, why would there be different sources for where configuration are kept? I understand you will have the options to store the configuration in different location, but the entire configuration namespace would be stored in 1 place, what is the usecase for having different parts stored in different places? i.e. network on disk1 and bluetooth on disk2? Maybe I am missing something here :)

But to touch briefly on what you are saying here, why would there be different sources for where configuration are kept? I understand you will have the options to store the configuration in different location, but the entire configuration namespace would be stored in 1 place, what is the usecase for having different parts stored in different places? i.e. network on disk1 and bluetooth on disk2? Maybe I am missing something here :)

As I understand it, the idea is to have different (read-only) backends generated from the static configuration. These backends can be used to supply default configurations given at build time, but would still allow for both provisioning and run-time backends to override the static "factory" configuration. If no provisioning/run-time configuration changes are needed, the overhead of a read-only backend for supplying the static configuration is very low, while keeping the exact same API for retrieving the settings from the subsystem/service/library/application code.

But to touch briefly on what you are saying here, why would there be different sources for where configuration are kept? I understand you will have the options to store the configuration in different location, but the entire configuration namespace would be stored in 1 place, what is the usecase for having different parts stored in different places? i.e. network on disk1 and bluetooth on disk2? Maybe I am missing something here :)

As I understand it, the idea is to have different (read-only) backends generated from the static configuration. These backends can be used to supply default configurations given at build time, but would still allow for both provisioning and run-time backends to override the static "factory" configuration. If no provisioning/run-time configuration changes are needed, the overhead of a read-only backend for supplying the static configuration is very low, while keeping the exact same API for retrieving the settings from the subsystem/service/library/application code.

@henrikbrixandersen it could even be simpler (but in essence your understanding is correct): the static configuration generates a default const struct (e.g. net0_cfg = {}) for the subsystem (or for a part of the subsystem). At the same time the "backend" is constructed:
const struct static_backend net0_backend[] = {
        .name = "net/net_cfg",
        .data = &net0_cfg,
        .data_size = sizeof(net0_cfg),

The "net/net_cfg" handler would copy the constant struct to the runtime struct. The static configuration init would just walk over the static_backend and use settings_runtime_set() to apply the static configuration settings.
The above is only a simple example, there could be multiple const structs defining the configuration and/or "real" data (not a pointer) in the definition of the static_backend.
The static_backend as defined above is so small that I don't know if this even deserves the name backend.

I understood right away that for the build-time configuration you personally seem to prefer the backend-less settings subsystem approach - it was already part of my RFC, remember? But have you also noticed that others seem to prefer usage of vanilla macros/structs for build-time use cases where dependency on a separate settings subsystem is neither required nor appropriate? Depending on the use case it may introduce unnecessary overhead (as little as that may be), unnecessary complexity and its settings cannot be fully checked for completeness and consistency at dev and build time due to its weakly typed API. We are trying to specify a system that caters for all our requirements, not just yours in particular. As long as your preferred choice is also supported, I think there's no need to convince everyone that your choice is the only good one.

This topic is for requirement gathering only, nothing here is to decide how it should be implemented

I have a working poc for the build-time macro/struct and no-backend settings-subsys use cases and I'm at about 50% of a poc for the mass-provisioning use case via independently built partitions. Give me a few weeks to finish it and I'll have a poc for all requirements that have been stated so far. It's not so difficult after all.

Then you have completely misread this topic and you're doing the wrong thing, you do not build a proof of concept for requirements, proof of concept comes after agreeing what an implementation should look like which comes after agreeing requirements

@carlescufi It would be really great if we could focus in the meeting today on what we are trying to solve, without getting too much into details about how. I think it is necessary that we shift the discussion a little bit towards the first one, and not discuss how we could solve the yet unknown requirements.
Once we agreed upon things like do we want to use the same approach for the static configuration as well as for runtime provisioning we can then figure out from which file format we for instance want to put information into the settings subsystem. These should just be a few examples of hot topics, which in my opinion are happening too early.

Architecture WG:

We all agree that this issue should cover both software configuration as well as device provisioning
We all agree that we want to support static (build-time) as well as runtime (either settings, const structs or other)
We all agree that

so, the result of still being confused how we ended up spending time on cbor, thrift and protobufs instead of the problem we are trying to originally solve, here are some thoughts to try and bring us back on track. There is no question that the above have a place in an overall architecture, but they are not the basic building blocks we need to be concerned about. The amount of protocols that can be supported and used is countless, if there is a need, they will be supported in some way or another.
Here is a digram to describe some of the layers and idea that have circulated in few issues and PRs:

More or less, looking at the left side and bottom up, you will see what @jukkar has been trying to achieve in #68127 (@jukkar , correct me if I am wrong). That is a usecase, where you provide a static configuration via  configuration file (yaml) to the networking subsystem, the configuration will be hardcoded and part of the binary. This does not require the settings subsystem and based on the changes in the above PR, not something you can change at runtime for example.
Based on the feedback in the above PR, in other issues and discussions, there is a desire to make this generic and not a networking thing and utilize the settings subsystem if desired to store such configuration and make them runtime configurable if there are no foot print constraints and if there is a way to change and save those changes on some storage.
This is where we start moving to the right of the diagram providing this functionalty. So, instead of having the data captured in some auto-generated header that is directly read by the networking config system, a configuration system layer is added to support additional (non-zero-overhead!) use cases, i.e. the use of the settings subsystem which in the simplest  case can be reading the data from memory and act as the interface for retireving the data, even without the ability sto store to some storage.
If we get this far, we will be almost there. The runtime configuration part can be exteded and enhanced in many different ways, starting from some shell backed, to a web backend and then  some other protocols that can feed configuration data that originated from a common format supported by a well defined schema.
Throwing this in to trigger some more discussion and input. Hope this helps. We can discuss details in the next meeting.

Formalization can be a pain in the butt, but unfortunately it is sometimes helpful. Therefore, let me try to collect and phrase some requirements.
Roles:

User: Person, which uses Zephyr to implement an application. Related requirements are URx
Contributor: Bucket for the Zephyr Roles Contributor, Collaborator and Maintainer, basically somebody developing and maintaining the Zephyr codebase itself. Related requirements are CRx

Requirements (subitems are examples):

UR1: As a user I want to be able to set a configuration which is used during build time.

Configuring a buffer size
Configuring an I2C address for a port expander

UR2: As a user I want to be able to set different configurations for multiple instances of the same type.

Configuring different network IP addresses for multiple network devices

UR3: As a user I want to be able to reuse my existing knowledge of Zephyr.

Devicetree syntax
Bindings
KConfig

UR4: As a user I want to have a minimal flash and runtime overhead for configurations which are used during build time.
UR5: As a user I want to be able to override the configuration from build time during runtime.

Setting default network settings, which are then overriden with settings from an SD card
Provisioning bluetooth mesh setups

UR6: As a user I want to be able to manually change configurations during runtime.

Setting a different network IP address, based on some application logic.

UR7: As a user I want to have minimal flash and runtime overhead for configurations which are set during runtime and override the ones from build time.
UR8: As a user I do not want to have to specify the initialization order of entities manually.

Automatically deriving the initialization priorities for instance from dependencies in the device tree.

UR9: As a user I want to automatically generate configurations from other systems, which are then equally treated to manual settings.

Generate build time configuration or runtime provisioning from a custom database.

UR10: As a user I want to have an easy to understand interface for all configurations.

Especially targeted for beginners

UR11: As a user I want to have an easy migration path for an already existing Zephyr application to a new configuration system.
UR12: As a user I want to be able to sign a firmware image.

use it in combination with MCUboot

UR13: As a user I want to ensure that configuration during runtime has no been tampered with.

When already using a signed and/or encrypted image, also the provisioning source should be verifiable and/or encrypted

CR1: As a contributor I want to support a minimum amount of input file formats.

Avoid multiple possible inputs at the same time like Kconfig, devicetree, yaml, Json, CBOR, ... whatever you can think of

CR2: As a contributor I want to have an easy and clear decision process where a new configuration should be located.

Should this configuration be in the devicetree or Kconfig?

CR3: As a contributor I want to automatically generated documentation for all available settings.

This list is definitely incomplete, but I gave my best to collect whatever I could find in the issues and PRs related to this topic.
From such a list we could then deduct how we want to provide a solution from them. A rather simple one would be for UR1: We just reuse our existing mechanism which turns a devicetree into C-macros. Of course some requirements might contradict each other, like CR1 and UR3. Or for the actual implementation it might make sense to prioritize them, as UR1 is probably easier to implement and provide than UR5. UR1 we basically already have, where on the other hand UR5 will require some sweat and tears.
Please consider this as a proposal, which is definitely incomplete. But I think if we first collect these requirements, probably best offline so that everybody can think quietly about their possible use cases and phrase them pointedly, we can then use this list to design a solution. I would also volunteer for updating this post with more proposals or different phrasings, if somebody has inputs and ideas. Or somebody can also steal this list and collect the information elsewhere, no hard feelings (@carlescufi in this RFC maybe?)
Btw, the Zephyr community did already use a similar approach for the west development:
https://docs.zephyrproject.org/latest/develop/west/why.html

Good list, thanks. few comments

UR1: As a user I want to be able to set a configuration which is used during build time.

Configuring a buffer size
Configuring an I2C address for a port expander

You can already do that using both Kconfig and DTS, and you do not need this new "configuration system" to accomplish this. what am I missing?

UR3: As a user I want to be able to reuse my existing knowledge of Zephyr.

Devicetree syntax
Bindings
KConfig

You forget to mention that Yaml is something else widely used in Zephyr

UR8: As a user I do not want to have to specify the initialization order of entities manually.

Automatically deriving the initialization priorities for instance from dependencies in the device tree.

not sure how this is related, what order are you referring to?

UR12: As a user I want to be able to sign a firmware image.

use it in combination with MCUboot

missing how is this related.

@benediktibk Thanks for consolidating. Good job summarizing what was contributed so far. I added mostly additional examples for illustration. My additions in italic.
@carlescufi Could you add that list to the RFC body to make it more visible?
@ All: For the moment being I propose we stay in brainstorming mode, i.e. everyone can add their requirements w/o major discussions/rejections. Once we have a list, we can still prioritize and tweak it as a group.

UR1: As a user I want to be able to set a configuration which is used during build time.

Configuring a buffer size (single and multi instance)
Configuring an I2C address for a port expander (single and multi instance)

UR2: As a user I want to be able to set different configurations for multiple instances of the same type.

Configuring different network IP addresses for multiple network devices
Requires referring from config to HW/peripheral instances in DT (1:1, n:1 and n:m)

UR3: As a user I want to be able to reuse my existing knowledge of Zephyr.

Devicetree syntax macros, and data model
YAML syntax
Bindings syntax  and features (support C types, inheritance, child bindings, allow/block, ...)
KConfig
Settings subsystem
snippets

UR4: As a user I want to have a minimal flash, RAM and runtime CPU overhead for configurations which are used during build time, provisioning and runtime.

initialization dependency graph - including overrides at all levels
build: zero footprint by default (i.e. only the size of data that would have to be allocated anyway)
provisioning: option to optimize for read-only (data compression), only store what is used, minimal performance overhead for metadata storage (no strings) and manipulations
provisioning/runtime: option to optimize for simplicity, e.g. existing r/w-settings backends
minimal flash and runtime overhead for configurations which are set during runtime, i.e. override the ones from build time'
usability, self-describing data vs. size: granular key/value pairs vs. one key + opaque struct per subsys?

UR5 -a: As a user I want to be able to override the configuration from build time during provisioning and runtime.

Setting default network settings, which are then overriden with settings from an SD card or from a separate .hex built and merged/flashed individually for each device at the end of the production line
Selective (per-application) provisioning e.g. sensing, bluetooth and bluetooth mesh setups over build defaults.
Overriding the gain of an ADC channel at runtime.

UR5-b: As a user I want to be able to override build (and possibly provisioning) configuration at all levels.

hardware (boards, shields)
custom drivers or subsystem extensions (overlays, composition)
application (overlays)
snippets

UR5-c: As a user I want to be able to easily extend, remove and or replace (parts of) the Kconfig, DT and config data model w/o having to fork zehyr.

custom out-of-tree application, driver and hardware development
custom hardware extensions
composition, inheritance and overriding of zephyr types with custom (partial) bindings
custom application-specific configuration model

UR6: As a user I want to be able to manually and persistently change configurations during runtime.

Setting a different network IP address, based on some application logic that will be remembered on reboot.

UR7: now a sub-item of UR4.

UR8: As a user I do not want to have to specify the initialization order of entities manually by default, but I want to be able to override them at any level (driver, subsystem, application, ...).

Automatically deriving the initialization priorities for instance from dependencies in the device tree.
build time only - i.e. definitely zero additional config footprint wanted here
config properties with early availability need to be supported in provisioning infrastructure (e.g. network flags at POST_KERNEL time with proper default/automatic dependency setup)

UR9: As a user I want to automatically generate configurations from other systems, which are then equally treated to manual settings.

Generate build time configuration or runtime provisioning from a custom database.
Very simple dev experience by default but extensible for power users (e.g. mass provisioning) - convention over configuration.

UR10: As a user I want to have an easy to understand interface for all configurations.

Especially targeted for beginners
automatically generated documentation that is always 100% up-to-date
tooling that allows me to inspect or even update configuration interactively (see Kconfig, DTShell, ...)
a smooth unboxing experience that leads first-time-users through initially required steps
single and multi instance configuration of the same object type in one place
no duplicate definition for build, provisioning and runtime of the same setting in different places (e.g. Kconfig vs config)
reference errors, e.g. between config and DT, and typing errors (e.g. between config image and settings subsys) discovered at build time rather than at runtime (integration test or production)
integrated task-centric model docs (consistent naming, global namespace) with well-defined mappings between representations of the same entities in DT and config (e.g. peripherals are the same for driver instantiation in DT and net config)

UR11: As a user I want to have an easy migration path for an already existing Zephyr application to a new configuration system.

long term backwards compat for properties that have been migrated away from Kconfig or DT

UR12: As a user I want to be able to sign a firmware image or provide other security material safely.

use it in combination with MCUboot
device-specific private or public keys for protocol encrypytion (e.g. IEEE 802.15.4, Bluetooth, Thread onboarding, ...)

UR13: As a user I want to ensure that configuration during runtime has not been tampered with.

When already using a signed and/or encrypted image, also the provisioning source should be verifiable and/or encrypted.

CR1: As a contributor I want to support a minimum amount of input file formats.

Avoid multiple possible inputs at the same time like Kconfig, devicetree, yaml, Json, CBOR, ... whatever you can think of

CR2: As a contributor I want to have an easy and clear decision process where a new configuration should be located.

Should this configuration be in the devicetree, config or Kconfig?
In which folder should it be placed?
Rules that can ideally be (partially) automated or at least are easy to agree upon objectively, e.g. by basing them on usage sites (arch/boards vs drivers vs subsystem), data normalization, encapsulation or section targets (text vs data).
What about subsystem thread priority, a subsystem thread stack size, logging, ...?
What about hardware offloading of e.g. MAC features by certain SoCs only while otherwise configured in the subsys - DT or config?

CR3: As a contributor I want automatically generated documentation for all available settings.

CR4: As a contributor I want a productive development environment (top developer experience).

Support in-memory provisioning and runtime override of the flash simulator for QEMU testing and development
Configuring tests at build time (e.g. net config)
Debugging and Development with minimal extra work and dependencies incl. easy switching between dev mode (build only) and production mode (build + provisioning).
Optionally trading size for subsystem maintenance effort, e.g. by only supporting the setting subsys rather than the macro target.
Re-using as much existing code as possible, avoid duplication (e.g. macros, several subsystem-facing config interfaces), re-inventing wheels (e.g. bindings) or inconsistencies (same entity modeled inconsistently in different sources)
Easy extension of existing tooling, e.g. DTShell -> ConfigShell, Nordic's VS-Code integration, ...

CR5: As a contributor I want an easy and backwards compatible migration path between Kconfig, devicetree and config

Include Kconfig or DT properties in the set of properties that we want to provision or change at runtime, e.g. CDC ECM MAC addresses.
Promote from single to multi-instance.
Support long-term backwards compat for migrated props with minimal maintenance effort.
Keep DT as Linux/U-boot/QEMU-compatible as possible - e.g. by being able to extract a fully compatible partial tree on demand.

@nashif Thanks for visualizing some aspects of the reqs. Questions/comments:

Could you elaborate how a shared/generic config integration API (the black box) could look like? As I'm tinkering around with this quite some time already across subsystems, it seems to me that subsystems will require individual integration approaches, i.e. access build and settings config directly. This also takes into account what people said in this discussion about subsystem specific requirements (build only vs. mix vs. settings only).
I think most of the sharing across subsystems will take place in the tooling layer rather than in the application.
I think build-time access to Kconfig/DT is missing in your data flow graph for validation and integration (e.g. to validate/establish struct device pointers per config instance, for backwards compatibility or to enforce and consistently document the global namespace).
We need separate data flows for OTA and end-of-line provisioning as they'll not be executed at the same time nor usually by the same people.
I think shell and web act like subsystems. They are probably interacting with the settings subsys directly.

As I cannot edit the picture source directly, could you consider the following changes?

Replacing the black layer in favor of individual links from the subsystem-specific config blocks to one or both of build and/or settings config.
Illustrating how large parts of build and provisioning tooling could be integrated for all use cases (build config, OTA, provisioning). I know that for sure because I have a working poc on that (for both, build and provisioning time tooling).
Add a data flow from Kconfig and DT to the config tooling (not runtime!) for verification and link establishment.
Add an additional "provisioning" box.
Move Shell and Web above the Settings subsys and treat them like other subsystems (with read-only links to build config and r/w links to settings).

@nashif Actually it's easy to redraw that. I'm going to post a proposal myself. Sorry for the noise.

UR1: As a user I want to be able to set a configuration which is used during build time.

Configuring a buffer size
Configuring an I2C address for a port expander

You can already do that using both Kconfig and DTS, and you do not need this new "configuration system" to accomplish this. what am I missing?

Absolutely correct, we already have. But it is a requirement which is part of the realm we are trying to change respectively work on, and therefore we should consider it. I would classify it as a very high priority with small effort to implement requirement.

UR3: As a user I want to be able to reuse my existing knowledge of Zephyr.

Devicetree syntax
Bindings
KConfig

You forget to mention that Yaml is something else widely used in Zephyr

I probably should have added a ... at the end, but you are definitely right. YAML is another noteworthy example, @fgrandel added it already to his version of the list.

UR8: As a user I do not want to have to specify the initialization order of entities manually.

Automatically deriving the initialization priorities for instance from dependencies in the device tree.

not sure how this is related, what order are you referring to?

This came from https://github.com/zephyrproject-rtos/zephyr/pull/73836. In general, these requirements are not necessarily mine, I just tried to collect them and consider all inputs.

UR12: As a user I want to be able to sign a firmware image.

use it in combination with MCUboot

missing how is this related.

This one came from here: https://github.com/zephyrproject-rtos/zephyr/issues/77638#issuecomment-2312987378

@fgrandel I intentionally tried to keep the requirements small and short, as I think it makes them easier to digest and work with. For this reason I have for instance tried to keep runtime and build time separate. Basically, sharply focused and language wise sticking to a short as possible sentence.
This would be my approach on phrasing the requirements, updated by you:

UR1: As a user I want to be able to set a configuration which is used during build time.

Configuring a buffer size
Configuring an I2C address for a port expander

UR2: As a user I want to be able to override the configuration from build time during provisioning and runtime.

Setting default network settings, which are then overriden with settings from an SD card or from a separate .hex built and merged/flashed individually for each device at the end of the production line
Selective (per-application) provisioning e.g. sensing, bluetooth and bluetooth mesh setups over build defaults.
Overriding the gain of an ADC channel at runtime.

UR3: As a user I want to be able to set different configurations for multiple instances of the same type.

Configuring different network IP addresses for multiple network devices
Configuring a different buffer size for each instance of an entity type
Configuring a different I2C address for each instance of a port expander

UR4: As a user I want to be able to refer to hardware instances from more software concentric configuration items.

Requires referring from config to HW/peripheral instances in DT (1:1, n:1 and n:m)

UR5: As a user I want to be able to reuse my existing knowledge of Zephyr.

Devicetree syntax macros, and data model
YAML syntax
Bindings syntax and features (support C types, inheritance, child bindings, allow/block, machine-readable docs, ...)
KConfig
Settings subsystem
snippets

UR6: As a user I want to have a minimal flash, RAM and runtime CPU overhead for configurations which are used during build time, provisioning and runtime.

build: zero footprint by default (i.e. only the size of data that would have to be allocated anyway)
provisioning: option to optimize for read-only (data compression), only store what is used, minimal performance overhead for metadata storage (no strings) and manipulations
provisioning/runtime: option to optimize for simplicity, e.g. existing r/w-settings backends
minimal flash and runtime overhead for configurations which are set during runtime, i.e. override the ones from build time'
usability, self-describing data vs. size: granular key/value pairs vs. one key + opaque struct per subsys?

UR7: As a user I want to be able to easily extend, remove and or replace parts of the config data model w/o having to fork Zephyr.

custom out-of-tree application, driver and hardware development
custom hardware extensions
composition, inheritance and overriding of zephyr types with custom (partial) bindings
custom application-specific configuration model

UR8: As a user I want to be able to manually and persistently change configurations during runtime.

Setting a different network IP address, based on some application logic that will be remembered on reboot.
Being able to override build (and possibly provisioning) configuration at all levels.

UR9: As a user I do not want to have to specify the initialization order of entities manually by default.

Automatically deriving the initialization priorities for instance from dependencies in the device tree.
build time only - i.e. definitely zero additional config footprint wanted here
config properties with early availability need to be supported in provisioning infrastructure (e.g. network flags at POST_KERNEL time with proper default/automatic dependency setup)

UR10: As a user I want to be able to override the automatically generated initialization order of entities at any level.

initialization dependency graph - including overrides at all levels

UR11: As a user I want to automatically generate configurations from other systems, which are then equally treated to manual settings.

Generate build time configuration or runtime provisioning from a custom database.
Very simple dev experience by default but extensible for power users (e.g. mass provisioning) - convention over configuration.

UR12: As a user I want to have an easy to understand interface for all configurations.

Especially targeted for beginners
automatically generated documentation that is always 100% up-to-date
tooling that allows me to inspect or even update configuration interactively (see Kconfig, DTShell, ...)
a smooth unboxing experience that leads first-time-users through initially required steps
single and multi instance configuration of the same object type in one place
no duplicate definition for build, provisioning and runtime of the same setting in different places (e.g. Kconfig vs config)
reference errors, e.g. between config and DT, and typing errors (e.g. between config image and settings subsys) discovered at build time rather than at runtime (integration test or production)
integrated task-centric model docs (consistent naming, global namespace) with well-defined mappings between representations of the same entities in DT and config (e.g. peripherals are the same for driver instantiation in DT and net config)

UR13: As a user I want to have an easy migration path for an already existing Zephyr application to a new configuration system.

long term backwards compat for properties that have been migrated away from Kconfig or DT

UR14: As a user I want to be able to sign a firmware image or provide other security material safely.

use it in combination with MCUboot
device-specific private or public keys for protocol encrypytion (e.g. IEEE 802.15.4, Bluetooth, Thread onboarding, ...)

UR15: As a user I want to ensure that configuration during runtime has not been tampered with.

When already using a signed and/or encrypted image, also the provisioning source should be verifiable and/or encrypted.

CR1: As a contributor I want to support a minimum amount of input file formats.

Avoid multiple possible inputs at the same time like Kconfig, devicetree, yaml, Json, CBOR, ... whatever you can think of

CR2: As a contributor I want to have an easy and clear decision process where a new configuration should be located.

Should this configuration be in the devicetree, config or Kconfig?
In which folder should it be placed?
Rules that can ideally be (partially) automated or at least are easy to agree upon objectively, e.g. by basing them on usage sites (arch/boards vs drivers vs subsystem), data normalization, encapsulation or section targets (text vs data).